Unlike jobs which are executed before sending normal messages, MDNs
from `smtp_mdns` table are sent after sending messages from `smtp`
table. This way normal messages have higher priority than MDNs.
There are no SMTP jobs anymore. All jobs are IMAP jobs, so
`jobs.thread` column is not used anymore.
MarkseenMsgOnImap job, that was responsible for marking messages as
seen on IMAP and sending MDNs, has been removed.
Messages waiting to be marked as seen are now stored in a
single-column imap_markseen table consisting of foreign keys pointing
to corresponding imap table records.
Messages are marked as seen in batches in the inbox loop. UIDs are
grouped by folders to reduce the number of requests, including folder
selection requests. UID grouping logic has been factored out of
move_delete_messages into UidGrouper iterator to avoid code duplication.
Messages are marked as seen right before fetching from the inbox
folder. This ensures that even if new messages arrive into inbox while
the connection has another folder selected to mark messages there, all
messages are fetched before going IDLE. Ideally marking messages as
seen should be done after fetching and moving, as it is a low-priority
task, but this requires skipping IDLE if UIDNEXT has advanced since
previous time inbox has been selected. This is outside of the scope of
this change.
MDNs are now queued independently of marking the messages as seen.
SendMdn job is created directly rather than after marking the message
as seen on IMAP. Previously sending MDNs was done in MarkseenMsgOnImap
avoid duplicate MDN sending by setting $MDNSent flag together with
\Seen flag and skipping MDN sending if the flag is already set. This
is not the case anymore as $MDNSent flag support has been removed in
9c077c98cd and duplicate MDN sending in
multi-device case is avoided by synchronizing Seen status since
833e5f46cc as long as the server
supports CONDSTORE extension.
Previously first try used an old connection and. If using stale
connection, an immediate retry was performed using a new connection.
Now if connection is stale, it is closed immediately without trying to
use it.
This should reduce the delay in cases when old connection is unusable.
This makes the contact ID its own newtype instead of being a plain
u32. The change purposefully does not yet try and reap any benefits
from this yet, instead aiming for a boring change that's easy to
review. Only exception is the ToSql/FromSql as not doing that yet
would also have created churn in the database code and it is easier to
go straight for the right solution here.
- Replace .ok_or_else() and .map_err() with anyhow::Context where possible.
- Use .context() to check Option for None when it's an error
- Resultify Chatlist.get_chat_id()
- Add useful .context() to some errors
- IMAP error handling cleanup
`imap` table maps Message-IDs to UIDs on the server. `dc_receive_imf`
no longer gets the UID of the message as an argument and does not
insert the folder and UID of the message into the `msgs`
table. `server_folder` and `server_uid` columns in `msgs` table are
deprecated.
MoveMsg and DeleteMsgOnImap jobs are removed. Now messages are moved
and deleted only in the `fetch_move_delete` procedure that consults
the `target` column of the `imap` table to determine where the message
should go.
Where the message should go is determined after prefetching by the
`imap::target_folder()` procedure. Messages are only downloaded once
they reach their target folder to avoid race conditions in multidevice
setting, such as:
1. One device trying to FETCH the message while the other tries to
MOVE it.
2. One device marking the message as \Seen in the Inbox while the
other has already copied unseen message to the Movebox and is going to
delete the \Seen message in the Inbox.
3. Device downloads the message from the Inbox while there are newer
messages in the Movebox placed there by the other device, thus
processing the messages out of order.
Update gossiped_timestamp when someone else sends autocrypt gossip in
the group, so we postpone sending gossip again ourselves.
- Warn about failures to parse Autocrypt-Gossip header
- Move gossip-related methods into ChatId impl
- Fix a "gossi_pp_ed" typo
* add basic multi-device-sync functions
* generate json
* add context.parse_sync_items()
* add context.execute_sync_items()
* piggyback sync-commands message, add body for human-readable part
* avoid double json renderings
* mimeparser parses incoming .json sync-files
* do not piggyback sync-files
* execute sync items
* return status of send_sync_msg()
* send sync messages as multipart/report
* add a per-item-timestamp and also allow adding other per-item-fields in the future
* if the self-chat does not exist, create it blocked/hidden
* create tokens closer to real qr-code needs
* respect bcc_self setting, add test for that
* sync qr code tokens after promoting groups
* send sync-messages only if an experimental switch is set
* trigger send_sync_msg() after sending messages and after creating/redraw/revive qr-code
* add DC_STR_* constants to deltachat.h
* adapt to refactored qr module as of #2729
* tweak test
* use SendSyncMsgs config name instead of SendExperimentalSyncMsgs - we can remove or rename the config nevertheless, but have the option to keep it without renaming
* tweak docs
* remove currently unused effective timestamp calculation
* clarify when send_sync_msg() is called
* make sure, sync-messages are encrypted and are sent by SELF
* tweak docs, fix typos
The problem was:
When opening the connectivity view while there is no network,
get_connectivity_html() calls schedule_quota_update(), which schedules a
UpdateRecentQuota job. But in update_recent_quota(), connecting fails
and a ConnectivityChanged event is emitted (connectivity changes from
Error to Connecting and back). Therefore the UI calls
get_connectivity_html() again, and the loop is complete.
This made the UI completely unresponsible. To reproduce, just turn wi-fi
off and open the connectivity view.
The fix is:
schedule_quota_update() now only schedules a new job if there is no old
job. To prevent the possible (though probably unlikely) problem that an
old quota update job has a backoff of, like, a day and therefore quota
is not updated, I reduced the backoff time for quota jobs to 10s.
Fixes possibly https://github.com/deltachat/deltachat-android/issues/2043, but we should "re-try" this
* draft a download-api
* basic implementation
* allow partial downloads for protected chats
* use a separate column for download_state
* force a minimal timeout for delete_server_after in combination with partial messages
* add a warning if a possible download may expire by delete_server_after
* test load_imap_deletion_msgid()
* add a test for a partial download
* improve documentation and visibility
* let get_download_limit() return Result<Option>
* rusty getters
* apply MIN_DELETE_SERVER_AFTER to shown availability time
* move stub-creation to download.rs, use stock-strings, nicer logging
* make clippy happy (cargo clippy --tests)
* refine tests and comments
* fix typo
* remove superfluous closure in ffi
* respect partial_download for immediately scheduled DeleteMsgOnImap jobs
* resultify update_recent_quota()
* add a device-message if quota exceeds QUOTA_WARN_THRESHOLD_PERCENTAGE
* check if a quota warning should be added during housekeeping, this is at least once a day
* dc_get_config("quota_exceeding") is useful for bots
* make clippy happy
* reword QuotaExceedingMsgBody message
* avoid lots of warnings if quota jitters around the warning threshold
* allow clippy::assertions_on_constants
these constants depend on each other, it makes sense to check that they are not changed in an incompatible way
* add imap::get_quota_roots()
* schedule quote-checking job on getting connectivity-html
* get quota and debug print it
* basic quota output
* update quota at most once per minute, emit event on changes
* use more meaningful names
* add some comments, move update_recent_quota() to quota.rs
* show root name only if there are several roots
* make clippy happy, some refactorings
* allow only one update-quota job per time
* add now supported QUOTA to standards.md
fix#2539
It's always a bit ambiguous which function should do set_err or set last_send_error, I used this rule here:
If some code returns Status::RetryLater, then it sets last_send_error, because Status::RetryLater means that the user won't see the error directly on the message (if we returned Status::Failed(Err(_)), then the message would be shown as failed to the user) Also, smtp_send always sets last_send_error because, well, sending just failed or succeeded.
Also, remove unused field pending_error.
Contact request chats are not merged into a single virtual "deaddrop"
chat anymore. Instead, they are shown in the chatlist the same way as
other chats, but sending of messages to them is not allowed and MDNs
are not sent automatically until the chat is "accepted" by the user.
New API:
- dc_chat_is_contact_request(): returns true if chat is a contact
request. In this case option to accept and block the chat via
dc_accept_chat() and dc_block_chat() should be shown in the UI.
- dc_accept_chat(): accept contact request and unblock the chat
- dc_block_chat(): decline contact request and block the chat
Removed API:
- dc_create_chat_by_msg_id(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_marknoticed_contact(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_decide_on_contact_request(): this call requires a message ID from
deaddrop chat as input. As deaddrop chat is removed, this call can't
be used anymore.
- dc_msg_get_real_chat_id(): use dc_msg_get_chat_id() instead, the
only difference between these calls was in handling of deaddrop chat
- removed DC_CHAT_ID_DEADDROP and DC_STR_DEADDROP constants
See https://support.delta.chat/t/discussion-how-to-show-error-states/1363/10 <!-- comment -->
It turns out that it's pretty easy to distinguish between lots of states (currently Error/NotConnected, Connecting…, Getting new messages… and Connected). What's not that easy is distinguishing between an actual error and no network, because if the server just doesn't respond, it could mean that we don't have network or that we are trying ipv6, but only ipv4 works.
**WRT debouncing:**
Sending of EVENT_CONNECTIVITY_CHANGED is not debounced, but emitted every time one of the 3 threads (Inbox, Mvbox and Sentbox) has a network error, starts fetching data, or is done fetching data.
This means that it is emitted:
- 9 times when dc_maybe_network() is called or we get network connection
- 12 times when we lose network connection
Some measurements: dc_get_connectivity() takes a little more than 1ms (in my measurements back in March), dc_get_connectivity_html() takes 10-20ms. This means that it's no immmediate problem to call them very often, might increase battery drain though. For the UI it may be a lot of work to update the title everytime; at least Android is smart enough to update the title only once.
Possible problems (we don't have to worry about them now I think):
- Due to the scan_folders feature, if the user has lots of folders, the state could be "Connecting..." for quite a long time, generally DC seemed a little unresponsive to me because it took so long for "Connecting..." to go away. Telegram has a state "Updating..." that sometimes comes after "Connecting...".
To be done in other PRs:
- Better handle the case that the password was changed on the server and authenticating fails, see https://github.com/deltachat/deltachat-core-rust/issues/1923 and https://github.com/deltachat/deltachat-core-rust/issues/1768
- maybe event debouncing (except for "Connected" connectivity events)
fix https://github.com/deltachat/deltachat-android/issues/1760
`Imap` structure is always created in a configured state now. There is
no default value for `ImapConfig` anymore.
Also resultify Scheduler::start() to fail on database errors, for
example if IMAP configuration cannot be read from the database during
`start_io()`. Previosuly errors during reading keys such as
`mvbox_watch` were simply ignored and folders were not watched until
the application is completely restarted, now start_io() will fail and
scheduler will only be started at the next start_io() call which
usually happens when app is brought to the foreground.
Using `impl AsRef<str>` as the argument instead of `&str` makes it
possible to call the function with `&str`, `String` and other types
that implement `AsRef` trait.
The cost of it is that compiled binary contains mulitple versions of
the same function, one for each variant of types. If function contains
multiple generic `impl AsRef` arguments, the number of versions possibly
compiled into binary grows exponentially with the number of arguments.
Simple way to avoid it is to call `.as_ref()` on the caller side to
convert the argument to `&str`. In most cases even adding a `&` and
relying on `Deref` coercion is sufficient.
This patch changes many functions that accepted `impl AsRef<str>` and
`impl AsRef<Path>` to accept `&str` and `&Path` instead.
In some places `.clone()` calls are removed. Calling `.clone()` on
`String` and passing `String` to a function accepting `impl
AsRef<str>` is completely unnecessary as `&str` reference could be
passed instead. There is no clippy warning against it yet, but
changing argument type to `&str` allowed to find these cases.
The result of debloating is not impressive, several hundred kilobytes
are saved, which is about 3% of the `.so` binary, but the code is
cleaner too.
This moves the module-level lookup and creation functions to the
types, which make the naming more consistent. Now the lookup_* get_*
and create_* functions all behave similarly.
Peraps even more important the API of the lookup now allows
distinguishing failure from not found. This in turn is important to
be able to remove reliance on a ChatId with a 0 or "unset" value. The
locations where this ChatId(0) is still used is in database queries
which should be solved in an independed commit.
* Remove sql::error submodule
Use anyhow errors instead.
* Remove explicit checks for open SQL connection
An error will be thrown anyway during attempt to execute query.
* Don't use `with_conn()` and remove it
* Remove unused `with_conn_async`
* Resultify markseen_msgs
