There are at least two user reports that fetching existing messages
sometimes results in infinite loop of retrying it. Account is working
if set up from the backup, but never starts working if set up
from scratch.
This change improves error reporting, but also sets FetchedExistingMsgs
before actually trying to do it. This way if the operation fails,
connection is reestablished, but fetching existing messages is not
retried again over and over.
IMAP capabilities and selected folder are IMAP session,
not IMAP client property.
Moving most operations into IMAP session structure
removes the need to constantly check whether IMAP session exists
and reduces number of invalid states, e.g. when a folder is selected but
there is no connection.
Capabilities are determined immediately after logging in,
so there is no need for `capabilities_determined` flag anymore.
Capabilities of the server are always known if there is a session.
`should_reconnect` flag and `disconnect()` function are removed: we
drop the session on error. Even though RFC 3501 says that a client
SHOULD NOT close the connection without a LOGOUT, it is more reliable
to always just drop the connection, especially after an error.
New ratelimiter module counts number of sent messages and calculates
the time until more messages can be sent.
Rate limiter is currently applied only to MDNs. Other messages are
sent without rate limiting even if quota is exceeded, but MDNs are not
sent until ratelimiter allows sending again.
Scheduler has no Stopped state anymore. If Scheduler exists, it is
always started. Scheduler is stopped via Scheduler.stop(), which
consumes Scheduler and cannot fail.
Unlike jobs which are executed before sending normal messages, MDNs
from `smtp_mdns` table are sent after sending messages from `smtp`
table. This way normal messages have higher priority than MDNs.
There are no SMTP jobs anymore. All jobs are IMAP jobs, so
`jobs.thread` column is not used anymore.
i was wondering mainly about the whole line
"Failed to start IO: scheduler is already stopped".
i think it has to be "already started" and not "already stopped".
Hold scheduler lock during the whole procedure of scheduler starting
and stopping. This ensures that two processes can't get two read locks
in parallel and start loops or send the stop signal twice.
Also remove shutdown channels: it is enough to wait
for the loop handle without receiving a shutdown signal
from the end of the loop.
The limit on the number of jobs executed in a row was introduced to
prevent large queues of small jobs like MarkseenMsgOnImap, MoveMsg and
DeleteMsgOnImap from delaying message fetching. Since all these jobs
are now removed and IMAP operations they did are now batched, it is
impossible to have 20 or more queued IMAP jobs.
MarkseenMsgOnImap job, that was responsible for marking messages as
seen on IMAP and sending MDNs, has been removed.
Messages waiting to be marked as seen are now stored in a
single-column imap_markseen table consisting of foreign keys pointing
to corresponding imap table records.
Messages are marked as seen in batches in the inbox loop. UIDs are
grouped by folders to reduce the number of requests, including folder
selection requests. UID grouping logic has been factored out of
move_delete_messages into UidGrouper iterator to avoid code duplication.
Messages are marked as seen right before fetching from the inbox
folder. This ensures that even if new messages arrive into inbox while
the connection has another folder selected to mark messages there, all
messages are fetched before going IDLE. Ideally marking messages as
seen should be done after fetching and moving, as it is a low-priority
task, but this requires skipping IDLE if UIDNEXT has advanced since
previous time inbox has been selected. This is outside of the scope of
this change.
MDNs are now queued independently of marking the messages as seen.
SendMdn job is created directly rather than after marking the message
as seen on IMAP. Previously sending MDNs was done in MarkseenMsgOnImap
avoid duplicate MDN sending by setting $MDNSent flag together with
\Seen flag and skipping MDN sending if the flag is already set. This
is not the case anymore as $MDNSent flag support has been removed in
9c077c98cd and duplicate MDN sending in
multi-device case is avoided by synchronizing Seen status since
833e5f46cc as long as the server
supports CONDSTORE extension.
* Do ephemeral deletion in background loop
1. in start_io start ephemeral async task, in stop_io cancel ephemeral async task
2. start ephemeral async task which loops like this:
- wait until next time a message deletion is needed or an interrupt occurs (see 3.)
- perform delete_expired_messages including sending MSGS_CHANGED events
3. on new messages (incoming or outgoing) with ephemeral timer:
- interrupt ephemeral async task
* Changelog
* Fix and improve test
* no return value needed
* address @link2xt review comments
* slight normalization: have only one place where we wait for interrupt_receiver
* simplify sql statement -- and don't exit the ephemeral_task if there is an sql problem but rather wait
* Remove now-unused `ephemeral_task` JoinHandle
The JoinHandle is now inside the Scheduler.
* fix clippy
* Revert accidental move of the line
* Add log
Co-authored-by: holger krekel <holger@merlinux.eu>
Co-authored-by: link2xt <link2xt@testrun.org>
- do not attempt to mark reserved meessages as seen when
messages with empty Message-ID are marked as seen on IMAP
- do not reconnect on Seen flag synchronization failures This avoid
reconnection loops in case of permanent errors in `sync_seen_flags`
* Make set_config() look a bit nicer
* Add OnlyFetchMvbox option
* Add test for the config
* Add option to only watch mvbox
This is supposed to support having a server-side rule which moves
emails to the mvbox already. The new option makes sure the mvbox is
wathched and also makes sure no messages are feched from folders other
than the mvbox and the spam folder if enabled. It does not interact
with the other settings.
* Fixup ignore conditions
* Cleanup some bits
* Watch the mvbox when `WatchMvboxOnly` is set
* Rename back to only_fetch_mvbox (flub said it's OK for him)
* typo
* clippy, more typos
Co-authored-by: Hocuri <hocuri@gmx.de>
Synchronizing seen flags doubles the time required to scan all
folders. Delta Chat only marks messages as Seen on Inbox or DeltaChat,
so there is no need to check for Seen flag on other folders.
This operation takes roughly 0.3 s on a moderate size database.
Calling it once before scanning all folders and scanning
the watched folder instead of each time after downloading
a message from a folder speeds up IMAP loop.
Also resultified `get_watched_folders`.
Python test `test_moved_markseen` is modified to test using incoming
message instead of BCC-self message, because BCC-self message is
detected immediately in the Inbox.
`imap` table maps Message-IDs to UIDs on the server. `dc_receive_imf`
no longer gets the UID of the message as an argument and does not
insert the folder and UID of the message into the `msgs`
table. `server_folder` and `server_uid` columns in `msgs` table are
deprecated.
MoveMsg and DeleteMsgOnImap jobs are removed. Now messages are moved
and deleted only in the `fetch_move_delete` procedure that consults
the `target` column of the `imap` table to determine where the message
should go.
Where the message should go is determined after prefetching by the
`imap::target_folder()` procedure. Messages are only downloaded once
they reach their target folder to avoid race conditions in multidevice
setting, such as:
1. One device trying to FETCH the message while the other tries to
MOVE it.
2. One device marking the message as \Seen in the Inbox while the
other has already copied unseen message to the Movebox and is going to
delete the \Seen message in the Inbox.
3. Device downloads the message from the Inbox while there are newer
messages in the Movebox placed there by the other device, thus
processing the messages out of order.
* In dc_maybe_network_lost() directly set the connectivity "Not connected"
r10s reported that without doing this, the connectivity would stay at
"Connected" for 16 more seconds after network is gone and
dc_maybe_network_lost() was called.
* Set the state for all connections
fix#2539
It's always a bit ambiguous which function should do set_err or set last_send_error, I used this rule here:
If some code returns Status::RetryLater, then it sets last_send_error, because Status::RetryLater means that the user won't see the error directly on the message (if we returned Status::Failed(Err(_)), then the message would be shown as failed to the user) Also, smtp_send always sets last_send_error because, well, sending just failed or succeeded.
Also, remove unused field pending_error.
See https://support.delta.chat/t/discussion-how-to-show-error-states/1363/10 <!-- comment -->
It turns out that it's pretty easy to distinguish between lots of states (currently Error/NotConnected, Connecting…, Getting new messages… and Connected). What's not that easy is distinguishing between an actual error and no network, because if the server just doesn't respond, it could mean that we don't have network or that we are trying ipv6, but only ipv4 works.
**WRT debouncing:**
Sending of EVENT_CONNECTIVITY_CHANGED is not debounced, but emitted every time one of the 3 threads (Inbox, Mvbox and Sentbox) has a network error, starts fetching data, or is done fetching data.
This means that it is emitted:
- 9 times when dc_maybe_network() is called or we get network connection
- 12 times when we lose network connection
Some measurements: dc_get_connectivity() takes a little more than 1ms (in my measurements back in March), dc_get_connectivity_html() takes 10-20ms. This means that it's no immmediate problem to call them very often, might increase battery drain though. For the UI it may be a lot of work to update the title everytime; at least Android is smart enough to update the title only once.
Possible problems (we don't have to worry about them now I think):
- Due to the scan_folders feature, if the user has lots of folders, the state could be "Connecting..." for quite a long time, generally DC seemed a little unresponsive to me because it took so long for "Connecting..." to go away. Telegram has a state "Updating..." that sometimes comes after "Connecting...".
To be done in other PRs:
- Better handle the case that the password was changed on the server and authenticating fails, see https://github.com/deltachat/deltachat-core-rust/issues/1923 and https://github.com/deltachat/deltachat-core-rust/issues/1768
- maybe event debouncing (except for "Connected" connectivity events)
fix https://github.com/deltachat/deltachat-android/issues/1760
`Imap` structure is always created in a configured state now. There is
no default value for `ImapConfig` anymore.
Also resultify Scheduler::start() to fail on database errors, for
example if IMAP configuration cannot be read from the database during
`start_io()`. Previosuly errors during reading keys such as
`mvbox_watch` were simply ignored and folders were not watched until
the application is completely restarted, now start_io() will fail and
scheduler will only be started at the next start_io() call which
usually happens when app is brought to the foreground.
Switches from rusqlite to sqlx to have a fully async based interface
to sqlite.
Co-authored-by: B. Petersen <r10s@b44t.com>
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: link2xt <link2xt@testrun.org>
