Connection establishment now happens only in one place in each IMAP loop.
Now all connection establishment happens in one place
and is limited by the ratelimit.
Backoff was removed from fake_idle
as it does not establish connections anymore.
If connection fails, fake_idle will return an error.
We then drop the connection and get back to the beginning of IMAP
loop.
Backoff may be still nice to have to delay retries
in case of constant connection failures
so we don't immediately hit ratelimit if the network is unusable
and returns immediate error on each connection attempt
(e.g. ICMP network unreachable error),
but adding backoff for connection failures is out of scope for this change.
If a Delta Chat message has the Message-ID already existing in the db, but a greater "Date", it's a
resent message that can be deleted. Messages having the same "Date" mustn't be deleted because they
can be already seen messages moved back to INBOX. Also don't delete messages having lesser "Date" to
avoid deleting both messages in a multi-device setting.
Previously the message was removed from `download` table,
but message bubble was stuck in InProgress state.
Now download state is updated by the caller,
so it cannot be accidentally skipped.
Also add a test on downloading a message later. Although it doesn't reproduce #4700 for some reason,
it fails w/o the fix because before a message state was changing to `InSeen` after a full download
which doesn't look correct. The result of a full message download should be such as if it was fully
downloaded initially.
Such a message may be assigned to a wrong chat (e.g. undecipherable group msgs often get assigned to
the 1:1 chat with the sender). Add `DownloadState::Undecipherable` so that messages referencing
undecipherable ones don't go to that wrong chat too. Also do not reply to not fully downloaded
messages. Before `Message.error` was checked for that purpose, but a message can be error for many
reasons.
Message.set_text() and Message.get_text() are modified accordingly
to accept String and return String.
Messages which previously contained None text
are now represented as messages with empty text.
Use Message.set_text("".to_string())
instead of Message.set_text(None).
Moved custom ToSql trait including Send + Sync from lib.rs to sql.rs.
Replaced most params! and paramsv! macro usage with tuples.
Replaced paramsv! and params_iterv! with params_slice!,
because there is no need to construct a vector.
Gmail archives messages marked as `\Deleted` by default if those messages aren't in the Trash. But
if move them to the Trash instead, they will be auto-deleted in 30 days.
get_chat_msgs() function is split into new get_chat_msgs() without flags
and get_chat_msgs_ex() which accepts booleans instead of bitflags.
FFI call dc_get_chat_msgs() is still using bitflags for compatibility.
JSON-RPC calls get_message_ids() and get_message_list_items()
accept booleans instead of bitflags now.
* Because both only make problems with mailing lists, it's easiest to just disable them. If we want, we can make them work properly with mailing lists one day and re-enable them, but this needs some further thoughts.
Part of #3701
* Use load_from_db() in more tests
* clippy
* Changelog
* Downgrade warning to info, improve message
* Use lifetimes instead of cloning
This way no temporary rows are created and it is easier to maintain
because UPDATE statement is right below the INSERT statement,
unlike `merge_messages` function which is easy to forget about.
* save webxdc-updates for not yet downloaded messages, that are probably webxdc instances then
* test webxdc updates received while instance is not yet downloaded
* keep msg_id on downloading messages
keeping msg_id on downloading messages
has the advantage that webxdc updates and other references to the msg_id
can be processed as usual.
if a message expands to multiple msg_id,
the last one is kept,
however, this does not affect webxdc at all.
(alternatives may be to update `msgs_status_updates`
but that seems more complicated and even less elegant,
another alternative would be to use different keys (eg. `rfc274_mid`),
but that also seems not to be much easier and would waste space as well.
also both alternatives would need adaption for other foreign keys)
* update CHANGELOG
* do not emit WebxdcStatusUpdate event in case the message is not yet downloaded
* move DELETE/UPDATE to an transaction
* make merge_msg_id() a little less confusing
* use some webxdc-update-param from placeholder
(the placeholder may be updated,
the just downloaded messages is not)
* more precise function name
* test not directly downloading status updates
* test not directly downloading mdn
Message-IDs are now retrieved only during fetching and saved into imap
table. dc_receive_imf_inner does not attempt to extract the Message-ID
anymore.
For messages without Message-ID the ID is now generated in
imap::fetch_new_messages rather than dc_receive_imf_inner,
so the same ID is used in the imap table (maintained by the imap
module) and msgs table (maintained by dc_receive_imf module).
Message-ID generation based on the Date, From and To field hashing has
been replaced with a simple dc_create_id() to avoid retrieving Date,
From, and To fields in the imap module, as it's hard to test that it
stays compatible between Delta Chat versions in this module. This
breaks jump-to-quote for quoted messages without Message-ID, which is
not critical.
Also prefetch X-Microsoft-Original-Message-ID, so retrieval of
duplicate messages with X-Microsoft-Original-Message-ID can be skipped
like it is done for messages with Message-ID header.
- Replace .ok_or_else() and .map_err() with anyhow::Context where possible.
- Use .context() to check Option for None when it's an error
- Resultify Chatlist.get_chat_id()
- Add useful .context() to some errors
- IMAP error handling cleanup
`imap` table maps Message-IDs to UIDs on the server. `dc_receive_imf`
no longer gets the UID of the message as an argument and does not
insert the folder and UID of the message into the `msgs`
table. `server_folder` and `server_uid` columns in `msgs` table are
deprecated.
MoveMsg and DeleteMsgOnImap jobs are removed. Now messages are moved
and deleted only in the `fetch_move_delete` procedure that consults
the `target` column of the `imap` table to determine where the message
should go.
Where the message should go is determined after prefetching by the
`imap::target_folder()` procedure. Messages are only downloaded once
they reach their target folder to avoid race conditions in multidevice
setting, such as:
1. One device trying to FETCH the message while the other tries to
MOVE it.
2. One device marking the message as \Seen in the Inbox while the
other has already copied unseen message to the Movebox and is going to
delete the \Seen message in the Inbox.
3. Device downloads the message from the Inbox while there are newer
messages in the Movebox placed there by the other device, thus
processing the messages out of order.
When there is an outgoing message in a chat, mark all older messages in this chat as seen.
Android already has a similar behavior, however, this led to the issue https://github.com/deltachat/deltachat-android/issues/2163 and should be changed back.
--
From the issue description at https://github.com/deltachat/deltachat-android/issues/2163, I implemented these fixes:
> Core should take care that if the last message in a chat is not fresh|noticed, no messages in the chat can be fresh. [...] Do this [...] in a function that's called at the end of fetch_new_messages(). Then dc_receive_imf() wouldn't get slower by this and we could re-use the same function for migration.
So, I didn't do this inside `dc_receive_imf()` in order not to make it take even longer. This obviously has the downside of higher complexity.
And I think we should implement this:
> On Androd, show the unread badge when unread!=0 again (see deltachat/deltachat-android@618af02). Then the user can see that there is a chat with an unread message and click it to get rid of it.
because it shouldn't be the UI's job to decide whether an unread badge is shown, but the core's.
* draft a download-api
* basic implementation
* allow partial downloads for protected chats
* use a separate column for download_state
* force a minimal timeout for delete_server_after in combination with partial messages
* add a warning if a possible download may expire by delete_server_after
* test load_imap_deletion_msgid()
* add a test for a partial download
* improve documentation and visibility
* let get_download_limit() return Result<Option>
* rusty getters
* apply MIN_DELETE_SERVER_AFTER to shown availability time
* move stub-creation to download.rs, use stock-strings, nicer logging
* make clippy happy (cargo clippy --tests)
* refine tests and comments
* fix typo
* remove superfluous closure in ffi
* respect partial_download for immediately scheduled DeleteMsgOnImap jobs
