Received messages shouldn't mingle with just sent ones and appear somewhere in the middle of the
chat, so we go after the newest non fresh message.
But if a received outgoing message is older than some `InSeen` message, better sort the received
message purely by timestamp (this is an heuristic in order not to break the Gmail-like case
simulated by `verified_chats::test_old_message_4()`). We could place the received message just
before that `InSeen` message, but anyway the user may not notice it.
At least this fixes outgoing messages sorting for shared accounts where messages from other devices
should be sorted the same way as incoming ones.
Otherwise backups exported from the current core and imported in versions < 1.144.0 have QR codes
not working. The breaking change which removed the column is
5a6efdff44.
Before file extensions were also limited to 32 chars, but extra chars in the beginning were just cut
off, e.g. "file.with_lots_of_characters_behind_point_and_double_ending.tar.gz" was considered to
have an extension "d_point_and_double_ending.tar.gz". Better to take only "tar.gz" then.
Also don't include whitespace-containing parts in extensions. File extensions generally don't
contain whitespaces.
This should fix ad-hoc groups splitting when messages are fetched out of order from different
folders or otherwise reordered, or some messages are missing so that the messages reference chain is
broken, or a member was removed from the thread and readded later, etc. Even if this way two
different threads are merged, it looks acceptable, having many threads with the same name/subject
and members isn't a common use case.
Why:
- With IMAP APPEND we can upload messages directly to the DeltaChat folder (for non-chatmail
accounts).
- We can set the `\Seen` flag immediately so that if the user has other MUA, it doesn't alert about
a new message if it's just a sync message (there were several such reports on the support
forum). Though this also isn't useful for chatmail.
- We don't need SMTP envelope and overall remove some overhead on processing sync messages.
Groups promotion to other devices and QR code tokens synchronisation are not synchronised processes,
so there are reasons why a QR code token may arrive earlier than the first group message:
- We are going to upload sync messages via IMAP while group messages are sent by SMTP.
- If sync messages go to the mvbox, they can be fetched earlier than group messages from Inbox.
`chat::create_send_msg_jobs()` already handles `Config::BccSelf` as needed. The only exception is
Autocrypt setup messages. This change unifies the logic for the self-chat and groups only containing
`SELF`.
This makes possible to schedule one more sending of the message, the existing jobs are not
cancelled. Otherwise it's complicated to implement bots that resend messages when a new member joins
the group.
And also:
- Make it `pub(crate)`.
- Use it in `should_request_mdns()` as using `config_exists()` there isn't correct because the
latter doesn't look at environment.
I.e. to all messages except "v{c,g}-request" as they sent out on a QR code scanning which is a
manual action and "vg-member-added" as formally this message is auto-submitted, but the member
addition is a result of an explicit user action. Otherwise it would be strange to have the
Auto-Submitted header in "member-added" messages of verified groups only.
`imap::Session::store_seen_flags_on_imap()` handles messages from multiple folders, so not being
able to select one folder mustn't fail the whole function.
- To avoid receiving undecryptable MDNs by bots and replying to them if the bot's key changes.
- MDNs from bots don't look useful in general, usually the user expects some reply from the bot, not
just that the message is read.
Recently there are many questions on the Delta Chat forum why some unexpected encrypted messages
appear in Inbox. Seems they are mainly sync messages, though that also obviously happens to
SecureJoin messages. Anyway, regardless of the `MvboxMove` setting, auto-generated outgoing messages
should be moved to the DeltaChat folder so as not to complicate co-using Delta Chat with other MUAs.
This way we can't get an account with missing blobs if there's not enough disk space.
Also delete already unpacked files if all files weren't unpacked successfully. Still, there are some
minor problems remaining:
- If a db wasn't imported successfully, unpacked blobs aren't deleted because we don't know at which
step the import failed and whether the db will reference the blobs after restart.
- If `delete_and_reset_all_device_msgs()` fails, the whole `import_backup()` fails also, but after a
restart delete_and_reset_all_device_msgs() isn't retried. Probably errors from it should be
ignored at all.
There are many reasons why we may fail to find valid signatures in a message, e.g. we don't yet know
a public key attached in the same message, anyway, if From is forged, the message must be rejected.
Also always take the displayname from encrypted From, even if no valid signatures are found.
This is a hint for apps that a WebXDC icon should be shown in the summary, e.g. in the
chatlist. Otherwise it's not clear when it should be shown, e.g. it shouldn't be shown in a reaction
summary.
Follow-up to 3cf78749df "Emit DC_EVENT_MSGS_CHANGED for
DC_CHAT_ID_ARCHIVED_LINK when the number of archived chats with unread messages increases (#3940)".
In general we don't want to make an extra db query to know if a noticied chat is
archived. Emitting events should be cheap, better to allow false-positive `MsgsChanged` events.
`Context::send_sync_msg()` mustn't be called from multiple tasks in parallel to avoid sending the
same sync items twice because sync items are removed from the db only after successful
sending. Let's guarantee this by calling `send_sync_msg()` only from the SMTP loop. Before
`send_sync_msg()` could be called in parallel from the SMTP loop and another task doing
e.g. `chat::sync()` which led to `test_multidevice_sync_chat` being flaky because of events
triggered by duplicated sync messages.
The "Chat-Group-Member-Removed" header is added to ad-hoc group messages as well, so we should check
for its presense before creating an ad-hoc group as we do for DC-style groups.
Before, update sending might be delayed due to rate limits and later merged into large
messages. This is undesirable for apps that want to send large files over WebXDC updates because the
message with aggregated update may be too large for actual sending and hit the provider limit or
require multiple attempts on a flaky SMTP connection.
So, don't aggregate updates if the size of an aggregated update will exceed the limit of 100
KiB. This is a soft limit, so it may be exceeded if a single update is larger and it limits only the
update JSON size, so the message with all envelopes still may be larger. Also the limit may be
exceeded when updates are sent together with the WebXDC instance when resending it as the instance
size isn't accounted to not complicate the code. At least this is not worse than the previous
behaviour when all updates were attached.
Bridge bots like matterdelta need to set a quoted text without referencing the quoted message, this
makes easier bridging messages from other platforms to Delta Chat or even bridging Delta Chat groups
in different accounts where you can not set a quoted message by the message id from another account.
Follow-up to 5fa7cff46. Let's still not send a sync message if the contact wasn't modified. This is
not very important, but just for consistency with the `chat::rename_ex()` behaviour.
This way it's clearer which key is which and also adding the key fingerprint to the file name avoids
overwriting another previously exported key. I think this is better than adding an incremental
number as we do for backups, there's no need to export a key several times to different files.
For purposes of building a message it's better to consider the self-chat as verified. Particularly,
this removes unencrypted name from the "From" header.
If currently there are no multi-device bots, let's disable sync messages for bots at all. Another
option is to auto-disable sync messages when `Config::Bot` is set, so sync messages can be reenabled
if needed. But let's leave this option for the future.
It's possible that when rebasing a PR adding a migration a merge-conflict doesn't occur if another
migration was added in the target branch. Better to have at least runtime checks that the migration
version is correct. Looks like compile-time checks are not possible because Rust doesn't allow to
redefine constants, only vars.
Bot processes are run asynchronously, so we shouldn't send messages to a bot before it's fully
initialised and skipped existing messages for processing, i.e. before DC_EVENT_IMAP_INBOX_IDLE is
emitted.
Looks like it has no sense to send any profile data (From/To names, self-status; self-avatar was
never sent even before) in MDNs, they aren't normal messages and aren't seen in a MUA. Better not to
reveal profile data to the network and even to contacts in MDNs and make them more lightweight.
Follow-up to b771311593. Since that commit names are not revealed in
verified chats, but during verification (i.e. SecureJoin) they are still sent unencrypted. Moreover,
all profile data mustn't be sent even encrypted before the contact verification, i.e. before
"v{c,g}-request-with-auth". That was done for the selfavatar in
304e902fce, now it's done for From/To names and the self-status as
well. Moreover, "v{c,g}-request" and "v{c,g}-auth-required" messages are deleted right after
processing, so other devices won't see the received profile data anyway.
Probably sync messages generated for a not yet configured account are useless because there are no
other devices yet. And even if this is not true, we don't want to depend on the order of setting
`Config::SyncMsgs` and other keys. Also w/o this the Python tests don't work if we start syncing
`Config::MvboxMove` because they don't expect that sync messages are sent while configuring
accounts.
NB: We don't restart IO from the synchronisation code, so `MvboxMove` isn't effective immediately if
`ConfiguredMvboxFolder` is unset, but only after a reconnect to IMAP.
Before, if the user deleted a message too quickly after sending, it was deleted only locally. The
fix is to remember for tombstones that the corresponding message should be deleted on the server
too.
This is a way to prevent redownloading locally deleted messages. Otherwise if a message is deleted
quickly after sending and `bcc_self` is configured, the BCC copy is downloaded and appears as a new
message as it happens for messages sent from another device.
Delta Chat -style groups have names w/o prefixes like "Re: " even if the user is added to an already
existing group, so let's remove prefixes from ad-hoc group names too. Usually it's not very
important that the group is a classic email thread existed before, this info just eats up screen
space. Also this way a group name is likely to preserve if the first message was missed.
SQLite search with `LIKE` is case-insensitive only for ASCII chars. To make it case-insensitive for
all messages, create a new column `msgs.txt_normalized` defaulting to `NULL` (so we do not bump up
the database size in a migration) and storing lowercased/normalized text there when the row is
created/updated. When doing a search, search over `IFNULL(txt_normalized, txt)`.
Before, if `Config::FetchExistingMsgs` is set, existing messages were received with the `InSeen`
state set, but for bots they must be `InFresh` and also `IncomingMsg` events should be emitted for
them so that they are processed by bots as it happens with new messages.
Otherwise it's impossible to remove a member with missing key from a protected group. In the worst
case a removed member will be added back due to the group membership consistency algo.
The "I left the group" message can't be sent to a protected group if some member's key is missing,
in this case we should remain in the group. The problem should be fixed first, then the user may
retry to leave the group.
There was a comment in `fetch_existing_msgs()`: "Bots don't want those messages". If a bot doesn't
want this setting, why enable it? It's disabled by default anyway.
`IsChatmail` is set also by `inbox_fetch_idle()`, but it isn't called during `configure()`. Setting
`IsChatmail` from `inbox_fetch_idle()` is necessary to handle client/server upgrades, but
`IsChatmail` also should be available for the app after configuring an account, e.g. DC Android
needs it to know whether to ask the user to disable battery optimisations.
When doing an AEAP transition, we mustn't just delete the old peerstate as this would break
encryption to it. This is critical for non-verified groups -- if we can't encrypt to the old
address, we can't securely remove it from the group (to add the new one instead).
iequidoo