Otherwise, e.g. if a message is a large GIF, but its viewtype is set to `Image` by the app, this GIF
will be recoded to JPEG to reduce its size. GIFs and other special viewtypes must be always detected
and sent as is.
... and fails if file already exists. The UI should open the file saving dialog, defaulting to
Downloads and original filename, when asked to save the file. After confirmation it should call
dc_msg_save_file().
This way we avoid an immediate retry if the network is not yet ready exhausting the ratelimiter's
quota of two connection attempts. Also notify the ratelimiter only after a successful connection so
that it only limits the server load, but not connection attempts.
As trashed messages can't be loaded from the db now by `Message::load_from_db()` returning an error
for such messages, errors from `Message::load_from_db()` should be logged as warnings. If it's
really an error like a db failure, it should be logged internally.
As trashed messages can't be loaded from the db now, `get_message_by_id()` returns None in some
tests e.g. in `test_deleted_msgs_dont_reappear()`. A `PerAccount` hook shouldn't be called if so.
There's no need to load an updated message state from the db to implement `is_outgoing()` and also
this function is implicitly called in some tests where a message is already trashed and a call to
`dc_get_msg()` generates an unexpected error.
This fixes things for Gmail f.e. Before, `Imap::fetch_move_delete()` was called before looking for
Trash and returned an error because of that failing the whole `fetch_idle()` which prevented
configuring Trash in turn.
If `delete_device_after` is configured, that period should be counted for webxdc instances from the
last status update, otherwise nothing prevents from deleting them. Use `msgs.timestamp_rcvd` to
store the last status update timestamp, it anyway isn't used for anything except displaying a
detailed message info. Also, as `ephemeral::select_expired_messages()` now also checks
`timestamp_rcvd`, we have an improvement that a message is guaranteed not to be deleted for the
`delete_device_after` period since its receipt. Before only the sort timestamp was checked which is
derived from the "sent" timestamp.
Let's add a 1-minute tolerance to `Params::MemberListTimestamp`.
This adds to the group membership consistency algo the following properties:
- If remote group membership changes were made by two members in parallel, both of them are applied,
no matter in which order the messages are received.
- If we remove a member locally, only explicit remote member additions/removals made in parallel are
allowed, but not the synchronisation of the member list from "To". Before, if somebody managed to
reply earlier than receiving our removal of a member, we added it back which doesn't look good.
- Always emit `ContactsChanged` from `contact::update_last_seen()` if a contact was seen recently
just for simplicity and symmetry with `RecentlySeenLoop::run()` which also may emit several events
for single contact.
- Fix sleep time calculation in `RecentlySeenLoop::run()` -- `now` must be updated on every
iteration, before the initial value was used every time which led to progressively long sleeps.
Instead, look up the 1:1 chat in `receive_imf::add_parts()`. This is a more generic approach to fix
assigning outgoing reactions to 1:1 chats in the multi-device setup. Although currently both
approaches give the same result, this way we can even implement a "react privately"
functionality. Maybe it sounds useless, but it seems better to have less reaction-specific code.
Don't attach selfavatar in "v{c,g}-request" and "v{c,g}-auth-required" messages:
- These messages are deleted right after processing, so other devices won't see the avatar.
- It's also good for privacy because the contact isn't yet verified and these messages are auto-sent
unlike usual unencrypted messages.
Add white background instead of the default black one to avatars when recoding to JPEG. But not for
"usual" images to spare the CPU. The motivation is to handle correctly
"black-on-transparent-background" avatars which are quite common.
`Param::MemberListTimestamp` was updated only from `receive_imf::apply_group_changes()` i.e. for
received messages. If we sent a message, that timestamp wasn't updated, so remote group membership
changes always overrode local ones. Especially that was a problem when a message is sent offline so
that it doesn't incorporate recent group membership changes.
An unencrypted message with already known Autocrypt key, but sent from another address, means that
it's rather a new contact sharing the same key than the existing one changed its address, otherwise
it would already have our key to encrypt.
If Subject is multiline-formatted, `mailparse` adds the leading whitespace to it. The solution is to
always remove the leading whitespace, because if Subject isn't multiline-formatted, it never
contains the leading whitespace anyway. But as for the trailing whitespace -- i checked -- it's
never removed, so let's keep this as is.
This fixes the following identity-misbinding attack:
It appears that Bob’s messages in the SecureJoin protocol do not properly “bind” to Alice’s public
key or fingerprint. Even though Bob’s messages carry Alice’s public key and address as a gossip in
the protected payload, Alice does not reject the message if the gossiped key is different from her
own key. As a result, Mallory could perform an identity-misbinding attack. If Mallory obtained
Alice’s QR invite code, she could change her own QR code to contain the same tokens as in Alice’s QR
code, and convince Bob to scan the modified QR code, possibly as an insider attacker. Mallory would
forward messages from Bob to Alice and craft appropriate responses for Bob on his own. In the end,
Bob would believe he is talking to Mallory, but Alice would believe she is talking to Bob.
If a message is sent from SELF, but signed with a foreign key, it mustn't be considered
Autocrypt-encrypted and shown with a padlock. Currently this is broken.
The way it was implemented it threw out all remaining messages after finding the next incoming
message. Better use FFIEventTracker functions, they are used in all the tests anyway.
Before group avatar was sent as an attachment. Let's do the same as with user avatar and send group
avatar as base64. Receiver code uses the same functions for user and chat avatars, so base64 avatars
are supported for most receivers already.
It was broken completely and before "fix: apply Autocrypt headers if timestamp is unchanged" that
didn't show up because the message from the second Bob's device never had "Date" greater than one
from the message sent before from the first device.
Use sync messages for that as it is done for e.g. Config::Displayname. Maybe we need to remove
self-status synchronisation via usual messages then, but let's think of it a bit.
Use sync messages for that as it is done for e.g. Config::Displayname. Maybe we need to remove
avatar synchronisation via usual messages then, but let's think of it a bit.
Even if `vc-request-with-auth` is received with a delay, the protection message must have the sort
timestamp equal to the Sent timestamp of `vc-request-with-auth`, otherwise subsequent chat messages
would also have greater sort timestamps and while it doesn't affect the chat itself (because Sent
timestamps are shown to a user), it affects the chat position in the chatlist because chats there
are sorted by sort timestamps of the last messages, so the user sees chats sorted out of
order. That's what happened in #5088 where a user restores the backup made before setting up a
verified chat with their contact and fetches new messages, including `vc-request-with-auth` and also
messages from other chats, after that.
Add a new crate `deltachat_time` with a fake `struct SystemTimeTools` for mocking
`SystemTime::now()` for test purposes. One still needs to use `std::time::SystemTime` as a struct
representing a system time. I think such a minimalistic approach is ok -- even if somebody uses the
original `SystemTime::now()` instead of the mock by mistake, that could break only tests but not the
program itself. The worst thing that can happen is that tests using `SystemTime::shift()` and
checking messages timestamps f.e. wouldn't catch the corresponding bugs, but now we don't have such
tests at all which is much worse.
The system clock may be adjusted and even go back, so caching system time in code sections where
it's not supposed to change may even protect from races/bugs.
If a time value doesn't need to be sent to another host, saved to the db or otherwise used across
program restarts, a monotonically nondecreasing clock (`Instant`) should be used. But as `Instant`
may use `libc::clock_gettime(CLOCK_MONOTONIC)`, e.g. on Android, and does not advance while being in
deep sleep mode, get rid of `Instant` in favor of using `SystemTime`, but add `tools::Time` as an
alias for it with the appropriate comment so that it's clear why `Instant` isn't used in those
places and to protect from unwanted usages of `Instant` in the future. Also this can help to switch
to another clock impl if we find any.
Restart the IO scheduler if needed to make the new config value effective (for `MvboxMove,
OnlyFetchMvbox, SentboxWatch` currently). Also add `set_config_internal()` which doesn't affect
running the IO scheduler. The reason is that `Scheduler::start()` itself calls `set_config()`,
although not for the mentioned keys, but still, and also Rust complains about recursive async calls.
Currently when a user sets up another device by logging in, a new key is created. If a message is
sent from either device outside, it cannot be decrypted by the other device.
The message is replaced with square bracket error like this:
```
<string name="systemmsg_cannot_decrypt">This message cannot be decrypted.\n\n• It might already help to simply reply to this message and ask the sender to send the message again.\n\n• If you just re-installed Delta Chat then it is best if you re-setup Delta Chat now and choose "Add as second device" or import a backup.</string>
```
(taken from Android repo `res/values/strings.xml`)
If the message is outgoing, it does not help to "simply reply to this message". Instead, we should
add a translatable device message of a special type so UI can link to the FAQ entry about second
device. But let's limit such notifications to 1 per day. And as for the undecryptable message
itself, let it go to Trash if it can't be assigned to a chat by its references.
Before it was emitted only on the source device and `test_sync()` didn't catch the bug because of
the interference of the previous call to `set_config_bool()`. Now there's a separate test that does
its job well.
Before moving emails to the mvbox we need to remember its UIDVALIDITY, otherwise emails moved before
that wouldn't be fetched but considered "old" instead.
Also:
- Don't use `session.create()` to create mvbox as `select_with_uidvalidity()` already creates mvbox
on its own.
- Don't try to create compat folders like "INBOX.DeltaChat", but only look for them.
If `escaper::decode_html_buf_sloppy()` just truncates the text (which happens when it fails to
html-decode it at some position), then it's probably not HTML at all and should be left as
is. That's what happens with hyperlinks f.e. and there was even a test on this wrong behaviour which
is fixed now. So, now hyperlinks are not truncated in messages and should work as expected.
If a Delta Chat message has the Message-ID already existing in the db, but a greater "Date", it's a
resent message that can be deleted. Messages having the same "Date" mustn't be deleted because they
can be already seen messages moved back to INBOX. Also don't delete messages having lesser "Date" to
avoid deleting both messages in a multi-device setting.
Use `create_smeared_timestamp()` for this. This allows to dedup messages on the receiver -- if it
sees the same Message-ID, but a different timestamp, then it's a resent message that can be deleted.
"Auto-Submitted: auto-replied" messages mustn't be considered as sent by either bots or non-bots,
e.g. MDNs have this header value and it's the same for bots and non-bots.
This allows to send existing messages (incoming and outgoing) taken from encrypted chats, to
unencrypted ones. `Param::ForcePlaintext` is removed as well -- if a message can be sent encrypted
this time, nothing bad with this.
Currently `Chat.send_msg()` modifies the source message and returns another message object
equivalent to the source one. That's how it works in the core and thus in Python bindings too.
This is needed to protect from ESPs (such as gmx.at) doing their own Quoted-Printable encoding and
thus breaking messages and signatures. It's unlikely that the reader uses a MUA not supporting
Quoted-Printable encoding. And RFC 2646 "4.6" also recommends it for encrypted messages.
Add an event for a case if a multi-device synced config value changed. Maybe the app needs to
refresh smth on such an event. For uniformity it is emitted on the source device too. The value is
omitted, otherwise it would be logged which might not be good for privacy.
Before, while a message is in OutPending state after resending is requested, the user still sees the
red marker with error and it is confusing, so the user don't know the sending state of the message.
a27e84ad89 "fix: Delete received outgoing messages from SMTP queue"
can break sending messages sent as several SMTP messages because they have a lot of recipients:
`pub(crate) const DEFAULT_MAX_SMTP_RCPT_TO: usize = 50;`
We should not cancel sending if it is such a message and we received BCC-self because it does not
mean the other part was sent successfully. For this, split such messages into separate jobs in the
`smtp` table so that only a job containing BCC-self is canceled from `receive_imf_inner()`. Although
this doesn't solve the initial problem with timed-out SMTP requests for such messages completely,
this enables fine-grained SMTP retries so we don't need to resend all SMTP messages if only some of
them failed to be sent.
The bug was made in 44227d7b86. Sql::execute() with placeholders must
be used to escape strings, one never should escape them manually as strings themselves can contain
escape symbols. Thanks to @link2xt for noticing.
Some SMTP servers are running slow before-queue filters, most commonly Postfix with `rspamd` filter
which is implemented as a [before-queue Milter](https://www.postfix.org/MILTER_README.html). Some of
`rspamd` plugin filters are slow on large mails.
We previously had problems with timing out during waiting for SMTP response:
https://github.com/deltachat/deltachat-core-rust/issues/1383. This is largely fixed by
https://github.com/async-email/async-smtp/pull/29 and currently we have 60-second timeout just for
reading a response but apparently it is not sufficient -- maybe connection gets killed by NAT while
we are waiting for response or `rspamd` takes more than 60 seconds for large messages.
As a result a message is resent multiple times and eventually fails with "too many retries" while
multiple BCC-self messages are received.
We should remove the message from the SMTP queue as soon as we receive it via IMAP as it is clear
the message was sent even if we did not manage to get actual SMTP server response.
First of all, it's just downloaded and hasn't been seen yet by the user. Also this changes nothing
as `msgs.state` isn't changed when replacing a message anyway.
Put a copy of Message-ID into hidden headers and prefer it over the one in the IMF header section
that servers mess up with.
This also reverts "Set X-Microsoft-Original-Message-ID on outgoing emails for amazonaws (#3077)".
As per the comment in `receive_imf.rs`, `chat.protected` must be maintained regardless of the
`Config::VerifiedOneOnOneChats`. The only thing that mustn't be done if `VerifiedOneOnOneChats` is
unset (i.e. for non-supporting UIs) is marking chats as "protection broken" because this needs
showing the corresponding dialog to a user.
Drafts mustn't affect sorting of any other messages, they aren't even displayed in the chat
window. Also hidden messages mustn't affect sorting of usual messages. But let hidden messages sort
together with protection messages because hidden messages also can be or not be verified, so let's
preserve this information -- even it's not useful currently, it can be useful in the future
versions.
Before in some places it was correctly calculated by passing the "sent" timestamp to
`calc_sort_timestamp()`, but in other places just the system time was used. In some complex
scenarios like #5088 (restoration of a backup made before a contact verification) it led to wrong
sort timestamps of protection messages and also messages following by them.
But to reduce number of args passed to functions needing to calculate the sort timestamp, add
message timestamps to `struct MimeMessage` which is anyway passed everywhere.
Ad-hoc groups don't have grpid-s that can be used to identify them across devices and thus wasn't
synced until now.
The same problem already exists for assigning messages to ad-hoc groups and this assignment is done
by `get_parent_message()` and `lookup_chat_by_reply()`. Let's reuse this logic for the
synchronisation, it works well enough and this way we have less surprises than if we try to
implement grpids for ad-hoc groups. I.e. add an `Msgids` variant to `chat::SyncId` analogous to the
"References" header in messages and put two following Message-IDs to a sync message:
- The latest message A having `DownloadState::Done` and the state to be one of `InFresh, InNoticed,
InSeen, OutDelivered, OutMdnRcvd`.
- The message that A references in `In-Reply-To`.
This way the logic is almost the same to what we have in `Chat::prepare_msg_raw()` (the difference
is that we don't use the oldest Message-ID) and it's easier to reuse the existing code.
NOTE: If a chat has only an OutPending message f.e., the synchronisation wouldn't work, but trying
to work in such a corner case has no significant value and isn't worth complicating the code.
Otherwise it looks like the message creating a protected group is not verified. For this, use
`sent_timestamp` of the received message as an upper limit of the sort timestamp (`msgs.timestamp`)
of the protection message. As the protection message is added to the chat earlier, this way its
timestamp is always less or eq than the received message's timestamp.
Test that on the second device of a protected group creator the first message is
`SystemMessage::ChatProtectionEnabled` and the second one is the message populating the group.
It's not necessary and in other places like add_contact_to_chat_ex() sync messages are also sent
only if there are no system messages sent like MemberAddedToGroup.
We already synchronise status/footer when we see a self-sent message with a Chat-Version
header. Would be nice to do the same for display name.
But let's do it the same way as for `Config::{MdnsEnabled,ShowEmails}`. Otherwise, if we sync the
display name using the "From" header, smth like `Param::StatusTimestamp` is needed then to reject
outdated display names. Also this timestamp needs to be updated when `Config::Displayname` is set
locally. Also this wouldn't work if system time isn't synchronised on devices. Also using multiple
approaches to sync different config values would lead to more code and bugs while having almost no
value -- using "From" only saves some bytes and allows to sync some things w/o the synchronisation
itself to be enabled. But the latter also can be a downside -- if it's usual synchonisation, you can
(potentially) disable it and share the same email account across people in some organisation
allowing them to have different display names. With using "From" for synchronisation such a
capability definitely requires a new config option.
Motivation: Syncing these options will improve UX in very most cases and should be done. Other
candidates are less clear or are advanced options, we can reconsider that at some point later.
Approach:
- Sync options one-by-one when the corresponding option is set (even if to the same value).
- Don't sync when an option is reset to a default as defaults may differ across client versions.
- Check on both sides that the option should be synced so that if there are different client
versions, the synchronisation of the option is either done or not done in both directions.
Moreover, receivers of a config value need to check if a key can be synced because some settings
(e.g. Avatar path) could otherwise lead to exfiltration of files from a receiver's device if we
assume an attacker to have control of a device in a multi-device setting or if multiple users are
sharing an account.
- Don't sync `SyncMsgs` itself.
If a message is encrypted, but unsigned:
- Don't set `MimeMessage::from_is_signed`.
- Remove "secure-join-fingerprint" and "chat-verified" headers from `MimeMessage`.
- Minor: Preserve "Subject" from the unencrypted top level if there's no "Subject" in the encrypted
part, this message is displayed w/o a padlock anyway.
Apparently it didn't lead to any vulnerabilities because there are checks for
`MimeMessage::signatures.is_empty()` in all necessary places, but still the code looked dangerous,
especially because `from_is_singed` var name didn't correspond to its actual value (it was rather
`from_is_encrypted_maybe_signed`).
Limit the number of IMAP connections to 1 per minute regardless of the reason of reconnection, but
allow one immediate retry. This is more reliable than ratelimiting only in error conditions because
ratelimiting can't be skipped by mistake. Anyway connections shouldn't be frequent in normal
operation mode.
Other devices should get the same chat name as the currently used device, i.e. the name a user sees
after renaming the chat. This fix is minor because `improve_single_line_input()` logic isn't going
to change often, but still, and also it simplifies the code.
Looks like this doesn't fix anything currently, because a better message from
`apply_group_changes()` doesn't appear in a context with another better message, but why drop it if
it's possible to add it, moreover, messages about implicit member additions are never dropped while
looking less important.
iequidoo