Headers like From and Autocrypt are now added late,
right before sending the message over SMTP.
This way we advertise the latest list of transports
and use the correct From address in the encrypted part
even for messages queued while being offline.
BCC-self recipients are also added late.
For unencrypted messages we only want to send a copy
to the sending address, but we don't know the sending address
when queueing the message.
Adding bcc-self recipients when dequeuing the message
also makes it possible to send copies to updated list of relays.
This did not affect actual message deletion,
because select_expired_messages already takes timestamp_rcvd
into account and does not delete system messages
that say "Messages are end-to-end encrypted" too early.
So it is a minor bug as estimate_deletion_cnt
is meant to only roughly estimate the number of messages
to be deleted. Still, there were no tests before,
so now estimate_deletion_cnt is tested.
"ex" means "used to be, but no longer is".
"ext" means "extended", which is the intended meaning.
"Ext" is more common not only in the Rust ecosystem but in general.
We already had several people (myself included) asking
what "ex" is supposed to mean.
To reproduce this commit, search and replace `_ex(?!\w)` -> `_ext`.
But don't change `list_transports_ex` because it's public API.
This value is not used for any logic
except the disappearing messages timer rollback prevention.
It is unlikely that non-chat clients
participate in chats with disappearing messages,
and the case was not tested anyway.
There was a bug in prefetch_should_download() that it made it return
true for pre-messages even when they were already downloaded. This meant
that pre-messages were downloaded from all relays, rather than just one,
wasting internet data.
The fix is in rfc724_mid_download_tried(), which is used by
prefetch_should_download() to determine whether a message was already
downloaded.
---------
Co-authored-by: l <link2xt@testrun.org>
Maybe it's even good that MDNs will be sent for pre-messages only having placeholder text with the
viewtype and size, at least this way we notify the contact that we've noticed the message. Anyway,
with adding message previews the problem will be solved for the corresponding viewtypes.
We have recently removed it from the normal "message info" dialog
at least on Delta Chat Desktop, so without this one can't easily find
the ID of a particular message,
which is useful for debugging.
Follow-up to 970222f. We shouldn't update the message state to `OutDelivered` if message sending
failed before and the message was re-sent successfully only to a new broadcast member.
receive_imf should not fail in this case,
but trash the post-message and updates.
The problem with previously used message::rfc724_mid_exists check
was that rfc724_mid_exists may return a trashed message.
Fix https://github.com/chatmail/core/issues/8195
Supersedes https://github.com/chatmail/core/pull/8217
The most interesting change is in `delete_expired_imap_messages()`
because there, messages are marked for deletion on single-device
chatmail profiles.
The logic in `delete_expired_imap_messages()` was wrong before, and
pre-messages were not deleted at all. This is fixed by also querying
`pre_rfc724_mid` in addition to `rfc724_mid` from the `msgs` table.
In order not to make the SQL statement too complex, I split it into two.
WRT tests:
- `test_immediate_autodelete()` tests the auto-deletion;
`test_imap_autodelete_fully_downloaded_msg()` tests that even for a
message that is split into pre- and post-message, both messages are
deleted.
- A lot of tests test that if bcc_self is on, messages are not
auto-deleted. E.g. `test_one_account_send_bcc_setting()` and
`test_markseen_message_and_mdn()` relies on the fact that messages stay
on the server when bcc_self is on.
- Unfortunately, it is not possible to test that messages are only
deleted for chatmail servers, because we don't have any tests that use a
non-chatmail server.
`OutPreparing` is deprecated since 2024-12-07, replace it with `OutFailed`, such messages are
probably not interesting anymore. `OutMdnRcvd` is not used in the db for new messages since
a30c6ae1f7, `OutDelivered` is stored instead.
E.g. rename location::is_sending_locations_to_chat to location::is_sending_to_chat.
Then import only the location module and use it with location:: prefix,
similarly to how e.g. std::fs or std::tokio::fs avoids using "fs" or "file"
in function names.
There's no check for `from_id` so another group member can delete the
message, but this can only happen in case of message reordering and the
problem already exists for usual messages, so we may ignore it and
overall a group represents a scope of trust.
Co-authored-by: Hocuri <hocuri@gmx.de>
This causes problems for Delta Chat Desktop at
<https://github.com/deltachat/deltachat-desktop/pull/6278>
even though the logic was originally introduced for iOS.
If the problem remains on iOS,
heuristics can be added into iOS UI.
Otherwise `Param::GuaranteeE2ee` is set only after rendering the message
and some UIs, e.g. DC Android, display the message as unencrypted while
preparing the blob and rendering and encrypting the message. NB: DC
Desktop doesn't display the message until `send_msg()` returns.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
this adds an api to make the newest incoming message of a chat as "fresh",
so that UI can offer a "mark chat unread" option as usual for messengers
(eg. swipe right on iOS toggels between "read" and "unread").
"mark unread" is one of the most requested missing features,
used by many ppl to organize their every day messenger usage -
tho "pinning" and "saved messages" are similar,
it seems to be missed often.
we follow a very simple approach here
and just reset the state to `MessageState::InFresh`.
this does not introduce new states or flows.
therefore, chats without any incoming message cannot be marked as fresh.
in practise, this is probably not really an issue,
as the "mark fresh" is usually used to undo a "mark noticed" operation -
and then you have incoming message.
also, most status messages as "all messages are e2ee" count as incoming.
to avoid double sending of MDN,
we remove `Param::WantsMdn` once the MDN is scheduled.
in case MDN are used for syncing, MDN is still sent as before.
many other messenger show a "badge without number",
if we want that as well,
we can always track the "manually set as fresh" state in a parameter.
but for now, it is fine without and showing a "1", which alsso makes sense as badges may be summed up.
there is an iOS pr that uses this new feature,
jsonrpc is left out until api is settled.
also out of scope is synchronisation -
main reason is that "mark noticed" is not synced as well, so we avoid an imbalance here.
both, "mark noticed" as well as "mark fresh" should be synced however,
as soon as this feature is merged.
Hidden messages are marked as seen
when chat is marked as noticed.
MDNs to such messages should not be sent
as this notifies the hidden message sender
that the chat was opened.
The issue discovered by Frank Seifferth.
Close https://github.com/chatmail/core/issues/7396. Before reviewing,
you should read the issue description of
https://github.com/chatmail/core/issues/7396.
I recommend to review with hidden whitespace changes.
TODO:
- [x] Implement the new protocol
- [x] Make Rust tests pass
- [x] Make Python tests pass
- [x] Test it manually on a phone
- [x] Print the sent messages, and check that they look how they should:
[test_secure_join_group_with_mime_printed.txt](https://github.com/user-attachments/files/24800556/test_secure_join_group.txt)
- [x] Fix bug: If Alice has a second device, then Bob's chat won't be
shown yet on that second device. Also, Bob's contact isn't shown in her
contact list. As soon as either party writes something into the chat,
the that shows up and everything is fine. All of this is still a way
better UX than in WhatsApp, where Bob always has to write first 😂
Still, I should fix that.
- This is actually caused by a larger bug: AUTH tokens aren't synced if
there is no corresponding INVITE token.
- Fixed by 6b658a0e0
- [x] Either make a new `auth_tokens` table with a proper UNIQUE bound,
or put a UNIQUE bound on the `tokens` table
- [x] Benchmarking
- [x] TODOs in the code, maybe change naming of the new functions
- [x] Write test for interop with older DC (esp. that the original
securejoin runs if you remove the &v=3 param)
- [x] From a cryptography perspective, is it fine that vc-request is
encrypted with AUTH, rather than a separate secret (like INVITE)?
- [x] Make sure that QR codes without INVITE work, so that we can remove
it eventually
- [x] Self-review, and comment on some of my code changes to explain
what they do
- [x] ~~Maybe use a new table rather than reusing AUTH token.~~ See
https://github.com/chatmail/core/pull/7754#discussion_r2728544725
- [ ] Update documentation; I'll do that in a separate PR. All necessary
information is in the https://github.com/chatmail/core/issues/7396 issue
description
- [ ] Update tests and other code to use the new names (e.g.
`request-pubkey` rather than `request` and `pubkey` rather than
`auth-required`); I'll do that in a follow-up PR
**Backwards compatibility:**
Everything works seamlessly in my tests. If both devices are updated,
then the new protocol is used; otherwise, the old protocol is used. If
there is a not-yet-updated second device, it will correctly observe the
protocol, and mark the chat partner as verified.
Note that I removed the `Auto-Submitted: auto-replied` header from
securejoin messages. We don't need it ourselves, it's a cleartext header
that leaks too much information, and I can't see any reason to have it.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
fix https://github.com/chatmail/core/issues/7766
Implementation notes:
- Descriptions are only sent with member additions, when the description
is changed, and when promoting a previously-unpromoted group, in order
not to waste bandwith.
- Descriptions are not loaded everytime a chat object is loaded, because
they are only needed for the profile. Instead, they are in their own
table, and can be loaded with their own JsonRPC call.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
This includes forwarding of long messages. Also this fixes sending, but more likely resending of
forwarded messages for which the original message was deleted, because now we save HTML to the db
immediately when creating a forwarded message.
Co-authored-by: Hocuri <hocuri@gmx.de>
We currently synchronize "seen" status of messages by setting `\Seen` flag on IMAP and then looking
for new `\Seen` flags using `CONDSTORE` IMAP extension. This approach has multiple disadvantages:
- It requires that the server supports `CONDSTORE` extension. For example Maddy does not support
CONDSTORE yet: https://github.com/foxcpp/maddy/issues/727
- It leaks the seen status to the server without any encryption.
- It requires more than just store-and-forward queues and prevents replacing IMAP with simpler
protocols like POP3 or UUCP or some HTTP-based API for queue polling.
A simpler approach is to send MDNs to self when `Config::BccSelf` (aka multidevice) is enabled,
regardless of whether the message requested and MDN. If MDN was requested and we have MDNs enabled,
then also send to the message sender, but MDN to self is sent regardless of whether read receipts
are actually enabled.
`sync_seen_flags()` and `CONDSTORE` check is better completely removed, maybe after one
release. `store_seen_flags_on_imap()` can be kept for unencrypted non-chat messages.
One potential problem with sending MDNs is that it may trigger ratelimits on some providers and
count as another recipient.
This makes `Contact::get_all()` and `Chatlist::try_load()` case-insensitive for non-ASCII chat and
contact names as well. The same approach as in f6f4ccc6ea "feat:
Case-insensitive search for non-ASCII messages (#5052)" is used: `chats.name_normalized` and
`contacts.name_normalized` colums are added which store lowercased/normalized names (for a contact,
if the name is unset, it's a normalized authname). If a normalized name is the same as the
chat/contact name, it's not stored to reduce the db size. A db migration is added for 10000 random
chats and the same number of the most recently seen contacts, for users it will probably migrate all
chats/contacts and for bots which may have more data it's not important.
This API allows to check if the message with
given ID exists and distinguish between
message not existing and database error.
It might also be faster than
checking messages one by one
if multiple messages need to be checked
because of using a single SQL transaction.
