Before, copying Auto-Submitted to the outer headers was needed for moving such messages,
e.g. multi-device sync messages, to the DeltaChat folder. Now all encrypted messages are moved.
Chat-Version is used sometimes by Sieve filters to move messages to DeltaChat folder:
37beed6ad9/data/conf/dovecot/global_sieve_before
This probably prevents notifications to MUAs that don't watch DeltaChat but watch INBOX.
There are however disadvantages to exposing Chat-Version:
1. Spam filters may not like this header and it is difficult or impossible to tell if `Chat-Version`
plays role in rejecting the message or delivering it into Spam folder. If there is no such header
visible to the spam filter, this possibility can be ruled out.
2. Replies to chat messages may have no `Chat-Version` but have to be moved anyway.
3. The user may have no control over the Sieve filter, but it comes preconfigured in mailcow, so it
is not possible to disable it on the client.
Thanks to link2xt for providing this motivation.
NOTE: Old Delta Chat will assign partially downloaded replies to an ad-hoc group with the sender
instead of the 1:1 chat, but we're removing partial downloads anyway.
Before, only replies to chat messages were moved to the mvbox because we're removing Chat-Version
from outer headers, but there's no much sense in moving only replies and not moving original
messages and MDNs. Instead, move all encrypted messages. Users should be informed about this in UIs,
so if a user has another PGP-capable MUA, probably they should disable MvboxMove. Moreover, untying
this logic from References and In-Reply-To allows to remove them from outer headers too, the "Header
Protection for Cryptographically Protected Email" RFC even suggests such a behavior:
https://datatracker.ietf.org/doc/html/rfc9788#name-offering-more-ambitious-hea.
There was a comment that group messages should always be downloaded to avoid inconsistent group
state, but this is solved by the group consistency algo nowadays in the sense that inconsistent
group state won't spread to other members if we send to the group. Moreover, encrypted messages are
now always downloaded, and unencrypted chat replies too, and as for ad-hoc groups,
`Config::ShowEmails` controls everything.
Instead, prefetch Secure-Join, Content-Type and Subject headers, try to find out if the message is encrypted, i.e.:
- if its Content-Type is "multipart/encrypted"
- or Subject is "..." or "[...]" as some MUAs use "multipart/mixed"; we can't only look at Subject
as it's not mandatory;
and depending on this decide on the target folder and whether the message should be
downloaded. There's no much sense in downloading unencrypted "Chat-Version"-containing messages if
`ShowEmails` is `Off` or `AcceptedContacts`, unencrypted Delta Chat messages should be considered as
usual emails, there's even the "New E-Mail" feature in UIs nowadays which sends such messages.
Don't prefetch Auto-Submitted as well, this becomes unnecessary.
Changed behavior: before, "Chat-Version"-containing messages were moved from INBOX to DeltaChat, now
such encrypted messages may remain in INBOX -- if there's no parent message or it's not
`MessengerMessage`. Don't unconditionally move encrypted messages yet because the account may be
shared with other software which doesn't and shouldn't look into the DeltaChat folder.
adb brought this up in an internal discussion.
With the recent introduction of channels it becomes easier to hit the
limit
and it becomes impossible to send messages to a channel with more than
1000 members, this pr fixes that.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
`Contact::get_color()` returns gray if own keypair doesn't exist yet and we don't want any UIs
displaying it. Keypair generation can't be done in `get_color()` or `get_by_id_optional()` to avoid
breaking Core tests on key import. Also this makes the API clearer, pure getters shouldn't modify
any visible state.
By the time you scan the QR code,
inviter may not be in the group already.
In this case securejoin protocol will never complete.
If you then join the group in some other way,
this results in you implicitly adding that inviter
to the group.
For multiple transports we will need to run
multiple IMAP clients in parallel.
UID validity change detected by one IMAP client
should not result in UID resync
for another IMAP client.
- sort garbage to the beginning, readable text to the end
- instead of `%20`, make use of `+` to encode spaces
- shorter invite links and smaller QR codes by truncation of the names
the truncation of the name uses chars() which does not respect grapheme clusters, so
that last character may be wrong. not sure if there is a nice and easy
alternative, but maybe it's good engoug - the real, full name will come
over the wire (exiting truncate() truncates on word boundaries, which is
maybe too soft here - names may be long, depending on the language, and
not contain any space)
moreover, this resolves the "name too long" issue from
https://github.com/chatmail/core/issues/7015
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
We use query_and_then() instead of query_map() function now.
The difference is that row processing function
returns anyhow::Result, so simple fallible processing
like JSON parsing can be done inside of it
when calling query_map_vec() and query_map_collect()
without having to resort to query_map()
and iterating over all rows again afterwards.
Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
`login_param` module is now for user-visible entered login parameters,
while the `transport` module contains structures for internal
representation of connection candidate list
created during transport configuration.
This is similar to old `dcaccount:` with URL,
but creates a 9-character username on the client
and avoids making an HTTPS request.
The scheme is reused to avoid the apps
needing to register for the new scheme.
`http` support is removed because it was
not working already, there is a check
that the scheme is `https` when the URL
is actually used and the core has
no way to make HTTP requests without TLS.
It's used in `fetch_existing_msgs()`, but we can remove it and tell users that they need to
move/copy messages from Sentbox to Inbox so that Delta Chat adds all contacts from them. This way
users will be also informed that Delta Chat needs users to CC/BCC/To themselves to see messages sent
from other MUAs.
The motivation is to reduce code complexity, get rid of the extra IMAP connection and cases when
messages are added to chats by Inbox and Sentbox loops in parallel which leads to various message
sorting bugs, particularly to outgoing messages breaking sorting of incoming ones which are fetched
later, but may have a smaller "Date".
We already have both rand 0.8 and rand 0.9
in our dependency tree.
We still need rand 0.8 because
public APIs of rPGP 0.17.0 and Iroh 0.35.0
use rand 0.8 types in public APIs,
so it is imported as rand_old.
Flakiness was introduced in e7348a4fd8.
This change removes a call to joining_chat_id() which created a chat,
now we check for existing group chat
without creating it if it does not exist.
Context: PR #7116 is backwards-incompatible with versions older than
v2.21, and since the release hasn't reached all users yet, we currently
can't release from main; for details see #7326.
Issue #7326 explains how we can make this less breaking, but this only
works if many contacts are verified. So, this PR here proposes to
postpone the stricter rules for who is verified a bit:
- Set verification timeout for invite codes to 1 week (this is still
stricter than no timeout at all, which we had in the past)
- Don't reset indirect verifications yet
In a few months (when everyone has v2.22.0), we can revert the PR here,
then.
---------
Co-authored-by: l <link2xt@testrun.org>
