This makes such contacts appear in the contact list. `IncomingTo` is used because
`ChatId::accept_ex()` does so for groups, so sending a group message is effectively accepting the
group again in regards to contact searchability.
This fixes up b549e7633d which made it impossible to find contacts
from groups even if we've written there.
The part of logic there adjusting the sort timestamp forward if the parent message has a greater
sort timestamp wasn't tested explicitly by any test. I only saw one unrelated "golden test" failure
when commented it out.
(Related to #8027)
We know which message was added from the return value
of receive_imf(). It may be that the first chat
in the chatlist is not the one where the message was received
if there is a pinned chat or if
just received message is old.
For now, do this only for `OneOneChat` and `MailingListOrBroadcast`, this is enough to correctly
support messages from modern Delta Chat versions sending Intended Recipient Fingerprint subpackets
and single-recipient messages from modern versions of other MUAs.
This is an important thing forgotten to be checked in 332527089.
Also there's another test which currently doesn't work as we want: outgoing encrypted messages
continue to arrive to ad-hoc group even if we already have contact's key. This should be fixed by
sending and receiving Intended Recipient Fingerprint subpackets.
The good thing is that apparently there are no scenarios requiring the contact to update their
software, the user should just update own devices.
I.e. create a non-replyable ad-hoc group in such cases. Unencrypted replies to encrypted messages
are a security issue and "Composing a Reply Message" from RFC 9787 and "Replying and Forwarding
Guidance" from RFC 9788 forbid such replies.
84161f4202 promotes group members to `Origin::IncomingTo` when
accepting it, instead of `CreateChat` as before, but this changes almost nothing because it happens
rarely that the user only accepts a group and writes nothing there soon. Now if a message has
multiple recipients, i.e. it's a 3-or-more-member group, or if it's a broadcast message, we don't
scale up its recipients to `Origin::OutgoingTo`.
When accepting a chat, its members are promoted to `Origin::CreateChat`, but for groups it makes
sense to use lower origin because users don't always check all members before accepting a chat and
may not want to have the group members mixed with existing contacts. `IncomingTo` fits here by its
definition: "additional To:'s of incoming message of known sender", i.e. we assume that the sender
of some message is known to the user. This way we can show contacts coming from groups in the bottom
of contact list, maybe even add some separator later. It makes sense not to hide such contacts
completely, otherwise if the user remembers the contact name, but not the chat it's a member of, it
would be difficult to find the contact.
This records the curent behavior: after accepting a group with unknown contacts the contact list
contains them mixed with contacts for which 1:1 chats exist, i.e. new contacts from the group are
neither hidden nor sorted down.
Before, outgoing self-sent unencrypted messages were assigned to the self-chat. Now we assign them
to ad-hoc groups with only SELF instead of 1:1 chats with address contacts corresponding to our own
addresses because we don't want to create such address contacts; we still use SELF for `from_id` of
such messages. Not assigning such messages to the encrypted chat should be safe enough and such
messages can actually be sent by the user from another MUA.
It must be verified by "unknown verifier" instead. But if the verifier has known verifier in turn,
it must reverify contacts having unknown verifier. Add a check for this also.
Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
We already have both rand 0.8 and rand 0.9
in our dependency tree.
We still need rand 0.8 because
public APIs of rPGP 0.17.0 and Iroh 0.35.0
use rand 0.8 types in public APIs,
so it is imported as rand_old.
We do not try to delete resent messages
anymore. Previously resent messages
were distinguised by having duplicate Message-ID,
but future Date, but now we need to download
the message before we even see the Date.
We now move the message to the destination folder
but do not fetch it.
It may not be a good idea to delete
the duplicate in multi-device setups anyway,
because the device which has a message
may delete the duplicate of a message
the other device missed.
To avoid triggering IMAP busy move loop
described in the comments
we now only move the messages
from INBOX and Spam folders.
If we use modules (which are actually namespaces), we can use shorter names. Another approach is to
only use modules for internal code incapsulation and use full names like deltachat-ffi does.
Create unprotected group in test_create_protected_grp_multidev
The test is renamed accordingly.
SystemMessage::ChatE2ee is added in encrypted groups
regardless of whether they are protected or not.
Previously new encrypted unprotected groups
had no message saying that messages are end-to-end encrypted
at all.
In messages with text/plain, text/html and text/calendar parts
within a multipart/alternative, text/calendar part is currently ignored,
text/plain is displayed and HTML is available via HTML API.
Also verify not yet verified contacts w/o setting a verifier for them (in the db it's stored as
`verifier_id=id` though) because we don't know who verified them for another device.
We haven't dropped verified groups yet, so we need to do smth with messages that can't be verified
yet which often occurs because of messages reordering, particularly in large groups. Apart from the
reported case #7059, i had other direct reports that sometimes messages can't be verified for
various reasons, but when the reason is already fixed, it should be possible to re-download failed
messages and see them.
Also remove the code replacing the message text with a verification error from
`apply_group_changes()` as `add_parts()` already does this.
Encrypted message may create unencrypted groups
if the message does not have a Chat-Group-ID.
This can happen if v1 client sends an encrypted
message to opportunistically encrypted ad hoc group.
In this case `from_id` corresponds to the key-contact,
but we should add address-contact of the sender
to the member list.
This adds a test for https://github.com/chatmail/core/pull/7032/.
The crash happened if you received a message that has the from contact
also in the "To: " and "Chat-Group-Member-Fpr: " headers. Not sure how
it happened that such a message was created; I managed to create one in
a test, but needed to access some internals of MimeFactory for that.
I then saved the email that is sent by Alice into the test-data
directory, and then made a test that Bob doesn't crash if he receives
this email. And as a sanity-check, also test that Bob marks Fiona as
verified after receiving this email.
this PR adds a info message "messages are end-to-end-encrypted" also for
chats created by eg. vcards. by the removal of lock icons, this is a
good place to hint for that in addition; this is also what eg. whatsapp
and others are doing
the wording itself is tweaked at
https://github.com/deltachat/deltachat-android/pull/3817 (and there is
also the rough idea to make the message a little more outstanding, by
some more dedicated colors)
~~did not test in practise, if this leads to double "e2ee info messages"
on secure join, tests look good, however.~~ EDIT: did lots of practise
tests meanwhile :)
most of the changes in this PR are about test ...
ftr, in another PR, after 2.0 reeases, there could probably quite some
code cleanup wrt set-protection, protection-disabled etc.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
Delta Chat always adds protected headers to the inner encrypted or signed message, so if a protected
header is only present in the outer part, it should be ignored because it's probably added by the
server or somebody else. The exceptions are Subject and List-ID because there are known cases when
they are only present in the outer message part.
Also treat any Chat-* headers as protected. This fixes e.g. a case when the server injects a
"Chat-Version" IMF header tricking Delta Chat into thinking that it's a chat message.
Also handle "Auto-Submitted" and "Autocrypt-Setup-Message" as protected headers on the receiver
side, this was apparently forgotten.
Non-members can't modify the member list (incl. adding themselves), modify an ephemeral timer, so
they shouldn't be able to change the group name or avatar, just for consistency. Even if messages
are reordered and a group name change from a new member arrives before its addition, the new group
name will be applied on a receipt of the next message following the addition message because
Chat-Group-Name-Timestamp increases. While Delta Chat groups aimed for chatting with trusted
contacts, accepting group changes from everyone knowing Chat-Group-Id means that if any of the past
members have the key compromised, the group should be recreated which looks impractical.
