Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
Delta Chat always adds protected headers to the inner encrypted or signed message, so if a protected
header is only present in the outer part, it should be ignored because it's probably added by the
server or somebody else. The exceptions are Subject and List-ID because there are known cases when
they are only present in the outer message part.
Also treat any Chat-* headers as protected. This fixes e.g. a case when the server injects a
"Chat-Version" IMF header tricking Delta Chat into thinking that it's a chat message.
Also handle "Auto-Submitted" and "Autocrypt-Setup-Message" as protected headers on the receiver
side, this was apparently forgotten.
This change introduces a new type of contacts
identified by their public key fingerprint
rather than an e-mail address.
Encrypted chats now stay encrypted
and unencrypted chats stay unencrypted.
For example, 1:1 chats with key-contacts
are encrypted and 1:1 chats with address-contacts
are unencrypted.
Groups that have a group ID are encrypted
and can only contain key-contacts
while groups that don't have a group ID ("adhoc groups")
are unencrypted and can only contain address-contacts.
JSON-RPC API `reset_contact_encryption` is removed.
Python API `Contact.reset_encryption` is removed.
"Group tracking plugin" in legacy Python API was removed because it
relied on parsing email addresses from system messages with regexps.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Co-authored-by: B. Petersen <r10s@b44t.com>
This change adds support for receiving
Autocrypt header in the protected part of encrypted message.
Autocrypt header is now also allowed in mailing lists.
Previously Autocrypt header was rejected when
List-Post header was present,
but the check for the address being equal to the From: address
is sufficient.
New experimental `protect_autocrypt` config is disabled
by default because Delta Chat with reception
support should be released first on all platforms.
Explicit check for `-----BEGIN PGP MESSAGE-----` is unnecessary
and not sufficient to ensure that the message is valid.
We have already checked the MIME type,
so ASCII-armored OpenPGP message should be inside.
If it's not, decryption will fail anyway.
An unencrypted message with already known Autocrypt key, but sent from another address, means that
it's rather a new contact sharing the same key than the existing one changed its address, otherwise
it would already have our key to encrypt.
- Remove "Detected Autocrypt-mime message" logs printed for every incoming Autocrypt message.
- Print only a single line at the beginning of receive_imf with both the Message-ID and seen flag.
- Print Securejoin step only once, inside handle_securejoin_handshake or observe_securejoin_on_other_device.
- Do not log "Not creating ad-hoc group" every time ad-hoc group is not created, log when it is created instead.
- Log ID of the chat where Autocrypt-Gossip for all members is received.
- Do not print "Secure-join requested." for {vg,vc}-request, we already log the step.
- Remove ">>>>>>>>>>>>>>>>>>>>>>>>>" noise from securejoin logs.
If a message is unsigned or signed with an unknown key, `MimeMessage::was_encrypted()` returns
false. So, it mustn't be checked when deciding whether to look into
`MimeMessage::decoded_data`. Looking through git history one can see that it's just a wrong check
left in the code for historical reasons.
Message.set_text() and Message.get_text() are modified accordingly
to accept String and return String.
Messages which previously contained None text
are now represented as messages with empty text.
Use Message.set_text("".to_string())
instead of Message.set_text(None).
If we move the detached signatures validation code out of try_decrypt(), we don't need to convert an
already parsed signed message part to Vec and then parse it back. Also this simplifies the
try_decrypt() semantics and return type. It can't make a good coffee anyway.
This way we don't need a separate code path for signatures validation for unencrypted
messages. Also, now we degrade encryption only if there are no valid signatures, so the code for
upgrading encryption back isn't needed.
Note that if the message is encrypted, we don't check whether it's signed with an attached key
currently, otherwise a massive refactoring of the code is needed because for encrypted messages a
signature is checked and discarded first now.
* Because both only make problems with mailing lists, it's easiest to just disable them. If we want, we can make them work properly with mailing lists one day and re-enable them, but this needs some further thoughts.
Part of #3701
* Use load_from_db() in more tests
* clippy
* Changelog
* Downgrade warning to info, improve message
* Use lifetimes instead of cloning
Since switch to async we don't have spurious "database is busy"
errors anymore. Since an error is irrecoverable in most cases,
we can skip the message. The cost of this is we may
accidentally skip a correct message if I/O fails, but
the advantage is that we are guaranteed to never confuse
irrecoverable error with recoverable one and get stuck in
infinite loop redownloading the same message over and over.
* Treat multiple From addresses as if there was no From: addr
* changelog
* Don't send invalid emails through the whole receive_imf pipeline
Instead, directly create a trash entry for them.
* Don't create trash entries for randomly generated Message-Id's
* clippy
* fix typo
Co-authored-by: link2xt <link2xt@testrun.org>
Fix#3507
Note that this is not intended for a release at this point! We first have to test whether it runs stable enough. If we want to make a release while we are not confident enough in authres-checking, then we have to disable it.
BTW, most of the 3000 new lines are in `test_data/messages/dkimchecks...`, not the actual code
da3a4b94 adds the results to the Message info. It currently does this by adding them to `hop_info`. Maybe we should rename `hop_info` to `extra_info` or something; this has the disadvantage that we can't rename the sql column name though.
Follow-ups for this could be:
- In `update_authservid_candidates()`: Implement the rest of the algorithm @hpk42 and me thought about. What's missing is remembering how sure we are that these are the right authserv-ids. Esp., when receiving a message sent from another account at the same domain, we can be quite sure that the authserv-ids in there are the ones of our email server. This will make authres-checking work with buzon.uy, disroot.org, yandex.ru, mailo.com, and riseup.net.
- Think about how we present this to the user - e.g. currently the only change is that we don't accept key changes, which will mean that the small lock on the message is not shown.
- And it will mean that we can fully enable AEAP, after revisiting the security implications of this, and assuming everyone (esp. @link2xt who pointed out the problems in the first place) feels comfortable with it.
