fix https://github.com/chatmail/core/issues/7766
Implementation notes:
- Descriptions are only sent with member additions, when the description
is changed, and when promoting a previously-unpromoted group, in order
not to waste bandwith.
- Descriptions are not loaded everytime a chat object is loaded, because
they are only needed for the profile. Instead, they are in their own
table, and can be loaded with their own JsonRPC call.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
There are quite some unneeded stock strings; this PR removes some of
them. None of these stock strings were actually set by the UI, or even
have translations in Transifex.
- We don't have AEAP anymore.
- The "I added/removed member" and "I left the group" strings are
anyways not meant to be shown to the user. Also, starting to translate
them now would leak the device language.
BREAKING CHANGE: This can theoretically be a breaking change because a
UI could reference one of the removed stock strings, so I marked it as
breaking just in case.
Add a stock string `%1$s invited you to join this channel.\n\nWaiting
for the device of %2$s to reply…`, which is shown when a user starts to
join a channel.
I did _not_ add an equivalent to `%1$s replied, waiting for being added
to the group…`, which is shown when vg-auth-required was received. I
don't think that this would add any information that's interesting to
the user, other than 'something is happening, hang on'. And the more
text on the screen, the less likely that anyone reads it. But if others
think differently, we can also add it.
With this PR, joining a channel looks like this:
```
Msg#2003: info (Contact#Contact#Info): Messages are end-to-end encrypted. [NOTICED][INFO]
Msg#2004: info (Contact#Contact#Info): Alice invited you to join this channel.
Waiting for the device of Alice to reply… [NOTICED][INFO]
Msg#2007🔒: (Contact#Contact#2001): You joined the channel. [FRESH][INFO]
```
Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
This way, the statistics / self-reporting bot will be made into an
opt-in regular sending of statistics, where you enable the setting once
and then they will be sent automatically. The statistics will be sent to
a bot, so that the user can see exactly which data is being sent, and
how often. The chat will be archived and muted by default, so that it
doesn't disturb the user.
The collected statistics will focus on the public-key-verification that
is performed while scanning a QR code. Later on, we can add more
statistics to collect.
**Context:**
_This is just to give a rough idea; I would need to write a lot more
than a few paragraphs in order to fully explain all the context here_.
End-to-end encrypted messengers are generally susceptible to MitM
attacks. In order to mitigate against this, messengers offer some way of
verifying the chat partner's public key. However, numerous studies found
that most popular messengers implement this public-key-verification in a
way that is not understood by users, and therefore ineffective - [a 2021
"State of Knowledge" paper
concludes:](https://dl.acm.org/doi/pdf/10.1145/3558482.3581773)
> Based on our evaluation, we have determined that all current E2EE
apps, particularly when operating in opportunistic E2EE mode, are
incapable of repelling active man-in-the-middle (MitM) attacks. In
addition, we find that none of the current E2EE apps provide better and
more usable [public key verification] ceremonies, resulting in insecure
E2EE communications against active MitM attacks.
This is why Delta Chat tries to go a different route: When the user
scans a QR code (regardless of whether the QR code creates a 1:1 chat,
invites to a group, or subscribes to a broadcast channel), a
public-key-verification is performed in the background, without the user
even having to know about this.
The statistics collected here are supposed to tell us whether Delta Chat
succeeds to nudge the users into using QR codes in a way that is secure
against MitM attacks.
**Plan for statistics-sending:**
- [x] Get this PR reviewed and merged (but don't make it available in
the UI yet; if Android wants to make a release in the meantime, I will
create a PR that removes the option there)
- [x] Set the interval to 1 week again (right now, it's 1 minute for
testing)
- [ ] Write something for people who are interested in what exactly we
count, and link to it (see `TODO[blog post]` in the code)
- [ ] Prepare a short survey for participants
- [ ] Fine-tune the texts at
https://github.com/deltachat/deltachat-android/pull/3794, and get it
reviewed and merged
- [ ] After the next release, ask people to enable the
statistics-sending
Create unprotected group in test_create_protected_grp_multidev
The test is renamed accordingly.
SystemMessage::ChatE2ee is added in encrypted groups
regardless of whether they are protected or not.
Previously new encrypted unprotected groups
had no message saying that messages are end-to-end encrypted
at all.
this PR adds a info message "messages are end-to-end-encrypted" also for
chats created by eg. vcards. by the removal of lock icons, this is a
good place to hint for that in addition; this is also what eg. whatsapp
and others are doing
the wording itself is tweaked at
https://github.com/deltachat/deltachat-android/pull/3817 (and there is
also the rough idea to make the message a little more outstanding, by
some more dedicated colors)
~~did not test in practise, if this leads to double "e2ee info messages"
on secure join, tests look good, however.~~ EDIT: did lots of practise
tests meanwhile :)
most of the changes in this PR are about test ...
ftr, in another PR, after 2.0 reeases, there could probably quite some
code cleanup wrt set-protection, protection-disabled etc.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
A donation request device message is added if >= 100 messages have been sent and delivered. The
condition is checked every 30 days since the first message is sent. The message is added only once.
Part of #6884.
The channel owner will not be notified in any way that you left, they
will only see that there is one member less.
For the translated stock strings, this is what we agreed on in the
group:
- Add a new "Leave Channel" stock string (will need to be done in UIs)
- Reword the existing "Are you sure you want to leave this group?"
string to "Are you sure you want to leave?" (the options are "Cancel"
and "Leave Group" / "Leave Channel", so it's clear what you are leaving)
(will need to be done in the deltachat-android repo, other UIs will pick
it up automatically)
- Reword the existing "You left the group." string to "You left". (done
here, I will adapt the strings in deltachat-android, too)
I adapted DC Android by pushing
6df2740884
to https://github.com/deltachat/deltachat-android/pull/3783.
---------
Co-authored-by: l <link2xt@testrun.org>
Part of #6884
----
- [x] Add new chat type `InBroadcastChannel` and `OutBroadcastChannel`
for incoming / outgoing channels, where the former is similar to a
`Mailinglist` and the latter is similar to a `Broadcast` (which is
removed)
- Consideration for naming: `InChannel`/`OutChannel` (without
"broadcast") would be shorter, but less greppable because we already
have a lot of occurences of `channel` in the code. Consistently calling
them `BcChannel`/`bc_channel` in the code would be both short and
greppable, but a bit arcane when reading it at first. Opinions are
welcome; if I hear none, I'll keep with `BroadcastChannel`.
- [x] api: Add create_broadcast_channel(), deprecate
create_broadcast_list() (or `create_channel()` / `create_bc_channel()`
if we decide to switch)
- Adjust code comments to match the new behavior.
- [x] Ask Desktop developers what they use `is_broadcast` field for, and
whether it should be true for both outgoing & incoming channels (or look
it up myself)
- I added `is_out_broadcast_channel`, and deprecated `is_broadcast`, for
now
- [x] When the user changes the broadcast channel name, immediately show
this change on receiving devices
- [x] Allow to change brodacast channel avatar, and immediately apply it
on the receiving device
- [x] Make it possible to block InBroadcastChannel
- [x] Make it possible to set the avatar of an OutgoingChannel, and
apply it on the receiving side
- [x] DECIDE whether we still want to use the broadcast icon as the
default icon or whether we want to use the letter-in-a-circle
- We decided to use the letter-in-a-circle for now, because it's easier
to implement, and I need to stay in the time plan
- [x] chat.rs: Return an error if the user tries to modify a
`InBroadcastChannel`
- [x] Add automated regression tests
- [x] Grep for `broadcast` and see whether there is any other work I
need to do
- [x] Bug: Don't show `~` in front of the sender's same in broadcast
lists
----
Note that I removed the following guard:
```rust
if !new_chat_contacts.contains(&ContactId::SELF) {
warn!(
context,
"Received group avatar update for group chat {} we are not a member of.", chat.id
);
} else if !new_chat_contacts.contains(&from_id) {
warn!(
context,
"Contact {from_id} attempts to modify group chat {} avatar without being a member.",
chat.id,
);
} else [...]
```
i.e. with this change, non-members will be able to modify the avatar.
Things were slightly easier this way, and I think that this is in line
with non-members being able to modify the group name and memberlist
(they need to know the Group-Chat-Id, anyway), but I can also change it
back.
This change introduces a new type of contacts
identified by their public key fingerprint
rather than an e-mail address.
Encrypted chats now stay encrypted
and unencrypted chats stay unencrypted.
For example, 1:1 chats with key-contacts
are encrypted and 1:1 chats with address-contacts
are unencrypted.
Groups that have a group ID are encrypted
and can only contain key-contacts
while groups that don't have a group ID ("adhoc groups")
are unencrypted and can only contain address-contacts.
JSON-RPC API `reset_contact_encryption` is removed.
Python API `Contact.reset_encryption` is removed.
"Group tracking plugin" in legacy Python API was removed because it
relied on parsing email addresses from system messages with regexps.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Co-authored-by: B. Petersen <r10s@b44t.com>
this PR moves now advanced/unsupported ASM strings to core, removing
work from translations, esp. as another hint is added which would
require retranslations. it is better to have that just in english, it is
a nerd feature anyways.
moverover, this PR removes special rendering of ASM in the summary,
which might be confusion, but mainly it is now unneeded, dead code
i'll do another android PR that will point to "Add Second Device"
already on ASM generation EDIT: done at
https://github.com/deltachat/deltachat-android/pull/3726
targets https://github.com/deltachat/deltachat-desktop/issues/4946
for tuning down email address everywhere, that bit is missing in core.
it was never useful, as it was never shown on the receivers side. and
for the sender side, the context the qr code is opened is clear
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
instead of showing addresses in info message, provide an API to get the
contact-id.
UI can then make the info message tappable and open the contact profile
in scope
the corresponding iOS PR - incl. **screencast** - is at
https://github.com/deltachat/deltachat-ios/pull/2652 ; jsonrpc can come
in a subsequent PR when things are settled on android/ios
the number of parameters in `add_info_msg_with_cmd` gets bigger and
bigger, however, i did not want to refactor this in this PR. it is also
not really adding complexity
closes#6702
---------
Co-authored-by: link2xt <link2xt@testrun.org>
Co-authored-by: Hocuri <hocuri@gmx.de>
Deduplicate:
- In the REPL
- In `store_from_base64()`, which writes avatars received in headers
- In a few tests
- The saved messages, broadcast, device, archive icons
- The autocrypt setup message
1-2 more PRs, and we can get rid of `BlobObject::create`,
`sanitise_name()`, and some others
A NDN may arrive days after the message is sent when it's already impossible to tell which message
wasn't delivered looking at the "Failed to send" info message, so it only clutters the chat and
makes the user think they tried to send some message recently which isn't true. Moreover, the info
message duplicates the info already displayed in the error message behind the exclamation mark and
info messages do not point to the message that is failed to be sent.
Moreover it works rarely because `mimeparser.rs` only parses recipients from `x-failed-recipients`,
so it likely only works for Gmail. Postfix does not add this `X-Failed-Recipients` header. Let's
remove this parsing too. Thanks to @link2xt for pointing this out.
while adapting strings for the recent change about read receipts,
https://github.com/deltachat/deltachat-core-rust/pull/5712 , it turns
out in discussions eg. at
https://github.com/deltachat/deltachat-android/issues/3179 that
untranslated english for the read receipts seem to be sufficient or even
better:
- do not reveal the sender's language
- unexpected languages are confusing - even if you chat in english, you
may get Chinese read receipts
- many clients do not show the text anyways, iirc, eg. Outlook display
the read receipts in context, and Delta Chat of course as well
- afaik, we're leaving comparable `multipart/report` untranslated as
well (sync, but also webxdc updates are practically english only)
- less code, fewer translations needed :)
