Remove unused info_only option from `get_chat_msgs_ex()`.
This option was meant to show an "audit log" of a group, i.e. only the
info messages. This feature was removed again from Desktop, but the
option still lingered around in Core.
This also adds a doc comment to the JsonRPC functions, because I
wanted to note somewhere that the parameter is deprecated, and I needed
some place to put this note.
After talking with r10s:
For spring cleaning, remove the largely-unused things that can be done a
bit.
Most private messengers (WhatsApp/Signal/...) do not have this feature,
and we do not want to become Matrix where every client has different,
partly-incompatible features.
I used some AI to draft a first version of this, and then reworked it.
This is one of multiple possibilities to fix
https://github.com/chatmail/core/issues/8041: For bots, the IncomingMsg
event is not emitted when the pre-message arrives, only when the
post-message arrives. Also, post-messages are downloaded immediately,
not after all the small messages are downloaded.
The `get_next_msgs()` API is deprecated. Instead, bots need to listen to
the IncomingMsg event in order to be notified about new events. Is this
acceptable for bots?
THE PROBLEM THAT WAS SOLVED BY THIS:
With pre-messages, it's hard for bots to wait for the message to be fully downloaded and then process it.
Up until now, bots used get_next_msgs() to query the unprocessed messages, then set last_msg_id after processing a message to that they won't process it again.
But this will now also return messages that were not fully downloaded.
ALTERNATIVES:
In the following, I will explain
the alternatives, and for why it's not so easy to just make the
`get_next_msgs()` API work. If it's not understandable, I'm happy to
elaborate more.
Core can't just completely ignore the pre-message for two reasons:
- If a post-message containing a Webxdc arrives later, and some webxdc updates arrive in the meantime, then these updates will be lost.
- The post-message doesn't contain the text (reasoning was to avoid duplicate text for people who didn't upgrade yet during the 2.43.0 rollout)
There are multiple solutions:
- Add the message as hidden in the database when the pre-message arrives.
- When the post-message arrives and we want to make it available for bots, we need to update the msg_id because of how the `get_next_msgs()` API works. This means that we need to update all webxdc updates that reference this msg_id.
- Alternatively, we could make webxdc's reference the rfc724_mid instead of the msg_id, so that we don't need stable msg_ids anymore
- Alternatively, we could deprecate `get_next_msgs()`, and ask bots to use plain events for message processing again. It's not that bad; worst case, the bot crashes and then forgets to react to some messages, but the user will just try again. And if some message makes the bot crash, then it might actually be good not to try and process it again.
- Store the pre-message text and `PostMsgMetadata` (or alternatively, the whole mime) in a new database table. Wait with processing it until the post-message arrives.
Additionally, the logic that small messages are downloaded before post-messages should be disabled for bots, in order to prevent reordering.
Rpc.start() now calls get_system_info() after launching the server
to verify it started successfully. If the server exits early (e.g.
due to an invalid accounts directory), the core error message from
stderr is captured and included in the raised JsonRpcError.
The reader_loop now unblocks pending RPC requests when the server
closes stdout, so callers never hang on a dead server.
Export JsonRpcError from the deltachat_rpc_client package.
Add test_early_failure verifying that Rpc.start() raises with
the actual core error message for invalid accounts directories.
This code does not expect the variable to be unset,
so use indexing to fail with KeyError instead.
Otherwise getenv() returns None which is then converted to "none" string by formatting
and the test only fails because of connection attempts to "none" domain.
This commit also makes testing hooks easier, as it allows to process
events and run hooks on them, until a certain event occurs.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
This is to fix tests failing with `OSError: [Errno 9] Bad file descriptor`. Maybe stdout closes
earlier than stderr, before the test finishes, not sure. For reference, the previous commit removing
print()s is 800edc6fce.
Actually it will be not as breaking if you used the constants, because
this pr also changes the constants.
closes#7029
Note that I had to change the constants from enum to namespace, this has
the side effect, that you can no longer also use the constants as types,
you need to instead prefix them with `typeof ` now.
Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
This is similar to old `dcaccount:` with URL,
but creates a 9-character username on the client
and avoids making an HTTPS request.
The scheme is reused to avoid the apps
needing to register for the new scheme.
`http` support is removed because it was
not working already, there is a check
that the scheme is `https` when the URL
is actually used and the core has
no way to make HTTP requests without TLS.
Create unprotected group in test_create_protected_grp_multidev
The test is renamed accordingly.
SystemMessage::ChatE2ee is added in encrypted groups
regardless of whether they are protected or not.
Previously new encrypted unprotected groups
had no message saying that messages are end-to-end encrypted
at all.
this PR adds a info message "messages are end-to-end-encrypted" also for
chats created by eg. vcards. by the removal of lock icons, this is a
good place to hint for that in addition; this is also what eg. whatsapp
and others are doing
the wording itself is tweaked at
https://github.com/deltachat/deltachat-android/pull/3817 (and there is
also the rough idea to make the message a little more outstanding, by
some more dedicated colors)
~~did not test in practise, if this leads to double "e2ee info messages"
on secure join, tests look good, however.~~ EDIT: did lots of practise
tests meanwhile :)
most of the changes in this PR are about test ...
ftr, in another PR, after 2.0 reeases, there could probably quite some
code cleanup wrt set-protection, protection-disabled etc.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
Part of #6884
----
- [x] Add new chat type `InBroadcastChannel` and `OutBroadcastChannel`
for incoming / outgoing channels, where the former is similar to a
`Mailinglist` and the latter is similar to a `Broadcast` (which is
removed)
- Consideration for naming: `InChannel`/`OutChannel` (without
"broadcast") would be shorter, but less greppable because we already
have a lot of occurences of `channel` in the code. Consistently calling
them `BcChannel`/`bc_channel` in the code would be both short and
greppable, but a bit arcane when reading it at first. Opinions are
welcome; if I hear none, I'll keep with `BroadcastChannel`.
- [x] api: Add create_broadcast_channel(), deprecate
create_broadcast_list() (or `create_channel()` / `create_bc_channel()`
if we decide to switch)
- Adjust code comments to match the new behavior.
- [x] Ask Desktop developers what they use `is_broadcast` field for, and
whether it should be true for both outgoing & incoming channels (or look
it up myself)
- I added `is_out_broadcast_channel`, and deprecated `is_broadcast`, for
now
- [x] When the user changes the broadcast channel name, immediately show
this change on receiving devices
- [x] Allow to change brodacast channel avatar, and immediately apply it
on the receiving device
- [x] Make it possible to block InBroadcastChannel
- [x] Make it possible to set the avatar of an OutgoingChannel, and
apply it on the receiving side
- [x] DECIDE whether we still want to use the broadcast icon as the
default icon or whether we want to use the letter-in-a-circle
- We decided to use the letter-in-a-circle for now, because it's easier
to implement, and I need to stay in the time plan
- [x] chat.rs: Return an error if the user tries to modify a
`InBroadcastChannel`
- [x] Add automated regression tests
- [x] Grep for `broadcast` and see whether there is any other work I
need to do
- [x] Bug: Don't show `~` in front of the sender's same in broadcast
lists
----
Note that I removed the following guard:
```rust
if !new_chat_contacts.contains(&ContactId::SELF) {
warn!(
context,
"Received group avatar update for group chat {} we are not a member of.", chat.id
);
} else if !new_chat_contacts.contains(&from_id) {
warn!(
context,
"Contact {from_id} attempts to modify group chat {} avatar without being a member.",
chat.id,
);
} else [...]
```
i.e. with this change, non-members will be able to modify the avatar.
Things were slightly easier this way, and I think that this is in line
with non-members being able to modify the group name and memberlist
(they need to know the Group-Chat-Id, anyway), but I can also change it
back.
This change introduces a new type of contacts
identified by their public key fingerprint
rather than an e-mail address.
Encrypted chats now stay encrypted
and unencrypted chats stay unencrypted.
For example, 1:1 chats with key-contacts
are encrypted and 1:1 chats with address-contacts
are unencrypted.
Groups that have a group ID are encrypted
and can only contain key-contacts
while groups that don't have a group ID ("adhoc groups")
are unencrypted and can only contain address-contacts.
JSON-RPC API `reset_contact_encryption` is removed.
Python API `Contact.reset_encryption` is removed.
"Group tracking plugin" in legacy Python API was removed because it
relied on parsing email addresses from system messages with regexps.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Co-authored-by: B. Petersen <r10s@b44t.com>
