Fix#7435
For most messages, `calc_sort_timestamp()` makes sure that they are at the correct place; esp. that they are not above system messages or other noticed/seen messages.
Most callers of `add_info_msg()`, however, didn't call `calc_sort_timestamp()`, and just used `time()` or `smeared_time()` to get the sort timestamp. Because of this, system messages could sometimes wrongly be sorted above other messages.
This PR fixes this by making the sort timestamp optional in `add_info_msg*()`. If the sort timestamp isn't passed, then the message is sorted to the bottom of the chat. `sent_rcvd_timestamp` is not optional anymore, because we need _some_ timestamp that can be shown to the user (most callers just pass `time()` there).
Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
Flakiness was introduced in e7348a4fd8.
This change removes a call to joining_chat_id() which created a chat,
now we check for existing group chat
without creating it if it does not exist.
Create unprotected group in test_create_protected_grp_multidev
The test is renamed accordingly.
SystemMessage::ChatE2ee is added in encrypted groups
regardless of whether they are protected or not.
Previously new encrypted unprotected groups
had no message saying that messages are end-to-end encrypted
at all.
This mechanism replaces `Chat-Verified` header.
New parameter `_verified=1` in `Autocrypt-Gossip`
header marks that the sender has the gossiped key
verified.
Using `_verified=1` instead of `_verified`
because it is less likely to cause troubles
with existing Autocrypt header parsers.
This is also how https://www.rfc-editor.org/rfc/rfc2045
defines parameter syntax.
In messages with text/plain, text/html and text/calendar parts
within a multipart/alternative, text/calendar part is currently ignored,
text/plain is displayed and HTML is available via HTML API.
DKIM-Signatures apply to the last headers, so start from the last header and take a valid one,
i.e. skip headers having unknown critical attributes, etc. Though this means that Autocrypt header
must be "oversigned" to guarantee that a not DKIM-signed header isn't taken, still start from the
last header for consistency with processing other headers. This isn't a security issue anyway.
Encrypted message may create unencrypted groups
if the message does not have a Chat-Group-ID.
This can happen if v1 client sends an encrypted
message to opportunistically encrypted ad hoc group.
In this case `from_id` corresponds to the key-contact,
but we should add address-contact of the sender
to the member list.
This adds a test for https://github.com/chatmail/core/pull/7032/.
The crash happened if you received a message that has the from contact
also in the "To: " and "Chat-Group-Member-Fpr: " headers. Not sure how
it happened that such a message was created; I managed to create one in
a test, but needed to access some internals of MimeFactory for that.
I then saved the email that is sent by Alice into the test-data
directory, and then made a test that Bob doesn't crash if he receives
this email. And as a sanity-check, also test that Bob marks Fiona as
verified after receiving this email.
this PR adds a info message "messages are end-to-end-encrypted" also for
chats created by eg. vcards. by the removal of lock icons, this is a
good place to hint for that in addition; this is also what eg. whatsapp
and others are doing
the wording itself is tweaked at
https://github.com/deltachat/deltachat-android/pull/3817 (and there is
also the rough idea to make the message a little more outstanding, by
some more dedicated colors)
~~did not test in practise, if this leads to double "e2ee info messages"
on secure join, tests look good, however.~~ EDIT: did lots of practise
tests meanwhile :)
most of the changes in this PR are about test ...
ftr, in another PR, after 2.0 reeases, there could probably quite some
code cleanup wrt set-protection, protection-disabled etc.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
Part of #6884.
The channel owner will not be notified in any way that you left, they
will only see that there is one member less.
For the translated stock strings, this is what we agreed on in the
group:
- Add a new "Leave Channel" stock string (will need to be done in UIs)
- Reword the existing "Are you sure you want to leave this group?"
string to "Are you sure you want to leave?" (the options are "Cancel"
and "Leave Group" / "Leave Channel", so it's clear what you are leaving)
(will need to be done in the deltachat-android repo, other UIs will pick
it up automatically)
- Reword the existing "You left the group." string to "You left". (done
here, I will adapt the strings in deltachat-android, too)
I adapted DC Android by pushing
6df2740884
to https://github.com/deltachat/deltachat-android/pull/3783.
---------
Co-authored-by: l <link2xt@testrun.org>
BREAKING CHANGE: messages with invalid images, images of unknown size,
huge images, will have Viewtype::File
After changing the logic of Viewtype selection, I had to fix 3 old tests
that used invalid Base64 image data.
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
This change introduces a new type of contacts
identified by their public key fingerprint
rather than an e-mail address.
Encrypted chats now stay encrypted
and unencrypted chats stay unencrypted.
For example, 1:1 chats with key-contacts
are encrypted and 1:1 chats with address-contacts
are unencrypted.
Groups that have a group ID are encrypted
and can only contain key-contacts
while groups that don't have a group ID ("adhoc groups")
are unencrypted and can only contain address-contacts.
JSON-RPC API `reset_contact_encryption` is removed.
Python API `Contact.reset_encryption` is removed.
"Group tracking plugin" in legacy Python API was removed because it
relied on parsing email addresses from system messages with regexps.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Co-authored-by: B. Petersen <r10s@b44t.com>
instead of showing addresses in info message, provide an API to get the
contact-id.
UI can then make the info message tappable and open the contact profile
in scope
the corresponding iOS PR - incl. **screencast** - is at
https://github.com/deltachat/deltachat-ios/pull/2652 ; jsonrpc can come
in a subsequent PR when things are settled on android/ios
the number of parameters in `add_info_msg_with_cmd` gets bigger and
bigger, however, i did not want to refactor this in this PR. it is also
not really adding complexity
closes#6702
---------
Co-authored-by: link2xt <link2xt@testrun.org>
Co-authored-by: Hocuri <hocuri@gmx.de>
This implements new group consistency algorithm described in
<https://github.com/deltachat/deltachat-core-rust/issues/6401>
New `Chat-Group-Member-Timestamps` header is added
to send timestamps of member additions and removals.
Member is part of the chat if its addition timestamp
is greater or equal to the removal timestamp.
A user reported to me that after they left a group, they were implicitly readded, but there's no any
readdition message, so currently it looks in the chat like leaving it has no effect, just new
messages continue to arrive. The readdition probably happened because some member didn't receive the
user's self-removal message, anyway, at least there must be a message that the user is readded, even
if it isn't known by whom.
with this PR, when an `.xdc` with `request_integration = map` in the
manifest is added to the "Saved Messages" chat, it is used _locally_ as
an replacement for the shipped maps.xdc (other devices will see the
`.xdc` but not use it)
this allows easy development and adapting the map to use services that
work better in some area.
there are lots of known discussions and ideas about adding more barriers
of safety. however, after internal discussions, we decided to move
forward and also to allow internet, if requested by an integration (as
discussed at
https://github.com/deltachat/deltachat-core-rust/pull/3516).
the gist is to ease development and to make users who want to adapt,
actionable _now_, without making things too hard and adding too high
barriers or stressing our own resources/power too much.
note, that things are still experimental and will be the next time -
without the corresponding switch being enabled, nothing will work at
all, so we can be quite relaxed here :)
for android/ios, things will work directly. for desktop, allow_internet
needs to be accepted unconditionally from core. for the future, we might
add a question before using an integration and/or add signing. or sth.
completely different - but for now, the thing is to get started.
nb: "integration" field in the webxdc-info is experimental as well and
should not be used in UIs at all currently, it may vanish again and is
there mainly for simplicity of the code; therefore, no need to document
that.
successor of https://github.com/deltachat/deltachat-core-rust/pull/5461
this is how it looks like currently - again, please note that all that
is an experiment!
<img width=320
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/f659c891-f46a-4e28-9d0a-b6783d69be8d>
<img width=320
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/54549b3c-a894-4568-9e27-d5f1caea2d22>
... when going out of experimental, there are loots of ideas, eg.
changing "Start" to "integrate"
I.e. add the "Messages are guaranteed to be end-to-end encrypted from now on." message and mark the
chat as protected again because no user action is required in this case. There are a couple of
problems though:
- If the program crashes earlier than the protection is restored, the chat remains
protection-broken. But this problem already exists because `ChatId::set_protection()` is never
retried.
- If multiple old unverified messages are received, protection messages added in between don't
annihilate, so they clutter the chat.
If a message from an old contact's setup is received, the outdated Autocrypt header isn't applied,
so the contact verification preserves. But the chat protection breaks because the old message is
sorted to the bottom as it mustn't be sorted over the protection info message (which is `InNoticed`
moreover). Would be nice to preserve the chat protection too e.g. add a "protection broken" message,
then the old message and then a new "protection enabled" message, but let's record the current
behaviour first.
Received messages shouldn't mingle with just sent ones and appear somewhere in the middle of the
chat, so we go after the newest non fresh message.
But if a received outgoing message is older than some `InSeen` message, better sort the received
message purely by timestamp (this is an heuristic in order not to break the Gmail-like case
simulated by `verified_chats::test_old_message_4()`). We could place the received message just
before that `InSeen` message, but anyway the user may not notice it.
At least this fixes outgoing messages sorting for shared accounts where messages from other devices
should be sorted the same way as incoming ones.
This reverts commit 1caf672904.
Otherwise public key signature is regenerated each time the key is
loaded and test `key::tests::test_load_self_existing` which loads the
key twice fails when two loads happen on different seconds.
Closes#5976
Otherwise it's impossible to remove a member with missing key from a protected group. In the worst
case a removed member will be added back due to the group membership consistency algo.
Otherwise, e.g. if a message is a large GIF, but its viewtype is set to `Image` by the app, this GIF
will be recoded to JPEG to reduce its size. GIFs and other special viewtypes must be always detected
and sent as is.
as discussed in several chats, this PR starts making it possible to use
Webxdc as integrations to the main app. In other word: selected parts of
the main app can be integrated as Webxdc, eg. Maps [^1]
this PR contains two parts:
- draft an Webxdc Integration API
- use the Webxdc Integration API to create a Maps Integration
to be clear: a Webxdc is not part of this PR. the PR is about marking a
Webxdc being used as a Map - and core then feeds the Webxdc with
location data. from the view of the Webxdc, the normal
`sendUpdate()`/`setUpdateListener()` is used.
things are still marked as "experimental", idea is to get that in to
allow @adbenitez and @nicodh to move forward on the integrations into
android and desktop, as well as improving the maps.xdc itself.
good news is that we currently can change the protocol between Webxdc
and core at any point :)
# Webxdc Integration API
see `dc_init_webxdc_integration()` in `deltachat.h` for overview and
documentation.
rust code is mostly in `webxdc/integration.rs` that is called by other
places as needed. current [user of the API is
deltachat-ios](https://github.com/deltachat/deltachat-ios/pull/1912),
android/desktop will probably follow.
the jsonrpc part is missing and can come in another PR when things are
settled and desktop is really starting [^2] (so we won't need to do all
iterations twice :) makes also sense, when this is done by someone
actually trying that out on desktop
while the API is prepared to allow other types of integrations (photo
editor, compose tools ...) internally, we currently ignore the type. if
that gets more crazy, we probably also need a dedicated table for the
integrations and not just a single param.
# Maps Integration
rust code is mostly in `webxdc/maps_integration.rs` that is called by
`webxdc/integration.rs` as needed.
EDIT: the idea of having a split here, is that
`webxdc/maps_integration.rs` really can focus on the json part, on the
communication with the .xdc, including tests
this PR is basic implementation, enabling to move forward on
integrations on iOS, but also on desktop and android.
the current implementation allows already the following:
- global and per-chat maps
- add and display POIs
- show positions and tracks of the last 24 hours
the current maps.xdc uses leaflet, and is in some regards better than
the current android/desktop implementations (much faster, show age of
positions, fade out positions, always show names of POIs, clearer UI).
however, we are also not bound to leaflet, it can be anything
> [**screenshots of the current
state**](https://github.com/deltachat/deltachat-ios/pull/1912)
> 👆
to move forward faster and to keep this PR small, the following will go
to a subsequent PR:
- consider allowing webxdc to use a different timewindow for the
location
- delete POIs
- jsonrpc
[^1]: maps are a good example as anyways barely native (see android
app), did cause a lot of pain on many levels in the past (technically,
bureaucratically), and have a comparable simple api
[^2]: only going for jsonrpc would only make sense if large parts of
android/ios would use jsonrpc, we're not there
---------
Co-authored-by: link2xt <link2xt@testrun.org>
Let's add a 1-minute tolerance to `Params::MemberListTimestamp`.
This adds to the group membership consistency algo the following properties:
- If remote group membership changes were made by two members in parallel, both of them are applied,
no matter in which order the messages are received.
- If we remove a member locally, only explicit remote member additions/removals made in parallel are
allowed, but not the synchronisation of the member list from "To". Before, if somebody managed to
reply earlier than receiving our removal of a member, we added it back which doesn't look good.
