This includes forwarding of long messages. Also this fixes sending, but more likely resending of
forwarded messages for which the original message was deleted, because now we save HTML to the db
immediately when creating a forwarded message.
Co-authored-by: Hocuri <hocuri@gmx.de>
This PR fixes a bug that old channel members were remembered in the
database even after they left the channel. Concretely, they remained in
the `past_members` table that's only meant for groups.
Though it was not a bad bug; we're anyways not cleaning up old contacts.
We currently synchronize "seen" status of messages by setting `\Seen` flag on IMAP and then looking
for new `\Seen` flags using `CONDSTORE` IMAP extension. This approach has multiple disadvantages:
- It requires that the server supports `CONDSTORE` extension. For example Maddy does not support
CONDSTORE yet: https://github.com/foxcpp/maddy/issues/727
- It leaks the seen status to the server without any encryption.
- It requires more than just store-and-forward queues and prevents replacing IMAP with simpler
protocols like POP3 or UUCP or some HTTP-based API for queue polling.
A simpler approach is to send MDNs to self when `Config::BccSelf` (aka multidevice) is enabled,
regardless of whether the message requested and MDN. If MDN was requested and we have MDNs enabled,
then also send to the message sender, but MDN to self is sent regardless of whether read receipts
are actually enabled.
`sync_seen_flags()` and `CONDSTORE` check is better completely removed, maybe after one
release. `store_seen_flags_on_imap()` can be kept for unencrypted non-chat messages.
One potential problem with sending MDNs is that it may trigger ratelimits on some providers and
count as another recipient.
84161f4202 promotes group members to `Origin::IncomingTo` when
accepting it, instead of `CreateChat` as before, but this changes almost nothing because it happens
rarely that the user only accepts a group and writes nothing there soon. Now if a message has
multiple recipients, i.e. it's a 3-or-more-member group, or if it's a broadcast message, we don't
scale up its recipients to `Origin::OutgoingTo`.
Before this PR, when a user with current main sends a large message to a
user with an old Delta Chat (before #7431), the text will be duplicated:
One message will arrive with only the text, and one message with
attachment+text.
This PR changes this - there will be one message with only the text, and
one message with only the attachment.
If we want to guard against lost pre-messages, then we can revert this
PR in a few months, though I'm not sure that's necessary - it's unlikely
that the small pre-message gets lost but the big post-message gets
through.
`chat::set_chat_profile_image()` already checks that the group has grpid, still it makes sense to
check that a message is encrypted when sending, in case if the chat has a profile image in the db
for some reason.
There are quite some unneeded stock strings; this PR removes some of
them. None of these stock strings were actually set by the UI, or even
have translations in Transifex.
- We don't have AEAP anymore.
- The "I added/removed member" and "I left the group" strings are
anyways not meant to be shown to the user. Also, starting to translate
them now would leak the device language.
BREAKING CHANGE: This can theoretically be a breaking change because a
UI could reference one of the removed stock strings, so I marked it as
breaking just in case.
Fix#7435
For most messages, `calc_sort_timestamp()` makes sure that they are at the correct place; esp. that they are not above system messages or other noticed/seen messages.
Most callers of `add_info_msg()`, however, didn't call `calc_sort_timestamp()`, and just used `time()` or `smeared_time()` to get the sort timestamp. Because of this, system messages could sometimes wrongly be sorted above other messages.
This PR fixes this by making the sort timestamp optional in `add_info_msg*()`. If the sort timestamp isn't passed, then the message is sorted to the bottom of the chat. `sent_rcvd_timestamp` is not optional anymore, because we need _some_ timestamp that can be shown to the user (most callers just pass `time()` there).
And enable it by default as the standard Header Protection is backward-compatible.
Also this tests extra IMF header removal when a message has standard Header Protection since now we
can send such messages.
Omit Legacy Display Elements from "text/plain" and "text/html" (implement 4.5.3.{2,3} of
https://www.rfc-editor.org/rfc/rfc9788 "Header Protection for Cryptographically Protected Email").
Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
We already have both rand 0.8 and rand 0.9
in our dependency tree.
We still need rand 0.8 because
public APIs of rPGP 0.17.0 and Iroh 0.35.0
use rand 0.8 types in public APIs,
so it is imported as rand_old.
This mechanism replaces `Chat-Verified` header.
New parameter `_verified=1` in `Autocrypt-Gossip`
header marks that the sender has the gossiped key
verified.
Using `_verified=1` instead of `_verified`
because it is less likely to cause troubles
with existing Autocrypt header parsers.
This is also how https://www.rfc-editor.org/rfc/rfc2045
defines parameter syntax.
With newer Iroh it is possible to obtain
own node address before home relay is selected
and accidentally send own address without relay URL.
It took me some time to debug why Iroh 0.92.0
did not work with iroh-relay 0.92.0
so I'm adding these assertions even
while we still use Iroh 0.35.0.
SDP offer and answer contain newlines.
Without the fix these newlines are not encoded at all
and break the header into multiple headers
or even prevent parsing of the following headers.
- sync declined calls from callee to caller, as usual in all larger
messengers
- introduce the call states "Missed call", "Declined call" and
"Cancelled all" ("Ended call" is gone)
- allow calling end_call()/accept_call() for already ended/accepted
calls, in practise, handling all cornercases is tricky in UI - and the
state needs anyways to be tracked.
- track and show the call duration
the duration calculation depends on local time, but it is displayed only
coarse and is not needed for any state. this can be improved as needed,
timestamps of the corresponding messages are probably better at some
point. or ending device sends its view of the time around. but for the
first throw, it seems good enough
if we finally want that set of states, it can be exposed to a json-info
in a subsequent call, so that the UI can render it more nicely. fallback
strings as follows will stay for now to make adaption in other UI easy,
and for debugging:
<img width="320" alt="IMG_0154"
src="https://github.com/user-attachments/assets/09a89bfb-66f4-4184-b05c-e8040b96cf44"
/>
successor of https://github.com/chatmail/core/pull/6650
a dedicated viewtype allows the UI to show a more advanced UI, but even
when using the defaults,
it has the advantage that incoming/outgoing and the date are directly
visible.
successor of https://github.com/chatmail/core/pull/6650
Sort recipients by `add_timestamp DESC` so that if the group is large and there are multiple SMTP
messages, a newly added member receives the member addition message earlier and has gossiped keys of
other members (otherwise the new member may receive messages from other members earlier and fail to
verify them).
This PR adds a test to reproduce a bug raised by @adbenitez that peer
channels break when the resend feature is used.
---------
Co-authored-by: iequidoo <dgreshilov@gmail.com>
We have some debug assertions already, but we also want the corresponding errors in the release
configuration so that it's not less reliable than non-optimized one. This doesn't change any
function signatures, only debug assertions in functions returning `Result` are replaced.
Co-authored-by: l <link2xt@testrun.org>
Part of #6884.
The channel owner will not be notified in any way that you left, they
will only see that there is one member less.
For the translated stock strings, this is what we agreed on in the
group:
- Add a new "Leave Channel" stock string (will need to be done in UIs)
- Reword the existing "Are you sure you want to leave this group?"
string to "Are you sure you want to leave?" (the options are "Cancel"
and "Leave Group" / "Leave Channel", so it's clear what you are leaving)
(will need to be done in the deltachat-android repo, other UIs will pick
it up automatically)
- Reword the existing "You left the group." string to "You left". (done
here, I will adapt the strings in deltachat-android, too)
I adapted DC Android by pushing
6df2740884
to https://github.com/deltachat/deltachat-android/pull/3783.
---------
Co-authored-by: l <link2xt@testrun.org>
If a user attaches an image as `File`, we should send the original filename. And vice versa, if it's
`Image` originally, we mustn't reveal the filename.
The filename used for sending is now also saved to the db, so all the sender's devices will display
the same filename in the message info.
Older versions of Delta Chat ignore the message if it contains a
ChatGroupId header. ("older versions" means all versions without #6901,
i.e.currently released versions)
This means that without this PR, broadcast channel messages sent from
current main don't arrive at a device running latest released DC.
Part of #6884.
Part of #6884
----
- [x] Add new chat type `InBroadcastChannel` and `OutBroadcastChannel`
for incoming / outgoing channels, where the former is similar to a
`Mailinglist` and the latter is similar to a `Broadcast` (which is
removed)
- Consideration for naming: `InChannel`/`OutChannel` (without
"broadcast") would be shorter, but less greppable because we already
have a lot of occurences of `channel` in the code. Consistently calling
them `BcChannel`/`bc_channel` in the code would be both short and
greppable, but a bit arcane when reading it at first. Opinions are
welcome; if I hear none, I'll keep with `BroadcastChannel`.
- [x] api: Add create_broadcast_channel(), deprecate
create_broadcast_list() (or `create_channel()` / `create_bc_channel()`
if we decide to switch)
- Adjust code comments to match the new behavior.
- [x] Ask Desktop developers what they use `is_broadcast` field for, and
whether it should be true for both outgoing & incoming channels (or look
it up myself)
- I added `is_out_broadcast_channel`, and deprecated `is_broadcast`, for
now
- [x] When the user changes the broadcast channel name, immediately show
this change on receiving devices
- [x] Allow to change brodacast channel avatar, and immediately apply it
on the receiving device
- [x] Make it possible to block InBroadcastChannel
- [x] Make it possible to set the avatar of an OutgoingChannel, and
apply it on the receiving side
- [x] DECIDE whether we still want to use the broadcast icon as the
default icon or whether we want to use the letter-in-a-circle
- We decided to use the letter-in-a-circle for now, because it's easier
to implement, and I need to stay in the time plan
- [x] chat.rs: Return an error if the user tries to modify a
`InBroadcastChannel`
- [x] Add automated regression tests
- [x] Grep for `broadcast` and see whether there is any other work I
need to do
- [x] Bug: Don't show `~` in front of the sender's same in broadcast
lists
----
Note that I removed the following guard:
```rust
if !new_chat_contacts.contains(&ContactId::SELF) {
warn!(
context,
"Received group avatar update for group chat {} we are not a member of.", chat.id
);
} else if !new_chat_contacts.contains(&from_id) {
warn!(
context,
"Contact {from_id} attempts to modify group chat {} avatar without being a member.",
chat.id,
);
} else [...]
```
i.e. with this change, non-members will be able to modify the avatar.
Things were slightly easier this way, and I think that this is in line
with non-members being able to modify the group name and memberlist
(they need to know the Group-Chat-Id, anyway), but I can also change it
back.
This change introduces a new type of contacts
identified by their public key fingerprint
rather than an e-mail address.
Encrypted chats now stay encrypted
and unencrypted chats stay unencrypted.
For example, 1:1 chats with key-contacts
are encrypted and 1:1 chats with address-contacts
are unencrypted.
Groups that have a group ID are encrypted
and can only contain key-contacts
while groups that don't have a group ID ("adhoc groups")
are unencrypted and can only contain address-contacts.
JSON-RPC API `reset_contact_encryption` is removed.
Python API `Contact.reset_encryption` is removed.
"Group tracking plugin" in legacy Python API was removed because it
relied on parsing email addresses from system messages with regexps.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Co-authored-by: B. Petersen <r10s@b44t.com>
