this adds an api to make the newest incoming message of a chat as "fresh",
so that UI can offer a "mark chat unread" option as usual for messengers
(eg. swipe right on iOS toggels between "read" and "unread").
"mark unread" is one of the most requested missing features,
used by many ppl to organize their every day messenger usage -
tho "pinning" and "saved messages" are similar,
it seems to be missed often.
we follow a very simple approach here
and just reset the state to `MessageState::InFresh`.
this does not introduce new states or flows.
therefore, chats without any incoming message cannot be marked as fresh.
in practise, this is probably not really an issue,
as the "mark fresh" is usually used to undo a "mark noticed" operation -
and then you have incoming message.
also, most status messages as "all messages are e2ee" count as incoming.
to avoid double sending of MDN,
we remove `Param::WantsMdn` once the MDN is scheduled.
in case MDN are used for syncing, MDN is still sent as before.
many other messenger show a "badge without number",
if we want that as well,
we can always track the "manually set as fresh" state in a parameter.
but for now, it is fine without and showing a "1", which alsso makes sense as badges may be summed up.
there is an iOS pr that uses this new feature,
jsonrpc is left out until api is settled.
also out of scope is synchronisation -
main reason is that "mark noticed" is not synced as well, so we avoid an imbalance here.
both, "mark noticed" as well as "mark fresh" should be synced however,
as soon as this feature is merged.
encryption info needs a dedicated string for "Messages are end-to-end encrypted"
as the UI will add more infomation to the info messages,
smth. as "Tap for more information".
an alternative fix would have been to let the UI render the info-message
differently, but adding another string to core causes less friction.
If one of broadcast owner's devices didn't add a new subscriber for any reason, e.g. because of
missing SecureJoin messages, this device shall add "member added" messages when syncing the member
list from the `SetPgpContacts` message.
Timestamp renewal was introduced in 1dbf924c6a "feat:
chat::resend_msgs: Guarantee strictly increasing time in the Date header" so that re-sent messages
can be deduplicated on the reciver side, but the deduplication logic doesn't depend on "Date"
anymore.
fix#7877
The bug was: If there is no chat description, and the chat description
is set to an empty string, the INSERT statement inserted a row with an
empty chat description, and therefore from the view of the INSERT
statement, something changed.
This PR fixes this by simply loading the chat description first, and
comparing it.
instead of Alice saying to Bob "You changed the chat description",
we now say "[Chat description changed, please update ...]
i was also considering to say "[Chat description changed to:\n\n...]"
but then there is no incentive for ppl to update, and chat descriptions
for chat creation would still be missing. and this is probably far more
often used.
successor of https://github.com/chatmail/core/pull/7829
fix https://github.com/chatmail/core/issues/7766
Implementation notes:
- Descriptions are only sent with member additions, when the description
is changed, and when promoting a previously-unpromoted group, in order
not to waste bandwith.
- Descriptions are not loaded everytime a chat object is loaded, because
they are only needed for the profile. Instead, they are in their own
table, and can be loaded with their own JsonRPC call.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
This includes forwarding of long messages. Also this fixes sending, but more likely resending of
forwarded messages for which the original message was deleted, because now we save HTML to the db
immediately when creating a forwarded message.
Co-authored-by: Hocuri <hocuri@gmx.de>
It has a really complex logic, so it's better to avoid calling it if possible than think which side
effects and performance penalties it has. It was never called here before adding forwarding messages
across contexts (accounts).
This is not possible for webxdcs and vCards currently however, so add workarounds for them:
- Use translated "Mini App" as the webxdc name.
- Use just "👤" instead of the vCard summary (i.e. the vCard contact name).
resolves#7724: When forwarding a message with file to another profile, the file was not copied to the target $blobdir and so the forwarded message missed it
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
This PR fixes a bug that old channel members were remembered in the
database even after they left the channel. Concretely, they remained in
the `past_members` table that's only meant for groups.
Though it was not a bad bug; we're anyways not cleaning up old contacts.
We currently synchronize "seen" status of messages by setting `\Seen` flag on IMAP and then looking
for new `\Seen` flags using `CONDSTORE` IMAP extension. This approach has multiple disadvantages:
- It requires that the server supports `CONDSTORE` extension. For example Maddy does not support
CONDSTORE yet: https://github.com/foxcpp/maddy/issues/727
- It leaks the seen status to the server without any encryption.
- It requires more than just store-and-forward queues and prevents replacing IMAP with simpler
protocols like POP3 or UUCP or some HTTP-based API for queue polling.
A simpler approach is to send MDNs to self when `Config::BccSelf` (aka multidevice) is enabled,
regardless of whether the message requested and MDN. If MDN was requested and we have MDNs enabled,
then also send to the message sender, but MDN to self is sent regardless of whether read receipts
are actually enabled.
`sync_seen_flags()` and `CONDSTORE` check is better completely removed, maybe after one
release. `store_seen_flags_on_imap()` can be kept for unencrypted non-chat messages.
One potential problem with sending MDNs is that it may trigger ratelimits on some providers and
count as another recipient.
This fixes the bug when a new transport doesn't become primary on the 2nd device because INBOX from
the new transport isn't fully fetched. Now the `Transports` sync message is received from the old
transport, but as it has updated "From", it updates the primary transport correspondingly. NB: I/O
for the new primary transport isn't immediately started however, this needs a separate fix.
When accepting a chat, its members are promoted to `Origin::CreateChat`, but for groups it makes
sense to use lower origin because users don't always check all members before accepting a chat and
may not want to have the group members mixed with existing contacts. `IncomingTo` fits here by its
definition: "additional To:'s of incoming message of known sender", i.e. we assume that the sender
of some message is known to the user. This way we can show contacts coming from groups in the bottom
of contact list, maybe even add some separator later. It makes sense not to hide such contacts
completely, otherwise if the user remembers the contact name, but not the chat it's a member of, it
would be difficult to find the contact.
Add `chat::forward_msgs_2ctx()` which takes another context as a parameter and forwards messages to
it and its jsonrpc wrapper `CommandApi::forward_messages_to_account()`.
We periodically forget to remove new params from forwarded messages as this can't be catched by
existing tests, some examples:
bfc08abe88a1837aeb8c56b2361f01
This may leak confidential data. Instead, it's better to explicitly list params that we want to
forward, then if we forget to forward some param, a test on forwarding messages carrying the new
functionality will break, or the bug will be reported earlier, it's easier to notice that some info
is missing than some extra info is leaked.
This makes `Contact::get_all()` and `Chatlist::try_load()` case-insensitive for non-ASCII chat and
contact names as well. The same approach as in f6f4ccc6ea "feat:
Case-insensitive search for non-ASCII messages (#5052)" is used: `chats.name_normalized` and
`contacts.name_normalized` colums are added which store lowercased/normalized names (for a contact,
if the name is unset, it's a normalized authname). If a normalized name is the same as the
chat/contact name, it's not stored to reduce the db size. A db migration is added for 10000 random
chats and the same number of the most recently seen contacts, for users it will probably migrate all
chats/contacts and for bots which may have more data it's not important.
Fix#7435
For most messages, `calc_sort_timestamp()` makes sure that they are at the correct place; esp. that they are not above system messages or other noticed/seen messages.
Most callers of `add_info_msg()`, however, didn't call `calc_sort_timestamp()`, and just used `time()` or `smeared_time()` to get the sort timestamp. Because of this, system messages could sometimes wrongly be sorted above other messages.
This PR fixes this by making the sort timestamp optional in `add_info_msg*()`. If the sort timestamp isn't passed, then the message is sorted to the bottom of the chat. `sent_rcvd_timestamp` is not optional anymore, because we need _some_ timestamp that can be shown to the user (most callers just pass `time()` there).
We use query_and_then() instead of query_map() function now.
The difference is that row processing function
returns anyhow::Result, so simple fallible processing
like JSON parsing can be done inside of it
when calling query_map_vec() and query_map_collect()
without having to resort to query_map()
and iterating over all rows again afterwards.
Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
Make it return the correct value for non-Group chats.
This also should improve performance, thanks to the fact that
we now don't have to query all the chat's contacts.
Instead we only need confirm that self-contact
is among the group members, and only when the chat type is `Group`.
If we use modules (which are actually namespaces), we can use shorter names. Another approach is to
only use modules for internal code incapsulation and use full names like deltachat-ffi does.
