Removing a relay now takes effect immediately:
- the profile stops fetching and advertising it,
- secondary devices immediately apply the removal through the transport sync,
Upgrading removes unpublished relays and triggers keyupdates.
BREAKING CHANGE: set_transport_unpublished() is removed: UIs call delete_transport() when the user removes a relay.
BREAKING CHANGE: list_transports_ex() and the TransportListEntry type are removed: use list_transports().
BREAKING CHANGE: delete_transport() no longer refuses to remove the primary transport: it refuses only to remove the last one and re-elects the sending transport as needed.
BREAKING CHANGE: TransportsModified is now also emitted on the device modifying the transports, not only on devices applying the synced change.
Deprecated: DC_STR_PHASING_OUT
When the published relay list changes, key-contacts are informed with an
unsigned message carrying the re-signed key, encrypted to a chunk of contacts
at a time. It is shaped like a receipt notification naming no message, so that
cores which know nothing about keyupdates trash it as well.
See the src/keyupdate.rs module docs for the design.
An unsigned message carries no intended recipient fingerprints,
so recipients of an encrypted unsigned message
learn nothing about other recipients from the PGP packets.
"ex" means "used to be, but no longer is".
"ext" means "extended", which is the intended meaning.
"Ext" is more common not only in the Rust ecosystem but in general.
We already had several people (myself included) asking
what "ex" is supposed to mean.
To reproduce this commit, search and replace `_ex(?!\w)` -> `_ext`.
But don't change `list_transports_ex` because it's public API.
Adds `assert_warn`, `assert_error` and `assert_warns_or_errors`
methods to `TestContext`, that let us assert that
a certain warning or error is logged during the test.
Also asserts test logs should not contain any other
errors or warnings.
Adjusts tests accordingly.
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
This change separates rendering into two separate steps:
1. Rendering of the message payload without the From, Date and Autocrypt headers.
2. Adding the From, Date and Autocrypt headers and possibly encrypting the message.
The goal is to have serializable result of the first step
that can be persisted in the database and sent later with any email address.
This way it will be possible to send queued messages over any relay.
This will make it possible not to remove all messages from the queue
when the sending relay is changed.
Currently changing `configured_addr` deletes everything from `smtp` table.
This change is however only a refactoring and does not implement any features.
Unifies internal naming of single/direct/1:1/normal chats
to match the naming used in the API.
Closes: #8368
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
---------
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
BREAKING CHANGE: removed oauth2 module, dc_get_oauth2_url FFI function, DC_LP_AUTH flags and configured/serverflags, and the oauth2 parameter/field from SMTP/IMAP clients, JSON-RPC interfaces, and CLI tools.
also contains regenerated provider data after dropping oauth in the update script.
Arbitrary timeouts often result in flaky tests,
especially if CI runners may be paused.
In this case however timeout was not used by any tests
and only slowed down the tests in cases where
no message is expected but non-zero timeout is set.
Follow-up to 970222f. We shouldn't update the message state to `OutDelivered` if message sending
failed before and the message was re-sent successfully only to a new broadcast member.
This change is a preparation for ignoring
unencrypted messages by default.
New test_utils::encrypt_raw_message and
test_utils::receive_encrypted_imf are
used to encrypt the messages before
"receiving" them with receive_imf.
This is needed to have support of v6 PQC keys
by the time users start generating profiles
using such keys.
Test key was generated with rsop/v0.10.0-16-gd98265f
(commit d98265f821e7bb181d06da1d634c5c4668d89e83)
using the command
cargo run --features draft-pqc generate-key \
--profile draft-ietf-openpgp-pqc-14-v6-ed25519-mlkem768x25519
Remove unused info_only option from `get_chat_msgs_ex()`.
This option was meant to show an "audit log" of a group, i.e. only the
info messages. This feature was removed again from Desktop, but the
option still lingered around in Core.
This also adds a doc comment to the JsonRPC functions, because I
wanted to note somewhere that the parameter is deprecated, and I needed
some place to put this note.
We shouldn't just revert ad7f873c68 because then we won't be able to
normally transfer a backup from a device having clock a bit in the future.
INDEXED BY msgs_index7 is to prevent SQLite from downgrading to using msgs_index2 which has less
ordering.
Last 10 messages in a broadcast channel are resent. They are sent and encrypted only to the new member, not to other subscribers.
Close#7678
Based on https://github.com/chatmail/core/pull/7854, with the following
changes:
- Refactor and simplify code, don't reuse the existing `get_chat_msgs()`
function
- Document that Param::Arg4 is also used for resent messages
cc818d9099
- It's unclear how exactly to resend webxdc status updates. After
discussing with @r10s, don't resend webxdc's at all for now.
38d57ebb30
- Don't set fake `msg_id` in resent messages
e7d0687d90
Setting the msg_id to `u32::MAX` is hacky, and may just as well break
things as it may fix things, because some code may use the msg.id to
load information from the database, like `get_iroh_topic_for_msg()`.
From reading the code, I couldn't find any problem with leaving the
correct `msg_id`, and if there is one, then we should add a function
parameter `is_resending` that is checked in the corresponding places.
Easiest to review file-by-file rather than individual commits, probably.
I'll squash-merge this.
---------
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Fix https://github.com/chatmail/core/issues/8042
The problem was that after receiving the bcc_self'ed pre-message in
`receive_imf`, the logic there only looked for a pending
`smtp`-table-entry that matches the rfc724_mid, and if there was none
then it thought "Great, apparently the message is fully sent out, we can
mark it as delivered!".
But with pre-messages, the same message can have two `smtp` entries (one
for the pre-message and one for the post-message), and the message
should only be marked as delivered once both of them are sent out.
Now, I changed the logic to look for all entries with the same msg_id.
This is actually the same SQL query used in smtp.rs, so, I extracted it
into a new function; feel free to suggest a better name for it.
I tested on Android that it now works fine.
I'll add a test in a follow-up PR.
There are a lot of other problems with sending large files, though:
- The pre-message is sent before the post-message, so that for the
receiver it looks as if the message arrived, but stays in
"downloading..." forever
- There is quite a time delay between clicking on "Send" and the
outgoing message appearing in the chat
- The message shortly gets a letter icon right after it is sent
- I'm wondering if there is a way to give feedback to the user
immediately if the message is too big
- It's unclear when exactly we want to send read receipts
I'll open a follow-up issue for these.
Closes https://github.com/chatmail/core/issues/7980.
Unpublished transports are not advertised to contacts, and self-sent messages are not sent there, so that we don't cause extra messages to the corresponding inbox, but can still receive messages from contacts who don't know the new relay addresses yet.
- This adds `list_transports_ex()` and `set_transport_unpublished()` JsonRPC functions
- By default, transports are published, but when updating, all existing transports except for the primary one become unpublished in order not to break existing users that followed https://delta.chat/legacy-move
- It is not possible to unpublish the primary transport, and setting a transport as primary automatically sets it to published
An alternative would be to change the existing list_transports API rather than adding a new one list_transports_ex. But to be honest, I don't mind the _ex prefix that much, and I am wary about compatibility issues. But maybe it would be fine; see b08ba4bb8 for how this would look.
There is no need to store copy of public key
next to the secret key because public key is a subset of the secret key
and can be obtained by using SignedSecretKey.public_key()
or SignedSecretKey.to_public_key().
For now, do this only for `OneOneChat` and `MailingListOrBroadcast`, this is enough to correctly
support messages from modern Delta Chat versions sending Intended Recipient Fingerprint subpackets
and single-recipient messages from modern versions of other MUAs.
- webxdc notify specifically to Bob: notifies Bob even when chat is
muted
- webxdc notify to everyone ("*"): notifies Bob if he does not have the
chat muted
This aligns with how we handle notifications with quote replies.
---------
Co-authored-by: link2xt <link2xt@testrun.org>
Omit Legacy Display Elements from "text/plain" and "text/html" (implement 4.5.3.{2,3} of
https://www.rfc-editor.org/rfc/rfc9788 "Header Protection for Cryptographically Protected Email").
Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
