This way they also can be processed by `markseen_msgs()` resulting in MDNs which improves
multi-device synchronization and updates contacts' `last_seen`. I.e. leave it up to the UIs.
Hidden messages are marked as seen
when chat is marked as noticed.
MDNs to such messages should not be sent
as this notifies the hidden message sender
that the chat was opened.
The issue discovered by Frank Seifferth.
Can be reviewed commit-by-commit.
This fixes another silly thing you can do with securejoinv3: show Bob a
QR code with auth token that is a broadcast channel secret of a known
channel, then never respond. Bob will decrypt messages from the channel
and drop them because they are sent by the "wrong" sender.
This can be avoided with domain separation, instead of
encrypting/decrypting securejoinv3 messages directly with auth token,
encrypt/decrypt them with `securejoin/<auth token>` as the secret or
even `securejoinv3/<alice's fingerprint>/<auth token>`. For existing
broadcast channels we cannot do this, but for securejoinv3 that is not
released yet this looks like an improvement that avoids at least this
problem.
Credits to link2xt for noticing the problem.
This also adds Alice's fingerprint to the auth tokens, which
was pretty easy to do. I find it hard to develop an intuition for
whether this is important, or whether we will be annoyed by it in the
future.
**Note:** This means that QR code scans will not work if one of the chat
partners uses a self-compiled core between c724e2981 and merging this PR
here. This is fine; we will just have to tell the other developers to
update their self-compiled cores.
The tests were originally generated with AI and then reworked.
Follow-up to https://github.com/chatmail/core/pull/7754 (c724e29)
This prevents the following attack:
/// Eve is subscribed to a channel and wants to know whether Alice is also subscribed to it.
/// To achieve this, Eve sends a message to Alice
/// encrypted with the symmetric secret of this broadcast channel.
///
/// If Alice sends an answer (or read receipt),
/// then Eve knows that Alice is in the broadcast channel.
///
/// A similar attack would be possible with auth tokens
/// that are also used to symmetrically encrypt messages.
///
/// To prevent this, a message that was unexpectedly
/// encrypted with a symmetric secret must be dropped.
encryption info needs a dedicated string for "Messages are end-to-end encrypted"
as the UI will add more infomation to the info messages,
smth. as "Tap for more information".
an alternative fix would have been to let the UI render the info-message
differently, but adding another string to core causes less friction.
If one of broadcast owner's devices didn't add a new subscriber for any reason, e.g. because of
missing SecureJoin messages, this device shall add "member added" messages when syncing the member
list from the `SetPgpContacts` message.
The test sometimes fails because of wrong message ordering for bob:
[...]
Waiting for the device of alice@example.org to reply… [NOTICED][INFO]
<Msg#2010🔒: (Contact#Contact#2001): hi [FRESH]
Msg#2008🔒: (Contact#Contact#2001): You joined the channel. [FRESH][INFO]
>Msg#2010🔒: (Contact#Contact#2001): hi [FRESH]
Msg#2011🔒: (Contact#Contact#2001): Member Me removed by alice@example.org. [FRESH][INFO]
This adds `SystemTime::shift(Duration::from_secs(1))` as a workaround.
Co-authored-by: Hocuri <hocuri@gmx.de>
dc_jsonrpc_init called Arc::from_raw on the account_manager pointer, which took ownership of the caller's refcount. When the local Arc dropped at the end of the function, the refcount was decremented, leaving the C side's pointer with a stolen refcount. This caused a use-after-free race between dc_accounts_unref and dc_jsonrpc_unref at shutdown.
Wrap in ManuallyDrop to prevent the implicit drop, keeping the caller's refcount intact.
Regression introduced in #7662.
This change is mainly to avoid exposing the write lock outside the pool module.
To avoid deadlocks, outside code should work only with the pooled connections
and use no more than one connection per thread.
With multiple transports there are multiple inbox loops on the same profile `Context`.
They tend to start running housekeeping at the same time, e.g. when deleting
a message with an attachment, and then `remove_unused_files()`
tries to remove the same files that are already deleted by another thread
and logs errors.
Don't use first-person form in placeholder texts,
as these can be misleading when broadcasted to group.
Additionally ensures that broadcasted system messages
are not localized to not leak locally-set language
to the group chat.
Fixes#7930
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
Don't depend on these 3 cleartext headers for the question whether we
download a message.
This PR will waste a bit of bandwidth for people who use the legacy
show_emails option; apart from that, there is no user-visible change
yet. It's a preparation for being able to remove these headers, in order
to further reduce unencrypted metadata.
Removing In-Reply-To and References will be easy; removing Chat-Version
must happen at least one release after the PR here is released, so that
people don't miss messages. Also, maybe some nerds depend on the
Chat-Version header for server-side filtering of messages, but we shall
have this discussion at some other time.
For the question whether a message should be moved, we do still depend
on them; this will be fixed with
https://github.com/chatmail/core/pull/7780.
When both this PR and #7780 are merged, we can stop requesting
Chat-Version header during prefetch.
I assume that the problem was that sometimes, alice2 or fiona doesn't
accept alice's smeared timestamp, because `calc_sort_timestamp()`
doesn't allow the timestamp of a received message to be in the future. I
tried this patch:
```diff
diff --cc src/chat.rs
index 9565437cf,9565437cf..a2e4f97d0
--- a/src/chat.rs
+++ b/src/chat.rs
@@@ -46,6 -46,6 +46,7 @@@ use crate::receive_imf::ReceivedMsg
use crate::smtp::{self, send_msg_to_smtp};
use crate::stock_str;
use crate::sync::{self, Sync::*, SyncData};
++use crate::timesmearing::MAX_SECONDS_TO_LEND_FROM_FUTURE;
use crate::tools::{
IsNoneOrEmpty, SystemTime, buf_compress, create_broadcast_secret, create_id,
create_outgoing_rfc724_mid, create_smeared_timestamp, create_smeared_timestamps, get_abs_path,
@@@ -1212,7 -1212,7 +1213,11 @@@ SELECT id, rfc724_mid, pre_rfc724_mid,
received: bool,
incoming: bool,
) -> Result<i64> {
-- let mut sort_timestamp = cmp::min(message_timestamp, smeared_time(context));
++ let mut sort_timestamp = cmp::min(
++ message_timestamp,
++ // Add MAX_SECONDS_TO_LEND_FROM_FUTURE in order to allow other senders to do timesmearing, too:
++ smeared_time(context) + MAX_SECONDS_TO_LEND_FROM_FUTURE,
++ );
let last_msg_time: Option<i64> = if always_sort_to_bottom {
// get newest message for this chat
```
...maybe this patch makes sense anyways, but you still get the problem
that the message sent by alice2 (i.e. the add-fiona message) will have
an earlier timestamp than the message sent by alice, because alice
already sent more messages, and therefore has more timesmearing-seconds.
It's unsure it makes sense to modify calc_sort_timestamp() this way because if some chat member has the clock in the future (even unintentionally), their fresh messages will be sorted to the bottom relatively to others' fresh messages. Maybe it's even better to limit the message timestamp ("Date") by the current system time there.
To really fix the problem, we could send a serial number together with the timestamp, that distinguishes two messages sent in the same second. But since we haven't gotten complaints about message ordering since some time, let's just leave things as they are.
Since all this timesmearing is a bit best-effort right now, I decided to
instead just make the test more relaxed.
fix https://github.com/chatmail/core/issues/7880
depends on #7754 (merged)
With this change, a securejoin message is just ignored if the contact
was deleted in the meantime; apparently the user is not interested in
the securejoin process anymore if they deleted the contact.
But other, parallel securejoin processes must not be affected; the test
also tests this.
This was initially done in the IMAP loop
to set is_chatmail for existing users.
They should all have the setting configured
by now unless they install some very old backup.
Setting during the configuration is needed
for Delta Chat Desktop because it caches the value
internally:
<https://github.com/deltachat/deltachat-desktop/issues/6068>
Close https://github.com/chatmail/core/issues/7396. Before reviewing,
you should read the issue description of
https://github.com/chatmail/core/issues/7396.
I recommend to review with hidden whitespace changes.
TODO:
- [x] Implement the new protocol
- [x] Make Rust tests pass
- [x] Make Python tests pass
- [x] Test it manually on a phone
- [x] Print the sent messages, and check that they look how they should:
[test_secure_join_group_with_mime_printed.txt](https://github.com/user-attachments/files/24800556/test_secure_join_group.txt)
- [x] Fix bug: If Alice has a second device, then Bob's chat won't be
shown yet on that second device. Also, Bob's contact isn't shown in her
contact list. As soon as either party writes something into the chat,
the that shows up and everything is fine. All of this is still a way
better UX than in WhatsApp, where Bob always has to write first 😂
Still, I should fix that.
- This is actually caused by a larger bug: AUTH tokens aren't synced if
there is no corresponding INVITE token.
- Fixed by 6b658a0e0
- [x] Either make a new `auth_tokens` table with a proper UNIQUE bound,
or put a UNIQUE bound on the `tokens` table
- [x] Benchmarking
- [x] TODOs in the code, maybe change naming of the new functions
- [x] Write test for interop with older DC (esp. that the original
securejoin runs if you remove the &v=3 param)
- [x] From a cryptography perspective, is it fine that vc-request is
encrypted with AUTH, rather than a separate secret (like INVITE)?
- [x] Make sure that QR codes without INVITE work, so that we can remove
it eventually
- [x] Self-review, and comment on some of my code changes to explain
what they do
- [x] ~~Maybe use a new table rather than reusing AUTH token.~~ See
https://github.com/chatmail/core/pull/7754#discussion_r2728544725
- [ ] Update documentation; I'll do that in a separate PR. All necessary
information is in the https://github.com/chatmail/core/issues/7396 issue
description
- [ ] Update tests and other code to use the new names (e.g.
`request-pubkey` rather than `request` and `pubkey` rather than
`auth-required`); I'll do that in a follow-up PR
**Backwards compatibility:**
Everything works seamlessly in my tests. If both devices are updated,
then the new protocol is used; otherwise, the old protocol is used. If
there is a not-yet-updated second device, it will correctly observe the
protocol, and mark the chat partner as verified.
Note that I removed the `Auto-Submitted: auto-replied` header from
securejoin messages. We don't need it ourselves, it's a cleartext header
that leaks too much information, and I can't see any reason to have it.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
There is no need to store copy of public key
next to the secret key because public key is a subset of the secret key
and can be obtained by using SignedSecretKey.public_key()
or SignedSecretKey.to_public_key().
Rpc.start() now calls get_system_info() after launching the server
to verify it started successfully. If the server exits early (e.g.
due to an invalid accounts directory), the core error message from
stderr is captured and included in the raised JsonRpcError.
The reader_loop now unblocks pending RPC requests when the server
closes stdout, so callers never hang on a dead server.
Export JsonRpcError from the deltachat_rpc_client package.
Add test_early_failure verifying that Rpc.start() raises with
the actual core error message for invalid accounts directories.
Timestamp renewal was introduced in 1dbf924c6a "feat:
chat::resend_msgs: Guarantee strictly increasing time in the Date header" so that re-sent messages
can be deduplicated on the reciver side, but the deduplication logic doesn't depend on "Date"
anymore.
fix https://github.com/chatmail/core/issues/7863
`test_import_encrypted_bak_into_encrypted_acct` CFFI test fails because
it tests that trying to import an encrypted account with a wrong
passphrase into an already-encrypted database will fail, but leave the
already-encrypted database (esp, leave it with the same password).
But in order to reset the database after a failed login attempt, I'm
using this code:
```rust
context.sql.close().await;
fs::remove_file(context.sql.dbfile.as_path())
.await
.log_err(context)
.ok();
context
.sql
.open(context, "".to_string()) // <-- NOTE THIS LINE
.await
.log_err(context)
.ok();
```
We're not remembering the password, so, we can't just pass the correct
password there.
Since password-protected databases are not really supported anyways, we
decided to just skip the test.
I also tried two tricks for deleting everything [found on
Stackoverflow](https://stackoverflow.com/questions/525512/drop-all-tables-command),
but neither of them managed to actually reset the database (i.e. they
led to a failed Rust test, because asserting
`!context2.is_configured().await?` failed):
```rust
context
.sql
.call_write(|conn| {
let mut stmt = conn.prepare(
"select 'drop table ' || name || ';' from sqlite_master where type = 'table';",
)?;
let mut iter = stmt.query(())?;
while iter.next()?.is_some() {}
Ok(())
})
.await
.log_err(context)
.ok();
context
.sql
.run_migrations(context)
.await
.log_err(context)
.ok();
```
```rust
context
.sql
.transaction(|t| {
t.execute_batch(
"
PRAGMA writable_schema = 1;
delete from sqlite_master where type in ('table', 'index', 'trigger');
PRAGMA writable_schema = 0",
)?;
Ok(())
})
.await
.log_err(context)
.ok();
context
.sql
.run_migrations(context)
.await
.log_err(context)
.ok();
```
---------
Co-authored-by: l <link2xt@testrun.org>
I sometimes find it hard to find the most recent migration.
This PR moves the migrations::run() function to the end of the file, so
that it's easy to find the most recent migration - it's at the end of
the file.
There are no changes except for switching the ordering of the functions.
The webxdc file name itself isn't informative for users. Still, send and display it if the webxdc
manifest can't be parsed, it's better than sending "Mini App" and this isn't a normal case anyway.
This code does not expect the variable to be unset,
so use indexing to fail with KeyError instead.
Otherwise getenv() returns None which is then converted to "none" string by formatting
and the test only fails because of connection attempts to "none" domain.
fix#7877
The bug was: If there is no chat description, and the chat description
is set to an empty string, the INSERT statement inserted a row with an
empty chat description, and therefore from the view of the INSERT
statement, something changed.
This PR fixes this by simply loading the chat description first, and
comparing it.
instead of Alice saying to Bob "You changed the chat description",
we now say "[Chat description changed, please update ...]
i was also considering to say "[Chat description changed to:\n\n...]"
but then there is no incentive for ppl to update, and chat descriptions
for chat creation would still be missing. and this is probably far more
often used.
successor of https://github.com/chatmail/core/pull/7829
Iroh-Gossip-Topic is sent in a post-message. Post-message goes to trash,
so topic should be associated with the existing pre-message that is
updated rather than with the post-message.
Fix https://github.com/chatmail/core/issues/7835.
The problem was most probably:
- `ac1_clone` receives the sync message, sends `TRANSPORTS_MODIFIED`
event, and launches a task that will restart IO
- After IO was stopped, but before it is started again,
`ac1_clone.add_transport_from_qr(qr)` is called
- this check fails:
```rust
ensure!(
!self.scheduler.is_running().await,
"cannot configure, already running"
);
```
fix https://github.com/chatmail/core/issues/7766
Implementation notes:
- Descriptions are only sent with member additions, when the description
is changed, and when promoting a previously-unpromoted group, in order
not to waste bandwith.
- Descriptions are not loaded everytime a chat object is loaded, because
they are only needed for the profile. Instead, they are in their own
table, and can be loaded with their own JsonRPC call.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Existing test relies on folder scanning.
We are going to remove the option to not show emails
(<https://github.com/chatmail/core/issues/7631>)
so the test will be removed eventually anyway.
This includes forwarding of long messages. Also this fixes sending, but more likely resending of
forwarded messages for which the original message was deleted, because now we save HTML to the db
immediately when creating a forwarded message.
Co-authored-by: Hocuri <hocuri@gmx.de>
It has a really complex logic, so it's better to avoid calling it if possible than think which side
effects and performance penalties it has. It was never called here before adding forwarding messages
across contexts (accounts).
For now, do this only for `OneOneChat` and `MailingListOrBroadcast`, this is enough to correctly
support messages from modern Delta Chat versions sending Intended Recipient Fingerprint subpackets
and single-recipient messages from modern versions of other MUAs.
It only looks up contacts by address in the given chat, so `_fallback_to_chat` suffix is more
informative. It's obvious that such a lookup is done by address, there are no other reasonable
options.
This is an important thing forgotten to be checked in 332527089.
Also there's another test which currently doesn't work as we want: outgoing encrypted messages
continue to arrive to ad-hoc group even if we already have contact's key. This should be fixed by
sending and receiving Intended Recipient Fingerprint subpackets.
The good thing is that apparently there are no scenarios requiring the contact to update their
software, the user should just update own devices.
This is not possible for webxdcs and vCards currently however, so add workarounds for them:
- Use translated "Mini App" as the webxdc name.
- Use just "👤" instead of the vCard summary (i.e. the vCard contact name).
I.e. create a non-replyable ad-hoc group in such cases. Unencrypted replies to encrypted messages
are a security issue and "Composing a Reply Message" from RFC 9787 and "Replying and Forwarding
Guidance" from RFC 9788 forbid such replies.
If From is an address-contact, it mustn't be able to modify an encrypted group. If From is a
key-contact, it mustn't be added to members of an unencrypted group.
See: https://support.delta.chat/t/high-quality-avatar/1052/8
This removes an additional resolution-reduction for images that fit
within the resolution-limit.
The additional reduction of the image-resolution was added in
b7864f232b.
It seems that back then only resolution-reduction was used to make the
file-size of images smaller, and the purpose was to skip an unnecessary
encoding-step.
However, encoding images with [jpeg-quality set to
`75`](a6b2a54e46/src/blob.rs (L392)),
as it is currently done, can significantly reduce the file-size, even if
the resolution is the same. As the resolution of the image that will be
encoded is rather low when it fits within the resolution-limit, further
reducing it can significantly reduce the quality of the image.
Thus it is better to not skip the encoding-step at the original
resolution.
this PR cleans up with some easy, deprecated stuff.
i roughly checked that they are no longer in use - by these checks,
other deprecated function were kept, eg.
dc_provider_new_from_email_with_dns() and dc_chat_is_protected() is
still used by deltatouch - to not add noise there, we remove them here
on the next cleanup ...
for DC_STR_*, however, if they are in used, the corresponding lines
should just be removed
this will cleanup https://c.delta.chat/deprecated.html as well
resolves#7724: When forwarding a message with file to another profile, the file was not copied to the target $blobdir and so the forwarded message missed it
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
If `Message::load_from_db_optional()` or `set_msg_failed()` fails, we
shouldn't early-return. Because it's important that the line
`execute("DELETE FROM smtp WHERE id=?", (rowid,))` is executed in order
to prevent an infinite loop, if one of these functions fails.
Otherwise if the user reads messages being offline and then the device comes online, sent MDNs will
remove all notifications from other devices even if new messages have arrived. Notifications not
removed at all look more acceptable.
This PR fixes a bug that old channel members were remembered in the
database even after they left the channel. Concretely, they remained in
the `past_members` table that's only meant for groups.
Though it was not a bad bug; we're anyways not cleaning up old contacts.
There is no difference for RSA and Ed25519,
the only signing keys that we generate.
The both use SHA256:
<7e3b6c0af2/src/types/params/public.rs (L231-L234)>
The only difference is for the possible future PQC signing keys
and imported NIST P-512 and NIST P-384 keys.
We currently synchronize "seen" status of messages by setting `\Seen` flag on IMAP and then looking
for new `\Seen` flags using `CONDSTORE` IMAP extension. This approach has multiple disadvantages:
- It requires that the server supports `CONDSTORE` extension. For example Maddy does not support
CONDSTORE yet: https://github.com/foxcpp/maddy/issues/727
- It leaks the seen status to the server without any encryption.
- It requires more than just store-and-forward queues and prevents replacing IMAP with simpler
protocols like POP3 or UUCP or some HTTP-based API for queue polling.
A simpler approach is to send MDNs to self when `Config::BccSelf` (aka multidevice) is enabled,
regardless of whether the message requested and MDN. If MDN was requested and we have MDNs enabled,
then also send to the message sender, but MDN to self is sent regardless of whether read receipts
are actually enabled.
`sync_seen_flags()` and `CONDSTORE` check is better completely removed, maybe after one
release. `store_seen_flags_on_imap()` can be kept for unencrypted non-chat messages.
One potential problem with sending MDNs is that it may trigger ratelimits on some providers and
count as another recipient.
Hostname resolution may timeout if DNS servers are not responding.
It is also not necessary to resolve fallback ICE server hostnames
if the user is not going to use calls.
If the user enables statistics-sending in the advanced settings, they
will be asked whether they also want to take part in a survey. We use a
short ID to then link the survey result to the sent statistics.
However, the survey website didn't like our base64 ids, especially the
fact that the id could contain a `-`. This PR makes it so that the id
only contains lowecase letters.
This commit also makes testing hooks easier, as it allows to process
events and run hooks on them, until a certain event occurs.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
84161f4202 promotes group members to `Origin::IncomingTo` when
accepting it, instead of `CreateChat` as before, but this changes almost nothing because it happens
rarely that the user only accepts a group and writes nothing there soon. Now if a message has
multiple recipients, i.e. it's a 3-or-more-member group, or if it's a broadcast message, we don't
scale up its recipients to `Origin::OutgoingTo`.
closes#6945
### Why not deprecate it instead?
Because empty contact list is a more annoying-to-find bug in your app
than failing to build or getting undefined at runtime.
Also you would not see the deprecated hints anyway because for that you
need autogenerated types and those only exist for typescript currently.
Before this PR, when a user with current main sends a large message to a
user with an old Delta Chat (before #7431), the text will be duplicated:
One message will arrive with only the text, and one message with
attachment+text.
This PR changes this - there will be one message with only the text, and
one message with only the attachment.
If we want to guard against lost pre-messages, then we can revert this
PR in a few months, though I'm not sure that's necessary - it's unlikely
that the small pre-message gets lost but the big post-message gets
through.
Also remove "you can now call 'configure'" from the REPL output, probably users of the REPL tool can
read the code documentation to know when 'configure' should be run.
Before this PR there were cases where error messages never reach the
user because logging was not initialized yet.
This PR moves log initialization to the start of the program, so that
all logged messages reach the user.
Otherwise it's not possible to write tests reliably because sync messages may be executed multiple
times if they arrive from different transports. This should fix flaky
`test_transport_synchronization`.
Also always emit `TransportsModified` if the primary transport is changed by a sync message, even if
it doesn't contain `SyncData::Transports`.
Also don't decrease `add_timestamp` in `save_transport()` if nothing else changes, this doesn't make
sense.
This way, if the sync message updates transports, the check for a new primary transport is done
against the updated transport list which is more reliable.
This fixes the bug when a new transport doesn't become primary on the 2nd device because INBOX from
the new transport isn't fully fetched. Now the `Transports` sync message is received from the old
transport, but as it has updated "From", it updates the primary transport correspondingly. NB: I/O
for the new primary transport isn't immediately started however, this needs a separate fix.
This is to fix tests failing with `OSError: [Errno 9] Bad file descriptor`. Maybe stdout closes
earlier than stderr, before the test finishes, not sure. For reference, the previous commit removing
print()s is 800edc6fce.
- DC_CHAT_TYPE_BROADCAST does no longer exist
- DC_CHAT_TYPE_MAILINGLIST is no longer always read-only
- avoid double docs by moving everything to DC_CHAT_TYPE group
(here, things were also updated recently)
Otherwise wrong IMAP client corresponding to a different transport
may pick up the job to mark the message as seen there,
and fail to do it as the message does not exist.
It may also mark the wrong message with the correct folder
and UID, but wrong IMAP server.
There was a comment that group messages should always be downloaded to avoid inconsistent group
state, but this is solved by the group consistency algo nowadays in the sense that inconsistent
group state won't spread to other members if we send to the group.
At urgent request from @hpk42, this adds a config option `team_profile`.
This option is only settable via SQLite (not exposed in the UI), and the
only thing it does is disabling synchronization of seen status.
I tested manually on my Android phone that it works.
Not straigthforward to write an automatic test, because we want to test that something
does _not_ happen (i.e. that the seen status is _not_ synchronized), and
it's not clear how long to wait before we check.
Probably it's fine to just not add a test.
This is what I tried:
```python
@pytest.mark.parametrize("team_profile", [True, False])
def test_markseen_basic(team_profile, acfactory):
"""
Test that seen status is synchronized iff `team_profile` isn't set.
"""
alice, bob = acfactory.get_online_accounts(2)
# Bob sets up a second device.
bob2 = bob.clone()
bob2.start_io()
alice_chat_bob = alice.create_chat(bob)
bob.create_chat(alice)
bob2.create_chat(alice)
alice_chat_bob.send_text("Hello Bob!")
message = bob.wait_for_incoming_msg()
message2 = bob2.wait_for_incoming_msg()
assert message2.get_snapshot().state == MessageState.IN_FRESH
message.mark_seen()
# PROBLEM: We're not waiting 'long enough',
# so, the 'state == MessageState.IN_SEEN' assertion below fails
bob.create_chat(bob).send_text("Self-sent message")
self_sent = bob2.wait_for_msg(EventType.MSGS_CHANGED)
assert self_sent.get_snapshot().text == "Self-sent message"
if team_profile:
assert message2.get_snapshot().state == MessageState.IN_FRESH
else:
assert message2.get_snapshot().state == MessageState.IN_SEEN
```
Adds an api to get all ui config keys. There already is an option to get
all normal config keys (`"sys.config_keys"`), but before this pr there
was no way to get all `ui.*` config keys.
#### Why is this api needed?
For webxdc cleanup on desktop, which stores window position in a ui
config key (such as `ui.desktop.webxdcBounds.676464`) as soon as a
webxdc is opened since many versions now. So listing all ui keys is a
good way for us to find out which webxdc may have web data stored.
unfortunately electron does not (yet?) have a way to list all origins
that have web-data like android does, so this is the next best thing we
can do before itterating all possible ids, see also
https://github.com/deltachat/deltachat-desktop/issues/5758.
#### Why is this only a jsonrpc api and not another special/virtual
config key like `"sys.config_keys"`?
r10s indicated that `ui.*`-config keys are barely used
(https://github.com/deltachat/deltachat-desktop/issues/5790#issuecomment-3598512802),
so I thought it makes more sense to add it as dedicated api which's
existentence is checked by the typechecker, so it will be easier to not
miss it when we should remove the api again in the future.
But we could also do a dedicated special/virtual config key for it, if
you think that is better, this is easy to change.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
- webxdc notify specifically to Bob: notifies Bob even when chat is
muted
- webxdc notify to everyone ("*"): notifies Bob if he does not have the
chat muted
This aligns with how we handle notifications with quote replies.
---------
Co-authored-by: link2xt <link2xt@testrun.org>
It is a follow-up to https://github.com/chatmail/core/pull/7643
Event is not emitted when the transports are modified on this device
and we should consistently say that this event is not only for testing.
`Config::OnlyFetchMvbox` should be checked before `MvboxMove` because the latter makes no sense in
presense of `OnlyFetchMvbox` and even grayed out in the UIs in this case. Otherwise users will see
an error mentioning the wrong setting.
When accepting a chat, its members are promoted to `Origin::CreateChat`, but for groups it makes
sense to use lower origin because users don't always check all members before accepting a chat and
may not want to have the group members mixed with existing contacts. `IncomingTo` fits here by its
definition: "additional To:'s of incoming message of known sender", i.e. we assume that the sender
of some message is known to the user. This way we can show contacts coming from groups in the bottom
of contact list, maybe even add some separator later. It makes sense not to hide such contacts
completely, otherwise if the user remembers the contact name, but not the chat it's a member of, it
would be difficult to find the contact.
This records the curent behavior: after accepting a group with unknown contacts the contact list
contains them mixed with contacts for which 1:1 chats exist, i.e. new contacts from the group are
neither hidden nor sorted down.
All users of this function expect a nonempty list to be returned, otherwise they fail with hard to
understand errors like "No connection attempts were made", so it's better to fail early if DNS
resolution failed and the cache doesn't contain resolutions as well.
#7587 removed "used_account_settings" and "entered_account_settings"
from Context.get_info(). link2xt pointed out that
entered_account_settings can still be useful to debug login issues, so
tis pr adds logs both on failed configuration attempts.
example warning event:
```
Warning src/configure.rs:292: configure failed: entered params myself@merlinux.eu imap:unset:***:unset:0:Automatic:AUTH_NORMAL smtp:unset:0:unset:0:Automatic:AUTH_NORMAL cert_automatic, used params myself@merlinux.eu imap:[mailcow.testrun.org:993:tls:myself@merlinux.eu, mailcow.testrun.org:143:starttls:myself@merlinux.eu] smtp:[mailcow.testrun.org:465:tls:myself@merlinux.eu, mailcow.testrun.org:587:starttls:myself@merlinux.eu] provider:none cert_automatic
```
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
We got a report that application is not responding after update
on Android and getting killed before it can start,
suspected to be a slow SQL migration:
<https://github.com/chatmail/core/issues/7602>
This change removes calculation of normalized names for
existing chats and contacts added in
<https://github.com/chatmail/core/pull/7548>
to exclude the possibility of this migration being slow.
New chats and contacts will still get normalized names
and all chats and contacts will get it when they are renamed.
And don't add a `SecurejoinWait` info message at all if we know Alice's key from the start. If we
don't remove this info message, it appears in the chat after "Messages are end-to-end encrypted..."
which is quite confusing when Bob can already send messages to Alice.
the ringing of 60 seconds is indeed a bit short,
esp. when messages are already delayed.
but also if everythings is fast and working, 60 seconds are short.
the 60 seconds also come from the first implementation,
where we did not had a "reject" message from callee to caller,
so that caller should not wait unnecessarily.
this has changed,
however, the other constraints are still valid -
phone may get offline, and we should not ring for stale calls.
sure, that can be fixed also differently,
but for the current implementation,
the 120 seconds seem to be a good compromise.
This adds information of all used transports to `Context.get_info` in
the key `used_transport_settings`.
The format is the same as in `used_account_settings`. The new property
also contains the primary account, so we could remove
`used_account_settings` now, though there is also
`entered_account_settings` in which it stays in relation.
- there is an alternative pr at
https://github.com/chatmail/core/pull/7584, which gives each transport
it's own key, which improves readability.
closes#7581
The events are needed when you are not using chatmail core from rust, if
you use chatmail core from your rust bot or from tauri, then you likely
already use the rust logging/tracing ecosystem. So it makes sense to use
it instead of listening to the events and logging them yourself.
This pr fixes a few cases where the event was direclty emitted instead
of using the macro and thus was not also automatically logged via
tracing.
Add `chat::forward_msgs_2ctx()` which takes another context as a parameter and forwards messages to
it and its jsonrpc wrapper `CommandApi::forward_messages_to_account()`.
We periodically forget to remove new params from forwarded messages as this can't be catched by
existing tests, some examples:
bfc08abe88a1837aeb8c56b2361f01
This may leak confidential data. Instead, it's better to explicitly list params that we want to
forward, then if we forget to forward some param, a test on forwarding messages carrying the new
functionality will break, or the bug will be reported earlier, it's easier to notice that some info
is missing than some extra info is leaked.
Fix get_secondary_addrs() which was using
`secondary_addrs` config that is not updated anymore.
Instead of using `secondary_addrs` config,
use the `transports` table which contains all the addresses.
This makes `Contact::get_all()` and `Chatlist::try_load()` case-insensitive for non-ASCII chat and
contact names as well. The same approach as in f6f4ccc6ea "feat:
Case-insensitive search for non-ASCII messages (#5052)" is used: `chats.name_normalized` and
`contacts.name_normalized` colums are added which store lowercased/normalized names (for a contact,
if the name is unset, it's a normalized authname). If a normalized name is the same as the
chat/contact name, it's not stored to reduce the db size. A db migration is added for 10000 random
chats and the same number of the most recently seen contacts, for users it will probably migrate all
chats/contacts and for bots which may have more data it's not important.
Many clients don't send it currently, so it is unlikely that servers depend on it:
https://mastodon.social/@cks/114690055923939576.
For "implicit TLS", do not turn it off yet, it will serve as a fallback in case of rare server that
needs it. If the server only supports STARTTLS and requires SNI then it is really weird, likely
should not happen.
Migration 140 merged in version 2.28.0
introduced `NOT NULL` transport_id
columns, so old versions of core not aware of it
fail e.g. when they expect conflict on `(folder)`
column rather than `(transport_id, folder)`.
Currently there is a race between transports
to upload sync messages and delete them from `imap_send` table.
Sometimes mulitple transports upload the same message
and sometimes only some of them "win".
With this change only the primary transport
will upload the sync message.
We do not have transport ID assigned until configuration finishes,
so we cannot save UID validitiy for folders and write
into any tables that have transport_id column yet.
This effectively readds the old `imap_rfc724_mid` built only on `rfc724_mid`, otherwise
`sql::prune_tombstones()` which is called from `housekeeping()` becomes very slow because of no
suitable index.
`chat::set_chat_profile_image()` already checks that the group has grpid, still it makes sense to
check that a message is encrypted when sending, in case if the chat has a profile image in the db
for some reason.
This updates `scripts/remote_tests_{rust,python}.sh`.
The scripts were previously used to run tests
from CI on remote faster machine,
but they are still usable to run tests remotely
e.g. from a laptop that is on battery.
There are quite some unneeded stock strings; this PR removes some of
them. None of these stock strings were actually set by the UI, or even
have translations in Transifex.
- We don't have AEAP anymore.
- The "I added/removed member" and "I left the group" strings are
anyways not meant to be shown to the user. Also, starting to translate
them now would leak the device language.
BREAKING CHANGE: This can theoretically be a breaking change because a
UI could reference one of the removed stock strings, so I marked it as
breaking just in case.
- Db encryption does nothing with blobs, so fs/disk encryption is recommended.
- Isolation from other apps is needed anyway.
- Experimental database encryption was removed on iOS and Android.
- Delta Touch is using CFFI API with a manually entered password because Ubuntu Touch does not offer
filesystem or disk encryption, but we don't want new users of these APIs, such as bot developers.
This API allows to check if the message with
given ID exists and distinguish between
message not existing and database error.
It might also be faster than
checking messages one by one
if multiple messages need to be checked
because of using a single SQL transaction.
Add a stock string `%1$s invited you to join this channel.\n\nWaiting
for the device of %2$s to reply…`, which is shown when a user starts to
join a channel.
I did _not_ add an equivalent to `%1$s replied, waiting for being added
to the group…`, which is shown when vg-auth-required was received. I
don't think that this would add any information that's interesting to
the user, other than 'something is happening, hang on'. And the more
text on the screen, the less likely that anyone reads it. But if others
think differently, we can also add it.
With this PR, joining a channel looks like this:
```
Msg#2003: info (Contact#Contact#Info): Messages are end-to-end encrypted. [NOTICED][INFO]
Msg#2004: info (Contact#Contact#Info): Alice invited you to join this channel.
Waiting for the device of Alice to reply… [NOTICED][INFO]
Msg#2007🔒: (Contact#Contact#2001): You joined the channel. [FRESH][INFO]
```
Before, outgoing self-sent unencrypted messages were assigned to the self-chat. Now we assign them
to ad-hoc groups with only SELF instead of 1:1 chats with address contacts corresponding to our own
addresses because we don't want to create such address contacts; we still use SELF for `from_id` of
such messages. Not assigning such messages to the encrypted chat should be safe enough and such
messages can actually be sent by the user from another MUA.
Fix#7435
For most messages, `calc_sort_timestamp()` makes sure that they are at the correct place; esp. that they are not above system messages or other noticed/seen messages.
Most callers of `add_info_msg()`, however, didn't call `calc_sort_timestamp()`, and just used `time()` or `smeared_time()` to get the sort timestamp. Because of this, system messages could sometimes wrongly be sorted above other messages.
This PR fixes this by making the sort timestamp optional in `add_info_msg*()`. If the sort timestamp isn't passed, then the message is sorted to the bottom of the chat. `sent_rcvd_timestamp` is not optional anymore, because we need _some_ timestamp that can be shown to the user (most callers just pass `time()` there).
It must be verified by "unknown verifier" instead. But if the verifier has known verifier in turn,
it must reverify contacts having unknown verifier. Add a check for this also.
New APIs are JSON-RPC method stop_background_fetch(),
Rust method Accounts.stop_background_fetch()
and C method dc_accounts_stop_background_fetch().
These APIs allow to cancel background fetch early
even before the initially set timeout,
for example on Android when the system calls
`Service.onTimeout()` for a `dataSync` foreground service.
And enable it by default as the standard Header Protection is backward-compatible.
Also this tests extra IMF header removal when a message has standard Header Protection since now we
can send such messages.
Omit Legacy Display Elements from "text/plain" and "text/html" (implement 4.5.3.{2,3} of
https://www.rfc-editor.org/rfc/rfc9788 "Header Protection for Cryptographically Protected Email").
Regardless of whether chatmail relay is used or not,
bcc_self should be enabled when second device is added.
It should also be enabled again even if the user
has turned it off manually.
adb brought this up in an internal discussion.
With the recent introduction of channels it becomes easier to hit the
limit
and it becomes impossible to send messages to a channel with more than
1000 members, this pr fixes that.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
Created new test_folders.py
Moved existing JSON-RPC tests:
- test_reactions_for_a_reordering_move
- test_delete_deltachat_folder
Ported tests:
- test_move_works_on_self_sent
- test_moved_markseen
- test_markseen_message_and_mdn
- test_mvbox_thread_and_trash (renamed to test_mvbox_and_trash)
- test_scan_folders
- test_move_works
- test_move_avoids_loop
- test_immediate_autodelete
- test_trash_multiple_messages
The change also contains fixes for direct_imap fixture
needed to use IMAP IDLE in JSON-RPC tests.
`Contact::get_color()` returns gray if own keypair doesn't exist yet and we don't want any UIs
displaying it. Keypair generation can't be done in `get_color()` or `get_by_id_optional()` to avoid
breaking Core tests on key import. Also this makes the API clearer, pure getters shouldn't modify
any visible state.
By the time you scan the QR code,
inviter may not be in the group already.
In this case securejoin protocol will never complete.
If you then join the group in some other way,
this results in you implicitly adding that inviter
to the group.
For multiple transports we will need to run
multiple IMAP clients in parallel.
UID validity change detected by one IMAP client
should not result in UID resync
for another IMAP client.
Actually it will be not as breaking if you used the constants, because
this pr also changes the constants.
closes#7029
Note that I had to change the constants from enum to namespace, this has
the side effect, that you can no longer also use the constants as types,
you need to instead prefix them with `typeof ` now.
- sort garbage to the beginning, readable text to the end
- instead of `%20`, make use of `+` to encode spaces
- shorter invite links and smaller QR codes by truncation of the names
the truncation of the name uses chars() which does not respect grapheme clusters, so
that last character may be wrong. not sure if there is a nice and easy
alternative, but maybe it's good engoug - the real, full name will come
over the wire (exiting truncate() truncates on word boundaries, which is
maybe too soft here - names may be long, depending on the language, and
not contain any space)
moreover, this resolves the "name too long" issue from
https://github.com/chatmail/core/issues/7015
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
We use query_and_then() instead of query_map() function now.
The difference is that row processing function
returns anyhow::Result, so simple fallible processing
like JSON parsing can be done inside of it
when calling query_map_vec() and query_map_collect()
without having to resort to query_map()
and iterating over all rows again afterwards.
Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.
This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.
- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)
## Backwards compatibility
This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:
> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
>
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again.
>
> Here is what to do:
> • Create a new channel
> • Tap on the channel name
> • Tap on "QR Invite Code"
> • Have all recipients scan the QR code, or send them the link
>
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.
## The symmetric encryption
Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).
The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).
Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].
## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code
This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.
The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.
[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`
BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)
`login_param` module is now for user-visible entered login parameters,
while the `transport` module contains structures for internal
representation of connection candidate list
created during transport configuration.
This is similar to old `dcaccount:` with URL,
but creates a 9-character username on the client
and avoids making an HTTPS request.
The scheme is reused to avoid the apps
needing to register for the new scheme.
`http` support is removed because it was
not working already, there is a check
that the scheme is `https` when the URL
is actually used and the core has
no way to make HTTP requests without TLS.
It's used in `fetch_existing_msgs()`, but we can remove it and tell users that they need to
move/copy messages from Sentbox to Inbox so that Delta Chat adds all contacts from them. This way
users will be also informed that Delta Chat needs users to CC/BCC/To themselves to see messages sent
from other MUAs.
The motivation is to reduce code complexity, get rid of the extra IMAP connection and cases when
messages are added to chats by Inbox and Sentbox loops in parallel which leads to various message
sorting bugs, particularly to outgoing messages breaking sorting of incoming ones which are fetched
later, but may have a smaller "Date".
We already have both rand 0.8 and rand 0.9
in our dependency tree.
We still need rand 0.8 because
public APIs of rPGP 0.17.0 and Iroh 0.35.0
use rand 0.8 types in public APIs,
so it is imported as rand_old.
Flakiness was introduced in e7348a4fd8.
This change removes a call to joining_chat_id() which created a chat,
now we check for existing group chat
without creating it if it does not exist.
Context: PR #7116 is backwards-incompatible with versions older than
v2.21, and since the release hasn't reached all users yet, we currently
can't release from main; for details see #7326.
Issue #7326 explains how we can make this less breaking, but this only
works if many contacts are verified. So, this PR here proposes to
postpone the stricter rules for who is verified a bit:
- Set verification timeout for invite codes to 1 week (this is still
stricter than no timeout at all, which we had in the past)
- Don't reset indirect verifications yet
In a few months (when everyone has v2.22.0), we can revert the PR here,
then.
---------
Co-authored-by: l <link2xt@testrun.org>
We do not try to delete resent messages
anymore. Previously resent messages
were distinguised by having duplicate Message-ID,
but future Date, but now we need to download
the message before we even see the Date.
We now move the message to the destination folder
but do not fetch it.
It may not be a good idea to delete
the duplicate in multi-device setups anyway,
because the device which has a message
may delete the duplicate of a message
the other device missed.
To avoid triggering IMAP busy move loop
described in the comments
we now only move the messages
from INBOX and Spam folders.
Make it return the correct value for non-Group chats.
This also should improve performance, thanks to the fact that
we now don't have to query all the chat's contacts.
Instead we only need confirm that self-contact
is among the group members, and only when the chat type is `Group`.
tokio-tar is unmaintained and has unpatched CVE-2025-62518.
More details on CVE are in <https://edera.dev/stories/tarmageddon>.
tokio-tar is only used for transferring backups
and worst case is that by manually inspecting
a carefully crafted backup user will not see
the same files as get unpacked when importing a backup.
This way, the statistics / self-reporting bot will be made into an
opt-in regular sending of statistics, where you enable the setting once
and then they will be sent automatically. The statistics will be sent to
a bot, so that the user can see exactly which data is being sent, and
how often. The chat will be archived and muted by default, so that it
doesn't disturb the user.
The collected statistics will focus on the public-key-verification that
is performed while scanning a QR code. Later on, we can add more
statistics to collect.
**Context:**
_This is just to give a rough idea; I would need to write a lot more
than a few paragraphs in order to fully explain all the context here_.
End-to-end encrypted messengers are generally susceptible to MitM
attacks. In order to mitigate against this, messengers offer some way of
verifying the chat partner's public key. However, numerous studies found
that most popular messengers implement this public-key-verification in a
way that is not understood by users, and therefore ineffective - [a 2021
"State of Knowledge" paper
concludes:](https://dl.acm.org/doi/pdf/10.1145/3558482.3581773)
> Based on our evaluation, we have determined that all current E2EE
apps, particularly when operating in opportunistic E2EE mode, are
incapable of repelling active man-in-the-middle (MitM) attacks. In
addition, we find that none of the current E2EE apps provide better and
more usable [public key verification] ceremonies, resulting in insecure
E2EE communications against active MitM attacks.
This is why Delta Chat tries to go a different route: When the user
scans a QR code (regardless of whether the QR code creates a 1:1 chat,
invites to a group, or subscribes to a broadcast channel), a
public-key-verification is performed in the background, without the user
even having to know about this.
The statistics collected here are supposed to tell us whether Delta Chat
succeeds to nudge the users into using QR codes in a way that is secure
against MitM attacks.
**Plan for statistics-sending:**
- [x] Get this PR reviewed and merged (but don't make it available in
the UI yet; if Android wants to make a release in the meantime, I will
create a PR that removes the option there)
- [x] Set the interval to 1 week again (right now, it's 1 minute for
testing)
- [ ] Write something for people who are interested in what exactly we
count, and link to it (see `TODO[blog post]` in the code)
- [ ] Prepare a short survey for participants
- [ ] Fine-tune the texts at
https://github.com/deltachat/deltachat-android/pull/3794, and get it
reviewed and merged
- [ ] After the next release, ask people to enable the
statistics-sending
If we use modules (which are actually namespaces), we can use shorter names. Another approach is to
only use modules for internal code incapsulation and use full names like deltachat-ffi does.
Create unprotected group in test_create_protected_grp_multidev
The test is renamed accordingly.
SystemMessage::ChatE2ee is added in encrypted groups
regardless of whether they are protected or not.
Previously new encrypted unprotected groups
had no message saying that messages are end-to-end encrypted
at all.
This mechanism replaces `Chat-Verified` header.
New parameter `_verified=1` in `Autocrypt-Gossip`
header marks that the sender has the gossiped key
verified.
Using `_verified=1` instead of `_verified`
because it is less likely to cause troubles
with existing Autocrypt header parsers.
This is also how https://www.rfc-editor.org/rfc/rfc2045
defines parameter syntax.
MX record lookup was only used to detect Google Workspace domains.
They can still be configured manually.
We anyway do not want to encourage creating new profiles
with Google Workspace as we don't have Gmail OAUTH2 token anymore
and new users can more easily onboard with a chatmail relay.
This allows to send send messages when disconnected
using `send_msg_sync` to create a dedicated connection
and send a single message.
The command can be used for measuring
the duration of SMTP connection establishment.
darwin.apple_sdk.frameworks is deprecated and does nothing according to
<https://nixos.org/manual/nixpkgs/stable/#sec-darwin-legacy-frameworks>
libiconv dependency also does not seem to be needed for `libdeltachat`,
I have checked that `nix build .#libdeltachat` still succeeds on macOS.
Before, the color was computed from the address, but as we've switched to fingerprint-based contact
colors, this logic became stale. Now `deltachat::contact::get_color()` is used. A test would be nice
to have, but as now all the logic is in Core, this isn't critical as there are Core tests at least.
Any control information from the message
should only be downloaded when the message
is fully downloaded to avoid processing it twice.
Besides, "partial" messages may actually be full messages
with an error that are only processed as partial
to add a message bubble allowing to download the message later.
In-Reply-To may refer to non-call message
as we do not control the sender.
It may also happen that call message
was received by older version and processed
as text, in which case correct In-Reply-To
appears to be referring to the text message.
Emitting an `AccountsItemChanged` event is needed for UIs to know when `Contact::get_color()` starts
returning the true color for `SELF`. Before, an address-based color was returned for a new account
which was changing to a fingerprint-based color e.g. on app restart. Now the self-color is our
favorite gray until own keypair is generated or imported e.g. via ASM.
It really depends on the screen,
but on Android phones with higher DPI
than laptop screen the contrast
looks a bit too low.
It should not be increased too high
because of clipping.
I have not updated Python interpreters for legacy bindings
in scripts/run_all.sh because I have not checked
if manylinux images already exist and work for building wheels.
Legacy Python bindings are deprecated
and we don't publish new releases for them.
Old Delta Chat clients don't provide timestamps for added and removed members in messages, so at
least implicit member changes may be ignored as it's not clear if they are newer than explicit
member changes from modern clients. Lost messages aren't so frequent anyway, and overall
compatibility with old versions may be limited already.
Determinate Systems GitHub action
installs Nix version from Determinate Systems
and Determinate Nix Installer has
dropped support for installing upstream Nix:
https://determinate.systems/blog/installer-dropping-upstream/
This commit switches to upstream Nix
to avoid accidentally depending on any features
of Determinate Nix.
With newer Iroh it is possible to obtain
own node address before home relay is selected
and accidentally send own address without relay URL.
It took me some time to debug why Iroh 0.92.0
did not work with iroh-relay 0.92.0
so I'm adding these assertions even
while we still use Iroh 0.35.0.
I have logs from a user where messages are prefetched for long minutes, and while it's not a problem
on its own, we can't rely that the connection overlives such a period, so make
`fetch_new_messages()` prefetch (and then actually download) messages in batches of 500 messages.
Regarding the `@deprecated` addition:
it's not clear to me why the `listAccounts` function exists,
why `getAllAccounts` gets special treatment.
It has been there ever since the introduction of JSON-RPC.
Maybe it's because of the existence of `getContextEvents`,
which has to do with accounts.
But hopefully the new doc on `getContextEvents`
compensates for this.
this PR clarifies some events, also updating to recent changes.
see changed code for details
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
In messages with text/plain, text/html and text/calendar parts
within a multipart/alternative, text/calendar part is currently ignored,
text/plain is displayed and HTML is available via HTML API.
Ignoring `receive_imf_inner()` errors, i.e. silently skipping messages on failures, leads to bugs
never fixed. As for temporary I/O errors, ignoring them leads to lost messages, in this case it's
better to bubble up the error and get the IMAP loop stuck. However if there's some logic error, it's
better to show it to the user so that it's more likely reported, and continue receiving messages. To
distinguish these cases, on error, try adding the message as partially downloaded with the error set
to `msgs.error`, this way the user also can retry downloading the message to finally see it if the
problem is fixed.
DKIM-Signatures apply to the last headers, so start from the last header and take a valid one,
i.e. skip headers having unknown critical attributes, etc. Though this means that Autocrypt header
must be "oversigned" to guarantee that a not DKIM-signed header isn't taken, still start from the
last header for consistency with processing other headers. This isn't a security issue anyway.
Some Delta Chat clients (Desktop, for example)
do `leave_webxdc_realtime`
regardless of whether we've ever joined a realtime channel
in the first place. Such as when closing a WebXDC window.
This might result in unexpected and suspicious firewall warnings.
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Actually this leads to fetching messages only from watched folders and Spam. Motivation:
- At least Gmail has virtual folders which aren't correctly detected as such, e.g. "Sent".
- At least Gmail has many virtual folders and scanning all of them takes significant time, 5-6 secs
in median for me. This slows down receiving new messages and consumes battery.
- Delta Chat shouldn't fetch messages from folders potentially created by other apps for their own
purposes. NB: All compatible Delta Chat forks should use the "DeltaChat" folder as mvbox.
- Fetching from folders that aren't watched, e.g. from "Sent", may lead to message ordering issues.
SDP offer and answer contain newlines.
Without the fix these newlines are not encoded at all
and break the header into multiple headers
or even prevent parsing of the following headers.
Quoting @adbenitez:
> I have been using the SecurejoinInviterProgress event to show a
welcome message when user scan the QR/link of the bot (== starts a chat
with the bot)
> but this have a big problem: in that event all you know is that a
contact completed the secure-join process, you don't know if it was via
certain 1:1 invite link or a group invitation, then a group-invite bot
would send you a help message in 1:1 every time you join a group with it
Since it's easy enough to add this information to the
SecurejoinInviterProgress event, I wrote a PR to do so.
This is a preparation for expiring authentication tokens.
If we make authentication token expire,
we need to generate new authentication tokens each time
QR code screen is opened in the UI,
so authentication token is fresh.
We however don't want to completely invalidate
old authentication codes at the same time,
e.g. they should still be valid for joining groups,
just not result in a verification on the inviter side.
Since a group now can have a lot of authentication tokens,
it is easy to lose track of them
without any way to remove them
as they are not displayed anywhere in the UI.
As a solution, we now remove all
tokens corresponding to a group ID
when one token is withdrawn,
or all non-group tokens
when a single non-group token is withdrawn.
"Reset QR code" option already present
in the UI which works by resetting
current QR code will work without any UI changes,
but will now result in invalidation
of all previously created QR codes and invite links.
- sync declined calls from callee to caller, as usual in all larger
messengers
- introduce the call states "Missed call", "Declined call" and
"Cancelled all" ("Ended call" is gone)
- allow calling end_call()/accept_call() for already ended/accepted
calls, in practise, handling all cornercases is tricky in UI - and the
state needs anyways to be tracked.
- track and show the call duration
the duration calculation depends on local time, but it is displayed only
coarse and is not needed for any state. this can be improved as needed,
timestamps of the corresponding messages are probably better at some
point. or ending device sends its view of the time around. but for the
first throw, it seems good enough
if we finally want that set of states, it can be exposed to a json-info
in a subsequent call, so that the UI can render it more nicely. fallback
strings as follows will stay for now to make adaption in other UI easy,
and for debugging:
<img width="320" alt="IMG_0154"
src="https://github.com/user-attachments/assets/09a89bfb-66f4-4184-b05c-e8040b96cf44"
/>
successor of https://github.com/chatmail/core/pull/6650
this PR uses the initial "call messages" (that has a separate viewtype
since #7174) to show all call status.
this is what most other messengers are doing as well. additional "info
messages" after a call are no longer needed.
on the wire, as we cannot pickpack on visible info messages, we use
hidden messages, similar to eg. webxdc status updates.
in future PR, it is planned to allow getting call state as a json, so
that UI can render nicely. it is then decided if we want to translate
the strings in the core.
<img width="320" alt="IMG_0150"
src="https://github.com/user-attachments/assets/41ee3fa3-8be4-42c3-8dd9-d20f49881650"
/>
successor of https://github.com/chatmail/core/pull/6650
a dedicated viewtype allows the UI to show a more advanced UI, but even
when using the defaults,
it has the advantage that incoming/outgoing and the date are directly
visible.
successor of https://github.com/chatmail/core/pull/6650
The setting is already removed from the UIs,
but users who had it disabled previously have
no way to enable it. After this change
encryption is effectively always preferred.
UIs now display green checkmark in a profile
if the contact is verified.
Chats with key-contacts cannot become unprotected,
so there is no need to check 1:1 chat.
3 months were proven to be too short some years ago, after that issue,
we went far up to 12 months.
however, 12 months were considered too long after recent discussions :)
so, 6 months seems to be a good compromise.
the warning is still repeated every months and the text is unchanged.
advantage is still that this approach does not require network or
opt-in, and catches really all lazy updaters with few effort, cmp
https://github.com/deltachat/deltachat-desktop/issues/5422
Now that the previous commit avoids creating incorrect reverse verification chains, we can do
this. Sure, existing users' dbs aready have verification chains ending with "unknown" roots, but at
least for new users updating `verifier_id` to a known verifier makes sense.
If this happens, mark the contact as verified by an unknown contact instead. This avoids introducing
incorrect reverse chains: if the verifier itself has an unknown verifier, it may be `contact_id`
actually (directly or indirectly) on the other device (which is needed for getting "verified by
unknown contact" in the first place).
Also verify not yet verified contacts w/o setting a verifier for them (in the db it's stored as
`verifier_id=id` though) because we don't know who verified them for another device.
While testing the previous commit i understood that it's better to try giving different colors to
groups, particularly if their names are equal so that they visually differ, and at the same time
preserve the color if the group is renamed. Using `grpid` solves this. So let groups change colors
once and forever.
We can't just fail on an invalid chat name because the user would lose the work already done in the
UI like selecting members. Sometimes happens to me when i put space into name.
Follow-up to https://github.com/chatmail/core/pull/7125: We now have a
mix of non-async (parking_lot) and async (tokio) Mutexes used for the
connectivity. We can just use non-async Mutexes, because we don't
attempt to hold them over an await point. I also tested that we get a
compiler error if we do try to hold one over an await point (rather than
just deadlocking/blocking the executor on runtime).
Not 100% sure about using the parking_lot rather than std Mutex, because
since https://github.com/rust-lang/rust/issues/93740, parking_lot
doesn't have a lot of advantages anymore. But as long as iroh depends on
it, we might as well use it ourselves.
`get_connectivity()` is expected to return immediately, not when the scheduler finishes updating its
state in `start_io()/stop_io()/pause_io()`, otherwise it causes app non-responsiveness.
Instead of read-locking `SchedulerState::inner`, store the `ConnectivityStore` collection in
`Context` and fetch it from there in `get_connectivity()`. Update it every time we release a write
lock on `SchedulerState::inner`.
If the contact is already introduced by someone,
usually by adding to a verified group,
it should not be reverified because of another
chat message is a verified group.
This usually results is verification loops
and is not meaningful because the verifier
likely got this same contact introduced
in the same group.
We haven't dropped verified groups yet, so we need to do smth with messages that can't be verified
yet which often occurs because of messages reordering, particularly in large groups. Apart from the
reported case #7059, i had other direct reports that sometimes messages can't be verified for
various reasons, but when the reason is already fixed, it should be possible to re-download failed
messages and see them.
Also remove the code replacing the message text with a verification error from
`apply_group_changes()` as `add_parts()` already does this.
This doesn't fix anything in UIs currently because they don't call `get_securejoin_qr()` for
unencrypted groups, but it's still better to log an error which will be shown in this case.
In order to debug some test failures wrt broadcast channels, I need to
read the log of some tests that have an alice0 and an alice1.
It's currently not possible to tell whether a line was logged by alice0
or alice1, so, I would like to change that with this PR.
Edit: Turns out that there are more tests that call their profiles
"alice1"/"alice2" (or "alice"/"alice2") than "alice0"/"alice1". So, I
changed the logging to count "alice", "alice2", "alice3", ….
Not sure whether I should adapt old tests; it might save someone some
confusion in the future, but I might also make a mistake and make it so
that the test doesn't properly test anymore.
Chat was only loaded to avoid removing GuaranteeE2ee
for protected chats, but resending a message
in protected chat is guaranteed to be encrypted anyway.
Otherwise if the message is loaded by the UI
after GuaranteeE2ee is reset but before SMTP queue item
is created, the message may appear as unencrypted
even if it was actually resent as encrypted.
Encrypted message may create unencrypted groups
if the message does not have a Chat-Group-ID.
This can happen if v1 client sends an encrypted
message to opportunistically encrypted ad hoc group.
In this case `from_id` corresponds to the key-contact,
but we should add address-contact of the sender
to the member list.
The icon is mainly used to identify unencrypted chats
in the chatlist where encrypted and unencrypted chats are mixed.
It is used for group chats rather than only for 1:1 chats
with address-contacts.
We don't want to prefer returning verified contacts because e.g. if a bot was reinstalled and its
key changed, it may not be verified, and we don't want to bring the user to the old chat if they
click on the bot email address. But trying to return accepted contacts increases security and
doesn't break the described scenario.
If an address-contact and a key-contact were seen at exactly the same time, that doesn't necessarily
mean that it's a random event, it might occur because some code updates contacts this way in some
scenario. While this is unlikely, prefer to look up the key-contact.
Work around possible checkpoint starvations (there were cases reported when a WAL file is bigger
than 200M) and also make sure we truncate the WAL periodically. Auto-checkponting does not normally
truncate the WAL (unless the `journal_size_limit` pragma is set), see
https://www.sqlite.org/wal.html.
- It's obvious that addresses are needed to add contacts, so `_by_address_list` looks excessive.
- The function only looks up `SELF` by address.
- The name was confusing because there's also `lookup_key_contacts_by_address_list()` that actually
looks up key contacts by addresses (not only `SELF`).
- Added import-vcard and make-vcard commands to deltachat-repl. These
wrap the corresponding import_vcard() and make_vcard() operations from
src/contact.rs. Each command takes a file path argument specifying the
location of the vCard file to be read or written. The make-vcard command
takes one or more IDs to write to the file.
- Documented commands in "Using the CLI client" section of README.md.
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
The info will look like:
```
debug_assertions=On - DO NOT RELEASE THIS BUILD
```
or:
```
debug_assertions=Off
```
I tested that this actually works when compiling on Android.
<details><summary>This is how it looked before the second
commit</summary>
<p>
The deltachat_core_version line in the info will look like:
```
deltachat_core_version=v2.5.0 [debug build]
```
or:
```
deltachat_core_version=v2.5.0 [release build]
```
I tested that this actually works when compiling on Android.
</p>
</details>
This adds a test for https://github.com/chatmail/core/pull/7032/.
The crash happened if you received a message that has the from contact
also in the "To: " and "Chat-Group-Member-Fpr: " headers. Not sure how
it happened that such a message was created; I managed to create one in
a test, but needed to access some internals of MimeFactory for that.
I then saved the email that is sent by Alice into the test-data
directory, and then made a test that Bob doesn't crash if he receives
this email. And as a sanity-check, also test that Bob marks Fiona as
verified after receiving this email.
Sort recipients by `add_timestamp DESC` so that if the group is large and there are multiple SMTP
messages, a newly added member receives the member addition message earlier and has gossiped keys of
other members (otherwise the new member may receive messages from other members earlier and fail to
verify them).
This PR adds a test to reproduce a bug raised by @adbenitez that peer
channels break when the resend feature is used.
---------
Co-authored-by: iequidoo <dgreshilov@gmail.com>
This should fix https://github.com/chatmail/core/issues/7030; @r10s if
you could test whether it fixes your problem
The crash happened if you received a message that has the from contact
also in the "To: " and "Chat-Group-Member-Fpr: " headers. Not sure how
it happened that such a message was created.
Previously, chat protection was only removed if the chat became an
email-chat because the key-contact has a different email, but not if the
chat became an email-chat for a different reason.
Therefore, it could happen that the migration produced a protected,
unencrypted chat, where you couldn't write any messages.
I tested that applying this fix actually fixes the bug I had.
tokio-io-timeout 1.2.0 used previously
did not reset the timeout when returning
timeout error. This resulted
in infinite loop spamming
the log with messages that look like this and using 100% CPU:
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Normally these messages should be separated by at least 1 minute timeout.
The reason for infinite loop is not figured out yet,
but this change should at least fix 100% CPU usage.
See <https://github.com/sfackler/tokio-io-timeout/issues/13>
for the bugreport and
<https://github.com/sfackler/tokio-io-timeout/pull/14>
for the bugfix.
Socket may lose peer address when it is disconnected from
the server side. In this case debug_assert! failed
for me when running the core in debug mode on desktop.
To avoid the case of peer_addr not being available,
we now store it when LoggingStream is created.
this PR adds a info message "messages are end-to-end-encrypted" also for
chats created by eg. vcards. by the removal of lock icons, this is a
good place to hint for that in addition; this is also what eg. whatsapp
and others are doing
the wording itself is tweaked at
https://github.com/deltachat/deltachat-android/pull/3817 (and there is
also the rough idea to make the message a little more outstanding, by
some more dedicated colors)
~~did not test in practise, if this leads to double "e2ee info messages"
on secure join, tests look good, however.~~ EDIT: did lots of practise
tests meanwhile :)
most of the changes in this PR are about test ...
ftr, in another PR, after 2.0 reeases, there could probably quite some
code cleanup wrt set-protection, protection-disabled etc.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
- After the revisions to support key contacts, the 'listcontacts'
command in the repl only lists key-contacts. A separate flag now
has to be passed to Contact::get_all() to list address contacts.
This makes it difficult to run the example in README.md because we
need to see the new contact so we can get its ID for the next
command in the example, that creates a chat by ID.
- Revised 'listcontacts' command to make a second call to
Contact::get_all() with the DC_GCL_ADDRESS flag, then print the
e-mail contacts after the key contacts.
- Revised configuration example in top-level README.md to reflect
current command output.
fixes#7011
Delta Chat always adds protected headers to the inner encrypted or signed message, so if a protected
header is only present in the outer part, it should be ignored because it's probably added by the
server or somebody else. The exceptions are Subject and List-ID because there are known cases when
they are only present in the outer message part.
Also treat any Chat-* headers as protected. This fixes e.g. a case when the server injects a
"Chat-Version" IMF header tricking Delta Chat into thinking that it's a chat message.
Also handle "Auto-Submitted" and "Autocrypt-Setup-Message" as protected headers on the receiver
side, this was apparently forgotten.
Headers are normally added at the top of the message,
e.g. when forwarding new `Received` headers are
added at the top.
When headers are protected with DKIM-Signature
and oversigning is not used,
forged headers may be added on top
so headers from the top are generally less trustworthy.
This is tested with `test_take_last_header`,
but so far last header was only preferred
for known headers. This change extends
preference of the last header to all headers.
Connection sometimes fails while processing FETCH
responses. In this case `fetch_new_messages` exits early
and does not advance next expected UID even if
some messages were processed.
This results in prefetching the same messages
after reconnection and log messages
similar to
"Not moving the message ab05c85a-e191-4fd2-a951-9972bc7e167f@localhost that we have seen before.".
With this change we advance next expected UID
even if `fetch_new_messages` returns a network error.
A donation request device message is added if >= 100 messages have been sent and delivered. The
condition is checked every 30 days since the first message is sent. The message is added only once.
Follow-up for https://github.com/chatmail/core/pull/6992
Since we're printing the hint to stderr now, rather than stdout (as per
link2xt's suggestion), it was too noisy. Also, it was printed once for
every test account rather than once per test.
Now, it integrates nicely with rust's hint to enable a backtrace:
```
stderr ───
thread 'chat::chat_tests::test_broadcasts_name_and_avatar' panicked at src/chat/chat_tests.rs:2757:5:
assertion failed: `(left == right)`
Diff < left / right > :
<true
>false
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: If you want to examine the database files, set environment variable DELTACHAT_SAVE_TMP_DB=1
FAIL [ 0.265s] deltachat chat::chat_tests::test_broadcast
```
Add `logged_debug_assert` macro logging a warning if a condition is not satisfied, before invoking
`debug_assert!`, and use this macro where `Context` is accessible (i.e. don't change function
signatures for now).
Follow-up to 0359481ba4.
We have some debug assertions already, but we also want the corresponding errors in the release
configuration so that it's not less reliable than non-optimized one. This doesn't change any
function signatures, only debug assertions in functions returning `Result` are replaced.
Co-authored-by: l <link2xt@testrun.org>
Part of #6884.
The channel owner will not be notified in any way that you left, they
will only see that there is one member less.
For the translated stock strings, this is what we agreed on in the
group:
- Add a new "Leave Channel" stock string (will need to be done in UIs)
- Reword the existing "Are you sure you want to leave this group?"
string to "Are you sure you want to leave?" (the options are "Cancel"
and "Leave Group" / "Leave Channel", so it's clear what you are leaving)
(will need to be done in the deltachat-android repo, other UIs will pick
it up automatically)
- Reword the existing "You left the group." string to "You left". (done
here, I will adapt the strings in deltachat-android, too)
I adapted DC Android by pushing
6df2740884
to https://github.com/deltachat/deltachat-android/pull/3783.
---------
Co-authored-by: l <link2xt@testrun.org>
I had it a few times now that I wanted to examine the database in order
to debug a test failure. Setting this environment variable makes this
easy in the future.
New public API `set_accounts_order` allows setting the order of accounts.
The account order is stored as a list of account IDs in `accounts.toml`
under a new `accounts_order: Vec<u32>` field.
BREAKING CHANGE: messages with invalid images, images of unknown size,
huge images, will have Viewtype::File
After changing the logic of Viewtype selection, I had to fix 3 old tests
that used invalid Base64 image data.
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Non-members can't modify the member list (incl. adding themselves), modify an ephemeral timer, so
they shouldn't be able to change the group name or avatar, just for consistency. Even if messages
are reordered and a group name change from a new member arrives before its addition, the new group
name will be applied on a receipt of the next message following the addition message because
Chat-Group-Name-Timestamp increases. While Delta Chat groups aimed for chatting with trusted
contacts, accepting group changes from everyone knowing Chat-Group-Id means that if any of the past
members have the key compromised, the group should be recreated which looks impractical.
There's a comment in `tools` that tells to use `tools::Time` instead of `Instant` because on Android
the latter doesn't advance in the deep sleep mode. The only place except `migrations` where
`Instant` is used is tests, but we don't run CI on Android. It's unlikely that Delta Chat goes to
the deep sleep while executing migrations, but still possible, so let's use `tools::Time` as
everywhere else.
Save:
- (old contact id) -> (new contact id) mapping.
- The message id starting from which all messages are already migrated.
Run the migration from `housekeeping()` for at least 500 ms and for >= 1000 messages per run.
We don't want images having unsupported format or corrupted ones to be sent as `Image` and appear in
the "Images" tab in UIs because they can't be displayed correctly.
If a user attaches an image as `File`, we should send the original filename. And vice versa, if it's
`Image` originally, we mustn't reveal the filename.
The filename used for sending is now also saved to the db, so all the sender's devices will display
the same filename in the message info.
Otherwise unsupported and corrupted images are displayed in the "Images" tab in UIs and that looks
as a Delta Chat bug. This should be a rare case though, so log it as error and let the user know
that metadata isn't removed from the image at least.
`Viewtype::Sticker` has special meaning: the file should be an image having fully transparent
pixels. But "tgs" (Telegram animated sticker) is a compressed JSON and isn't recognized by Core as
image.
Older versions of Delta Chat ignore the message if it contains a
ChatGroupId header. ("older versions" means all versions without #6901,
i.e.currently released versions)
This means that without this PR, broadcast channel messages sent from
current main don't arrive at a device running latest released DC.
Part of #6884.
In https://github.com/chatmail/core/pull/6901, I unfortunately forgot to
document the API change when squash-merging, so, I'm doing this with the
PR here.
The API change is breaking because not adapting to the new channel types
would lead to errors.
Part of #6884
----
- [x] Add new chat type `InBroadcastChannel` and `OutBroadcastChannel`
for incoming / outgoing channels, where the former is similar to a
`Mailinglist` and the latter is similar to a `Broadcast` (which is
removed)
- Consideration for naming: `InChannel`/`OutChannel` (without
"broadcast") would be shorter, but less greppable because we already
have a lot of occurences of `channel` in the code. Consistently calling
them `BcChannel`/`bc_channel` in the code would be both short and
greppable, but a bit arcane when reading it at first. Opinions are
welcome; if I hear none, I'll keep with `BroadcastChannel`.
- [x] api: Add create_broadcast_channel(), deprecate
create_broadcast_list() (or `create_channel()` / `create_bc_channel()`
if we decide to switch)
- Adjust code comments to match the new behavior.
- [x] Ask Desktop developers what they use `is_broadcast` field for, and
whether it should be true for both outgoing & incoming channels (or look
it up myself)
- I added `is_out_broadcast_channel`, and deprecated `is_broadcast`, for
now
- [x] When the user changes the broadcast channel name, immediately show
this change on receiving devices
- [x] Allow to change brodacast channel avatar, and immediately apply it
on the receiving device
- [x] Make it possible to block InBroadcastChannel
- [x] Make it possible to set the avatar of an OutgoingChannel, and
apply it on the receiving side
- [x] DECIDE whether we still want to use the broadcast icon as the
default icon or whether we want to use the letter-in-a-circle
- We decided to use the letter-in-a-circle for now, because it's easier
to implement, and I need to stay in the time plan
- [x] chat.rs: Return an error if the user tries to modify a
`InBroadcastChannel`
- [x] Add automated regression tests
- [x] Grep for `broadcast` and see whether there is any other work I
need to do
- [x] Bug: Don't show `~` in front of the sender's same in broadcast
lists
----
Note that I removed the following guard:
```rust
if !new_chat_contacts.contains(&ContactId::SELF) {
warn!(
context,
"Received group avatar update for group chat {} we are not a member of.", chat.id
);
} else if !new_chat_contacts.contains(&from_id) {
warn!(
context,
"Contact {from_id} attempts to modify group chat {} avatar without being a member.",
chat.id,
);
} else [...]
```
i.e. with this change, non-members will be able to modify the avatar.
Things were slightly easier this way, and I think that this is in line
with non-members being able to modify the group name and memberlist
(they need to know the Group-Chat-Id, anyway), but I can also change it
back.
- The implementation of listverified was removed in commit
37dc1f5ca0, but it still shows up in
the help and in the auto-complete grammar.
- Removed listverified where it still appears.
closes#6971
We need this because it's not clear whether Android should switch to
JsonRPC for everything, because of concerns that JsonRPC might be a lot
slower than the CFFI (although we still need to measure that).
SQL migration to key contacts generates a lot of events,
and they are dropped in desktop logs because it does
not read the events fast enough.
This at least reduces the number of dropped messages.
This change introduces a new type of contacts
identified by their public key fingerprint
rather than an e-mail address.
Encrypted chats now stay encrypted
and unencrypted chats stay unencrypted.
For example, 1:1 chats with key-contacts
are encrypted and 1:1 chats with address-contacts
are unencrypted.
Groups that have a group ID are encrypted
and can only contain key-contacts
while groups that don't have a group ID ("adhoc groups")
are unencrypted and can only contain address-contacts.
JSON-RPC API `reset_contact_encryption` is removed.
Python API `Contact.reset_encryption` is removed.
"Group tracking plugin" in legacy Python API was removed because it
relied on parsing email addresses from system messages with regexps.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Co-authored-by: B. Petersen <r10s@b44t.com>
Display name is rarely needed for debugging,
so there is no need to include it in the logs.
Display name is even already listed in `skip_from_get_info`,
but the test only allowed the values to be skipped
without checking that they are always skipped.
Before this PR, ConfiguredAddr (which will be used to store the primary
transport) would have been changed when adding a new transport. Doesn't
matter yet because it's not possible yet to have multiple transports.
But I wanted to fix this bug already so that I'm not suprised by it
later.
Citing @link2xt:
> RFC examples sometimes don't escape commas, but there is errata that fixes some of them.
Also this unescapes commas in all fields. This can lead to, say, an email address with commas, but
anyway the caller should check parsed `VcardContact`'s fields for correctness.
OpenSSL is vendored, but because of rusqlite feature
transitively enabling vendoring feature.
This change makes vendoring explicit
even if we disable SQLCipher in the future.
New `fuzz` profile is added
because cargo-bolero now requires it and uses
by default, while `--release` option is removed.
Instructions for running AFL are removed from the README
because it requires some system reconfiguration
and I did not test it this time.
The main reason for this change is the app picker
that Delta Chat clients use, which utilizes
the `fetch_url` function.
Sometimes we get an error from the server,
but we have no way to figure out that it's an error,
other than inspecting the body, which we don't (and shouldn't) do.
This results in us attempting to send webxdc apps
that are not even valid .zip files.
Another, arguably even worse thing is that
we also put the error responses to the cache,
so it's not easy to recover from such an error.
So, let's just return an error if the response code
is not a successful response code.
If we receive a message from non-verified contact
in a non-protected chat with a Chat-Verified header,
there is no need to upgrade the chat
to verified and display an error.
If it was an attack, an attacker could
just not send the Chat-Verified header.
Most of the time, however, it is just
message reordering.
The test was still WIP but got merged together with the fix. I suggest
to keep the fix in main and add the test in a follow-up RP. The test
should suffice becaues I tested it manually.
When an invalid webxdc is set as draft, json-rpc's `get_draft` fails,
because `get_webxdc_info` which it calls, fails because the zip reader
can not read a non-zip file. With this change, any error occurring in
`get_webxdc_info` is ignored and the None-variant is returned instead. I
also added a test, that setting invalid xdcs is draft is fine core-wise
and checked that the input field stays responsive when a fake.xdc
produced like in #6826 is added to draft
close#6826
closes#6873 , see there for reasoning.
tested that on iOS already, works like a charm - and was much easier
than expected as @iequidoo already updated `timestamp_rcvd` on status
updates in https://github.com/chatmail/core/pull/5388
~~a test is missing, ordering is not tested at all, will check if that
is doable reasonably easy~~ EDIT: added a test
Fix https://github.com/chatmail/core/issues/6621; I also tested on
Android that the webxdc self-addr actually stays the same when staging a
draft and then sending.
Follow-up to https://github.com/chatmail/core/pull/6704; #6704 made sure
that the webxdc self-addr doesn't change when creating a message and
then sending it. This PR here makes sure that the rfc724_mid (which is
needed to compute the self-addr) is saved when setting a draft, so that
it's loaded properly after a call to get_draft().
cc @adbenitez @r10s @Septias
FuturesUnordered is likely buggy and iroh previously switched
to JoinSet in <https://github.com/n0-computer/iroh/pull/1647>.
We also have reports with logs of background_fetch getting
stuck so apparently task cancellation after timeout does not work
as intended with FuturesUnordered.
Set `rfc724_mid` in `Message::new()`, `Message::new_text()`, and
`Message::default()` instead of when sending the message. This way the
rfc724 mid can be read in the draft stage which makes it more consistent
for bots. Tests had to be adjusted to create multiple messages to get
unique mid, otherwise core would not send the messages out.
this PR scaled avatars using the Triangle-filter,
resulting in often better image quality and smaller files (5%).
it comes at high costs,
therefore, we do not do that unconditionally for each image sent, see
comment in the code
and https://github.com/chatmail/core/pull/6815
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Revert the biggest part of https://github.com/chatmail/core/pull/6722/
in order to fix#6816. Reopens
https://github.com/chatmail/core/issues/6706.
Rationale for reverting instead of fixing is that it's not trivial to
implement "if the chat is encrypted, can_send() returns true": When
sending a message, in order to check whether to encrypt, we load all
peerstates and check whether all of them can be encrypted to
(`should_encrypt()`). We could do this in `can_send()`, but this would
make it quite slow for groups. With multi-transport, the ways of
checking whether to encrypt will be different, so in order not to do
unnecessary work now, this PR just revert parts of
[https://github.com/chatmail/core/pull/6722/](https://github.com/chatmail/core/pull/6817#),
so that we can make things work nicely when multi-transport is merged.
As a quick mitigation, we could increase the timeout from 15s to
something like 1 minute or 1 day: Long enough that usually securejoin
will finish before, but short enough that it's possible to send to old
chats that had a failed securejoin long in the past.
Move all `configured_*` parameters into a new SQL table `transports`.
All `configured_*` parameters are deprecated; the only exception is
`configured_addr`, which is used to store the address of the primary
transport. Currently, there can only ever be one primary transport (i.e.
the `transports` table only ever has one row); this PR is not supposed
to change DC's behavior in any meaningful way.
This is a preparation for mt.
---------
Co-authored-by: l <link2xt@testrun.org>
In the `test` cfg, introduce `MimeMessage::headers_removed` hash set and `header_exists()` function
returning whether the header exists in any part of the parsed message. `get_header()` shouldn't be
used in tests for checking absense of headers because it returns `None` for removed ("ignored")
headers.
This change simplifies
updating the gossip timestamps
when we receive a message
because we only need to know
the keys received in Autocrypt-Gossip
header and which chat the message is
assigned to.
We no longer need to iterate
over the member list.
This is a preparation
for PGP contacts
and member lists that contain
key fingerprints rather than
email addresses.
This change also removes encryption preference
from Autocrypt-Gossip header.
It SHOULD NOT be gossiped
according to the Autocrypt specification
and we ignore encryption preference anyway
since 1.157.0.
test_gossip_optimization is removed
because it relied on a per-chat gossip_timestamp.
Broadcast lists are encrypted since 1.159.0,
but Autocrypt-Gossip was not disabled.
As Autocrypt-Gossip contains the email address
and the key of the recipient, it should
not be sent to broadcast lists.
this PR moves now advanced/unsupported ASM strings to core, removing
work from translations, esp. as another hint is added which would
require retranslations. it is better to have that just in english, it is
a nerd feature anyways.
moverover, this PR removes special rendering of ASM in the summary,
which might be confusion, but mainly it is now unneeded, dead code
i'll do another android PR that will point to "Add Second Device"
already on ASM generation EDIT: done at
https://github.com/deltachat/deltachat-android/pull/3726
targets https://github.com/deltachat/deltachat-desktop/issues/4946
it was all the time questionable if not encrypting broadcast lists
rules the issue that recipients may know each other cryptographically.
however, meanwhile with chatmail, unncrypted broadcasts are no longer possible,
and we actively broke workflows eg. from this teacher:
https://support.delta.chat/t/broadcast-funktioniert-nach-update-nicht-meht/3694
this basically reverts commit
7e5907daf2
which was that time added last-minute and without lots discussions :)
let the students get their homework again :)
Primary key is usually used for certification.
It is possible to make a certification- and encryption-
capable key with RSA, but RFC 9580 says
that implementations SHOULD NOT generate RSA keys.
The test works most of the time, but essentially tests that splitting
the public key from a private key
generates the same result.
However, it fails if two signatures are generated
at different seconds.
Closes#6762
fetch_existing option is not enabled in existing clients
and does not work with encrypted messages
without importing the key into a newely created account.
next android/desktop/ios releases won't have the "Show Classic Emails"
option for chatmail.
to avoid issues with user that have set sth else than "All", we ignore
the option alltogether for chatmail profiles.
ftr, i do not expect ppl having that option changed for chatmail much,
it does not make much sense. so this PR is mainly to save our limited
support resources :) (usecase: "look, i am using chatmail to sign up at
SERVICE, but for security reasons i set show=all only when i reset my
password" :)
one could also do that in a migration, however, (a) migrations always
come with some risk, even the easiest ones, and (b) the show_emails
option is subject to change or disappear anyways, subsequent changes are
easier in code than in additional or removed migrations, and (c) it is
really only one line, that does not add much with complexity
Otherwise if an invite link is generated and the group is renamed then before the promotion, the
joined member will have the group name from the invite link, not the new one.
for tuning down email address everywhere, that bit is missing in core.
it was never useful, as it was never shown on the receivers side. and
for the sender side, the context the qr code is opened is clear
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
Without tox any python `scripts/make-python-env.sh` does not run. Maybe
at some point we can even generate the environment for testing with
`venvHook` like functionality. We could also add `python3` instead of
fixing it to 3.11, but I don't know which version core expects
`self.accounts.read().await.get_all()` acquires a read lock
and does not release it until the end of `for` loop.
After that, a writer may get into the queue,
e.g. because of the concurrent `add_account` call.
In this case `let context_option = self.accounts.read().await.get_account(id);`
tries to acquire another read lock and deadlocks
because tokio RwLock is write-preferring and will not
give another read lock while there is a writer in the queue.
At the same time, writer never gets a write lock
because the first read lock is not released.
The fix is to get a single read lock
for the whole `get_all_accounts()` call.
This is described in <https://docs.rs/tokio/1.44.1/tokio/sync/struct.RwLock.html#method.read>:
"Note that under the priority policy of RwLock, read locks are not
granted until prior write locks, to prevent starvation. Therefore
deadlock may occur if a read lock is held by the current task, a write
lock attempt is made, and then a subsequent read lock attempt is made by
the current task."
instead of showing addresses in info message, provide an API to get the
contact-id.
UI can then make the info message tappable and open the contact profile
in scope
the corresponding iOS PR - incl. **screencast** - is at
https://github.com/deltachat/deltachat-ios/pull/2652 ; jsonrpc can come
in a subsequent PR when things are settled on android/ios
the number of parameters in `add_info_msg_with_cmd` gets bigger and
bigger, however, i did not want to refactor this in this PR. it is also
not really adding complexity
closes#6702
---------
Co-authored-by: link2xt <link2xt@testrun.org>
Co-authored-by: Hocuri <hocuri@gmx.de>
otherwise, by tuning down the email addresses,
one does not really has and idea who is SELF.
maybe the dialog is the only way at the end to get the transport
adresses of contacts,
this is unclear atm, this PR fixes the issue at hand
Targets https://github.com/chatmail/core/issues/6636
Right now the error message is:
> Error: Delta Chat is already running. To use Delta Chat, you must
first close the existing Delta Chat process, or restart your device.
>
> (accounts.lock lock file is already locked)
other suggestions welcome!
vc-request is an unencrypted message
that Bob sends when he does not have Alice's key.
It also does not contain
Bob's avatar and name,
so the contact has only the email address
at this point and it is too early
to show it.
The test is mostly testing that groups can be matched
even if Message-ID is replaced.
Delta Chat no longer places group ID into Message-ID
or References, so the test is not
testing anything other than the ability
to match groups based on References header.
This API allows to explicitly set
a name of the contact
instead of trying to create a new contact
with the same address.
Not all contacts are identified
by the email address
and we are going to introduce
contacts identified by their keys.
Four new APIs `add_transport_from_qr()`, `add_transport()`,
`list_transports()`, `delete_transport()`, as described in the draft at
"API".
The `add_tranport*()` APIs automatically stops and starts I/O; for
`configure()` the stopping and starting is done in the JsonRPC bindings,
which is not where things like this should be done I think, the bindings
should just translate the APIs.
This also completely disables AEAP for now.
I won't be available for a week, but if you want to merge this already,
feel free to just commit all review suggestions and squash-merge.
Found and fixed a bug while investigating
https://github.com/chatmail/core/issues/6656. It's not the same bug,
though.
Steps to reproduce this bug:
- Create a new profile
- Transfer it to a second device
- Send a message from the first device
- -> It will never arrive on the second device, instead a warning will
be printed that you are using DC on multiple devices.
The bug was that the key wasn't created before the backup transfer, so
that the second device then created its own key instead of using the
same key as the first device.
In order to regression-test, this PR now changes `clone()` to use "Add
second device" instead of exporting and importing a backup. Exporting
and importing a backup has enough tests already.
This PR also adds an unrelated test `test_selfavatar_sync()`.
The bug was introduced by https://github.com/chatmail/core/pull/6574 in
v1.156.0
WebSocket support is not used
and is not maintained. It still uses
outdated axum 0.7 version
and does not have any authentication.
Delta Chat Desktop has a new browser target
that implements WebSocket support on top
of stdio server, supports blobs
and is tested in CI.
When it is possible to test that no unhidden contact
is creating by looking through the contact list
or get the contact ID as the from_id of received message,
do it to avoid acidentally creating a contact
or changing its origin before testing.
async-compression 0.4.21 fixes a bug in the encoder
where it did not flush all the internal state sometimes,
resulting in IMAP APPEND command timing out
waiting for response when uploading large sync messages.
See <https://github.com/Nullus157/async-compression/pull/333>
for details.
this PR deletes all known messages belonging to a chat when the chat is
deleted.
this may not be an exhaustive list as a client might not know all
message-ids (eg. when using different times for "delete from device").
in this case, other devices may know more IDs. otherwise, the chatmail
server will eventually clean up at some point. for non-chatmail, this is
up to the user then.
the deletion sql commands were inspired by
[`delete_msgs_ex`](https://github.com/chatmail/core/blob/main/src/message.rs#L1743)
(in fact, [a first
try](https://github.com/chatmail/core/compare/r10s/clear-chat-on-delete)
was adapting that part, however, that seems less performant as lots of
sql commands are needed)
successor of #5007
SecureJoin and importing a vCard are the primary
ways we want to support for creating contacts.
Typing in an email address and relying on Autocrypt
results in sending the first message unencrypted
and we want to clearly separate unencrypted and encrypted
chats in the future.
To make the tests more stable, we set up test contacts
with vCards as this always immediately
results in creating a single encrypted chat
and this is not going to change.
up to now, avatars were restricted to 20k, 256x256 pixel.
resulting in often poor avatar quality.
(the limitation comes from times
where unencrypted outlook inner headers were a thing;
this has changed meanwhile)
to increase quality while keeping a reasonable small size,
for "balanced quality",
we increase the allowed width to 512x512 and the allowed size to 60k.
for "worse quality", things stay unchanged at 128x128 pixel and 20k.
Currently broadcast lists creation is synced across devices. Groups creation, in some means, too --
on a group promotion by a first message. Otoh, messages deletion is synced now as well. So, for
feature-completeness, sync chats deletion too.
Say in the README that cargo-bolero@0.8.0
should be installed as newer versions
do not work currently.
Update `mailparse` to 0.16.0
because this is what Delta Chat currently uses.
I also added fuzzer outputs to .gitignore
Add "Chat-Group-Name-Timestamp" message header and use the last-write-wins logic when updating group
names (similar to group member timestamps). Note that if the "Chat-Group-Name-Changed" header is
absent though, we don't add a system message (`MsgGrpNameChangedBy`) because we don't want to blame
anyone.
Replacing default key
when a profile is already part of
verified groups results in
`[The message was sent with non-verified encryption. See 'Info' for more details]`
messages for other users.
It is still possible
to import the default key before
Delta Chat generates the key.
When there is a broken group (which might happen with multi-transport),
people want to leave it.
The problem is that every "Group left" message notifies all other
members and pops up the chat, so that other members also want to leave
the group.
This PR makes it so that "Group left" messages don't create a
notification, don't cause a number-in-a-cirle badge counter on the chat,
and don't sort up the chat in the chatlist.
If a group is deleted, then the group won't pop up when someone leaves
it; this worked fine already before this PR, and there also is a test
for it.
Instead of being trashed, the message containing a reaction remains in the chat, hidden and InFresh. When the chat is opened, it will be marked as Seen on the server, so that a second device removes the notifications for the reaction.
Close https://github.com/deltachat/deltachat-core-rust/issues/6210.
Also, this adds a benchmark.
Now that we are deduplicating everywhere, we can get rid of some code.
The old python bindings did not get an optional `name` parameter because
they are deprecated anyway, but it would be easy to add it.
> _greetings from the ice of the deutsche bahn 🚂🚃🚃🚃 always a pleasure to
see how well delta chat meanwhile performs in bad networks :)_
this PR adds an API to request other chat members to replace the message
text of an already sent message. scope is mainly to fix typos. this
feature is known from whatsapp, telegram, signal, and is
[requested](https://support.delta.chat/t/retract-edit-sent-messages/1918)
[since](https://support.delta.chat/t/edit-messages-in-delta-chat/899)
[years](https://github.com/deltachat/deltachat-android/issues/198).
technically, a message with an
[`Obsoletes:`](https://datatracker.ietf.org/doc/html/rfc2076#section-3.6)
header is sent out.
```
From: alice@nine
To: bob@nine
Message-ID: 2000@nine
In-Reply-To: 1000@nine
Obsoletes: 1000@nine
Edited: this is the new text
```
the body is the new text, prefixed by the static text `Edited:` (which
is not a header). the latter is to make the message appear more nicely
in Non-Delta-MUA. save for the `In-Reply-To` header. the `Edited:`
prefix is removed by Delta Chat on receiving.
headers should be protected and moved to e2ee part as usual.
corrected message text is flagged, and UI should show this state, in
practise as "Edited" beside the date.
in case, the original message is not found, nothing happens and the
correction message is trashes (assuming the original was deleted).
question: is the `Obsoletes:` header a good choice? i _thought_ there is
some more specifica RFC, but i cannot find sth. in any case, it should
be an header that is not used otherwise by MUA, to make sure no wanted
messages disappear.
what is NOT done and out of scope:
- optimise if messages are not yet sent out. this is doable, but
introduces quite some cornercaes and may not be worth the effort
- replaces images or other attachments. this is also a bit cornercasy
and beyond "typo fixing", and better be handled by "delete for me and
others" (which may come soon, having the idea now, it seems easy :)
- get edit history in any way. not sure if this is worth the effort,
remember, as being a private messenger, we assume trust among chat
members. it is also questionable wrt privacy, seized devices etc.
- add text where nothing was before; again, scope is "typo fixing",
better avoid cornercases
- saved messages are not edited (this is anyway questionable)
- quoted texts, that are used for the case the original message is
deleted, are not updated
- edits are ignored when the original message is not there yet (out of
order, not yet downloaded)
- message status indicator does not show if edits are sent out or not -
similar to reactions, webxdc updates, sync messages. signal has the same
issue :) still, connectivity should show if there are messages pending
<img width="366" alt="Screenshot 2025-02-17 at 17 25 02"
src="https://github.com/user-attachments/assets/a4a53996-438b-47ef-9004-2c9062eea5d7"
/>
corresponding iOS branch (no PR yet):
https://github.com/deltachat/deltachat-ios/compare/main...r10s/edit-messages
---------
Co-authored-by: l <link2xt@testrun.org>
This makes it so that files will be deduplicated when using the JsonRPC
API.
@nicodh and @WofWca you know the Desktop code and how it is using the
API, so, you can probably tell me whether this is a good way of changing
the JsonRPC code - feel free to push changes directly to this PR here!
This PR here changes the existing functions instead of creating new
ones; we can alternatively create new ones if it allows for a smoother
transition.
This brings a few changes:
- If you pass a file that is already in the blobdir, it will be renamed
to `<hash>.<extension>` immediately (previously, the filename on the
disk stayed the same)
- If you pass a file that's not in the blobdir yet, it will be copied to
the blobdir immediately (previously, it was copied to the blobdir later,
when sending)
- If you create a file and then pass it to `create_message()`, it's
better to directly create it in the blobdir, since it doesn't need to be
copied.
- You must not write to the files after they were passed to core,
because otherwise, the hash will be wrong. So, if Desktop recodes videos
or so, then the video file mustn't just be overwritten. What you can do
instead is write the recoded video to a file with a random name in the
blobdir and then create a new message with the new attachment. If
needed, we can also create a JsonRPC for `set_file_and_deduplicate()`
that replaces the file on an existing message.
In order to test whether everything still works, the desktop issue has a
list of things to test:
https://github.com/deltachat/deltachat-desktop/issues/4498
Core issue: #6265
---------
Co-authored-by: l <link2xt@testrun.org>
This hopefully does not trigger Apple spam filter
which seems to pass thorugh Message-IDs with hyphens.
It is not confirmed, but better blend in
by mimicking existing popular email
where possible.
With iOS and Desktop copying the file to a to a temp file with the name
of `get_filename()`, it should be sanitized first.
The PR can be reviewed commit-by-commit or all at once.
This task is not guaranteed to be cancellation-safe
and provides a stop token for safe cancellation instead.
We should always cancel the task properly
and not by racing against another future.
Otherwise following call to Handle.done()
may work on IMAP session that is in the middle
of response, for example.
this converts old QR codes to the new format, in an hacky, but simple
way, see #6518 for more details and for code snippet
then QR code change is esp. bad as ppl will have different versions for
some days at least, weakening overall UX, esp. of first-time-users that
may come to delta because of praised, seamless multidevice ... :)
i tested in https://github.com/deltachat/deltachat-ios/pull/2595 that
this actually fixes the problem, and there is no deeper issue with
changed chashes or so - seemed not to be the case, at least, with this
hack, core accepts QR codes from the released 1.52-and-before series
this hack gives user time to update, it can be removed after some months
(we can also remove the old BACKUP qr code alltogether then)
we should still not wait too long with the PR as there are already
versions out with the "new/bad" QR code (and growing, as new iOS
installations will get the new format, one cannot revert a version, only
pause rollout)
---------
Co-authored-by: link2xt <link2xt@testrun.org>
This change adds a test and updates mailparse from 0.15.0 to 0.16.0.
mailparse 0.16.0 includes a fix for the bug
that resulted in CRLF being included at the end of the body.
Workaround for the bug in the `pk_validate` function is also removed.
APNS tokens never expire unless
the user uninstalls the application.
Because of this in most cases
the token remains valid forever
and chatmail server never removes the token
even if it is unencrypted
or the user has removed Delta Chat profile
from the device but not the whole application.
We want to modify chatmail servers
to remember the last time the token was stored
and remove them after some time.
Before we do this, we need to modify
the client to store the device token
each time so the server knows which tokens are used
and can update their timestamps.
QR code format changed because `NodeAddr` serialization
changed between iroh 0.29.0 and iroh 0.30.0.
We have already released iroh 0.30.0,
so the test at least makes change we don't break compatibility again.
There was a bug that file extensions were removed when recoding an
avatar. The problem was that `recode_avatar` used `name` to check for
the extension, but some functions passed an empty string.
There even were two tests from before the decision to keep the
extensions that tested for the faulty behavior.
These were the last places in the `deltachat` crate where files were
stored without deduplication. The CFFI python bindings are the last
thing that's still missing.
Deduplicate:
- In the REPL
- In `store_from_base64()`, which writes avatars received in headers
- In a few tests
- The saved messages, broadcast, device, archive icons
- The autocrypt setup message
1-2 more PRs, and we can get rid of `BlobObject::create`,
`sanitise_name()`, and some others
When receiving messages, blobs will be deduplicated with the new
function `create_and_deduplicate_from_bytes()`. For sending files, this
adds a new function `set_file_and_deduplicate()` instead of
deduplicating by default.
This is for
https://github.com/deltachat/deltachat-core-rust/issues/6265; read the
issue description there for more details.
TODO:
- [x] Set files as read-only
- [x] Don't do a write when the file is already identical
- [x] The first 32 chars or so of the 64-character hash are enough. I
calculated that if 10b people (i.e. all of humanity) use DC, and each of
them has 200k distinct blob files (I have 4k in my day-to-day account),
and we used 20 chars, then the expected value for the number of name
collisions would be ~0.0002 (and the probability that there is a least
one name collision is lower than that) [^1]. I added 12 more characters
to be on the super safe side, but this wouldn't be necessary and I could
also make it 20 instead of 32.
- Not 100% sure whether that's necessary at all - it would mainly be
necessary if we might hit a length limit on some file systems (the
blobdir is usually sth like
`accounts/2ff9fc096d2f46b6832b24a1ed99c0d6/dc.db-blobs` (53 chars), plus
64 chars for the filename would be 117).
- [x] "touch" the files to prevent them from being deleted
- [x] TODOs in the code
For later PRs:
- Replace `BlobObject::create(…)` with
`BlobObject::create_and_deduplicate(…)` in order to deduplicate
everytime core creates a file
- Modify JsonRPC to deduplicate blob files
- Possibly rename BlobObject.name to BlobObject.file in order to prevent
confusion (because `name` usually means "user-visible-name", not "name
of the file on disk").
[^1]: Calculated with both https://printfn.github.io/fend/ and
https://www.geogebra.org/calculator, both of which came to the same
result
([1](https://github.com/user-attachments/assets/bbb62550-3781-48b5-88b1-ba0e29c28c0d),
[2](https://github.com/user-attachments/assets/82171212-b797-4117-a39f-0e132eac7252))
---------
Co-authored-by: l <link2xt@testrun.org>
- This [is said to lead improve compilation
speed](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html#Assorted-Tricks)
- When grepping for a function invocation, this makes it easy to see whether it's from a test or "real" code
- We're calling the files e.g. `chat_tests.rs` instead of `tests.rs` for the same reason why we moved `imap/mod.rs` to `imap.rs`: Otherwise, your editor always shows you that you're in the file `tests.rs` and you don't know which one.
This is only moving mimeparser and chat tests, because these were the
biggest files; we can move more files in subsequent PRs if we like it.
Fix https://github.com/deltachat/deltachat-core-rust/issues/6433
I at first only wanted to do it any outgoing messages, but @link2xt was
concerned that this may accidentally enable bcc_self, e.g. in the
following case:
- you send out a message
- it's deleted, e.g. via ephemeral messages
- Someone forwards this outgoing message to you again, e.g. via a
mailing list.
If we create an unpromoted group,
add a member there and then remove it
before we promote a group, there is no need to
add such member to the list of past members
and send the address of this member to the group
when it is promoted.
> _took quite some time until i found the time to finish this PR and to
find a time window that does not disturb other developments too much,
but here we are:_
this PR enables UI to improve "Saved messages" hugely, bringing it on
WhatsApp's "Starred Messages" or Telegram's "Saved Messages" level. with
this PR, UIs can add the following functionality with few effort ([~100
loc on iOS](https://github.com/deltachat/deltachat-ios/pull/2527)):
- add a "Save" button in the messages context menu, allowing to save a
message
- show directly in the chat if a message was saved, eg. by a little star
★
- in "Saved Messages", show the message in its context - so with author,
avatar, date and keep incoming/outgoing state
- in "Saved Messages", a button to go to the original message can be
added
- if the original message was deleted, one can still go to the original
chat
these features were often requested, in the forum, but also in many
one-to-one discussions, recently at the global gathering.
moreover, in contrast to the old method with "forward to saved", no
traffic is wasted - the messages are saved locally, and only a small
sync messages is sent around
this is how it looks out on iOS:
<img width="260" alt="Screenshot 2025-01-17 at 00 02 51"
src="https://github.com/user-attachments/assets/902741b6-167f-4af1-9f9a-bd0439dd20d0"
/> <img width="353" alt="Screenshot 2025-01-17 at 00 05
33"
src="https://github.com/user-attachments/assets/97eceb79-6e43-4a89-9792-2e077e7141cd"
/>
technically, still a copy is done from the original message (with
already now deduplicated blobs), so that things work nicely with
deletion modes; eg. you can save an important message and preserve it
from deletion that way.
jsonrpc can be done in a subsequent PR, i was implementing the UI on iOS
where that was not needed (and most API were part of message object that
is not in use in jsonrpc atm)
@hpk42 the forward issue we discussed earller that day is already solved
(first implementation did not had an explict save_msgs() but was using
forward_msgs(SELF) as saving - with the disadvantage that forwarding to
SELF is not working, eg. if one wants the old behaviour) acutally, that
made the PR a lot nicer, as now very few existing code is changed
<details>
<summary>previous considerations and abandoned things</summary>
while working on this PR, there was also the idea to just set a flag
“starred” in the message table and not copy anything. however, while
tempting, that would result in more complexity, questions and drawbacks
in UI:
- delete-message, delete-chat, auto-deletion, clear-chat would raise
questions - what do do with the “Starred”? having a copy in “Saved
Messages” does not raise this question
- newly saved messages appear naturally as “new” in “Saved Messages”,
simply setting a flag would show them somewhere in between - unless we
do additional effort
- “Saved Messages” already has its place in the UI - and allows to
_directly_ save things there as well - not easily doable with “starring”
- one would need to re-do many things that already exist in “Saved
Messages”, at least in core
- one idea to solve some of the issues could be to have “Starred” as
well as “Saved Messages” - however, that would irritate user, one would
remember exactly what was done with a message, and understand the fine
differences
whatsapp does this “starred”, btw, so when original is deleted, starred
is deleted as well. Telegram does things similar to us, Signal does
nothing. Whatsapp has a per-chat view for starred messages - if needed,
we could do sth. like that as well, however, let’s first see how far the
“all view” goes (in contrast to whatsapp, we have profiles for
separation as well)
for the wording: “saving” is what we’re doing here, this is much more on
point as “starring” - which is more the idea of a “bookmark”, and i
think, whatsapp uses this wording to not raise false expectations
(still, i know of ppl that were quite upset that a “starred” message was
deleted when eg. the chat was cleared to save some memory)
wrt webxdc app updates: options that come into mind were: _empty_ (as
now), _snapshot_ (copy all updates) or _shortcut_ (always open
original). i am not sure what the best solution is, the easiest was
_empty_, so i went for that, as it is (a) obvious, and what is already
done with forwarding and (b) the original is easy to open now (in
contrast to forwarding).
so, might totally be that we need or want to tweak things here, but i
would leave that outside the first iteration, things are not worsened in
that area
wrt reactions: as things are detached, similar to webxdc updates, we do
not not to show the original reactions - that way, one can use reactions
for private markers (telegram is selling that feature :)
to the icon: a disk or a bookmark might be other options, but the star
is nicer and also know from other apps - and anyways a but vague UX
wise. so i think, it is fine
finally, known issue is that if a message was saved that does not exist
on another device, it does not get there. i think, that issue is a weak
one, and can be ignored mostly, most times, user will save messages soon
after receiving, and if on some devices auto-deletion is done, it is
maybe not even expected to have suddenly another copy there
</details>
EDIT: once this is merged, detailed issues about what to do should be
filed for android/desktop (however, they do not have urgency to adapt,
things will continue working as is)
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
This partially restores the fix from c9cf2b7f2e
that was removed during the addition of new group consistency at de63527d94
but only for "Member added" messages.
Multiple "Member added" messages happen
when the same QR code is processed multiple times
by multiple devices.
Some migrations change the `config` table, but they don't update the
cache. While this wasn't the cause for
https://github.com/deltachat/deltachat-core-rust/issues/6432, it might
have caused a similar bug, so, let's clear the config cache after every
migration.
Users report that in a setup with Android (1.50.4 from F-Droid) and Desktop (1.48.0 x86_64 .deb
release) and chatmail account `bcc_self` was reverted to 0 on Android, resulting in messages sent
from Android not appearing on Desktop. This might happen because of the bug in migration #127, it
doesn't handle `delete_server_after` > 1. Existing chatmail configurations having
`delete_server_after` != 1 ("delete at once") should get `bcc_self` enabled, they may be multidevice
configurations:
- Before migration #127, `delete_server_after` was set to 0 upon a backup export, but
then `bcc_self` is enabled instead (whose default is changed to 0 for chatmail).
- The user might set `delete_server_after` to a value other than 0 or 1 when that was
possible in UIs.
So let's add another migration fixing this. But still don't check `is_chatmail` for simplicity.
Using `repeat_vars()` to generate SQL statements led to some of them having more than
`SQLITE_MAX_VARIABLE_NUMBER` parameters and thus failing, so let's get rid of this pattern. But
let's not optimise for now and just repeat executing an SQL statement in a loop, all the places
where `repeat_vars()` is used seem not performance-critical and containing functions execute other
SQL statements in loops. If needed, performance can be improved by preparing a statement and
executing it in a loop. An exception is `lookup_chat_or_create_adhoc_group()` where `repeat_vars()`
can't be replaced with a query loop, there we need to replace the `SELECT` query with a read
transaction creating a temporary table which is used to perform the SELECT query then.
Before, `Chat::why_cant_send()` just returned `CantSendReason` after the first unsuccessful check
allowing to handle the result and finally send the message if the condition is acceptable in which
case the remaining checks are not done. This didn't result in any bugs, but to make the code more
robust let's add a functional parameter to filter failed checks without early return.
This implements new group consistency algorithm described in
<https://github.com/deltachat/deltachat-core-rust/issues/6401>
New `Chat-Group-Member-Timestamps` header is added
to send timestamps of member additions and removals.
Member is part of the chat if its addition timestamp
is greater or equal to the removal timestamp.
Python 3.7 is not supported on GitHub Actions ubuntu-latest runner:
https://github.com/actions/setup-python/issues/962
Python 3.7 has reached EOL more than 1 year ago anyway,
so not worth the effort supporting it.
This test keeps failing on macOS CI,
capturing events like `DC_EVENT_ACCOUNTS_ITEM_CHANGED`
before FailPlugin is setup.
These CI runners likely get less resources
because there is a limited number of them,
and this triggers this race condition.
Race is fixed by setting up fail plugin
before starting to capture events.
Currently Delta Chat puts self address in the To field
to avoid the To field being empty.
There is a plan to put empty `hidden-recipients`
group there, this fix prepares the receiver for such messages.
New Delta Chat is going to send self-sent messages
with undisclosed recipients instead of placing self into the `To` field.
To avoid assigning broadcast list messages to Saved Messages chat,
we should check the mailing list headers
before attempting to assign to Saved Messages.
First of all, chatmail servers normally forbid to send unencrypted mail, so if we know the peer's
key, we should encrypt to it. Chatmail setups have `E2eeEnabled=1` by default and this isn't
possible to change in UIs, so this change fixes the chatmail case. Additionally, for chatmail, if a
peer has `EncryptPreference::Reset`, let's handle it as `EncryptPreference::NoPreference` for the
reason above. Still, if `E2eeEnabled` is 0 for a chatmail setup somehow, e.g. the user set it via
environment, let's assume that the user knows what they do and ignore `IsChatmail` flag.
NB:
- If we don't know the peer's key, we should try to send an unencrypted message as before for a
chatmail setup.
- This change doesn't remove the "majority rule", but now the majority with
`EncryptPreference::NoPreference` can't disable encryption if the local preference is `Mutual`. To
disable encryption, some peer should have a missing peerstate or, for the non-chatmail case, the
majority should have `EncryptPreference::Reset`.
Change `BccSelf` default to 0 for chatmail configurations and enable it upon a backup export. As for
`DeleteServerAfter` who was set to 0 upon a backup export before, make its default dependent on
`BccSelf` for chatmail. We don't need `BccSelf` for chatmail by default because we assume
single-device use. Also `BccSelf` is needed for "classic" email accounts even if `DeleteServerAfter`
is set to "immediately" to detect that a message was sent if SMTP server is slow to respond and
connection is lost before receiving the status line which isn't a problem for chatmail servers.
Even if UIs don't call `Message::force_sticker()`, they don't want conversions of `Sticker` to
`Image` if it's obviously not an image, particularly, has non-image extension. Also UIs don't want
conversions of `Sticker` to anything other than `Image`, so let's keep the `Sticker` viewtype in
this case.
In multi-device case `vg-request-with-auth` left on IMAP may result in situation when Bob joins the
group, then leaves it, then second Alice device comes online and processes `vg-request-with-auth`
again and adds Bob back. So we should IMAP-delete `vg-request-with-auth`. Another device will know
the Bob's key from Autocrypt-Gossip. It's not a problem if Alice loses state (restores from an old
backup) or goes offline for long before sending `vg-member-added`, anyway it may not be delivered by
the server, rather Bob should retry sending SecureJoin messages as he is a part which wants to join,
so let's not solve this for now.
Add a `create` param to `select_with_uidvalidity()` instead of always trying to create the folder
and return `Ok(false)` from it if the folder doesn't exist and shouldn't be created, and handle this
in `store_seen_flags_on_imap()` by just removing "jobs" from the `imap_markseen` table. Also don't
create the folder in other code paths where it's not necessary.
This is to avoid revalidating HTTP cache too frequently (and have many parallel revalidation tasks)
if revalidation fails or the HTTP request takes some time. The stale period >= 1 hour, so 1 more
minute won't be a problem.
It was called from `receive_imf` when an outgoing message is received. But
`Imap::fetch_new_messages()` already calls `chat::mark_old_messages_as_noticed()` which does the job
better (per-message).
SQL statements fail if the number of variables
exceeds `SQLITE_LIMIT_VARIABLE_NUMBER`.
Remaining repeat_vars() calls are difficult to replace
and use arrays passed from the UI,
e.g. forwarded message IDs or marked as seen IDs.
UIs may want to display smth like "Transferring..." after "Establishing connection between
devices..." on nonzero progress. Before, progress on the receiver side was starting with 2 after
receiving enough data.
It only contained proxy
that some users ran in Termux
to look at IMAP traffic.
The same can be achieved with `socat`, e.g.:
socat -v TCP-LISTEN:9999,bind=127.0.0.1 OPENSSL:nine.testrun.org:993
--> libdeltachat is going to be deprecated and only exists because Android, iOS and Ubuntu Touch are still using it. If you build a new project, then please use the jsonrpc api instead.
[^1]: Out of date / unmaintained, if you like those languages feel free to start maintaining them. If you have questions we'll help you, please ask in the issues.
10. Update the version to the next development version:
`scripts/set_core_version.py 1.117.0-dev`.
11. Commit and push the change:
`git commit -m "chore: bump version to 1.117.0-dev" && git push origin main`.
12. Once the binaries are generated and [published](https://github.com/chatmail/core/releases),
check Windows binaries for false positive detections at [VirusTotal].
Either upload the binaries directly or submit a direct link to the artifact.
You can use [old browsers interface](https://www.virustotal.com/old-browsers/)
if there are problems with using the default website.
If you submit a direct link and get to the page saying
"No security vendors flagged this URL as malicious",
it does not mean that the file itself is not detected.
You need to go to the "details" tab
and click on the SHA-256 hash in the "Body SHA-256" section.
If any false positive is detected,
open an issue to track removing it.
See <https://github.com/chatmail/core/issues/7847>
for an example of false positive detection issue.
If there is a false positive "Microsoft" detection,
mark the issue as a blocker.
[VirusTotal]: https://www.virustotal.com/
## Dealing with antivirus false positives
If Windows release is incorrectly detected by some antivirus, submit requests to remove detection.
"Microsoft" antivirus is built in Windows and will break user setups so removing its detection should be highest priority.
To submit false positive to Microsoft, go to <https://www.microsoft.com/en-us/wdsi/filesubmission> and select "Submit file as a ... Software developer" option.
False positive contacts for other vendors can be found at <https://docs.virustotal.com/docs/false-positive-contacts>.
Not all of them may be up to date, so check the links below first.
Previously we successfully used the following contacts:
- [ESET-NOD32](mailto:samples@eset.com)
- [Symantec](https://symsubmit.symantec.com/)
## Dealing with failed releases
Once you make a GitHub release,
CI will try to build and publish [PyPI](https://pypi.org/) and [npm](https://www.npmjs.com/) packages.
If this fails for some reason, do not modify the failed tag, do not delete it and do not force-push to the `main` branch.
Fix the build process and tag a new release instead.
style="fill:#8c8c8c;fill-opacity:1;stroke:none;stroke-width:680.523;stroke-dasharray:none;paint-order:markers fill stroke"
id="rect1"
width="9951.9541"
height="9767.4756"
x="-71.697792"
y="-1012.83"
ry="0.43547946"/>
<path
d="m 2948.0033,5553.6941 q -130.7292,0 -228.7761,-96.3953 -98.0468,-96.3953 -98.0468,-224.9223 V 2447.6234 q 0,-128.527 98.0468,-224.9223 98.0469,-96.3953 228.7761,-96.3953 h 3703.9934 q 130.7292,0 228.776,96.3953 98.0469,96.3953 98.0469,224.9223 v 2784.7531 q 0,128.527 -98.0469,224.9223 -98.0468,96.3953 -228.776,96.3953 z M 4800,3936.3952 2948.0033,2742.1646 V 5232.3765 H 6651.9967 V 2742.1646 Z m 0,-321.3176 1830.2085,-1167.4541 h -3654.97 z m -1851.9967,-872.913 v -294.5412 2784.7531 z"
@@ -4,46 +4,16 @@ This crate provides a [JSON-RPC 2.0](https://www.jsonrpc.org/specification) inte
The JSON-RPC API is exposed in two fashions:
* A executable that exposes the JSON-RPC API through a [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) server running on localhost.
* The JSON-RPC API can also be called through the [C FFI](../deltachat-ffi). The C FFI needs to be built with the `jsonrpc` feature. It will then expose the functions `dc_jsonrpc_init`, `dc_jsonrpc_request`, `dc_jsonrpc_next_response` and `dc_jsonrpc_unref`. See the docs in the [header file](../deltachat-ffi/deltachat.h) for details.
* A executable`deltachat-rpc-server` that exposes the JSON-RPC API through stdio.
* The JSON-RPC API can also be called through the [C FFI](../deltachat-ffi). It exposes the functions `dc_jsonrpc_init`, `dc_jsonrpc_request`, `dc_jsonrpc_next_response` and `dc_jsonrpc_unref`. See the docs in the [header file](../deltachat-ffi/deltachat.h) for details.
We also include a JavaScript and TypeScript client for the JSON-RPC API. The source for this is in the [`typescript`](typescript) folder. The client can easily be used with the WebSocket server to build DeltaChat apps for web browsers or Node.js. See the [examples](typescript/example) for details.
We also include a JavaScript and TypeScript client for the JSON-RPC API. The source for this is in the [`typescript`](typescript) folder.
## Usage
#### Running the WebSocket server
From within this folder, you can start the WebSocket server with the following command:
```sh
cargo run --features webserver
```
If you want to use the server in a production setup, first build it in release mode:
```sh
cargo build --features webserver --release
```
You will then find the `deltachat-jsonrpc-server` executable in your `target/release` folder.
The executable currently does not support any command-line arguments. By default, once started it will accept WebSocket connections on `ws://localhost:20808/ws`. It will store the persistent configuration and databases in a `./accounts` folder relative to the directory from where it is started.
The server can be configured with environment variables:
|variable|default|description|
|-|-|-|
|`DC_PORT`|`20808`|port to listen on|
|`DC_ACCOUNTS_PATH`|`./accounts`|path to storage directory|
If you are targeting other architectures (like KaiOS or Android), the webserver binary can be cross-compiled easily with [rust-cross](https://github.com/cross-rs/cross):
The package includes a JavaScript/TypeScript client which is partially auto-generated through the JSON-RPC library used by this crate ([yerpc](https://github.com/Frando/yerpc/)). Find the source in the [`typescript`](typescript) folder.
The package includes a JavaScript/TypeScript client which is partially auto-generated through the JSON-RPC library used by this crate ([yerpc](https://github.com/chatmail/yerpc)). Find the source in the [`typescript`](typescript) folder.
To use it locally, first install the dependencies and compile the TypeScript code to JavaScript:
```sh
@@ -52,15 +22,7 @@ npm install
npm run build
```
The JavaScript client is not yet published on NPM (but will likely be soon). Currently, it is recommended to vendor the bundled build. After running `npm run build` as documented above, there will be a file `dist/deltachat.bundle.js`. This is an ESM module containing all dependencies. Copy this file to your project and import the DeltaChat class.
```typescript
import{DeltaChat}from'./deltachat.bundle.js'
constdc=newDeltaChat('ws://localhost:20808/ws')
constaccounts=awaitdc.rpc.getAllAccounts()
console.log('accounts',accounts)
```
The JavaScript client is [published on NPM](https://www.npmjs.com/package/@deltachat/jsonrpc-client).
A script is included to build autogenerated documentation, which includes all RPC methods:
```sh
@@ -73,18 +35,6 @@ Then open the [`typescript/docs`](typescript/docs) folder in a web browser.
#### Running the example app
We include a small demo web application that talks to the WebSocket server. It can be used for testing. Feel invited to expand this.
```sh
cd typescript
npm run build
npm run example:build
npm run example:start
```
Then, open [`http://localhost:8080/example.html`](http://localhost:8080/example.html) in a web browser.
Run `npm run example:dev` to live-rebuild the example app when files changes.
### Testing
The crate includes both a basic Rust smoke test and more featureful integration tests that use the TypeScript client.
@@ -104,14 +54,12 @@ cd typescript
npm run test
```
This will build the `deltachat-jsonrpc-server` binary and then run a test suite against the WebSocket server.
This will build the `deltachat-jsonrpc-server` binary and then run a test suite.
The test suite includes some tests that need online connectivity and a way to create test email accounts. To run these tests, talk to DeltaChat developers to get a token for the `testrun.org` service, or use a local instance of [`mailadm`](https://github.com/deltachat/docker-mailadm).
Then, set the `CHATMAIL_DOMAIN` environment variable to your testing email server domain.
The test suite includes some tests that need online connectivity and a way to create test email accounts. To run these tests, set the `CHATMAIL_DOMAIN` environment variable to your testing email server domain.
```
CHATMAIL_DOMAIN=chat.example.org npm run test
CHATMAIL_DOMAIN=ci-chatmail.testrun.org npm run test
- [ ] different test type to simulate two devices: to test autocrypt_initiate_key_transfer & autocrypt_continue_key_transfer
## MVP - Websocket server&client
For kaiOS and other experiments, like a deltachat "web" over network from an android phone.
- [ ] coverage for a majority of the API
- [ ] Blobs served
- [ ] Blob upload (for attachments, setting profile-picture, importing backup and so on)
- [ ] other way blobs can be addressed when using websocket vs. jsonrpc over dc-node
- [ ] Web push API? At least some kind of notification hook closure this lib can accept.
### Other Ideas for the Websocket server
- [ ] make sure there can only be one connection at a time to the ws
- why? , it could give problems if its commanded from multiple connections
- [ ] encrypted connection?
- [ ] authenticated connection?
- [ ] Look into unit-testing for the proc macros?
- [ ] proc macro taking over doc comments to generated typescript file
## Desktop Apis
Incomplete todo for desktop api porting, just some remainders for points that might need more work:
- [ ] manual start/stop io functions in the api for context and accounts, so "not syncing all accounts" can still be done in desktop -> webserver should then not do start io on all accounts by default
/// 400=vg-/vc-request-with-auth sent, typically shown as "alice@addr verified, introducing myself."
/// (Bob has verified alice and waits until Alice does the same for him)
/// 1000=vg-member-added/vc-contact-confirm received
progress: u16,
},
/// The connectivity to the server changed.
/// This means that you should refresh the connectivity view
@@ -253,22 +351,37 @@ pub enum EventType {
#[serde(rename_all = "camelCase")]
WebxdcStatusUpdate{
/// Message ID.
msg_id: u32,
/// Status update ID.
status_update_serial: u32,
},
/// Data received over an ephemeral peer channel.
#[serde(rename_all = "camelCase")]
WebxdcRealtimeData{msg_id: u32,data: Vec<u8>},
WebxdcRealtimeData{
/// Message ID.
msg_id: u32,
/// Realtime data.
data: Vec<u8>,
},
/// Advertisement received over an ephemeral peer channel.
/// This can be used by bots to initiate peer-to-peer communication from their side.
#[serde(rename_all = "camelCase")]
WebxdcRealtimeAdvertisementReceived{msg_id: u32},
WebxdcRealtimeAdvertisementReceived{
/// Message ID of the webxdc instance.
msg_id: u32,
},
/// Inform that a message containing a webxdc instance has been deleted
#[serde(rename_all = "camelCase")]
WebxdcInstanceDeleted{msg_id: u32},
WebxdcInstanceDeleted{
/// ID of the deleted message.
msg_id: u32,
},
/// Tells that the Background fetch was completed (or timed out).
/// This event acts as a marker, when you reach this event you can be sure
@@ -284,7 +397,10 @@ pub enum EventType {
/// Inform that a single chat list item changed and needs to be rerendered.
/// If `chat_id` is set to None, then all currently visible chats need to be rerendered, and all not-visible items need to be cleared from cache if the UI has a cache.
#[serde(rename_all = "camelCase")]
ChatlistItemChanged{chat_id: Option<u32>},
ChatlistItemChanged{
/// ID of the changed chat
chat_id: Option<u32>,
},
/// Inform that the list of accounts has changed (an account removed or added or (not yet implemented) the account order changes)
///
@@ -301,7 +417,60 @@ pub enum EventType {
AccountsItemChanged,
/// Inform than some events have been skipped due to event channel overflow.
EventChannelOverflow{n: u64},
EventChannelOverflow{
/// Number of events skipped.
n: u64,
},
/// Incoming call.
IncomingCall{
/// ID of the info message referring to the call.
msg_id: u32,
/// ID of the chat which the message belongs to.
chat_id: u32,
/// User-defined info as passed to place_outgoing_call()
place_call_info: String,
/// True if incoming call is a video call.
has_video: bool,
},
/// Incoming call accepted.
/// This is esp. interesting to stop ringing on other devices.
IncomingCallAccepted{
/// ID of the info message referring to the call.
msg_id: u32,
/// ID of the chat which the message belongs to.
chat_id: u32,
/// The call was accepted from this device (process).
from_this_device: bool,
},
/// Outgoing call accepted.
OutgoingCallAccepted{
/// ID of the info message referring to the call.
msg_id: u32,
/// ID of the chat which the message belongs to.
chat_id: u32,
/// User-defined info passed to dc_accept_incoming_call(
accept_call_info: String,
},
/// Call ended.
CallEnded{
/// ID of the info message referring to the call.
msg_id: u32,
/// ID of the chat which the message belongs to.
chat_id: u32,
},
/// One or more transports has changed.
///
/// UI should update the list.
///
/// This event is emitted when transport
/// synchronization messages arrives,
/// but not when the UI modifies the transport list by itself.
TransportsModified,
}
implFrom<CoreEventType>forEventType{
@@ -334,20 +503,24 @@ impl From<CoreEventType> for EventType {
contact_id: contact_id.to_u32(),
},
CoreEventType::IncomingReaction{
chat_id,
contact_id,
msg_id,
reaction,
}=>IncomingReaction{
chat_id: chat_id.to_u32(),
contact_id: contact_id.to_u32(),
msg_id: msg_id.to_u32(),
reaction: reaction.as_str().to_string(),
},
CoreEventType::IncomingWebxdcNotify{
chat_id,
contact_id,
msg_id,
text,
href,
}=>IncomingWebxdcNotify{
chat_id: chat_id.to_u32(),
contact_id: contact_id.to_u32(),
msg_id: msg_id.to_u32(),
text,
@@ -386,6 +559,9 @@ impl From<CoreEventType> for EventType {
"CAN NOT RUN COVERAGE correctly: Missing CHATMAIL_DOMAIN environment variable!\n\n",
"You can set COVERAGE_OFFLINE=1 to circumvent this check and skip the online tests, but those coverage results will be wrong, because some functions can only be tested in the online test"
"You can set COVERAGE_OFFLINE=1 to circumvent this check and skip the online tests, but those coverage results will be wrong, because some functions can only be tested in the online test",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
