`version_string` is meant to be displayed by UI, and comes from the
wire.
therefore, as a general precaution, ensure a string that is
regarded as a typical version string. all versions in scope are
currently v123.456.789-shortsuffix, where suffix is a-z and mostly
unused in production. that is the base. we can adapt if there is really
a need, but not for theoretical version strings. as we do not stop
processing, things are not bad even if we missed a valid usecase herr.
moreover, if `version_string` is empty, we skip the candidate - as we
cannot display something useful to the user. that little bit of care is
expected from relays :)
This replaces independent data structures
which allowed invalid state of having both the public keys
and the shared secret, or having a shared secret
but not encrypting the message because encryption_pubkeys
was None instead of Some(Vec::new()).
Creating QR code or sending a message will still fail
with a shorter error, but no info message will be created anymore.
Most users should have migrated the channels by recreating them by now.
This change is needed for moving loading of the shared secret
to earlier stages of message preparation,
otherwise mimefactory will have to create these info messages
when loading the message into memory.
"ex" means "used to be, but no longer is".
"ext" means "extended", which is the intended meaning.
"Ext" is more common not only in the Rust ecosystem but in general.
We already had several people (myself included) asking
what "ex" is supposed to mean.
To reproduce this commit, search and replace `_ex(?!\w)` -> `_ext`.
But don't change `list_transports_ex` because it's public API.
Unencrypted messages are no longer sent as multipart/mixed
with Message-ID header duplicated inside.
This was useful as non-standard protection
against servers that rewrite the Message-ID,
but now most messages are sent encrypted
and for them standard RFC 9788 header protection
works the same way.
it is outdated and incomplete, and always was experimental.
it is not in use in any UI but to get the name of the chat in Ubuntu
Touch, i created an issue for that at
https://codeberg.org/lk108/deltatouch/issues/305
moreover it is used for python bindings as get_summary(), not sure how
much that is in use,
we can leave the API for that for now, but we should not encourage
further usage of the API
---------
Co-authored-by: adb <adb@merlinux.eu>
> we aim to inform about updates for installations outside of any
appstore soon.
>
> there is already a PR on android at
https://github.com/deltachat/deltachat-android/pull/4582, however, the
information about "update available" is a mockup there.
>
> in general, there are 3 ideas around about how to gather the "update
available" infomation - (1) checking a central url, (2) let contacts
provide information, (3) let relay provide information. on various
one-to-one discussions, outcome is that (3) is the most reasonable way
to go.
this PR is about reading update information via IMAP metadata from the
relay.
it is up to the UI to call `get_app_version()` at a reasonable time and
frequency, see comment in the code. when called, `get_app_version()`
iterates over all known profiles and relays and checks for version
information, returning the newest for the given scope.
we do not use an event, as that is tricky wrt changes - we do not know
if other relays report later a newer version. we also do not cache
anything, to prevent bad relays avoiding us to update permanently. also
it is easier :)
<details>
<summary>outdated notes and questions</summary>
- ~~it is up to the clients to get the needed information, we could let
core filter, but it seems easy enough the other way round, and may have
debug advantages, one can iterate etc.~~ EDIT: we now filter in core,
this also makes the jsonrpc part easier, see review comments
- when is IMAP METADATA actually read? when are they ready? is that
really the correct place? i am up to change that, but beware, this is
not really my expertise, so someone else may need to take over :)
EDIT: see below, IMAP METADATA is read on connection, before fetching
starts, usually fast enough
- relay part is missing. once the format is settled and discussed
shortly here, that should be done soon as well. but this is definitely
not my expertise and needed to be done by someone else :)
- key for IMAP METADATA is `/shared/vendor/deltachat/appversions` -
shall we continue use `deltachat` for compatibility or so? `chatmail`
seems to be more correct
EDIT: we stay with the current
</details>
relay counterpart issue: https://github.com/chatmail/relay/issues/1037
cc @link2xt @Hocuri @hpk42
---------
Co-authored-by: holger krekel <holger@merlinux.eu>
There is currently no place to document up to date SQL schema.
New database is initialized with src/sql/tables.sql
and then updated by migrations,
So tracking down how some column is used requires grepping the code,
looking for a migration adding it and corresponding commits.
Co-Authored-By: biørn <r10s@b44t.com>
also core src/imap.rs retries connecting (2s with backoff) but direct_imap is only used in (CI) tests
and we can just keep trying every second until pytest timeout hits.
As we are using a dedicated CI relay instance, we don't need to be careful with creating test addresses.
But re-using test addresses in a test running in a single worker can leak events/messages which compromises test isolation.
Instead of encountering random flaky tests because let's just not cache addresses to maximize test isolation.
The chat is always going to be encrypted, this is enough.
We actually do mark the contact as verified,
but the meaning of this verification is unclear
and we don't even display it visibly in the chat anymore.
This value is not used for any logic
except the disappearing messages timer rollback prevention.
It is unlikely that non-chat clients
participate in chats with disappearing messages,
and the case was not tested anyway.
bail!() bubbles up the error until it lands in the device chat,
which is way over the top for a reaction that is filtered out by channel owner.
currently this happens, as arcance chat uses different default reactions.
in the future, this may happen even more often when we allow custom filters,
that may change over time.
Adds following stock strings:
- You were removed by %1$s.
- You were added by %1$s.
and implicit equivalents:
- You were removed.
- You were added.
Closes: #8419
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
previously only 10 messages were sent but if joinining did not succeed (due to slow CI sometimes) nothing would be received and the test would hang.
Also cleans up the tests to not set webxdc realtime config (true by default) and restructure helpers to reduce redundant code.
this PR adds the "message pinned state" to the "reaction state" that is
already sent together with resent channel messages since #8496.
this change is done in the first commit, and in fact only changes few
lines (one can consider to rename "broadcast_reactions.rs" and related
stuff to "broadcast_state.rs" in another PR: i did not to that in this
PR to keep diff and review simple).
moreover, in the second commit ,the "selection of messages to resent" is
changed to keep an additional focus on the pinned messages, so that the
most recent pinned messages are resent as well.
successor of #8496 and #8546
Adds `assert_warn`, `assert_error` and `assert_warns_or_errors`
methods to `TestContext`, that let us assert that
a certain warning or error is logged during the test.
Also asserts test logs should not contain any other
errors or warnings.
Adjusts tests accordingly.
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
Currently if there is an error fetching quota
then the HTML displays something like
```
example.com: Connected
Failed to parse
```
It's not clear that "Failed to parse" only refers to quota info.
Devices no longer implicitely use the From address of sync messages
to determine their primary transport. Receivers have no concept of it and own devices
may disagree on which relay is reachable because of VPN or different networks:
1. Make setting a primary transport (`configured_addr`) a per-device non-synced operation.
2. Transport rows (add/remove/unpublish) keep syncing like before.
3. A device reelects a primary if a sync message unpublished/removed the current primary
if there is a better candidate.
4. `TransportsModified` event is emitted at most once on an incoming transport sync message.
Users will notice the change in that changing primary transport in settings/advanced/relays
will not synchronize to other devices anymore.
Came across this while investigating more test_transport_synchronization flakiness,
sometimes missing TransportsModified events or getting a missing configured_addr.
The underlying problem was that stopping IO was triggered immediately during
receiving sync messages, potentially *canceling* the processing of the sync message,
effectively de-syncing the device's view on transports.
this PR, created together with @adbenitez, improves the connectivity
state passed to UI, which has changed a lot since multi relay:
- change the algorithm for `get_connectivity()`: this is roughly the
"best" connectivity of all relays now, so if one is connected, we're
already fine. this was discussed widely one to one, and that part
already closes#8554
- the PR adds a test for that, previously, that was untested
- additionally, do not regard unpublished relays in
`get_connectivity()`: e.g. unpublished relays are no longer given to
peers - so if only that is connected, the overall state should not be
"connected". therefore, we just ignore unpublished relays there
- in `get_connectivity_html()`, we continue showing unpublished relays,
however, we tune them down visually and flag them as such
- for the docs, remove the "range for some future use" wording. it was
never used like that, and that future will probably not arrive :)
<img width="320"
src="https://github.com/user-attachments/assets/1805c525-72d1-4197-b865-6e2d6fbc8819"
/>
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
the wire format is already prepared to carry information additionally to reactions,
so let the header name reflect that as well.
in practise, we might want to use a `pinned` flag very soon
in case pinned messages got resent in channels.
it makes sense if that flag goes to the message as such,
and not to an extra message.
and then it makes sense to not introduce a new things.
ftr, this is about resending state to other members,
it is not about syncing across devices for the same user.
the code as such is not changed by this PR,
it is only a wording preparation, of a header that is not even visible.
if we semantically put things other than reactions to the wire format,
we may want to move it out of `broadcast_reactions.rs`
the UIs are currently advised
to only allow the five default reactions in broadcast channels.
this PR ensures that from sending site as well as receiving site.
soon, we probably want to make the possible reactions configurable, this
PR is mainly for some safety until then. once we have an API to change
default reactions, we can also easily test "invalid" reactions on the
receiving side; currently only sending is tested
successor of https://github.com/chatmail/core/pull/8450
---------
Co-authored-by: Jagoda Estera Ślązak <128227338+j-g00da@users.noreply.github.com>
this PR adds support for reactions in broadcast channels.
> the idea of broadcast reactions is that they are sent as usual from
subscribers to owner. after some time, the owner broadcasts them to all
subscribers, who only get to see reaction+count, not who-reacted-what
the PR is quite large, but a good share are tests and otherwise many
things are straight forward.
review should be done by-file, not by-commit. to make review easier,
here is a high-level overview:
first a change in the existing internal `Reactions` object was required.
before this PR, `Reactions` had a "contact to reaction map" only, and
the "frequencies map", that are actually mainly needed for UI, were
calculated as needed. with this PR, the "frequencies map" is the field
that always exist, the "contact map" is only available on top of that.
moreover, this PR shifts that part to the core, it was unfortunately in
the bindings before.
with that preparation things done as follows:
1. reactions from broadcast channel subscriber (`Chattype::InBroadcast`)
to broadcast channel owner (`Chattype::OutBroadcast`) are sent as usual,
only change for that step was to allow sending them at all
2. the owner receives reactions and saves them to the existing
`reactions` table as usual. additionally, the changed message is
remembered in `reactions_need_broadcast` table
3. in the IMAP loop, when ~10 minutes have passed, and
`reactions_need_broadcast` contains entries, a single, hidden message
with accumulated reactions is sent. this message may contain reactions
to different messages. for each message, all known reactions are sent as
reaction+count.
4. subscriber receive that message and save the accumulated reactions in
`reactions_broadcasted`
6. `get_message_reactions` is adapted so that `frequencies` are set
independently of who-reacted-what (the old and only field).
who-reacted-what is called `by_contact` now. it is always set for
compatibility reasons, however, it is not exhaustive for subscribers.
in general, UI should work with frequencies, the API itself, however,
has not changed.
other tweaks:
- outgoing channels are muted on creation, and UI shall allow to
unmute/mute them as all other chats. reason is that reactions are
notified, but in many cases not of large interest. this is also what
telegram is doing
- to have an intermediate feedback when reacting, the local state should
include ones own reaction, even if it is not yet broadcasted. for that,
we modify `reactions_broadcasted` using `modify_frequencies()` as needed
when sending an reaction. there are still some situations where the
update may not include ones own reaction, in this case it is added
lately by `refine_frequencies()`, so that `get_message_reactions()`
always contain SELF.
(in a first implementation, we always increased SELF reaction in
refine_frequencies(), however, that was worse and led to SELF counted
twice once the owner sent broadcast)
<details>
<summary>wire format</summary>
wire format is a JSON in the `Broadcast-Reactions:` header.
additionally, `Content-Disposition: reaction` is set to not show the
hidden message on existing devices.
using a header also allows us to broadcast reactions with resent channel
messages (on joining) later.
```
{
"messages": [
{
"id": "12345678",
"reactions": [
{ "emoji": "👍", "count": 4 },
{ "emoji": "🎉", "count": 2 }
]
},
{
"id": "23456789",
"reactions": []
}
]
}
```
for `id`, the wire format needs to use `rfc724_mid` as `msg_id` are
local only.
</details>
### known issues
- if the channel owner uses multiple devices, broadcasted reaction
updates are sent from each device. the updates are not that big, so that
is probably not a big deal. if it turns out that this is an issue, we
can think about fixes in another PR. might be done by restarting our
10-minute-wait once we see an update from another device
- we cannot set contact_id for DC_EVENT_REACTIONS_CHANGED - but i doubt
it was ever used
### for another pr
- ~~add `Broadcast-Reactions:` header also for resent channel messages,
so that new subscriber do not only get the latest messages, but also
their reactions. for that, the `Broadcast-Reactions:` header can go to
the corresponding message, no need to send extra messages. we would need
to change the sending part to send all reactions for a given message. on
receiving part, we need to make sure, `receive_broadcast_reactions()` is
called when the message actually exist.~~
EDIT: subsequent PR for resending broadcast reactions at
https://github.com/chatmail/core/pull/8496
- add api to allow only a subset of reactions, fiter incoming reactions
before broadcasting
### misc.
ui pr: https://github.com/deltachat/deltachat-ios/pull/3225 and
https://github.com/deltachat/deltachat-android/pull/4560 , which both
were tested successfully with this core PR already. desktop is meant to
be done once this is merged
---------
Co-authored-by: l <link2xt@testrun.org>
This adds a test for the unencrypted headers, because
https://github.com/chatmail/core/pull/8345/ changes how these are
rendered, and so far we didn't have any tests for them.
This change separates rendering into two separate steps:
1. Rendering of the message payload without the From, Date and Autocrypt headers.
2. Adding the From, Date and Autocrypt headers and possibly encrypting the message.
The goal is to have serializable result of the first step
that can be persisted in the database and sent later with any email address.
This way it will be possible to send queued messages over any relay.
This will make it possible not to remove all messages from the queue
when the sending relay is changed.
Currently changing `configured_addr` deletes everything from `smtp` table.
This change is however only a refactoring and does not implement any features.
Otherwise users may add up to 5 relays, but 2 oldest relays
are actually ignored.
Having the maximum number of published relays
and the number of relays used the same between all clients
makes the order of relays in the key irrelevant.
We may even remove sorting by `add_timestamp`
from `get_all_self_addrs()` in the future.
There was a bug in prefetch_should_download() that it made it return
true for pre-messages even when they were already downloaded. This meant
that pre-messages were downloaded from all relays, rather than just one,
wasting internet data.
The fix is in rfc724_mid_download_tried(), which is used by
prefetch_should_download() to determine whether a message was already
downloaded.
---------
Co-authored-by: l <link2xt@testrun.org>
EventThread is created in Account.__init__,
but the thread start may be scheduled later.
If the thread only cals dc_get_event_emitter()
after some events have been emitted,
EventThread will never capture such events
and the tests will timeout waiting for the event.
This happened in CI on a Linux runner
with test_markseen_invalid_message_ids
timing out waiting for DC_EVENT_MSGS_CHANGED
right after sending a message.
If multi-relay onboarding config is set from UIs, automatically add
relays until there are 3 relays. For now, there will be a hardcoded list
of relay candidates.
- We need a list of chatmail relays that we somehow trust, and that
agree to be in the list. Then, we add all of them to the candidate list
(a new SQL table with colums "host" and "last_tried").
- Before going to IMAP IDLE: When there are less than 3 relays, fill it
up with relays from the candidate list. If creating an account fails,
try again with another relay from the list. For each candidate, we need
to remember the last time we tried to add a transport there, and try at
most once a week or so per candidate.
- For now, this will be behind an off-by-default config option, which at
least DC Android will enable when creating a new profile. UIs can then
opt in on their own pace but will likely need to disable it for tests.
- Right now, the backoff times are: Try to add a relay at most once per
hour, and try to add the same relay at most once per week
register device token if XDELTAPUSH IMAP capability is available (current relay setup provides it)
or if "maxsmtprecipients" IMAP metadata key is available (relays since May 2026 provide it),
allowing chatmail relay setups to drop XDELTAPUSH capability marker with the next release
while retaining push notification support.
The lint is reasonable, but without some work
to bring the future sizes way below the limit,
this lint is frequently triggered in unrelated PRs
that just add some local variables to async functions
or extend some structure that is passed on the stack.
Largest change is in the FFI crate.
With 2024 (but not 2021) edition unsafe code
inside unsafe functions should be marked separately
so we can mark exactly the code that is unsafe.
Some CFFI functions even have no unsafe code inside.
Most interesting change is that .strdup()
functions are not marked as unsafe anymore.
They are allocating memory and return raw pointers,
but there is nothing unsafe about it.
Only using the returned raw pointers is unsafe.
This way calls to .strdup() don't have to be marked
with unsafe{} blocks.
- treat an Autocrypt header as a kind of Autocrypt-Gossip: addr=<self>
letting it participate in the existing cooperative key gossipping mechanics.
- speed up processing of incoming autocrypt-gossip headers by using
1 sql commit for N gossip headers instead of the current N commits.
unsuspecting consumers will otherwise require execnet to run pytest with deltachat-rpc-client installed.
"import py" is discouraged for pytest these days but still supported in compatibility mode.
Unifies internal naming of single/direct/1:1/normal chats
to match the naming used in the API.
Closes: #8368
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
---------
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
`unwrap_or` is eagerly evaluated,
which caused always printing the WARN log,
even if EXIF orientation value is correct
(and not ignored).
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
The `FolderMeaning` and the `target_folder()` function and related code
were needed for the message moving logic. The message moving logic isn't
needed anymore, so, this PR is a first step to remove it.
We only ever look at one folder by now (usually the Inbox folder), so,
any message moving logic isn't needed anymore because we will never
notice a message that is supposed to be moved.
The `target` column of the `imap` table is mostly the same as the
`folder` column now. Except when the message is supposed to be deleted,
then it's an empty string.
fix https://github.com/chatmail/core/issues/8463
Followup to https://github.com/chatmail/core/pull/8431: Re-dd `oauth2`
to the two structs that serialized and sent over the wire. This solves
an incompatibility problem where transports could not be synced to older
versions of DC, and profiles could not be transferred, because the
deserializer expected the field to be present.
I now added `#[serde(default)]`, so that with this PR, Delta Chat will
be compatible with both v2.56 and with older versions.
BREAKING CHANGE: provider lookup APIs were removed from CFFI and JSON-RPC.
also removes offline provider database code and generated provider data,
provider-specific fields in configure/transport paths, and REPL providerinfo.
Heartbeat notifications are only used on iOS for classic mail servers
but both code and privacy wise (notification server sees IP addresses from those users)
not something we want to support any longer (was discussed with bjoern).
Remove is_chatmail flag, and rely on IMAP metadata
advertising the recipients limit (usually 1000),
falling back to 50 (or fewer in some exceptional cases)
just as before when is_chatmail was false.
Also, server metadata is now keyed per transport,
which in the future eases collecting ICE servers
from all relays (instead of just any first relay).
Enabling brotli does not add new dependencies,
we have it already.
Documentation at <https://docs.rs/rustls/0.23.42/rustls/compress/>
recommends enabling at least brotli.
zlib feature is not enabled as it pulls in duplicate zlib-rs 0.6.6
zlib was even never supported in Chromium (while brotli is).
It might still be interesting to enable it in the future
because it seems to be the only option supported by OpenSSL
and there may be servers that support zlib but not brotli.
RFC 8879 specifies zstd as well, but there is no feature to enable it yet.
This also brings back explicit TLS 1.2 feature that was reverted.
We don't want TLS 1.2 support to be accidentally dropped
if no dependencies explicitly require it anymore.
BREAKING CHANGE: removed oauth2 module, dc_get_oauth2_url FFI function, DC_LP_AUTH flags and configured/serverflags, and the oauth2 parameter/field from SMTP/IMAP clients, JSON-RPC interfaces, and CLI tools.
also contains regenerated provider data after dropping oauth in the update script.
https://docs.npmjs.com/trusted-publishers recommends node-version 24
and not updating npm separately, this change copies the recommended configuration.
Also update node version to 24 in non-release workflows, version 18 is EOL.
There is a version 26 already, but 24 is LTS.
Also disable cache for release pipelines.
A new table `pending_reactions` is now used,
to store reactions that arrived before the referenced message.
When receiving a new message, pending_reactions is checked and any
reactions that arrived earlier are applied.
Fixes: #8367
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
Unpublished relays aren't shown in the list of relays anymore, and can't
be deleted manually. Therefore, we want to automatically delete them in
order to make space for more relays when the limit (5) is reached and
the user wants to add another relay.
Before this change "Err" case spams the logs with
"Failed to download message rfc724_mid=...: IMAP location for ... post-message is unknown."
logged by every transports when the message is not available on any transport.
Un-escapes footer marks in long plain-text messages when retrieved with
`get_message_html`/`dc_get_msg_html`.
Additionally, makes escaping stricter,
only matching lines starting with `-- ` instead of `--`.
Fixes: #8269
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
Not clear why `GROUP BY` was needed -- a chat can't appear in the selected rows multiple times
because we do JOIN on the last message in the chat, there can't be multiple last messages.
This speeds up chatlisting by 3--4 times on my biggest profile, so the difference is visually
noticeable, particularly after dropping disk caches. Before, `msgs_index2` was used which has less
ordering and requires a full scan of `msgs` table.
Network errors such as timeouts should not break out of the loop
by bubbling up errors. There may be more candidates to try,
breaking out of the loop skips trying them.
Automatically remove relays that are hidden (`is_published=0`) and
haven't been used to receive new messages for over 90 days.
Closes: #8384
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
This is a follow-up to 8f51c7b9d5
which removed sync_seen_flags().
MODSEQ is a message attribute related to
CONDSTORE IMAP extension and is not used anymore
since CONDSTORE is not used.
BREAKING CHANGE: UIs should use `list_transports()` rather than `list_transports_ex()`, because unpublished transports count as removed from the user point of view, and should not be shown in the relay list anymore.
We use the timestamp of the latest transport modification
as the timestamp of the Direct Key Signature.
However, first transport is created before the public key
and existing transports may even have zero timestamp
if they were created before migration 142.
For interoperability with Sequoia-PGP,
make sure signatures always have a timestamp
that is not in the past compared to the timestamp
of the primary key.
This is necessary because
https://github.com/deltachat/sysadmin/issues/270 switches our
server-side authentication to rrsync, and that uploads to a restricted
path already; keeping it like this will result in a destination path
like `/var/www/html/var/www/html`.
Co-authored-by: j4n <j4n@systemli.org>
These messages are AUTH-encrypted and shalln't contain profile data as per the SecureJoin v3 spec,
otherwise the profile data can be seen e.g. by other broadcast subscribers.
An Iroh-Node-Addr message (realtime advertisement) may arrive earlier than the referenced webxdc in
case of multi-transport. By not creating a tombstone we still can receive it later on another
transport.
It's unclear whether this logic works, but I think we should at least
properly document why it exists (I already sent this text while figuring
out what the code does, so it was easy making this PR now)
Could add a XXX note that it's unsure whether this works
Alternative to https://github.com/chatmail/core/pull/8355/, fixes
https://github.com/chatmail/core/issues/8329
This fixes the following bug:
Scanning QR for existing contact doesn't update relay list
Two chat partners suddenly lost connectivity to all of their relays. They added another relay, which is accessible, and set it as main. Of course, they still couldn't communicate, because there was no way to announce a new relay list to each other (clients know only about old relays so they try to send updates there).
That's why the friends used a different channel (e.g. met IRL) to send their QR codes once again.
Expected behavior: They would expect that clients update their relay lists according to the contact info inside the QR codes, and they can continue talking in Delta.
Actual behavior: But Delta Chat client doesn't update a list of announced relays on scanning a QR in case a contact already exists. It simply opens the existing chat without altering any contact info.
---------
Co-authored-by: holger krekel <holger@merlinux.eu>
It is needed to speed up "DELETE FROM imap_markseen_new WHERE id = ?".
Otherwise if imap_markseen table grows large,
marking a lot of messages at once as seen and trying to delete them one by one
from imap_markseen table may have quadratic complexity.
BREAKING CHANGE: UIs should not behave differently for chatmail relays than for classical email servers; most usages of `is_chatmail` can be replaced by `force_encryption`
Prevents showing "empty" chat on other devices,
when pinning a chat that did not complete securejoin yet.
Fixes: #7713
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
This way they also can be processed by `markseen_msgs()` resulting in self-MDNs which improves
multi-device synchronization. MDNs to contacts won't really be sent because silent group changes
messages are self-removal messages which don't request MDNs.
Otherwise it's not possible to see the full message text if it's too long in a received edit. On the
sender side let it still be text so that the message can be edited again.
Maybe it's even good that MDNs will be sent for pre-messages only having placeholder text with the
viewtype and size, at least this way we notify the contact that we've noticed the message. Anyway,
with adding message previews the problem will be solved for the corresponding viewtypes.
With madmail 2.2.2 from https://github.com/themadorg/madmail/releases/download/v2.2.2/madmail-linux-amd64
using IMAP command SETMETADATA to set /private/devicetoken
returns error "A0003 BAD command not supported".
With this change connection is not aborted anymore
and the error is logged like this:
src/scheduler.rs:458: Transport 1: Failed to register push token: SETMETADATA command failed: bad response: code: None, info: Some("command not supported").
madmail 2.2.2 however fails to select the INBOX afterwards
and the following command `A0004 SELECT "INBOX"`
results in `* BAD command not supported` response.
Fixes https://github.com/chatmail/core/issues/8334
Arbitrary timeouts often result in flaky tests,
especially if CI runners may be paused.
In this case however timeout was not used by any tests
and only slowed down the tests in cases where
no message is expected but non-zero timeout is set.
- Ignore "vc-request-with-auth" if the sender is blocked.
- Ignore SecureJoin messages on an observing device if the joiner is blocked. Even if it's a
"vc-contact-confirm" message, it might be issued by another device before the joiner was blocked
on the observing device. Still, to avoid membership inconsistency on devices, don't ignore
"vg-member-added".
- Ignore "vc-request-pubkey" if the sender is blocked by address. We don't know sender's key yet, so
we should handle the sender as an address contact at this point.
The sender can generate a new key and still join, but it's at least an extra work and also, in case
of a group, other members will see that there's a new unknown member and be more careful.
Currently, the resolution of a resized image that was sent in a chat,
depends on the aspect-ratio.
Assuming the `balanced`-quality-setting is used,
a square image, that is larger than the limits for resolution and file-size,
will be resized to 1280x1280 (1,638,400 pixels),
an image with an aspect-ratio of 16:9,
will be resized to 1280x720 (921,600 pixels),
and if the aspect-ratio is 32:9, to 1280x360 (460,800 pixels).
This change makes it so, that the number of pixels,
in images with different aspect-ratios, will be similar.
Webxdc updates go to pre-message and when the message has no text,
the whole pre-message is trashed if it has a webxdc update.
The solution is not to trash pre-message in this case so preview is displayed.
I.e. don't fail `remove_contact_from_chat()` for such a contact because there may be a race
condition with a remote removal of the contact done without trace, but don't send a removal message
and sync message in this case and don't emit a `ChatModified` event.
If a contact is already a past member, we still send a removal message to the chat, this is safe and
protects from lost removal messages, so there's no need to complicate the code in this case.
aws-lc-rs is the default provider for Rustls.
We have been using ring as the default provider
to avoid duplicate dependency
because iroh supports only ring in 0.35.0,
but with version 1.0 it will be possible
to select other crypto providers
and we can switch to using more common default.
This change also enables "tls12" feature explicitly,
otherwise if there is no other dependency
requiring it, only TLS 1.3 will be supported
and we want to keep supporting TLS 1.2.
As of 3b29469102, all public keys except
for the currently-used one are ignored. But Delta Chat still tries to
use all _private_ keys for decryption.
This PR adds the information how many keys there are in the database. If
there is more than one, we know that the user imported their own key
back when that was possible. This info is interesting both for ourselves
and for the Cispa researchers that want to write a paper about Delta
Chat.
An alternative to this PR would be to make `key_create_timestamps`
actually contain the full list of key creation timestamps. I'm not sure
if that's worth the effort, though.
Follow-up to #8293
Fix https://github.com/chatmail/core/issues/8250. Not sure why anyone
would tamper with the link, but maybe the name gets ellipsized because
it is very long, or maybe we decide to ellipsize more aggressively in
the future.
The fix is easy enough, just set the GroupNameTimestamp when creating
the channel, so that `HeaderDef::ChatGroupNameTimestamp` gets set when
sending a message and the logic in
`apply_chat_name_avatar_and_description_changes()` recognizes the
incoming name's timestamp as newer than `chat_group_name_timestamp`
(which is 0 right after joining a chat).
The test is somewhat hacky and depends on the title being the last
parameter in the invite code URL, but this is fine; if we ever change
that, then the test will fail and we need to modify it.
This also removes some wrong doc comments.
Since groups are unpromoted until sending a first message, the same fix isn't needed for groups.
This hack was needed to "build" Android packages
by repackaging Linux packages on-the-fly,
but PyPI has since added Android support
and we are using it directly.
We have recently removed it from the normal "message info" dialog
at least on Delta Chat Desktop, so without this one can't easily find
the ID of a particular message,
which is useful for debugging.
crates.io recently started rejecting requests to https://crates.io/api/v1/crates
made by Nix: https://github.com/rust-lang/crates.io/issues/13482
This resulted in failing to download any dependencies
from crates.io with 403 error during Rust package builds.
The problem was solved in nixpkgs by switching
to CDN URL https://static.crates.io/crates
in https://github.com/NixOS/nixpkgs/pull/524985
As of writing this on 2026-05-29
the fix is on "master" branch of nixpkgs,
but not on "nixos-unstable" or "nixpkgs-unstable"
branches yet according to https://nixpk.gs/pr-tracker.html?pr=524985
so I have switched nixpkgs to "master" branch to get the problem fixed.
naersk needs a similar fix. I opened a PR for it
https://github.com/nix-community/naersk/pull/391
but because the fix is not merged yet,
switched to my PR branch in flake.nix.
Updating nixpkgs required some minor changes,
e.g. sphinx_rtd_theme has been renamed to sphinx-rtd-theme,
pthreads in Windows builds had to be moved to buildInputs to fix
"Refusing to evaluate package 'mingw_w64-pthreads-13.0.0'
in /nix/store/f78lkqnk63pd0kf52zf2wcx35p1nnalr-source/pkgs/os-specific/windows/mingw-w64/headers.nix:35
because it is not available on the requested hostPlatform"
Follow-up to 970222f. We shouldn't update the message state to `OutDelivered` if message sending
failed before and the message was re-sent successfully only to a new broadcast member.
receive_imf should not fail in this case,
but trash the post-message and updates.
The problem with previously used message::rfc724_mid_exists check
was that rfc724_mid_exists may return a trashed message.
"[::1]" is not a hostname, but an IPv6 address,
and should be passed as "::1" to tokio::net functions.
Otherwise connecting to proxy addresses such as `socks5://[::1]:9150`
fails by trying to resolve IPv6 address with DNS.
Fix https://github.com/chatmail/core/issues/8195
Supersedes https://github.com/chatmail/core/pull/8217
The most interesting change is in `delete_expired_imap_messages()`
because there, messages are marked for deletion on single-device
chatmail profiles.
The logic in `delete_expired_imap_messages()` was wrong before, and
pre-messages were not deleted at all. This is fixed by also querying
`pre_rfc724_mid` in addition to `rfc724_mid` from the `msgs` table.
In order not to make the SQL statement too complex, I split it into two.
WRT tests:
- `test_immediate_autodelete()` tests the auto-deletion;
`test_imap_autodelete_fully_downloaded_msg()` tests that even for a
message that is split into pre- and post-message, both messages are
deleted.
- A lot of tests test that if bcc_self is on, messages are not
auto-deleted. E.g. `test_one_account_send_bcc_setting()` and
`test_markseen_message_and_mdn()` relies on the fact that messages stay
on the server when bcc_self is on.
- Unfortunately, it is not possible to test that messages are only
deleted for chatmail servers, because we don't have any tests that use a
non-chatmail server.
This info is added at the top of the logs
and we don't want users to send us the logs
with the key fingerprint for debugging,
it is not really useful but identifies the profile.
This also solves the problem with the fact that
it's not clear from the resulting HTML
that this error message is referring to quota and not something else.
See https://github.com/chatmail/core/pull/8130.
This change is a preparation for ignoring
unencrypted messages by default.
New test_utils::encrypt_raw_message and
test_utils::receive_encrypted_imf are
used to encrypt the messages before
"receiving" them with receive_imf.
It turned out for desktop the style unicode-bidi: plaintext is needed.
dir="auto" didn't work. The style only without dir attribute worked too
but maybe it's better to have both. With this change
https://github.com/deltachat/deltachat-desktop/issues/6345 is also
solved for long messages that have a "Show full message" dialog
We log all connection attempts errors as they fail already,
but once all attempts are exhausted, we only log the first error
without specifying which address failed.
The first error is frequently the least interesting
"Network is unreachable (os error 101)" that happens
when trying to connect to IPv6 address from
a network that does not support IPv6.
To make reading the logs easier,
log all errors together with the addresses
again once all connection attempts are exhausted.
Then it will be visible that IPv6 failed
with "Network is unreachable (os error 101)"
and IPv4 failed with "Connection timeout: deadline has elapsed"
or similar error.
Before the change error looked like this:
IMAP failed to connect to example.org:143:starttls: Connection failure: Network is unreachable (os error 101).
With the change the error looks like this:
IMAP failed to connect to example.org:143:starttls: All connection attempts failed: Connection to [***::1]:143 failed: Network is unreachable (os error 101); Connection to [***::2]:143 failed: Network is unreachable (os error 101); Connection to x.x.x.1:143 timed out: deadline has elapsed; Connection to x.x.x.2:143 timed out: deadline has elapsed; Connection to x.x.x.3:143 timed out: deadline has elapsed.
This is needed to have support of v6 PQC keys
by the time users start generating profiles
using such keys.
Test key was generated with rsop/v0.10.0-16-gd98265f
(commit d98265f821e7bb181d06da1d634c5c4668d89e83)
using the command
cargo run --features draft-pqc generate-key \
--profile draft-ietf-openpgp-pqc-14-v6-ed25519-mlkem768x25519
This is not required by <https://datatracker.ietf.org/doc/html/rfc3156>.
Looks like Content-Description and Content-Disposition are
the same as what Thunderbird produces,
e.g. in `test-data/message/thunderbird_encrypted_signed.eml`,
and the same values also made it into Autocrypt spec.
Content-Description is likely not used by anyone.
For Content-Disposition specification see <https://datatracker.ietf.org/doc/html/rfc2183>.
While it is explicitly allowed to set the filename for inline attachment,
it does not look useful to specify generic "encrypted.asc" explicitly.
Content-Disposition for the second part results in some webmail clients
showing the second part as an attachment with "encrypted.asc" filename.
They will likely show it differently after this change,
e.g. with a different filename or actually inline.
`OutPreparing` is deprecated since 2024-12-07, replace it with `OutFailed`, such messages are
probably not interesting anymore. `OutMdnRcvd` is not used in the db for new messages since
a30c6ae1f7, `OutDelivered` is stored instead.
If we missed a message changing the primary transport, we shouldn't remove it when applying
a SyncTransports message, such a state doesn't look correct even if it's temporary.
Otherwise if it's forwarded to a device that already has the original
message, the pre-message isn't received and this results in a message
having no text and `Forwarded` param.
Co-authored-by: Hocuri <hocuri@gmx.de>
<https://github.com/swatinem/rust-cache>
caches only dependencies, and
dependencies don't change most of the time,
so PRs store the same cache
as already stored
by the main branch commit PRs are based on.
Hash of Cargo.{toml,lock} is part of the
cache key, so for dependency updating PRs
the cache key is new. Such PRs
rebuild everything from scratch,
which is a separate problem.
Storing such cache from PR is however
not useful because most of the time
dependency updating PR is going
to be rebased again before merging
as Dependabot PRs are opened in a batch
and then merged one by one
while Dependabot rebases remaining PRs.
Use add_or_update_transport() instead of deprecated configure(),
do not use deprecated get_next_messages(),
make use of AttrDict,
print invite link,
do not use % formatting for logging.
Currently, the Fingerprint type implements Display, but this doesn't get
you the canonical fingerprint representation, but something
human-readable. This is confusing, and back when I first used
`Fingerprint`, I immediately wrote a bug because of this. So, instead,
make a function `human_readable()` on Fingerprint.
This comes from the discussion at
https://github.com/chatmail/core/pull/8174#discussion_r3143130722.
Implementing `FromStr` and then calling `parse()` creates an
indirection, which is hard to follow for people who are not familiar
with Rust. @r10s recently voiced this problem when we were
pair-programming, and I agree.
We can decide what exactly to call the new function.
I didn't remove all `FromStr` implementations yet; `FromStr for
Fingerprint`, `FromStr for Params`, `FromStr for MozConfigTag` and
`FromStr for EphemeralTimer` are still left.
E.g. rename location::is_sending_locations_to_chat to location::is_sending_to_chat.
Then import only the location module and use it with location:: prefix,
similarly to how e.g. std::fs or std::tokio::fs avoids using "fs" or "file"
in function names.
There's no check for `from_id` so another group member can delete the
message, but this can only happen in case of message reordering and the
problem already exists for usual messages, so we may ignore it and
overall a group represents a scope of trust.
Co-authored-by: Hocuri <hocuri@gmx.de>
Remove unused info_only option from `get_chat_msgs_ex()`.
This option was meant to show an "audit log" of a group, i.e. only the
info messages. This feature was removed again from Desktop, but the
option still lingered around in Core.
This also adds a doc comment to the JsonRPC functions, because I
wanted to note somewhere that the parameter is deprecated, and I needed
some place to put this note.
Looking forward, it is unclear how exactly we want to handle errors, but
in the meantime, this is a simple thing we can do to improve the
situation.
If you have a broadcast channel, chances are that some of your
recipients have some problems receiving your messages. You are probably
not too interested in that.
In reality, some channel operators were quite confused by these errors,
and we told them to just ignore them, but at this point we may as well
just hide them.
Theoretically we can start removing recipients that never get our
messages at some point.
This causes problems for Delta Chat Desktop at
<https://github.com/deltachat/deltachat-desktop/pull/6278>
even though the logic was originally introduced for iOS.
If the problem remains on iOS,
heuristics can be added into iOS UI.
We shouldn't just revert ad7f873c68 because then we won't be able to
normally transfer a backup from a device having clock a bit in the future.
INDEXED BY msgs_index7 is to prevent SQLite from downgrading to using msgs_index2 which has less
ordering.
Last 10 messages in a broadcast channel are resent. They are sent and encrypted only to the new member, not to other subscribers.
Close#7678
Based on https://github.com/chatmail/core/pull/7854, with the following
changes:
- Refactor and simplify code, don't reuse the existing `get_chat_msgs()`
function
- Document that Param::Arg4 is also used for resent messages
cc818d9099
- It's unclear how exactly to resend webxdc status updates. After
discussing with @r10s, don't resend webxdc's at all for now.
38d57ebb30
- Don't set fake `msg_id` in resent messages
e7d0687d90
Setting the msg_id to `u32::MAX` is hacky, and may just as well break
things as it may fix things, because some code may use the msg.id to
load information from the database, like `get_iroh_topic_for_msg()`.
From reading the code, I couldn't find any problem with leaving the
correct `msg_id`, and if there is one, then we should add a function
parameter `is_resending` that is checked in the corresponding places.
Easiest to review file-by-file rather than individual commits, probably.
I'll squash-merge this.
---------
Co-authored-by: iequidoo <dgreshilov@gmail.com>
For SMTP errors we already format `last_send_error` with {:#},
but for IMAP errors we have formatted the errors with .to_string().
This resulted in errors such as
"Error: IMAP failed to connect to example.org:443:tls"
instead of
"Error: IMAP failed to connect to example.org:443:tls: Connection failure: Network is unreachable (os error 101)."
in the connectivity view HTML.
Broadcast subscribers can't change the chat name, so sending the "Chat-Group-Name{,-Timestamp}"
headers looks unnecessary. That could be useful for other subscriber's devices, but having only the
chat name isn't enough anyway, at least knowing the secret is necessary which is sent by the
broadcast owner.
Messages are iterated over in fetch_new_msg_batch()
and largest_uid_fetched variable is updated there
assuming that messages come in the order of increasing UID.
If UIDs are not increasing, it is possible
that largest_uid_fetched will be updated
even though smaller UID is not fetched yet
and the message will be lost.
INTERNALDATE sorting was introduced to
deal with email providers such as Gmail
that keep INTERNALDATE but not the UID
order when moving the messages.
Since we don't move the messages anymore
after commit 04c0e7da16,
there is no need for ordering by INTERNALDATE.
This change weakens TLS checks.
Every time we make a successful TLS connection,
we remember public key hash from the certificate
in relation to the hostname.
If later we connect to the same hostname and the public key does not change,
we skip checking certificate chain.
This way we will still connect successfully
even if certificate expires or becomes invalid for another reason,
but keeps the key.
We always check that certificate corresponds to the hostname.
We also do this for certificates starting with _
where we allow self-signed certificates,
so self-signed certificates with mismatching domains are not allowed.
Previously we did not check this for domains starting with _.
Since commit 25750de4e1
released in 2.36.0 we do not move messages to mvbox without explicit
mvbox_move setting, so do not need to watch it as well
as long as other devices are updated to the same change.
This makes such contacts appear in the contact list. `IncomingTo` is used because
`ChatId::accept_ex()` does so for groups, so sending a group message is effectively accepting the
group again in regards to contact searchability.
This fixes up b549e7633d which made it impossible to find contacts
from groups even if we've written there.
After talking with r10s:
For spring cleaning, remove the largely-unused things that can be done a
bit.
Most private messengers (WhatsApp/Signal/...) do not have this feature,
and we do not want to become Matrix where every client has different,
partly-incompatible features.
rfc724_mid_exists() returns the message id not only when the message is fully downloaded, so we need
to check this additionally. Also improve test_receive_pre_message to check that receiving the
pre-message again doesn't affect the existing message.
Otherwise `Param::GuaranteeE2ee` is set only after rendering the message
and some UIs, e.g. DC Android, display the message as unencrypted while
preparing the blob and rendering and encrypting the message. NB: DC
Desktop doesn't display the message until `send_msg()` returns.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
The resolution-limits for avatar-images are currently 512x512 or 256x256.
Reducing the resolution further, to 2/3 each step, can reduce the quality
much more than is necessary to fit within the file-size-limits,
which are currently 60 kB or 20 kB.
An image made entirely of noise (which results in unusually
large file-sizes), encoded with jpeg-quality 75,
and 4:2:2-colour-subsampling (the format currently used
for encoding images), can be below 60 kB at 227x227.
For the lower file-size-limit of 20 kB,
such images can be too large at 170x170, but fit at 149x149.
More normal images will have a lower file-size at the same resolution.
Before this change, the target-resolutions for resampling were:
512x512 -> 341x341 -> 227x227.
And for the lower file-size-limit:
256x256 -> 170x170 -> 113x113.
After this change, the target-resolutions for resampling will be:
512x512 -> 448x448 -> 392x392 -> 343x343 -> 300x300 -> 262x262 -> 229x229.
And for the lower file-size-limit, those will be:
256x256 -> 224x224 -> 196x196 -> 171x171 -> 149x149 -> 130x130 -> 113x113.
This does add 2 steps between the previous target-resolutions,
while still reaching target-resolutions close to the previous ones,
to reduce situations in which the quality will be lower than before.
Large size of Mimefactor.render() futures
increases the size of all callers
down to set_config() and background_fetch().
I also had to Box::pin one call to fetch_new_msg_batch
and one call to fetch_single_msg.
Before, if the user fixed their clock incorrectly set to the future, they needed to delete
previously sent messages or wait until this future comes again so that new sent messages are added
to the bottom. Strictly speaking, the problem isn't fixable because we don't know if messages were
incorrectly timestamped into the future or they are timestamped correctly and the clock is now
incorrectly set to the past. Anyway, adding messages to the middle of the chat isn't a good way to
inform the user about the problem.
I used some AI to draft a first version of this, and then reworked it.
This is one of multiple possibilities to fix
https://github.com/chatmail/core/issues/8041: For bots, the IncomingMsg
event is not emitted when the pre-message arrives, only when the
post-message arrives. Also, post-messages are downloaded immediately,
not after all the small messages are downloaded.
The `get_next_msgs()` API is deprecated. Instead, bots need to listen to
the IncomingMsg event in order to be notified about new events. Is this
acceptable for bots?
THE PROBLEM THAT WAS SOLVED BY THIS:
With pre-messages, it's hard for bots to wait for the message to be fully downloaded and then process it.
Up until now, bots used get_next_msgs() to query the unprocessed messages, then set last_msg_id after processing a message to that they won't process it again.
But this will now also return messages that were not fully downloaded.
ALTERNATIVES:
In the following, I will explain
the alternatives, and for why it's not so easy to just make the
`get_next_msgs()` API work. If it's not understandable, I'm happy to
elaborate more.
Core can't just completely ignore the pre-message for two reasons:
- If a post-message containing a Webxdc arrives later, and some webxdc updates arrive in the meantime, then these updates will be lost.
- The post-message doesn't contain the text (reasoning was to avoid duplicate text for people who didn't upgrade yet during the 2.43.0 rollout)
There are multiple solutions:
- Add the message as hidden in the database when the pre-message arrives.
- When the post-message arrives and we want to make it available for bots, we need to update the msg_id because of how the `get_next_msgs()` API works. This means that we need to update all webxdc updates that reference this msg_id.
- Alternatively, we could make webxdc's reference the rfc724_mid instead of the msg_id, so that we don't need stable msg_ids anymore
- Alternatively, we could deprecate `get_next_msgs()`, and ask bots to use plain events for message processing again. It's not that bad; worst case, the bot crashes and then forgets to react to some messages, but the user will just try again. And if some message makes the bot crash, then it might actually be good not to try and process it again.
- Store the pre-message text and `PostMsgMetadata` (or alternatively, the whole mime) in a new database table. Wait with processing it until the post-message arrives.
Additionally, the logic that small messages are downloaded before post-messages should be disabled for bots, in order to prevent reordering.
We already set sort_timestamp to 0 for "Messages are end-to-end encrypted."
since 8f1bf963b4.
Do this for "Others will only see this group after you sent a first message."
and "Messages in this chat use classic email and are not encrypted." as well
so no messages can be added on top.
Workflow checks that PRs are made only when current version ends with -dev
If this fails, a commit bumping the version to -dev should be pushed to main branch.
This avoids creating 1:1 chat on a second device when joining a channel.
Now when joining a channel there may be no 1:1 chat with the inviter
when the channel is created. In this case we still create the channel
as unblocked even if 1:1 chat would be a contact request
because joining the channel is an explicit action
and it is not possible to add someone who did not scan a QR
to the channel manually.
We don't want to send an outgoing message from the 2nd device (`ac1_clone`) before receiving the
incoming one and expect that the messages will be ordered correctly on the 1st device (`ac1`). Let's
ensure the correct message order locally in the first place. I checked logs of a failed test run and
it indeed happened that `ac1_clone` sent the message earlier, so it can't reference the incoming
message and `tweak_sort_timestamp()` does nothing on `ac1`, so the messages can't be ordered
correctly considering that smeared clocks on the devices are diferent.
The part of logic there adjusting the sort timestamp forward if the parent message has a greater
sort timestamp wasn't tested explicitly by any test. I only saw one unrelated "golden test" failure
when commented it out.
(Related to #8027)
Fix https://github.com/chatmail/core/issues/8042
The problem was that after receiving the bcc_self'ed pre-message in
`receive_imf`, the logic there only looked for a pending
`smtp`-table-entry that matches the rfc724_mid, and if there was none
then it thought "Great, apparently the message is fully sent out, we can
mark it as delivered!".
But with pre-messages, the same message can have two `smtp` entries (one
for the pre-message and one for the post-message), and the message
should only be marked as delivered once both of them are sent out.
Now, I changed the logic to look for all entries with the same msg_id.
This is actually the same SQL query used in smtp.rs, so, I extracted it
into a new function; feel free to suggest a better name for it.
I tested on Android that it now works fine.
I'll add a test in a follow-up PR.
There are a lot of other problems with sending large files, though:
- The pre-message is sent before the post-message, so that for the
receiver it looks as if the message arrived, but stays in
"downloading..." forever
- There is quite a time delay between clicking on "Send" and the
outgoing message appearing in the chat
- The message shortly gets a letter icon right after it is sent
- I'm wondering if there is a way to give feedback to the user
immediately if the message is too big
- It's unclear when exactly we want to send read receipts
I'll open a follow-up issue for these.
This is to avoid sorting incoming messages that
are slightly in the past above system messages
about SecureJoin. SecureJoin messages are
timed according to smeared timestamp,
so even in the local tests they are in the future
by a few seconds.
We set timestamp of this info message to 0
to make it always appear in the beginning of the chat.
To avoid new chats being sorted to the end of the chatlist,
we ignore such 0 and use chat creation timestamp
when sorting the chatlist.
We know which message was added from the return value
of receive_imf(). It may be that the first chat
in the chatlist is not the one where the message was received
if there is a pinned chat or if
just received message is old.
It is not clear now what this is testing.
Golden test shows messages ordered
incorrectly according to the timestamps,
they should be ordered the other way round.
Comment talks about fetching from mvbox and inbox
in paralell which is a rare case that
could have happened if one message is left in the inbox
and the other message is a chat message moved to mvbox.
We never download anything that is not moved to the target folder.
The test also resides in "verified chats" tests
which are all legacy tests we kept after
replacing the concept of verified/protected chats
with key contacts in 2.x.
If our key is gossiped, the message is intended for us.
The check for address is redundant for incoming messages as
if we received the message then it was addressed to us.
This whole protection code can eventually be removed
as we have intended recipient fingerprints already,
it only protects against forwarding of messages
sent by old clients.
With `ORDER BY` statement SQLite searches
the `imap` table by `transport_id` and for each found row
scans the whole `imap_markseen` table.
Number of `imap` entries for each `transport_id`
is usually large as we need to know
which UIDs to delete on IMAP server
when deleting a message.
```
sqlite> EXPLAIN QUERY PLAN
SELECT imap.id, uid, folder FROM imap, imap_markseen
WHERE imap.id = imap_markseen.id
AND imap.transport_id=?
AND target = folder
ORDER BY folder, uid;
QUERY PLAN
|--SEARCH imap USING INDEX sqlite_autoindex_imap_1 (transport_id=?)
`--SCAN imap_markseen
```
Without `ORDER BY` statement SQLite scans `imap_markseen`
table which is expected to be small,
and then searches `imap` table by `rowid` for each found result.
```
sqlite> EXPLAIN QUERY PLAN
SELECT imap.id, uid, folder FROM imap, imap_markseen
WHERE imap.id = imap_markseen.id
AND imap.transport_id=?
AND target = folder;
QUERY PLAN
|--SCAN imap_markseen
`--SEARCH imap USING INTEGER PRIMARY KEY (rowid=?)
```
Query planning was tested with SQLite 3.52.0.
It is possible to explictly make
query planner move sorting to the last step
with `ORDER +folder, +uid`, but this is not recommended
in SQLite documentation
(see <https://www.sqlite.org/optoverview.html#uplus>).
It is also possible to add indexes,
but indexes use space,
adding them requires an SQL migration,
and each index needs to be updated so it will slow down writes.
Only inbox loop is changed because non-inbox loop is going to be removed
together with `mvbox_move`.
Added transport IDs to the log and logging around quota updates.
Removed some logs that add noise,
like logging that IDLE is supported each time right before using it.
The original idea was to always lock `write_mutex` before acquiring an `InnerPool.semaphore` permit
to avoid ABBA deadlocks, but when refactoring a PR for b696a242fc,
that was forgotten.
This doesn't really change the program flow as we have `Context::housekeeping_mutex` anyway,
just simplifies the code.
If we learn about this message being available on IMAP later,
we will add another available_post_msgs row.
If we don't delete the row, we will keep failing each time
until IMAP entry becomes available and it may not happen.
We cannot upgrade the crate because it is a transitive dependency
and the issue described in
<https://rustsec.org/advisories/RUSTSEC-2026-0049>
is not dangerous because it requiers a compromised CA
and revoked certificate. Worst case that happens
with iroh is that outer layer of encryption to
iroh relay is compromised, but iroh traffic is
still encrypted between peers without relying on CAs.
Previously, if the mvbox_move folder is missing, then core will loop
infinitely, because `new_mail` is never set to false.
The fix is to first set `new_mail` to false, then return if the folder
is missing.
This is the bug @hpk42 experienced when commenting in
https://github.com/chatmail/core/issues/7989
---------
Co-authored-by: holger krekel <holger@merlinux.eu>
Follow-up to https://github.com/chatmail/core/pull/7994/, in order to
prevent clashes with other things that are called `Transport`, and in
order to make the struct name more greppable
Closes https://github.com/chatmail/core/issues/7980.
Unpublished transports are not advertised to contacts, and self-sent messages are not sent there, so that we don't cause extra messages to the corresponding inbox, but can still receive messages from contacts who don't know the new relay addresses yet.
- This adds `list_transports_ex()` and `set_transport_unpublished()` JsonRPC functions
- By default, transports are published, but when updating, all existing transports except for the primary one become unpublished in order not to break existing users that followed https://delta.chat/legacy-move
- It is not possible to unpublish the primary transport, and setting a transport as primary automatically sets it to published
An alternative would be to change the existing list_transports API rather than adding a new one list_transports_ex. But to be honest, I don't mind the _ex prefix that much, and I am wary about compatibility issues. But maybe it would be fine; see b08ba4bb8 for how this would look.
this adds an api to make the newest incoming message of a chat as "fresh",
so that UI can offer a "mark chat unread" option as usual for messengers
(eg. swipe right on iOS toggels between "read" and "unread").
"mark unread" is one of the most requested missing features,
used by many ppl to organize their every day messenger usage -
tho "pinning" and "saved messages" are similar,
it seems to be missed often.
we follow a very simple approach here
and just reset the state to `MessageState::InFresh`.
this does not introduce new states or flows.
therefore, chats without any incoming message cannot be marked as fresh.
in practise, this is probably not really an issue,
as the "mark fresh" is usually used to undo a "mark noticed" operation -
and then you have incoming message.
also, most status messages as "all messages are e2ee" count as incoming.
to avoid double sending of MDN,
we remove `Param::WantsMdn` once the MDN is scheduled.
in case MDN are used for syncing, MDN is still sent as before.
many other messenger show a "badge without number",
if we want that as well,
we can always track the "manually set as fresh" state in a parameter.
but for now, it is fine without and showing a "1", which alsso makes sense as badges may be summed up.
there is an iOS pr that uses this new feature,
jsonrpc is left out until api is settled.
also out of scope is synchronisation -
main reason is that "mark noticed" is not synced as well, so we avoid an imbalance here.
both, "mark noticed" as well as "mark fresh" should be synced however,
as soon as this feature is merged.
Hidden messages are marked as seen
when chat is marked as noticed.
MDNs to such messages should not be sent
as this notifies the hidden message sender
that the chat was opened.
The issue discovered by Frank Seifferth.
Can be reviewed commit-by-commit.
This fixes another silly thing you can do with securejoinv3: show Bob a
QR code with auth token that is a broadcast channel secret of a known
channel, then never respond. Bob will decrypt messages from the channel
and drop them because they are sent by the "wrong" sender.
This can be avoided with domain separation, instead of
encrypting/decrypting securejoinv3 messages directly with auth token,
encrypt/decrypt them with `securejoin/<auth token>` as the secret or
even `securejoinv3/<alice's fingerprint>/<auth token>`. For existing
broadcast channels we cannot do this, but for securejoinv3 that is not
released yet this looks like an improvement that avoids at least this
problem.
Credits to link2xt for noticing the problem.
This also adds Alice's fingerprint to the auth tokens, which
was pretty easy to do. I find it hard to develop an intuition for
whether this is important, or whether we will be annoyed by it in the
future.
**Note:** This means that QR code scans will not work if one of the chat
partners uses a self-compiled core between c724e2981 and merging this PR
here. This is fine; we will just have to tell the other developers to
update their self-compiled cores.
The tests were originally generated with AI and then reworked.
Follow-up to https://github.com/chatmail/core/pull/7754 (c724e29)
This prevents the following attack:
/// Eve is subscribed to a channel and wants to know whether Alice is also subscribed to it.
/// To achieve this, Eve sends a message to Alice
/// encrypted with the symmetric secret of this broadcast channel.
///
/// If Alice sends an answer (or read receipt),
/// then Eve knows that Alice is in the broadcast channel.
///
/// A similar attack would be possible with auth tokens
/// that are also used to symmetrically encrypt messages.
///
/// To prevent this, a message that was unexpectedly
/// encrypted with a symmetric secret must be dropped.
encryption info needs a dedicated string for "Messages are end-to-end encrypted"
as the UI will add more infomation to the info messages,
smth. as "Tap for more information".
an alternative fix would have been to let the UI render the info-message
differently, but adding another string to core causes less friction.
If one of broadcast owner's devices didn't add a new subscriber for any reason, e.g. because of
missing SecureJoin messages, this device shall add "member added" messages when syncing the member
list from the `SetPgpContacts` message.
The test sometimes fails because of wrong message ordering for bob:
[...]
Waiting for the device of alice@example.org to reply… [NOTICED][INFO]
<Msg#2010🔒: (Contact#Contact#2001): hi [FRESH]
Msg#2008🔒: (Contact#Contact#2001): You joined the channel. [FRESH][INFO]
>Msg#2010🔒: (Contact#Contact#2001): hi [FRESH]
Msg#2011🔒: (Contact#Contact#2001): Member Me removed by alice@example.org. [FRESH][INFO]
This adds `SystemTime::shift(Duration::from_secs(1))` as a workaround.
Co-authored-by: Hocuri <hocuri@gmx.de>
dc_jsonrpc_init called Arc::from_raw on the account_manager pointer, which took ownership of the caller's refcount. When the local Arc dropped at the end of the function, the refcount was decremented, leaving the C side's pointer with a stolen refcount. This caused a use-after-free race between dc_accounts_unref and dc_jsonrpc_unref at shutdown.
Wrap in ManuallyDrop to prevent the implicit drop, keeping the caller's refcount intact.
Regression introduced in #7662.
This change is mainly to avoid exposing the write lock outside the pool module.
To avoid deadlocks, outside code should work only with the pooled connections
and use no more than one connection per thread.
With multiple transports there are multiple inbox loops on the same profile `Context`.
They tend to start running housekeeping at the same time, e.g. when deleting
a message with an attachment, and then `remove_unused_files()`
tries to remove the same files that are already deleted by another thread
and logs errors.
Don't use first-person form in placeholder texts,
as these can be misleading when broadcasted to group.
Additionally ensures that broadcasted system messages
are not localized to not leak locally-set language
to the group chat.
Fixes#7930
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
Don't depend on these 3 cleartext headers for the question whether we
download a message.
This PR will waste a bit of bandwidth for people who use the legacy
show_emails option; apart from that, there is no user-visible change
yet. It's a preparation for being able to remove these headers, in order
to further reduce unencrypted metadata.
Removing In-Reply-To and References will be easy; removing Chat-Version
must happen at least one release after the PR here is released, so that
people don't miss messages. Also, maybe some nerds depend on the
Chat-Version header for server-side filtering of messages, but we shall
have this discussion at some other time.
For the question whether a message should be moved, we do still depend
on them; this will be fixed with
https://github.com/chatmail/core/pull/7780.
When both this PR and #7780 are merged, we can stop requesting
Chat-Version header during prefetch.
I assume that the problem was that sometimes, alice2 or fiona doesn't
accept alice's smeared timestamp, because `calc_sort_timestamp()`
doesn't allow the timestamp of a received message to be in the future. I
tried this patch:
```diff
diff --cc src/chat.rs
index 9565437cf,9565437cf..a2e4f97d0
--- a/src/chat.rs
+++ b/src/chat.rs
@@@ -46,6 -46,6 +46,7 @@@ use crate::receive_imf::ReceivedMsg
use crate::smtp::{self, send_msg_to_smtp};
use crate::stock_str;
use crate::sync::{self, Sync::*, SyncData};
++use crate::timesmearing::MAX_SECONDS_TO_LEND_FROM_FUTURE;
use crate::tools::{
IsNoneOrEmpty, SystemTime, buf_compress, create_broadcast_secret, create_id,
create_outgoing_rfc724_mid, create_smeared_timestamp, create_smeared_timestamps, get_abs_path,
@@@ -1212,7 -1212,7 +1213,11 @@@ SELECT id, rfc724_mid, pre_rfc724_mid,
received: bool,
incoming: bool,
) -> Result<i64> {
-- let mut sort_timestamp = cmp::min(message_timestamp, smeared_time(context));
++ let mut sort_timestamp = cmp::min(
++ message_timestamp,
++ // Add MAX_SECONDS_TO_LEND_FROM_FUTURE in order to allow other senders to do timesmearing, too:
++ smeared_time(context) + MAX_SECONDS_TO_LEND_FROM_FUTURE,
++ );
let last_msg_time: Option<i64> = if always_sort_to_bottom {
// get newest message for this chat
```
...maybe this patch makes sense anyways, but you still get the problem
that the message sent by alice2 (i.e. the add-fiona message) will have
an earlier timestamp than the message sent by alice, because alice
already sent more messages, and therefore has more timesmearing-seconds.
It's unsure it makes sense to modify calc_sort_timestamp() this way because if some chat member has the clock in the future (even unintentionally), their fresh messages will be sorted to the bottom relatively to others' fresh messages. Maybe it's even better to limit the message timestamp ("Date") by the current system time there.
To really fix the problem, we could send a serial number together with the timestamp, that distinguishes two messages sent in the same second. But since we haven't gotten complaints about message ordering since some time, let's just leave things as they are.
Since all this timesmearing is a bit best-effort right now, I decided to
instead just make the test more relaxed.
fix https://github.com/chatmail/core/issues/7880
depends on #7754 (merged)
With this change, a securejoin message is just ignored if the contact
was deleted in the meantime; apparently the user is not interested in
the securejoin process anymore if they deleted the contact.
But other, parallel securejoin processes must not be affected; the test
also tests this.
This was initially done in the IMAP loop
to set is_chatmail for existing users.
They should all have the setting configured
by now unless they install some very old backup.
Setting during the configuration is needed
for Delta Chat Desktop because it caches the value
internally:
<https://github.com/deltachat/deltachat-desktop/issues/6068>
Close https://github.com/chatmail/core/issues/7396. Before reviewing,
you should read the issue description of
https://github.com/chatmail/core/issues/7396.
I recommend to review with hidden whitespace changes.
TODO:
- [x] Implement the new protocol
- [x] Make Rust tests pass
- [x] Make Python tests pass
- [x] Test it manually on a phone
- [x] Print the sent messages, and check that they look how they should:
[test_secure_join_group_with_mime_printed.txt](https://github.com/user-attachments/files/24800556/test_secure_join_group.txt)
- [x] Fix bug: If Alice has a second device, then Bob's chat won't be
shown yet on that second device. Also, Bob's contact isn't shown in her
contact list. As soon as either party writes something into the chat,
the that shows up and everything is fine. All of this is still a way
better UX than in WhatsApp, where Bob always has to write first 😂
Still, I should fix that.
- This is actually caused by a larger bug: AUTH tokens aren't synced if
there is no corresponding INVITE token.
- Fixed by 6b658a0e0
- [x] Either make a new `auth_tokens` table with a proper UNIQUE bound,
or put a UNIQUE bound on the `tokens` table
- [x] Benchmarking
- [x] TODOs in the code, maybe change naming of the new functions
- [x] Write test for interop with older DC (esp. that the original
securejoin runs if you remove the &v=3 param)
- [x] From a cryptography perspective, is it fine that vc-request is
encrypted with AUTH, rather than a separate secret (like INVITE)?
- [x] Make sure that QR codes without INVITE work, so that we can remove
it eventually
- [x] Self-review, and comment on some of my code changes to explain
what they do
- [x] ~~Maybe use a new table rather than reusing AUTH token.~~ See
https://github.com/chatmail/core/pull/7754#discussion_r2728544725
- [ ] Update documentation; I'll do that in a separate PR. All necessary
information is in the https://github.com/chatmail/core/issues/7396 issue
description
- [ ] Update tests and other code to use the new names (e.g.
`request-pubkey` rather than `request` and `pubkey` rather than
`auth-required`); I'll do that in a follow-up PR
**Backwards compatibility:**
Everything works seamlessly in my tests. If both devices are updated,
then the new protocol is used; otherwise, the old protocol is used. If
there is a not-yet-updated second device, it will correctly observe the
protocol, and mark the chat partner as verified.
Note that I removed the `Auto-Submitted: auto-replied` header from
securejoin messages. We don't need it ourselves, it's a cleartext header
that leaks too much information, and I can't see any reason to have it.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
There is no need to store copy of public key
next to the secret key because public key is a subset of the secret key
and can be obtained by using SignedSecretKey.public_key()
or SignedSecretKey.to_public_key().
Rpc.start() now calls get_system_info() after launching the server
to verify it started successfully. If the server exits early (e.g.
due to an invalid accounts directory), the core error message from
stderr is captured and included in the raised JsonRpcError.
The reader_loop now unblocks pending RPC requests when the server
closes stdout, so callers never hang on a dead server.
Export JsonRpcError from the deltachat_rpc_client package.
Add test_early_failure verifying that Rpc.start() raises with
the actual core error message for invalid accounts directories.
Timestamp renewal was introduced in 1dbf924c6a "feat:
chat::resend_msgs: Guarantee strictly increasing time in the Date header" so that re-sent messages
can be deduplicated on the reciver side, but the deduplication logic doesn't depend on "Date"
anymore.
fix https://github.com/chatmail/core/issues/7863
`test_import_encrypted_bak_into_encrypted_acct` CFFI test fails because
it tests that trying to import an encrypted account with a wrong
passphrase into an already-encrypted database will fail, but leave the
already-encrypted database (esp, leave it with the same password).
But in order to reset the database after a failed login attempt, I'm
using this code:
```rust
context.sql.close().await;
fs::remove_file(context.sql.dbfile.as_path())
.await
.log_err(context)
.ok();
context
.sql
.open(context, "".to_string()) // <-- NOTE THIS LINE
.await
.log_err(context)
.ok();
```
We're not remembering the password, so, we can't just pass the correct
password there.
Since password-protected databases are not really supported anyways, we
decided to just skip the test.
I also tried two tricks for deleting everything [found on
Stackoverflow](https://stackoverflow.com/questions/525512/drop-all-tables-command),
but neither of them managed to actually reset the database (i.e. they
led to a failed Rust test, because asserting
`!context2.is_configured().await?` failed):
```rust
context
.sql
.call_write(|conn| {
let mut stmt = conn.prepare(
"select 'drop table ' || name || ';' from sqlite_master where type = 'table';",
)?;
let mut iter = stmt.query(())?;
while iter.next()?.is_some() {}
Ok(())
})
.await
.log_err(context)
.ok();
context
.sql
.run_migrations(context)
.await
.log_err(context)
.ok();
```
```rust
context
.sql
.transaction(|t| {
t.execute_batch(
"
PRAGMA writable_schema = 1;
delete from sqlite_master where type in ('table', 'index', 'trigger');
PRAGMA writable_schema = 0",
)?;
Ok(())
})
.await
.log_err(context)
.ok();
context
.sql
.run_migrations(context)
.await
.log_err(context)
.ok();
```
---------
Co-authored-by: l <link2xt@testrun.org>
I sometimes find it hard to find the most recent migration.
This PR moves the migrations::run() function to the end of the file, so
that it's easy to find the most recent migration - it's at the end of
the file.
There are no changes except for switching the ordering of the functions.
The webxdc file name itself isn't informative for users. Still, send and display it if the webxdc
manifest can't be parsed, it's better than sending "Mini App" and this isn't a normal case anyway.
This code does not expect the variable to be unset,
so use indexing to fail with KeyError instead.
Otherwise getenv() returns None which is then converted to "none" string by formatting
and the test only fails because of connection attempts to "none" domain.
fix#7877
The bug was: If there is no chat description, and the chat description
is set to an empty string, the INSERT statement inserted a row with an
empty chat description, and therefore from the view of the INSERT
statement, something changed.
This PR fixes this by simply loading the chat description first, and
comparing it.
instead of Alice saying to Bob "You changed the chat description",
we now say "[Chat description changed, please update ...]
i was also considering to say "[Chat description changed to:\n\n...]"
but then there is no incentive for ppl to update, and chat descriptions
for chat creation would still be missing. and this is probably far more
often used.
successor of https://github.com/chatmail/core/pull/7829
Iroh-Gossip-Topic is sent in a post-message. Post-message goes to trash,
so topic should be associated with the existing pre-message that is
updated rather than with the post-message.
Fix https://github.com/chatmail/core/issues/7835.
The problem was most probably:
- `ac1_clone` receives the sync message, sends `TRANSPORTS_MODIFIED`
event, and launches a task that will restart IO
- After IO was stopped, but before it is started again,
`ac1_clone.add_transport_from_qr(qr)` is called
- this check fails:
```rust
ensure!(
!self.scheduler.is_running().await,
"cannot configure, already running"
);
```
fix https://github.com/chatmail/core/issues/7766
Implementation notes:
- Descriptions are only sent with member additions, when the description
is changed, and when promoting a previously-unpromoted group, in order
not to waste bandwith.
- Descriptions are not loaded everytime a chat object is loaded, because
they are only needed for the profile. Instead, they are in their own
table, and can be loaded with their own JsonRPC call.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Existing test relies on folder scanning.
We are going to remove the option to not show emails
(<https://github.com/chatmail/core/issues/7631>)
so the test will be removed eventually anyway.
This includes forwarding of long messages. Also this fixes sending, but more likely resending of
forwarded messages for which the original message was deleted, because now we save HTML to the db
immediately when creating a forwarded message.
Co-authored-by: Hocuri <hocuri@gmx.de>
It has a really complex logic, so it's better to avoid calling it if possible than think which side
effects and performance penalties it has. It was never called here before adding forwarding messages
across contexts (accounts).
For now, do this only for `OneOneChat` and `MailingListOrBroadcast`, this is enough to correctly
support messages from modern Delta Chat versions sending Intended Recipient Fingerprint subpackets
and single-recipient messages from modern versions of other MUAs.
It only looks up contacts by address in the given chat, so `_fallback_to_chat` suffix is more
informative. It's obvious that such a lookup is done by address, there are no other reasonable
options.
This is an important thing forgotten to be checked in 332527089.
Also there's another test which currently doesn't work as we want: outgoing encrypted messages
continue to arrive to ad-hoc group even if we already have contact's key. This should be fixed by
sending and receiving Intended Recipient Fingerprint subpackets.
The good thing is that apparently there are no scenarios requiring the contact to update their
software, the user should just update own devices.
This is not possible for webxdcs and vCards currently however, so add workarounds for them:
- Use translated "Mini App" as the webxdc name.
- Use just "👤" instead of the vCard summary (i.e. the vCard contact name).
I.e. create a non-replyable ad-hoc group in such cases. Unencrypted replies to encrypted messages
are a security issue and "Composing a Reply Message" from RFC 9787 and "Replying and Forwarding
Guidance" from RFC 9788 forbid such replies.
If From is an address-contact, it mustn't be able to modify an encrypted group. If From is a
key-contact, it mustn't be added to members of an unencrypted group.
See: https://support.delta.chat/t/high-quality-avatar/1052/8
This removes an additional resolution-reduction for images that fit
within the resolution-limit.
The additional reduction of the image-resolution was added in
b7864f232b.
It seems that back then only resolution-reduction was used to make the
file-size of images smaller, and the purpose was to skip an unnecessary
encoding-step.
However, encoding images with [jpeg-quality set to
`75`](a6b2a54e46/src/blob.rs (L392)),
as it is currently done, can significantly reduce the file-size, even if
the resolution is the same. As the resolution of the image that will be
encoded is rather low when it fits within the resolution-limit, further
reducing it can significantly reduce the quality of the image.
Thus it is better to not skip the encoding-step at the original
resolution.
this PR cleans up with some easy, deprecated stuff.
i roughly checked that they are no longer in use - by these checks,
other deprecated function were kept, eg.
dc_provider_new_from_email_with_dns() and dc_chat_is_protected() is
still used by deltatouch - to not add noise there, we remove them here
on the next cleanup ...
for DC_STR_*, however, if they are in used, the corresponding lines
should just be removed
this will cleanup https://c.delta.chat/deprecated.html as well
resolves#7724: When forwarding a message with file to another profile, the file was not copied to the target $blobdir and so the forwarded message missed it
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
If `Message::load_from_db_optional()` or `set_msg_failed()` fails, we
shouldn't early-return. Because it's important that the line
`execute("DELETE FROM smtp WHERE id=?", (rowid,))` is executed in order
to prevent an infinite loop, if one of these functions fails.
Otherwise if the user reads messages being offline and then the device comes online, sent MDNs will
remove all notifications from other devices even if new messages have arrived. Notifications not
removed at all look more acceptable.
2026-01-20 18:34:10 -03:00
599 changed files with 26271 additions and 25362 deletions
10. Update the version to the next development version:
`scripts/set_core_version.py 1.117.0-dev`.
11. Commit and push the change:
`git commit -m "chore: bump version to 1.117.0-dev" && git push origin main`.
12. Once the binaries are generated and [published](https://github.com/chatmail/core/releases),
check Windows binaries for false positive detections at [VirusTotal].
Either upload the binaries directly or submit a direct link to the artifact.
You can use [old browsers interface](https://www.virustotal.com/old-browsers/)
if there are problems with using the default website.
If you submit a direct link and get to the page saying
"No security vendors flagged this URL as malicious",
it does not mean that the file itself is not detected.
You need to go to the "details" tab
and click on the SHA-256 hash in the "Body SHA-256" section.
If any false positive is detected,
open an issue to track removing it.
See <https://github.com/chatmail/core/issues/7847>
for an example of false positive detection issue.
If there is a false positive "Microsoft" detection,
mark the issue as a blocker.
[VirusTotal]: https://www.virustotal.com/
## Dealing with antivirus false positives
If Windows release is incorrectly detected by some antivirus, submit requests to remove detection.
"Microsoft" antivirus is built in Windows and will break user setups so removing its detection should be highest priority.
To submit false positive to Microsoft, go to <https://www.microsoft.com/en-us/wdsi/filesubmission> and select "Submit file as a ... Software developer" option.
False positive contacts for other vendors can be found at <https://docs.virustotal.com/docs/false-positive-contacts>.
Not all of them may be up to date, so check the links below first.
Previously we successfully used the following contacts:
- [ESET-NOD32](mailto:samples@eset.com)
- [Symantec](https://symsubmit.symantec.com/)
## Dealing with failed releases
Once you make a GitHub release,
CI will try to build and publish [PyPI](https://pypi.org/) and [npm](https://www.npmjs.com/) packages.
If this fails for some reason, do not modify the failed tag, do not delete it and do not force-push to the `main` branch.
Fix the build process and tag a new release instead.
@@ -59,6 +59,13 @@ If column is already declared without `NOT NULL`, use `IFNULL` function to provi
Use `HAVING COUNT(*) > 0` clause
to [prevent aggregate functions such as `MIN` and `MAX` from returning `NULL`](https://stackoverflow.com/questions/66527856/aggregate-functions-max-etc-return-null-instead-of-no-rows).
List columns explicitly in `INSERT` statements:
```
INSERT OR IGNORE INTO download (rfc724_mid, msg_id) VALUES (?,0);
```
Otherwise if a new column with default value is added in a future DB version, an upgraded DB can't
be used with the old code, e.g. after transferring a DB from a device running a newer version.
Don't delete unused columns too early, but maybe after several months/releases, unused columns are
still used by older versions, so deleting them breaks downgrading the core or importing a backup in
an older version. Also don't change the column type, consider adding a new column with another name
@@ -68,6 +75,12 @@ keyword doesn't help here.
Consider adding context to `anyhow` errors for SQL statements using `.context()` so that it's
possible to understand from logs which statement failed. See [Errors](#errors) for more info.
When changing complex SQL queries, test them on a new database with `EXPLAIN QUERY PLAN`
to make sure that indexes are used and large tables are not going to be scanned.
Never run `ANALYZE` on the databases,
this makes query planner unpredictable
and may make performance significantly worse: <https://github.com/chatmail/core/issues/6585>
- The secondary self addresses (see below) are shown in the UI, but not editable.
- When the user changed the email address in the configure screen, show a dialog to the user, either directly explaining things or with a link to the FAQ (see "Other" below)
Changes in the core
-------------------
- [x] We have one primary self address and any number of secondary self addresses. `is_self_addr()` checks all of them.
- [x] If the user does a reconfigure and changes the email address, the previous address is added as a secondary self address.
- don't forget to deduplicate secondary self addresses in case the user switches back and forth between addresses).
- The key stays the same.
- [x] No changes for 1:1 chats, there simply is a new one. (This works since, contrary to group messages, messages sent to a 1:1 chat are not assigned to the group chat but always to the 1:1 chat with the sender. So it's not a problem that the new messages might be put into the old chat if they are a reply to a message there.)
- [ ] When sending a message: If any of the secondary self addrs is in the chat's member list, remove it locally (because we just transitioned away from it). We add a log message for this (alternatively, a system message in the chat would be more visible).
- [x] ([#3385](https://github.com/deltachat/deltachat-core-rust/pull/3385)) When receiving a message: If the key exists, but belongs to another address (we may want to benchmark this)
AND there is a `Chat-Version` header\
AND the message is signed correctly
AND the From address is (also) in the encrypted (and therefore signed) headers <sup>[[1]](#myfootnote1)</sup>\
AND the message timestamp is newer than the contact's `lastseen` (to prevent changing the address back when messages arrive out of order) (this condition is not that important since we will have eventual consistency even without it):
Replace the contact in _all_ groups, possibly deduplicate the members list, and add a system message to all of these chats.
- Note that we can't simply compare the keys byte-by-byte, since the UID may have changed, or the sender may have rotated the key and signed the new key with the old one.
<aname="myfootnote1">[1]</a>: Without this check, an attacker could replay a message from Alice to Bob. Then Bob's device would do an AEAP transition from Alice's to the attacker's address, allowing for easier phishing.
<details>
<summary>More details about this</summary>
Suppose Alice sends a message to Evil (or to a group with both Evil and Bob). Evil then forwards the message to Bob, changing the From and To headers (and if necessary Message-Id) and replacing `addr=alice@example.org;` in the autocrypt header with `addr=evil@example.org;`.
Then Bob's device sees that there is a message which is signed by Alice's key and comes from Evil's address and would do the AEAP transition, i.e. replace Alice with Evil in all groups and show a message "Alice changed their address from alice@example.org to evil@example.org". Disadvantages for Evil are that Bob's message will be shown on Alice's device, possibly creating confusion/suspicion, and that the usual "Setup changed for..." message will be shown the next time Evil sends a message (because Evil doesn't know Alice's private key).
Possible mitigations:
- if we make the AEAP device message sth. like "Automatically removed alice@example.org and added evil@example.org", then this will create more suspicion, making the phishing harder (we didn't talk about what what the wording should be at all yet).
- Add something similar to replay protection to our Autocrypt implementation. This could be done e.g. by adding a second `From` header to the protected headers. If it's present, the receiver then requires it to be the same as the outer `From`, and if it's not present, we don't do AEAP --> **That's what we implemented**
Note that usually a mail is signed by a key that has a UID matching the from address.
That's not mandatory for Autocrypt (and in fact, we just keep the old UID when changing the self address, so with AEAP the UID will actually be different than the from address sometimes)
> The content of the user id packet is only decorative
</details>
### Notes:
- We treat protected and non-protected chats the same
- We leave the aeap transition statement away since it seems not to be needed, makes things harder on the sending side, wastes some network traffic, and is worse for privacy (since more people know what old addresses you had).
- As soon as we encrypt read receipts, sending a read receipt will be enough to tell a lot of people that you transitioned
- AEAP will make the problem of inconsistent group state worse, both because it doesn't work if the message is unencrypted (even if the design allowed it, it would be problematic security-wise) and because some chat partners may have gotten the transition and some not. We should do something against this at some point in the future, like asking the user whether they want to add/remove the members to restore consistent group state.
#### Downsides of this design:
- Inconsistent group state: Suppose Alice does an AEAP transition and sends a 1:1 message to Bob, so Bob rewrites Alice's contact. Alice, Bob and Charlie are together in a group. Before Alice writes to this group, Bob and Charlie will have different membership lists, and Bob will send messages to Alice's new address, while Charlie will send them to her old address.
#### Upsides:
- With this approach, it's easy to switch to a model where the info about the transition is encoded in the PGP key. Since the key is gossiped, the information about the transition will spread virally.
- Faster transition: If you send a message to e.g. "Delta Chat Dev", all members of the "sub-group" "delta android" will know of your transition.
### Alternatives and old discussions/plans:
- Change the contact instead of rewriting the group member lists. This seems to call for more trouble since we will end up with multiple contacts having the same email address.
- If needed, we could add a header a) indicating that the sender did an address transition or b) listing all the secondary (old) addresses. For now, there is no big enough benefit to warrant introducing another header and its processing on the receiver side (including all the necessary checks and handling of error cases). Instead, we only check for the `Chat-Version` header to prevent accidental transitions when an MUA user sends a message from another email address with the same key.
- The condition for a transition temporarily was:
> When receiving a message: If we are going to assign a message to a chat, but the sender is not a member of this chat\
> AND the signing key is the same as the direct (non-gossiped) key of one of the chat members\
> AND ...
However, this would mean that in 1:1 messages can't trigger a transition, since we don't assign private messages to the parent chat, but always to the 1:1 chat with the sender.
<details>
<summary>Some previous state of the discussion, which temporarily lived in an issue description</summary>
Summarizing the discussions from https://github.com/deltachat/deltachat-core-rust/pull/2896, mostly quoting @hpk42:
1. (DONE) At the time of configure we push the current primary to become a secondary.
2. When a message is sent out to a chat, and the message is encrypted, and we have secondary addresses, then we
a) add a protected "AEAP-Replacement" header that contains all secondary addresses
b) if any of the secondary addresses is in the chat's member list, we remove it and leave a system message that we did so
3. When an encrypted message with a replacement header is received, replace the e-mail address of all secondary contacts (if they exist) with the new primary and drop a sysmessage in all chats the secondary is member off. This might (in edge cases) result in chats that have two or more contacts with the same e-mail address. We might ignore this for a first release and just log a warning. Let's maybe not get hung up on this case before everything else works.
Notes:
- for now we will send out aeap replacement headers forever, there is no termination condition other than lack of secondary addresses. I think that's fine for now. Later on we might introduce options to remove secondary addresses but i wouldn't do this for a first release/PR.
- the design is resilient against changing e-mail providers from A to B to C and then back to A, with partially updated chats and diverging views from recipients/contacts on this transition. In the end, you will have a primary and some secondaries, and when you start sending out messages everybody will eventually synchronize when they receive the current state of primaries/secondaries.
- of course on incoming message for need to check for each stated secondary address in the replacement header that it uses the same signature as the signature we verified as valid with the incoming message **--> Also we have to somehow make sure that the signing key was not just gossiped from some random other person in some group.**
- there are no extra flags/columns in the database needed (i hope)
#### Downsides of the chosen approach:
- Inconsistent group state: Suppose Alice does an AEAP transition and sends a 1:1 message to Bob, so Bob rewrites Alice's contact. Alice, Bob and Charlie are together in a group. Before Alice writes to this group, Bob and Charlie will have different membership lists, and Bob will send messages to Alice's new address, while Charlie will send them to her old address.
- There will be multiple contacts with the same address in the database. We will have to do something against this at some point.
The most obvious alternative would be to create a new contact with the new address and replace the old contact in the groups.
#### Upsides:
- With this approach, it's easier to switch to a model where the info about the transition is encoded in the PGP key. Since the key is gossiped, the information about the transition will spread virally.
- (Also, less important: Slightly faster transition: If you send a message to e.g. "Delta Chat Dev", all members of the "sub-group" "delta android" will know of your transition.)
- It's easier to implement (if too many problems turn up, we can still switch to another approach and didn't waste that much development time.)
- The user is responsible that messages to the old address arrive at the new address, for example by configuring the old provider to forward all emails to the new one.
Notes during implementing
========================
- As far as I understand the code, unencrypted messages are unsigned. So, the transition only works if both sides have the other side's key.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
