With multi-relay, it can happen that one of the relays can't connect or is stuck in "Connecting...". In this case, right now we show "Not Connected"/"Connecting" as the connectivity, which multiple users already complained about, esp. since it's not obvious how to remove an unpublished, not working relay.
Instead, the new logic is:
- If any relay is working, then we signal this to the user, to signal that new messages may come in.
- Only show "Connecting..." or "Not Connected" if all of the relays are failing to connect, because if any of the relays can connect then things are mostly fine.
the wire format is already prepared to carry information additionally to reactions,
so let the header name reflect that as well.
in practise, we might want to use a `pinned` flag very soon
in case pinned messages got resent in channels.
it makes sense if that flag goes to the message as such,
and not to an extra message.
and then it makes sense to not introduce a new things.
ftr, this is about resending state to other members,
it is not about syncing across devices for the same user.
the code as such is not changed by this PR,
it is only a wording preparation, of a header that is not even visible.
if we semantically put things other than reactions to the wire format,
we may want to move it out of `broadcast_reactions.rs`
the UIs are currently advised
to only allow the five default reactions in broadcast channels.
this PR ensures that from sending site as well as receiving site.
soon, we probably want to make the possible reactions configurable, this
PR is mainly for some safety until then. once we have an API to change
default reactions, we can also easily test "invalid" reactions on the
receiving side; currently only sending is tested
successor of https://github.com/chatmail/core/pull/8450
---------
Co-authored-by: Jagoda Estera Ślązak <128227338+j-g00da@users.noreply.github.com>
this PR adds support for reactions in broadcast channels.
> the idea of broadcast reactions is that they are sent as usual from
subscribers to owner. after some time, the owner broadcasts them to all
subscribers, who only get to see reaction+count, not who-reacted-what
the PR is quite large, but a good share are tests and otherwise many
things are straight forward.
review should be done by-file, not by-commit. to make review easier,
here is a high-level overview:
first a change in the existing internal `Reactions` object was required.
before this PR, `Reactions` had a "contact to reaction map" only, and
the "frequencies map", that are actually mainly needed for UI, were
calculated as needed. with this PR, the "frequencies map" is the field
that always exist, the "contact map" is only available on top of that.
moreover, this PR shifts that part to the core, it was unfortunately in
the bindings before.
with that preparation things done as follows:
1. reactions from broadcast channel subscriber (`Chattype::InBroadcast`)
to broadcast channel owner (`Chattype::OutBroadcast`) are sent as usual,
only change for that step was to allow sending them at all
2. the owner receives reactions and saves them to the existing
`reactions` table as usual. additionally, the changed message is
remembered in `reactions_need_broadcast` table
3. in the IMAP loop, when ~10 minutes have passed, and
`reactions_need_broadcast` contains entries, a single, hidden message
with accumulated reactions is sent. this message may contain reactions
to different messages. for each message, all known reactions are sent as
reaction+count.
4. subscriber receive that message and save the accumulated reactions in
`reactions_broadcasted`
6. `get_message_reactions` is adapted so that `frequencies` are set
independently of who-reacted-what (the old and only field).
who-reacted-what is called `by_contact` now. it is always set for
compatibility reasons, however, it is not exhaustive for subscribers.
in general, UI should work with frequencies, the API itself, however,
has not changed.
other tweaks:
- outgoing channels are muted on creation, and UI shall allow to
unmute/mute them as all other chats. reason is that reactions are
notified, but in many cases not of large interest. this is also what
telegram is doing
- to have an intermediate feedback when reacting, the local state should
include ones own reaction, even if it is not yet broadcasted. for that,
we modify `reactions_broadcasted` using `modify_frequencies()` as needed
when sending an reaction. there are still some situations where the
update may not include ones own reaction, in this case it is added
lately by `refine_frequencies()`, so that `get_message_reactions()`
always contain SELF.
(in a first implementation, we always increased SELF reaction in
refine_frequencies(), however, that was worse and led to SELF counted
twice once the owner sent broadcast)
<details>
<summary>wire format</summary>
wire format is a JSON in the `Broadcast-Reactions:` header.
additionally, `Content-Disposition: reaction` is set to not show the
hidden message on existing devices.
using a header also allows us to broadcast reactions with resent channel
messages (on joining) later.
```
{
"messages": [
{
"id": "12345678",
"reactions": [
{ "emoji": "👍", "count": 4 },
{ "emoji": "🎉", "count": 2 }
]
},
{
"id": "23456789",
"reactions": []
}
]
}
```
for `id`, the wire format needs to use `rfc724_mid` as `msg_id` are
local only.
</details>
### known issues
- if the channel owner uses multiple devices, broadcasted reaction
updates are sent from each device. the updates are not that big, so that
is probably not a big deal. if it turns out that this is an issue, we
can think about fixes in another PR. might be done by restarting our
10-minute-wait once we see an update from another device
- we cannot set contact_id for DC_EVENT_REACTIONS_CHANGED - but i doubt
it was ever used
### for another pr
- ~~add `Broadcast-Reactions:` header also for resent channel messages,
so that new subscriber do not only get the latest messages, but also
their reactions. for that, the `Broadcast-Reactions:` header can go to
the corresponding message, no need to send extra messages. we would need
to change the sending part to send all reactions for a given message. on
receiving part, we need to make sure, `receive_broadcast_reactions()` is
called when the message actually exist.~~
EDIT: subsequent PR for resending broadcast reactions at
https://github.com/chatmail/core/pull/8496
- add api to allow only a subset of reactions, fiter incoming reactions
before broadcasting
### misc.
ui pr: https://github.com/deltachat/deltachat-ios/pull/3225 and
https://github.com/deltachat/deltachat-android/pull/4560 , which both
were tested successfully with this core PR already. desktop is meant to
be done once this is merged
---------
Co-authored-by: l <link2xt@testrun.org>
This adds a test for the unencrypted headers, because
https://github.com/chatmail/core/pull/8345/ changes how these are
rendered, and so far we didn't have any tests for them.
This change separates rendering into two separate steps:
1. Rendering of the message payload without the From, Date and Autocrypt headers.
2. Adding the From, Date and Autocrypt headers and possibly encrypting the message.
The goal is to have serializable result of the first step
that can be persisted in the database and sent later with any email address.
This way it will be possible to send queued messages over any relay.
This will make it possible not to remove all messages from the queue
when the sending relay is changed.
Currently changing `configured_addr` deletes everything from `smtp` table.
This change is however only a refactoring and does not implement any features.
Otherwise users may add up to 5 relays, but 2 oldest relays
are actually ignored.
Having the maximum number of published relays
and the number of relays used the same between all clients
makes the order of relays in the key irrelevant.
We may even remove sorting by `add_timestamp`
from `get_all_self_addrs()` in the future.
There was a bug in prefetch_should_download() that it made it return
true for pre-messages even when they were already downloaded. This meant
that pre-messages were downloaded from all relays, rather than just one,
wasting internet data.
The fix is in rfc724_mid_download_tried(), which is used by
prefetch_should_download() to determine whether a message was already
downloaded.
---------
Co-authored-by: l <link2xt@testrun.org>
EventThread is created in Account.__init__,
but the thread start may be scheduled later.
If the thread only cals dc_get_event_emitter()
after some events have been emitted,
EventThread will never capture such events
and the tests will timeout waiting for the event.
This happened in CI on a Linux runner
with test_markseen_invalid_message_ids
timing out waiting for DC_EVENT_MSGS_CHANGED
right after sending a message.
If multi-relay onboarding config is set from UIs, automatically add
relays until there are 3 relays. For now, there will be a hardcoded list
of relay candidates.
- We need a list of chatmail relays that we somehow trust, and that
agree to be in the list. Then, we add all of them to the candidate list
(a new SQL table with colums "host" and "last_tried").
- Before going to IMAP IDLE: When there are less than 3 relays, fill it
up with relays from the candidate list. If creating an account fails,
try again with another relay from the list. For each candidate, we need
to remember the last time we tried to add a transport there, and try at
most once a week or so per candidate.
- For now, this will be behind an off-by-default config option, which at
least DC Android will enable when creating a new profile. UIs can then
opt in on their own pace but will likely need to disable it for tests.
- Right now, the backoff times are: Try to add a relay at most once per
hour, and try to add the same relay at most once per week
register device token if XDELTAPUSH IMAP capability is available (current relay setup provides it)
or if "maxsmtprecipients" IMAP metadata key is available (relays since May 2026 provide it),
allowing chatmail relay setups to drop XDELTAPUSH capability marker with the next release
while retaining push notification support.
The lint is reasonable, but without some work
to bring the future sizes way below the limit,
this lint is frequently triggered in unrelated PRs
that just add some local variables to async functions
or extend some structure that is passed on the stack.
Largest change is in the FFI crate.
With 2024 (but not 2021) edition unsafe code
inside unsafe functions should be marked separately
so we can mark exactly the code that is unsafe.
Some CFFI functions even have no unsafe code inside.
Most interesting change is that .strdup()
functions are not marked as unsafe anymore.
They are allocating memory and return raw pointers,
but there is nothing unsafe about it.
Only using the returned raw pointers is unsafe.
This way calls to .strdup() don't have to be marked
with unsafe{} blocks.
- treat an Autocrypt header as a kind of Autocrypt-Gossip: addr=<self>
letting it participate in the existing cooperative key gossipping mechanics.
- speed up processing of incoming autocrypt-gossip headers by using
1 sql commit for N gossip headers instead of the current N commits.
unsuspecting consumers will otherwise require execnet to run pytest with deltachat-rpc-client installed.
"import py" is discouraged for pytest these days but still supported in compatibility mode.
