This removes the last usages of the deprecated `is_chatmail` and
no `XCHATMAIL` IMAP capability is read anymore.
UIs are not using it for a longer time anymore.
The self-reporting statistics now set `is_chatmail` field to
`true`: all transports have relay-typical metadata
`false`: at least one doesn't
`null`: we don't know
BREAKING CHANGE: `is_chatmail` is no longer a known config key.
"addr" is already deprecated and its default value was using "configured_addr".
We want to get rid of "configured_addr" too, but users should at least
not access it through deprecated config.
Headers like From and Autocrypt are now added late,
right before sending the message over SMTP.
This way we advertise the latest list of transports
and use the correct From address in the encrypted part
even for messages queued while being offline.
BCC-self recipients are also added late.
For unencrypted messages we only want to send a copy
to the sending address, but we don't know the sending address
when queueing the message.
Adding bcc-self recipients when dequeuing the message
also makes it possible to send copies to updated list of relays.
Removing a relay now takes effect immediately:
- the profile stops fetching and advertising it,
- secondary devices immediately apply the removal through the transport sync,
Upgrading removes unpublished relays and triggers keyupdates.
BREAKING CHANGE: set_transport_unpublished() is removed: UIs call delete_transport() when the user removes a relay.
BREAKING CHANGE: list_transports_ex() and the TransportListEntry type are removed: use list_transports().
BREAKING CHANGE: delete_transport() no longer refuses to remove the primary transport: it refuses only to remove the last one and re-elects the sending transport as needed.
BREAKING CHANGE: TransportsModified is now also emitted on the device modifying the transports, not only on devices applying the synced change.
Deprecated: DC_STR_PHASING_OUT
When the published relay list changes, key-contacts are informed with an
unsigned message carrying the re-signed key, encrypted to a chunk of contacts
at a time. It is shaped like a receipt notification naming no message, so that
cores which know nothing about keyupdates trash it as well.
See the src/keyupdate.rs module docs for the design.
Authentication failures may happen because of internal server errors.
Device message saying "Please check if the email address and the password are correct"
was written for classic email setups when the user knows the password.
For users of chatmail relays this message is not actionable,
but still appears when relay fails to check the password.
"ex" means "used to be, but no longer is".
"ext" means "extended", which is the intended meaning.
"Ext" is more common not only in the Rust ecosystem but in general.
We already had several people (myself included) asking
what "ex" is supposed to mean.
To reproduce this commit, search and replace `_ex(?!\w)` -> `_ext`.
But don't change `list_transports_ex` because it's public API.
Devices no longer implicitely use the From address of sync messages
to determine their primary transport. Receivers have no concept of it and own devices
may disagree on which relay is reachable because of VPN or different networks:
1. Make setting a primary transport (`configured_addr`) a per-device non-synced operation.
2. Transport rows (add/remove/unpublish) keep syncing like before.
3. A device reelects a primary if a sync message unpublished/removed the current primary
if there is a better candidate.
4. `TransportsModified` event is emitted at most once on an incoming transport sync message.
Users will notice the change in that changing primary transport in settings/advanced/relays
will not synchronize to other devices anymore.
this PR adds support for reactions in broadcast channels.
> the idea of broadcast reactions is that they are sent as usual from
subscribers to owner. after some time, the owner broadcasts them to all
subscribers, who only get to see reaction+count, not who-reacted-what
the PR is quite large, but a good share are tests and otherwise many
things are straight forward.
review should be done by-file, not by-commit. to make review easier,
here is a high-level overview:
first a change in the existing internal `Reactions` object was required.
before this PR, `Reactions` had a "contact to reaction map" only, and
the "frequencies map", that are actually mainly needed for UI, were
calculated as needed. with this PR, the "frequencies map" is the field
that always exist, the "contact map" is only available on top of that.
moreover, this PR shifts that part to the core, it was unfortunately in
the bindings before.
with that preparation things done as follows:
1. reactions from broadcast channel subscriber (`Chattype::InBroadcast`)
to broadcast channel owner (`Chattype::OutBroadcast`) are sent as usual,
only change for that step was to allow sending them at all
2. the owner receives reactions and saves them to the existing
`reactions` table as usual. additionally, the changed message is
remembered in `reactions_need_broadcast` table
3. in the IMAP loop, when ~10 minutes have passed, and
`reactions_need_broadcast` contains entries, a single, hidden message
with accumulated reactions is sent. this message may contain reactions
to different messages. for each message, all known reactions are sent as
reaction+count.
4. subscriber receive that message and save the accumulated reactions in
`reactions_broadcasted`
6. `get_message_reactions` is adapted so that `frequencies` are set
independently of who-reacted-what (the old and only field).
who-reacted-what is called `by_contact` now. it is always set for
compatibility reasons, however, it is not exhaustive for subscribers.
in general, UI should work with frequencies, the API itself, however,
has not changed.
other tweaks:
- outgoing channels are muted on creation, and UI shall allow to
unmute/mute them as all other chats. reason is that reactions are
notified, but in many cases not of large interest. this is also what
telegram is doing
- to have an intermediate feedback when reacting, the local state should
include ones own reaction, even if it is not yet broadcasted. for that,
we modify `reactions_broadcasted` using `modify_frequencies()` as needed
when sending an reaction. there are still some situations where the
update may not include ones own reaction, in this case it is added
lately by `refine_frequencies()`, so that `get_message_reactions()`
always contain SELF.
(in a first implementation, we always increased SELF reaction in
refine_frequencies(), however, that was worse and led to SELF counted
twice once the owner sent broadcast)
<details>
<summary>wire format</summary>
wire format is a JSON in the `Broadcast-Reactions:` header.
additionally, `Content-Disposition: reaction` is set to not show the
hidden message on existing devices.
using a header also allows us to broadcast reactions with resent channel
messages (on joining) later.
```
{
"messages": [
{
"id": "12345678",
"reactions": [
{ "emoji": "👍", "count": 4 },
{ "emoji": "🎉", "count": 2 }
]
},
{
"id": "23456789",
"reactions": []
}
]
}
```
for `id`, the wire format needs to use `rfc724_mid` as `msg_id` are
local only.
</details>
### known issues
- if the channel owner uses multiple devices, broadcasted reaction
updates are sent from each device. the updates are not that big, so that
is probably not a big deal. if it turns out that this is an issue, we
can think about fixes in another PR. might be done by restarting our
10-minute-wait once we see an update from another device
- we cannot set contact_id for DC_EVENT_REACTIONS_CHANGED - but i doubt
it was ever used
### for another pr
- ~~add `Broadcast-Reactions:` header also for resent channel messages,
so that new subscriber do not only get the latest messages, but also
their reactions. for that, the `Broadcast-Reactions:` header can go to
the corresponding message, no need to send extra messages. we would need
to change the sending part to send all reactions for a given message. on
receiving part, we need to make sure, `receive_broadcast_reactions()` is
called when the message actually exist.~~
EDIT: subsequent PR for resending broadcast reactions at
https://github.com/chatmail/core/pull/8496
- add api to allow only a subset of reactions, fiter incoming reactions
before broadcasting
### misc.
ui pr: https://github.com/deltachat/deltachat-ios/pull/3225 and
https://github.com/deltachat/deltachat-android/pull/4560 , which both
were tested successfully with this core PR already. desktop is meant to
be done once this is merged
---------
Co-authored-by: l <link2xt@testrun.org>
If multi-relay onboarding config is set from UIs, automatically add
relays until there are 3 relays. For now, there will be a hardcoded list
of relay candidates.
- We need a list of chatmail relays that we somehow trust, and that
agree to be in the list. Then, we add all of them to the candidate list
(a new SQL table with colums "host" and "last_tried").
- Before going to IMAP IDLE: When there are less than 3 relays, fill it
up with relays from the candidate list. If creating an account fails,
try again with another relay from the list. For each candidate, we need
to remember the last time we tried to add a transport there, and try at
most once a week or so per candidate.
- For now, this will be behind an off-by-default config option, which at
least DC Android will enable when creating a new profile. UIs can then
opt in on their own pace but will likely need to disable it for tests.
- Right now, the backoff times are: Try to add a relay at most once per
hour, and try to add the same relay at most once per week
BREAKING CHANGE: provider lookup APIs were removed from CFFI and JSON-RPC.
also removes offline provider database code and generated provider data,
provider-specific fields in configure/transport paths, and REPL providerinfo.
Remove is_chatmail flag, and rely on IMAP metadata
advertising the recipients limit (usually 1000),
falling back to 50 (or fewer in some exceptional cases)
just as before when is_chatmail was false.
Also, server metadata is now keyed per transport,
which in the future eases collecting ICE servers
from all relays (instead of just any first relay).
BREAKING CHANGE: removed oauth2 module, dc_get_oauth2_url FFI function, DC_LP_AUTH flags and configured/serverflags, and the oauth2 parameter/field from SMTP/IMAP clients, JSON-RPC interfaces, and CLI tools.
also contains regenerated provider data after dropping oauth in the update script.
BREAKING CHANGE: UIs should not behave differently for chatmail relays than for classical email servers; most usages of `is_chatmail` can be replaced by `force_encryption`
Fix https://github.com/chatmail/core/issues/8195
Supersedes https://github.com/chatmail/core/pull/8217
The most interesting change is in `delete_expired_imap_messages()`
because there, messages are marked for deletion on single-device
chatmail profiles.
The logic in `delete_expired_imap_messages()` was wrong before, and
pre-messages were not deleted at all. This is fixed by also querying
`pre_rfc724_mid` in addition to `rfc724_mid` from the `msgs` table.
In order not to make the SQL statement too complex, I split it into two.
WRT tests:
- `test_immediate_autodelete()` tests the auto-deletion;
`test_imap_autodelete_fully_downloaded_msg()` tests that even for a
message that is split into pre- and post-message, both messages are
deleted.
- A lot of tests test that if bcc_self is on, messages are not
auto-deleted. E.g. `test_one_account_send_bcc_setting()` and
`test_markseen_message_and_mdn()` relies on the fact that messages stay
on the server when bcc_self is on.
- Unfortunately, it is not possible to test that messages are only
deleted for chatmail servers, because we don't have any tests that use a
non-chatmail server.
Since commit 25750de4e1
released in 2.36.0 we do not move messages to mvbox without explicit
mvbox_move setting, so do not need to watch it as well
as long as other devices are updated to the same change.
Closes https://github.com/chatmail/core/issues/7980.
Unpublished transports are not advertised to contacts, and self-sent messages are not sent there, so that we don't cause extra messages to the corresponding inbox, but can still receive messages from contacts who don't know the new relay addresses yet.
- This adds `list_transports_ex()` and `set_transport_unpublished()` JsonRPC functions
- By default, transports are published, but when updating, all existing transports except for the primary one become unpublished in order not to break existing users that followed https://delta.chat/legacy-move
- It is not possible to unpublish the primary transport, and setting a transport as primary automatically sets it to published
An alternative would be to change the existing list_transports API rather than adding a new one list_transports_ex. But to be honest, I don't mind the _ex prefix that much, and I am wary about compatibility issues. But maybe it would be fine; see b08ba4bb8 for how this would look.
This fixes the bug when a new transport doesn't become primary on the 2nd device because INBOX from
the new transport isn't fully fetched. Now the `Transports` sync message is received from the old
transport, but as it has updated "From", it updates the primary transport correspondingly. NB: I/O
for the new primary transport isn't immediately started however, this needs a separate fix.
At urgent request from @hpk42, this adds a config option `team_profile`.
This option is only settable via SQLite (not exposed in the UI), and the
only thing it does is disabling synchronization of seen status.
I tested manually on my Android phone that it works.
Not straigthforward to write an automatic test, because we want to test that something
does _not_ happen (i.e. that the seen status is _not_ synchronized), and
it's not clear how long to wait before we check.
Probably it's fine to just not add a test.
This is what I tried:
```python
@pytest.mark.parametrize("team_profile", [True, False])
def test_markseen_basic(team_profile, acfactory):
"""
Test that seen status is synchronized iff `team_profile` isn't set.
"""
alice, bob = acfactory.get_online_accounts(2)
# Bob sets up a second device.
bob2 = bob.clone()
bob2.start_io()
alice_chat_bob = alice.create_chat(bob)
bob.create_chat(alice)
bob2.create_chat(alice)
alice_chat_bob.send_text("Hello Bob!")
message = bob.wait_for_incoming_msg()
message2 = bob2.wait_for_incoming_msg()
assert message2.get_snapshot().state == MessageState.IN_FRESH
message.mark_seen()
# PROBLEM: We're not waiting 'long enough',
# so, the 'state == MessageState.IN_SEEN' assertion below fails
bob.create_chat(bob).send_text("Self-sent message")
self_sent = bob2.wait_for_msg(EventType.MSGS_CHANGED)
assert self_sent.get_snapshot().text == "Self-sent message"
if team_profile:
assert message2.get_snapshot().state == MessageState.IN_FRESH
else:
assert message2.get_snapshot().state == MessageState.IN_SEEN
```
Adds an api to get all ui config keys. There already is an option to get
all normal config keys (`"sys.config_keys"`), but before this pr there
was no way to get all `ui.*` config keys.
#### Why is this api needed?
For webxdc cleanup on desktop, which stores window position in a ui
config key (such as `ui.desktop.webxdcBounds.676464`) as soon as a
webxdc is opened since many versions now. So listing all ui keys is a
good way for us to find out which webxdc may have web data stored.
unfortunately electron does not (yet?) have a way to list all origins
that have web-data like android does, so this is the next best thing we
can do before itterating all possible ids, see also
https://github.com/deltachat/deltachat-desktop/issues/5758.
#### Why is this only a jsonrpc api and not another special/virtual
config key like `"sys.config_keys"`?
r10s indicated that `ui.*`-config keys are barely used
(https://github.com/deltachat/deltachat-desktop/issues/5790#issuecomment-3598512802),
so I thought it makes more sense to add it as dedicated api which's
existentence is checked by the typechecker, so it will be easier to not
miss it when we should remove the api again in the future.
But we could also do a dedicated special/virtual config key for it, if
you think that is better, this is easy to change.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
