Closes https://github.com/chatmail/core/issues/7980.
Unpublished transports are not advertised to contacts, and self-sent messages are not sent there, so that we don't cause extra messages to the corresponding inbox, but can still receive messages from contacts who don't know the new relay addresses yet.
- This adds `list_transports_ex()` and `set_transport_unpublished()` JsonRPC functions
- By default, transports are published, but when updating, all existing transports except for the primary one become unpublished in order not to break existing users that followed https://delta.chat/legacy-move
- It is not possible to unpublish the primary transport, and setting a transport as primary automatically sets it to published
An alternative would be to change the existing list_transports API rather than adding a new one list_transports_ex. But to be honest, I don't mind the _ex prefix that much, and I am wary about compatibility issues. But maybe it would be fine; see b08ba4bb8 for how this would look.
This fixes the bug when a new transport doesn't become primary on the 2nd device because INBOX from
the new transport isn't fully fetched. Now the `Transports` sync message is received from the old
transport, but as it has updated "From", it updates the primary transport correspondingly. NB: I/O
for the new primary transport isn't immediately started however, this needs a separate fix.
At urgent request from @hpk42, this adds a config option `team_profile`.
This option is only settable via SQLite (not exposed in the UI), and the
only thing it does is disabling synchronization of seen status.
I tested manually on my Android phone that it works.
Not straigthforward to write an automatic test, because we want to test that something
does _not_ happen (i.e. that the seen status is _not_ synchronized), and
it's not clear how long to wait before we check.
Probably it's fine to just not add a test.
This is what I tried:
```python
@pytest.mark.parametrize("team_profile", [True, False])
def test_markseen_basic(team_profile, acfactory):
"""
Test that seen status is synchronized iff `team_profile` isn't set.
"""
alice, bob = acfactory.get_online_accounts(2)
# Bob sets up a second device.
bob2 = bob.clone()
bob2.start_io()
alice_chat_bob = alice.create_chat(bob)
bob.create_chat(alice)
bob2.create_chat(alice)
alice_chat_bob.send_text("Hello Bob!")
message = bob.wait_for_incoming_msg()
message2 = bob2.wait_for_incoming_msg()
assert message2.get_snapshot().state == MessageState.IN_FRESH
message.mark_seen()
# PROBLEM: We're not waiting 'long enough',
# so, the 'state == MessageState.IN_SEEN' assertion below fails
bob.create_chat(bob).send_text("Self-sent message")
self_sent = bob2.wait_for_msg(EventType.MSGS_CHANGED)
assert self_sent.get_snapshot().text == "Self-sent message"
if team_profile:
assert message2.get_snapshot().state == MessageState.IN_FRESH
else:
assert message2.get_snapshot().state == MessageState.IN_SEEN
```
Adds an api to get all ui config keys. There already is an option to get
all normal config keys (`"sys.config_keys"`), but before this pr there
was no way to get all `ui.*` config keys.
#### Why is this api needed?
For webxdc cleanup on desktop, which stores window position in a ui
config key (such as `ui.desktop.webxdcBounds.676464`) as soon as a
webxdc is opened since many versions now. So listing all ui keys is a
good way for us to find out which webxdc may have web data stored.
unfortunately electron does not (yet?) have a way to list all origins
that have web-data like android does, so this is the next best thing we
can do before itterating all possible ids, see also
https://github.com/deltachat/deltachat-desktop/issues/5758.
#### Why is this only a jsonrpc api and not another special/virtual
config key like `"sys.config_keys"`?
r10s indicated that `ui.*`-config keys are barely used
(https://github.com/deltachat/deltachat-desktop/issues/5790#issuecomment-3598512802),
so I thought it makes more sense to add it as dedicated api which's
existentence is checked by the typechecker, so it will be easier to not
miss it when we should remove the api again in the future.
But we could also do a dedicated special/virtual config key for it, if
you think that is better, this is easy to change.
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Fix get_secondary_addrs() which was using
`secondary_addrs` config that is not updated anymore.
Instead of using `secondary_addrs` config,
use the `transports` table which contains all the addresses.
And enable it by default as the standard Header Protection is backward-compatible.
Also this tests extra IMF header removal when a message has standard Header Protection since now we
can send such messages.
Omit Legacy Display Elements from "text/plain" and "text/html" (implement 4.5.3.{2,3} of
https://www.rfc-editor.org/rfc/rfc9788 "Header Protection for Cryptographically Protected Email").
`login_param` module is now for user-visible entered login parameters,
while the `transport` module contains structures for internal
representation of connection candidate list
created during transport configuration.
It's used in `fetch_existing_msgs()`, but we can remove it and tell users that they need to
move/copy messages from Sentbox to Inbox so that Delta Chat adds all contacts from them. This way
users will be also informed that Delta Chat needs users to CC/BCC/To themselves to see messages sent
from other MUAs.
The motivation is to reduce code complexity, get rid of the extra IMAP connection and cases when
messages are added to chats by Inbox and Sentbox loops in parallel which leads to various message
sorting bugs, particularly to outgoing messages breaking sorting of incoming ones which are fetched
later, but may have a smaller "Date".
We do not try to delete resent messages
anymore. Previously resent messages
were distinguised by having duplicate Message-ID,
but future Date, but now we need to download
the message before we even see the Date.
We now move the message to the destination folder
but do not fetch it.
It may not be a good idea to delete
the duplicate in multi-device setups anyway,
because the device which has a message
may delete the duplicate of a message
the other device missed.
To avoid triggering IMAP busy move loop
described in the comments
we now only move the messages
from INBOX and Spam folders.
This way, the statistics / self-reporting bot will be made into an
opt-in regular sending of statistics, where you enable the setting once
and then they will be sent automatically. The statistics will be sent to
a bot, so that the user can see exactly which data is being sent, and
how often. The chat will be archived and muted by default, so that it
doesn't disturb the user.
The collected statistics will focus on the public-key-verification that
is performed while scanning a QR code. Later on, we can add more
statistics to collect.
**Context:**
_This is just to give a rough idea; I would need to write a lot more
than a few paragraphs in order to fully explain all the context here_.
End-to-end encrypted messengers are generally susceptible to MitM
attacks. In order to mitigate against this, messengers offer some way of
verifying the chat partner's public key. However, numerous studies found
that most popular messengers implement this public-key-verification in a
way that is not understood by users, and therefore ineffective - [a 2021
"State of Knowledge" paper
concludes:](https://dl.acm.org/doi/pdf/10.1145/3558482.3581773)
> Based on our evaluation, we have determined that all current E2EE
apps, particularly when operating in opportunistic E2EE mode, are
incapable of repelling active man-in-the-middle (MitM) attacks. In
addition, we find that none of the current E2EE apps provide better and
more usable [public key verification] ceremonies, resulting in insecure
E2EE communications against active MitM attacks.
This is why Delta Chat tries to go a different route: When the user
scans a QR code (regardless of whether the QR code creates a 1:1 chat,
invites to a group, or subscribes to a broadcast channel), a
public-key-verification is performed in the background, without the user
even having to know about this.
The statistics collected here are supposed to tell us whether Delta Chat
succeeds to nudge the users into using QR codes in a way that is secure
against MitM attacks.
**Plan for statistics-sending:**
- [x] Get this PR reviewed and merged (but don't make it available in
the UI yet; if Android wants to make a release in the meantime, I will
create a PR that removes the option there)
- [x] Set the interval to 1 week again (right now, it's 1 minute for
testing)
- [ ] Write something for people who are interested in what exactly we
count, and link to it (see `TODO[blog post]` in the code)
- [ ] Prepare a short survey for participants
- [ ] Fine-tune the texts at
https://github.com/deltachat/deltachat-android/pull/3794, and get it
reviewed and merged
- [ ] After the next release, ask people to enable the
statistics-sending
Ignoring `receive_imf_inner()` errors, i.e. silently skipping messages on failures, leads to bugs
never fixed. As for temporary I/O errors, ignoring them leads to lost messages, in this case it's
better to bubble up the error and get the IMAP loop stuck. However if there's some logic error, it's
better to show it to the user so that it's more likely reported, and continue receiving messages. To
distinguish these cases, on error, try adding the message as partially downloaded with the error set
to `msgs.error`, this way the user also can retry downloading the message to finally see it if the
problem is fixed.
The setting is already removed from the UIs,
but users who had it disabled previously have
no way to enable it. After this change
encryption is effectively always preferred.
`get_connectivity()` is expected to return immediately, not when the scheduler finishes updating its
state in `start_io()/stop_io()/pause_io()`, otherwise it causes app non-responsiveness.
Instead of read-locking `SchedulerState::inner`, store the `ConnectivityStore` collection in
`Context` and fetch it from there in `get_connectivity()`. Update it every time we release a write
lock on `SchedulerState::inner`.
A donation request device message is added if >= 100 messages have been sent and delivered. The
condition is checked every 30 days since the first message is sent. The message is added only once.
Move all `configured_*` parameters into a new SQL table `transports`.
All `configured_*` parameters are deprecated; the only exception is
`configured_addr`, which is used to store the address of the primary
transport. Currently, there can only ever be one primary transport (i.e.
the `transports` table only ever has one row); this PR is not supposed
to change DC's behavior in any meaningful way.
This is a preparation for mt.
---------
Co-authored-by: l <link2xt@testrun.org>
fetch_existing option is not enabled in existing clients
and does not work with encrypted messages
without importing the key into a newely created account.
APNS tokens never expire unless
the user uninstalls the application.
Because of this in most cases
the token remains valid forever
and chatmail server never removes the token
even if it is unencrypted
or the user has removed Delta Chat profile
from the device but not the whole application.
We want to modify chatmail servers
to remember the last time the token was stored
and remove them after some time.
Before we do this, we need to modify
the client to store the device token
each time so the server knows which tokens are used
and can update their timestamps.
These were the last places in the `deltachat` crate where files were
stored without deduplication. The CFFI python bindings are the last
thing that's still missing.
