When there are no parent references,
Delta Chat inserts Message-ID into References.
Such references should be ignored
because otherwise fully downloaded message
may be assigned to the same chat as previously incorrectly assigned
partially downloaded message.
Fixes receive_imf::tests::test_create_group_with_big_msg
Chat-Group-ID always correctly identifies the chat
message was sent to, while In-Reply-To and References
may point to a message that has itself been incorrectly
assigned to a chat.
as discussed in several chats, this PR starts making it possible to use
Webxdc as integrations to the main app. In other word: selected parts of
the main app can be integrated as Webxdc, eg. Maps [^1]
this PR contains two parts:
- draft an Webxdc Integration API
- use the Webxdc Integration API to create a Maps Integration
to be clear: a Webxdc is not part of this PR. the PR is about marking a
Webxdc being used as a Map - and core then feeds the Webxdc with
location data. from the view of the Webxdc, the normal
`sendUpdate()`/`setUpdateListener()` is used.
things are still marked as "experimental", idea is to get that in to
allow @adbenitez and @nicodh to move forward on the integrations into
android and desktop, as well as improving the maps.xdc itself.
good news is that we currently can change the protocol between Webxdc
and core at any point :)
# Webxdc Integration API
see `dc_init_webxdc_integration()` in `deltachat.h` for overview and
documentation.
rust code is mostly in `webxdc/integration.rs` that is called by other
places as needed. current [user of the API is
deltachat-ios](https://github.com/deltachat/deltachat-ios/pull/1912),
android/desktop will probably follow.
the jsonrpc part is missing and can come in another PR when things are
settled and desktop is really starting [^2] (so we won't need to do all
iterations twice :) makes also sense, when this is done by someone
actually trying that out on desktop
while the API is prepared to allow other types of integrations (photo
editor, compose tools ...) internally, we currently ignore the type. if
that gets more crazy, we probably also need a dedicated table for the
integrations and not just a single param.
# Maps Integration
rust code is mostly in `webxdc/maps_integration.rs` that is called by
`webxdc/integration.rs` as needed.
EDIT: the idea of having a split here, is that
`webxdc/maps_integration.rs` really can focus on the json part, on the
communication with the .xdc, including tests
this PR is basic implementation, enabling to move forward on
integrations on iOS, but also on desktop and android.
the current implementation allows already the following:
- global and per-chat maps
- add and display POIs
- show positions and tracks of the last 24 hours
the current maps.xdc uses leaflet, and is in some regards better than
the current android/desktop implementations (much faster, show age of
positions, fade out positions, always show names of POIs, clearer UI).
however, we are also not bound to leaflet, it can be anything
> [**screenshots of the current
state**](https://github.com/deltachat/deltachat-ios/pull/1912)
> 👆
to move forward faster and to keep this PR small, the following will go
to a subsequent PR:
- consider allowing webxdc to use a different timewindow for the
location
- delete POIs
- jsonrpc
[^1]: maps are a good example as anyways barely native (see android
app), did cause a lot of pain on many levels in the past (technically,
bureaucratically), and have a comparable simple api
[^2]: only going for jsonrpc would only make sense if large parts of
android/ios would use jsonrpc, we're not there
---------
Co-authored-by: link2xt <link2xt@testrun.org>
API now pretends that trashed messages don't exist.
This way callers don't have to check if loaded message
belongs to trash chat.
If message may be trashed by the time it is attempted to be loaded,
callers should use Message::load_from_db_optional.
Most changes are around receive_status_update() function
because previously it relied on loading trashed status update
messages immediately after adding them to the database.
`a.clone_from(&b)` is equivalent to `a = b.clone()` in functionality,
but can be overridden to reuse the resources of a to avoid unnecessary
allocations.
Let's add a 1-minute tolerance to `Params::MemberListTimestamp`.
This adds to the group membership consistency algo the following properties:
- If remote group membership changes were made by two members in parallel, both of them are applied,
no matter in which order the messages are received.
- If we remove a member locally, only explicit remote member additions/removals made in parallel are
allowed, but not the synchronisation of the member list from "To". Before, if somebody managed to
reply earlier than receiving our removal of a member, we added it back which doesn't look good.
Instead, look up the 1:1 chat in `receive_imf::add_parts()`. This is a more generic approach to fix
assigning outgoing reactions to 1:1 chats in the multi-device setup. Although currently both
approaches give the same result, this way we can even implement a "react privately"
functionality. Maybe it sounds useless, but it seems better to have less reaction-specific code.
shows the last reaction in chatlist's summaries if there is no
newer message.
the reason to show reactions in the summary, is to make them a _little_
more visible when one is not in the chat. esp. in not-so-chatty or in
one-to-ones chats this becomes handy: imaging a question and someone
"answers" with "thumbs up" ...
otoh, reactions are still tuned down on purpose: no notifications, chats
are opend as usual, the chatlist is not sorted by reactions and also the
date in the summary refer to the last message - i thought quite a bit
about that, this seems to be good compromise and will raise the fewest
questions. it is somehow clear to the users that reactions are not the
same as a real message. also, it is comparable easy to implement - no
UI changes required :)
all that is very close to what whatsapp is doing (figured that out by
quite some testing ... to cite @adbenitez: if in doubt, we can blame
whatsapp :)
technically, i first wanted to go for the "big solution" and add two
more columns, chat_id and timestamp, however, it seemed a bit bloated if
we really only need the last one. therefore, i just added the last
reaction information to the chat's param, which seems more performant
but also easier to code :)
`is_probably_private_reply()` checks
if a message should better go to the one to one chat
instead of the chat already identified by the `In-Reply-To`.
this functionality is needed to make "Reply Privately" work.
however, this functionality is never true for reactions.
if we would return `true` here, own reactions seen by a second device
would not get the correct chat assiged.
Do not include oldest reference, because chat members
which have been added later and have not seen the first message
do not have referenced message in the database.
Instead, include up to 3 recent Message-IDs.
Restart the IO scheduler if needed to make the new config value effective (for `MvboxMove,
OnlyFetchMvbox, SentboxWatch` currently). Also add `set_config_internal()` which doesn't affect
running the IO scheduler. The reason is that `Scheduler::start()` itself calls `set_config()`,
although not for the mentioned keys, but still, and also Rust complains about recursive async calls.
Currently when a user sets up another device by logging in, a new key is created. If a message is
sent from either device outside, it cannot be decrypted by the other device.
The message is replaced with square bracket error like this:
```
<string name="systemmsg_cannot_decrypt">This message cannot be decrypted.\n\n• It might already help to simply reply to this message and ask the sender to send the message again.\n\n• If you just re-installed Delta Chat then it is best if you re-setup Delta Chat now and choose "Add as second device" or import a backup.</string>
```
(taken from Android repo `res/values/strings.xml`)
If the message is outgoing, it does not help to "simply reply to this message". Instead, we should
add a translatable device message of a special type so UI can link to the FAQ entry about second
device. But let's limit such notifications to 1 per day. And as for the undecryptable message
itself, let it go to Trash if it can't be assigned to a chat by its references.
If a Delta Chat message has the Message-ID already existing in the db, but a greater "Date", it's a
resent message that can be deleted. Messages having the same "Date" mustn't be deleted because they
can be already seen messages moved back to INBOX. Also don't delete messages having lesser "Date" to
avoid deleting both messages in a multi-device setting.
"Auto-Submitted: auto-replied" messages mustn't be considered as sent by either bots or non-bots,
e.g. MDNs have this header value and it's the same for bots and non-bots.
a27e84ad89 "fix: Delete received outgoing messages from SMTP queue"
can break sending messages sent as several SMTP messages because they have a lot of recipients:
`pub(crate) const DEFAULT_MAX_SMTP_RCPT_TO: usize = 50;`
We should not cancel sending if it is such a message and we received BCC-self because it does not
mean the other part was sent successfully. For this, split such messages into separate jobs in the
`smtp` table so that only a job containing BCC-self is canceled from `receive_imf_inner()`. Although
this doesn't solve the initial problem with timed-out SMTP requests for such messages completely,
this enables fine-grained SMTP retries so we don't need to resend all SMTP messages if only some of
them failed to be sent.
The bug was made in 44227d7b86. Sql::execute() with placeholders must
be used to escape strings, one never should escape them manually as strings themselves can contain
escape symbols. Thanks to @link2xt for noticing.
Some SMTP servers are running slow before-queue filters, most commonly Postfix with `rspamd` filter
which is implemented as a [before-queue Milter](https://www.postfix.org/MILTER_README.html). Some of
`rspamd` plugin filters are slow on large mails.
We previously had problems with timing out during waiting for SMTP response:
https://github.com/deltachat/deltachat-core-rust/issues/1383. This is largely fixed by
https://github.com/async-email/async-smtp/pull/29 and currently we have 60-second timeout just for
reading a response but apparently it is not sufficient -- maybe connection gets killed by NAT while
we are waiting for response or `rspamd` takes more than 60 seconds for large messages.
As a result a message is resent multiple times and eventually fails with "too many retries" while
multiple BCC-self messages are received.
We should remove the message from the SMTP queue as soon as we receive it via IMAP as it is clear
the message was sent even if we did not manage to get actual SMTP server response.
First of all, it's just downloaded and hasn't been seen yet by the user. Also this changes nothing
as `msgs.state` isn't changed when replacing a message anyway.
Put a copy of Message-ID into hidden headers and prefer it over the one in the IMF header section
that servers mess up with.
This also reverts "Set X-Microsoft-Original-Message-ID on outgoing emails for amazonaws (#3077)".
As per the comment in `receive_imf.rs`, `chat.protected` must be maintained regardless of the
`Config::VerifiedOneOnOneChats`. The only thing that mustn't be done if `VerifiedOneOnOneChats` is
unset (i.e. for non-supporting UIs) is marking chats as "protection broken" because this needs
showing the corresponding dialog to a user.
Drafts mustn't affect sorting of any other messages, they aren't even displayed in the chat
window. Also hidden messages mustn't affect sorting of usual messages. But let hidden messages sort
together with protection messages because hidden messages also can be or not be verified, so let's
preserve this information -- even it's not useful currently, it can be useful in the future
versions.
Before in some places it was correctly calculated by passing the "sent" timestamp to
`calc_sort_timestamp()`, but in other places just the system time was used. In some complex
scenarios like #5088 (restoration of a backup made before a contact verification) it led to wrong
sort timestamps of protection messages and also messages following by them.
But to reduce number of args passed to functions needing to calculate the sort timestamp, add
message timestamps to `struct MimeMessage` which is anyway passed everywhere.
Ad-hoc groups don't have grpid-s that can be used to identify them across devices and thus wasn't
synced until now.
The same problem already exists for assigning messages to ad-hoc groups and this assignment is done
by `get_parent_message()` and `lookup_chat_by_reply()`. Let's reuse this logic for the
synchronisation, it works well enough and this way we have less surprises than if we try to
implement grpids for ad-hoc groups. I.e. add an `Msgids` variant to `chat::SyncId` analogous to the
"References" header in messages and put two following Message-IDs to a sync message:
- The latest message A having `DownloadState::Done` and the state to be one of `InFresh, InNoticed,
InSeen, OutDelivered, OutMdnRcvd`.
- The message that A references in `In-Reply-To`.
This way the logic is almost the same to what we have in `Chat::prepare_msg_raw()` (the difference
is that we don't use the oldest Message-ID) and it's easier to reuse the existing code.
NOTE: If a chat has only an OutPending message f.e., the synchronisation wouldn't work, but trying
to work in such a corner case has no significant value and isn't worth complicating the code.
- Remove "Detected Autocrypt-mime message" logs printed for every incoming Autocrypt message.
- Print only a single line at the beginning of receive_imf with both the Message-ID and seen flag.
- Print Securejoin step only once, inside handle_securejoin_handshake or observe_securejoin_on_other_device.
- Do not log "Not creating ad-hoc group" every time ad-hoc group is not created, log when it is created instead.
- Log ID of the chat where Autocrypt-Gossip for all members is received.
- Do not print "Secure-join requested." for {vg,vc}-request, we already log the step.
- Remove ">>>>>>>>>>>>>>>>>>>>>>>>>" noise from securejoin logs.
Otherwise it looks like the message creating a protected group is not verified. For this, use
`sent_timestamp` of the received message as an upper limit of the sort timestamp (`msgs.timestamp`)
of the protection message. As the protection message is added to the chat earlier, this way its
timestamp is always less or eq than the received message's timestamp.
Allowing outgoing unencrypted messages in groups with 2 members
breaks the test
`python/tests/test_0_complex_or_slow.py::test_verified_group_vs_delete_server_after`
This centralizes all Securejoin/verification checks and updates in one
place right before add_parts() even before we assign the message to
the chat, so we can decouple chat logic from verification logic.
