Add "Chat-Group-Name-Timestamp" message header and use the last-write-wins logic when updating group
names (similar to group member timestamps). Note that if the "Chat-Group-Name-Changed" header is
absent though, we don't add a system message (`MsgGrpNameChangedBy`) because we don't want to blame
anyone.
> _greetings from the ice of the deutsche bahn 🚂🚃🚃🚃 always a pleasure to
see how well delta chat meanwhile performs in bad networks :)_
this PR adds an API to request other chat members to replace the message
text of an already sent message. scope is mainly to fix typos. this
feature is known from whatsapp, telegram, signal, and is
[requested](https://support.delta.chat/t/retract-edit-sent-messages/1918)
[since](https://support.delta.chat/t/edit-messages-in-delta-chat/899)
[years](https://github.com/deltachat/deltachat-android/issues/198).
technically, a message with an
[`Obsoletes:`](https://datatracker.ietf.org/doc/html/rfc2076#section-3.6)
header is sent out.
```
From: alice@nine
To: bob@nine
Message-ID: 2000@nine
In-Reply-To: 1000@nine
Obsoletes: 1000@nine
Edited: this is the new text
```
the body is the new text, prefixed by the static text `Edited:` (which
is not a header). the latter is to make the message appear more nicely
in Non-Delta-MUA. save for the `In-Reply-To` header. the `Edited:`
prefix is removed by Delta Chat on receiving.
headers should be protected and moved to e2ee part as usual.
corrected message text is flagged, and UI should show this state, in
practise as "Edited" beside the date.
in case, the original message is not found, nothing happens and the
correction message is trashes (assuming the original was deleted).
question: is the `Obsoletes:` header a good choice? i _thought_ there is
some more specifica RFC, but i cannot find sth. in any case, it should
be an header that is not used otherwise by MUA, to make sure no wanted
messages disappear.
what is NOT done and out of scope:
- optimise if messages are not yet sent out. this is doable, but
introduces quite some cornercaes and may not be worth the effort
- replaces images or other attachments. this is also a bit cornercasy
and beyond "typo fixing", and better be handled by "delete for me and
others" (which may come soon, having the idea now, it seems easy :)
- get edit history in any way. not sure if this is worth the effort,
remember, as being a private messenger, we assume trust among chat
members. it is also questionable wrt privacy, seized devices etc.
- add text where nothing was before; again, scope is "typo fixing",
better avoid cornercases
- saved messages are not edited (this is anyway questionable)
- quoted texts, that are used for the case the original message is
deleted, are not updated
- edits are ignored when the original message is not there yet (out of
order, not yet downloaded)
- message status indicator does not show if edits are sent out or not -
similar to reactions, webxdc updates, sync messages. signal has the same
issue :) still, connectivity should show if there are messages pending
<img width="366" alt="Screenshot 2025-02-17 at 17 25 02"
src="https://github.com/user-attachments/assets/a4a53996-438b-47ef-9004-2c9062eea5d7"
/>
corresponding iOS branch (no PR yet):
https://github.com/deltachat/deltachat-ios/compare/main...r10s/edit-messages
---------
Co-authored-by: l <link2xt@testrun.org>
This implements new group consistency algorithm described in
<https://github.com/deltachat/deltachat-core-rust/issues/6401>
New `Chat-Group-Member-Timestamps` header is added
to send timestamps of member additions and removals.
Member is part of the chat if its addition timestamp
is greater or equal to the removal timestamp.
In multi-device case `vg-request-with-auth` left on IMAP may result in situation when Bob joins the
group, then leaves it, then second Alice device comes online and processes `vg-request-with-auth`
again and adds Bob back. So we should IMAP-delete `vg-request-with-auth`. Another device will know
the Bob's key from Autocrypt-Gossip. It's not a problem if Alice loses state (restores from an old
backup) or goes offline for long before sending `vg-member-added`, anyway it may not be delivered by
the server, rather Bob should retry sending SecureJoin messages as he is a part which wants to join,
so let's not solve this for now.
Recently there are many questions on the Delta Chat forum why some unexpected encrypted messages
appear in Inbox. Seems they are mainly sync messages, though that also obviously happens to
SecureJoin messages. Anyway, regardless of the `MvboxMove` setting, auto-generated outgoing messages
should be moved to the DeltaChat folder so as not to complicate co-using Delta Chat with other MUAs.
Before this fix actual contents of the message
reposted by Schleuder is considered a mailing list footer and removed,
not visible even in the "Show Full Message..." view.
With this change there will be two message bubbles,
one for header and one for the contents,
but it is still better than losing the contents completely.
Attempting to parse header part is out of scope for this change.
Fix#3507
Note that this is not intended for a release at this point! We first have to test whether it runs stable enough. If we want to make a release while we are not confident enough in authres-checking, then we have to disable it.
BTW, most of the 3000 new lines are in `test_data/messages/dkimchecks...`, not the actual code
da3a4b94 adds the results to the Message info. It currently does this by adding them to `hop_info`. Maybe we should rename `hop_info` to `extra_info` or something; this has the disadvantage that we can't rename the sql column name though.
Follow-ups for this could be:
- In `update_authservid_candidates()`: Implement the rest of the algorithm @hpk42 and me thought about. What's missing is remembering how sure we are that these are the right authserv-ids. Esp., when receiving a message sent from another account at the same domain, we can be quite sure that the authserv-ids in there are the ones of our email server. This will make authres-checking work with buzon.uy, disroot.org, yandex.ru, mailo.com, and riseup.net.
- Think about how we present this to the user - e.g. currently the only change is that we don't accept key changes, which will mean that the small lock on the message is not shown.
- And it will mean that we can fully enable AEAP, after revisiting the security implications of this, and assuming everyone (esp. @link2xt who pointed out the problems in the first place) feels comfortable with it.
systemli.org seems not to include Received: header in messages sent to
self, including BCC-self and messages sent to Saved message chat. This
results in all these messages not being displayed on other devices in
multi-device setting. To work around this problem we restrict draft
detection based on Received: header to classical mails.
Delta Chat already implements proper draft detection based on the
\Draft message flag and \Drafts folder flag. However, Thunderbird does
not set \Draft flag when message is stored as "Template". To make
draft detection for Thunderbird more robust, in case Received: header
check is removed altogether later, we check for X-Mozilla-Draft-Info
header which Thunderbird sets both for "Drafts" and "Templates".
outlooks SMTP-server change the Message-ID of messages
and put the original Message-ID to X-Microsoft-Original-Message-ID.
the changed Message-ID has some issues:
- outgoing messages with bcc_self enabled are shown twice
as the self-copy got the changed Message-ID while the database uses
the original one
- read receipts do not work as they refer to the changed message id
- in general, sender and recipient see different Message-IDs
the issues can be fixed by
(1) let all receivers use the original Message-ID
this is what this pr is doing
and this should fix all issues with delta-to-delta communication,
including groups, group-images etc.
there may be issues left in communication
with other MUAs as they are using another Message-ID.
(2) ftr: updating the Message-ID in the database of the sender to the new one
this requires bcc_self always enabled (which is not the case)
and may also result easily in race conditions
(Bob answers before Alice sees its self-sent message),
however, has the advantage of better compatibility with other MUA.
if needed, the compatibility with other MUA could be improved by remembering
both Messages-IDs, maybe we could treat the modified as References or so,
however, i think, this could be part of another PR if we know better about
real, in the wild issues.
Start implementing #1994
TODO (in later PRs):
- Add a hint to the watch settings that all folders are fetched from time to time (to be done in the individual UIs)
- folder names are case-insensitive, so double-check that all comparisons are case-insensitive
- The `scan_folders.rs` file didn't get as large as I expected and it's probably not worth it having an extra file for it. But if there are no objections, I'll make another PR to rename it to `folders.rs` and also put into it `configure_folders()` from `imap/mod.rs` and `needs_move()` with all its tests from `message.rs`.
Done:
- Most mailboxes have a "Drafts" folder where constantly new emails appear but we don't actually want to show them, what do we do about this? The most reliable way to detect such messages that we found up to now is:
If there is no `Received` header AND it's not in the `ConfiguredSentbox`, then ignore the email.
- before or after INBOX idle trigger a new "scan all folders for messages". It does a "list folders" and then goes through all folders with select-statements, checking if "next-uid" was changed since checked last time. This might be batchable but in any case should not consume a lot of traffic. We might debounce this scan activity to happen at most every N minutes
- if next-uid changed for a folder, we "prefetch" and "fetch" DC-messages as as needed ("dc-messages" are not just those with "Chat-Version" headers, but can also be regular emails)
- if we discover DC-messages in folders that have the "/Spam" flag (maybe excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
- for now, we do not change any user visible option, but introduce this "scan all" automatically and on top of what exists. The DeltaChat folder-watching does not perform scan-all-folders (maybe with the exception to trigger scan-all also with DeltaChat if INBOX is not watched)
- Tests (except if you have ideas to improve them)
- all folders, their last uidvalidity, next-seen etc. are kept in a separate "imap-sync" sqlite table. Maybe this can be used to streamline some of the "Sent" folder and "DeltaChat" folder detection code we already have.
- We now also move self-sent messages from the Inbox to the Sent folder if `mvbox_move` is off, as this was very easy to do now. This way, we now behave more like a normal MUA if the user wants this.
FOR LATER PRs:
- maybe for the first 50 messages or so, we could reduce the IDLE-timeout (currently 23 minutes or so) to faster detect messages sent to non-inbox folders. However, on Android and iOS, we would likely trigger scan-all when the app moves to foreground, and so it might not be neccessary to reduce the current idle-timeout at least for them. We can leave this "faster discovery" question for the end, after we move to real-life testing.
- (Later on, after the above works, we can consider heuristics on which folders to perform IDLE on, and remove the Watch-folder options (inbox, deltachat, sent). We tried to find a safe scheme for already doing it but failed to fine one, too many unknowns, also some questions regarding multi-device (you might have different settings with each of it, one moves, the other doesn't etc.) so we postponed this in favor of the above incremental suggestion.)
* Start implementing #1994
* Add debug logs, it seems like the SQL migration can go into another pr
* Let fetch_new_messages return whether there are new emails
* Code style
* Don't prefetch if there are no new emails
* clippy
* Even more debug logs
* If the folder was not newly selected, return always try to fetch as
uid_next is probably outdated
* Fix new bug
* Recognize spam folder
* if we discover DC-messages in folders that have the "/Spam" flag (excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
* Clippy, prioritize folder_meaning over folder_name_meaning
* Add a first test, for the first day after installation only debounce to 2s
* Start adding two tests (both of them fail)
* Don't abort folder scan if one folder fails
* More consts
* Replace bool return value by enum
* Split test up into multiple tests
* Print logs during rust tests
* Rust tests pass now
* .
* One of the Python tests passes now - reconfigure folders during scanning
* Make the last test pass - Delete emails in all folders when starting the test, not only inbox and mvbox
The problem had been that emails were left in the folder "xyz"
* lint
* DB migration (untested)
* Store uid_next in SQL instead of lastseen in a config
* Revert "If Inbox-watch is disabled and enabled again, do not fetch emails from in between"
all folders are always watched, anyway
* clippy, rm debug logs, comments
* Codestyle, comments
* fixing things again
* Fix another test: don't fetch from uid_next-1 but uid_next; make some {} to {:#} so that we can use `.context(...)`
* move self-sent, non-setupmessage chat messages to the Sent folder if `mvbox_move` is off
* comment
* Comments, make sure things work even if there is no uid_next
* Style
* Comments
* The rust test tested wrongly
* comments, small codestyle change
* Ignore emails that are probably only drafts
Most mailboxes have a "Drafts" folder where constantly new emails appear
but we don't actually want to show them.
So: If there is no Received header AND it's not in the ConfiguredSentbox,
then ignore the email.
Also: Add test.
* Fix occasional test failure, it was introduced as DC now moves messages from Inbox to Sent
* Add `Received` header to the rust tests
* After this PR we will always watch all folders and delete messages there if server_delete is enabled. So, for people who have server_delete on, disable it and add a hint to the devicechat
* comment, small fix
* link2xt's first review
* Use ON CONFLICT(FOLDER) DO to update and if it doesn't exist, then insert
Reason from link2xt: We had a problem with multiple peerstates inserted due to key fingerprint parsing error previously. With logic in Rust a similar problem can occur: an UPDATE can fail for reasons other than a conflict. PRIMARY KEY should ensure uniqueness in this case, but anyway.
* Remove two TODO statements, remove fetch_new_messages: ignoring uid {}, uid_next was {} log
* Next TODO: Make uidvalidity and uid_next DEFAULT 0
* rm two TODOs, Seems like we are not going to `exclude folders that are watched anyway` in this PR
* small tweak: Handle instants more carefully
* Add scan_all_folders_debounce_secs config for tests, set debounce to 60s (before it was just 2s during the first day)
* Don't use bold letters for the device message
* React to changes in the folders better
Before, if there was a configured Sent folder, but then it got
removed and replaced with another folder with a name meaning "Sent" but
without Sent flag, it would be ignored.
So, instead of checking against ConfiguredSentboxFolder,
create two Option variables at the beginning of the loop and replace
them with Some if it is None. At the end of the loop, store the new
values into ConfiguredSendboxFolder and ConfiguredSpamFolder, even if it
is None.
Also, derive some useful traits.
* move job: Return a meaningful error if server_folder is None instead of panicing
* small error-handling fix
* Fix test_fetch_existing() python test
Before, we sometimes got a race condition where scan_folders() sees that
there is a Sentbox and saves this info after we set the
ConfiguredSentbox to None and before the message is sent.
So, just expect that the message is moved to the sentbox.
* migration is 72 now
* rm 2 TODOs, Don't infinitely retry when dc_receive_imf() returns Err
* clippy: Remove glob imports
* Delete created folders at the beginning of tests
(some created folders made problems in the next tests because)
* Improve resetting accounts between tests
* draft API to deal with uncut message texts
* add column mime_modified
* add mime_modified flag to MimeParser and save it in the database
* save mime_headers also when mime_modified is set
* cargo fmt
* set mime_modified on parsed html-texts and when there are multiple alternative-parts; add test for that
* prototype functions, add to repl and ffi
* use correct mime_modified flag
* basically parse Mime-Structure to HTML
* add basic tests for HTML-parsing
* convert text/plain to html for getting original
* respect charset for plain texts
* make test more specific
* fix handling non-utf-8 charsets for plain messages
* add test for plain_to_html()
* add failing test for plaintext linkify
* linkify urls in plain text
* fix regex
* plain text linkify: add failing test for encapsulated links as <https://domain.com>
* plain text linkify: make encapsulated links as <https://domain.com> work
* plain text linkify: require word boundary at beginning of link, add tests for that
* plain text linkify: linkify emails
* plain text: support format=flowed
* plain text: support quotes
* make clippy happy
* set mime-modified also when simplify() cuts non-html messages, add tests for that
* streamline mime recursion
* repl tool: write original html to file for further processing
* convert cid:- to data:-protocol
* add a test for cid: to data: conversion
* make clippy happy
* fix html-tests to work with windows-lineends
* clarify what the returned html-code may contain
* add some more detailed doc comments
* add mime_modified column only if not exist
this additional check is needed
as the column may added with another dbversion in
some shipped beta-versions.
* incorporate documentation suggestions from review
* rename get_original_mime_html() to more simple get_html()
* rename api is_mime_modified() to more simple has_html(); internally, mime_modified-flag stays as-is, however
* rename MimeS to MimeMultipartType
* do not set mime-modified flag for encrypted messages that need extra-handling for saved mime-structure
* fix typo
* move get_msg_html() to MsgId.get_html()
* incorporate more documentation suggestions from review
* remove unused return value from collect_texts_recursive()
* avoid mime_modified being mutable in write-parts-loop
* move 'use futures::future::FutureExt' atop of html.rs
* move attributes defining plain-text to a dedicated structure
* more PlainText to separate file
* escape cid when building regex
* let dc_get_msg_html() return NULL when calling with bad param
* First try making get_recipients use MailHeader (nice and functional)
* Get it to compile by using not-so-functional style
* Add "empty-from" test, drop unnecessary check for error; continue using addrparse_header() instead of addrparse()
* Try to use functional style, unfortunately, I can't get the compiler to accept it
* Do it imperative-style: Do not overwrite To with Cc and vice versa
* Use addrparse_header() once more
* Still addrparse_header()
* Clippy
* Fix compile errors in tests
* Fix typo
* Fix tests again ;-)
* Code style
* Code style; try a HashMap<addr: String, display_name: String> as an address list but I am not convinced
* Code style; Use Vec<SingleInfo> as address list
* Clippy
* Add tests
* Add another test
* Remove stale comments
we change the name on the wire as the old Chat-Group-Image header
could not be used on random mails, it was the marker for a "Changed" message,
if we would keep this names, things will fail for exising installations
as messages are dropped and a "Group image changed" message is shown instead.
