Contact request chats are not merged into a single virtual "deaddrop"
chat anymore. Instead, they are shown in the chatlist the same way as
other chats, but sending of messages to them is not allowed and MDNs
are not sent automatically until the chat is "accepted" by the user.
New API:
- dc_chat_is_contact_request(): returns true if chat is a contact
request. In this case option to accept and block the chat via
dc_accept_chat() and dc_block_chat() should be shown in the UI.
- dc_accept_chat(): accept contact request and unblock the chat
- dc_block_chat(): decline contact request and block the chat
Removed API:
- dc_create_chat_by_msg_id(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_marknoticed_contact(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_decide_on_contact_request(): this call requires a message ID from
deaddrop chat as input. As deaddrop chat is removed, this call can't
be used anymore.
- dc_msg_get_real_chat_id(): use dc_msg_get_chat_id() instead, the
only difference between these calls was in handling of deaddrop chat
- removed DC_CHAT_ID_DEADDROP and DC_STR_DEADDROP constants
See https://support.delta.chat/t/discussion-how-to-show-error-states/1363/10 <!-- comment -->
It turns out that it's pretty easy to distinguish between lots of states (currently Error/NotConnected, Connecting…, Getting new messages… and Connected). What's not that easy is distinguishing between an actual error and no network, because if the server just doesn't respond, it could mean that we don't have network or that we are trying ipv6, but only ipv4 works.
**WRT debouncing:**
Sending of EVENT_CONNECTIVITY_CHANGED is not debounced, but emitted every time one of the 3 threads (Inbox, Mvbox and Sentbox) has a network error, starts fetching data, or is done fetching data.
This means that it is emitted:
- 9 times when dc_maybe_network() is called or we get network connection
- 12 times when we lose network connection
Some measurements: dc_get_connectivity() takes a little more than 1ms (in my measurements back in March), dc_get_connectivity_html() takes 10-20ms. This means that it's no immmediate problem to call them very often, might increase battery drain though. For the UI it may be a lot of work to update the title everytime; at least Android is smart enough to update the title only once.
Possible problems (we don't have to worry about them now I think):
- Due to the scan_folders feature, if the user has lots of folders, the state could be "Connecting..." for quite a long time, generally DC seemed a little unresponsive to me because it took so long for "Connecting..." to go away. Telegram has a state "Updating..." that sometimes comes after "Connecting...".
To be done in other PRs:
- Better handle the case that the password was changed on the server and authenticating fails, see https://github.com/deltachat/deltachat-core-rust/issues/1923 and https://github.com/deltachat/deltachat-core-rust/issues/1768
- maybe event debouncing (except for "Connected" connectivity events)
fix https://github.com/deltachat/deltachat-android/issues/1760
New `dc_msg_is_bot()` C API and corresponding `Message.is_bot()`
Python API can be used to check if incoming message is sent by a bot,
e.g. to avoid two echo bots replying indefinitely to each other.
"Bot" flag is not set for outgoing messages, but may be set for
BCC-self messages. For now documentation says that `dc_msg_is_bot()`
return value is unspecified for outgoing messages. It can be better
specified later if needed for specific applications, e.g. sharing an
account with a helper bot.
* Add failing test for #2369
* Completely ignore Drafts folder
fix#2369
* Also ignore messages that have the Draft flag set but are not in the Drafts folder
There was a bugreport for previous versions of DC that BCC-self
messages are not marked as seen if server-side Sieve rule moves them.
This test is an unsuccessful attempt to reproduce the bug,
which was confirmed to be fixed.
This test checks that MDN is marked as seen on ac1. Because ac1 also
receives BCC-self, it is possible that event of marking this message
as seen is confused with marking MDN as seen, and the test passes even
if MDN is not marked as seen. Explicitly disabling BCC-self for ac1
ensures it receives only one message (MDN).
Also start direct_imap as early as possible, before sending the
message. It is possible now, because BCC-self is not sent.
When watch_inbox was off, scan_folders failed and a toast "IMAP operation attempted while it is torn down" was shown.
--
The problem was:
When inbox_watch is off, scan_folders() is called at 244260a978/src/scheduler.rs (L107) but connect_configured() is not called before.
* Add test
* Don't only setup handle, but connect_configured() in scan_folders()
The problem was:
If a message has the \Deleted flag set, we ignore it. But we forgot to
update last_uid, so that uid_next was not updated at 47e639b777/src/imap.rs (L730) and the same message was
fetched over and over again.
Fix#2429
This PR originally contained a fix for sqlx which turned out not not to be necessary. But on the way, I made some general improvements:
- Under some circumstances, a "normal" test failure led to a timeout, without printing a decent error message. See e.g. https://app.circleci.com/pipelines/github/deltachat/deltachat-core-rust/8069/workflows/ba2a9949-b4ad-4ceb-a930-073bba05e2db/jobs/30965.
(The problem was: if there is an exception in dc_account_extra_configure(), when trying to handle the exception the line account.log("===================", e, "===================") was called, which can't work as log() only expects one parameter)
- When a test fails: Call `dump_account_info()` even if there is no direct_imap
- In test_import_export_online_all(), add another 100KB file to the backup. This adds resilience against future size changes of the sql db file: The test tests the smoothness of the progress bar. And if there are there are not enough about-equally-sized files, the progress bar can't be smooth.
save subject for messages:
- new api `dc_msg_get_subject()`,
- when quoting, use the subject of the quoted message as the new subject, instead of the
last subject in the chat
fix#2254: if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message), the db was corrupted.
* Add result.log() for logging with less boilerplate code
* Bugfix: Resultify housekeeping() to make it abort if the db is closed instead of just deleting everything
* Require the UI to call dc_stop_io() before backup export
* Prepare a bit better for closed-db: Resultify get_uidvalidity and get_uid_next and let job::load_next() wait until the db is open
About the bug (before this PR):
if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message):
- I don't know if it downloads the message, but of course at some point the process of receiving the message will fail
- In my test, DC is just in the process of moving a message when the imex starts, but then can't delete the job or update the msg server_uid
- Then, when job::load_next() is called, no job can be loaded. That's why it calls `load_housekeeping_job()`. As `load_housekeeping_job()` can't load the time of the last housekeeping, it assumes we never ran housekeeping and returns a new Housekeeping job, which is immediately executed.
- housekeeping can't find any blobs referenced in the db and therefore deletes almost all blobs.
Now I know why the tests failed before 48c58a7 (i.e. after c923670) but
not on master (i.e. before c923670):
because of a bug, scan_folders() set the configured_sentbox to None if
it was set before. If it was None before, it restored the correct value.
On master, there was another bug that led to two runs of
scan_folders() being started at the same time. Therefore, the first run
set configured_sentbox to None, the second one restored the correct
value.
c923670 fixed the latter bug, so that only one run of scan_folders() was
started. Therefore, configured_sentbox stayed None incorrectly and
test_fetch_existing() failed.
48c58a7 fixed the former bug.
This commit adds checks to test_fetch_existing(), so that it definitely
checks for the former bug.
The following rules apply now:
1. "name" column is only updated manually and never over the network
2. "authname" column is only updated over the network and never manually
3. Displayname is a "name" if it is non-empty, otherwise it is "authname".
This fixes a known (pytest.xfail) problem of "name" being changed over
the network when user has set it to non-empty string manually.
This also fixes the problem when "name" and "authname"
became unsynchronized accidentally, when they were equal and then
Origin::IncomingUnknownTo update arrived, setting "name" but not
"authname". Rust regression test is added for this case.
Encryption preference was already applied since commit 78d855c5ca,
but only for already existing peerstates.
As a result, new users ignored gossiped encryption preference in the
"member added" message and had to wait until someone gossiped encryption
preferences for inactive users the second time.
It is misleading and should never be checked.
dc_stop_io() also stops pending ephemeral message deletion tasks, so it
should be called in any case before releasing context.
Start implementing #1994
TODO (in later PRs):
- Add a hint to the watch settings that all folders are fetched from time to time (to be done in the individual UIs)
- folder names are case-insensitive, so double-check that all comparisons are case-insensitive
- The `scan_folders.rs` file didn't get as large as I expected and it's probably not worth it having an extra file for it. But if there are no objections, I'll make another PR to rename it to `folders.rs` and also put into it `configure_folders()` from `imap/mod.rs` and `needs_move()` with all its tests from `message.rs`.
Done:
- Most mailboxes have a "Drafts" folder where constantly new emails appear but we don't actually want to show them, what do we do about this? The most reliable way to detect such messages that we found up to now is:
If there is no `Received` header AND it's not in the `ConfiguredSentbox`, then ignore the email.
- before or after INBOX idle trigger a new "scan all folders for messages". It does a "list folders" and then goes through all folders with select-statements, checking if "next-uid" was changed since checked last time. This might be batchable but in any case should not consume a lot of traffic. We might debounce this scan activity to happen at most every N minutes
- if next-uid changed for a folder, we "prefetch" and "fetch" DC-messages as as needed ("dc-messages" are not just those with "Chat-Version" headers, but can also be regular emails)
- if we discover DC-messages in folders that have the "/Spam" flag (maybe excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
- for now, we do not change any user visible option, but introduce this "scan all" automatically and on top of what exists. The DeltaChat folder-watching does not perform scan-all-folders (maybe with the exception to trigger scan-all also with DeltaChat if INBOX is not watched)
- Tests (except if you have ideas to improve them)
- all folders, their last uidvalidity, next-seen etc. are kept in a separate "imap-sync" sqlite table. Maybe this can be used to streamline some of the "Sent" folder and "DeltaChat" folder detection code we already have.
- We now also move self-sent messages from the Inbox to the Sent folder if `mvbox_move` is off, as this was very easy to do now. This way, we now behave more like a normal MUA if the user wants this.
FOR LATER PRs:
- maybe for the first 50 messages or so, we could reduce the IDLE-timeout (currently 23 minutes or so) to faster detect messages sent to non-inbox folders. However, on Android and iOS, we would likely trigger scan-all when the app moves to foreground, and so it might not be neccessary to reduce the current idle-timeout at least for them. We can leave this "faster discovery" question for the end, after we move to real-life testing.
- (Later on, after the above works, we can consider heuristics on which folders to perform IDLE on, and remove the Watch-folder options (inbox, deltachat, sent). We tried to find a safe scheme for already doing it but failed to fine one, too many unknowns, also some questions regarding multi-device (you might have different settings with each of it, one moves, the other doesn't etc.) so we postponed this in favor of the above incremental suggestion.)
* Start implementing #1994
* Add debug logs, it seems like the SQL migration can go into another pr
* Let fetch_new_messages return whether there are new emails
* Code style
* Don't prefetch if there are no new emails
* clippy
* Even more debug logs
* If the folder was not newly selected, return always try to fetch as
uid_next is probably outdated
* Fix new bug
* Recognize spam folder
* if we discover DC-messages in folders that have the "/Spam" flag (excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
* Clippy, prioritize folder_meaning over folder_name_meaning
* Add a first test, for the first day after installation only debounce to 2s
* Start adding two tests (both of them fail)
* Don't abort folder scan if one folder fails
* More consts
* Replace bool return value by enum
* Split test up into multiple tests
* Print logs during rust tests
* Rust tests pass now
* .
* One of the Python tests passes now - reconfigure folders during scanning
* Make the last test pass - Delete emails in all folders when starting the test, not only inbox and mvbox
The problem had been that emails were left in the folder "xyz"
* lint
* DB migration (untested)
* Store uid_next in SQL instead of lastseen in a config
* Revert "If Inbox-watch is disabled and enabled again, do not fetch emails from in between"
all folders are always watched, anyway
* clippy, rm debug logs, comments
* Codestyle, comments
* fixing things again
* Fix another test: don't fetch from uid_next-1 but uid_next; make some {} to {:#} so that we can use `.context(...)`
* move self-sent, non-setupmessage chat messages to the Sent folder if `mvbox_move` is off
* comment
* Comments, make sure things work even if there is no uid_next
* Style
* Comments
* The rust test tested wrongly
* comments, small codestyle change
* Ignore emails that are probably only drafts
Most mailboxes have a "Drafts" folder where constantly new emails appear
but we don't actually want to show them.
So: If there is no Received header AND it's not in the ConfiguredSentbox,
then ignore the email.
Also: Add test.
* Fix occasional test failure, it was introduced as DC now moves messages from Inbox to Sent
* Add `Received` header to the rust tests
* After this PR we will always watch all folders and delete messages there if server_delete is enabled. So, for people who have server_delete on, disable it and add a hint to the devicechat
* comment, small fix
* link2xt's first review
* Use ON CONFLICT(FOLDER) DO to update and if it doesn't exist, then insert
Reason from link2xt: We had a problem with multiple peerstates inserted due to key fingerprint parsing error previously. With logic in Rust a similar problem can occur: an UPDATE can fail for reasons other than a conflict. PRIMARY KEY should ensure uniqueness in this case, but anyway.
* Remove two TODO statements, remove fetch_new_messages: ignoring uid {}, uid_next was {} log
* Next TODO: Make uidvalidity and uid_next DEFAULT 0
* rm two TODOs, Seems like we are not going to `exclude folders that are watched anyway` in this PR
* small tweak: Handle instants more carefully
* Add scan_all_folders_debounce_secs config for tests, set debounce to 60s (before it was just 2s during the first day)
* Don't use bold letters for the device message
* React to changes in the folders better
Before, if there was a configured Sent folder, but then it got
removed and replaced with another folder with a name meaning "Sent" but
without Sent flag, it would be ignored.
So, instead of checking against ConfiguredSentboxFolder,
create two Option variables at the beginning of the loop and replace
them with Some if it is None. At the end of the loop, store the new
values into ConfiguredSendboxFolder and ConfiguredSpamFolder, even if it
is None.
Also, derive some useful traits.
* move job: Return a meaningful error if server_folder is None instead of panicing
* small error-handling fix
* Fix test_fetch_existing() python test
Before, we sometimes got a race condition where scan_folders() sees that
there is a Sentbox and saves this info after we set the
ConfiguredSentbox to None and before the message is sent.
So, just expect that the message is moved to the sentbox.
* migration is 72 now
* rm 2 TODOs, Don't infinitely retry when dc_receive_imf() returns Err
* clippy: Remove glob imports
* Delete created folders at the beginning of tests
(some created folders made problems in the next tests because)
* Improve resetting accounts between tests
If mvbox_move was on and an mdn was received, it was not marked as read.
Also remove a confusing log that said that we are adding a markseen job, even if we weren't. As job::save() logs itself, there is no need to log this again.
