Commit Graph
100 Commits
Author SHA1 Message Date
Hocuri 6c8da526a0 Fix: Only show the "Cannot login" device message if it's actually authentication that failed (#2804)
Generally we could also just remove the device message as we have the
connectivity view, OTOH if you are not using DC a lot, it may be useful
to be notified without opening DC.

And apart from this one bug it's working fine.
2021-11-13 19:20:01 +01:00
Hocuri b5d0907090 Get back stack traces and error messages when tests fail (#2758)
Before this PR, when a test failed, we often got:

```
thread panicked while processing panic. aborting.
error: test failed, to rerun pass '-p deltachat --lib'

Caused by:
  process didn't exit successfully: `/home/user/deltachat-android/jni/deltachat-core-rust/target/debug/deps/deltachat-33648fc4aaad608c 'contact::tests::test_selfavatar_changed_event' --nocapture` (signal: 4, SIGILL: illegal instruction)
```

instead of the error message and stack trace.

This PR fixes this.
2021-10-18 10:48:13 +02:00
Hocuri 6613fa67ee Add DC_EVENT_SELFAVATAR_CHANGED (#2742)
* Add DC_EVENT_SELFAVATAR_CHANGED

* Add test.

Unfortunately I can't easily also test that the avatar is not copied
from unencrypted messages:

In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.

* Unfortunately I can't easily also test that the avatar is not copied from unencrypted messages:

In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.
2021-10-18 10:42:16 +02:00
Hocuri 5e29cae81a Fix: Don't update quota in an endless loop (#2726)
The problem was:
When opening the connectivity view while there is no network,
get_connectivity_html() calls schedule_quota_update(), which schedules a
UpdateRecentQuota job. But in update_recent_quota(), connecting fails
and a ConnectivityChanged event is emitted (connectivity changes from
Error to Connecting and back). Therefore the UI calls
get_connectivity_html() again, and the loop is complete.

This made the UI completely unresponsible. To reproduce, just turn wi-fi
off and open the connectivity view.

The fix is:
schedule_quota_update() now only schedules a new job if there is no old
job. To prevent the possible (though probably unlikely) problem that an
old quota update job has a backoff of, like, a day and therefore quota
is not updated, I reduced the backoff time for quota jobs to 10s.

Fixes possibly https://github.com/deltachat/deltachat-android/issues/2043, but we should "re-try" this
2021-10-05 10:57:34 +02:00
Hocuri 46956caf75 Fix: Recognize ndns that put the headers into "message/global-headers" part (Improve ndn detection) (#2598)
I sent a message, and the ndn (Non Delivery Notification) was not parsed correctly, so here comes
the fix and the test.
2021-09-12 21:02:51 +02:00
HocuriandB. Petersen 223aeb7b1a Fix: Make emails forwarded by GMX readable (#2600)
Recognizing these emails as forwarded would probably be too complicated and require too much special-casing, but now the user can access the email text via "Show full message".

fix #2599

Co-authored-by: B. Petersen <r10s@b44t.com>
2021-08-14 18:05:17 +02:00
Hocuri 481276cf46 In dc_maybe_network_lost() directly set the connectivity "Not connected" (#2551)
* In dc_maybe_network_lost() directly set the connectivity "Not connected"

r10s reported that without doing this, the connectivity would stay at
"Connected" for 16 more seconds after network is gone and
dc_maybe_network_lost() was called.

* Set the state for all connections
2021-08-09 17:25:34 +02:00
Hocuri faab61b0d4 Connectivity view: Only set smtp to "connected" if the last message was actually sent (#2541)
fix #2539

It's always a bit ambiguous which function should do set_err or set last_send_error, I used this rule here:

If some code returns Status::RetryLater, then it sets last_send_error, because Status::RetryLater means that the user won't see the error directly on the message (if we returned Status::Failed(Err(_)), then the message would be shown as failed to the user) Also, smtp_send always sets last_send_error because, well, sending just failed or succeeded.

Also, remove unused field pending_error.
2021-08-09 12:31:33 +02:00
Hocuri 6e7f63dba7 Fix which chats messages are assigned to (#2465)
fix https://github.com/deltachat/deltachat-core-rust/issues/2463
fix #2116

The email could be private (i.e. only sent to me) or non-private (i.e. also sent to sb else). Also, it could be a classical email or a chat message. Below, I'm describing for each of the four combinations whether they should be assigned to a chat by create_or_lookup_group() or lookup_chat_by_reply(). Because I needed to use these function names a lot, I shortened them to l:group() and l:by_reply() in this PR description.

(!) means todo, (! -> Done) means I fixed something.

## Private classical email: 
l:group() and l:by_reply() must both take care not to put it into group
l:group() no (! -> Done), l:by_reply no (! -> Done)
except for classical MUA replies to two-member-groups, they should be put into the parent group

### wrt alias-support:

A private classical email is very probably not going to be an answer to email that went to an alias address:

Suppose Alice writes to support@example.com.
support@example.com forwards to bob@example.com and claire@example.com.
When Bob answers, he will _probably_ answer `To: alice@example.com, support@example.com` (=> it's a non-private classical email).

With this PR, if he does only answer `To: alice@example.com`, (=> it's a private classical email), Alice's DC will show the answer in the private chat with Bob. Which actually makes more sense than showing it in the support@example.com chat I think. Also, if it was shown in the support@example.com chat, then Alice would answer in the support@example.com chat, then Claire would get Alice's message but not Bob's and therefore miss some context.
That being said - **I could change this**. Pretty easily actually, I would just have to remove the call to `is_probably_private_reply()` from `lookup_chat_by_reply()`. Didn't think through all edge cases yet, but should work.

## Private chat message:
l:group() has to put private chat messages into the group: It won't mistakenly put a message there (because it can rely on Chat-Group-Id). Currently, `is_probably_private_reply()` returns true for private chat messages, but l:group still has to put them into the group chat. (it's not nice that the function called is_probably_private_reply returns true for all private chat messages, but I didn't find any nicer solution) l:by_reply() must not assign it to sth special
l:group yes, l:by_reply no

By the way, for chat messages, `try_getting_grpid()` doesn't look at InReplyTo or References, in order not to put private chat-message replies into the group chat.

For alias-support, the same goes as for the private classical emails.

## Non-private classical email:
Just put it into any group, and if there is none, create one. 

_Off-topic:
We currently don't look at the recipients lists, which means that a message can easily be assigned to a group although it was not sent to all group members. One day we could somehow compare the recipients list with the members list, but that needs some more discussing (esp. what do we do if they don't match? Create a new group? Show a hint in the UI?) and is nothing for a bug-fixing PR like this one._

l_group yes, l_by_reply yes, even for outgoing messages (! -> Done, this also was the issue reported by @gerryfrancis in the testing group:

Outgoing messages were not put into a chat by In-Reply-To/References, which is correct for chat messages, but for non-private classical outgoing emails, a new ad-hoc group was created everytime.

So, I added this: https://github.com/deltachat/deltachat-core-rust/pull/2465/files#diff-e7606b521f6710ddc6e5236ba5d7eefc917b7ad744b9e71762fd42830c55485bR703-R711)

## Non-private chat message:
Can be put into chat by l:by_reply() because it must be a group message
l:group yes, l:by_reply yes (to make alias support work if the support person uses DC)

Nothing to test or fix here; we have to put chat group messages into the group, which is trivial. And we have to make sure that alias-support works, which already was well tested.
2021-07-19 16:02:11 +02:00
Hocuri 308403ad99 Connectivity view (instead of spamming the user with error_network when sth fails) (#2319)
See https://support.delta.chat/t/discussion-how-to-show-error-states/1363/10 <!-- comment -->

It turns out that it's pretty easy to distinguish between lots of states (currently Error/NotConnected, Connecting…, Getting new messages… and Connected). What's not that easy is distinguishing between an actual error and no network, because if the server just doesn't respond, it could mean that we don't have network or that we are trying ipv6, but only ipv4 works.

**WRT debouncing:**

Sending of EVENT_CONNECTIVITY_CHANGED is not debounced, but emitted every time one of the 3 threads (Inbox, Mvbox and Sentbox) has a network error, starts fetching data, or is done fetching data.
This means that it is emitted:
- 9 times when dc_maybe_network() is called or we get network connection
- 12 times when we lose network connection

Some measurements: dc_get_connectivity() takes a little more than 1ms (in my measurements back in March), dc_get_connectivity_html() takes 10-20ms. This means that it's no immmediate problem to call them very often, might increase battery drain though. For the UI it may be a lot of work to update the title everytime; at least Android is smart enough to update the title only once.

Possible problems (we don't have to worry about them now I think):
- Due to the scan_folders feature, if the user has lots of folders, the state could be "Connecting..." for quite a long time, generally DC seemed a little unresponsive to me because it took so long for "Connecting..." to go away. Telegram has a state "Updating..." that sometimes comes after "Connecting...".

To be done in other PRs:
- Better handle the case that the password was changed on the server and authenticating fails, see https://github.com/deltachat/deltachat-core-rust/issues/1923 and https://github.com/deltachat/deltachat-core-rust/issues/1768
- maybe event debouncing  (except for "Connected" connectivity events)

fix https://github.com/deltachat/deltachat-android/issues/1760
2021-07-08 22:50:11 +02:00
Hocuri b16785bb62 Tweak test 2021-06-12 16:03:55 +02:00
Hocuri d12d9d94d6 Improve test_migration_flags, add EvTracker to test_utils 2021-06-12 16:03:55 +02:00
Hocuri 25fed9ab52 comment 2021-06-11 13:09:22 +02:00
Hocuri 751b9add09 remove unneeded files 2021-06-11 13:09:22 +02:00
Hocuri b727190da5 test low media quality a little more 2021-06-11 13:09:22 +02:00
Hocuri 368fa9fc44 Test image rotation 2021-06-11 13:09:22 +02:00
Hocuri 3b897eac53 Fix downscaling images (#2469)
* Add failing test

* typo

* Fix the bug
2021-06-07 10:14:03 +02:00
Hocuri bf7f64d50b Ignore Drafts folder when scanning (#2454)
* Add failing test for #2369

* Completely ignore Drafts folder

fix #2369

* Also ignore messages that have the Draft flag set but are not in the Drafts folder
2021-06-03 21:14:39 +02:00
Hocuri e3e2adeea5 Fix all outgoing messages popping up in selfchat (#2456)
Fix https://github.com/deltachat/deltachat-android/issues/1940, fix https://github.com/deltachat/deltachat-core-rust/issues/2220 (I assume these are the same bug)

The problem was:
- Gmail adds a header `Bcc: <self-address>` to our bcc-self message
- `to_id` was just set to the first recipient, in this case _self_
- it was seen that `to_id` is _self_, so it's a self-sent message
2021-05-28 20:02:46 +02:00
Hocuri 46e901be78 Fix benchmark compile error (#2457) 2021-05-28 18:48:13 +02:00
Hocuri a9f5077cf9 Make scan_folders work when inbox is not watched (#2446)
When watch_inbox was off, scan_folders failed and a toast "IMAP operation attempted while it is torn down" was shown.

--

The problem was:

When inbox_watch is off, scan_folders() is called at https://github.com/deltachat/deltachat-core-rust/blob/244260a9789a34c8cdd50c73b96557a8ced40345/src/scheduler.rs#L107 but connect_configured() is not called before.



* Add test

* Don't only setup handle, but connect_configured() in scan_folders()
2021-05-16 22:18:38 +02:00
Hocuri 1c436777e0 Fix #2429 (message was downloaded multiple times a second), add test (#2430)
The problem was:

If a message has the \Deleted flag set, we ignore it. But we forgot to
update last_uid, so that uid_next was not updated at https://github.com/deltachat/deltachat-core-rust/blob/47e639b777428a8104ada1e9bb94502e72092311/src/imap.rs#L730 and the same message was
fetched over and over again.

Fix #2429
2021-05-09 18:43:55 +02:00
Hocuri 296c230bc9 Correct typo 2021-05-09 10:27:11 +02:00
Hocuri b7864f232b Compress avatar to below 20k (#2384)
- Currently, group images are compressed as well because it was easier to implement that way.
- Currently, in the unlikely case that the avatar is compressed down to 20x20 pixels but still bigger than 20KB, the user doesn't get any indication of this, the avatar simply isn't changed (at least on Android).

  If we want to change this, the easiest way is probably to let `dc_set_config()` in the ffi call `error!()` if `Selfavatar` can't be set. The same might make sense for some or all other configs. BUUUUUT: At least Android doesn't show error!() toasts anymore, probably because they were used too often and too spammy.
- The factor by which we scale down if the file is too big is 1.5.
2021-05-02 19:54:13 +02:00
Hocuri 819775ac39 Revert "More logging for "core spams imap events""
This reverts commit 5394327bf6.
2021-04-23 09:15:15 +02:00
Hocuri a1ef32170d Make logging less verbose 2021-04-22 18:03:37 +02:00
Hocuri a4486d8c30 Fix #2335 (delete job flooding) (#2372)
* Fix #2335 (delete job flooding)

The problem was:

- You are offline, and an ephemeral message is due to delete.
- load_imap_deletion_job() is called and returns a deletion job for it.
- The job fails because, well, we are offline.
- The job is saved into the database.
- load_imap_deletion_job() is called again, and so on.

* Add test
2021-04-22 16:37:22 +02:00
Hocuri 553d3936a9 Some general Python test improvements (#2316)
This PR originally contained a fix for sqlx which turned out not not to be necessary. But on the way, I made some general improvements:

- Under some circumstances, a "normal" test failure led to a timeout, without printing a decent error message. See e.g. https://app.circleci.com/pipelines/github/deltachat/deltachat-core-rust/8069/workflows/ba2a9949-b4ad-4ceb-a930-073bba05e2db/jobs/30965.
  (The problem was: if there is an exception in dc_account_extra_configure(), when trying to handle the exception the line account.log("===================", e, "===================") was called, which can't work as log() only expects one parameter)
- When a test fails: Call `dump_account_info()` even if there is no direct_imap
- In test_import_export_online_all(), add another 100KB file to the backup. This adds resilience against future size changes of the sql db file: The test tests the smoothness of the progress bar. And if there are there are not enough about-equally-sized files, the progress bar can't be smooth.
2021-04-18 19:20:31 +02:00
Hocuri 3707471266 Add alias support 2 (#2297)
fix  #2073
fix #2292 (I think)

- Messages can be assigned to any chat by the References and In-Reply-To, also to 1:1 chats; this has higher priority than the group id because with ad-hoc groups, it can happen that two devices have different group ids for the same conversation thread.
- If `From` is not in the chat (we call this "shadow sender"), `OverrideSenderDisplayname` is set. This communicates to the UI that:
  - A `~`should be added in front of the sender's displayname.
  - Also in 1:1 chats, the sender's displayname and avatar is shown, as if this was a group.

  The "Unknown sender for this chat" messages are completely removed for unprotected groups.

For protected chats, everything stays as it was before.

POSTPONED:

- Maybe (if it turns out to be still necessary):
  - The ad-hoc group id is computed by the the References, instead of the member list, as it is currently done
  - How do we prevent that the message can't be assigned to the correct chat as the parent message was deleted?
2021-04-10 22:06:22 +02:00
Hocuri 5394327bf6 More logging for "core spams imap events"
TODO: revert
2021-04-10 17:08:41 +03:00
Hocuri df277b374d Ignore unknown classical emails from spam folder (#2311) 2021-04-10 10:45:47 +02:00
Hocuri 179a2a50e6 Parse <blockquote> tags for better quote detection (#2313) 2021-04-07 18:45:00 +02:00
Hocuri 0de8b6a7e5 Update uid_next if the server rewinded it
fix #2188

Also, if we notice that the server started reusing old UIDs, _also_ do a `ResyncFolders`, because the server likely forgot to change uid_validity
2021-03-18 16:14:56 +03:00
Hocuri 7bc919fad5 Save subject for messages 2: Outgoing messages (#2283)
* Save subject for sent-out messages

* Test that subject is saved (outgoing)

* Correctly set subject when forwarding messages, add test for this
2021-03-14 15:07:49 +01:00
HocuriandFloris Bruynooghe 98fc559536 Even nicer logging: Add ok_or_log() and more (#2284)
Co-authored-by: Floris Bruynooghe <flub@devork.be>
2021-03-13 21:06:37 +01:00
Hocuri 99b2d79312 When forwarding a message that is an impersonating message, the forwarder should not be impersonating
fix #2287
2021-03-12 11:47:02 +01:00
Hocuri 04891238d4 save subject for messages (#2274)
save subject for messages:

- new api `dc_msg_get_subject()`,

- when quoting, use the subject of the quoted message as the new subject, instead of the
last subject in the chat
2021-03-07 16:57:12 +01:00
Hocuri 165c57f0a4 Rust-tests: Don't panic while panicking 2021-03-03 11:58:21 +01:00
Hocuri 476e613377 Trash messages more thoroughly (#2273)
Esp. remove some information for newly-arriving messages
2021-03-02 12:04:53 +01:00
Hocuri 2a39dc06e9 Fix imex race condition, (#2255)
fix #2254: if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message), the db was corrupted.

* Add result.log() for logging with less boilerplate code

* Bugfix: Resultify housekeeping() to make it abort if the db is closed instead of just deleting everything

* Require the UI to call dc_stop_io() before backup export

* Prepare a bit better for closed-db: Resultify get_uidvalidity and get_uid_next and let job::load_next() wait until the db is open

About the bug (before this PR):
if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message):
- I don't know if it downloads the message, but of course at some point the process of receiving the message will fail
- In my test, DC is just in the process of moving a message when the imex starts, but then can't delete the job or update the msg server_uid
- Then, when job::load_next() is called, no job can be loaded. That's why it calls `load_housekeeping_job()`. As `load_housekeeping_job()` can't load the time of the last housekeeping, it assumes we never ran housekeeping and returns a new Housekeeping job, which is immediately executed.
- housekeeping can't find any blobs referenced in the db and therefore deletes almost all blobs.
2021-03-02 10:25:02 +01:00
Hocuri 6132cc2a42 Update rust-email, test that no empty lines are produced (#2119)
* Add a failing test for #2118

* Update rust-email to fix #2118
2021-02-17 13:34:54 +01:00
Hocuri cda2fc4fea Assert that sentbox and mvbox folders stay configured during test_fetch_existing
Now I know why the tests failed before 48c58a7 (i.e. after c923670) but
not on master (i.e. before c923670):

because of a bug, scan_folders() set the configured_sentbox to None if
it was set before. If it was None before, it restored the correct value.

On master, there was another bug that led to two runs of
scan_folders() being started at the same time. Therefore, the first run
set configured_sentbox to None, the second one restored the correct
value.

c923670 fixed the latter bug, so that only one run of scan_folders() was
started. Therefore, configured_sentbox stayed None incorrectly and
test_fetch_existing() failed.

48c58a7 fixed the former bug.

This commit adds checks to test_fetch_existing(), so that it definitely
checks for the former bug.
2021-02-14 20:41:06 +01:00
Hocuri 95a0481b63 scan_folders() bugfix: Don't exclude watched folders from being set as sent/spam folder
This does fix a bug and it makes the tests pass, but I'm not sure why it
makes the tests pass; maybe there is a race condition that made the
tests fail and my commit just leads to another timing.
2021-02-14 20:41:06 +01:00
Hocuri 7d27c2bfea Remove confusing "Not scanning" log msg 2021-02-14 20:41:06 +01:00
Hocuri c0023cb54d Only scan on the Inbox thread
This should prevent race conditions when multiple threads fetch at once.
2021-02-14 20:41:06 +01:00
Hocuri e3f7b31501 Untested: If we can't find the newest backup file by opening the sql file, find it by name comparison 2021-02-14 19:48:15 +01:00
Hocuri 35b0f00a88 legacy (.bak) import: If backup files can't be opened, just try one that couldn't be opened
fix https://github.com/deltachat/deltachat-android/issues/1768
2021-02-14 19:48:15 +01:00
Hocuri 10435a10e9 Don't scan watched folders
This should fix the duplicate-messages-bug
2021-02-10 16:56:52 +03:00
HocuriandB. Petersen 396ec131fc Implement receiving mailing lists (#1964)
* Copypaste-merge my old work

* Start implementing mailinglists the new way

* Create pseudo contact

* Fine-tune docs

* Remove some unnecessary changes

* style

* Make a stock str

* Fix a crash. Yes, this line caused a panic when reconfiguring on Android

(without a reasonable error log). Also, I could not receive any messages
anymore.

* rfmt

* Add tests and make them pass

* Even more tests

* rfmt

* Enhance test and fix bug

* Don't update the sender name when prefetching because maybe it's a mailing list

* Use an enum instead of numbers for the decision

* Don't remove anyone from mailing lists

* Fix bug in the regex

* Adjust error msg

* Compile error after rebase

* Correctly emit event

* Add dc_msg_is_mailing_list so that you can find out whether messages in the deaddrop belong to mailing lists.

* Add received headers to unit tests

* Comments, small tweaks

* Use dc_msg_get_override_sender_name instead of dc_msg_get_sender_name

* Add dc_msg_get_sender_first_name() because sometimes the first name was not correctly shown in mailing lists

* small fixes, don't let the user modify mailing list groups

* Hide contacts for addresses like noreply@github.com and reply+AEJ...@reply.github.com

When testing mailing lists, I noticed that sometimes a mailing list
contact got a name (like, hocuri <noreply@github.com>). It turned out
that ages ago, I had accidentally written an email to - in this example
- hocuri <noreply@github.com> and it had been added to the contacts
list.

This hides email addresses from the contacts list that are obviously not
meant to be written at and prevents updating the names.

* Comment, clippy

* Replace u32 with ChatId

* Resolve lost of small issues from the reviews

* remove dc_msg_get_sender_first_name

* add dc_msg_get_real_chat_id()

this allows to check if a contact request belongs to a mailing list
and to show name/avatar/whatever of the mailinglist.

another approach was to duplicate some chat-apis for messages
(eg. dc_msg_is_mailing_list()) however that would require far more new apis.

the idea to change the behavior of dc_msg_get_chat_id() would be
more clean, however, that easily breaks existing implementations
and can led to hard to find bugs.

* remove now unused Message.is_mailing_list()

* if a name for a mailing list is missing, use List-ID

* fix comment

* fix error message

* document how dc_get_chat_contacts() works for mailing lists

* refine decide api (#2185)

* add DC_DECIDE* constants to deltachat.h, tweak documentation

* use StartChat/Block/NotNow instead of Yes/Never/NotNow

* decide_on_contact_request works on ctx/msg-id

functions working on message-objects
usually do not read or write directly on the database.

therefore, decide_on_contact_request()
should not work with message-objects as well,
it is even a bit misleading, as eg. chat-id of the object is not modified.

instead, the function works on context,
similar to dc_send_msg(), dc_create_chat() and so on.
for now, i moved it to context, could maybe be part od MsgId.

* Update src/chatlist.rs

Co-authored-by: Hocuri <hocuri@gmx.de>

Co-authored-by: Hocuri <hocuri@gmx.de>

* refine documentation

* re-add accidentally deleted Param::MailingList

* remove pseudo-contact in domain @mailing.list

1. the pseudo-contact was added to the mailing list contacts,
   which is not needed.
   might be that we want to add the recent contacts there in a subsequent pr
   (we do not know all contacts of a mailing list)

2. the pseudo-contact was used to block
   mailing lists; this is done by setting the chat to Blocked::Manually now

3- the pseudo-contact was used for unblocking;
   as it is very neat not to require additional ui for mailing list unblocking,
   might be that we introduce a similar pseudo-contact for this limited purpose
   in a subsequent pr, however, the pseudo-contact needs to exist only
   during unblocking then, maybe also the special domain is not needed,
   we'll see :)

* Move dc_decide_on_contact_request() up to the dc_context section as it's a member of dc_context now

More specifically, to the "handle messages" section

* re-introduce Chattype::Mailinglist (#2195)

* re-introduce Chattype::Mailinglist

* exhaustive chattype-check in fetch_existing_msgs() and get_summary2()

* exhaustive chattype-check in ndn_maybe_add_info_msg()

* exhaustive chattype-check in message::fill()

* remove dc_chat_is_mailing_list() from ffi

Co-authored-by: B. Petersen <r10s@b44t.com>
2021-02-07 23:37:04 +01:00
Hocuri b0a3a0046c Add new sentbox_move to get_info() 2021-02-04 20:26:00 +01:00
Hocuri 8ac2cdd929 Add set_config_bool, fix rust test to test the current expected behavior 2021-02-04 20:26:00 +01:00
Hocuri e820d072f5 Disable moving to Sentbox for now
fix https://github.com/deltachat/deltachat-core-rust/issues/2176

we can re-enable it when DC determines automatically which folders to watch and
this is deployed for at least 1 release.
2021-02-04 20:26:00 +01:00
Hocuri b67fbedcef Create get_configured_provider(), remove delete_server_after == "1" check 2021-01-21 10:36:00 +01:00
Hocuri 687db252b6 do not disable server-deletion when enabled by provider-db
fix #2157
2021-01-21 10:36:00 +01:00
Hocuri ccee289a5c Don't ignore incoming mails without Received-header
fix #2149
2021-01-17 18:17:19 +01:00
Hocuri 08c46af3aa Switch to Rust 1.49.0
Notable improvements:
- [track_caller](https://github.com/rust-lang/rfcs/blob/master/text/2091-inline-semantic.md): We can get the location of the caller of a message, which makes logging easier as you can have dedicated functions (not only macros), e.g. write a function `log(self: Result, msg: string) -> Option`
- The output after a panic is shorter and hides lots of uninteresting
function calls
- Test output is better captured

After merging, we should probably wait a bit with using the track_caller
feature to make going back easier, should we get problems again.
2021-01-17 04:43:41 +03:00
Hocuri 4636785449 Revert "Add verbose logging for #2065"
This reverts commit 83df69f43c.
2021-01-15 15:06:05 +01:00
Hocuri ebccdbbcb9 Improve onboarding by scanning all folders from time to time (#2067)
Start implementing #1994

TODO (in later PRs):

- Add a hint to the watch settings that all folders are fetched from time to time (to be done in the individual UIs)
- folder names are case-insensitive, so double-check that all comparisons are case-insensitive
- The `scan_folders.rs` file didn't get as large as I expected and it's probably not worth it having an extra file for it. But if there are no objections, I'll make another PR to rename it to `folders.rs` and also put into it `configure_folders()` from `imap/mod.rs` and `needs_move()` with all its tests from `message.rs`.

Done:

- Most mailboxes have a "Drafts" folder where constantly new emails appear but we don't actually want to show them, what do we do about this? The most reliable way to detect such messages that we found up to now is:
  If there is no `Received` header AND it's not in the `ConfiguredSentbox`, then ignore the email.
- before or after INBOX idle trigger a new "scan all folders for messages".  It does a "list folders" and then goes through all folders with select-statements, checking if "next-uid" was changed since checked last time.  This might be batchable but in any case should not consume a lot of traffic. We might debounce this scan activity to happen at most every N minutes 

- if next-uid changed for a folder, we "prefetch" and "fetch" DC-messages as as needed ("dc-messages" are not just those with "Chat-Version" headers, but can also be regular emails) 

- if we discover DC-messages in folders that have the "/Spam" flag (maybe excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help  provider-spam-systems to regard this contact/mail as non-spam 

- for now, we do not change any user visible option, but introduce this "scan all" automatically and on top of what exists.   The DeltaChat folder-watching does not perform scan-all-folders (maybe with the exception to trigger scan-all also with DeltaChat if INBOX is not watched)

- Tests (except if you have ideas to improve them)
- all folders, their last uidvalidity, next-seen etc. are kept in a separate "imap-sync" sqlite table.  Maybe this can be used to streamline some of the "Sent" folder and "DeltaChat" folder detection code we already have. 

- We now also move self-sent messages from the Inbox to the Sent folder if `mvbox_move` is off, as this was very easy to do now. This way, we now behave more like a normal MUA if the user wants this.

FOR LATER PRs:

- maybe for the first 50 messages or so, we could reduce the IDLE-timeout (currently 23 minutes or so) to faster detect messages sent to non-inbox folders. However, on Android and iOS, we would likely trigger scan-all when the app moves to foreground, and so  it might not be neccessary to reduce the current idle-timeout at least for them.  We can leave this "faster discovery" question for the end, after we move to real-life testing. 

- (Later on, after the above works, we can consider heuristics on which folders to perform IDLE on, and remove the Watch-folder options (inbox, deltachat, sent). We tried to find a safe scheme for already doing it but failed to fine one, too many unknowns, also some questions regarding multi-device (you might have different settings with each of it, one moves, the other doesn't etc.) so we postponed this in favor of the above incremental suggestion.)




* Start implementing #1994

* Add debug logs, it seems like the SQL migration can go into another pr

* Let fetch_new_messages return whether there are new emails

* Code style

* Don't prefetch if there are no new emails

* clippy

* Even more debug logs

* If the folder was not newly selected, return always try to fetch as

uid_next is probably outdated

* Fix new bug

* Recognize spam folder

* if we discover DC-messages in folders that have the "/Spam" flag (excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam

* Clippy, prioritize folder_meaning over folder_name_meaning

* Add a first test, for the first day after installation only debounce to 2s

* Start adding two tests (both of them fail)

* Don't abort folder scan if one folder fails

* More consts

* Replace bool return value by enum

* Split test up into multiple tests

* Print logs during rust tests

* Rust tests pass now

* .

* One of the Python tests passes now - reconfigure folders during scanning

* Make the last test pass - Delete emails in all folders when starting the test, not only inbox and mvbox

The problem had been that emails were left in the folder "xyz"

* lint

* DB migration (untested)

* Store uid_next in SQL instead of lastseen in a config

* Revert "If Inbox-watch is disabled and enabled again, do not fetch emails from in between"

all folders are always watched, anyway

* clippy, rm debug logs, comments

* Codestyle, comments

* fixing things again

* Fix another test: don't fetch from uid_next-1 but uid_next; make some {} to {:#} so that we can use `.context(...)`

* move self-sent, non-setupmessage chat messages to the Sent folder if `mvbox_move` is off

* comment

* Comments, make sure things work even if there is no uid_next

* Style

* Comments

* The rust test tested wrongly

* comments, small codestyle change

* Ignore emails that are probably only drafts

Most mailboxes have a "Drafts" folder where constantly new emails appear
but we don't actually want to show them.
So: If there is no Received header AND it's not in the ConfiguredSentbox,
then ignore the email.

Also: Add test.

* Fix occasional test failure, it was introduced as DC now moves messages from Inbox to Sent

* Add `Received` header to the rust tests

* After this PR we will always watch all folders and delete messages there if server_delete is enabled. So, for people who have server_delete on, disable it and add a hint to the devicechat

* comment, small fix

* link2xt's first review

* Use ON CONFLICT(FOLDER) DO to update and if it doesn't exist, then insert

Reason from link2xt: We had a problem with multiple peerstates inserted due to key fingerprint parsing error previously. With logic in Rust a similar problem can occur: an UPDATE can fail for reasons other than a conflict. PRIMARY KEY should ensure uniqueness in this case, but anyway.

* Remove two TODO statements, remove fetch_new_messages: ignoring uid {}, uid_next was {} log

* Next TODO: Make uidvalidity and uid_next DEFAULT 0

* rm two TODOs, Seems like we are not going to `exclude folders that are watched anyway` in this PR

* small tweak: Handle instants more carefully

* Add scan_all_folders_debounce_secs config for tests, set debounce to 60s (before it was just 2s during the first day)

* Don't use bold letters for the device message

* React to changes in the folders better

Before, if there was a configured Sent folder, but then it got
removed and replaced with another folder with a name meaning "Sent" but
without Sent flag, it would be ignored.

So, instead of checking against ConfiguredSentboxFolder,
create two Option variables at the beginning of the loop and replace
them with Some if it is None. At the end of the loop, store the new
values into ConfiguredSendboxFolder and ConfiguredSpamFolder, even if it
is None.

Also, derive some useful traits.

* move job: Return a meaningful error if server_folder is None instead of panicing

* small error-handling fix

* Fix test_fetch_existing() python test

Before, we sometimes got a race condition where scan_folders() sees that
there is a Sentbox and saves this info after we set the
ConfiguredSentbox to None and before the message is sent.

So, just expect that the message is moved to the sentbox.

* migration is 72 now

* rm 2 TODOs, Don't infinitely retry when dc_receive_imf() returns Err

* clippy: Remove glob imports

* Delete created folders at the beginning of tests

(some created folders made problems in the next tests because)

* Improve resetting accounts between tests
2021-01-13 14:09:51 +01:00
Hocuri 78fd0c285b Delete messages more thoroughly (and at all) (#2114)
- Make sure delete_expired_messages and housekeeping runs once a day
- delete more info about messages when putting them to trash (esp. also
delete txt_raw, from_id and to_id as we don't need those anymore, so
they are data that was unnecessarily kept)

fix #1926, fix #2090

Also:

* Nicer test_utils: add send_text() and print_chat()

* Adapt ephemeral messages for testing (make them accurate to the second)

* Add test for ephemeral messages

* Make pop_sent_msg() really pop the last sent message
2020-12-26 18:29:43 +01:00
Hocuri d3aba4e817 Simplify if condition 2020-12-20 20:04:19 +01:00
Hocuri 53fed91a17 Fix marking read receipts as seen
If mvbox_move was on and an mdn was received, it was not marked as read.

Also remove a confusing log that said that we are adding a markseen job, even if we weren't. As job::save() logs itself, there is no need to log this again.
2020-12-20 20:04:19 +01:00
Hocuri 03232eb79c Adapt other tests to the fact that now the subject is prepended more often 2020-12-18 18:44:16 +01:00
Hocuri 9edc6702f1 Fix #2078 2020-12-18 18:44:16 +01:00
Hocuri 2b207e1375 rustfmt 2020-12-18 18:35:54 +01:00
Hocuri 4d2c2130e8 Replace all t.ctx with t
Not sure if that's the way to go because, you know, merge conflicts, but
we could do this now.
2020-12-18 18:35:54 +01:00
Hocuri af8a6d7722 Create and use some test helper functions 2020-12-18 18:35:54 +01:00
Hocuri 29991f1caf Add test 2020-12-16 13:41:14 +01:00
Hocuri e982549046 Next try to re-enable fetch-existing-msgs, fixing #2097 2020-12-16 13:41:14 +01:00
Hocuri ec83fae314 Parse name="quote" divs (#2104)
fix #1560 Replies in html-only format are not converted nicely wrt Quoting
2020-12-13 18:02:20 +01:00
Hocuri 7c9624e822 Don't download ranges of messages (i.e. first:last) (#2061) 2020-12-03 10:19:56 +01:00
Hocuri 83df69f43c Add verbose logging for #2065 2020-12-01 09:06:56 +01:00
Hocuri 2a9d06d817 Re-enable Export to the new backup format, add backup progress, add a test for the backup progress (#2023)
* Add progress for backup import/export

* Export to the new backup format

* Add tests for the new imex progress
2020-12-01 09:05:25 +01:00
Hocuriandbjoern 38ed94367c Update src/config.rs
Co-authored-by: bjoern <r10s@b44t.com>
2020-11-18 12:08:28 +01:00
Hocuri 3a7bd8b49d Don't fetch messages that arrived in the meantime if InboxWatch is
disabled and re-enabled

That's another narrow-stitching patch for a scenario where many emails could
be deleted all at once and surprisingly: user disables inbox-watch, enables delete-from-server, after a moth enables inbox-watch again -> currently all emails that arrived in the meantime will be deleted (if emails are not fetched, they won't be deleted)
2020-11-17 14:13:27 +01:00
Hocuri c8d4eee794 Don't fetch from INBOX if it is disabled
Before, if there were more than 20 jobs at once, we unconditionally
fetched from inbox
2020-11-16 17:03:52 +01:00
Hocuri c8242b12fe Add docs for clone_online_account 2020-10-27 08:00:26 +01:00
Hocuri 31bf34890a Always show the cause for which a msg landed in trash 2020-10-25 17:42:26 +01:00
Hocuri 34af492afb Rename fetch_existing to fetch_existing_msgs, add comment (#2042) 2020-10-24 12:10:36 +02:00
Hocuri 41b2dee4ca Use get_contact() instead of create_contact() 2020-10-23 21:10:12 +02:00
Hocuri e5da5c48f1 second review 2020-10-23 17:19:23 +02:00
Hocuri 5b5c6a9c31 Alex' review 2020-10-23 17:19:23 +02:00
Hocuri 4ae1a17cc0 Add test for "Saved-messages do not pop up in original chat in multi-device anymore" 2020-10-23 17:19:23 +02:00
Hocuri e6b7a7e292 saved-messages do not pop up in original chat in multi-device anymore
fix #2020

When forwarding a message, the original `in_reply_to` stays in place.
I did not find any recent commit that changed anything about this,
but IIRC there was a bug that prevented setting the `in_reply_to` at
all.

So, for chat messages, do not look at the InReplyTo header (and also not
at the References header)
2020-10-23 12:33:25 +03:00
Hocuri c438691b73 Disable fetch-existing for now 2020-10-23 08:31:49 +02:00
Hocuri 6c5eaaed2c Don't peek-receipients/fetch-existing if this is a bot 2020-10-19 18:10:54 +02:00
Hocuri dcc00075b0 extract variable 2020-10-19 15:23:18 +02:00
Hocuri a320fb9d6c Doesn't look great, but this was the only way to make compiler happy 2020-10-19 15:23:18 +02:00
Hocuriandholger krekel 3bef4909d5 Update src/message.rs
Co-authored-by: holger krekel  <holger@merlinux.eu>
2020-10-19 15:23:18 +02:00
Hocuri 26aeacc6be @link2xt's review 2020-10-19 15:23:18 +02:00
Hocuri 0b1288fc17 Mark all failed messages as failed when receiving an NDN
There may be multiple messages with the same `Message-Id`-header alias
rfc724_mid because an email with multiple attachments was split up.
2020-10-19 15:23:18 +02:00
Hocuri c005f756d6 Do not allow to save drafts in non-writeable chats, fix #1986 (#1997) 2020-10-19 13:15:48 +02:00
Hocuri 3c6d52842e Doc comments are show in HTML documentation.
This is not a proper documentation, just a note on implementation.
2020-10-19 13:07:55 +02:00
Hocuri 4d2542cee5 Don't show HTML if there is no content and there is a file attached
Fix https://github.com/deltachat/deltachat-core-rust/issues/1982
2020-10-19 13:07:55 +02:00
Hocuri b8e98c0b81 Also add missing suffix (#1973) 2020-10-16 18:42:21 +02:00
Hocuri 07e4762f71 Call update_device_chats automatically during configure (#1957)
update_device_chats() takes about 2 seconds on a modern device (Android) because the
welcome image file has to be written to the disk as a blob. The problem
was that this was done after the progress bar had vanished and before
anything else happened so that I thought that something had gone wrong
multiple times.

The UIs have to remove update_device_chats(), too..
2020-10-15 17:43:22 +02:00
Hocuri bb50b9abe4 Show more errors (#1967) 2020-10-10 18:19:31 +02:00
Hocuri be88b946b6 Peek reipients, fetch existing messages
Read all of an e-mail accounts messages and extract all To/CC addresses
if the From was from our own account.
Then, fetch existing messages from the server and show them.

Also, I fixed two other things:
- just by chance my test failed because of an completely unrelated bug.
The bug: bcc_self messages were not marked as read if mvbox_move was set
to true.
- add some color to the test output (minor change)
2020-10-10 15:56:30 +03:00
Hocuri cf5342c367 Allow drafts without text if there is a quote 2020-10-10 12:41:43 +03:00
Hocuri 990ab739cc Save quote as draft 2020-10-10 12:41:43 +03:00
Hocuri ea731a3619 Code style 2020-10-10 12:41:43 +03:00