* Add DC_EVENT_SELFAVATAR_CHANGED
* Add test.
Unfortunately I can't easily also test that the avatar is not copied
from unencrypted messages:
In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.
* Unfortunately I can't easily also test that the avatar is not copied from unencrypted messages:
In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.
See https://support.delta.chat/t/discussion-how-to-show-error-states/1363/10 <!-- comment -->
It turns out that it's pretty easy to distinguish between lots of states (currently Error/NotConnected, Connecting…, Getting new messages… and Connected). What's not that easy is distinguishing between an actual error and no network, because if the server just doesn't respond, it could mean that we don't have network or that we are trying ipv6, but only ipv4 works.
**WRT debouncing:**
Sending of EVENT_CONNECTIVITY_CHANGED is not debounced, but emitted every time one of the 3 threads (Inbox, Mvbox and Sentbox) has a network error, starts fetching data, or is done fetching data.
This means that it is emitted:
- 9 times when dc_maybe_network() is called or we get network connection
- 12 times when we lose network connection
Some measurements: dc_get_connectivity() takes a little more than 1ms (in my measurements back in March), dc_get_connectivity_html() takes 10-20ms. This means that it's no immmediate problem to call them very often, might increase battery drain though. For the UI it may be a lot of work to update the title everytime; at least Android is smart enough to update the title only once.
Possible problems (we don't have to worry about them now I think):
- Due to the scan_folders feature, if the user has lots of folders, the state could be "Connecting..." for quite a long time, generally DC seemed a little unresponsive to me because it took so long for "Connecting..." to go away. Telegram has a state "Updating..." that sometimes comes after "Connecting...".
To be done in other PRs:
- Better handle the case that the password was changed on the server and authenticating fails, see https://github.com/deltachat/deltachat-core-rust/issues/1923 and https://github.com/deltachat/deltachat-core-rust/issues/1768
- maybe event debouncing (except for "Connected" connectivity events)
fix https://github.com/deltachat/deltachat-android/issues/1760
Using `impl AsRef<str>` as the argument instead of `&str` makes it
possible to call the function with `&str`, `String` and other types
that implement `AsRef` trait.
The cost of it is that compiled binary contains mulitple versions of
the same function, one for each variant of types. If function contains
multiple generic `impl AsRef` arguments, the number of versions possibly
compiled into binary grows exponentially with the number of arguments.
Simple way to avoid it is to call `.as_ref()` on the caller side to
convert the argument to `&str`. In most cases even adding a `&` and
relying on `Deref` coercion is sufficient.
This patch changes many functions that accepted `impl AsRef<str>` and
`impl AsRef<Path>` to accept `&str` and `&Path` instead.
In some places `.clone()` calls are removed. Calling `.clone()` on
`String` and passing `String` to a function accepting `impl
AsRef<str>` is completely unnecessary as `&str` reference could be
passed instead. There is no clippy warning against it yet, but
changing argument type to `&str` allowed to find these cases.
The result of debloating is not impressive, several hundred kilobytes
are saved, which is about 3% of the `.so` binary, but the code is
cleaner too.
This moves the module-level lookup and creation functions to the
types, which make the naming more consistent. Now the lookup_* get_*
and create_* functions all behave similarly.
Peraps even more important the API of the lookup now allows
distinguishing failure from not found. This in turn is important to
be able to remove reliance on a ChatId with a 0 or "unset" value. The
locations where this ChatId(0) is still used is in database queries
which should be solved in an independed commit.
Switches from rusqlite to sqlx to have a fully async based interface
to sqlite.
Co-authored-by: B. Petersen <r10s@b44t.com>
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: link2xt <link2xt@testrun.org>
save subject for messages:
- new api `dc_msg_get_subject()`,
- when quoting, use the subject of the quoted message as the new subject, instead of the
last subject in the chat
fix#2254: if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message), the db was corrupted.
* Add result.log() for logging with less boilerplate code
* Bugfix: Resultify housekeeping() to make it abort if the db is closed instead of just deleting everything
* Require the UI to call dc_stop_io() before backup export
* Prepare a bit better for closed-db: Resultify get_uidvalidity and get_uid_next and let job::load_next() wait until the db is open
About the bug (before this PR):
if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message):
- I don't know if it downloads the message, but of course at some point the process of receiving the message will fail
- In my test, DC is just in the process of moving a message when the imex starts, but then can't delete the job or update the msg server_uid
- Then, when job::load_next() is called, no job can be loaded. That's why it calls `load_housekeeping_job()`. As `load_housekeeping_job()` can't load the time of the last housekeeping, it assumes we never ran housekeeping and returns a new Housekeeping job, which is immediately executed.
- housekeeping can't find any blobs referenced in the db and therefore deletes almost all blobs.
* Copypaste-merge my old work
* Start implementing mailinglists the new way
* Create pseudo contact
* Fine-tune docs
* Remove some unnecessary changes
* style
* Make a stock str
* Fix a crash. Yes, this line caused a panic when reconfiguring on Android
(without a reasonable error log). Also, I could not receive any messages
anymore.
* rfmt
* Add tests and make them pass
* Even more tests
* rfmt
* Enhance test and fix bug
* Don't update the sender name when prefetching because maybe it's a mailing list
* Use an enum instead of numbers for the decision
* Don't remove anyone from mailing lists
* Fix bug in the regex
* Adjust error msg
* Compile error after rebase
* Correctly emit event
* Add dc_msg_is_mailing_list so that you can find out whether messages in the deaddrop belong to mailing lists.
* Add received headers to unit tests
* Comments, small tweaks
* Use dc_msg_get_override_sender_name instead of dc_msg_get_sender_name
* Add dc_msg_get_sender_first_name() because sometimes the first name was not correctly shown in mailing lists
* small fixes, don't let the user modify mailing list groups
* Hide contacts for addresses like noreply@github.com and reply+AEJ...@reply.github.com
When testing mailing lists, I noticed that sometimes a mailing list
contact got a name (like, hocuri <noreply@github.com>). It turned out
that ages ago, I had accidentally written an email to - in this example
- hocuri <noreply@github.com> and it had been added to the contacts
list.
This hides email addresses from the contacts list that are obviously not
meant to be written at and prevents updating the names.
* Comment, clippy
* Replace u32 with ChatId
* Resolve lost of small issues from the reviews
* remove dc_msg_get_sender_first_name
* add dc_msg_get_real_chat_id()
this allows to check if a contact request belongs to a mailing list
and to show name/avatar/whatever of the mailinglist.
another approach was to duplicate some chat-apis for messages
(eg. dc_msg_is_mailing_list()) however that would require far more new apis.
the idea to change the behavior of dc_msg_get_chat_id() would be
more clean, however, that easily breaks existing implementations
and can led to hard to find bugs.
* remove now unused Message.is_mailing_list()
* if a name for a mailing list is missing, use List-ID
* fix comment
* fix error message
* document how dc_get_chat_contacts() works for mailing lists
* refine decide api (#2185)
* add DC_DECIDE* constants to deltachat.h, tweak documentation
* use StartChat/Block/NotNow instead of Yes/Never/NotNow
* decide_on_contact_request works on ctx/msg-id
functions working on message-objects
usually do not read or write directly on the database.
therefore, decide_on_contact_request()
should not work with message-objects as well,
it is even a bit misleading, as eg. chat-id of the object is not modified.
instead, the function works on context,
similar to dc_send_msg(), dc_create_chat() and so on.
for now, i moved it to context, could maybe be part od MsgId.
* Update src/chatlist.rs
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: Hocuri <hocuri@gmx.de>
* refine documentation
* re-add accidentally deleted Param::MailingList
* remove pseudo-contact in domain @mailing.list
1. the pseudo-contact was added to the mailing list contacts,
which is not needed.
might be that we want to add the recent contacts there in a subsequent pr
(we do not know all contacts of a mailing list)
2. the pseudo-contact was used to block
mailing lists; this is done by setting the chat to Blocked::Manually now
3- the pseudo-contact was used for unblocking;
as it is very neat not to require additional ui for mailing list unblocking,
might be that we introduce a similar pseudo-contact for this limited purpose
in a subsequent pr, however, the pseudo-contact needs to exist only
during unblocking then, maybe also the special domain is not needed,
we'll see :)
* Move dc_decide_on_contact_request() up to the dc_context section as it's a member of dc_context now
More specifically, to the "handle messages" section
* re-introduce Chattype::Mailinglist (#2195)
* re-introduce Chattype::Mailinglist
* exhaustive chattype-check in fetch_existing_msgs() and get_summary2()
* exhaustive chattype-check in ndn_maybe_add_info_msg()
* exhaustive chattype-check in message::fill()
* remove dc_chat_is_mailing_list() from ffi
Co-authored-by: B. Petersen <r10s@b44t.com>
- doc fixes
- make BobStateHandle safer by moving the state out of the handle.
- handle more match cases explicit in BobState returns
- fewer mutable variables
This introduces a state machine which takes care of managing the
handshake transitions in the secure-join protocol. This separates
user interactions from the protocol state handling.
This means that while handling the protocol state there are a bunch of
failures no longer possible due to all state information being
guaranteed to be present. As part of this the QR-code state has been
extracted from the generic Lot structure to something suitable just
for the SecureJoin protocol.
A LogSink has been added to the testing tools allowing log messages to
be correctly displayed on test failures.
This also checks that some of the correct user interactions happen,
checking we get a joiner event and the verified chat messages.
It also extends the test utils with the ability to distinguish the
different context logs by having them named.
Lots of new clippy lints due to toolchain upgrade.
Made the Message::error field pub(crate) again, it was the odd one out
and it seemed a reasonable way to shut up clippy.
Start implementing #1994
TODO (in later PRs):
- Add a hint to the watch settings that all folders are fetched from time to time (to be done in the individual UIs)
- folder names are case-insensitive, so double-check that all comparisons are case-insensitive
- The `scan_folders.rs` file didn't get as large as I expected and it's probably not worth it having an extra file for it. But if there are no objections, I'll make another PR to rename it to `folders.rs` and also put into it `configure_folders()` from `imap/mod.rs` and `needs_move()` with all its tests from `message.rs`.
Done:
- Most mailboxes have a "Drafts" folder where constantly new emails appear but we don't actually want to show them, what do we do about this? The most reliable way to detect such messages that we found up to now is:
If there is no `Received` header AND it's not in the `ConfiguredSentbox`, then ignore the email.
- before or after INBOX idle trigger a new "scan all folders for messages". It does a "list folders" and then goes through all folders with select-statements, checking if "next-uid" was changed since checked last time. This might be batchable but in any case should not consume a lot of traffic. We might debounce this scan activity to happen at most every N minutes
- if next-uid changed for a folder, we "prefetch" and "fetch" DC-messages as as needed ("dc-messages" are not just those with "Chat-Version" headers, but can also be regular emails)
- if we discover DC-messages in folders that have the "/Spam" flag (maybe excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
- for now, we do not change any user visible option, but introduce this "scan all" automatically and on top of what exists. The DeltaChat folder-watching does not perform scan-all-folders (maybe with the exception to trigger scan-all also with DeltaChat if INBOX is not watched)
- Tests (except if you have ideas to improve them)
- all folders, their last uidvalidity, next-seen etc. are kept in a separate "imap-sync" sqlite table. Maybe this can be used to streamline some of the "Sent" folder and "DeltaChat" folder detection code we already have.
- We now also move self-sent messages from the Inbox to the Sent folder if `mvbox_move` is off, as this was very easy to do now. This way, we now behave more like a normal MUA if the user wants this.
FOR LATER PRs:
- maybe for the first 50 messages or so, we could reduce the IDLE-timeout (currently 23 minutes or so) to faster detect messages sent to non-inbox folders. However, on Android and iOS, we would likely trigger scan-all when the app moves to foreground, and so it might not be neccessary to reduce the current idle-timeout at least for them. We can leave this "faster discovery" question for the end, after we move to real-life testing.
- (Later on, after the above works, we can consider heuristics on which folders to perform IDLE on, and remove the Watch-folder options (inbox, deltachat, sent). We tried to find a safe scheme for already doing it but failed to fine one, too many unknowns, also some questions regarding multi-device (you might have different settings with each of it, one moves, the other doesn't etc.) so we postponed this in favor of the above incremental suggestion.)
* Start implementing #1994
* Add debug logs, it seems like the SQL migration can go into another pr
* Let fetch_new_messages return whether there are new emails
* Code style
* Don't prefetch if there are no new emails
* clippy
* Even more debug logs
* If the folder was not newly selected, return always try to fetch as
uid_next is probably outdated
* Fix new bug
* Recognize spam folder
* if we discover DC-messages in folders that have the "/Spam" flag (excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
* Clippy, prioritize folder_meaning over folder_name_meaning
* Add a first test, for the first day after installation only debounce to 2s
* Start adding two tests (both of them fail)
* Don't abort folder scan if one folder fails
* More consts
* Replace bool return value by enum
* Split test up into multiple tests
* Print logs during rust tests
* Rust tests pass now
* .
* One of the Python tests passes now - reconfigure folders during scanning
* Make the last test pass - Delete emails in all folders when starting the test, not only inbox and mvbox
The problem had been that emails were left in the folder "xyz"
* lint
* DB migration (untested)
* Store uid_next in SQL instead of lastseen in a config
* Revert "If Inbox-watch is disabled and enabled again, do not fetch emails from in between"
all folders are always watched, anyway
* clippy, rm debug logs, comments
* Codestyle, comments
* fixing things again
* Fix another test: don't fetch from uid_next-1 but uid_next; make some {} to {:#} so that we can use `.context(...)`
* move self-sent, non-setupmessage chat messages to the Sent folder if `mvbox_move` is off
* comment
* Comments, make sure things work even if there is no uid_next
* Style
* Comments
* The rust test tested wrongly
* comments, small codestyle change
* Ignore emails that are probably only drafts
Most mailboxes have a "Drafts" folder where constantly new emails appear
but we don't actually want to show them.
So: If there is no Received header AND it's not in the ConfiguredSentbox,
then ignore the email.
Also: Add test.
* Fix occasional test failure, it was introduced as DC now moves messages from Inbox to Sent
* Add `Received` header to the rust tests
* After this PR we will always watch all folders and delete messages there if server_delete is enabled. So, for people who have server_delete on, disable it and add a hint to the devicechat
* comment, small fix
* link2xt's first review
* Use ON CONFLICT(FOLDER) DO to update and if it doesn't exist, then insert
Reason from link2xt: We had a problem with multiple peerstates inserted due to key fingerprint parsing error previously. With logic in Rust a similar problem can occur: an UPDATE can fail for reasons other than a conflict. PRIMARY KEY should ensure uniqueness in this case, but anyway.
* Remove two TODO statements, remove fetch_new_messages: ignoring uid {}, uid_next was {} log
* Next TODO: Make uidvalidity and uid_next DEFAULT 0
* rm two TODOs, Seems like we are not going to `exclude folders that are watched anyway` in this PR
* small tweak: Handle instants more carefully
* Add scan_all_folders_debounce_secs config for tests, set debounce to 60s (before it was just 2s during the first day)
* Don't use bold letters for the device message
* React to changes in the folders better
Before, if there was a configured Sent folder, but then it got
removed and replaced with another folder with a name meaning "Sent" but
without Sent flag, it would be ignored.
So, instead of checking against ConfiguredSentboxFolder,
create two Option variables at the beginning of the loop and replace
them with Some if it is None. At the end of the loop, store the new
values into ConfiguredSendboxFolder and ConfiguredSpamFolder, even if it
is None.
Also, derive some useful traits.
* move job: Return a meaningful error if server_folder is None instead of panicing
* small error-handling fix
* Fix test_fetch_existing() python test
Before, we sometimes got a race condition where scan_folders() sees that
there is a Sentbox and saves this info after we set the
ConfiguredSentbox to None and before the message is sent.
So, just expect that the message is moved to the sentbox.
* migration is 72 now
* rm 2 TODOs, Don't infinitely retry when dc_receive_imf() returns Err
* clippy: Remove glob imports
* Delete created folders at the beginning of tests
(some created folders made problems in the next tests because)
* Improve resetting accounts between tests
- Make sure delete_expired_messages and housekeeping runs once a day
- delete more info about messages when putting them to trash (esp. also
delete txt_raw, from_id and to_id as we don't need those anymore, so
they are data that was unnecessarily kept)
fix#1926, fix#2090
Also:
* Nicer test_utils: add send_text() and print_chat()
* Adapt ephemeral messages for testing (make them accurate to the second)
* Add test for ephemeral messages
* Make pop_sent_msg() really pop the last sent message
This currently tests the setup-contact full flow and the shortcut
flow. More securejoin tests are needed but this puts some
infrastructure in place to start writing these.
This tidies up our testing tools a little bit. We had several
functions which through various changes ended up doing the same and
some more which did very similar stuff, so I merged them to have
things simpler. Also moved towards methods on the TestContext struct
while cleaning this up anyway, seems like this structure is going to
stay around for a bit anyway.
The intersting change is in `test_utils.rs`, everything else is just
updating callers. A few tests used example.org which I moved to
example.com to be able to re-use more configuration of the test
context.
The user-visible change here is that it allows the FFI API to save
keys in the database for a context. This is primarily intended for
testing purposes as it allows you to get a key without having to
generate it.
Internally the most important change is to start using the
SignedPublicKey and SignedPrivateKey types from rpgp instead of
wrapping them into a single Key object. This allows APIs to be
specific about which they want instead of having to do runtime checks
like .is_public() or so. This means some of the functionality of the
Key impl now needs to be a trait.
A thid API change is to introduce the KeyPair struct, which binds
together the email address, public and private key for a keypair.
All these changes result in a bunch of cleanups, though more more
should be done to completely replace the Key type with the
SignedPublicKye/SignedPrivateKey + traits. But this change is large
enough already.
Testing-wise this adds two new keys which can be loaded from disk and
and avoids a few more key-generating tests. The encrypt/decrypt tests
are moved from the stress tests into the pgp tests and split up.
Since stock string callback has been deprecated, all event callbacks
return 0.
For compatibility, C declarations are not changed and FFI users are
expected to return 0 from their callbacks.
For consistency with other tests that use example public and private
keys, replace use of `concat!` macro with `include_str!`. This way, key
data embedded into source with single line of code.
A new context is now created by calling Context::new and therefore you
always have a valid context. This is much more in Rust style and will
allow a lot of furture simplifications on the context itself.
The FFI layer has not yet been adjusted in this commit and thus will
fail.
This marks the function safe and returns Result, it also now returns
the ConfiguredAddr since it has to look this up anyway and it makes
testing more easy. Turns out it reduces some duplicate SQL query in
some callers too.
More test code has been moved from dc_imex to test_utils as it's
more genrally applicable.
