We do not make all transactions
[IMMEDIATE](https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions)
for more parallelism -- at least read transactions can be made DEFERRED to run in parallel
w/o any drawbacks. But if we make write transactions DEFERRED also w/o any external locking,
then they are upgraded from read to write ones on the first write statement. This has some
drawbacks:
- If there are other write transactions, we block the thread and the db connection until
upgraded. Also if some reader comes then, it has to get next, less used connection with a
worse per-connection page cache.
- If a transaction is blocked for more than busy_timeout, it fails with SQLITE_BUSY.
- Configuring busy_timeout is not the best way to manage transaction timeouts, we would
prefer it to be integrated with Rust/tokio asyncs. Moreover, SQLite implements waiting
using sleeps.
- If upon a successful upgrade to a write transaction the db has been modified by another
one, the transaction has to be rolled back and retried. It is an extra work in terms of
CPU/battery.
- Maybe minor, but we lose some fairness in servicing write transactions, i.e. we service
them in the order of the first write statement, not in the order they come.
The only pro of making write transactions DEFERRED w/o the external locking is some
parallelism between them. Also we have an option to make write transactions IMMEDIATE, also
w/o the external locking. But then the most of cons above are still valid. Instead, if we
perform all write transactions under an async mutex, the only cons is losing some
parallelism for write transactions.
.call() interface is safer because it ensures
that blocking operations on SQL connection
are called within tokio::task::block_in_place().
Previously some code called blocking operations
in async context, e.g. add_parts() in receive_imf module.
The underlying implementation of .call()
can later be replaced with an implementation
that does not require block_in_place(),
e.g. a worker pool,
without changing the code using the .call() interface.
Gmail archives messages marked as `\Deleted` by default if those messages aren't in the Trash. But
if move them to the Trash instead, they will be auto-deleted in 30 days.
- Return Result from set_verified() so that it can't be missed.
- Pass Fingerprint to set_verified() by value to avoid cloning it inside. This optimises out an
extra clone() if we already have a value that can be moved at the caller side. However, this may
add an extra clone() if set_verified() fails, but let's not optimise the fail scenario.
This allows to distinguish exceptions,
such as database errors, from invalid user input.
For example, if the From: field of the message
does not look like an email address, the mail
should be ignored. But if there is a database
failure while writing a new contact for the address,
this error should be bubbled up.
Note that if the message is encrypted, we don't check whether it's signed with an attached key
currently, otherwise a massive refactoring of the code is needed because for encrypted messages a
signature is checked and discarded first now.
This esp. speeds up receive_imf a bit when we recreate the member list (recreate_member_list == true).
It's a preparation for https://github.com/deltachat/deltachat-core-rust/issues/3768, which will be a one-line-fix, but recreate the member list more often, so that we want to optimize this case a bit.
It also adds a benchmark for this case. It's not that easy to make the benchmark non-flaky, but by closing all other programs and locking the CPU to 1.5GHz it worked. It is consistently a few percent faster than ./without-optim:
```
Receive messages/Receive 100 Chat-Group-Member-{Added|Removed} messages
time: [52.257 ms 52.569 ms 52.941 ms]
change: [-3.5301% -2.6181% -1.6697%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
```
It's a w/a for "Space added before long group names after MIME serialization/deserialization"
issue. DC itself never creates group names with leading/trailing whitespace, so it can be safely
removed. On the sender side there's no trim() because group names anyway go through
improve_single_line_input(). And I believe we should send the exact name we have in our db. Also
there's no check for leading/trailing whitespace because there may be existing user databases with
group names having such whitespaces.
Since switch to async we don't have spurious "database is busy"
errors anymore. Since an error is irrecoverable in most cases,
we can skip the message. The cost of this is we may
accidentally skip a correct message if I/O fails, but
the advantage is that we are guaranteed to never confuse
irrecoverable error with recoverable one and get stuck in
infinite loop redownloading the same message over and over.
That's a bug which @Simon-Laux and probably also @hpk42 had, where one malformed incoming (Spam-) mail blocked the receiving of all emails coming after it.
The problem was that from_field_to_contact_id() returned ContactId::UNDEFINED, and then lookup_by_contact() returned Err.
* Treat multiple From addresses as if there was no From: addr
* changelog
* Don't send invalid emails through the whole receive_imf pipeline
Instead, directly create a trash entry for them.
* Don't create trash entries for randomly generated Message-Id's
* clippy
* fix typo
Co-authored-by: link2xt <link2xt@testrun.org>
Seems like consume_events() didn't work properly, i.e. in some cases it
didn't see the latest events and failed to consume them. So, the
IncomingMsg event from receiving DC_MAILINGLIST stayed in the events
channel, which made this fail:
```rust
// Check that no notification is displayed for blocked mailing list message.
while let Ok(event) = t.evtracker.try_recv() {
assert!(!matches!(event.typ, EventType::IncomingMsg { .. }));
}
```
Fix it by explicitly waiting for the first IncomingMsg event.
This way no temporary rows are created and it is easier to maintain
because UPDATE statement is right below the INSERT statement,
unlike `merge_messages` function which is easy to forget about.
