We do not make all transactions
[IMMEDIATE](https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions)
for more parallelism -- at least read transactions can be made DEFERRED to run in parallel
w/o any drawbacks. But if we make write transactions DEFERRED also w/o any external locking,
then they are upgraded from read to write ones on the first write statement. This has some
drawbacks:
- If there are other write transactions, we block the thread and the db connection until
upgraded. Also if some reader comes then, it has to get next, less used connection with a
worse per-connection page cache.
- If a transaction is blocked for more than busy_timeout, it fails with SQLITE_BUSY.
- Configuring busy_timeout is not the best way to manage transaction timeouts, we would
prefer it to be integrated with Rust/tokio asyncs. Moreover, SQLite implements waiting
using sleeps.
- If upon a successful upgrade to a write transaction the db has been modified by another
one, the transaction has to be rolled back and retried. It is an extra work in terms of
CPU/battery.
- Maybe minor, but we lose some fairness in servicing write transactions, i.e. we service
them in the order of the first write statement, not in the order they come.
The only pro of making write transactions DEFERRED w/o the external locking is some
parallelism between them. Also we have an option to make write transactions IMMEDIATE, also
w/o the external locking. But then the most of cons above are still valid. Instead, if we
perform all write transactions under an async mutex, the only cons is losing some
parallelism for write transactions.
scripts/update-provider-database.sh checks out the provider database
and updates data.rs file,
making it easier to update.
CI uses this script to check that checked in data.rs
corresponds to the provider database repository.
the slowest test was `test_modify_chat_disordered`;
due to several sleep(), it lasts at least 11 seconds.
however, many of the sleep() are not needed and were added
just to get things done that time.
this pr removes 7 superfluous sleeps,
making the test finish in about 3 seconds,
so that this test is no longer the bottleneck when running
`cargo test` on fast machines.
(this is not only useful to safe some seconds -
but also to notice degradations as of
https://github.com/deltachat/deltachat-core-rust/issues/4051#issuecomment-1436995166 )
Although it does a little for security, it will help to protect from unwanted server-side
modifications and bugs. And now we have a time to test "multipart/signed" messages compatibility
with other MUAs.
This makes DC compatible with "multipart/signed" messages thus allowing switching to them someday
from the current "multipart/mixed" unencrypted message format.
try_decrypt() is a CPU-bound task.
When called from async function,
it should be wrapped in tokio::task::spawn_blocking().
Using tokio::task::spawn_blocking() is difficult here
because of &mail, &private_keyring and &public_keyring borrows,
so we should at least use tokio::task::block_in_place()
to avoid blocking the executor.
.call() interface is safer because it ensures
that blocking operations on SQL connection
are called within tokio::task::block_in_place().
Previously some code called blocking operations
in async context, e.g. add_parts() in receive_imf module.
The underlying implementation of .call()
can later be replaced with an implementation
that does not require block_in_place(),
e.g. a worker pool,
without changing the code using the .call() interface.
IIRC, this was written this way back when we didn't have config caching,
in order to save database accesses by getting the config outside the
for loop.
Gmail archives messages marked as `\Deleted` by default if those messages aren't in the Trash. But
if move them to the Trash instead, they will be auto-deleted in 30 days.
With the introduction of transactions in Contact::add_or_lookup(),
python tests sometimes fail to create contacts with the folowing error:
Cannot create contact: add_or_lookup: database is locked: Error code 5: The database file is locked
`PRAGMA busy_timeout=60000` does not affect
this case as the error is returned before 60 seconds pass.
DEFERRED transactions with write operations need
to be retried from scratch
if they are rolled back
due to a write operation on another connection.
Using IMMEDIATE transactions for writing
is an attempt to fix this problem
without a retry loop.
If we later need DEFERRED transactions,
e.g. for reading a snapshot without locking the database,
we may introduce another function for this.
When connection pool is organized as a stack,
it always returns most recently used connection.
Because each connection has its own page cache,
using the connection with fresh cache improves performance.
I commented out `oauth2::tests::test_oauth_from_mx`
because it requires network connection,
turned off the Wi-Fi and ran the tests.
Before the change, with a queue:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 56.424 s ± 4.515 s [User: 183.181 s, System: 128.156 s]
Range (min … max): 52.123 s … 68.193 s 10 runs
```
With a stack:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 29.887 s ± 1.377 s [User: 101.226 s, System: 45.573 s]
Range (min … max): 26.591 s … 31.010 s 10 runs
```
On version 1.107.1:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 43.658 s ± 1.079 s [User: 202.582 s, System: 50.723 s]
Range (min … max): 41.531 s … 45.170 s 10 runs
```
This also removes BackupProvider::join in favour of implementing
Future directly. I wondered about implementing a FusedFutre to make
this a little safer but it would introduce a dependency on the futures
crate in deltachat-ffi which did not exist yet, so I didn't do that.
