PR #5099 removed some columns in the database that were actually in use.
usually, to not worsen UX unnecessarily
(releases take time - in between, "Add Second Device", "Backup" etc.
would fail), we try to avoid such schema changes (checking for
db-version would avoid import etc. but would still worse UX),
see discussion at #2294.
these are the errors, the user will be confronted with otherwise:
<img width=400
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/e3f0fd6e-a7a9-43f6-9023-0ae003985425>
it is not great to maintain the old columns, but well :)
as no official releases with newer cores are rolled out yet, i think, it
is fine to change the "107" migration
and not copy things a second time in a newer migration.
(this issue happens to me during testing, and is probably also the issue
reported by @lk108 for ubuntu-touch)
Restart the IO scheduler if needed to make the new config value effective (for `MvboxMove,
OnlyFetchMvbox, SentboxWatch` currently). Also add `set_config_internal()` which doesn't affect
running the IO scheduler. The reason is that `Scheduler::start()` itself calls `set_config()`,
although not for the mentioned keys, but still, and also Rust complains about recursive async calls.
a27e84ad89 "fix: Delete received outgoing messages from SMTP queue"
can break sending messages sent as several SMTP messages because they have a lot of recipients:
`pub(crate) const DEFAULT_MAX_SMTP_RCPT_TO: usize = 50;`
We should not cancel sending if it is such a message and we received BCC-self because it does not
mean the other part was sent successfully. For this, split such messages into separate jobs in the
`smtp` table so that only a job containing BCC-self is canceled from `receive_imf_inner()`. Although
this doesn't solve the initial problem with timed-out SMTP requests for such messages completely,
this enables fine-grained SMTP retries so we don't need to resend all SMTP messages if only some of
them failed to be sent.
When a key is gossiped for the contact in a verified chat,
it is stored in the secondary verified key slot.
The messages are then encrypted to the secondary verified key
if they are also encrypted to the contact introducing this secondary key.
Chat-Group-Member-Added no longer updates the verified key.
Verified group recovery only relies on the secondary verified key.
When a message is received from a contact
signed with a secondary verified key,
secondary verified key replaces the primary verified key.
When verified key is changed for the contact
in response to receiving a message
signed with a secondary verified key,
"Setup changed" message is added
to the same chat where the message is received.
Moved custom ToSql trait including Send + Sync from lib.rs to sql.rs.
Replaced most params! and paramsv! macro usage with tuples.
Replaced paramsv! and params_iterv! with params_slice!,
because there is no need to construct a vector.
We are currently using libsqlite3-sys 0.25.2,
corresponding to SQLcipher 4.5.2
and SQLite 3.39.2.
SQLite supports ALTER TABLE DROP COLUMN since version 3.35.0,
and it has received critical database corruption bugfixes in 3.35.5.
There have been no fixes to it between SQLite 3.36.0 and 3.41.0,
so it appears stable now.
When connection pool is organized as a stack,
it always returns most recently used connection.
Because each connection has its own page cache,
using the connection with fresh cache improves performance.
I commented out `oauth2::tests::test_oauth_from_mx`
because it requires network connection,
turned off the Wi-Fi and ran the tests.
Before the change, with a queue:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 56.424 s ± 4.515 s [User: 183.181 s, System: 128.156 s]
Range (min … max): 52.123 s … 68.193 s 10 runs
```
With a stack:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 29.887 s ± 1.377 s [User: 101.226 s, System: 45.573 s]
Range (min … max): 26.591 s … 31.010 s 10 runs
```
On version 1.107.1:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 43.658 s ± 1.079 s [User: 202.582 s, System: 50.723 s]
Range (min … max): 41.531 s … 45.170 s 10 runs
```
In execute_migration transaction first update the version, and only
then execute the rest of the migration. This ensures that transaction
is immediately recognized as write transaction and there is no need
to promote it from read transaction to write transaction later, e.g.
in the case of "DROP TABLE IF EXISTS" that is a read only operation if
the table does not exist. Promoting a read transaction to write
transaction may result in an error if database is locked.
* do not `SELECT *` on old tables to fill new ones
the old table may contain deprecrated columns for whatever reason;
as a result the query fails as the statement tries to insert eg.
16 columns into 12 colums
(concrete error for acpeerstate that have several deprecated columns)
* update CHANGELOG
This esp. speeds up receive_imf a bit when we recreate the member list (recreate_member_list == true).
It's a preparation for https://github.com/deltachat/deltachat-core-rust/issues/3768, which will be a one-line-fix, but recreate the member list more often, so that we want to optimize this case a bit.
It also adds a benchmark for this case. It's not that easy to make the benchmark non-flaky, but by closing all other programs and locking the CPU to 1.5GHz it worked. It is consistently a few percent faster than ./without-optim:
```
Receive messages/Receive 100 Chat-Group-Member-{Added|Removed} messages
time: [52.257 ms 52.569 ms 52.941 ms]
change: [-3.5301% -2.6181% -1.6697%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
```
Fix#3507
Note that this is not intended for a release at this point! We first have to test whether it runs stable enough. If we want to make a release while we are not confident enough in authres-checking, then we have to disable it.
BTW, most of the 3000 new lines are in `test_data/messages/dkimchecks...`, not the actual code
da3a4b94 adds the results to the Message info. It currently does this by adding them to `hop_info`. Maybe we should rename `hop_info` to `extra_info` or something; this has the disadvantage that we can't rename the sql column name though.
Follow-ups for this could be:
- In `update_authservid_candidates()`: Implement the rest of the algorithm @hpk42 and me thought about. What's missing is remembering how sure we are that these are the right authserv-ids. Esp., when receiving a message sent from another account at the same domain, we can be quite sure that the authserv-ids in there are the ones of our email server. This will make authres-checking work with buzon.uy, disroot.org, yandex.ru, mailo.com, and riseup.net.
- Think about how we present this to the user - e.g. currently the only change is that we don't accept key changes, which will mean that the small lock on the message is not shown.
- And it will mean that we can fully enable AEAP, after revisiting the security implications of this, and assuming everyone (esp. @link2xt who pointed out the problems in the first place) feels comfortable with it.
Very small PR; Motivation: Easier navigation using Go-To-definition.
Because, using go-to-definition of rust-analyzer on parse() doesn't take you to the actual parse() implementation but its trait definiton. On the other hand, it's very easy to find EmailAddress::new().
* more flexible render_webxdc_status_update_object()
* delay webxdc updates when ratelimit is reached
* inject updates messages to the SMTP loop as needed
this avoids starting several tasks
and allows sending updates out after a restart of the app.
* use mutex to prevent race conditions in status updates
* check ratelimiter only before the sending loop; it won't change until messages are actually sent out
* fix typo
* prefer standard type declaration over turbofish syntax
* use UNIQUE and ON CONFLICT for query smtp_status_updates
* combine DELETE+SELECT to one atomic statement
* as all operations on smtp_status_updates are now atomic, a mutex is no longer needed
* test DELETE+RETURNING statement
* simplify calls to can_send()
* comment about ratelimit boolean in send_smtp_messages()
Unlike jobs which are executed before sending normal messages, MDNs
from `smtp_mdns` table are sent after sending messages from `smtp`
table. This way normal messages have higher priority than MDNs.
There are no SMTP jobs anymore. All jobs are IMAP jobs, so
`jobs.thread` column is not used anymore.
MarkseenMsgOnImap job, that was responsible for marking messages as
seen on IMAP and sending MDNs, has been removed.
Messages waiting to be marked as seen are now stored in a
single-column imap_markseen table consisting of foreign keys pointing
to corresponding imap table records.
Messages are marked as seen in batches in the inbox loop. UIDs are
grouped by folders to reduce the number of requests, including folder
selection requests. UID grouping logic has been factored out of
move_delete_messages into UidGrouper iterator to avoid code duplication.
Messages are marked as seen right before fetching from the inbox
folder. This ensures that even if new messages arrive into inbox while
the connection has another folder selected to mark messages there, all
messages are fetched before going IDLE. Ideally marking messages as
seen should be done after fetching and moving, as it is a low-priority
task, but this requires skipping IDLE if UIDNEXT has advanced since
previous time inbox has been selected. This is outside of the scope of
this change.
MDNs are now queued independently of marking the messages as seen.
SendMdn job is created directly rather than after marking the message
as seen on IMAP. Previously sending MDNs was done in MarkseenMsgOnImap
avoid duplicate MDN sending by setting $MDNSent flag together with
\Seen flag and skipping MDN sending if the flag is already set. This
is not the case anymore as $MDNSent flag support has been removed in
9c077c98cd and duplicate MDN sending in
multi-device case is avoided by synchronizing Seen status since
833e5f46cc as long as the server
supports CONDSTORE extension.
