This esp. speeds up receive_imf a bit when we recreate the member list (recreate_member_list == true).
It's a preparation for https://github.com/deltachat/deltachat-core-rust/issues/3768, which will be a one-line-fix, but recreate the member list more often, so that we want to optimize this case a bit.
It also adds a benchmark for this case. It's not that easy to make the benchmark non-flaky, but by closing all other programs and locking the CPU to 1.5GHz it worked. It is consistently a few percent faster than ./without-optim:
```
Receive messages/Receive 100 Chat-Group-Member-{Added|Removed} messages
time: [52.257 ms 52.569 ms 52.941 ms]
change: [-3.5301% -2.6181% -1.6697%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
```
MarkseenMsgOnImap job, that was responsible for marking messages as
seen on IMAP and sending MDNs, has been removed.
Messages waiting to be marked as seen are now stored in a
single-column imap_markseen table consisting of foreign keys pointing
to corresponding imap table records.
Messages are marked as seen in batches in the inbox loop. UIDs are
grouped by folders to reduce the number of requests, including folder
selection requests. UID grouping logic has been factored out of
move_delete_messages into UidGrouper iterator to avoid code duplication.
Messages are marked as seen right before fetching from the inbox
folder. This ensures that even if new messages arrive into inbox while
the connection has another folder selected to mark messages there, all
messages are fetched before going IDLE. Ideally marking messages as
seen should be done after fetching and moving, as it is a low-priority
task, but this requires skipping IDLE if UIDNEXT has advanced since
previous time inbox has been selected. This is outside of the scope of
this change.
MDNs are now queued independently of marking the messages as seen.
SendMdn job is created directly rather than after marking the message
as seen on IMAP. Previously sending MDNs was done in MarkseenMsgOnImap
avoid duplicate MDN sending by setting $MDNSent flag together with
\Seen flag and skipping MDN sending if the flag is already set. This
is not the case anymore as $MDNSent flag support has been removed in
9c077c98cd and duplicate MDN sending in
multi-device case is avoided by synchronizing Seen status since
833e5f46cc as long as the server
supports CONDSTORE extension.
* Do ephemeral deletion in background loop
1. in start_io start ephemeral async task, in stop_io cancel ephemeral async task
2. start ephemeral async task which loops like this:
- wait until next time a message deletion is needed or an interrupt occurs (see 3.)
- perform delete_expired_messages including sending MSGS_CHANGED events
3. on new messages (incoming or outgoing) with ephemeral timer:
- interrupt ephemeral async task
* Changelog
* Fix and improve test
* no return value needed
* address @link2xt review comments
* slight normalization: have only one place where we wait for interrupt_receiver
* simplify sql statement -- and don't exit the ephemeral_task if there is an sql problem but rather wait
* Remove now-unused `ephemeral_task` JoinHandle
The JoinHandle is now inside the Scheduler.
* fix clippy
* Revert accidental move of the line
* Add log
Co-authored-by: holger krekel <holger@merlinux.eu>
Co-authored-by: link2xt <link2xt@testrun.org>
* add simple backup export/import test
this test fails on current master
until the context is recrated.
* avoid config_cache races
adds needed SQL-statements to config_cache locking.
otherwise, another thread may alter the database
eg. between SELECT and the config_cache update -
resulting in the wrong value being written to config_cache.
* also update config_cache on initializing tables
VERSION_CFG is also set later, however,
not doing it here will result in bugs when we change DBVERSION at some point.
as this alters only VERSION_CFG and that is executed sequentially anyway,
race conditions between SQL and config_cache
seems not to be an issue in this case.
* clear config_cache after backup import
import replaces the whole database,
so config_cache needs to be invalidated as well.
we do that before import,
so in case a backup is imported only partly,
the cache does not add additional problems.
* update CHANGELOG
This brings the Display of ContactId in line with those of ChatId etc,
which is a bit clearer is logs and such places.
It also updates an SQL query to rely on the ToSql impl of ContactId
rather than it's Display when building the query.
As of #2980, the migrations were not run after importing a backup. This lead to errors like "Failed to send message: no such table: smtp" if you imported a backup from a previous DC version.
To create encrypted account with account manager, call
dc_accounts_add_closed_account(). Open this account with
dc_context_open() using the passphrase you want to use for encryption.
When application is loaded next time and account manager is created,
it will open all accounts that have no passphrase set. For encrypted
accounts dc_context_is_open() will return 0. To open them, call
dc_context_open() with the correct passphrase. After opening, call
dc_context_start_io() on this account or just dc_accounts_start_io()
to start all accounts that are not started yet.
Support for legacy SQLite-based backup format is removed in this
commit.
- Replace .ok_or_else() and .map_err() with anyhow::Context where possible.
- Use .context() to check Option for None when it's an error
- Resultify Chatlist.get_chat_id()
- Add useful .context() to some errors
- IMAP error handling cleanup
It slows down dc_get_chat_msgs() from ~50ms to seconds on Android. It
is disabled by default in SQLCipher 4.5.0, but currently SQLCipher 4.4.3
is bundled so this PRAGMA has to be disabled manually.
Previously default of `auto_vacuum=NONE` was used.
`PRAGMA auto_vacuum=INCERMENTAL` allows running `PRAGMA
incremental_vacuum`. Unlike `VACUUM`, `PRAGMA incremental_vacuum`
frees unused pages without rebuilding the whole database, like the
`VACUUM` does, so it does not require additional disk space to build a
vacuumed copy of the database. This may even free enough space on the
disk to run `VACUUM` afterwards.
New setting will only be enabled for new databases or if the `VACUUM`
command runs successfully. Currently `VACUUM` is only executed on
backup export, but may fail nevertheless if there is not enough space
on the disk.
Also try to run `PRAGMA incremental_vacuum` during housekeeping. It
may not be the best strategy, but likely does not make any difference
under normal usage when the database only grows and there are no free
pages. Free pages are created only if enough data is deleted to free
at least one database page of 4096 bytes, for example when automatic
deletion of messages is deleted for the first time. In the future if
more data is placed into the database, like avatars and other blobs,
it may be necessary to revise this strategy, for example to keep some
free pages instead of removing all of them each time by querying
`PRAGMA freelist_pages` and running `PRAGMA incremental_vacuum(N)`.
I considered removing it from the context by default, but the
migration test really wants to have the tracker initialised from the
very first event and not after the context is initialised. It is
easier for now to leave it hardcoded instead of adding an API to
explicitly require enabling it via the builder.
This replaces the EventSink callbacks with simple channel senders.
This simplifies the TestContext a lot as that is much simpler to
handle. It then also removes the special-casing of the LogSink since
it now is another even sender, only injected at the very start.
`imap` table maps Message-IDs to UIDs on the server. `dc_receive_imf`
no longer gets the UID of the message as an argument and does not
insert the folder and UID of the message into the `msgs`
table. `server_folder` and `server_uid` columns in `msgs` table are
deprecated.
MoveMsg and DeleteMsgOnImap jobs are removed. Now messages are moved
and deleted only in the `fetch_move_delete` procedure that consults
the `target` column of the `imap` table to determine where the message
should go.
Where the message should go is determined after prefetching by the
`imap::target_folder()` procedure. Messages are only downloaded once
they reach their target folder to avoid race conditions in multidevice
setting, such as:
1. One device trying to FETCH the message while the other tries to
MOVE it.
2. One device marking the message as \Seen in the Inbox while the
other has already copied unseen message to the Movebox and is going to
delete the \Seen message in the Inbox.
3. Device downloads the message from the Inbox while there are newer
messages in the Movebox placed there by the other device, thus
processing the messages out of order.
* resultify update_recent_quota()
* add a device-message if quota exceeds QUOTA_WARN_THRESHOLD_PERCENTAGE
* check if a quota warning should be added during housekeeping, this is at least once a day
* dc_get_config("quota_exceeding") is useful for bots
* make clippy happy
* reword QuotaExceedingMsgBody message
* avoid lots of warnings if quota jitters around the warning threshold
* allow clippy::assertions_on_constants
these constants depend on each other, it makes sense to check that they are not changed in an incompatible way
Using `impl AsRef<str>` as the argument instead of `&str` makes it
possible to call the function with `&str`, `String` and other types
that implement `AsRef` trait.
The cost of it is that compiled binary contains mulitple versions of
the same function, one for each variant of types. If function contains
multiple generic `impl AsRef` arguments, the number of versions possibly
compiled into binary grows exponentially with the number of arguments.
Simple way to avoid it is to call `.as_ref()` on the caller side to
convert the argument to `&str`. In most cases even adding a `&` and
relying on `Deref` coercion is sufficient.
This patch changes many functions that accepted `impl AsRef<str>` and
`impl AsRef<Path>` to accept `&str` and `&Path` instead.
In some places `.clone()` calls are removed. Calling `.clone()` on
`String` and passing `String` to a function accepting `impl
AsRef<str>` is completely unnecessary as `&str` reference could be
passed instead. There is no clippy warning against it yet, but
changing argument type to `&str` allowed to find these cases.
The result of debloating is not impressive, several hundred kilobytes
are saved, which is about 3% of the `.so` binary, but the code is
cleaner too.
* Remove sql::error submodule
Use anyhow errors instead.
* Remove explicit checks for open SQL connection
An error will be thrown anyway during attempt to execute query.
* Don't use `with_conn()` and remove it
* Remove unused `with_conn_async`
* Resultify markseen_msgs
- Currently, group images are compressed as well because it was easier to implement that way.
- Currently, in the unlikely case that the avatar is compressed down to 20x20 pixels but still bigger than 20KB, the user doesn't get any indication of this, the avatar simply isn't changed (at least on Android).
If we want to change this, the easiest way is probably to let `dc_set_config()` in the ffi call `error!()` if `Selfavatar` can't be set. The same might make sense for some or all other configs. BUUUUUT: At least Android doesn't show error!() toasts anymore, probably because they were used too often and too spammy.
- The factor by which we scale down if the file is too big is 1.5.
