To handle backups the UIs have to make sure they do stop the IO
scheduler and also don't accidentally restart it while working on it.
Since they have to call start_io from a bunch of locations this can be
a bit difficult to manage.
This introduces a mechanism for the core to pause IO for some time,
which is used by the imex function. It interacts well with other
calls to dc_start_io() and dc_stop_io() making sure that when resumed
the scheduler will be running or not as the latest calls to them.
This was a little more invasive then hoped due to the scheduler. The
additional abstraction of the scheduler on the context seems a nice
improvement though.
We do not make all transactions
[IMMEDIATE](https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions)
for more parallelism -- at least read transactions can be made DEFERRED to run in parallel
w/o any drawbacks. But if we make write transactions DEFERRED also w/o any external locking,
then they are upgraded from read to write ones on the first write statement. This has some
drawbacks:
- If there are other write transactions, we block the thread and the db connection until
upgraded. Also if some reader comes then, it has to get next, less used connection with a
worse per-connection page cache.
- If a transaction is blocked for more than busy_timeout, it fails with SQLITE_BUSY.
- Configuring busy_timeout is not the best way to manage transaction timeouts, we would
prefer it to be integrated with Rust/tokio asyncs. Moreover, SQLite implements waiting
using sleeps.
- If upon a successful upgrade to a write transaction the db has been modified by another
one, the transaction has to be rolled back and retried. It is an extra work in terms of
CPU/battery.
- Maybe minor, but we lose some fairness in servicing write transactions, i.e. we service
them in the order of the first write statement, not in the order they come.
The only pro of making write transactions DEFERRED w/o the external locking is some
parallelism between them. Also we have an option to make write transactions IMMEDIATE, also
w/o the external locking. But then the most of cons above are still valid. Instead, if we
perform all write transactions under an async mutex, the only cons is losing some
parallelism for write transactions.
* Because both only make problems with mailing lists, it's easiest to just disable them. If we want, we can make them work properly with mailing lists one day and re-enable them, but this needs some further thoughts.
Part of #3701
* Use load_from_db() in more tests
* clippy
* Changelog
* Downgrade warning to info, improve message
* Use lifetimes instead of cloning
* enable `BccSelf` by default
enabling `BccSelf` improves user experience as
it is easier to set up another device
and ppl will also see "all" messages in other user agents directly.
for uncounted user problems, after diving into the issue,
the resulting device was "turn on BccSelf".
disabled `BccSelf` was probably the the number one single reason
of user problems.
main drawback of the change are potentially double notifications
when using a shared account and having another mail app on the same device.
however, we meanwhile do not recommend shared accounts at all,
the issue is also fixable by the other mail apps (as done by K-9)
and could be even regarded as a feature (you can decide which app to use for ansering).
but at the end the drawback is probably much smaller than the issues reported above.
* adapt tests to `BccSelf` enabled
* update CHANGELOG
* add request_internet_access manifest option
* test request_internet_access
* update CHANGELOG
* force warning when internet access is enabled
if internet access is enabled,
show a warning instead of the normal summary
(the internet access is currently mainly to test out integrations
as maps for video chat; the summary is dispensable in the cases currently)
* adapt json-rpc's WebxdcMessageInfo
* save webxdc-updates for not yet downloaded messages, that are probably webxdc instances then
* test webxdc updates received while instance is not yet downloaded
* keep msg_id on downloading messages
keeping msg_id on downloading messages
has the advantage that webxdc updates and other references to the msg_id
can be processed as usual.
if a message expands to multiple msg_id,
the last one is kept,
however, this does not affect webxdc at all.
(alternatives may be to update `msgs_status_updates`
but that seems more complicated and even less elegant,
another alternative would be to use different keys (eg. `rfc274_mid`),
but that also seems not to be much easier and would waste space as well.
also both alternatives would need adaption for other foreign keys)
* update CHANGELOG
* do not emit WebxdcStatusUpdate event in case the message is not yet downloaded
* move DELETE/UPDATE to an transaction
* make merge_msg_id() a little less confusing
* use some webxdc-update-param from placeholder
(the placeholder may be updated,
the just downloaded messages is not)
* more precise function name
* test not directly downloading status updates
* test not directly downloading mdn
* more flexible render_webxdc_status_update_object()
* delay webxdc updates when ratelimit is reached
* inject updates messages to the SMTP loop as needed
this avoids starting several tasks
and allows sending updates out after a restart of the app.
* use mutex to prevent race conditions in status updates
* check ratelimiter only before the sending loop; it won't change until messages are actually sent out
* fix typo
* prefer standard type declaration over turbofish syntax
* use UNIQUE and ON CONFLICT for query smtp_status_updates
* combine DELETE+SELECT to one atomic statement
* as all operations on smtp_status_updates are now atomic, a mutex is no longer needed
* test DELETE+RETURNING statement
* simplify calls to can_send()
* comment about ratelimit boolean in send_smtp_messages()
* clarify webxdc reference wrt info-messages
* add from_id parameter to add_info_msg_with_cmd()
* flag webxdc-info-messages as such
* set from_id to sender for webxdc-info-messages
* test additional webxdc info properties
* do not add series of similar info messages
instead, if on adding the last info message
is already from the same webxdc and sender,
just update the text
* test cleanup of webxdc info messages series
* update changelog
* make clippy happy
there is no real complexity in the args,
so allowing one more arg is probably fine.
if really wanted, we can refactor the function in another pr;
this pr is already complex enough :)
* use cleaner function names and comments
* clarify CHANGELOG
* do not wipe info for drafts
* drafts and received webxdc-instances, resent or not, do not trigger info-messages
* test that resending a webxdc does not not add legacy info-messages
* make clippy happy
* update CHANGELOG
bcc_self-updates are not received
due to the normal prevention of not even downloading these messages.
there is no extra defense of that on webxdc-level;
status-updates do not even have a "global-unique" id
(they have a local id and the message where they're wrappted into have the
"global-unique" Message-Id)
this bypasses the replication safety introduced by letting the caller track the last serial,
however, in case of bots that do not track much state and do not playback updates anyway,
it is still useful.
* add min_api to manifest.toml, define WEBXDC_API_VERSION
* return an error instead of index.html if the webxdc requires a newer api
* add a test with an webxdc requiring a newer api
* update CHANGELOG
This seems not only wasteful but genuinly has the risk someone makes
their device useless by accidentally adding a huge file.
This also re-structures the checks a little: The if-conditions are
flattened out and cheap checks are done before more expensive ones.
* (r10s, adb, hpk) remove getAllUpdates() and add a typical replica-API that works with increasing serials. Streamline docs a bit.
* adapt ffi to new api
* documentation: updates serials may have gaps
* get_webxdc_status_updates() return updates larger than a given serial
* remove status_update_id from status-update-event; it is not needed (ui should update from the last known serial) and easily gets confused with last_serial
* unify wording to 'StatusUpdateSerial'
* remove legacy payload format, all known webxdc should be adapted
* add serial and max_serial to status updates
* avoid races when getting max_serial by avoiding two SQL calls
* update changelog
Co-authored-by: B. Petersen <r10s@b44t.com>
this uses `get_webxdc_info().name` for chatlist etc.
the previuosly used static strings comes from a time
where we just did not had the correct name.
i was also thinking about adding `get_webxdc_info().summary`,
however, as this information is dynamic,
that may open several issues, eg. quoted text may change
so that the answer is out of context.
as discussed in dev chat,
100 kb is too low if one wants to use any game framework.
also, many game that doesn't have poor graphics
or uses magic CSS-tricks to do fancy draws,
will have more than 100 kb probably just in tiles and sprites.
even for regions with low resources,
100 kb is low for a game -
esp. if we compare that with stickers that are send around as well
are not deduplicated and also have several 100 kb in size -
for much less effect.
we should still encourage ppl to create tiny apps,
however, a too low limit also restricts possibilities wrt adaption.
already before,
we did _not_ sent updates to contact requests or other read-only-chats.
however, we _did_ modify the local database,
so that getAllUpdates() returns an update that was not sent out to other peers.
this is fixed by checking can_send() soon.
that way, all peers have the same state
also for contact requests or other read-only-chats
and webxdc in contact requests can be opened as usual.
further ui improvements may be needed for contact requests
(maybe allow the webxdc to know about that state or
maybe show a warning in the ui somewhere),
however, this is not part of this pr.
