With `ORDER BY` statement SQLite searches
the `imap` table by `transport_id` and for each found row
scans the whole `imap_markseen` table.
Number of `imap` entries for each `transport_id`
is usually large as we need to know
which UIDs to delete on IMAP server
when deleting a message.
```
sqlite> EXPLAIN QUERY PLAN
SELECT imap.id, uid, folder FROM imap, imap_markseen
WHERE imap.id = imap_markseen.id
AND imap.transport_id=?
AND target = folder
ORDER BY folder, uid;
QUERY PLAN
|--SEARCH imap USING INDEX sqlite_autoindex_imap_1 (transport_id=?)
`--SCAN imap_markseen
```
Without `ORDER BY` statement SQLite scans `imap_markseen`
table which is expected to be small,
and then searches `imap` table by `rowid` for each found result.
```
sqlite> EXPLAIN QUERY PLAN
SELECT imap.id, uid, folder FROM imap, imap_markseen
WHERE imap.id = imap_markseen.id
AND imap.transport_id=?
AND target = folder;
QUERY PLAN
|--SCAN imap_markseen
`--SEARCH imap USING INTEGER PRIMARY KEY (rowid=?)
```
Query planning was tested with SQLite 3.52.0.
It is possible to explictly make
query planner move sorting to the last step
with `ORDER +folder, +uid`, but this is not recommended
in SQLite documentation
(see <https://www.sqlite.org/optoverview.html#uplus>).
It is also possible to add indexes,
but indexes use space,
adding them requires an SQL migration,
and each index needs to be updated so it will slow down writes.
Only inbox loop is changed because non-inbox loop is going to be removed
together with `mvbox_move`.
Added transport IDs to the log and logging around quota updates.
Removed some logs that add noise,
like logging that IDLE is supported each time right before using it.
The original idea was to always lock `write_mutex` before acquiring an `InnerPool.semaphore` permit
to avoid ABBA deadlocks, but when refactoring a PR for b696a242fc,
that was forgotten.
This doesn't really change the program flow as we have `Context::housekeeping_mutex` anyway,
just simplifies the code.
If we learn about this message being available on IMAP later,
we will add another available_post_msgs row.
If we don't delete the row, we will keep failing each time
until IMAP entry becomes available and it may not happen.
Previously, if the mvbox_move folder is missing, then core will loop
infinitely, because `new_mail` is never set to false.
The fix is to first set `new_mail` to false, then return if the folder
is missing.
This is the bug @hpk42 experienced when commenting in
https://github.com/chatmail/core/issues/7989
---------
Co-authored-by: holger krekel <holger@merlinux.eu>
Follow-up to https://github.com/chatmail/core/pull/7994/, in order to
prevent clashes with other things that are called `Transport`, and in
order to make the struct name more greppable
Closes https://github.com/chatmail/core/issues/7980.
Unpublished transports are not advertised to contacts, and self-sent messages are not sent there, so that we don't cause extra messages to the corresponding inbox, but can still receive messages from contacts who don't know the new relay addresses yet.
- This adds `list_transports_ex()` and `set_transport_unpublished()` JsonRPC functions
- By default, transports are published, but when updating, all existing transports except for the primary one become unpublished in order not to break existing users that followed https://delta.chat/legacy-move
- It is not possible to unpublish the primary transport, and setting a transport as primary automatically sets it to published
An alternative would be to change the existing list_transports API rather than adding a new one list_transports_ex. But to be honest, I don't mind the _ex prefix that much, and I am wary about compatibility issues. But maybe it would be fine; see b08ba4bb8 for how this would look.
this adds an api to make the newest incoming message of a chat as "fresh",
so that UI can offer a "mark chat unread" option as usual for messengers
(eg. swipe right on iOS toggels between "read" and "unread").
"mark unread" is one of the most requested missing features,
used by many ppl to organize their every day messenger usage -
tho "pinning" and "saved messages" are similar,
it seems to be missed often.
we follow a very simple approach here
and just reset the state to `MessageState::InFresh`.
this does not introduce new states or flows.
therefore, chats without any incoming message cannot be marked as fresh.
in practise, this is probably not really an issue,
as the "mark fresh" is usually used to undo a "mark noticed" operation -
and then you have incoming message.
also, most status messages as "all messages are e2ee" count as incoming.
to avoid double sending of MDN,
we remove `Param::WantsMdn` once the MDN is scheduled.
in case MDN are used for syncing, MDN is still sent as before.
many other messenger show a "badge without number",
if we want that as well,
we can always track the "manually set as fresh" state in a parameter.
but for now, it is fine without and showing a "1", which alsso makes sense as badges may be summed up.
there is an iOS pr that uses this new feature,
jsonrpc is left out until api is settled.
also out of scope is synchronisation -
main reason is that "mark noticed" is not synced as well, so we avoid an imbalance here.
both, "mark noticed" as well as "mark fresh" should be synced however,
as soon as this feature is merged.
Hidden messages are marked as seen
when chat is marked as noticed.
MDNs to such messages should not be sent
as this notifies the hidden message sender
that the chat was opened.
The issue discovered by Frank Seifferth.
Can be reviewed commit-by-commit.
This fixes another silly thing you can do with securejoinv3: show Bob a
QR code with auth token that is a broadcast channel secret of a known
channel, then never respond. Bob will decrypt messages from the channel
and drop them because they are sent by the "wrong" sender.
This can be avoided with domain separation, instead of
encrypting/decrypting securejoinv3 messages directly with auth token,
encrypt/decrypt them with `securejoin/<auth token>` as the secret or
even `securejoinv3/<alice's fingerprint>/<auth token>`. For existing
broadcast channels we cannot do this, but for securejoinv3 that is not
released yet this looks like an improvement that avoids at least this
problem.
Credits to link2xt for noticing the problem.
This also adds Alice's fingerprint to the auth tokens, which
was pretty easy to do. I find it hard to develop an intuition for
whether this is important, or whether we will be annoyed by it in the
future.
**Note:** This means that QR code scans will not work if one of the chat
partners uses a self-compiled core between c724e2981 and merging this PR
here. This is fine; we will just have to tell the other developers to
update their self-compiled cores.
The tests were originally generated with AI and then reworked.
Follow-up to https://github.com/chatmail/core/pull/7754 (c724e29)
This prevents the following attack:
/// Eve is subscribed to a channel and wants to know whether Alice is also subscribed to it.
/// To achieve this, Eve sends a message to Alice
/// encrypted with the symmetric secret of this broadcast channel.
///
/// If Alice sends an answer (or read receipt),
/// then Eve knows that Alice is in the broadcast channel.
///
/// A similar attack would be possible with auth tokens
/// that are also used to symmetrically encrypt messages.
///
/// To prevent this, a message that was unexpectedly
/// encrypted with a symmetric secret must be dropped.
encryption info needs a dedicated string for "Messages are end-to-end encrypted"
as the UI will add more infomation to the info messages,
smth. as "Tap for more information".
an alternative fix would have been to let the UI render the info-message
differently, but adding another string to core causes less friction.
If one of broadcast owner's devices didn't add a new subscriber for any reason, e.g. because of
missing SecureJoin messages, this device shall add "member added" messages when syncing the member
list from the `SetPgpContacts` message.
The test sometimes fails because of wrong message ordering for bob:
[...]
Waiting for the device of alice@example.org to reply… [NOTICED][INFO]
<Msg#2010🔒: (Contact#Contact#2001): hi [FRESH]
Msg#2008🔒: (Contact#Contact#2001): You joined the channel. [FRESH][INFO]
>Msg#2010🔒: (Contact#Contact#2001): hi [FRESH]
Msg#2011🔒: (Contact#Contact#2001): Member Me removed by alice@example.org. [FRESH][INFO]
This adds `SystemTime::shift(Duration::from_secs(1))` as a workaround.
Co-authored-by: Hocuri <hocuri@gmx.de>
This change is mainly to avoid exposing the write lock outside the pool module.
To avoid deadlocks, outside code should work only with the pooled connections
and use no more than one connection per thread.
With multiple transports there are multiple inbox loops on the same profile `Context`.
They tend to start running housekeeping at the same time, e.g. when deleting
a message with an attachment, and then `remove_unused_files()`
tries to remove the same files that are already deleted by another thread
and logs errors.
Don't use first-person form in placeholder texts,
as these can be misleading when broadcasted to group.
Additionally ensures that broadcasted system messages
are not localized to not leak locally-set language
to the group chat.
Fixes#7930
Signed-off-by: Jagoda Ślązak <jslazak@jslazak.com>
