fix: move sorting outside of SQL query in store_seen_flags_on_imap

With `ORDER BY` statement SQLite searches
the `imap` table by `transport_id` and for each found row
scans the whole `imap_markseen` table.
Number of `imap` entries for each `transport_id`
is usually large as we need to know
which UIDs to delete on IMAP server
when deleting a message.

```
sqlite> EXPLAIN QUERY PLAN
SELECT imap.id, uid, folder FROM imap, imap_markseen
WHERE imap.id = imap_markseen.id
AND imap.transport_id=?
AND target = folder
ORDER BY folder, uid;
QUERY PLAN
|--SEARCH imap USING INDEX sqlite_autoindex_imap_1 (transport_id=?)
`--SCAN imap_markseen
```

Without `ORDER BY` statement SQLite scans `imap_markseen`
table which is expected to be small,
and then searches `imap` table by `rowid` for each found result.

```
sqlite> EXPLAIN QUERY PLAN
SELECT imap.id, uid, folder FROM imap, imap_markseen
WHERE imap.id = imap_markseen.id
AND imap.transport_id=?
AND target = folder;
QUERY PLAN
|--SCAN imap_markseen
`--SEARCH imap USING INTEGER PRIMARY KEY (rowid=?)
```

Query planning was tested with SQLite 3.52.0.
It is possible to explictly make
query planner move sorting to the last step
with `ORDER +folder, +uid`, but this is not recommended
in SQLite documentation
(see <https://www.sqlite.org/optoverview.html#uplus>).

It is also possible to add indexes,
but indexes use space,
adding them requires an SQL migration,
and each index needs to be updated so it will slow down writes.
This commit is contained in:
link2xt
2026-03-26 03:37:59 +01:00
committed by l
parent a24248a90b
commit 4dd0ba2c72

View File

@@ -1104,14 +1104,13 @@ impl Session {
}
let transport_id = self.transport_id();
let rows = context
let mut rows = context
.sql
.query_map_vec(
"SELECT imap.id, uid, folder FROM imap, imap_markseen
WHERE imap.id = imap_markseen.id
AND imap.transport_id=?
AND target = folder
ORDER BY folder, uid",
AND target = folder",
(transport_id,),
|row| {
let rowid: i64 = row.get(0)?;
@@ -1122,6 +1121,16 @@ impl Session {
)
.await?;
// Number of SQL results is expected to be low as
// we usually don't have many messages to mark on IMAP at once.
// We are sorting outside of SQL to avoid SQLite constructing a query plan
// that scans the whole `imap` table. Scanning `imap_markseen` is fine
// as it should not have many items.
// If you change the SQL query, test it with `EXPLAIN QUERY PLAN`.
rows.sort_unstable_by(|(_rowid1, uid1, folder1), (_rowid2, uid2, folder2)| {
(folder1, uid1).cmp(&(folder2, uid2))
});
for (folder, rowid_set, uid_set) in UidGrouper::from(rows) {
let folder_exists = match self.select_with_uidvalidity(context, &folder).await {
Err(err) => {