Simplify logic

fix: Downgrade message state to InNoticed when it's fully downloaded (#2970 )
test: Mark not downloaded message as seen (#2970 )
2026-10-17 18:33:01 +03:00 · 2024-10-15 18:02:55 +02:00 · 2024-10-14 20:20:02 -03:00 · 2024-10-14 20:20:02 -03:00 · 2024-10-14 14:01:22 +00:00 · 2024-10-14 10:24:01 -03:00
30 changed files with 287 additions and 107 deletions
@@ -211,9 +211,9 @@ jobs:
        include:
          # Currently used Rust version.
          - os: ubuntu-latest
-            python: 3.12
+            python: 3.13
          - os: macos-latest
-            python: 3.12
+            python: 3.13

          # PyPy tests
          - os: ubuntu-latest
@@ -263,11 +263,11 @@ jobs:
      matrix:
        include:
          - os: ubuntu-latest
-            python: 3.12
+            python: 3.13
          - os: macos-latest
-            python: 3.12
+            python: 3.13
          - os: windows-latest
-            python: 3.12
+            python: 3.13

          # PyPy tests
          - os: ubuntu-latest
@@ -1,5 +1,36 @@
 # Changelog

+## [1.147.1] - 2024-10-13
+
+### Build system
+
+- Build Python 3.13 wheels.
+- deltachat-rpc-client: Add classifiers for all supported Python versions.
+
+### CI
+
+- Update to Python 3.13.
+
+### Documentation
+
+- CONTRIBUTING.md: Add a note on deleting/changing db columns.
+
+### Fixes
+
+- Reset quota on configured address change ([#5908](https://github.com/deltachat/deltachat-core-rust/pull/5908)).
+- Do not emit progress 1000 when configuration is cancelled.
+- Assume file extensions are 32 chars max and don't contain whitespace ([#5338](https://github.com/deltachat/deltachat-core-rust/pull/5338)).
+- Readd tokens.foreign_id column ([#6038](https://github.com/deltachat/deltachat-core-rust/pull/6038)).
+
+### Miscellaneous Tasks
+
+- cargo: Bump futures-* from 0.3.30 to 0.3.31.
+- cargo: Upgrade async_zip to 0.0.17 ([#6035](https://github.com/deltachat/deltachat-core-rust/pull/6035)).
+
+### Refactor
+
+- MsgId::update_download_state: Don't fail if the message doesn't exist anymore.
+
 ## [1.147.0] - 2024-10-05

 ### API-Changes
@@ -4976,3 +5007,4 @@ https://github.com/deltachat/deltachat-core-rust/pulls?q=is%3Apr+is%3Aclosed
 [1.145.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.144.0..v1.145.0
 [1.146.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.145.0..v1.146.0
 [1.147.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.146.0..v1.147.0
+[1.147.1]: https://github.com/deltachat/deltachat-core-rust/compare/v1.147.0..v1.147.1
@@ -84,6 +84,12 @@ If column is already declared without `NOT NULL`, use `IFNULL` function to provi
 Use `HAVING COUNT(*) > 0` clause
 to [prevent aggregate functions such as `MIN` and `MAX` from returning `NULL`](https://stackoverflow.com/questions/66527856/aggregate-functions-max-etc-return-null-instead-of-no-rows).

+Don't delete unused columns too early, but maybe after several months/releases, unused columns are
+still used by older versions, so deleting them breaks downgrading the core or importing a backup in
+an older version. Also don't change the column type, consider adding a new column with another name
+instead. Finally, never change column semantics, this is especially dangerous because the `STRICT`
+keyword doesn't help here.
+
 ### Commit messages

 Commit messages follow the [Conventional Commits] notation.
@@ -273,12 +273,13 @@ dependencies = [

 [[package]]
 name = "async-compression"
-version = "0.3.15"
+version = "0.4.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "942c7cd7ae39e91bde4820d74132e9862e62c2f386c3aa90ccf55949f5bad63a"
+checksum = "e26a9844c659a2a293d239c7910b752f8487fe122c6c8bd1659bf85a6507c302"
 dependencies = [
 "flate2",
 "futures-core",
+ "futures-io",
 "memchr",
 "pin-project-lite",
 "tokio",
@@ -286,11 +287,12 @@ dependencies = [

 [[package]]
 name = "async-imap"
-version = "0.10.1"
+version = "0.10.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2162818f7b394e342a6591864bfc960824ed13e3f6609fee0d19e770ebcd99e1"
+checksum = "5488cd022c3c7bc41a9b34a540d9ac0d9c5cd42fdb106a67616521b7592d5b4e"
 dependencies = [
 "async-channel 2.3.1",
+ "async-compression",
 "base64 0.21.7",
 "bytes",
 "chrono",
@@ -299,6 +301,7 @@ dependencies = [
 "log",
 "nom",
 "once_cell",
+ "pin-project",
 "pin-utils",
 "self_cell",
 "stop-token",
@@ -368,16 +371,17 @@ dependencies = [

 [[package]]
 name = "async_zip"
-version = "0.0.12"
+version = "0.0.17"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b2105142db9c6203b9dadc83b0553394589a6cb31b1449a3b46b42f47c3434d0"
+checksum = "00b9f7252833d5ed4b00aa9604b563529dd5e11de9c23615de2dcdf91eb87b52"
 dependencies = [
 "async-compression",
 "crc32fast",
- "log",
+ "futures-lite 2.3.0",
 "pin-project",
 "thiserror",
 "tokio",
+ "tokio-util",
 ]

 [[package]]
@@ -1267,7 +1271,7 @@ dependencies = [

 [[package]]
 name = "deltachat"
-version = "1.147.0"
+version = "1.147.1"
 dependencies = [
 "anyhow",
 "async-broadcast",
@@ -1367,7 +1371,7 @@ dependencies = [

 [[package]]
 name = "deltachat-jsonrpc"
-version = "1.147.0"
+version = "1.147.1"
 dependencies = [
 "anyhow",
 "async-channel 2.3.1",
@@ -1392,7 +1396,7 @@ dependencies = [

 [[package]]
 name = "deltachat-repl"
-version = "1.147.0"
+version = "1.147.1"
 dependencies = [
 "anyhow",
 "deltachat",
@@ -1407,7 +1411,7 @@ dependencies = [

 [[package]]
 name = "deltachat-rpc-server"
-version = "1.147.0"
+version = "1.147.1"
 dependencies = [
 "anyhow",
 "deltachat",
@@ -1436,7 +1440,7 @@ dependencies = [

 [[package]]
 name = "deltachat_ffi"
-version = "1.147.0"
+version = "1.147.1"
 dependencies = [
 "anyhow",
 "deltachat",
@@ -2196,9 +2200,9 @@ dependencies = [

 [[package]]
 name = "futures-channel"
-version = "0.3.30"
+version = "0.3.31"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "eac8f7d7865dcb88bd4373ab671c8cf4508703796caa2b1985a9ca867b3fcb78"
+checksum = "2dff15bf788c671c1934e366d07e30c1814a8ef514e1af724a602e8a2fbe1b10"
 dependencies = [
 "futures-core",
 "futures-sink",
@@ -2221,9 +2225,9 @@ dependencies = [

 [[package]]
 name = "futures-core"
-version = "0.3.30"
+version = "0.3.31"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dfc6580bb841c5a68e9ef15c77ccc837b40a7504914d52e47b8b0e9bbda25a1d"
+checksum = "05f29059c0c2090612e8d742178b0580d2dc940c837851ad723096f87af6663e"

 [[package]]
 name = "futures-executor"
@@ -2238,9 +2242,9 @@ dependencies = [

 [[package]]
 name = "futures-io"
-version = "0.3.30"
+version = "0.3.31"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a44623e20b9681a318efdd71c299b6b222ed6f231972bfe2f224ebad6311f0c1"
+checksum = "9e5c1b78ca4aae1ac06c48a526a655760685149f0d465d21f37abfe57ce075c6"

 [[package]]
 name = "futures-lite"
@@ -2272,9 +2276,9 @@ dependencies = [

 [[package]]
 name = "futures-macro"
-version = "0.3.30"
+version = "0.3.31"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "87750cf4b7a4c0625b1529e4c543c2182106e4dedc60a2a6455e00d212c489ac"
+checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -2283,15 +2287,15 @@ dependencies = [

 [[package]]
 name = "futures-sink"
-version = "0.3.30"
+version = "0.3.31"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9fb8e00e87438d937621c1c6269e53f536c14d3fbd6a042bb24879e57d474fb5"
+checksum = "e575fab7d1e0dcb8d0c7bcf9a63ee213816ab51902e6d244a95819acacf1d4f7"

 [[package]]
 name = "futures-task"
-version = "0.3.30"
+version = "0.3.31"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "38d84fa142264698cdce1a9f9172cf383a0c82de1bddcf3092901442c4097004"
+checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988"

 [[package]]
 name = "futures-timer"
@@ -2301,9 +2305,9 @@ checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24"

 [[package]]
 name = "futures-util"
-version = "0.3.30"
+version = "0.3.31"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3d6401deb83407ab3da39eba7e33987a73c3df0c82b4bb5813ee871c19c41d48"
+checksum = "9fa08315bb612088cc391249efdc3bc77536f16c91f6cf495e6fbe85b20a4a81"
 dependencies = [
 "futures-channel",
 "futures-core",
@@ -6074,6 +6078,7 @@ checksum = "61e7c3654c13bcd040d4a03abee2c75b1d14a37b423cf5a813ceae1cc903ec6a"
 dependencies = [
 "bytes",
 "futures-core",
+ "futures-io",
 "futures-sink",
 "futures-util",
 "hashbrown",
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat"
-version = "1.147.0"
+version = "1.147.1"
 edition = "2021"
 license = "MPL-2.0"
 rust-version = "1.77"
@@ -41,10 +41,10 @@ ratelimit = { path = "./deltachat-ratelimit" }
 anyhow = { workspace = true }
 async-broadcast = "0.7.1"
 async-channel = { workspace = true }
-async-imap = { version = "0.10.1", default-features = false, features = ["runtime-tokio"] }
+async-imap = { version = "0.10.2", default-features = false, features = ["runtime-tokio", "compress"] }
 async-native-tls = { version = "0.5", default-features = false, features = ["runtime-tokio"] }
 async-smtp = { version = "0.9", default-features = false, features = ["runtime-tokio"] }
-async_zip = { version = "0.0.12", default-features = false, features = ["deflate", "fs"] }
+async_zip = { version = "0.0.17", default-features = false, features = ["deflate", "tokio-fs"] }
 base64 = { workspace = true }
 brotli = { version = "6", default-features=false, features = ["std"] }
 bytes = "1"
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat_ffi"
-version = "1.147.0"
+version = "1.147.1"
 description = "Deltachat FFI"
 edition = "2018"
 readme = "README.md"
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-jsonrpc"
-version = "1.147.0"
+version = "1.147.1"
 description = "DeltaChat JSON-RPC API"
 edition = "2021"
 default-run = "deltachat-jsonrpc-server"
@@ -58,5 +58,5 @@
  },
  "type": "module",
  "types": "dist/deltachat.d.ts",
-  "version": "1.147.0"
+  "version": "1.147.1"
 }
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-repl"
-version = "1.147.0"
+version = "1.147.1"
 license = "MPL-2.0"
 edition = "2021"
 repository = "https://github.com/deltachat/deltachat-core-rust"
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

 [project]
 name = "deltachat-rpc-client"
-version = "1.147.0"
+version = "1.147.1"
 description = "Python client for Delta Chat core JSON-RPC interface"
 classifiers = [
    "Development Status :: 5 - Production/Stable",
@@ -13,10 +13,13 @@ classifiers = [
    "Operating System :: POSIX :: Linux",
    "Operating System :: MacOS :: MacOS X",
    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
    "Topic :: Communications :: Chat",
    "Topic :: Communications :: Email"
 ]
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-rpc-server"
-version = "1.147.0"
+version = "1.147.1"
 description = "DeltaChat JSON-RPC server"
 edition = "2021"
 readme = "README.md"
@@ -15,5 +15,5 @@
  },
  "type": "module",
  "types": "index.d.ts",
-  "version": "1.147.0"
+  "version": "1.147.1"
 }
@@ -55,5 +55,5 @@
    "test:mocha": "mocha node/test/test.mjs --growl --reporter=spec --bail --exit"
  },
  "types": "node/dist/index.d.ts",
-  "version": "1.147.0"
+  "version": "1.147.1"
 }
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

 [project]
 name = "deltachat"
-version = "1.147.0"
+version = "1.147.1"
 description = "Python bindings for the Delta Chat Core library using CFFI against the Rust-implemented libdeltachat"
 readme = "README.rst"
 requires-python = ">=3.7"
@@ -1 +1 @@
-2024-10-05
+2024-10-13
@@ -31,6 +31,6 @@ unset CHATMAIL_DOMAIN

 # Try to build wheels for a range of interpreters, but don't fail if they are not available.
 # E.g. musllinux_1_1 does not have PyPy interpreters as of 2022-07-10
-tox --workdir "$TOXWORKDIR" -e py37,py38,py39,py310,py311,py312,pypy37,pypy38,pypy39,pypy310 --skip-missing-interpreters true
+tox --workdir "$TOXWORKDIR" -e py37,py38,py39,py310,py311,py312,py313,pypy37,pypy38,pypy39,pypy310 --skip-missing-interpreters true

 auditwheel repair "$TOXWORKDIR"/wheelhouse/deltachat* -w "$TOXWORKDIR/wheelhouse"
@@ -253,16 +253,16 @@ impl<'a> BlobObject<'a> {
    ///
    /// The extension part will always be lowercased.
    fn sanitise_name(name: &str) -> (String, String) {
-        let mut name = name.to_string();
+        let mut name = name;
        for part in name.rsplit('/') {
            if !part.is_empty() {
-                name = part.to_string();
+                name = part;
                break;
            }
        }
        for part in name.rsplit('\\') {
            if !part.is_empty() {
-                name = part.to_string();
+                name = part;
                break;
            }
        }
@@ -272,32 +272,39 @@ impl<'a> BlobObject<'a> {
            replacement: "",
        };

-        let clean = sanitize_filename::sanitize_with_options(name, opts);
-        // Let's take the tricky filename
+        let name = sanitize_filename::sanitize_with_options(name, opts);
+        // Let's take a tricky filename,
        // "file.with_lots_of_characters_behind_point_and_double_ending.tar.gz" as an example.
-        // Split it into "file" and "with_lots_of_characters_behind_point_and_double_ending.tar.gz":
-        let mut iter = clean.splitn(2, '.');
-
-        let stem: String = iter.next().unwrap_or_default().chars().take(64).collect();
-        // stem == "file"
-
-        let ext_chars = iter.next().unwrap_or_default().chars();
-        let ext: String = ext_chars
+        // Assume that the extension is 32 chars maximum.
+        let ext: String = name
+            .chars()
            .rev()
-            .take(32)
+            .take_while(|c| !c.is_whitespace())
+            .take(33)
            .collect::<Vec<_>>()
            .iter()
            .rev()
            .collect();
-        // ext == "d_point_and_double_ending.tar.gz"
+        // ext == "nd_point_and_double_ending.tar.gz"

-        if ext.is_empty() {
-            (stem, "".to_string())
+        // Split it into "nd_point_and_double_ending" and "tar.gz":
+        let mut iter = ext.splitn(2, '.');
+        iter.next();
+
+        let ext = iter.next().unwrap_or_default();
+        let ext = if ext.is_empty() {
+            String::new()
        } else {
-            (stem, format!(".{ext}").to_lowercase())
-            // Return ("file", ".d_point_and_double_ending.tar.gz")
-            // which is not perfect but acceptable.
-        }
+            format!(".{ext}")
+            // ".tar.gz"
+        };
+        let stem = name
+            .strip_suffix(&ext)
+            .unwrap_or_default()
+            .chars()
+            .take(64)
+            .collect();
+        (stem, ext.to_lowercase())
    }

    /// Checks whether a name is a valid blob name.
@@ -963,6 +970,19 @@ mod tests {
        assert!(!stem.contains(':'));
        assert!(!stem.contains('*'));
        assert!(!stem.contains('?'));
+
+        let (stem, ext) = BlobObject::sanitise_name(
+            "file.with_lots_of_characters_behind_point_and_double_ending.tar.gz",
+        );
+        assert_eq!(
+            stem,
+            "file.with_lots_of_characters_behind_point_and_double_ending"
+        );
+        assert_eq!(ext, ".tar.gz");
+
+        let (stem, ext) = BlobObject::sanitise_name("a. tar.tar.gz");
+        assert_eq!(stem, "a. tar");
+        assert_eq!(ext, ".tar.gz");
    }

    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -857,6 +857,8 @@ impl Context {
    ///
    /// This should only be used by test code and during configure.
    pub(crate) async fn set_primary_self_addr(&self, primary_new: &str) -> Result<()> {
+        self.quota.write().await.take();
+
        // add old primary address (if exists) to secondary addresses
        let mut secondary_addrs = self.get_all_self_addrs().await?;
        // never store a primary address also as a secondary
@@ -869,7 +871,7 @@ impl Context {

        self.set_config_internal(Config::ConfiguredAddr, Some(primary_new))
            .await?;
-
+        self.emit_event(EventType::ConnectivityChanged);
        Ok(())
    }

@@ -13,7 +13,7 @@ mod auto_mozilla;
 mod auto_outlook;
 pub(crate) mod server_params;

-use anyhow::{bail, ensure, Context as _, Result};
+use anyhow::{bail, ensure, format_err, Context as _, Result};
 use auto_mozilla::moz_autoconfigure;
 use auto_outlook::outlk_autodiscover;
 use deltachat_contact_tools::EmailAddress;
@@ -80,10 +80,7 @@ impl Context {

        let res = self
            .inner_configure()
-            .race(cancel_channel.recv().map(|_| {
-                progress!(self, 0);
-                Ok(())
-            }))
+            .race(cancel_channel.recv().map(|_| Err(format_err!("Cancelled"))))
            .await;

        self.free_ongoing().await;
@@ -98,19 +98,26 @@ impl MsgId {
        Ok(())
    }

+    /// Updates the message download state. Returns `Ok` if the message doesn't exist anymore.
    pub(crate) async fn update_download_state(
        self,
        context: &Context,
        download_state: DownloadState,
    ) -> Result<()> {
-        let msg = Message::load_from_db(context, self).await?;
-        context
+        if context
            .sql
            .execute(
                "UPDATE msgs SET download_state=? WHERE id=?;",
                (download_state, self),
            )
-            .await?;
+            .await?
+            == 0
+        {
+            return Ok(());
+        }
+        let Some(msg) = Message::load_from_db_optional(context, self).await? else {
+            return Ok(());
+        };
        context.emit_event(EventType::MsgsChanged {
            chat_id: msg.chat_id,
            msg_id: self,
@@ -322,11 +329,17 @@ mod tests {
            DownloadState::InProgress,
            DownloadState::Failure,
            DownloadState::Done,
+            DownloadState::Done,
        ] {
            msg_id.update_download_state(&t, *s).await?;
            let msg = Message::load_from_db(&t, msg_id).await?;
            assert_eq!(msg.download_state(), *s);
        }
+        msg_id.delete_from_db(&t).await?;
+        // Nothing to do is ok.
+        msg_id
+            .update_download_state(&t, DownloadState::Done)
+            .await?;

        Ok(())
    }
@@ -39,6 +39,7 @@ use crate::login_param::{
 use crate::message::{self, Message, MessageState, MessengerMessage, MsgId, Viewtype};
 use crate::mimeparser;
 use crate::net::proxy::ProxyConfig;
+use crate::net::session::SessionStream;
 use crate::oauth2::get_oauth2_access_token;
 use crate::receive_imf::{
    from_field_to_contact_id, get_prefetch_parent_message, receive_imf_inner, ReceivedMsg,
@@ -55,7 +56,7 @@ pub mod scan_folders;
 pub mod select_folder;
 pub(crate) mod session;

-use client::Client;
+use client::{determine_capabilities, Client};
 use mailparse::SingleInfo;
 use session::Session;

@@ -376,7 +377,23 @@ impl Imap {
            };

            match login_res {
-                Ok(session) => {
+                Ok(mut session) => {
+                    let capabilities = determine_capabilities(&mut session).await?;
+
+                    let session = if capabilities.can_compress {
+                        info!(context, "Enabling IMAP compression.");
+                        let compressed_session = session
+                            .compress(|s| {
+                                let session_stream: Box<dyn SessionStream> = Box::new(s);
+                                session_stream
+                            })
+                            .await
+                            .context("Failed to enable IMAP compression")?;
+                        Session::new(compressed_session, capabilities)
+                    } else {
+                        Session::new(session, capabilities)
+                    };
+
                    // Store server ID in the context to display in account info.
                    let mut lock = context.server_id.write().await;
                    lock.clone_from(&session.capabilities.server_id);
@@ -25,6 +25,10 @@ pub(crate) struct Capabilities {
    /// <https://tools.ietf.org/html/rfc5464>
    pub can_metadata: bool,

+    /// True if the server has COMPRESS=DEFLATE capability as defined in
+    /// <https://tools.ietf.org/html/rfc4978>
+    pub can_compress: bool,
+
    /// True if the server supports XDELTAPUSH capability.
    /// This capability means setting /private/devicetoken IMAP METADATA
    /// on the INBOX results in new mail notifications
@@ -7,7 +7,6 @@ use async_imap::Session as ImapSession;
 use tokio::io::BufWriter;

 use super::capabilities::Capabilities;
-use super::session::Session;
 use crate::context::Context;
 use crate::login_param::{ConnectionCandidate, ConnectionSecurity};
 use crate::net::dns::{lookup_host_with_cache, update_connect_timestamp};
@@ -51,7 +50,7 @@ fn alpn(port: u16) -> &'static [&'static str] {
 /// Determine server capabilities.
 ///
 /// If server supports ID capability, send our client ID.
-async fn determine_capabilities(
+pub(crate) async fn determine_capabilities(
    session: &mut ImapSession<Box<dyn SessionStream>>,
 ) -> Result<Capabilities> {
    let caps = session
@@ -69,6 +68,7 @@ async fn determine_capabilities(
        can_check_quota: caps.has_str("QUOTA"),
        can_condstore: caps.has_str("CONDSTORE"),
        can_metadata: caps.has_str("METADATA"),
+        can_compress: caps.has_str("COMPRESS=DEFLATE"),
        can_push: caps.has_str("XDELTAPUSH"),
        is_chatmail: caps.has_str("XCHATMAIL"),
        server_id,
@@ -83,28 +83,31 @@ impl Client {
        }
    }

-    pub(crate) async fn login(self, username: &str, password: &str) -> Result<Session> {
+    pub(crate) async fn login(
+        self,
+        username: &str,
+        password: &str,
+    ) -> Result<ImapSession<Box<dyn SessionStream>>> {
        let Client { inner, .. } = self;
-        let mut session = inner
+
+        let session = inner
            .login(username, password)
            .await
            .map_err(|(err, _client)| err)?;
-        let capabilities = determine_capabilities(&mut session).await?;
-        Ok(Session::new(session, capabilities))
+        Ok(session)
    }

    pub(crate) async fn authenticate(
        self,
        auth_type: &str,
        authenticator: impl async_imap::Authenticator,
-    ) -> Result<Session> {
+    ) -> Result<ImapSession<Box<dyn SessionStream>>> {
        let Client { inner, .. } = self;
-        let mut session = inner
+        let session = inner
            .authenticate(auth_type, authenticator)
            .await
            .map_err(|(err, _client)| err)?;
-        let capabilities = determine_capabilities(&mut session).await?;
-        Ok(Session::new(session, capabilities))
+        Ok(session)
    }

    async fn connection_attempt(
@@ -1826,8 +1826,8 @@ pub(crate) async fn update_msg_state(
    context
        .sql
        .execute(
-            &format!("UPDATE msgs SET state=?1 {error_subst} WHERE id=?2 AND (?1!=?3 OR state<?3)"),
-            (state, msg_id, MessageState::OutDelivered),
+            &format!("UPDATE msgs SET state=? {error_subst} WHERE id=?"),
+            (state, msg_id),
        )
        .await?;
    Ok(())
@@ -2590,6 +2590,60 @@ mod tests {
        Ok(())
    }

+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_markseen_not_downloaded_msg() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let alice = &tcm.alice().await;
+        alice.set_config(Config::DownloadLimit, Some("1")).await?;
+        let bob = &tcm.bob().await;
+        let bob_chat_id = tcm.send_recv_accept(alice, bob, "hi").await.chat_id;
+
+        let file_bytes = include_bytes!("../test-data/image/screenshot.png");
+        let mut msg = Message::new(Viewtype::Image);
+        msg.set_file_from_bytes(bob, "a.jpg", file_bytes, None)
+            .await?;
+        let sent_msg = bob.send_msg(bob_chat_id, &mut msg).await;
+        let msg = alice.recv_msg(&sent_msg).await;
+        assert_eq!(msg.download_state, DownloadState::Available);
+        assert!(!msg.param.get_bool(Param::WantsMdn).unwrap_or_default());
+        assert_eq!(msg.state, MessageState::InFresh);
+        markseen_msgs(alice, vec![msg.id]).await?;
+        let msg = Message::load_from_db(alice, msg.id).await?;
+        assert_eq!(msg.state, MessageState::InSeen);
+        assert!(
+            !alice
+                .sql
+                .exists("SELECT COUNT(*) FROM smtp_mdns", ())
+                .await?
+        );
+
+        alice.set_config(Config::DownloadLimit, None).await?;
+        // Simulate that the message is even marked as `\Seen` on IMAP.
+        let rcvd_msg = receive_imf(alice, sent_msg.payload().as_bytes(), true)
+            .await
+            .unwrap()
+            .unwrap();
+        assert_eq!(rcvd_msg.chat_id, msg.chat_id);
+        let msg = Message::load_from_db(alice, *rcvd_msg.msg_ids.last().unwrap())
+            .await
+            .unwrap();
+        assert_eq!(msg.download_state, DownloadState::Done);
+        assert!(msg.param.get_bool(Param::WantsMdn).unwrap_or_default());
+        assert!(msg.get_showpadlock());
+        assert_eq!(msg.state, MessageState::InNoticed);
+        markseen_msgs(alice, vec![msg.id]).await?;
+        let msg = Message::load_from_db(alice, msg.id).await?;
+        assert_eq!(msg.state, MessageState::InSeen);
+        assert_eq!(
+            alice
+                .sql
+                .count("SELECT COUNT(*) FROM smtp_mdns", ())
+                .await?,
+            1
+        );
+        Ok(())
+    }
+
    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
    async fn test_get_state() -> Result<()> {
        let alice = TestContext::new_alice().await;
@@ -53,6 +53,11 @@ impl<T: SessionStream> SessionStream for shadowsocks::ProxyClientStream<T> {
        self.get_mut().set_read_timeout(timeout)
    }
 }
+impl<T: SessionStream> SessionStream for async_imap::DeflateStream<T> {
+    fn set_read_timeout(&mut self, timeout: Option<Duration>) {
+        self.get_mut().set_read_timeout(timeout)
+    }
+}

 /// Session stream with a read buffer.
 pub(crate) trait SessionBufStream: SessionStream + AsyncBufRead {}
@@ -197,7 +197,7 @@ mod tests {
    }

    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-    async fn test_quota_needs_update() {
+    async fn test_quota_needs_update() -> Result<()> {
        let mut tcm = TestContextManager::new();
        let t = &tcm.unconfigured().await;
        const TIMEOUT: u64 = 60;
@@ -214,5 +214,14 @@ mod tests {
            modified: tools::Time::now(),
        });
        assert!(!t.quota_needs_update(TIMEOUT).await);
+
+        t.evtracker.clear_events();
+        t.set_primary_self_addr("new@addr").await?;
+        assert!(t.quota.read().await.is_none());
+        t.evtracker
+            .get_matching(|evt| matches!(evt, EventType::ConnectivityChanged))
+            .await;
+        assert!(t.quota_needs_update(TIMEOUT).await);
+        Ok(())
    }
 }
@@ -1014,7 +1014,10 @@ async fn add_parts(
            }
        }

-        state = if seen
+        state = if replace_msg_id.is_some() {
+            // TODO a comment explaining why we're doing this would be nice
+            MessageState::InNoticed
+        } else if seen
            || fetching_existing_messages
            || is_mdn
            || is_reaction
@@ -1562,7 +1565,7 @@ INSERT INTO msgs
 ON CONFLICT (id) DO UPDATE
 SET rfc724_mid=excluded.rfc724_mid, chat_id=excluded.chat_id,
    from_id=excluded.from_id, to_id=excluded.to_id, timestamp_sent=excluded.timestamp_sent,
-    type=excluded.type, msgrmsg=excluded.msgrmsg,
+    type=excluded.type, state=min(state,excluded.state), msgrmsg=excluded.msgrmsg,
    txt=excluded.txt, txt_normalized=excluded.txt_normalized, subject=excluded.subject,
    txt_raw=excluded.txt_raw, param=excluded.param,
    hidden=excluded.hidden,bytes=excluded.bytes, mime_headers=excluded.mime_headers,
@@ -1613,7 +1616,7 @@ RETURNING id
                    } else {
                        DownloadState::Done
                    },
-                    mime_parser.hop_info
+                    mime_parser.hop_info,
                ],
                |row| {
                    let msg_id: MsgId = row.get(0)?;
@@ -3163,20 +3163,21 @@ Reply from different address
 }

 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-async fn test_long_and_duplicated_filenames() -> Result<()> {
+async fn test_weird_and_duplicated_filenames() -> Result<()> {
    let mut tcm = TestContextManager::new();
    let alice = tcm.alice().await;
    let bob = tcm.bob().await;

    for filename_sent in &[
        "foo.bar very long file name test baz.tar.gz",
-        "foobarabababababababbababababverylongfilenametestbaz.tar.gz",
+        "foo.barabababababababbababababverylongfilenametestbaz.tar.gz",
        "fooo...tar.gz",
        "foo. .tar.gz",
        "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.tar.gz",
        "a.tar.gz",
        "a.tar.gz",
        "a.a..a.a.a.a.tar.gz",
+        "a. tar.tar.gz",
    ] {
        let attachment = alice.blobdir.join(filename_sent);
        let content = format!("File content of {filename_sent}");
@@ -1037,6 +1037,15 @@ CREATE INDEX msgs_status_updates_index2 ON msgs_status_updates (uid);
        .await?;
    }

+    inc_and_check(&mut migration_version, 122)?;
+    if dbversion < migration_version {
+        sql.execute_migration(
+            "ALTER TABLE tokens ADD COLUMN foreign_id INTEGER NOT NULL DEFAULT 0",
+            migration_version,
+        )
+        .await?;
+    }
+
    let new_version = sql
        .get_raw_config_int(VERSION_CFG)
        .await?
@@ -23,13 +23,13 @@ use std::path::Path;

 use anyhow::{anyhow, bail, ensure, format_err, Context as _, Result};

+use async_zip::tokio::read::fs::ZipFileReader as FsZipFileReader;
 use deltachat_contact_tools::sanitize_bidi_characters;
 use deltachat_derive::FromSql;
 use lettre_email::PartBuilder;
 use rusqlite::OptionalExtension;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
-use tokio::io::AsyncReadExt;

 use crate::chat::{self, Chat};
 use crate::constants::Chattype;
@@ -195,7 +195,7 @@ fn find_zip_entry<'a>(
    name: &str,
 ) -> Option<(usize, &'a async_zip::StoredZipEntry)> {
    for (i, ent) in file.entries().iter().enumerate() {
-        if ent.entry().filename() == name {
+        if ent.filename().as_bytes() == name.as_bytes() {
            return Some((i, ent));
        }
    }
@@ -212,7 +212,7 @@ impl Context {
            return Ok(false);
        }

-        let archive = match async_zip::read::mem::ZipFileReader::new(file.to_vec()).await {
+        let archive = match async_zip::base::read::mem::ZipFileReader::new(file.to_vec()).await {
            Ok(archive) => archive,
            Err(_) => {
                info!(self, "{} cannot be opened as zip-file", &filename);
@@ -235,7 +235,7 @@ impl Context {
            bail!("{} is not a valid webxdc file", filename);
        }

-        let valid = match async_zip::read::fs::ZipFileReader::new(path).await {
+        let valid = match FsZipFileReader::new(path).await {
            Ok(archive) => {
                if find_zip_entry(archive.file(), "index.html").is_none() {
                    warn!(self, "{} misses index.html", filename);
@@ -791,27 +791,24 @@ fn parse_webxdc_manifest(bytes: &[u8]) -> Result<WebxdcManifest> {
    Ok(manifest)
 }

-async fn get_blob(archive: &async_zip::read::fs::ZipFileReader, name: &str) -> Result<Vec<u8>> {
+async fn get_blob(archive: &FsZipFileReader, name: &str) -> Result<Vec<u8>> {
    let (i, _) = find_zip_entry(archive.file(), name)
        .ok_or_else(|| anyhow!("no entry found for {}", name))?;
-    let mut reader = archive.entry(i).await?;
+    let mut reader = archive.reader_with_entry(i).await?;
    let mut buf = Vec::new();
-    reader.read_to_end(&mut buf).await?;
+    reader.read_to_end_checked(&mut buf).await?;
    Ok(buf)
 }

 impl Message {
    /// Get handle to a webxdc ZIP-archive.
    /// To check for file existence use archive.by_name(), to read a file, use get_blob(archive).
-    async fn get_webxdc_archive(
-        &self,
-        context: &Context,
-    ) -> Result<async_zip::read::fs::ZipFileReader> {
+    async fn get_webxdc_archive(&self, context: &Context) -> Result<FsZipFileReader> {
        let path = self
            .get_file(context)
            .ok_or_else(|| format_err!("No webxdc instance file."))?;
        let path_abs = get_abs_path(context, &path);
-        let archive = async_zip::read::fs::ZipFileReader::new(path_abs).await?;
+        let archive = FsZipFileReader::new(path_abs).await?;
        Ok(archive)
    }
Author	SHA1	Message	Date
Hocuri	d94fa41b94	Simplify logic	2024-10-15 18:02:55 +02:00
iequidoo	be59fd473e	fix: Downgrade message state to InNoticed when it's fully downloaded (#2970 )	2024-10-14 20:20:02 -03:00
iequidoo	e37dd8470b	test: Mark not downloaded message as seen (#2970 ) Add a test on what happens currently when apps call `markseen_msgs()` for not downloaded messages. Such messages are marked as seen, but MDNs aren't sent for them. This is probably correct, but when a message is downloaded, its state should be downgraded to `InNoticed` and when the user really sees it, the app should issue `markseen_msgs()` again and that should trigger sending an MDN. Currently when such a message is downloaded, it remains `InSeen`.	2024-10-14 20:20:02 -03:00
link2xt	f1ca689f99	feat: IMAP COMPRESS support	2024-10-14 14:01:22 +00:00
iequidoo	796b0d7752	refactor: update_msg_state: Don't avoid downgrading OutMdnRcvd to OutDelivered `OutMdnRcvd` is a "virtual" message state now, only old messages can have this state in the db, so `update_msg_state()` can be simplified.	2024-10-14 10:24:01 -03:00
link2xt	2ea5c86a5a	chore(release): prepare for 1.147.1	2024-10-13 18:40:33 +00:00
iequidoo	50b250cf78	docs(CONTRIBUTING.md): Add a note on deleting/changing db columns	2024-10-13 15:34:15 -03:00
iequidoo	3c03370589	fix: Readd tokens.foreign_id column (#6038 ) Otherwise backups exported from the current core and imported in versions < 1.144.0 have QR codes not working. The breaking change which removed the column is `5a6efdff44`.	2024-10-13 15:34:15 -03:00
iequidoo	8f41aed917	fix: Assume file extensions are 32 chars max and don't contain whitespace (#5338 ) Before file extensions were also limited to 32 chars, but extra chars in the beginning were just cut off, e.g. "file.with_lots_of_characters_behind_point_and_double_ending.tar.gz" was considered to have an extension "d_point_and_double_ending.tar.gz". Better to take only "tar.gz" then. Also don't include whitespace-containing parts in extensions. File extensions generally don't contain whitespaces.	2024-10-11 11:04:41 -03:00
link2xt	19be12a25d	chore(cargo): upgrade async_zip to 0.0.17 (#6035 )	2024-10-11 00:17:40 +00:00
link2xt	6a121b87eb	fix: do not emit progress 1000 when configuration is cancelled There is already code below that emits progress 0 or 1000 depending on whether configuration succeeded or failed. Before this change cancelling resulted in progress 0 emitted, immediately followed by progress 1000.	2024-10-10 10:34:58 +00:00
link2xt	420c0ed9b0	build(deltachat-rpc-client): add classifiers for all supported Python versions	2024-10-10 07:18:06 +00:00
link2xt	e05bb03db6	build: build Python 3.13 wheels	2024-10-10 07:18:06 +00:00
link2xt	73fcb97eef	ci: update to Python 3.13	2024-10-10 07:18:06 +00:00
iequidoo	8acf391ffe	refactor: MsgId::update_download_state: Don't fail if the message doesn't exist anymore If a race happens and the message disappears, there's just nothing to do and no sense to fail. Follow-up to `22e5bf8571`.	2024-10-08 12:31:41 -03:00
iequidoo	aacea2de25	fix: Reset quota on configured address change (#5908 )	2024-10-07 18:04:53 -03:00
iequidoo	b713e8cd94	chore(cargo): bump futures-* from 0.3.30 to 0.3.31 futures-util 0.3.30 is yanked.	2024-10-07 15:33:09 -03:00
@@ -1 +1 @@
 -10-05
 -10-13