feat(sql): truncate WAL on stop_io()

This results in 0xdead10cc crashes on suspend.
iOS itself ensures that multiple instances of Delta Chat are not running.

.github/workflows/ci.yml

@@ -24,7 +24,7 @@ jobs:
     name: Lint Rust
     runs-on: ubuntu-latest
     env:
-      RUSTUP_TOOLCHAIN: 1.73.0
+      RUSTUP_TOOLCHAIN: 1.74.0
     steps:
       - uses: actions/checkout@v3
       - name: Install rustfmt and clippy
@@ -76,11 +76,11 @@ jobs:
       matrix:
         include:
           - os: ubuntu-latest
-            rust: 1.73.0
+            rust: 1.74.0
           - os: windows-latest
-            rust: 1.73.0
+            rust: 1.74.0
           - os: macos-latest
-            rust: 1.73.0
+            rust: 1.74.0
 
           # Minimum Supported Rust Version = 1.70.0
           - os: ubuntu-latest

.github/workflows/upload-python-docs.yml

@@ -0,0 +1,25 @@
+name: Build & Deploy Documentation on py.delta.chat
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 0 # Fetch history to calculate VCS version number.
+      - name: Build Python documentation
+        run: scripts/build-python-docs.sh
+      - name: Upload to py.delta.chat
+        uses: up9cloud/action-rsync@v1.3
+        env:
+          USER: delta
+          KEY: ${{ secrets.CODESPEAK_KEY }}
+          HOST: "lists.codespeak.net"
+          SOURCE: "dist/html/"
+          TARGET: "/home/delta/build/master"

CHANGELOG.md

@@ -1,5 +1,121 @@
 # Changelog
 
+## [1.131.9] - 2023-12-02
+
+### API-Changes
+
+- Remove `dc_get_http_response()`, `dc_http_response_get_mimetype()`, `dc_http_response_get_encoding()`, `dc_http_response_get_blob()`, `dc_http_response_get_size()`, `dc_http_response_unref()` and `dc_http_response_t` from cffi.
+- Deprecate CFFI APIs `dc_send_reaction()`, `dc_get_msg_reactions()`, `dc_reactions_get_contacts()`, `dc_reactions_get_by_contact_id()`, `dc_reactions_unref` and `dc_reactions_t`.
+- Make `Contact.is_verified()` return bool.
+
+### Build system
+
+- Switch from fork of iroh to iroh 0.4.2 pre-release.
+
+### Features / Changes
+
+- Send `Chat-Verified` headers in 1:1 chats.
+- Ratelimit IMAP connections ([#4940](https://github.com/deltachat/deltachat-core-rust/pull/4940)).
+- Remove receiver limit on `.xdc` size.
+- Don't affect MimeMessage with "From" and secured headers from encrypted unsigned messages.
+- Sync `Config::{MdnsEnabled,ShowEmails}` across devices ([#4954](https://github.com/deltachat/deltachat-core-rust/pull/4954)).
+- Sync `Config::Displayname` across devices ([#4893](https://github.com/deltachat/deltachat-core-rust/pull/4893)).
+- `Chat::rename_ex`: Don't send sync message if usual message is sent.
+
+### Fixes
+
+- Lock the database when INSERTing a webxdc update, avoid "Database is locked" errors.
+- Use keyring with all private keys when decrypting a message ([#5046](https://github.com/deltachat/deltachat-core-rust/pull/5046)).
+
+### Tests
+
+- Make Result-returning tests produce a line number.
+- Add `test_utils::sync()`.
+- Test inserting lots of webxdc updates.
+- Split `test_sync_alter_chat()` into smaller tests.
+
+## [1.131.8] - 2023-11-27
+
+### Features / Changes
+
+- webxdc: Add unique IDs to status updates sent outside and deduplicate based on IDs.
+
+### Fixes
+
+- Allow IMAP servers not returning UIDNEXT on SELECT and STATUS such as mail.163.com.
+- Use the correct securejoin strings used in the UI, remove old TODO ([#5047](https://github.com/deltachat/deltachat-core-rust/pull/5047)).
+- Do not emit events about webxdc update events logged into debug log webxdc.
+
+### Tests
+
+- Check that `receive_status_update` has forward compatibility and unique webxdc IDs will be ignored by previous Delta Chat versions.
+
+## [1.131.7] - 2023-11-24
+
+### Fixes
+
+- Revert "fix: check UIDNEXT with a STATUS command before going IDLE". This attempts to fix mail.163.com which has broken STATUS command.
+
+## [1.131.6] - 2023-11-21
+
+### Fixes
+
+- Fail fast if IMAP FETCH cannot be parsed instead of getting stuck in infinite loop.
+
+### Documentation
+
+- Generate deltachat-rpc-client documentation and publish it to <https://py.delta.chat>.
+
+## [1.131.5] - 2023-11-20
+
+### API-Changes
+
+- deltachat-rpc-client: Add `Message.get_sender_contact()`.
+- Turn `ContactAddress` into an owned type.
+
+### Features / Changes
+
+- Lowercase addresses in Autocrypt and Autocrypt-Gossip headers.
+- Lowercase the address in member added/removed messages.
+- Lowercase `addr` when it is set.
+- Do not replace the message with an error in square brackets when the sender is not a member of the protected group.
+
+### Fixes
+
+- `Chat::sync_contacts()`: Fetch contact addresses in a single query.
+- `Chat::rename_ex()`: Sync improved chat name to other devices.
+- Recognize `Chat-Group-Member-Added` of self case-insensitively.
+- Compare verifier addr to peerstate addr case-insensitively.
+
+### Tests
+
+- Port [Secure-Join](https://securejoin.readthedocs.io/) tests to JSON-RPC.
+
+### CI
+
+- Test with Rust 1.74.
+
+
+## [1.131.4] - 2023-11-16
+
+### Documentation
+
+- Document DC_DOWNLOAD_UNDECIPHERABLE.
+
+### Fixes
+
+- Always add "Member added" as system message.
+
+## [1.131.3] - 2023-11-15
+
+### Fixes
+
+- Update async-imap to 0.9.4 which does not ignore EOF on FETCH.
+- Reset gossiped timestamp on securejoin.
+- sync: Ignore unknown sync items to provide forward compatibility and avoid creating empty message bubbles.
+- sync: Skip sync when chat name is set to the current one.
+- Return connectivity HTML with an error when IO is stopped.
+
 ## [1.131.2] - 2023-11-14
 
 ### API-Changes
@@ -3199,3 +3315,10 @@ https://github.com/deltachat/deltachat-core-rust/pulls?q=is%3Apr+is%3Aclosed
 [1.131.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.130.0...v1.131.0
 [1.131.1]: https://github.com/deltachat/deltachat-core-rust/compare/v1.131.0...v1.131.1
 [1.131.2]: https://github.com/deltachat/deltachat-core-rust/compare/v1.131.1...v1.131.2
+[1.131.3]: https://github.com/deltachat/deltachat-core-rust/compare/v1.131.2...v1.131.3
+[1.131.4]: https://github.com/deltachat/deltachat-core-rust/compare/v1.131.3...v1.131.4
+[1.131.5]: https://github.com/deltachat/deltachat-core-rust/compare/v1.131.4...v1.131.5
+[1.131.6]: https://github.com/deltachat/deltachat-core-rust/compare/v1.131.5...v1.131.6
+[1.131.7]: https://github.com/deltachat/deltachat-core-rust/compare/v1.131.6...v1.131.7
+[1.131.8]: https://github.com/deltachat/deltachat-core-rust/compare/v1.131.7...v1.131.8
+[1.131.9]: https://github.com/deltachat/deltachat-core-rust/compare/v1.131.8...v1.131.9

CONTRIBUTING.md

@@ -86,6 +86,17 @@ For example:
 .with_context(|| format!("Unable to trash message {msg_id}"))
 ```
 
+All errors should be handled in one of these ways:
+- With `if let Err() =` (incl. logging them into `warn!()`/`err!()`).
+- With `.log_err().ok()`.
+- Bubbled up with `?`.
+
+`backtrace` feature is enabled for `anyhow` crate
+and `debug = 1` option is set in the test profile.
+This allows to run `RUST_BACKTRACE=1 cargo test`
+and get a backtrace with line numbers in resultified tests
+which return `anyhow::Result`.
+
 ### Logging
 
 For logging, use `info!`, `warn!` and `error!` macros.

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat"
-version = "1.131.2"
+version = "1.131.9"
 edition = "2021"
 license = "MPL-2.0"
 rust-version = "1.70"
@@ -11,6 +11,10 @@ panic = 'abort'
 opt-level = 1
 
 [profile.test]
+# Make anyhow `backtrace` feature useful.
+# With `debug = 0` there are no line numbers in the backtrace
+# produced with RUST_BACKTRACE=1.
+debug = 1
 opt-level = 0
 
 # Always optimize dependencies.
@@ -26,6 +30,9 @@ opt-level = "z"
 codegen-units = 1
 strip = true
 
+[patch.crates-io]
+imap-proto = { git = "https://github.com/djc/tokio-imap.git", rev = "01ff256a7e42a9f7d2732706f8b71a16ce93427e" }
+
 [dependencies]
 deltachat_derive = { path = "./deltachat_derive" }
 format-flowed = { path = "./format-flowed" }
@@ -44,15 +51,15 @@ chrono = { version = "0.4", default-features=false, features = ["clock", "std"]
 email = { git = "https://github.com/deltachat/rust-email", branch = "master" }
 encoded-words = { git = "https://github.com/async-email/encoded-words", branch = "master" }
 escaper = "0.1"
-fast-socks5 = "0.8"
-fd-lock = "3.0.11"
+fast-socks5 = "0.9"
+fd-lock = "4"
 futures = "0.3"
 futures-lite = "2.0.0"
 hex = "0.4.0"
 hickory-resolver = "0.24"
 humansize = "2"
 image = { version = "0.24.7", default-features=false, features = ["gif", "jpeg", "ico", "png", "pnm", "webp", "bmp"] }
-iroh = { git = "https://github.com/deltachat/iroh", branch = "0.4-update-quic", default-features = false }
+iroh = { git = "https://github.com/n0-computer/iroh", branch = "maint-0.4", default-features = false }
 kamadak-exif = "0.5"
 lettre_email = { git = "https://github.com/deltachat/lettre", branch = "master" }
 libc = "0.2"
@@ -96,13 +103,14 @@ uuid = { version = "1", features = ["serde", "v4"] }
 
 [dev-dependencies]
 ansi_term = "0.12.0"
+anyhow = { version = "1", features = ["backtrace"] } # Enable `backtrace` feature in tests.
 criterion = { version = "0.5.1", features = ["async_tokio"] }
 futures-lite = "2.0.0"
 log = "0.4"
 pretty_env_logger = "0.5"
 proptest = { version = "1", default-features = false, features = ["std"] }
 tempfile = "3"
-testdir = "0.8.0"
+testdir = "0.9.0"
 tokio = { version = "1", features = ["parking_lot", "rt-multi-thread", "macros"] }
 pretty_assertions = "1.3.0"
 

deltachat-ffi/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat_ffi"
-version = "1.131.2"
+version = "1.131.9"
 description = "Deltachat FFI"
 edition = "2018"
 readme = "README.md"

deltachat-ffi/deltachat.h

@@ -25,7 +25,6 @@ typedef struct _dc_event     dc_event_t;
 typedef struct _dc_event_emitter dc_event_emitter_t;
 typedef struct _dc_jsonrpc_instance dc_jsonrpc_instance_t;
 typedef struct _dc_backup_provider dc_backup_provider_t;
-typedef struct _dc_http_response dc_http_response_t;
 
 // Alias for backwards compatibility, use dc_event_emitter_t instead.
 typedef struct _dc_event_emitter dc_accounts_event_emitter_t;
@@ -1112,6 +1111,7 @@ uint32_t dc_send_videochat_invitation (dc_context_t* context, uint32_t chat_id);
  * received overrides all previously received reactions. It is
  * possible to remove all reactions by sending an empty string.
  *
+ * @deprecated 2023-11-27, use jsonrpc method `send_reaction` instead
  * @memberof dc_context_t
  * @param context The context object.
  * @param msg_id ID of the message you react to.
@@ -1124,6 +1124,7 @@ uint32_t dc_send_reaction (dc_context_t* context, uint32_t msg_id, char *reactio
 /**
  * Get a structure with reactions to the message.
  *
+ * @deprecated 2023-11-27, use jsonrpc method `get_message_reactions` instead
  * @memberof dc_context_t
  * @param context The context object.
  * @param msg_id The message ID to get reactions for.
@@ -2561,7 +2562,7 @@ dc_lot_t*       dc_check_qr                  (dc_context_t* context, const char*
  *     the Verified-Group-Invite protocol is offered in the QR code;
  *     works for protected groups as well as for normal groups.
  *     If set to 0, the Setup-Contact protocol is offered in the QR code.
- *     See https://countermitm.readthedocs.io/en/latest/new.html
+ *     See https://securejoin.readthedocs.io/en/latest/new.html
  *     for details about both protocols.
  * @return The text that should go to the QR code,
  *     On errors, an empty QR code is returned, NULL is never returned.
@@ -2597,7 +2598,7 @@ char*           dc_get_securejoin_qr_svg         (dc_context_t* context, uint32_
  *
  * Subsequent calls of dc_join_securejoin() will abort previous, unfinished handshakes.
  *
- * See https://countermitm.readthedocs.io/en/latest/new.html
+ * See https://securejoin.readthedocs.io/en/latest/new.html
  * for details about both protocols.
  *
  * @memberof dc_context_t
@@ -4580,15 +4581,18 @@ int dc_msg_has_html (dc_msg_t* msg);
   * if they are larger than the limit set by the dc_set_config()-option `download_limit`.
   *
   * The function returns one of:
-  * - @ref DC_DOWNLOAD_DONE        - The message does not need any further download action
-  *                                  and should be rendered as usual.
-  * - @ref DC_DOWNLOAD_AVAILABLE   - There is additional content to download.
-  *                                  In addition to the usual message rendering,
-  *                                  the UI shall show a download button that calls dc_download_full_msg()
-  * - @ref DC_DOWNLOAD_IN_PROGRESS - Download was started with dc_download_full_msg() and is still in progress.
-  *                                  If the download fails or succeeds,
-  *                                  the event @ref DC_EVENT_MSGS_CHANGED is emitted.
-  * - @ref DC_DOWNLOAD_FAILURE     - Download error, the user may start over calling dc_download_full_msg() again.
+  * - @ref DC_DOWNLOAD_DONE           - The message does not need any further download action
+  *                                     and should be rendered as usual.
+  * - @ref DC_DOWNLOAD_AVAILABLE      - There is additional content to download.
+  *                                     In addition to the usual message rendering,
+  *                                     the UI shall show a download button that calls dc_download_full_msg()
+  * - @ref DC_DOWNLOAD_IN_PROGRESS    - Download was started with dc_download_full_msg() and is still in progress.
+  *                                     If the download fails or succeeds,
+  *                                     the event @ref DC_EVENT_MSGS_CHANGED is emitted.
+  *
+  * - @ref DC_DOWNLOAD_UNDECIPHERABLE - The message does not need any futher download action.
+  *                                     It was fully downloaded, but we failed to decrypt it.
+  * - @ref DC_DOWNLOAD_FAILURE        - Download error, the user may start over calling dc_download_full_msg() again.
   *
   * @memberof dc_msg_t
   * @param msg The message object.
@@ -5182,72 +5186,6 @@ int             dc_provider_get_status                (const dc_provider_t* prov
 void            dc_provider_unref                     (dc_provider_t* provider);
 
 
-/**
- * Return an HTTP(S) GET response.
- * This function can be used to download remote content for HTML emails.
- *
- * @memberof dc_context_t
- * @param context The context object to take proxy settings from.
- * @param url HTTP or HTTPS URL.
- * @return The response must be released using dc_http_response_unref() after usage.
- *     NULL is returned on errors.
- */
-dc_http_response_t*     dc_get_http_response      (const dc_context_t* context, const char* url);
-
-
-/**
- * @class dc_http_response_t
- *
- * An object containing an HTTP(S) GET response.
- * Created by dc_get_http_response().
- */
-
-
-/**
- * Returns HTTP response MIME type as a string, e.g. "text/plain" or "text/html".
- *
- * @memberof dc_http_response_t
- * @param response HTTP response as returned by dc_get_http_response().
- * @return The string which must be released using dc_str_unref() after usage. May be NULL.
- */
-char*                   dc_http_response_get_mimetype (const dc_http_response_t* response);
-
-/**
- * Returns HTTP response encoding, e.g. "utf-8".
- *
- * @memberof dc_http_response_t
- * @param response HTTP response as returned by dc_get_http_response().
- * @return The string which must be released using dc_str_unref() after usage. May be NULL.
- */
-char*                   dc_http_response_get_encoding (const dc_http_response_t* response);
-
-/**
- * Returns HTTP response contents.
- *
- * @memberof dc_http_response_t
- * @param response HTTP response as returned by dc_get_http_response().
- * @return The blob which must be released using dc_str_unref() after usage. NULL is never returned.
- */
-uint8_t*                dc_http_response_get_blob     (const dc_http_response_t* response);
-
-/**
- * Returns HTTP response content size.
- *
- * @memberof dc_http_response_t
- * @param response HTTP response as returned by dc_get_http_response().
- * @return The blob size.
- */
-size_t                  dc_http_response_get_size     (const dc_http_response_t* response);
-
-/**
- * Free an HTTP response object.
- *
- * @memberof dc_http_response_t
- * @param response HTTP response as returned by dc_get_http_response().
- */
-void                    dc_http_response_unref        (const dc_http_response_t* response);
-
-
 /**
  * @class dc_lot_t
  *
@@ -5347,6 +5285,7 @@ int64_t         dc_lot_get_timestamp     (const dc_lot_t* lot);
 
 /**
  * @class dc_reactions_t
+ * @deprecated 2023-11-27, use jsonrpc method `get_message_reactions` instead
  *
  * An object representing all reactions for a single message.
  */
@@ -5354,6 +5293,7 @@ int64_t         dc_lot_get_timestamp     (const dc_lot_t* lot);
 /**
  * Returns array of contacts which reacted to the given message.
  *
+ * @deprecated 2023-11-27, use jsonrpc method `get_message_reactions` instead
  * @memberof dc_reactions_t
  * @param reactions The object containing message reactions.
  * @return array of contact IDs. Use dc_array_get_cnt() to get array length and
@@ -5365,6 +5305,7 @@ dc_array_t*     dc_reactions_get_contacts(dc_reactions_t* reactions);
 /**
  * Returns a string containing space-separated reactions of a single contact.
  *
+ * @deprecated 2023-11-27, use jsonrpc method `get_message_reactions` instead
  * @memberof dc_reactions_t
  * @param reactions The object containing message reactions.
  * @param contact_id ID of the contact.
@@ -5380,6 +5321,7 @@ char*           dc_reactions_get_by_contact_id(dc_reactions_t* reactions, uint32
  *
  * Reactions objects are created by dc_get_msg_reactions().
  *
+ * @deprecated 2023-11-27
  * @memberof dc_reactions_t
  * @param reactions The object to free.
  *     If NULL is given, nothing is done.
@@ -6433,22 +6375,27 @@ void dc_event_unref(dc_event_t* event);
 /**
  * Download not needed, see dc_msg_get_download_state() for details.
  */
-#define DC_DOWNLOAD_DONE          0
+#define DC_DOWNLOAD_DONE           0
 
 /**
  * Download available, see dc_msg_get_download_state() for details.
  */
-#define DC_DOWNLOAD_AVAILABLE    10
+#define DC_DOWNLOAD_AVAILABLE      10
 
 /**
  * Download failed, see dc_msg_get_download_state() for details.
  */
-#define DC_DOWNLOAD_FAILURE      20
+#define DC_DOWNLOAD_FAILURE        20
+
+/**
+ * Download not needed, see dc_msg_get_download_state() for details.
+ */
+#define DC_DOWNLOAD_UNDECIPHERABLE 30
 
 /**
  * Download in progress, see dc_msg_get_download_state() for details.
  */
-#define DC_DOWNLOAD_IN_PROGRESS  1000
+#define DC_DOWNLOAD_IN_PROGRESS    1000
 
 
 
@@ -6613,7 +6560,7 @@ void dc_event_unref(dc_event_t* event);
 /// - %1$s will be replaced by the name of the verified contact
 #define DC_STR_CONTACT_VERIFIED           35
 
-/// "Cannot verify %1$s."
+/// "Cannot establish guaranteed end-to-end encryption with %1$s."
 ///
 /// Used in status messages.
 /// - %1$s will be replaced by the name of the contact that cannot be verified

deltachat-ffi/src/lib.rs

@@ -31,7 +31,6 @@ use deltachat::ephemeral::Timer as EphemeralTimer;
 use deltachat::imex::BackupProvider;
 use deltachat::key::preconfigure_keypair;
 use deltachat::message::MsgId;
-use deltachat::net::read_url_blob;
 use deltachat::qr_code_generator::{generate_backup_qr, get_securejoin_qr_svg};
 use deltachat::reaction::{get_msg_reactions, send_reaction, Reactions};
 use deltachat::stock_str::StockMessage;
@@ -4113,10 +4112,17 @@ pub unsafe extern "C" fn dc_contact_is_verified(contact: *mut dc_contact_t) -> l
     let ffi_contact = &*contact;
     let ctx = &*ffi_contact.context;
 
-    block_on(ffi_contact.contact.is_verified(ctx))
+    if block_on(ffi_contact.contact.is_verified(ctx))
         .context("is_verified failed")
         .log_err(ctx)
-        .unwrap_or_default() as libc::c_int
+        .unwrap_or_default()
+    {
+        // Return value is essentially a boolean,
+        // but we return 2 for true for backwards compatibility.
+        2
+    } else {
+        0
+    }
 }
 
 #[no_mangle]
@@ -4590,96 +4596,6 @@ pub unsafe extern "C" fn dc_provider_unref(provider: *mut dc_provider_t) {
     // this may change once we start localizing string.
 }
 
-// dc_http_response_t
-
-pub type dc_http_response_t = net::HttpResponse;
-
-#[no_mangle]
-pub unsafe extern "C" fn dc_get_http_response(
-    context: *const dc_context_t,
-    url: *const libc::c_char,
-) -> *mut dc_http_response_t {
-    if context.is_null() || url.is_null() {
-        eprintln!("ignoring careless call to dc_get_http_response()");
-        return ptr::null_mut();
-    }
-
-    let context = &*context;
-    let url = to_string_lossy(url);
-    if let Ok(response) = block_on(read_url_blob(context, &url))
-        .context("read_url_blob")
-        .log_err(context)
-    {
-        Box::into_raw(Box::new(response))
-    } else {
-        ptr::null_mut()
-    }
-}
-
-#[no_mangle]
-pub unsafe extern "C" fn dc_http_response_get_mimetype(
-    response: *const dc_http_response_t,
-) -> *mut libc::c_char {
-    if response.is_null() {
-        eprintln!("ignoring careless call to dc_http_response_get_mimetype()");
-        return ptr::null_mut();
-    }
-
-    let response = &*response;
-    response.mimetype.strdup()
-}
-
-#[no_mangle]
-pub unsafe extern "C" fn dc_http_response_get_encoding(
-    response: *const dc_http_response_t,
-) -> *mut libc::c_char {
-    if response.is_null() {
-        eprintln!("ignoring careless call to dc_http_response_get_encoding()");
-        return ptr::null_mut();
-    }
-
-    let response = &*response;
-    response.encoding.strdup()
-}
-
-#[no_mangle]
-pub unsafe extern "C" fn dc_http_response_get_blob(
-    response: *const dc_http_response_t,
-) -> *mut libc::c_char {
-    if response.is_null() {
-        eprintln!("ignoring careless call to dc_http_response_get_blob()");
-        return ptr::null_mut();
-    }
-
-    let response = &*response;
-    let blob_len = response.blob.len();
-    let ptr = libc::malloc(blob_len);
-    libc::memcpy(ptr, response.blob.as_ptr() as *mut libc::c_void, blob_len);
-    ptr as *mut libc::c_char
-}
-
-#[no_mangle]
-pub unsafe extern "C" fn dc_http_response_get_size(
-    response: *const dc_http_response_t,
-) -> libc::size_t {
-    if response.is_null() {
-        eprintln!("ignoring careless call to dc_http_response_get_size()");
-        return 0;
-    }
-
-    let response = &*response;
-    response.blob.len()
-}
-
-#[no_mangle]
-pub unsafe extern "C" fn dc_http_response_unref(response: *mut dc_http_response_t) {
-    if response.is_null() {
-        eprintln!("ignoring careless call to dc_http_response_unref()");
-        return;
-    }
-    drop(Box::from_raw(response));
-}
-
 // -- Accounts
 
 /// Reader-writer lock wrapper for accounts manager to guarantee thread safety when using

deltachat-jsonrpc/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-jsonrpc"
-version = "1.131.2"
+version = "1.131.9"
 description = "DeltaChat JSON-RPC API"
 edition = "2021"
 default-run = "deltachat-jsonrpc-server"
@@ -30,7 +30,7 @@ walkdir = "2.3.3"
 base64 = "0.21"
 
 # optional dependencies
-axum = { version = "0.6.20", optional = true, features = ["ws"] }
+axum = { version = "0.7", optional = true, features = ["ws"] }
 env_logger = { version = "0.10.0", optional = true }
 
 [dev-dependencies]

deltachat-jsonrpc/src/api.rs

@@ -678,7 +678,7 @@ impl CommandApi {
     ///     the Verified-Group-Invite protocol is offered in the QR code;
     ///     works for protected groups as well as for normal groups.
     ///     If not set, the Setup-Contact protocol is offered in the QR code.
-    ///     See https://countermitm.readthedocs.io/en/latest/new.html
+    ///     See https://securejoin.readthedocs.io/en/latest/new.html
     ///     for details about both protocols.
     ///
     /// return format: `[code, svg]`
@@ -707,7 +707,7 @@ impl CommandApi {
     ///
     /// Subsequent calls of `secure_join()` will abort previous, unfinished handshakes.
     ///
-    /// See https://countermitm.readthedocs.io/en/latest/new.html
+    /// See https://securejoin.readthedocs.io/en/latest/new.html
     /// for details about both protocols.
     ///
     /// **qr**: The text of the scanned QR code. Typically, the same string as given

deltachat-jsonrpc/src/api/types/contact.rs

@@ -1,5 +1,4 @@
 use anyhow::Result;
-use deltachat::contact::VerifiedStatus;
 use deltachat::context::Context;
 use serde::Serialize;
 use typescript_type_def::TypeDef;
@@ -57,7 +56,7 @@ impl ContactObject {
             Some(path_buf) => path_buf.to_str().map(|s| s.to_owned()),
             None => None,
         };
-        let is_verified = contact.is_verified(context).await? == VerifiedStatus::BidirectVerified;
+        let is_verified = contact.is_verified(context).await?;
         let is_profile_verified = contact.is_profile_verified(context).await?;
 
         let verifier_id = contact

deltachat-jsonrpc/src/webserver.rs

@@ -33,10 +33,8 @@ async fn main() -> Result<(), std::io::Error> {
 
     let addr = SocketAddr::from(([127, 0, 0, 1], port));
     log::info!("JSON-RPC WebSocket server listening on {}", addr);
-    axum::Server::bind(&addr)
-        .serve(app.into_make_service())
-        .await
-        .unwrap();
+    let listener = tokio::net::TcpListener::bind(addr).await.unwrap();
+    axum::serve(listener, app).await.unwrap();
 
     Ok(())
 }

deltachat-jsonrpc/typescript/package.json

@@ -53,5 +53,5 @@
   },
   "type": "module",
   "types": "dist/deltachat.d.ts",
-  "version": "1.131.2"
+  "version": "1.131.9"
 }

deltachat-repl/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-repl"
-version = "1.131.2"
+version = "1.131.9"
 license = "MPL-2.0"
 edition = "2021"
 

deltachat-repl/src/cmdline.rs

@@ -284,13 +284,8 @@ async fn log_contactlist(context: &Context, contacts: &[ContactId]) -> Result<()
         let contact = Contact::get_by_id(context, *contact_id).await?;
         let name = contact.get_display_name();
         let addr = contact.get_addr();
-        let verified_state = contact.is_verified(context).await?;
-        let verified_str = if VerifiedStatus::Unverified != verified_state {
-            if verified_state == VerifiedStatus::BidirectVerified {
-                " √√"
-            } else {
-                " √"
-            }
+        let verified_str = if contact.is_verified(context).await? {
+            " √"
         } else {
             ""
         };

deltachat-rpc-client/examples/echobot_no_hooks.py

@@ -40,13 +40,13 @@ def main():
 
         while True:
             event = account.wait_for_event()
-            if event["type"] == EventType.INFO:
+            if event["kind"] == EventType.INFO:
                 logging.info("%s", event["msg"])
-            elif event["type"] == EventType.WARNING:
+            elif event["kind"] == EventType.WARNING:
                 logging.warning("%s", event["msg"])
-            elif event["type"] == EventType.ERROR:
+            elif event["kind"] == EventType.ERROR:
                 logging.error("%s", event["msg"])
-            elif event["type"] == EventType.INCOMING_MSG:
+            elif event["kind"] == EventType.INCOMING_MSG:
                 logging.info("Got an incoming message")
                 process_messages()
 

deltachat-rpc-client/src/deltachat_rpc_client/account.py

@@ -218,7 +218,7 @@ class Account:
         The function returns immediately and the handshake runs in background, sending
         and receiving several messages.
         Subsequent calls of `secure_join()` will abort previous, unfinished handshakes.
-        See https://countermitm.readthedocs.io/en/latest/new.html for protocol details.
+        See https://securejoin.readthedocs.io/en/latest/new.html for protocol details.
 
         :param qrdata: The text of the scanned QR code.
         """
@@ -271,6 +271,18 @@ class Account:
             if event.kind == EventType.INCOMING_MSG:
                 return event
 
+    def wait_for_securejoin_inviter_success(self):
+        while True:
+            event = self.wait_for_event()
+            if event["kind"] == "SecurejoinInviterProgress" and event["progress"] == 1000:
+                break
+
+    def wait_for_securejoin_joiner_success(self):
+        while True:
+            event = self.wait_for_event()
+            if event["kind"] == "SecurejoinJoinerProgress" and event["progress"] == 1000:
+                break
+
     def get_fresh_messages_in_arrival_order(self) -> List[Message]:
         """Return fresh messages list sorted in the order of their arrival, with ascending IDs."""
         warn(

deltachat-rpc-client/src/deltachat_rpc_client/client.py

@@ -195,7 +195,7 @@ class Client:
 
 
 class Bot(Client):
-    """Simple bot implementation that listent to events of a single account."""
+    """Simple bot implementation that listens to events of a single account."""
 
     def configure(self, email: str, password: str, **kwargs) -> None:
         kwargs.setdefault("bot", "1")

deltachat-rpc-client/src/deltachat_rpc_client/message.py

@@ -42,6 +42,10 @@ class Message:
             return AttrDict(reactions)
         return None
 
+    def get_sender_contact(self) -> Contact:
+        from_id = self.get_snapshot().from_id
+        return self.account.get_contact_by_id(from_id)
+
     def mark_seen(self) -> None:
         """Mark the message as seen."""
         self._rpc.markseen_msgs(self.account.id, [self.id])

deltachat-rpc-client/tests/test_securejoin.py

@@ -9,20 +9,14 @@ def test_qr_setup_contact(acfactory) -> None:
     qr_code, _svg = alice.get_qr_code()
     bob.secure_join(qr_code)
 
-    while True:
-        event = alice.wait_for_event()
-        if event["kind"] == "SecurejoinInviterProgress" and event["progress"] == 1000:
-            break
+    alice.wait_for_securejoin_inviter_success()
 
     # Test that Alice verified Bob's profile.
     alice_contact_bob = alice.get_contact_by_addr(bob.get_config("addr"))
     alice_contact_bob_snapshot = alice_contact_bob.get_snapshot()
     assert alice_contact_bob_snapshot.is_verified
 
-    while True:
-        event = bob.wait_for_event()
-        if event["kind"] == "SecurejoinJoinerProgress" and event["progress"] == 1000:
-            break
+    bob.wait_for_securejoin_joiner_success()
 
     # Test that Bob verified Alice's profile.
     bob_contact_alice = bob.get_contact_by_addr(alice.get_config("addr"))
@@ -35,24 +29,31 @@ def test_qr_securejoin(acfactory):
 
     logging.info("Alice creates a verified group")
     alice_chat = alice.create_group("Verified group", protect=True)
+    assert alice_chat.get_basic_snapshot().is_protected
 
     logging.info("Bob joins verified group")
     qr_code, _svg = alice_chat.get_qr_code()
     bob.secure_join(qr_code)
-    while True:
-        event = alice.wait_for_event()
-        if event.kind == "SecurejoinInviterProgress" and event["progress"] == 1000:
-            break
+
+    # Check that at least some of the handshake messages are deleted.
+    for ac in [alice, bob]:
+        while True:
+            event = ac.wait_for_event()
+            if event["kind"] == "ImapMessageDeleted":
+                break
+
+    alice.wait_for_securejoin_inviter_success()
 
     # Test that Alice verified Bob's profile.
     alice_contact_bob = alice.get_contact_by_addr(bob.get_config("addr"))
     alice_contact_bob_snapshot = alice_contact_bob.get_snapshot()
     assert alice_contact_bob_snapshot.is_verified
 
-    while True:
-        event = bob.wait_for_event()
-        if event["kind"] == "SecurejoinJoinerProgress" and event["progress"] == 1000:
-            break
+    bob.wait_for_securejoin_joiner_success()
+
+    snapshot = bob.get_message_by_id(bob.wait_for_incoming_msg_event().msg_id).get_snapshot()
+    assert snapshot.text == "Member Me ({}) added by {}.".format(bob.get_config("addr"), alice.get_config("addr"))
+    assert snapshot.chat.get_basic_snapshot().is_protected
 
     # Test that Bob verified Alice's profile.
     bob_contact_alice = bob.get_contact_by_addr(alice.get_config("addr"))
@@ -97,10 +98,7 @@ def test_qr_readreceipt(acfactory) -> None:
     charlie.secure_join(qr_code)
 
     for joiner in [bob, charlie]:
-        while True:
-            event = joiner.wait_for_event()
-            if event["kind"] == "SecurejoinJoinerProgress" and event["progress"] == 1000:
-                break
+        joiner.wait_for_securejoin_joiner_success()
 
     logging.info("Alice creates a verified group")
     group = alice.create_group("Group", protect=True)
@@ -152,7 +150,8 @@ def test_qr_readreceipt(acfactory) -> None:
     assert not bob.get_chat_by_contact(bob_contact_charlie)
 
 
-def test_verified_group_recovery(acfactory, rpc) -> None:
+def test_verified_group_recovery(acfactory) -> None:
+    """Tests verified group recovery by reverifying a member and sending a message in a group."""
     ac1, ac2, ac3 = acfactory.get_online_accounts(3)
 
     logging.info("ac1 creates verified group")
@@ -162,10 +161,7 @@ def test_verified_group_recovery(acfactory, rpc) -> None:
     logging.info("ac2 joins verified group")
     qr_code, _svg = chat.get_qr_code()
     ac2.secure_join(qr_code)
-    while True:
-        event = ac1.wait_for_event()
-        if event.kind == "SecurejoinInviterProgress" and event["progress"] == 1000:
-            break
+    ac1.wait_for_securejoin_inviter_success()
 
     # ac1 has ac2 directly verified.
     ac1_contact_ac2 = ac1.get_contact_by_addr(ac2.get_config("addr"))
@@ -173,10 +169,7 @@ def test_verified_group_recovery(acfactory, rpc) -> None:
 
     logging.info("ac3 joins verified group")
     ac3_chat = ac3.secure_join(qr_code)
-    while True:
-        event = ac1.wait_for_event()
-        if event.kind == "SecurejoinInviterProgress" and event["progress"] == 1000:
-            break
+    ac1.wait_for_securejoin_inviter_success()
 
     logging.info("ac2 logs in on a new device")
     ac2 = acfactory.resetup_account(ac2)
@@ -185,69 +178,44 @@ def test_verified_group_recovery(acfactory, rpc) -> None:
     qr_code, _svg = ac3.get_qr_code()
     ac2.secure_join(qr_code)
 
-    while True:
-        event = ac3.wait_for_event()
-        if event.kind == "SecurejoinInviterProgress" and event["progress"] == 1000:
-            break
+    ac3.wait_for_securejoin_inviter_success()
 
     logging.info("ac3 sends a message to the group")
     assert len(ac3_chat.get_contacts()) == 3
     ac3_chat.send_text("Hi!")
 
-    msg_id = ac2.wait_for_incoming_msg_event().msg_id
-    message = ac2.get_message_by_id(msg_id)
-    snapshot = message.get_snapshot()
-    logging.info("Received message %s", snapshot.text)
+    snapshot = ac1.get_message_by_id(ac1.wait_for_incoming_msg_event().msg_id).get_snapshot()
     assert snapshot.text == "Hi!"
 
-    # ac1 contact cannot be verified by ac2 because ac3 did not gossip ac1 key in the "Hi!" message.
-    ac1_contact = ac2.get_contact_by_addr(ac1.get_config("addr"))
-    assert not ac1_contact.get_snapshot().is_verified
-
-    ac3_contact_id_ac1 = rpc.lookup_contact_id_by_addr(ac3.id, ac1.get_config("addr"))
-    ac3_chat.remove_contact(ac3_contact_id_ac1)
-    ac3_chat.add_contact(ac3_contact_id_ac1)
-
     msg_id = ac2.wait_for_incoming_msg_event().msg_id
     message = ac2.get_message_by_id(msg_id)
     snapshot = message.get_snapshot()
-    logging.info("ac2 got event message: %s", snapshot.text)
-    assert "removed" in snapshot.text
-
-    event = ac2.wait_for_incoming_msg_event()
-    msg_id = event.msg_id
-    chat_id = event.chat_id
-    message = ac2.get_message_by_id(msg_id)
-    snapshot = message.get_snapshot()
-    logging.info("ac2 got event message: %s", snapshot.text)
-    assert "added" in snapshot.text
+    assert snapshot.text == "Hi!"
 
+    # ac1 contact is verified for ac2 because ac3 gossiped ac1 key in the "Hi!" message.
+    ac1_contact = ac2.get_contact_by_addr(ac1.get_config("addr"))
     assert ac1_contact.get_snapshot().is_verified
 
-    chat = Chat(ac2, chat_id)
-    chat.send_text("Works again!")
+    # ac2 can write messages to the group.
+    snapshot.chat.send_text("Works again!")
 
-    msg_id = ac3.wait_for_incoming_msg_event().msg_id
-    message = ac3.get_message_by_id(msg_id)
-    snapshot = message.get_snapshot()
+    snapshot = ac3.get_message_by_id(ac3.wait_for_incoming_msg_event().msg_id).get_snapshot()
     assert snapshot.text == "Works again!"
 
-    ac1.wait_for_incoming_msg_event()  # Hi!
-    ac1.wait_for_incoming_msg_event()  # Member removed
-    ac1.wait_for_incoming_msg_event()  # Member added
     snapshot = ac1.get_message_by_id(ac1.wait_for_incoming_msg_event().msg_id).get_snapshot()
     assert snapshot.text == "Works again!"
 
-    # ac2 is now verified by ac3 for ac1
-    ac1_contact_ac3 = ac1.get_contact_by_addr(ac3.get_config("addr"))
-    assert ac1_contact_ac2.get_snapshot().verifier_id == ac1_contact_ac3.id
-
     ac1_chat_messages = snapshot.chat.get_messages()
     ac2_addr = ac2.get_config("addr")
     assert ac1_chat_messages[-2].get_snapshot().text == f"Changed setup for {ac2_addr}"
 
+    # ac2 is now verified by ac3 for ac1
+    ac1_contact_ac3 = ac1.get_contact_by_addr(ac3.get_config("addr"))
+    assert ac1_contact_ac2.get_snapshot().verifier_id == ac1_contact_ac3.id
+
 
 def test_verified_group_member_added_recovery(acfactory) -> None:
+    """Tests verified group recovery by reverifiying than removing and adding a member back."""
     ac1, ac2, ac3 = acfactory.get_online_accounts(3)
 
     logging.info("ac1 creates verified group")
@@ -257,10 +225,7 @@ def test_verified_group_member_added_recovery(acfactory) -> None:
     logging.info("ac2 joins verified group")
     qr_code, _svg = chat.get_qr_code()
     ac2.secure_join(qr_code)
-    while True:
-        event = ac1.wait_for_event()
-        if event.kind == "SecurejoinInviterProgress" and event["progress"] == 1000:
-            break
+    ac1.wait_for_securejoin_inviter_success()
 
     # ac1 has ac2 directly verified.
     ac1_contact_ac2 = ac1.get_contact_by_addr(ac2.get_config("addr"))
@@ -268,10 +233,7 @@ def test_verified_group_member_added_recovery(acfactory) -> None:
 
     logging.info("ac3 joins verified group")
     ac3_chat = ac3.secure_join(qr_code)
-    while True:
-        event = ac1.wait_for_event()
-        if event.kind == "SecurejoinInviterProgress" and event["progress"] == 1000:
-            break
+    ac1.wait_for_securejoin_inviter_success()
 
     logging.info("ac2 logs in on a new device")
     ac2 = acfactory.resetup_account(ac2)
@@ -280,10 +242,7 @@ def test_verified_group_member_added_recovery(acfactory) -> None:
     qr_code, _svg = ac3.get_qr_code()
     ac2.secure_join(qr_code)
 
-    while True:
-        event = ac3.wait_for_event()
-        if event.kind == "SecurejoinInviterProgress" and event["progress"] == 1000:
-            break
+    ac3.wait_for_securejoin_inviter_success()
 
     logging.info("ac3 sends a message to the group")
     assert len(ac3_chat.get_contacts()) == 3
@@ -339,3 +298,182 @@ def test_verified_group_member_added_recovery(acfactory) -> None:
     # ac2 is now verified by ac3 for ac1
     ac1_contact_ac3 = ac1.get_contact_by_addr(ac3.get_config("addr"))
     assert ac1_contact_ac2.get_snapshot().verifier_id == ac1_contact_ac3.id
+
+
+def test_qr_join_chat_with_pending_bobstate_issue4894(acfactory):
+    """Regression test for
+    issue <https://github.com/deltachat/deltachat-core-rust/issues/4894>.
+    """
+    ac1, ac2, ac3, ac4 = acfactory.get_online_accounts(4)
+
+    logging.info("ac3: verify with ac2")
+    qr_code, _svg = ac2.get_qr_code()
+    ac3.secure_join(qr_code)
+    ac2.wait_for_securejoin_inviter_success()
+
+    # in order for ac2 to have pending bobstate with a verified group
+    # we first create a fully joined verified group, and then start
+    # joining a second time but interrupt it, to create pending bob state
+
+    logging.info("ac1: create verified group that ac2 fully joins")
+    ch1 = ac1.create_group("Group", protect=True)
+    qr_code, _svg = ch1.get_qr_code()
+    ac2.secure_join(qr_code)
+    ac1.wait_for_securejoin_inviter_success()
+
+    # ensure ac1 can write and ac2 receives messages in verified chat
+    ch1.send_text("ac1 says hello")
+    while 1:
+        snapshot = ac2.get_message_by_id(ac2.wait_for_incoming_msg_event().msg_id).get_snapshot()
+        if snapshot.text == "ac1 says hello":
+            assert snapshot.chat.get_basic_snapshot().is_protected
+            break
+
+    logging.info("ac1: let ac2 join again but shutoff ac1 in the middle of securejoin")
+    qr_code, _svg = ch1.get_qr_code()
+    ac2.secure_join(qr_code)
+    ac1.remove()
+    logging.info("ac2 now has pending bobstate but ac1 is shutoff")
+
+    # we meanwhile expect ac3/ac2 verification started in the beginning to have completed
+    assert ac3.get_contact_by_addr(ac2.get_config("addr")).get_snapshot().is_verified
+    assert ac2.get_contact_by_addr(ac3.get_config("addr")).get_snapshot().is_verified
+
+    logging.info("ac3: create a verified group VG with ac2")
+    vg = ac3.create_group("ac3-created", protect=True)
+    vg.add_contact(ac3.get_contact_by_addr(ac2.get_config("addr")))
+
+    # ensure ac2 receives message in VG
+    vg.send_text("hello")
+    while 1:
+        msg = ac2.get_message_by_id(ac2.wait_for_incoming_msg_event().msg_id).get_snapshot()
+        if msg.text == "hello":
+            assert msg.chat.get_basic_snapshot().is_protected
+            break
+
+    logging.info("ac3: create a join-code for group VG and let ac4 join, check that ac2 got it")
+    qr_code, _svg = vg.get_qr_code()
+    ac4.secure_join(qr_code)
+    ac3.wait_for_securejoin_inviter_success()
+    while 1:
+        ev = ac2.wait_for_event()
+        if "added by unrelated SecureJoin" in str(ev):
+            return
+
+
+def test_qr_new_group_unblocked(acfactory):
+    """Regression test for a bug introduced in core v1.113.0.
+    ac2 scans a verified group QR code created by ac1.
+    This results in creation of a blocked 1:1 chat with ac1 on ac2,
+    but ac1 contact is not blocked on ac2.
+    Then ac1 creates a group, adds ac2 there and promotes it by sending a message.
+    ac2 should receive a message and create a contact request for the group.
+    Due to a bug previously ac2 created a blocked group.
+    """
+
+    ac1, ac2 = acfactory.get_online_accounts(2)
+    ac1_chat = ac1.create_group("Group for joining", protect=True)
+    qr_code, _svg = ac1_chat.get_qr_code()
+    ac2.secure_join(qr_code)
+
+    ac1.wait_for_securejoin_inviter_success()
+
+    ac1_new_chat = ac1.create_group("Another group")
+    ac1_new_chat.add_contact(ac1.get_contact_by_addr(ac2.get_config("addr")))
+    # Receive "Member added" message.
+    ac2.wait_for_incoming_msg_event()
+
+    ac1_new_chat.send_text("Hello!")
+    ac2_msg = ac2.get_message_by_id(ac2.wait_for_incoming_msg_event().msg_id).get_snapshot()
+    assert ac2_msg.text == "Hello!"
+    assert ac2_msg.chat.get_basic_snapshot().is_contact_request
+
+
+def test_aeap_flow_verified(acfactory):
+    """Test that a new address is added to a contact when it changes its address."""
+    ac1, ac2, ac1new = acfactory.get_online_accounts(3)
+
+    logging.info("ac1: create verified-group QR, ac2 scans and joins")
+    chat = ac1.create_group("hello", protect=True)
+    assert chat.get_basic_snapshot().is_protected
+    qr_code, _svg = chat.get_qr_code()
+    logging.info("ac2: start QR-code based join-group protocol")
+    ac2.secure_join(qr_code)
+    ac1.wait_for_securejoin_inviter_success()
+
+    logging.info("sending first message")
+    msg_out = chat.send_text("old address").get_snapshot()
+
+    logging.info("receiving first message")
+    ac2.wait_for_incoming_msg_event()  # member added message
+    msg_in_1 = ac2.get_message_by_id(ac2.wait_for_incoming_msg_event().msg_id).get_snapshot()
+    assert msg_in_1.text == msg_out.text
+
+    logging.info("changing email account")
+    ac1.set_config("addr", ac1new.get_config("addr"))
+    ac1.set_config("mail_pw", ac1new.get_config("mail_pw"))
+    ac1.stop_io()
+    ac1.configure()
+    ac1.start_io()
+
+    logging.info("sending second message")
+    msg_out = chat.send_text("changed address").get_snapshot()
+
+    logging.info("receiving second message")
+    msg_in_2 = ac2.get_message_by_id(ac2.wait_for_incoming_msg_event().msg_id)
+    msg_in_2_snapshot = msg_in_2.get_snapshot()
+    assert msg_in_2_snapshot.text == msg_out.text
+    assert msg_in_2_snapshot.chat.id == msg_in_1.chat.id
+    assert msg_in_2.get_sender_contact().get_snapshot().address == ac1new.get_config("addr")
+    assert len(msg_in_2_snapshot.chat.get_contacts()) == 2
+    assert ac1new.get_config("addr") in [
+        contact.get_snapshot().address for contact in msg_in_2_snapshot.chat.get_contacts()
+    ]
+
+
+def test_gossip_verification(acfactory) -> None:
+    alice, bob, carol = acfactory.get_online_accounts(3)
+
+    # Bob verifies Alice.
+    qr_code, _svg = alice.get_qr_code()
+    bob.secure_join(qr_code)
+    bob.wait_for_securejoin_joiner_success()
+
+    # Bob verifies Carol.
+    qr_code, _svg = carol.get_qr_code()
+    bob.secure_join(qr_code)
+    bob.wait_for_securejoin_joiner_success()
+
+    bob_contact_alice = bob.create_contact(alice.get_config("addr"), "Alice")
+    bob_contact_carol = bob.create_contact(carol.get_config("addr"), "Carol")
+    carol_contact_alice = carol.create_contact(alice.get_config("addr"), "Alice")
+
+    logging.info("Bob creates an Autocrypt group")
+    bob_group_chat = bob.create_group("Autocrypt Group")
+    assert not bob_group_chat.get_basic_snapshot().is_protected
+    bob_group_chat.add_contact(bob_contact_alice)
+    bob_group_chat.add_contact(bob_contact_carol)
+    bob_group_chat.send_message(text="Hello Autocrypt group")
+
+    snapshot = carol.get_message_by_id(carol.wait_for_incoming_msg_event().msg_id).get_snapshot()
+    assert snapshot.text == "Hello Autocrypt group"
+    assert snapshot.show_padlock
+
+    # Autocrypt group does not propagate verification.
+    carol_contact_alice_snapshot = carol_contact_alice.get_snapshot()
+    assert not carol_contact_alice_snapshot.is_verified
+
+    logging.info("Bob creates a Securejoin group")
+    bob_group_chat = bob.create_group("Securejoin Group", protect=True)
+    assert bob_group_chat.get_basic_snapshot().is_protected
+    bob_group_chat.add_contact(bob_contact_alice)
+    bob_group_chat.add_contact(bob_contact_carol)
+    bob_group_chat.send_message(text="Hello Securejoin group")
+
+    snapshot = carol.get_message_by_id(carol.wait_for_incoming_msg_event().msg_id).get_snapshot()
+    assert snapshot.text == "Hello Securejoin group"
+    assert snapshot.show_padlock
+
+    # Securejoin propagates verification.
+    carol_contact_alice_snapshot = carol_contact_alice.get_snapshot()
+    assert carol_contact_alice_snapshot.is_verified

deltachat-rpc-client/tests/test_webxdc.py

@@ -43,3 +43,15 @@ def test_webxdc(acfactory) -> None:
     assert status_updates == [
         {"payload": "Second update", "serial": 2, "max_serial": 2},
     ]
+
+
+def test_webxdc_insert_lots_of_updates(acfactory) -> None:
+    alice, bob = acfactory.get_online_accounts(2)
+
+    bob_addr = bob.get_config("addr")
+    alice_contact_bob = alice.create_contact(bob_addr, "Bob")
+    alice_chat_bob = alice_contact_bob.create_chat()
+    message = alice_chat_bob.send_message(text="Let's play chess!", file="../test-data/webxdc/chess.xdc")
+
+    for i in range(2000):
+        message.send_webxdc_status_update({"payload": str(i)}, "description")

deltachat-rpc-server/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-rpc-server"
-version = "1.131.2"
+version = "1.131.9"
 description = "DeltaChat JSON-RPC server"
 edition = "2021"
 readme = "README.md"

deny.toml

@@ -3,6 +3,13 @@ unmaintained = "allow"
 ignore = [
     "RUSTSEC-2020-0071",
     "RUSTSEC-2022-0093",
+
+    # Timing attack on RSA.
+    # Delta Chat does not use RSA for new keys
+    # and this requires precise measurement of the decryption time by the attacker.
+    # There is no fix at the time of writing this (2023-11-28).
+    # <https://rustsec.org/advisories/RUSTSEC-2023-0071>
+    "RUSTSEC-2023-0071",
 ]
 
 [bans]
@@ -26,9 +33,13 @@ skip = [
      { name = "ed25519-dalek", version = "1.0.1" },
      { name = "ed25519", version = "1.5.3" },
      { name = "event-listener", version = "2.5.3" },
+     { name = "fd-lock", version = "3.0.13" },
      { name = "getrandom", version = "<0.2" },
-     { name = "hashbrown", version = "<0.14.0" },
-     { name = "indexmap", version = "<2.0.0" },
+     { name = "h2", version = "0.3.22" },
+     { name = "http-body", version = "0.4.5" },
+     { name = "http", version = "0.2.11" },
+     { name = "hyper", version = "0.14.27" },
+     { name = "idna", version = "0.4.0" },
      { name = "pem-rfc7468", version = "0.6.0" },
      { name = "pkcs8", version = "0.9.0" },
      { name = "quick-error", version = "<2.0" },
@@ -49,12 +60,16 @@ skip = [
      { name = "time", version = "<0.3" },
      { name = "untrusted", version = "0.7.1" },
      { name = "wasi", version = "<0.11" },
-     { name = "windows_aarch64_msvc", version = "<0.48" },
-     { name = "windows_i686_gnu", version = "<0.48" },
-     { name = "windows_i686_msvc", version = "<0.48" },
+     { name = "windows_aarch64_gnullvm", version = "<0.52" },
+     { name = "windows_aarch64_msvc", version = "<0.52" },
+     { name = "windows_i686_gnu", version = "<0.52" },
+     { name = "windows_i686_msvc", version = "<0.52" },
+     { name = "windows-sys", version = "<0.52" },
+     { name = "windows-targets", version = "<0.52" },
      { name = "windows", version = "0.32.0" },
-     { name = "windows_x86_64_gnu", version = "<0.48" },
-     { name = "windows_x86_64_msvc", version = "<0.48" },
+     { name = "windows_x86_64_gnullvm", version = "<0.52" },
+     { name = "windows_x86_64_gnu", version = "<0.52" },
+     { name = "windows_x86_64_msvc", version = "<0.52" },
 ]
 
 
@@ -86,4 +101,6 @@ license-files = [
 github = [
        "async-email",
        "deltachat",
+       "djc",
+       "n0-computer", # iroh
 ]

fuzz/Cargo.lock

@@ -47,7 +47,7 @@ version = "0.7.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fcb51a0695d8f838b1ee009b3fbf66bda078cd64590202a864a8f3e8c4315c47"
 dependencies = [
- "getrandom 0.2.8",
+ "getrandom 0.2.11",
  "once_cell",
  "version_check",
 ]
@@ -158,10 +158,23 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cf46fee83e5ccffc220104713af3292ff9bc7c64c7de289f66dae8e38d826833"
 dependencies = [
  "concurrent-queue",
- "event-listener",
+ "event-listener 2.5.3",
  "futures-core",
 ]
 
+[[package]]
+name = "async-channel"
+version = "2.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1ca33f4bc4ed1babef42cad36cc1f51fa88be00420404e5b1e80ab1b18f7678c"
+dependencies = [
+ "concurrent-queue",
+ "event-listener 4.0.0",
+ "event-listener-strategy",
+ "futures-core",
+ "pin-project-lite",
+]
+
 [[package]]
 name = "async-compression"
 version = "0.3.15"
@@ -181,7 +194,7 @@ version = "0.9.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b538b767cbf9c162a6c5795d4b932bd2c20ba10b5a91a94d2b2b6886c1dce6a8"
 dependencies = [
- "async-channel",
+ "async-channel 1.8.0",
  "base64 0.21.0",
  "bytes",
  "chrono",
@@ -484,9 +497,9 @@ dependencies = [
 
 [[package]]
 name = "brotli"
-version = "3.3.4"
+version = "3.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a1a0b1dbcc8ae29329621f8d4f0d835787c1c38bb1401979b49d13b0b305ff68"
+checksum = "516074a47ef4bce09577a3b379392300159ce5b1ba2e501ff1c819950066100f"
 dependencies = [
  "alloc-no-stdlib",
  "alloc-stdlib",
@@ -495,9 +508,9 @@ dependencies = [
 
 [[package]]
 name = "brotli-decompressor"
-version = "2.3.4"
+version = "2.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4b6561fd3f895a11e8f72af2cb7d22e08366bebc2b6b57f7744c4bda27034744"
+checksum = "4e2e4afe60d7dd600fdd3de8d0f08c2b7ec039712e3b6137ff98b7004e82de4f"
 dependencies = [
  "alloc-no-stdlib",
  "alloc-stdlib",
@@ -568,9 +581,12 @@ dependencies = [
 
 [[package]]
 name = "cc"
-version = "1.0.78"
+version = "1.0.83"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a20104e2335ce8a659d6dd92a51a767a0c062599c73b343fd152cb401e828c3d"
+checksum = "f1174fb0b6ec23863f8b971027804a42614e347eafb0a95bf0b12cdae21fc4d0"
+dependencies = [
+ "libc",
+]
 
 [[package]]
 name = "cfb-mode"
@@ -646,9 +662,9 @@ checksum = "3d7b894f5411737b7867f4827955924d7c254fc9f4d91a6aad6b097804b1018b"
 
 [[package]]
 name = "concurrent-queue"
-version = "2.0.0"
+version = "2.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bd7bef69dc86e3c610e4e7aed41035e2a7ed12e72dd7530f61327a6579a4390b"
+checksum = "f057a694a54f12365049b0958a1685bb52d567f5593b355fbf685838e873d400"
 dependencies = [
  "crossbeam-utils",
 ]
@@ -906,10 +922,10 @@ dependencies = [
 
 [[package]]
 name = "deltachat"
-version = "1.123.0"
+version = "1.131.9"
 dependencies = [
  "anyhow",
- "async-channel",
+ "async-channel 2.1.1",
  "async-imap",
  "async-native-tls",
  "async-smtp",
@@ -928,6 +944,7 @@ dependencies = [
  "futures",
  "futures-lite",
  "hex",
+ "hickory-resolver",
  "humansize",
  "image",
  "iroh",
@@ -943,6 +960,7 @@ dependencies = [
  "parking_lot",
  "percent-encoding",
  "pgp",
+ "pin-project",
  "qrcodegen",
  "quick-xml",
  "rand 0.8.5",
@@ -968,7 +986,6 @@ dependencies = [
  "tokio-tar",
  "tokio-util",
  "toml",
- "trust-dns-resolver",
  "url",
  "uuid 1.2.2",
 ]
@@ -1434,6 +1451,12 @@ dependencies = [
  "syn 1.0.107",
 ]
 
+[[package]]
+name = "equivalent"
+version = "1.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5443807d6dff69373d433ab9ef5378ad8df50ca6298caf15de6e52e24aaf54d5"
+
 [[package]]
 name = "errno"
 version = "0.2.8"
@@ -1482,10 +1505,31 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0206175f82b8d6bf6652ff7d71a1e27fd2e4efde587fd368662814d6ec1d9ce0"
 
 [[package]]
-name = "fallible-iterator"
-version = "0.2.0"
+name = "event-listener"
+version = "4.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4443176a9f2c162692bd3d352d745ef9413eec5782a80d8fd6f8a1ac692a07f7"
+checksum = "770d968249b5d99410d61f5bf89057f3199a077a04d087092f58e7d10692baae"
+dependencies = [
+ "concurrent-queue",
+ "parking",
+ "pin-project-lite",
+]
+
+[[package]]
+name = "event-listener-strategy"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "958e4d70b6d5e81971bebec42271ec641e7ff4e170a6fa605f2b8a8b65cb97d3"
+dependencies = [
+ "event-listener 4.0.0",
+ "pin-project-lite",
+]
+
+[[package]]
+name = "fallible-iterator"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649"
 
 [[package]]
 name = "fallible-streaming-iterator"
@@ -1495,11 +1539,12 @@ checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
 
 [[package]]
 name = "fast-socks5"
-version = "0.8.1"
+version = "0.9.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d2687b5a6108f18ba8621e0e618a3be1dcc2768632dad24b7cea1f87975375a9"
+checksum = "d449e348301d5fb9b0e5781510d8235ffe3bbac3286bd305462736a9e7043039"
 dependencies = [
  "anyhow",
+ "async-trait",
  "log",
  "thiserror",
  "tokio",
@@ -1525,14 +1570,20 @@ dependencies = [
 ]
 
 [[package]]
-name = "fd-lock"
-version = "3.0.13"
+name = "fastrand"
+version = "2.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ef033ed5e9bad94e55838ca0ca906db0e043f517adda0c8b79c7a8c66c93c1b5"
+checksum = "25cbce373ec4653f1a01a31e8a5e5ec0c622dc27ff9c4e6606eefef5cbbed4a5"
+
+[[package]]
+name = "fd-lock"
+version = "4.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b93f7a0db71c99f68398f80653ed05afb0b00e062e1a20c7ff849c4edfabbbcc"
 dependencies = [
  "cfg-if",
  "rustix 0.38.14",
- "windows-sys 0.48.0",
+ "windows-sys 0.52.0",
 ]
 
 [[package]]
@@ -1593,7 +1644,7 @@ dependencies = [
  "futures-sink",
  "nanorand",
  "pin-project",
- "spin 0.9.6",
+ "spin 0.9.8",
 ]
 
 [[package]]
@@ -1680,17 +1731,16 @@ checksum = "00f5fb52a06bdcadeb54e8d3671f8888a39697dcb0b81b23b55174030427f4eb"
 
 [[package]]
 name = "futures-lite"
-version = "1.13.0"
+version = "2.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "49a9d51ce47660b1e808d3c990b4709f2f415d928835a17dfd16991515c46bce"
+checksum = "d3831c2651acb5177cbd83943f3d9c8912c5ad03c76afcc0e9511ba568ec5ebb"
 dependencies = [
- "fastrand",
+ "fastrand 2.0.1",
  "futures-core",
  "futures-io",
  "memchr",
  "parking",
  "pin-project-lite",
- "waker-fn",
 ]
 
 [[package]]
@@ -1758,9 +1808,9 @@ dependencies = [
 
 [[package]]
 name = "getrandom"
-version = "0.2.8"
+version = "0.2.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c05aeb6a22b8f62540c194aac980f2115af067bfe15a0734d7277a768d396b31"
+checksum = "fe9006bed769170c11f845cf00c7c1e9092aeb3f268e007c3e760ac68008070f"
 dependencies = [
  "cfg-if",
  "js-sys",
@@ -1819,7 +1869,7 @@ dependencies = [
  "futures-sink",
  "futures-util",
  "http",
- "indexmap",
+ "indexmap 1.9.2",
  "slab",
  "tokio",
  "tokio-util",
@@ -1835,13 +1885,19 @@ dependencies = [
  "ahash",
 ]
 
+[[package]]
+name = "hashbrown"
+version = "0.14.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "290f1a1d9242c78d09ce40a5e87e7554ee637af1351968159f4952f028f75604"
+
 [[package]]
 name = "hashlink"
 version = "0.8.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "69fe1fcf8b4278d860ad0548329f892a3631fb63f82574df68275f34cdbe0ffa"
 dependencies = [
- "hashbrown",
+ "hashbrown 0.12.3",
 ]
 
 [[package]]
@@ -1862,6 +1918,51 @@ version = "0.4.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
 
+[[package]]
+name = "hickory-proto"
+version = "0.24.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "091a6fbccf4860009355e3efc52ff4acf37a63489aad7435372d44ceeb6fbbcf"
+dependencies = [
+ "async-trait",
+ "cfg-if",
+ "data-encoding",
+ "enum-as-inner",
+ "futures-channel",
+ "futures-io",
+ "futures-util",
+ "idna",
+ "ipnet",
+ "once_cell",
+ "rand 0.8.5",
+ "thiserror",
+ "tinyvec",
+ "tokio",
+ "tracing",
+ "url",
+]
+
+[[package]]
+name = "hickory-resolver"
+version = "0.24.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "35b8f021164e6a984c9030023544c57789c51760065cd510572fedcfb04164e8"
+dependencies = [
+ "cfg-if",
+ "futures-util",
+ "hickory-proto",
+ "ipconfig",
+ "lru-cache",
+ "once_cell",
+ "parking_lot",
+ "rand 0.8.5",
+ "resolv-conf",
+ "smallvec",
+ "thiserror",
+ "tokio",
+ "tracing",
+]
+
 [[package]]
 name = "hkdf"
 version = "0.12.3"
@@ -2052,7 +2153,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "1885e79c1fc4b10f0e172c475f458b7f7b93061064d98c3293e98c5ba0c8b399"
 dependencies = [
  "autocfg",
- "hashbrown",
+ "hashbrown 0.12.3",
+]
+
+[[package]]
+name = "indexmap"
+version = "2.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d530e1a18b1cb4c484e6e34556a0d948706958449fca0cab753d649f2bce3d1f"
+dependencies = [
+ "equivalent",
+ "hashbrown 0.14.3",
 ]
 
 [[package]]
@@ -2103,9 +2214,8 @@ checksum = "11b0d96e660696543b251e58030cf9787df56da39dab19ad60eae7353040917e"
 
 [[package]]
 name = "iroh"
-version = "0.4.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e4fb9858c8cd3dd924a5da5bc511363845a9bcfdfac066bb2ef8454eb6111546"
+version = "0.4.2"
+source = "git+https://github.com/n0-computer/iroh?branch=maint-0.4#9881b7886235035a1124e4371f7a4cd59379e51b"
 dependencies = [
  "abao",
  "anyhow",
@@ -2126,8 +2236,9 @@ dependencies = [
  "quinn",
  "rand 0.7.3",
  "rcgen",
- "ring",
+ "ring 0.16.20",
  "rustls",
+ "rustls-webpki",
  "serde",
  "serde-error",
  "ssh-key",
@@ -2140,7 +2251,6 @@ dependencies = [
  "tracing-futures",
  "tracing-subscriber",
  "walkdir",
- "webpki",
  "x509-parser",
  "zeroize",
 ]
@@ -2219,9 +2329,9 @@ dependencies = [
 
 [[package]]
 name = "libc"
-version = "0.2.148"
+version = "0.2.150"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9cdc71e17332e86d2e1d38c1f99edcb6288ee11b815fb1a4b049eaa2114d369b"
+checksum = "89d92a4743f9a61002fae18374ed11e7973f530cb3a3255fb354818118b2203c"
 
 [[package]]
 name = "libm"
@@ -2237,9 +2347,9 @@ checksum = "348108ab3fba42ec82ff6e9564fc4ca0247bdccdc68dd8af9764bbc79c3c8ffb"
 
 [[package]]
 name = "libsqlite3-sys"
-version = "0.26.0"
+version = "0.27.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "afc22eff61b133b115c6e8c74e818c628d6d5e7a502afea6f64dee076dd94326"
+checksum = "cf4e226dcd58b4be396f7bd3c20da8fdee2911400705297ba7d2d7cc2c30f716"
 dependencies = [
  "cc",
  "openssl-sys",
@@ -2404,7 +2514,7 @@ version = "0.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "6a51313c5820b0b02bd422f4b44776fbf47961755c74ce64afc73bfad10226c3"
 dependencies = [
- "getrandom 0.2.8",
+ "getrandom 0.2.11",
 ]
 
 [[package]]
@@ -2484,15 +2594,6 @@ dependencies = [
  "minimal-lexical",
 ]
 
-[[package]]
-name = "nom8"
-version = "0.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ae01545c9c7fc4486ab7debaf2aad7003ac19431791868fb2e8066df97fad2f8"
-dependencies = [
- "memchr",
-]
-
 [[package]]
 name = "nu-ansi-term"
 version = "0.46.0"
@@ -2638,11 +2739,11 @@ checksum = "624a8340c38c1b80fd549087862da4ba43e08858af025b236e509b6649fc13d5"
 
 [[package]]
 name = "openssl"
-version = "0.10.55"
+version = "0.10.60"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "345df152bc43501c5eb9e4654ff05f794effb78d4efe3d53abc158baddc0703d"
+checksum = "79a4c6c3a2b158f7f8f2a2fc5a969fa3a068df6fc9dbb4a43845436e3af7c800"
 dependencies = [
- "bitflags 1.3.2",
+ "bitflags 2.4.0",
  "cfg-if",
  "foreign-types",
  "libc",
@@ -2670,18 +2771,18 @@ checksum = "ff011a302c396a5197692431fc1948019154afc178baf7d8e37367442a4601cf"
 
 [[package]]
 name = "openssl-src"
-version = "111.25.0+1.1.1t"
+version = "300.1.6+3.1.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3173cd3626c43e3854b1b727422a276e568d9ec5fe8cec197822cf52cfb743d6"
+checksum = "439fac53e092cd7442a3660c85dde4643ab3b5bd39040912388dcdabf6b88085"
 dependencies = [
  "cc",
 ]
 
 [[package]]
 name = "openssl-sys"
-version = "0.9.90"
+version = "0.9.96"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "374533b0e45f3a7ced10fcaeccca020e66656bc03dac384f852e4e5a7a8104a6"
+checksum = "3812c071ba60da8b5677cc12bcb1d42989a65553772897a7e0355545a819838f"
 dependencies = [
  "cc",
  "libc",
@@ -2754,9 +2855,9 @@ dependencies = [
 
 [[package]]
 name = "parking"
-version = "2.0.0"
+version = "2.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "427c3892f9e783d91cc128285287e70a59e206ca452770ece88a76f7a3eddd72"
+checksum = "bb813b8af86854136c6922af0598d719255ecb2179515e6e7730d468f05c9cae"
 
 [[package]]
 name = "parking_lot"
@@ -3062,11 +3163,12 @@ checksum = "4339fc7a1021c9c1621d87f5e3505f2805c8c105420ba2f2a4df86814590c142"
 
 [[package]]
 name = "quic-rpc"
-version = "0.5.2"
+version = "0.6.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d453504fc3e456160ae3b9ebe4d83c1f6477af167aa9b67e2d7bf11a096f179d"
+checksum = "6d60c2fc2390baad4b9d41ae9957ae88c3095496f88e252ef50722df8b5b78d7"
 dependencies = [
  "bincode",
+ "educe",
  "flume",
  "futures",
  "pin-project",
@@ -3086,18 +3188,18 @@ checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
 
 [[package]]
 name = "quick-xml"
-version = "0.30.0"
+version = "0.31.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "eff6510e86862b57b210fd8cbe8ed3f0d7d600b9c2863cd4549a2e033c66e956"
+checksum = "1004a344b30a54e2ee58d66a71b32d2db2feb0a31f9a2d302bf0536f15de2a33"
 dependencies = [
  "memchr",
 ]
 
 [[package]]
 name = "quinn"
-version = "0.9.3"
+version = "0.10.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "445cbfe2382fa023c4f2f3c7e1c95c03dcc1df2bf23cebcb2b13e1402c4394d1"
+checksum = "8cc2c5017e4b43d5995dcea317bc46c1e09404c0a9664d2908f7f02dfe943d75"
 dependencies = [
  "bytes",
  "pin-project-lite",
@@ -3108,18 +3210,17 @@ dependencies = [
  "thiserror",
  "tokio",
  "tracing",
- "webpki",
 ]
 
 [[package]]
 name = "quinn-proto"
-version = "0.9.5"
+version = "0.10.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c956be1b23f4261676aed05a0046e204e8a6836e50203902683a718af0797989"
+checksum = "141bf7dfde2fbc246bfd3fe12f2455aa24b0fbd9af535d8c86c7bd1381ff2b1a"
 dependencies = [
  "bytes",
  "rand 0.8.5",
- "ring",
+ "ring 0.16.20",
  "rustc-hash",
  "rustls",
  "rustls-native-certs",
@@ -3127,20 +3228,19 @@ dependencies = [
  "thiserror",
  "tinyvec",
  "tracing",
- "webpki",
 ]
 
 [[package]]
 name = "quinn-udp"
-version = "0.3.2"
+version = "0.4.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "641538578b21f5e5c8ea733b736895576d0fe329bb883b937db6f4d163dbaaf4"
+checksum = "055b4e778e8feb9f93c4e439f71dc2156ef13360b432b799e179a8c4cdf0b1d7"
 dependencies = [
+ "bytes",
  "libc",
- "quinn-proto",
- "socket2 0.4.7",
+ "socket2 0.5.4",
  "tracing",
- "windows-sys 0.42.0",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -3217,7 +3317,7 @@ version = "0.6.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
 dependencies = [
- "getrandom 0.2.8",
+ "getrandom 0.2.11",
 ]
 
 [[package]]
@@ -3240,7 +3340,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ffbe84efe2f38dea12e9bfc1f65377fdf03e53a18cb3b995faedf7934c7e785b"
 dependencies = [
  "pem",
- "ring",
+ "ring 0.16.20",
  "time 0.3.20",
  "yasna",
 ]
@@ -3260,7 +3360,7 @@ version = "0.4.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b033d837a7cf162d7993aded9304e30a83213c648b6e389db233191f891e5c2b"
 dependencies = [
- "getrandom 0.2.8",
+ "getrandom 0.2.11",
  "redox_syscall",
  "thiserror",
 ]
@@ -3387,11 +3487,25 @@ dependencies = [
  "libc",
  "once_cell",
  "spin 0.5.2",
- "untrusted",
+ "untrusted 0.7.1",
  "web-sys",
  "winapi",
 ]
 
+[[package]]
+name = "ring"
+version = "0.17.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "684d5e6e18f669ccebf64a92236bb7db9a34f07be010e3627368182027180866"
+dependencies = [
+ "cc",
+ "getrandom 0.2.11",
+ "libc",
+ "spin 0.9.8",
+ "untrusted 0.9.0",
+ "windows-sys 0.48.0",
+]
+
 [[package]]
 name = "ripemd"
 version = "0.1.3"
@@ -3445,9 +3559,9 @@ dependencies = [
 
 [[package]]
 name = "rusqlite"
-version = "0.29.0"
+version = "0.30.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "549b9d036d571d42e6e85d1c1425e2ac83491075078ca9a15be021c56b1641f2"
+checksum = "a78046161564f5e7cd9008aff3b2990b3850dc8e0349119b98e8f251e099f24d"
 dependencies = [
  "bitflags 2.4.0",
  "fallible-iterator",
@@ -3522,13 +3636,13 @@ dependencies = [
 
 [[package]]
 name = "rustls"
-version = "0.20.8"
+version = "0.21.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fff78fc74d175294f4e83b28343315ffcfb114b156f0185e9741cb5570f50e2f"
+checksum = "629648aced5775d558af50b2b4c7b02983a04b312126d45eeead26e7caa498b9"
 dependencies = [
- "ring",
+ "ring 0.17.6",
+ "rustls-webpki",
  "sct",
- "webpki",
 ]
 
 [[package]]
@@ -3552,6 +3666,16 @@ dependencies = [
  "base64 0.21.0",
 ]
 
+[[package]]
+name = "rustls-webpki"
+version = "0.101.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8b6275d1ee7a1cd780b64aca7726599a1dbc893b1e64144529e55c3c2f745765"
+dependencies = [
+ "ring 0.17.6",
+ "untrusted 0.9.0",
+]
+
 [[package]]
 name = "rustversion"
 version = "1.0.11"
@@ -3617,8 +3741,8 @@ version = "0.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d53dcdb7c9f8158937a7981b48accfd39a43af418591a5d008c7b22b5e1b7ca4"
 dependencies = [
- "ring",
- "untrusted",
+ "ring 0.16.20",
+ "untrusted 0.7.1",
 ]
 
 [[package]]
@@ -3674,9 +3798,9 @@ dependencies = [
 
 [[package]]
 name = "self_cell"
-version = "1.0.1"
+version = "1.0.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4c309e515543e67811222dbc9e3dd7e1056279b782e1dacffe4242b718734fb6"
+checksum = "e388332cd64eb80cd595a00941baf513caffae8dce9cfd0467fc9c66397dade6"
 
 [[package]]
 name = "semver"
@@ -3735,9 +3859,9 @@ dependencies = [
 
 [[package]]
 name = "serde_spanned"
-version = "0.6.1"
+version = "0.6.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0efd8caf556a6cebd3b285caf480045fcc1ac04f6bd786b09a6f11af30c4fcf4"
+checksum = "12022b835073e5b11e90a14f86838ceb1c8fb0325b72416845c487ac0fa95e80"
 dependencies = [
  "serde",
 ]
@@ -3897,9 +4021,9 @@ checksum = "6e63cff320ae2c57904679ba7cb63280a3dc4613885beafb148ee7bf9aa9042d"
 
 [[package]]
 name = "spin"
-version = "0.9.6"
+version = "0.9.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b5d6e0250b93c8427a177b849d144a96d5acc57006149479403d7861ab721e34"
+checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67"
 dependencies = [
  "lock_api",
 ]
@@ -3959,7 +4083,7 @@ version = "0.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "af91f480ee899ab2d9f8435bfdfc14d08a5754bd9d3fef1f1a1c23336aad6c8b"
 dependencies = [
- "async-channel",
+ "async-channel 1.8.0",
  "cfg-if",
  "futures-core",
  "pin-project-lite",
@@ -4064,7 +4188,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "af18f7ae1acd354b992402e9ec5864359d693cd8a79dcbef59f76891701c1e95"
 dependencies = [
  "cfg-if",
- "fastrand",
+ "fastrand 1.8.0",
  "redox_syscall",
  "rustix 0.36.7",
  "windows-sys 0.42.0",
@@ -4280,9 +4404,9 @@ dependencies = [
 
 [[package]]
 name = "toml"
-version = "0.7.2"
+version = "0.8.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f7afcae9e3f0fe2c370fd4657108972cbb2fa9db1b9f84849cefd80741b01cb6"
+checksum = "a1a195ec8c9da26928f773888e0742ca3ca1040c6cd859c919c9f59c1954ab35"
 dependencies = [
  "serde",
  "serde_spanned",
@@ -4292,24 +4416,24 @@ dependencies = [
 
 [[package]]
 name = "toml_datetime"
-version = "0.6.1"
+version = "0.6.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3ab8ed2edee10b50132aed5f331333428b011c99402b5a534154ed15746f9622"
+checksum = "3550f4e9685620ac18a50ed434eb3aec30db8ba93b0287467bca5826ea25baf1"
 dependencies = [
  "serde",
 ]
 
 [[package]]
 name = "toml_edit"
-version = "0.19.3"
+version = "0.21.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5e6a7712b49e1775fb9a7b998de6635b299237f48b404dde71704f2e0e7f37e5"
+checksum = "d34d383cd00a163b4a5b85053df514d45bc330f6de7737edfe0a93311d1eaa03"
 dependencies = [
- "indexmap",
- "nom8",
+ "indexmap 2.1.0",
  "serde",
  "serde_spanned",
  "toml_datetime",
+ "winnow",
 ]
 
 [[package]]
@@ -4325,6 +4449,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8ce8c33a8d48bd45d624a6e523445fd21ec13d3653cd51f681abf67418f54eb8"
 dependencies = [
  "cfg-if",
+ "log",
  "pin-project-lite",
  "tracing-attributes",
  "tracing-core",
@@ -4390,52 +4515,6 @@ dependencies = [
  "tracing-log",
 ]
 
-[[package]]
-name = "trust-dns-proto"
-version = "0.23.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0dc775440033cb114085f6f2437682b194fa7546466024b1037e82a48a052a69"
-dependencies = [
- "async-trait",
- "cfg-if",
- "data-encoding",
- "enum-as-inner",
- "futures-channel",
- "futures-io",
- "futures-util",
- "idna",
- "ipnet",
- "once_cell",
- "rand 0.8.5",
- "smallvec",
- "thiserror",
- "tinyvec",
- "tokio",
- "tracing",
- "url",
-]
-
-[[package]]
-name = "trust-dns-resolver"
-version = "0.23.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2dff7aed33ef3e8bf2c9966fccdfed93f93d46f432282ea875cd66faabc6ef2f"
-dependencies = [
- "cfg-if",
- "futures-util",
- "ipconfig",
- "lru-cache",
- "once_cell",
- "parking_lot",
- "rand 0.8.5",
- "resolv-conf",
- "smallvec",
- "thiserror",
- "tokio",
- "tracing",
- "trust-dns-proto",
-]
-
 [[package]]
 name = "try-lock"
 version = "0.2.3"
@@ -4475,7 +4554,7 @@ version = "0.1.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c5faade31a542b8b35855fff6e8def199853b2da8da256da52f52f1316ee3137"
 dependencies = [
- "hashbrown",
+ "hashbrown 0.12.3",
  "regex",
 ]
 
@@ -4506,6 +4585,12 @@ version = "0.7.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "a156c684c91ea7d62626509bce3cb4e1d9ed5c4d978f7b4352658f96a4c26b4a"
 
+[[package]]
+name = "untrusted"
+version = "0.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
+
 [[package]]
 name = "url"
 version = "2.4.1"
@@ -4523,7 +4608,7 @@ version = "0.8.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bc5cf98d8186244414c848017f0e2676b3fcb46807f6668a97dfe67359a3c4b7"
 dependencies = [
- "getrandom 0.2.8",
+ "getrandom 0.2.11",
 ]
 
 [[package]]
@@ -4532,7 +4617,7 @@ version = "1.2.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "422ee0de9031b5b948b97a8fc04e3aa35230001a722ddd27943e0be31564ce4c"
 dependencies = [
- "getrandom 0.2.8",
+ "getrandom 0.2.11",
  "serde",
 ]
 
@@ -4554,12 +4639,6 @@ version = "0.9.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "49874b5167b65d7193b8aba1567f5c7d93d001cafc34600cee003eda787e483f"
 
-[[package]]
-name = "waker-fn"
-version = "1.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9d5b2c62b4012a3e1eca5a7e077d13b3bf498c4073e33ccd58626607748ceeca"
-
 [[package]]
 name = "walkdir"
 version = "2.3.3"
@@ -4674,16 +4753,6 @@ dependencies = [
  "wasm-bindgen",
 ]
 
-[[package]]
-name = "webpki"
-version = "0.22.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "07ecc0cd7cac091bf682ec5efa18b1cff79d617b84181f38b3951dbe135f607f"
-dependencies = [
- "ring",
- "untrusted",
-]
-
 [[package]]
 name = "weezl"
 version = "0.1.7"
@@ -4774,7 +4843,16 @@ version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9"
 dependencies = [
- "windows-targets",
+ "windows-targets 0.48.5",
+]
+
+[[package]]
+name = "windows-sys"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
+dependencies = [
+ "windows-targets 0.52.0",
 ]
 
 [[package]]
@@ -4792,6 +4870,21 @@ dependencies = [
  "windows_x86_64_msvc 0.48.5",
 ]
 
+[[package]]
+name = "windows-targets"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8a18201040b24831fbb9e4eb208f8892e1f50a37feb53cc7ff887feb8f50e7cd"
+dependencies = [
+ "windows_aarch64_gnullvm 0.52.0",
+ "windows_aarch64_msvc 0.52.0",
+ "windows_i686_gnu 0.52.0",
+ "windows_i686_msvc 0.52.0",
+ "windows_x86_64_gnu 0.52.0",
+ "windows_x86_64_gnullvm 0.52.0",
+ "windows_x86_64_msvc 0.52.0",
+]
+
 [[package]]
 name = "windows_aarch64_gnullvm"
 version = "0.42.0"
@@ -4804,6 +4897,12 @@ version = "0.48.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8"
 
+[[package]]
+name = "windows_aarch64_gnullvm"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cb7764e35d4db8a7921e09562a0304bf2f93e0a51bfccee0bd0bb0b666b015ea"
+
 [[package]]
 name = "windows_aarch64_msvc"
 version = "0.32.0"
@@ -4828,6 +4927,12 @@ version = "0.48.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc"
 
+[[package]]
+name = "windows_aarch64_msvc"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bbaa0368d4f1d2aaefc55b6fcfee13f41544ddf36801e793edbbfd7d7df075ef"
+
 [[package]]
 name = "windows_i686_gnu"
 version = "0.32.0"
@@ -4852,6 +4957,12 @@ version = "0.48.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e"
 
+[[package]]
+name = "windows_i686_gnu"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a28637cb1fa3560a16915793afb20081aba2c92ee8af57b4d5f28e4b3e7df313"
+
 [[package]]
 name = "windows_i686_msvc"
 version = "0.32.0"
@@ -4876,6 +4987,12 @@ version = "0.48.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406"
 
+[[package]]
+name = "windows_i686_msvc"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ffe5e8e31046ce6230cc7215707b816e339ff4d4d67c65dffa206fd0f7aa7b9a"
+
 [[package]]
 name = "windows_x86_64_gnu"
 version = "0.32.0"
@@ -4900,6 +5017,12 @@ version = "0.48.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e"
 
+[[package]]
+name = "windows_x86_64_gnu"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3d6fa32db2bc4a2f5abeacf2b69f7992cd09dca97498da74a151a3132c26befd"
+
 [[package]]
 name = "windows_x86_64_gnullvm"
 version = "0.42.0"
@@ -4912,6 +5035,12 @@ version = "0.48.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc"
 
+[[package]]
+name = "windows_x86_64_gnullvm"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1a657e1e9d3f514745a572a6846d3c7aa7dbe1658c056ed9c3344c4109a6949e"
+
 [[package]]
 name = "windows_x86_64_msvc"
 version = "0.32.0"
@@ -4936,6 +5065,21 @@ version = "0.48.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538"
 
+[[package]]
+name = "windows_x86_64_msvc"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dff9641d1cd4be8d1a070daf9e3773c5f67e78b4d9d42263020c057706765c04"
+
+[[package]]
+name = "winnow"
+version = "0.5.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "829846f3e3db426d4cee4510841b71a8e58aa2a76b1132579487ae430ccd9c7b"
+dependencies = [
+ "memchr",
+]
+
 [[package]]
 name = "winreg"
 version = "0.10.1"

node/constants.js

@@ -28,6 +28,7 @@ module.exports = {
   DC_DOWNLOAD_DONE: 0,
   DC_DOWNLOAD_FAILURE: 20,
   DC_DOWNLOAD_IN_PROGRESS: 1000,
+  DC_DOWNLOAD_UNDECIPHERABLE: 30,
   DC_EVENT_CHAT_EPHEMERAL_TIMER_MODIFIED: 2021,
   DC_EVENT_CHAT_MODIFIED: 2020,
   DC_EVENT_CONFIGURE_PROGRESS: 2041,

node/lib/constants.ts

@@ -28,6 +28,7 @@ export enum C {
   DC_DOWNLOAD_DONE = 0,
   DC_DOWNLOAD_FAILURE = 20,
   DC_DOWNLOAD_IN_PROGRESS = 1000,
+  DC_DOWNLOAD_UNDECIPHERABLE = 30,
   DC_EVENT_CHAT_EPHEMERAL_TIMER_MODIFIED = 2021,
   DC_EVENT_CHAT_MODIFIED = 2020,
   DC_EVENT_CONFIGURE_PROGRESS = 2041,

package.json

@@ -56,5 +56,5 @@
     "test:mocha": "mocha -r esm node/test/test.js --growl --reporter=spec --bail --exit"
   },
   "types": "node/dist/index.d.ts",
-  "version": "1.131.2"
+  "version": "1.131.9"
 }

python/README.rst

@@ -1,6 +1,6 @@
-=========================
-DeltaChat Python bindings
-=========================
+============================
+CFFI Python Bindings
+============================
 
 This package provides `Python bindings`_ to the `deltachat-core library`_
 which implements IMAP/SMTP/MIME/OpenPGP e-mail standards and offers
@@ -8,157 +8,3 @@ a low-level Chat/Contact/Message API to user interfaces and bots.
 
 .. _`deltachat-core library`: https://github.com/deltachat/deltachat-core-rust
 .. _`Python bindings`: https://py.delta.chat/
-
-Installing pre-built packages (Linux-only)
-==========================================
-
-If you have a Linux system you may install the ``deltachat`` binary "wheel" packages
-without any "build-from-source" steps.
-Otherwise you need to `compile the Delta Chat bindings yourself`__.
-
-__ sourceinstall_
-
-We recommend to first create a fresh Python virtual environment
-and activate it in your shell::
-
-    python -m venv env
-    source env/bin/activate
-
-Afterwards, invoking ``python`` or ``pip install`` only
-modifies files in your ``env`` directory and leaves
-your system installation alone.
-
-For Linux we build wheels for all releases and push them to a python package
-index. To install the latest release::
-
-    pip install deltachat
-
-To verify it worked::
-
-    python -c "import deltachat"
-
-Running tests
-=============
-
-Recommended way to run tests is using `scripts/run-python-test.sh`
-script provided in the core repository.
-
-This script compiles the library in debug mode and runs the tests using `tox`_.
-By default it will run all "offline" tests and skip all functional
-end-to-end tests that require accounts on real e-mail servers.
-
-.. _`tox`: https://tox.wiki
-.. _livetests:
-
-Running "live" tests with temporary accounts
---------------------------------------------
-
-If you want to run live functional tests
-you can set ``CHATMAIL_DOMAIN`` to a domain of the email server
-that creates e-mail accounts like this::
-
-    export CHATMAIL_DOMAIN=nine.testrun.org
-
-With this account-creation setting, pytest runs create ephemeral e-mail accounts on the server.
-These accounts have the pattern `ci-{6 characters}@{CHATMAIL_DOMAIN}`.
-After setting the variable, either rerun `scripts/run-python-test.sh`
-or run offline and online tests with `tox` directly::
-
-    tox -e py
-
-Each test run creates new accounts.
-
-Developing the bindings
------------------------
-
-If you want to develop or debug the bindings,
-you can create a testing development environment using `tox`::
-
-    export DCC_RS_DEV="$PWD"
-    export DCC_RS_TARGET=debug
-    tox -c python --devenv env -e py
-    . env/bin/activate
-
-Inside this environment the bindings are installed
-in editable mode (as if installed with `python -m pip install -e`)
-together with the testing dependencies like `pytest` and its plugins.
-
-You can then edit the source code in the development tree
-and quickly run `pytest` manually without waiting  for `tox`
-to recreating the virtual environment each time.
-
-.. _sourceinstall:
-
-Installing bindings from source
-===============================
-
-Install Rust and Cargo first.
-The easiest is probably to use `rustup <https://rustup.rs/>`_.
-
-Bootstrap Rust and Cargo by using rustup::
-
-   curl https://sh.rustup.rs -sSf | sh
-
-Then clone the deltachat-core-rust repo::
-
-   git clone https://github.com/deltachat/deltachat-core-rust
-   cd deltachat-core-rust
-
-To install the Delta Chat Python bindings make sure you have Python3 installed.
-E.g. on Debian-based systems `apt install python3 python3-pip
-python3-venv` should give you a usable python installation.
-
-First, build the core library::
-
-   cargo build --release -p deltachat_ffi --features jsonrpc
-
-`jsonrpc` feature is required even if not used by the bindings
-because `deltachat.h` includes JSON-RPC functions unconditionally.
-
-Create the virtual environment and activate it:
-
-   python -m venv env
-   source env/bin/activate
-
-Build and install the bindings:
-
-   export DCC_RS_DEV="$PWD"
-   export DCC_RS_TARGET=release
-   python -m pip install ./python
-
-`DCC_RS_DEV` environment variable specifies the location of
-the core development tree. If this variable is not set,
-`libdeltachat` library and `deltachat.h` header are expected
-to be installed system-wide.
-
-When `DCC_RS_DEV` is set, `DCC_RS_TARGET` specifies
-the build profile name to look up the artifacts
-in the target directory.
-In this case setting it can be skipped because
-`DCC_RS_TARGET=release` is the default.
-
-Building manylinux based wheels
-===============================
-
-Building portable manylinux wheels which come with libdeltachat.so
-can be done with Docker_ or Podman_.
-
-.. _Docker: https://www.docker.com/
-.. _Podman: https://podman.io/
-
-If you want to build your own wheels, build container image first::
-
-   $ cd deltachat-core-rust # cd to deltachat-core-rust working tree
-   $ docker build -t deltachat/coredeps scripts/coredeps
-
-This will use the ``scripts/coredeps/Dockerfile`` to build
-container image called ``deltachat/coredeps``.  You can afterwards
-find it with::
-
-   $ docker images
-
-This docker image can be used to run tests and build Python wheels for all interpreters::
-
-    $ docker run -e CHATMAIL_DOMAIN \
-       --rm -it -v $(pwd):/mnt -w /mnt \
-       deltachat/coredeps scripts/run_all.sh

python/doc/Makefile

@@ -1,197 +0,0 @@
-# Makefile for Sphinx documentation
-#
-
-VERSION       = $(shell python -c "import conf ; print(conf.version)")
-DOCZIP        = devpi-$(VERSION).doc.zip
-# You can set these variables from the command line.
-SPHINXOPTS    =
-SPHINXBUILD   = sphinx-build
-PAPER         =
-BUILDDIR      = _build
-RSYNCOPTS     = -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
-
-export HOME=/tmp/home
-export TESTHOME=$(HOME)
-
-# Internal variables.
-PAPEROPT_a4     = -D latex_paper_size=a4
-PAPEROPT_letter = -D latex_paper_size=letter
-ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
-# the i18n builder cannot share the environment and doctrees with the others
-I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
-
-# This variable is not auto generated as the order is important.
-USER_MAN_CHAPTERS = commands\
-                    user\
-                    indices\
-                    packages\
-#                 userman/index.rst\
-#                 userman/devpi_misc.rst\
-#                 userman/devpi_concepts.rst\
-
-
-#export DEVPI_CLIENTDIR=$(CURDIR)/.tmp_devpi_user_man/client
-#export DEVPI_SERVERDIR=$(CURDIR)/.tmp_devpi_user_man/server
-
-chapter = commands
-
-.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp \
-        epub latex latexpdf text man changes linkcheck doctest gettext install \
-        quickstart-releaseprocess quickstart-pypimirror quickstart-server regen \
-        prepare-quickstart\
-        regen.server-fresh regen.server-restart regen.server-clean\
-        regen.uman-all regen.uman
-
-help:
-	@echo "Please use \`make <target>' where <target> is one of"
-	@echo "  html       to make standalone HTML files"
-	@echo "  dirhtml    to make HTML files named index.html in directories"
-	@echo "  singlehtml to make a single large HTML file"
-	@echo "  pickle     to make pickle files"
-	@echo "  json       to make JSON files"
-	@echo "  htmlhelp   to make HTML files and a HTML help project"
-	@echo "  qthelp     to make HTML files and a qthelp project"
-	@echo "  devhelp    to make HTML files and a Devhelp project"
-	@echo "  epub       to make an epub"
-	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
-	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
-	@echo "  text       to make text files"
-	@echo "  man        to make manual pages"
-	@echo "  texinfo    to make Texinfo files"
-	@echo "  info       to make Texinfo files and run them through makeinfo"
-	@echo "  gettext    to make PO message catalogs"
-	@echo "  changes    to make an overview of all changed/added/deprecated items"
-	@echo "  linkcheck  to check all external links for integrity"
-	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
-	@echo
-	@echo "User Manual Regen Targets"
-	@echo "  regen.uman        regenerates page. of the user manual chapeter e.g. regen.uman chapter=..."
-	@echo "  regen.uman-all    regenerates the user manual"
-	@echo "  regen.uman-clean  stop temp server and clean up directory"
-	@echo "  Chapter List: $(USER_MAN_CHAPTERS)"
-
-clean:
-	-rm -rf $(BUILDDIR)/*
-
-version:
-	@echo "version $(VERSION)"
-
-doczip: html
-	python doczip.py $(DOCZIP) _build/html
-
-install: html
-	rsync -avz $(RSYNCOPTS) _build/html/ delta@py.delta.chat:build/master
-
-
-html:
-	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
-	@echo
-	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
-
-dirhtml:
-	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
-	@echo
-	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
-
-singlehtml:
-	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
-	@echo
-	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
-
-pickle:
-	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
-	@echo
-	@echo "Build finished; now you can process the pickle files."
-
-json:
-	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
-	@echo
-	@echo "Build finished; now you can process the JSON files."
-
-htmlhelp:
-	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
-	@echo
-	@echo "Build finished; now you can run HTML Help Workshop with the" \
-	      ".hhp project file in $(BUILDDIR)/htmlhelp."
-
-qthelp:
-	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
-	@echo
-	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
-	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
-	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/devpi.qhcp"
-	@echo "To view the help file:"
-	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/devpi.qhc"
-
-devhelp:
-	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
-	@echo
-	@echo "Build finished."
-	@echo "To view the help file:"
-	@echo "# mkdir -p $$HOME/.local/share/devhelp/devpi"
-	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/devpi"
-	@echo "# devhelp"
-
-epub:
-	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
-	@echo
-	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
-
-latex:
-	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
-	@echo
-	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
-	@echo "Run \`make' in that directory to run these through (pdf)latex" \
-	      "(use \`make latexpdf' here to do that automatically)."
-
-latexpdf:
-	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
-	@echo "Running LaTeX files through pdflatex..."
-	$(MAKE) -C $(BUILDDIR)/latex all-pdf
-	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
-
-text:
-	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
-	@echo
-	@echo "Build finished. The text files are in $(BUILDDIR)/text."
-
-man:
-	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
-	@echo
-	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
-
-texinfo:
-	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
-	@echo
-	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
-	@echo "Run \`make' in that directory to run these through makeinfo" \
-	      "(use \`make info' here to do that automatically)."
-
-info:
-	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
-	@echo "Running Texinfo files through makeinfo..."
-	make -C $(BUILDDIR)/texinfo info
-	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
-
-gettext:
-	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
-	@echo
-	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
-
-changes:
-	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
-	@echo
-	@echo "The overview file is in $(BUILDDIR)/changes."
-
-linkcheck:
-	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
-	@echo
-	@echo "Link check complete; look for any errors in the above output " \
-	      "or in $(BUILDDIR)/linkcheck/output.txt."
-
-doctest:
-	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
-	@echo "Testing of doctests in the sources finished, look at the " \
-	      "results in $(BUILDDIR)/doctest/output.txt."
-
-

python/doc/_templates/globaltoc.html

@@ -1,17 +0,0 @@
-
-<div class="globaltoc">
-
-<ul>
-  <li><a href="{{ pathto('index') }}">index</a></li>
-  <li><a href="{{ pathto('install') }}">install</a></li>
-  <li><a href="{{ pathto('api') }}">high level API</a></li>
-  <li><a href="{{ pathto('lapi') }}">low level API</a></li>
-</ul>
-<b>external links:</b>
-<ul>
-  <li><a href="https://github.com/deltachat/deltachat-core-rust">github repository</a></li>
-  <li><a href="https://pypi.python.org/pypi/deltachat">pypi: deltachat</a></li>
-  <li><a href="https://web.libera.chat/#deltachat">#deltachat</a></li>
-</ul>
-
-</div>

python/doc/_templates/sidebarintro.html

@@ -1 +0,0 @@
-<h3>deltachat {{release}}</h3>

python/doc/cffi/api.rst

@@ -1,5 +1,4 @@
-
-high level API reference
+High Level API Reference
 ========================
 
 - :class:`deltachat.Account` (your main entry point, creates the
@@ -8,28 +7,14 @@ high level API reference
 - :class:`deltachat.Chat`
 - :class:`deltachat.Message`
 
-Account
--------
-
 .. autoclass:: deltachat.Account
-    :members:
-
-
-Contact
--------
+   :members:
 
 .. autoclass:: deltachat.Contact
-    :members:
-
-Chat
-----
+   :members:
 
 .. autoclass:: deltachat.Chat
-    :members:
-
-Message
--------
+   :members:
 
 .. autoclass:: deltachat.Message
-    :members:
-
+   :members:

python/doc/cffi/examples.rst

@@ -1,11 +1,10 @@
-
-examples
+Examples
 ========
 
 Once you have :doc:`installed deltachat bindings <install>`
 you need email/password credentials for an IMAP/SMTP account.
 Delta Chat developers and the CI system use a special URL to create
-temporary e-mail accounts on [testrun.org](https://testrun.org) for testing.
+temporary email accounts on `testrun.org <https://testrun.org/>`_ for testing.
 
 Receiving a Chat message from the command line
 ----------------------------------------------
@@ -16,11 +15,11 @@ Here is a simple bot that:
 
 - terminates the bot if the message `/quit` is sent
 
-.. include:: ../examples/echo_and_quit.py
+.. include:: ../../examples/echo_and_quit.py
     :literal:
 
 With this file in your working directory you can run the bot
-by specifying a database path, an e-mail address and password of
+by specifying a database path, an email address and password of
 a SMTP-IMAP account::
 
     $ cd examples
@@ -40,11 +39,11 @@ Here is a simple bot that:
 
 - tracks member additions and removals for all chat groups
 
-.. include:: ../examples/group_tracking.py
+.. include:: ../../examples/group_tracking.py
     :literal:
 
 With this file in your working directory you can run the bot
-by specifying a database path, an e-mail address and password of
+by specifying a database path, an email address and password of
 a SMTP-IMAP account::
 
     python group_tracking.py --email ADDRESS --password PASSWORD /tmp/db

python/doc/cffi/install.rst

@@ -0,0 +1,80 @@
+Install
+=======
+
+Installing pre-built packages (Linux-only)
+------------------------------------------
+
+If you have a Linux system you may install the ``deltachat`` binary "wheel" packages
+without any "build-from-source" steps.
+Otherwise you need to `compile the Delta Chat bindings yourself`__.
+
+__ sourceinstall_
+
+We recommend to first create a fresh Python virtual environment
+and activate it in your shell::
+
+    python -m venv env
+    source env/bin/activate
+
+Afterwards, invoking ``python`` or ``pip install`` only
+modifies files in your ``env`` directory and leaves
+your system installation alone.
+
+For Linux we build wheels for all releases and push them to a python package
+index. To install the latest release::
+
+    pip install deltachat
+
+To verify it worked::
+
+    python -c "import deltachat"
+
+.. _sourceinstall:
+
+Installing bindings from source
+-------------------------------
+
+Install Rust and Cargo first.
+The easiest is probably to use `rustup <https://rustup.rs/>`_.
+
+Bootstrap Rust and Cargo by using rustup::
+
+   curl https://sh.rustup.rs -sSf | sh
+
+Then clone the deltachat-core-rust repo::
+
+   git clone https://github.com/deltachat/deltachat-core-rust
+   cd deltachat-core-rust
+
+To install the Delta Chat Python bindings make sure you have Python3 installed.
+E.g. on Debian-based systems `apt install python3 python3-pip
+python3-venv` should give you a usable python installation.
+
+First, build the core library::
+
+   cargo build --release -p deltachat_ffi --features jsonrpc
+
+`jsonrpc` feature is required even if not used by the bindings
+because `deltachat.h` includes JSON-RPC functions unconditionally.
+
+Create the virtual environment and activate it::
+
+   python -m venv env
+   source env/bin/activate
+
+Build and install the bindings::
+
+   export DCC_RS_DEV="$PWD"
+   export DCC_RS_TARGET=release
+   python -m pip install ./python
+
+`DCC_RS_DEV` environment variable specifies the location of
+the core development tree. If this variable is not set,
+`libdeltachat` library and `deltachat.h` header are expected
+to be installed system-wide.
+
+When `DCC_RS_DEV` is set, `DCC_RS_TARGET` specifies
+the build profile name to look up the artifacts
+in the target directory.
+In this case setting it can be skipped because
+`DCC_RS_TARGET=release` is the default.

python/doc/cffi/intro.rst

@@ -0,0 +1,11 @@
+Introduction
+============
+
+CFFI bindings are available via the `deltachat <https://pypi.org/project/deltachat/>`_ Python package.
+The package contains both the Python bindings and the Delta Chat core.
+It is provided only for Linux.
+
+The ``deltachat`` Python package provides two layers of bindings for the
+core Rust-library of the https://delta.chat messaging ecosystem:
+low-level CFFI bindings to the C interface of the Delta Chat core
+and high-level Python bindings built on top of CFFI bindings.

python/doc/cffi/lapi.rst

@@ -1,8 +1,7 @@
+Low Level API Reference
+=======================
 
-low level API reference
-===================================
-
-for full doxygen-generated C-docs, defines and functions please checkout
+For full doxygen-generated C-docs, defines and functions please checkout
 
 https://c.delta.chat
 

python/doc/cffi/manylinux.rst

@@ -0,0 +1,25 @@
+Building Manylinux-Based Wheels
+===============================
+
+Building portable manylinux wheels which come with libdeltachat.so
+can be done with Docker_ or Podman_.
+
+.. _Docker: https://www.docker.com/
+.. _Podman: https://podman.io/
+
+If you want to build your own wheels, build container image first::
+
+   $ cd deltachat-core-rust # cd to deltachat-core-rust working tree
+   $ docker build -t deltachat/coredeps scripts/coredeps
+
+This will use the ``scripts/coredeps/Dockerfile`` to build
+container image called ``deltachat/coredeps``.  You can afterwards
+find it with::
+
+   $ docker images
+
+This docker image can be used to run tests and build Python wheels for all interpreters::
+
+    $ docker run -e CHATMAIL_DOMAIN \
+       --rm -it -v $(pwd):/mnt -w /mnt \
+       deltachat/coredeps scripts/run_all.sh

python/doc/cffi/tests.rst

@@ -0,0 +1,49 @@
+Running Tests
+=============
+
+Recommended way to run tests is using `scripts/run-python-test.sh`
+script provided in the core repository.
+
+This script compiles the library in debug mode and runs the tests using `tox`_.
+By default it will run all "offline" tests and skip all functional
+end-to-end tests that require accounts on real email servers.
+
+.. _`tox`: https://tox.wiki
+.. _livetests:
+
+Running "Live" Tests With Temporary Accounts
+--------------------------------------------
+
+If you want to run live functional tests
+you can set ``CHATMAIL_DOMAIN`` to a domain of the email server
+that creates email accounts like this::
+
+    export CHATMAIL_DOMAIN=nine.testrun.org
+
+With this account-creation setting, pytest runs create ephemeral email accounts on the server.
+These accounts have the pattern `ci-{6 characters}@{CHATMAIL_DOMAIN}`.
+After setting the variable, either rerun `scripts/run-python-test.sh`
+or run offline and online tests with `tox` directly::
+
+    tox -e py
+
+Each test run creates new accounts.
+
+Developing the Bindings
+-----------------------
+
+If you want to develop or debug the bindings,
+you can create a testing development environment using `tox`::
+
+    export DCC_RS_DEV="$PWD"
+    export DCC_RS_TARGET=debug
+    tox -c python --devenv env -e py
+    . env/bin/activate
+
+Inside this environment the bindings are installed
+in editable mode (as if installed with `python -m pip install -e`)
+together with the testing dependencies like `pytest` and its plugins.
+
+You can then edit the source code in the development tree
+and quickly run `pytest` manually without waiting  for `tox`
+to recreating the virtual environment each time.

python/doc/changelog.rst

@@ -1,4 +0,0 @@
-Changelog for deltachat-core's Python bindings
-==============================================
-
-.. include:: ../CHANGELOG

python/doc/conf.py

@@ -1,138 +1,94 @@
-# -*- coding: utf-8 -*-
-#
-# devpi documentation build configuration file, created by
-# sphinx-quickstart on Mon Jun  3 16:11:22 2013.
-#
-# This file is execfile()d with the current directory set to its containing dir.
-#
-# Note that not all possible configuration values are present in this
-# autogenerated file.
-#
-# All configuration values have a default; values that are commented out
-# serve to show the default.
-
-import sys
-import os
+from pathlib import Path
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 from deltachat import __version__ as release
+
 version = ".".join(release.split(".")[:2])
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
-#sys.path.insert(0, os.path.abspath('.'))
+# sys.path.insert(0, os.path.abspath('.'))
 
 # -- General configuration -----------------------------------------------------
 
-# If your documentation needs a minimal Sphinx version, state it here.
-#needs_sphinx = '1.0'
-
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
-    'sphinx.ext.autodoc',
-    'sphinx.ext.autosummary',
-    #'sphinx.ext.intersphinx',
-    'sphinx.ext.todo',
-    'sphinx.ext.viewcode',
-    'breathe',
-    #'sphinx.ext.githubpages',
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.viewcode",
+    "breathe",
+    "sphinx_rtd_theme",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # The suffix of source filenames.
-source_suffix = '.rst'
+source_suffix = ".rst"
 
 # The encoding of source files.
-#source_encoding = 'utf-8-sig'
+# source_encoding = 'utf-8-sig'
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = "index"
 
 # General information about the project.
-project = u'deltachat'
-copyright = u'2020, holger krekel and contributors'
+project = "Delta Chat"
+copyright = "2023, Delta Chat contributors"
+author = "Delta Chat contributors"
 
 
-# The language for content autogenerated by Sphinx. Refer to documentation
-# for a list of supported languages.
-#language = None
-
 # There are two options for replacing |today|: either, you set today to some
 # non-false value, then it is used:
-#today = ''
+# today = ''
 # Else, today_fmt is used as the format for a strftime call.
-#today_fmt = '%B %d, %Y'
+# today_fmt = '%B %d, %Y'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-exclude_patterns = ['sketch', '_build', "attic"]
+exclude_patterns = ["sketch", "_build", "attic", "Thumbs.db", ".DS_Store"]
 
 # The reST default role (used for this markup: `text`) to use for all documents.
-#default_role = None
+# default_role = None
 
 # If true, '()' will be appended to :func: etc. cross-reference text.
-#add_function_parentheses = True
+# add_function_parentheses = True
 
 # If true, the current module name will be prepended to all description
 # unit titles (such as .. function::).
-#add_module_names = True
+# add_module_names = True
 
 # If true, sectionauthor and moduleauthor directives will be shown in the
 # output. They are ignored by default.
-#show_authors = False
+# show_authors = False
 
 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+pygments_style = "sphinx"
 
 # A list of ignored prefixes for module index sorting.
-#modindex_common_prefix = []
+# modindex_common_prefix = []
 
 
 # -- breathe options ------
 
-breathe_projects = {
-    "deltachat": "../../docs/xml/"
-}
+breathe_projects = {"deltachat": Path("../../docs/xml/")}
 
 breathe_default_project = "deltachat"
 
 # -- Options for HTML output ---------------------------------------------------
 
-sys.path.append(os.path.abspath('_themes'))
-html_theme_path = ['_themes']
-
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
-# html_theme = 'flask'
-html_theme = 'alabaster'
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further.  For a list of options available for each theme, see the
-# documentation.
-html_theme_options = {
-    'logo': '_static/delta-chat.svg',
-    'font_size': "1.1em",
-    'caption_font_size': "0.9em",
-    'code_font_size': "1.1em",
-
-
-}
-
-# Add any paths that contain custom themes here, relative to this directory.
-#html_theme_path = ["_themes"]
+html_theme = "sphinx_rtd_theme"
 
 # The name for this set of Sphinx documents.  If None, it defaults to
 # "<project> v<release> documentation".
-#html_title = None
+# html_title = None
 
 # A shorter title for the navigation bar.  Default is the same as html_title.
-#html_short_title = None
+# html_short_title = None
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
@@ -141,51 +97,34 @@ html_logo = "_static/delta-chat.svg"
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
 # pixels large.
-html_favicon = '_static/favicon.ico'
+html_favicon = "_static/favicon.ico"
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = ["_static"]
 
 # If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
 # using the given strftime format.
-#html_last_updated_fmt = '%b %d, %Y'
+# html_last_updated_fmt = '%b %d, %Y'
 
 # If true, SmartyPants will be used to convert quotes and dashes to
 # typographically correct entities.
-#html_use_smartypants = True
-
-# Custom sidebar templates, maps document names to template names.
-#
-html_sidebars = {
-    'index': [
-        'sidebarintro.html',
-        'globaltoc.html',
-        'searchbox.html'
-    ],
-    '**': [
-        'sidebarintro.html',
-        'globaltoc.html',
-        'relations.html',
-        'searchbox.html'
-    ]
-}
-
+# html_use_smartypants = True
 
 
 # Additional templates that should be rendered to pages, maps page names to
 # template names.
-#html_additional_pages = {}
+# html_additional_pages = {}
 
 # If false, no module index is generated.
-#html_domain_indices = True
+# html_domain_indices = True
 
 # If false, no index is generated.
-#html_use_index = True
+# html_use_index = True
 
 # If true, the index is split into individual pages for each letter.
-#html_split_index = False
+# html_split_index = False
 
 # If true, links to the reST sources are added to the pages.
 html_show_sourcelink = False
@@ -194,71 +133,65 @@ html_show_sourcelink = False
 html_show_sphinx = False
 
 # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
-#html_show_copyright = True
+# html_show_copyright = True
 
 # If true, an OpenSearch description file will be output, and all pages will
 # contain a <link> tag referring to it.  The value of this option must be the
 # base URL from which the finished HTML is served.
-html_use_opensearch = 'https://doc.devpi.net'
+html_use_opensearch = "https://doc.devpi.net"
 
 # This is the file name suffix for HTML files (e.g. ".xhtml").
-#html_file_suffix = None
+# html_file_suffix = None
 
 # Output file base name for HTML help builder.
-htmlhelp_basename = 'deltachat-python'
+htmlhelp_basename = "deltachat-python"
 
 # -- Options for LaTeX output --------------------------------------------------
 
 latex_elements = {
-# The paper size ('letterpaper' or 'a4paper').
-#'papersize': 'letterpaper',
-
-# The font size ('10pt', '11pt' or '12pt').
-'pointsize': '12pt',
-
-# Additional stuff for the LaTeX preamble.
-#'preamble': '',
+    # The paper size ('letterpaper' or 'a4paper').
+    #'papersize': 'letterpaper',
+    # The font size ('10pt', '11pt' or '12pt').
+    #'pointsize': '12pt',
+    # Additional stuff for the LaTeX preamble.
+    #'preamble': '',
 }
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title, author, documentclass [howto/manual]).
 latex_documents = [
-  ('index', 'devpi.tex', u'deltachat documentation',
-   u'holger krekel', 'manual'),
+    ("index", "devpi.tex", "deltachat documentation", "holger krekel", "manual"),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
 # the title page.
-#latex_logo = None
+# latex_logo = None
 
 # For "manual" documents, if this is true, then toplevel headings are parts,
 # not chapters.
-#latex_use_parts = False
+# latex_use_parts = False
 
 # If true, show page references after internal links.
-#latex_show_pagerefs = False
+# latex_show_pagerefs = False
 
 # If true, show URL addresses after external links.
-#latex_show_urls = False
+# latex_show_urls = False
 
 # Documents to append as an appendix to all manuals.
-#latex_appendices = []
+# latex_appendices = []
 
 # If false, no module index is generated.
-#latex_domain_indices = True
+# latex_domain_indices = True
 
 
 # -- Options for manual page output --------------------------------------------
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [
-    ('index', 'deltachat', u'deltachat documentation',
-     [u'holger krekel'], 1)
-]
+man_pages = [("index", "deltachat", "deltachat documentation", ["holger krekel"], 1)]
 
 # If true, show URL addresses after external links.
-#man_show_urls = False
+# man_show_urls = False
 
 
 # -- Options for Texinfo output ------------------------------------------------
@@ -267,30 +200,38 @@ man_pages = [
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-  ('index', 'devpi', u'devpi Documentation',
-   u'holger krekel', 'devpi', 'One line description of project.',
-   'Miscellaneous'),
+    (
+        "index",
+        "devpi",
+        "devpi Documentation",
+        "holger krekel",
+        "devpi",
+        "One line description of project.",
+        "Miscellaneous",
+    ),
 ]
 
 # Documents to append as an appendix to all manuals.
-#texinfo_appendices = []
+# texinfo_appendices = []
 
 # If false, no module index is generated.
-#texinfo_domain_indices = True
+# texinfo_domain_indices = True
 
 # How to display URL addresses: 'footnote', 'no', or 'inline'.
-#texinfo_show_urls = 'footnote'
+# texinfo_show_urls = 'footnote'
 
 
 # Example configuration for intersphinx: refer to the Python standard library.
-intersphinx_mapping = {'http://docs.python.org/': None}
+intersphinx_mapping = {"http://docs.python.org/": None}
 
 # autodoc options
 autodoc_member_order = "bysource"
+
+
 # always document __init__ functions
 def skip(app, what, name, obj, skip, options):
     return skip
 
+
 def setup(app):
     app.connect("autodoc-skip-member", skip)
-

python/doc/index.rst

@@ -1,41 +1,44 @@
-deltachat python bindings
-=========================
+Delta Chat Python bindings, new and old
+=======
 
-The ``deltachat`` Python package provides two layers of bindings for the
-core Rust-library of the https://delta.chat messaging ecosystem:
-
-- :doc:`api` is a high level interface to deltachat-core.
-
-- :doc:`plugins` is a brief introduction into implementing plugin hooks.
-
-- :doc:`lapi` is a lowlevel CFFI-binding to the `Rust Core
-  <https://github.com/deltachat/deltachat-core-rust>`_.
-
-
-
-getting started
----------------
+`Delta Chat <https://delta.chat/>`_ provides two kinds of Python bindings
+to the `Rust Core <https://github.com/deltachat/deltachat-core-rust>`_:
+JSON-RPC bindings and CFFI bindings.
+When starting a new project it is recommended to use JSON-RPC bindings,
+which are used in the Delta Chat Desktop app through generated Typescript-bindings. 
+The Python JSON-RPC bindings are maintained by Delta Chat core developers. 
+Most existing bot projects and many tests in Delta Chat's own core library
+still use the CFFI-bindings, and it is going to be maintained certainly also in 2024. 
+New APIs might however only appear in the JSON-RPC bindings, 
+as the CFFI bindings are increasingly in maintenance-only mode. 
 
 .. toctree::
    :maxdepth: 2
+   :caption: JSON-RPC Bindings
 
-   install
-   examples
+   jsonrpc/intro
+   jsonrpc/install
+   jsonrpc/examples
+   jsonrpc/reference
+   jsonrpc/develop
 
 .. toctree::
-   :hidden:
+   :maxdepth: 2
+   :caption: CFFI Bindings
 
-   links
-   changelog
-   api
-   lapi
-   plugins
-
-..
-    Indices and tables
-    ==================
-
-    * :ref:`genindex`
-    * :ref:`modindex`
-    * :ref:`search`
+   cffi/intro
+   cffi/install
+   cffi/examples
+   cffi/manylinux
+   cffi/tests
+   cffi/api
+   cffi/lapi
+   cffi/plugins
 
+.. _`deltachat`: https://delta.chat
+.. _`deltachat-core repo`: https://github.com/deltachat
+.. _pip: http://pypi.org/project/pip/
+.. _virtualenv: http://pypi.org/project/virtualenv/
+.. _merlinux: http://merlinux.eu
+.. _pypi: http://pypi.org/
+.. _`issue-tracker`: https://github.com/deltachat/deltachat-core-rust

python/doc/install.rst

@@ -1,2 +0,0 @@
-
-.. include:: ../README.rst

python/doc/jsonrpc/develop.rst

@@ -0,0 +1,68 @@
+===========
+Development
+===========
+
+To develop JSON-RPC bindings,
+clone the `deltachat-core-rust <https://github.com/deltachat/deltachat-core-rust/>`_ repository::
+
+   git clone https://github.com/deltachat/deltachat-core-rust.git
+
+Testing
+=======
+
+To run online tests, set ``CHATMAIL_DOMAIN``
+to a domain of the email server
+that can be used to create testing accounts::
+
+    export CHATMAIL_DOMAIN=nine.testrun.org
+
+Then run ``scripts/run-rpc-test.sh``
+to build debug version of ``deltachat-rpc-server``
+and run ``deltachat-rpc-client`` tests
+in a separate virtual environment managed by `tox <https://tox.wiki/>`_.
+
+Development Environment
+=======================
+
+Creating a new virtual environment
+to run the tests each time
+as ``scripts/run-rpc-test.sh`` does is slow
+if you are changing the tests or the code
+and want to rerun the tests each time.
+
+If you are developing the tests,
+it is better to create a persistent virtual environment.
+You can do this by running ``scripts/make-rpc-testenv.sh``.
+This creates a virtual environment ``venv`` which you can then enter with::
+
+   . venv/bin/activate
+
+Then you can run the tests with
+
+::
+
+    pytest deltachat-rpc-client/tests/
+
+Refer to `pytest documentation <https://docs.pytest.org/>` for details.
+
+If make the changes to Delta Chat core
+or Python bindings, you can rebuild the environment by rerunning
+``scripts/make-rpc-testenv.sh``.
+It is ok to rebuild the activated environment this way,
+you do not need to deactivate or reactivate the environment each time.
+
+Using REPL
+==========
+
+Once you have a development environment,
+you can quickly test things in REPL::
+
+   $ python
+   >>> from deltachat_rpc_client import *
+   >>> rpc = Rpc()
+   >>> rpc.start()
+   >>> dc = DeltaChat(rpc)
+   >>> system_info = dc.get_system_info()
+   >>> system_info["level"]
+   'awesome'
+   >>> rpc.close()

python/doc/jsonrpc/examples.rst

@@ -0,0 +1,19 @@
+Examples
+========
+
+Echo bot
+--------
+.. include:: ../../../deltachat-rpc-client/examples/echobot_no_hooks.py
+    :literal:
+
+Echo bot with hooks
+-------------------
+.. include:: ../../../deltachat-rpc-client/examples/echobot.py
+    :literal:
+
+Advanced echo bot
+-----------------
+
+.. include:: ../../../deltachat-rpc-client/examples/echobot_advanced.py
+    :literal:
+

python/doc/jsonrpc/install.rst

@@ -0,0 +1,36 @@
+Install
+=======
+
+To use JSON-RPC bindings for Delta Chat core you will need
+a ``deltachat-rpc-server`` binary which provides Delta Chat core API over JSON-RPC
+and a ``deltachat-rpc-client`` Python package which is a JSON-RPC client that starts ``deltachat-rpc-server`` process and uses JSON-RPC API.
+
+`Create a virtual environment <https://docs.python.org/3/library/venv.html>`__ if you
+don’t have one already and activate it::
+
+   $ python -m venv venv
+   $ . venv/bin/activate
+
+Install ``deltachat-rpc-server``
+--------------------------------
+
+To get ``deltachat-rpc-server`` binary you have three options:
+
+1. Install ``deltachat-rpc-server`` from PyPI using ``pip install deltachat-rpc-server``.
+2. Build and install ``deltachat-rpc-server`` from source with ``cargo install --git https://github.com/deltachat/deltachat-core-rust/ deltachat-rpc-server``.
+3. Download prebuilt release from https://github.com/deltachat/deltachat-core-rust/releases and install it into ``PATH``.
+
+Check that ``deltachat-rpc-server`` is installed and can run::
+
+   $ deltachat-rpc-server --version
+   1.131.4
+
+Then install ``deltachat-rpc-client`` with ``pip install deltachat-rpc-client``.
+
+Install ``deltachat-rpc-client``
+--------------------------------
+
+To get ``deltachat-rpc-client`` Python library you can:
+
+1. Install ``deltachat-rpc-client`` from PyPI using ``pip install deltachat-rpc-client``.
+2. Install ``deltachat-rpc-client`` from source with ``pip install git+https://github.com/deltachat/deltachat-core-rust.git@main#subdirectory=deltachat-rpc-client``.

python/doc/jsonrpc/intro.rst

@@ -0,0 +1,8 @@
+Introduction
+============
+
+JSON-RPC bindings are available via the `deltachat-rpc-client <https://pypi.org/project/deltachat-rpc-client/>`_ Python package.
+This package provides only the Python bindings and requires ``deltachat-rpc-server`` binary to be installed.
+`deltachat-rpc-server <https://pypi.org/project/deltachat-rpc-server/>`_ package provides ``deltachat-rpc-server`` binary for Linux, Windows, macOS and Android.
+
+RPC client connects to standalone Delta Chat RPC server ``deltachat-rpc-server`` and provides Python interface to it.

python/doc/jsonrpc/reference.rst

@@ -0,0 +1,5 @@
+API Reference
+=============
+
+.. automodule:: deltachat_rpc_client
+   :members:

python/doc/links.rst

@@ -1,11 +0,0 @@
-
-links
-================================
-
-.. _`deltachat`: https://delta.chat
-.. _`deltachat-core repo`: https://github.com/deltachat
-.. _pip: http://pypi.org/project/pip/
-.. _virtualenv: http://pypi.org/project/virtualenv/
-.. _merlinux: http://merlinux.eu
-.. _pypi: http://pypi.org/
-.. _`issue-tracker`: https://github.com/deltachat/deltachat-core

python/doc/make.bat

@@ -1,190 +0,0 @@
-@ECHO OFF
-
-REM Command file for Sphinx documentation
-
-if "%SPHINXBUILD%" == "" (
-	set SPHINXBUILD=sphinx-build
-)
-set BUILDDIR=_build
-set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% .
-set I18NSPHINXOPTS=%SPHINXOPTS% .
-if NOT "%PAPER%" == "" (
-	set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS%
-	set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS%
-)
-
-if "%1" == "" goto help
-
-if "%1" == "help" (
-	:help
-	echo.Please use `make ^<target^>` where ^<target^> is one of
-	echo.  html       to make standalone HTML files
-	echo.  dirhtml    to make HTML files named index.html in directories
-	echo.  singlehtml to make a single large HTML file
-	echo.  pickle     to make pickle files
-	echo.  json       to make JSON files
-	echo.  htmlhelp   to make HTML files and a HTML help project
-	echo.  qthelp     to make HTML files and a qthelp project
-	echo.  devhelp    to make HTML files and a Devhelp project
-	echo.  epub       to make an epub
-	echo.  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter
-	echo.  text       to make text files
-	echo.  man        to make manual pages
-	echo.  texinfo    to make Texinfo files
-	echo.  gettext    to make PO message catalogs
-	echo.  changes    to make an overview over all changed/added/deprecated items
-	echo.  linkcheck  to check all external links for integrity
-	echo.  doctest    to run all doctests embedded in the documentation if enabled
-	goto end
-)
-
-if "%1" == "clean" (
-	for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i
-	del /q /s %BUILDDIR%\*
-	goto end
-)
-
-if "%1" == "html" (
-	%SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The HTML pages are in %BUILDDIR%/html.
-	goto end
-)
-
-if "%1" == "dirhtml" (
-	%SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml.
-	goto end
-)
-
-if "%1" == "singlehtml" (
-	%SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml.
-	goto end
-)
-
-if "%1" == "pickle" (
-	%SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; now you can process the pickle files.
-	goto end
-)
-
-if "%1" == "json" (
-	%SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; now you can process the JSON files.
-	goto end
-)
-
-if "%1" == "htmlhelp" (
-	%SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; now you can run HTML Help Workshop with the ^
-.hhp project file in %BUILDDIR%/htmlhelp.
-	goto end
-)
-
-if "%1" == "qthelp" (
-	%SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; now you can run "qcollectiongenerator" with the ^
-.qhcp project file in %BUILDDIR%/qthelp, like this:
-	echo.^> qcollectiongenerator %BUILDDIR%\qthelp\devpi.qhcp
-	echo.To view the help file:
-	echo.^> assistant -collectionFile %BUILDDIR%\qthelp\devpi.ghc
-	goto end
-)
-
-if "%1" == "devhelp" (
-	%SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished.
-	goto end
-)
-
-if "%1" == "epub" (
-	%SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The epub file is in %BUILDDIR%/epub.
-	goto end
-)
-
-if "%1" == "latex" (
-	%SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished; the LaTeX files are in %BUILDDIR%/latex.
-	goto end
-)
-
-if "%1" == "text" (
-	%SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The text files are in %BUILDDIR%/text.
-	goto end
-)
-
-if "%1" == "man" (
-	%SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The manual pages are in %BUILDDIR%/man.
-	goto end
-)
-
-if "%1" == "texinfo" (
-	%SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo.
-	goto end
-)
-
-if "%1" == "gettext" (
-	%SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Build finished. The message catalogs are in %BUILDDIR%/locale.
-	goto end
-)
-
-if "%1" == "changes" (
-	%SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.The overview file is in %BUILDDIR%/changes.
-	goto end
-)
-
-if "%1" == "linkcheck" (
-	%SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Link check complete; look for any errors in the above output ^
-or in %BUILDDIR%/linkcheck/output.txt.
-	goto end
-)
-
-if "%1" == "doctest" (
-	%SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest
-	if errorlevel 1 exit /b 1
-	echo.
-	echo.Testing of doctests in the sources finished, look at the ^
-results in %BUILDDIR%/doctest/output.txt.
-	goto end
-)
-
-:end

python/tests/test_1_online.py

@@ -9,7 +9,7 @@ import pytest
 from imap_tools import AND, U
 
 import deltachat as dc
-from deltachat import account_hookimpl, Message, Chat
+from deltachat import account_hookimpl, Message
 from deltachat.tracker import ImexTracker
 
 
@@ -1658,128 +1658,6 @@ def test_ac_setup_message_twice(acfactory, lp):
     assert ac1.get_info()["fingerprint"] == ac2.get_info()["fingerprint"]
 
 
-def test_qr_setup_contact(acfactory, lp):
-    ac1, ac2 = acfactory.get_online_accounts(2)
-    lp.sec("ac1: create QR code and let ac2 scan it, starting the securejoin")
-    qr = ac1.get_setup_contact_qr()
-
-    lp.sec("ac2: start QR-code based setup contact protocol")
-    ch = ac2.qr_setup_contact(qr)
-    assert ch.id >= 10
-    ac1._evtracker.wait_securejoin_inviter_progress(1000)
-
-
-@pytest.mark.parametrize("verified_one_on_one_chats", [0, 1])
-def test_qr_join_chat(acfactory, lp, verified_one_on_one_chats):
-    ac1, ac2 = acfactory.get_online_accounts(2)
-    ac1.set_config("verified_one_on_one_chats", verified_one_on_one_chats)
-    ac2.set_config("verified_one_on_one_chats", verified_one_on_one_chats)
-
-    lp.sec("ac1: create QR code and let ac2 scan it, starting the securejoin")
-    chat = ac1.create_group_chat("hello")
-    qr = chat.get_join_qr()
-    lp.sec("ac2: start QR-code based join-group protocol")
-    ch = ac2.qr_join_chat(qr)
-    lp.sec("ac2: qr_join_chat() returned")
-    assert ch.id >= 10
-    # check that at least some of the handshake messages are deleted
-    ac1._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_DELETED")
-    ac2._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_DELETED")
-    ac1._evtracker.wait_securejoin_inviter_progress(1000)
-
-    msg = ac2._evtracker.wait_next_incoming_message()
-    assert msg.text == "Member Me ({}) added by {}.".format(ac2.get_config("addr"), ac1.get_config("addr"))
-
-    # ac1 reloads the chat.
-    chat = Chat(chat.account, chat.id)
-    assert not chat.is_protected()
-
-    # ac2 reloads the chat.
-    ch = Chat(ch.account, ch.id)
-    assert not ch.is_protected()
-
-
-def test_qr_join_chat_with_pending_bobstate_issue4894(acfactory, lp):
-    ac1, ac2, ac3, ac4 = acfactory.get_online_accounts(4)
-
-    lp.sec("ac3: verify with ac2")
-    ac3.qr_setup_contact(ac2.get_setup_contact_qr())
-    ac2._evtracker.wait_securejoin_inviter_progress(1000)
-
-    # in order for ac2 to have pending bobstate with a verified group
-    # we first create a fully joined verified group, and then start
-    # joining a second time but interrupt it, to create pending bob state
-
-    lp.sec("ac1: create verified group that ac2 fully joins")
-    ch1 = ac1.create_group_chat("ac1-shutoff group", verified=True)
-    ac2.qr_join_chat(ch1.get_join_qr())
-    ac1._evtracker.wait_securejoin_inviter_progress(1000)
-
-    # ensure ac1 can write and ac2 receives messages in verified chat
-    ch1.send_text("ac1 says hello")
-    while 1:
-        msg = ac2.wait_next_incoming_message()
-        if msg.text == "ac1 says hello":
-            assert msg.chat.is_protected()
-            break
-
-    lp.sec("ac1: let ac2 join again but shutoff ac1 in the middle of securejoin")
-    ac2.qr_join_chat(ch1.get_join_qr())
-    ac1.shutdown()
-    lp.sec("ac2 now has pending bobstate but ac1 is shutoff")
-
-    # we meanwhile expect ac3/ac2 verification started in the beginning to have completed
-    assert ac3.get_contact(ac2).is_verified()
-    assert ac2.get_contact(ac3).is_verified()
-
-    lp.sec("ac3: create a verified group VG with ac2")
-    vg = ac3.create_group_chat("ac3-created", [ac2], verified=True)
-
-    # ensure ac2 receives message in VG
-    vg.send_text("hello")
-    while 1:
-        msg = ac2.wait_next_incoming_message()
-        if msg.text == "hello":
-            assert msg.chat.is_protected()
-            break
-
-    lp.sec("ac3: create a join-code for group VG and let ac4 join, check that ac2 got it")
-    ac4.qr_join_chat(vg.get_join_qr())
-    ac3._evtracker.wait_securejoin_inviter_progress(1000)
-    while 1:
-        ev = ac2._evtracker.get()
-        if "added by unrelated SecureJoin" in str(ev):
-            return
-
-
-def test_qr_new_group_unblocked(acfactory, lp):
-    """Regression test for a bug intoduced in core v1.113.0.
-    ac2 scans a verified group QR code created by ac1.
-    This results in creation of a blocked 1:1 chat with ac1 on ac2,
-    but ac1 contact is not blocked on ac2.
-    Then ac1 creates a group, adds ac2 there and promotes it by sending a message.
-    ac2 should receive a message and create a contact request for the group.
-    Due to a bug previously ac2 created a blocked group.
-    """
-
-    ac1, ac2 = acfactory.get_online_accounts(2)
-    ac1_chat = ac1.create_group_chat("Group for joining", verified=True)
-    qr = ac1_chat.get_join_qr()
-    ac2.qr_join_chat(qr)
-
-    ac1._evtracker.wait_securejoin_inviter_progress(1000)
-
-    ac1_new_chat = ac1.create_group_chat("Another group")
-    ac1_new_chat.add_contact(ac2)
-    # Receive "Member added" message.
-    ac2._evtracker.wait_next_incoming_message()
-
-    ac1_new_chat.send_text("Hello!")
-    ac2_msg = ac2._evtracker.wait_next_incoming_message()
-    assert ac2_msg.text == "Hello!"
-    assert ac2_msg.chat.is_contact_request()
-
-
 def test_qr_email_capitalization(acfactory, lp):
     """Regression test for a bug
     that resulted in failure to propagate verification via gossip in a verified group
@@ -2527,47 +2405,6 @@ def test_delete_deltachat_folder(acfactory):
     assert "DeltaChat" in ac1.direct_imap.list_folders()
 
 
-def test_aeap_flow_verified(acfactory, lp):
-    """Test that a new address is added to a contact when it changes its address."""
-    ac1, ac2, ac1new = acfactory.get_online_accounts(3)
-
-    lp.sec("ac1: create verified-group QR, ac2 scans and joins")
-    chat = ac1.create_group_chat("hello", verified=True)
-    assert chat.is_protected()
-    qr = chat.get_join_qr()
-    lp.sec("ac2: start QR-code based join-group protocol")
-    chat2 = ac2.qr_join_chat(qr)
-    assert chat2.id >= 10
-    ac1._evtracker.wait_securejoin_inviter_progress(1000)
-
-    lp.sec("sending first message")
-    msg_out = chat.send_text("old address")
-
-    lp.sec("receiving first message")
-    ac2._evtracker.wait_next_incoming_message()  # member added message
-    msg_in_1 = ac2._evtracker.wait_next_incoming_message()
-    assert msg_in_1.text == msg_out.text
-
-    lp.sec("changing email account")
-    ac1.set_config("addr", ac1new.get_config("addr"))
-    ac1.set_config("mail_pw", ac1new.get_config("mail_pw"))
-    ac1.stop_io()
-    configtracker = ac1.configure()
-    configtracker.wait_finish()
-    ac1.start_io()
-
-    lp.sec("sending second message")
-    msg_out = chat.send_text("changed address")
-
-    lp.sec("receiving second message")
-    msg_in_2 = ac2._evtracker.wait_next_incoming_message()
-    assert msg_in_2.text == msg_out.text
-    assert msg_in_2.chat.id == msg_in_1.chat.id
-    assert msg_in_2.get_sender_contact().addr == ac1new.get_config("addr")
-    assert len(msg_in_2.chat.get_contacts()) == 2
-    assert ac1new.get_config("addr") in [contact.addr for contact in msg_in_2.chat.get_contacts()]
-
-
 def test_archived_muted_chat(acfactory, lp):
     """If an archived and muted chat receives a new message, DC_EVENT_MSGS_CHANGED for
     DC_CHAT_ID_ARCHIVED_LINK must be generated if the chat had only seen messages previously.

python/tests/test_4_lowlevel.py

@@ -156,6 +156,8 @@ def test_markseen_invalid_message_ids(acfactory):
     chat = contact1.create_chat()
     chat.send_text("one message")
     ac1._evtracker.get_matching("DC_EVENT_MSGS_CHANGED")
+    # Skip configuration-related warnings, but not errors.
+    ac1._evtracker.ensure_event_not_queued("DC_EVENT_ERROR")
     msg_ids = [9]
     lib.dc_markseen_msgs(ac1._dc_context, msg_ids, len(msg_ids))
     ac1._evtracker.ensure_event_not_queued("DC_EVENT_WARNING|DC_EVENT_ERROR")

python/tox.ini

@@ -62,9 +62,9 @@ commands =
 [testenv:doc]
 changedir=doc
 deps =
-# Pinned due to incompatibility of breathe with sphinx 7.2: <https://github.com/breathe-doc/breathe/issues/943>
-    sphinx<=7.1.2
+    sphinx
     breathe
+    sphinx_rtd_theme
 commands =
     sphinx-build -Q -w toxdoc-warnings.log -b html . _build/html
 

release-date.in

@@ -1 +1 @@
-2023-11-14
+2023-12-02

scripts/README.md

@@ -39,6 +39,8 @@ and an own build machine.
 
 - `android-rpc-server.sh` compiles binaries of `deltachat-rpc-server` using Android NDK.
 
+- `build-python-docs.sh` builds Python documentation into `dist/html/`.
+
 ## Triggering runs on the build machine locally (fast!)
 
 There is experimental support for triggering a remote Python or Rust test run 

scripts/build-python-docs.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+export DCC_RS_TARGET=debug
+export DCC_RS_DEV="$PWD"
+cargo build -p deltachat_ffi --features jsonrpc
+
+python3 -m venv venv
+venv/bin/pip install ./python
+venv/bin/pip install ./deltachat-rpc-client
+venv/bin/pip install sphinx breathe sphinx_rtd_theme
+venv/bin/pip install ./deltachat-rpc-client
+venv/bin/sphinx-build -b html -a python/doc/ dist/html

scripts/concourse/docs_wheels.yml

@@ -102,8 +102,6 @@ jobs:
             - name: deltachat-core-rust-release
               path: .
           outputs:
-            - name: py-docs
-              path: ./python/doc/_build/
             # Binary wheels
             - name: py-wheels
               path: ./python/.docker-tox/wheelhouse/
@@ -115,28 +113,6 @@ jobs:
               - |
                 scripts/run_all.sh
 
-      # Upload python docs to py.delta.chat
-      - task: upload-py-docs
-        config:
-          inputs:
-            - name: py-docs
-          image_resource:
-            type: registry-image
-            source:
-              repository: alpine
-          platform: linux
-          run:
-            path: sh
-            args:
-              - -ec
-              - |
-                apk add --no-cache rsync openssh-client
-                mkdir -p ~/.ssh
-                chmod 700 ~/.ssh
-                echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
-                chmod 600 ~/.ssh/id_ed25519
-                rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete py-docs/html/ delta@py.delta.chat:build/master
-
       # Upload x86_64 wheels and source packages
       - task: upload-wheels
         config:

scripts/run_all.sh

@@ -1,6 +1,6 @@
 #!/bin/sh
 #
-# Build the Delta Chat Core Rust library, Python wheels and docs 
+# Build the Delta Chat Core Rust library and Python wheels
 
 set -e -x
 
@@ -34,9 +34,3 @@ unset CHATMAIL_DOMAIN
 tox --workdir "$TOXWORKDIR" -e py37,py38,py39,py310,py311,py312,pypy37,pypy38,pypy39,pypy310 --skip-missing-interpreters true
 
 auditwheel repair "$TOXWORKDIR"/wheelhouse/deltachat* -w "$TOXWORKDIR/wheelhouse"
-
-
-echo -----------------------
-echo generating python docs
-echo -----------------------
-tox --workdir "$TOXWORKDIR" -e doc

src/accounts.rs

@@ -8,11 +8,14 @@ use anyhow::{ensure, Context as _, Result};
 use serde::{Deserialize, Serialize};
 use tokio::fs;
 use tokio::io::AsyncWriteExt;
-use tokio::sync::oneshot;
 use tokio::task::JoinHandle;
-use tokio::time::{sleep, Duration};
 use uuid::Uuid;
 
+#[cfg(not(target_os = "ios"))]
+use tokio::sync::oneshot;
+#[cfg(not(target_os = "ios"))]
+use tokio::time::{sleep, Duration};
+
 use crate::context::Context;
 use crate::events::{Event, EventEmitter, EventType, Events};
 use crate::stock_str::StockStrings;
@@ -303,6 +306,7 @@ impl Accounts {
 const CONFIG_NAME: &str = "accounts.toml";
 
 /// Lockfile name.
+#[cfg(not(target_os = "ios"))]
 const LOCKFILE_NAME: &str = "accounts.lock";
 
 /// Database file name.
@@ -338,22 +342,16 @@ impl Drop for Config {
 }
 
 impl Config {
-    /// Creates a new Config for `file`, but doesn't open/sync it.
-    async fn new_nosync(file: PathBuf, lock: bool) -> Result<Self> {
-        let dir = file.parent().context("Cannot get config file directory")?;
-        let inner = InnerConfig {
-            accounts: Vec::new(),
-            selected_account: 0,
-            next_id: 1,
-        };
-        if !lock {
-            let cfg = Self {
-                file,
-                inner,
-                lock_task: None,
-            };
-            return Ok(cfg);
-        }
+    #[cfg(target_os = "ios")]
+    async fn create_lock_task(_dir: PathBuf) -> Result<Option<JoinHandle<anyhow::Result<()>>>> {
+        // Do not lock accounts.toml on iOS.
+        // This results in 0xdead10cc crashes on suspend.
+        // iOS itself ensures that multiple instances of Delta Chat are not running.
+        Ok(None)
+    }
+
+    #[cfg(not(target_os = "ios"))]
+    async fn create_lock_task(dir: PathBuf) -> Result<Option<JoinHandle<anyhow::Result<()>>>> {
         let lockfile = dir.join(LOCKFILE_NAME);
         let mut lock = fd_lock::RwLock::new(fs::File::create(lockfile).await?);
         let (locked_tx, locked_rx) = oneshot::channel();
@@ -384,12 +382,32 @@ impl Config {
             rx.await?;
             Ok(())
         });
+        locked_rx.await?;
+        Ok(Some(lock_task))
+    }
+
+    /// Creates a new Config for `file`, but doesn't open/sync it.
+    async fn new_nosync(file: PathBuf, lock: bool) -> Result<Self> {
+        let dir = file.parent().context("Cannot get config file directory")?;
+        let inner = InnerConfig {
+            accounts: Vec::new(),
+            selected_account: 0,
+            next_id: 1,
+        };
+        if !lock {
+            let cfg = Self {
+                file,
+                inner,
+                lock_task: None,
+            };
+            return Ok(cfg);
+        }
+        let lock_task = Self::create_lock_task(dir.to_path_buf()).await?;
         let cfg = Self {
             file,
             inner,
-            lock_task: Some(lock_task),
+            lock_task,
         };
-        locked_rx.await?;
         Ok(cfg)
     }
 

src/aheader.rs

@@ -67,7 +67,7 @@ impl Aheader {
 
 impl fmt::Display for Aheader {
     fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
-        write!(fmt, "addr={};", self.addr)?;
+        write!(fmt, "addr={};", self.addr.to_lowercase())?;
         if self.prefer_encrypt == EncryptPreference::Mutual {
             write!(fmt, " prefer-encrypt=mutual;")?;
         }
@@ -262,5 +262,16 @@ mod tests {
             )
         )
         .contains("prefer-encrypt"));
+
+        // Always lowercase the address in the header.
+        assert!(format!(
+            "{}",
+            Aheader::new(
+                "TeSt@eXaMpLe.cOm".to_string(),
+                SignedPublicKey::from_base64(RAWKEY).unwrap(),
+                EncryptPreference::Mutual
+            )
+        )
+        .contains("test@example.com"));
     }
 }

src/chat.rs

@@ -1,5 +1,6 @@
 //! # Chat module.
 
+use std::cmp;
 use std::collections::{HashMap, HashSet};
 use std::convert::{TryFrom, TryInto};
 use std::fmt;
@@ -21,7 +22,7 @@ use crate::constants::{
     Blocked, Chattype, DC_CHAT_ID_ALLDONE_HINT, DC_CHAT_ID_ARCHIVED_LINK, DC_CHAT_ID_LAST_SPECIAL,
     DC_CHAT_ID_TRASH, DC_RESEND_USER_AVATAR_DAYS,
 };
-use crate::contact::{self, Contact, ContactAddress, ContactId, Origin, VerifiedStatus};
+use crate::contact::{self, Contact, ContactAddress, ContactId, Origin};
 use crate::context::Context;
 use crate::debug_logging::maybe_set_logging_xdc;
 use crate::download::DownloadState;
@@ -34,7 +35,7 @@ use crate::message::{self, Message, MessageState, MsgId, Viewtype};
 use crate::mimefactory::MimeFactory;
 use crate::mimeparser::SystemMessage;
 use crate::param::{Param, Params};
-use crate::peerstate::{Peerstate, PeerstateVerifiedStatus};
+use crate::peerstate::Peerstate;
 use crate::receive_imf::ReceivedMsg;
 use crate::smtp::send_msg_to_smtp;
 use crate::sql;
@@ -289,6 +290,7 @@ impl ChatId {
 
     /// Create a group or mailinglist raw database record with the given parameters.
     /// The function does not add SELF nor checks if the record already exists.
+    #[allow(clippy::too_many_arguments)]
     pub(crate) async fn create_multiuser_record(
         context: &Context,
         chattype: Chattype,
@@ -297,9 +299,10 @@ impl ChatId {
         create_blocked: Blocked,
         create_protected: ProtectionStatus,
         param: Option<String>,
+        timestamp: i64,
     ) -> Result<Self> {
         let grpname = strip_rtlo_characters(grpname);
-        let smeared_time = create_smeared_timestamp(context);
+        let timestamp = cmp::min(timestamp, smeared_time(context));
         let row_id =
             context.sql.insert(
                 "INSERT INTO chats (type, name, grpid, blocked, created_timestamp, protected, param) VALUES(?, ?, ?, ?, ?, ?, ?);",
@@ -308,7 +311,7 @@ impl ChatId {
                     &grpname,
                     grpid,
                     create_blocked,
-                    smeared_time,
+                    timestamp,
                     create_protected,
                     param.unwrap_or_default(),
                 ),
@@ -318,7 +321,7 @@ impl ChatId {
 
         if create_protected == ProtectionStatus::Protected {
             chat_id
-                .add_protection_msg(context, ProtectionStatus::Protected, None, smeared_time)
+                .add_protection_msg(context, ProtectionStatus::Protected, None, timestamp)
                 .await?;
         }
 
@@ -503,7 +506,7 @@ impl ChatId {
                     let contact_ids = get_chat_contacts(context, self).await?;
                     for contact_id in contact_ids {
                         let contact = Contact::get_by_id(context, contact_id).await?;
-                        if contact.is_verified(context).await? != VerifiedStatus::BidirectVerified {
+                        if !contact.is_verified(context).await? {
                             bail!("{} is not verified.", contact.get_display_name());
                         }
                     }
@@ -1202,11 +1205,7 @@ impl ChatId {
             let peerstate = Peerstate::from_addr(context, addr).await?;
 
             match peerstate
-                .filter(|peerstate| {
-                    peerstate
-                        .peek_key(PeerstateVerifiedStatus::Unverified)
-                        .is_some()
-                })
+                .filter(|peerstate| peerstate.peek_key(false).is_some())
                 .map(|peerstate| peerstate.prefer_encrypt)
             {
                 Some(EncryptPreference::Mutual) => ret_mutual += &format!("{addr}\n"),
@@ -1935,11 +1934,18 @@ impl Chat {
 
     /// Sends a `SyncAction` synchronising chat contacts to other devices.
     pub(crate) async fn sync_contacts(&self, context: &Context) -> Result<()> {
-        let mut addrs = Vec::new();
-        for contact_id in get_chat_contacts(context, self.id).await? {
-            let contact = Contact::get_by_id(context, contact_id).await?;
-            addrs.push(contact.get_addr().to_string());
-        }
+        let addrs = context
+            .sql
+            .query_map(
+                "SELECT c.addr \
+                FROM contacts c INNER JOIN chats_contacts cc \
+                ON c.id=cc.contact_id \
+                WHERE cc.chat_id=?",
+                (self.id,),
+                |row| row.get::<_, String>(0),
+                |addrs| addrs.collect::<Result<Vec<_>, _>>().map_err(Into::into),
+            )
+            .await?;
         self.sync(context, SyncAction::SetContacts(addrs)).await
     }
 
@@ -3437,9 +3443,7 @@ pub(crate) async fn add_contact_to_chat_ex(
         }
     } else {
         // else continue and send status mail
-        if chat.is_protected()
-            && contact.is_verified(context).await? != VerifiedStatus::BidirectVerified
-        {
+        if chat.is_protected() && !contact.is_verified(context).await? {
             error!(
                 context,
                 "Only bidirectional verified contacts can be added to protected chats."
@@ -3454,8 +3458,8 @@ pub(crate) async fn add_contact_to_chat_ex(
     if chat.typ == Chattype::Group && chat.is_promoted() {
         msg.viewtype = Viewtype::Text;
 
-        let contact_addr = contact.get_addr();
-        msg.text = stock_str::msg_add_member_local(context, contact_addr, ContactId::SELF).await;
+        let contact_addr = contact.get_addr().to_lowercase();
+        msg.text = stock_str::msg_add_member_local(context, &contact_addr, ContactId::SELF).await;
         msg.param.set_cmd(SystemMessage::MemberAddedToGroup);
         msg.param.set(Param::Arg, contact_addr);
         msg.param.set_int(Param::Arg2, from_handshake.into());
@@ -3626,7 +3630,7 @@ pub async fn remove_contact_from_chat(
                         .await;
                     }
                     msg.param.set_cmd(SystemMessage::MemberRemovedFromGroup);
-                    msg.param.set(Param::Arg, contact.get_addr());
+                    msg.param.set(Param::Arg, contact.get_addr().to_lowercase());
                     msg.id = send_msg(context, chat_id, &mut msg).await?;
                 } else {
                     sync = Sync;
@@ -3681,11 +3685,10 @@ pub async fn set_chat_name(context: &Context, chat_id: ChatId, new_name: &str) -
 
 async fn rename_ex(
     context: &Context,
-    sync: sync::Sync,
+    mut sync: sync::Sync,
     chat_id: ChatId,
     new_name: &str,
 ) -> Result<()> {
-    let sync_name = new_name;
     let new_name = improve_single_line_input(new_name);
     /* the function only sets the names of group chats; normal chats get their names from the contacts */
     let mut success = false;
@@ -3728,6 +3731,7 @@ async fn rename_ex(
                 }
                 msg.id = send_msg(context, chat_id, &mut msg).await?;
                 context.emit_msgs_changed(chat_id, msg.id);
+                sync = Nosync;
             }
             context.emit_event(EventType::ChatModified(chat_id));
             success = true;
@@ -3737,8 +3741,8 @@ async fn rename_ex(
     if !success {
         bail!("Failed to set name");
     }
-    if sync.into() {
-        let sync_name = sync_name.to_string();
+    if sync.into() && chat.name != new_name {
+        let sync_name = new_name.to_string();
         chat.sync(context, SyncAction::Rename(sync_name))
             .await
             .log_err(context)
@@ -4227,7 +4231,7 @@ async fn set_contacts_by_addrs(context: &Context, id: ChatId, addrs: &[String])
     let mut contacts = HashSet::new();
     for addr in addrs {
         let contact_addr = ContactAddress::new(addr)?;
-        let contact = Contact::add_or_lookup(context, "", contact_addr, Origin::Hidden)
+        let contact = Contact::add_or_lookup(context, "", &contact_addr, Origin::Hidden)
             .await?
             .0;
         contacts.insert(contact);
@@ -4326,7 +4330,7 @@ mod tests {
     use crate::contact::{Contact, ContactAddress};
     use crate::message::delete_msgs;
     use crate::receive_imf::receive_imf;
-    use crate::test_utils::{TestContext, TestContextManager};
+    use crate::test_utils::{sync, TestContext, TestContextManager};
     use strum::IntoEnumIterator;
     use tokio::fs;
 
@@ -5585,7 +5589,7 @@ mod tests {
         let (contact_id, _) = Contact::add_or_lookup(
             &t,
             "",
-            ContactAddress::new("foo@bar.org")?,
+            &ContactAddress::new("foo@bar.org")?,
             Origin::IncomingUnknownTo,
         )
         .await?;
@@ -6596,7 +6600,7 @@ mod tests {
         let (contact_id, _) = Contact::add_or_lookup(
             &t,
             "",
-            ContactAddress::new("foo@bar.org")?,
+            &ContactAddress::new("foo@bar.org")?,
             Origin::ManuallyCreated,
         )
         .await?;
@@ -6880,102 +6884,113 @@ mod tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-    async fn test_sync_alter_chat() -> Result<()> {
-        let alices = [
-            TestContext::new_alice().await,
-            TestContext::new_alice().await,
-        ];
-        for a in &alices {
+    async fn test_sync_blocked() -> Result<()> {
+        let alice0 = &TestContext::new_alice().await;
+        let alice1 = &TestContext::new_alice().await;
+        for a in [alice0, alice1] {
             a.set_config_bool(Config::SyncMsgs, true).await?;
         }
         let bob = TestContext::new_bob().await;
 
-        let ba_chat = bob.create_chat(&alices[0]).await;
+        let ba_chat = bob.create_chat(alice0).await;
         let sent_msg = bob.send_text(ba_chat.id, "hi").await;
-        let a0b_chat_id = alices[0].recv_msg(&sent_msg).await.chat_id;
-        alices[1].recv_msg(&sent_msg).await;
-        let ab_contact_ids = [
-            alices[0].add_or_lookup_contact(&bob).await.id,
-            alices[1].add_or_lookup_contact(&bob).await.id,
-        ];
+        let a0b_chat_id = alice0.recv_msg(&sent_msg).await.chat_id;
+        alice1.recv_msg(&sent_msg).await;
+        let a0b_contact_id = alice0.add_or_lookup_contact(&bob).await.id;
 
-        async fn sync(alices: &[TestContext]) -> Result<()> {
-            let sync_msg = alices.get(0).unwrap().pop_sent_msg().await;
-            alices.get(1).unwrap().recv_msg(&sync_msg).await;
-            Ok(())
-        }
-
-        assert_eq!(alices[1].get_chat(&bob).await.blocked, Blocked::Request);
-        a0b_chat_id.accept(&alices[0]).await?;
-        sync(&alices).await?;
-        assert_eq!(alices[1].get_chat(&bob).await.blocked, Blocked::Not);
-        a0b_chat_id.block(&alices[0]).await?;
-        sync(&alices).await?;
-        assert_eq!(alices[1].get_chat(&bob).await.blocked, Blocked::Yes);
-        a0b_chat_id.unblock(&alices[0]).await?;
-        sync(&alices).await?;
-        assert_eq!(alices[1].get_chat(&bob).await.blocked, Blocked::Not);
+        assert_eq!(alice1.get_chat(&bob).await.blocked, Blocked::Request);
+        a0b_chat_id.accept(alice0).await?;
+        sync(alice0, alice1).await;
+        assert_eq!(alice1.get_chat(&bob).await.blocked, Blocked::Not);
+        a0b_chat_id.block(alice0).await?;
+        sync(alice0, alice1).await;
+        assert_eq!(alice1.get_chat(&bob).await.blocked, Blocked::Yes);
+        a0b_chat_id.unblock(alice0).await?;
+        sync(alice0, alice1).await;
+        assert_eq!(alice1.get_chat(&bob).await.blocked, Blocked::Not);
 
         // Unblocking a 1:1 chat doesn't unblock the contact currently.
-        Contact::unblock(&alices[0], ab_contact_ids[0]).await?;
+        Contact::unblock(alice0, a0b_contact_id).await?;
 
-        assert!(!alices[1].add_or_lookup_contact(&bob).await.is_blocked());
-        Contact::block(&alices[0], ab_contact_ids[0]).await?;
-        sync(&alices).await?;
-        assert!(alices[1].add_or_lookup_contact(&bob).await.is_blocked());
-        Contact::unblock(&alices[0], ab_contact_ids[0]).await?;
-        sync(&alices).await?;
-        assert!(!alices[1].add_or_lookup_contact(&bob).await.is_blocked());
+        assert!(!alice1.add_or_lookup_contact(&bob).await.is_blocked());
+        Contact::block(alice0, a0b_contact_id).await?;
+        sync(alice0, alice1).await;
+        assert!(alice1.add_or_lookup_contact(&bob).await.is_blocked());
+        Contact::unblock(alice0, a0b_contact_id).await?;
+        sync(alice0, alice1).await;
+        assert!(!alice1.add_or_lookup_contact(&bob).await.is_blocked());
 
         // Test accepting and blocking groups. This way we test:
         // - Group chats synchronisation.
         // - That blocking a group deletes it on other devices.
         let fiona = TestContext::new_fiona().await;
         let fiona_grp_chat_id = fiona
-            .create_group_with_members(ProtectionStatus::Unprotected, "grp", &[&alices[0]])
+            .create_group_with_members(ProtectionStatus::Unprotected, "grp", &[alice0])
             .await;
         let sent_msg = fiona.send_text(fiona_grp_chat_id, "hi").await;
-        let a0_grp_chat_id = alices[0].recv_msg(&sent_msg).await.chat_id;
-        let a1_grp_chat_id = alices[1].recv_msg(&sent_msg).await.chat_id;
-        let a1_grp_chat = Chat::load_from_db(&alices[1], a1_grp_chat_id).await?;
+        let a0_grp_chat_id = alice0.recv_msg(&sent_msg).await.chat_id;
+        let a1_grp_chat_id = alice1.recv_msg(&sent_msg).await.chat_id;
+        let a1_grp_chat = Chat::load_from_db(alice1, a1_grp_chat_id).await?;
         assert_eq!(a1_grp_chat.blocked, Blocked::Request);
-        a0_grp_chat_id.accept(&alices[0]).await?;
-        sync(&alices).await?;
-        let a1_grp_chat = Chat::load_from_db(&alices[1], a1_grp_chat_id).await?;
+        a0_grp_chat_id.accept(alice0).await?;
+        sync(alice0, alice1).await;
+        let a1_grp_chat = Chat::load_from_db(alice1, a1_grp_chat_id).await?;
         assert_eq!(a1_grp_chat.blocked, Blocked::Not);
-        a0_grp_chat_id.block(&alices[0]).await?;
-        sync(&alices).await?;
-        assert!(Chat::load_from_db(&alices[1], a1_grp_chat_id)
-            .await
-            .is_err());
+        a0_grp_chat_id.block(alice0).await?;
+        sync(alice0, alice1).await;
+        assert!(Chat::load_from_db(alice1, a1_grp_chat_id).await.is_err());
         assert!(
-            !alices[1]
+            !alice1
                 .sql
                 .exists("SELECT COUNT(*) FROM chats WHERE id=?", (a1_grp_chat_id,))
                 .await?
         );
 
-        // Test syncing of chat visibility on a self-chat. This way we test:
-        // - Self-chat synchronisation.
-        // - That sync messages don't unarchive the self-chat.
-        let a0self_chat_id = alices[0].get_self_chat().await.id;
+        Ok(())
+    }
+
+    /// Tests syncing of chat visibility on a self-chat. This way we test:
+    /// - Self-chat synchronisation.
+    /// - That sync messages don't unarchive the self-chat.
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_sync_visibility() -> Result<()> {
+        let alice0 = &TestContext::new_alice().await;
+        let alice1 = &TestContext::new_alice().await;
+        for a in [alice0, alice1] {
+            a.set_config_bool(Config::SyncMsgs, true).await?;
+        }
+        let a0self_chat_id = alice0.get_self_chat().await.id;
+
         assert_eq!(
-            alices[1].get_self_chat().await.get_visibility(),
+            alice1.get_self_chat().await.get_visibility(),
             ChatVisibility::Normal
         );
         let mut visibilities =
             ChatVisibility::iter().chain(std::iter::once(ChatVisibility::Normal));
         visibilities.next();
         for v in visibilities {
-            a0self_chat_id.set_visibility(&alices[0], v).await?;
-            sync(&alices).await?;
-            for a in &alices {
+            a0self_chat_id.set_visibility(alice0, v).await?;
+            sync(alice0, alice1).await;
+            for a in [alice0, alice1] {
                 assert_eq!(a.get_self_chat().await.get_visibility(), v);
             }
         }
+        Ok(())
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_sync_muted() -> Result<()> {
+        let alice0 = &TestContext::new_alice().await;
+        let alice1 = &TestContext::new_alice().await;
+        for a in [alice0, alice1] {
+            a.set_config_bool(Config::SyncMsgs, true).await?;
+        }
+        let bob = TestContext::new_bob().await;
+        let a0b_chat_id = alice0.create_chat(&bob).await.id;
+        alice1.create_chat(&bob).await;
 
         assert_eq!(
-            alices[1].get_chat(&bob).await.mute_duration,
+            alice1.get_chat(&bob).await.mute_duration,
             MuteDuration::NotMuted
         );
         let mute_durations = [
@@ -6984,8 +6999,8 @@ mod tests {
             MuteDuration::NotMuted,
         ];
         for m in mute_durations {
-            set_muted(&alices[0], a0b_chat_id, m).await?;
-            sync(&alices).await?;
+            set_muted(alice0, a0b_chat_id, m).await?;
+            sync(alice0, alice1).await;
             let m = match m {
                 MuteDuration::Until(time) => MuteDuration::Until(
                     SystemTime::UNIX_EPOCH
@@ -6995,42 +7010,76 @@ mod tests {
                 ),
                 _ => m,
             };
-            assert_eq!(alices[1].get_chat(&bob).await.mute_duration, m);
+            assert_eq!(alice1.get_chat(&bob).await.mute_duration, m);
         }
+        Ok(())
+    }
 
-        let a0_broadcast_id = create_broadcast_list(&alices[0]).await?;
-        sync(&alices).await?;
-        let a0_broadcast_chat = Chat::load_from_db(&alices[0], a0_broadcast_id).await?;
-        set_chat_name(&alices[0], a0_broadcast_id, "Broadcast list 42").await?;
-        sync(&alices).await?;
-        let a1_broadcast_id = get_chat_id_by_grpid(&alices[1], &a0_broadcast_chat.grpid)
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_sync_broadcast() -> Result<()> {
+        let alice0 = &TestContext::new_alice().await;
+        let alice1 = &TestContext::new_alice().await;
+        for a in [alice0, alice1] {
+            a.set_config_bool(Config::SyncMsgs, true).await?;
+        }
+        let bob = TestContext::new_bob().await;
+        let a0b_contact_id = alice0.add_or_lookup_contact(&bob).await.id;
+
+        let a0_broadcast_id = create_broadcast_list(alice0).await?;
+        sync(alice0, alice1).await;
+        let a0_broadcast_chat = Chat::load_from_db(alice0, a0_broadcast_id).await?;
+        let a1_broadcast_id = get_chat_id_by_grpid(alice1, &a0_broadcast_chat.grpid)
             .await?
             .unwrap()
             .0;
-        let a1_broadcast_chat = Chat::load_from_db(&alices[1], a1_broadcast_id).await?;
+        let a1_broadcast_chat = Chat::load_from_db(alice1, a1_broadcast_id).await?;
         assert_eq!(a1_broadcast_chat.get_type(), Chattype::Broadcast);
-        assert_eq!(a1_broadcast_chat.get_name(), "Broadcast list 42");
-        assert!(get_chat_contacts(&alices[1], a1_broadcast_id)
-            .await?
-            .is_empty());
-        add_contact_to_chat(&alices[0], a0_broadcast_id, ab_contact_ids[0]).await?;
-        sync(&alices).await?;
+        assert_eq!(a1_broadcast_chat.get_name(), a0_broadcast_chat.get_name());
+        assert!(get_chat_contacts(alice1, a1_broadcast_id).await?.is_empty());
+        add_contact_to_chat(alice0, a0_broadcast_id, a0b_contact_id).await?;
+        sync(alice0, alice1).await;
+        let a1b_contact_id = Contact::lookup_id_by_addr(
+            alice1,
+            &bob.get_config(Config::Addr).await?.unwrap(),
+            Origin::Hidden,
+        )
+        .await?
+        .unwrap();
         assert_eq!(
-            get_chat_contacts(&alices[1], a1_broadcast_id).await?,
-            vec![ab_contact_ids[1]]
+            get_chat_contacts(alice1, a1_broadcast_id).await?,
+            vec![a1b_contact_id]
         );
-        let sent_msg = alices[1].send_text(a1_broadcast_id, "hi").await;
+        let sent_msg = alice1.send_text(a1_broadcast_id, "hi").await;
         let msg = bob.recv_msg(&sent_msg).await;
         let chat = Chat::load_from_db(&bob, msg.chat_id).await?;
         assert_eq!(chat.get_type(), Chattype::Mailinglist);
-        let msg = alices[0].recv_msg(&sent_msg).await;
+        let msg = alice0.recv_msg(&sent_msg).await;
         assert_eq!(msg.chat_id, a0_broadcast_id);
-        remove_contact_from_chat(&alices[0], a0_broadcast_id, ab_contact_ids[0]).await?;
-        sync(&alices).await?;
-        assert!(get_chat_contacts(&alices[1], a1_broadcast_id)
-            .await?
-            .is_empty());
+        remove_contact_from_chat(alice0, a0_broadcast_id, a0b_contact_id).await?;
+        sync(alice0, alice1).await;
+        assert!(get_chat_contacts(alice1, a1_broadcast_id).await?.is_empty());
+        Ok(())
+    }
 
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_sync_name() -> Result<()> {
+        let alice0 = &TestContext::new_alice().await;
+        let alice1 = &TestContext::new_alice().await;
+        for a in [alice0, alice1] {
+            a.set_config_bool(Config::SyncMsgs, true).await?;
+        }
+        let a0_broadcast_id = create_broadcast_list(alice0).await?;
+        sync(alice0, alice1).await;
+        let a0_broadcast_chat = Chat::load_from_db(alice0, a0_broadcast_id).await?;
+        set_chat_name(alice0, a0_broadcast_id, "Broadcast list 42").await?;
+        sync(alice0, alice1).await;
+        let a1_broadcast_id = get_chat_id_by_grpid(alice1, &a0_broadcast_chat.grpid)
+            .await?
+            .unwrap()
+            .0;
+        let a1_broadcast_chat = Chat::load_from_db(alice1, a1_broadcast_id).await?;
+        assert_eq!(a1_broadcast_chat.get_type(), Chattype::Broadcast);
+        assert_eq!(a1_broadcast_chat.get_name(), "Broadcast list 42");
         Ok(())
     }
 }

src/config.rs

@@ -5,6 +5,7 @@ use std::path::Path;
 use std::str::FromStr;
 
 use anyhow::{ensure, Context as _, Result};
+use serde::{Deserialize, Serialize};
 use strum::{EnumProperty, IntoEnumIterator};
 use strum_macros::{AsRefStr, Display, EnumIter, EnumProperty, EnumString};
 
@@ -13,8 +14,10 @@ use crate::constants::DC_VERSION_STR;
 use crate::contact::addr_cmp;
 use crate::context::Context;
 use crate::events::EventType;
+use crate::log::LogExt;
 use crate::mimefactory::RECOMMENDED_FILE_SIZE;
 use crate::provider::{get_provider_by_id, Provider};
+use crate::sync::{self, Sync::*, SyncData};
 use crate::tools::{get_abs_path, improve_single_line_input, EmailAddress};
 
 /// The available configuration keys.
@@ -31,6 +34,8 @@ use crate::tools::{get_abs_path, improve_single_line_input, EmailAddress};
     EnumProperty,
     PartialOrd,
     Ord,
+    Serialize,
+    Deserialize,
 )]
 #[strum(serialize_all = "snake_case")]
 pub enum Config {
@@ -340,6 +345,24 @@ pub enum Config {
     VerifiedOneOnOneChats,
 }
 
+impl Config {
+    /// Whether the config option is synced across devices.
+    ///
+    /// This must be checked on both sides so that if there are different client versions, the
+    /// synchronisation of a particular option is either done or not done in both directions.
+    /// Moreover, receivers of a config value need to check if a key can be synced because some
+    /// settings (e.g. Avatar path) could otherwise lead to exfiltration of files from a receiver's
+    /// device if we assume an attacker to have control of a device in a multi-device setting or if
+    /// multiple users are sharing an account. Another example is `Self::SyncMsgs` itself which
+    /// mustn't be controlled by other devices.
+    pub(crate) fn is_synced(&self) -> bool {
+        matches!(
+            self,
+            Self::Displayname | Self::MdnsEnabled | Self::ShowEmails
+        )
+    }
+}
+
 impl Context {
     /// Returns true if configuration value is set for the given key.
     pub async fn config_exists(&self, key: Config) -> Result<bool> {
@@ -460,6 +483,16 @@ impl Context {
     /// Set the given config key.
     /// If `None` is passed as a value the value is cleared and set to the default if there is one.
     pub async fn set_config(&self, key: Config, value: Option<&str>) -> Result<()> {
+        self.set_config_ex(key.is_synced().into(), key, value).await
+    }
+
+    pub(crate) async fn set_config_ex(
+        &self,
+        sync: sync::Sync,
+        key: Config,
+        mut value: Option<&str>,
+    ) -> Result<()> {
+        let better_value;
         match key {
             Config::Selfavatar => {
                 self.sql
@@ -486,10 +519,11 @@ impl Context {
                 ret?
             }
             Config::Displayname => {
-                let value = value.map(improve_single_line_input);
-                self.sql
-                    .set_raw_config(key.as_ref(), value.as_deref())
-                    .await?;
+                if let Some(v) = value {
+                    better_value = improve_single_line_input(v);
+                    value = Some(&better_value);
+                }
+                self.sql.set_raw_config(key.as_ref(), value).await?;
             }
             Config::Socks5Enabled
             | Config::BccSelf
@@ -513,10 +547,32 @@ impl Context {
                 );
                 self.sql.set_raw_config(key.as_ref(), value).await?;
             }
+            Config::Addr => {
+                self.sql
+                    .set_raw_config(key.as_ref(), value.map(|s| s.to_lowercase()).as_deref())
+                    .await?;
+            }
             _ => {
                 self.sql.set_raw_config(key.as_ref(), value).await?;
             }
         }
+
+        if sync != Sync {
+            return Ok(());
+        }
+        let Some(val) = value else {
+            return Ok(());
+        };
+        let val = val.to_string();
+        if self
+            .add_sync_item(SyncData::Config { key, val })
+            .await
+            .log_err(self)
+            .is_err()
+        {
+            return Ok(());
+        }
+        self.send_sync_msg().await.log_err(self).ok();
         Ok(())
     }
 
@@ -648,7 +704,7 @@ mod tests {
 
     use super::*;
     use crate::constants;
-    use crate::test_utils::TestContext;
+    use crate::test_utils::{sync, TestContext};
 
     #[test]
     fn test_to_string() {
@@ -662,6 +718,21 @@ mod tests {
         );
     }
 
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_set_config_addr() {
+        let t = TestContext::new().await;
+
+        // Test that uppercase address get lowercased.
+        assert!(t
+            .set_config(Config::Addr, Some("Foobar@eXample.oRg"))
+            .await
+            .is_ok());
+        assert_eq!(
+            t.get_config(Config::Addr).await.unwrap().unwrap(),
+            "foobar@example.org"
+        );
+    }
+
     /// Tests that "bot" config can only be set to "0" or "1".
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_set_config_bot() {
@@ -777,4 +848,65 @@ mod tests {
 
         Ok(())
     }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_sync() -> Result<()> {
+        let alice0 = TestContext::new_alice().await;
+        let alice1 = TestContext::new_alice().await;
+        for a in [&alice0, &alice1] {
+            a.set_config_bool(Config::SyncMsgs, true).await?;
+        }
+
+        let mdns_enabled = alice0.get_config_bool(Config::MdnsEnabled).await?;
+        // Alice1 has a different config value.
+        alice1
+            .set_config_bool(Config::MdnsEnabled, !mdns_enabled)
+            .await?;
+        // This changes nothing, but still sends a sync message.
+        alice0
+            .set_config_bool(Config::MdnsEnabled, mdns_enabled)
+            .await?;
+        sync(&alice0, &alice1).await;
+        assert_eq!(
+            alice1.get_config_bool(Config::MdnsEnabled).await?,
+            mdns_enabled
+        );
+
+        // Reset to default. Test that it's not synced because defaults may differ across client
+        // versions.
+        alice0.set_config(Config::MdnsEnabled, None).await?;
+        assert!(alice0.get_config_bool(Config::MdnsEnabled).await?);
+        alice0.set_config_bool(Config::MdnsEnabled, false).await?;
+        sync(&alice0, &alice1).await;
+        assert!(!alice1.get_config_bool(Config::MdnsEnabled).await?);
+
+        let show_emails = alice0.get_config_bool(Config::ShowEmails).await?;
+        alice0
+            .set_config_bool(Config::ShowEmails, !show_emails)
+            .await?;
+        sync(&alice0, &alice1).await;
+        assert_eq!(
+            alice1.get_config_bool(Config::ShowEmails).await?,
+            !show_emails
+        );
+
+        // `Config::SyncMsgs` mustn't be synced.
+        alice0.set_config_bool(Config::SyncMsgs, false).await?;
+        alice0.set_config_bool(Config::SyncMsgs, true).await?;
+        alice0.set_config_bool(Config::MdnsEnabled, true).await?;
+        sync(&alice0, &alice1).await;
+        assert!(alice1.get_config_bool(Config::MdnsEnabled).await?);
+
+        // Usual sync scenario.
+        alice0
+            .set_config(Config::Displayname, Some("Alice Sync"))
+            .await?;
+        sync(&alice0, &alice1).await;
+        assert_eq!(
+            alice1.get_config(Config::Displayname).await?,
+            Some("Alice Sync".to_string())
+        );
+
+        Ok(())
+    }
 }

src/configure.rs

@@ -34,6 +34,7 @@ use crate::provider::{Protocol, Socket, UsernamePattern};
 use crate::smtp::Smtp;
 use crate::socks::Socks5Config;
 use crate::stock_str;
+use crate::sync::Sync::*;
 use crate::tools::{time, EmailAddress};
 use crate::{chat, e2ee, provider};
 
@@ -132,7 +133,9 @@ async fn on_configure_completed(
             for def in config_defaults {
                 if !context.config_exists(def.key).await? {
                     info!(context, "apply config_defaults {}={}", def.key, def.value);
-                    context.set_config(def.key, Some(def.value)).await?;
+                    context
+                        .set_config_ex(Nosync, def.key, Some(def.value))
+                        .await?;
                 } else {
                     info!(
                         context,

src/contact.rs

@@ -30,7 +30,7 @@ use crate::login_param::LoginParam;
 use crate::message::MessageState;
 use crate::mimeparser::AvatarAction;
 use crate::param::{Param, Params};
-use crate::peerstate::{Peerstate, PeerstateVerifiedStatus};
+use crate::peerstate::Peerstate;
 use crate::sql::{self, params_iter};
 use crate::sync::{self, Sync::*, SyncData};
 use crate::tools::{
@@ -43,43 +43,43 @@ use crate::{chat, stock_str};
 const SEEN_RECENTLY_SECONDS: i64 = 600;
 
 /// Valid contact address.
-#[derive(Debug, Clone, Copy)]
-pub(crate) struct ContactAddress<'a>(&'a str);
+#[derive(Debug, Clone)]
+pub(crate) struct ContactAddress(String);
 
-impl Deref for ContactAddress<'_> {
+impl Deref for ContactAddress {
     type Target = str;
 
     fn deref(&self) -> &Self::Target {
-        self.0
+        &self.0
     }
 }
 
-impl AsRef<str> for ContactAddress<'_> {
+impl AsRef<str> for ContactAddress {
     fn as_ref(&self) -> &str {
-        self.0
+        &self.0
     }
 }
 
-impl fmt::Display for ContactAddress<'_> {
+impl fmt::Display for ContactAddress {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         write!(f, "{}", self.0)
     }
 }
 
-impl<'a> ContactAddress<'a> {
+impl ContactAddress {
     /// Constructs a new contact address from string,
     /// normalizing and validating it.
-    pub fn new(s: &'a str) -> Result<Self> {
+    pub fn new(s: &str) -> Result<Self> {
         let addr = addr_normalize(s);
-        if !may_be_valid_addr(addr) {
+        if !may_be_valid_addr(&addr) {
             bail!("invalid address {:?}", s);
         }
-        Ok(Self(addr))
+        Ok(Self(addr.to_string()))
     }
 }
 
 /// Allow converting [`ContactAddress`] to an SQLite type.
-impl rusqlite::types::ToSql for ContactAddress<'_> {
+impl rusqlite::types::ToSql for ContactAddress {
     fn to_sql(&self) -> rusqlite::Result<rusqlite::types::ToSqlOutput> {
         let val = rusqlite::types::Value::Text(self.0.to_string());
         let out = rusqlite::types::ToSqlOutput::Owned(val);
@@ -149,6 +149,22 @@ impl ContactId {
             .await?;
         Ok(())
     }
+
+    /// Reset gossip timestamp in all chats with this contact.
+    pub(crate) async fn regossip_keys(&self, context: &Context) -> Result<()> {
+        context
+            .sql
+            .execute(
+                "UPDATE chats
+                 SET gossiped_timestamp=0
+                 WHERE EXISTS (SELECT 1 FROM chats_contacts
+                               WHERE chats_contacts.chat_id=chats.id
+                               AND chats_contacts.contact_id=?)",
+                (self,),
+            )
+            .await?;
+        Ok(())
+    }
 }
 
 impl fmt::Display for ContactId {
@@ -332,24 +348,6 @@ pub(crate) enum Modifier {
     Created,
 }
 
-/// Verification status of the contact.
-#[derive(Debug, PartialEq, Eq, Clone, Copy, FromPrimitive)]
-#[repr(u8)]
-pub enum VerifiedStatus {
-    /// Contact is not verified.
-    Unverified = 0,
-    /// SELF has verified the fingerprint of a contact. Currently unused.
-    Verified = 1,
-    /// SELF and contact have verified their fingerprints in both directions; in the UI typically checkmarks are shown.
-    BidirectVerified = 2,
-}
-
-impl Default for VerifiedStatus {
-    fn default() -> Self {
-        Self::Unverified
-    }
-}
-
 impl Contact {
     /// Loads a single contact object from the database.
     ///
@@ -484,7 +482,7 @@ impl Contact {
         let addr = ContactAddress::new(&addr)?;
 
         let (contact_id, sth_modified) =
-            Contact::add_or_lookup(context, &name, addr, Origin::ManuallyCreated)
+            Contact::add_or_lookup(context, &name, &addr, Origin::ManuallyCreated)
                 .await
                 .context("add_or_lookup")?;
         let blocked = Contact::is_blocked_load(context, contact_id).await?;
@@ -549,7 +547,7 @@ impl Contact {
 
         let addr_normalized = addr_normalize(addr);
 
-        if context.is_self_addr(addr_normalized).await? {
+        if context.is_self_addr(&addr_normalized).await? {
             return Ok(Some(ContactId::SELF));
         }
 
@@ -599,7 +597,7 @@ impl Contact {
     pub(crate) async fn add_or_lookup(
         context: &Context,
         name: &str,
-        addr: ContactAddress<'_>,
+        addr: &ContactAddress,
         mut origin: Origin,
     ) -> Result<(ContactId, Modifier)> {
         let mut sth_modified = Modifier::None;
@@ -607,7 +605,7 @@ impl Contact {
         ensure!(!addr.is_empty(), "Can not add_or_lookup empty address");
         ensure!(origin != Origin::Unknown, "Missing valid origin");
 
-        if context.is_self_addr(&addr).await? {
+        if context.is_self_addr(addr).await? {
             return Ok((ContactId::SELF, sth_modified));
         }
 
@@ -762,7 +760,7 @@ impl Contact {
                             } else {
                                 "".to_string()
                             },
-                            addr,
+                            &addr,
                             origin,
                             if update_authname {
                                 name.to_string()
@@ -807,7 +805,7 @@ impl Contact {
             let name = normalize_name(&name);
             match ContactAddress::new(&addr) {
                 Ok(addr) => {
-                    match Contact::add_or_lookup(context, &name, addr, Origin::AddressBook).await {
+                    match Contact::add_or_lookup(context, &name, &addr, Origin::AddressBook).await {
                         Ok((_, modified)) => {
                             if modified != Modifier::None {
                                 modify_cnt += 1
@@ -1039,11 +1037,9 @@ impl Contact {
             let loginparam = LoginParam::load_configured_params(context).await?;
             let peerstate = Peerstate::from_addr(context, &contact.addr).await?;
 
-            if let Some(peerstate) = peerstate.filter(|peerstate| {
-                peerstate
-                    .peek_key(PeerstateVerifiedStatus::Unverified)
-                    .is_some()
-            }) {
+            if let Some(peerstate) =
+                peerstate.filter(|peerstate| peerstate.peek_key(false).is_some())
+            {
                 let stock_message = match peerstate.prefer_encrypt {
                     EncryptPreference::Mutual => stock_str::e2e_preferred(context).await,
                     EncryptPreference::NoPreference => stock_str::e2e_available(context).await,
@@ -1058,11 +1054,11 @@ impl Contact {
                     .fingerprint()
                     .to_string();
                 let fingerprint_other_verified = peerstate
-                    .peek_key(PeerstateVerifiedStatus::BidirectVerified)
+                    .peek_key(true)
                     .map(|k| k.fingerprint().to_string())
                     .unwrap_or_default();
                 let fingerprint_other_unverified = peerstate
-                    .peek_key(PeerstateVerifiedStatus::Unverified)
+                    .peek_key(false)
                     .map(|k| k.fingerprint().to_string())
                     .unwrap_or_default();
                 if loginparam.addr < peerstate.addr {
@@ -1265,20 +1261,20 @@ impl Contact {
     /// otherwise use is_chat_protected().
     /// Use [Self::get_verifier_id] to display the verifier contact
     /// in the info section of the contact profile.
-    pub async fn is_verified(&self, context: &Context) -> Result<VerifiedStatus> {
+    pub async fn is_verified(&self, context: &Context) -> Result<bool> {
         // We're always sort of secured-verified as we could verify the key on this device any time with the key
         // on this device
         if self.id == ContactId::SELF {
-            return Ok(VerifiedStatus::BidirectVerified);
+            return Ok(true);
         }
 
         if let Some(peerstate) = Peerstate::from_addr(context, &self.addr).await? {
             if peerstate.is_using_verified_key() {
-                return Ok(VerifiedStatus::BidirectVerified);
+                return Ok(true);
             }
         }
 
-        Ok(VerifiedStatus::Unverified)
+        Ok(false)
     }
 
     /// Returns the `ContactId` that verified the contact.
@@ -1301,7 +1297,7 @@ impl Contact {
             return Ok(None);
         };
 
-        if verifier_addr == self.addr {
+        if addr_cmp(&verifier_addr, &self.addr) {
             // Contact is directly verified via QR code.
             return Ok(Some(ContactId::SELF));
         }
@@ -1333,7 +1329,7 @@ impl Contact {
             Ok(chat_id.is_protected(context).await? == ProtectionStatus::Protected)
         } else {
             // 1:1 chat does not exist.
-            Ok(self.is_verified(context).await? == VerifiedStatus::BidirectVerified)
+            Ok(self.is_verified(context).await?)
         }
     }
 
@@ -1389,12 +1385,13 @@ pub fn may_be_valid_addr(addr: &str) -> bool {
     res.is_ok()
 }
 
-/// Returns address with whitespace trimmed and `mailto:` prefix removed.
-pub fn addr_normalize(addr: &str) -> &str {
-    let norm = addr.trim();
+/// Returns address lowercased,
+/// with whitespace trimmed and `mailto:` prefix removed.
+pub fn addr_normalize(addr: &str) -> String {
+    let norm = addr.trim().to_lowercase();
 
     if norm.starts_with("mailto:") {
-        norm.get(7..).unwrap_or(norm)
+        norm.get(7..).unwrap_or(&norm).to_string()
     } else {
         norm
     }
@@ -1649,8 +1646,8 @@ fn cat_fingerprint(
 
 /// Compares two email addresses, normalizing them beforehand.
 pub fn addr_cmp(addr1: &str, addr2: &str) -> bool {
-    let norm1 = addr_normalize(addr1).to_lowercase();
-    let norm2 = addr_normalize(addr2).to_lowercase();
+    let norm1 = addr_normalize(addr1);
+    let norm2 = addr_normalize(addr2);
 
     norm1 == norm2
 }
@@ -1848,10 +1845,7 @@ mod tests {
     fn test_normalize_addr() {
         assert_eq!(addr_normalize("mailto:john@doe.com"), "john@doe.com");
         assert_eq!(addr_normalize("  hello@world.com   "), "hello@world.com");
-
-        // normalisation preserves case to allow user-defined spelling.
-        // however, case is ignored on addr_cmp()
-        assert_ne!(addr_normalize("John@Doe.com"), "john@doe.com");
+        assert_eq!(addr_normalize("John@Doe.com"), "john@doe.com");
     }
 
     #[test]
@@ -1877,7 +1871,7 @@ mod tests {
         let (id, _modified) = Contact::add_or_lookup(
             &context.ctx,
             "bob",
-            ContactAddress::new("user@example.org")?,
+            &ContactAddress::new("user@example.org")?,
             Origin::IncomingReplyTo,
         )
         .await?;
@@ -1905,7 +1899,7 @@ mod tests {
         let (contact_bob_id, modified) = Contact::add_or_lookup(
             &context.ctx,
             "someone",
-            ContactAddress::new("user@example.org")?,
+            &ContactAddress::new("user@example.org")?,
             Origin::ManuallyCreated,
         )
         .await?;
@@ -1970,7 +1964,7 @@ mod tests {
         let (contact_id, sth_modified) = Contact::add_or_lookup(
             &t,
             "bla foo",
-            ContactAddress::new("one@eins.org").unwrap(),
+            &ContactAddress::new("one@eins.org").unwrap(),
             Origin::IncomingUnknownTo,
         )
         .await
@@ -1989,7 +1983,7 @@ mod tests {
         let (contact_id_test, sth_modified) = Contact::add_or_lookup(
             &t,
             "Real one",
-            ContactAddress::new(" one@eins.org  ").unwrap(),
+            &ContactAddress::new(" one@eins.org  ").unwrap(),
             Origin::ManuallyCreated,
         )
         .await
@@ -2005,7 +1999,7 @@ mod tests {
         let (contact_id, sth_modified) = Contact::add_or_lookup(
             &t,
             "",
-            ContactAddress::new("three@drei.sam").unwrap(),
+            &ContactAddress::new("three@drei.sam").unwrap(),
             Origin::IncomingUnknownTo,
         )
         .await
@@ -2022,7 +2016,7 @@ mod tests {
         let (contact_id_test, sth_modified) = Contact::add_or_lookup(
             &t,
             "m. serious",
-            ContactAddress::new("three@drei.sam").unwrap(),
+            &ContactAddress::new("three@drei.sam").unwrap(),
             Origin::IncomingUnknownFrom,
         )
         .await
@@ -2037,7 +2031,7 @@ mod tests {
         let (contact_id_test, sth_modified) = Contact::add_or_lookup(
             &t,
             "schnucki",
-            ContactAddress::new("three@drei.sam").unwrap(),
+            &ContactAddress::new("three@drei.sam").unwrap(),
             Origin::ManuallyCreated,
         )
         .await
@@ -2053,7 +2047,7 @@ mod tests {
         let (contact_id, sth_modified) = Contact::add_or_lookup(
             &t,
             "",
-            ContactAddress::new("alice@w.de").unwrap(),
+            &ContactAddress::new("alice@w.de").unwrap(),
             Origin::IncomingUnknownTo,
         )
         .await
@@ -2195,7 +2189,7 @@ mod tests {
         let (contact_id, _) = Contact::add_or_lookup(
             &alice,
             "Bob",
-            ContactAddress::new("bob@example.net")?,
+            &ContactAddress::new("bob@example.net")?,
             Origin::ManuallyCreated,
         )
         .await?;
@@ -2274,7 +2268,7 @@ mod tests {
         let (contact_id, sth_modified) = Contact::add_or_lookup(
             &t,
             "bob1",
-            ContactAddress::new("bob@example.org").unwrap(),
+            &ContactAddress::new("bob@example.org").unwrap(),
             Origin::IncomingUnknownFrom,
         )
         .await
@@ -2290,7 +2284,7 @@ mod tests {
         let (contact_id, sth_modified) = Contact::add_or_lookup(
             &t,
             "bob2",
-            ContactAddress::new("bob@example.org").unwrap(),
+            &ContactAddress::new("bob@example.org").unwrap(),
             Origin::IncomingUnknownFrom,
         )
         .await
@@ -2316,7 +2310,7 @@ mod tests {
         let (contact_id, sth_modified) = Contact::add_or_lookup(
             &t,
             "bob4",
-            ContactAddress::new("bob@example.org").unwrap(),
+            &ContactAddress::new("bob@example.org").unwrap(),
             Origin::IncomingUnknownFrom,
         )
         .await
@@ -2345,7 +2339,7 @@ mod tests {
         let (contact_id_same, sth_modified) = Contact::add_or_lookup(
             &t,
             "claire1",
-            ContactAddress::new("claire@example.org").unwrap(),
+            &ContactAddress::new("claire@example.org").unwrap(),
             Origin::IncomingUnknownFrom,
         )
         .await
@@ -2361,7 +2355,7 @@ mod tests {
         let (contact_id_same, sth_modified) = Contact::add_or_lookup(
             &t,
             "claire2",
-            ContactAddress::new("claire@example.org").unwrap(),
+            &ContactAddress::new("claire@example.org").unwrap(),
             Origin::IncomingUnknownFrom,
         )
         .await
@@ -2386,7 +2380,7 @@ mod tests {
         let (contact_id, sth_modified) = Contact::add_or_lookup(
             &t,
             "Bob",
-            ContactAddress::new("bob@example.org")?,
+            &ContactAddress::new("bob@example.org")?,
             Origin::IncomingUnknownFrom,
         )
         .await?;
@@ -2398,7 +2392,7 @@ mod tests {
         let (contact_id_same, sth_modified) = Contact::add_or_lookup(
             &t,
             "Not Bob",
-            ContactAddress::new("bob@example.org")?,
+            &ContactAddress::new("bob@example.org")?,
             Origin::IncomingUnknownTo,
         )
         .await?;
@@ -2411,7 +2405,7 @@ mod tests {
         let (contact_id_same, sth_modified) = Contact::add_or_lookup(
             &t,
             "Bob",
-            ContactAddress::new("bob@example.org")?,
+            &ContactAddress::new("bob@example.org")?,
             Origin::IncomingUnknownFrom,
         )
         .await?;
@@ -2440,7 +2434,7 @@ mod tests {
         Contact::add_or_lookup(
             &t,
             "dave2",
-            ContactAddress::new("dave@example.org").unwrap(),
+            &ContactAddress::new("dave@example.org").unwrap(),
             Origin::IncomingUnknownFrom,
         )
         .await
@@ -2561,7 +2555,7 @@ mod tests {
         let (contact_bob_id, _modified) = Contact::add_or_lookup(
             &alice,
             "Bob",
-            ContactAddress::new("bob@example.net")?,
+            &ContactAddress::new("bob@example.net")?,
             Origin::ManuallyCreated,
         )
         .await?;
@@ -2724,7 +2718,7 @@ CCCB 5AA9 F6E1 141C 9431
         let (contact_id, _) = Contact::add_or_lookup(
             &alice,
             "Bob",
-            ContactAddress::new("bob@example.net")?,
+            &ContactAddress::new("bob@example.net")?,
             Origin::ManuallyCreated,
         )
         .await?;

src/context.rs

@@ -413,7 +413,8 @@ impl Context {
                 .is_some()
             {
                 let mut lock = self.ratelimit.write().await;
-                *lock = Ratelimit::new(Duration::new(40, 0), 6.0);
+                // Allow at least 1 message every second + a burst of 3.
+                *lock = Ratelimit::new(Duration::new(3, 0), 3.0);
             }
         }
         self.scheduler.start(self.clone()).await;
@@ -422,6 +423,9 @@ impl Context {
     /// Stops the IO scheduler.
     pub async fn stop_io(&self) {
         self.scheduler.stop(self).await;
+        if let Err(err) = self.sql.checkpoint(self).await {
+            error!(self, "Failed to checkpoint the database: {err:#}.");
+        }
     }
 
     /// Restarts the IO scheduler if it was running before

src/debug_logging.rs

@@ -54,7 +54,7 @@ pub async fn debug_logging_loop(context: &Context, events: Receiver<DebugEventLo
         match context
             .write_status_update_inner(
                 &msg_id,
-                StatusUpdateItem {
+                &StatusUpdateItem {
                     payload: json!({
                         "event": event,
                         "time": time,
@@ -62,6 +62,7 @@ pub async fn debug_logging_loop(context: &Context, events: Receiver<DebugEventLo
                     info: None,
                     summary: None,
                     document: None,
+                    uid: None,
                 },
             )
             .await
@@ -70,10 +71,17 @@ pub async fn debug_logging_loop(context: &Context, events: Receiver<DebugEventLo
                 eprintln!("Can't log event to webxdc status update: {err:#}");
             }
             Ok(serial) => {
-                context.emit_event(EventType::WebxdcStatusUpdate {
-                    msg_id,
-                    status_update_serial: serial,
-                });
+                if let Some(serial) = serial {
+                    if !matches!(event, EventType::WebxdcStatusUpdate { .. }) {
+                        context.emit_event(EventType::WebxdcStatusUpdate {
+                            msg_id,
+                            status_update_serial: serial,
+                        });
+                    }
+                } else {
+                    // This should not happen as the update has no `uid`.
+                    error!(context, "Debug logging update is not created.");
+                };
             }
         }
     }

src/e2ee.rs

@@ -7,7 +7,7 @@ use crate::aheader::{Aheader, EncryptPreference};
 use crate::config::Config;
 use crate::context::Context;
 use crate::key::{load_self_public_key, load_self_secret_key, SignedPublicKey};
-use crate::peerstate::{Peerstate, PeerstateVerifiedStatus};
+use crate::peerstate::Peerstate;
 use crate::pgp;
 
 #[derive(Debug)]
@@ -94,7 +94,7 @@ impl EncryptHelper {
     pub async fn encrypt(
         self,
         context: &Context,
-        min_verified: PeerstateVerifiedStatus,
+        verified: bool,
         mail_to_encrypt: lettre_email::PartBuilder,
         peerstates: Vec<(Option<Peerstate>, &str)>,
     ) -> Result<String> {
@@ -107,7 +107,7 @@ impl EncryptHelper {
             .filter_map(|(state, addr)| state.clone().map(|s| (s, addr)))
         {
             let key = peerstate
-                .take_key(min_verified)
+                .take_key(verified)
                 .with_context(|| format!("proper enc-key for {addr} missing, cannot encrypt"))?;
             keyring.push(key);
             verifier_addresses.push(addr);
@@ -118,7 +118,7 @@ impl EncryptHelper {
 
         // Encrypt to secondary verified keys
         // if we also encrypt to the introducer ("verifier") of the key.
-        if min_verified == PeerstateVerifiedStatus::BidirectVerified {
+        if verified {
             for (peerstate, _addr) in peerstates {
                 if let Some(peerstate) = peerstate {
                     if let (Some(key), Some(verifier)) = (

src/imap.rs

@@ -9,6 +9,7 @@ use std::{
     collections::{BTreeMap, BTreeSet, HashMap},
     iter::Peekable,
     mem::take,
+    time::Duration,
 };
 
 use anyhow::{bail, format_err, Context as _, Result};
@@ -16,6 +17,8 @@ use async_channel::Receiver;
 use async_imap::types::{Fetch, Flag, Name, NameAttribute, UnsolicitedResponse};
 use futures::{StreamExt, TryStreamExt};
 use num_traits::FromPrimitive;
+use ratelimit::Ratelimit;
+use tokio::sync::RwLock;
 
 use crate::chat::{self, ChatId, ChatIdBlocked};
 use crate::config::Config;
@@ -38,7 +41,7 @@ use crate::scheduler::connectivity::ConnectivityStore;
 use crate::socks::Socks5Config;
 use crate::sql;
 use crate::stock_str;
-use crate::tools::create_id;
+use crate::tools::{create_id, duration_to_str};
 
 pub(crate) mod capabilities;
 mod client;
@@ -91,6 +94,7 @@ pub struct Imap {
     login_failed_once: bool,
 
     pub(crate) connectivity: ConnectivityStore,
+    ratelimit: RwLock<Ratelimit>,
 }
 
 #[derive(Debug)]
@@ -252,6 +256,8 @@ impl Imap {
             session: None,
             login_failed_once: false,
             connectivity: Default::default(),
+            // 1 connection per minute + a burst of 2.
+            ratelimit: RwLock::new(Ratelimit::new(Duration::new(120, 0), 2.0)),
         };
 
         Ok(imap)
@@ -300,10 +306,20 @@ impl Imap {
         }
 
         self.connectivity.set_connecting(context).await;
+        let ratelimit_duration = self.ratelimit.read().await.until_can_send();
+        if !ratelimit_duration.is_zero() {
+            warn!(
+                context,
+                "IMAP got rate limited, waiting for {} until can connect",
+                duration_to_str(ratelimit_duration),
+            );
+            tokio::time::sleep(ratelimit_duration).await;
+        }
 
         let oauth2 = self.config.lp.oauth2;
 
         info!(context, "Connecting to IMAP server");
+        self.ratelimit.write().await.send();
         let connection_res: Result<Client> = if self.config.lp.security == Socket::Starttls
             || self.config.lp.security == Socket::Plain
         {
@@ -572,9 +588,6 @@ impl Imap {
     /// When selecting a folder for the first time, sets the uid_next to the current
     /// mailbox.uid_next so that no old emails are fetched.
     ///
-    /// Makes sure that UIDNEXT is known for `selected_mailbox`
-    /// and errors out if UIDNEXT cannot be determined.
-    ///
     /// Returns Result<new_emails> (i.e. whether new emails arrived),
     /// if in doubt, returns new_emails=true so emails are fetched.
     pub(crate) async fn select_with_uidvalidity(
@@ -592,11 +605,18 @@ impl Imap {
             .as_mut()
             .with_context(|| format!("No mailbox selected, folder: {folder}"))?;
 
+        let old_uid_validity = get_uidvalidity(context, folder)
+            .await
+            .with_context(|| format!("failed to get old UID validity for folder {folder}"))?;
+        let old_uid_next = get_uid_next(context, folder)
+            .await
+            .with_context(|| format!("failed to get old UID NEXT for folder {folder}"))?;
+
         let new_uid_validity = mailbox
             .uid_validity
             .with_context(|| format!("No UIDVALIDITY for folder {folder}"))?;
         let new_uid_next = if let Some(uid_next) = mailbox.uid_next {
-            uid_next
+            Some(uid_next)
         } else {
             warn!(
                 context,
@@ -619,20 +639,17 @@ impl Imap {
                 .inner
                 .status(folder, "(UIDNEXT)")
                 .await
-                .context("STATUS (UIDNEXT) error for {folder:?}")?;
+                .with_context(|| format!("STATUS (UIDNEXT) error for {folder:?}"))?;
 
-            status
-                .uid_next
-                .context("STATUS {folder} (UIDNEXT) did not return UIDNEXT")?
+            if status.uid_next.is_none() {
+                // This happens with mail.163.com as of 2023-11-26.
+                // It does not return UIDNEXT on SELECT and returns invalid
+                // `* STATUS "INBOX" ()` response on explicit request for UIDNEXT.
+                warn!(context, "STATUS {folder} (UIDNEXT) did not return UIDNEXT.");
+            }
+            status.uid_next
         };
-        mailbox.uid_next = Some(new_uid_next);
-
-        let old_uid_validity = get_uidvalidity(context, folder)
-            .await
-            .with_context(|| format!("failed to get old UID validity for folder {folder}"))?;
-        let old_uid_next = get_uid_next(context, folder)
-            .await
-            .with_context(|| format!("failed to get old UID NEXT for folder {folder}"))?;
+        mailbox.uid_next = new_uid_next;
 
         if new_uid_validity == old_uid_validity {
             let new_emails = if newly_selected == NewlySelected::No {
@@ -641,7 +658,7 @@ impl Imap {
                 // the caller tries to fetch new messages (we could of course run a SELECT command now, but trying to fetch
                 // new messages is only one command, just as a SELECT command)
                 true
-            } else {
+            } else if let Some(new_uid_next) = new_uid_next {
                 if new_uid_next < old_uid_next {
                     warn!(
                         context,
@@ -651,7 +668,11 @@ impl Imap {
                     context.schedule_resync().await?;
                 }
                 new_uid_next != old_uid_next // If UIDNEXT changed, there are new emails
+            } else {
+                // We have no UIDNEXT and if in doubt, return true.
+                true
             };
+
             return Ok(new_emails);
         }
 
@@ -660,6 +681,7 @@ impl Imap {
 
         // ==============  uid_validity has changed or is being set the first time.  ==============
 
+        let new_uid_next = new_uid_next.unwrap_or_default();
         set_uid_next(context, folder, new_uid_next).await?;
         set_uidvalidity(context, folder, new_uid_validity).await?;
 
@@ -876,11 +898,7 @@ impl Imap {
             .as_ref()
             .with_context(|| format!("Expected {folder:?} to be selected"))?
             .uid_next
-            .with_context(|| {
-                format!(
-                    "Expected UIDNEXT to be determined for {folder:?} by select_with_uidvalidity"
-                )
-            })?;
+            .unwrap_or_default();
         let new_uid_next = max(
             max(largest_uid_fetched, largest_uid_skipped.unwrap_or(0)) + 1,
             mailbox_uid_next,
@@ -1451,13 +1469,8 @@ impl Imap {
                             break;
                         };
 
-                    let next_fetch_response = match next_fetch_response {
-                        Ok(next_fetch_response) => next_fetch_response,
-                        Err(err) => {
-                            warn!(context, "Failed to process IMAP FETCH result: {}.", err);
-                            continue;
-                        }
-                    };
+                    let next_fetch_response =
+                        next_fetch_response.context("Failed to process IMAP FETCH result")?;
 
                     if let Some(next_uid) = next_fetch_response.uid {
                         if next_uid == request_uid {
@@ -2519,7 +2532,7 @@ async fn add_all_recipients_as_contacts(
         let (_, modified) = Contact::add_or_lookup(
             context,
             &display_name_normalized,
-            recipient_addr,
+            &recipient_addr,
             Origin::OutgoingTo,
         )
         .await?;

src/imap/client.rs

@@ -17,8 +17,8 @@ use crate::net::tls::wrap_tls;
 use crate::socks::Socks5Config;
 use fast_socks5::client::Socks5Stream;
 
-/// IMAP write and read timeout.
-pub(crate) const IMAP_TIMEOUT: Duration = Duration::from_secs(30);
+/// IMAP connection, write and read timeout.
+pub(crate) const IMAP_TIMEOUT: Duration = Duration::from_secs(60);
 
 #[derive(Debug)]
 pub(crate) struct Client {

src/imap/idle.rs

@@ -9,7 +9,7 @@ use super::session::Session;
 use super::Imap;
 use crate::config::Config;
 use crate::context::Context;
-use crate::imap::{client::IMAP_TIMEOUT, get_uid_next, FolderMeaning};
+use crate::imap::{client::IMAP_TIMEOUT, FolderMeaning};
 use crate::log::LogExt;
 
 const IDLE_TIMEOUT: Duration = Duration::from_secs(23 * 60);
@@ -29,29 +29,6 @@ impl Session {
             return Ok(self);
         }
 
-        // Despite checking for unsolicited EXISTS above,
-        // we may have missed EXISTS if the message was
-        // received when the folder was not selected.
-        let status = self
-            .status(folder, "(UIDNEXT)")
-            .await
-            .context("STATUS (UIDNEXT) error for {folder:?}")?;
-        if let Some(uid_next) = status.uid_next {
-            let expected_uid_next = get_uid_next(context, folder)
-                .await
-                .with_context(|| format!("failed to get old UID NEXT for folder {folder}"))?;
-            if uid_next > expected_uid_next {
-                info!(
-                    context,
-                    "Skipping IDLE on {folder:?} because UIDNEXT {uid_next}>{expected_uid_next} indicates there are new messages."
-                );
-                return Ok(self);
-            }
-        } else {
-            warn!(context, "STATUS {folder} (UIDNEXT) did not return UIDNEXT");
-            // Go to IDLE anyway if STATUS is broken.
-        }
-
         if let Ok(()) = idle_interrupt_receiver.try_recv() {
             info!(context, "skip idle, got interrupt");
             return Ok(self);

src/imex.rs

@@ -808,6 +808,7 @@ mod tests {
     use tokio::task;
 
     use super::*;
+    use crate::key;
     use crate::pgp::{split_armored_data, HEADER_AUTOCRYPT, HEADER_SETUPCODE};
     use crate::stock_str::StockMessage;
     use crate::test_utils::{alice_keypair, TestContext};
@@ -920,6 +921,37 @@ mod tests {
         }
     }
 
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_import_second_key() -> Result<()> {
+        let alice = &TestContext::new_alice().await;
+        let chat = alice.create_chat(alice).await;
+        let sent = alice.send_text(chat.id, "Encrypted with old key").await;
+        let export_dir = tempfile::tempdir().unwrap();
+
+        let alice = &TestContext::new().await;
+        alice.configure_addr("alice@example.org").await;
+        imex(alice, ImexMode::ExportSelfKeys, export_dir.path(), None).await?;
+
+        let alice = &TestContext::new_alice().await;
+        let old_key = key::load_self_secret_key(alice).await?;
+
+        imex(alice, ImexMode::ImportSelfKeys, export_dir.path(), None).await?;
+
+        let new_key = key::load_self_secret_key(alice).await?;
+        assert_ne!(new_key, old_key);
+        assert_eq!(
+            key::load_self_secret_keyring(alice).await?,
+            vec![new_key, old_key]
+        );
+
+        let msg = alice.recv_msg(&sent).await;
+        assert!(msg.get_showpadlock());
+        assert_eq!(msg.chat_id, alice.get_self_chat().await.id);
+        assert_eq!(msg.get_text(), "Encrypted with old key");
+
+        Ok(())
+    }
+
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_export_and_import_backup() -> Result<()> {
         for set_verified_oneonone_chats in [true, false] {

src/key.rs

@@ -16,6 +16,7 @@ use tokio::runtime::Handle;
 use crate::config::Config;
 use crate::constants::KeyGenType;
 use crate::context::Context;
+use crate::log::LogExt;
 use crate::pgp::KeyPair;
 use crate::tools::{time, EmailAddress};
 
@@ -125,6 +126,25 @@ pub(crate) async fn load_self_secret_key(context: &Context) -> Result<SignedSecr
     }
 }
 
+pub(crate) async fn load_self_secret_keyring(context: &Context) -> Result<Vec<SignedSecretKey>> {
+    let keys = context
+        .sql
+        .query_map(
+            r#"SELECT private_key
+               FROM keypairs
+               WHERE addr=(SELECT value FROM config WHERE keyname="configured_addr")
+               ORDER BY is_default DESC"#,
+            (),
+            |row| row.get::<_, Vec<u8>>(0),
+            |keys| keys.collect::<Result<Vec<_>, _>>().map_err(Into::into),
+        )
+        .await?
+        .into_iter()
+        .filter_map(|bytes| SignedSecretKey::from_slice(&bytes).log_err(context).ok())
+        .collect();
+    Ok(keys)
+}
+
 impl DcKey for SignedPublicKey {
     fn to_asc(&self, header: Option<(&str, &str)>) -> String {
         // Not using .to_armored_string() to make clear *why* it is

src/mimefactory.rs

@@ -22,7 +22,7 @@ use crate::location;
 use crate::message::{self, Message, MsgId, Viewtype};
 use crate::mimeparser::SystemMessage;
 use crate::param::Param;
-use crate::peerstate::{Peerstate, PeerstateVerifiedStatus};
+use crate::peerstate::Peerstate;
 use crate::simplify::escape_message_footer_marks;
 use crate::stock_str;
 use crate::tools::IsNoneOrEmpty;
@@ -312,7 +312,7 @@ impl<'a> MimeFactory<'a> {
         }
     }
 
-    fn min_verified(&self) -> PeerstateVerifiedStatus {
+    fn verified(&self) -> bool {
         match &self.loaded {
             Loaded::Message { chat } => {
                 if chat.is_protected() {
@@ -321,15 +321,15 @@ impl<'a> MimeFactory<'a> {
                         // In order to do this, it is necessary that they can be sent
                         // to a key that is not yet verified.
                         // This has to work independently of whether the chat is protected right now.
-                        PeerstateVerifiedStatus::Unverified
+                        false
                     } else {
-                        PeerstateVerifiedStatus::BidirectVerified
+                        true
                     }
                 } else {
-                    PeerstateVerifiedStatus::Unverified
+                    false
                 }
             }
-            Loaded::Mdn { .. } => PeerstateVerifiedStatus::Unverified,
+            Loaded::Mdn { .. } => false,
         }
     }
 
@@ -627,7 +627,7 @@ impl<'a> MimeFactory<'a> {
             ));
         }
 
-        let min_verified = self.min_verified();
+        let verified = self.verified();
         let grpimage = self.grpimage();
         let force_plaintext = self.should_force_plaintext();
         let skip_autocrypt = self.should_skip_autocrypt();
@@ -723,7 +723,7 @@ impl<'a> MimeFactory<'a> {
                 && self.should_do_gossip(context).await?
             {
                 for peerstate in peerstates.iter().filter_map(|(state, _)| state.as_ref()) {
-                    if let Some(header) = peerstate.render_gossip_header(min_verified) {
+                    if let Some(header) = peerstate.render_gossip_header(verified) {
                         message = message.header(Header::new("Autocrypt-Gossip".into(), header));
                         is_gossiped = true;
                     }
@@ -756,7 +756,7 @@ impl<'a> MimeFactory<'a> {
             }
 
             let encrypted = encrypt_helper
-                .encrypt(context, min_verified, message, peerstates)
+                .encrypt(context, verified, message, peerstates)
                 .await?;
 
             outer_message
@@ -924,9 +924,7 @@ impl<'a> MimeFactory<'a> {
         let mut meta_part = None;
 
         let send_verified_headers = match chat.typ {
-            // In single chats, the protection status isn't necessarily the same for both sides,
-            // so we don't send the Chat-Verified header:
-            Chattype::Single => false,
+            Chattype::Single => true,
             Chattype::Group => true,
             // Mailinglists and broadcast lists can actually never be verified:
             Chattype::Mailinglist => false,
@@ -1919,7 +1917,7 @@ mod tests {
         let contact_id = Contact::add_or_lookup(
             &t,
             "Dave",
-            ContactAddress::new("dave@example.com").unwrap(),
+            &ContactAddress::new("dave@example.com").unwrap(),
             Origin::ManuallyCreated,
         )
         .await

src/mimeparser.rs

@@ -27,7 +27,7 @@ use crate::decrypt::{
 use crate::dehtml::dehtml;
 use crate::events::EventType;
 use crate::headerdef::{HeaderDef, HeaderDefMap};
-use crate::key::{load_self_secret_key, DcKey, Fingerprint, SignedPublicKey};
+use crate::key::{load_self_secret_keyring, DcKey, Fingerprint, SignedPublicKey};
 use crate::message::{
     self, set_msg_failed, update_msg_state, Message, MessageState, MsgId, Viewtype,
 };
@@ -259,14 +259,12 @@ impl MimeMessage {
             }
         }
 
-        // remove headers that are allowed _only_ in the encrypted part
-        headers.remove("secure-join-fingerprint");
-        headers.remove("chat-verified");
+        // Remove headers that are allowed _only_ in the encrypted+signed part. It's ok to leave
+        // them in signed-only emails, but has no value currently.
+        Self::remove_secured_headers(&mut headers);
 
         let from = from.context("No from in message")?;
-        let private_keyring = vec![load_self_secret_key(context)
-            .await
-            .context("Failed to get own key")?];
+        let private_keyring = load_self_secret_keyring(context).await?;
 
         let mut decryption_info =
             prepare_decryption(context, &mail, &from.addr, message_time).await?;
@@ -307,10 +305,11 @@ impl MimeMessage {
             content
         });
         if let (Ok(mail), true) = (mail, encrypted) {
-            // Handle any gossip headers if the mail was encrypted.  See section
-            // "3.6 Key Gossip" of <https://autocrypt.org/autocrypt-spec-1.1.0.pdf>
-            // but only if the mail was correctly signed:
             if !signatures.is_empty() {
+                // Handle any gossip headers if the mail was encrypted. See section
+                // "3.6 Key Gossip" of <https://autocrypt.org/autocrypt-spec-1.1.0.pdf>
+                // but only if the mail was correctly signed. Probably it's ok to not require
+                // encryption here, but let's follow the standard.
                 let gossip_headers = mail.headers.get_all_values("Autocrypt-Gossip");
                 gossiped_addr = update_gossip_peerstates(
                     context,
@@ -320,6 +319,9 @@ impl MimeMessage {
                     gossip_headers,
                 )
                 .await?;
+                // Remove unsigned subject from messages displayed with padlock.
+                // See <https://github.com/deltachat/deltachat-core-rust/issues/1790>.
+                headers.remove("subject");
             }
 
             // let known protected headers from the decrypted
@@ -327,24 +329,20 @@ impl MimeMessage {
 
             // Signature was checked for original From, so we
             // do not allow overriding it.
-            let mut signed_from = None;
-
-            // We do not want to allow unencrypted subject in encrypted emails because the
-            // user might falsely think that the subject is safe.
-            // See <https://github.com/deltachat/deltachat-core-rust/issues/1790>.
-            headers.remove("subject");
+            let mut inner_from = None;
 
             MimeMessage::merge_headers(
                 context,
                 &mut headers,
                 &mut recipients,
-                &mut signed_from,
+                &mut inner_from,
                 &mut list_post,
                 &mut chat_disposition_notification_to,
                 &mail.headers,
             );
-            if let Some(signed_from) = signed_from {
-                if addr_cmp(&signed_from.addr, &from.addr) {
+
+            if let (Some(inner_from), true) = (inner_from, !signatures.is_empty()) {
+                if addr_cmp(&inner_from.addr, &from.addr) {
                     from_is_signed = true;
                 } else {
                     // There is a From: header in the encrypted &
@@ -362,6 +360,8 @@ impl MimeMessage {
             }
         }
         if signatures.is_empty() {
+            Self::remove_secured_headers(&mut headers);
+
             // If it is not a read receipt, degrade encryption.
             if let (Some(peerstate), Ok(mail)) = (&mut decryption_info.peerstate, mail) {
                 if message_time > peerstate.last_seen_autocrypt
@@ -1379,6 +1379,11 @@ impl MimeMessage {
             .and_then(|msgid| parse_message_id(msgid).ok())
     }
 
+    fn remove_secured_headers(headers: &mut HashMap<String, String>) {
+        headers.remove("secure-join-fingerprint");
+        headers.remove("chat-verified");
+    }
+
     fn merge_headers(
         context: &Context,
         headers: &mut HashMap<String, String>,

src/peerstate.rs

@@ -26,17 +26,6 @@ pub enum PeerstateKeyType {
     PublicKey,
 }
 
-/// Verification status of the contact peerstate.
-#[derive(Debug, PartialEq, Eq, Clone, Copy, FromPrimitive)]
-#[repr(u8)]
-pub enum PeerstateVerifiedStatus {
-    /// Peerstate is not verified.
-    Unverified = 0,
-    //Verified = 1, // not used
-    /// Peerstate is verified and we assume that the contact has verified our peerstate.
-    BidirectVerified = 2,
-}
-
 /// Peerstate represents the state of an Autocrypt peer.
 #[derive(Debug, PartialEq, Eq, Clone)]
 pub struct Peerstate {
@@ -373,8 +362,8 @@ impl Peerstate {
     }
 
     /// Returns the contents of the `Autocrypt-Gossip` header for outgoing messages.
-    pub fn render_gossip_header(&self, min_verified: PeerstateVerifiedStatus) -> Option<String> {
-        if let Some(key) = self.peek_key(min_verified) {
+    pub fn render_gossip_header(&self, verified: bool) -> Option<String> {
+        if let Some(key) = self.peek_key(verified) {
             let header = Aheader::new(
                 self.addr.clone(),
                 key.clone(), // TODO: avoid cloning
@@ -397,43 +386,41 @@ impl Peerstate {
     /// Converts the peerstate into the contact public key.
     ///
     /// Similar to [`Self::peek_key`], but consumes the peerstate and returns owned key.
-    pub fn take_key(mut self, min_verified: PeerstateVerifiedStatus) -> Option<SignedPublicKey> {
-        match min_verified {
-            PeerstateVerifiedStatus::BidirectVerified => self.verified_key.take(),
-            PeerstateVerifiedStatus::Unverified => {
-                self.public_key.take().or_else(|| self.gossip_key.take())
-            }
+    pub fn take_key(mut self, verified: bool) -> Option<SignedPublicKey> {
+        if verified {
+            self.verified_key.take()
+        } else {
+            self.public_key.take().or_else(|| self.gossip_key.take())
         }
     }
 
     /// Returns a reference to the contact public key.
     ///
-    /// `min_verified` determines the minimum required verification status of the key.
+    /// `verified` determines the required verification status of the key.
     /// If verified key is requested, returns the verified key,
     /// otherwise returns the Autocrypt key.
     ///
     /// Returned key is suitable for sending in `Autocrypt-Gossip` header.
     ///
     /// Returns `None` if there is no suitable public key.
-    pub fn peek_key(&self, min_verified: PeerstateVerifiedStatus) -> Option<&SignedPublicKey> {
-        match min_verified {
-            PeerstateVerifiedStatus::BidirectVerified => self.verified_key.as_ref(),
-            PeerstateVerifiedStatus::Unverified => {
-                self.public_key.as_ref().or(self.gossip_key.as_ref())
-            }
+    pub fn peek_key(&self, verified: bool) -> Option<&SignedPublicKey> {
+        if verified {
+            self.verified_key.as_ref()
+        } else {
+            self.public_key.as_ref().or(self.gossip_key.as_ref())
         }
     }
 
     /// Returns a reference to the contact's public key fingerprint.
     ///
     /// Similar to [`Self::peek_key`], but returns the fingerprint instead of the key.
-    fn peek_key_fingerprint(&self, min_verified: PeerstateVerifiedStatus) -> Option<&Fingerprint> {
-        match min_verified {
-            PeerstateVerifiedStatus::BidirectVerified => self.verified_key_fingerprint.as_ref(),
-            PeerstateVerifiedStatus::Unverified => self
-                .public_key_fingerprint
+    fn peek_key_fingerprint(&self, verified: bool) -> Option<&Fingerprint> {
+        if verified {
+            self.verified_key_fingerprint.as_ref()
+        } else {
+            self.public_key_fingerprint
                 .as_ref()
-                .or(self.gossip_key_fingerprint.as_ref()),
+                .or(self.gossip_key_fingerprint.as_ref())
         }
     }
 
@@ -443,10 +430,9 @@ impl Peerstate {
     /// Note that verified groups always use the verified key no matter if the
     /// opportunistic key matches or not.
     pub(crate) fn is_using_verified_key(&self) -> bool {
-        let verified = self.peek_key_fingerprint(PeerstateVerifiedStatus::BidirectVerified);
+        let verified = self.peek_key_fingerprint(true);
 
-        verified.is_some()
-            && verified == self.peek_key_fingerprint(PeerstateVerifiedStatus::Unverified)
+        verified.is_some() && verified == self.peek_key_fingerprint(false)
     }
 
     /// Set this peerstate to verified
@@ -648,7 +634,7 @@ impl Peerstate {
                             let (new_contact_id, _) = Contact::add_or_lookup(
                                 context,
                                 "",
-                                new_addr,
+                                &new_addr,
                                 Origin::IncomingUnknownFrom,
                             )
                             .await?;
@@ -719,9 +705,7 @@ pub(crate) async fn maybe_do_aeap_transition(
                 // addresses with an MUA.
                 && mime_parser.has_chat_version()
                 // Check if the message is signed correctly.
-                // If it's not signed correctly, the whole autocrypt header will be mostly
-                // ignored anyway and the message shown as not encrypted, so we don't
-                // have to handle this case.
+                // Although checking `from_is_signed` below is sufficient, let's play it safe.
                 && !mime_parser.signatures.is_empty()
                 // Check if the From: address was also in the signed part of the email.
                 // Without this check, an attacker could replay a message from Alice 

src/qr.rs

@@ -365,9 +365,10 @@ async fn decode_openpgp(context: &Context, qr: &str) -> Result<Qr> {
 
     if let (Some(addr), Some(invitenumber), Some(authcode)) = (&addr, invitenumber, authcode) {
         let addr = ContactAddress::new(addr)?;
-        let (contact_id, _) = Contact::add_or_lookup(context, &name, addr, Origin::UnhandledQrScan)
-            .await
-            .with_context(|| format!("failed to add or lookup contact for address {addr:?}"))?;
+        let (contact_id, _) =
+            Contact::add_or_lookup(context, &name, &addr, Origin::UnhandledQrScan)
+                .await
+                .with_context(|| format!("failed to add or lookup contact for address {addr:?}"))?;
 
         if let (Some(grpid), Some(grpname)) = (grpid, grpname) {
             if context
@@ -432,7 +433,7 @@ async fn decode_openpgp(context: &Context, qr: &str) -> Result<Qr> {
         if let Some(peerstate) = peerstate {
             let peerstate_addr = ContactAddress::new(&peerstate.addr)?;
             let (contact_id, _) =
-                Contact::add_or_lookup(context, &name, peerstate_addr, Origin::UnhandledQrScan)
+                Contact::add_or_lookup(context, &name, &peerstate_addr, Origin::UnhandledQrScan)
                     .await
                     .context("add_or_lookup")?;
             ChatIdBlocked::get_for_contact(context, contact_id, Blocked::Request)
@@ -777,7 +778,7 @@ impl Qr {
     ) -> Result<Self> {
         let addr = ContactAddress::new(addr)?;
         let (contact_id, _) =
-            Contact::add_or_lookup(context, name, addr, Origin::UnhandledQrScan).await?;
+            Contact::add_or_lookup(context, name, &addr, Origin::UnhandledQrScan).await?;
         Ok(Qr::Addr { contact_id, draft })
     }
 }
@@ -788,7 +789,7 @@ fn normalize_address(addr: &str) -> Result<String> {
     let new_addr = percent_decode_str(addr).decode_utf8()?;
     let new_addr = addr_normalize(&new_addr);
 
-    ensure!(may_be_valid_addr(new_addr), "Bad e-mail address");
+    ensure!(may_be_valid_addr(&new_addr), "Bad e-mail address");
 
     Ok(new_addr.to_string())
 }

src/reaction.rs

@@ -392,7 +392,7 @@ Can we chat at 1pm pacific, today?"
         let bob_id = Contact::add_or_lookup(
             &alice,
             "",
-            ContactAddress::new("bob@example.net")?,
+            &ContactAddress::new("bob@example.net")?,
             Origin::ManuallyCreated,
         )
         .await?

src/receive_imf.rs

@@ -14,7 +14,7 @@ use crate::chat::{self, Chat, ChatId, ChatIdBlocked, ProtectionStatus};
 use crate::config::Config;
 use crate::constants::{Blocked, Chattype, ShowEmails, DC_CHAT_ID_TRASH};
 use crate::contact::{
-    may_be_valid_addr, normalize_name, Contact, ContactAddress, ContactId, Origin,
+    addr_cmp, may_be_valid_addr, normalize_name, Contact, ContactAddress, ContactId, Origin,
 };
 use crate::context::Context;
 use crate::debug_logging::maybe_set_logging_xdc_inner;
@@ -63,6 +63,11 @@ pub struct ReceivedMsg {
 
     /// Whether IMAP messages should be immediately deleted.
     pub needs_delete_job: bool,
+
+    /// Whether the From address was repeated in the signed part
+    /// (and we know that the signer intended to send from this address).
+    #[cfg(test)]
+    pub(crate) from_is_signed: bool,
 }
 
 /// Emulates reception of a message from the network.
@@ -98,6 +103,22 @@ pub async fn receive_imf(
     receive_imf_inner(context, &rfc724_mid, imf_raw, seen, None, false).await
 }
 
+/// Inserts a tombstone into `msgs` table
+/// to prevent downloading the same message in the future.
+///
+/// Returns tombstone database row ID.
+async fn insert_tombstone(context: &Context, rfc724_mid: &str) -> Result<MsgId> {
+    let row_id = context
+        .sql
+        .insert(
+            "INSERT INTO msgs(rfc724_mid, chat_id) VALUES (?,?)",
+            (rfc724_mid, DC_CHAT_ID_TRASH),
+        )
+        .await?;
+    let msg_id = MsgId::new(u32::try_from(row_id)?);
+    Ok(msg_id)
+}
+
 /// Receive a message and add it to the database.
 ///
 /// Returns an error on database failure or if the message is broken,
@@ -137,14 +158,7 @@ pub(crate) async fn receive_imf_inner(
                 return Ok(None);
             }
 
-            let row_id = context
-                .sql
-                .execute(
-                    "INSERT INTO msgs(rfc724_mid, chat_id) VALUES (?,?)",
-                    (rfc724_mid, DC_CHAT_ID_TRASH),
-                )
-                .await?;
-            let msg_ids = vec![MsgId::new(u32::try_from(row_id)?)];
+            let msg_ids = vec![insert_tombstone(context, rfc724_mid).await?];
 
             return Ok(Some(ReceivedMsg {
                 chat_id: DC_CHAT_ID_TRASH,
@@ -152,6 +166,8 @@ pub(crate) async fn receive_imf_inner(
                 sort_timestamp: 0,
                 msg_ids,
                 needs_delete_job: false,
+                #[cfg(test)]
+                from_is_signed: false,
             }));
         }
         Ok(mime_parser) => mime_parser,
@@ -229,25 +245,76 @@ pub(crate) async fn receive_imf_inner(
 
     update_verified_keys(context, &mut mime_parser, from_id).await?;
 
-    // Add parts
-    let received_msg = add_parts(
-        context,
-        &mut mime_parser,
-        imf_raw,
-        incoming,
-        &to_ids,
-        rfc724_mid,
-        sent_timestamp,
-        rcvd_timestamp,
-        from_id,
-        seen || replace_partial_download.is_some(),
-        is_partial_download,
-        replace_partial_download,
-        fetching_existing_messages,
-        prevent_rename,
-    )
-    .await
-    .context("add_parts error")?;
+    let received_msg;
+    if let Some(securejoin_step) = mime_parser.get_header(HeaderDef::SecureJoin) {
+        info!(context, "Received securejoin step {securejoin_step}.");
+
+        let res;
+        if incoming {
+            res = handle_securejoin_handshake(context, &mime_parser, from_id)
+                .await
+                .context("error in Secure-Join message handling")?;
+
+            // Peerstate could be updated by handling the Securejoin handshake.
+            let contact = Contact::get_by_id(context, from_id).await?;
+            mime_parser.decryption_info.peerstate =
+                Peerstate::from_addr(context, contact.get_addr()).await?;
+        } else {
+            let to_id = to_ids.get(0).copied().unwrap_or_default();
+            // handshake may mark contacts as verified and must be processed before chats are created
+            res = observe_securejoin_on_other_device(context, &mime_parser, to_id)
+                .await
+                .context("error in Secure-Join watching")?
+        }
+
+        match res {
+            securejoin::HandshakeMessage::Done | securejoin::HandshakeMessage::Ignore => {
+                let msg_id = insert_tombstone(context, rfc724_mid).await?;
+                received_msg = Some(ReceivedMsg {
+                    chat_id: DC_CHAT_ID_TRASH,
+                    state: MessageState::InSeen,
+                    sort_timestamp: sent_timestamp,
+                    msg_ids: vec![msg_id],
+                    needs_delete_job: res == securejoin::HandshakeMessage::Done,
+                    #[cfg(test)]
+                    from_is_signed: mime_parser.from_is_signed,
+                });
+            }
+            securejoin::HandshakeMessage::Propagate => {
+                received_msg = None;
+            }
+        }
+    } else {
+        received_msg = None;
+    }
+
+    let verified_encryption =
+        has_verified_encryption(context, &mime_parser, from_id, &to_ids).await?;
+
+    let received_msg = if let Some(received_msg) = received_msg {
+        received_msg
+    } else {
+        // Add parts
+        add_parts(
+            context,
+            &mut mime_parser,
+            imf_raw,
+            incoming,
+            &to_ids,
+            rfc724_mid,
+            sent_timestamp,
+            rcvd_timestamp,
+            from_id,
+            seen || replace_partial_download.is_some(),
+            is_partial_download,
+            replace_partial_download,
+            fetching_existing_messages,
+            prevent_rename,
+            verified_encryption,
+        )
+        .await
+        .context("add_parts error")?
+    };
 
     if !from_id.is_special() {
         contact::update_last_seen(context, from_id, sent_timestamp).await?;
@@ -388,7 +455,10 @@ pub(crate) async fn receive_imf_inner(
 ///
 /// Also returns whether it is blocked or not and its origin.
 ///
-/// * `prevent_rename`: passed through to `add_or_lookup_contacts_by_address_list()`
+/// * `prevent_rename`: if true, the display_name of this contact will not be changed. Useful for
+/// mailing lists: In some mailing lists, many users write from the same address but with different
+/// display names. We don't want the display name to change every time the user gets a new email from
+/// a mailing list.
 ///
 /// Returns `None` if From field does not contain a valid contact address.
 pub async fn from_field_to_contact_id(
@@ -452,6 +522,7 @@ async fn add_parts(
     mut replace_msg_id: Option<MsgId>,
     fetching_existing_messages: bool,
     prevent_rename: bool,
+    verified_encryption: VerifiedEncryption,
 ) -> Result<ReceivedMsg> {
     let mut chat_id = None;
     let mut chat_id_blocked = Blocked::Not;
@@ -510,38 +581,6 @@ async fn add_parts(
     if incoming {
         to_id = ContactId::SELF;
 
-        // Whether the message is a part of securejoin handshake that should be marked as seen
-        // automatically.
-        let securejoin_seen;
-
-        // handshake may mark contacts as verified and must be processed before chats are created
-        if mime_parser.get_header(HeaderDef::SecureJoin).is_some() {
-            match handle_securejoin_handshake(context, mime_parser, from_id)
-                .await
-                .context("error in Secure-Join message handling")?
-            {
-                securejoin::HandshakeMessage::Done => {
-                    chat_id = Some(DC_CHAT_ID_TRASH);
-                    needs_delete_job = true;
-                    securejoin_seen = true;
-                }
-                securejoin::HandshakeMessage::Ignore => {
-                    chat_id = Some(DC_CHAT_ID_TRASH);
-                    securejoin_seen = true;
-                }
-                securejoin::HandshakeMessage::Propagate => {
-                    // process messages as "member added" normally
-                    securejoin_seen = false;
-                }
-            }
-            // Peerstate could be updated by handling the Securejoin handshake.
-            let contact = Contact::get_by_id(context, from_id).await?;
-            mime_parser.decryption_info.peerstate =
-                Peerstate::from_addr(context, contact.get_addr()).await?;
-        } else {
-            securejoin_seen = false;
-        }
-
         let test_normal_chat = if from_id == ContactId::UNDEFINED {
             None
         } else {
@@ -614,6 +653,8 @@ async fn add_parts(
                 create_blocked,
                 from_id,
                 to_ids,
+                &verified_encryption,
+                sent_timestamp,
             )
             .await?
             {
@@ -636,14 +677,9 @@ async fn add_parts(
         if let Some(group_chat_id) = chat_id {
             if !chat::is_contact_in_chat(context, group_chat_id, from_id).await? {
                 let chat = Chat::load_from_db(context, group_chat_id).await?;
-                if chat.is_protected() {
-                    if chat.typ == Chattype::Single {
-                        // Just assign the message to the 1:1 chat with the actual sender instead.
-                        chat_id = None;
-                    } else {
-                        let s = stock_str::unknown_sender_for_chat(context).await;
-                        mime_parser.replace_msg_by_error(&s);
-                    }
+                if chat.is_protected() && chat.typ == Chattype::Single {
+                    // Just assign the message to the 1:1 chat with the actual sender instead.
+                    chat_id = None;
                 } else {
                     // In non-protected chats, just mark the sender as overridden. Therefore, the UI will prepend `~`
                     // to the sender's name, indicating to the user that he/she is not part of the group.
@@ -651,6 +687,12 @@ async fn add_parts(
                     let name: &str = from.display_name.as_ref().unwrap_or(&from.addr);
                     for part in &mut mime_parser.parts {
                         part.param.set(Param::OverrideSenderDisplayname, name);
+
+                        if chat.is_protected() {
+                            // In protected chat, also mark the message with an error.
+                            let s = stock_str::unknown_sender_for_chat(context).await;
+                            part.error = Some(s);
+                        }
                     }
                 }
             }
@@ -663,6 +705,7 @@ async fn add_parts(
                 from_id,
                 to_ids,
                 is_partial_download.is_some(),
+                &verified_encryption,
             )
             .await?;
         }
@@ -675,6 +718,7 @@ async fn add_parts(
                     allow_creation,
                     mailinglist_header,
                     mime_parser,
+                    sent_timestamp,
                 )
                 .await?
                 {
@@ -753,15 +797,8 @@ async fn add_parts(
                     false => None,
                 };
                 if let Some(chat) = chat {
-                    let mut new_protection = match has_verified_encryption(
-                        context,
-                        mime_parser,
-                        from_id,
-                        to_ids,
-                        Chattype::Single,
-                    )
-                    .await?
-                    {
+                    debug_assert!(chat.typ == Chattype::Single);
+                    let mut new_protection = match verified_encryption {
                         VerifiedEncryption::Verified => ProtectionStatus::Protected,
                         VerifiedEncryption::NotVerified(_) => ProtectionStatus::Unprotected,
                     };
@@ -794,7 +831,6 @@ async fn add_parts(
             || is_mdn
             || is_reaction
             || is_location_kml
-            || securejoin_seen
             || chat_id_blocked == Blocked::Yes
         {
             MessageState::InSeen
@@ -812,21 +848,7 @@ async fn add_parts(
         let self_sent =
             from_id == ContactId::SELF && to_ids.len() == 1 && to_ids.contains(&ContactId::SELF);
 
-        // handshake may mark contacts as verified and must be processed before chats are created
-        if mime_parser.get_header(HeaderDef::SecureJoin).is_some() {
-            match observe_securejoin_on_other_device(context, mime_parser, to_id)
-                .await
-                .context("error in Secure-Join watching")?
-            {
-                securejoin::HandshakeMessage::Done | securejoin::HandshakeMessage::Ignore => {
-                    chat_id = Some(DC_CHAT_ID_TRASH);
-                }
-                securejoin::HandshakeMessage::Propagate => {
-                    // process messages as "member added" normally
-                    chat_id = None;
-                }
-            }
-        } else if mime_parser.sync_items.is_some() && self_sent {
+        if mime_parser.sync_items.is_some() && self_sent {
             chat_id = Some(DC_CHAT_ID_TRASH);
         }
 
@@ -864,6 +886,8 @@ async fn add_parts(
                     Blocked::Not,
                     from_id,
                     to_ids,
+                    &verified_encryption,
+                    sent_timestamp,
                 )
                 .await?
                 {
@@ -906,6 +930,7 @@ async fn add_parts(
                 from_id,
                 to_ids,
                 is_partial_download.is_some(),
+                &verified_encryption,
             )
             .await?;
         }
@@ -1070,10 +1095,16 @@ async fn add_parts(
     if !chat_id.is_special() && is_partial_download.is_none() {
         let chat = Chat::load_from_db(context, chat_id).await?;
 
-        if chat.is_protected() {
-            if let VerifiedEncryption::NotVerified(err) =
-                has_verified_encryption(context, mime_parser, from_id, to_ids, chat.typ).await?
-            {
+        // For outgoing emails in the 1:1 chat we have an exception that
+        // they are allowed to be unencrypted:
+        // 1. They can't be an attack (they are outgoing, not incoming)
+        // 2. Probably the unencryptedness is just a temporary state, after all
+        //    the user obviously still uses DC
+        //    -> Showing info messages everytime would be a lot of noise
+        // 3. The info messages that are shown to the user ("Your chat partner
+        //    likely reinstalled DC" or similar) would be wrong.
+        if chat.is_protected() && (incoming || chat.typ != Chattype::Single) {
+            if let VerifiedEncryption::NotVerified(err) = verified_encryption {
                 warn!(context, "Verification problem: {err:#}.");
                 let s = format!("{err}. See 'Info' for more details");
                 mime_parser.replace_msg_by_error(&s);
@@ -1143,9 +1174,23 @@ async fn add_parts(
             Some(_) => group_changes_msgs.0.push(msg),
         }
     }
-    let mut group_changes_msgs: Vec<_> = group_changes_msgs.0.into_iter().rev().collect();
-    let mut parts = mime_parser.parts.iter_mut().peekable();
-    while let Some(part) = parts.peek() {
+
+    for group_changes_msg in group_changes_msgs.0 {
+        // Currently all additional group changes messages are "Member added".
+        chat::add_info_msg_with_cmd(
+            context,
+            chat_id,
+            &group_changes_msg,
+            SystemMessage::MemberAddedToGroup,
+            sort_timestamp,
+            None,
+            None,
+            None,
+        )
+        .await?;
+    }
+
+    for part in &mime_parser.parts {
         if part.is_reaction {
             let reaction_str = simplify::remove_footers(part.msg.as_str());
             set_msg_reaction(
@@ -1178,10 +1223,7 @@ async fn add_parts(
         }
 
         let mut txt_raw = "".to_string();
-        let group_changes_msg = group_changes_msgs.pop();
-        let (msg, typ): (&str, Viewtype) = if let Some(msg) = &group_changes_msg {
-            (msg, Viewtype::Text)
-        } else if let Some(better_msg) = &better_msg {
+        let (msg, typ): (&str, Viewtype) = if let Some(better_msg) = &better_msg {
             (better_msg, Viewtype::Text)
         } else {
             (&part.msg, part.typ)
@@ -1309,10 +1351,6 @@ RETURNING id
 
         debug_assert!(!row_id.is_special());
         created_db_entries.push(row_id);
-
-        if group_changes_msg.is_none() {
-            parts.next();
-        }
     }
 
     // check all parts whether they contain a new logging webxdc
@@ -1378,6 +1416,8 @@ RETURNING id
         sort_timestamp,
         msg_ids: created_db_entries,
         needs_delete_job,
+        #[cfg(test)]
+        from_is_signed: mime_parser.from_is_signed,
     })
 }
 
@@ -1441,7 +1481,7 @@ async fn calc_sort_timestamp(
     always_sort_to_bottom: bool,
     incoming: bool,
 ) -> Result<i64> {
-    let mut sort_timestamp = message_timestamp;
+    let mut sort_timestamp = min(message_timestamp, smeared_time(context));
 
     let last_msg_time: Option<i64> = if always_sort_to_bottom {
         // get newest message for this chat
@@ -1478,7 +1518,7 @@ async fn calc_sort_timestamp(
         }
     }
 
-    Ok(min(sort_timestamp, smeared_time(context)))
+    Ok(sort_timestamp)
 }
 
 async fn lookup_chat_by_reply(
@@ -1576,6 +1616,7 @@ async fn is_probably_private_reply(
 /// than two members, a new ad hoc group is created.
 ///
 /// On success the function returns the found/created (chat_id, chat_blocked) tuple.
+#[allow(clippy::too_many_arguments)]
 async fn create_or_lookup_group(
     context: &Context,
     mime_parser: &mut MimeMessage,
@@ -1584,6 +1625,8 @@ async fn create_or_lookup_group(
     create_blocked: Blocked,
     from_id: ContactId,
     to_ids: &[ContactId],
+    verified_encryption: &VerifiedEncryption,
+    timestamp: i64,
 ) -> Result<Option<(ChatId, Blocked)>> {
     let grpid = if let Some(grpid) = try_getting_grpid(mime_parser) {
         grpid
@@ -1596,7 +1639,7 @@ async fn create_or_lookup_group(
             member_ids.push(ContactId::SELF);
         }
 
-        let res = create_adhoc_group(context, mime_parser, create_blocked, &member_ids)
+        let res = create_adhoc_group(context, mime_parser, create_blocked, &member_ids, timestamp)
             .await
             .context("could not create ad hoc group")?
             .map(|chat_id| (chat_id, create_blocked));
@@ -1627,9 +1670,7 @@ async fn create_or_lookup_group(
     }
 
     let create_protected = if mime_parser.get_header(HeaderDef::ChatVerified).is_some() {
-        if let VerifiedEncryption::NotVerified(err) =
-            has_verified_encryption(context, mime_parser, from_id, to_ids, Chattype::Group).await?
-        {
+        if let VerifiedEncryption::NotVerified(err) = verified_encryption {
             warn!(context, "Verification problem: {err:#}.");
             let s = format!("{err}. See 'Info' for more details");
             mime_parser.replace_msg_by_error(&s);
@@ -1680,6 +1721,7 @@ async fn create_or_lookup_group(
             create_blocked,
             create_protected,
             None,
+            timestamp,
         )
         .await
         .with_context(|| format!("Failed to create group '{grpname}' for grpid={grpid}"))?;
@@ -1722,6 +1764,7 @@ async fn create_or_lookup_group(
 ///
 /// Optionally returns better message to replace the original system message.
 /// is_partial_download: whether the message is not fully downloaded.
+#[allow(clippy::too_many_arguments)]
 async fn apply_group_changes(
     context: &Context,
     mime_parser: &mut MimeMessage,
@@ -1730,6 +1773,7 @@ async fn apply_group_changes(
     from_id: ContactId,
     to_ids: &[ContactId],
     is_partial_download: bool,
+    verified_encryption: &VerifiedEncryption,
 ) -> Result<(Vec<String>, Option<String>)> {
     if chat_id.is_special() {
         // Do not apply group changes to the trash chat.
@@ -1748,7 +1792,7 @@ async fn apply_group_changes(
     // True if a Delta Chat client has explicitly added our current primary address.
     let self_added =
         if let Some(added_addr) = mime_parser.get_header(HeaderDef::ChatGroupMemberAdded) {
-            context.get_primary_self_addr().await? == *added_addr
+            addr_cmp(&context.get_primary_self_addr().await?, added_addr)
         } else {
             false
         };
@@ -1790,9 +1834,7 @@ async fn apply_group_changes(
     };
 
     if mime_parser.get_header(HeaderDef::ChatVerified).is_some() {
-        if let VerifiedEncryption::NotVerified(err) =
-            has_verified_encryption(context, mime_parser, from_id, to_ids, chat.typ).await?
-        {
+        if let VerifiedEncryption::NotVerified(err) = verified_encryption {
             warn!(context, "Verification problem: {err:#}.");
             let s = format!("{err}. See 'Info' for more details");
             mime_parser.replace_msg_by_error(&s);
@@ -2013,6 +2055,7 @@ async fn create_or_lookup_mailinglist(
     allow_creation: bool,
     list_id_header: &str,
     mime_parser: &MimeMessage,
+    timestamp: i64,
 ) -> Result<Option<(ChatId, Blocked)>> {
     let listid = mailinglist_header_listid(list_id_header)?;
 
@@ -2044,6 +2087,7 @@ async fn create_or_lookup_mailinglist(
             blocked,
             ProtectionStatus::Unprotected,
             param,
+            timestamp,
         )
         .await
         .with_context(|| {
@@ -2169,7 +2213,7 @@ async fn apply_mailinglist_changes(
             return Ok(());
         }
     };
-    let (contact_id, _) = Contact::add_or_lookup(context, "", list_post, Origin::Hidden).await?;
+    let (contact_id, _) = Contact::add_or_lookup(context, "", &list_post, Origin::Hidden).await?;
     let mut contact = Contact::get_by_id(context, contact_id).await?;
     if contact.param.get(Param::ListId) != Some(listid) {
         contact.param.set(Param::ListId, listid);
@@ -2228,6 +2272,7 @@ async fn create_adhoc_group(
     mime_parser: &MimeMessage,
     create_blocked: Blocked,
     member_ids: &[ContactId],
+    timestamp: i64,
 ) -> Result<Option<ChatId>> {
     if mime_parser.is_mailinglist_message() {
         info!(
@@ -2272,6 +2317,7 @@ async fn create_adhoc_group(
         create_blocked,
         ProtectionStatus::Unprotected,
         None,
+        timestamp,
     )
     .await?;
     chat::add_to_chat_contacts_table(context, new_chat_id, member_ids).await?;
@@ -2348,8 +2394,7 @@ async fn update_verified_keys(
 
 /// Checks whether the message is allowed to appear in a protected chat.
 ///
-/// This means that it is encrypted, signed with a verified key,
-/// and if it's a group, all the recipients are verified.
+/// This means that it is encrypted and signed with a verified key.
 ///
 /// Also propagates gossiped keys to verified if needed.
 async fn has_verified_encryption(
@@ -2357,21 +2402,15 @@ async fn has_verified_encryption(
     mimeparser: &MimeMessage,
     from_id: ContactId,
     to_ids: &[ContactId],
-    chat_type: Chattype,
 ) -> Result<VerifiedEncryption> {
     use VerifiedEncryption::*;
 
-    if from_id == ContactId::SELF && chat_type == Chattype::Single {
-        // For outgoing emails in the 1:1 chat, we have an exception that
-        // they are allowed to be unencrypted:
-        // 1. They can't be an attack (they are outgoing, not incoming)
-        // 2. Probably the unencryptedness is just a temporary state, after all
-        //    the user obviously still uses DC
-        //    -> Showing info messages everytime would be a lot of noise
-        // 3. The info messages that are shown to the user ("Your chat partner
-        //    likely reinstalled DC" or similar) would be wrong.
-        return Ok(Verified);
-    }
+    // We do not need to check if we are verified with ourself.
+    let to_ids = to_ids
+        .iter()
+        .copied()
+        .filter(|id| *id != ContactId::SELF)
+        .collect::<Vec<ContactId>>();
 
     if !mimeparser.was_encrypted() {
         return Ok(NotVerified("This message is not encrypted".to_string()));
@@ -2401,17 +2440,6 @@ async fn has_verified_encryption(
         }
     }
 
-    // we do not need to check if we are verified with ourself
-    let to_ids = to_ids
-        .iter()
-        .copied()
-        .filter(|id| *id != ContactId::SELF)
-        .collect::<Vec<ContactId>>();
-
-    if to_ids.is_empty() {
-        return Ok(Verified);
-    }
-
     mark_recipients_as_verified(context, from_id, to_ids, mimeparser).await?;
     Ok(Verified)
 }
@@ -2421,7 +2449,15 @@ async fn mark_recipients_as_verified(
     from_id: ContactId,
     to_ids: Vec<ContactId>,
     mimeparser: &MimeMessage,
-) -> Result<(), anyhow::Error> {
+) -> Result<()> {
+    if to_ids.is_empty() {
+        return Ok(());
+    }
+
+    if mimeparser.get_header(HeaderDef::ChatVerified).is_none() {
+        return Ok(());
+    }
+
     let rows = context
         .sql
         .query_map(
@@ -2469,7 +2505,7 @@ async fn mark_recipients_as_verified(
                             let (to_contact_id, _) = Contact::add_or_lookup(
                                 context,
                                 "",
-                                ContactAddress::new(&to_addr)?,
+                                &ContactAddress::new(&to_addr)?,
                                 Origin::Hidden,
                             )
                             .await?;
@@ -2574,11 +2610,6 @@ pub(crate) async fn get_prefetch_parent_message(
 /// Looks up contact IDs from the database given the list of recipients.
 ///
 /// Returns vector of IDs guaranteed to be unique.
-///
-/// * param `prevent_rename`: if true, the display_name of this contact will not be changed. Useful for
-/// mailing lists: In some mailing lists, many users write from the same address but with different
-/// display names. We don't want the display name to change every time the user gets a new email from
-/// a mailing list.
 async fn add_or_lookup_contacts_by_address_list(
     context: &Context,
     address_list: &[SingleInfo],
@@ -2607,7 +2638,7 @@ async fn add_or_lookup_contacts_by_address_list(
 async fn add_or_lookup_contact_by_addr(
     context: &Context,
     display_name: Option<&str>,
-    addr: ContactAddress<'_>,
+    addr: ContactAddress,
     origin: Origin,
 ) -> Result<ContactId> {
     if context.is_self_addr(&addr).await? {
@@ -2616,7 +2647,7 @@ async fn add_or_lookup_contact_by_addr(
     let display_name_normalized = display_name.map(normalize_name).unwrap_or_default();
 
     let (contact_id, _modified) =
-        Contact::add_or_lookup(context, &display_name_normalized, addr, origin).await?;
+        Contact::add_or_lookup(context, &display_name_normalized, &addr, origin).await?;
     Ok(contact_id)
 }
 

src/receive_imf/tests.rs

@@ -322,7 +322,7 @@ async fn test_no_from() {
     let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
     assert!(chats.get_msg_id(0).is_err());
 
-    receive_imf(
+    let received = receive_imf(
         context,
         b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                  To: bob@example.com\n\
@@ -335,8 +335,13 @@ async fn test_no_from() {
         false,
     )
     .await
+    .unwrap()
     .unwrap();
 
+    // Check that tombstone MsgId is returned.
+    assert_eq!(received.msg_ids.len(), 1);
+    assert!(!received.msg_ids[0].is_special());
+
     let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
     // Check that the message is not shown to the user:
     assert!(chats.is_empty());
@@ -431,7 +436,7 @@ async fn test_escaped_recipients() {
     let carl_contact_id = Contact::add_or_lookup(
         &t,
         "Carl",
-        ContactAddress::new("carl@host.tld").unwrap(),
+        &ContactAddress::new("carl@host.tld").unwrap(),
         Origin::IncomingUnknownFrom,
     )
     .await
@@ -477,7 +482,7 @@ async fn test_cc_to_contact() {
     let carl_contact_id = Contact::add_or_lookup(
         &t,
         "garabage",
-        ContactAddress::new("carl@host.tld").unwrap(),
+        &ContactAddress::new("carl@host.tld").unwrap(),
         Origin::IncomingUnknownFrom,
     )
     .await
@@ -2003,7 +2008,7 @@ async fn test_duplicate_message() -> Result<()> {
     let bob_contact_id = Contact::add_or_lookup(
         &alice,
         "Bob",
-        ContactAddress::new("bob@example.org").unwrap(),
+        &ContactAddress::new("bob@example.org").unwrap(),
         Origin::IncomingUnknownFrom,
     )
     .await?
@@ -2060,7 +2065,7 @@ async fn test_ignore_footer_status_from_mailinglist() -> Result<()> {
     let bob_id = Contact::add_or_lookup(
         &t,
         "",
-        ContactAddress::new("bob@example.net").unwrap(),
+        &ContactAddress::new("bob@example.net").unwrap(),
         Origin::IncomingUnknownCc,
     )
     .await?
@@ -2139,7 +2144,7 @@ async fn test_ignore_old_status_updates() -> Result<()> {
     let bob_id = Contact::add_or_lookup(
         &t,
         "",
-        ContactAddress::new("bob@example.net")?,
+        &ContactAddress::new("bob@example.net")?,
         Origin::AddressBook,
     )
     .await?
@@ -2623,19 +2628,17 @@ async fn test_incoming_contact_request() -> Result<()> {
     let chat = chat::Chat::load_from_db(&t, msg.chat_id).await?;
     assert!(chat.is_contact_request());
 
-    loop {
-        let event = t
-            .evtracker
-            .get_matching(|evt| matches!(evt, EventType::IncomingMsg { .. }))
-            .await;
-        match event {
-            EventType::IncomingMsg { chat_id, msg_id } => {
-                assert_eq!(msg.chat_id, chat_id);
-                assert_eq!(msg.id, msg_id);
-                return Ok(());
-            }
-            _ => unreachable!(),
+    let event = t
+        .evtracker
+        .get_matching(|evt| matches!(evt, EventType::IncomingMsg { .. }))
+        .await;
+    match event {
+        EventType::IncomingMsg { chat_id, msg_id } => {
+            assert_eq!(msg.chat_id, chat_id);
+            assert_eq!(msg.id, msg_id);
+            Ok(())
         }
+        _ => unreachable!(),
     }
 }
 
@@ -3107,7 +3110,8 @@ async fn test_thunderbird_autocrypt() -> Result<()> {
     let t = TestContext::new_bob().await;
 
     let raw = include_bytes!("../../test-data/message/thunderbird_with_autocrypt.eml");
-    receive_imf(&t, raw, false).await?;
+    let received_msg = receive_imf(&t, raw, false).await?.unwrap();
+    assert!(received_msg.from_is_signed);
 
     let peerstate = Peerstate::from_addr(&t, "alice@example.org")
         .await?
@@ -3188,7 +3192,8 @@ async fn test_thunderbird_unsigned() -> Result<()> {
 
     // Alice receives an unsigned message from Bob.
     let raw = include_bytes!("../../test-data/message/thunderbird_encrypted_unsigned.eml");
-    receive_imf(&alice, raw, false).await?;
+    let received_msg = receive_imf(&alice, raw, false).await?.unwrap();
+    assert!(!received_msg.from_is_signed);
 
     let msg = alice.get_last_msg().await;
     assert!(!msg.get_showpadlock());
@@ -3197,6 +3202,27 @@ async fn test_thunderbird_unsigned() -> Result<()> {
     Ok(())
 }
 
+/// Bob receives an encrypted unsigned message with only an unencrypted Subject.
+///
+/// Test that the message is displayed without any errors,
+/// but also without a padlock, but with the Subject.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_thunderbird_unsigned_with_unencrypted_subject() -> Result<()> {
+    let bob = TestContext::new_bob().await;
+
+    let raw = include_bytes!(
+        "../../test-data/message/thunderbird_encrypted_unsigned_with_unencrypted_subject.eml"
+    );
+    receive_imf(&bob, raw, false).await?;
+
+    let msg = bob.get_last_msg().await;
+    assert!(!msg.get_showpadlock());
+    assert!(msg.error().is_none());
+    assert_eq!(msg.get_subject(), "Hello!");
+
+    Ok(())
+}
+
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn test_mua_user_adds_member() -> Result<()> {
     let t = TestContext::new_alice().await;

src/scheduler/connectivity.rs

@@ -2,7 +2,7 @@ use core::fmt;
 use std::cmp::min;
 use std::{iter::once, ops::Deref, sync::Arc};
 
-use anyhow::{anyhow, Result};
+use anyhow::Result;
 use humansize::{format_size, BINARY};
 use tokio::sync::Mutex;
 
@@ -299,6 +299,10 @@ impl Context {
                     .yellow {
                         background-color: #fdc625;
                     }
+                    .not-started-error {
+                        font-size: 2em;
+                        color: red;
+                    }
                 </style>
             </head>
             <body>"#
@@ -318,7 +322,8 @@ impl Context {
                 sched.smtp.state.connectivity.clone(),
             ),
             _ => {
-                return Err(anyhow!("Not started"));
+                ret += "<div class=\"not-started-error\">Error: IO Not Started</div><p>Please report this issue to the app developer.</p>\n</body></html>\n";
+                return Ok(ret);
             }
         };
         drop(lock);

src/securejoin.rs

@@ -1,4 +1,4 @@
-//! Verified contact protocol implementation as [specified by countermitm project](https://countermitm.readthedocs.io/en/stable/new.html#setup-contact-protocol).
+//! Verified contact protocol implementation as [specified by countermitm project](https://securejoin.readthedocs.io/en/latest/new.html#setup-contact-protocol).
 
 use std::convert::TryFrom;
 
@@ -243,6 +243,7 @@ async fn fingerprint_equals_sender(
 /// next with this incoming setup-contact/secure-join handshake message.
 ///
 /// [`receive_imf`]: crate::receive_imf::receive_imf
+#[derive(Debug, PartialEq, Eq)]
 pub(crate) enum HandshakeMessage {
     /// The message has been fully handled and should be removed/delete.
     ///
@@ -426,6 +427,7 @@ pub(crate) async fn handle_securejoin_handshake(
                 .await?;
                 return Ok(HandshakeMessage::Ignore);
             }
+            contact_id.regossip_keys(context).await?;
             Contact::scaleup_origin_by_id(context, contact_id, Origin::SecurejoinInvited).await?;
             info!(context, "Auth verified.",);
             context.emit_event(EventType::ContactsChanged(Some(contact_id)));
@@ -514,12 +516,8 @@ pub(crate) async fn handle_securejoin_handshake(
         }
 
         "vg-member-added-received" | "vc-contact-confirm-received" => {
-            /*==========================================================
-            ====              Alice - the inviter side              ====
-            ====  Step 8 in "Out-of-band verified groups" protocol  ====
-            ==========================================================*/
-
-            Ok(HandshakeMessage::Done) // "Done" deletes the message
+            // Deprecated steps, delete them immediately.
+            Ok(HandshakeMessage::Done)
         }
         _ => {
             warn!(context, "invalid step: {}", step);
@@ -540,10 +538,10 @@ pub(crate) async fn handle_securejoin_handshake(
 ///   before sending vg-member-added/vc-contact-confirm - so, if we observe vg-member-added/vc-contact-confirm,
 ///   we can mark the peer as verified as well.
 ///
-/// - if we see the self-sent-message vg-member-added-received
+/// - if we see the self-sent-message vg-request-with-auth/vc-request-with-auth
 ///   we know that we're an joiner-observer.
-///   the joining device has marked the peer as verified on vg-member-added/vc-contact-confirm
-///   before sending vg-member-added-received - so, if we observe vg-member-added-received,
+///   the joining device has marked the peer as verified
+///   before sending vg-request-with-auth/vc-request-with-auth - so, if we observe vg-member-added-received,
 ///   we can mark the peer as verified as well.
 pub(crate) async fn observe_securejoin_on_other_device(
     context: &Context,
@@ -562,9 +560,7 @@ pub(crate) async fn observe_securejoin_on_other_device(
         "vg-request-with-auth"
         | "vc-request-with-auth"
         | "vg-member-added"
-        | "vc-contact-confirm"
-        | "vg-member-added-received"
-        | "vc-contact-confirm-received" => {
+        | "vc-contact-confirm" => {
             if !encrypted_and_signed(
                 context,
                 mime_message,
@@ -769,11 +765,10 @@ fn encrypted_and_signed(
 mod tests {
     use super::*;
     use crate::chat;
-    use crate::chat::ProtectionStatus;
+    use crate::chat::{remove_contact_from_chat, ProtectionStatus};
     use crate::chatlist::Chatlist;
     use crate::constants::Chattype;
     use crate::contact::ContactAddress;
-    use crate::contact::VerifiedStatus;
     use crate::peerstate::Peerstate;
     use crate::receive_imf::receive_imf;
     use crate::stock_str::chat_protection_enabled;
@@ -892,17 +887,11 @@ mod tests {
         let contact_bob = Contact::get_by_id(&alice.ctx, contact_bob_id)
             .await
             .unwrap();
-        assert_eq!(
-            contact_bob.is_verified(&alice.ctx).await.unwrap(),
-            VerifiedStatus::Unverified
-        );
+        assert_eq!(contact_bob.is_verified(&alice.ctx).await.unwrap(), false);
 
         // Step 5+6: Alice receives vc-request-with-auth, sends vc-contact-confirm
         alice.recv_msg(&sent).await;
-        assert_eq!(
-            contact_bob.is_verified(&alice.ctx).await.unwrap(),
-            VerifiedStatus::BidirectVerified
-        );
+        assert_eq!(contact_bob.is_verified(&alice.ctx).await.unwrap(), true);
 
         // exactly one one-to-one chat should be visible for both now
         // (check this before calling alice.create_chat() explicitly below)
@@ -945,35 +934,18 @@ mod tests {
         let contact_alice = Contact::get_by_id(&bob.ctx, contact_alice_id)
             .await
             .unwrap();
-        assert_eq!(
-            contact_bob.is_verified(&bob.ctx).await.unwrap(),
-            VerifiedStatus::Unverified
-        );
+        assert_eq!(contact_bob.is_verified(&bob.ctx).await.unwrap(), false);
 
-        // Step 7: Bob receives vc-contact-confirm, sends vc-contact-confirm-received
+        // Step 7: Bob receives vc-contact-confirm
         bob.recv_msg(&sent).await;
-        assert_eq!(
-            contact_alice.is_verified(&bob.ctx).await.unwrap(),
-            VerifiedStatus::BidirectVerified
-        );
+        assert_eq!(contact_alice.is_verified(&bob.ctx).await.unwrap(), true);
 
         // Check Bob got the verified message in his 1:1 chat.
-        {
-            let chat = bob.create_chat(&alice).await;
-            let msg = get_chat_msg(&bob, chat.get_id(), 0, 1).await;
-            assert!(msg.is_info());
-            let expected_text = chat_protection_enabled(&bob).await;
-            assert_eq!(msg.get_text(), expected_text);
-        }
-
-        // Check Bob sent the final message
-        let sent = bob.pop_sent_msg().await;
-        let msg = alice.parse_msg(&sent).await;
-        assert!(msg.was_encrypted());
-        assert_eq!(
-            msg.get_header(HeaderDef::SecureJoin).unwrap(),
-            "vc-contact-confirm-received"
-        );
+        let chat = bob.create_chat(&alice).await;
+        let msg = get_chat_msg(&bob, chat.get_id(), 0, 1).await;
+        assert!(msg.is_info());
+        let expected_text = chat_protection_enabled(&bob).await;
+        assert_eq!(msg.get_text(), expected_text);
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -1057,22 +1029,16 @@ mod tests {
         let (contact_bob_id, _modified) = Contact::add_or_lookup(
             &alice.ctx,
             "Bob",
-            ContactAddress::new("bob@example.net")?,
+            &ContactAddress::new("bob@example.net")?,
             Origin::ManuallyCreated,
         )
         .await?;
         let contact_bob = Contact::get_by_id(&alice.ctx, contact_bob_id).await?;
-        assert_eq!(
-            contact_bob.is_verified(&alice.ctx).await?,
-            VerifiedStatus::Unverified
-        );
+        assert_eq!(contact_bob.is_verified(&alice.ctx).await?, false);
 
         // Step 5+6: Alice receives vc-request-with-auth, sends vc-contact-confirm
         alice.recv_msg(&sent).await;
-        assert_eq!(
-            contact_bob.is_verified(&alice.ctx).await?,
-            VerifiedStatus::BidirectVerified
-        );
+        assert_eq!(contact_bob.is_verified(&alice.ctx).await?, true);
 
         let sent = alice.pop_sent_msg().await;
         let msg = bob.parse_msg(&sent).await;
@@ -1089,25 +1055,12 @@ mod tests {
                 .expect("Error looking up contact")
                 .expect("Contact not found");
         let contact_alice = Contact::get_by_id(&bob.ctx, contact_alice_id).await?;
-        assert_eq!(
-            contact_bob.is_verified(&bob.ctx).await?,
-            VerifiedStatus::Unverified
-        );
+        assert_eq!(contact_bob.is_verified(&bob.ctx).await?, false);
 
-        // Step 7: Bob receives vc-contact-confirm, sends vc-contact-confirm-received
+        // Step 7: Bob receives vc-contact-confirm
         bob.recv_msg(&sent).await;
-        assert_eq!(
-            contact_alice.is_verified(&bob.ctx).await?,
-            VerifiedStatus::BidirectVerified
-        );
+        assert_eq!(contact_alice.is_verified(&bob.ctx).await?, true);
 
-        let sent = bob.pop_sent_msg().await;
-        let msg = alice.parse_msg(&sent).await;
-        assert!(msg.was_encrypted());
-        assert_eq!(
-            msg.get_header(HeaderDef::SecureJoin).unwrap(),
-            "vc-contact-confirm-received"
-        );
         Ok(())
     }
 
@@ -1230,17 +1183,11 @@ mod tests {
                 .await?
                 .expect("Contact not found");
         let contact_bob = Contact::get_by_id(&alice.ctx, contact_bob_id).await?;
-        assert_eq!(
-            contact_bob.is_verified(&alice.ctx).await?,
-            VerifiedStatus::Unverified
-        );
+        assert_eq!(contact_bob.is_verified(&alice.ctx).await?, false);
 
         // Step 5+6: Alice receives vg-request-with-auth, sends vg-member-added
         alice.recv_msg(&sent).await;
-        assert_eq!(
-            contact_bob.is_verified(&alice.ctx).await?,
-            VerifiedStatus::BidirectVerified
-        );
+        assert_eq!(contact_bob.is_verified(&alice.ctx).await?, true);
 
         let sent = alice.pop_sent_msg().await;
         let msg = bob.parse_msg(&sent).await;
@@ -1276,19 +1223,13 @@ mod tests {
                 .expect("Error looking up contact")
                 .expect("Contact not found");
         let contact_alice = Contact::get_by_id(&bob.ctx, contact_alice_id).await?;
-        assert_eq!(
-            contact_bob.is_verified(&bob.ctx).await?,
-            VerifiedStatus::Unverified
-        );
+        assert_eq!(contact_bob.is_verified(&bob.ctx).await?, false);
 
-        // Step 7: Bob receives vg-member-added, sends vg-member-added-received
+        // Step 7: Bob receives vg-member-added
         bob.recv_msg(&sent).await;
         {
             // Bob has Alice verified, message shows up in the group chat.
-            assert_eq!(
-                contact_alice.is_verified(&bob.ctx).await?,
-                VerifiedStatus::BidirectVerified
-            );
+            assert_eq!(contact_alice.is_verified(&bob.ctx).await?, true);
             let chat = bob.get_chat(&alice).await;
             assert_eq!(
                 chat.blocked,
@@ -1304,14 +1245,6 @@ mod tests {
             }
         }
 
-        let sent = bob.pop_sent_msg().await;
-        let msg = alice.parse_msg(&sent).await;
-        assert!(msg.was_encrypted());
-        assert_eq!(
-            msg.get_header(HeaderDef::SecureJoin).unwrap(),
-            "vg-member-added-received"
-        );
-
         let bob_chat = Chat::load_from_db(&bob.ctx, bob_chatid).await?;
         assert!(bob_chat.is_protected());
         assert!(bob_chat.typ == Chattype::Group);
@@ -1351,4 +1284,32 @@ First thread."#;
         assert!(get_securejoin_qr(&alice, Some(chat_id)).await.is_err());
         Ok(())
     }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_unknown_sender() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let alice = tcm.alice().await;
+        let bob = tcm.bob().await;
+
+        tcm.execute_securejoin(&alice, &bob).await;
+
+        let alice_chat_id = alice
+            .create_group_with_members(ProtectionStatus::Protected, "Group with Bob", &[&bob])
+            .await;
+
+        let sent = alice.send_text(alice_chat_id, "Hi!").await;
+        let bob_chat_id = bob.recv_msg(&sent).await.chat_id;
+
+        let sent = bob.send_text(bob_chat_id, "Hi hi!").await;
+
+        let alice_bob_contact_id = Contact::create(&alice, "Bob", "bob@example.net").await?;
+        remove_contact_from_chat(&alice, alice_chat_id, alice_bob_contact_id).await?;
+
+        // The message from Bob is delivered late, Bob is already removed.
+        let msg = alice.recv_msg(&sent).await;
+        assert_eq!(msg.text, "Hi hi!");
+        assert_eq!(msg.error.unwrap(), "Unknown sender for this chat.");
+
+        Ok(())
+    }
 }

src/securejoin/bob.rs

@@ -16,7 +16,7 @@ use crate::context::Context;
 use crate::events::EventType;
 use crate::mimeparser::MimeMessage;
 use crate::sync::Sync::*;
-use crate::tools::time;
+use crate::tools::{create_smeared_timestamp, time};
 use crate::{chat, stock_str};
 
 /// Starts the securejoin protocol with the QR `invite`.
@@ -58,7 +58,6 @@ pub(super) async fn start_protocol(context: &Context, invite: QrInvite) -> Resul
         QrInvite::Group { .. } => {
             // For a secure-join we need to create the group and add the contact.  The group will
             // only become usable once the protocol is finished.
-            // TODO: how does this group become usable?
             let group_chat_id = state.joining_chat_id(context).await?;
             if !is_contact_in_chat(context, group_chat_id, invite.contact_id()).await? {
                 chat::add_to_chat_contacts_table(context, group_chat_id, &[invite.contact_id()])
@@ -193,6 +192,7 @@ impl BobState {
                             Blocked::Not,
                             ProtectionStatus::Unprotected, // protection is added later as needed
                             None,
+                            create_smeared_timestamp(context),
                         )
                         .await?
                     }

src/securejoin/bobstate.rs

@@ -345,17 +345,6 @@ impl BobState {
             .await?;
         context.emit_event(EventType::ContactsChanged(None));
 
-        self.send_handshake_message(context, BobHandshakeMsg::ContactConfirmReceived)
-            .await
-            .map_err(|_| {
-                warn!(
-                    context,
-                    "Failed to send vc-contact-confirm-received/vg-member-added-received"
-                );
-            })
-            // This is not an error affecting the protocol outcome.
-            .ok();
-
         self.update_next(&context.sql, SecureJoinStep::Completed)
             .await?;
         Ok(Some(BobHandshakeStage::Completed))
@@ -401,9 +390,6 @@ async fn send_handshake_message(
             msg.param.set(Param::Arg2, invite.authcode());
             msg.param.set_int(Param::GuaranteeE2ee, 1);
         }
-        BobHandshakeMsg::ContactConfirmReceived => {
-            msg.param.set_int(Param::GuaranteeE2ee, 1);
-        }
     };
 
     // Sends our own fingerprint in the Secure-Join-Fingerprint header.
@@ -425,8 +411,6 @@ enum BobHandshakeMsg {
     Request,
     /// vc-request-with-auth or vg-request-with-auth
     RequestWithAuth,
-    /// vc-contact-confirm-received or vg-member-added-received
-    ContactConfirmReceived,
 }
 
 impl BobHandshakeMsg {
@@ -454,10 +438,6 @@ impl BobHandshakeMsg {
                 QrInvite::Contact { .. } => "vc-request-with-auth",
                 QrInvite::Group { .. } => "vg-request-with-auth",
             },
-            Self::ContactConfirmReceived => match invite {
-                QrInvite::Contact { .. } => "vc-contact-confirm-received",
-                QrInvite::Group { .. } => "vg-member-added-received",
-            },
         }
     }
 }

src/smtp.rs

@@ -27,8 +27,8 @@ use crate::scheduler::connectivity::ConnectivityStore;
 use crate::socks::Socks5Config;
 use crate::sql;
 
-/// SMTP write and read timeout.
-const SMTP_TIMEOUT: Duration = Duration::from_secs(30);
+/// SMTP connection, write and read timeout.
+const SMTP_TIMEOUT: Duration = Duration::from_secs(60);
 
 #[derive(Default)]
 pub(crate) struct Smtp {

src/sql.rs

@@ -131,6 +131,23 @@ impl Sql {
         // drop closes the connection
     }
 
+    /// Flushes the WAL file.
+    pub(crate) async fn checkpoint(&self, context: &Context) -> Result<()> {
+        let busy = self
+            .call_write(move |conn| {
+                let busy = conn.query_row("PRAGMA wal_checkpoint(TRUNCATE)", (), |row| {
+                    let busy: bool = row.get(0)?;
+                    Ok(busy)
+                })?;
+                Ok(busy)
+            })
+            .await?;
+        if busy {
+            warn!(context, "Failed to checkpoint WAL");
+        }
+        Ok(())
+    }
+
     /// Imports the database from a separate file with the given passphrase.
     pub(crate) async fn import(&self, path: &Path, passphrase: String) -> Result<()> {
         let path_str = path

src/sql/migrations.rs

@@ -763,6 +763,28 @@ CREATE INDEX smtp_messageid ON imap(rfc724_mid);
         .await?;
     }
 
+    if dbversion < 105 {
+        // Create UNIQUE uid column and drop unused update_item_read column.
+        sql.execute_migration(
+            r#"CREATE TABLE new_msgs_status_updates (
+id INTEGER PRIMARY KEY AUTOINCREMENT,
+msg_id INTEGER,
+update_item TEXT DEFAULT '',
+uid TEXT UNIQUE
+);
+INSERT OR IGNORE INTO new_msgs_status_updates SELECT
+  id, msg_id, update_item, NULL
+FROM msgs_status_updates;
+DROP TABLE msgs_status_updates;
+ALTER TABLE new_msgs_status_updates RENAME TO msgs_status_updates;
+CREATE INDEX msgs_status_updates_index1 ON msgs_status_updates (msg_id);
+CREATE INDEX msgs_status_updates_index2 ON msgs_status_updates (uid);
+"#,
+            105,
+        )
+        .await?;
+    }
+
     let new_version = sql
         .get_raw_config_int(VERSION_CFG)
         .await?

src/stock_str.rs

@@ -92,7 +92,7 @@ pub enum StockMessage {
     #[strum(props(fallback = "%1$s verified."))]
     ContactVerified = 35,
 
-    #[strum(props(fallback = "Cannot verify %1$s"))]
+    #[strum(props(fallback = "Cannot establish guaranteed end-to-end encryption with %1$s"))]
     ContactNotVerified = 36,
 
     #[strum(props(fallback = "Changed setup for %1$s"))]
@@ -149,7 +149,7 @@ pub enum StockMessage {
                     however, of course, if they like, you may point them to 👉 https://get.delta.chat"))]
     WelcomeMessage = 71,
 
-    #[strum(props(fallback = "Unknown sender for this chat. See 'info' for more details."))]
+    #[strum(props(fallback = "Unknown sender for this chat."))]
     UnknownSenderForChat = 72,
 
     #[strum(props(fallback = "Message from %1$s"))]
@@ -832,7 +832,7 @@ pub(crate) async fn contact_verified(context: &Context, contact: &Contact) -> St
         .replace1(addr)
 }
 
-/// Stock string: `Cannot verify %1$s`.
+/// Stock string: `Cannot establish guaranteed end-to-end encryption with %1$s`.
 pub(crate) async fn contact_not_verified(context: &Context, contact: &Contact) -> String {
     let addr = &contact.get_name_n_addr();
     translated(context, StockMessage::ContactNotVerified)
@@ -930,7 +930,7 @@ pub(crate) async fn welcome_message(context: &Context) -> String {
     translated(context, StockMessage::WelcomeMessage).await
 }
 
-/// Stock string: `Unknown sender for this chat. See 'info' for more details.`.
+/// Stock string: `Unknown sender for this chat.`.
 pub(crate) async fn unknown_sender_for_chat(context: &Context) -> String {
     translated(context, StockMessage::UnknownSenderForChat).await
 }

src/sync.rs

@@ -20,7 +20,7 @@ use crate::tools::time;
 use crate::{stock_str, token};
 
 /// Whether to send device sync messages. Aimed for usage in the internal API.
-#[derive(Debug)]
+#[derive(Debug, PartialEq)]
 pub(crate) enum Sync {
     Nosync,
     Sync,
@@ -35,6 +35,15 @@ impl From<Sync> for bool {
     }
 }
 
+impl From<bool> for Sync {
+    fn from(sync: bool) -> Sync {
+        match sync {
+            false => Sync::Nosync,
+            true => Sync::Sync,
+        }
+    }
+}
+
 #[derive(Debug, Serialize, Deserialize)]
 pub(crate) struct QrTokenData {
     pub(crate) invitenumber: String,
@@ -50,12 +59,24 @@ pub(crate) enum SyncData {
         id: chat::SyncId,
         action: chat::SyncAction,
     },
+    Config {
+        key: Config,
+        val: String,
+    },
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+#[serde(untagged)]
+pub(crate) enum SyncDataOrUnknown {
+    SyncData(SyncData),
+    Unknown(serde_json::Value),
 }
 
 #[derive(Debug, Serialize, Deserialize)]
 pub(crate) struct SyncItem {
     timestamp: i64,
-    data: SyncData,
+
+    data: SyncDataOrUnknown,
 }
 
 #[derive(Debug, Deserialize)]
@@ -63,6 +84,12 @@ pub(crate) struct SyncItems {
     items: Vec<SyncItem>,
 }
 
+impl From<SyncData> for SyncDataOrUnknown {
+    fn from(sync_data: SyncData) -> Self {
+        Self::SyncData(sync_data)
+    }
+}
+
 impl Context {
     /// Adds an item to the list of items that should be synchronized to other devices.
     ///
@@ -79,7 +106,10 @@ impl Context {
             return Ok(());
         }
 
-        let item = SyncItem { timestamp, data };
+        let item = SyncItem {
+            timestamp,
+            data: data.into(),
+        };
         let item = serde_json::to_string(&item)?;
         self.sql
             .execute("INSERT INTO multi_device_sync (item) VALUES(?);", (item,))
@@ -242,9 +272,19 @@ impl Context {
         info!(self, "executing {} sync item(s)", items.items.len());
         for item in &items.items {
             match &item.data {
-                AddQrToken(token) => self.add_qr_token(token).await,
-                DeleteQrToken(token) => self.delete_qr_token(token).await,
-                AlterChat { id, action } => self.sync_alter_chat(id, action).await,
+                SyncDataOrUnknown::SyncData(data) => match data {
+                    AddQrToken(token) => self.add_qr_token(token).await,
+                    DeleteQrToken(token) => self.delete_qr_token(token).await,
+                    AlterChat { id, action } => self.sync_alter_chat(id, action).await,
+                    SyncData::Config { key, val } => match key.is_synced() {
+                        true => self.set_config_ex(Sync::Nosync, *key, Some(val)).await,
+                        false => Ok(()),
+                    },
+                },
+                SyncDataOrUnknown::Unknown(data) => {
+                    warn!(self, "Ignored unknown sync item: {data}.");
+                    Ok(())
+                }
             }
             .log_err(self)
             .ok();
@@ -383,48 +423,32 @@ mod tests {
 
         assert!(t.parse_sync_items(r#"{"badname":[]}"#.to_string()).is_err());
 
-        assert!(t.parse_sync_items(
-                r#"{"items":[{"timestamp":1631781316,"data":{"BadItem":{"invitenumber":"in","auth":"a","grpid":null}}}]}"#
-                    .to_string(),
-            )
-            .is_err());
-
-        assert!(t.parse_sync_items(
-                r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","auth":123}}}]}"#.to_string(),
-            )
-            .is_err()); // `123` is invalid for `String`
-
-        assert!(t.parse_sync_items(
-                r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","auth":true}}}]}"#.to_string(),
-            )
-            .is_err()); // `true` is invalid for `String`
-
-        assert!(t.parse_sync_items(
-                r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","auth":[]}}}]}"#.to_string(),
-            )
-            .is_err()); // `[]` is invalid for `String`
-
-        assert!(t.parse_sync_items(
-                r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","auth":{}}}}]}"#.to_string(),
-            )
-            .is_err()); // `{}` is invalid for `String`
-
-        assert!(t.parse_sync_items(
-                r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","grpid":null}}}]}"#.to_string(),
-            )
-            .is_err()); // missing field
-
-        assert!(t.parse_sync_items(
-                r#"{"items":[{"timestamp":1631781318,"data":{"AlterChat":{"id":{"ContactAddr":"bob@example.net"},"action":"Burn"}}}]}"#.to_string(),
-            )
-            .is_err()); // Unknown enum value
+        for bad_item_example in [
+            r#"{"items":[{"timestamp":1631781316,"data":{"BadItem":{"invitenumber":"in","auth":"a","grpid":null}}}]}"#,
+            r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","auth":123}}}]}"#, // `123` is invalid for `String`
+            r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","auth":true}}}]}"#, // `true` is invalid for `String`
+            r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","auth":[]}}}]}"#, // `[]` is invalid for `String`
+            r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","auth":{}}}}]}"#, // `{}` is invalid for `String`
+            r#"{"items":[{"timestamp":1631781316,"data":{"AddQrToken":{"invitenumber":"in","grpid":null}}}]}"#, // missing field
+            r#"{"items":[{"timestamp":1631781316,"data":{"AlterChat":{"id":{"ContactAddr":"bob@example.net"},"action":"Burn"}}}]}"#, // Unknown enum value
+        ] {
+            let sync_items = t.parse_sync_items(bad_item_example.to_string()).unwrap();
+            assert_eq!(sync_items.items.len(), 1);
+            assert!(matches!(sync_items.items[0].timestamp, 1631781316));
+            assert!(matches!(
+                sync_items.items[0].data,
+                SyncDataOrUnknown::Unknown(_)
+            ));
+        }
 
         // Test enums inside items and SystemTime
         let sync_items = t.parse_sync_items(
             r#"{"items":[{"timestamp":1631781318,"data":{"AlterChat":{"id":{"ContactAddr":"bob@example.net"},"action":{"SetMuted":{"Until":{"secs_since_epoch":42,"nanos_since_epoch":999000000}}}}}}]}"#.to_string(),
         )?;
         assert_eq!(sync_items.items.len(), 1);
-        let AlterChat { id, action } = &sync_items.items.get(0).unwrap().data else {
+        let SyncDataOrUnknown::SyncData(AlterChat { id, action }) =
+            &sync_items.items.get(0).unwrap().data
+        else {
             bail!("bad item");
         };
         assert_eq!(
@@ -466,7 +490,9 @@ mod tests {
         )?;
 
         assert_eq!(sync_items.items.len(), 1);
-        if let AddQrToken(token) = &sync_items.items.get(0).unwrap().data {
+        if let SyncDataOrUnknown::SyncData(AddQrToken(token)) =
+            &sync_items.items.get(0).unwrap().data
+        {
             assert_eq!(token.invitenumber, "in");
             assert_eq!(token.auth, "yip");
             assert_eq!(token.grpid, None);

src/test_utils.rs

@@ -580,7 +580,7 @@ impl TestContext {
         // MailinglistAddress is the lowest allowed origin, we'd prefer to not modify the
         // origin when creating this contact.
         let (contact_id, modified) =
-            Contact::add_or_lookup(self, &name, addr, Origin::MailinglistAddress)
+            Contact::add_or_lookup(self, &name, &addr, Origin::MailinglistAddress)
                 .await
                 .expect("add_or_lookup");
         match modified {
@@ -1067,6 +1067,13 @@ pub(crate) async fn mark_as_verified(this: &TestContext, other: &TestContext) {
     peerstate.save_to_db(&this.sql).await.unwrap();
 }
 
+/// Pops a sync message from alice0 and receives it on alice1. Should be used after an action on
+/// alice0's side that implies sending a sync message.
+pub(crate) async fn sync(alice0: &TestContext, alice1: &TestContext) {
+    let sync_msg = alice0.pop_sent_msg().await;
+    alice1.recv_msg(&sync_msg).await;
+}
+
 /// Pretty-print an event to stdout
 ///
 /// Done during tests this is captured by `cargo test` and associated with the test itself.

src/tests/verified_chats.rs

@@ -1,12 +1,11 @@
 use anyhow::Result;
 use pretty_assertions::assert_eq;
 
-use crate::chat::ProtectionStatus;
+use crate::chat::{self, Chat, ProtectionStatus};
 use crate::chatlist::Chatlist;
 use crate::config::Config;
-use crate::constants::DC_GCL_FOR_FORWARDING;
-use crate::contact::VerifiedStatus;
-use crate::contact::{Contact, Origin};
+use crate::constants::{Chattype, DC_GCL_FOR_FORWARDING};
+use crate::contact::{Contact, ContactId, Origin};
 use crate::message::{Message, Viewtype};
 use crate::mimefactory::MimeFactory;
 use crate::mimeparser::SystemMessage;
@@ -70,10 +69,7 @@ async fn check_verified_oneonone_chat(broken_by_classical_email: bool) {
     tcm.send_recv(&bob, &alice, "Using DC again").await;
 
     let contact = alice.add_or_lookup_contact(&bob).await;
-    assert_eq!(
-        contact.is_verified(&alice.ctx).await.unwrap(),
-        VerifiedStatus::BidirectVerified
-    );
+    assert_eq!(contact.is_verified(&alice.ctx).await.unwrap(), true);
 
     // Bob's chat is marked as verified again
     assert_verified(&alice, &bob, ProtectionStatus::Protected).await;
@@ -121,10 +117,7 @@ async fn test_create_verified_oneonone_chat() -> Result<()> {
 
     // Alice and Fiona should now be verified because of gossip
     let alice_fiona_contact = alice.add_or_lookup_contact(&fiona).await;
-    assert_eq!(
-        alice_fiona_contact.is_verified(&alice).await.unwrap(),
-        VerifiedStatus::BidirectVerified
-    );
+    assert!(alice_fiona_contact.is_verified(&alice).await.unwrap(),);
 
     // Alice should have a hidden protected chat with Fiona
     {
@@ -684,7 +677,7 @@ async fn test_break_protection_then_verify_again() -> Result<()> {
         // Bob sent a message with a new key, so he most likely doesn't have
         // the old key anymore. This means that Alice's device should show
         // him as unverified:
-        VerifiedStatus::Unverified
+        false
     );
     let chat = alice.get_chat(&bob_new).await;
     assert_eq!(chat.is_protected(), false);
@@ -782,14 +775,49 @@ async fn test_create_oneonone_chat_with_former_verified_contact() -> Result<()>
     Ok(())
 }
 
+/// Tests that on the second device of a protected group creator the first message is
+/// `SystemMessage::ChatProtectionEnabled` and the second one is the message populating the group.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_create_protected_grp_multidev() -> Result<()> {
+    let mut tcm = TestContextManager::new();
+    let alice = &tcm.alice().await;
+    let alice1 = &tcm.alice().await;
+
+    let group_id = alice
+        .create_group_with_members(ProtectionStatus::Protected, "Group", &[])
+        .await;
+    assert_eq!(
+        get_chat_msg(alice, group_id, 0, 1).await.get_info_type(),
+        SystemMessage::ChatProtectionEnabled
+    );
+
+    let sent = alice.send_text(group_id, "Hey").await;
+    // This sleep is necessary to reproduce the bug when the original message is sorted over the
+    // "protection enabled" message so that these messages have different timestamps. The better way
+    // would be to adjust the system time here if we could mock the system clock for the tests.
+    tokio::time::sleep(std::time::Duration::from_millis(2000)).await;
+    let msg = alice1.recv_msg(&sent).await;
+    let group1 = Chat::load_from_db(alice1, msg.chat_id).await?;
+    assert_eq!(group1.get_type(), Chattype::Group);
+    assert!(group1.is_protected());
+    assert_eq!(
+        chat::get_chat_contacts(alice1, group1.id).await?,
+        vec![ContactId::SELF]
+    );
+    assert_eq!(
+        get_chat_msg(alice1, group1.id, 0, 2).await.get_info_type(),
+        SystemMessage::ChatProtectionEnabled
+    );
+    assert_eq!(get_chat_msg(alice1, group1.id, 1, 2).await.id, msg.id);
+
+    Ok(())
+}
+
 // ============== Helper Functions ==============
 
 async fn assert_verified(this: &TestContext, other: &TestContext, protected: ProtectionStatus) {
     let contact = this.add_or_lookup_contact(other).await;
-    assert_eq!(
-        contact.is_verified(this).await.unwrap(),
-        VerifiedStatus::BidirectVerified
-    );
+    assert_eq!(contact.is_verified(this).await.unwrap(), true);
 
     let chat = this.get_chat(other).await;
     let (expect_protected, expect_broken) = match protected {

src/webxdc.rs

@@ -5,6 +5,7 @@
 //! - `id` - status update serial number
 //! - `msg_id` - ID of the message in the `msgs` table
 //! - `update_item` - JSON representation of the status update
+//! - `uid` - "id" field of the update, used for deduplication
 //!
 //! Status updates are scheduled for sending by adding a record
 //! to `smtp_status_updates_table` SQL table.
@@ -14,7 +15,6 @@
 //! - `last_serial` - serial number of the last status update to send
 //! - `descr` - text to send along with the updates
 
-use std::convert::TryFrom;
 use std::path::Path;
 
 use anyhow::{anyhow, bail, ensure, format_err, Context as _, Result};
@@ -37,6 +37,7 @@ use crate::mimefactory::wrapped_base64_encode;
 use crate::mimeparser::SystemMessage;
 use crate::param::Param;
 use crate::param::Params;
+use crate::tools::create_id;
 use crate::tools::strip_rtlo_characters;
 use crate::tools::{create_smeared_timestamp, get_abs_path};
 
@@ -62,11 +63,6 @@ const WEBXDC_DEFAULT_ICON: &str = "__webxdc__/default-icon.png";
 /// it is planned to raise that limit as needed in subsequent versions.
 const WEBXDC_SENDING_LIMIT: u64 = 655360;
 
-/// Be more tolerant for .xdc sizes on receiving -
-/// might be, the senders version uses already a larger limit
-/// and not showing the .xdc on some devices would be even worse ux.
-const WEBXDC_RECEIVING_LIMIT: u64 = 4194304;
-
 /// Raw information read from manifest.toml
 #[derive(Debug, Deserialize, Default)]
 #[non_exhaustive]
@@ -178,6 +174,13 @@ pub struct StatusUpdateItem {
     /// for a voting app.
     #[serde(skip_serializing_if = "Option::is_none")]
     pub summary: Option<String>,
+
+    /// Unique ID for deduplication.
+    /// This can be used if the message is sent over multiple transports.
+    ///
+    /// If there is no ID, message is always considered to be unique.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub uid: Option<String>,
 }
 
 /// Update items as passed to the UIs.
@@ -210,14 +213,6 @@ impl Context {
             return Ok(false);
         }
 
-        if file.len() as u64 > WEBXDC_RECEIVING_LIMIT {
-            info!(
-                self,
-                "{} exceeds receiving limit of {} bytes", &filename, WEBXDC_RECEIVING_LIMIT
-            );
-            return Ok(false);
-        }
-
         let archive = match async_zip::read::mem::ZipFileReader::new(file.to_vec()).await {
             Ok(archive) => archive,
             Err(_) => {
@@ -317,7 +312,14 @@ impl Context {
         timestamp: i64,
         can_info_msg: bool,
         from_id: ContactId,
-    ) -> Result<StatusUpdateSerial> {
+    ) -> Result<Option<StatusUpdateSerial>> {
+        let Some(status_update_serial) = self
+            .write_status_update_inner(&instance.id, &status_update_item)
+            .await?
+        else {
+            return Ok(None);
+        };
+
         if can_info_msg {
             if let Some(ref info) = status_update_item.info {
                 if let Some(info_msg_id) =
@@ -376,10 +378,6 @@ impl Context {
             self.emit_msgs_changed(instance.chat_id, instance.id);
         }
 
-        let status_update_serial = self
-            .write_status_update_inner(&instance.id, status_update_item)
-            .await?;
-
         if instance.viewtype == Viewtype::Webxdc {
             self.emit_event(EventType::WebxdcStatusUpdate {
                 msg_id: instance.id,
@@ -387,23 +385,43 @@ impl Context {
             });
         }
 
-        Ok(status_update_serial)
+        Ok(Some(status_update_serial))
     }
 
+    /// Inserts a status update item into `msgs_status_updates` table.
+    ///
+    /// Returns serial ID of the status update if a new item is inserted.
     pub(crate) async fn write_status_update_inner(
         &self,
         instance_id: &MsgId,
-        status_update_item: StatusUpdateItem,
-    ) -> Result<StatusUpdateSerial> {
-        let rowid = self
+        status_update_item: &StatusUpdateItem,
+    ) -> Result<Option<StatusUpdateSerial>> {
+        let _lock = self.sql.write_lock().await;
+        let uid = status_update_item.uid.as_deref();
+        let Some(rowid) = self
             .sql
-            .insert(
-                "INSERT INTO msgs_status_updates (msg_id, update_item) VALUES(?, ?);",
-                (instance_id, serde_json::to_string(&status_update_item)?),
+            .query_row_optional(
+                "INSERT INTO msgs_status_updates (msg_id, update_item, uid) VALUES(?, ?, ?)
+                 ON CONFLICT (uid) DO NOTHING
+                 RETURNING id",
+                (
+                    instance_id,
+                    serde_json::to_string(&status_update_item)?,
+                    uid,
+                ),
+                |row| {
+                    let id: u32 = row.get(0)?;
+                    Ok(id)
+                },
             )
-            .await?;
-        let status_update_serial = StatusUpdateSerial(u32::try_from(rowid)?);
-        Ok(status_update_serial)
+            .await?
+        else {
+            let uid = uid.unwrap_or("-");
+            info!(self, "Ignoring duplicate status update with uid={uid}");
+            return Ok(None);
+        };
+        let status_update_serial = StatusUpdateSerial(rowid);
+        Ok(Some(status_update_serial))
     }
 
     /// Returns the update_item with `status_update_serial` from the webxdc with message id `msg_id`.
@@ -432,13 +450,8 @@ impl Context {
         update_str: &str,
         descr: &str,
     ) -> Result<()> {
-        let status_update_item: StatusUpdateItem =
-            if let Ok(item) = serde_json::from_str::<StatusUpdateItem>(update_str) {
-                item
-            } else {
-                bail!("create_status_update_record: no valid update item.");
-            };
-
+        let status_update_item: StatusUpdateItem = serde_json::from_str(update_str)
+            .with_context(|| format!("Failed to parse webxdc update item from {update_str:?}"))?;
         self.send_webxdc_status_update_struct(instance_msg_id, status_update_item, descr)
             .await?;
         Ok(())
@@ -449,17 +462,27 @@ impl Context {
     pub async fn send_webxdc_status_update_struct(
         &self,
         instance_msg_id: MsgId,
-        status_update: StatusUpdateItem,
+        mut status_update: StatusUpdateItem,
         descr: &str,
     ) -> Result<()> {
-        let mut instance = Message::load_from_db(self, instance_msg_id).await?;
-        if instance.viewtype != Viewtype::Webxdc {
-            bail!("send_webxdc_status_update: is no webxdc message");
+        let mut instance = Message::load_from_db(self, instance_msg_id)
+            .await
+            .with_context(|| {
+                format!("Failed to load message {instance_msg_id} from the database")
+            })?;
+        let viewtype = instance.viewtype;
+        if viewtype != Viewtype::Webxdc {
+            bail!("send_webxdc_status_update: message {instance_msg_id} is not a webxdc message, but a {viewtype} message.");
         }
 
-        let chat = Chat::load_from_db(self, instance.chat_id).await?;
-        if let Some(reason) = chat.why_cant_send(self).await? {
-            bail!("cannot send to {}: {}", chat.id, reason);
+        let chat_id = instance.chat_id;
+        let chat = Chat::load_from_db(self, chat_id)
+            .await
+            .with_context(|| format!("Failed to load chat {chat_id} from the database"))?;
+        if let Some(reason) = chat.why_cant_send(self).await.with_context(|| {
+            format!("Failed to check if webxdc update can be sent to chat {chat_id}")
+        })? {
+            bail!("Cannot send to {chat_id}: {reason}.");
         }
 
         let send_now = !matches!(
@@ -467,6 +490,7 @@ impl Context {
             MessageState::Undefined | MessageState::OutPreparing | MessageState::OutDraft
         );
 
+        status_update.uid = Some(create_id());
         let status_update_serial: StatusUpdateSerial = self
             .create_status_update_record(
                 &mut instance,
@@ -475,7 +499,9 @@ impl Context {
                 send_now,
                 ContactId::SELF,
             )
-            .await?;
+            .await
+            .context("Failed to create status update")?
+            .context("Duplicate status update UID was generated")?;
 
         if send_now {
             self.sql.insert(
@@ -483,7 +509,7 @@ impl Context {
                  ON CONFLICT(msg_id)
                  DO UPDATE SET last_serial=excluded.last_serial, descr=excluded.descr",
                 (instance.id, status_update_serial, status_update_serial, descr),
-            ).await?;
+            ).await.context("Failed to insert webxdc update into SMTP queue")?;
             self.scheduler.interrupt_smtp().await;
         }
         Ok(())
@@ -655,7 +681,10 @@ impl Context {
                         let (update_item_str, serial) = row;
                         let update_item = StatusUpdateItemAndSerial
                         {
-                            item: serde_json::from_str(&update_item_str)?,
+                            item: StatusUpdateItem {
+                                uid: None, // Erase UIDs, apps, bots and tests don't need to know them.
+                                ..serde_json::from_str(&update_item_str)?
+                            },
                             serial,
                             max_serial,
                         };
@@ -863,8 +892,6 @@ mod tests {
     async fn test_webxdc_file_limits() -> Result<()> {
         assert!(WEBXDC_SENDING_LIMIT >= 32768);
         assert!(WEBXDC_SENDING_LIMIT < 16777216);
-        assert!(WEBXDC_RECEIVING_LIMIT >= WEBXDC_SENDING_LIMIT * 2);
-        assert!(WEBXDC_RECEIVING_LIMIT < 16777216);
         Ok(())
     }
 
@@ -1348,17 +1375,38 @@ mod tests {
                     info: None,
                     document: None,
                     summary: None,
+                    uid: Some("iecie2Ze".to_string()),
+                },
+                1640178619,
+                true,
+                ContactId::SELF,
+            )
+            .await?
+            .unwrap();
+        assert_eq!(
+            t.get_webxdc_status_updates(instance.id, StatusUpdateSerial(0))
+                .await?,
+            r#"[{"payload":{"foo":"bar"},"serial":1,"max_serial":1}]"#
+        );
+
+        // Update with duplicate update ID is received.
+        // Whatever the payload is, update should be ignored just because ID is duplicate.
+        let update_id1_duplicate = t
+            .create_status_update_record(
+                &mut instance,
+                StatusUpdateItem {
+                    payload: json!({"nothing": "this should be ignored"}),
+                    info: None,
+                    document: None,
+                    summary: None,
+                    uid: Some("iecie2Ze".to_string()),
                 },
                 1640178619,
                 true,
                 ContactId::SELF,
             )
             .await?;
-        assert_eq!(
-            t.get_webxdc_status_updates(instance.id, StatusUpdateSerial(0))
-                .await?,
-            r#"[{"payload":{"foo":"bar"},"serial":1,"max_serial":1}]"#
-        );
+        assert_eq!(update_id1_duplicate, None);
 
         assert!(t
             .send_webxdc_status_update(instance.id, "\n\n\n", "")
@@ -1384,15 +1432,17 @@ mod tests {
                     info: None,
                     document: None,
                     summary: None,
+                    uid: None,
                 },
                 1640178619,
                 true,
                 ContactId::SELF,
             )
-            .await?;
+            .await?
+            .unwrap();
         assert_eq!(
             t.get_webxdc_status_updates(instance.id, update_id1).await?,
-            r#"[{"payload":{"foo2":"bar2"},"serial":2,"max_serial":2}]"#
+            r#"[{"payload":{"foo2":"bar2"},"serial":3,"max_serial":3}]"#
         );
         t.create_status_update_record(
             &mut instance,
@@ -1401,6 +1451,7 @@ mod tests {
                 info: None,
                 document: None,
                 summary: None,
+                uid: None,
             },
             1640178619,
             true,
@@ -1410,9 +1461,9 @@ mod tests {
         assert_eq!(
             t.get_webxdc_status_updates(instance.id, StatusUpdateSerial(0))
                 .await?,
-            r#"[{"payload":{"foo":"bar"},"serial":1,"max_serial":3},
-{"payload":{"foo2":"bar2"},"serial":2,"max_serial":3},
-{"payload":true,"serial":3,"max_serial":3}]"#
+            r#"[{"payload":{"foo":"bar"},"serial":1,"max_serial":4},
+{"payload":{"foo2":"bar2"},"serial":3,"max_serial":4},
+{"payload":true,"serial":4,"max_serial":4}]"#
         );
 
         t.send_webxdc_status_update(
@@ -1423,8 +1474,8 @@ mod tests {
         .await?;
         assert_eq!(
             t.get_webxdc_status_updates(instance.id, update_id2).await?,
-            r#"[{"payload":true,"serial":3,"max_serial":4},
-{"payload":1,"serial":4,"max_serial":4}]"#
+            r#"[{"payload":true,"serial":4,"max_serial":5},
+{"payload":1,"serial":5,"max_serial":5}]"#
         );
 
         Ok(())
@@ -1468,7 +1519,7 @@ mod tests {
         t.receive_status_update(
             ContactId::SELF,
             instance.id,
-            r#"{"updates":[{"payload":{"foo":"bar"}}]}"#,
+            r#"{"updates":[{"payload":{"foo":"bar"}, "someTrash": "definitely TrAsH"}]}"#,
         )
         .await?;
         assert_eq!(
@@ -1654,6 +1705,8 @@ mod tests {
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_render_webxdc_status_update_object_range() -> Result<()> {
+        use regex::Regex;
+
         let t = TestContext::new_alice().await;
         let chat_id = create_group_chat(&t, ProtectionStatus::Unprotected, "a chat").await?;
         let instance = send_webxdc_instance(&t, chat_id).await?;
@@ -1672,7 +1725,13 @@ mod tests {
             )
             .await?
             .unwrap();
-        assert_eq!(json, "{\"updates\":[{\"payload\":2},\n{\"payload\":3}]}");
+        let json = Regex::new(r#""uid":"[^"]*""#)
+            .unwrap()
+            .replace_all(&json, "XXX");
+        assert_eq!(
+            json,
+            "{\"updates\":[{\"payload\":2,XXX},\n{\"payload\":3,XXX}]}"
+        );
 
         assert_eq!(
             t.sql

test-data/golden/chat_test_msg_with_implicit_member_add

@@ -3,6 +3,6 @@ Group#Chat#10: Group chat [3 member(s)]
 Msg#10:  (Contact#Contact#11): I created a group [FRESH]
 Msg#11:  (Contact#Contact#11): Member Fiona (fiona@example.net) added by alice@example.org. [FRESH][INFO]
 Msg#12: Me (Contact#Contact#Self): You removed member Fiona (fiona@example.net). [INFO] o
-Msg#13:  (Contact#Contact#11): Member Fiona (fiona@example.net) added. [FRESH]
+Msg#13: info (Contact#Contact#Info): Member Fiona (fiona@example.net) added. [NOTICED][INFO]
 Msg#14:  (Contact#Contact#11): Welcome, Fiona! [FRESH]
 --------------------------------------------------------------------------------

test-data/golden/chat_test_simultaneous_member_remove

@@ -2,6 +2,6 @@ Group#Chat#10: Group chat [4 member(s)]
 --------------------------------------------------------------------------------
 Msg#10:  (Contact#Contact#10): Hi! I created a group. [FRESH]
 Msg#11: Me (Contact#Contact#Self): You left the group. [INFO] o
-Msg#12:  (Contact#Contact#10): Member Me (bob@example.net) added. [FRESH][INFO]
+Msg#12: info (Contact#Contact#Info): Member Me (bob@example.net) added. [NOTICED][INFO]
 Msg#13:  (Contact#Contact#10): Member claire@example.net added by alice@example.org. [FRESH][INFO]
 --------------------------------------------------------------------------------

test-data/message/thunderbird_encrypted_unsigned_with_unencrypted_subject.eml

@@ -0,0 +1,91 @@
+From - Sun, 19 Nov 2023 01:08:24 GMT
+X-Mozilla-Status: 0800
+X-Mozilla-Status2: 00000000
+Message-ID: <38a2a29b-8261-403b-abb5-56b0a87d2ff4@example.org>
+Date: Sat, 18 Nov 2023 22:08:23 -0300
+MIME-Version: 1.0
+User-Agent: Mozilla Thunderbird
+Content-Language: en-US
+To: bob@example.net
+From: Alice <alice@example.org>
+Autocrypt: addr=alice@example.org; keydata=
+ xsDNBGOaGzQBDADCFtBNMHRDJQRkd2tNm7CJm1Yo3Y5r3qP6v0FSwP1BIHbiIf0E/jFiKZWj
+ 1uL68J2mGUuUu+Qi4ovf1l9/QQYzg/DCaLZxlbc0LKu2LXcpUL5DPu37mdw+DKs0YvNIlc+A
+ RjyFUwd3rsZN3k58inf1mYzKuKU6NpbdXULbOEYwnVEwzQsrtS2JgJ+tLSYUvNJeMJXm/cDL
+ XKJSApAyvVVdxxteG8uWcDqWV/HcXuopXLILf3yJF0De11/7G62dHNHuhmtgRLsTN4Q372Q9
+ KNdYEFLHaN91jEzyD/+aHNskATxtcGhppI8OQsU3NzNgHyd8Smzx5oTyZ/6NdhYoh0pKB8yf
+ VAyA69t5fctQRb4+bTwL+sS9KDobQOvcyOMUSccDfUhsWMghwsMCwU4Sz9hIY6dCAfroDAiL
+ vYUfdNJstAqvLf04mZtMmkI7Q2BYLETEgu4KQzQHRQekmOE/3EaSiojNa4ZTVURMdJ9U+I3E
+ q8e6TbOY7Xa4V8krAt/F2wMAEQEAAc0ZQWxpY2UgPGFsaWNlQGV4YW1wbGUub3JnPsLBDQQT
+ AQgANxYhBBSrP2X8J0u9tfp2jCXwByRZ5HriBQJjmhs1BQm7+B4AAhsDBAsJCAcFFQgJCgsF
+ FgIDAQAACgkQJfAHJFnkeuKQ2wwAgDgiCI6bz9PjqE1GoDcy/xQdy+nnYq5pOuHGUndZ7jYK
+ cOqM8LDEaG7GgrFsbs9vGhTA1fyqncM41pB7SmwQ7zBVaMdtHoulEG4RPGVboDaY9tuMOL3/
+ GVxFbovVHyU5Lr1euryNh/0JvMITY0UHaEY1k1M7izYUMyFu8I1ODZ9Iws2trUyU3Omw/sTJ
+ x15zzCsK8Aq+r3JmB+Q33SFSgWr/YWH0dQVIQ0I5iLN2q14oucmLBaKc9EXdRLiu8S8lLSQl
+ nfISJ17GBLmH1YxmPPZ3CRHC6iEKCLR6G9wzhsTPNdK7dRCYR5wTI27RVPLBcSnCKAeTopAJ
+ YskyNndtv0iaNRT7YLOfhrsBAofSjuLegR04CNiqBHtYQ3LO3WKhJ7riRcQ/Ksv0wYkmj1gJ
+ 8myMwA+ybfYrpNqO4devnCvE3Eo5gzeYbvYU2Z17n9y6HAOG9/Tm/daiGEP2ni6iwV0kqLzw
+ eC48R1D75T66PxX/jQooujrTph8+K3ckV/q+zsDNBGOaGzUBDADV+DGgKxvCpfVFuPGrSdRU
+ 06dxowdKOKavO6WGMvN3g/+CFrIsjUFy4S0Soo5ARnLh23i49ZSjacXFpgtZUNV3iGOSOcSE
+ LldLtZk5BV9w/ATqqgu4/LVdNA9rm+o197bIeSQCRTnY/QV6FdKYxVd4NBVH9abZ7t8Tm4qC
+ urZj56MjPCg3fqT+Q6sjxH+nKBrs8s8iCJkYhGBgU3q5W+wrtZ56kI9mxJec62KHpyLZ0rTE
+ xEAeVbChUJOo11vUtJfTrDhI6lhqyr72o/A6bY1OV7WzkxtiBRl35eewQ+RDLJ4yxaNj/XTS
+ UxOz60xNggEfDVtfgfjBZrBbiHXqf8iKVV1ZPGm0ycvXZGYFw2zXLI2PwevhQCm+t4Ywty1h
+ 8l019MYmGadpQgbuA4ZippuzOSzSGMQ+S4uYEzeeymR9ksxVSXn90HEzqC7LdHCcd2IO6rfu
+ g2fuRf258Adfuoh3s8YUlWyXjEHLXKo9SRgGMfGs7qgCOL/ReAwFPtKACvEAEQEAAcLA/AQY
+ AQgAJhYhBBSrP2X8J0u9tfp2jCXwByRZ5HriBQJjmhs2BQm7+B4AAhsMAAoJECXwByRZ5Hri
+ EOkMALtq4DVYX8RfoPdU0Dt6y+yDj1NALv5GefvHbgfuaVT8PaOP0gxCjWrnUDvvJEwP1W3j
+ UXYqDwKP42hiGWsnXk2hbgXbplArgP3H987x7c8bu1wIAmkJ9eLjEM++rbOD4vWbYXRwaDiH
+ LetFJ5tGHDAIfL48NYpz2o3XZ3/O7WdTZphsAcvgPxTC+zU7WkbUl2SQlj0/qwsoD+qe9RYT
+ XhVXR7q7sjcGB4TpeqzRT7YKVLoVNq+bQw2lUX4W561gAYbZvVo/XByfDCoxmkxwuMlSmajj
+ Wy7b9TuT38t1HArv4m/LyVuBHiikX0/MUNBeSSIiKDvTL6NdHTjnZM6ptZvdvW3+ou6ET0pK
+ MGDpk/1NVuMnIHJESRg/SSFV6sElgq38k9wAT2oUqLcYvYI07nHmnuciaGygkCcGt+l2PvAa
+ j4mkQQvMU0cNRDBybk5aKi820oGIJjT7e+5RnD2mYZQdOAbQhDVCHvrfS1I60bsHT1MHqyAa
+ /qMLjKwBpKEd/w==
+X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0;
+ attachmentreminder=0; deliveryformat=0
+X-Identity-Key: id3
+Fcc: imap://alice%40example.org@in.example.org/Sent
+Subject: Hello!
+Content-Type: multipart/encrypted;
+ protocol="application/pgp-encrypted";
+ boundary="------------HVIlnYsnz3YPWPVlor40isrE"
+
+This is an OpenPGP/MIME encrypted message (RFC 4880 and 3156)
+--------------HVIlnYsnz3YPWPVlor40isrE
+Content-Type: application/pgp-encrypted
+Content-Description: PGP/MIME version identification
+
+Version: 1
+
+--------------HVIlnYsnz3YPWPVlor40isrE
+Content-Type: application/octet-stream; name="encrypted.asc"
+Content-Description: OpenPGP encrypted message
+Content-Disposition: inline; filename="encrypted.asc"
+
+-----BEGIN PGP MESSAGE-----
+
+wcDMA/K57StIWPW6AQv/cTRuyZRuX+Qkmxcbo34Df32PL2O3LSuwChdd4pd+WRNU5r08E0eLeFGv
+k/C/2iFrNhgmvsJEnDtTZVc7+qykMeXwwC1f2qT7OAvTbAF9Me1dBCQy8QAPSzOowNu8434qOPwz
+fZkoB4pdkKlAhGMUBdtbWYg3EDdpBUCpAuT8l3qhlfdSTRh+da2UVh9kix3SXSmLarMZGlJikh3T
+pRLAlPOtRa7k5R4rc/OCD9HUL9EHGUk4rV86+mr4Wpp1aIhdTNksfrFXvgRFx0dcU33uS74X/zme
+c1AOuJK8ed3F2Zc5IYTyE+ps1jBqVH1+CM697d5FnIMx96VyB4txDBBBjyq+oh+SY+xUxvtR/EzN
+mR0qfiSCwOKwGdzPWDkBoCePYnykiR4shVdVV3qjYZwBmY5IWNahyBtIB470V4wMiJXdZFMQu5XD
+NO4DJ6/x2UCcUiUokAwvKTSHflAHocZJ2ICfzhlJjJSiqEttwyG0xU0ZhTdZIbwzRWzYzDgMakFO
+wcBMA+PY3JvEjuMiAQf+MXJB44iQ/Ti9WDH3MOX0N+X0APf5mzeqU9MpSZ+7mAnTRVmz/iMD989J
+ngxu1mv3vQBjdNokIAfBYk29qyRBULXaof+6x5VJSWoopV6t1vNd+DB2HgLkbdCJuzikapCE+QAm
+gmgoknQap+3l4D1RkMys7w1awsMYK0wR1Iucjb8M2I4f5ObPSMS5211ZBJuYOf1OQt0jX1jCNTOi
+Q5tbufJ6EjAvP6XOYTY3in8+p7yocBgQhXaK8NB8jdg+h6IE/NNX1W5v1tMx17WRtIQGwh4cOlB6
+Fsu87eMx8x9Ew2YdpN6yvIHddy9M9k3NCROT+5rIq7+1GTy0WoI/KxNwtdLBEwE2h7VFmJ5qEWhi
+lmunWBIlA71IdNqpi+9lbg/QPwCgvRow05Gv0FyKbSvDA/fN18+CLuz8RicNFRbiPgwxMLRE3lZB
+jj7BEDa53fjnjBJG4lA9mUaB/ALScQwkGXqKPpeDN4Iexy8eBsZoBczJcPna7GNgSYiXxbWo2P8+
+T71oSht+igsyi8gYDNwmhcsQxSWtF7f53irOKWgEpgz2hMjWi764GGBzAJ231suyV2eZXVJm2Im7
+RUDu3bJNu/Q/CvxQQdcUXHEoHrTzDZ3KQPW0/oYln2WZwkRQcVVGSLVINgVTdTkI8GEuXH/4yjYF
+tUq68esMQ2WOWZ7IiTNPU9T2F5kEwR2sf55XgLckj0OLJn5CkSAWBJhHdL/9er06u8ksTs6V9UEo
+i+15XUAiNzsuIcydoGF9sSuO9nm62xYO8uY8yQl/z2Q7CXKefkXBSeaNcYyQKrCnLfwnuvR2WFE5
+OJ0HM5i/qcWmqWSvSYzdDqUbI95q9Fgn78F/bcJ9ZTdPDkWLi//SUzcWeUWgRC57U4CFTApj74EJ
+lT/gVlH/y88RNcvGGAPHHS11adLmln7CbLEF9tmRM3ou78nMm7VhXT8C/QZiTvJgG3yZbeRFlC39
+=hdiO
+-----END PGP MESSAGE-----
+
+--------------HVIlnYsnz3YPWPVlor40isrE--