No need to set the origin anymore

This test keeps failing on macOS CI,
capturing events like `DC_EVENT_ACCOUNTS_ITEM_CHANGED`
before FailPlugin is setup.
These CI runners likely get less resources
because there is a limited number of them,
and this triggers this race condition.

Race is fixed by setting up fail plugin
before starting to capture events.

CHANGELOG.md

@@ -1,5 +1,156 @@
 # Changelog
 
+## [1.153.0] - 2025-01-05
+
+### Features / Changes
+
+- Remove "jobs" from imap_markseen if folder doesn't exist ([#5870](https://github.com/deltachat/deltachat-core-rust/pull/5870)).
+- Delete `vg-request-with-auth` from IMAP after processing ([#6208](https://github.com/deltachat/deltachat-core-rust/pull/6208)).
+
+### API-Changes
+
+- Add `IncomingWebxdcNotify.chat_id` ([#6356](https://github.com/deltachat/deltachat-core-rust/pull/6356)).
+- rpc-client: Add INCOMING_REACTION to const.EventType ([#6349](https://github.com/deltachat/deltachat-core-rust/pull/6349)).
+
+### Documentation
+
+- Viewtype::Sticker may be changed to Image and how to disable that ([#6352](https://github.com/deltachat/deltachat-core-rust/pull/6352)).
+
+### Fixes
+
+- Never change Viewtype::Sticker to Image if file has non-image extension ([#6352](https://github.com/deltachat/deltachat-core-rust/pull/6352)).
+- Change BccSelf default to 0 for chatmail ([#6340](https://github.com/deltachat/deltachat-core-rust/pull/6340)).
+- Mark holiday notice messages as bot-generated.
+- Don't treat location-only and sync messages as bot ones ([#6357](https://github.com/deltachat/deltachat-core-rust/pull/6357)).
+- Update shadowsocks crate to 1.22.0 to avoid panic when parsing some QR codes.
+- Prefer to encrypt if E2eeEnabled even if peers have EncryptPreference::NoPreference.
+- Prioritize mailing list over self-sent messages.
+- Allow empty `To` field for self-sent messages.
+- Default `to_id` to self instead of 0.
+
+### Refactor
+
+- Remove unused parameter and return value from `build_body_file(…)` ([#6369](https://github.com/deltachat/deltachat-core-rust/pull/6369)).
+- Deprecate Param::ErroneousE2ee.
+- Add `emit_msgs_changed_without_msg_id`.
+- Add_parts: Remove excessive `is_mdn` checks.
+- Simplify `self_sent` condition.
+- Don't ignore get_for_contact errors.
+
+### Tests
+
+- Messages without recipients are assigned to self chat.
+- Message with empty To: field should have a valid to_id.
+- Fix `test_logged_ac_process_ffi_failure` flakiness.
+
+## [1.152.2] - 2024-12-24
+
+### Features / Changes
+
+- Emit ImexProgress(1) after receiving backup size.
+- `delete_msgs`: Use `transaction()` instead of `call_write()`.
+- Start ephemeral timers when the chat is noticed.
+- Start ephemeral timers when the chat is archived.
+- Revalidate HTTP cache entries once per minute maximum.
+
+### Fixes
+
+- Reduce number of `repeat_vars()` calls.
+- `sanitise_name`: Don't consider punctuation and control chars as part of file extension ([#6362](https://github.com/deltachat/deltachat-core-rust/pull/6362)).
+
+### Refactor
+
+- Remove marknoticed_chat_if_older_than().
+
+### Miscellaneous Tasks
+
+- Remove contrib/ directory.
+
+## [1.152.1] - 2024-12-17
+
+### Build system
+
+- Downgrade Rust version used to build binaries.
+- Reduce MSRV to 1.77.0.
+
+## [1.152.0] - 2024-12-12
+
+### API-Changes
+
+- [**breaking**] Remove `dc_prepare_msg` and `dc_msg_is_increation`.
+
+### Build system
+
+- Increase MSRV to 1.81.0.
+
+### Features / Changes
+
+- Cache HTTP GET requests.
+- Prefix server-url in info.
+- Set `mime_modified` for the last message part, not the first ([#4462](https://github.com/deltachat/deltachat-core-rust/pull/4462)).
+
+### Fixes
+
+- Render "message" parts in multipart messages' HTML ([#4462](https://github.com/deltachat/deltachat-core-rust/pull/4462)).
+- Ignore garbage at the end of the keys.
+
+## [1.151.6] - 2024-12-11
+
+### Features / Changes
+
+- Don't add "Failed to send message to ..." info messages to group chats.
+- Add info messages about implicit membership changes if group member list is recreated ([#6314](https://github.com/deltachat/deltachat-core-rust/pull/6314)).
+
+### Fixes
+
+- Add self-addition message to chat when recreating member list.
+- Do not subscribe to heartbeat if already subscribed via metadata.
+
+### Build system
+
+- Add idna 0.5.0 exception into deny.toml.
+
+### Documentation
+
+- Update links to Node.js bindings in the README.
+
+### Refactor
+
+- Factor out `wait_for_all_work_done()`.
+
+### Tests
+
+- Notifiy more prominently & in more tests about false positives when running `cargo test` ([#6308](https://github.com/deltachat/deltachat-core-rust/pull/6308)).
+
+## [1.151.5] - 2024-12-05
+
+### API-Changes
+
+- [**breaking**] Remove dc_all_work_done().
+
+### Security
+
+- cargo: Update rPGP to 0.14.2.
+
+  This fixes [Panics on Malformed Untrusted Input](https://github.com/rpgp/rpgp/security/advisories/GHSA-9rmp-2568-59rv)
+  and [Potential Resource Exhaustion when handling Untrusted Messages](https://github.com/rpgp/rpgp/security/advisories/GHSA-4grw-m28r-q285).
+  This allows the attacker to crash the application via specially crafted messages and keys.
+  We recommend all users and bot operators to upgrade to the latest version.
+  There is no impact on the confidentiality of the messages and keys so no action other than upgrading is needed.
+
+### Fixes
+
+- Store plaintext in mime_headers of truncated sent messages ([#6273](https://github.com/deltachat/deltachat-core-rust/pull/6273)).
+
+### Documentation
+
+- Document `push` module.
+- Remove mention of non-existent `nightly` feature.
+
+### Tests
+
+- Fix panic in `receive_emails` benchmark ([#6306](https://github.com/deltachat/deltachat-core-rust/pull/6306)).
+
 ## [1.151.4] - 2024-12-03
 
 ### Features / Changes
@@ -5428,3 +5579,9 @@ https://github.com/deltachat/deltachat-core-rust/pulls?q=is%3Apr+is%3Aclosed
 [1.151.2]: https://github.com/deltachat/deltachat-core-rust/compare/v1.151.1..v1.151.2
 [1.151.3]: https://github.com/deltachat/deltachat-core-rust/compare/v1.151.2..v1.151.3
 [1.151.4]: https://github.com/deltachat/deltachat-core-rust/compare/v1.151.3..v1.151.4
+[1.151.5]: https://github.com/deltachat/deltachat-core-rust/compare/v1.151.4..v1.151.5
+[1.151.6]: https://github.com/deltachat/deltachat-core-rust/compare/v1.151.5..v1.151.6
+[1.152.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.151.6..v1.152.0
+[1.152.1]: https://github.com/deltachat/deltachat-core-rust/compare/v1.152.0..v1.152.1
+[1.152.2]: https://github.com/deltachat/deltachat-core-rust/compare/v1.152.1..v1.152.2
+[1.153.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.152.2..v1.153.0

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat"
-version = "1.151.4"
+version = "1.153.0"
 edition = "2021"
 license = "MPL-2.0"
 rust-version = "1.77"
@@ -39,7 +39,7 @@ format-flowed = { path = "./format-flowed" }
 ratelimit = { path = "./deltachat-ratelimit" }
 
 anyhow = { workspace = true }
-async-broadcast = "0.7.1"
+async-broadcast = "0.7.2"
 async-channel = { workspace = true }
 async-imap = { version = "0.10.2", default-features = false, features = ["runtime-tokio", "compress"] }
 async-native-tls = { version = "0.5", default-features = false, features = ["runtime-tokio"] }
@@ -52,7 +52,7 @@ chrono = { workspace = true, features = ["alloc", "clock", "std"] }
 email = { git = "https://github.com/deltachat/rust-email", branch = "master" }
 encoded-words = { git = "https://github.com/async-email/encoded-words", branch = "master" }
 escaper = "0.1"
-fast-socks5 = "0.9"
+fast-socks5 = "0.10"
 fd-lock = "4"
 futures-lite = { workspace = true }
 futures = { workspace = true }
@@ -76,7 +76,7 @@ num-traits = { workspace = true }
 once_cell = { workspace = true }
 parking_lot = "0.12"
 percent-encoding = "2.3"
-pgp = { version = "0.14.0", default-features = false }
+pgp = { version = "0.14.2", default-features = false }
 pin-project = "1"
 qrcodegen = "1.7.0"
 quick-xml = "0.37"
@@ -85,15 +85,15 @@ rand = { workspace = true }
 regex = { workspace = true }
 rusqlite = { workspace = true, features = ["sqlcipher"] }
 rust-hsluv = "0.1"
-rustls-pki-types = "1.10.0"
-rustls = { version = "0.23.19", default-features = false }
+rustls-pki-types = "1.10.1"
+rustls = { version = "0.23.20", default-features = false }
 sanitize-filename = { workspace = true }
 serde_json = { workspace = true }
 serde_urlencoded = "0.7.1"
 serde = { workspace = true, features = ["derive"] }
 sha-1 = "0.10"
 sha2 = "0.10"
-shadowsocks = { version = "1.21.0", default-features = false, features = ["aead-cipher-2022"] }
+shadowsocks = { version = "1.22.0", default-features = false, features = ["aead-cipher", "aead-cipher-2022"] }
 smallvec = "1.13.2"
 strum = "0.26"
 strum_macros = "0.26"
@@ -101,8 +101,8 @@ tagger = "4.3.4"
 textwrap = "0.16.1"
 thiserror = { workspace = true }
 tokio-io-timeout = "1.2.0"
-tokio-rustls = { version = "0.26.0", default-features = false }
-tokio-stream = { version = "0.1.16", features = ["fs"] }
+tokio-rustls = { version = "0.26.1", default-features = false }
+tokio-stream = { version = "0.1.17", features = ["fs"] }
 tokio-tar = { version = "0.3" } # TODO: integrate tokio into async-tar
 tokio-util = { workspace = true }
 tokio = { workspace = true, features = ["fs", "rt-multi-thread", "macros"] }
@@ -120,7 +120,7 @@ nu-ansi-term = { workspace = true }
 pretty_assertions = "1.4.1"
 proptest = { version = "1", default-features = false, features = ["std"] }
 tempfile = { workspace = true }
-testdir = "0.9.0"
+testdir = "0.9.3"
 tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
 
 [workspace]
@@ -169,7 +169,7 @@ harness = false
 anyhow = "1"
 async-channel = "2.3.1"
 base64 = "0.22"
-chrono = { version = "0.4.38", default-features = false }
+chrono = { version = "0.4.39", default-features = false }
 deltachat-contact-tools = { path = "deltachat-contact-tools" }
 deltachat-jsonrpc = { path = "deltachat-jsonrpc", default-features = false }
 deltachat = { path = ".", default-features = false }
@@ -189,7 +189,7 @@ serde_json = "1"
 tempfile = "3.14.0"
 thiserror = "1"
 tokio = "1"
-tokio-util = "0.7.11"
+tokio-util = "0.7.13"
 tracing-subscriber = "0.3"
 yerpc = "0.6.2"
 

README.md

@@ -161,7 +161,6 @@ $ cargo bolero test fuzz_format_flowed --release=false -e afl -s NONE
 ## Features
 
 - `vendored`: When using Openssl for TLS, this bundles a vendored version.
-- `nightly`: Enable nightly only performance and security related features.
 
 ## Update Provider Data
 
@@ -178,8 +177,8 @@ Language bindings are available for:
 
 - **C** \[[📂 source](./deltachat-ffi) | [📚 docs](https://c.delta.chat)\]
 - **Node.js** 
-  - over cffi: \[[📂 source](./node) | [📦 npm](https://www.npmjs.com/package/deltachat-node) | [📚 docs](https://js.delta.chat)\]
-  - over jsonrpc built with napi.rs (experimental): \[[📂 source](https://github.com/deltachat/napi-jsonrpc) | [📦 npm](https://www.npmjs.com/package/@deltachat/napi-jsonrpc)\]
+  - over JSON-RPC: \[[📂 source](./deltachat-rpc-client) | [📦 npm](https://www.npmjs.com/package/@deltachat/jsonrpc-client) | [📚 docs](https://js.jsonrpc.delta.chat/)\]
+  - over CFFI[^1]: \[[📂 source](./node) | [📦 npm](https://www.npmjs.com/package/deltachat-node) | [📚 docs](https://js.delta.chat/)\]
 - **Python** \[[📂 source](./python) | [📦 pypi](https://pypi.org/project/deltachat) | [📚 docs](https://py.delta.chat)\]
 - **Go**
   - over jsonrpc: \[[📂 source](https://github.com/deltachat/deltachat-rpc-client-go/)\]

benches/receive_emails.rs

@@ -12,18 +12,18 @@ use deltachat::{
 };
 use tempfile::tempdir;
 
-async fn recv_all_emails(context: Context) -> Context {
+async fn recv_all_emails(context: Context, iteration: u32) -> Context {
     for i in 0..100 {
         let imf_raw = format!(
             "Subject: Benchmark
-Message-ID: Mr.OssSYnOFkhR.{i}@testrun.org
+Message-ID: Mr.{iteration}.{i}@testrun.org
 Date: Sat, 07 Dec 2019 19:00:27 +0000
 To: alice@example.com
 From: sender@testrun.org
 Chat-Version: 1.0
 Chat-Disposition-Notification-To: sender@testrun.org
 Chat-User-Avatar: 0
-In-Reply-To: Mr.OssSYnOFkhR.{i_dec}@testrun.org
+In-Reply-To: Mr.{iteration}.{i_dec}@testrun.org
 MIME-Version: 1.0
 
 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
@@ -41,11 +41,11 @@ Hello {i}",
 
 /// Receive 100 emails that remove charlie@example.com and add
 /// him back
-async fn recv_groupmembership_emails(context: Context) -> Context {
+async fn recv_groupmembership_emails(context: Context, iteration: u32) -> Context {
     for i in 0..50 {
         let imf_raw = format!(
             "Subject: Benchmark
-Message-ID: Gr.OssSYnOFkhR.{i}@testrun.org
+Message-ID: Gr.{iteration}.ADD.{i}@testrun.org
 Date: Sat, 07 Dec 2019 19:00:27 +0000
 To: alice@example.com, b@example.com, c@example.com, d@example.com, e@example.com, f@example.com
 From: sender@testrun.org
@@ -53,13 +53,12 @@ Chat-Version: 1.0
 Chat-Disposition-Notification-To: sender@testrun.org
 Chat-User-Avatar: 0
 Chat-Group-Member-Added: charlie@example.com
-In-Reply-To: Gr.OssSYnOFkhR.{i_dec}@testrun.org
+In-Reply-To: Gr.{iteration}.REMOVE.{i_dec}@testrun.org
 MIME-Version: 1.0
 
 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
 
 Hello {i}",
-            i = i,
             i_dec = i - 1,
         );
         receive_imf(&context, black_box(imf_raw.as_bytes()), false)
@@ -68,7 +67,7 @@ Hello {i}",
 
         let imf_raw = format!(
             "Subject: Benchmark
-Message-ID: Gr.OssSYnOFkhR.{i}@testrun.org
+Message-ID: Gr.{iteration}.REMOVE.{i}@testrun.org
 Date: Sat, 07 Dec 2019 19:00:27 +0000
 To: alice@example.com, b@example.com, c@example.com, d@example.com, e@example.com, f@example.com
 From: sender@testrun.org
@@ -76,14 +75,12 @@ Chat-Version: 1.0
 Chat-Disposition-Notification-To: sender@testrun.org
 Chat-User-Avatar: 0
 Chat-Group-Member-Removed: charlie@example.com
-In-Reply-To: Gr.OssSYnOFkhR.{i_dec}@testrun.org
+In-Reply-To: Gr.{iteration}.ADD.{i}@testrun.org
 MIME-Version: 1.0
 
 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
 
-Hello {i}",
-            i = i,
-            i_dec = i - 1,
+Hello {i}"
         );
         receive_imf(&context, black_box(imf_raw.as_bytes()), false)
             .await
@@ -129,11 +126,13 @@ fn criterion_benchmark(c: &mut Criterion) {
     group.bench_function("Receive 100 simple text msgs", |b| {
         let rt = tokio::runtime::Runtime::new().unwrap();
         let context = rt.block_on(create_context());
+        let mut i = 0;
 
         b.to_async(&rt).iter(|| {
             let ctx = context.clone();
+            i += 1;
             async move {
-                recv_all_emails(black_box(ctx)).await;
+                recv_all_emails(black_box(ctx), i).await;
             }
         });
     });
@@ -142,11 +141,13 @@ fn criterion_benchmark(c: &mut Criterion) {
         |b| {
             let rt = tokio::runtime::Runtime::new().unwrap();
             let context = rt.block_on(create_context());
+            let mut i = 0;
 
             b.to_async(&rt).iter(|| {
                 let ctx = context.clone();
+                i += 1;
                 async move {
-                    recv_groupmembership_emails(black_box(ctx)).await;
+                    recv_groupmembership_emails(black_box(ctx), i).await;
                 }
             });
         },

contrib/proxy.py

@@ -1,80 +0,0 @@
-#!/usr/bin/env python3
-# Examples:
-#
-# Original server that doesn't use SSL:
-# ./proxy.py 8080 imap.nauta.cu 143
-# ./proxy.py 8081 smtp.nauta.cu 25
-#
-# Original server that uses SSL:
-# ./proxy.py 8080 testrun.org 993 --ssl
-# ./proxy.py 8081 testrun.org 465 --ssl
-
-from datetime import datetime
-import argparse
-import selectors
-import ssl
-import socket
-import socketserver
-
-
-class Proxy(socketserver.ThreadingTCPServer):
-    allow_reuse_address = True
-
-    def __init__(self, proxy_host, proxy_port, real_host, real_port, use_ssl):
-        self.real_host = real_host
-        self.real_port = real_port
-        self.use_ssl = use_ssl
-        super().__init__((proxy_host, proxy_port), RequestHandler)
-
-
-class RequestHandler(socketserver.BaseRequestHandler):
-
-    def handle(self):
-        print('{} - {} CONNECTED.'.format(datetime.now(), self.client_address))
-
-        total = 0
-        real_server = (self.server.real_host, self.server.real_port)
-        with socket.create_connection(real_server) as sock:
-            if self.server.use_ssl:
-                context = ssl.create_default_context()
-                sock = context.wrap_socket(
-                    sock, server_hostname=real_server[0])
-
-            forward = {self.request: sock, sock: self.request}
-
-            sel = selectors.DefaultSelector()
-            sel.register(self.request, selectors.EVENT_READ,
-                         self.client_address)
-            sel.register(sock, selectors.EVENT_READ, real_server)
-
-            active = True
-            while active:
-                events = sel.select()
-                for key, mask in events:
-                    print('\n{} - {} wrote:'.format(datetime.now(), key.data))
-                    data = key.fileobj.recv(1024)
-                    received = len(data)
-                    total += received
-                    print(data)
-                    print('{} Bytes\nTotal: {} Bytes'.format(received, total))
-                    if data:
-                        forward[key.fileobj].sendall(data)
-                    else:
-                        print('\nCLOSING CONNECTION.\n\n')
-                        forward[key.fileobj].close()
-                        key.fileobj.close()
-                        active = False
-
-
-if __name__ == '__main__':
-    p = argparse.ArgumentParser(description='Simple Python Proxy')
-    p.add_argument(
-        "proxy_port", help="the port where the proxy will listen", type=int)
-    p.add_argument('host', help="the real host")
-    p.add_argument('port', help="the port of the real host", type=int)
-    p.add_argument("--ssl", help="use ssl to connect to the real host",
-                   action="store_true")
-    args = p.parse_args()
-
-    with Proxy('', args.proxy_port, args.host, args.port, args.ssl) as proxy:
-        proxy.serve_forever()

deltachat-ffi/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat_ffi"
-version = "1.151.4"
+version = "1.153.0"
 description = "Deltachat FFI"
 edition = "2018"
 readme = "README.md"

deltachat-ffi/deltachat.h

@@ -722,12 +722,6 @@ char*           dc_get_connectivity_html     (dc_context_t* context);
 int              dc_get_push_state           (dc_context_t* context);
 
 
-/**
- * Only used by the python tests.
- */
-int             dc_all_work_done             (dc_context_t* context);
-
-
 // connect
 
 /**
@@ -969,54 +963,6 @@ uint32_t        dc_create_chat_by_contact_id (dc_context_t* context, uint32_t co
 uint32_t        dc_get_chat_id_by_contact_id (dc_context_t* context, uint32_t contact_id);
 
 
-/**
- * Prepare a message for sending.
- *
- * Call this function if the file to be sent is still in creation.
- * Once you're done with creating the file, call dc_send_msg() as usual
- * and the message will really be sent.
- *
- * This is useful as the user can already send the next messages while
- * e.g. the recoding of a video is not yet finished. Or the user can even forward
- * the message with the file being still in creation to other groups.
- *
- * Files being sent with the increation-method must be placed in the
- * blob directory, see dc_get_blobdir().
- * If the increation-method is not used - which is probably the normal case -
- * dc_send_msg() copies the file to the blob directory if it is not yet there.
- * To distinguish the two cases, msg->state must be set properly. The easiest
- * way to ensure this is to reuse the same object for both calls.
- *
- * Example:
- * ~~~
- * char* blobdir = dc_get_blobdir(context);
- * char* file_to_send = mprintf("%s/%s", blobdir, "send.mp4")
- *
- * dc_msg_t* msg = dc_msg_new(context, DC_MSG_VIDEO);
- * dc_msg_set_file(msg, file_to_send, NULL);
- * dc_prepare_msg(context, chat_id, msg);
- *
- * // ... create the file ...
- *
- * dc_send_msg(context, chat_id, msg);
- *
- * dc_msg_unref(msg);
- * free(file_to_send);
- * dc_str_unref(file_to_send);
- * ~~~
- *
- * @memberof dc_context_t
- * @param context The context object as returned from dc_context_new().
- * @param chat_id The chat ID to send the message to.
- * @param msg The message object to send to the chat defined by the chat ID.
- *     On success, msg_id and state of the object are set up,
- *     The function does not take ownership of the object,
- *     so you have to free it using dc_msg_unref() as usual.
- * @return The ID of the message that is being prepared.
- */
-uint32_t        dc_prepare_msg               (dc_context_t* context, uint32_t chat_id, dc_msg_t* msg);
-
-
 /**
  * Send a message defined by a dc_msg_t object to a chat.
  *
@@ -1041,13 +987,11 @@ uint32_t        dc_prepare_msg               (dc_context_t* context, uint32_t ch
  * If that fails, is not possible, or the image is already small enough, the image is sent as original.
  * If you want images to be always sent as the original file, use the #DC_MSG_FILE type.
  *
- * Videos and other file types are currently not recoded by the library,
- * with dc_prepare_msg(), however, you can do that from the UI.
+ * Videos and other file types are currently not recoded by the library.
  *
  * @memberof dc_context_t
  * @param context The context object as returned from dc_context_new().
  * @param chat_id The chat ID to send the message to.
- *     If dc_prepare_msg() was called before, this parameter can be 0.
  * @param msg The message object to send to the chat defined by the chat ID.
  *     On success, msg_id of the object is set up,
  *     The function does not take ownership of the object,
@@ -1064,7 +1008,6 @@ uint32_t        dc_send_msg                  (dc_context_t* context, uint32_t ch
  * @memberof dc_context_t
  * @param context The context object as returned from dc_context_new().
  * @param chat_id The chat ID to send the message to.
- *     If dc_prepare_msg() was called before, this parameter can be 0.
  * @param msg The message object to send to the chat defined by the chat ID.
  *     On success, msg_id of the object is set up,
  *     The function does not take ownership of the object,
@@ -3991,7 +3934,7 @@ int             dc_msg_get_viewtype           (const dc_msg_t* msg);
  *
  * Outgoing message states:
  * - @ref DC_STATE_OUT_PREPARING - For files which need time to be prepared before they can be sent,
- *   the message enters this state before @ref DC_STATE_OUT_PENDING.
+ *   the message enters this state before @ref DC_STATE_OUT_PENDING. Deprecated.
  * - @ref DC_STATE_OUT_DRAFT - Message saved as draft using dc_set_draft()
  * - @ref DC_STATE_OUT_PENDING - The user has pressed the "send" button but the
  *   message is not yet sent and is pending in some way. Maybe we're offline (no checkmark).
@@ -4541,20 +4484,6 @@ int             dc_msg_get_info_type          (const dc_msg_t* msg);
  */
 char*           dc_msg_get_webxdc_href        (const dc_msg_t* msg);
 
-/**
- * Check if a message is still in creation. A message is in creation between
- * the calls to dc_prepare_msg() and dc_send_msg().
- *
- * Typically, this is used for videos that are recoded by the UI before
- * they can be sent.
- *
- * @memberof dc_msg_t
- * @param msg The message object.
- * @return 1=message is still in creation (dc_send_msg() was not called yet),
- *     0=message no longer in creation.
- */
-int             dc_msg_is_increation          (const dc_msg_t* msg);
-
 
 /**
  * Check if the message is an Autocrypt Setup Message.
@@ -5464,6 +5393,8 @@ int64_t         dc_lot_get_timestamp     (const dc_lot_t* lot);
 
 /**
  * Message containing a sticker, similar to image.
+ * NB: When sending, the message viewtype may be changed to `Image` by some heuristics like checking
+ * for transparent pixels.
  * If possible, the UI should display the image without borders in a transparent way.
  * A click on a sticker will offer to install the sticker set in some future.
  */
@@ -5568,6 +5499,8 @@ int64_t         dc_lot_get_timestamp     (const dc_lot_t* lot);
 
 /**
  * Outgoing message being prepared. See dc_msg_get_state() for details.
+ *
+ * @deprecated 2024-12-07
  */
 #define         DC_STATE_OUT_PREPARING       18
 
@@ -6917,7 +6850,7 @@ void dc_event_unref(dc_event_t* event);
 
 /// "Failed to send message to %1$s."
 ///
-/// Used in status messages.
+/// Unused. Was used in group chat status messages.
 /// - %1$s will be replaced by the name of the contact the message cannot be sent to
 #define DC_STR_FAILED_SENDING_TO          74
 

deltachat-ffi/src/lib.rs

@@ -413,16 +413,6 @@ pub unsafe extern "C" fn dc_get_push_state(context: *const dc_context_t) -> libc
     block_on(ctx.push_state()) as libc::c_int
 }
 
-#[no_mangle]
-pub unsafe extern "C" fn dc_all_work_done(context: *mut dc_context_t) -> libc::c_int {
-    if context.is_null() {
-        eprintln!("ignoring careless call to dc_all_work_done()");
-        return 0;
-    }
-    let ctx = &*context;
-    block_on(async move { ctx.all_work_done().await as libc::c_int })
-}
-
 #[no_mangle]
 pub unsafe extern "C" fn dc_get_oauth2_url(
     context: *mut dc_context_t,
@@ -986,27 +976,6 @@ pub unsafe extern "C" fn dc_get_chat_id_by_contact_id(
     })
 }
 
-#[no_mangle]
-pub unsafe extern "C" fn dc_prepare_msg(
-    context: *mut dc_context_t,
-    chat_id: u32,
-    msg: *mut dc_msg_t,
-) -> u32 {
-    if context.is_null() || chat_id == 0 || msg.is_null() {
-        eprintln!("ignoring careless call to dc_prepare_msg()");
-        return 0;
-    }
-    let ctx = &mut *context;
-    let ffi_msg: &mut MessageWrapper = &mut *msg;
-
-    block_on(async move {
-        chat::prepare_msg(ctx, ChatId::new(chat_id), &mut ffi_msg.message)
-            .await
-            .unwrap_or_log_default(ctx, "Failed to prepare message")
-    })
-    .to_u32()
-}
-
 #[no_mangle]
 pub unsafe extern "C" fn dc_send_msg(
     context: *mut dc_context_t,
@@ -3723,16 +3692,6 @@ pub unsafe extern "C" fn dc_msg_get_webxdc_href(msg: *mut dc_msg_t) -> *mut libc
     ffi_msg.message.get_webxdc_href().strdup()
 }
 
-#[no_mangle]
-pub unsafe extern "C" fn dc_msg_is_increation(msg: *mut dc_msg_t) -> libc::c_int {
-    if msg.is_null() {
-        eprintln!("ignoring careless call to dc_msg_is_increation()");
-        return 0;
-    }
-    let ffi_msg = &*msg;
-    ffi_msg.message.is_increation().into()
-}
-
 #[no_mangle]
 pub unsafe extern "C" fn dc_msg_is_setupmessage(msg: *mut dc_msg_t) -> libc::c_int {
     if msg.is_null() {

deltachat-jsonrpc/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-jsonrpc"
-version = "1.151.4"
+version = "1.153.0"
 description = "DeltaChat JSON-RPC API"
 edition = "2021"
 default-run = "deltachat-jsonrpc-server"
@@ -33,7 +33,7 @@ base64 = { workspace = true }
 
 # optional dependencies
 axum = { version = "0.7", optional = true, features = ["ws"] }
-env_logger = { version = "0.11.5", optional = true }
+env_logger = { version = "0.11.6", optional = true }
 
 [dev-dependencies]
 tokio = { workspace = true, features = ["full", "rt-multi-thread"] }

deltachat-jsonrpc/src/api/types/events.rs

@@ -109,6 +109,7 @@ pub enum EventType {
     /// Incoming webxdc info or summary update, should be notified.
     #[serde(rename_all = "camelCase")]
     IncomingWebxdcNotify {
+        chat_id: u32,
         contact_id: u32,
         msg_id: u32,
         text: String,
@@ -343,11 +344,13 @@ impl From<CoreEventType> for EventType {
                 reaction: reaction.as_str().to_string(),
             },
             CoreEventType::IncomingWebxdcNotify {
+                chat_id,
                 contact_id,
                 msg_id,
                 text,
                 href,
             } => IncomingWebxdcNotify {
+                chat_id: chat_id.to_u32(),
                 contact_id: contact_id.to_u32(),
                 msg_id: msg_id.to_u32(),
                 text,

deltachat-jsonrpc/src/api/types/message.rs

@@ -273,6 +273,9 @@ pub enum MessageViewtype {
     Gif,
 
     /// Message containing a sticker, similar to image.
+    /// NB: When sending, the message viewtype may be changed to `Image` by some heuristics like
+    /// checking for transparent pixels. Use `Message::force_sticker()` to disable them.
+    ///
     /// If possible, the ui should display the image without borders in a transparent way.
     /// A click on a sticker will offer to install the sticker set in some future.
     Sticker,

deltachat-jsonrpc/typescript/package.json

@@ -58,5 +58,5 @@
   },
   "type": "module",
   "types": "dist/deltachat.d.ts",
-  "version": "1.151.4"
+  "version": "1.153.0"
 }

deltachat-repl/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-repl"
-version = "1.151.4"
+version = "1.153.0"
 license = "MPL-2.0"
 edition = "2021"
 repository = "https://github.com/deltachat/deltachat-core-rust"

deltachat-rpc-client/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "deltachat-rpc-client"
-version = "1.151.4"
+version = "1.153.0"
 description = "Python client for Delta Chat core JSON-RPC interface"
 classifiers = [
     "Development Status :: 5 - Production/Stable",

deltachat-rpc-client/src/deltachat_rpc_client/const.py

@@ -41,6 +41,7 @@ class EventType(str, Enum):
     REACTIONS_CHANGED = "ReactionsChanged"
     INCOMING_MSG = "IncomingMsg"
     INCOMING_MSG_BUNCH = "IncomingMsgBunch"
+    INCOMING_REACTION = "IncomingReaction"
     MSGS_NOTICED = "MsgsNoticed"
     MSG_DELIVERED = "MsgDelivered"
     MSG_FAILED = "MsgFailed"

deltachat-rpc-client/tests/test_securejoin.py

@@ -73,22 +73,25 @@ def test_qr_securejoin(acfactory, protect, tmp_path):
     qr_code = alice_chat.get_qr_code()
     bob.secure_join(qr_code)
 
-    # Check that at least some of the handshake messages are deleted.
+    # Alice deletes "vg-request".
+    while True:
+        event = alice.wait_for_event()
+        if event["kind"] == "ImapMessageDeleted":
+            break
+    alice.wait_for_securejoin_inviter_success()
+    # Bob deletes "vg-auth-required", Alice deletes "vg-request-with-auth".
     for ac in [alice, bob]:
         while True:
             event = ac.wait_for_event()
             if event["kind"] == "ImapMessageDeleted":
                 break
-
-    alice.wait_for_securejoin_inviter_success()
+    bob.wait_for_securejoin_joiner_success()
 
     # Test that Alice verified Bob's profile.
     alice_contact_bob = alice.get_contact_by_addr(bob.get_config("addr"))
     alice_contact_bob_snapshot = alice_contact_bob.get_snapshot()
     assert alice_contact_bob_snapshot.is_verified
 
-    bob.wait_for_securejoin_joiner_success()
-
     snapshot = bob.get_message_by_id(bob.wait_for_incoming_msg_event().msg_id).get_snapshot()
     assert snapshot.text == "Member Me ({}) added by {}.".format(bob.get_config("addr"), alice.get_config("addr"))
     assert snapshot.chat.get_basic_snapshot().is_protected == protect

deltachat-rpc-server/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-rpc-server"
-version = "1.151.4"
+version = "1.153.0"
 description = "DeltaChat JSON-RPC server"
 edition = "2021"
 readme = "README.md"

deltachat-rpc-server/npm-package/package.json

@@ -15,5 +15,5 @@
   },
   "type": "module",
   "types": "index.d.ts",
-  "version": "1.151.4"
+  "version": "1.153.0"
 }

deny.toml

@@ -18,6 +18,9 @@ ignore = [
 
     # Unmaintained instant
     "RUSTSEC-2024-0384",
+
+    # idna 0.5.0
+    "RUSTSEC-2024-0421",
 ]
 
 [bans]
@@ -39,9 +42,6 @@ skip = [
      { name = "http", version = "0.2.12" },
      { name = "idna", version = "0.5.0" },
      { name = "nix", version = "0.26.4" },
-     { name = "num_enum_derive", version = "0.5.11" },
-     { name = "num_enum", version = "0.5.11" },
-     { name = "proc-macro-crate", version = "1.3.1" },
      { name = "quick-error", version = "<2.0" },
      { name = "rand_chacha", version = "<0.3" },
      { name = "rand_core", version = "<0.6" },
@@ -51,8 +51,9 @@ skip = [
      { name = "regex-syntax", version = "0.6.29" },
      { name = "sync_wrapper", version = "0.1.2" },
      { name = "syn", version = "1.0.109" },
+     { name = "thiserror-impl", version = "1.0.69" },
+     { name = "thiserror", version = "1.0.69" },
      { name = "time", version = "<0.3" },
-     { name = "toml_edit", version = "0.19.15" },
      { name = "wasi", version = "<0.11" },
      { name = "windows_aarch64_gnullvm", version = "<0.52" },
      { name = "windows_aarch64_msvc", version = "<0.52" },
@@ -65,7 +66,6 @@ skip = [
      { name = "windows_x86_64_gnullvm", version = "<0.52" },
      { name = "windows_x86_64_gnu", version = "<0.52" },
      { name = "windows_x86_64_msvc", version = "<0.52" },
-     { name = "winnow", version = "0.5.40" },
      { name = "winreg", version = "0.50.0" },
 ]
 

flake.lock

@@ -47,16 +47,17 @@
         "rust-analyzer-src": "rust-analyzer-src"
       },
       "locked": {
-        "lastModified": 1731393059,
-        "narHash": "sha256-rmzi0GHEwpzg1LGfGPO4SRD7D6QGV3UYGQxkJvn+J5U=",
+        "lastModified": 1711088506,
+        "narHash": "sha256-USdlY7Tx2oJWqFBpp10+03+h7eVhpkQ4s9t1ERjeIJE=",
         "owner": "nix-community",
         "repo": "fenix",
-        "rev": "fda8d5b59bb0dc0021ad3ba1d722f9ef6d36e4d9",
+        "rev": "85f4139f3c092cf4afd9f9906d7ed218ef262c97",
         "type": "github"
       },
       "original": {
         "owner": "nix-community",
         "repo": "fenix",
+        "rev": "85f4139f3c092cf4afd9f9906d7ed218ef262c97",
         "type": "github"
       }
     },
@@ -114,6 +115,25 @@
         "type": "github"
       }
     },
+    "new-fenix": {
+      "inputs": {
+        "nixpkgs": "nixpkgs_4",
+        "rust-analyzer-src": "rust-analyzer-src_2"
+      },
+      "locked": {
+        "lastModified": 1734417396,
+        "narHash": "sha256-32x1Z+Pz3Jv0cK9EG56cFTKXy/mZ/c+Ikxw+aVfKHp4=",
+        "owner": "nix-community",
+        "repo": "fenix",
+        "rev": "a18d41b26e998e95a598858fdb86ba22fb5da47d",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "fenix",
+        "type": "github"
+      }
+    },
     "nix-filter": {
       "locked": {
         "lastModified": 1730207686,
@@ -174,6 +194,22 @@
       }
     },
     "nixpkgs_4": {
+      "locked": {
+        "lastModified": 1734119587,
+        "narHash": "sha256-AKU6qqskl0yf2+JdRdD0cfxX4b9x3KKV5RqA6wijmPM=",
+        "owner": "nixos",
+        "repo": "nixpkgs",
+        "rev": "3566ab7246670a43abd2ffa913cc62dad9cdf7d5",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nixos",
+        "ref": "nixos-unstable",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs_5": {
       "locked": {
         "lastModified": 1731139594,
         "narHash": "sha256-IigrKK3vYRpUu+HEjPL/phrfh7Ox881er1UEsZvw9Q4=",
@@ -195,8 +231,9 @@
         "fenix": "fenix",
         "flake-utils": "flake-utils_2",
         "naersk": "naersk",
+        "new-fenix": "new-fenix",
         "nix-filter": "nix-filter",
-        "nixpkgs": "nixpkgs_4"
+        "nixpkgs": "nixpkgs_5"
       }
     },
     "rust-analyzer-src": {
@@ -216,6 +253,23 @@
         "type": "github"
       }
     },
+    "rust-analyzer-src_2": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1734386068,
+        "narHash": "sha256-Py025JiD9lcPmldB7X1AEjq3WBTS60jZUJRtTDonmaE=",
+        "owner": "rust-lang",
+        "repo": "rust-analyzer",
+        "rev": "0a706f7d2ac093985eae317781200689cfd48b78",
+        "type": "github"
+      },
+      "original": {
+        "owner": "rust-lang",
+        "ref": "nightly",
+        "repo": "rust-analyzer",
+        "type": "github"
+      }
+    },
     "systems": {
       "locked": {
         "lastModified": 1681028828,

flake.nix

@@ -1,14 +1,19 @@
 {
   description = "Delta Chat core";
   inputs = {
-    fenix.url = "github:nix-community/fenix";
+    # Old Rust to build releases.
+    fenix.url = "github:nix-community/fenix?rev=85f4139f3c092cf4afd9f9906d7ed218ef262c97";
+
+    # New Rust for development shell.
+    new-fenix.url = "github:nix-community/fenix";
+
     flake-utils.url = "github:numtide/flake-utils";
     naersk.url = "github:nix-community/naersk";
     nix-filter.url = "github:numtide/nix-filter";
     nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
     android.url = "github:tadfisher/android-nixpkgs";
   };
-  outputs = { self, nixpkgs, flake-utils, nix-filter, naersk, fenix, android }:
+  outputs = { self, nixpkgs, flake-utils, nix-filter, naersk, fenix, new-fenix, android }:
     flake-utils.lib.eachDefaultSystem (system:
       let
         pkgs = nixpkgs.legacyPackages.${system};
@@ -539,13 +544,13 @@
           let
             pkgs = import nixpkgs {
               system = system;
-              overlays = [ fenix.overlays.default ];
+              overlays = [ new-fenix.overlays.default ];
             };
           in
           pkgs.mkShell {
 
             buildInputs = with pkgs; [
-              (fenix.packages.${system}.complete.withComponents [
+              (new-fenix.packages.${system}.complete.withComponents [
                 "cargo"
                 "clippy"
                 "rust-src"

fuzz/Cargo.lock

@@ -146,6 +146,7 @@ dependencies = [
  "blake2",
  "cpufeatures",
  "password-hash",
+ "zeroize",
 ]
 
 [[package]]
@@ -178,7 +179,7 @@ dependencies = [
  "nom",
  "num-traits",
  "rusticata-macros",
- "thiserror",
+ "thiserror 1.0.58",
  "time 0.3.36",
 ]
 
@@ -190,7 +191,7 @@ checksum = "7378575ff571966e99a744addeff0bff98b8ada0dedf1956d59e634db95eaac1"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
  "synstructure 0.13.1",
 ]
 
@@ -202,7 +203,7 @@ checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -274,7 +275,7 @@ dependencies = [
  "pin-utils",
  "self_cell",
  "stop-token",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
 ]
 
@@ -285,7 +286,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "9343dc5acf07e79ff82d0c37899f079db3534d99f189a1837c8e549c99405bec"
 dependencies = [
  "native-tls",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "url",
 ]
@@ -298,23 +299,22 @@ checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
 name = "async-smtp"
-version = "0.9.0"
+version = "0.10.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8709c0d4432be428a88a06746689a9cb543e8e27ef7f61ca4d0455003a3d8c5b"
+checksum = "3ee04bcf0a7ebf5594f9aff84935dc8cb0490b65055913a7a4c4d08f81e181d6"
 dependencies = [
  "anyhow",
  "base64 0.13.1",
  "futures",
- "hostname",
  "log",
  "nom",
  "pin-project",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
 ]
 
@@ -326,7 +326,7 @@ checksum = "6e0c28dcc82d7c8ead5cb13beb15405b57b8546e93215673ff8ca0349a028107"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -339,7 +339,7 @@ dependencies = [
  "crc32fast",
  "futures-lite 2.5.0",
  "pin-project",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "tokio-util",
 ]
@@ -1040,7 +1040,7 @@ checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -1108,7 +1108,7 @@ dependencies = [
  "proc-macro2",
  "quote",
  "strsim",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -1119,7 +1119,7 @@ checksum = "d336a2a514f6ccccaa3e09b02d41d35330c07ddf03a62165fcec10bb561c7806"
 dependencies = [
  "darling_core",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -1152,7 +1152,7 @@ dependencies = [
 
 [[package]]
 name = "deltachat"
-version = "1.150.0"
+version = "1.151.5"
 dependencies = [
  "anyhow",
  "async-broadcast",
@@ -1220,7 +1220,7 @@ dependencies = [
  "strum_macros",
  "tagger",
  "textwrap",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "tokio-io-timeout",
  "tokio-rustls",
@@ -1263,7 +1263,7 @@ name = "deltachat_derive"
 version = "2.0.0"
 dependencies = [
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -1300,7 +1300,7 @@ checksum = "8034092389675178f570469e6c3b0465d3d30b4505c294a6550db47f3c17ad18"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -1331,7 +1331,7 @@ dependencies = [
  "darling",
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -1341,7 +1341,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "206868b8242f27cecce124c19fd88157fbd0dd334df2587f36417bafbc85097b"
 dependencies = [
  "derive_builder_core",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -1361,7 +1361,7 @@ checksum = "cb7330aeadfbe296029522e6c40f315320aba36fc43a5b3632f3795348f3bd22"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
  "unicode-xid",
 ]
 
@@ -1579,7 +1579,7 @@ dependencies = [
  "hex",
  "lazy_static",
  "regex",
- "thiserror",
+ "thiserror 1.0.58",
 ]
 
 [[package]]
@@ -1670,7 +1670,7 @@ dependencies = [
  "heck 0.4.1",
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -1690,7 +1690,7 @@ checksum = "de0d48a183585823424a4ce1aa132d174a6a81bd540895822eb4c8373a8e49e8"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -1814,7 +1814,7 @@ dependencies = [
  "anyhow",
  "async-trait",
  "log",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "tokio-stream",
 ]
@@ -2061,7 +2061,7 @@ checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -2331,7 +2331,7 @@ dependencies = [
  "ipnet",
  "once_cell",
  "rand 0.8.5",
- "thiserror",
+ "thiserror 1.0.58",
  "time 0.3.36",
  "tinyvec",
  "tokio",
@@ -2355,7 +2355,7 @@ dependencies = [
  "rand 0.8.5",
  "resolv-conf",
  "smallvec",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "tracing",
 ]
@@ -2718,7 +2718,7 @@ dependencies = [
  "rand_core 0.6.4",
  "serde",
  "ssh-key",
- "thiserror",
+ "thiserror 1.0.58",
  "ttl_cache",
  "url",
  "zeroize",
@@ -2828,7 +2828,7 @@ dependencies = [
  "netlink-packet-route",
  "netlink-sys",
  "netwatch",
- "num_enum 0.7.2",
+ "num_enum",
  "once_cell",
  "parking_lot",
  "pin-project",
@@ -2848,7 +2848,7 @@ dependencies = [
  "strum",
  "stun-rs",
  "surge-ping",
- "thiserror",
+ "thiserror 1.0.58",
  "time 0.3.36",
  "tokio",
  "tokio-rustls",
@@ -2880,7 +2880,7 @@ dependencies = [
  "rustc-hash 2.0.0",
  "rustls",
  "socket2 0.5.6",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "tracing",
 ]
@@ -2898,7 +2898,7 @@ dependencies = [
  "rustls",
  "rustls-platform-verifier",
  "slab",
- "thiserror",
+ "thiserror 1.0.58",
  "tinyvec",
  "tracing",
 ]
@@ -2954,7 +2954,7 @@ dependencies = [
  "combine",
  "jni-sys",
  "log",
- "thiserror",
+ "thiserror 1.0.58",
  "walkdir",
 ]
 
@@ -3186,7 +3186,7 @@ dependencies = [
  "serde_bencode",
  "serde_bytes",
  "sha1_smol",
- "thiserror",
+ "thiserror 1.0.58",
  "tracing",
 ]
 
@@ -3349,7 +3349,7 @@ dependencies = [
  "anyhow",
  "byteorder",
  "paste",
- "thiserror",
+ "thiserror 1.0.58",
 ]
 
 [[package]]
@@ -3363,7 +3363,7 @@ dependencies = [
  "log",
  "netlink-packet-core",
  "netlink-sys",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
 ]
 
@@ -3401,7 +3401,7 @@ dependencies = [
  "rtnetlink",
  "serde",
  "socket2 0.5.6",
- "thiserror",
+ "thiserror 1.0.58",
  "time 0.3.36",
  "tokio",
  "tracing",
@@ -3500,7 +3500,7 @@ checksum = "9e6a0fd4f737c707bd9086cc16c925f294943eb62eb71499e9fd4cf71f8b9f4e"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -3543,34 +3543,13 @@ dependencies = [
  "libc",
 ]
 
-[[package]]
-name = "num_enum"
-version = "0.5.11"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1f646caf906c20226733ed5b1374287eb97e3c2a5c227ce668c1f2ce20ae57c9"
-dependencies = [
- "num_enum_derive 0.5.11",
-]
-
 [[package]]
 name = "num_enum"
 version = "0.7.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "02339744ee7253741199f897151b38e72257d13802d4ee837285cc2990a90845"
 dependencies = [
- "num_enum_derive 0.7.2",
-]
-
-[[package]]
-name = "num_enum_derive"
-version = "0.5.11"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dcbff9bc912032c62bf65ef1d5aea88983b420f4f839db1e9b0c281a25c9c799"
-dependencies = [
- "proc-macro-crate 1.3.1",
- "proc-macro2",
- "quote",
- "syn 1.0.107",
+ "num_enum_derive",
 ]
 
 [[package]]
@@ -3579,10 +3558,10 @@ version = "0.7.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "681030a937600a36906c185595136d26abfebb4aa9c65701cefcaf8578bb982b"
 dependencies = [
- "proc-macro-crate 3.1.0",
+ "proc-macro-crate",
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -3813,7 +3792,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cd53dff83f26735fdc1ca837098ccf133605d794cdae66acfc2bfac3ec809d95"
 dependencies = [
  "memchr",
- "thiserror",
+ "thiserror 1.0.58",
  "ucd-trie",
 ]
 
@@ -3837,7 +3816,7 @@ dependencies = [
  "pest_meta",
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -3853,15 +3832,15 @@ dependencies = [
 
 [[package]]
 name = "pgp"
-version = "0.14.0"
+version = "0.14.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "49bb5f77aaf8ae1ed6fe63387ad513b10cd44716fd053ecc227b9493c096cdb2"
+checksum = "1877a97fd422433220ad272eb008ec55691944b1200e9eb204e3cb2cb69d34e9"
 dependencies = [
  "aes",
  "aes-gcm",
  "aes-kw",
  "argon2",
- "base64 0.21.7",
+ "base64 0.22.1",
  "bitfield",
  "block-padding",
  "blowfish",
@@ -3897,7 +3876,7 @@ dependencies = [
  "nom",
  "num-bigint-dig",
  "num-traits",
- "num_enum 0.5.11",
+ "num_enum",
  "ocb3",
  "p256",
  "p384",
@@ -3911,7 +3890,7 @@ dependencies = [
  "sha3",
  "signature",
  "smallvec",
- "thiserror",
+ "thiserror 1.0.58",
  "twofish",
  "x25519-dalek",
  "x448",
@@ -3935,7 +3914,7 @@ checksum = "3c0f5fad0874fc7abcd4d750e76917eaebbecaa2c20bde22e1dbeeba8beb758c"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -3967,7 +3946,7 @@ dependencies = [
  "mainline",
  "self_cell",
  "simple-dns",
- "thiserror",
+ "thiserror 1.0.58",
  "tracing",
  "ureq",
  "wasm-bindgen",
@@ -4021,7 +4000,7 @@ dependencies = [
  "proc-macro2",
  "quote",
  "regex",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -4102,12 +4081,12 @@ dependencies = [
  "iroh-metrics",
  "libc",
  "netwatch",
- "num_enum 0.7.2",
+ "num_enum",
  "rand 0.8.5",
  "serde",
  "smallvec",
  "socket2 0.5.6",
- "thiserror",
+ "thiserror 1.0.58",
  "time 0.3.36",
  "tokio",
  "tokio-util",
@@ -4199,23 +4178,13 @@ dependencies = [
  "elliptic-curve",
 ]
 
-[[package]]
-name = "proc-macro-crate"
-version = "1.3.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7f4c021e1093a56626774e81216a4ce732a735e5bad4868a03f3ed65ca0c3919"
-dependencies = [
- "once_cell",
- "toml_edit 0.19.15",
-]
-
 [[package]]
 name = "proc-macro-crate"
 version = "3.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "6d37c51ca738a55da99dc0c4a34860fd675453b8b36209178c2249bb13651284"
 dependencies = [
- "toml_edit 0.21.0",
+ "toml_edit",
 ]
 
 [[package]]
@@ -4252,9 +4221,9 @@ checksum = "dc375e1527247fe1a97d8b7156678dfe7c1af2fc075c9a4db3690ecd2a148068"
 
 [[package]]
 name = "proc-macro2"
-version = "1.0.78"
+version = "1.0.92"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e2422ad645d89c99f8f3e6b88a9fdeca7fabeac836b1002371c4367c8f984aae"
+checksum = "37d3544b3f2748c54e147655edb5025752e2303145b5aefb3c3ea2c78b973bb0"
 dependencies = [
  "unicode-ident",
 ]
@@ -4279,7 +4248,7 @@ checksum = "440f724eba9f6996b75d63681b0a92b06947f1457076d503a4d2e2c8f56442b8"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -4336,26 +4305,29 @@ dependencies = [
  "quinn-udp",
  "rustc-hash 1.1.0",
  "rustls",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "tracing",
 ]
 
 [[package]]
 name = "quinn-proto"
-version = "0.11.3"
+version = "0.11.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ddf517c03a109db8100448a4be38d498df8a210a99fe0e1b9eaf39e78c640efe"
+checksum = "a2fe5ef3495d7d2e377ff17b1a8ce2ee2ec2a18cde8b6ad6619d65d0701c135d"
 dependencies = [
  "bytes",
+ "getrandom 0.2.11",
  "rand 0.8.5",
  "ring",
- "rustc-hash 1.1.0",
+ "rustc-hash 2.0.0",
  "rustls",
+ "rustls-pki-types",
  "slab",
- "thiserror",
+ "thiserror 2.0.6",
  "tinyvec",
  "tracing",
+ "web-time",
 ]
 
 [[package]]
@@ -4668,22 +4640,21 @@ dependencies = [
 
 [[package]]
 name = "rsa"
-version = "0.9.1"
+version = "0.9.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "72f1471dbb4be5de45050e8ef7040625298ccb9efe941419ac2697088715925f"
+checksum = "47c75d7c5c6b673e58bf54d8544a9f432e3a925b0e80f7cd3602ab5c50c55519"
 dependencies = [
- "byteorder",
  "const-oid",
  "digest",
  "num-bigint-dig",
  "num-integer",
- "num-iter",
  "num-traits",
  "pkcs1",
  "pkcs8",
  "rand_core 0.6.4",
  "sha2",
  "signature",
+ "spki",
  "subtle",
  "zeroize",
 ]
@@ -4702,7 +4673,7 @@ dependencies = [
  "netlink-proto",
  "netlink-sys",
  "nix",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
 ]
 
@@ -4791,9 +4762,9 @@ dependencies = [
 
 [[package]]
 name = "rustls"
-version = "0.23.18"
+version = "0.23.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9c9cc1d47e243d655ace55ed38201c19ae02c148ae56412ab8750e8f0166ab7f"
+checksum = "934b404430bb06b3fae2cba809eb45a1ab1aecd64491213d7c3301b88393f8d1"
 dependencies = [
  "log",
  "once_cell",
@@ -4832,6 +4803,9 @@ name = "rustls-pki-types"
 version = "1.10.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "16f1201b3c9a7ee8039bcadc17b7e605e2945b27eee7631788c1bd2b0643674b"
+dependencies = [
+ "web-time",
+]
 
 [[package]]
 name = "rustls-platform-verifier"
@@ -5050,7 +5024,7 @@ checksum = "e0cd7e117be63d3c3678776753929474f3b04a43a080c744d6b0ae2a8c28e222"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -5125,6 +5099,7 @@ checksum = "89f599ac0c323ebb1c6082821a54962b839832b03984598375bff3975b804423"
 dependencies = [
  "digest",
  "sha1",
+ "zeroize",
 ]
 
 [[package]]
@@ -5182,7 +5157,7 @@ dependencies = [
  "shadowsocks-crypto",
  "socket2 0.5.6",
  "spin 0.9.8",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "tokio-tfo",
  "url",
@@ -5328,9 +5303,9 @@ dependencies = [
 
 [[package]]
 name = "spki"
-version = "0.7.1"
+version = "0.7.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "37a5be806ab6f127c3da44b7378837ebf01dadca8510a0e572460216b228bd0e"
+checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d"
 dependencies = [
  "base64ct",
  "der",
@@ -5415,7 +5390,7 @@ dependencies = [
  "proc-macro2",
  "quote",
  "struct_iterable_internal",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -5443,7 +5418,7 @@ dependencies = [
  "proc-macro2",
  "quote",
  "rustversion",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -5487,7 +5462,7 @@ dependencies = [
  "pnet_packet",
  "rand 0.8.5",
  "socket2 0.5.6",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "tracing",
 ]
@@ -5505,9 +5480,9 @@ dependencies = [
 
 [[package]]
 name = "syn"
-version = "2.0.52"
+version = "2.0.90"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b699d15b36d1f02c3e7c69f8ffef53de37aefae075d8488d4ba1a7788d574a07"
+checksum = "919d3b74a5dd0ccd15aeb8f93e7006bd9e14c295087c9896a110f490752bcf31"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -5551,7 +5526,7 @@ checksum = "c8af7666ab7b6390ab78131fb5b0fce11d6b7a6951602017c35fa82800708971"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -5626,7 +5601,16 @@ version = "1.0.58"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "03468839009160513471e86a034bb2c5c0e4baae3b43f79ffc55c4a5427b3297"
 dependencies = [
- "thiserror-impl",
+ "thiserror-impl 1.0.58",
+]
+
+[[package]]
+name = "thiserror"
+version = "2.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8fec2a1820ebd077e2b90c4df007bebf344cd394098a13c563957d0afc83ea47"
+dependencies = [
+ "thiserror-impl 2.0.6",
 ]
 
 [[package]]
@@ -5637,7 +5621,18 @@ checksum = "c61f3ba182994efc43764a46c018c347bc492c79f024e705f46567b418f6d4f7"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
+]
+
+[[package]]
+name = "thiserror-impl"
+version = "2.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d65750cab40f4ff1929fb1ba509e9914eb756131cef4210da8d5d700d26f6312"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -5743,7 +5738,7 @@ checksum = "693d596312e88961bc67d7f1f97af8a70227d9f90c31bba5806eec004978d752"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -5823,7 +5818,7 @@ dependencies = [
  "http 1.1.0",
  "httparse",
  "js-sys",
- "thiserror",
+ "thiserror 1.0.58",
  "tokio",
  "tokio-tungstenite",
  "wasm-bindgen",
@@ -5855,7 +5850,7 @@ dependencies = [
  "serde",
  "serde_spanned",
  "toml_datetime",
- "toml_edit 0.21.0",
+ "toml_edit",
 ]
 
 [[package]]
@@ -5867,17 +5862,6 @@ dependencies = [
  "serde",
 ]
 
-[[package]]
-name = "toml_edit"
-version = "0.19.15"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1b5bb770da30e5cbfde35a2d7b9b8a2c4b8ef89548a7a6aeab5c9a576e3e7421"
-dependencies = [
- "indexmap",
- "toml_datetime",
- "winnow",
-]
-
 [[package]]
 name = "toml_edit"
 version = "0.21.0"
@@ -5917,7 +5901,7 @@ checksum = "34704c8d6ebcbc939824180af020566b01a7c01f80641264eba0999f6c2b6be7"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -5988,7 +5972,7 @@ dependencies = [
  "log",
  "rand 0.8.5",
  "sha1",
- "thiserror",
+ "thiserror 1.0.58",
  "url",
  "utf-8",
 ]
@@ -6217,7 +6201,7 @@ dependencies = [
  "once_cell",
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
  "wasm-bindgen-shared",
 ]
 
@@ -6251,7 +6235,7 @@ checksum = "e94f17b526d0a461a191c78ea52bbce64071ed5c04c9ffe424dcb38f74171bb7"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
  "wasm-bindgen-backend",
  "wasm-bindgen-shared",
 ]
@@ -6271,7 +6255,7 @@ dependencies = [
  "event-listener 4.0.3",
  "futures-util",
  "parking_lot",
- "thiserror",
+ "thiserror 1.0.58",
 ]
 
 [[package]]
@@ -6284,6 +6268,16 @@ dependencies = [
  "wasm-bindgen",
 ]
 
+[[package]]
+name = "web-time"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
+dependencies = [
+ "js-sys",
+ "wasm-bindgen",
+]
+
 [[package]]
 name = "webpki-roots"
 version = "0.26.7"
@@ -6393,7 +6387,7 @@ checksum = "12168c33176773b86799be25e2a2ba07c7aab9968b37541f1094dbd7a60c8946"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -6404,7 +6398,7 @@ checksum = "9d8dc32e0095a7eeccebd0e3f09e9509365ecb3fc6ac4d6f5f14a3f6392942d1"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]
@@ -6693,7 +6687,7 @@ dependencies = [
  "futures",
  "log",
  "serde",
- "thiserror",
+ "thiserror 1.0.58",
  "windows 0.52.0",
 ]
 
@@ -6742,7 +6736,7 @@ dependencies = [
  "nom",
  "oid-registry",
  "rusticata-macros",
- "thiserror",
+ "thiserror 1.0.58",
  "time 0.3.36",
 ]
 
@@ -6802,7 +6796,7 @@ checksum = "9ce1b18ccd8e73a9321186f97e46f9f04b778851177567b1975109d26a08d2a6"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.52",
+ "syn 2.0.90",
 ]
 
 [[package]]

node/lib/message.ts

@@ -299,10 +299,6 @@ export class Message {
     return Boolean(binding.dcn_msg_is_forwarded(this.dc_msg))
   }
 
-  isIncreation() {
-    return Boolean(binding.dcn_msg_is_increation(this.dc_msg))
-  }
-
   isInfo() {
     return Boolean(binding.dcn_msg_is_info(this.dc_msg))
   }

node/src/module.c

@@ -2374,17 +2374,6 @@ NAPI_METHOD(dcn_msg_is_forwarded) {
   NAPI_RETURN_INT32(is_forwarded);
 }
 
-NAPI_METHOD(dcn_msg_is_increation) {
-  NAPI_ARGV(1);
-  NAPI_DC_MSG();
-
-  //TRACE("calling..");
-  int is_increation = dc_msg_is_increation(dc_msg);
-  //TRACE("result %d", is_increation);
-
-  NAPI_RETURN_INT32(is_increation);
-}
-
 NAPI_METHOD(dcn_msg_is_info) {
   NAPI_ARGV(1);
   NAPI_DC_MSG();
@@ -3555,7 +3544,6 @@ NAPI_INIT() {
   NAPI_EXPORT_FUNCTION(dcn_msg_has_location);
   NAPI_EXPORT_FUNCTION(dcn_msg_has_html);
   NAPI_EXPORT_FUNCTION(dcn_msg_is_forwarded);
-  NAPI_EXPORT_FUNCTION(dcn_msg_is_increation);
   NAPI_EXPORT_FUNCTION(dcn_msg_is_info);
   NAPI_EXPORT_FUNCTION(dcn_msg_is_sent);
   NAPI_EXPORT_FUNCTION(dcn_msg_is_setupmessage);

node/test/test.mjs

@@ -536,7 +536,6 @@ describe('Offline Tests with unconfigured account', function () {
     strictEqual(msg.getWidth(), 0, 'no message width')
     strictEqual(msg.isDeadDrop(), false, 'not deaddrop')
     strictEqual(msg.isForwarded(), false, 'not forwarded')
-    strictEqual(msg.isIncreation(), false, 'not in creation')
     strictEqual(msg.isInfo(), false, 'not an info message')
     strictEqual(msg.isSent(), false, 'messge is not sent')
     strictEqual(msg.isSetupmessage(), false, 'not an autocrypt setup message')

package.json

@@ -55,5 +55,5 @@
     "test:mocha": "mocha node/test/test.mjs --growl --reporter=spec --bail --exit"
   },
   "types": "node/dist/index.d.ts",
-  "version": "1.151.4"
+  "version": "1.153.0"
 }

python/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "deltachat"
-version = "1.151.4"
+version = "1.153.0"
 description = "Python bindings for the Delta Chat Core library using CFFI against the Rust-implemented libdeltachat"
 readme = "README.rst"
 requires-python = ">=3.7"

python/src/deltachat/account.py

@@ -671,9 +671,6 @@ class Account:
     def get_connectivity_html(self) -> str:
         return from_dc_charpointer(lib.dc_get_connectivity_html(self._dc_context))
 
-    def all_work_done(self):
-        return lib.dc_all_work_done(self._dc_context)
-
     def start_io(self):
         """start this account's IO scheduling (Rust-core async scheduler).
 

python/src/deltachat/chat.py

@@ -271,8 +271,7 @@ class Chat:
 
         :param msg: a :class:`deltachat.message.Message` instance
            previously returned by
-           e.g. :meth:`deltachat.message.Message.new_empty` or
-           :meth:`prepare_file`.
+           e.g. :meth:`deltachat.message.Message.new_empty`.
         :raises ValueError: if message can not be sent.
 
         :returns: a :class:`deltachat.message.Message` instance as
@@ -341,37 +340,6 @@ class Chat:
             raise ValueError("message could not be sent")
         return Message.from_db(self.account, sent_id)
 
-    def prepare_message(self, msg):
-        """prepare a message for sending.
-
-        :param msg: the message to be prepared.
-        :returns: a :class:`deltachat.message.Message` instance.
-           This is the same object that was passed in, which
-           has been modified with the new state of the core.
-        """
-        msg_id = lib.dc_prepare_msg(self.account._dc_context, self.id, msg._dc_msg)
-        if msg_id == 0:
-            raise ValueError("message could not be prepared")
-        # modify message in place to avoid bad state for the caller
-        msg._dc_msg = Message.from_db(self.account, msg_id)._dc_msg
-        return msg
-
-    def prepare_message_file(self, path, mime_type=None, view_type="file"):
-        """prepare a message for sending and return the resulting Message instance.
-
-        To actually send the message, call :meth:`send_prepared`.
-        The file must be inside the blob directory.
-
-        :param path: path to the file.
-        :param mime_type: the mime-type of this file, defaults to auto-detection.
-        :param view_type: "text", "image", "gif", "audio", "video", "file"
-        :raises ValueError: if message can not be prepared/chat does not exist.
-        :returns: the resulting :class:`Message` instance
-        """
-        msg = Message.new_empty(self.account, view_type)
-        msg.set_file(path, mime_type)
-        return self.prepare_message(msg)
-
     def send_prepared(self, message):
         """send a previously prepared message.
 

python/src/deltachat/events.py

@@ -158,12 +158,6 @@ class FFIEventTracker:
 
             self.get_matching("DC_EVENT_CONNECTIVITY_CHANGED")
 
-    def wait_for_all_work_done(self):
-        while True:
-            if self.account.all_work_done():
-                return
-            self.get_matching("DC_EVENT_CONNECTIVITY_CHANGED")
-
     def ensure_event_not_queued(self, event_name_regex):
         __tracebackhide__ = True
         rex = re.compile(f"(?:{event_name_regex}).*")

python/tests/test_1_online.py

@@ -1253,7 +1253,10 @@ def test_no_old_msg_is_fresh(acfactory, lp):
 
 def test_prefer_encrypt(acfactory, lp):
     """Test quorum rule for encryption preference in 1:1 and group chat."""
-    ac1, ac2, ac3 = acfactory.get_online_accounts(3)
+    ac1 = acfactory.new_online_configuring_account(fix_is_chatmail=True)
+    ac2 = acfactory.new_online_configuring_account(fix_is_chatmail=True)
+    ac3 = acfactory.new_online_configuring_account(fix_is_chatmail=True)
+    acfactory.bring_accounts_online()
     ac1.set_config("e2ee_enabled", "0")
     ac2.set_config("e2ee_enabled", "1")
     ac3.set_config("e2ee_enabled", "0")
@@ -1276,7 +1279,8 @@ def test_prefer_encrypt(acfactory, lp):
     lp.sec("ac2: sending message to ac1")
     chat2 = ac2.create_chat(ac1)
     msg2 = chat2.send_text("message2")
-    assert not msg2.is_encrypted()
+    # Own preference is `Mutual` and we have the peer's key.
+    assert msg2.is_encrypted()
     ac1._evtracker.wait_next_incoming_message()
 
     lp.sec("ac1: sending message to group chat with ac2 and ac3")
@@ -1292,8 +1296,8 @@ def test_prefer_encrypt(acfactory, lp):
     ac3.set_config("e2ee_enabled", "1")
     chat3 = ac3.create_chat(ac1)
     msg4 = chat3.send_text("message4")
-    # ac1 still does not prefer encryption
-    assert not msg4.is_encrypted()
+    # Own preference is `Mutual` and we have the peer's key.
+    assert msg4.is_encrypted()
     ac1._evtracker.wait_next_incoming_message()
 
     lp.sec("ac1: sending another message to group chat with ac2 and ac3")
@@ -1366,10 +1370,9 @@ def test_quote_encrypted(acfactory, lp):
         msg_draft.quote = quoted_msg
         chat.set_draft(msg_draft)
 
-        # Get the draft, prepare and send it.
+        # Get the draft and send it.
         msg_draft = chat.get_draft()
-        msg_out = chat.prepare_message(msg_draft)
-        chat.send_prepared(msg_out)
+        chat.send_msg(msg_draft)
 
         chat.set_draft(None)
         assert chat.get_draft() is None
@@ -1900,9 +1903,10 @@ def test_connectivity(acfactory, lp):
     ac1.start_io()
     ac1._evtracker.wait_for_connectivity(dc.const.DC_CONNECTIVITY_CONNECTING)
     ac1._evtracker.wait_for_connectivity_change(dc.const.DC_CONNECTIVITY_CONNECTING, dc.const.DC_CONNECTIVITY_WORKING)
+    ac1._evtracker.wait_for_connectivity_change(dc.const.DC_CONNECTIVITY_WORKING, dc.const.DC_CONNECTIVITY_CONNECTED)
 
     lp.sec(
-        "Test that after calling start_io(), maybe_network() and waiting for `all_work_done()`, "
+        "Test that after calling start_io(), maybe_network() and waiting for `DC_CONNECTIVITY_CONNECTED`, "
         "all messages are fetched",
     )
 
@@ -1911,7 +1915,7 @@ def test_connectivity(acfactory, lp):
         ac2.create_chat(ac1).send_text("Hi")
         idle1.wait_for_new_message()
     ac1.maybe_network()
-    ac1._evtracker.wait_for_all_work_done()
+    ac1._evtracker.wait_for_connectivity(dc.const.DC_CONNECTIVITY_CONNECTED)
     msgs = ac1.create_chat(ac2).get_messages()
     assert len(msgs) == 1
     assert msgs[0].text == "Hi"
@@ -1927,30 +1931,6 @@ def test_connectivity(acfactory, lp):
     assert len(msgs) == 2
     assert msgs[1].text == "Hi 2"
 
-    lp.sec("Test that the connectivity doesn't flicker to WORKING if there are no new messages")
-
-    ac1.maybe_network()
-    while 1:
-        assert ac1.get_connectivity() == dc.const.DC_CONNECTIVITY_CONNECTED
-        if ac1.all_work_done():
-            break
-        ac1._evtracker.get_matching("DC_EVENT_CONNECTIVITY_CHANGED")
-
-    lp.sec("Test that the connectivity doesn't flicker to WORKING if the sender of the message is blocked")
-    ac1.create_contact(ac2).block()
-
-    ac1.direct_imap.select_config_folder("inbox")
-    with ac1.direct_imap.idle() as idle1:
-        ac2.create_chat(ac1).send_text("Hi")
-        idle1.wait_for_new_message()
-    ac1.maybe_network()
-
-    while 1:
-        assert ac1.get_connectivity() == dc.const.DC_CONNECTIVITY_CONNECTED
-        if ac1.all_work_done():
-            break
-        ac1._evtracker.get_matching("DC_EVENT_CONNECTIVITY_CHANGED")
-
     lp.sec("Test that the connectivity is NOT_CONNECTED if the password is wrong")
 
     ac1.set_config("configured_mail_pw", "abc")
@@ -1961,32 +1941,6 @@ def test_connectivity(acfactory, lp):
     ac1._evtracker.wait_for_connectivity(dc.const.DC_CONNECTIVITY_NOT_CONNECTED)
 
 
-def test_all_work_done(acfactory, lp):
-    """
-    Tests that calling start_io() immediately followed by maybe_network()
-    and then waiting for all_work_done() reliably fetches the messages
-    delivered while account was offline.
-    In other words, connectivity should not change to a state
-    where all_work_done() returns true until IMAP connection goes idle.
-    """
-    ac1, ac2 = acfactory.get_online_accounts(2)
-
-    ac1.stop_io()
-    ac1._evtracker.wait_for_connectivity(dc.const.DC_CONNECTIVITY_NOT_CONNECTED)
-
-    ac1.direct_imap.select_config_folder("inbox")
-    with ac1.direct_imap.idle() as idle1:
-        ac2.create_chat(ac1).send_text("Hi")
-        idle1.wait_for_new_message()
-
-    ac1.start_io()
-    ac1.maybe_network()
-    ac1._evtracker.wait_for_all_work_done()
-    msgs = ac1.create_chat(ac2).get_messages()
-    assert len(msgs) == 1
-    assert msgs[0].text == "Hi"
-
-
 def test_fetch_deleted_msg(acfactory, lp):
     """This is a regression test: Messages with \\Deleted flag were downloaded again and again,
     hundreds of times, because uid_next was not updated.
@@ -2340,9 +2294,8 @@ def test_group_quote(acfactory, lp):
     reply_msg = Message.new_empty(msg.chat.account, "text")
     reply_msg.set_text("reply")
     reply_msg.quote = msg
-    reply_msg = msg.chat.prepare_message(reply_msg)
     assert reply_msg.quoted_text == "hello"
-    msg.chat.send_prepared(reply_msg)
+    msg.chat.send_msg(reply_msg)
 
     lp.sec("ac3: receiving reply")
     received_reply = ac3._evtracker.wait_next_incoming_message()

python/tests/test_2_increation.py

@@ -1,107 +0,0 @@
-import os.path
-import shutil
-from filecmp import cmp
-
-import pytest
-
-
-def wait_msg_delivered(account, msg_list):
-    """wait for one or more MSG_DELIVERED events to match msg_list contents."""
-    msg_list = list(msg_list)
-    while msg_list:
-        ev = account._evtracker.get_matching("DC_EVENT_MSG_DELIVERED")
-        msg_list.remove((ev.data1, ev.data2))
-
-
-def wait_msgs_changed(account, msgs_list):
-    """wait for one or more MSGS_CHANGED events to match msgs_list contents."""
-    account.log(f"waiting for msgs_list={msgs_list}")
-    msgs_list = list(msgs_list)
-    while msgs_list:
-        ev = account._evtracker.get_matching("DC_EVENT_MSGS_CHANGED")
-        for i, (data1, data2) in enumerate(msgs_list):
-            if ev.data1 == data1:
-                if data2 is None or ev.data2 == data2:
-                    del msgs_list[i]
-                    break
-        else:
-            account.log(f"waiting mismatch data1={data1} data2={data2}")
-    return ev.data2
-
-
-class TestOnlineInCreation:
-    def test_increation_not_blobdir(self, tmp_path, acfactory, lp):
-        ac1, ac2 = acfactory.get_online_accounts(2)
-        chat = ac1.create_chat(ac2)
-
-        lp.sec("Creating in-creation file outside of blobdir")
-        assert str(tmp_path) != ac1.get_blobdir()
-        src = tmp_path / "file.txt"
-        src.touch()
-        with pytest.raises(Exception):
-            chat.prepare_message_file(str(src))
-
-    def test_no_increation_copies_to_blobdir(self, tmp_path, acfactory, lp):
-        ac1, ac2 = acfactory.get_online_accounts(2)
-        chat = ac1.create_chat(ac2)
-
-        lp.sec("Creating file outside of blobdir")
-        assert str(tmp_path) != ac1.get_blobdir()
-        src = tmp_path / "file.txt"
-        src.write_text("hello there\n")
-        msg = chat.send_file(str(src))
-        assert msg.filename.startswith(os.path.join(ac1.get_blobdir(), "file"))
-        assert msg.filename.endswith(".txt")
-
-    def test_forward_increation(self, acfactory, data, lp):
-        ac1, ac2 = acfactory.get_online_accounts(2)
-
-        chat = ac1.create_chat(ac2)
-        wait_msgs_changed(ac1, [(0, 0)])  # why no chat id?
-
-        lp.sec("create a message with a file in creation")
-        orig = data.get_path("d.png")
-        path = os.path.join(ac1.get_blobdir(), "d.png")
-        with open(path, "x") as fp:
-            fp.write("preparing")
-        prepared_original = chat.prepare_message_file(path)
-        assert prepared_original.is_out_preparing()
-        wait_msgs_changed(ac1, [(chat.id, prepared_original.id)])
-
-        lp.sec("create a new group")
-        chat2 = ac1.create_group_chat("newgroup")
-        wait_msgs_changed(ac1, [(0, 0)])
-
-        lp.sec("add a contact to new group")
-        chat2.add_contact(ac2)
-        wait_msgs_changed(ac1, [(chat2.id, None)])
-
-        lp.sec("forward the message while still in creation")
-        ac1.forward_messages([prepared_original], chat2)
-        forwarded_id = wait_msgs_changed(ac1, [(chat2.id, None)])
-        forwarded_msg = ac1.get_message_by_id(forwarded_id)
-        assert forwarded_msg.is_out_preparing()
-
-        lp.sec("finish creating the file and send it")
-        assert prepared_original.is_out_preparing()
-        shutil.copyfile(orig, path)
-        chat.send_prepared(prepared_original)
-        assert prepared_original.is_out_pending() or prepared_original.is_out_delivered()
-
-        lp.sec("check that both forwarded and original message are proper.")
-        wait_msgs_changed(ac1, [(chat2.id, forwarded_id), (chat.id, prepared_original.id)])
-
-        fwd_msg = ac1.get_message_by_id(forwarded_id)
-        assert fwd_msg.is_out_pending() or fwd_msg.is_out_delivered()
-
-        lp.sec("wait for both messages to be delivered to SMTP")
-        wait_msg_delivered(ac1, [(chat2.id, forwarded_id), (chat.id, prepared_original.id)])
-
-        lp.sec("wait1 for original or forwarded messages to arrive")
-        received_original = ac2._evtracker.wait_next_incoming_message()
-        assert cmp(received_original.filename, orig, shallow=False)
-
-        lp.sec("wait2 for original or forwarded messages to arrive")
-        received_copy = ac2._evtracker.wait_next_incoming_message()
-        assert received_copy.id != received_original.id
-        assert cmp(received_copy.filename, orig, shallow=False)

python/tests/test_3_offline.py

@@ -378,30 +378,6 @@ class TestOfflineChat:
         with pytest.raises(ValueError):
             chat1.send_text("msg1")
 
-    def test_prepare_message_and_send(self, ac1, chat1):
-        msg = chat1.prepare_message(Message.new_empty(chat1.account, "text"))
-        msg.set_text("hello world")
-        assert msg.text == "hello world"
-        assert msg.id > 0
-        chat1.send_prepared(msg)
-        assert "Sent" in msg.get_message_info()
-        str(msg)
-        repr(msg)
-        assert msg == ac1.get_message_by_id(msg.id)
-
-    def test_prepare_file(self, ac1, chat1):
-        blobdir = ac1.get_blobdir()
-        p = os.path.join(blobdir, "somedata.txt")
-        with open(p, "w") as f:
-            f.write("some data")
-        message = chat1.prepare_message_file(p)
-        assert message.id > 0
-        message.set_text("hello world")
-        assert message.is_out_preparing()
-        assert message.text == "hello world"
-        chat1.send_prepared(message)
-        assert "Sent" in message.get_message_info()
-
     def test_message_eq_contains(self, chat1):
         msg = chat1.send_text("msg1")
         msg2 = None
@@ -691,8 +667,7 @@ class TestOfflineChat:
         assert os.path.exists(messages[1].filename)
 
     def test_set_get_draft(self, chat1):
-        msg = Message.new_empty(chat1.account, "text")
-        msg1 = chat1.prepare_message(msg)
+        msg1 = Message.new_empty(chat1.account, "text")
         msg1.set_text("hello")
         chat1.set_draft(msg1)
         msg1.set_text("obsolete")
@@ -711,21 +686,6 @@ class TestOfflineChat:
         assert not res.is_ask_verifygroup()
         assert res.contact_id == 10
 
-    def test_quote(self, chat1):
-        """Offline quoting test"""
-        msg = Message.new_empty(chat1.account, "text")
-        msg.set_text("Multi\nline\nmessage")
-        assert msg.quoted_text is None
-
-        # Prepare message to assign it a Message-Id.
-        # Messages without Message-Id cannot be quoted.
-        msg = chat1.prepare_message(msg)
-
-        reply_msg = Message.new_empty(chat1.account, "text")
-        reply_msg.set_text("reply")
-        reply_msg.quote = msg
-        assert reply_msg.quoted_text == "Multi\nline\nmessage"
-
     def test_group_chat_many_members_add_remove(self, ac1, lp):
         lp.sec("ac1: creating group chat with 10 other members")
         chat = ac1.create_group_chat(name="title1")

python/tests/test_4_lowlevel.py

@@ -212,8 +212,13 @@ def test_logged_ac_process_ffi_failure(acfactory):
             0 / 0
 
     cap = Queue()
-    ac1.log = cap.put
+
+    # Make sure the next attempt to log an event fails.
     ac1.add_account_plugin(FailPlugin())
+
+    # Start capturing events.
+    ac1.log = cap.put
+
     # cause any event eg contact added/changed
     ac1.create_contact("something@example.org")
     res = cap.get(timeout=10)

release-date.in

@@ -1 +1 @@
-2024-12-03
+2025-01-05

src/blob.rs

@@ -279,7 +279,9 @@ impl<'a> BlobObject<'a> {
         let ext: String = name
             .chars()
             .rev()
-            .take_while(|c| !c.is_whitespace())
+            .take_while(|c| {
+                (!c.is_ascii_punctuation() || *c == '.') && !c.is_whitespace() && !c.is_control()
+            })
             .take(33)
             .collect::<Vec<_>>()
             .iter()
@@ -763,7 +765,6 @@ mod tests {
     use fs::File;
 
     use super::*;
-    use crate::chat::{self, create_group_chat, ProtectionStatus};
     use crate::message::{Message, Viewtype};
     use crate::test_utils::{self, TestContext};
 
@@ -983,6 +984,10 @@ mod tests {
         let (stem, ext) = BlobObject::sanitise_name("a. tar.tar.gz");
         assert_eq!(stem, "a. tar");
         assert_eq!(ext, ".tar.gz");
+
+        let (stem, ext) = BlobObject::sanitise_name("Guia_uso_GNB (v0.8).pdf");
+        assert_eq!(stem, "Guia_uso_GNB (v0.8)");
+        assert_eq!(ext, ".pdf");
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -1458,36 +1463,21 @@ mod tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-    async fn test_increation_in_blobdir() -> Result<()> {
-        let t = TestContext::new_alice().await;
-        let chat_id = create_group_chat(&t, ProtectionStatus::Unprotected, "abc").await?;
-
-        let file = t.get_blobdir().join("anyfile.dat");
-        fs::write(&file, b"bla").await?;
-        let mut msg = Message::new(Viewtype::File);
+    async fn test_send_gif_as_sticker() -> Result<()> {
+        let bytes = include_bytes!("../test-data/image/image100x50.gif");
+        let alice = &TestContext::new_alice().await;
+        let file = alice.get_blobdir().join("file").with_extension("gif");
+        fs::write(&file, &bytes)
+            .await
+            .context("failed to write file")?;
+        let mut msg = Message::new(Viewtype::Sticker);
         msg.set_file(file.to_str().unwrap(), None);
-        let prepared_id = chat::prepare_msg(&t, chat_id, &mut msg).await?;
-        assert_eq!(prepared_id, msg.id);
-        assert!(msg.is_increation());
-
-        let msg = Message::load_from_db(&t, prepared_id).await?;
-        assert!(msg.is_increation());
-
-        Ok(())
-    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-    async fn test_increation_not_blobdir() -> Result<()> {
-        let t = TestContext::new_alice().await;
-        let chat_id = create_group_chat(&t, ProtectionStatus::Unprotected, "abc").await?;
-        assert_ne!(t.get_blobdir().to_str(), t.dir.path().to_str());
-
-        let file = t.dir.path().join("anyfile.dat");
-        fs::write(&file, b"bla").await?;
-        let mut msg = Message::new(Viewtype::File);
-        msg.set_file(file.to_str().unwrap(), None);
-        assert!(chat::prepare_msg(&t, chat_id, &mut msg).await.is_err());
-
+        let chat = alice.get_self_chat().await;
+        let sent = alice.send_msg(chat.id, &mut msg).await;
+        let msg = Message::load_from_db(alice, sent.sender_msg_id).await?;
+        // Message::force_sticker() wasn't used, still Viewtype::Sticker is preserved because of the
+        // extension.
+        assert_eq!(msg.get_viewtype(), Viewtype::Sticker);
         Ok(())
     }
 }

src/chat.rs

@@ -28,7 +28,7 @@ use crate::contact::{self, Contact, ContactId, Origin};
 use crate::context::Context;
 use crate::debug_logging::maybe_set_logging_xdc;
 use crate::download::DownloadState;
-use crate::ephemeral::Timer as EphemeralTimer;
+use crate::ephemeral::{start_chat_ephemeral_timers, Timer as EphemeralTimer};
 use crate::events::EventType;
 use crate::html::new_html_mimepart;
 use crate::location;
@@ -688,6 +688,10 @@ impl ChatId {
             })
             .await?;
 
+        if visibility == ChatVisibility::Archived {
+            start_chat_ephemeral_timers(context, self).await?;
+        }
+
         context.emit_msgs_changed_without_ids();
         chatlist_events::emit_chatlist_changed(context);
         chatlist_events::emit_chatlist_item_changed(context, self);
@@ -743,7 +747,7 @@ impl ChatId {
                 .await?;
             if unread_cnt == 1 {
                 // Added the first unread message in the chat.
-                context.emit_msgs_changed(DC_CHAT_ID_ARCHIVED_LINK, MsgId::new(0));
+                context.emit_msgs_changed_without_msg_id(DC_CHAT_ID_ARCHIVED_LINK);
             }
             return Ok(());
         }
@@ -758,6 +762,8 @@ impl ChatId {
     /// shown.
     pub(crate) fn emit_msg_event(self, context: &Context, msg_id: MsgId, important: bool) {
         if important {
+            debug_assert!(!msg_id.is_unset());
+
             context.emit_incoming_msg(self, msg_id);
         } else {
             context.emit_msgs_changed(self, msg_id);
@@ -819,17 +825,14 @@ impl ChatId {
         };
 
         if changed {
-            context.emit_msgs_changed(
-                self,
-                if msg.is_some() {
-                    match self.get_draft_msg_id(context).await? {
-                        Some(msg_id) => msg_id,
-                        None => MsgId::new(0),
-                    }
-                } else {
-                    MsgId::new(0)
-                },
-            );
+            if msg.is_some() {
+                match self.get_draft_msg_id(context).await? {
+                    Some(msg_id) => context.emit_msgs_changed(self, msg_id),
+                    None => context.emit_msgs_changed_without_msg_id(self),
+                }
+            } else {
+                context.emit_msgs_changed_without_msg_id(self)
+            }
         }
 
         Ok(())
@@ -888,7 +891,7 @@ impl ChatId {
             _ => {
                 let blob = msg
                     .param
-                    .get_blob(Param::File, context, !msg.is_increation())
+                    .get_blob(Param::File, context)
                     .await?
                     .context("no file stored in params")?;
                 msg.param.set(Param::File, blob.as_name());
@@ -1987,9 +1990,6 @@ impl Chat {
                 .ok();
         }
 
-        // reset encrypt error state eg. for forwarding
-        msg.param.remove(Param::ErroneousE2ee);
-
         let is_bot = context.get_config_bool(Config::Bot).await?;
         msg.param
             .set_optional(Param::Bot, Some("1").filter(|_| is_bot));
@@ -2104,13 +2104,19 @@ impl Chat {
         } else {
             None
         };
+        let new_mime_headers = new_mime_headers.map(|s| new_html_mimepart(s).build().as_string());
         let new_mime_headers = new_mime_headers.or_else(|| match was_truncated {
-            true => Some(msg.text.clone()),
+            // We need to add some headers so that they are stripped before formatting HTML by
+            // `MsgId::get_html()`, not a part of the actual text. Let's add "Content-Type", it's
+            // anyway a useful metadata about the stored text.
+            true => Some(
+                "Content-Type: text/plain; charset=utf-8\r\n\r\n".to_string() + &msg.text + "\r\n",
+            ),
             false => None,
         });
         let new_mime_headers = match new_mime_headers {
             Some(h) => Some(tokio::task::block_in_place(move || {
-                buf_compress(new_html_mimepart(h).build().as_string().as_bytes())
+                buf_compress(h.as_bytes())
             })?),
             None => None,
         };
@@ -2671,31 +2677,21 @@ impl ChatIdBlocked {
     }
 }
 
-/// Prepares a message for sending.
-pub async fn prepare_msg(context: &Context, chat_id: ChatId, msg: &mut Message) -> Result<MsgId> {
-    ensure!(
-        !chat_id.is_special(),
-        "Cannot prepare message for special chat"
-    );
-
-    let msg_id = prepare_msg_common(context, chat_id, msg, MessageState::OutPreparing).await?;
-    context.emit_msgs_changed(msg.chat_id, msg.id);
-
-    Ok(msg_id)
-}
-
 async fn prepare_msg_blob(context: &Context, msg: &mut Message) -> Result<()> {
     if msg.viewtype == Viewtype::Text || msg.viewtype == Viewtype::VideochatInvitation {
         // the caller should check if the message text is empty
     } else if msg.viewtype.has_file() {
         let mut blob = msg
             .param
-            .get_blob(Param::File, context, !msg.is_increation())
+            .get_blob(Param::File, context)
             .await?
             .with_context(|| format!("attachment missing for message of type #{}", msg.viewtype))?;
         let send_as_is = msg.viewtype == Viewtype::File;
 
-        if msg.viewtype == Viewtype::File || msg.viewtype == Viewtype::Image {
+        if msg.viewtype == Viewtype::File
+            || msg.viewtype == Viewtype::Image
+            || msg.viewtype == Viewtype::Sticker && !msg.param.exists(Param::ForceSticker)
+        {
             // Correct the type, take care not to correct already very special
             // formats as GIF or VOICE.
             //
@@ -2704,7 +2700,12 @@ async fn prepare_msg_blob(context: &Context, msg: &mut Message) -> Result<()> {
             // - from FILE/IMAGE to GIF */
             if let Some((better_type, _)) = message::guess_msgtype_from_suffix(&blob.to_abs_path())
             {
-                if better_type != Viewtype::Webxdc
+                if msg.viewtype == Viewtype::Sticker {
+                    if better_type != Viewtype::Image {
+                        // UIs don't want conversions of `Sticker` to anything other than `Image`.
+                        msg.param.set_int(Param::ForceSticker, 1);
+                    }
+                } else if better_type != Viewtype::Webxdc
                     || context
                         .ensure_sendable_webxdc_file(&blob.to_abs_path())
                         .await
@@ -2765,13 +2766,92 @@ async fn prepare_msg_blob(context: &Context, msg: &mut Message) -> Result<()> {
     Ok(())
 }
 
+/// Returns whether a contact is in a chat or not.
+pub async fn is_contact_in_chat(
+    context: &Context,
+    chat_id: ChatId,
+    contact_id: ContactId,
+) -> Result<bool> {
+    // this function works for group and for normal chats, however, it is more useful
+    // for group chats.
+    // ContactId::SELF may be used to check, if the user itself is in a group
+    // chat (ContactId::SELF is not added to normal chats)
+
+    let exists = context
+        .sql
+        .exists(
+            "SELECT COUNT(*) FROM chats_contacts WHERE chat_id=? AND contact_id=?;",
+            (chat_id, contact_id),
+        )
+        .await?;
+    Ok(exists)
+}
+
+/// Sends a message object to a chat.
+///
+/// Sends the event #DC_EVENT_MSGS_CHANGED on success.
+/// However, this does not imply, the message really reached the recipient -
+/// sending may be delayed eg. due to network problems. However, from your
+/// view, you're done with the message. Sooner or later it will find its way.
+pub async fn send_msg(context: &Context, chat_id: ChatId, msg: &mut Message) -> Result<MsgId> {
+    ensure!(
+        !chat_id.is_special(),
+        "chat_id cannot be a special chat: {chat_id}"
+    );
+
+    if msg.state != MessageState::Undefined && msg.state != MessageState::OutPreparing {
+        msg.param.remove(Param::GuaranteeE2ee);
+        msg.param.remove(Param::ForcePlaintext);
+        msg.update_param(context).await?;
+    }
+
+    // protect all system messages against RTLO attacks
+    if msg.is_system_message() {
+        msg.text = sanitize_bidi_characters(&msg.text);
+    }
+
+    if !prepare_send_msg(context, chat_id, msg).await?.is_empty() {
+        if !msg.hidden {
+            context.emit_msgs_changed(msg.chat_id, msg.id);
+        }
+
+        if msg.param.exists(Param::SetLatitude) {
+            context.emit_location_changed(Some(ContactId::SELF)).await?;
+        }
+
+        context.scheduler.interrupt_smtp().await;
+    }
+
+    Ok(msg.id)
+}
+
+/// Tries to send a message synchronously.
+///
+/// Creates jobs in the `smtp` table, then drectly opens an SMTP connection and sends the
+/// message. If this fails, the jobs remain in the database for later sending.
+pub async fn send_msg_sync(context: &Context, chat_id: ChatId, msg: &mut Message) -> Result<MsgId> {
+    let rowids = prepare_send_msg(context, chat_id, msg).await?;
+    if rowids.is_empty() {
+        return Ok(msg.id);
+    }
+    let mut smtp = crate::smtp::Smtp::new();
+    for rowid in rowids {
+        send_msg_to_smtp(context, &mut smtp, rowid)
+            .await
+            .context("failed to send message, queued for later sending")?;
+    }
+    context.emit_msgs_changed(msg.chat_id, msg.id);
+    Ok(msg.id)
+}
+
 /// Prepares a message to be sent out.
-async fn prepare_msg_common(
+///
+/// Returns row ids of the `smtp` table.
+async fn prepare_send_msg(
     context: &Context,
     chat_id: ChatId,
     msg: &mut Message,
-    change_state_to: MessageState,
-) -> Result<MsgId> {
+) -> Result<Vec<i64>> {
     let mut chat = Chat::load_from_db(context, chat_id).await?;
 
     // Check if the chat can be sent to.
@@ -2815,7 +2895,7 @@ async fn prepare_msg_common(
     };
 
     // ... then change the MessageState in the message object
-    msg.state = change_state_to;
+    msg.state = MessageState::OutPending;
 
     prepare_msg_blob(context, msg).await?;
     if !msg.hidden {
@@ -2831,125 +2911,6 @@ async fn prepare_msg_common(
         .await?;
     msg.chat_id = chat_id;
 
-    Ok(msg.id)
-}
-
-/// Returns whether a contact is in a chat or not.
-pub async fn is_contact_in_chat(
-    context: &Context,
-    chat_id: ChatId,
-    contact_id: ContactId,
-) -> Result<bool> {
-    // this function works for group and for normal chats, however, it is more useful
-    // for group chats.
-    // ContactId::SELF may be used to check, if the user itself is in a group
-    // chat (ContactId::SELF is not added to normal chats)
-
-    let exists = context
-        .sql
-        .exists(
-            "SELECT COUNT(*) FROM chats_contacts WHERE chat_id=? AND contact_id=?;",
-            (chat_id, contact_id),
-        )
-        .await?;
-    Ok(exists)
-}
-
-/// Sends a message object to a chat.
-///
-/// Sends the event #DC_EVENT_MSGS_CHANGED on success.
-/// However, this does not imply, the message really reached the recipient -
-/// sending may be delayed eg. due to network problems. However, from your
-/// view, you're done with the message. Sooner or later it will find its way.
-// TODO: Do not allow ChatId to be 0, if prepare_msg had been called
-//   the caller can get it from msg.chat_id.  Forwards would need to
-//   be fixed for this somehow too.
-pub async fn send_msg(context: &Context, chat_id: ChatId, msg: &mut Message) -> Result<MsgId> {
-    if chat_id.is_unset() {
-        let forwards = msg.param.get(Param::PrepForwards);
-        if let Some(forwards) = forwards {
-            for forward in forwards.split(' ') {
-                if let Ok(msg_id) = forward.parse::<u32>().map(MsgId::new) {
-                    if let Ok(mut msg) = Message::load_from_db(context, msg_id).await {
-                        send_msg_inner(context, chat_id, &mut msg).await?;
-                    };
-                }
-            }
-            msg.param.remove(Param::PrepForwards);
-            msg.update_param(context).await?;
-        }
-        return send_msg_inner(context, chat_id, msg).await;
-    }
-
-    if msg.state != MessageState::Undefined && msg.state != MessageState::OutPreparing {
-        msg.param.remove(Param::GuaranteeE2ee);
-        msg.param.remove(Param::ForcePlaintext);
-        msg.update_param(context).await?;
-    }
-    send_msg_inner(context, chat_id, msg).await
-}
-
-/// Tries to send a message synchronously.
-///
-/// Creates jobs in the `smtp` table, then drectly opens an SMTP connection and sends the
-/// message. If this fails, the jobs remain in the database for later sending.
-pub async fn send_msg_sync(context: &Context, chat_id: ChatId, msg: &mut Message) -> Result<MsgId> {
-    let rowids = prepare_send_msg(context, chat_id, msg).await?;
-    if rowids.is_empty() {
-        return Ok(msg.id);
-    }
-    let mut smtp = crate::smtp::Smtp::new();
-    for rowid in rowids {
-        send_msg_to_smtp(context, &mut smtp, rowid)
-            .await
-            .context("failed to send message, queued for later sending")?;
-    }
-    context.emit_msgs_changed(msg.chat_id, msg.id);
-    Ok(msg.id)
-}
-
-async fn send_msg_inner(context: &Context, chat_id: ChatId, msg: &mut Message) -> Result<MsgId> {
-    // protect all system messages against RTLO attacks
-    if msg.is_system_message() {
-        msg.text = sanitize_bidi_characters(&msg.text);
-    }
-
-    if !prepare_send_msg(context, chat_id, msg).await?.is_empty() {
-        if !msg.hidden {
-            context.emit_msgs_changed(msg.chat_id, msg.id);
-        }
-
-        if msg.param.exists(Param::SetLatitude) {
-            context.emit_location_changed(Some(ContactId::SELF)).await?;
-        }
-
-        context.scheduler.interrupt_smtp().await;
-    }
-
-    Ok(msg.id)
-}
-
-/// Returns row ids of the `smtp` table.
-async fn prepare_send_msg(
-    context: &Context,
-    chat_id: ChatId,
-    msg: &mut Message,
-) -> Result<Vec<i64>> {
-    // prepare_msg() leaves the message state to OutPreparing, we
-    // only have to change the state to OutPending in this case.
-    // Otherwise we still have to prepare the message, which will set
-    // the state to OutPending.
-    if msg.state != MessageState::OutPreparing {
-        // automatically prepare normal messages
-        prepare_msg_common(context, chat_id, msg, MessageState::OutPending).await?;
-    } else {
-        // update message state of separately prepared messages
-        ensure!(
-            chat_id.is_unset() || chat_id == msg.chat_id,
-            "Inconsistent chat ID"
-        );
-        message::update_msg_state(context, msg.id, MessageState::OutPending).await?;
-    }
     let row_ids = create_send_msg_jobs(context, msg)
         .await
         .context("Failed to create send jobs")?;
@@ -2978,7 +2939,8 @@ pub(crate) async fn create_send_msg_jobs(context: &Context, msg: &mut Message) -
     // because BCC-self messages are also used to detect
     // that message was sent if SMTP server is slow to respond
     // and connection is frequently lost
-    // before receiving status line.
+    // before receiving status line. NB: This is not a problem for chatmail servers, so `BccSelf`
+    // disabled by default is fine.
     //
     // `from` must be the last addr, see `receive_imf_inner()` why.
     if context.get_config_bool(Config::BccSelf).await?
@@ -3278,19 +3240,6 @@ pub async fn get_chat_msgs_ex(
     Ok(items)
 }
 
-pub(crate) async fn marknoticed_chat_if_older_than(
-    context: &Context,
-    chat_id: ChatId,
-    timestamp: i64,
-) -> Result<()> {
-    if let Some(chat_timestamp) = chat_id.get_timestamp(context).await? {
-        if timestamp > chat_timestamp {
-            marknoticed_chat(context, chat_id).await?;
-        }
-    }
-    Ok(())
-}
-
 /// Marks all messages in the chat as noticed.
 /// If the given chat-id is the archive-link, marks all messages in all archived chats as noticed.
 pub async fn marknoticed_chat(context: &Context, chat_id: ChatId) -> Result<()> {
@@ -3302,10 +3251,10 @@ pub async fn marknoticed_chat(context: &Context, chat_id: ChatId) -> Result<()>
             .query_map(
                 "SELECT DISTINCT(m.chat_id) FROM msgs m
                     LEFT JOIN chats c ON m.chat_id=c.id
-                    WHERE m.state=10 AND m.hidden=0 AND m.chat_id>9 AND c.blocked=0 AND c.archived=1",
-                    (),
+                    WHERE m.state=10 AND m.hidden=0 AND m.chat_id>9 AND c.archived=1",
+                (),
                 |row| row.get::<_, ChatId>(0),
-                |ids| ids.collect::<Result<Vec<_>, _>>().map_err(Into::into)
+                |ids| ids.collect::<Result<Vec<_>, _>>().map_err(Into::into),
             )
             .await?;
         if chat_ids_in_archive.is_empty() {
@@ -3314,32 +3263,40 @@ pub async fn marknoticed_chat(context: &Context, chat_id: ChatId) -> Result<()>
 
         context
             .sql
-            .execute(
-                &format!(
-                    "UPDATE msgs SET state=13 WHERE state=10 AND hidden=0 AND chat_id IN ({});",
-                    sql::repeat_vars(chat_ids_in_archive.len())
-                ),
-                rusqlite::params_from_iter(&chat_ids_in_archive),
-            )
+            .transaction(|transaction| {
+                let mut stmt = transaction.prepare(
+                    "UPDATE msgs SET state=13 WHERE state=10 AND hidden=0 AND chat_id = ?",
+                )?;
+                for chat_id_in_archive in &chat_ids_in_archive {
+                    stmt.execute((chat_id_in_archive,))?;
+                }
+                Ok(())
+            })
             .await?;
+
         for chat_id_in_archive in chat_ids_in_archive {
+            start_chat_ephemeral_timers(context, chat_id_in_archive).await?;
             context.emit_event(EventType::MsgsNoticed(chat_id_in_archive));
             chatlist_events::emit_chatlist_item_changed(context, chat_id_in_archive);
         }
-    } else if context
-        .sql
-        .execute(
-            "UPDATE msgs
+    } else {
+        start_chat_ephemeral_timers(context, chat_id).await?;
+
+        if context
+            .sql
+            .execute(
+                "UPDATE msgs
             SET state=?
           WHERE state=?
             AND hidden=0
             AND chat_id=?;",
-            (MessageState::InNoticed, MessageState::InFresh, chat_id),
-        )
-        .await?
-        == 0
-    {
-        return Ok(());
+                (MessageState::InNoticed, MessageState::InFresh, chat_id),
+            )
+            .await?
+            == 0
+        {
+            return Ok(());
+        }
     }
 
     context.emit_event(EventType::MsgsNoticed(chat_id));
@@ -3411,6 +3368,7 @@ pub(crate) async fn mark_old_messages_as_noticed(
     }
 
     for c in changed_chats {
+        start_chat_ephemeral_timers(context, c).await?;
         context.emit_event(EventType::MsgsNoticed(c));
         chatlist_events::emit_chatlist_item_changed(context, c);
     }
@@ -4167,8 +4125,6 @@ pub async fn forward_msgs(context: &Context, msg_ids: &[MsgId], chat_id: ChatId)
             bail!("cannot forward drafts.");
         }
 
-        let original_param = msg.param.clone();
-
         // we tested a sort of broadcast
         // by not marking own forwarded messages as such,
         // however, this turned out to be to confusing and unclear.
@@ -4191,33 +4147,13 @@ pub async fn forward_msgs(context: &Context, msg_ids: &[MsgId], chat_id: ChatId)
         // do not leak data as group names; a default subject is generated by mimefactory
         msg.subject = "".to_string();
 
-        let new_msg_id: MsgId;
-        if msg.state == MessageState::OutPreparing {
-            new_msg_id = chat
-                .prepare_msg_raw(context, &mut msg, None, curr_timestamp)
-                .await?;
-            curr_timestamp += 1;
-            msg.param = original_param;
-            msg.id = src_msg_id;
-
-            if let Some(old_fwd) = msg.param.get(Param::PrepForwards) {
-                let new_fwd = format!("{} {}", old_fwd, new_msg_id.to_u32());
-                msg.param.set(Param::PrepForwards, new_fwd);
-            } else {
-                msg.param
-                    .set(Param::PrepForwards, new_msg_id.to_u32().to_string());
-            }
-
-            msg.update_param(context).await?;
-        } else {
-            msg.state = MessageState::OutPending;
-            new_msg_id = chat
-                .prepare_msg_raw(context, &mut msg, None, curr_timestamp)
-                .await?;
-            curr_timestamp += 1;
-            if !create_send_msg_jobs(context, &mut msg).await?.is_empty() {
-                context.scheduler.interrupt_smtp().await;
-            }
+        msg.state = MessageState::OutPending;
+        let new_msg_id = chat
+            .prepare_msg_raw(context, &mut msg, None, curr_timestamp)
+            .await?;
+        curr_timestamp += 1;
+        if !create_send_msg_jobs(context, &mut msg).await?.is_empty() {
+            context.scheduler.interrupt_smtp().await;
         }
         created_chats.push(chat_id);
         created_msgs.push(new_msg_id);
@@ -4712,7 +4648,7 @@ impl Context {
     /// a noticed chat is archived. Emitting events should be cheap, a false-positive `MsgsChanged`
     /// is ok.
     pub(crate) fn on_archived_chats_maybe_noticed(&self) {
-        self.emit_msgs_changed(DC_CHAT_ID_ARCHIVED_LINK, MsgId::new(0));
+        self.emit_msgs_changed_without_msg_id(DC_CHAT_ID_ARCHIVED_LINK);
     }
 }
 
@@ -4724,7 +4660,7 @@ mod tests {
     use crate::headerdef::HeaderDef;
     use crate::message::delete_msgs;
     use crate::receive_imf::receive_imf;
-    use crate::test_utils::{sync, TestContext, TestContextManager};
+    use crate::test_utils::{sync, TestContext, TestContextManager, TimeShiftFalsePositiveNote};
     use strum::IntoEnumIterator;
     use tokio::fs;
 
@@ -4860,15 +4796,12 @@ mod tests {
         assert_eq!(test.text, "hello2".to_string());
         assert_eq!(test.state, MessageState::OutDraft);
 
-        let id_after_prepare = prepare_msg(&t, *chat_id, &mut msg).await?;
-        assert_eq!(id_after_prepare, id_after_1st_set);
-        let test = Message::load_from_db(&t, id_after_prepare).await?;
-        assert_eq!(test.state, MessageState::OutPreparing);
-        assert!(!test.hidden); // sent draft must no longer be hidden
-
         let id_after_send = send_msg(&t, *chat_id, &mut msg).await?;
         assert_eq!(id_after_send, id_after_1st_set);
 
+        let test = Message::load_from_db(&t, id_after_send).await?;
+        assert!(!test.hidden); // sent draft must no longer be hidden
+
         Ok(())
     }
 
@@ -5268,6 +5201,8 @@ mod tests {
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_modify_chat_disordered() -> Result<()> {
+        let _n = TimeShiftFalsePositiveNote;
+
         // Alice creates a group with Bob, Claire and Daisy and then removes Claire and Daisy
         // (sleep() is needed as otherwise smeared time from Alice looks to Bob like messages from the future which are all set to "now" then)
         let alice = TestContext::new_alice().await;
@@ -5618,7 +5553,6 @@ mod tests {
 
         let mut msg = Message::new_text("message text".to_string());
         assert!(send_msg(&t, device_chat_id, &mut msg).await.is_err());
-        assert!(prepare_msg(&t, device_chat_id, &mut msg).await.is_err());
 
         let msg_id = add_device_msg(&t, None, Some(&mut msg)).await.unwrap();
         assert!(forward_msgs(&t, &[msg_id], device_chat_id).await.is_err());

src/chatlist.rs

@@ -661,7 +661,7 @@ mod tests {
         let contacts = get_chat_contacts(&t, chat_id).await?;
         let contact_id = *contacts.first().unwrap();
         let chat = Chat::load_from_db(&t, chat_id).await?;
-        assert_eq!(chat.get_name(), "Bob Authname");
+        assert_eq!(chat.get_name(), "~Bob Authname");
 
         // check, the one-to-one-chat can be found using chatlist search query
         let chats = Chatlist::try_load(&t, 0, Some("bob authname"), None).await?;
@@ -682,7 +682,7 @@ mod tests {
         let test_id = Contact::create(&t, "", "bob@example.org").await?;
         assert_eq!(contact_id, test_id);
         let chat = Chat::load_from_db(&t, chat_id).await?;
-        assert_eq!(chat.get_name(), "Bob Authname");
+        assert_eq!(chat.get_name(), "~Bob Authname");
         let chats = Chatlist::try_load(&t, 0, Some("bob authname"), None).await?;
         assert_eq!(chats.len(), 1);
         let chats = Chatlist::try_load(&t, 0, Some("bob nickname"), None).await?;

src/config.rs

@@ -143,7 +143,7 @@ pub enum Config {
     /// Send BCC copy to self.
     ///
     /// Should be enabled for multidevice setups.
-    #[strum(props(default = "1"))]
+    /// Default is 0 for chatmail accounts before a backup export, 1 otherwise.
     BccSelf,
 
     /// True if encryption is preferred according to Autocrypt standard.
@@ -202,7 +202,7 @@ pub enum Config {
     /// Value 1 is treated as "delete at once": messages are deleted
     /// immediately, without moving to DeltaChat folder.
     ///
-    /// Default is 1 for chatmail accounts before a backup export, 0 otherwise.
+    /// Default is 1 for chatmail accounts without `BccSelf`, 0 otherwise.
     DeleteServerAfter,
 
     /// Timer in seconds after which the message is deleted from the
@@ -519,11 +519,19 @@ impl Context {
 
         // Default values
         let val = match key {
-            Config::ConfiguredInboxFolder => Some("INBOX"),
-            Config::DeleteServerAfter => match Box::pin(self.is_chatmail()).await? {
-                false => Some("0"),
-                true => Some("1"),
+            Config::BccSelf => match Box::pin(self.is_chatmail()).await? {
+                false => Some("1"),
+                true => Some("0"),
             },
+            Config::ConfiguredInboxFolder => Some("INBOX"),
+            Config::DeleteServerAfter => {
+                match !Box::pin(self.get_config_bool(Config::BccSelf)).await?
+                    && Box::pin(self.is_chatmail()).await?
+                {
+                    true => Some("1"),
+                    false => Some("0"),
+                }
+            }
             _ => key.get_str("default"),
         };
         Ok(val.map(|s| s.to_string()))
@@ -1105,6 +1113,28 @@ mod tests {
         Ok(())
     }
 
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_delete_server_after_default() -> Result<()> {
+        let t = &TestContext::new_alice().await;
+
+        // Check that the settings are displayed correctly.
+        assert_eq!(t.get_config(Config::BccSelf).await?, Some("1".to_string()));
+        assert_eq!(
+            t.get_config(Config::DeleteServerAfter).await?,
+            Some("0".to_string())
+        );
+
+        // Leaving emails on the server even w/o `BccSelf` is a good default at least because other
+        // MUAs do so even if the server doesn't save sent messages to some sentbox (like Gmail
+        // does).
+        t.set_config_bool(Config::BccSelf, false).await?;
+        assert_eq!(
+            t.get_config(Config::DeleteServerAfter).await?,
+            Some("0".to_string())
+        );
+        Ok(())
+    }
+
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_sync() -> Result<()> {
         let alice0 = TestContext::new_alice().await;

src/configure.rs

@@ -452,8 +452,9 @@ async fn configure(ctx: &Context, param: &EnteredLoginParam) -> Result<Configure
     imap.configure_folders(ctx, &mut imap_session, create_mvbox)
         .await?;
 
+    let create = true;
     imap_session
-        .select_with_uidvalidity(ctx, "INBOX")
+        .select_with_uidvalidity(ctx, "INBOX", create)
         .await
         .context("could not read INBOX status")?;
 

src/contact.rs

@@ -129,17 +129,14 @@ impl ContactId {
     ) -> Result<()> {
         context
             .sql
-            .execute(
-                &format!(
-                    "UPDATE contacts SET origin=? WHERE id IN ({}) AND origin<?",
-                    sql::repeat_vars(ids.len())
-                ),
-                rusqlite::params_from_iter(
-                    params_iter(&[origin])
-                        .chain(params_iter(ids))
-                        .chain(params_iter(&[origin])),
-                ),
-            )
+            .transaction(|transaction| {
+                let mut stmt = transaction
+                    .prepare("UPDATE contacts SET origin=?1 WHERE id = ?2 AND origin < ?1")?;
+                for id in ids {
+                    stmt.execute((origin, id))?;
+                }
+                Ok(())
+            })
             .await?;
         Ok(())
     }
@@ -924,7 +921,7 @@ impl Contact {
                             let chat_name = if !name.is_empty() {
                                 name
                             } else if !authname.is_empty() {
-                                authname
+                                format!("~{}", authname)
                             } else {
                                 addr
                             };
@@ -1368,14 +1365,14 @@ impl Contact {
     ///
     /// This name is typically used in lists.
     /// To get the name editable in a formular, use `Contact::get_name`.
-    pub fn get_display_name(&self) -> &str {
+    pub fn get_display_name(&self) -> String {
         if !self.name.is_empty() {
-            return &self.name;
+            return self.name.clone();
         }
         if !self.authname.is_empty() {
-            return &self.authname;
+            return format!("~{}", self.authname);
         }
-        &self.addr
+        self.addr.clone()
     }
 
     /// Get a summary of authorized name and address.
@@ -1407,7 +1404,7 @@ impl Contact {
         if !self.name.is_empty() {
             format!("{} ({})", self.name, self.addr)
         } else if !self.authname.is_empty() {
-            format!("{} ({})", self.authname, self.addr)
+            format!("~{} ({})", self.authname, self.addr)
         } else {
             (&self.addr).into()
         }
@@ -1984,7 +1981,7 @@ mod tests {
     use crate::chat::{get_chat_contacts, send_text_msg, Chat};
     use crate::chatlist::Chatlist;
     use crate::receive_imf::receive_imf;
-    use crate::test_utils::{self, TestContext, TestContextManager};
+    use crate::test_utils::{self, TestContext, TestContextManager, TimeShiftFalsePositiveNote};
 
     #[test]
     fn test_contact_id_values() {
@@ -2056,7 +2053,7 @@ mod tests {
         let contact = Contact::get_by_id(&context.ctx, id).await.unwrap();
         assert_eq!(contact.get_name(), "");
         assert_eq!(contact.get_authname(), "bob");
-        assert_eq!(contact.get_display_name(), "bob");
+        assert_eq!(contact.get_display_name(), "~bob");
 
         // Search by name.
         let contacts = Contact::get_all(&context.ctx, 0, Some("bob")).await?;
@@ -2188,7 +2185,7 @@ mod tests {
         assert_eq!(contact_id, contact_id_test);
         assert_eq!(sth_modified, Modifier::Modified);
         let contact = Contact::get_by_id(&t, contact_id).await.unwrap();
-        assert_eq!(contact.get_name_n_addr(), "m. serious (three@drei.sam)");
+        assert_eq!(contact.get_name_n_addr(), "~m. serious (three@drei.sam)");
         assert!(!contact.is_blocked());
 
         // manually edit name of third contact (does not changed authorized name)
@@ -2279,14 +2276,14 @@ mod tests {
         )
         .await?;
         let chat_id = t.get_last_msg().await.get_chat_id();
-        assert_eq!(Chat::load_from_db(&t, chat_id).await?.name, "Flobbyfoo");
+        assert_eq!(Chat::load_from_db(&t, chat_id).await?.name, "~Flobbyfoo");
         let chatlist = Chatlist::try_load(&t, 0, Some("flobbyfoo"), None).await?;
         assert_eq!(chatlist.len(), 1);
         let contact = Contact::get_by_id(&t, *contact_id).await?;
         assert_eq!(contact.get_authname(), "Flobbyfoo");
         assert_eq!(contact.get_name(), "");
-        assert_eq!(contact.get_display_name(), "Flobbyfoo");
-        assert_eq!(contact.get_name_n_addr(), "Flobbyfoo (f@example.org)");
+        assert_eq!(contact.get_display_name(), "~Flobbyfoo");
+        assert_eq!(contact.get_name_n_addr(), "~Flobbyfoo (f@example.org)");
         let contacts = Contact::get_all(&t, 0, Some("f@example.org")).await?;
         assert_eq!(contacts.len(), 1);
         let contacts = Contact::get_all(&t, 0, Some("flobbyfoo")).await?;
@@ -2307,7 +2304,7 @@ mod tests {
         )
         .await?;
         let chat_id = t.get_last_msg().await.get_chat_id();
-        assert_eq!(Chat::load_from_db(&t, chat_id).await?.name, "Foo Flobby");
+        assert_eq!(Chat::load_from_db(&t, chat_id).await?.name, "~Foo Flobby");
         let chatlist = Chatlist::try_load(&t, 0, Some("Flobbyfoo"), None).await?;
         assert_eq!(chatlist.len(), 0);
         let chatlist = Chatlist::try_load(&t, 0, Some("Foo Flobby"), None).await?;
@@ -2315,8 +2312,8 @@ mod tests {
         let contact = Contact::get_by_id(&t, *contact_id).await?;
         assert_eq!(contact.get_authname(), "Foo Flobby");
         assert_eq!(contact.get_name(), "");
-        assert_eq!(contact.get_display_name(), "Foo Flobby");
-        assert_eq!(contact.get_name_n_addr(), "Foo Flobby (f@example.org)");
+        assert_eq!(contact.get_display_name(), "~Foo Flobby");
+        assert_eq!(contact.get_name_n_addr(), "~Foo Flobby (f@example.org)");
         let contacts = Contact::get_all(&t, 0, Some("f@example.org")).await?;
         assert_eq!(contacts.len(), 1);
         let contacts = Contact::get_all(&t, 0, Some("flobbyfoo")).await?;
@@ -2442,7 +2439,7 @@ mod tests {
         let contact = Contact::get_by_id(&t, contact_id).await.unwrap();
         assert_eq!(contact.get_authname(), "bob1");
         assert_eq!(contact.get_name(), "");
-        assert_eq!(contact.get_display_name(), "bob1");
+        assert_eq!(contact.get_display_name(), "~bob1");
 
         // incoming mail `From: bob2 <bob@example.org>` - this should update authname
         let (contact_id, sth_modified) = Contact::add_or_lookup(
@@ -2458,7 +2455,7 @@ mod tests {
         let contact = Contact::get_by_id(&t, contact_id).await.unwrap();
         assert_eq!(contact.get_authname(), "bob2");
         assert_eq!(contact.get_name(), "");
-        assert_eq!(contact.get_display_name(), "bob2");
+        assert_eq!(contact.get_display_name(), "~bob2");
 
         // manually edit name to "bob3" - authname should be still be "bob2" as given in `From:` above
         let contact_id = Contact::create(&t, "bob3", "bob@example.org")
@@ -2513,7 +2510,7 @@ mod tests {
         let contact = Contact::get_by_id(&t, contact_id).await.unwrap();
         assert_eq!(contact.get_authname(), "claire1");
         assert_eq!(contact.get_name(), "");
-        assert_eq!(contact.get_display_name(), "claire1");
+        assert_eq!(contact.get_display_name(), "~claire1");
 
         // incoming mail `From: claire2 <claire@example.org>` - this should update authname
         let (contact_id_same, sth_modified) = Contact::add_or_lookup(
@@ -2529,7 +2526,7 @@ mod tests {
         let contact = Contact::get_by_id(&t, contact_id).await.unwrap();
         assert_eq!(contact.get_authname(), "claire2");
         assert_eq!(contact.get_name(), "");
-        assert_eq!(contact.get_display_name(), "claire2");
+        assert_eq!(contact.get_display_name(), "~claire2");
     }
 
     /// Regression test.
@@ -2550,7 +2547,7 @@ mod tests {
         .await?;
         assert_eq!(sth_modified, Modifier::Created);
         let contact = Contact::get_by_id(&t, contact_id).await?;
-        assert_eq!(contact.get_display_name(), "Bob");
+        assert_eq!(contact.get_display_name(), "~Bob");
 
         // Incoming message from someone else with "Not Bob" <bob@example.org> in the "To:" field.
         let (contact_id_same, sth_modified) = Contact::add_or_lookup(
@@ -2563,7 +2560,7 @@ mod tests {
         assert_eq!(contact_id, contact_id_same);
         assert_eq!(sth_modified, Modifier::Modified);
         let contact = Contact::get_by_id(&t, contact_id).await?;
-        assert_eq!(contact.get_display_name(), "Not Bob");
+        assert_eq!(contact.get_display_name(), "~Not Bob");
 
         // Incoming message from Bob, changing the name back.
         let (contact_id_same, sth_modified) = Contact::add_or_lookup(
@@ -2576,7 +2573,7 @@ mod tests {
         assert_eq!(contact_id, contact_id_same);
         assert_eq!(sth_modified, Modifier::Modified); // This was None until the bugfix
         let contact = Contact::get_by_id(&t, contact_id).await?;
-        assert_eq!(contact.get_display_name(), "Bob");
+        assert_eq!(contact.get_display_name(), "~Bob");
 
         Ok(())
     }
@@ -2613,7 +2610,7 @@ mod tests {
         let contact = Contact::get_by_id(&t, contact_id).await.unwrap();
         assert_eq!(contact.get_authname(), "dave2");
         assert_eq!(contact.get_name(), "");
-        assert_eq!(contact.get_display_name(), "dave2");
+        assert_eq!(contact.get_display_name(), "~dave2");
     }
 
     #[test]
@@ -2915,6 +2912,8 @@ Hi."#;
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_was_seen_recently() -> Result<()> {
+        let _n = TimeShiftFalsePositiveNote;
+
         let mut tcm = TestContextManager::new();
         let alice = tcm.alice().await;
         let bob = tcm.bob().await;
@@ -2930,18 +2929,7 @@ Hi."#;
         bob.recv_msg(&sent_msg).await;
         let contact = Contact::get_by_id(&bob, *contacts.first().unwrap()).await?;
 
-        let green = nu_ansi_term::Color::Green.normal();
-        assert!(
-            contact.was_seen_recently(),
-            "{}",
-            green.paint(
-                "\nNOTE: This test failure is probably a false-positive, caused by tests running in parallel.
-The issue is that `SystemTime::shift()` (a utility function for tests) changes the time for all threads doing tests, and not only for the running test.
-Until the false-positive is fixed:
-- Use `cargo test -- --test-threads 1` instead of `cargo test`
-- Or use `cargo nextest run` (install with `cargo install cargo-nextest --locked`)\n"
-            )
-        );
+        assert!(contact.was_seen_recently());
 
         let self_contact = Contact::get_by_id(&bob, ContactId::SELF).await?;
         assert!(!self_contact.was_seen_recently());

src/context.rs

@@ -553,23 +553,7 @@ impl Context {
 
         if self.scheduler.is_running().await {
             self.scheduler.maybe_network().await;
-
-            // Wait until fetching is finished.
-            // Ideally we could wait for connectivity change events,
-            // but sleep loop is good enough.
-
-            // First 100 ms sleep in chunks of 10 ms.
-            for _ in 0..10 {
-                if self.all_work_done().await {
-                    break;
-                }
-                tokio::time::sleep(std::time::Duration::from_millis(10)).await;
-            }
-
-            // If we are not finished in 100 ms, keep waking up every 100 ms.
-            while !self.all_work_done().await {
-                tokio::time::sleep(std::time::Duration::from_millis(100)).await;
-            }
+            self.wait_for_all_work_done().await;
         } else {
             // Pause the scheduler to ensure another connection does not start
             // while we are fetching on a dedicated connection.
@@ -659,14 +643,36 @@ impl Context {
     }
 
     /// Emits a MsgsChanged event with specified chat and message ids
+    ///
+    /// If IDs are unset, [`Self::emit_msgs_changed_without_ids`]
+    /// or [`Self::emit_msgs_changed_without_msg_id`] should be used
+    /// instead of this function.
     pub fn emit_msgs_changed(&self, chat_id: ChatId, msg_id: MsgId) {
+        debug_assert!(!chat_id.is_unset());
+        debug_assert!(!msg_id.is_unset());
+
         self.emit_event(EventType::MsgsChanged { chat_id, msg_id });
         chatlist_events::emit_chatlist_changed(self);
         chatlist_events::emit_chatlist_item_changed(self, chat_id);
     }
 
+    /// Emits a MsgsChanged event with specified chat and without message id.
+    pub fn emit_msgs_changed_without_msg_id(&self, chat_id: ChatId) {
+        debug_assert!(!chat_id.is_unset());
+
+        self.emit_event(EventType::MsgsChanged {
+            chat_id,
+            msg_id: MsgId::new(0),
+        });
+        chatlist_events::emit_chatlist_changed(self);
+        chatlist_events::emit_chatlist_item_changed(self, chat_id);
+    }
+
     /// Emits an IncomingMsg event with specified chat and message ids
     pub fn emit_incoming_msg(&self, chat_id: ChatId, msg_id: MsgId) {
+        debug_assert!(!chat_id.is_unset());
+        debug_assert!(!msg_id.is_unset());
+
         self.emit_event(EventType::IncomingMsg { chat_id, msg_id });
         chatlist_events::emit_chatlist_changed(self);
         chatlist_events::emit_chatlist_item_changed(self, chat_id);

src/download.rs

@@ -3,7 +3,7 @@
 use std::cmp::max;
 use std::collections::BTreeMap;
 
-use anyhow::{anyhow, bail, Result};
+use anyhow::{anyhow, bail, ensure, Result};
 use deltachat_derive::{FromSql, ToSql};
 use serde::{Deserialize, Serialize};
 
@@ -201,7 +201,11 @@ impl Session {
             bail!("Attempt to fetch UID 0");
         }
 
-        self.select_with_uidvalidity(context, folder).await?;
+        let create = false;
+        let folder_exists = self
+            .select_with_uidvalidity(context, folder, create)
+            .await?;
+        ensure!(folder_exists, "No folder {folder}");
 
         // we are connected, and the folder is selected
         info!(context, "Downloading message {}/{} fully...", folder, uid);

src/e2ee.rs

@@ -40,20 +40,17 @@ impl EncryptHelper {
 
     /// Determines if we can and should encrypt.
     ///
-    /// For encryption to be enabled, `e2ee_guaranteed` should be true, or strictly more than a half
-    /// of peerstates should prefer encryption. Own preference is counted equally to peer
-    /// preferences, even if message copy is not sent to self.
-    ///
     /// `e2ee_guaranteed` should be set to true for replies to encrypted messages (as required by
     /// Autocrypt Level 1, version 1.1) and for messages sent in protected groups.
     ///
     /// Returns an error if `e2ee_guaranteed` is true, but one or more keys are missing.
-    pub fn should_encrypt(
+    pub(crate) async fn should_encrypt(
         &self,
         context: &Context,
         e2ee_guaranteed: bool,
         peerstates: &[(Option<Peerstate>, String)],
     ) -> Result<bool> {
+        let is_chatmail = context.is_chatmail().await?;
         let mut prefer_encrypt_count = if self.prefer_encrypt == EncryptPreference::Mutual {
             1
         } else {
@@ -64,10 +61,15 @@ impl EncryptHelper {
                 Some(peerstate) => {
                     let prefer_encrypt = peerstate.prefer_encrypt;
                     info!(context, "Peerstate for {addr:?} is {prefer_encrypt}.");
-                    match peerstate.prefer_encrypt {
-                        EncryptPreference::NoPreference | EncryptPreference::Reset => {}
-                        EncryptPreference::Mutual => prefer_encrypt_count += 1,
-                    };
+                    if match peerstate.prefer_encrypt {
+                        EncryptPreference::NoPreference | EncryptPreference::Reset => {
+                            (peerstate.prefer_encrypt != EncryptPreference::Reset || is_chatmail)
+                                && self.prefer_encrypt == EncryptPreference::Mutual
+                        }
+                        EncryptPreference::Mutual => true,
+                    } {
+                        prefer_encrypt_count += 1;
+                    }
                 }
                 None => {
                     let msg = format!("Peerstate for {addr:?} missing, cannot encrypt");
@@ -170,9 +172,11 @@ pub async fn ensure_secret_key_exists(context: &Context) -> Result<()> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::chat::send_text_msg;
     use crate::key::DcKey;
     use crate::message::{Message, Viewtype};
     use crate::param::Param;
+    use crate::receive_imf::receive_imf;
     use crate::test_utils::{bob_keypair, TestContext, TestContextManager};
 
     mod ensure_secret_key_exists {
@@ -320,29 +324,109 @@ Sent with my Delta Chat Messenger: https://delta.chat";
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-    async fn test_should_encrypt() {
+    async fn test_should_encrypt() -> Result<()> {
         let t = TestContext::new_alice().await;
+        assert!(t.get_config_bool(Config::E2eeEnabled).await?);
         let encrypt_helper = EncryptHelper::new(&t).await.unwrap();
 
-        // test with EncryptPreference::NoPreference:
-        // if e2ee_eguaranteed is unset, there is no encryption as not more than half of peers want encryption
         let ps = new_peerstates(EncryptPreference::NoPreference);
-        assert!(encrypt_helper.should_encrypt(&t, true, &ps).unwrap());
-        assert!(!encrypt_helper.should_encrypt(&t, false, &ps).unwrap());
+        assert!(encrypt_helper.should_encrypt(&t, true, &ps).await?);
+        // Own preference is `Mutual` and we have the peer's key.
+        assert!(encrypt_helper.should_encrypt(&t, false, &ps).await?);
 
-        // test with EncryptPreference::Reset
         let ps = new_peerstates(EncryptPreference::Reset);
-        assert!(encrypt_helper.should_encrypt(&t, true, &ps).unwrap());
-        assert!(!encrypt_helper.should_encrypt(&t, false, &ps).unwrap());
+        assert!(encrypt_helper.should_encrypt(&t, true, &ps).await?);
+        assert!(!encrypt_helper.should_encrypt(&t, false, &ps).await?);
 
-        // test with EncryptPreference::Mutual (self is also Mutual)
         let ps = new_peerstates(EncryptPreference::Mutual);
-        assert!(encrypt_helper.should_encrypt(&t, true, &ps).unwrap());
-        assert!(encrypt_helper.should_encrypt(&t, false, &ps).unwrap());
+        assert!(encrypt_helper.should_encrypt(&t, true, &ps).await?);
+        assert!(encrypt_helper.should_encrypt(&t, false, &ps).await?);
 
         // test with missing peerstate
         let ps = vec![(None, "bob@foo.bar".to_string())];
-        assert!(encrypt_helper.should_encrypt(&t, true, &ps).is_err());
-        assert!(!encrypt_helper.should_encrypt(&t, false, &ps).unwrap());
+        assert!(encrypt_helper.should_encrypt(&t, true, &ps).await.is_err());
+        assert!(!encrypt_helper.should_encrypt(&t, false, &ps).await?);
+        Ok(())
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_should_encrypt_e2ee_disabled() -> Result<()> {
+        let t = &TestContext::new_alice().await;
+        t.set_config_bool(Config::E2eeEnabled, false).await?;
+        let encrypt_helper = EncryptHelper::new(t).await.unwrap();
+
+        let ps = new_peerstates(EncryptPreference::NoPreference);
+        assert!(!encrypt_helper.should_encrypt(t, false, &ps).await?);
+
+        let ps = new_peerstates(EncryptPreference::Reset);
+        assert!(encrypt_helper.should_encrypt(t, true, &ps).await?);
+
+        let mut ps = new_peerstates(EncryptPreference::Mutual);
+        // Own preference is `NoPreference` and there's no majority with `Mutual`.
+        assert!(!encrypt_helper.should_encrypt(t, false, &ps).await?);
+        // Now the majority wants to encrypt. Let's encrypt, anyway there are other cases when we
+        // can't send unencrypted, e.g. protected groups.
+        ps.push(ps[0].clone());
+        assert!(encrypt_helper.should_encrypt(t, false, &ps).await?);
+
+        // Test with missing peerstate.
+        let ps = vec![(None, "bob@foo.bar".to_string())];
+        assert!(encrypt_helper.should_encrypt(t, true, &ps).await.is_err());
+        Ok(())
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_chatmail_prefers_to_encrypt() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let alice = &tcm.alice().await;
+        let bob = &tcm.bob().await;
+        bob.set_config_bool(Config::IsChatmail, true).await?;
+
+        let bob_chat_id = tcm
+            .send_recv_accept(alice, bob, "Hello from DC")
+            .await
+            .chat_id;
+        receive_imf(
+            bob,
+            b"From: alice@example.org\n\
+            To: bob@example.net\n\
+            Message-ID: <2222@example.org>\n\
+            Date: Sun, 22 Mar 3000 22:37:58 +0000\n\
+            \n\
+            Hello from another MUA\n",
+            false,
+        )
+        .await?;
+        send_text_msg(bob, bob_chat_id, "hi".to_string()).await?;
+        let sent_msg = bob.pop_sent_msg().await;
+        let msg = Message::load_from_db(bob, sent_msg.sender_msg_id).await?;
+        assert!(msg.get_showpadlock());
+        Ok(())
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_chatmail_can_send_unencrypted() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let bob = &tcm.bob().await;
+        bob.set_config_bool(Config::IsChatmail, true).await?;
+        let bob_chat_id = receive_imf(
+            bob,
+            b"From: alice@example.org\n\
+            To: bob@example.net\n\
+            Message-ID: <2222@example.org>\n\
+            Date: Sun, 22 Mar 3000 22:37:58 +0000\n\
+            \n\
+            Hello\n",
+            false,
+        )
+        .await?
+        .unwrap()
+        .chat_id;
+        bob_chat_id.accept(bob).await?;
+        send_text_msg(bob, bob_chat_id, "hi".to_string()).await?;
+        let sent_msg = bob.pop_sent_msg().await;
+        let msg = Message::load_from_db(bob, sent_msg.sender_msg_id).await?;
+        assert!(!msg.get_showpadlock());
+        Ok(())
     }
 }

src/ephemeral.rs

@@ -84,7 +84,6 @@ use crate::location;
 use crate::log::LogExt;
 use crate::message::{Message, MessageState, MsgId, Viewtype};
 use crate::mimeparser::SystemMessage;
-use crate::sql::{self, params_iter};
 use crate::stock_str;
 use crate::tools::{duration_to_str, time, SystemTime};
 
@@ -329,23 +328,44 @@ pub(crate) async fn start_ephemeral_timers_msgids(
     msg_ids: &[MsgId],
 ) -> Result<()> {
     let now = time();
-    let count = context
+    let should_interrupt =
+    context
+        .sql
+        .transaction(move |transaction| {
+            let mut should_interrupt = false;
+            let mut stmt =
+                transaction.prepare(
+                    "UPDATE msgs SET ephemeral_timestamp = ?1 + ephemeral_timer
+                     WHERE (ephemeral_timestamp == 0 OR ephemeral_timestamp > ?1 + ephemeral_timer) AND ephemeral_timer > 0
+                     AND id=?2")?;
+            for msg_id in msg_ids {
+                should_interrupt |= stmt.execute((now, msg_id))? > 0;
+            }
+            Ok(should_interrupt)
+        }).await?;
+    if should_interrupt {
+        context.scheduler.interrupt_ephemeral_task().await;
+    }
+    Ok(())
+}
+
+/// Starts ephemeral timer for all messages in the chat.
+///
+/// This should be called when chat is marked as noticed.
+pub(crate) async fn start_chat_ephemeral_timers(context: &Context, chat_id: ChatId) -> Result<()> {
+    let now = time();
+    let should_interrupt = context
         .sql
         .execute(
-            &format!(
-                "UPDATE msgs SET ephemeral_timestamp = ? + ephemeral_timer
-         WHERE (ephemeral_timestamp == 0 OR ephemeral_timestamp > ? + ephemeral_timer) AND ephemeral_timer > 0
-         AND id IN ({})",
-                sql::repeat_vars(msg_ids.len())
-            ),
-            rusqlite::params_from_iter(
-                std::iter::once(&now as &dyn crate::sql::ToSql)
-                    .chain(std::iter::once(&now as &dyn crate::sql::ToSql))
-                    .chain(params_iter(msg_ids)),
-            ),
+            "UPDATE msgs SET ephemeral_timestamp = ?1 + ephemeral_timer
+             WHERE chat_id = ?2
+             AND ephemeral_timer > 0
+             AND (ephemeral_timestamp == 0 OR ephemeral_timestamp > ?1 + ephemeral_timer)",
+            (now, chat_id),
         )
-        .await?;
-    if count > 0 {
+        .await?
+        > 0;
+    if should_interrupt {
         context.scheduler.interrupt_ephemeral_task().await;
     }
     Ok(())
@@ -482,7 +502,7 @@ pub(crate) async fn delete_expired_messages(context: &Context, now: i64) -> Resu
         }
 
         for modified_chat_id in modified_chat_ids {
-            context.emit_msgs_changed(modified_chat_id, MsgId::new(0));
+            context.emit_msgs_changed_without_msg_id(modified_chat_id);
         }
 
         for msg_id in webxdc_deleted {
@@ -695,7 +715,9 @@ pub(crate) async fn start_ephemeral_timers(context: &Context) -> Result<()> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::chat::{marknoticed_chat, set_muted, ChatVisibility, MuteDuration};
     use crate::config::Config;
+    use crate::constants::DC_CHAT_ID_ARCHIVED_LINK;
     use crate::download::DownloadState;
     use crate::location;
     use crate::message::markseen_msgs;
@@ -930,7 +952,6 @@ mod tests {
 
         // Alice sends a text message.
         let mut msg = Message::new(Viewtype::Text);
-        chat::prepare_msg(&alice.ctx, chat_alice, &mut msg).await?;
         chat::send_msg(&alice.ctx, chat_alice, &mut msg).await?;
         let sent = alice.pop_sent_msg().await;
 
@@ -957,14 +978,12 @@ mod tests {
 
         // Alice sends message to Bob
         let mut msg = Message::new(Viewtype::Text);
-        chat::prepare_msg(&alice.ctx, chat_alice, &mut msg).await?;
         chat::send_msg(&alice.ctx, chat_alice, &mut msg).await?;
         let sent = alice.pop_sent_msg().await;
         bob.recv_msg(&sent).await;
 
         // Alice sends second message to Bob, with no timer
         let mut msg = Message::new(Viewtype::Text);
-        chat::prepare_msg(&alice.ctx, chat_alice, &mut msg).await?;
         chat::send_msg(&alice.ctx, chat_alice, &mut msg).await?;
         let sent = alice.pop_sent_msg().await;
 
@@ -1425,4 +1444,77 @@ mod tests {
 
         Ok(())
     }
+
+    /// Tests that ephemeral timer is started when the chat is noticed.
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_noticed_ephemeral_timer() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let alice = &tcm.alice().await;
+        let bob = &tcm.bob().await;
+
+        let chat = alice.create_chat(bob).await;
+        let duration = 60;
+        chat.id
+            .set_ephemeral_timer(alice, Timer::Enabled { duration })
+            .await?;
+        let bob_received_message = tcm.send_recv(alice, bob, "Hello!").await;
+
+        marknoticed_chat(bob, bob_received_message.chat_id).await?;
+        SystemTime::shift(Duration::from_secs(100));
+
+        delete_expired_messages(bob, time()).await?;
+
+        assert!(Message::load_from_db_optional(bob, bob_received_message.id)
+            .await?
+            .is_none());
+        Ok(())
+    }
+
+    /// Tests that archiving the chat starts ephemeral timer.
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_archived_ephemeral_timer() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let alice = &tcm.alice().await;
+        let bob = &tcm.bob().await;
+
+        let chat = alice.create_chat(bob).await;
+        let duration = 60;
+        chat.id
+            .set_ephemeral_timer(alice, Timer::Enabled { duration })
+            .await?;
+        let bob_received_message = tcm.send_recv(alice, bob, "Hello!").await;
+
+        bob_received_message
+            .chat_id
+            .set_visibility(bob, ChatVisibility::Archived)
+            .await?;
+        SystemTime::shift(Duration::from_secs(100));
+
+        delete_expired_messages(bob, time()).await?;
+
+        assert!(Message::load_from_db_optional(bob, bob_received_message.id)
+            .await?
+            .is_none());
+
+        // Bob mutes the chat so it is not unarchived.
+        set_muted(bob, bob_received_message.chat_id, MuteDuration::Forever).await?;
+
+        // Now test that for already archived chat
+        // timer is started if all archived chats are marked as noticed.
+        let bob_received_message_2 = tcm.send_recv(alice, bob, "Hello again!").await;
+        assert_eq!(bob_received_message_2.state, MessageState::InFresh);
+
+        marknoticed_chat(bob, DC_CHAT_ID_ARCHIVED_LINK).await?;
+        SystemTime::shift(Duration::from_secs(100));
+
+        delete_expired_messages(bob, time()).await?;
+
+        assert!(
+            Message::load_from_db_optional(bob, bob_received_message_2.id)
+                .await?
+                .is_none()
+        );
+
+        Ok(())
+    }
 }

src/events/chatlist_events.rs

@@ -241,7 +241,7 @@ mod test_chatlist_events {
             .await;
         bob.recv_msg(&sent_msg).await;
         let alice_on_bob = bob.add_or_lookup_contact(&alice).await;
-        assert!(alice_on_bob.get_display_name() == "Alice");
+        assert_eq!(alice_on_bob.get_display_name(), "~Alice");
 
         wait_for_chatlist_all_items(&bob).await;
 

src/events/payload.rs

@@ -109,6 +109,9 @@ pub enum EventType {
 
     /// A webxdc wants an info message or a changed summary to be notified.
     IncomingWebxdcNotify {
+        /// ID of the chat.
+        chat_id: ChatId,
+
         /// ID of the contact sending.
         contact_id: ContactId,
 

src/headerdef.rs

@@ -73,6 +73,7 @@ pub enum HeaderDef {
 
     /// [Autocrypt](https://autocrypt.org/) header.
     Autocrypt,
+    AutocryptGossip,
     AutocryptSetupMessage,
     SecureJoin,
 

src/html.rs

@@ -7,6 +7,8 @@
 //! `MsgId.get_html()` will return HTML -
 //! this allows nice quoting, handling linebreaks properly etc.
 
+use std::mem;
+
 use anyhow::{Context as _, Result};
 use base64::Engine as _;
 use lettre_email::mime::Mime;
@@ -77,21 +79,26 @@ fn get_mime_multipart_type(ctype: &ParsedContentType) -> MimeMultipartType {
 struct HtmlMsgParser {
     pub html: String,
     pub plain: Option<PlainText>,
+    pub(crate) msg_html: String,
 }
 
 impl HtmlMsgParser {
     /// Function takes a raw mime-message string,
     /// searches for the main-text part
     /// and returns that as parser.html
-    pub async fn from_bytes(context: &Context, rawmime: &[u8]) -> Result<Self> {
+    pub async fn from_bytes<'a>(
+        context: &Context,
+        rawmime: &'a [u8],
+    ) -> Result<(Self, mailparse::ParsedMail<'a>)> {
         let mut parser = HtmlMsgParser {
             html: "".to_string(),
             plain: None,
+            msg_html: "".to_string(),
         };
 
-        let parsedmail = mailparse::parse_mail(rawmime)?;
+        let parsedmail = mailparse::parse_mail(rawmime).context("Failed to parse mail")?;
 
-        parser.collect_texts_recursive(&parsedmail).await?;
+        parser.collect_texts_recursive(context, &parsedmail).await?;
 
         if parser.html.is_empty() {
             if let Some(plain) = &parser.plain {
@@ -100,8 +107,8 @@ impl HtmlMsgParser {
         } else {
             parser.cid_to_data_recursive(context, &parsedmail).await?;
         }
-
-        Ok(parser)
+        parser.html += &mem::take(&mut parser.msg_html);
+        Ok((parser, parsedmail))
     }
 
     /// Function iterates over all mime-parts
@@ -114,12 +121,13 @@ impl HtmlMsgParser {
     /// therefore we use the first one.
     async fn collect_texts_recursive<'a>(
         &'a mut self,
+        context: &'a Context,
         mail: &'a mailparse::ParsedMail<'a>,
     ) -> Result<()> {
         match get_mime_multipart_type(&mail.ctype) {
             MimeMultipartType::Multiple => {
                 for cur_data in &mail.subparts {
-                    Box::pin(self.collect_texts_recursive(cur_data)).await?
+                    Box::pin(self.collect_texts_recursive(context, cur_data)).await?
                 }
                 Ok(())
             }
@@ -128,8 +136,35 @@ impl HtmlMsgParser {
                 if raw.is_empty() {
                     return Ok(());
                 }
-                let mail = mailparse::parse_mail(&raw).context("failed to parse mail")?;
-                Box::pin(self.collect_texts_recursive(&mail)).await
+                let (parser, mail) = Box::pin(HtmlMsgParser::from_bytes(context, &raw)).await?;
+                if !parser.html.is_empty() {
+                    let mut text = "\r\n\r\n".to_string();
+                    for h in mail.headers {
+                        let key = h.get_key();
+                        if matches!(
+                            key.to_lowercase().as_str(),
+                            "date"
+                                | "from"
+                                | "sender"
+                                | "reply-to"
+                                | "to"
+                                | "cc"
+                                | "bcc"
+                                | "subject"
+                        ) {
+                            text += &format!("{key}: {}\r\n", h.get_value());
+                        }
+                    }
+                    text += "\r\n";
+                    self.msg_html += &PlainText {
+                        text,
+                        flowed: false,
+                        delsp: false,
+                    }
+                    .to_html();
+                    self.msg_html += &parser.html;
+                }
+                Ok(())
             }
             MimeMultipartType::Single => {
                 let mimetype = mail.ctype.mimetype.parse::<Mime>()?;
@@ -175,14 +210,7 @@ impl HtmlMsgParser {
                 }
                 Ok(())
             }
-            MimeMultipartType::Message => {
-                let raw = mail.get_body_raw()?;
-                if raw.is_empty() {
-                    return Ok(());
-                }
-                let mail = mailparse::parse_mail(&raw).context("failed to parse mail")?;
-                Box::pin(self.cid_to_data_recursive(context, &mail)).await
-            }
+            MimeMultipartType::Message => Ok(()),
             MimeMultipartType::Single => {
                 let mimetype = mail.ctype.mimetype.parse::<Mime>()?;
                 if mimetype.type_() == mime::IMAGE {
@@ -240,7 +268,7 @@ impl MsgId {
                     warn!(context, "get_html: parser error: {:#}", err);
                     Ok(None)
                 }
-                Ok(parser) => Ok(Some(parser.html)),
+                Ok((parser, _)) => Ok(Some(parser.html)),
             }
         } else {
             warn!(context, "get_html: no mime for {}", self);
@@ -274,7 +302,7 @@ mod tests {
     async fn test_htmlparse_plain_unspecified() {
         let t = TestContext::new().await;
         let raw = include_bytes!("../test-data/message/text_plain_unspecified.eml");
-        let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
+        let (parser, _) = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
         assert_eq!(
             parser.html,
             r#"<!DOCTYPE html>
@@ -292,7 +320,7 @@ This message does not have Content-Type nor Subject.<br/>
     async fn test_htmlparse_plain_iso88591() {
         let t = TestContext::new().await;
         let raw = include_bytes!("../test-data/message/text_plain_iso88591.eml");
-        let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
+        let (parser, _) = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
         assert_eq!(
             parser.html,
             r#"<!DOCTYPE html>
@@ -310,7 +338,7 @@ message with a non-UTF-8 encoding: äöüßÄÖÜ<br/>
     async fn test_htmlparse_plain_flowed() {
         let t = TestContext::new().await;
         let raw = include_bytes!("../test-data/message/text_plain_flowed.eml");
-        let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
+        let (parser, _) = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
         assert!(parser.plain.unwrap().flowed);
         assert_eq!(
             parser.html,
@@ -332,7 +360,7 @@ and will be wrapped as usual.<br/>
     async fn test_htmlparse_alt_plain() {
         let t = TestContext::new().await;
         let raw = include_bytes!("../test-data/message/text_alt_plain.eml");
-        let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
+        let (parser, _) = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
         assert_eq!(
             parser.html,
             r#"<!DOCTYPE html>
@@ -353,7 +381,7 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
     async fn test_htmlparse_html() {
         let t = TestContext::new().await;
         let raw = include_bytes!("../test-data/message/text_html.eml");
-        let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
+        let (parser, _) = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
 
         // on windows, `\r\n` linends are returned from mimeparser,
         // however, rust multiline-strings use just `\n`;
@@ -371,7 +399,7 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
     async fn test_htmlparse_alt_html() {
         let t = TestContext::new().await;
         let raw = include_bytes!("../test-data/message/text_alt_html.eml");
-        let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
+        let (parser, _) = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
         assert_eq!(
             parser.html.replace('\r', ""), // see comment in test_htmlparse_html()
             r##"<html>
@@ -386,7 +414,7 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
     async fn test_htmlparse_alt_plain_html() {
         let t = TestContext::new().await;
         let raw = include_bytes!("../test-data/message/text_alt_plain_html.eml");
-        let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
+        let (parser, _) = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
         assert_eq!(
             parser.html.replace('\r', ""), // see comment in test_htmlparse_html()
             r##"<html>
@@ -411,7 +439,7 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
         assert!(test.find("data:").is_none());
 
         // parsing converts cid: to data:
-        let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
+        let (parser, _) = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
         assert!(parser.html.contains("<html>"));
         assert!(!parser.html.contains("Content-Id:"));
         assert!(parser.html.contains("data:image/jpeg;base64,/9j/4AAQ"));

src/imap.rs

@@ -13,7 +13,7 @@ use std::{
     time::{Duration, UNIX_EPOCH},
 };
 
-use anyhow::{bail, format_err, Context as _, Result};
+use anyhow::{bail, ensure, format_err, Context as _, Result};
 use async_channel::Receiver;
 use async_imap::types::{Fetch, Flag, Name, NameAttribute, UnsolicitedResponse};
 use deltachat_contact_tools::ContactAddress;
@@ -46,7 +46,6 @@ use crate::receive_imf::{
     from_field_to_contact_id, get_prefetch_parent_message, receive_imf_inner, ReceivedMsg,
 };
 use crate::scheduler::connectivity::ConnectivityStore;
-use crate::sql;
 use crate::stock_str;
 use crate::tools::{self, create_id, duration_to_str};
 
@@ -541,10 +540,14 @@ impl Imap {
             return Ok(false);
         }
 
-        session
-            .select_with_uidvalidity(context, folder)
+        let create = false;
+        let folder_exists = session
+            .select_with_uidvalidity(context, folder, create)
             .await
             .with_context(|| format!("Failed to select folder {folder:?}"))?;
+        if !folder_exists {
+            return Ok(false);
+        }
 
         if !session.new_mail && !fetch_existing_msgs {
             info!(context, "No new emails in folder {folder:?}.");
@@ -836,45 +839,52 @@ impl Session {
         folder: &str,
         folder_meaning: FolderMeaning,
     ) -> Result<()> {
+        let uid_validity;
         // Collect pairs of UID and Message-ID.
         let mut msgs = BTreeMap::new();
 
-        self.select_with_uidvalidity(context, folder).await?;
+        let create = false;
+        let folder_exists = self
+            .select_with_uidvalidity(context, folder, create)
+            .await?;
+        if folder_exists {
+            let mut list = self
+                .uid_fetch("1:*", RFC724MID_UID)
+                .await
+                .with_context(|| format!("Can't resync folder {folder}"))?;
+            while let Some(fetch) = list.try_next().await? {
+                let headers = match get_fetch_headers(&fetch) {
+                    Ok(headers) => headers,
+                    Err(err) => {
+                        warn!(context, "Failed to parse FETCH headers: {}", err);
+                        continue;
+                    }
+                };
+                let message_id = prefetch_get_message_id(&headers);
 
-        let mut list = self
-            .uid_fetch("1:*", RFC724MID_UID)
-            .await
-            .with_context(|| format!("can't resync folder {folder}"))?;
-        while let Some(fetch) = list.try_next().await? {
-            let headers = match get_fetch_headers(&fetch) {
-                Ok(headers) => headers,
-                Err(err) => {
-                    warn!(context, "Failed to parse FETCH headers: {}", err);
-                    continue;
+                if let (Some(uid), Some(rfc724_mid)) = (fetch.uid, message_id) {
+                    msgs.insert(
+                        uid,
+                        (
+                            rfc724_mid,
+                            target_folder(context, folder, folder_meaning, &headers).await?,
+                        ),
+                    );
                 }
-            };
-            let message_id = prefetch_get_message_id(&headers);
-
-            if let (Some(uid), Some(rfc724_mid)) = (fetch.uid, message_id) {
-                msgs.insert(
-                    uid,
-                    (
-                        rfc724_mid,
-                        target_folder(context, folder, folder_meaning, &headers).await?,
-                    ),
-                );
             }
+
+            info!(
+                context,
+                "resync_folder_uids: Collected {} message IDs in {folder}.",
+                msgs.len(),
+            );
+
+            uid_validity = get_uidvalidity(context, folder).await?;
+        } else {
+            warn!(context, "resync_folder_uids: No folder {folder}.");
+            uid_validity = 0;
         }
 
-        info!(
-            context,
-            "Resync: collected {} message IDs in folder {}",
-            msgs.len(),
-            folder,
-        );
-
-        let uid_validity = get_uidvalidity(context, folder).await?;
-
         // Write collected UIDs to SQLite database.
         context
             .sql
@@ -911,15 +921,15 @@ impl Session {
             .await?;
         context
             .sql
-            .execute(
-                &format!(
-                    "DELETE FROM imap WHERE id IN ({})",
-                    sql::repeat_vars(row_ids.len())
-                ),
-                rusqlite::params_from_iter(row_ids),
-            )
+            .transaction(|transaction| {
+                let mut stmt = transaction.prepare("DELETE FROM imap WHERE id = ?")?;
+                for row_id in row_ids {
+                    stmt.execute((row_id,))?;
+                }
+                Ok(())
+            })
             .await
-            .context("cannot remove deleted messages from imap table")?;
+            .context("Cannot remove deleted messages from imap table")?;
 
         context.emit_event(EventType::ImapMessageDeleted(format!(
             "IMAP messages {uid_set} marked as deleted"
@@ -942,15 +952,15 @@ impl Session {
                     // Messages are moved or don't exist, IMAP returns OK response in both cases.
                     context
                         .sql
-                        .execute(
-                            &format!(
-                                "DELETE FROM imap WHERE id IN ({})",
-                                sql::repeat_vars(row_ids.len())
-                            ),
-                            rusqlite::params_from_iter(row_ids),
-                        )
+                        .transaction(|transaction| {
+                            let mut stmt = transaction.prepare("DELETE FROM imap WHERE id = ?")?;
+                            for row_id in row_ids {
+                                stmt.execute((row_id,))?;
+                            }
+                            Ok(())
+                        })
                         .await
-                        .context("cannot delete moved messages from imap table")?;
+                        .context("Cannot delete moved messages from imap table")?;
                     context.emit_event(EventType::ImapMessageMoved(format!(
                         "IMAP messages {set} moved to {target}"
                     )));
@@ -996,15 +1006,15 @@ impl Session {
         }
         context
             .sql
-            .execute(
-                &format!(
-                    "UPDATE imap SET target='' WHERE id IN ({})",
-                    sql::repeat_vars(row_ids.len())
-                ),
-                rusqlite::params_from_iter(row_ids),
-            )
+            .transaction(|transaction| {
+                let mut stmt = transaction.prepare("UPDATE imap SET target='' WHERE id = ?")?;
+                for row_id in row_ids {
+                    stmt.execute((row_id,))?;
+                }
+                Ok(())
+            })
             .await
-            .context("cannot plan deletion of messages")?;
+            .context("Cannot plan deletion of messages")?;
         if copy {
             context.emit_event(EventType::ImapMessageMoved(format!(
                 "IMAP messages {set} copied to {target}"
@@ -1040,7 +1050,11 @@ impl Session {
             // MOVE/DELETE operations. This does not result in multiple SELECT commands
             // being sent because `select_folder()` does nothing if the folder is already
             // selected.
-            self.select_with_uidvalidity(context, folder).await?;
+            let create = false;
+            let folder_exists = self
+                .select_with_uidvalidity(context, folder, create)
+                .await?;
+            ensure!(folder_exists, "No folder {folder}");
 
             // Empty target folder name means messages should be deleted.
             if target.is_empty() {
@@ -1134,29 +1148,40 @@ impl Session {
             .await?;
 
         for (folder, rowid_set, uid_set) in UidGrouper::from(rows) {
-            if let Err(err) = self.select_with_uidvalidity(context, &folder).await {
-                warn!(context, "store_seen_flags_on_imap: Failed to select {folder}, will retry later: {err:#}.");
+            let create = false;
+            let folder_exists = match self.select_with_uidvalidity(context, &folder, create).await {
+                Err(err) => {
+                    warn!(
+                        context,
+                        "store_seen_flags_on_imap: Failed to select {folder}, will retry later: {err:#}.");
+                    continue;
+                }
+                Ok(folder_exists) => folder_exists,
+            };
+            if !folder_exists {
+                warn!(context, "store_seen_flags_on_imap: No folder {folder}.");
             } else if let Err(err) = self.add_flag_finalized_with_set(&uid_set, "\\Seen").await {
                 warn!(
                     context,
                     "Cannot mark messages {uid_set} in {folder} as seen, will retry later: {err:#}.");
+                continue;
             } else {
                 info!(
                     context,
                     "Marked messages {} in folder {} as seen.", uid_set, folder
                 );
-                context
-                    .sql
-                    .execute(
-                        &format!(
-                            "DELETE FROM imap_markseen WHERE id IN ({})",
-                            sql::repeat_vars(rowid_set.len())
-                        ),
-                        rusqlite::params_from_iter(rowid_set),
-                    )
-                    .await
-                    .context("cannot remove messages marked as seen from imap_markseen table")?;
             }
+            context
+                .sql
+                .transaction(|transaction| {
+                    let mut stmt = transaction.prepare("DELETE FROM imap_markseen WHERE id = ?")?;
+                    for rowid in rowid_set {
+                        stmt.execute((rowid,))?;
+                    }
+                    Ok(())
+                })
+                .await
+                .context("Cannot remove messages marked as seen from imap_markseen table")?;
         }
 
         Ok(())
@@ -1172,9 +1197,14 @@ impl Session {
             return Ok(());
         }
 
-        self.select_with_uidvalidity(context, folder)
+        let create = false;
+        let folder_exists = self
+            .select_with_uidvalidity(context, folder, create)
             .await
-            .context("failed to select folder")?;
+            .context("Failed to select folder")?;
+        if !folder_exists {
+            return Ok(());
+        }
 
         let mailbox = self
             .selected_mailbox
@@ -1560,52 +1590,54 @@ impl Session {
             return Ok(());
         };
 
-        let device_token_changed = context
-            .get_config(Config::DeviceToken)
-            .await?
-            .map_or(true, |config_token| device_token != config_token);
-
-        if device_token_changed && self.can_metadata() && self.can_push() {
-            let folder = context
-                .get_config(Config::ConfiguredInboxFolder)
+        if self.can_metadata() && self.can_push() {
+            let device_token_changed = context
+                .get_config(Config::DeviceToken)
                 .await?
-                .context("INBOX is not configured")?;
+                .map_or(true, |config_token| device_token != config_token);
 
-            let encrypted_device_token =
-                encrypt_device_token(&device_token).context("Failed to encrypt device token")?;
+            if device_token_changed {
+                let folder = context
+                    .get_config(Config::ConfiguredInboxFolder)
+                    .await?
+                    .context("INBOX is not configured")?;
 
-            // We expect that the server supporting `XDELTAPUSH` capability
-            // has non-synchronizing literals support as well:
-            // <https://www.rfc-editor.org/rfc/rfc7888>.
-            let encrypted_device_token_len = encrypted_device_token.len();
+                let encrypted_device_token = encrypt_device_token(&device_token)
+                    .context("Failed to encrypt device token")?;
 
-            if encrypted_device_token_len <= 4096 {
-                self.run_command_and_check_ok(&format_setmetadata(
-                    &folder,
-                    &encrypted_device_token,
-                ))
-                .await
-                .context("SETMETADATA command failed")?;
+                // We expect that the server supporting `XDELTAPUSH` capability
+                // has non-synchronizing literals support as well:
+                // <https://www.rfc-editor.org/rfc/rfc7888>.
+                let encrypted_device_token_len = encrypted_device_token.len();
 
-                // Store device token saved on the server
-                // to prevent storing duplicate tokens.
-                // The server cannot deduplicate on its own
-                // because encryption gives a different
-                // result each time.
-                context
-                    .set_config_internal(Config::DeviceToken, Some(&device_token))
-                    .await?;
-            } else {
-                // If Apple or Google (FCM) gives us a very large token,
-                // do not even try to give it to IMAP servers.
-                //
-                // Limit of 4096 is arbitrarily selected
-                // to be the same as required by LITERAL- IMAP extension.
-                //
-                // Dovecot supports LITERAL+ and non-synchronizing literals
-                // of any length, but there is no reason for tokens
-                // to be that large even after OpenPGP encryption.
-                warn!(context, "Device token is too long for LITERAL-, ignoring.");
+                if encrypted_device_token_len <= 4096 {
+                    self.run_command_and_check_ok(&format_setmetadata(
+                        &folder,
+                        &encrypted_device_token,
+                    ))
+                    .await
+                    .context("SETMETADATA command failed")?;
+
+                    // Store device token saved on the server
+                    // to prevent storing duplicate tokens.
+                    // The server cannot deduplicate on its own
+                    // because encryption gives a different
+                    // result each time.
+                    context
+                        .set_config_internal(Config::DeviceToken, Some(&device_token))
+                        .await?;
+                } else {
+                    // If Apple or Google (FCM) gives us a very large token,
+                    // do not even try to give it to IMAP servers.
+                    //
+                    // Limit of 4096 is arbitrarily selected
+                    // to be the same as required by LITERAL- IMAP extension.
+                    //
+                    // Dovecot supports LITERAL+ and non-synchronizing literals
+                    // of any length, but there is no reason for tokens
+                    // to be that large even after OpenPGP encryption.
+                    warn!(context, "Device token is too long for LITERAL-, ignoring.");
+                }
             }
             context.push_subscribed.store(true, Ordering::Relaxed);
         } else if !context.push_subscriber.heartbeat_subscribed().await {
@@ -1628,7 +1660,7 @@ fn format_setmetadata(folder: &str, device_token: &str) -> String {
 impl Session {
     /// Returns success if we successfully set the flag or we otherwise
     /// think add_flag should not be retried: Disconnection during setting
-    /// the flag, or other imap-errors, returns true as well.
+    /// the flag, or other imap-errors, returns Ok as well.
     ///
     /// Returning error means that the operation can be retried.
     async fn add_flag_finalized_with_set(&mut self, uid_set: &str, flag: &str) -> Result<()> {
@@ -1675,7 +1707,11 @@ impl Session {
                 self.close().await?;
                 // Before moving emails to the mvbox we need to remember its UIDVALIDITY, otherwise
                 // emails moved before that wouldn't be fetched but considered "old" instead.
-                self.select_with_uidvalidity(context, folder).await?;
+                let create = false;
+                let folder_exists = self
+                    .select_with_uidvalidity(context, folder, create)
+                    .await?;
+                ensure!(folder_exists, "No MVBOX folder {:?}??", &folder);
                 return Ok(Some(folder));
             }
         }
@@ -1686,7 +1722,10 @@ impl Session {
         // Some servers require namespace-style folder names like "INBOX.DeltaChat", so we try all
         // the variants here.
         for folder in folders {
-            match self.select_with_uidvalidity(context, folder).await {
+            match self
+                .select_with_uidvalidity(context, folder, create_mvbox)
+                .await
+            {
                 Ok(_) => {
                     info!(context, "MVBOX-folder {} created.", folder);
                     return Ok(Some(folder));
@@ -2537,10 +2576,14 @@ async fn add_all_recipients_as_contacts(
         );
         return Ok(());
     };
-    session
-        .select_with_uidvalidity(context, &mailbox)
+    let create = false;
+    let folder_exists = session
+        .select_with_uidvalidity(context, &mailbox, create)
         .await
         .with_context(|| format!("could not select {mailbox}"))?;
+    if !folder_exists {
+        return Ok(());
+    }
 
     let recipients = session
         .get_all_recipients(context)

src/imap/idle.rs

@@ -29,7 +29,9 @@ impl Session {
     ) -> Result<Self> {
         use futures::future::FutureExt;
 
-        self.select_with_uidvalidity(context, folder).await?;
+        let create = true;
+        self.select_with_uidvalidity(context, folder, create)
+            .await?;
 
         if self.drain_unsolicited_responses(context)? {
             self.new_mail = true;

src/imap/scan_folders.rs

@@ -34,6 +34,7 @@ impl Imap {
         let watched_folders = get_watched_folders(context).await?;
 
         let mut folder_configs = BTreeMap::new();
+        let mut folder_names = Vec::new();
 
         for folder in folders {
             let folder_meaning = get_folder_meaning_by_attrs(folder.attributes());
@@ -44,6 +45,7 @@ impl Imap {
                 // already been moved and left it in the inbox.
                 continue;
             }
+            folder_names.push(folder.name().to_string());
             let folder_name_meaning = get_folder_meaning_by_name(folder.name());
 
             if let Some(config) = folder_meaning.to_config() {
@@ -91,6 +93,7 @@ impl Imap {
             }
         }
 
+        info!(context, "Found folders: {folder_names:?}.");
         last_scan.replace(tools::Time::now());
         Ok(true)
     }

src/imap/select_folder.rs

@@ -111,7 +111,8 @@ impl ImapSession {
         }
     }
 
-    /// Selects a folder and takes care of UIDVALIDITY changes.
+    /// Selects a folder optionally creating it and takes care of UIDVALIDITY changes. Returns false
+    /// iff `folder` doesn't exist.
     ///
     /// When selecting a folder for the first time, sets the uid_next to the current
     /// mailbox.uid_next so that no old emails are fetched.
@@ -123,11 +124,24 @@ impl ImapSession {
         &mut self,
         context: &Context,
         folder: &str,
-    ) -> Result<()> {
-        let newly_selected = self
-            .select_or_create_folder(context, folder)
-            .await
-            .with_context(|| format!("failed to select or create folder {folder}"))?;
+        create: bool,
+    ) -> Result<bool> {
+        let newly_selected = if create {
+            self.select_or_create_folder(context, folder)
+                .await
+                .with_context(|| format!("failed to select or create folder {folder}"))?
+        } else {
+            match self.select_folder(context, folder).await {
+                Ok(newly_selected) => newly_selected,
+                Err(err) => match err {
+                    Error::NoFolder(..) => return Ok(false),
+                    _ => {
+                        return Err(err)
+                            .with_context(|| format!("failed to select folder {folder}"))?
+                    }
+                },
+            }
+        };
         let mailbox = self
             .selected_mailbox
             .as_mut()
@@ -199,7 +213,7 @@ impl ImapSession {
                 }
             }
 
-            return Ok(());
+            return Ok(true);
         }
 
         // UIDVALIDITY is modified, reset highest seen MODSEQ.
@@ -233,7 +247,7 @@ impl ImapSession {
             old_uid_next,
             old_uid_validity,
         );
-        Ok(())
+        Ok(true)
     }
 }
 

src/imex.rs

@@ -416,7 +416,7 @@ async fn import_backup_stream_inner<R: tokio::io::AsyncRead + Unpin>(
             .context("cannot import unpacked database");
     }
     if res.is_ok() {
-        res = adjust_delete_server_after(context).await;
+        res = adjust_bcc_self(context).await;
     }
     fs::remove_file(unpacked_database)
         .await
@@ -796,7 +796,7 @@ async fn export_database(
         .to_str()
         .with_context(|| format!("path {} is not valid unicode", dest.display()))?;
 
-    adjust_delete_server_after(context).await?;
+    adjust_bcc_self(context).await?;
     context
         .sql
         .set_raw_config_int("backup_time", timestamp)
@@ -826,15 +826,14 @@ async fn export_database(
         .await
 }
 
-/// Sets `Config::DeleteServerAfter` to "never" if needed so that new messages are present on the
-/// server after a backup restoration or available for all devices in multi-device case.
-/// NB: Calling this after a backup import isn't reliable as we can crash in between, but this is a
-/// problem only for old backups, new backups already have `DeleteServerAfter` set if necessary.
-async fn adjust_delete_server_after(context: &Context) -> Result<()> {
-    if context.is_chatmail().await? && !context.config_exists(Config::DeleteServerAfter).await? {
-        context
-            .set_config(Config::DeleteServerAfter, Some("0"))
-            .await?;
+/// Sets `Config::BccSelf` (and `DeleteServerAfter` to "never" in effect) if needed so that new
+/// messages are present on the server after a backup restoration or available for all devices in
+/// multi-device case. NB: Calling this after a backup import isn't reliable as we can crash in
+/// between, but this is a problem only for old backups, new backups already have `BccSelf` set if
+/// necessary.
+async fn adjust_bcc_self(context: &Context) -> Result<()> {
+    if context.is_chatmail().await? && !context.config_exists(Config::BccSelf).await? {
+        context.set_config(Config::BccSelf, Some("1")).await?;
     }
     Ok(())
 }
@@ -1030,12 +1029,20 @@ mod tests {
 
         let context1 = &TestContext::new_alice().await;
 
-        // Check that the setting is displayed correctly.
+        // Check that the settings are displayed correctly.
+        assert_eq!(
+            context1.get_config(Config::BccSelf).await?,
+            Some("1".to_string())
+        );
         assert_eq!(
             context1.get_config(Config::DeleteServerAfter).await?,
             Some("0".to_string())
         );
         context1.set_config_bool(Config::IsChatmail, true).await?;
+        assert_eq!(
+            context1.get_config(Config::BccSelf).await?,
+            Some("0".to_string())
+        );
         assert_eq!(
             context1.get_config(Config::DeleteServerAfter).await?,
             Some("1".to_string())
@@ -1058,6 +1065,10 @@ mod tests {
         assert!(context2.is_configured().await?);
         assert!(context2.is_chatmail().await?);
         for ctx in [context1, context2] {
+            assert_eq!(
+                ctx.get_config(Config::BccSelf).await?,
+                Some("1".to_string())
+            );
             assert_eq!(
                 ctx.get_config(Config::DeleteServerAfter).await?,
                 Some("0".to_string())

src/imex/transfer.rs

@@ -178,6 +178,7 @@ impl BackupProvider {
         }
 
         info!(context, "Received valid backup authentication token.");
+        // Emit a nonzero progress so that UIs can display smth like "Transferring...".
         context.emit_event(EventType::ImexProgress(1));
 
         let blobdir = BlobDirContents::new(&context).await?;
@@ -309,6 +310,10 @@ pub async fn get_backup2(
     let mut file_size_buf = [0u8; 8];
     recv_stream.read_exact(&mut file_size_buf).await?;
     let file_size = u64::from_be_bytes(file_size_buf);
+    info!(context, "Received backup file size.");
+    // Emit a nonzero progress so that UIs can display smth like "Transferring...".
+    context.emit_event(EventType::ImexProgress(1));
+
     import_backup_stream(context, recv_stream, file_size, passphrase)
         .await
         .context("Failed to import backup from QUIC stream")?;
@@ -434,12 +439,14 @@ mod tests {
         assert!(msg.save_file(&ctx1, &path).await.is_err());
 
         // Check that both received the ImexProgress events.
-        ctx0.evtracker
-            .get_matching(|ev| matches!(ev, EventType::ImexProgress(1000)))
-            .await;
-        ctx1.evtracker
-            .get_matching(|ev| matches!(ev, EventType::ImexProgress(1000)))
-            .await;
+        for ctx in [&ctx0, &ctx1] {
+            ctx.evtracker
+                .get_matching(|ev| matches!(ev, EventType::ImexProgress(1)))
+                .await;
+            ctx.evtracker
+                .get_matching(|ev| matches!(ev, EventType::ImexProgress(1000)))
+                .await;
+        }
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]

src/key.rs

@@ -30,7 +30,39 @@ use crate::tools::{self, time_elapsed};
 pub(crate) trait DcKey: Serialize + Deserializable + PublicKeyTrait + Clone {
     /// Create a key from some bytes.
     fn from_slice(bytes: &[u8]) -> Result<Self> {
-        Ok(<Self as Deserializable>::from_bytes(Cursor::new(bytes))?)
+        let res = <Self as Deserializable>::from_bytes(Cursor::new(bytes));
+        if let Ok(res) = res {
+            return Ok(res);
+        }
+
+        // Workaround for keys imported using
+        // Delta Chat core < 1.0.0.
+        // Old Delta Chat core had a bug
+        // that resulted in treating CRC24 checksum
+        // as part of the key when reading ASCII Armor.
+        // Some users that started using Delta Chat in 2019
+        // have such corrupted keys with garbage bytes at the end.
+        //
+        // Garbage is at least 3 bytes long
+        // and may be longer due to padding
+        // at the end of the real key data
+        // and importing the key multiple times.
+        //
+        // If removing 10 bytes is not enough,
+        // the key is likely actually corrupted.
+        for garbage_bytes in 3..std::cmp::min(bytes.len(), 10) {
+            let res = <Self as Deserializable>::from_bytes(Cursor::new(
+                bytes
+                    .get(..bytes.len().saturating_sub(garbage_bytes))
+                    .unwrap_or_default(),
+            ));
+            if let Ok(res) = res {
+                return Ok(res);
+            }
+        }
+
+        // Removing garbage bytes did not help, return the error.
+        Ok(res?)
     }
 
     /// Create a key from a base64 string.
@@ -565,6 +597,36 @@ i8pcjGO+IZffvyZJVRWfVooBJmWWbPB1pueo3tx8w3+fcuzpxz+RLFKaPyqXO+dD
         }
     }
 
+    /// Tests workaround for Delta Chat core < 1.0.0
+    /// which parsed CRC24 at the end of ASCII Armor
+    /// as the part of the key.
+    /// Depending on the alignment and the number of
+    /// `=` characters at the end of the key,
+    /// this resulted in various number of garbage
+    /// octets at the end of the key, starting from 3 octets,
+    /// but possibly 4 or 5 and maybe more octets
+    /// if the key is imported or transferred
+    /// using Autocrypt Setup Message multiple times.
+    #[test]
+    fn test_ignore_trailing_garbage() {
+        // Test several variants of garbage.
+        for garbage in [
+            b"\x02\xfc\xaa\x38\x4b\x5c".as_slice(),
+            b"\x02\xfc\xaa".as_slice(),
+            b"\x01\x02\x03\x04\x05".as_slice(),
+        ] {
+            let private_key = KEYPAIR.secret.clone();
+
+            let mut binary = DcKey::to_bytes(&private_key);
+            binary.extend(garbage);
+
+            let private_key2 =
+                SignedSecretKey::from_slice(&binary).expect("Failed to ignore garbage");
+
+            assert_eq!(private_key.dc_fingerprint(), private_key2.dc_fingerprint());
+        }
+    }
+
     #[test]
     fn test_base64_roundtrip() {
         let key = KEYPAIR.public.clone();

src/location.rs

@@ -707,9 +707,6 @@ pub(crate) async fn save(
                     ))?;
 
                     if timestamp > newest_timestamp {
-                        // okay to drop, as we use cached prepared statements
-                        drop(stmt_test);
-                        drop(stmt_insert);
                         newest_timestamp = timestamp;
                         newest_location_id = Some(u32::try_from(conn.last_insert_rowid())?);
                     }
@@ -1130,6 +1127,10 @@ Content-Disposition: attachment; filename="location.kml"
         assert_eq!(get_range(alice, None, None, 0, 0).await?.len(), 1);
         assert_eq!(get_range(bob, None, None, 0, 0).await?.len(), 1);
 
+        // Location-only messages are "auto-generated", but they mustn't make the contact a bot.
+        let contact = bob.add_or_lookup_contact(alice).await;
+        assert!(!contact.is_bot());
+
         // Day later Bob removes location.
         SystemTime::shift(Duration::from_secs(86400));
         delete_expired(alice, time()).await?;

src/message.rs

@@ -230,7 +230,7 @@ impl MsgId {
         let name = from_contact.get_name_n_addr();
         if let Some(override_sender_name) = msg.get_override_sender_name() {
             let addr = from_contact.get_addr();
-            ret += &format!(" by ~{override_sender_name} ({addr})");
+            ret += &format!(" by {override_sender_name} ({addr})");
         } else {
             ret += &format!(" by {name}");
         }
@@ -293,13 +293,7 @@ impl MsgId {
             ret += ", Location sent";
         }
 
-        let e2ee_errors = msg.param.get_int(Param::ErroneousE2ee).unwrap_or_default();
-
-        if 0 != e2ee_errors {
-            if 0 != e2ee_errors & 0x2 {
-                ret += ", Encrypted, no valid signature";
-            }
-        } else if 0 != msg.param.get_int(Param::GuaranteeE2ee).unwrap_or_default() {
+        if 0 != msg.param.get_int(Param::GuaranteeE2ee).unwrap_or_default() {
             ret += ", Encrypted";
         }
 
@@ -348,7 +342,7 @@ impl MsgId {
             let server_urls = Self::get_info_server_urls(context, msg.rfc724_mid).await?;
             for server_url in server_urls {
                 // Format as RFC 5092 relative IMAP URL.
-                ret += &format!("\n{server_url}");
+                ret += &format!("\nServer-URL: {server_url}");
             }
         }
         let hop_info = self.hop_info(context).await?;
@@ -901,7 +895,7 @@ impl Message {
     pub fn get_override_sender_name(&self) -> Option<String> {
         self.param
             .get(Param::OverrideSenderDisplayname)
-            .map(|name| name.to_string())
+            .map(|name| format!("~{name}"))
     }
 
     // Exposing this function over the ffi instead of get_override_sender_name() would mean that at least Android Java code has
@@ -953,18 +947,6 @@ impl Message {
         cmd != SystemMessage::Unknown
     }
 
-    /// Whether the message is still being created.
-    ///
-    /// Messages with attachments might be created before the
-    /// attachment is ready.  In this case some more restrictions on
-    /// the attachment apply, e.g. if the file to be attached is still
-    /// being written to or otherwise will still change it can not be
-    /// copied to the blobdir.  Thus those attachments need to be
-    /// created immediately in the blobdir with a valid filename.
-    pub fn is_increation(&self) -> bool {
-        self.viewtype.has_file() && self.state == MessageState::OutPreparing
-    }
-
     /// Returns true if the message is an Autocrypt Setup Message.
     pub fn is_setupmessage(&self) -> bool {
         if self.viewtype != Viewtype::File {
@@ -1625,15 +1607,15 @@ pub async fn delete_msgs(context: &Context, msg_ids: &[MsgId]) -> Result<()> {
         modified_chat_ids.insert(msg.chat_id);
 
         let target = context.get_delete_msgs_target().await?;
-        let update_db = |conn: &mut rusqlite::Connection| {
-            conn.execute(
+        let update_db = |trans: &mut rusqlite::Transaction| {
+            trans.execute(
                 "UPDATE imap SET target=? WHERE rfc724_mid=?",
                 (target, msg.rfc724_mid),
             )?;
-            conn.execute("DELETE FROM smtp WHERE msg_id=?", (msg_id,))?;
+            trans.execute("DELETE FROM smtp WHERE msg_id=?", (msg_id,))?;
             Ok(())
         };
-        if let Err(e) = context.sql.call_write(update_db).await {
+        if let Err(e) = context.sql.transaction(update_db).await {
             error!(context, "delete_msgs: failed to update db: {e:#}.");
             res = Err(e);
             continue;
@@ -1655,7 +1637,7 @@ pub async fn delete_msgs(context: &Context, msg_ids: &[MsgId]) -> Result<()> {
     res?;
 
     for modified_chat_id in modified_chat_ids {
-        context.emit_msgs_changed(modified_chat_id, MsgId::new(0));
+        context.emit_msgs_changed_without_msg_id(modified_chat_id);
         chatlist_events::emit_chatlist_item_changed(context, modified_chat_id);
     }
 
@@ -2106,6 +2088,9 @@ pub enum Viewtype {
     Gif = 21,
 
     /// Message containing a sticker, similar to image.
+    /// NB: When sending, the message viewtype may be changed to `Image` by some heuristics like
+    /// checking for transparent pixels. Use `Message::force_sticker()` to disable them.
+    ///
     /// If possible, the ui should display the image without borders in a transparent way.
     /// A click on a sticker will offer to install the sticker set in some future.
     Sticker = 23,
@@ -2206,38 +2191,6 @@ mod tests {
         );
     }
 
-    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-    async fn test_prepare_message_and_send() {
-        let d = test::TestContext::new().await;
-        let ctx = &d.ctx;
-
-        ctx.set_config(Config::ConfiguredAddr, Some("self@example.com"))
-            .await
-            .unwrap();
-
-        let chat = d.create_chat_with_contact("", "dest@example.com").await;
-
-        let mut msg = Message::new(Viewtype::Text);
-
-        let msg_id = chat::prepare_msg(ctx, chat.id, &mut msg).await.unwrap();
-
-        let _msg2 = Message::load_from_db(ctx, msg_id).await.unwrap();
-        assert_eq!(_msg2.get_filemime(), None);
-    }
-
-    /// Tests that message can be prepared even if account has no configured address.
-    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-    async fn test_prepare_not_configured() {
-        let d = test::TestContext::new().await;
-        let ctx = &d.ctx;
-
-        let chat = d.create_chat_with_contact("", "dest@example.com").await;
-
-        let mut msg = Message::new(Viewtype::Text);
-
-        assert!(chat::prepare_msg(ctx, chat.id, &mut msg).await.is_ok());
-    }
-
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_parse_webrtc_instance() {
         let (webrtc_type, url) = Message::parse_webrtc_instance("basicwebrtc:https://foo/bar");
@@ -2357,9 +2310,9 @@ mod tests {
 
         let mut msg = Message::new_text("Quoted message".to_string());
 
-        // Prepare message for sending, so it gets a Message-Id.
+        // Send message, so it gets a Message-Id.
         assert!(msg.rfc724_mid.is_empty());
-        let msg_id = chat::prepare_msg(ctx, chat.id, &mut msg).await.unwrap();
+        let msg_id = chat::send_msg(ctx, chat.id, &mut msg).await.unwrap();
         let msg = Message::load_from_db(ctx, msg_id).await.unwrap();
         assert!(!msg.rfc724_mid.is_empty());
 
@@ -2484,10 +2437,10 @@ mod tests {
         msg.set_override_sender_name(Some("over ride".to_string()));
         assert_eq!(
             msg.get_override_sender_name(),
-            Some("over ride".to_string())
+            Some("~over ride".to_string())
         );
-        assert_eq!(msg.get_sender_name(&contact), "over ride".to_string());
-        assert_ne!(contact.get_display_name(), "over ride".to_string());
+        assert_eq!(msg.get_sender_name(&contact), "~over ride".to_string());
+        assert_ne!(contact.get_display_name(), "~over ride".to_string());
         chat::send_msg(&alice, chat.id, &mut msg).await.unwrap();
         let sent_msg = alice.pop_sent_msg().await;
 
@@ -2504,10 +2457,10 @@ mod tests {
         assert_eq!(msg.text, "bla blubb");
         assert_eq!(
             msg.get_override_sender_name(),
-            Some("over ride".to_string())
+            Some("~over ride".to_string())
         );
-        assert_eq!(msg.get_sender_name(&contact), "over ride".to_string());
-        assert_ne!(contact.get_display_name(), "over ride".to_string());
+        assert_eq!(msg.get_sender_name(&contact), "~over ride".to_string());
+        assert_ne!(contact.get_display_name(), "~over ride".to_string());
 
         // explicitly check that the message does not create a mailing list
         // (mailing lists may also use `Sender:`-header)
@@ -2518,7 +2471,7 @@ mod tests {
         let msg = alice2.recv_msg(&sent_msg).await;
         assert_eq!(
             msg.get_override_sender_name(),
-            Some("over ride".to_string())
+            Some("~over ride".to_string())
         );
     }
 
@@ -2754,6 +2707,29 @@ mod tests {
     async fn test_is_bot() -> Result<()> {
         let alice = TestContext::new_alice().await;
 
+        // Alice receives an auto-generated non-chat message.
+        //
+        // This could be a holiday notice,
+        // in which case the message should be marked as bot-generated,
+        // but the contact should not.
+        receive_imf(
+            &alice,
+            b"From: Claire <claire@example.com>\n\
+                    To: alice@example.org\n\
+                    Message-ID: <789@example.com>\n\
+                    Auto-Submitted: auto-generated\n\
+                    Date: Fri, 29 Jan 2021 21:37:55 +0000\n\
+                    \n\
+                    hello\n",
+            false,
+        )
+        .await?;
+        let msg = alice.get_last_msg().await;
+        assert_eq!(msg.get_text(), "hello".to_string());
+        assert!(msg.is_bot());
+        let contact = Contact::get_by_id(&alice, msg.from_id).await?;
+        assert!(!contact.is_bot());
+
         // Alice receives a message from Bob the bot.
         receive_imf(
             &alice,

src/mimefactory.rs

@@ -652,7 +652,9 @@ impl MimeFactory {
 
         let peerstates = self.peerstates_for_recipients(context).await?;
         let is_encrypted = !self.should_force_plaintext()
-            && encrypt_helper.should_encrypt(context, e2ee_guaranteed, &peerstates)?;
+            && encrypt_helper
+                .should_encrypt(context, e2ee_guaranteed, &peerstates)
+                .await?;
         let is_securejoin_message = if let Loaded::Message { msg, .. } = &self.loaded {
             msg.param.get_cmd() == SystemMessage::SecurejoinMessage
         } else {
@@ -1369,7 +1371,7 @@ impl MimeFactory {
 
         // add attachment part
         if msg.viewtype.has_file() {
-            let (file_part, _) = build_body_file(context, &msg, "").await?;
+            let file_part = build_body_file(context, &msg).await?;
             parts.push(file_part);
         }
 
@@ -1509,14 +1511,10 @@ pub(crate) fn wrapped_base64_encode(buf: &[u8]) -> String {
         .join("\r\n")
 }
 
-async fn build_body_file(
-    context: &Context,
-    msg: &Message,
-    base_name: &str,
-) -> Result<(PartBuilder, String)> {
+async fn build_body_file(context: &Context, msg: &Message) -> Result<PartBuilder> {
     let blob = msg
         .param
-        .get_blob(Param::File, context, true)
+        .get_blob(Param::File, context)
         .await?
         .context("msg has no file")?;
     let suffix = blob.suffix().unwrap_or("dat");
@@ -1539,17 +1537,13 @@ async fn build_body_file(
         ),
         Viewtype::Image | Viewtype::Gif => format!(
             "image_{}.{}",
-            if base_name.is_empty() {
-                chrono::Utc
-                    .timestamp_opt(msg.timestamp_sort, 0)
-                    .single()
-                    .map_or_else(
-                        || "YY-mm-dd_hh:mm:ss".to_string(),
-                        |ts| ts.format("%Y-%m-%d_%H-%M-%S").to_string(),
-                    )
-            } else {
-                base_name.to_string()
-            },
+            chrono::Utc
+                .timestamp_opt(msg.timestamp_sort, 0)
+                .single()
+                .map_or_else(
+                    || "YY-mm-dd_hh:mm:ss".to_string(),
+                    |ts| ts.format("%Y-%m-%d_%H-%M-%S").to_string(),
+                ),
             &suffix,
         ),
         Viewtype::Video => format!(
@@ -1601,7 +1595,7 @@ async fn build_body_file(
         .header(("Content-Transfer-Encoding", "base64"))
         .body(encoded_body);
 
-    Ok((mail, filename_to_send))
+    Ok(mail)
 }
 
 async fn build_avatar_file(context: &Context, path: &str) -> Result<String> {
@@ -1905,7 +1899,7 @@ mod tests {
         )
         .await
         .unwrap();
-        let new_msg = incoming_msg_to_reply_msg(
+        let mut new_msg = incoming_msg_to_reply_msg(
             b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                  From: bob@example.com\n\
                  To: alice@example.org\n\
@@ -1931,6 +1925,9 @@ mod tests {
                  Original-Message-ID: <2893@example.com>\n\
                  Disposition: manual-action/MDN-sent-automatically; displayed\n\
                  \n", &t).await;
+        chat::send_msg(&t, new_msg.chat_id, &mut new_msg)
+            .await
+            .unwrap();
         let mf = MimeFactory::from_msg(&t, new_msg).await.unwrap();
         // The subject string should not be "Re: message opened"
         assert_eq!("Re: Hello, Bob", mf.subject_str(&t).await.unwrap());
@@ -2077,7 +2074,7 @@ mod tests {
 
         let mut new_msg = Message::new_text("Hi".to_string());
         new_msg.chat_id = chat_id;
-        chat::prepare_msg(&t, chat_id, &mut new_msg).await.unwrap();
+        chat::send_msg(&t, chat_id, &mut new_msg).await.unwrap();
 
         let mf = MimeFactory::from_msg(&t, new_msg).await.unwrap();
 
@@ -2134,7 +2131,7 @@ mod tests {
     ) -> String {
         let t = TestContext::new_alice().await;
         let mut new_msg = incoming_msg_to_reply_msg(imf_raw, &t).await;
-        let incoming_msg = get_chat_msg(&t, new_msg.chat_id, 0, 2).await;
+        let incoming_msg = get_chat_msg(&t, new_msg.chat_id, 0, 1).await;
 
         if delete_original_msg {
             incoming_msg.id.trash(&t, false).await.unwrap();
@@ -2164,6 +2161,9 @@ mod tests {
             new_msg.set_quote(&t, Some(&incoming_msg)).await.unwrap();
         }
 
+        chat::send_msg(&t, new_msg.chat_id, &mut new_msg)
+            .await
+            .unwrap();
         let mf = MimeFactory::from_msg(&t, new_msg).await.unwrap();
         mf.subject_str(&t).await.unwrap()
     }
@@ -2184,9 +2184,6 @@ mod tests {
 
         let mut new_msg = Message::new_text("Hi".to_string());
         new_msg.chat_id = chat_id;
-        chat::prepare_msg(context, chat_id, &mut new_msg)
-            .await
-            .unwrap();
 
         new_msg
     }
@@ -2197,7 +2194,7 @@ mod tests {
         let t = TestContext::new_alice().await;
         let context = &t;
 
-        let msg = incoming_msg_to_reply_msg(
+        let mut msg = incoming_msg_to_reply_msg(
             b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                 From: Charlie <charlie@example.com>\n\
                 To: alice@example.org\n\
@@ -2210,6 +2207,7 @@ mod tests {
             context,
         )
         .await;
+        chat::send_msg(&t, msg.chat_id, &mut msg).await.unwrap();
 
         let mimefactory = MimeFactory::from_msg(&t, msg).await.unwrap();
 

src/mimeparser.rs

@@ -17,10 +17,10 @@ use rand::distributions::{Alphanumeric, DistString};
 use crate::aheader::{Aheader, EncryptPreference};
 use crate::authres::handle_authres;
 use crate::blob::BlobObject;
-use crate::chat::{add_info_msg, ChatId};
+use crate::chat::ChatId;
 use crate::config::Config;
-use crate::constants::{self, Chattype};
-use crate::contact::{Contact, ContactId, Origin};
+use crate::constants;
+use crate::contact::ContactId;
 use crate::context::Context;
 use crate::decrypt::{
     get_autocrypt_peerstate, get_encrypted_mime, keyring_from_peerstate, try_decrypt,
@@ -36,8 +36,7 @@ use crate::peerstate::Peerstate;
 use crate::simplify::{simplify, SimplifiedText};
 use crate::sync::SyncItems;
 use crate::tools::{
-    create_smeared_timestamp, get_filemeta, parse_receive_headers, smeared_time, truncate_msg_text,
-    validate_id,
+    get_filemeta, parse_receive_headers, smeared_time, truncate_msg_text, validate_id,
 };
 use crate::{chatlist_events, location, stock_str, tools};
 
@@ -106,20 +105,26 @@ pub(crate) struct MimeMessage {
     /// received.
     pub(crate) footer: Option<String>,
 
-    // if this flag is set, the parts/text/etc. are just close to the original mime-message;
-    // clients should offer a way to view the original message in this case
+    /// If set, this is a modified MIME message; clients should offer a way to view the original
+    /// MIME message in this case.
     pub is_mime_modified: bool,
 
-    /// The decrypted, raw mime structure.
-    ///
-    /// This is non-empty iff `is_mime_modified` and the message was actually encrypted. It is used
-    /// for e.g. late-parsing HTML.
+    /// Decrypted, raw MIME structure. Nonempty iff `is_mime_modified` and the message was actually
+    /// encrypted.
     pub decoded_data: Vec<u8>,
 
     /// Hop info for debugging.
     pub(crate) hop_info: String,
 
-    /// Whether the contact sending this should be marked as bot or non-bot.
+    /// Whether the message is auto-generated.
+    ///
+    /// If chat message (with `Chat-Version` header) is auto-generated,
+    /// the contact sending this should be marked as bot.
+    ///
+    /// If non-chat message is auto-generated,
+    /// it could be a holiday notice auto-reply,
+    /// in which case the message should be marked as bot-generated,
+    /// but the contact should not be.
     pub(crate) is_bot: Option<bool>,
 
     /// When the message was received, in secs since epoch.
@@ -565,11 +570,14 @@ impl MimeMessage {
             },
         };
 
-        if parser.mdn_reports.is_empty() && parser.webxdc_status_update.is_none() {
-            // "Auto-Submitted" is also set by holiday-notices so we also check "chat-version".
-            let is_bot = parser.headers.get("auto-submitted")
-                == Some(&"auto-generated".to_string())
-                && parser.headers.contains_key("chat-version");
+        let is_location_only = parser.location_kml.is_some() && parser.parts.is_empty();
+        if parser.mdn_reports.is_empty()
+            && !is_location_only
+            && parser.sync_items.is_none()
+            && parser.webxdc_status_update.is_none()
+        {
+            let is_bot =
+                parser.headers.get("auto-submitted") == Some(&"auto-generated".to_string());
             parser.is_bot = Some(is_bot);
         }
         parser.maybe_remove_bad_parts();
@@ -1665,18 +1673,8 @@ impl MimeMessage {
                 .get_header_value(HeaderDef::MessageId)
                 .and_then(|v| parse_message_id(&v).ok())
             {
-                let mut to_list =
-                    get_all_addresses_from_header(&report.headers, "x-failed-recipients");
-                let to = if to_list.len() != 1 {
-                    // We do not know which recipient failed
-                    None
-                } else {
-                    to_list.pop()
-                };
-
                 return Ok(Some(DeliveryReport {
                     rfc724_mid: original_message_id,
-                    failed_recipient: to.map(|s| s.addr),
                     failure,
                 }));
             }
@@ -1774,7 +1772,6 @@ impl MimeMessage {
                 {
                     self.delivery_report = Some(DeliveryReport {
                         rfc724_mid: original_message_id,
-                        failed_recipient: None,
                         failure: true,
                     })
                 }
@@ -1912,7 +1909,6 @@ pub(crate) struct Report {
 #[derive(Debug)]
 pub(crate) struct DeliveryReport {
     pub rfc724_mid: String,
-    pub failed_recipient: Option<String>,
     pub failure: bool,
 }
 
@@ -2278,20 +2274,12 @@ async fn handle_ndn(
     let msgs: Vec<_> = context
         .sql
         .query_map(
-            concat!(
-                "SELECT",
-                "    m.id AS msg_id,",
-                "    c.id AS chat_id,",
-                "    c.type AS type",
-                " FROM msgs m LEFT JOIN chats c ON m.chat_id=c.id",
-                " WHERE rfc724_mid=? AND from_id=1",
-            ),
+            "SELECT id FROM msgs
+                WHERE rfc724_mid=? AND from_id=1",
             (&failed.rfc724_mid,),
             |row| {
-                let msg_id: MsgId = row.get("msg_id")?;
-                let chat_id: ChatId = row.get("chat_id")?;
-                let chat_type: Chattype = row.get("type")?;
-                Ok((msg_id, chat_id, chat_type))
+                let msg_id: MsgId = row.get(0)?;
+                Ok(msg_id)
             },
             |rows| Ok(rows.collect::<Vec<_>>()),
         )
@@ -2299,16 +2287,13 @@ async fn handle_ndn(
 
     let error = if let Some(error) = error {
         error
-    } else if let Some(failed_recipient) = &failed.failed_recipient {
-        format!("Delivery to {failed_recipient} failed.").clone()
     } else {
         "Delivery to at least one recipient failed.".to_string()
     };
     let err_msg = &error;
 
-    let mut first = true;
     for msg in msgs {
-        let (msg_id, chat_id, chat_type) = msg?;
+        let msg_id = msg?;
         let mut message = Message::load_from_db(context, msg_id).await?;
         let aggregated_error = message
             .error
@@ -2320,47 +2305,11 @@ async fn handle_ndn(
             aggregated_error.as_ref().unwrap_or(err_msg),
         )
         .await?;
-        if first {
-            // Add only one info msg for all failed messages
-            ndn_maybe_add_info_msg(context, failed, chat_id, chat_type).await?;
-        }
-        first = false;
     }
 
     Ok(())
 }
 
-async fn ndn_maybe_add_info_msg(
-    context: &Context,
-    failed: &DeliveryReport,
-    chat_id: ChatId,
-    chat_type: Chattype,
-) -> Result<()> {
-    match chat_type {
-        Chattype::Group | Chattype::Broadcast => {
-            if let Some(failed_recipient) = &failed.failed_recipient {
-                let contact_id =
-                    Contact::lookup_id_by_addr(context, failed_recipient, Origin::Unknown)
-                        .await?
-                        .context("contact ID not found")?;
-                let contact = Contact::get_by_id(context, contact_id).await?;
-                // Tell the user which of the recipients failed if we know that (because in
-                // a group, this might otherwise be unclear)
-                let text = stock_str::failed_sending_to(context, contact.get_display_name()).await;
-                add_info_msg(context, chat_id, &text, create_smeared_timestamp(context)).await?;
-                context.emit_event(EventType::ChatModified(chat_id));
-            }
-        }
-        Chattype::Mailinglist => {
-            // ndn_maybe_add_info_msg() is about the case when delivery to the group failed.
-            // If we get an NDN for the mailing list, just issue a warning.
-            warn!(context, "ignoring NDN for mailing list.");
-        }
-        Chattype::Single => {}
-    }
-    Ok(())
-}
-
 #[cfg(test)]
 mod tests {
     use mailparse::ParsedMail;
@@ -3151,11 +3100,7 @@ MDYyMDYxNTE1RTlDOEE4Cj4+CnN0YXJ0eHJlZgo4Mjc4CiUlRU9GCg==
 
         // Make sure the file is there even though the html is wrong:
         let param = &message.parts[0].param;
-        let blob: BlobObject = param
-            .get_blob(Param::File, &t, false)
-            .await
-            .unwrap()
-            .unwrap();
+        let blob: BlobObject = param.get_blob(Param::File, &t).await.unwrap().unwrap();
         let f = tokio::fs::File::open(blob.to_abs_path()).await.unwrap();
         let size = f.metadata().await.unwrap().len();
         assert_eq!(size, 154);
@@ -3653,9 +3598,10 @@ On 2020-10-25, Bob wrote:
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_mime_modified_large_plain() -> Result<()> {
         let t = TestContext::new_alice().await;
+        let t1 = TestContext::new_alice().await;
 
         static REPEAT_TXT: &str = "this text with 42 chars is just repeated.\n";
-        static REPEAT_CNT: usize = 2000; // results in a text of 84k, should be more than DC_DESIRED_TEXT_LEN
+        static REPEAT_CNT: usize = DC_DESIRED_TEXT_LEN / REPEAT_TXT.len() + 2;
         let long_txt = format!("From: alice@c.de\n\n{}", REPEAT_TXT.repeat(REPEAT_CNT));
         assert_eq!(long_txt.matches("just repeated").count(), REPEAT_CNT);
         assert!(long_txt.len() > DC_DESIRED_TEXT_LEN);
@@ -3676,22 +3622,21 @@ On 2020-10-25, Bob wrote:
             if draft {
                 chat.id.set_draft(&t, Some(&mut msg)).await?;
             }
-            t.send_msg(chat.id, &mut msg).await;
+            let sent_msg = t.send_msg(chat.id, &mut msg).await;
             let msg = t.get_last_msg_in(chat.id).await;
             assert!(msg.has_html());
-            assert_eq!(
-                msg.id
-                    .get_html(&t)
-                    .await?
-                    .unwrap()
-                    .matches("just repeated")
-                    .count(),
-                REPEAT_CNT
-            );
+            let html = msg.id.get_html(&t).await?.unwrap();
+            assert_eq!(html.matches("<!DOCTYPE html>").count(), 1);
+            assert_eq!(html.matches("just repeated.<br/>").count(), REPEAT_CNT);
             assert!(
-                msg.text.matches("just repeated").count() <= DC_DESIRED_TEXT_LEN / REPEAT_TXT.len()
+                msg.text.matches("just repeated.").count()
+                    <= DC_DESIRED_TEXT_LEN / REPEAT_TXT.len()
             );
             assert!(msg.text.len() <= DC_DESIRED_TEXT_LEN + DC_ELLIPSIS.len());
+
+            let msg = t1.recv_msg(&sent_msg).await;
+            assert!(msg.has_html());
+            assert_eq!(msg.id.get_html(&t1).await?.unwrap(), html);
         }
 
         t.set_config(Config::Bot, Some("1")).await?;

src/net/http.rs

@@ -6,14 +6,17 @@ use http_body_util::BodyExt;
 use hyper_util::rt::TokioIo;
 use mime::Mime;
 use serde::Serialize;
+use tokio::fs;
 
+use crate::blob::BlobObject;
 use crate::context::Context;
 use crate::net::proxy::ProxyConfig;
 use crate::net::session::SessionStream;
 use crate::net::tls::wrap_rustls;
+use crate::tools::{create_id, time};
 
 /// HTTP(S) GET response.
-#[derive(Debug)]
+#[derive(Debug, Clone, PartialEq, Eq)]
 pub struct Response {
     /// Response body.
     pub blob: Vec<u8>,
@@ -90,9 +93,146 @@ where
     Ok(sender)
 }
 
-/// Retrieves the binary contents of URL using HTTP GET request.
-pub async fn read_url_blob(context: &Context, url: &str) -> Result<Response> {
-    let mut url = url.to_string();
+/// Converts the URL to expiration and stale timestamps.
+fn http_url_cache_timestamps(url: &str, mimetype: Option<&str>) -> (i64, i64) {
+    let now = time();
+
+    let expires = now + 3600 * 24 * 35;
+    let stale = if url.ends_with(".xdc") {
+        // WebXDCs are never stale, they just expire.
+        expires
+    } else if mimetype.is_some_and(|s| s.starts_with("image/")) {
+        // Cache images for 1 day.
+        //
+        // As of 2024-12-12 WebXDC icons at <https://webxdc.org/apps/>
+        // use the same path for all app versions,
+        // so may change, but it is not critical if outdated icon is displayed.
+        now + 3600 * 24
+    } else {
+        // Revalidate everything else after 1 hour.
+        //
+        // This includes HTML, CSS and JS.
+        now + 3600
+    };
+    (expires, stale)
+}
+
+/// Places the binary into HTTP cache.
+async fn http_cache_put(context: &Context, url: &str, response: &Response) -> Result<()> {
+    let blob = BlobObject::create(
+        context,
+        &format!("http_cache_{}", create_id()),
+        response.blob.as_slice(),
+    )
+    .await?;
+
+    let (expires, stale) = http_url_cache_timestamps(url, response.mimetype.as_deref());
+    context
+        .sql
+        .insert(
+            "INSERT OR REPLACE INTO http_cache (url, expires, stale, blobname, mimetype, encoding)
+             VALUES (?, ?, ?, ?, ?, ?)",
+            (
+                url,
+                expires,
+                stale,
+                blob.as_name(),
+                response.mimetype.as_deref().unwrap_or_default(),
+                response.encoding.as_deref().unwrap_or_default(),
+            ),
+        )
+        .await?;
+
+    Ok(())
+}
+
+/// Retrieves the binary from HTTP cache.
+///
+/// Also returns if the response is stale and should be revalidated in the background.
+async fn http_cache_get(context: &Context, url: &str) -> Result<Option<(Response, bool)>> {
+    let now = time();
+    let Some((blob_name, mimetype, encoding, stale_timestamp)) = context
+        .sql
+        .query_row_optional(
+            "SELECT blobname, mimetype, encoding, stale
+             FROM http_cache WHERE url=? AND expires > ?",
+            (url, now),
+            |row| {
+                let blob_name: String = row.get(0)?;
+                let mimetype: Option<String> = Some(row.get(1)?).filter(|s: &String| !s.is_empty());
+                let encoding: Option<String> = Some(row.get(2)?).filter(|s: &String| !s.is_empty());
+                let stale_timestamp: i64 = row.get(3)?;
+                Ok((blob_name, mimetype, encoding, stale_timestamp))
+            },
+        )
+        .await?
+    else {
+        return Ok(None);
+    };
+    let is_stale = now > stale_timestamp;
+
+    let blob_object = BlobObject::from_name(context, blob_name)?;
+    let blob_abs_path = blob_object.to_abs_path();
+    let blob = match fs::read(blob_abs_path)
+        .await
+        .with_context(|| format!("Failed to read blob for {url:?} cache entry."))
+    {
+        Ok(blob) => blob,
+        Err(err) => {
+            // This should not happen, but user may go into the blobdir and remove files,
+            // antivirus may delete the file or there may be a bug in housekeeping.
+            warn!(context, "{err:?}.");
+            return Ok(None);
+        }
+    };
+
+    let (expires, _stale) = http_url_cache_timestamps(url, mimetype.as_deref());
+    let response = Response {
+        blob,
+        mimetype,
+        encoding,
+    };
+
+    // Update expiration timestamp
+    // to prevent deletion of the file still in use.
+    //
+    // If the response is stale, the caller should revalidate it in the background, so update
+    // `stale` timestamp to avoid revalidating too frequently (and have many parallel revalidation
+    // tasks) if revalidation fails or the HTTP request takes some time. The stale period >= 1 hour,
+    // so 1 more minute won't be a problem.
+    let stale_timestamp = if is_stale { now + 60 } else { stale_timestamp };
+    context
+        .sql
+        .execute(
+            "UPDATE http_cache SET expires=?, stale=? WHERE url=?",
+            (expires, stale_timestamp, url),
+        )
+        .await?;
+
+    Ok(Some((response, is_stale)))
+}
+
+/// Removes expired cache entries.
+pub(crate) async fn http_cache_cleanup(context: &Context) -> Result<()> {
+    // Remove cache entries that are already expired
+    // or entries that will not expire in a year
+    // to make sure we don't have invalid timestamps that are way forward in the future.
+    context
+        .sql
+        .execute(
+            "DELETE FROM http_cache
+             WHERE ?1 > expires OR expires > ?1 + 31536000",
+            (time(),),
+        )
+        .await?;
+    Ok(())
+}
+
+/// Fetches URL and updates the cache.
+///
+/// URL is fetched regardless of whether there is an existing result in the cache.
+async fn fetch_url(context: &Context, original_url: &str) -> Result<Response> {
+    let mut url = original_url.to_string();
 
     // Follow up to 10 http-redirects
     for _i in 0..10 {
@@ -139,16 +279,42 @@ pub async fn read_url_blob(context: &Context, url: &str) -> Result<Response> {
         });
         let body = response.collect().await?.to_bytes();
         let blob: Vec<u8> = body.to_vec();
-        return Ok(Response {
+        let response = Response {
             blob,
             mimetype,
             encoding,
-        });
+        };
+        info!(context, "Inserting {original_url:?} into cache.");
+        http_cache_put(context, &url, &response).await?;
+        return Ok(response);
     }
 
     Err(anyhow!("Followed 10 redirections"))
 }
 
+/// Retrieves the binary contents of URL using HTTP GET request.
+pub async fn read_url_blob(context: &Context, url: &str) -> Result<Response> {
+    if let Some((response, is_stale)) = http_cache_get(context, url).await? {
+        info!(context, "Returning {url:?} from cache.");
+        if is_stale {
+            let context = context.clone();
+            let url = url.to_string();
+            tokio::spawn(async move {
+                // Fetch URL in background to update the cache.
+                info!(context, "Fetching stale {url:?} in background.");
+                if let Err(err) = fetch_url(&context, &url).await {
+                    warn!(context, "Failed to revalidate {url:?}: {err:#}.");
+                }
+            });
+        }
+        return Ok(response);
+    }
+
+    info!(context, "Not found {url:?} in cache, fetching.");
+    let response = fetch_url(context, url).await?;
+    Ok(response)
+}
+
 /// Sends an empty POST request to the URL.
 ///
 /// Returns response text and whether request was successful or not.
@@ -241,3 +407,125 @@ pub(crate) async fn post_form<T: Serialize + ?Sized>(
     let bytes = response.collect().await?.to_bytes();
     Ok(bytes)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::time::Duration;
+
+    use crate::sql::housekeeping;
+    use crate::test_utils::TestContext;
+    use crate::tools::SystemTime;
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_http_cache() -> Result<()> {
+        let t = &TestContext::new().await;
+
+        assert_eq!(http_cache_get(t, "https://webxdc.org/").await?, None);
+
+        let html_response = Response {
+            blob: b"<!DOCTYPE html> ...".to_vec(),
+            mimetype: Some("text/html".to_string()),
+            encoding: None,
+        };
+
+        let xdc_response = Response {
+            blob: b"PK...".to_vec(),
+            mimetype: Some("application/octet-stream".to_string()),
+            encoding: None,
+        };
+        let xdc_editor_url = "https://apps.testrun.org/webxdc-editor-v3.2.0.xdc";
+        let xdc_pixel_url = "https://apps.testrun.org/webxdc-pixel-v2.xdc";
+
+        http_cache_put(t, "https://webxdc.org/", &html_response).await?;
+
+        assert_eq!(http_cache_get(t, xdc_editor_url).await?, None);
+        assert_eq!(http_cache_get(t, xdc_pixel_url).await?, None);
+        assert_eq!(
+            http_cache_get(t, "https://webxdc.org/").await?,
+            Some((html_response.clone(), false))
+        );
+
+        http_cache_put(t, xdc_editor_url, &xdc_response).await?;
+        http_cache_put(t, xdc_pixel_url, &xdc_response).await?;
+        assert_eq!(
+            http_cache_get(t, xdc_editor_url).await?,
+            Some((xdc_response.clone(), false))
+        );
+        assert_eq!(
+            http_cache_get(t, xdc_pixel_url).await?,
+            Some((xdc_response.clone(), false))
+        );
+
+        assert_eq!(
+            http_cache_get(t, "https://webxdc.org/").await?,
+            Some((html_response.clone(), false))
+        );
+
+        // HTML is stale after 1 hour, but .xdc is not.
+        SystemTime::shift(Duration::from_secs(3600 + 100));
+        assert_eq!(
+            http_cache_get(t, "https://webxdc.org/").await?,
+            Some((html_response.clone(), true))
+        );
+        assert_eq!(
+            http_cache_get(t, xdc_editor_url).await?,
+            Some((xdc_response.clone(), false))
+        );
+
+        // Stale cache entry can be renewed
+        // even before housekeeping removes old one.
+        http_cache_put(t, "https://webxdc.org/", &html_response).await?;
+        assert_eq!(
+            http_cache_get(t, "https://webxdc.org/").await?,
+            Some((html_response.clone(), false))
+        );
+
+        // 35 days later pixel .xdc expires because we did not request it for 35 days and 1 hour.
+        // But editor is still there because we did not request it for just 35 days.
+        // We have not renewed the editor however, so it becomes stale.
+        SystemTime::shift(Duration::from_secs(3600 * 24 * 35 - 100));
+
+        // Run housekeeping to test that it does not delete the blob too early.
+        housekeeping(t).await?;
+
+        assert_eq!(
+            http_cache_get(t, xdc_editor_url).await?,
+            Some((xdc_response.clone(), true))
+        );
+        assert_eq!(http_cache_get(t, xdc_pixel_url).await?, None);
+
+        // If we get the blob the second time quickly, it shouldn't be stale because it's supposed
+        // that we've already run a revalidation task which will update the blob soon.
+        assert_eq!(
+            http_cache_get(t, xdc_editor_url).await?,
+            Some((xdc_response.clone(), false))
+        );
+        // But if the revalidation task hasn't succeeded after some time, the blob is stale again
+        // even if we continue to get it frequently.
+        for i in (0..100).rev() {
+            SystemTime::shift(Duration::from_secs(6));
+            if let Some((_, true)) = http_cache_get(t, xdc_editor_url).await? {
+                break;
+            }
+            assert!(i > 0);
+        }
+
+        // Test that if the file is accidentally removed from the blobdir,
+        // there is no error when trying to load the cache entry.
+        for entry in std::fs::read_dir(t.get_blobdir())? {
+            let entry = entry.unwrap();
+            let path = entry.path();
+            std::fs::remove_file(path).expect("Failed to remove blob");
+        }
+
+        assert_eq!(
+            http_cache_get(t, xdc_editor_url)
+                .await
+                .context("Failed to get no cache response")?,
+            None
+        );
+
+        Ok(())
+    }
+}

src/net/proxy.rs

@@ -640,6 +640,9 @@ mod tests {
     fn test_invalid_proxy_url() {
         assert!(ProxyConfig::from_url("foobar://127.0.0.1:9050").is_err());
         assert!(ProxyConfig::from_url("abc").is_err());
+
+        // This caused panic before shadowsocks 1.22.0.
+        assert!(ProxyConfig::from_url("ss://foo:bar@127.0.0.1:9999").is_err());
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]

src/param.rs

@@ -55,6 +55,8 @@ pub enum Param {
 
     /// For Messages: decrypted with validation errors or without mutual set, if neither
     /// 'c' nor 'e' are preset, the messages is only transport encrypted.
+    ///
+    /// Deprecated on 2024-12-25.
     ErroneousE2ee = b'e',
 
     /// For Messages: force unencrypted message, a value from `ForcePlaintext` enum.
@@ -366,20 +368,16 @@ impl Params {
     ///
     /// This parses the parameter value as a [ParamsFile] and than
     /// tries to return a [BlobObject] for that file.  If the file is
-    /// not yet a valid blob, one will be created by copying the file
-    /// only if `create` is set to `true`, otherwise an error is
-    /// returned.
+    /// not yet a valid blob, one will be created by copying the file.
     ///
     /// Note that in the [ParamsFile::FsPath] case the blob can be
     /// created without copying if the path already refers to a valid
-    /// blob.  If so a [BlobObject] will be returned regardless of the
-    /// `create` argument.
+    /// blob.  If so a [BlobObject] will be returned.
     #[allow(clippy::needless_lifetimes)]
     pub async fn get_blob<'a>(
         &self,
         key: Param,
         context: &'a Context,
-        create: bool,
     ) -> Result<Option<BlobObject<'a>>> {
         let val = match self.get(key) {
             Some(val) => val,
@@ -387,10 +385,7 @@ impl Params {
         };
         let file = ParamsFile::from_param(context, val)?;
         let blob = match file {
-            ParamsFile::FsPath(path) => match create {
-                true => BlobObject::new_from_path(context, &path).await?,
-                false => BlobObject::from_path(context, &path)?,
-            },
+            ParamsFile::FsPath(path) => BlobObject::new_from_path(context, &path).await?,
             ParamsFile::Blob(blob) => blob,
         };
         Ok(Some(blob))
@@ -546,23 +541,20 @@ mod tests {
         let path: PathBuf = p.get_path(Param::File, &t).unwrap().unwrap();
         assert_eq!(path, fname);
 
-        // Blob does not exist yet, expect error.
-        assert!(p.get_blob(Param::File, &t, false).await.is_err());
-
         fs::write(fname, b"boo").await.unwrap();
-        let blob = p.get_blob(Param::File, &t, true).await.unwrap().unwrap();
+        let blob = p.get_blob(Param::File, &t).await.unwrap().unwrap();
         assert!(blob.as_file_name().starts_with("foo"));
 
         // Blob in blobdir, expect blob.
         let bar_path = t.get_blobdir().join("bar");
         p.set(Param::File, bar_path.to_str().unwrap());
-        let blob = p.get_blob(Param::File, &t, false).await.unwrap().unwrap();
+        let blob = p.get_blob(Param::File, &t).await.unwrap().unwrap();
         assert_eq!(blob, BlobObject::from_name(&t, "bar".to_string()).unwrap());
 
         p.remove(Param::File);
         assert!(p.get_file(Param::File, &t).unwrap().is_none());
         assert!(p.get_path(Param::File, &t).unwrap().is_none());
-        assert!(p.get_blob(Param::File, &t, false).await.unwrap().is_none());
+        assert!(p.get_blob(Param::File, &t).await.unwrap().is_none());
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]

src/peerstate.rs

@@ -664,7 +664,7 @@ impl Peerstate {
                 let old_contact = Contact::get_by_id(context, contact_id).await?;
                 stock_str::aeap_addr_changed(
                     context,
-                    old_contact.get_display_name(),
+                    &old_contact.get_display_name(),
                     &self.addr,
                     new_addr,
                 )

src/push.rs

@@ -1,3 +1,11 @@
+//! # Push notifications module.
+//!
+//! This module is responsible for Apple Push Notification Service
+//! and Firebase Cloud Messaging push notifications.
+//!
+//! It provides [`PushSubscriber`] type
+//! which holds push notification token for the device,
+//! shared by all accounts.
 use std::sync::atomic::Ordering;
 use std::sync::Arc;
 

src/reaction.rs

@@ -731,7 +731,7 @@ Here's my footer -- bob@example.net"
         assert_eq!(summary.state, MessageState::InFresh); // state refers to message, not to reaction
         assert!(summary.prefix.is_none());
         assert!(summary.thumbnail_path.is_none());
-        assert_summary(&alice, "BOB reacted 👍 to \"Party?\"").await;
+        assert_summary(&alice, "~BOB reacted 👍 to \"Party?\"").await;
 
         // Alice reacts to own message as well
         SystemTime::shift(Duration::from_secs(10));
@@ -742,7 +742,7 @@ Here's my footer -- bob@example.net"
         expect_no_unwanted_events(&bob).await;
 
         assert_summary(&alice, "You reacted 🍿 to \"Party?\"").await;
-        assert_summary(&bob, "ALICE reacted 🍿 to \"Party?\"").await;
+        assert_summary(&bob, "~ALICE reacted 🍿 to \"Party?\"").await;
 
         // Alice sends a newer message, this overwrites reaction summaries
         SystemTime::shift(Duration::from_secs(10));
@@ -759,7 +759,7 @@ Here's my footer -- bob@example.net"
         bob.recv_msg_opt(&alice_send_reaction).await;
 
         assert_summary(&alice, "You reacted 🤘 to \"Party?\"").await;
-        assert_summary(&bob, "ALICE reacted 🤘 to \"Party?\"").await;
+        assert_summary(&bob, "~ALICE reacted 🤘 to \"Party?\"").await;
 
         // Retracted reactions remove all summary reactions
         SystemTime::shift(Duration::from_secs(10));

src/receive_imf.rs

@@ -360,7 +360,7 @@ pub(crate) async fn receive_imf_inner(
             let contact = Contact::get_by_id(context, from_id).await?;
             mime_parser.peerstate = Peerstate::from_addr(context, contact.get_addr()).await?;
         } else {
-            let to_id = to_ids.first().copied().unwrap_or_default();
+            let to_id = to_ids.first().copied().unwrap_or(ContactId::SELF);
             // handshake may mark contacts as verified and must be processed before chats are created
             res = observe_securejoin_on_other_device(context, &mime_parser, to_id)
                 .await
@@ -607,7 +607,7 @@ pub(crate) async fn receive_imf_inner(
     }
 
     if let Some(replace_chat_id) = replace_chat_id {
-        context.emit_msgs_changed(replace_chat_id, MsgId::new(0));
+        context.emit_msgs_changed_without_msg_id(replace_chat_id);
     } else if !chat_id.is_trash() {
         let fresh = received_msg.state == MessageState::InFresh;
         for msg_id in &received_msg.msg_ids {
@@ -621,7 +621,11 @@ pub(crate) async fn receive_imf_inner(
         .await;
 
     if let Some(is_bot) = mime_parser.is_bot {
-        from_id.mark_bot(context, is_bot).await?;
+        // If the message is auto-generated and was generated by Delta Chat,
+        // mark the contact as a bot.
+        if mime_parser.get_header(HeaderDef::ChatVersion).is_some() {
+            from_id.mark_bot(context, is_bot).await?;
+        }
     }
 
     Ok(Some(received_msg))
@@ -774,6 +778,11 @@ async fn add_parts(
         }
     }
 
+    if chat_id.is_none() && is_mdn {
+        chat_id = Some(DC_CHAT_ID_TRASH);
+        info!(context, "Message is an MDN (TRASH).",);
+    }
+
     if mime_parser.incoming {
         to_id = ContactId::SELF;
 
@@ -785,11 +794,6 @@ async fn add_parts(
             markseen_on_imap_table(context, rfc724_mid).await.ok();
         }
 
-        if chat_id.is_none() && is_mdn {
-            chat_id = Some(DC_CHAT_ID_TRASH);
-            info!(context, "Message is an MDN (TRASH).",);
-        }
-
         let create_blocked_default = if is_bot {
             Blocked::Not
         } else {
@@ -942,14 +946,11 @@ async fn add_parts(
                 chat_id = Some(chat.id);
                 chat_id_blocked = chat.blocked;
             } else if allow_creation {
-                if let Ok(chat) = ChatIdBlocked::get_for_contact(context, from_id, create_blocked)
+                let chat = ChatIdBlocked::get_for_contact(context, from_id, create_blocked)
                     .await
-                    .context("Failed to get (new) chat for contact")
-                    .log_err(context)
-                {
-                    chat_id = Some(chat.id);
-                    chat_id_blocked = chat.blocked;
-                }
+                    .context("Failed to get (new) chat for contact")?;
+                chat_id = Some(chat.id);
+                chat_id_blocked = chat.blocked;
             }
 
             if let Some(chat_id) = chat_id {
@@ -973,7 +974,6 @@ async fn add_parts(
                 // the 1:1 chat accordingly.
                 let chat = match is_partial_download.is_none()
                     && mime_parser.get_header(HeaderDef::SecureJoin).is_none()
-                    && !is_mdn
                 {
                     true => Some(Chat::load_from_db(context, chat_id).await?)
                         .filter(|chat| chat.typ == Chattype::Single),
@@ -1034,9 +1034,13 @@ async fn add_parts(
         // the mail is on the IMAP server, probably it is also delivered.
         // We cannot recreate other states (read, error).
         state = MessageState::OutDelivered;
-        to_id = to_ids.first().copied().unwrap_or_default();
+        to_id = to_ids.first().copied().unwrap_or(ContactId::SELF);
 
-        let self_sent = to_ids.len() == 1 && to_ids.contains(&ContactId::SELF);
+        // Older Delta Chat versions with core <=1.152.2 only accepted
+        // self-sent messages in Saved Messages with own address in the `To` field.
+        // New Delta Chat versions may use empty `To` field
+        // with only a single `hidden-recipients` group in this case.
+        let self_sent = to_ids.len() <= 1 && to_id == ContactId::SELF;
 
         if mime_parser.sync_items.is_some() && self_sent {
             chat_id = Some(DC_CHAT_ID_TRASH);
@@ -1142,9 +1146,8 @@ async fn add_parts(
                         chat_id = Some(id);
                         chat_id_blocked = blocked;
                     }
-                } else if let Ok(chat) =
-                    ChatIdBlocked::get_for_contact(context, to_id, Blocked::Not).await
-                {
+                } else {
+                    let chat = ChatIdBlocked::get_for_contact(context, to_id, Blocked::Not).await?;
                     chat_id = Some(chat.id);
                     chat_id_blocked = chat.blocked;
                 }
@@ -1160,7 +1163,7 @@ async fn add_parts(
             if chat_id_blocked != Blocked::Not {
                 if let Some(chat_id) = chat_id {
                     chat_id.unblock_ex(context, Nosync).await?;
-                    chat_id_blocked = Blocked::Not;
+                    // Not assigning `chat_id_blocked = Blocked::Not` to avoid unused_assignments warning.
                 }
             }
         }
@@ -1178,26 +1181,6 @@ async fn add_parts(
             .await?;
         }
 
-        if chat_id.is_none() && self_sent {
-            // from_id==to_id==ContactId::SELF - this is a self-sent messages,
-            // maybe an Autocrypt Setup Message
-            if let Ok(chat) = ChatIdBlocked::get_for_contact(context, ContactId::SELF, Blocked::Not)
-                .await
-                .context("Failed to get (new) chat for contact")
-                .log_err(context)
-            {
-                chat_id = Some(chat.id);
-                chat_id_blocked = chat.blocked;
-            }
-
-            if let Some(chat_id) = chat_id {
-                if Blocked::Not != chat_id_blocked {
-                    chat_id.unblock_ex(context, Nosync).await?;
-                    // Not assigning `chat_id_blocked = Blocked::Not` to avoid unused_assignments warning.
-                }
-            }
-        }
-
         if chat_id.is_none() {
             // Check if the message belongs to a broadcast list.
             if let Some(mailinglist_header) = mime_parser.get_mailinglist_header() {
@@ -1213,6 +1196,21 @@ async fn add_parts(
                 );
             }
         }
+
+        if chat_id.is_none() && self_sent {
+            // from_id==to_id==ContactId::SELF - this is a self-sent messages,
+            // maybe an Autocrypt Setup Message
+            let chat = ChatIdBlocked::get_for_contact(context, ContactId::SELF, Blocked::Not)
+                .await
+                .context("Failed to get (new) chat for contact")?;
+
+            chat_id = Some(chat.id);
+            // Not assigning `chat_id_blocked = chat.blocked` to avoid unused_assignments warning.
+
+            if Blocked::Not != chat.blocked {
+                chat.id.unblock_ex(context, Nosync).await?;
+            }
+        }
     }
 
     if fetching_existing_messages && mime_parser.decrypting_failed {
@@ -1232,7 +1230,7 @@ async fn add_parts(
     }
 
     let orig_chat_id = chat_id;
-    let mut chat_id = if is_mdn || is_reaction {
+    let mut chat_id = if is_reaction {
         DC_CHAT_ID_TRASH
     } else {
         chat_id.unwrap_or_else(|| {
@@ -1406,10 +1404,11 @@ async fn add_parts(
     // we save the full mime-message and add a flag
     // that the ui should show button to display the full message.
 
-    // a flag used to avoid adding "show full message" button to multiple parts of the message.
-    let mut save_mime_modified = mime_parser.is_mime_modified;
+    // We add "Show Full Message" button to the last message bubble (part) if this flag evaluates to
+    // `true` finally.
+    let mut save_mime_modified = false;
 
-    let mime_headers = if save_mime_headers || save_mime_modified {
+    let mime_headers = if save_mime_headers || mime_parser.is_mime_modified {
         let headers = if !mime_parser.decoded_data.is_empty() {
             mime_parser.decoded_data.clone()
         } else {
@@ -1475,7 +1474,8 @@ async fn add_parts(
         }
     }
 
-    for part in &mime_parser.parts {
+    let mut parts = mime_parser.parts.iter().peekable();
+    while let Some(part) = parts.next() {
         if part.is_reaction {
             let reaction_str = simplify::remove_footers(part.msg.as_str());
             let is_incoming_fresh = mime_parser.incoming && !seen && !fetching_existing_messages;
@@ -1519,14 +1519,11 @@ async fn add_parts(
         } else {
             (&part.msg, part.typ)
         };
-
         let part_is_empty =
             typ == Viewtype::Text && msg.is_empty() && part.param.get(Param::Quote).is_none();
-        let mime_modified = save_mime_modified && !part_is_empty;
-        if mime_modified {
-            // Avoid setting mime_modified for more than one part.
-            save_mime_modified = false;
-        }
+
+        save_mime_modified |= mime_parser.is_mime_modified && !part_is_empty && !hidden;
+        let save_mime_modified = save_mime_modified && parts.peek().is_none();
 
         if part.typ == Viewtype::Text {
             let msg_raw = part.msg_raw.as_ref().cloned().unwrap_or_default();
@@ -1546,8 +1543,7 @@ async fn add_parts(
 
         // If you change which information is skipped if the message is trashed,
         // also change `MsgId::trash()` and `delete_expired_messages()`
-        let trash =
-            chat_id.is_trash() || (is_location_kml && msg.is_empty() && typ == Viewtype::Text);
+        let trash = chat_id.is_trash() || (is_location_kml && part_is_empty && !save_mime_modified);
 
         let row_id = context
             .sql
@@ -1610,14 +1606,14 @@ RETURNING id
                     },
                     hidden,
                     part.bytes as isize,
-                    if (save_mime_headers || mime_modified) && !trash {
+                    if (save_mime_headers || save_mime_modified) && !trash {
                         mime_headers.clone()
                     } else {
                         Vec::new()
                     },
                     mime_in_reply_to,
                     mime_references,
-                    mime_modified,
+                    save_mime_modified,
                     part.error.as_deref().unwrap_or_default(),
                     ephemeral_timer,
                     ephemeral_timestamp,
@@ -1692,12 +1688,7 @@ RETURNING id
         "Message has {icnt} parts and is assigned to chat #{chat_id}."
     );
 
-    // new outgoing message from another device marks the chat as noticed.
-    if !mime_parser.incoming && !chat_id.is_special() {
-        chat::marknoticed_chat_if_older_than(context, chat_id, sort_timestamp).await?;
-    }
-
-    if !is_mdn {
+    if !chat_id.is_trash() {
         let mut chat = Chat::load_from_db(context, chat_id).await?;
 
         // In contrast to most other update-timestamps,
@@ -2216,9 +2207,7 @@ async fn apply_group_changes(
             if let Some(contact_id) =
                 Contact::lookup_id_by_addr(context, added_addr, Origin::Unknown).await?
             {
-                if !recreate_member_list {
-                    added_id = Some(contact_id);
-                }
+                added_id = Some(contact_id);
                 is_new_member = !chat_contacts.contains(&contact_id);
             } else {
                 warn!(context, "Added {added_addr:?} has no contact id.");
@@ -2286,12 +2275,16 @@ async fn apply_group_changes(
             new_members.insert(from_id);
         }
 
+        // These are for adding info messages about implicit membership changes, so they are only
+        // filled when such messages are needed.
+        let mut added_ids = HashSet::<ContactId>::new();
+        let mut removed_ids = HashSet::<ContactId>::new();
+
         if !recreate_member_list {
-            let mut diff = HashSet::<ContactId>::new();
             if sync_member_list {
-                diff = new_members.difference(&chat_contacts).copied().collect();
+                added_ids = new_members.difference(&chat_contacts).copied().collect();
             } else if let Some(added_id) = added_id {
-                diff.insert(added_id);
+                added_ids.insert(added_id);
             }
             new_members.clone_from(&chat_contacts);
             // Don't delete any members locally, but instead add absent ones to provide group
@@ -2305,33 +2298,62 @@ async fn apply_group_changes(
             // will likely recreate the member list from the next received message. The problem
             // occurs only if that "somebody" managed to reply earlier. Really, it's a problem for
             // big groups with high message rate, but let it be for now.
-            new_members.extend(diff.clone());
-            if let Some(added_id) = added_id {
-                diff.remove(&added_id);
-            }
-            if !diff.is_empty() {
-                warn!(context, "Implicit addition of {diff:?} to chat {chat_id}.");
-            }
-            group_changes_msgs.reserve(diff.len());
-            for contact_id in diff {
-                let contact = Contact::get_by_id(context, contact_id).await?;
-                group_changes_msgs.push(
-                    stock_str::msg_add_member_local(
-                        context,
-                        contact.get_addr(),
-                        ContactId::UNDEFINED,
-                    )
-                    .await,
-                );
-            }
+            new_members.extend(added_ids.clone());
         }
         if let Some(removed_id) = removed_id {
             new_members.remove(&removed_id);
         }
         if recreate_member_list {
-            info!(
+            if self_added {
+                // ... then `better_msg` is already set.
+            } else if chat.blocked == Blocked::Request || !chat_contacts.contains(&ContactId::SELF)
+            {
+                warn!(context, "Implicit addition of SELF to chat {chat_id}.");
+                group_changes_msgs.push(
+                    stock_str::msg_add_member_local(
+                        context,
+                        &context.get_primary_self_addr().await?,
+                        ContactId::UNDEFINED,
+                    )
+                    .await,
+                );
+            } else {
+                added_ids = new_members.difference(&chat_contacts).copied().collect();
+                removed_ids = chat_contacts.difference(&new_members).copied().collect();
+            }
+        }
+
+        if let Some(added_id) = added_id {
+            added_ids.remove(&added_id);
+        }
+        if let Some(removed_id) = removed_id {
+            removed_ids.remove(&removed_id);
+        }
+        if !added_ids.is_empty() {
+            warn!(
                 context,
-                "Recreating chat {chat_id} member list with {new_members:?}.",
+                "Implicit addition of {added_ids:?} to chat {chat_id}."
+            );
+        }
+        if !removed_ids.is_empty() {
+            warn!(
+                context,
+                "Implicit removal of {removed_ids:?} from chat {chat_id}."
+            );
+        }
+        group_changes_msgs.reserve(added_ids.len() + removed_ids.len());
+        for contact_id in added_ids {
+            let contact = Contact::get_by_id(context, contact_id).await?;
+            group_changes_msgs.push(
+                stock_str::msg_add_member_local(context, contact.get_addr(), ContactId::UNDEFINED)
+                    .await,
+            );
+        }
+        for contact_id in removed_ids {
+            let contact = Contact::get_by_id(context, contact_id).await?;
+            group_changes_msgs.push(
+                stock_str::msg_del_member_local(context, contact.get_addr(), ContactId::UNDEFINED)
+                    .await,
             );
         }
 

src/receive_imf/tests.rs

@@ -165,7 +165,7 @@ async fn test_adhoc_group_show_accepted_contact_accepted() {
     chat_id.accept(&t).await.unwrap();
     let chat = chat::Chat::load_from_db(&t, chat_id).await.unwrap();
     assert_eq!(chat.typ, Chattype::Single);
-    assert_eq!(chat.name, "Bob");
+    assert_eq!(chat.name, "~Bob");
     assert_eq!(chat::get_chat_contacts(&t, chat_id).await.unwrap().len(), 1);
     assert_eq!(chat::get_chat_msgs(&t, chat_id).await.unwrap().len(), 1);
 
@@ -584,7 +584,7 @@ async fn test_escaped_recipients() {
     .unwrap();
     let contact = Contact::get_by_id(&t, carl_contact_id).await.unwrap();
     assert_eq!(contact.get_name(), "");
-    assert_eq!(contact.get_display_name(), "h2");
+    assert_eq!(contact.get_display_name(), "~h2");
 
     let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
     let msg = Message::load_from_db(&t, chats.get_msg_id(0).unwrap().unwrap())
@@ -631,7 +631,7 @@ async fn test_cc_to_contact() {
     .unwrap();
     let contact = Contact::get_by_id(&t, carl_contact_id).await.unwrap();
     assert_eq!(contact.get_name(), "");
-    assert_eq!(contact.get_display_name(), "Carl");
+    assert_eq!(contact.get_display_name(), "~Carl");
 }
 
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -868,18 +868,10 @@ async fn test_parse_ndn_group_msg() -> Result<()> {
     assert_eq!(msg.state, MessageState::OutFailed);
 
     let msgs = chat::get_chat_msgs(&t, msg.chat_id).await?;
-    let msg_id = if let ChatItem::Message { msg_id } = msgs.last().unwrap() {
-        msg_id
-    } else {
+    let ChatItem::Message { msg_id } = *msgs.last().unwrap() else {
         panic!("Wrong item type");
     };
-    let last_msg = Message::load_from_db(&t, *msg_id).await?;
-
-    assert_eq!(
-        last_msg.text,
-        stock_str::failed_sending_to(&t, "assidhfaaspocwaeofi@gmail.com").await
-    );
-    assert_eq!(last_msg.from_id, ContactId::INFO);
+    assert_eq!(msg_id, msg.id);
     Ok(())
 }
 
@@ -1019,8 +1011,8 @@ async fn test_github_mailing_list() -> Result<()> {
     let contact2 = Contact::get_by_id(&t.ctx, msg2.from_id).await?;
     assert_eq!(contact2.get_addr(), "notifications@github.com");
 
-    assert_eq!(msg1.get_override_sender_name().unwrap(), "Max Mustermann");
-    assert_eq!(msg2.get_override_sender_name().unwrap(), "Github");
+    assert_eq!(msg1.get_override_sender_name().unwrap(), "~Max Mustermann");
+    assert_eq!(msg2.get_override_sender_name().unwrap(), "~Github");
     Ok(())
 }
 
@@ -2085,7 +2077,7 @@ async fn check_alias_reply(from_dc: bool, chat_request: bool, group_request: boo
     }
     assert_eq!(
         answer.get_override_sender_name().unwrap(),
-        "bob@example.net"
+        "~bob@example.net"
     ); // Bob is not part of the group, so override-sender-name should be set
 
     // Check that Claire also gets the message in the same chat.
@@ -2097,7 +2089,7 @@ async fn check_alias_reply(from_dc: bool, chat_request: bool, group_request: boo
     assert_eq!(answer.chat_id, request.chat_id);
     assert_eq!(
         answer.get_override_sender_name().unwrap(),
-        "bob@example.net"
+        "~bob@example.net"
     );
 }
 
@@ -2208,6 +2200,30 @@ Message content",
     assert_ne!(msg.chat_id, t.get_self_chat().await.id);
 }
 
+/// Tests that message with hidden recipients is assigned to Saved Messages chat.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_hidden_recipients_self_chat() {
+    let t = TestContext::new_alice().await;
+
+    receive_imf(
+        &t,
+        b"Subject: s
+Chat-Version: 1.0
+Message-ID: <foobar@localhost>
+To: hidden-recipients:;
+From: <alice@example.org>
+
+Message content",
+        false,
+    )
+    .await
+    .unwrap();
+
+    let msg = t.get_last_msg().await;
+    assert_eq!(msg.chat_id, t.get_self_chat().await.id);
+    assert_eq!(msg.to_id, ContactId::SELF);
+}
+
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn test_no_unencrypted_name_in_self_chat() -> Result<()> {
     let mut tcm = TestContextManager::new();
@@ -2297,12 +2313,12 @@ Second signature";
     receive_imf(&alice, first_message, false).await?;
     let contact = Contact::get_by_id(&alice, bob_contact_id).await?;
     assert_eq!(contact.get_status(), "First signature");
-    assert_eq!(contact.get_display_name(), "Bob1");
+    assert_eq!(contact.get_display_name(), "~Bob1");
 
     receive_imf(&alice, second_message, false).await?;
     let contact = Contact::get_by_id(&alice, bob_contact_id).await?;
     assert_eq!(contact.get_status(), "Second signature");
-    assert_eq!(contact.get_display_name(), "Bob2");
+    assert_eq!(contact.get_display_name(), "~Bob2");
 
     // Duplicate message, should be ignored
     receive_imf(&alice, first_message, false).await?;
@@ -2310,7 +2326,7 @@ Second signature";
     // No change because last message is duplicate of the first.
     let contact = Contact::get_by_id(&alice, bob_contact_id).await?;
     assert_eq!(contact.get_status(), "Second signature");
-    assert_eq!(contact.get_display_name(), "Bob2");
+    assert_eq!(contact.get_display_name(), "~Bob2");
 
     Ok(())
 }
@@ -3835,6 +3851,61 @@ async fn test_messed_up_message_id() -> Result<()> {
     Ok(())
 }
 
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn test_big_forwarded_with_big_attachment() -> Result<()> {
+    let t = &TestContext::new_bob().await;
+
+    let raw = include_bytes!("../../test-data/message/big_forwarded_with_big_attachment.eml");
+    let rcvd = receive_imf(t, raw, false).await?.unwrap();
+    assert_eq!(rcvd.msg_ids.len(), 3);
+
+    let msg = Message::load_from_db(t, rcvd.msg_ids[0]).await?;
+    assert_eq!(msg.get_viewtype(), Viewtype::Text);
+    assert_eq!(msg.get_text(), "Hello!");
+    assert!(!msg.has_html());
+
+    let msg = Message::load_from_db(t, rcvd.msg_ids[1]).await?;
+    assert_eq!(msg.get_viewtype(), Viewtype::Text);
+    assert!(msg
+        .get_text()
+        .starts_with("this text with 42 chars is just repeated."));
+    assert!(msg.get_text().ends_with("[...]"));
+    assert!(!msg.has_html());
+
+    let msg = Message::load_from_db(t, rcvd.msg_ids[2]).await?;
+    assert_eq!(msg.get_viewtype(), Viewtype::File);
+    assert!(msg.has_html());
+    let html = msg.id.get_html(t).await?.unwrap();
+    let tail = html
+        .split_once("Hello!")
+        .unwrap()
+        .1
+        .split_once("From: AAA")
+        .unwrap()
+        .1
+        .split_once("aaa@example.org")
+        .unwrap()
+        .1
+        .split_once("To: Alice")
+        .unwrap()
+        .1
+        .split_once("alice@example.org")
+        .unwrap()
+        .1
+        .split_once("Subject: Some subject")
+        .unwrap()
+        .1
+        .split_once("Date: Fri, 2 Jun 2023 12:29:17 +0000")
+        .unwrap()
+        .1;
+    assert_eq!(
+        tail.matches("this text with 42 chars is just repeated.")
+            .count(),
+        128
+    );
+    Ok(())
+}
+
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn test_mua_user_adds_member() -> Result<()> {
     let t = TestContext::new_alice().await;
@@ -4145,7 +4216,7 @@ async fn test_dont_recreate_contacts_on_add_remove() -> Result<()> {
 }
 
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-async fn test_recreate_contact_list_on_missing_message() -> Result<()> {
+async fn test_recreate_contact_list_on_missing_messages() -> Result<()> {
     let alice = TestContext::new_alice().await;
     let bob = TestContext::new_bob().await;
     let chat_id = create_group_chat(&alice, ProtectionStatus::Unprotected, "Group").await?;
@@ -4170,25 +4241,33 @@ async fn test_recreate_contact_list_on_missing_message() -> Result<()> {
     remove_contact_from_chat(&bob, bob_chat_id, bob_contact_fiona).await?;
     let remove_msg = bob.pop_sent_msg().await;
 
-    // bob adds a new member
+    // bob adds new members
     let bob_blue = Contact::create(&bob, "blue", "blue@example.net").await?;
     add_contact_to_chat(&bob, bob_chat_id, bob_blue).await?;
-
+    bob.pop_sent_msg().await;
+    let bob_orange = Contact::create(&bob, "orange", "orange@example.net").await?;
+    add_contact_to_chat(&bob, bob_chat_id, bob_orange).await?;
     let add_msg = bob.pop_sent_msg().await;
 
-    // alice only receives the addition of the member
+    // alice only receives the second member addition
     alice.recv_msg(&add_msg).await;
 
-    // since we missed a message, a new contact list should be build
-    assert_eq!(get_chat_contacts(&alice, chat_id).await?.len(), 3);
+    // since we missed messages, a new contact list should be build
+    assert_eq!(get_chat_contacts(&alice, chat_id).await?.len(), 4);
 
     // re-add fiona
     add_contact_to_chat(&alice, chat_id, alice_fiona).await?;
 
     // delayed removal of fiona shouldn't remove her
     alice.recv_msg_trash(&remove_msg).await;
-    assert_eq!(get_chat_contacts(&alice, chat_id).await?.len(), 4);
+    assert_eq!(get_chat_contacts(&alice, chat_id).await?.len(), 5);
 
+    alice
+        .golden_test_chat(
+            chat_id,
+            "receive_imf_recreate_contact_list_on_missing_messages",
+        )
+        .await;
     Ok(())
 }
 
@@ -4455,6 +4534,20 @@ async fn test_recreate_member_list_on_missing_add_of_self() -> Result<()> {
     send_text_msg(&alice, alice_chat_id, "4th message".to_string()).await?;
     bob.recv_msg(&alice.pop_sent_msg().await).await;
     assert!(!is_contact_in_chat(&bob, bob_chat_id, ContactId::SELF).await?);
+
+    // But if Bob left a long time ago, they must recreate the member list after missing a message.
+    SystemTime::shift(Duration::from_secs(3600));
+    send_text_msg(&alice, alice_chat_id, "5th message".to_string()).await?;
+    alice.pop_sent_msg().await;
+    send_text_msg(&alice, alice_chat_id, "6th message".to_string()).await?;
+    bob.recv_msg(&alice.pop_sent_msg().await).await;
+    assert!(is_contact_in_chat(&bob, bob_chat_id, ContactId::SELF).await?);
+
+    bob.golden_test_chat(
+        bob_chat_id,
+        "receive_imf_recreate_member_list_on_missing_add_of_self",
+    )
+    .await;
     Ok(())
 }
 
@@ -5137,7 +5230,7 @@ async fn test_list_from() -> Result<()> {
     let raw = include_bytes!("../../test-data/message/list-from.eml");
     let received = receive_imf(t, raw, false).await?.unwrap();
     let msg = Message::load_from_db(t, *received.msg_ids.last().unwrap()).await?;
-    assert_eq!(msg.get_override_sender_name().unwrap(), "ÖAMTC");
+    assert_eq!(msg.get_override_sender_name().unwrap(), "~ÖAMTC");
     let sender_contact = Contact::get_by_id(t, msg.from_id).await?;
     assert_eq!(
         sender_contact.get_display_name(),

src/scheduler/connectivity.rs

@@ -80,7 +80,7 @@ impl DetailedConnectivity {
             DetailedConnectivity::Uninitialized => Some(Connectivity::NotConnected),
             DetailedConnectivity::Connecting => Some(Connectivity::Connecting),
             DetailedConnectivity::Working => Some(Connectivity::Working),
-            DetailedConnectivity::InterruptingIdle => Some(Connectivity::Connected),
+            DetailedConnectivity::InterruptingIdle => Some(Connectivity::Working),
 
             // At this point IMAP has just connected,
             // but does not know yet if there are messages to download.
@@ -201,7 +201,7 @@ impl ConnectivityStore {
 }
 
 /// Set all folder states to InterruptingIdle in case they were `Idle` before.
-/// Called during `dc_maybe_network()` to make sure that `dc_all_work_done()`
+/// Called during `dc_maybe_network()` to make sure that `all_work_done()`
 /// returns false immediately after `dc_maybe_network()`.
 pub(crate) async fn idle_interrupted(inbox: ConnectivityStore, oboxes: Vec<ConnectivityStore>) {
     let mut connectivity_lock = inbox.0.lock().await;
@@ -535,7 +535,7 @@ impl Context {
     }
 
     /// Returns true if all background work is done.
-    pub async fn all_work_done(&self) -> bool {
+    async fn all_work_done(&self) -> bool {
         let lock = self.scheduler.inner.read().await;
         let stores: Vec<_> = match *lock {
             InnerSchedulerState::Started(ref sched) => sched
@@ -555,4 +555,23 @@ impl Context {
         }
         true
     }
+
+    /// Waits until background work is finished.
+    pub async fn wait_for_all_work_done(&self) {
+        // Ideally we could wait for connectivity change events,
+        // but sleep loop is good enough.
+
+        // First 100 ms sleep in chunks of 10 ms.
+        for _ in 0..10 {
+            if self.all_work_done().await {
+                break;
+            }
+            tokio::time::sleep(std::time::Duration::from_millis(10)).await;
+        }
+
+        // If we are not finished in 100 ms, keep waking up every 100 ms.
+        while !self.all_work_done().await {
+            tokio::time::sleep(std::time::Duration::from_millis(100)).await;
+        }
+    }
 }

src/securejoin.rs

@@ -454,6 +454,9 @@ pub(crate) async fn handle_securejoin_handshake(
                     .await?;
                 inviter_progress(context, contact_id, 800);
                 inviter_progress(context, contact_id, 1000);
+                // IMAP-delete the message to avoid handling it by another device and adding the
+                // member twice. Another device will know the member's key from Autocrypt-Gossip.
+                Ok(HandshakeMessage::Done)
             } else {
                 // Setup verified contact.
                 secure_connection_established(
@@ -468,8 +471,8 @@ pub(crate) async fn handle_securejoin_handshake(
                     .context("failed sending vc-contact-confirm message")?;
 
                 inviter_progress(context, contact_id, 1000);
+                Ok(HandshakeMessage::Ignore) // "Done" would delete the message and break multi-device (the key from Autocrypt-header is needed)
             }
-            Ok(HandshakeMessage::Ignore) // "Done" would delete the message and break multi-device (the key from Autocrypt-header is needed)
         }
         /*=======================================================
         ====             Bob - the joiner's side             ====
@@ -751,7 +754,7 @@ mod tests {
     use crate::imex::{imex, ImexMode};
     use crate::receive_imf::receive_imf;
     use crate::stock_str::{self, chat_protection_enabled};
-    use crate::test_utils::get_chat_msg;
+    use crate::test_utils::{get_chat_msg, TimeShiftFalsePositiveNote};
     use crate::test_utils::{TestContext, TestContextManager};
     use crate::tools::SystemTime;
     use std::collections::HashSet;
@@ -798,6 +801,8 @@ mod tests {
     }
 
     async fn test_setup_contact_ex(case: SetupContactCase) {
+        let _n = TimeShiftFalsePositiveNote;
+
         let mut tcm = TestContextManager::new();
         let alice = tcm.alice().await;
         let alice_addr = &alice.get_config(Config::Addr).await.unwrap().unwrap();
@@ -1351,6 +1356,8 @@ mod tests {
         // explicit user action, the Auto-Submitted header shouldn't be present. Otherwise it would
         // be strange to have it in "member-added" messages of verified groups only.
         assert!(msg.get_header(HeaderDef::AutoSubmitted).is_none());
+        // This is a two-member group, but Alice must Autocrypt-gossip to her other devices.
+        assert!(msg.get_header(HeaderDef::AutocryptGossip).is_some());
 
         {
             // Now Alice's chat with Bob should still be hidden, the verified message should

src/smtp.rs

@@ -21,7 +21,6 @@ use crate::mimefactory::MimeFactory;
 use crate::net::proxy::ProxyConfig;
 use crate::net::session::SessionBufStream;
 use crate::scheduler::connectivity::ConnectivityStore;
-use crate::sql;
 use crate::stock_str::unencrypted_email;
 use crate::tools::{self, time_elapsed};
 
@@ -585,18 +584,16 @@ async fn send_mdn_rfc724_mid(
             info!(context, "Successfully sent MDN for {rfc724_mid}.");
             context
                 .sql
-                .execute("DELETE FROM smtp_mdns WHERE rfc724_mid = ?", (rfc724_mid,))
+                .transaction(|transaction| {
+                    let mut stmt =
+                        transaction.prepare("DELETE FROM smtp_mdns WHERE rfc724_mid = ?")?;
+                    stmt.execute((rfc724_mid,))?;
+                    for additional_rfc724_mid in additional_rfc724_mids {
+                        stmt.execute((additional_rfc724_mid,))?;
+                    }
+                    Ok(())
+                })
                 .await?;
-            if !additional_rfc724_mids.is_empty() {
-                let q = format!(
-                    "DELETE FROM smtp_mdns WHERE rfc724_mid IN({})",
-                    sql::repeat_vars(additional_rfc724_mids.len())
-                );
-                context
-                    .sql
-                    .execute(&q, rusqlite::params_from_iter(additional_rfc724_mids))
-                    .await?;
-            }
             Ok(true)
         }
         SendResult::Retry => {

src/sql.rs

@@ -19,6 +19,7 @@ use crate::location::delete_orphaned_poi_locations;
 use crate::log::LogExt;
 use crate::message::{Message, MsgId};
 use crate::net::dns::prune_dns_cache;
+use crate::net::http::http_cache_cleanup;
 use crate::net::prune_connection_history;
 use crate::param::{Param, Params};
 use crate::peerstate::Peerstate;
@@ -720,6 +721,12 @@ pub async fn housekeeping(context: &Context) -> Result<()> {
         warn!(context, "Can't set config: {e:#}.");
     }
 
+    http_cache_cleanup(context)
+        .await
+        .context("Failed to cleanup HTTP cache")
+        .log_err(context)
+        .ok();
+
     if let Err(err) = remove_unused_files(context).await {
         warn!(
             context,
@@ -846,6 +853,22 @@ pub async fn remove_unused_files(context: &Context) -> Result<()> {
         .await
         .context("housekeeping: failed to SELECT value FROM config")?;
 
+    context
+        .sql
+        .query_map(
+            "SELECT blobname FROM http_cache",
+            (),
+            |row| row.get::<_, String>(0),
+            |rows| {
+                for row in rows {
+                    maybe_add_file(&mut files_in_use, &row?);
+                }
+                Ok(())
+            },
+        )
+        .await
+        .context("Failed to SELECT blobname FROM http_cache")?;
+
     info!(context, "{} files in use.", files_in_use.len());
     /* go through directories and delete unused files */
     let blobdir = context.get_blobdir();
@@ -864,7 +887,6 @@ pub async fn remove_unused_files(context: &Context) -> Result<()> {
 
                     if p == blobdir
                         && (is_file_in_use(&files_in_use, None, &name_s)
-                            || is_file_in_use(&files_in_use, Some(".increation"), &name_s)
                             || is_file_in_use(&files_in_use, Some(".waveform"), &name_s)
                             || is_file_in_use(&files_in_use, Some("-preview.jpg"), &name_s))
                     {

src/sql/migrations.rs

@@ -1088,6 +1088,56 @@ CREATE INDEX msgs_status_updates_index2 ON msgs_status_updates (uid);
         .await?;
     }
 
+    inc_and_check(&mut migration_version, 125)?;
+    if dbversion < migration_version {
+        sql.execute_migration(
+            "CREATE TABLE http_cache (
+                url TEXT PRIMARY KEY,
+                expires INTEGER NOT NULL, -- When the cache entry is considered expired, timestamp in seconds.
+                blobname TEXT NOT NULL,
+                mimetype TEXT NOT NULL DEFAULT '', -- MIME type extracted from Content-Type header.
+                encoding TEXT NOT NULL DEFAULT '' -- Encoding from Content-Type header.
+            ) STRICT",
+            migration_version,
+        )
+        .await?;
+    }
+
+    inc_and_check(&mut migration_version, 126)?;
+    if dbversion < migration_version {
+        // Recreate http_cache table with new `stale` column.
+        sql.execute_migration(
+            "DROP TABLE http_cache;
+             CREATE TABLE http_cache (
+                url TEXT PRIMARY KEY,
+                expires INTEGER NOT NULL, -- When the cache entry is considered expired, timestamp in seconds.
+                stale INTEGER NOT NULL, -- When the cache entry is considered stale, timestamp in seconds.
+                blobname TEXT NOT NULL,
+                mimetype TEXT NOT NULL DEFAULT '', -- MIME type extracted from Content-Type header.
+                encoding TEXT NOT NULL DEFAULT '' -- Encoding from Content-Type header.
+            ) STRICT",
+            migration_version,
+        )
+        .await?;
+    }
+
+    inc_and_check(&mut migration_version, 127)?;
+    if dbversion < migration_version {
+        // Existing chatmail configurations having `delete_server_after` disabled should get
+        // `bcc_self` enabled, they may be multidevice configurations because before,
+        // `delete_server_after` was set to 0 upon a backup export for them, but together with this
+        // migration `bcc_self` is enabled instead (whose default is changed to 0 for chatmail). We
+        // don't check `is_chatmail` for simplicity.
+        sql.execute_migration(
+            "INSERT OR IGNORE INTO config (keyname, value)
+             SELECT 'bcc_self', '1'
+             FROM config WHERE keyname='delete_server_after' AND value='0'
+            ",
+            migration_version,
+        )
+        .await?;
+    }
+
     let new_version = sql
         .get_raw_config_int(VERSION_CFG)
         .await?

src/stock_str.rs

@@ -149,6 +149,7 @@ pub enum StockMessage {
     #[strum(props(fallback = "Message from %1$s"))]
     SubjectForNewContact = 73,
 
+    /// Unused. Was used in group chat status messages.
     #[strum(props(fallback = "Failed to send message to %1$s."))]
     FailedSendingTo = 74,
 
@@ -430,6 +431,9 @@ pub enum StockMessage {
     #[strum(props(fallback = "%1$s reacted %2$s to \"%3$s\""))]
     MsgReactedBy = 177,
 
+    #[strum(props(fallback = "Member %1$s removed."))]
+    MsgDelMember = 178,
+
     #[strum(props(fallback = "Establishing guaranteed end-to-end encryption, please wait…"))]
     SecurejoinWait = 190,
 
@@ -710,7 +714,11 @@ pub(crate) async fn msg_del_member_local(
             .unwrap_or_else(|_| addr.to_string()),
         _ => addr.to_string(),
     };
-    if by_contact == ContactId::SELF {
+    if by_contact == ContactId::UNDEFINED {
+        translated(context, StockMessage::MsgDelMember)
+            .await
+            .replace1(whom)
+    } else if by_contact == ContactId::SELF {
         translated(context, StockMessage::MsgYouDelMember)
             .await
             .replace1(whom)
@@ -814,7 +822,7 @@ pub(crate) async fn secure_join_started(
         translated(context, StockMessage::SecureJoinStarted)
             .await
             .replace1(&contact.get_name_n_addr())
-            .replace2(contact.get_display_name())
+            .replace2(&contact.get_display_name())
     } else {
         format!("secure_join_started: unknown contact {inviter_contact_id}")
     }
@@ -980,13 +988,6 @@ pub(crate) async fn subject_for_new_contact(context: &Context, self_name: &str)
         .replace1(self_name)
 }
 
-/// Stock string: `Failed to send message to %1$s.`.
-pub(crate) async fn failed_sending_to(context: &Context, name: &str) -> String {
-    translated(context, StockMessage::FailedSendingTo)
-        .await
-        .replace1(name)
-}
-
 /// Stock string: `Message deletion timer is disabled.`.
 pub(crate) async fn msg_ephemeral_timer_disabled(
     context: &Context,

src/sync.rs

@@ -566,6 +566,10 @@ mod tests {
         assert!(token::exists(&alice2, token::Namespace::Auth, "testtoken").await?);
         assert_eq!(Chatlist::try_load(&alice2, 0, None, None).await?.len(), 0);
 
+        // Sync messages are "auto-generated", but they mustn't make the self-contact a bot.
+        let self_contact = alice2.add_or_lookup_contact(&alice2).await;
+        assert!(!self_contact.is_bot());
+
         // the same sync message sent to bob must not be executed
         let bob = TestContext::new_bob().await;
         bob.recv_msg(&sent_msg).await;

src/test_utils.rs

@@ -770,7 +770,7 @@ impl TestContext {
         } else {
             assert_eq!(
                 actual, expected,
-                "To update the expected value, run `UPDATE_GOLDEN_TESTS=1 cargo test`"
+                "To update the expected value, run `UPDATE_GOLDEN_TESTS=1 cargo nextest run`"
             );
         }
     }
@@ -1351,6 +1351,24 @@ async fn write_msg(context: &Context, prefix: &str, msg: &Message, buf: &mut Str
     .unwrap();
 }
 
+/// When dropped after a test failure,
+/// prints a note about a possible false-possible caused by SystemTime::shift().
+pub(crate) struct TimeShiftFalsePositiveNote;
+impl Drop for TimeShiftFalsePositiveNote {
+    fn drop(&mut self) {
+        if std::thread::panicking() {
+            let green = nu_ansi_term::Color::Green.normal();
+            println!("{}", green.paint(
+            "\nNOTE: This test failure may be a false-positive, caused by tests running in parallel.
+The issue is that `SystemTime::shift()` (a utility function for tests) changes the time for all threads doing tests, and not only for the running test.
+Until the false-positive is fixed:
+- Use `cargo test -- --test-threads 1` instead of `cargo test`
+- Or use `cargo nextest run` (install with `cargo install cargo-nextest --locked`)\n")
+            );
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;

src/tests/aeap.rs

@@ -288,7 +288,8 @@ async fn check_that_transition_worked(
 
         let info_msg = get_last_info_msg(bob, *group).await.unwrap();
         let expected_text =
-            stock_str::aeap_addr_changed(bob, name, old_alice_addr, new_alice_addr).await;
+            stock_str::aeap_addr_changed(bob, &format!("{name}"), old_alice_addr, new_alice_addr)
+                .await;
         assert_eq!(info_msg.text, expected_text);
         assert_eq!(info_msg.from_id, ContactId::INFO);
 

src/tests/verified_chats.rs

@@ -947,8 +947,7 @@ async fn test_no_unencrypted_name_if_encrypted() -> Result<()> {
 
         let msg = alice.recv_msg(msg).await;
         let contact = Contact::get_by_id(&alice, msg.from_id).await?;
-
-        assert_eq!(Contact::get_display_name(&contact), "Bob Smith");
+        assert_eq!(Contact::get_display_name(&contact), "~Bob Smith");
     }
     Ok(())
 }

src/webxdc.rs

@@ -421,6 +421,7 @@ impl Context {
                     notify_list.get(&self_addr).or_else(|| notify_list.get("*"))
                 {
                     self.emit_event(EventType::IncomingWebxdcNotify {
+                        chat_id: instance.chat_id,
                         contact_id: from_id,
                         msg_id: notify_msg_id,
                         text: notify_text.clone(),

test-data/golden/receive_imf_recreate_contact_list_on_missing_messages

@@ -0,0 +1,8 @@
+Group#Chat#10: Group [5 member(s)] 
+--------------------------------------------------------------------------------
+Msg#10: Me (Contact#Contact#Self): populate  √
+Msg#11: info (Contact#Contact#Info): Member blue@example.net added. [NOTICED][INFO]
+Msg#12: info (Contact#Contact#Info): Member fiona (fiona@example.net) removed. [NOTICED][INFO]
+Msg#13: bob (Contact#Contact#11): Member orange@example.net added by bob (bob@example.net). [FRESH][INFO]
+Msg#14: Me (Contact#Contact#Self): You added member fiona (fiona@example.net). [INFO] o
+--------------------------------------------------------------------------------

test-data/golden/receive_imf_recreate_member_list_on_missing_add_of_self

@@ -0,0 +1,9 @@
+Group#Chat#10: Group [2 member(s)] 
+--------------------------------------------------------------------------------
+Msg#10: info (Contact#Contact#Info): Member Me (bob@example.net) added. [NOTICED][INFO]
+Msg#11:  (Contact#Contact#10): second message [FRESH]
+Msg#12🔒: Me (Contact#Contact#Self): You left the group. [INFO] √
+Msg#13:  (Contact#Contact#10): 4th message [FRESH]
+Msg#14: info (Contact#Contact#Info): Member Me (bob@example.net) added. [NOTICED][INFO]
+Msg#15:  (Contact#Contact#10): 6th message [FRESH]
+--------------------------------------------------------------------------------

test-data/golden/test_old_message_1

@@ -1,6 +1,6 @@
-Single#Chat#10: Bob [bob@example.net] 
+Single#Chat#10: ~Bob [bob@example.net] 
 --------------------------------------------------------------------------------
 Msg#10: info (Contact#Contact#Info): Messages are guaranteed to be end-to-end encrypted from now on. [NOTICED][INFO 🛡️]
-Msg#11: info (Contact#Contact#Info): Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
+Msg#11: info (Contact#Contact#Info): ~Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
 Msg#12:  (Contact#Contact#10): Message from Thunderbird [SEEN]
 --------------------------------------------------------------------------------

test-data/golden/test_old_message_2

@@ -1,7 +1,7 @@
-Single#Chat#10: Bob [bob@example.net] 
+Single#Chat#10: ~Bob [bob@example.net] 
 --------------------------------------------------------------------------------
 Msg#10: info (Contact#Contact#Info): Messages are guaranteed to be end-to-end encrypted from now on. [NOTICED][INFO 🛡️]
-Msg#11: info (Contact#Contact#Info): Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
+Msg#11: info (Contact#Contact#Info): ~Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
 Msg#12:  (Contact#Contact#10): Somewhat old message [FRESH]
 Msg#13:  (Contact#Contact#10): Even older message, that must NOT be shown before the info message [SEEN]
 --------------------------------------------------------------------------------

test-data/golden/test_old_message_3

@@ -1,8 +1,8 @@
-Single#Chat#10: Bob [bob@example.net] 🛡️
+Single#Chat#10: ~Bob [bob@example.net] 🛡️
 --------------------------------------------------------------------------------
 Msg#10: info (Contact#Contact#Info): Messages are guaranteed to be end-to-end encrypted from now on. [NOTICED][INFO 🛡️]
 Msg#11🔒:  (Contact#Contact#10): Heyho from my verified device! [FRESH]
-Msg#12: info (Contact#Contact#Info): Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
+Msg#12: info (Contact#Contact#Info): ~Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
 Msg#13:  (Contact#Contact#10): Old, unverified message [SEEN]
 Msg#14: info (Contact#Contact#Info): Messages are guaranteed to be end-to-end encrypted from now on. [NOTICED][INFO 🛡️]
 --------------------------------------------------------------------------------

test-data/golden/test_old_message_5

@@ -1,4 +1,4 @@
-Single#Chat#10: Bob [bob@example.net] 
+Single#Chat#10: ~Bob [bob@example.net] 
 --------------------------------------------------------------------------------
 Msg#10: Me (Contact#Contact#Self): Happy birthday, Bob!  √
 Msg#11:  (Contact#Contact#10): Happy birthday to me, Alice! [FRESH]

test-data/golden/test_verified_oneonone_chat_enable_disable

@@ -1,11 +1,11 @@
-Single#Chat#10: Bob [bob@example.net] 🛡️
+Single#Chat#10: ~Bob [bob@example.net] 🛡️
 --------------------------------------------------------------------------------
 Msg#10: info (Contact#Contact#Info): Messages are guaranteed to be end-to-end encrypted from now on. [NOTICED][INFO 🛡️]
-Msg#11: info (Contact#Contact#Info): Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
+Msg#11: info (Contact#Contact#Info): ~Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
 Msg#12:  (Contact#Contact#10): Message from Thunderbird [FRESH]
 Msg#13: info (Contact#Contact#Info): Messages are guaranteed to be end-to-end encrypted from now on. [NOTICED][INFO 🛡️]
 Msg#14🔒:  (Contact#Contact#10): Hello from DC [FRESH]
-Msg#15: info (Contact#Contact#Info): Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
+Msg#15: info (Contact#Contact#Info): ~Bob sent a message from another device. [NOTICED][INFO 🛡️❌]
 Msg#16:  (Contact#Contact#10): Message from Thunderbird [FRESH]
 Msg#17: info (Contact#Contact#Info): Messages are guaranteed to be end-to-end encrypted from now on. [NOTICED][INFO 🛡️]
 Msg#18🔒:  (Contact#Contact#10): Hello from DC [FRESH]

test-data/message/big_forwarded_with_big_attachment.eml

@@ -0,0 +1,305 @@
+From: Alice <alice@example.org>
+To: Bob <bob@example.net>
+Date: Fri, 2 Jun 2023 13:29:17 +0000
+Message-ID: <foobar1@localhost>
+Content-Type: multipart/mixed; boundary="zRs3OquGy6eU58KF"
+
+
+--zRs3OquGy6eU58KF
+Content-Type: text/plain; charset=us-ascii
+Content-Disposition: inline
+
+Hello!
+
+--zRs3OquGy6eU58KF
+Content-Type: message/rfc822
+Content-Disposition: inline
+
+From: AAA <aaa@example.org>
+To: Alice <alice@example.org>
+Subject: Some subject
+Date: Fri, 2 Jun 2023 12:29:17 +0000
+Message-ID: <foobar@localhost>
+In-Reply-To: <barbaz@localhost>
+Content-Type: multipart/mixed;
+	boundary="_innerboundary_"
+MIME-Version: 1.0
+
+--_innerboundary_
+Content-Type: text/plain; charset="us-ascii"
+Content-Transfer-Encoding: quoted-printable
+
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+
+--=20
+
+Kind regards,
+
+Bob
+
+--_innerboundary_
+Content-Type: text/plain; name="deltachat-log.txt"
+Content-Description: deltachat-log.txt
+Content-Disposition: attachment; filename="deltachat-log.txt";
+	size=55254; creation-date="Fri, 02 Jun 2023 11:33:49 GMT";
+	modification-date="Fri, 02 Jun 2023 12:29:17 GMT"
+
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+this text with 42 chars is just repeated.
+
+--_innerboundary_--
+
+--zRs3OquGy6eU58KF--