feat: compress backups with gzip

chore(release): prepare for 1.148.4
fix: do not wait for connections in maybe_add_gossip_peers()
2026-10-10 15:10:33 +03:00 · 2024-10-25 01:15:55 +00:00 · 2024-10-24 20:25:58 +00:00 · 2024-10-24 19:59:00 +00:00 · 2024-10-24 16:43:10 +00:00 · 2024-10-24 16:00:27 +00:00
32 changed files with 526 additions and 154 deletions
@@ -1,5 +1,131 @@
 # Changelog

+## [1.148.4] - 2024-10-24
+
+### Features / Changes
+
+- Jsonrpc: add `private_tag` to `Account::Configured` Object ([#6107](https://github.com/deltachat/deltachat-core-rust/pull/6107)).
+
+### Fixes
+
+- Normalize proxy URLs before saving into proxy_url.
+- Do not wait for connections in maybe_add_gossip_peers().
+
+## [1.148.3] - 2024-10-24
+
+### Fixes
+
+- Fix reception of realtime advertisements.
+
+### Features / Changes
+
+- Allow sending realtime messages up to 128 KB in size.
+
+### API-Changes
+
+- deltachat-rpc-client: Add EventType.WEBXDC_REALTIME_ADVERTISEMENT_RECEIVED.
+
+### Documentation
+
+- Fix DC_QR_PROXY docs ([#6099](https://github.com/deltachat/deltachat-core-rust/pull/6099)).
+
+### Refactor
+
+- Generate topic inside create_iroh_header().
+
+### Tests
+
+- Test that realtime advertisements work after chatting.
+
+## [1.148.2] - 2024-10-23
+
+### Fixes
+
+- Never initialize Iroh if realtime is disabled.
+
+### Features / Changes
+
+- Add more logging for iroh initialization and peer addition.
+
+### Build system
+
+- `nix flake update nixpkgs`.
+- `nix flake update fenix`.
+
+## [1.148.1] - 2024-10-23
+
+### Build system
+
+- Revert "build: nix flake update"
+
+This reverts commit 6f22ce2722b51773d7fbb0d89e4764f963cafd91..
+
+## [1.148.0] - 2024-10-22
+
+### API-Changes
+
+- Create QR codes from any data ([#6090](https://github.com/deltachat/deltachat-core-rust/pull/6090)).
+- Add delta chat logo to QR codes ([#6093](https://github.com/deltachat/deltachat-core-rust/pull/6093)).
+- Add realtime advertisement received event ([#6043](https://github.com/deltachat/deltachat-core-rust/pull/6043)).
+- Notify adding reactions ([#6072](https://github.com/deltachat/deltachat-core-rust/pull/6072))
+- Internal profile names ([#6088](https://github.com/deltachat/deltachat-core-rust/pull/6088)).
+
+### Features / Changes
+
+- IMAP COMPRESS support.
+- Sort received outgoing message down if it's fresher than all non fresh messages.
+- Prioritize cached results if DNS resolver returns many results.
+- Add in-memory cache for DNS.
+- deltachat-repl: Built-in QR code printer.
+- Log the logic for (not) doing AEAP.
+- Log when late Autocrypt header is ignored.
+- Add more context to `send_msg` errors.
+
+### Fixes
+
+- Replace old draft with a new one atomically.
+- ChatId::maybe_delete_draft: Don't delete message if it's not a draft anymore ([#6053](https://github.com/deltachat/deltachat-core-rust/pull/6053)).
+- Call update_connection_history for proxified connections.
+- sql: Set PRAGMA query_only to avoid writing on read-only connections.
+- sql: Run `PRAGMA incremental_vacuum` on a write connection.
+- Increase MAX_SECONDS_TO_LEND_FROM_FUTURE to 30.
+
+### Build system
+
+- Nix flake update.
+- Resolve warning about default-features, and make it possible to disable vendoring ([#6079](https://github.com/deltachat/deltachat-core-rust/pull/6079)).
+- Silence a rust-analyzer false-positive ([#6077](https://github.com/deltachat/deltachat-core-rust/pull/6077)).
+
+### CI
+
+- Update Rust to 1.82.0.
+
+### Documentation
+
+- Set_protection_for_timestamp_sort does not send messages.
+- Document MimeFactory.req_mdn.
+- Fix `too_long_first_doc_paragraph` clippy lint.
+
+### Refactor
+
+- Update_msg_state: Don't avoid downgrading OutMdnRcvd to OutDelivered.
+- Fix elided_named_lifetimes warning.
+- set_protection_for_timestamp_sort: Do not log bubbled up errors.
+- Fix clippy::needless_lifetimes warnings.
+- Use `HeaderDef` constant for Chat-Disposition-Notification-To.
+- Resultify get_self_fingerprint().
+- sql: Move write mutex into connection pool.
+
+### Tests
+
+- test_qr_setup_contact_svg: Stop testing for no display name.
+- Always gossip if gossip_period is set to 0.
+- test_aeap_flow_verified: Wait for "member added" before sending messages ([#6057](https://github.com/deltachat/deltachat-core-rust/pull/6057)).
+- Make test_verified_group_member_added_recovery more reliable.
+- test_aeap_flow_verified: Do not start ac1new.
+- Fix `test_securejoin_after_contact_resetup` flakiness.
+- Message from old setup preserves contact verification, but breaks 1:1 protection.
+
 ## [1.147.1] - 2024-10-13

 ### Build system
@@ -5008,3 +5134,8 @@ https://github.com/deltachat/deltachat-core-rust/pulls?q=is%3Apr+is%3Aclosed
 [1.146.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.145.0..v1.146.0
 [1.147.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.146.0..v1.147.0
 [1.147.1]: https://github.com/deltachat/deltachat-core-rust/compare/v1.147.0..v1.147.1
+[1.148.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.147.1..v1.148.0
+[1.148.1]: https://github.com/deltachat/deltachat-core-rust/compare/v1.148.0..v1.148.1
+[1.148.2]: https://github.com/deltachat/deltachat-core-rust/compare/v1.148.1..v1.148.2
+[1.148.3]: https://github.com/deltachat/deltachat-core-rust/compare/v1.148.2..v1.148.3
+[1.148.4]: https://github.com/deltachat/deltachat-core-rust/compare/v1.148.3..v1.148.4
@@ -1293,11 +1293,12 @@ dependencies = [

 [[package]]
 name = "deltachat"
-version = "1.147.1"
+version = "1.148.4"
 dependencies = [
 "anyhow",
 "async-broadcast",
 "async-channel 2.3.1",
+ "async-compression",
 "async-imap",
 "async-native-tls",
 "async-smtp",
@@ -1393,7 +1394,7 @@ dependencies = [

 [[package]]
 name = "deltachat-jsonrpc"
-version = "1.147.1"
+version = "1.148.4"
 dependencies = [
 "anyhow",
 "async-channel 2.3.1",
@@ -1418,7 +1419,7 @@ dependencies = [

 [[package]]
 name = "deltachat-repl"
-version = "1.147.1"
+version = "1.148.4"
 dependencies = [
 "anyhow",
 "deltachat",
@@ -1434,7 +1435,7 @@ dependencies = [

 [[package]]
 name = "deltachat-rpc-server"
-version = "1.147.1"
+version = "1.148.4"
 dependencies = [
 "anyhow",
 "deltachat",
@@ -1463,7 +1464,7 @@ dependencies = [

 [[package]]
 name = "deltachat_ffi"
-version = "1.147.1"
+version = "1.148.4"
 dependencies = [
 "anyhow",
 "deltachat",
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat"
-version = "1.147.1"
+version = "1.148.4"
 edition = "2021"
 license = "MPL-2.0"
 rust-version = "1.77"
@@ -42,6 +42,7 @@ anyhow = { workspace = true }
 async-broadcast = "0.7.1"
 async-channel = { workspace = true }
 async-imap = { version = "0.10.2", default-features = false, features = ["runtime-tokio", "compress"] }
+async-compression = { version = "0.4.15", default-features = false, features = ["tokio", "gzip"] }
 async-native-tls = { version = "0.5", default-features = false, features = ["runtime-tokio"] }
 async-smtp = { version = "0.9", default-features = false, features = ["runtime-tokio"] }
 async_zip = { version = "0.0.17", default-features = false, features = ["deflate", "tokio-fs"] }
@@ -0,0 +1,12 @@
+<path
+   style="fill:#ffffff;fill-opacity:1;stroke:none"
+   d="m 24.015419,1.2870249 c -12.549421,0 -22.7283936,10.1789711 -22.7283936,22.7283931 0,12.549422 10.1789726,22.728395 22.7283936,22.728395 14.337742,-0.342877 9.614352,-4.702705 23.697556,0.969161 -7.545453,-13.001555 -1.082973,-13.32964 -0.969161,-23.697556 0,-12.549422 -10.178973,-22.7283931 -22.728395,-22.7283931 z" />
+<path
+   style="fill:#000000;fill-opacity:1;stroke:none"
+   d="M 23.982249,5.3106163 C 13.645822,5.4364005 5.2618355,13.92999 5.2618355,24.275753 c 0,10.345764 8.3839865,18.635301 18.7204135,18.509516 9.827724,-0.03951 7.516769,-5.489695 18.380082,-0.443187 -5.950849,-9.296115 0.201753,-10.533667 0.340336,-18.521947 0,-10.345766 -8.383989,-18.6353031 -18.720418,-18.5095187 z" />
+<g
+    style="fill:#ffffff"
+    transform="scale(1.1342891,0.88160947)">
+  <path
+     d="m 21.360141,23.513382 q -1.218487,-1.364705 -3.387392,-3.265543 -2.388233,-2.095797 -3.216804,-3.289913 -0.828571,-1.218486 -0.828571,-2.6563 0,-2.144536 1.998318,-3.363022 1.998317,-1.2428565 5.215121,-1.2428565 3.216804,0 5.605037,1.0966375 2.412603,1.096638 2.412603,3.021846 0,0.92605 -0.584873,1.535293 -0.584874,0.609243 -1.364705,0.609243 -1.121008,0 -2.631931,-1.681511 -1.535292,-1.705881 -2.60756,-2.388233 -1.047898,-0.706722 -2.461343,-0.706722 -1.803359,0 -2.973106,0.804201 -1.145377,0.804201 -1.145377,2.047057 0,1.169747 0.950419,2.193275 0.950419,1.023529 4.898315,3.728568 4.215963,2.899998 5.946213,4.532769 1.75462,1.632772 2.851258,3.972265 1.096638,2.339494 1.096638,4.947055 0,4.581508 -3.241174,8.090749 -3.216804,3.484871 -7.530245,3.484871 -3.923526,0 -6.628566,-2.802519 -2.705039,-2.802518 -2.705039,-7.481506 0,-4.508399 2.973106,-7.530245 2.997477,-3.021846 7.359658,-3.655459 z m 1.072268,1.121008 q -6.994112,1.145377 -6.994112,9.601672 0,4.36218 1.730251,6.774783 1.75462,2.412603 4.069744,2.412603 2.412603,0 3.972265,-2.315124 1.559663,-2.339493 1.559663,-6.311759 0,-5.751255 -4.337811,-10.162175 z" />
+</g>
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat_ffi"
-version = "1.147.1"
+version = "1.148.4"
 description = "Deltachat FFI"
 edition = "2018"
 readme = "README.md"
@@ -2533,8 +2533,8 @@ void            dc_stop_ongoing_process      (dc_context_t* context);
 *   ask the user if they want to use the given service for video chats;
 *   if so, call dc_set_config_from_qr().
 *
- * - DC_QR_SOCKS5_PROXY with dc_lot_t::text1=host, dc_lot_t::text2=port:
- *   ask the user if they want to use the given proxy and overwrite the previous one, if any.
+ * - DC_QR_PROXY with dc_lot_t::text1=address:
+ *   ask the user if they want to use the given proxy.
 *   if so, call dc_set_config_from_qr() and restart I/O.
 *
 * - DC_QR_ADDR with dc_lot_t::id=Contact ID:
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-jsonrpc"
-version = "1.147.1"
+version = "1.148.4"
 description = "DeltaChat JSON-RPC API"
 edition = "2021"
 default-run = "deltachat-jsonrpc-server"
@@ -17,6 +17,9 @@ pub enum Account {
        // size: u32,
        profile_image: Option<String>, // TODO: This needs to be converted to work with blob http server.
        color: String,
+        /// Optional tag as "Work", "Family".
+        /// Meant to help profile owner to differ between profiles with similar names.
+        private_tag: Option<String>,
    },
    #[serde(rename_all = "camelCase")]
    Unconfigured { id: u32 },
@@ -31,12 +34,14 @@ impl Account {
            let color = color_int_to_hex_string(
                Contact::get_by_id(ctx, ContactId::SELF).await?.get_color(),
            );
+            let private_tag = ctx.get_config(Config::PrivateTag).await?;
            Ok(Account::Configured {
                id,
                display_name,
                addr,
                profile_image,
                color,
+                private_tag,
            })
        } else {
            Ok(Account::Unconfigured { id })
@@ -58,5 +58,5 @@
  },
  "type": "module",
  "types": "dist/deltachat.d.ts",
-  "version": "1.147.1"
+  "version": "1.148.4"
 }
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-repl"
-version = "1.147.1"
+version = "1.148.4"
 license = "MPL-2.0"
 edition = "2021"
 repository = "https://github.com/deltachat/deltachat-core-rust"
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

 [project]
 name = "deltachat-rpc-client"
-version = "1.147.1"
+version = "1.148.4"
 description = "Python client for Delta Chat core JSON-RPC interface"
 classifiers = [
    "Development Status :: 5 - Production/Stable",
@@ -63,6 +63,7 @@ class EventType(str, Enum):
    CHATLIST_ITEM_CHANGED = "ChatlistItemChanged"
    CONFIG_SYNCED = "ConfigSynced"
    WEBXDC_REALTIME_DATA = "WebxdcRealtimeData"
+    WEBXDC_REALTIME_ADVERTISEMENT_RECEIVED = "WebxdcRealtimeAdvertisementReceived"


 class ChatId(IntEnum):
@@ -7,6 +7,7 @@ If you want to debug iroh at rust-trace/log level set
    RUST_LOG=iroh_net=trace,iroh_gossip=trace
 """

+import os
 import sys
 import threading
 import time
@@ -107,13 +108,15 @@ def test_realtime_sequentially(acfactory, path_to_webxdc):
    assert snapshot.text == "ping2"

    log("sending realtime data ac1 -> ac2")
-    ac1_webxdc_msg.send_webxdc_realtime_data(b"foo")
+    # Test that 128 KB of data can be sent in a single message.
+    data = os.urandom(128000)
+    ac1_webxdc_msg.send_webxdc_realtime_data(data)

    log("ac2: waiting for realtime data")
    while 1:
        event = ac2.wait_for_event()
        if event.kind == EventType.WEBXDC_REALTIME_DATA:
-            assert event.data == list(b"foo")
+            assert event.data == list(data)
            break


@@ -208,3 +211,28 @@ def test_no_reordering(acfactory, path_to_webxdc):
                if event.data[0] == i:
                    break
                pytest.fail("Reordering detected")
+
+
+def test_advertisement_after_chatting(acfactory, path_to_webxdc):
+    """Test that realtime advertisement is assigned to the correct message after chatting."""
+    ac1, ac2 = acfactory.get_online_accounts(2)
+    ac1.set_config("webxdc_realtime_enabled", "1")
+    ac2.set_config("webxdc_realtime_enabled", "1")
+
+    ac1_ac2_chat = ac1.create_chat(ac2)
+    ac1_webxdc_msg = ac1_ac2_chat.send_message(text="WebXDC", file=path_to_webxdc)
+    ac2_webxdc_msg = ac2.wait_for_incoming_msg()
+    assert ac2_webxdc_msg.get_snapshot().text == "WebXDC"
+
+    ac1_ac2_chat.send_text("Hello!")
+    ac2_hello_msg = ac2.wait_for_incoming_msg()
+    ac2_hello_msg_snapshot = ac2_hello_msg.get_snapshot()
+    assert ac2_hello_msg_snapshot.text == "Hello!"
+    ac2_hello_msg_snapshot.chat.accept()
+
+    ac2_webxdc_msg.send_webxdc_realtime_advertisement()
+    while 1:
+        event = ac1.wait_for_event()
+        if event.kind == EventType.WEBXDC_REALTIME_ADVERTISEMENT_RECEIVED:
+            assert event.msg_id == ac1_webxdc_msg.id
+            break
@@ -61,7 +61,7 @@ def test_qr_securejoin(acfactory, protect, tmp_path):
    # Setup second device for Alice
    # to test observing securejoin protocol.
    alice.export_backup(tmp_path)
-    files = list(tmp_path.glob("*.tar"))
+    files = list(tmp_path.glob("*.tar.gz"))
    alice2 = acfactory.get_unconfigured_account()
    alice2.import_backup(files[0])

@@ -379,7 +379,7 @@ def test_import_export_backup(acfactory, tmp_path) -> None:
    alice = acfactory.new_configured_account()
    alice.export_backup(tmp_path)

-    files = list(tmp_path.glob("*.tar"))
+    files = list(tmp_path.glob("*.tar.gz"))
    alice2 = acfactory.get_unconfigured_account()
    alice2.import_backup(files[0])

@@ -630,7 +630,7 @@ def test_markseen_contact_request(acfactory, tmp_path):

    # Bob sets up a second device.
    bob.export_backup(tmp_path)
-    files = list(tmp_path.glob("*.tar"))
+    files = list(tmp_path.glob("*.tar.gz"))
    bob2 = acfactory.get_unconfigured_account()
    bob2.import_backup(files[0])
    bob2.start_io()
@@ -1,6 +1,6 @@
 [package]
 name = "deltachat-rpc-server"
-version = "1.147.1"
+version = "1.148.4"
 description = "DeltaChat JSON-RPC server"
 edition = "2021"
 readme = "README.md"
@@ -15,5 +15,5 @@
  },
  "type": "module",
  "types": "index.d.ts",
-  "version": "1.147.1"
+  "version": "1.148.4"
 }
@@ -3,15 +3,15 @@
    "android": {
      "inputs": {
        "devshell": "devshell",
-        "flake-utils": "flake-utils",
+        "flake-utils": "flake-utils_2",
        "nixpkgs": "nixpkgs"
      },
      "locked": {
-        "lastModified": 1729369131,
-        "narHash": "sha256-PtfScp+nQd1PsT5rf0Qgjdbsh4Iag6R1ivYMWLizyIc=",
+        "lastModified": 1712088936,
+        "narHash": "sha256-mVjeSWQiR/t4UZ9fUawY9OEPAhY1R3meYG+0oh8DUBs=",
        "owner": "tadfisher",
        "repo": "android-nixpkgs",
-        "rev": "82bffbf3f06bdccf44fc62a9bd4f152ac80a55b0",
+        "rev": "2d8181caef279f19c4a33dc694723f89ffc195d4",
        "type": "github"
      },
      "original": {
@@ -22,17 +22,18 @@
    },
    "devshell": {
      "inputs": {
+        "flake-utils": "flake-utils",
        "nixpkgs": [
          "android",
          "nixpkgs"
        ]
      },
      "locked": {
-        "lastModified": 1728330715,
-        "narHash": "sha256-xRJ2nPOXb//u1jaBnDP56M7v5ldavjbtR6lfGqSvcKg=",
+        "lastModified": 1711099426,
+        "narHash": "sha256-HzpgM/wc3aqpnHJJ2oDqPBkNsqWbW0WfWUO8lKu8nGk=",
        "owner": "numtide",
        "repo": "devshell",
-        "rev": "dd6b80932022cea34a019e2bb32f6fa9e494dfef",
+        "rev": "2d45b54ca4a183f2fdcf4b19c895b64fbf620ee8",
        "type": "github"
      },
      "original": {
@@ -47,11 +48,11 @@
        "rust-analyzer-src": "rust-analyzer-src"
      },
      "locked": {
-        "lastModified": 1729375822,
-        "narHash": "sha256-bRo4xVwUhvJ4Gz+OhWMREFMdBOYSw4Yi1Apj01ebbug=",
+        "lastModified": 1729578683,
+        "narHash": "sha256-h0Wmvrkadbyi3IJXFLPi+QyYjCAKDr2xQ6dLxlQ8cXY=",
        "owner": "nix-community",
        "repo": "fenix",
-        "rev": "2853e7d9b5c52a148a9fb824bfe4f9f433f557ab",
+        "rev": "d66cda53e8193a878742dcadb5bb75f4df7c3c0a",
        "type": "github"
      },
      "original": {
@@ -65,11 +66,11 @@
        "systems": "systems"
      },
      "locked": {
-        "lastModified": 1726560853,
-        "narHash": "sha256-X6rJYSESBVr3hBoH0WbKE5KvhPU5bloyZ2L4K60/fPQ=",
+        "lastModified": 1701680307,
+        "narHash": "sha256-kAuep2h5ajznlPMD9rnQyffWG8EM/C73lejGofXvdM8=",
        "owner": "numtide",
        "repo": "flake-utils",
-        "rev": "c1dfcf08411b08f6b8615f7d8971a2bfa81d5e8a",
+        "rev": "4022d587cbbfd70fe950c1e2083a02621806a725",
        "type": "github"
      },
      "original": {
@@ -83,11 +84,29 @@
        "systems": "systems_2"
      },
      "locked": {
-        "lastModified": 1726560853,
-        "narHash": "sha256-X6rJYSESBVr3hBoH0WbKE5KvhPU5bloyZ2L4K60/fPQ=",
+        "lastModified": 1710146030,
+        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
        "owner": "numtide",
        "repo": "flake-utils",
-        "rev": "c1dfcf08411b08f6b8615f7d8971a2bfa81d5e8a",
+        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
+        "type": "github"
+      },
+      "original": {
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "type": "github"
+      }
+    },
+    "flake-utils_3": {
+      "inputs": {
+        "systems": "systems_3"
+      },
+      "locked": {
+        "lastModified": 1710146030,
+        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
        "type": "github"
      },
      "original": {
@@ -101,11 +120,11 @@
        "nixpkgs": "nixpkgs_3"
      },
      "locked": {
-        "lastModified": 1721727458,
-        "narHash": "sha256-r/xppY958gmZ4oTfLiHN0ZGuQ+RSTijDblVgVLFi1mw=",
+        "lastModified": 1713520724,
+        "narHash": "sha256-CO8MmVDmqZX2FovL75pu5BvwhW+Vugc7Q6ze7Hj8heI=",
        "owner": "nix-community",
        "repo": "naersk",
-        "rev": "3fb418eaf352498f6b6c30592e3beb63df42ef11",
+        "rev": "c5037590290c6c7dae2e42e7da1e247e54ed2d49",
        "type": "github"
      },
      "original": {
@@ -131,11 +150,11 @@
    },
    "nixpkgs": {
      "locked": {
-        "lastModified": 1729256560,
-        "narHash": "sha256-/uilDXvCIEs3C9l73JTACm4quuHUsIHcns1c+cHUJwA=",
+        "lastModified": 1711703276,
+        "narHash": "sha256-iMUFArF0WCatKK6RzfUJknjem0H9m4KgorO/p3Dopkk=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "4c2fcb090b1f3e5b47eaa7bd33913b574a11e0a0",
+        "rev": "d8fe5e6c92d0d190646fb9f1056741a229980089",
        "type": "github"
      },
      "original": {
@@ -147,11 +166,11 @@
    },
    "nixpkgs_2": {
      "locked": {
-        "lastModified": 1729070438,
-        "narHash": "sha256-KOTTUfPkugH52avUvXGxvWy8ibKKj4genodIYUED+Kc=",
+        "lastModified": 1729256560,
+        "narHash": "sha256-/uilDXvCIEs3C9l73JTACm4quuHUsIHcns1c+cHUJwA=",
        "owner": "nixos",
        "repo": "nixpkgs",
-        "rev": "5785b6bb5eaae44e627d541023034e1601455827",
+        "rev": "4c2fcb090b1f3e5b47eaa7bd33913b574a11e0a0",
        "type": "github"
      },
      "original": {
@@ -163,12 +182,11 @@
    },
    "nixpkgs_3": {
      "locked": {
-        "lastModified": 1729265718,
-        "narHash": "sha256-4HQI+6LsO3kpWTYuVGIzhJs1cetFcwT7quWCk/6rqeo=",
-        "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "ccc0c2126893dd20963580b6478d1a10a4512185",
-        "type": "github"
+        "lastModified": 1711668574,
+        "narHash": "sha256-u1dfs0ASQIEr1icTVrsKwg2xToIpn7ZXxW3RHfHxshg=",
+        "path": "/nix/store/9fpv0kjq9a80isa1wkkvrdqsh9dpcn05-source",
+        "rev": "219951b495fc2eac67b1456824cc1ec1fd2ee659",
+        "type": "path"
      },
      "original": {
        "id": "nixpkgs",
@@ -177,11 +195,11 @@
    },
    "nixpkgs_4": {
      "locked": {
-        "lastModified": 1729256560,
-        "narHash": "sha256-/uilDXvCIEs3C9l73JTACm4quuHUsIHcns1c+cHUJwA=",
+        "lastModified": 1729413321,
+        "narHash": "sha256-I4tuhRpZFa6Fu6dcH9Dlo5LlH17peT79vx1y1SpeKt0=",
        "owner": "nixos",
        "repo": "nixpkgs",
-        "rev": "4c2fcb090b1f3e5b47eaa7bd33913b574a11e0a0",
+        "rev": "1997e4aa514312c1af7e2bda7fad1644e778ff26",
        "type": "github"
      },
      "original": {
@@ -195,7 +213,7 @@
      "inputs": {
        "android": "android",
        "fenix": "fenix",
-        "flake-utils": "flake-utils_2",
+        "flake-utils": "flake-utils_3",
        "naersk": "naersk",
        "nix-filter": "nix-filter",
        "nixpkgs": "nixpkgs_4"
@@ -204,11 +222,11 @@
    "rust-analyzer-src": {
      "flake": false,
      "locked": {
-        "lastModified": 1729255720,
-        "narHash": "sha256-yODOuZxBkS0UfqMa6nmbqNbVfIbsu0tYLbV5vZzmsqI=",
+        "lastModified": 1729533545,
+        "narHash": "sha256-A/AuEWcGwwjpfBCZqWDNNg5GwYrJduzLvlMe+A7xG5U=",
        "owner": "rust-lang",
        "repo": "rust-analyzer",
-        "rev": "72b214fbfbe6f7b95a7877b962783bd42062cc0a",
+        "rev": "de2ff17bc513807412d7bbaba1d995a774938583",
        "type": "github"
      },
      "original": {
@@ -247,6 +265,21 @@
        "repo": "default",
        "type": "github"
      }
+    },
+    "systems_3": {
+      "locked": {
+        "lastModified": 1681028828,
+        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "owner": "nix-systems",
+        "repo": "default",
+        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-systems",
+        "repo": "default",
+        "type": "github"
+      }
    }
  },
  "root": "root",
@@ -55,5 +55,5 @@
    "test:mocha": "mocha node/test/test.mjs --growl --reporter=spec --bail --exit"
  },
  "types": "node/dist/index.d.ts",
-  "version": "1.147.1"
+  "version": "1.148.4"
 }
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

 [project]
 name = "deltachat"
-version = "1.147.1"
+version = "1.148.4"
 description = "Python bindings for the Delta Chat Core library using CFFI against the Rust-implemented libdeltachat"
 readme = "README.rst"
 requires-python = ">=3.7"
@@ -1 +1 @@
-2024-10-13
+2024-10-24
@@ -4,12 +4,16 @@

 use deltachat_derive::{FromSql, ToSql};
 use once_cell::sync::Lazy;
+use percent_encoding::{AsciiSet, NON_ALPHANUMERIC};
 use serde::{Deserialize, Serialize};

 use crate::chat::ChatId;

 pub static DC_VERSION_STR: Lazy<String> = Lazy::new(|| env!("CARGO_PKG_VERSION").to_string());

+/// Set of characters to percent-encode in email addresses and names.
+pub(crate) const NON_ALPHANUMERIC_WITHOUT_DOT: &AsciiSet = &NON_ALPHANUMERIC.remove(b'.');
+
 #[derive(
    Debug,
    Default,
@@ -11,7 +11,7 @@ use futures_lite::FutureExt;
 use pin_project::pin_project;

 use tokio::fs::{self, File};
-use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
+use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt, ReadBuf};
 use tokio_tar::Archive;

 use crate::blob::BlobDirContents;
@@ -123,7 +123,7 @@ pub async fn has_backup(_context: &Context, dir_name: &Path) -> Result<String> {
        let name = dirent.file_name();
        let name: String = name.to_string_lossy().into();
        if name.starts_with("delta-chat")
-            && name.ends_with(".tar")
+            && (name.ends_with(".tar") || name.ends_with(".tar.gz"))
            && (newest_backup_name.is_empty() || name > newest_backup_name)
        {
            // We just use string comparison to determine which backup is newer.
@@ -269,30 +269,24 @@ async fn import_backup(
        context.get_dbfile().display()
    );

-    import_backup_stream(context, backup_file, file_size, passphrase).await?;
+    let backup_file = ProgressReader::new(backup_file, context.clone(), file_size);
+    if backup_to_import.extension() == Some(OsStr::new("gz")) {
+        let backup_file = tokio::io::BufReader::new(backup_file);
+        let backup_file = async_compression::tokio::bufread::GzipDecoder::new(backup_file);
+        import_backup_stream(context, backup_file, passphrase).await?;
+    } else {
+        import_backup_stream(context, backup_file, passphrase).await?;
+    }
    Ok(())
 }

 /// Imports backup by reading a tar file from a stream.
-///
-/// `file_size` is used to calculate the progress
-/// and emit progress events.
-/// Ideally it is the sum of the entry
-/// sizes without the header overhead,
-/// but can be estimated as tar file size
-/// in which case the progress is underestimated
-/// and may not reach 99.9% by the end of import.
-/// Underestimating is better than
-/// overestimating because the progress
-/// jumps to 100% instead of getting stuck at 99.9%
-/// for some time.
 pub(crate) async fn import_backup_stream<R: tokio::io::AsyncRead + Unpin>(
    context: &Context,
    backup_file: R,
-    file_size: u64,
    passphrase: String,
 ) -> Result<()> {
-    import_backup_stream_inner(context, backup_file, file_size, passphrase)
+    import_backup_stream_inner(context, backup_file, passphrase)
        .await
        .0
 }
@@ -319,6 +313,19 @@ struct ProgressReader<R> {
 }

 impl<R> ProgressReader<R> {
+    /// Creates a new `ProgressReader`.
+    ///
+    /// `file_size` is used to calculate the progress
+    /// and emit progress events.
+    /// Ideally it is the sum of the entry
+    /// sizes without the header overhead,
+    /// but can be estimated as tar file size
+    /// in which case the progress is underestimated
+    /// and may not reach 99.9% by the end of import.
+    /// Underestimating is better than
+    /// overestimating because the progress
+    /// jumps to 100% instead of getting stuck at 99.9%
+    /// for some time.
    fn new(r: R, context: Context, file_size: u64) -> Self {
        Self {
            inner: r,
@@ -358,10 +365,8 @@ where
 async fn import_backup_stream_inner<R: tokio::io::AsyncRead + Unpin>(
    context: &Context,
    backup_file: R,
-    file_size: u64,
    passphrase: String,
 ) -> (Result<()>,) {
-    let backup_file = ProgressReader::new(backup_file, context.clone(), file_size);
    let mut archive = Archive::new(backup_file);

    let mut entries = match archive.entries() {
@@ -461,10 +466,10 @@ fn get_next_backup_path(
        tempdbfile.push(format!("{stem}-{i:02}-{addr}.db"));

        let mut tempfile = folder.clone();
-        tempfile.push(format!("{stem}-{i:02}-{addr}.tar.part"));
+        tempfile.push(format!("{stem}-{i:02}-{addr}.tar.gz.part"));

        let mut destfile = folder.clone();
-        destfile.push(format!("{stem}-{i:02}-{addr}.tar"));
+        destfile.push(format!("{stem}-{i:02}-{addr}.tar.gz"));

        if !tempdbfile.exists() && !tempfile.exists() && !destfile.exists() {
            return Ok((tempdbfile, tempfile, destfile));
@@ -504,9 +509,13 @@ async fn export_backup(context: &Context, dir: &Path, passphrase: String) -> Res
        file_size += blob.to_abs_path().metadata()?.len()
    }

-    export_backup_stream(context, &temp_db_path, blobdir, file, file_size)
-        .await
-        .context("Exporting backup to file failed")?;
+    let gzip_encoder = async_compression::tokio::write::GzipEncoder::new(file);
+    let mut gzip_encoder =
+        export_backup_stream(context, &temp_db_path, blobdir, gzip_encoder, file_size)
+            .await
+            .context("Exporting backup to file failed")?;
+    gzip_encoder.shutdown().await?;
+
    fs::rename(temp_path, &dest_path).await?;
    context.emit_event(EventType::ImexFileWritten(dest_path));
    Ok(())
@@ -543,6 +552,10 @@ impl<W> ProgressWriter<W> {
            context,
        }
    }
+
+    fn into_inner(self) -> W {
+        self.inner
+    }
 }

 impl<W> AsyncWrite for ProgressWriter<W>
@@ -590,12 +603,12 @@ pub(crate) async fn export_backup_stream<'a, W>(
    blobdir: BlobDirContents<'a>,
    writer: W,
    file_size: u64,
-) -> Result<()>
+) -> Result<W>
 where
    W: tokio::io::AsyncWrite + tokio::io::AsyncWriteExt + Unpin + Send + 'static,
 {
-    let writer = ProgressWriter::new(writer, context.clone(), file_size);
-    let mut builder = tokio_tar::Builder::new(writer);
+    let progress_writer = ProgressWriter::new(writer, context.clone(), file_size);
+    let mut builder = tokio_tar::Builder::new(progress_writer);

    builder
        .append_path_with_name(temp_db_path, DBFILE_BACKUP_NAME)
@@ -607,8 +620,9 @@ where
        builder.append_file(path_in_archive, &mut file).await?;
    }

-    builder.finish().await?;
-    Ok(())
+    // Convert tar builder back into the underlying stream.
+    let progress_writer = builder.into_inner().await?;
+    Ok(progress_writer.into_inner())
 }

 /// Imports secret key from a file.
@@ -36,12 +36,13 @@ use futures_lite::FutureExt;
 use iroh_net::relay::RelayMode;
 use iroh_net::Endpoint;
 use tokio::fs;
+use tokio::io::AsyncWriteExt;
 use tokio::task::JoinHandle;
 use tokio_util::sync::CancellationToken;

 use crate::chat::add_device_msg;
 use crate::context::Context;
-use crate::imex::BlobDirContents;
+use crate::imex::{BlobDirContents, ProgressReader};
 use crate::message::{Message, Viewtype};
 use crate::qr::Qr;
 use crate::stock_str::backup_transfer_msg_body;
@@ -190,9 +191,11 @@ impl BackupProvider {

        send_stream.write_all(&file_size.to_be_bytes()).await?;

-        export_backup_stream(&context, &dbfile, blobdir, send_stream, file_size)
-            .await
-            .context("Failed to write backup into QUIC stream")?;
+        let mut send_stream =
+            export_backup_stream(&context, &dbfile, blobdir, send_stream, file_size)
+                .await
+                .context("Failed to write backup into QUIC stream")?;
+        send_stream.shutdown().await?;
        info!(context, "Finished writing backup into QUIC stream.");
        let mut buf = [0u8; 1];
        info!(context, "Waiting for acknowledgment.");
@@ -310,7 +313,8 @@ pub async fn get_backup2(
    let mut file_size_buf = [0u8; 8];
    recv_stream.read_exact(&mut file_size_buf).await?;
    let file_size = u64::from_be_bytes(file_size_buf);
-    import_backup_stream(context, recv_stream, file_size, passphrase)
+    let recv_stream = ProgressReader::new(recv_stream, context.clone(), file_size);
+    import_backup_stream(context, recv_stream, passphrase)
        .await
        .context("Failed to import backup from QUIC stream")?;
    info!(context, "Finished importing backup from the stream.");
@@ -20,6 +20,7 @@ use crate::e2ee::EncryptHelper;
 use crate::ephemeral::Timer as EphemeralTimer;
 use crate::headerdef::HeaderDef;
 use crate::html::new_html_mimepart;
+use crate::location;
 use crate::message::{self, Message, MsgId, Viewtype};
 use crate::mimeparser::SystemMessage;
 use crate::param::Param;
@@ -32,7 +33,6 @@ use crate::tools::{
    create_outgoing_rfc724_mid, create_smeared_timestamp, remove_subject_prefix, time,
 };
 use crate::webxdc::StatusUpdateSerial;
-use crate::{location, peer_channels};

 // attachments of 25 mb brutto should work on the majority of providers
 // (brutto examples: web.de=50, 1&1=40, t-online.de=32, gmail=25, posteo=50, yahoo=25, all-inkl=100).
@@ -1387,8 +1387,7 @@ impl MimeFactory {
            let json = msg.param.get(Param::Arg).unwrap_or_default();
            parts.push(context.build_status_update_part(json));
        } else if msg.viewtype == Viewtype::Webxdc {
-            let topic = peer_channels::create_random_topic();
-            headers.push(create_iroh_header(context, topic, msg.id).await?);
+            headers.push(create_iroh_header(context, msg.id).await?);
            if let (Some(json), _) = context
                .render_webxdc_status_update_object(
                    msg.id,
@@ -12,13 +12,14 @@ use fast_socks5::client::Socks5Stream;
 use fast_socks5::util::target_addr::ToTargetAddr;
 use fast_socks5::AuthenticationMethod;
 use fast_socks5::Socks5Command;
-use percent_encoding::{percent_encode, NON_ALPHANUMERIC};
+use percent_encoding::{percent_encode, utf8_percent_encode, NON_ALPHANUMERIC};
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::net::TcpStream;
 use tokio_io_timeout::TimeoutStream;
 use url::Url;

 use crate::config::Config;
+use crate::constants::NON_ALPHANUMERIC_WITHOUT_DOT;
 use crate::context::Context;
 use crate::net::connect_tcp;
 use crate::net::session::SessionStream;
@@ -41,6 +42,12 @@ impl PartialEq for ShadowsocksConfig {

 impl Eq for ShadowsocksConfig {}

+impl ShadowsocksConfig {
+    fn to_url(&self) -> String {
+        self.server_config.to_url()
+    }
+}
+
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct HttpConfig {
    /// HTTP proxy host.
@@ -84,6 +91,17 @@ impl HttpConfig {
        };
        Ok(http_config)
    }
+
+    fn to_url(&self, scheme: &str) -> String {
+        let host = utf8_percent_encode(&self.host, NON_ALPHANUMERIC_WITHOUT_DOT);
+        if let Some((user, password)) = &self.user_password {
+            let user = utf8_percent_encode(user, NON_ALPHANUMERIC);
+            let password = utf8_percent_encode(password, NON_ALPHANUMERIC);
+            format!("{scheme}://{user}:{password}@{host}:{}", self.port)
+        } else {
+            format!("{scheme}://{host}:{}", self.port)
+        }
+    }
 }

 #[derive(Debug, Clone, PartialEq, Eq)]
@@ -123,6 +141,17 @@ impl Socks5Config {

        Ok(socks_stream)
    }
+
+    fn to_url(&self) -> String {
+        let host = utf8_percent_encode(&self.host, NON_ALPHANUMERIC_WITHOUT_DOT);
+        if let Some((user, password)) = &self.user_password {
+            let user = utf8_percent_encode(user, NON_ALPHANUMERIC);
+            let password = utf8_percent_encode(password, NON_ALPHANUMERIC);
+            format!("socks5://{user}:{password}@{host}:{}", self.port)
+        } else {
+            format!("socks5://{host}:{}", self.port)
+        }
+    }
 }

 #[derive(Debug, Clone, PartialEq, Eq)]
@@ -217,7 +246,7 @@ where

 impl ProxyConfig {
    /// Creates a new proxy configuration by parsing given proxy URL.
-    fn from_url(url: &str) -> Result<Self> {
+    pub(crate) fn from_url(url: &str) -> Result<Self> {
        let url = Url::parse(url).context("Cannot parse proxy URL")?;
        match url.scheme() {
            "http" => {
@@ -272,6 +301,19 @@ impl ProxyConfig {
        }
    }

+    /// Serializes proxy config into an URL.
+    ///
+    /// This function can be used to normalize proxy URL
+    /// by parsing it and serializing back.
+    pub(crate) fn to_url(&self) -> String {
+        match self {
+            Self::Http(http_config) => http_config.to_url("http"),
+            Self::Https(http_config) => http_config.to_url("https"),
+            Self::Socks5(socks5_config) => socks5_config.to_url(),
+            Self::Shadowsocks(shadowsocks_config) => shadowsocks_config.to_url(),
+        }
+    }
+
    /// Migrates legacy `socks5_host`, `socks5_port`, `socks5_user` and `socks5_password`
    /// config into `proxy_url` if `proxy_url` is unset or empty.
    ///
@@ -23,7 +23,7 @@
 //!    (scoped per WebXDC app instance/message-id). The other peers can then join the gossip with `joinRealtimeChannel().setListener()`
 //!    and `joinRealtimeChannel().send()` just like the other peers.

-use anyhow::{anyhow, Context as _, Result};
+use anyhow::{anyhow, bail, Context as _, Result};
 use email::Header;
 use futures_lite::StreamExt;
 use iroh_gossip::net::{Event, Gossip, GossipEvent, JoinOptions, GOSSIP_ALPN};
@@ -143,9 +143,10 @@ impl Iroh {
                self.endpoint.add_node_addr(peer.clone())?;
            }

-            self.gossip
-                .join(topic, peers.into_iter().map(|peer| peer.node_id).collect())
-                .await?;
+            self.gossip.join_with_opts(
+                topic,
+                JoinOptions::with_bootstrap(peers.into_iter().map(|peer| peer.node_id)),
+            );
        }
        Ok(())
    }
@@ -232,6 +233,7 @@ impl ChannelState {
 impl Context {
    /// Create iroh endpoint and gossip.
    async fn init_peer_channels(&self) -> Result<Iroh> {
+        info!(self, "Initializing peer channels.");
        let secret_key = SecretKey::generate();
        let public_key = secret_key.public();

@@ -258,7 +260,14 @@ impl Context {

        // create gossip
        let my_addr = endpoint.node_addr().await?;
-        let gossip = Gossip::from_endpoint(endpoint.clone(), Default::default(), &my_addr.info);
+        let gossip_config = iroh_gossip::proto::topic::Config {
+            // Allow messages up to 128 KB in size.
+            // We set the limit to 128 KiB to account for internal overhead,
+            // but only guarantee 128 KB of payload to WebXDC developers.
+            max_message_size: 128 * 1024,
+            ..Default::default()
+        };
+        let gossip = Gossip::from_endpoint(endpoint.clone(), gossip_config, &my_addr.info);

        // spawn endpoint loop that forwards incoming connections to the gossiper
        let context = self.clone();
@@ -277,6 +286,10 @@ impl Context {

    /// Get or initialize the iroh peer channel.
    pub async fn get_or_try_init_peer_channel(&self) -> Result<&Iroh> {
+        if !self.get_config_bool(Config::WebxdcRealtimeEnabled).await? {
+            bail!("Attempt to get Iroh when realtime is disabled");
+        }
+
        let ctx = self.clone();
        self.iroh
            .get_or_try_init(|| async { ctx.init_peer_channels().await })
@@ -414,15 +427,15 @@ pub async fn leave_webxdc_realtime(ctx: &Context, msg_id: MsgId) -> Result<()> {
    Ok(())
 }

-pub(crate) fn create_random_topic() -> TopicId {
+/// Creates a new random gossip topic.
+fn create_random_topic() -> TopicId {
    TopicId::from_bytes(rand::random())
 }

-pub(crate) async fn create_iroh_header(
-    ctx: &Context,
-    topic: TopicId,
-    msg_id: MsgId,
-) -> Result<Header> {
+/// Creates `Iroh-Gossip-Header` with a new random topic
+/// and stores the topic for the message.
+pub(crate) async fn create_iroh_header(ctx: &Context, msg_id: MsgId) -> Result<Header> {
+    let topic = create_random_topic();
    insert_topic_stub(ctx, msg_id, topic).await?;
    Ok(Header::new(
        HeaderDef::IrohGossipTopic.get_headername().to_string(),
@@ -949,6 +962,10 @@ mod tests {
        // creates iroh endpoint as side effect
        leave_webxdc_realtime(alice, MsgId::new(1)).await.unwrap();

-        assert!(alice.ctx.iroh.get().is_none())
+        assert!(alice.ctx.iroh.get().is_none());
+
+        // This internal function should return error
+        // if accidentally called with the setting disabled.
+        assert!(alice.ctx.get_or_try_init_peer_channel().await.is_err());
    }
 }
@@ -20,7 +20,7 @@ use crate::events::EventType;
 use crate::key::Fingerprint;
 use crate::message::Message;
 use crate::net::http::post_empty;
-use crate::net::proxy::DEFAULT_SOCKS_PORT;
+use crate::net::proxy::{ProxyConfig, DEFAULT_SOCKS_PORT};
 use crate::peerstate::Peerstate;
 use crate::token;
 use crate::tools::validate_id;
@@ -723,6 +723,10 @@ pub async fn set_config_from_qr(context: &Context, qr: &str) -> Result<()> {
                .get_config(Config::ProxyUrl)
                .await?
                .unwrap_or_default();
+
+            // Normalize the URL.
+            let url = ProxyConfig::from_url(&url)?.to_url();
+
            let proxy_urls: Vec<&str> = std::iter::once(url.as_str())
                .chain(
                    old_proxy_url_value
@@ -1787,6 +1791,17 @@ mod tests {
            )
        );

+        // SOCKS5 config does not have port 1080 explicitly specified,
+        // but should bring `socks5://1.2.3.4:1080` to the top instead of creating another entry.
+        set_config_from_qr(&t, "socks5://1.2.3.4").await?;
+        assert_eq!(
+            t.get_config(Config::ProxyUrl).await?,
+            Some(
+                "socks5://1.2.3.4:1080\nss://YWVzLTEyOC1nY206dGVzdA@192.168.100.1:8888#Example1\nsocks5://foo:666\nsocks5://Da:x%26%25%24X@jau:1080"
+                    .to_string()
+            )
+        );
+
        Ok(())
    }

@@ -18,6 +18,7 @@ use crate::stock_str::{self, backup_transfer_qr};
 pub fn create_qr_svg(qrcode_content: &str) -> Result<String> {
    let all_size = 512.0;
    let qr_code_size = 416.0;
+    let logo_size = 96.0;

    let qr = QrCode::encode_text(qrcode_content, QrCodeEcc::Medium)?;
    let mut svg = String::with_capacity(28000);
@@ -67,7 +68,18 @@ pub fn create_qr_svg(qrcode_content: &str) -> Result<String> {
                d.attr("d", path_data)?;
                d.attr("transform", format!("scale({scale})"))
            })
-        })
+        })?;
+        w.elem("g", |d| {
+            d.attr(
+                "transform",
+                format!(
+                    "translate({},{}) scale(2)", // data in qr_overlay_delta.svg-part are 48 x 48, scaling by 2 results in desired logo_size of 96
+                    (all_size - logo_size) / 2.0,
+                    (all_size - logo_size) / 2.0
+                ),
+            )
+        })?
+        .build(|w| w.put_raw_escapable(include_str!("../assets/qr_overlay_delta.svg-part")))
    })?;

    Ok(svg)
@@ -1442,27 +1442,59 @@ async fn add_parts(
    }

    if let Some(node_addr) = mime_parser.get_header(HeaderDef::IrohNodeAddr) {
+        chat_id = DC_CHAT_ID_TRASH;
        match serde_json::from_str::<NodeAddr>(node_addr).context("Failed to parse node address") {
            Ok(node_addr) => {
                info!(context, "Adding iroh peer with address {node_addr:?}.");
-                let instance_id = parent.context("Failed to get parent message")?.id;
-                context.emit_event(EventType::WebxdcRealtimeAdvertisementReceived {
-                    msg_id: instance_id,
-                });
-                if let Some(topic) = get_iroh_topic_for_msg(context, instance_id).await? {
-                    let node_id = node_addr.node_id;
-                    let relay_server = node_addr.relay_url().map(|relay| relay.as_str());
-                    iroh_add_peer_for_topic(context, instance_id, topic, node_id, relay_server)
-                        .await?;
-                    let iroh = context.get_or_try_init_peer_channel().await?;
-                    iroh.maybe_add_gossip_peers(topic, vec![node_addr]).await?;
-                } else {
-                    warn!(
-                        context,
-                        "Could not add iroh peer because {instance_id} has no topic"
-                    );
+                match mime_parser.get_header(HeaderDef::InReplyTo) {
+                    Some(in_reply_to) => match rfc724_mid_exists(context, in_reply_to).await? {
+                        Some((instance_id, _ts_sent)) => {
+                            context.emit_event(EventType::WebxdcRealtimeAdvertisementReceived {
+                                msg_id: instance_id,
+                            });
+                            if let Some(topic) =
+                                get_iroh_topic_for_msg(context, instance_id).await?
+                            {
+                                let node_id = node_addr.node_id;
+                                let relay_server =
+                                    node_addr.relay_url().map(|relay| relay.as_str());
+                                iroh_add_peer_for_topic(
+                                    context,
+                                    instance_id,
+                                    topic,
+                                    node_id,
+                                    relay_server,
+                                )
+                                .await?;
+                                if context
+                                    .get_config_bool(Config::WebxdcRealtimeEnabled)
+                                    .await?
+                                {
+                                    let iroh = context.get_or_try_init_peer_channel().await?;
+                                    iroh.maybe_add_gossip_peers(topic, vec![node_addr]).await?;
+                                }
+                                info!(context, "Added iroh peer to the topic of {instance_id}.");
+                            } else {
+                                warn!(
+                                    context,
+                                    "Could not add iroh peer because {instance_id} has no topic."
+                                );
+                            }
+                        }
+                        None => {
+                            warn!(
+                                context,
+                                "Cannot add iroh peer because WebXDC instance does not exist."
+                            );
+                        }
+                    },
+                    None => {
+                        warn!(
+                            context,
+                            "Cannot add iroh peer because the message has no In-Reply-To."
+                        );
+                    }
                }
-                chat_id = DC_CHAT_ID_TRASH;
            }
            Err(err) => {
                warn!(context, "Couldn't parse NodeAddr: {err:#}.");
@@ -1,13 +1,13 @@
 //! Implementation of [SecureJoin protocols](https://securejoin.delta.chat/).

 use anyhow::{ensure, Context as _, Error, Result};
-use percent_encoding::{utf8_percent_encode, AsciiSet, NON_ALPHANUMERIC};
+use percent_encoding::{utf8_percent_encode, NON_ALPHANUMERIC};

 use crate::aheader::EncryptPreference;
 use crate::chat::{self, get_chat_id_by_grpid, Chat, ChatId, ChatIdBlocked, ProtectionStatus};
 use crate::chatlist_events;
 use crate::config::Config;
-use crate::constants::{Blocked, Chattype};
+use crate::constants::{Blocked, Chattype, NON_ALPHANUMERIC_WITHOUT_DOT};
 use crate::contact::{Contact, ContactId, Origin};
 use crate::context::Context;
 use crate::e2ee::ensure_secret_key_exists;
@@ -34,9 +34,6 @@ use qrinvite::QrInvite;

 use crate::token::Namespace;

-/// Set of characters to percent-encode in email addresses and names.
-pub const NON_ALPHANUMERIC_WITHOUT_DOT: &AsciiSet = &NON_ALPHANUMERIC.remove(b'.');
-
 fn inviter_progress(context: &Context, contact_id: ContactId, progress: usize) {
    debug_assert!(
        progress <= 1000,
@@ -681,6 +681,36 @@ fn new_connection(path: &Path, passphrase: &str) -> Result<Connection> {
    Ok(conn)
 }

+// Tries to clear the freelist to free some space on the disk.
+//
+// This only works if auto_vacuum is enabled.
+async fn incremental_vacuum(context: &Context) -> Result<()> {
+    context
+        .sql
+        .call_write(move |conn| {
+            let mut stmt = conn
+                .prepare("PRAGMA incremental_vacuum")
+                .context("Failed to prepare incremental_vacuum statement")?;
+
+            // It is important to step the statement until it returns no more rows.
+            // Otherwise it will not free as many pages as it can:
+            // <https://stackoverflow.com/questions/53746807/sqlite-incremental-vacuum-removing-only-one-free-page>.
+            let mut rows = stmt
+                .query(())
+                .context("Failed to run incremental_vacuum statement")?;
+            let mut row_count = 0;
+            while let Some(_row) = rows
+                .next()
+                .context("Failed to step incremental_vacuum statement")?
+            {
+                row_count += 1;
+            }
+            info!(context, "Incremental vacuum freed {row_count} pages.");
+            Ok(())
+        })
+        .await
+}
+
 /// Cleanup the account to restore some storage and optimize the database.
 pub async fn housekeeping(context: &Context) -> Result<()> {
    // Setting `Config::LastHousekeeping` at the beginning avoids endless loops when things do not
@@ -713,24 +743,8 @@ pub async fn housekeeping(context: &Context) -> Result<()> {
        );
    }

-    // Try to clear the freelist to free some space on the disk. This
-    // only works if auto_vacuum is enabled.
-    match context
-        .sql
-        .query_row_optional("PRAGMA incremental_vacuum", (), |_row| Ok(()))
-        .await
-    {
-        Err(err) => {
-            warn!(context, "Failed to run incremental vacuum: {err:#}.");
-        }
-        Ok(Some(())) => {
-            // Incremental vacuum returns a zero-column result if it did anything.
-            info!(context, "Successfully ran incremental vacuum.");
-        }
-        Ok(None) => {
-            // Incremental vacuum returned `SQLITE_DONE` immediately,
-            // there were no pages to remove.
-        }
+    if let Err(err) = incremental_vacuum(context).await {
+        warn!(context, "Failed to run incremental vacuum: {err:#}.");
    }

    context
@@ -1370,4 +1384,14 @@ mod tests {

        Ok(())
    }
+
+    /// Tests that incremental_vacuum does not fail.
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_incremental_vacuum() -> Result<()> {
+        let t = TestContext::new().await;
+
+        incremental_vacuum(&t).await?;
+
+        Ok(())
+    }
 }
Author	SHA1	Message	Date
link2xt	189a093e02	feat: compress backups with gzip	2024-10-25 01:15:55 +00:00
link2xt	fc2b111f5d	chore(release): prepare for 1.148.4	2024-10-24 20:25:58 +00:00
link2xt	913d2c45b3	fix: do not wait for connections in maybe_add_gossip_peers() join() method of Gossip [1] waits for at least one connection and this is not what we want because it may block receive_imf() forever if no connection arrives. [1] https://docs.rs/iroh-gossip/0.25.0/iroh_gossip/net/struct.Gossip.html#method.join	2024-10-24 19:59:00 +00:00
link2xt	e32d676a08	fix: normalize proxy URLs before saving into proxy_url	2024-10-24 16:43:10 +00:00
Simon Lauxandiequidoo	9812d5ba75	feat: jsonrpc: add `private_tag` to `Account::Configured` Object (#6107 ) Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>	2024-10-24 16:00:27 +00:00
link2xt	bc7568e39b	chore(release): prepare for 1.148.3	2024-10-24 14:08:59 +00:00
link2xt	11bf1c45d2	test: test that realtime advertisements work after chatting	2024-10-24 13:56:04 +00:00
link2xt	122c23ad4e	api(deltachat-rpc-client): add EventType.WEBXDC_REALTIME_ADVERTISEMENT_RECEIVED	2024-10-24 13:56:04 +00:00
link2xt	a0bde4699e	fix: fix reception of realtime advertisements	2024-10-24 13:56:04 +00:00
link2xt	ac01a4a771	feat: allow sending realtime messages up to 128 KB in size Previous default value was 4 KiB.	2024-10-24 13:55:28 +00:00
link2xt	51f2a8d59e	refactor: generate topic inside create_iroh_header()	2024-10-23 22:33:09 +00:00
bjoern	f208c31cdf	docs: fix DC_QR_PROXY docs (#6099 )	2024-10-23 22:29:06 +02:00
link2xt	acd7a1d17e	chore(release): prepare for 1.148.2	2024-10-23 17:52:24 +00:00
link2xt	db6d451c90	feat: add more logging for iroh initialization and peer addition	2024-10-23 17:48:33 +00:00
link2xt	4b3a6445fb	fix: never initialize Iroh if realtime is disabled	2024-10-23 17:48:33 +00:00
link2xt	aa3ef5011b	build: nix flake update fenix	2024-10-23 03:31:39 +00:00
link2xt	1d3072c287	build: nix flake update nixpkgs	2024-10-23 03:19:33 +00:00
link2xt	4fb59177fa	chore(release): prepare for 1.148.1	2024-10-23 02:37:22 +00:00
link2xt	d841bcb41e	Revert "build: nix flake update" This reverts commit `6f22ce2722`.	2024-10-23 02:07:22 +00:00
link2xt	d205bc410b	chore(release): prepare for 1.148.0	2024-10-23 00:27:49 +00:00
bjoern	0d573ac037	feat: add delta chat logo to QR codes (#6093 ) the chosen error correction allows tolerates about 15% "erroneous codewords", the logo is of a similar size as the old avatars, so it should be fine.	2024-10-23 01:43:50 +02:00
link2xt	a55e33fbc7	fix(sql): run `PRAGMA incremental_vacuum` on a write connection Otherwise it always fails with SQLITE_READONLY: ``` WARNING src/sql.rs:769: Failed to run incremental vacuum: attempt to write a readonly database: Error code 8: Attempt to write a readonly database. ```	2024-10-22 23:23:57 +00:00
bjoern	839b0e94af	api: create QR codes from any data (#6090 ) this PR adds a function that can be used to create any QR code, in a raw form. this can be used to create add-contact as well as add-second-device QR codes (eg. `dc_create_qr_svg(dc_get_securejoin_qr())`) - as well as for other QR codes as proxies. the disadvantage of the rich-formatted QR codes as created by `dc_get_securejoin_qr_svg()` and `dc_backup_provider_get_qr_svg()` were: - they do not look good and cannot interact with UI layout wise (but also tapping eg. an address is not easily possible) - esp. text really looks bad. even with [some](https://github.com/deltachat/deltachat-android/pull/2138/commits/e5dc8fe3d82e8b72631ae5806e9c4c7eec93a37a) [hacks](https://github.com/deltachat/deltachat-android/pull/2215) it [stays buggy](https://github.com/deltachat/deltachat-ios/issues/2200); the bugs mainly come from different SVG implementation, all need their own quirks - accessibility is probably bad as well we thought that time, SVG is a great thing for QR codes, but apart from basic geometrics, it is not. so, we avoid text, this also means to avoid putting an avatar in the middle of the QR code (we can put some generic symbol there, eg. different ones for add-contact and add-second-device). while this looks like a degradation, also other messengers use more raw QR codes. also, we removed many data from the QR code anyway, eg. the email address is no longer there. that time, sharing QR images was more a thing, meanwhile we have invite links, that are much better for that purpose. in theory, we could also leave the SVG path completely and go for PNG - which we did not that time as PNG and text looks bad, as the system font is not easily usable :) but going for PNG would add further challenges as passing binary data around, and also UI-implemtation-wise, that would be a larger step. so, let's stay with SVG in a form we know is compatible. the old QR code functions are deprecated.	2024-10-22 21:49:45 +02:00
@@ -1 +1 @@
 -10-13
 -10-24