fix speed issues with ORDER BY clause

Bumps [native-tls](https://github.com/sfackler/rust-native-tls) from 0.2.8 to 0.2.10.
- [Release notes](https://github.com/sfackler/rust-native-tls/releases)
- [Changelog](https://github.com/sfackler/rust-native-tls/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sfackler/rust-native-tls/compare/v0.2.8...v0.2.10)

---
updated-dependencies:
- dependency-name: native-tls
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

.github/mergeable.yml

@@ -0,0 +1,26 @@
+version: 2
+mergeable:
+  - when: pull_request.*
+    name: "Changelog check"
+    validate:
+      - do: or
+        validate:
+        - do: description
+          must_include:
+            regex: '#skip-changelog'
+        - do: and
+          validate:
+            - do: dependent
+              changed:
+                file: 'src/**'
+                required: ['CHANGELOG.md']
+            - do: dependent
+              changed:
+                file: 'deltachat-ffi/**'
+                required: ['CHANGELOG.md']
+    fail:
+      - do: checks
+        status: 'action_required' 
+        payload:
+          title: Changlog might need an update
+          summary: "Check if CHANGELOG.md needs an update or add #skip-changelog to the PR description."

.github/workflows/ci.yml

@@ -8,6 +8,9 @@ on:
       - staging
       - trying
 
+env:
+  RUSTFLAGS: -Dwarnings
+
 jobs:
 
   fmt:
@@ -21,6 +24,8 @@ jobs:
           toolchain: stable
           override: true
       - run: rustup component add rustfmt
+      - name: Cache rust cargo artifacts
+        uses: swatinem/rust-cache@v1
       - uses: actions-rs/cargo@v1
         with:
           command: fmt
@@ -35,6 +40,8 @@ jobs:
             toolchain: stable
             components: clippy
             override: true
+      - name: Cache rust cargo artifacts
+        uses: swatinem/rust-cache@v1
       - uses: actions-rs/clippy-check@v1
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
@@ -70,19 +77,19 @@ jobs:
         include:
           # Currently used Rust version, same as in `rust-toolchain` file.
           - os: ubuntu-latest
-            rust: 1.54.0
+            rust: 1.59.0
             python: 3.9
           - os: windows-latest
-            rust: 1.54.0
+            rust: 1.59.0
             python: false # Python bindings compilation on Windows is not supported.
 
-          # Minimum Supported Rust Version = 1.51.0
+          # Minimum Supported Rust Version = 1.56.0
           #
           # Minimum Supported Python Version = 3.7
           # This is the minimum version for which manylinux Python wheels are
           # built.
           - os: ubuntu-latest
-            rust: 1.51.0
+            rust: 1.56.0
             python: 3.7
     runs-on: ${{ matrix.os }}
     steps:
@@ -94,31 +101,14 @@ jobs:
         toolchain: ${{ matrix.rust }}
         override: true
 
-    - name: Cache cargo registry
-      uses: actions/cache@v2
-      with:
-        path: ~/.cargo/registry
-        key: ${{ matrix.os }}-${{ matrix.rust }}-cargo-registry-${{ hashFiles('**/Cargo.toml') }}
-
-    - name: Cache cargo index
-      uses: actions/cache@v2
-      with:
-        path: ~/.cargo/git
-        key: ${{ matrix.os }}-${{ matrix.rust }}-cargo-index-${{ hashFiles('**/Cargo.toml') }}
-
-    - name: Cache cargo build
-      uses: actions/cache@v2
-      with:
-        path: target
-        key: ${{ matrix.os }}-${{ matrix.rust }}-cargo-build-target-${{ hashFiles('**/Cargo.toml') }}
+    - name: Cache rust cargo artifacts
+      uses: swatinem/rust-cache@v1
 
     - name: check
       uses: actions-rs/cargo@v1
-      env:
-        RUSTFLAGS: -D warnings
       with:
         command: check
-        args: --all --bins --examples --tests --features repl
+        args: --all --bins --examples --tests --features repl --benches
 
     - name: tests
       uses: actions-rs/cargo@v1

.github/workflows/upload-docs.yml

@@ -0,0 +1,28 @@
+name: Build & Deploy Documentation on rs.delta.chat
+
+on:
+  push:
+    branches:
+      - master
+      - docs-gh-action
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v1
+    - uses: actions-rs/toolchain@v1
+    - name: Build the documentation with cargo
+      run: |
+        cargo doc --package deltachat --no-deps
+    - name: Upload to rs.delta.chat
+      uses: up9cloud/action-rsync@v1.3
+      env:
+        USER: ${{ secrets.USERNAME }}
+        KEY: ${{ secrets.KEY }}
+        HOST: "delta.chat"
+        SOURCE: "target/doc"
+        TARGET: "/var/www/html/rs/"
+

.github/workflows/upload-ffi-docs.yml

@@ -0,0 +1,28 @@
+name: Build & Deploy Documentation on cffi.delta.chat
+
+on:
+  push:
+    branches:
+      - master
+      - docs-gh-action
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v1
+    - uses: actions-rs/toolchain@v1
+    - name: Build the documentation with cargo
+      run: |
+        cargo doc --package deltachat_ffi --no-deps
+    - name: Upload to cffi.delta.chat
+      uses: up9cloud/action-rsync@v1.3
+      env:
+        USER: ${{ secrets.USERNAME }}
+        KEY: ${{ secrets.KEY }}
+        HOST: "delta.chat"
+        SOURCE: "target/doc"
+        TARGET: "/var/www/html/cffi/"
+

CHANGELOG.md

@@ -1,5 +1,162 @@
 # Changelog
 
+## Unreleased
+
+### API changes
+- change semantics of `dc_get_webxdc_status_updates()` second parameter
+  and remove update-id from `DC_EVENT_WEBXDC_STATUS_UPDATE` #3081
+
+### Fixes
+- Hopefully fix a bug where outgoing messages appear twice with Amazon SES #3077
+- do not delete messages without Message-IDs as duplicates #3095
+- Assign replies from a different email address to the correct chat #3119
+- start ephemeral timer when seen status is synchronized via IMAP #3122
+- do not delete duplicate messages on IMAP immediately to accidentally deleting
+  the last copy #3138
+
+### Changes
+- add more SMTP logging #3093
+- place common headers like `From:` before the large `Autocrypt:` header #3079
+- keep track of securejoin joiner status in database to survive restarts #2920
+- remove never used `SentboxMove` option #3111
+- improve speed by caching config values #3131 #3145
+- optimize `markseen_msgs` #3141
+- automatically accept chats with outgoing messages #3143
+- `dc_receive_imf` refactorings #3154 #3156
+- add index to speedup deletion of expired ephemeral messages #3155
+
+
+### Fixes
+- Fix a bug where sometimes the file extension of a long filename containing a dot was cropped #3098
+
+## 1.76.0
+
+### Changes
+- move messages in batches #3058
+- delete messages in batches #3060
+- python: remove arbitrary timeouts from tests #3059
+- refactorings #3026
+
+### Fixes
+- avoid archived, fresh chats #3053
+- Also resync UIDs in folders that are not configured #2289
+- treat "NO" IMAP response to MOVE and COPY commands as an error #3058
+- Fix a bug where messages in the Spam folder created contact requests #3015
+- Fix a bug where drafts disappeared after some days #3067
+- Parse MS Exchange read receipts and mark the original message as read #3075
+- do not retry message sending infinitely in case of permanent SMTP failure #3070
+- set message state to failed when retry limit is exceeded #3072
+
+
+## 1.75.0
+
+### Changes
+- optimize `delete_expired_imap_messages()` #3047
+
+
+## 1.74.0
+
+### Fixes
+- avoid reconnection loop when message without Message-ID is marked as seen #3044
+
+
+## 1.73.0
+
+### API changes
+- added `only_fetch_mvbox` config #3028
+
+### Changes
+- don't watch Sent folder by default #3025
+- use webxdc app name in chatlist/quotes/replies etc. #3027
+- make it possible to cancel message sending by removing the message #3034,
+  this was previosuly removed in 1.71.0 #2939
+- synchronize Seen flags only on watched folders to speed up
+  folder scanning #3041
+- remove direct dependency on `byteorder` crate #3031
+- refactorings #3023 #3013
+- update provider database #3043
+- improve documentation #3017 #3018 #3021
+
+### Fixes
+- fix splitting off text from webxdc messages #3032
+- call slow `delete_expired_imap_messages()` less often #3037
+- make synchronization of Seen status more robust in case unsolicited FETCH
+  result without UID is returned #3022
+- fetch Inbox before scanning folders to ensure iOS does
+  not kill the app before it gets to fetch the Inbox in background #3040
+
+
+## 1.72.0
+
+### Fixes
+- run migrations on backup import #3006
+
+
+## 1.71.0
+
+### API Changes
+- added APIs to handle database passwords: `dc_context_new_closed()`, `dc_context_open()`,
+  `dc_context_is_open()` and `dc_accounts_add_closed_account()` #2956 #2972
+- use second parameter of `dc_imex` to provide backup passphrase #2980
+- added `DC_MSG_WEBXDC`, `dc_send_webxdc_status_update()`,
+  `dc_get_webxdc_status_updates()`, `dc_msg_get_webxdc_blob()`, `dc_msg_get_webxdc_info()`
+  and `DC_EVENT_WEBXDC_STATUS_UPDATE` #2826 #2971 #2975 #2977 #2979 #2993 #2994 #2998 #3001 #3003
+- added `dc_msg_get_parent()` #2984
+- added `dc_msg_force_plaintext()` API for bots #2847
+- allow removing quotes on drafts `dc_msg_set_quote(msg, NULL)` #2950
+- removed `mvbox_watch` option; watching is enabled when `mvbox_move` is enabled #2906
+- removed `inbox_watch` option #2922
+- deprecated `os_name` in `dc_context_new()`, pass `NULL` or an empty string #2956
+
+### Changes
+- start making it possible to write to mailing lists #2736
+- add `hop_info` to `dc_get_info()` #2751 #2914 #2923
+- add information about whether the database is encrypted or not to `dc_get_info()` #3000
+- selfstatus now defaults to empty #2951 #2960
+- validate detached cryptographic signatures as used eg. by Thunderbird #2865
+- do not change the draft's `msg_id` on updates and sending #2887
+- add `imap` table to keep track of message UIDs #2909 #2938
+- replace `SendMsgToSmtp` jobs which stored outgoing messages in blobdir with `smtp` SQL table #2939 #2996
+- sql: enable `auto_vacuum=INCREMENTAL` #2931
+- sql: build rusqlite with sqlcipher #2934
+- synchronize Seen status across devices #2942
+- `dc_preconfigure_keypair` now takes ascii armored keys instead of base64 #2862
+- put removed member in Bcc instead of To in the message about removal #2864
+- improve group updates #2889
+- re-write the blob filename creation loop #2981
+- update provider database (11 Jan 2022) #2959
+- python: allow timeout for internal configure tracker API #2967
+- python: remove API deprecated in Python 3.10 #2907
+- refactorings #2932 #2957 #2947
+- improve tests #2863 #2866 #2881 #2908 #2918 #2901 #2973
+- improve documentation #2880 #2886 #2895
+- improve ci #2919 #2926 #2969 #2999
+
+### Fixes
+- fix leaving groups #2929
+- fix unread count #2861
+- make `add_parts()` not early-exit #2879
+- recognize MS Exchange read receipts as read receipts #2890
+- create parent directory if creating a new file fails #2978
+- save "configured" flag later #2974
+- improve log #2928
+- `dc_receive_imf`: don't fail on invalid address in the To field #2940
+
+
+## 1.70.0
+
+### Fixes
+- fix: do not abort Param parsing on unknown keys #2856
+- fix: execute `Chat-Group-Member-Removed:` even when arriving disordered #2857
+
+
+## 1.69.0
+
+### Fixes
+- fix group-related system messages in multi-device setups #2848
+- fix "Google Workspace" (former "G Suite") issues related to bad resolvers #2852
+
+
 ## 1.68.0
 
 ### Fixes

Cargo.toml

@@ -1,10 +1,10 @@
 [package]
 name = "deltachat"
-version = "1.68.0"
+version = "1.76.0"
 authors = ["Delta Chat Developers (ML) <delta@codespeak.net>"]
-edition = "2018"
+edition = "2021"
 license = "MPL-2.0"
-resolver = "2"
+rust-version = "1.56"
 
 [profile.dev]
 debug = 0
@@ -20,14 +20,13 @@ anyhow = "1"
 async-imap = { git = "https://github.com/async-email/async-imap" }
 async-native-tls = { version = "0.3" }
 async-smtp = { git = "https://github.com/async-email/async-smtp", branch="master", features = ["socks5"] }
-async-std-resolver = "0.20"
-async-std = { version = "1", features = ["unstable"] }
+async-std-resolver = "0.21"
+async-std = { version = "1" }
 async-tar = { version = "0.4", default-features=false }
 async-trait = "0.1"
 backtrace = "0.3"
 base64 = "0.13"
 bitflags = "1.3"
-byteorder = "1.3"
 chrono = "0.4"
 dirs = { version = "4", optional=true }
 email = { git = "https://github.com/deltachat/rust-email", branch = "master" }
@@ -35,37 +34,37 @@ encoded-words = { git = "https://github.com/async-email/encoded-words", branch="
 escaper = "0.1"
 futures = "0.3"
 hex = "0.4.0"
-image = { version = "0.23.5", default-features=false, features = ["gif", "jpeg", "ico", "png", "pnm", "webp", "bmp"] }
+image = { version = "0.24.1", default-features=false, features = ["gif", "jpeg", "ico", "png", "pnm", "webp", "bmp"] }
 kamadak-exif = "0.5"
 lettre_email = { git = "https://github.com/deltachat/lettre", branch = "master" }
 libc = "0.2"
-log = {version = "0.4.8", optional = true }
+log = {version = "0.4.16", optional = true }
 mailparse = "0.13"
 native-tls = "0.2"
 num_cpus = "1.13"
 num-derive = "0.3"
 num-traits = "0.2"
-once_cell = "1.8.0"
+once_cell = "1.10.0"
 percent-encoding = "2.0"
 pgp = { version = "0.7", default-features = false }
 pretty_env_logger = { version = "0.4", optional = true }
 quick-xml = "0.22"
 r2d2 = "0.8"
-r2d2_sqlite = "0.19"
+r2d2_sqlite = "0.20"
 rand = "0.7"
 regex = "1.5"
-rusqlite = "0.26"
+rusqlite = { version = "0.27", features = ["sqlcipher"] }
 rust-hsluv = "0.1"
-rustyline = { version = "9.0", optional = true }
+rustyline = { version = "9", optional = true }
 sanitize-filename = "0.3"
 serde_json = "1.0"
 serde = { version = "1.0", features = ["derive"] }
-sha-1 = "0.9"
-sha2 = "0.9"
+sha-1 = "0.10"
+sha2 = "0.10"
 smallvec = "1"
-stop-token = "0.6"
-strum = "0.23"
-strum_macros = "0.23"
+stop-token = "0.7"
+strum = "0.24"
+strum_macros = "0.24"
 surf = { version = "2.3", default-features = false, features = ["h1-client"] }
 thiserror = "1"
 toml = "0.5"
@@ -74,13 +73,14 @@ uuid = { version = "0.8", features = ["serde", "v4"] }
 fast-socks5 = "0.4"
 humansize = "1"
 qrcodegen = "1.7.0"
-tagger = "3.2.1"
-textwrap = "0.14.2"
+tagger = "4.3.3"
+textwrap = "0.15.0"
+zip = { version = "0.6.0", default-features = false, features = ["deflate"] }
 
 [dev-dependencies]
 ansi_term = "0.12.0"
 async-std = { version = "1", features = ["unstable", "attributes"] }
-criterion = "0.3"
+criterion = { version = "0.3.4", features = ["async_std"] }
 futures-lite = "1.12"
 log = "0.4"
 pretty_env_logger = "0.4"
@@ -116,9 +116,17 @@ harness = false
 name = "search_msgs"
 harness = false
 
+[[bench]]
+name = "receive_emails"
+harness = false
+
+[[bench]]
+name = "get_chat_msgs"
+harness = false
+
 [features]
 default = ["vendored"]
 internals = []
 repl = ["internals", "rustyline", "log", "pretty_env_logger", "ansi_term", "dirs"]
-vendored = ["async-native-tls/vendored", "async-smtp/native-tls-vendored", "rusqlite/bundled"]
+vendored = ["async-native-tls/vendored", "async-smtp/native-tls-vendored", "rusqlite/bundled-sqlcipher-vendored-openssl"]
 nightly = ["pgp/nightly"]

README.md

@@ -12,6 +12,8 @@ To download and install the official compiler for the Rust programming language,
 $ curl https://sh.rustup.rs -sSf | sh
 ```
 
+> On Windows, you may need to also install **Perl** to be able to compile deltachat-core.
+
 ## Using the CLI client
 
 Compile and run Delta Chat Core command line utility, using `cargo`:
@@ -125,11 +127,11 @@ $ cargo test -- --ignored
 
 Language bindings are available for:
 
-- [C](https://c.delta.chat)
-- [Node.js](https://www.npmjs.com/package/deltachat-node)
-- [Python](https://py.delta.chat)
-- [Go](https://github.com/deltachat/go-deltachat/)
-- [Free Pascal](https://github.com/deltachat/deltachat-fp/)
+- **C** \[[📂 source](./deltachat-ffi) | [📚 docs](https://c.delta.chat)\]
+- **Node.js** \[[📂 source](https://github.com/deltachat/deltachat-node) | [📦 npm](https://www.npmjs.com/package/deltachat-node) | [📚 docs](https://js.delta.chat)\]
+- **Python** \[[📂 source](./python) | [📦 pypi](https://pypi.org/project/deltachat) | [📚 docs](https://py.delta.chat)\]
+- **Go** \[[📂 source](https://github.com/deltachat/go-deltachat/)\]
+- **Free Pascal** \[[📂 source](https://github.com/deltachat/deltachat-fp/)\]
 - **Java** and **Swift** (contained in the Android/iOS repos)
 
 The following "frontend" projects make use of the Rust-library

assets/icon-webxdc.svg

@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="80mm"
+   height="297mm"
+   viewBox="0 0 80 297"
+   version="1.1"
+   id="svg71"
+   inkscape:version="1.0.2 (e86c8708, 2021-01-15)"
+   sodipodi:docname="icon-webxdc.svg"
+   inkscape:export-filename="C:\Users\user\OneDrive - BFW-Leipzig\Documents\LogoDC\finalohnerand.png"
+   inkscape:export-xdpi="96"
+   inkscape:export-ydpi="96">
+  <metadata
+     id="metadata856">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <sodipodi:namedview
+     id="namedview73"
+     pagecolor="#767676"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0"
+     inkscape:pagecheckerboard="true"
+     inkscape:document-units="mm"
+     showgrid="false"
+     showborder="false"
+     inkscape:snap-bbox="true"
+     inkscape:bbox-paths="true"
+     inkscape:bbox-nodes="true"
+     inkscape:snap-bbox-edge-midpoints="true"
+     inkscape:snap-bbox-midpoints="true"
+     inkscape:object-paths="true"
+     inkscape:snap-intersection-paths="true"
+     inkscape:zoom="1.4142136"
+     inkscape:cx="-90.271136"
+     inkscape:cy="-1233.1209"
+     inkscape:window-width="1864"
+     inkscape:window-height="1027"
+     inkscape:window-x="56"
+     inkscape:window-y="25"
+     inkscape:window-maximized="1"
+     inkscape:current-layer="layer1"
+     inkscape:snap-global="false"
+     showguides="false"
+     inkscape:guide-bbox="true"
+     inkscape:document-rotation="0"
+     units="px">
+    <sodipodi:guide
+       position="-154.76097,641.11689"
+       orientation="0,-1"
+       id="guide21118" />
+    <sodipodi:guide
+       position="-60.286487,633.36619"
+       orientation="0,-1"
+       id="guide21120" />
+  </sodipodi:namedview>
+  <defs
+     id="defs68">
+    <linearGradient
+       id="linearGradient4375">
+      <stop
+         style="stop-color:#364e59;stop-opacity:1;"
+         offset="0"
+         id="stop4377" />
+      <stop
+         style="stop-color:#364e59;stop-opacity:0;"
+         offset="1"
+         id="stop4379" />
+    </linearGradient>
+  </defs>
+  <g
+     inkscape:label="Ebene 1"
+     inkscape:groupmode="layer"
+     id="layer1">
+    <rect
+       style="fill:#1a1a1a;stroke:#000000;stroke-width:0.167903"
+       id="rect880"
+       width="79.8321"
+       height="79.8321"
+       x="-64.03286"
+       y="-375.9097"
+       ry="0" />
+    <path
+       inkscape:connector-curvature="0"
+       id="path3799-2"
+       d="m -24.089585,-372.59579 c -19.986026,0.24336 -36.196903,16.666 -36.196903,36.67011 0,20.00409 16.210877,36.03233 36.196903,35.78912 19.0024236,-0.076 14.5340713,-10.6146 35.538854,-0.85693 -11.50627538,-17.97454 0.390097,-20.36737 0.658079,-35.81316 0,-20.00411 -16.2108788,-36.03235 -36.196911,-35.78914 z"
+       style="fill:#364e59;fill-opacity:1;stroke:none;stroke-width:1.93355"
+       sodipodi:nodetypes="sscccs" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="M -54.193871,-325.26419 Z"
+       id="path3846" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="M -49.397951,-326.67773 Z"
+       id="path3848" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="m -49.397951,-326.67773 v 0 0"
+       id="path3850" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:0.01;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+       d="m -51.35133,-325.0334 -7.964067,5.98895 z"
+       id="path3965" />
+    <path
+       inkscape:connector-curvature="0"
+       id="path11037"
+       d="m -24.089585,-372.19891 c -19.986026,0.24156 -36.196903,16.54352 -36.196903,36.40062 0,7.86524 2.543315,15.1113 6.857155,20.97971 6.577146,8.94734 11.123515,9.77363 11.123515,9.77363 1.343237,1.78324 10.270932,4.3223 10.270932,4.3223 l 16.791727,-70.86654 -0.468369,-0.33457 c 0.458597,0.26445 0.428277,-0.27515 -8.378035,-0.27515 z"
+       style="fill:#7cc5cc;fill-opacity:1;stroke:none;stroke-width:1.92643"
+       sodipodi:nodetypes="sssccccss" />
+    <path
+       style="fill:none;stroke:#000000;stroke-width:0.264583px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       d="M -49.944239,-310.69957 Z"
+       id="path13674" />
+    <g
+       id="g15178"
+       transform="matrix(0.79975737,0,0,0.79975737,53.088959,-63.716396)">
+      <rect
+         style="fill:#364e59;fill-opacity:1;stroke-width:0.01;stroke-miterlimit:4;stroke-dasharray:none"
+         id="rect15072"
+         width="29.897917"
+         height="6.8791666"
+         x="-334.4964"
+         y="-154.51025"
+         transform="rotate(45)" />
+      <rect
+         style="fill:#364e59;fill-opacity:1;stroke-width:0.01;stroke-miterlimit:4;stroke-dasharray:none"
+         id="rect15072-5"
+         width="29.897917"
+         height="6.8791666"
+         x="147.63107"
+         y="-334.4964"
+         transform="rotate(-45)"
+         inkscape:transform-center-x="-0.74835017"
+         inkscape:transform-center-y="0.37417525" />
+    </g>
+    <g
+       id="g22468"
+       transform="translate(3.3033974)">
+      <g
+         id="g15178-0"
+         transform="matrix(-0.79975737,0,0,0.79975737,-103.11028,-63.716404)"
+         style="fill:#7cc5cc;fill-opacity:1">
+        <rect
+           style="fill:#7cc5cc;fill-opacity:1;stroke-width:0.01;stroke-miterlimit:4;stroke-dasharray:none"
+           id="rect15072-2"
+           width="29.897917"
+           height="6.8791666"
+           x="-334.4964"
+           y="-154.51025"
+           transform="rotate(45)" />
+        <rect
+           style="fill:#7cc5cc;fill-opacity:1;stroke-width:0.01;stroke-miterlimit:4;stroke-dasharray:none"
+           id="rect15072-5-5"
+           width="29.897917"
+           height="6.8791666"
+           x="147.63107"
+           y="-334.4964"
+           transform="rotate(-45)"
+           inkscape:transform-center-x="-0.74835017"
+           inkscape:transform-center-y="0.37417525" />
+      </g>
+    </g>
+  </g>
+</svg>

benches/contacts.rs

@@ -8,9 +8,7 @@ async fn address_book_benchmark(n: u32, read_count: u32) {
     let dir = tempdir().unwrap();
     let dbfile = dir.path().join("db.sqlite");
     let id = 100;
-    let context = Context::new("FakeOS".into(), dbfile.into(), id)
-        .await
-        .unwrap();
+    let context = Context::new(dbfile.into(), id).await.unwrap();
 
     let book = (0..n)
         .map(|i| format!("Name {}\naddr{}@example.org\n", i, i))

benches/create_account.rs

@@ -8,7 +8,7 @@ async fn create_accounts(n: u32) {
     let dir = tempdir().unwrap();
     let p: PathBuf = dir.path().join("accounts").into();
 
-    let accounts = Accounts::new("my_os".into(), p.clone()).await.unwrap();
+    let mut accounts = Accounts::new(p.clone()).await.unwrap();
 
     for expected_id in 2..n {
         let id = accounts.add_account().await.unwrap();

benches/get_chat_msgs.rs

@@ -0,0 +1,38 @@
+use async_std::path::Path;
+
+use criterion::async_executor::AsyncStdExecutor;
+use criterion::{black_box, criterion_group, criterion_main, Criterion};
+
+use deltachat::chat::{self, ChatId};
+use deltachat::chatlist::Chatlist;
+use deltachat::context::Context;
+
+async fn get_chat_msgs_benchmark(dbfile: &Path, chats: &[ChatId]) {
+    let id = 100;
+    let context = Context::new(dbfile.into(), id).await.unwrap();
+
+    for c in chats.iter().take(10) {
+        black_box(chat::get_chat_msgs(&context, *c, 0, None).await.ok());
+    }
+}
+
+fn criterion_benchmark(c: &mut Criterion) {
+    // To enable this benchmark, set `DELTACHAT_BENCHMARK_DATABASE` to some large database with many
+    // messages, such as your primary account.
+    if let Ok(path) = std::env::var("DELTACHAT_BENCHMARK_DATABASE") {
+        let chats: Vec<_> = async_std::task::block_on(async {
+            let context = Context::new((&path).into(), 100).await.unwrap();
+            let chatlist = Chatlist::try_load(&context, 0, None, None).await.unwrap();
+            let len = chatlist.len();
+            (0..len).map(|i| chatlist.get_chat_id(i).unwrap()).collect()
+        });
+
+        c.bench_function("Load all chats", |b| {
+            b.to_async(AsyncStdExecutor)
+                .iter(|| get_chat_msgs_benchmark(black_box(&path.as_ref()), black_box(&chats)))
+        });
+    }
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);

benches/receive_emails.rs

@@ -0,0 +1,84 @@
+use async_std::{path::PathBuf, task::block_on};
+use criterion::{
+    async_executor::AsyncStdExecutor, black_box, criterion_group, criterion_main, BatchSize,
+    Criterion,
+};
+use deltachat::{
+    config::Config,
+    context::Context,
+    dc_receive_imf::dc_receive_imf,
+    imex::{imex, ImexMode},
+};
+use tempfile::tempdir;
+
+async fn recv_all_emails(context: Context) -> Context {
+    for i in 0..100 {
+        let imf_raw = format!(
+            "Subject: Benchmark
+Message-ID: Mr.OssSYnOFkhR.{i}@testrun.org
+Date: Sat, 07 Dec 2019 19:00:27 +0000
+To: alice@example.com
+From: sender@testrun.org
+Chat-Version: 1.0
+Chat-Disposition-Notification-To: sender@testrun.org
+Chat-User-Avatar: 0
+In-Reply-To: Mr.OssSYnOFkhR.{i_dec}@testrun.org
+MIME-Version: 1.0
+
+Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
+
+Hello {i}",
+            i = i,
+            i_dec = i - 1,
+        );
+        dc_receive_imf(&context, black_box(imf_raw.as_bytes()), "INBOX", false)
+            .await
+            .unwrap();
+    }
+    context
+}
+
+async fn create_context() -> Context {
+    let dir = tempdir().unwrap();
+    let dbfile = dir.path().join("db.sqlite");
+    let id = 100;
+    let context = Context::new(dbfile.into(), id).await.unwrap();
+
+    let backup: PathBuf = std::env::current_dir()
+        .unwrap()
+        .join("delta-chat-backup.tar")
+        .into();
+    if backup.exists().await {
+        println!("Importing backup");
+        imex(&context, ImexMode::ImportBackup, &backup, None)
+            .await
+            .unwrap();
+    }
+
+    let addr = "alice@example.com";
+    context.set_config(Config::Addr, Some(addr)).await.unwrap();
+    context
+        .set_config(Config::ConfiguredAddr, Some(addr))
+        .await
+        .unwrap();
+    context
+        .set_config(Config::Configured, Some("1"))
+        .await
+        .unwrap();
+    context
+}
+
+fn criterion_benchmark(c: &mut Criterion) {
+    let mut group = c.benchmark_group("Receive messages");
+    group.bench_function("Receive 100 simple text msgs", |b| {
+        b.to_async(AsyncStdExecutor).iter_batched(
+            || block_on(create_context()),
+            |context| recv_all_emails(black_box(context)),
+            BatchSize::LargeInput,
+        );
+    });
+    group.finish();
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);

benches/search_msgs.rs

@@ -6,9 +6,7 @@ use std::path::Path;
 async fn search_benchmark(path: impl AsRef<Path>) {
     let dbfile = path.as_ref();
     let id = 100;
-    let context = Context::new("FakeOS".into(), dbfile.into(), id)
-        .await
-        .unwrap();
+    let context = Context::new(dbfile.into(), id).await.unwrap();
 
     for _ in 0..10u32 {
         context.search_msgs(None, "hello").await.unwrap();

deltachat-ffi/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat_ffi"
-version = "1.68.0"
+version = "1.76.0"
 description = "Deltachat FFI"
 authors = ["Delta Chat Developers (ML) <delta@codespeak.net>"]
 edition = "2018"

deltachat-ffi/README.md

@@ -1,5 +1,9 @@
 # Delta Chat C Interface
 
+## Installation
+
+see `Installing libdeltachat system wide` in [../README.md](../README.md)
+
 ## Documentation
 
 To generate the C Interface documentation,

deltachat-ffi/src/lib.rs

@@ -27,15 +27,17 @@ use async_std::sync::RwLock;
 use async_std::task::{block_on, spawn};
 use deltachat::qr_code_generator::get_securejoin_qr_svg;
 use num_traits::{FromPrimitive, ToPrimitive};
+use rand::Rng;
 
 use deltachat::chat::{ChatId, ChatVisibility, MuteDuration, ProtectionStatus};
 use deltachat::constants::DC_MSG_ID_LAST_SPECIAL;
-use deltachat::contact::{Contact, Origin};
+use deltachat::contact::{Contact, ContactId, Origin};
 use deltachat::context::Context;
 use deltachat::ephemeral::Timer as EphemeralTimer;
 use deltachat::key::DcKey;
 use deltachat::message::MsgId;
 use deltachat::stock_str::StockMessage;
+use deltachat::webxdc::StatusUpdateSerial;
 use deltachat::*;
 use deltachat::{accounts::Accounts, log::LogExt};
 
@@ -63,7 +65,7 @@ pub type dc_context_t = Context;
 
 #[no_mangle]
 pub unsafe extern "C" fn dc_context_new(
-    os_name: *const libc::c_char,
+    _os_name: *const libc::c_char,
     dbfile: *const libc::c_char,
     blobdir: *const libc::c_char,
 ) -> *mut dc_context_t {
@@ -74,21 +76,10 @@ pub unsafe extern "C" fn dc_context_new(
         return ptr::null_mut();
     }
 
-    let os_name = if os_name.is_null() {
-        String::from("DcFFI")
-    } else {
-        to_string_lossy(os_name)
-    };
-
     let ctx = if blobdir.is_null() || *blobdir == 0 {
-        use rand::Rng;
         // generate random ID as this functionality is not yet available on the C-api.
         let id = rand::thread_rng().gen();
-        block_on(Context::new(
-            os_name,
-            as_path(dbfile).to_path_buf().into(),
-            id,
-        ))
+        block_on(Context::new(as_path(dbfile).to_path_buf().into(), id))
     } else {
         eprintln!("blobdir can not be defined explicitly anymore");
         return ptr::null_mut();
@@ -96,12 +87,63 @@ pub unsafe extern "C" fn dc_context_new(
     match ctx {
         Ok(ctx) => Box::into_raw(Box::new(ctx)),
         Err(err) => {
-            eprintln!("failed to create context: {}", err);
+            eprintln!("failed to create context: {:#}", err);
             ptr::null_mut()
         }
     }
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn dc_context_new_closed(dbfile: *const libc::c_char) -> *mut dc_context_t {
+    setup_panic!();
+
+    if dbfile.is_null() {
+        eprintln!("ignoring careless call to dc_context_new_closed()");
+        return ptr::null_mut();
+    }
+
+    let id = rand::thread_rng().gen();
+    match block_on(Context::new_closed(
+        as_path(dbfile).to_path_buf().into(),
+        id,
+    )) {
+        Ok(context) => Box::into_raw(Box::new(context)),
+        Err(err) => {
+            eprintln!("failed to create context: {:#}", err);
+            ptr::null_mut()
+        }
+    }
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn dc_context_open(
+    context: *mut dc_context_t,
+    passphrase: *const libc::c_char,
+) -> libc::c_int {
+    if context.is_null() {
+        eprintln!("ignoring careless call to dc_context_open()");
+        return 0;
+    }
+
+    let ctx = &*context;
+    let passphrase = to_string_lossy(passphrase);
+    block_on(ctx.open(passphrase))
+        .log_err(ctx, "dc_context_open() failed")
+        .map(|b| b as libc::c_int)
+        .unwrap_or(0)
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn dc_context_is_open(context: *mut dc_context_t) -> libc::c_int {
+    if context.is_null() {
+        eprintln!("ignoring careless call to dc_context_is_open()");
+        return 0;
+    }
+
+    let ctx = &*context;
+    block_on(ctx.is_open()) as libc::c_int
+}
+
 /// Release the context structure.
 ///
 /// This function releases the memory of the `dc_context_t` structure.
@@ -193,7 +235,7 @@ pub unsafe extern "C" fn dc_get_config(
                     .unwrap_or_default()
                     .strdup(),
                 Err(_) => {
-                    warn!(ctx, "dc_get_config(): invalid key");
+                    warn!(ctx, "dc_get_config(): invalid key '{}'", &key);
                     "".strdup()
                 }
             }
@@ -451,14 +493,17 @@ pub unsafe extern "C" fn dc_event_get_data1_int(event: *mut dc_event_t) -> libc:
         | EventType::ChatEphemeralTimerModified { chat_id, .. } => chat_id.to_u32() as libc::c_int,
         EventType::ContactsChanged(id) | EventType::LocationChanged(id) => {
             let id = id.unwrap_or_default();
-            id as libc::c_int
+            id.to_u32() as libc::c_int
         }
         EventType::ConfigureProgress { progress, .. } | EventType::ImexProgress(progress) => {
             *progress as libc::c_int
         }
         EventType::ImexFileWritten(_) => 0,
         EventType::SecurejoinInviterProgress { contact_id, .. }
-        | EventType::SecurejoinJoinerProgress { contact_id, .. } => *contact_id as libc::c_int,
+        | EventType::SecurejoinJoinerProgress { contact_id, .. } => {
+            contact_id.to_u32() as libc::c_int
+        }
+        EventType::WebxdcStatusUpdate(msg_id) => msg_id.to_u32() as libc::c_int,
     }
 }
 
@@ -491,6 +536,7 @@ pub unsafe extern "C" fn dc_event_get_data2_int(event: *mut dc_event_t) -> libc:
         | EventType::MsgsNoticed(_)
         | EventType::ConnectivityChanged
         | EventType::SelfavatarChanged
+        | EventType::WebxdcStatusUpdate(_)
         | EventType::ChatModified(_) => 0,
         EventType::MsgsChanged { msg_id, .. }
         | EventType::IncomingMsg { msg_id, .. }
@@ -541,6 +587,7 @@ pub unsafe extern "C" fn dc_event_get_data2_str(event: *mut dc_event_t) -> *mut
         | EventType::SecurejoinJoinerProgress { .. }
         | EventType::ConnectivityChanged
         | EventType::SelfavatarChanged
+        | EventType::WebxdcStatusUpdate(_)
         | EventType::ChatEphemeralTimerModified { .. } => ptr::null_mut(),
         EventType::ConfigureProgress { comment, .. } => {
             if let Some(comment) = comment {
@@ -642,8 +689,8 @@ pub unsafe extern "C" fn dc_preconfigure_keypair(
     let ctx = &*context;
     block_on(async move {
         let addr = dc_tools::EmailAddress::new(&to_string_lossy(addr))?;
-        let public = key::SignedPublicKey::from_base64(&to_string_lossy(public_data))?;
-        let secret = key::SignedSecretKey::from_base64(&to_string_lossy(secret_data))?;
+        let public = key::SignedPublicKey::from_asc(&to_string_lossy(public_data))?.0;
+        let secret = key::SignedSecretKey::from_asc(&to_string_lossy(secret_data))?.0;
         let keypair = key::KeyPair {
             addr,
             public,
@@ -670,7 +717,11 @@ pub unsafe extern "C" fn dc_get_chatlist(
     let ctx = &*context;
     let qs = to_opt_string_lossy(query_str);
 
-    let qi = if query_id == 0 { None } else { Some(query_id) };
+    let qi = if query_id == 0 {
+        None
+    } else {
+        Some(ContactId::new(query_id))
+    };
 
     block_on(async move {
         match chatlist::Chatlist::try_load(ctx, flags as usize, qs.as_deref(), qi)
@@ -698,7 +749,7 @@ pub unsafe extern "C" fn dc_create_chat_by_contact_id(
     let ctx = &*context;
 
     block_on(async move {
-        ChatId::create_for_contact(ctx, contact_id)
+        ChatId::create_for_contact(ctx, ContactId::new(contact_id))
             .await
             .log_err(ctx, "Failed to create chat from contact_id")
             .map(|id| id.to_u32())
@@ -718,7 +769,7 @@ pub unsafe extern "C" fn dc_get_chat_id_by_contact_id(
     let ctx = &*context;
 
     block_on(async move {
-        ChatId::lookup_by_contact(ctx, contact_id)
+        ChatId::lookup_by_contact(ctx, ContactId::new(contact_id))
             .await
             .log_err(ctx, "Failed to get chat for contact_id")
             .unwrap_or_default() // unwraps the Result
@@ -830,6 +881,48 @@ pub unsafe extern "C" fn dc_send_videochat_invitation(
     })
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn dc_send_webxdc_status_update(
+    context: *mut dc_context_t,
+    msg_id: u32,
+    json: *const libc::c_char,
+    descr: *const libc::c_char,
+) -> libc::c_int {
+    if context.is_null() {
+        eprintln!("ignoring careless call to dc_send_webxdc_status_update()");
+        return 0;
+    }
+    let ctx = &*context;
+
+    block_on(ctx.send_webxdc_status_update(
+        MsgId::new(msg_id),
+        &to_string_lossy(json),
+        &to_string_lossy(descr),
+    ))
+    .log_err(ctx, "Failed to send webxdc update")
+    .is_ok() as libc::c_int
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn dc_get_webxdc_status_updates(
+    context: *mut dc_context_t,
+    msg_id: u32,
+    last_known_serial: u32,
+) -> *mut libc::c_char {
+    if context.is_null() {
+        eprintln!("ignoring careless call to dc_get_webxdc_status_updates()");
+        return "".strdup();
+    }
+    let ctx = &*context;
+
+    block_on(ctx.get_webxdc_status_updates(
+        MsgId::new(msg_id),
+        StatusUpdateSerial::new(last_known_serial),
+    ))
+    .unwrap_or_else(|_| "".to_string())
+    .strdup()
+}
+
 #[no_mangle]
 pub unsafe extern "C" fn dc_set_draft(
     context: *mut dc_context_t,
@@ -1253,7 +1346,10 @@ pub unsafe extern "C" fn dc_get_chat_contacts(
         let arr = dc_array_t::from(
             chat::get_chat_contacts(ctx, ChatId::new(chat_id))
                 .await
-                .unwrap_or_log_default(ctx, "Failed get_chat_contacts"),
+                .unwrap_or_log_default(ctx, "Failed get_chat_contacts")
+                .iter()
+                .map(|id| id.to_u32())
+                .collect::<Vec<u32>>(),
         );
         Box::into_raw(Box::new(arr))
     })
@@ -1363,7 +1459,7 @@ pub unsafe extern "C" fn dc_is_contact_in_chat(
     block_on(chat::is_contact_in_chat(
         ctx,
         ChatId::new(chat_id),
-        contact_id,
+        ContactId::new(contact_id),
     ))
     .log_err(ctx, "is_contact_in_chat failed")
     .unwrap_or_default() as libc::c_int
@@ -1384,7 +1480,7 @@ pub unsafe extern "C" fn dc_add_contact_to_chat(
     block_on(chat::add_contact_to_chat(
         ctx,
         ChatId::new(chat_id),
-        contact_id,
+        ContactId::new(contact_id),
     ))
     .log_err(ctx, "Failed to add contact")
     .is_ok() as libc::c_int
@@ -1405,7 +1501,7 @@ pub unsafe extern "C" fn dc_remove_contact_from_chat(
     block_on(chat::remove_contact_from_chat(
         ctx,
         ChatId::new(chat_id),
-        contact_id,
+        ContactId::new(contact_id),
     ))
     .log_err(ctx, "Failed to remove contact")
     .is_ok() as libc::c_int
@@ -1737,10 +1833,11 @@ pub unsafe extern "C" fn dc_lookup_contact_id_by_addr(
     let ctx = &*context;
 
     block_on(async move {
-        Contact::lookup_id_by_addr(ctx, to_string_lossy(addr), Origin::IncomingReplyTo)
+        Contact::lookup_id_by_addr(ctx, &to_string_lossy(addr), Origin::IncomingReplyTo)
             .await
             .unwrap_or_log_default(ctx, "failed to lookup id")
-            .unwrap_or(0)
+            .map(|id| id.to_u32())
+            .unwrap_or_default()
     })
 }
 
@@ -1760,6 +1857,7 @@ pub unsafe extern "C" fn dc_create_contact(
     block_on(async move {
         Contact::create(ctx, &name, &to_string_lossy(addr))
             .await
+            .map(|id| id.to_u32())
             .unwrap_or(0)
     })
 }
@@ -1798,7 +1896,9 @@ pub unsafe extern "C" fn dc_get_contacts(
 
     block_on(async move {
         match Contact::get_all(ctx, flags, query).await {
-            Ok(contacts) => Box::into_raw(Box::new(dc_array_t::from(contacts))),
+            Ok(contacts) => Box::into_raw(Box::new(dc_array_t::from(
+                contacts.iter().map(|id| id.to_u32()).collect::<Vec<u32>>(),
+            ))),
             Err(_) => ptr::null_mut(),
         }
     })
@@ -1835,7 +1935,10 @@ pub unsafe extern "C" fn dc_get_blocked_contacts(
             Contact::get_all_blocked(ctx)
                 .await
                 .log_err(ctx, "Can't get blocked contacts")
-                .unwrap_or_default(),
+                .unwrap_or_default()
+                .iter()
+                .map(|id| id.to_u32())
+                .collect::<Vec<u32>>(),
         )))
     })
 }
@@ -1846,18 +1949,18 @@ pub unsafe extern "C" fn dc_block_contact(
     contact_id: u32,
     block: libc::c_int,
 ) {
-    if context.is_null() || contact_id <= constants::DC_CONTACT_ID_LAST_SPECIAL as u32 {
+    if context.is_null() || contact_id <= constants::DC_CONTACT_ID_LAST_SPECIAL.to_u32() {
         eprintln!("ignoring careless call to dc_block_contact()");
         return;
     }
     let ctx = &*context;
     block_on(async move {
         if block == 0 {
-            Contact::unblock(ctx, contact_id)
+            Contact::unblock(ctx, ContactId::new(contact_id))
                 .await
                 .ok_or_log_msg(ctx, "Can't unblock contact");
         } else {
-            Contact::block(ctx, contact_id)
+            Contact::block(ctx, ContactId::new(contact_id))
                 .await
                 .ok_or_log_msg(ctx, "Can't block contact");
         }
@@ -1876,7 +1979,7 @@ pub unsafe extern "C" fn dc_get_contact_encrinfo(
     let ctx = &*context;
 
     block_on(async move {
-        Contact::get_encrinfo(ctx, contact_id)
+        Contact::get_encrinfo(ctx, ContactId::new(contact_id))
             .await
             .map(|s| s.strdup())
             .unwrap_or_else(|e| {
@@ -1891,14 +1994,14 @@ pub unsafe extern "C" fn dc_delete_contact(
     context: *mut dc_context_t,
     contact_id: u32,
 ) -> libc::c_int {
-    if context.is_null() || contact_id <= constants::DC_CONTACT_ID_LAST_SPECIAL as u32 {
+    if context.is_null() || contact_id <= constants::DC_CONTACT_ID_LAST_SPECIAL.to_u32() {
         eprintln!("ignoring careless call to dc_delete_contact()");
         return 0;
     }
     let ctx = &*context;
 
     block_on(async move {
-        match Contact::delete(ctx, contact_id).await {
+        match Contact::delete(ctx, ContactId::new(contact_id)).await {
             Ok(_) => 1,
             Err(_) => 0,
         }
@@ -1917,7 +2020,7 @@ pub unsafe extern "C" fn dc_get_contact(
     let ctx = &*context;
 
     block_on(async move {
-        Contact::get_by_id(ctx, contact_id)
+        Contact::get_by_id(ctx, ContactId::new(contact_id))
             .await
             .map(|contact| Box::into_raw(Box::new(ContactWrapper { context, contact })))
             .unwrap_or_else(|_| ptr::null_mut())
@@ -1929,7 +2032,7 @@ pub unsafe extern "C" fn dc_imex(
     context: *mut dc_context_t,
     what_raw: libc::c_int,
     param1: *const libc::c_char,
-    _param2: *const libc::c_char,
+    param2: *const libc::c_char,
 ) {
     if context.is_null() {
         eprintln!("ignoring careless call to dc_imex()");
@@ -1942,12 +2045,13 @@ pub unsafe extern "C" fn dc_imex(
             return;
         }
     };
+    let passphrase = to_opt_string_lossy(param2);
 
     let ctx = &*context;
 
     if let Some(param1) = to_opt_string_lossy(param1) {
         spawn(async move {
-            imex::imex(ctx, what, param1.as_ref())
+            imex::imex(ctx, what, param1.as_ref(), passphrase)
                 .await
                 .log_err(ctx, "IMEX failed")
         });
@@ -2339,7 +2443,7 @@ pub unsafe extern "C" fn dc_array_get_contact_id(
         return 0;
     }
 
-    (*array).get_location(index).contact_id
+    (*array).get_location(index).contact_id.to_u32()
 }
 #[no_mangle]
 pub unsafe extern "C" fn dc_array_get_msg_id(
@@ -2453,7 +2557,14 @@ pub unsafe extern "C" fn dc_chatlist_get_chat_id(
         return 0;
     }
     let ffi_list = &*chatlist;
-    ffi_list.list.get_chat_id(index as usize).to_u32()
+    let ctx = &*ffi_list.context;
+    match ffi_list.list.get_chat_id(index as usize) {
+        Ok(chat_id) => chat_id.to_u32(),
+        Err(err) => {
+            warn!(ctx, "get_chat_id failed: {}", err);
+            0
+        }
+    }
 }
 
 #[no_mangle]
@@ -2848,7 +2959,7 @@ pub unsafe extern "C" fn dc_msg_get_from_id(msg: *mut dc_msg_t) -> u32 {
         return 0;
     }
     let ffi_msg = &*msg;
-    ffi_msg.message.get_from_id()
+    ffi_msg.message.get_from_id().to_u32()
 }
 
 #[no_mangle]
@@ -2970,6 +3081,61 @@ pub unsafe extern "C" fn dc_msg_get_filename(msg: *mut dc_msg_t) -> *mut libc::c
     ffi_msg.message.get_filename().unwrap_or_default().strdup()
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn dc_msg_get_webxdc_blob(
+    msg: *mut dc_msg_t,
+    filename: *const libc::c_char,
+    ret_bytes: *mut libc::size_t,
+) -> *mut libc::c_char {
+    if msg.is_null() || filename.is_null() || ret_bytes.is_null() {
+        eprintln!("ignoring careless call to dc_msg_get_webxdc_blob()");
+        return ptr::null_mut();
+    }
+    let ffi_msg = &*msg;
+    let ctx = &*ffi_msg.context;
+    let blob = block_on(async move {
+        ffi_msg
+            .message
+            .get_webxdc_blob(ctx, &to_string_lossy(filename))
+            .await
+    });
+    match blob {
+        Ok(blob) => {
+            *ret_bytes = blob.len();
+            let ptr = libc::malloc(*ret_bytes);
+            libc::memcpy(ptr, blob.as_ptr() as *mut libc::c_void, *ret_bytes);
+            ptr as *mut libc::c_char
+        }
+        Err(err) => {
+            eprintln!("failed read blob from archive: {}", err);
+            ptr::null_mut()
+        }
+    }
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn dc_msg_get_webxdc_info(msg: *mut dc_msg_t) -> *mut libc::c_char {
+    if msg.is_null() {
+        eprintln!("ignoring careless call to dc_msg_get_webxdc_info()");
+        return "".strdup();
+    }
+    let ffi_msg = &*msg;
+    let ctx = &*ffi_msg.context;
+
+    block_on(async move {
+        let info = match ffi_msg.message.get_webxdc_info(ctx).await {
+            Ok(info) => info,
+            Err(err) => {
+                error!(ctx, "dc_msg_get_webxdc_info() failed to get info: {}", err);
+                return "".strdup();
+            }
+        };
+        serde_json::to_string(&info)
+            .unwrap_or_log_default(ctx, "dc_msg_get_webxdc_info() failed to serialise to json")
+            .strdup()
+    })
+}
+
 #[no_mangle]
 pub unsafe extern "C" fn dc_msg_get_filemime(msg: *mut dc_msg_t) -> *mut libc::c_char {
     if msg.is_null() {
@@ -3382,17 +3548,21 @@ pub unsafe extern "C" fn dc_msg_set_quote(msg: *mut dc_msg_t, quote: *const dc_m
         return;
     }
     let ffi_msg = &mut *msg;
-    let ffi_quote = &*quote;
-
-    if ffi_msg.context != ffi_quote.context {
-        eprintln!("ignoring attempt to quote message from a different context");
-        return;
-    }
+    let quote_msg = if quote.is_null() {
+        None
+    } else {
+        let ffi_quote = &*quote;
+        if ffi_msg.context != ffi_quote.context {
+            eprintln!("ignoring attempt to quote message from a different context");
+            return;
+        }
+        Some(&ffi_quote.message)
+    };
 
     block_on(async move {
         ffi_msg
             .message
-            .set_quote(&*ffi_msg.context, &ffi_quote.message)
+            .set_quote(&*ffi_msg.context, quote_msg)
             .await
             .log_err(&*ffi_msg.context, "failed to set quote")
             .ok();
@@ -3435,6 +3605,39 @@ pub unsafe extern "C" fn dc_msg_get_quoted_msg(msg: *const dc_msg_t) -> *mut dc_
     }
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn dc_msg_get_parent(msg: *const dc_msg_t) -> *mut dc_msg_t {
+    if msg.is_null() {
+        eprintln!("ignoring careless call to dc_msg_get_parent()");
+        return ptr::null_mut();
+    }
+    let ffi_msg: &MessageWrapper = &*msg;
+    let context = &*ffi_msg.context;
+    let res = block_on(async move {
+        ffi_msg
+            .message
+            .parent(context)
+            .await
+            .log_err(context, "failed to get parent message")
+            .unwrap_or(None)
+    });
+
+    match res {
+        Some(message) => Box::into_raw(Box::new(MessageWrapper { context, message })),
+        None => ptr::null_mut(),
+    }
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn dc_msg_force_plaintext(msg: *mut dc_msg_t) {
+    if msg.is_null() {
+        eprintln!("ignoring careless call to dc_msg_force_plaintext()");
+        return;
+    }
+    let ffi_msg = &mut *msg;
+    ffi_msg.message.force_plaintext();
+}
+
 // dc_contact_t
 
 /// FFI struct for [dc_contact_t]
@@ -3467,7 +3670,7 @@ pub unsafe extern "C" fn dc_contact_get_id(contact: *mut dc_contact_t) -> u32 {
         return 0;
     }
     let ffi_contact = &*contact;
-    ffi_contact.contact.get_id()
+    ffi_contact.contact.get_id().to_u32()
 }
 
 #[no_mangle]
@@ -3753,7 +3956,11 @@ pub unsafe extern "C" fn dc_provider_new_from_email(
 
     match socks5_enabled {
         Ok(socks5_enabled) => {
-            match block_on(provider::get_provider_info(addr.as_str(), socks5_enabled)) {
+            match block_on(provider::get_provider_info(
+                ctx,
+                addr.as_str(),
+                socks5_enabled,
+            )) {
                 Some(provider) => provider,
                 None => ptr::null_mut(),
             }
@@ -3835,7 +4042,7 @@ pub type dc_accounts_t = AccountsWrapper;
 
 #[no_mangle]
 pub unsafe extern "C" fn dc_accounts_new(
-    os_name: *const libc::c_char,
+    _os_name: *const libc::c_char,
     dbfile: *const libc::c_char,
 ) -> *mut dc_accounts_t {
     setup_panic!();
@@ -3845,13 +4052,7 @@ pub unsafe extern "C" fn dc_accounts_new(
         return ptr::null_mut();
     }
 
-    let os_name = if os_name.is_null() {
-        String::from("DcFFI")
-    } else {
-        to_string_lossy(os_name)
-    };
-
-    let accs = block_on(Accounts::new(os_name, as_path(dbfile).to_path_buf().into()));
+    let accs = block_on(Accounts::new(as_path(dbfile).to_path_buf().into()));
 
     match accs {
         Ok(accs) => Box::into_raw(Box::new(AccountsWrapper::new(accs))),
@@ -3956,6 +4157,30 @@ pub unsafe extern "C" fn dc_accounts_add_account(accounts: *mut dc_accounts_t) -
     })
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn dc_accounts_add_closed_account(accounts: *mut dc_accounts_t) -> u32 {
+    if accounts.is_null() {
+        eprintln!("ignoring careless call to dc_accounts_add_closed_account()");
+        return 0;
+    }
+
+    let accounts = &mut *accounts;
+
+    block_on(async move {
+        let mut accounts = accounts.write().await;
+        match accounts.add_closed_account().await {
+            Ok(id) => id,
+            Err(err) => {
+                accounts.emit_event(EventType::Error(format!(
+                    "Failed to add account: {:#}",
+                    err
+                )));
+                0
+            }
+        }
+    })
+}
+
 #[no_mangle]
 pub unsafe extern "C" fn dc_accounts_remove_account(
     accounts: *mut dc_accounts_t,

deltachat-ffi/src/lot.rs

@@ -111,19 +111,19 @@ impl Lot {
         match self {
             Self::Summary(_) => Default::default(),
             Self::Qr(qr) => match qr {
-                Qr::AskVerifyContact { contact_id, .. } => *contact_id,
+                Qr::AskVerifyContact { contact_id, .. } => contact_id.to_u32(),
                 Qr::AskVerifyGroup { .. } => Default::default(),
-                Qr::FprOk { contact_id } => *contact_id,
-                Qr::FprMismatch { contact_id } => contact_id.unwrap_or_default(),
+                Qr::FprOk { contact_id } => contact_id.to_u32(),
+                Qr::FprMismatch { contact_id } => contact_id.unwrap_or_default().to_u32(),
                 Qr::FprWithoutAddr { .. } => Default::default(),
                 Qr::Account { .. } => Default::default(),
                 Qr::WebrtcInstance { .. } => Default::default(),
-                Qr::Addr { contact_id } => *contact_id,
+                Qr::Addr { contact_id } => contact_id.to_u32(),
                 Qr::Url { .. } => Default::default(),
                 Qr::Text { .. } => Default::default(),
-                Qr::WithdrawVerifyContact { contact_id, .. } => *contact_id,
+                Qr::WithdrawVerifyContact { contact_id, .. } => contact_id.to_u32(),
                 Qr::WithdrawVerifyGroup { .. } => Default::default(),
-                Qr::ReviveVerifyContact { contact_id, .. } => *contact_id,
+                Qr::ReviveVerifyContact { contact_id, .. } => contact_id.to_u32(),
                 Qr::ReviveVerifyGroup { .. } => Default::default(),
             },
             Self::Error(_) => Default::default(),

draft/webxdc-dev-reference.md

@@ -0,0 +1,181 @@
+# Webxdc Developer Reference
+
+## Webxdc File Format
+
+- a **Webxdc app** is a **ZIP-file** with the extension `.xdc`
+- the ZIP-file must use the default compression methods as of RFC 1950,
+  this is "Deflate" or "Store"
+- the ZIP-file must contain at least the file `index.html`
+- if the Webxdc app is started, `index.html` is opened in a restricted webview
+  that allow accessing resources only from the ZIP-file
+
+
+## Webxdc API
+
+There are some additional APIs available once `webxdc.js` is included
+(the file will be provided by the concrete implementations,
+no need to add `webxdc.js` to your ZIP-file):
+
+```html
+<script src="webxdc.js"></script>
+```
+
+### sendUpdate()
+
+```js
+window.webxdc.sendUpdate(update, descr);
+```
+
+Webxdc apps are usually shared in a chat and run independently on each peer.
+To get a shared state, the peers use `sendUpdate()` to send updates to each other.
+
+- `update`: an object with the following properties:  
+    - `update.payload`: any javascript primitive, array or object.
+    - `update.info`: optional, short, informational message that will be added to the chat,
+       eg. "Alice voted" or "Bob scored 123 in MyGame";
+       usually only one line of text is shown,
+       use this option sparingly to not spam the chat.
+    - `update.summary`: optional, short text, shown beside app icon;
+       it is recommended to use some aggregated value,  eg. "8 votes", "Highscore: 123"
+
+- `descr`: short, human-readable description what this update is about.
+  this is shown eg. as a fallback text in an email program.
+
+All peers, including the sending one,
+will receive the update by the callback given to `setUpdateListener()`.
+
+There are situations where the user cannot send messages to a chat,
+eg. if the webxdc instance comes as a contact request or if the user has left a group.
+In these cases, you can still call `sendUpdate()`,
+however, the update won't be sent to other peers
+and you won't get the update by `setUpdateListener()`.
+
+
+### setUpdateListener()
+
+```js
+window.webxdc.setUpdateListener((update) => {}, serial);
+```
+
+With `setUpdateListener()` you define a callback that receives the updates
+sent by `sendUpdate()`. The callback is called for updates sent by you or other peers.
+The `serial` specifies the last serial that you know about (defaults to 0). 
+
+Each `update` which is passed to the callback comes with the following properties: 
+
+- `update.payload`: equals the payload given to `sendUpdate()`
+
+- `update.serial`: the serial number of this update.
+  Serials are larger `0` and newer serials have higher numbers.
+  There may be gaps in the serials
+  and it is not guaranteed that the next serial is exactly incremented by one.
+
+- `update.max_serial`: the maximum serial currently known.
+  If `max_serial` equals `serial` this update is the last update (until new network messages arrive).
+
+- `update.info`: optional, short, informational message (see `send_update`) 
+
+- `update.summary`: optional, short text, shown beside app icon (see `send_update`)
+
+
+### selfAddr
+
+```js
+window.webxdc.selfAddr
+```
+
+Property with the peer's own address.
+This is esp. useful if you want to differ between different peers -
+just send the address along with the payload,
+and, if needed, compare the payload addresses against selfAddr() later on.
+
+
+### selfName
+
+```js
+window.webxdc.selfName
+```
+
+Property with the peer's own name.
+This is name chosen by the user in their settings,
+if there is nothing set, that defaults to the peer's address.
+
+
+## manifest.toml
+
+If the ZIP-file contains a `manifest.toml` in its root directory,
+some basic information are read and used from there.
+
+the `manifest.toml` has the following format
+
+```toml
+name = "My App Name"
+```
+
+- **name** - The name of the app.
+  If no name is set or if there is no manifest, the filename is used as the app name.
+
+
+## App Icon
+
+If the ZIP-root contains an `icon.png` or `icon.jpg`,
+these files are used as the icon for the app.
+The icon should be a square at reasonable width/height;
+round corners etc. will be added by the implementations as needed.
+If no icon is set, a default icon will be used.
+
+
+## Webxdc Examples
+
+The following example shows an input field and  every input is show on all peers.
+
+```html
+<!DOCTYPE html>
+<html>
+  <head>
+    <meta charset="utf-8"/>
+    <script src="webxdc.js"></script>
+  </head>
+  <body>
+    <input id="input" type="text"/>
+    <a href="" onclick="sendMsg(); return false;">Send</a>
+    <p id="output"></p>
+    <script>
+    
+      function sendMsg() {
+        msg = document.getElementById("input").value;
+        window.webxdc.sendUpdate({payload: msg}, 'Someone typed "'+msg+'".');
+      }
+    
+      function receiveUpdate(update) {
+        document.getElementById('output').innerHTML += update.payload + "<br>";
+      }
+    
+      window.webxdc.setUpdateListener(receiveUpdate, 0);
+    </script>
+  </body>
+</html>
+```
+
+[Webxdc Development Tool](https://github.com/deltachat/webxdc-dev)
+offers an **Webxdc Simulator** that can be used in many browsers without any installation needed.
+You can also use that repository as a template for your own app -
+just clone and start adapting things to your need.
+
+
+### Advanced Examples
+
+- [2048](https://github.com/adbenitez/2048.xdc)
+- [Draw](https://github.com/adbenitez/draw.xdc)
+- [Poll](https://github.com/r10s/webxdc-poll/)
+- [Tic Tac Toe](https://github.com/Simon-Laux/tictactoe.xdc)
+- Even more with [Topic #webxdc on Github](https://github.com/topics/webxdc)
+
+
+## Closing Remarks
+
+- older devices might not have the newest js features in their webview,
+  you may want to transpile your code down to an older js version eg. with https://babeljs.io
+- there are tons of ideas for enhancements of the API and the file format,
+  eg. in the future, we will may define icon- and manifest-files,
+  allow to aggregate the state or add metadata.

examples/repl/cmdline.rs

@@ -17,7 +17,7 @@ use deltachat::download::DownloadState;
 use deltachat::imex::*;
 use deltachat::location;
 use deltachat::log::LogExt;
-use deltachat::message::{self, Message, MessageState, MsgId};
+use deltachat::message::{self, Message, MessageState, MsgId, Viewtype};
 use deltachat::peerstate::*;
 use deltachat::qr::*;
 use deltachat::sql;
@@ -84,6 +84,7 @@ async fn reset_tables(context: &Context, bits: i32) {
             )
             .await
             .unwrap();
+        context.sql().config_cache().write().await.clear();
         context
             .sql()
             .execute("DELETE FROM leftgrps;", paramsv![])
@@ -101,7 +102,7 @@ async fn reset_tables(context: &Context, bits: i32) {
 async fn poke_eml_file(context: &Context, filename: impl AsRef<Path>) -> Result<()> {
     let data = dc_read_file(context, filename).await?;
 
-    if let Err(err) = dc_receive_imf(context, &data, "import", 0, false).await {
+    if let Err(err) = dc_receive_imf(context, &data, "import", false).await {
         println!("dc_receive_imf errored: {:?}", err);
     }
     Ok(())
@@ -209,7 +210,7 @@ async fn log_msg(context: &Context, prefix: impl AsRef<str>, msg: &Message) {
         contact_id,
         msgtext.unwrap_or_default(),
         if msg.has_html() { "[HAS-HTML]️" } else { "" },
-        if msg.get_from_id() == 1 {
+        if msg.get_from_id() == DC_CONTACT_ID_SELF {
             ""
         } else if msg.get_state() == MessageState::InSeen {
             "[SEEN]"
@@ -267,9 +268,8 @@ async fn log_msglist(context: &Context, msglist: &[MsgId]) -> Result<()> {
     Ok(())
 }
 
-async fn log_contactlist(context: &Context, contacts: &[u32]) -> Result<()> {
+async fn log_contactlist(context: &Context, contacts: &[ContactId]) -> Result<()> {
     for contact_id in contacts {
-        let line;
         let mut line2 = "".to_string();
         let contact = Contact::get_by_id(context, *contact_id).await?;
         let name = contact.get_display_name();
@@ -284,24 +284,20 @@ async fn log_contactlist(context: &Context, contacts: &[u32]) -> Result<()> {
         } else {
             ""
         };
-        line = format!(
+        let line = format!(
             "{}{} <{}>",
             if !name.is_empty() {
-                &name
+                name
             } else {
                 "<name unset>"
             },
             verified_str,
-            if !addr.is_empty() {
-                &addr
-            } else {
-                "addr unset"
-            }
+            if !addr.is_empty() { addr } else { "addr unset" }
         );
-        let peerstate = Peerstate::from_addr(context, &addr)
+        let peerstate = Peerstate::from_addr(context, addr)
             .await
             .expect("peerstate error");
-        if peerstate.is_some() && *contact_id != 1 {
+        if peerstate.is_some() && *contact_id != DC_CONTACT_ID_SELF {
             line2 = format!(
                 ", prefer-encrypt={}",
                 peerstate.as_ref().unwrap().prefer_encrypt
@@ -387,6 +383,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                  sendfile <file> [<text>]\n\
                  sendhtml <file for html-part> [<text for plain-part>]\n\
                  sendsyncmsg\n\
+                 sendupdate <msg-id> <json status update>\n\
                  videochat\n\
                  draft [<text>]\n\
                  devicemsg <text>\n\
@@ -429,7 +426,6 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                  joinqr <qr-content>\n\
                  setqr <qr-content>\n\
                  providerinfo <addr>\n\
-                 event <event-id to test>\n\
                  fileinfo <file>\n\
                  estimatedeletion <seconds>\n\
                  clear -- clear screen\n\
@@ -471,20 +467,32 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
         }
         "export-backup" => {
             let dir = dirs::home_dir().unwrap_or_default();
-            imex(&context, ImexMode::ExportBackup, dir.as_ref()).await?;
+            imex(
+                &context,
+                ImexMode::ExportBackup,
+                dir.as_ref(),
+                Some(arg2.to_string()),
+            )
+            .await?;
             println!("Exported to {}.", dir.to_string_lossy());
         }
         "import-backup" => {
             ensure!(!arg1.is_empty(), "Argument <backup-file> missing.");
-            imex(&context, ImexMode::ImportBackup, arg1.as_ref()).await?;
+            imex(
+                &context,
+                ImexMode::ImportBackup,
+                arg1.as_ref(),
+                Some(arg2.to_string()),
+            )
+            .await?;
         }
         "export-keys" => {
             let dir = dirs::home_dir().unwrap_or_default();
-            imex(&context, ImexMode::ExportSelfKeys, dir.as_ref()).await?;
+            imex(&context, ImexMode::ExportSelfKeys, dir.as_ref(), None).await?;
             println!("Exported to {}.", dir.to_string_lossy());
         }
         "import-keys" => {
-            imex(&context, ImexMode::ImportSelfKeys, arg1.as_ref()).await?;
+            imex(&context, ImexMode::ImportSelfKeys, arg1.as_ref(), None).await?;
         }
         "export-setup" => {
             let setup_code = create_setup_code(&context);
@@ -563,7 +571,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                 );
 
                 for i in (0..cnt).rev() {
-                    let chat = Chat::load_from_db(&context, chatlist.get_chat_id(i)).await?;
+                    let chat = Chat::load_from_db(&context, chatlist.get_chat_id(i)?).await?;
                     println!(
                         "{}#{}: {} [{} fresh] {}{}{}{}",
                         chat_prefix(&chat),
@@ -706,7 +714,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
         }
         "createchat" => {
             ensure!(!arg1.is_empty(), "Argument <contact-id> missing.");
-            let contact_id: u32 = arg1.parse()?;
+            let contact_id = ContactId::new(arg1.parse()?);
             let chat_id = ChatId::create_for_contact(&context, contact_id).await?;
 
             println!("Single#{} created successfully.", chat_id,);
@@ -734,7 +742,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
             ensure!(sel_chat.is_some(), "No chat selected");
             ensure!(!arg1.is_empty(), "Argument <contact-id> missing.");
 
-            let contact_id_0: u32 = arg1.parse()?;
+            let contact_id_0 = ContactId::new(arg1.parse()?);
             chat::add_contact_to_chat(&context, sel_chat.as_ref().unwrap().get_id(), contact_id_0)
                 .await?;
             println!("Contact added to chat.");
@@ -742,7 +750,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
         "removemember" => {
             ensure!(sel_chat.is_some(), "No chat selected.");
             ensure!(!arg1.is_empty(), "Argument <contact-id> missing.");
-            let contact_id_1: u32 = arg1.parse()?;
+            let contact_id_1 = ContactId::new(arg1.parse()?);
             chat::remove_contact_from_chat(
                 &context,
                 sel_chat.as_ref().unwrap().get_id(),
@@ -758,7 +766,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
             chat::set_chat_name(
                 &context,
                 sel_chat.as_ref().unwrap().get_id(),
-                &format!("{} {}", arg1, arg2).trim(),
+                format!("{} {}", arg1, arg2).trim(),
             )
             .await?;
 
@@ -907,6 +915,16 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
             Some(msg_id) => println!("sync message sent as {}.", msg_id),
             None => println!("sync message not needed."),
         },
+        "sendupdate" => {
+            ensure!(
+                !arg1.is_empty() && !arg2.is_empty(),
+                "Arguments <msg-id> <json status update> expected"
+            );
+            let msg_id = MsgId::new(arg1.parse()?);
+            context
+                .send_webxdc_status_update(msg_id, arg2, "this is a webxdc status update")
+                .await?;
+        }
         "videochat" => {
             ensure!(sel_chat.is_some(), "No chat selected.");
             chat::send_videochat_invitation(&context, sel_chat.as_ref().unwrap().get_id()).await?;
@@ -1116,7 +1134,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
         "contactinfo" => {
             ensure!(!arg1.is_empty(), "Argument <contact-id> missing.");
 
-            let contact_id: u32 = arg1.parse()?;
+            let contact_id = ContactId::new(arg1.parse()?);
             let contact = Contact::get_by_id(&context, contact_id).await?;
             let name_n_addr = contact.get_name_n_addr();
 
@@ -1142,7 +1160,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                     if 0 != i {
                         res += ", ";
                     }
-                    let chat = Chat::load_from_db(&context, chatlist.get_chat_id(i)).await?;
+                    let chat = Chat::load_from_db(&context, chatlist.get_chat_id(i)?).await?;
                     res += &format!("{}#{}", chat_prefix(&chat), chat.get_id());
                 }
             }
@@ -1151,16 +1169,16 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
         }
         "delcontact" => {
             ensure!(!arg1.is_empty(), "Argument <contact-id> missing.");
-            Contact::delete(&context, arg1.parse()?).await?;
+            Contact::delete(&context, ContactId::new(arg1.parse()?)).await?;
         }
         "block" => {
             ensure!(!arg1.is_empty(), "Argument <contact-id> missing.");
-            let contact_id = arg1.parse()?;
+            let contact_id = ContactId::new(arg1.parse()?);
             Contact::block(&context, contact_id).await?;
         }
         "unblock" => {
             ensure!(!arg1.is_empty(), "Argument <contact-id> missing.");
-            let contact_id = arg1.parse()?;
+            let contact_id = ContactId::new(arg1.parse()?);
             Contact::unblock(&context, contact_id).await?;
         }
         "listblocked" => {
@@ -1185,7 +1203,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
             let socks5_enabled = context
                 .get_config_bool(config::Config::Socks5Enabled)
                 .await?;
-            match provider::get_provider_info(arg1, socks5_enabled).await {
+            match provider::get_provider_info(&context, arg1, socks5_enabled).await {
                 Some(info) => {
                     println!("Information for provider belonging to {}:", arg1);
                     println!("status: {}", info.status as u32);
@@ -1201,17 +1219,6 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                 }
             }
         }
-        // TODO: implement this again, unclear how to match this through though, without writing a parser.
-        // "event" => {
-        //     ensure!(!arg1.is_empty(), "Argument <id> missing.");
-        //     let event = arg1.parse()?;
-        //     let event = EventType::from_u32(event).ok_or(format_err!("EventType::from_u32({})", event))?;
-        //     let r = context.emit_event(event, 0 as libc::uintptr_t, 0 as libc::uintptr_t);
-        //     println!(
-        //         "Sending event {:?}({}), received value {}.",
-        //         event, event as usize, r,
-        //     );
-        // }
         "fileinfo" => {
             ensure!(!arg1.is_empty(), "Argument <file> missing.");
 

examples/repl/main.rs

@@ -169,7 +169,7 @@ const DB_COMMANDS: [&str; 10] = [
     "housekeeping",
 ];
 
-const CHAT_COMMANDS: [&str; 35] = [
+const CHAT_COMMANDS: [&str; 36] = [
     "listchats",
     "listarchived",
     "chat",
@@ -191,6 +191,7 @@ const CHAT_COMMANDS: [&str; 35] = [
     "sendfile",
     "sendhtml",
     "sendsyncmsg",
+    "sendupdate",
     "videochat",
     "draft",
     "listmedia",
@@ -226,13 +227,12 @@ const CONTACT_COMMANDS: [&str; 9] = [
     "unblock",
     "listblocked",
 ];
-const MISC_COMMANDS: [&str; 12] = [
+const MISC_COMMANDS: [&str; 11] = [
     "getqr",
     "getqrsvg",
     "getbadqr",
     "checkqr",
     "joinqr",
-    "event",
     "fileinfo",
     "clear",
     "exit",
@@ -297,7 +297,7 @@ async fn start(args: Vec<String>) -> Result<(), Error> {
         println!("Error: Bad arguments, expected [db-name].");
         bail!("No db-name specified");
     }
-    let context = Context::new("CLI".into(), Path::new(&args[1]).to_path_buf(), 0).await?;
+    let context = Context::new(Path::new(&args[1]).to_path_buf(), 0).await?;
 
     let events = context.get_event_emitter();
     async_std::task::spawn(async move {
@@ -415,7 +415,7 @@ async fn handle_cmd(
         }
         "getqr" | "getbadqr" => {
             ctx.start_io().await;
-            let group = arg1.parse::<u32>().ok().map(|id| ChatId::new(id));
+            let group = arg1.parse::<u32>().ok().map(ChatId::new);
             let mut qr = dc_get_securejoin_qr(&ctx, group).await?;
             if !qr.is_empty() {
                 if arg0 == "getbadqr" && qr.len() > 40 {
@@ -432,7 +432,7 @@ async fn handle_cmd(
         }
         "getqrsvg" => {
             ctx.start_io().await;
-            let group = arg1.parse::<u32>().ok().map(|id| ChatId::new(id));
+            let group = arg1.parse::<u32>().ok().map(ChatId::new);
             let file = dirs::home_dir().unwrap_or_default().join("qr.svg");
             match get_securejoin_qr_svg(&ctx, group).await {
                 Ok(svg) => {

examples/simple.rs

@@ -36,7 +36,7 @@ async fn main() {
     let dir = tempdir().unwrap();
     let dbfile = dir.path().join("db.sqlite");
     log::info!("creating database {:?}", dbfile);
-    let ctx = Context::new("FakeOs".into(), dbfile.into(), 0)
+    let ctx = Context::new(dbfile.into(), 0)
         .await
         .expect("Failed to create context");
     let info = ctx.get_info().await;

python/src/deltachat/__init__.py

@@ -75,7 +75,6 @@ def run_cmdline(argv=None, account_plugins=None):
         ac.set_config("addr", args.email)
         ac.set_config("mail_pw", args.password)
         ac.set_config("mvbox_move", "0")
-        ac.set_config("mvbox_watch", "0")
         ac.set_config("sentbox_watch", "0")
         ac.set_config("bot", "1")
         configtracker = ac.configure()

python/src/deltachat/account.py

@@ -179,6 +179,12 @@ class Account(object):
         """
         return True if lib.dc_is_configured(self._dc_context) else False
 
+    def is_open(self) -> bool:
+        """Determine if account is open
+
+        :returns True if account is open."""
+        return True if lib.dc_context_is_open(self._dc_context) else False
+
     def set_avatar(self, img_path: Optional[str]) -> None:
         """Set self avatar.
 
@@ -403,7 +409,10 @@ class Account(object):
         """
         arr = array("i")
         for msg in messages:
-            arr.append(getattr(msg, "id", msg))
+            if isinstance(msg, Message):
+                arr.append(msg.id)
+            else:
+                arr.append(msg)
         msg_ids = ffi.cast("uint32_t*", ffi.from_buffer(arr))
         lib.dc_markseen_msgs(self._dc_context, msg_ids, len(messages))
 

python/src/deltachat/direct_imap.py

@@ -224,9 +224,7 @@ class DirectImap:
         """ (blocking) wait for next idle message from server. """
         assert self._idling
         self.account.log("imap-direct: calling idle_check")
-        res = self.conn.idle_check(timeout=30)
-        if len(res) == 0:
-            raise TimeoutError
+        res = self.conn.idle_check()
         if terminate:
             self.idle_done()
         self.account.log("imap-direct: idle_check returned {!r}".format(res))

python/src/deltachat/events.py

@@ -40,7 +40,7 @@ class FFIEventLogger:
 
     @account_hookimpl
     def ac_log_line(self, message):
-        t = threading.currentThread()
+        t = threading.current_thread()
         tname = getattr(t, "name", t)
         if tname == "MainThread":
             tname = "MAIN"
@@ -193,7 +193,7 @@ class EventThread(threading.Thread):
     def __init__(self, account) -> None:
         self.account = account
         super(EventThread, self).__init__(name="events")
-        self.setDaemon(True)
+        self.daemon = True
         self._marked_for_shutdown = False
         self.start()
 

python/src/deltachat/message.py

@@ -225,6 +225,10 @@ class Message(object):
         """Quote setter"""
         lib.dc_msg_set_quote(self._dc_msg, quoted_message._dc_msg)
 
+    def force_plaintext(self) -> None:
+        """Force the message to be sent in plain text."""
+        lib.dc_msg_force_plaintext(self._dc_msg)
+
     def get_mime_headers(self):
         """ return mime-header object for an incoming message.
 

python/src/deltachat/testplugin.py

@@ -241,7 +241,6 @@ def acfactory(pytestconfig, tmpdir, request, session_liveconfig, data):
         def make_account(self, path, logid, quiet=False):
             ac = Account(path, logging=self._logging)
             ac._evtracker = ac.add_account_plugin(FFIEventTracker(ac))
-            ac._evtracker.set_timeout(30)
             ac.addr = ac.get_self_contact().addr
             ac.set_config("displayname", logid)
             if not quiet:
@@ -303,21 +302,20 @@ def acfactory(pytestconfig, tmpdir, request, session_liveconfig, data):
                 self._preconfigure_key(ac, configdict['addr'])
             return ac, dict(configdict)
 
-        def get_online_configuring_account(self, mvbox=False, sentbox=False, move=False,
+        def get_online_configuring_account(self, sentbox=False, move=False,
                                            pre_generated_key=True, quiet=False, config={}):
             ac, configdict = self.get_online_config(
                 pre_generated_key=pre_generated_key, quiet=quiet)
             configdict.update(config)
-            configdict["mvbox_watch"] = str(int(mvbox))
             configdict["mvbox_move"] = str(int(move))
             configdict["sentbox_watch"] = str(int(sentbox))
             ac.update_config(configdict)
             ac._configtracker = ac.configure()
             return ac
 
-        def get_one_online_account(self, pre_generated_key=True, mvbox=False, move=False):
+        def get_one_online_account(self, pre_generated_key=True, move=False):
             ac1 = self.get_online_configuring_account(
-                pre_generated_key=pre_generated_key, mvbox=mvbox, move=move)
+                pre_generated_key=pre_generated_key, move=move)
             self.wait_configure_and_start_io([ac1])
             return ac1
 
@@ -336,7 +334,7 @@ def acfactory(pytestconfig, tmpdir, request, session_liveconfig, data):
             return accounts
 
         def clone_online_account(self, account, pre_generated_key=True):
-            """ Clones addr, mail_pw, mvbox_watch, mvbox_move, sentbox_watch and the
+            """ Clones addr, mail_pw, mvbox_move, sentbox_watch and the
             direct_imap object of an online account. This simulates the user setting
             up a new device without importing a backup.
 
@@ -351,7 +349,6 @@ def acfactory(pytestconfig, tmpdir, request, session_liveconfig, data):
             ac.update_config(dict(
                 addr=account.get_config("addr"),
                 mail_pw=account.get_config("mail_pw"),
-                mvbox_watch=account.get_config("mvbox_watch"),
                 mvbox_move=account.get_config("mvbox_move"),
                 sentbox_watch=account.get_config("sentbox_watch"),
             ))
@@ -467,7 +464,7 @@ class BotProcess:
         # the (unicode) lines available for readers through a queue.
         self.stdout_queue = queue.Queue()
         self.stdout_thread = t = threading.Thread(target=self._run_stdout_thread, name="bot-stdout-thread")
-        t.setDaemon(True)
+        t.daemon = True
         t.start()
 
     def _run_stdout_thread(self) -> None:
@@ -485,7 +482,7 @@ class BotProcess:
     def kill(self) -> None:
         self.popen.kill()
 
-    def wait(self, timeout=30) -> None:
+    def wait(self, timeout=None) -> None:
         self.popen.wait(timeout=timeout)
 
     def fnmatch_lines(self, pattern_lines):
@@ -494,7 +491,7 @@ class BotProcess:
             print("+++FNMATCH:", next_pattern)
             ignored = []
             while 1:
-                line = self.stdout_queue.get(timeout=15)
+                line = self.stdout_queue.get()
                 if line is None:
                     if ignored:
                         print("BOT stdout terminated after these lines")

python/src/deltachat/tracker.py

@@ -90,11 +90,11 @@ class ConfigureTracker:
             if data1 is None or evdata == data1:
                 break
 
-    def wait_finish(self):
+    def wait_finish(self, timeout=None):
         """ wait until configure is completed.
 
         Raise Exception if Configure failed
         """
-        if not self._configure_events.get():
+        if not self._configure_events.get(timeout=timeout):
             content = "\n".join(map(str, self._ffi_events))
             raise ConfigureFailed(content)

python/tests/test_account.py

@@ -41,8 +41,8 @@ class TestOfflineAccountBasic:
     def test_wrong_db(self, tmpdir):
         p = tmpdir.join("hello.db")
         p.write("123")
-        with pytest.raises(ValueError):
-            Account(p.strpath)
+        account = Account(p.strpath)
+        assert not account.is_open()
 
     def test_os_name(self, tmpdir):
         p = tmpdir.join("hello.db")
@@ -57,7 +57,7 @@ class TestOfflineAccountBasic:
         alice_public = data.read_path("key/alice-public.asc")
         alice_secret = data.read_path("key/alice-secret.asc")
         assert alice_public and alice_secret
-        ac._preconfigure_keypair("alice@example.com", alice_public, alice_secret)
+        ac._preconfigure_keypair("alice@example.org", alice_public, alice_secret)
 
     def test_getinfo(self, acfactory):
         ac1 = acfactory.get_unconfigured_account()
@@ -652,8 +652,6 @@ class TestOnlineAccount:
             pre_generated_key=False,
             config={"key_gen_type": str(const.DC_KEY_GEN_ED25519)}
         )
-        # rsa key gen can be slow especially on CI, adjust timeout
-        ac1._evtracker.set_timeout(240)
         acfactory.wait_configure_and_start_io()
         chat = acfactory.get_accepted_chat(ac1, ac2)
 
@@ -737,7 +735,6 @@ class TestOnlineAccount:
         # make sure we are not sending message to ourselves
         assert self_addr not in ev.data2
         assert other_addr in ev.data2
-        ev = ac1._evtracker.get_matching("DC_EVENT_DELETED_BLOB_FILE")
 
         lp.sec("ac1: setting bcc_self=1")
         ac1.set_config("bcc_self", "1")
@@ -753,7 +750,6 @@ class TestOnlineAccount:
         # now make sure we are sending message to ourselves too
         assert self_addr in ev.data2
         assert other_addr in ev.data2
-        ev = ac1._evtracker.get_matching("DC_EVENT_DELETED_BLOB_FILE")
         assert ac1.direct_imap.idle_wait_for_seen()
 
         # Second client receives only second message, but not the first
@@ -860,7 +856,7 @@ class TestOnlineAccount:
 
     def test_mvbox_sentbox_threads(self, acfactory, lp):
         lp.sec("ac1: start with mvbox thread")
-        ac1 = acfactory.get_online_configuring_account(mvbox=True, move=True, sentbox=True)
+        ac1 = acfactory.get_online_configuring_account(move=True, sentbox=True)
 
         lp.sec("ac2: start without mvbox/sentbox threads")
         ac2 = acfactory.get_online_configuring_account()
@@ -874,27 +870,31 @@ class TestOnlineAccount:
 
     def test_move_works(self, acfactory):
         ac1 = acfactory.get_online_configuring_account()
-        ac2 = acfactory.get_online_configuring_account(mvbox=True, move=True)
+        ac2 = acfactory.get_online_configuring_account(move=True)
         acfactory.wait_configure_and_start_io()
         chat = acfactory.get_accepted_chat(ac1, ac2)
         chat.send_text("message1")
-        ev = ac2._evtracker.get_matching("DC_EVENT_INCOMING_MSG")
-        assert ev.data2 > const.DC_CHAT_ID_LAST_SPECIAL
+
+        # Message is moved to the movebox
         ac2._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
 
+        # Message is downloaded
+        ev = ac2._evtracker.get_matching("DC_EVENT_INCOMING_MSG")
+        assert ev.data2 > const.DC_CHAT_ID_LAST_SPECIAL
+
     def test_move_works_on_self_sent(self, acfactory):
-        ac1 = acfactory.get_online_configuring_account(mvbox=True, move=True)
+        ac1 = acfactory.get_online_configuring_account(move=True)
         ac2 = acfactory.get_online_configuring_account()
         acfactory.wait_configure_and_start_io()
         ac1.set_config("bcc_self", "1")
 
         chat = acfactory.get_accepted_chat(ac1, ac2)
         chat.send_text("message1")
+        ac1._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
         chat.send_text("message2")
+        ac1._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
         chat.send_text("message3")
         ac1._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
-        ac1._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
-        ac1._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
 
     def test_forward_messages(self, acfactory, lp):
         ac1, ac2 = acfactory.get_two_online_accounts()
@@ -953,7 +953,7 @@ class TestOnlineAccount:
         assert msg_in.is_forwarded()
 
     def test_send_self_message(self, acfactory, lp):
-        ac1 = acfactory.get_one_online_account(mvbox=True, move=True)
+        ac1 = acfactory.get_one_online_account(move=True)
         lp.sec("ac1: create self chat")
         chat = ac1.get_self_contact().create_chat()
         chat.send_text("hello")
@@ -983,7 +983,8 @@ class TestOnlineAccount:
         assert msg2 in chat2.get_messages()
         assert chat2.is_contact_request()
         assert chat2.count_fresh_messages() == 1
-        assert msg2.time_received >= msg1.time_sent
+        # Like it or not, this assert is flaky
+        # assert msg2.time_received >= msg1.time_sent
 
         lp.sec("create new chat with contact and verify it's proper")
         chat2b = msg2.create_chat()
@@ -1034,31 +1035,94 @@ class TestOnlineAccount:
 
     def test_moved_markseen(self, acfactory, lp):
         """Test that message already moved to DeltaChat folder is marked as seen."""
-        ac1 = acfactory.get_online_configuring_account(mvbox=True, config={"inbox_watch": "0"})
-        ac2 = acfactory.get_online_configuring_account()
+        ac1 = acfactory.get_online_configuring_account()
+        ac2 = acfactory.get_online_configuring_account(move=True)
         acfactory.wait_configure_and_start_io([ac1, ac2])
-        ac1.set_config("bcc_self", "1")
 
-        ac1.direct_imap.idle_start()
+        ac2.stop_io()
+        ac2.direct_imap.idle_start()
+
         ac1.create_chat(ac2).send_text("Hello!")
-        ac1.direct_imap.idle_check(terminate=True)
-        ac1.stop_io()
+
+        # Wait for the message to arrive.
+        ac2.direct_imap.idle_check(terminate=True)
 
         # Emulate moving of the message to DeltaChat folder by Sieve rule.
         # mailcow server contains this rule by default.
-        ac1.direct_imap.conn.move(["*"], "DeltaChat")
+        ac2.direct_imap.conn.move(["*"], "DeltaChat")
 
-        ac1.direct_imap.select_folder("DeltaChat")
-        ac1.direct_imap.idle_start()
-        ac1.start_io()
-        ac1.direct_imap.idle_wait_for_seen()
-        ac1.direct_imap.idle_done()
+        ac2.direct_imap.select_folder("DeltaChat")
+        ac2.direct_imap.idle_start()
+        ac2.start_io()
+        msg = ac2._evtracker.wait_next_incoming_message()
 
-        fetch = list(ac1.direct_imap.conn.fetch("*", b'FLAGS').values())
+        # Accept the contact request.
+        msg.chat.accept()
+        ac2.mark_seen_messages([msg])
+        ac2.direct_imap.idle_wait_for_seen()
+        ac2.direct_imap.idle_done()
+
+        fetch = list(ac2.direct_imap.conn.fetch("*", b'FLAGS').values())
         flags = fetch[-1][b'FLAGS']
         is_seen = b'\\Seen' in flags
         assert is_seen
 
+    def test_multidevice_sync_seen(self, acfactory, lp):
+        """Test that message marked as seen on one device is marked as seen on another."""
+        ac1 = acfactory.get_online_configuring_account()
+        ac2 = acfactory.get_online_configuring_account()
+        ac1_clone = acfactory.clone_online_account(ac1)
+        acfactory.wait_configure_and_start_io()
+
+        ac1.set_config("bcc_self", "1")
+        ac1_clone.set_config("bcc_self", "1")
+
+        ac1_chat = ac1.create_chat(ac2)
+        ac1_clone_chat = ac1_clone.create_chat(ac2)
+        ac2_chat = ac2.create_chat(ac1)
+
+        lp.sec("Send a message from ac2 to ac1 and check that it's 'fresh'")
+        ac2_chat.send_text("Hi")
+        ac1_message = ac1._evtracker.wait_next_incoming_message()
+        ac1_clone_message = ac1_clone._evtracker.wait_next_incoming_message()
+        assert ac1_chat.count_fresh_messages() == 1
+        assert ac1_clone_chat.count_fresh_messages() == 1
+        assert ac1_message.is_in_fresh
+        assert ac1_clone_message.is_in_fresh
+
+        lp.sec("ac1 marks message as seen on the first device")
+        ac1.mark_seen_messages([ac1_message])
+        assert ac1_message.is_in_seen
+
+        lp.sec("ac1 clone detects that message is marked as seen")
+        ev = ac1_clone._evtracker.get_matching("DC_EVENT_MSGS_NOTICED")
+        assert ev.data1 == ac1_clone_chat.id
+        assert ac1_clone_message.is_in_seen
+
+        lp.sec("Send an ephemeral message from ac2 to ac1")
+        ac2_chat.set_ephemeral_timer(60)
+        ac1._evtracker.get_matching("DC_EVENT_CHAT_EPHEMERAL_TIMER_MODIFIED")
+        ac1._evtracker.wait_next_incoming_message()
+        ac1_clone._evtracker.get_matching("DC_EVENT_CHAT_EPHEMERAL_TIMER_MODIFIED")
+        ac1_clone._evtracker.wait_next_incoming_message()
+
+        ac2_chat.send_text("Foobar")
+        ac1_message = ac1._evtracker.wait_next_incoming_message()
+        ac1_clone_message = ac1_clone._evtracker.wait_next_incoming_message()
+        assert "Ephemeral timer: 60\n" in ac1_message.get_message_info()
+        assert "Expires: " not in ac1_clone_message.get_message_info()
+        assert "Ephemeral timer: 60\n" in ac1_message.get_message_info()
+        assert "Expires: " not in ac1_clone_message.get_message_info()
+
+        ac1.mark_seen_messages([ac1_message])
+        assert ac1_message.is_in_seen
+        assert "Expires: " in ac1_message.get_message_info()
+        ev = ac1_clone._evtracker.get_matching("DC_EVENT_MSGS_NOTICED")
+        assert ev.data1 == ac1_clone_chat.id
+        assert ac1_clone_message.is_in_seen
+        # Test that the timer is started on the second device after synchronizing the seen status.
+        assert "Expires: " in ac1_clone_message.get_message_info()
+
     def test_message_override_sender_name(self, acfactory, lp):
         ac1, ac2 = acfactory.get_two_online_accounts()
         chat = acfactory.get_accepted_chat(ac1, ac2)
@@ -1095,8 +1159,8 @@ class TestOnlineAccount:
     def test_markseen_message_and_mdn(self, acfactory, mvbox_move):
         # Please only change this test if you are very sure that it will still catch the issues it catches now.
         # We had so many problems with markseen, if in doubt, rather create another test, it can't harm.
-        ac1 = acfactory.get_online_configuring_account(move=mvbox_move, mvbox=mvbox_move)
-        ac2 = acfactory.get_online_configuring_account(move=mvbox_move, mvbox=mvbox_move)
+        ac1 = acfactory.get_online_configuring_account(move=mvbox_move)
+        ac2 = acfactory.get_online_configuring_account(move=mvbox_move)
 
         acfactory.wait_configure_and_start_io()
         # Do not send BCC to self, we only want to test MDN on ac1.
@@ -1142,7 +1206,7 @@ class TestOnlineAccount:
         assert not msg_reply1.chat.is_group()
         assert msg_reply1.chat.id == private_chat1.id
 
-    def test_mdn_asymetric(self, acfactory, lp):
+    def test_mdn_asymmetric(self, acfactory, lp):
         ac1, ac2 = acfactory.get_two_online_accounts(move=True)
 
         lp.sec("ac1: create chat with ac2")
@@ -1166,6 +1230,9 @@ class TestOnlineAccount:
 
         assert len(msg.chat.get_messages()) == 1
 
+        ac1.direct_imap.select_config_folder("mvbox")
+        ac1.direct_imap.idle_start()
+
         lp.sec("ac2: mark incoming message as seen")
         ac2.mark_seen_messages([msg])
 
@@ -1175,6 +1242,9 @@ class TestOnlineAccount:
 
         assert len(chat.get_messages()) == 1
 
+        # Wait for the message to be marked as seen on IMAP.
+        assert ac1.direct_imap.idle_wait_for_seen()
+
         # MDN is received even though MDNs are already disabled
         assert msg_out.is_out_mdn_received()
 
@@ -1352,18 +1422,21 @@ class TestOnlineAccount:
         assert not device_chat.can_send()
         assert device_chat.get_draft() is None
 
-    def test_dont_show_emails_in_draft_folder(self, acfactory, lp):
+    def test_dont_show_emails(self, acfactory, lp):
         """Most mailboxes have a "Drafts" folder where constantly new emails appear but we don't actually want to show them.
         So: If it's outgoing AND there is no Received header AND it's not in the sentbox, then ignore the email.
 
-        If the draft email is sent out later (i.e. moved to "Sent"), it must be shown."""
+        If the draft email is sent out later (i.e. moved to "Sent"), it must be shown.
+
+        Also, test that unknown emails in the Spam folder are not shown."""
         ac1 = acfactory.get_online_configuring_account()
         ac1.set_config("show_emails", "2")
-        ac1.create_contact("alice@example.com").create_chat()
+        ac1.create_contact("alice@example.org").create_chat()
 
         acfactory.wait_configure(ac1)
         ac1.direct_imap.create_folder("Drafts")
         ac1.direct_imap.create_folder("Sent")
+        ac1.direct_imap.create_folder("Spam")
 
         acfactory.wait_configure_and_start_io()
         # Wait until each folder was selected once and we are IDLEing again:
@@ -1373,7 +1446,7 @@ class TestOnlineAccount:
         ac1.direct_imap.append("Drafts", """
             From: ac1 <{}>
             Subject: subj
-            To: alice@example.com
+            To: alice@example.org
             Message-ID: <aepiors@example.org>
             Content-Type: text/plain; charset=utf-8
 
@@ -1382,12 +1455,21 @@ class TestOnlineAccount:
         ac1.direct_imap.append("Sent", """
             From: ac1 <{}>
             Subject: subj
-            To: alice@example.com
+            To: alice@example.org
             Message-ID: <hsabaeni@example.org>
             Content-Type: text/plain; charset=utf-8
 
             message in Sent
         """.format(ac1.get_config("configured_addr")))
+        ac1.direct_imap.append("Spam", """
+            From: unknown.address@junk.org
+            Subject: subj
+            To: {}
+            Message-ID: <spam.message@junk.org>
+            Content-Type: text/plain; charset=utf-8
+
+            Unknown message in Spam
+        """.format(ac1.get_config("configured_addr")))
 
         ac1.set_config("scan_all_folders_debounce_secs", "0")
         lp.sec("All prepared, now let DC find the message")
@@ -1401,6 +1483,10 @@ class TestOnlineAccount:
         assert msg.text == "subj – message in Sent"
         assert len(msg.chat.get_messages()) == 1
 
+        assert not any("unknown.address" in c.get_name() for c in ac1.get_chats())
+        ac1.direct_imap.select_folder("Spam")
+        assert ac1.direct_imap.get_uid_by_message_id("spam.message@junk.org")
+
         ac1.stop_io()
         lp.sec("'Send out' the draft, i.e. move it to the Sent folder, and wait for DC to display it this time")
         ac1.direct_imap.select_folder("Drafts")
@@ -1413,6 +1499,35 @@ class TestOnlineAccount:
         assert msg2.text == "subj – message in Drafts that is moved to Sent later"
         assert len(msg.chat.get_messages()) == 2
 
+    def test_no_old_msg_is_fresh(self, acfactory, lp):
+        ac1 = acfactory.get_online_configuring_account()
+        ac2 = acfactory.get_online_configuring_account()
+        ac1_clone = acfactory.clone_online_account(ac1)
+        acfactory.wait_configure_and_start_io()
+
+        ac1.set_config("e2ee_enabled", "0")
+        ac1_clone.set_config("e2ee_enabled", "0")
+        ac2.set_config("e2ee_enabled", "0")
+
+        ac1_clone.set_config("bcc_self", "1")
+
+        ac1.create_chat(ac2)
+        ac1_clone.create_chat(ac2)
+
+        lp.sec("Send a first message from ac2 to ac1 and check that it's 'fresh'")
+        first_msg_id = ac2.create_chat(ac1).send_text("Hi")
+        ac1._evtracker.wait_next_incoming_message()
+        assert ac1.create_chat(ac2).count_fresh_messages() == 1
+        assert len(list(ac1.get_fresh_messages())) == 1
+
+        lp.sec("Send a message from ac1_clone to ac2 and check that ac1 marks the first message as 'noticed'")
+        ac1_clone.create_chat(ac2).send_text("Hi back")
+        ev = ac1._evtracker.get_matching("DC_EVENT_MSGS_NOTICED")
+
+        assert ev.data1 == first_msg_id.chat.id
+        assert ac1.create_chat(ac2).count_fresh_messages() == 0
+        assert len(list(ac1.get_fresh_messages())) == 0
+
     def test_prefer_encrypt(self, acfactory, lp):
         """Test quorum rule for encryption preference in 1:1 and group chat."""
         ac1, ac2, ac3 = acfactory.get_many_online_accounts(3)
@@ -1757,7 +1872,6 @@ class TestOnlineAccount:
         lp.sec("trigger ac setup message and return setupcode")
         assert ac1.get_info()["fingerprint"] != ac2.get_info()["fingerprint"]
         setup_code = ac1.initiate_key_transfer()
-        ac2._evtracker.set_timeout(30)
         ev = ac2._evtracker.get_matching("DC_EVENT_INCOMING_MSG|DC_EVENT_MSGS_CHANGED")
         msg = ac2.get_message_by_id(ev.data2)
         assert msg.is_setup_message()
@@ -1774,7 +1888,6 @@ class TestOnlineAccount:
     def test_ac_setup_message_twice(self, acfactory, lp):
         ac1 = acfactory.get_online_configuring_account()
         ac2 = acfactory.clone_online_account(ac1)
-        ac2._evtracker.set_timeout(30)
         acfactory.wait_configure_and_start_io()
 
         lp.sec("trigger ac setup message but ignore")
@@ -1948,7 +2061,7 @@ class TestOnlineAccount:
 
         lp.sec("ac1: send a message to group chat to promote the group")
         chat.send_text("afterwards promoted")
-        ev = in_list.get(timeout=10)
+        ev = in_list.get()
         assert ev.action == "chat-modified"
         assert chat.is_promoted()
         assert sorted(x.addr for x in chat.get_contacts()) == \
@@ -1958,29 +2071,29 @@ class TestOnlineAccount:
         # note that if the above create_chat() would not
         # happen we would not receive a proper member_added event
         contact2 = chat.add_contact("devnull@testrun.org")
-        ev = in_list.get(timeout=10)
+        ev = in_list.get()
         assert ev.action == "chat-modified"
-        ev = in_list.get(timeout=10)
+        ev = in_list.get()
         assert ev.action == "chat-modified"
-        ev = in_list.get(timeout=10)
+        ev = in_list.get()
         assert ev.action == "added"
         assert ev.message.get_sender_contact().addr == ac1_addr
         assert ev.contact.addr == "devnull@testrun.org"
 
         lp.sec("ac1: remove address2")
         chat.remove_contact(contact2)
-        ev = in_list.get(timeout=10)
+        ev = in_list.get()
         assert ev.action == "chat-modified"
-        ev = in_list.get(timeout=10)
+        ev = in_list.get()
         assert ev.action == "removed"
         assert ev.contact.addr == contact2.addr
         assert ev.message.get_sender_contact().addr == ac1_addr
 
         lp.sec("ac1: remove ac2 contact from chat")
         chat.remove_contact(ac2)
-        ev = in_list.get(timeout=10)
+        ev = in_list.get()
         assert ev.action == "chat-modified"
-        ev = in_list.get(timeout=10)
+        ev = in_list.get()
         assert ev.action == "removed"
         assert ev.message.get_sender_contact().addr == ac1_addr
 
@@ -2077,10 +2190,8 @@ class TestOnlineAccount:
         assert msg_back.chat == chat
         assert chat.get_profile_image() is None
 
-    @pytest.mark.parametrize("inbox_watch", ["0", "1"])
-    def test_connectivity(self, acfactory, lp, inbox_watch):
+    def test_connectivity(self, acfactory, lp):
         ac1, ac2 = acfactory.get_two_online_accounts()
-        ac1.set_config("inbox_watch", inbox_watch)
         ac1.set_config("scan_all_folders_debounce_secs", "0")
 
         ac1._evtracker.wait_for_connectivity(const.DC_CONNECTIVITY_CONNECTED)
@@ -2320,7 +2431,7 @@ class TestOnlineAccount:
 
     def test_immediate_autodelete(self, acfactory, lp):
         ac1 = acfactory.get_online_configuring_account()
-        ac2 = acfactory.get_online_configuring_account(mvbox=False, move=False, sentbox=False)
+        ac2 = acfactory.get_online_configuring_account(move=False, sentbox=False)
 
         # "1" means delete immediately, while "0" means do not delete
         ac2.set_config("delete_server_after", "1")
@@ -2429,8 +2540,7 @@ class TestOnlineAccount:
         lp.sec("ac2: deleting all messages except third")
         assert len(to_delete) == len(texts) - 1
         ac2.delete_messages(to_delete)
-        for msg in to_delete:
-            ac2._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_DELETED")
+        ac2._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_DELETED")
 
         ac2._evtracker.get_info_contains("close/expunge succeeded")
 
@@ -2581,38 +2691,39 @@ class TestOnlineAccount:
         assert received_reply.quoted_text == "hello"
         assert received_reply.quote.id == out_msg.id
 
-    @pytest.mark.parametrize("folder,move,expected_destination,inbox_watch,", [
-        ("xyz", False, "xyz", "1"),  # Test that emails are recognized in a random folder but not moved
-        ("xyz", True, "DeltaChat", "1"),  # ...emails are found in a random folder and moved to DeltaChat
-        ("Spam", False, "INBOX", "1"),  # ...emails are moved from the spam folder to the Inbox
-        ("INBOX", False, "INBOX", "0"),  # ...emails are found in the `Inbox` folder even if `inbox_watch` is "0"
+    @pytest.mark.parametrize("folder,move,expected_destination,", [
+        ("xyz", False, "xyz"),  # Test that emails are recognized in a random folder but not moved
+        ("xyz", True, "DeltaChat"),  # ...emails are found in a random folder and moved to DeltaChat
+        ("Spam", False, "INBOX"),  # ...emails are moved from the spam folder to the Inbox
     ])
     # Testrun.org does not support the CREATE-SPECIAL-USE capability, which means that we can't create a folder with
     # the "\Junk" flag (see https://tools.ietf.org/html/rfc6154). So, we can't test spam folder detection by flag.
-    def test_scan_folders(self, acfactory, lp, folder, move, expected_destination, inbox_watch):
+    def test_scan_folders(self, acfactory, lp, folder, move, expected_destination):
         """Delta Chat periodically scans all folders for new messages to make sure we don't miss any."""
         variant = folder + "-" + str(move) + "-" + expected_destination
         lp.sec("Testing variant " + variant)
         ac1 = acfactory.get_online_configuring_account(move=move)
         ac2 = acfactory.get_online_configuring_account()
-        ac1.set_config("inbox_watch", inbox_watch)
 
         acfactory.wait_configure(ac1)
         ac1.direct_imap.create_folder(folder)
 
         acfactory.wait_configure_and_start_io()
         # Wait until each folder was selected once and we are IDLEing:
-        if inbox_watch == "1":
-            ac1._evtracker.get_info_contains("INBOX: Idle entering wait-on-remote state")
-        else:
-            ac1._evtracker.get_info_contains("IMAP-fake-IDLE: no folder, waiting for interrupt")
+        ac1._evtracker.get_info_contains("INBOX: Idle entering wait-on-remote state")
         ac1.stop_io()
 
         # Send a message to ac1 and move it to the mvbox:
         ac1.direct_imap.select_config_folder("inbox")
         ac1.direct_imap.idle_start()
         acfactory.get_accepted_chat(ac2, ac1).send_text("hello")
-        ac1.direct_imap.idle_check(terminate=True)
+        while True:
+            if len(ac1.direct_imap.idle_check(terminate=True)) > 1:
+                # If length is 1, it's [(b'OK', b'Still here')]
+                # Could happen on very slow network.
+                #
+                # More is usually [(1, b'EXISTS'), (1, b'RECENT')]
+                break
         ac1.direct_imap.conn.move(["*"], folder)  # "*" means "biggest UID in mailbox"
 
         lp.sec("Everything prepared, now see if DeltaChat finds the message (" + variant + ")")
@@ -2621,11 +2732,7 @@ class TestOnlineAccount:
         msg = ac1._evtracker.wait_next_incoming_message()
         assert msg.text == "hello"
 
-        # Wait until the message was moved (if at all) and we are IDLEing again:
-        if inbox_watch == "1":
-            ac1._evtracker.get_info_contains("INBOX: Idle entering wait-on-remote state")
-        else:
-            ac1._evtracker.get_info_contains("IMAP-fake-IDLE: no folder, waiting for interrupt")
+        # The message has been downloaded, which means it has reached its destination.
         ac1.direct_imap.select_folder(expected_destination)
         assert len(ac1.direct_imap.get_all_messages()) == 1
         if folder != expected_destination:
@@ -2648,8 +2755,7 @@ class TestOnlineAccount:
             if mvbox_move:
                 assert ac.get_config("configured_mvbox_folder")
 
-        ac1 = acfactory.get_online_configuring_account(mvbox=mvbox_move, move=mvbox_move)
-        ac1.set_config("sentbox_move", "1")
+        ac1 = acfactory.get_online_configuring_account(move=mvbox_move)
         ac2 = acfactory.get_online_configuring_account()
 
         acfactory.wait_configure(ac1)
@@ -2669,7 +2775,7 @@ class TestOnlineAccount:
         if mvbox_move:
             ac1.direct_imap.select_config_folder("mvbox")
         else:
-            ac1.direct_imap.select_config_folder("sentbox")
+            ac1.direct_imap.select_folder("INBOX")
         ac1.direct_imap.idle_start()
 
         lp.sec("send out message with bcc to ourselves")
@@ -2748,7 +2854,7 @@ class TestOnlineAccount:
 
     def test_delete_deltachat_folder(self, acfactory):
         """Test that DeltaChat folder is recreated if user deletes it manually."""
-        ac1 = acfactory.get_online_configuring_account(mvbox=True)
+        ac1 = acfactory.get_online_configuring_account(move=True)
         ac2 = acfactory.get_online_configuring_account()
         acfactory.wait_configure(ac1)
 
@@ -2876,7 +2982,8 @@ class TestGroupStressTests:
         lp.sec("ac2: check that ac3 is removed")
         msg = ac2._evtracker.wait_next_incoming_message()
 
-        assert msg.chat.num_contacts() == chat.num_contacts()
+        assert chat.num_contacts() == 2
+        assert msg.chat.num_contacts() == 2
         acfactory.dump_imap_summary(sys.stdout)
 
 

python/tests/test_lowlevel.py

@@ -36,7 +36,8 @@ def test_wrong_db(tmpdir):
     # write an invalid database file
     p.write("x123" * 10)
 
-    assert ffi.NULL == lib.dc_context_new(ffi.NULL, p.strpath.encode("ascii"), ffi.NULL)
+    context = lib.dc_context_new(ffi.NULL, p.strpath.encode("ascii"), ffi.NULL)
+    assert not lib.dc_context_is_open(context)
 
 
 def test_empty_blobdir(tmpdir):

python/tox.ini

@@ -78,8 +78,8 @@ commands =
 addopts = -v -ra --strict-markers
 norecursedirs = .tox 
 xfail_strict=true
-timeout = 90
-timeout_method = thread
+timeout = 150
+timeout_func_only = True
 markers = 
     ignored: ignore this test in default test runs, use --ignored to run.
 

rust-toolchain

@@ -1 +1 @@
-1.54.0
+1.59.0

scripts/docker-coredeps-arm64/deps/build_rust.sh

@@ -8,7 +8,7 @@ set -e -x
 #
 # Avoid using rustup here as it depends on reading /proc/self/exe and
 # has problems running under QEMU.
-RUST_VERSION=1.54.0
+RUST_VERSION=1.59.0
 
 curl "https://static.rust-lang.org/dist/rust-${RUST_VERSION}-$(uname -m)-unknown-linux-gnu.tar.gz" | tar xz
 cd "rust-${RUST_VERSION}-$(uname -m)-unknown-linux-gnu"

scripts/docker-coredeps/deps/build_rust.sh

@@ -8,7 +8,7 @@ set -e -x
 #
 # Avoid using rustup here as it depends on reading /proc/self/exe and
 # has problems running under QEMU.
-RUST_VERSION=1.54.0
+RUST_VERSION=1.59.0
 
 curl "https://static.rust-lang.org/dist/rust-${RUST_VERSION}-$(uname -m)-unknown-linux-gnu.tar.gz" | tar xz
 cd "rust-${RUST_VERSION}-$(uname -m)-unknown-linux-gnu"

src/accounts.rs

@@ -29,21 +29,21 @@ pub struct Accounts {
 
 impl Accounts {
     /// Loads or creates an accounts folder at the given `dir`.
-    pub async fn new(os_name: String, dir: PathBuf) -> Result<Self> {
+    pub async fn new(dir: PathBuf) -> Result<Self> {
         if !dir.exists().await {
-            Accounts::create(os_name, &dir).await?;
+            Accounts::create(&dir).await?;
         }
 
         Accounts::open(dir).await
     }
 
     /// Creates a new default structure.
-    pub async fn create(os_name: String, dir: &PathBuf) -> Result<()> {
+    pub async fn create(dir: &PathBuf) -> Result<()> {
         fs::create_dir_all(dir)
             .await
             .context("failed to create folder")?;
 
-        Config::new(os_name.clone(), dir).await?;
+        Config::new(dir).await?;
 
         Ok(())
     }
@@ -56,8 +56,13 @@ impl Accounts {
         let config_file = dir.join(CONFIG_NAME);
         ensure!(config_file.exists().await, "accounts.toml does not exist");
 
-        let config = Config::from_file(config_file).await?;
-        let accounts = config.load_accounts().await?;
+        let config = Config::from_file(config_file)
+            .await
+            .context("failed to load accounts config")?;
+        let accounts = config
+            .load_accounts()
+            .await
+            .context("failed to load accounts")?;
 
         let emitter = EventEmitter::new();
 
@@ -66,7 +71,9 @@ impl Accounts {
         emitter.sender.send(events.get_emitter()).await?;
 
         for account in accounts.values() {
-            emitter.add_account(account).await?;
+            emitter.add_account(account).await.with_context(|| {
+                format!("failed to add account {} to event emitter", account.id)
+            })?;
         }
 
         Ok(Self {
@@ -104,12 +111,24 @@ impl Accounts {
         Ok(())
     }
 
-    /// Add a new account.
+    /// Add a new account and opens it.
+    ///
+    /// Returns account ID.
     pub async fn add_account(&mut self) -> Result<u32> {
-        let os_name = self.config.os_name().await;
         let account_config = self.config.new_account(&self.dir).await?;
 
-        let ctx = Context::new(os_name, account_config.dbfile().into(), account_config.id).await?;
+        let ctx = Context::new(account_config.dbfile().into(), account_config.id).await?;
+        self.emitter.add_account(&ctx).await?;
+        self.accounts.insert(account_config.id, ctx);
+
+        Ok(account_config.id)
+    }
+
+    /// Adds a new closed account.
+    pub async fn add_closed_account(&mut self) -> Result<u32> {
+        let account_config = self.config.new_account(&self.dir).await?;
+
+        let ctx = Context::new_closed(account_config.dbfile().into(), account_config.id).await?;
         self.emitter.add_account(&ctx).await?;
         self.accounts.insert(account_config.id, ctx);
 
@@ -183,13 +202,7 @@ impl Accounts {
 
         match res {
             Ok(_) => {
-                let ctx = Context::with_blobdir(
-                    self.config.os_name().await,
-                    new_dbfile,
-                    new_blobdir,
-                    account_config.id,
-                )
-                .await?;
+                let ctx = Context::new(new_dbfile, account_config.id).await?;
                 self.emitter.add_account(&ctx).await?;
                 self.accounts.insert(account_config.id, ctx);
                 Ok(account_config.id)
@@ -349,7 +362,6 @@ pub struct Config {
 /// This is serialized into TOML.
 #[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]
 struct InnerConfig {
-    pub os_name: String,
     /// The currently selected account.
     pub selected_account: u32,
     pub next_id: u32,
@@ -357,9 +369,8 @@ struct InnerConfig {
 }
 
 impl Config {
-    pub async fn new(os_name: String, dir: &PathBuf) -> Result<Self> {
+    pub async fn new(dir: &PathBuf) -> Result<Self> {
         let inner = InnerConfig {
-            os_name,
             accounts: Vec::new(),
             selected_account: 0,
             next_id: 1,
@@ -374,10 +385,6 @@ impl Config {
         Ok(cfg)
     }
 
-    pub async fn os_name(&self) -> String {
-        self.inner.os_name.clone()
-    }
-
     /// Sync the inmemory representation to disk.
     async fn sync(&self) -> Result<()> {
         fs::write(&self.file, toml::to_string_pretty(&self.inner)?)
@@ -396,12 +403,15 @@ impl Config {
     pub async fn load_accounts(&self) -> Result<BTreeMap<u32, Context>> {
         let mut accounts = BTreeMap::new();
         for account_config in &self.inner.accounts {
-            let ctx = Context::new(
-                self.inner.os_name.clone(),
-                account_config.dbfile().into(),
-                account_config.id,
-            )
-            .await?;
+            let ctx = Context::new(account_config.dbfile().into(), account_config.id)
+                .await
+                .with_context(|| {
+                    format!(
+                        "failed to create context from file {:?}",
+                        account_config.dbfile()
+                    )
+                })?;
+
             accounts.insert(account_config.id, ctx);
         }
 
@@ -426,8 +436,13 @@ impl Config {
 
         self.sync().await?;
 
-        self.select_account(id).await.expect("just added");
-        let cfg = self.get_account(id).await.expect("just added");
+        self.select_account(id)
+            .await
+            .context("failed to select just added account")?;
+        let cfg = self
+            .get_account(id)
+            .await
+            .context("failed to get just added account")?;
         Ok(cfg)
     }
 
@@ -498,7 +513,7 @@ mod tests {
         let dir = tempfile::tempdir().unwrap();
         let p: PathBuf = dir.path().join("accounts1").into();
 
-        let mut accounts1 = Accounts::new("my_os".into(), p.clone()).await.unwrap();
+        let mut accounts1 = Accounts::new(p.clone()).await.unwrap();
         accounts1.add_account().await.unwrap();
 
         let accounts2 = Accounts::open(p).await.unwrap();
@@ -516,7 +531,7 @@ mod tests {
         let dir = tempfile::tempdir().unwrap();
         let p: PathBuf = dir.path().join("accounts").into();
 
-        let mut accounts = Accounts::new("my_os".into(), p.clone()).await.unwrap();
+        let mut accounts = Accounts::new(p.clone()).await.unwrap();
         assert_eq!(accounts.accounts.len(), 0);
         assert_eq!(accounts.config.get_selected_account().await, 0);
 
@@ -543,7 +558,7 @@ mod tests {
         let dir = tempfile::tempdir()?;
         let p: PathBuf = dir.path().join("accounts").into();
 
-        let mut accounts = Accounts::new("my_os".into(), p.clone()).await?;
+        let mut accounts = Accounts::new(p.clone()).await?;
         assert!(accounts.get_selected_account().await.is_none());
         assert_eq!(accounts.config.get_selected_account().await, 0);
 
@@ -564,14 +579,12 @@ mod tests {
         let dir = tempfile::tempdir().unwrap();
         let p: PathBuf = dir.path().join("accounts").into();
 
-        let mut accounts = Accounts::new("my_os".into(), p.clone()).await.unwrap();
+        let mut accounts = Accounts::new(p.clone()).await.unwrap();
         assert_eq!(accounts.accounts.len(), 0);
         assert_eq!(accounts.config.get_selected_account().await, 0);
 
         let extern_dbfile: PathBuf = dir.path().join("other").into();
-        let ctx = Context::new("my_os".into(), extern_dbfile.clone(), 0)
-            .await
-            .unwrap();
+        let ctx = Context::new(extern_dbfile.clone(), 0).await.unwrap();
         ctx.set_config(crate::config::Config::Addr, Some("me@mail.com"))
             .await
             .unwrap();
@@ -601,7 +614,7 @@ mod tests {
         let dir = tempfile::tempdir().unwrap();
         let p: PathBuf = dir.path().join("accounts").into();
 
-        let mut accounts = Accounts::new("my_os".into(), p.clone()).await.unwrap();
+        let mut accounts = Accounts::new(p.clone()).await.unwrap();
 
         for expected_id in 1..10 {
             let id = accounts.add_account().await.unwrap();
@@ -621,7 +634,7 @@ mod tests {
         let dummy_accounts = 10;
 
         let (id0, id1, id2) = {
-            let mut accounts = Accounts::new("my_os".into(), p.clone()).await?;
+            let mut accounts = Accounts::new(p.clone()).await?;
             accounts.add_account().await?;
             let ids = accounts.get_all().await;
             assert_eq!(ids.len(), 1);
@@ -656,7 +669,7 @@ mod tests {
         assert!(id2 > id1 + dummy_accounts);
 
         let (id0_reopened, id1_reopened, id2_reopened) = {
-            let accounts = Accounts::new("my_os".into(), p.clone()).await?;
+            let accounts = Accounts::new(p.clone()).await?;
             let ctx = accounts.get_selected_account().await.unwrap();
             assert_eq!(
                 ctx.get_config(crate::config::Config::Addr).await?,
@@ -701,7 +714,7 @@ mod tests {
         let dir = tempfile::tempdir().unwrap();
         let p: PathBuf = dir.path().join("accounts").into();
 
-        let accounts = Accounts::new("my_os".into(), p.clone()).await?;
+        let accounts = Accounts::new(p.clone()).await?;
 
         // Make sure there are no accounts.
         assert_eq!(accounts.accounts.len(), 0);
@@ -721,4 +734,49 @@ mod tests {
 
         Ok(())
     }
+
+    #[async_std::test]
+    async fn test_encrypted_account() -> Result<()> {
+        let dir = tempfile::tempdir().context("failed to create tempdir")?;
+        let p: PathBuf = dir.path().join("accounts").into();
+
+        let mut accounts = Accounts::new(p.clone())
+            .await
+            .context("failed to create accounts manager")?;
+
+        assert_eq!(accounts.accounts.len(), 0);
+        let account_id = accounts
+            .add_closed_account()
+            .await
+            .context("failed to add closed account")?;
+        let account = accounts
+            .get_selected_account()
+            .await
+            .context("failed to get account")?;
+        assert_eq!(account.id, account_id);
+        let passphrase_set_success = account
+            .open("foobar".to_string())
+            .await
+            .context("failed to set passphrase")?;
+        assert!(passphrase_set_success);
+        drop(accounts);
+
+        let accounts = Accounts::new(p.clone())
+            .await
+            .context("failed to create second accounts manager")?;
+        let account = accounts
+            .get_selected_account()
+            .await
+            .context("failed to get account")?;
+        assert_eq!(account.is_open().await, false);
+
+        // Try wrong passphrase.
+        assert_eq!(account.open("barfoo".to_string()).await?, false);
+        assert_eq!(account.open("".to_string()).await?, false);
+
+        assert_eq!(account.open("foobar".to_string()).await?, true);
+        assert_eq!(account.is_open().await, true);
+
+        Ok(())
+    }
 }

src/aheader.rs

@@ -2,7 +2,7 @@
 //!
 //! Parse and create [Autocrypt-headers](https://autocrypt.org/en/latest/level1.html#the-autocrypt-header).
 
-use anyhow::{bail, format_err, Error, Result};
+use anyhow::{bail, Context as _, Error, Result};
 use std::collections::BTreeMap;
 use std::str::FromStr;
 use std::{fmt, str};
@@ -139,15 +139,14 @@ impl str::FromStr for Aheader {
         };
         let public_key: SignedPublicKey = attributes
             .remove("keydata")
-            .ok_or_else(|| format_err!("keydata attribute is not found"))
+            .context("keydata attribute is not found")
             .and_then(|raw| {
-                SignedPublicKey::from_base64(&raw)
-                    .map_err(|_| format_err!("Autocrypt key cannot be decoded"))
+                SignedPublicKey::from_base64(&raw).context("autocrypt key cannot be decoded")
             })
             .and_then(|key| {
                 key.verify()
                     .and(Ok(key))
-                    .map_err(|_| format_err!("Autocrypt key cannot be verified"))
+                    .context("autocrypt key cannot be verified")
             })?;
 
         let prefer_encrypt = attributes

src/blob.rs

@@ -3,28 +3,26 @@
 use core::cmp::max;
 use std::ffi::OsStr;
 use std::fmt;
+use std::io::Cursor;
 
 use async_std::path::{Path, PathBuf};
 use async_std::prelude::*;
 use async_std::{fs, io};
 
-use anyhow::format_err;
-use anyhow::Context as _;
-use anyhow::Error;
-use image::DynamicImage;
-use image::GenericImageView;
-use image::ImageFormat;
+use anyhow::{format_err, Context as _, Error};
+use image::{DynamicImage, ImageFormat};
 use num_traits::FromPrimitive;
 use thiserror::Error;
 
 use crate::config::Config;
 use crate::constants::{
-    MediaQuality, Viewtype, BALANCED_AVATAR_SIZE, BALANCED_IMAGE_SIZE, WORSE_AVATAR_SIZE,
-    WORSE_IMAGE_SIZE,
+    MediaQuality, BALANCED_AVATAR_SIZE, BALANCED_IMAGE_SIZE, WORSE_AVATAR_SIZE, WORSE_IMAGE_SIZE,
 };
 use crate::context::Context;
 use crate::events::EventType;
+use crate::log::LogExt;
 use crate::message;
+use crate::message::Viewtype;
 
 /// Represents a file in the blob directory.
 ///
@@ -63,7 +61,7 @@ impl<'a> BlobObject<'a> {
     ) -> std::result::Result<BlobObject<'a>, BlobError> {
         let blobdir = context.get_blobdir();
         let (stem, ext) = BlobObject::sanitise_name(suggested_name);
-        let (name, mut file) = BlobObject::create_new_file(blobdir, &stem, &ext).await?;
+        let (name, mut file) = BlobObject::create_new_file(context, blobdir, &stem, &ext).await?;
         file.write_all(data)
             .await
             .map_err(|err| BlobError::WriteFailure {
@@ -87,13 +85,16 @@ impl<'a> BlobObject<'a> {
 
     // Creates a new file, returning a tuple of the name and the handle.
     async fn create_new_file(
+        context: &Context,
         dir: &Path,
         stem: &str,
         ext: &str,
     ) -> Result<(String, fs::File), BlobError> {
-        let max_attempt = 15;
+        const MAX_ATTEMPT: u32 = 16;
+        let mut attempt = 0;
         let mut name = format!("{}{}", stem, ext);
-        for attempt in 0..max_attempt {
+        loop {
+            attempt += 1;
             let path = dir.join(&name);
             match fs::OpenOptions::new()
                 .create_new(true)
@@ -103,24 +104,20 @@ impl<'a> BlobObject<'a> {
             {
                 Ok(file) => return Ok((name, file)),
                 Err(err) => {
-                    if attempt == max_attempt {
+                    if attempt >= MAX_ATTEMPT {
                         return Err(BlobError::CreateFailure {
                             blobdir: dir.to_path_buf(),
                             blobname: name,
                             cause: err,
                         });
+                    } else if attempt == 1 && !dir.exists().await {
+                        fs::create_dir_all(dir).await.ok_or_log(context);
                     } else {
                         name = format!("{}-{}{}", stem, rand::random::<u32>(), ext);
                     }
                 }
             }
         }
-        // This is supposed to be unreachable, but the compiler doesn't know.
-        Err(BlobError::CreateFailure {
-            blobdir: dir.to_path_buf(),
-            blobname: name,
-            cause: std::io::Error::new(std::io::ErrorKind::Other, "supposedly unreachable"),
-        })
     }
 
     /// Creates a new blob object with unique name by copying an existing file.
@@ -149,7 +146,7 @@ impl<'a> BlobObject<'a> {
             })?;
         let (stem, ext) = BlobObject::sanitise_name(&src.to_string_lossy());
         let (name, mut dst_file) =
-            BlobObject::create_new_file(context.get_blobdir(), &stem, &ext).await?;
+            BlobObject::create_new_file(context, context.get_blobdir(), &stem, &ext).await?;
         let name_for_err = name.clone();
         if let Err(err) = io::copy(&mut src_file, &mut dst_file).await {
             {
@@ -292,7 +289,7 @@ impl<'a> BlobObject<'a> {
 
     /// Returns the filename of the blob.
     pub fn as_file_name(&self) -> &str {
-        self.name.rsplitn(2, '/').next().unwrap()
+        self.name.rsplit('/').next().unwrap()
     }
 
     /// The path relative in the blob directory.
@@ -305,7 +302,7 @@ impl<'a> BlobObject<'a> {
     /// If a blob's filename has an extension, it is always guaranteed
     /// to be lowercase.
     pub fn suffix(&self) -> Option<&str> {
-        let ext = self.name.rsplitn(2, '.').next();
+        let ext = self.name.rsplit('.').next();
         if ext == Some(&self.name) {
             None
         } else {
@@ -348,13 +345,30 @@ impl<'a> BlobObject<'a> {
         };
 
         let clean = sanitize_filename::sanitize_with_options(name, opts);
+        // Let's take the tricky filename
+        // "file.with_lots_of_characters_behind_point_and_double_ending.tar.gz" as an example.
+        // Split it into "file" and "with_lots_of_characters_behind_point_and_double_ending.tar.gz":
         let mut iter = clean.splitn(2, '.');
+
         let stem: String = iter.next().unwrap_or_default().chars().take(64).collect();
-        let ext: String = iter.next().unwrap_or_default().chars().take(32).collect();
+        // stem == "file"
+
+        let ext_chars = iter.next().unwrap_or_default().chars();
+        let ext: String = ext_chars
+            .rev()
+            .take(32)
+            .collect::<Vec<_>>()
+            .iter()
+            .rev()
+            .collect();
+        // ext == "d_point_and_double_ending.tar.gz"
+
         if ext.is_empty() {
             (stem, "".to_string())
         } else {
             (stem, format!(".{}", ext).to_lowercase())
+            // Return ("file", ".d_point_and_double_ending.tar.gz")
+            // which is not perfect but acceptable.
         }
     }
 
@@ -449,7 +463,8 @@ impl<'a> BlobObject<'a> {
 
         fn encode_img(img: &DynamicImage, encoded: &mut Vec<u8>) -> anyhow::Result<()> {
             encoded.clear();
-            img.write_to(encoded, image::ImageFormat::Jpeg)?;
+            let mut buf = Cursor::new(encoded);
+            img.write_to(&mut buf, image::ImageFormat::Jpeg)?;
             Ok(())
         }
         fn encoded_img_exceeds_bytes(
@@ -619,13 +634,14 @@ pub enum BlobError {
 mod tests {
     use fs::File;
 
-    use super::*;
+    use anyhow::Result;
+    use image::{GenericImageView, Pixel};
 
-    use crate::{
-        message::Message,
-        test_utils::{self, TestContext},
-    };
-    use image::Pixel;
+    use crate::chat::{self, create_group_chat, ProtectionStatus};
+    use crate::message::Message;
+    use crate::test_utils::{self, TestContext};
+
+    use super::*;
 
     #[async_std::test]
     async fn test_create() {
@@ -957,10 +973,11 @@ mod tests {
         .unwrap();
         assert_correct_rotation(&img_rotated);
 
-        let mut bytes = vec![];
+        let mut buf = Cursor::new(vec![]);
         img_rotated
-            .write_to(&mut bytes, image::ImageFormat::Jpeg)
+            .write_to(&mut buf, image::ImageFormat::Jpeg)
             .unwrap();
+        let bytes = buf.into_inner();
         let img_rotated = send_image_check_mediaquality(
             Some("0"),
             &bytes,
@@ -1066,4 +1083,38 @@ mod tests {
         assert_eq!(img.height() as u32, compressed_height);
         Ok(img)
     }
+
+    #[async_std::test]
+    async fn test_increation_in_blobdir() -> Result<()> {
+        let t = TestContext::new_alice().await;
+        let chat_id = create_group_chat(&t, ProtectionStatus::Unprotected, "abc").await?;
+
+        let file = t.get_blobdir().join("anyfile.dat");
+        File::create(&file).await?.write_all("bla".as_ref()).await?;
+        let mut msg = Message::new(Viewtype::File);
+        msg.set_file(file.to_str().unwrap(), None);
+        let prepared_id = chat::prepare_msg(&t, chat_id, &mut msg).await?;
+        assert_eq!(prepared_id, msg.id);
+        assert!(msg.is_increation());
+
+        let msg = Message::load_from_db(&t, prepared_id).await?;
+        assert!(msg.is_increation());
+
+        Ok(())
+    }
+
+    #[async_std::test]
+    async fn test_increation_not_blobdir() -> Result<()> {
+        let t = TestContext::new_alice().await;
+        let chat_id = create_group_chat(&t, ProtectionStatus::Unprotected, "abc").await?;
+        assert_ne!(t.get_blobdir().to_str(), t.dir.path().to_str());
+
+        let file = t.dir.path().join("anyfile.dat");
+        File::create(&file).await?.write_all("bla".as_ref()).await?;
+        let mut msg = Message::new(Viewtype::File);
+        msg.set_file(file.to_str().unwrap(), None);
+        assert!(chat::prepare_msg(&t, chat_id, &mut msg).await.is_err());
+
+        Ok(())
+    }
 }

src/chatlist.rs

@@ -1,6 +1,6 @@
 //! # Chat list module.
 
-use anyhow::{bail, ensure, Result};
+use anyhow::{ensure, Context as _, Result};
 
 use crate::chat::{update_special_chat_names, Chat, ChatId, ChatVisibility};
 use crate::constants::{
@@ -8,7 +8,7 @@ use crate::constants::{
     DC_CONTACT_ID_SELF, DC_CONTACT_ID_UNDEFINED, DC_GCL_ADD_ALLDONE_HINT, DC_GCL_ARCHIVED_ONLY,
     DC_GCL_FOR_FORWARDING, DC_GCL_NO_SPECIALS,
 };
-use crate::contact::Contact;
+use crate::contact::{Contact, ContactId};
 use crate::context::Context;
 use crate::ephemeral::delete_expired_messages;
 use crate::message::{Message, MessageState, MsgId};
@@ -85,7 +85,7 @@ impl Chatlist {
         context: &Context,
         listflags: usize,
         query: Option<&str>,
-        query_contact_id: Option<u32>,
+        query_contact_id: Option<ContactId>,
     ) -> Result<Self> {
         let flag_archived_only = 0 != listflags & DC_GCL_ARCHIVED_ONLY;
         let flag_for_forwarding = 0 != listflags & DC_GCL_FOR_FORWARDING;
@@ -147,7 +147,7 @@ impl Chatlist {
                    AND c.id IN(SELECT chat_id FROM chats_contacts WHERE contact_id=?2)
                  GROUP BY c.id
                  ORDER BY c.archived=?3 DESC, IFNULL(m.timestamp,c.created_timestamp) DESC, m.id DESC;",
-                paramsv![MessageState::OutDraft, query_contact_id as i32, ChatVisibility::Pinned],
+                paramsv![MessageState::OutDraft, query_contact_id, ChatVisibility::Pinned],
                 process_row,
                 process_rows,
             ).await?
@@ -271,21 +271,23 @@ impl Chatlist {
     /// Get a single chat ID of a chatlist.
     ///
     /// To get the message object from the message ID, use dc_get_chat().
-    pub fn get_chat_id(&self, index: usize) -> ChatId {
-        match self.ids.get(index) {
-            Some((chat_id, _msg_id)) => *chat_id,
-            None => ChatId::new(0),
-        }
+    pub fn get_chat_id(&self, index: usize) -> Result<ChatId> {
+        let (chat_id, _msg_id) = self
+            .ids
+            .get(index)
+            .context("chatlist index is out of range")?;
+        Ok(*chat_id)
     }
 
     /// Get a single message ID of a chatlist.
     ///
     /// To get the message object from the message ID, use dc_get_msg().
     pub fn get_msg_id(&self, index: usize) -> Result<Option<MsgId>> {
-        match self.ids.get(index) {
-            Some((_chat_id, msg_id)) => Ok(*msg_id),
-            None => bail!("Chatlist index out of range"),
-        }
+        let (_chat_id, msg_id) = self
+            .ids
+            .get(index)
+            .context("chatlist index is out of range")?;
+        Ok(*msg_id)
     }
 
     /// Returns a summary for a given chatlist index.
@@ -299,11 +301,10 @@ impl Chatlist {
         // This is because we may want to display drafts here or stuff as
         // "is typing".
         // Also, sth. as "No messages" would not work if the summary comes from a message.
-        let (chat_id, lastmsg_id) = match self.ids.get(index) {
-            Some(ids) => ids,
-            None => bail!("Chatlist index out of range"),
-        };
-
+        let (chat_id, lastmsg_id) = self
+            .ids
+            .get(index)
+            .context("chatlist index is out of range")?;
         Chatlist::get_summary2(context, *chat_id, *lastmsg_id, chat).await
     }
 
@@ -374,8 +375,8 @@ mod tests {
     use super::*;
 
     use crate::chat::{create_group_chat, get_chat_contacts, ProtectionStatus};
-    use crate::constants::Viewtype;
     use crate::dc_receive_imf::dc_receive_imf;
+    use crate::message::Viewtype;
     use crate::stock_str::StockMessage;
     use crate::test_utils::TestContext;
 
@@ -395,9 +396,9 @@ mod tests {
         // check that the chatlist starts with the most recent message
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
         assert_eq!(chats.len(), 3);
-        assert_eq!(chats.get_chat_id(0), chat_id3);
-        assert_eq!(chats.get_chat_id(1), chat_id2);
-        assert_eq!(chats.get_chat_id(2), chat_id1);
+        assert_eq!(chats.get_chat_id(0).unwrap(), chat_id3);
+        assert_eq!(chats.get_chat_id(1).unwrap(), chat_id2);
+        assert_eq!(chats.get_chat_id(2).unwrap(), chat_id1);
 
         // New drafts are sorted to the top
         // We have to set a draft on the other two messages, too, as
@@ -414,7 +415,7 @@ mod tests {
         }
 
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
-        assert_eq!(chats.get_chat_id(0), chat_id2);
+        assert_eq!(chats.get_chat_id(0).unwrap(), chat_id2);
 
         // check chatlist query and archive functionality
         let chats = Chatlist::try_load(&t, 0, Some("b"), None).await.unwrap();
@@ -445,7 +446,7 @@ mod tests {
 
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
         assert!(chats.len() == 3);
-        assert!(!Chat::load_from_db(&t, chats.get_chat_id(0))
+        assert!(!Chat::load_from_db(&t, chats.get_chat_id(0).unwrap())
             .await
             .unwrap()
             .is_self_talk());
@@ -454,7 +455,7 @@ mod tests {
             .await
             .unwrap();
         assert!(chats.len() == 2); // device chat cannot be written and is skipped on forwarding
-        assert!(Chat::load_from_db(&t, chats.get_chat_id(0))
+        assert!(Chat::load_from_db(&t, chats.get_chat_id(0).unwrap())
             .await
             .unwrap()
             .is_self_talk());
@@ -499,7 +500,7 @@ mod tests {
         dc_receive_imf(
             &t,
             b"From: Bob Authname <bob@example.org>\n\
-                 To: alice@example.com\n\
+                 To: alice@example.org\n\
                  Subject: foo\n\
                  Message-ID: <msg1234@example.org>\n\
                  Chat-Version: 1.0\n\
@@ -507,7 +508,6 @@ mod tests {
                  \n\
                  hello foo\n",
             "INBOX",
-            1,
             false,
         )
         .await?;
@@ -528,7 +528,7 @@ mod tests {
         // check, the one-to-one-chat can be found using chatlist search query
         let chats = Chatlist::try_load(&t, 0, Some("bob authname"), None).await?;
         assert_eq!(chats.len(), 1);
-        assert_eq!(chats.get_chat_id(0), chat_id);
+        assert_eq!(chats.get_chat_id(0).unwrap(), chat_id);
 
         // change the name of the contact; this also changes the name of the one-to-one-chat
         let test_id = Contact::create(&t, "Bob Nickname", "bob@example.org").await?;
@@ -561,7 +561,7 @@ mod tests {
         dc_receive_imf(
             &t,
             b"From: bob@example.org\n\
-                 To: alice@example.com\n\
+                 To: alice@example.org\n\
                  Subject: foo\n\
                  Message-ID: <msg5678@example.org>\n\
                  Chat-Version: 1.0\n\
@@ -569,7 +569,6 @@ mod tests {
                  \n\
                  hello foo\n",
             "INBOX",
-            1,
             false,
         )
         .await?;
@@ -585,7 +584,7 @@ mod tests {
         // check, the one-to-one-chat can be found using chatlist search query
         let chats = Chatlist::try_load(&t, 0, Some("bob@example.org"), None).await?;
         assert_eq!(chats.len(), 1);
-        assert_eq!(chats.get_chat_id(0), chat_id);
+        assert_eq!(chats.get_chat_id(0)?, chat_id);
 
         // change the name of the contact; this also changes the name of the one-to-one-chat
         let test_id = Contact::create(&t, "Bob Nickname", "bob@example.org").await?;
@@ -596,7 +595,7 @@ mod tests {
         assert_eq!(chats.len(), 0); // email-addresses are searchable in contacts, not in chats
         let chats = Chatlist::try_load(&t, 0, Some("Bob Nickname"), None).await?;
         assert_eq!(chats.len(), 1);
-        assert_eq!(chats.get_chat_id(0), chat_id);
+        assert_eq!(chats.get_chat_id(0)?, chat_id);
 
         // revert name change, this again changes the name of the one-to-one-chat to the email-address
         let test_id = Contact::create(&t, "", "bob@example.org").await?;

src/config.rs

@@ -10,11 +10,9 @@ use crate::constants::DC_VERSION_STR;
 use crate::context::Context;
 use crate::dc_tools::{dc_get_abs_path, improve_single_line_input};
 use crate::events::EventType;
-use crate::job;
 use crate::message::MsgId;
 use crate::mimefactory::RECOMMENDED_FILE_SIZE;
 use crate::provider::{get_provider_by_id, Provider};
-use crate::stock_str;
 
 /// The available configuration keys.
 #[derive(
@@ -64,30 +62,21 @@ pub enum Config {
     #[strum(props(default = "1"))]
     E2eeEnabled,
 
-    /// Ignore Autocrypt recommendation for message encryption if possible.
-    ///
-    /// The only expection is when recommendation is "disable", i.e. encryption is not possible
-    /// because some recipient has no OpenPGP key.
-    #[strum(props(default = "0"))]
-    E2eeForce,
-
     #[strum(props(default = "1"))]
     MdnsEnabled,
 
-    #[strum(props(default = "1"))]
-    InboxWatch,
-
-    #[strum(props(default = "1"))]
+    #[strum(props(default = "0"))]
     SentboxWatch,
 
-    #[strum(props(default = "1"))]
-    MvboxWatch,
-
     #[strum(props(default = "1"))]
     MvboxMove,
 
+    /// Watch for new messages in the "Mvbox" (aka DeltaChat folder) only.
+    ///
+    /// This will not entirely disable other folders, e.g. the spam folder will also still
+    /// be watched for new messages.
     #[strum(props(default = "0"))]
-    SentboxMove, // If `MvboxMove` is true, this config is ignored. Currently only used in tests.
+    OnlyFetchMvbox,
 
     #[strum(props(default = "0"))] // also change ShowEmails.default() on changes
     ShowEmails,
@@ -213,7 +202,6 @@ impl Context {
 
         // Default values
         match key {
-            Config::Selfstatus => Ok(Some(stock_str::status_line(self).await)),
             Config::ConfiguredInboxFolder => Ok(Some("INBOX".to_owned())),
             _ => Ok(key.get_str("default").map(|s| s.to_string())),
         }
@@ -241,6 +229,11 @@ impl Context {
         Ok(self.get_config_int(key).await? != 0)
     }
 
+    pub(crate) async fn should_watch_mvbox(&self) -> Result<bool> {
+        Ok(self.get_config_bool(Config::MvboxMove).await?
+            || self.get_config_bool(Config::OnlyFetchMvbox).await?)
+    }
+
     /// Gets configured "delete_server_after" value.
     ///
     /// `None` means never delete the message, `Some(0)` means delete
@@ -249,7 +242,7 @@ impl Context {
         match self.get_config_int(Config::DeleteServerAfter).await? {
             0 => Ok(None),
             1 => Ok(Some(0)),
-            x => Ok(Some(x as i64)),
+            x => Ok(Some(i64::from(x))),
         }
     }
 
@@ -271,7 +264,7 @@ impl Context {
     pub async fn get_config_delete_device_after(&self) -> Result<Option<i64>> {
         match self.get_config_int(Config::DeleteDeviceAfter).await? {
             0 => Ok(None),
-            x => Ok(Some(x as i64)),
+            x => Ok(Some(i64::from(x))),
         }
     }
 
@@ -297,55 +290,29 @@ impl Context {
                     }
                 }
                 self.emit_event(EventType::SelfavatarChanged);
-                Ok(())
-            }
-            Config::Selfstatus => {
-                let def = stock_str::status_line(self).await;
-                let val = if value.is_none() || value.unwrap() == def {
-                    None
-                } else {
-                    value
-                };
-
-                self.sql.set_raw_config(key, val).await?;
-                Ok(())
             }
             Config::DeleteDeviceAfter => {
-                let ret = self
-                    .sql
-                    .set_raw_config(key, value)
-                    .await
-                    .map_err(Into::into);
+                let ret = self.sql.set_raw_config(key, value).await;
                 // Force chatlist reload to delete old messages immediately.
                 self.emit_event(EventType::MsgsChanged {
                     msg_id: MsgId::new(0),
                     chat_id: ChatId::new(0),
                 });
-                ret
+                ret?
             }
             Config::Displayname => {
                 let value = value.map(improve_single_line_input);
                 self.sql.set_raw_config(key, value.as_deref()).await?;
-                Ok(())
-            }
-            Config::DeleteServerAfter => {
-                let ret = self
-                    .sql
-                    .set_raw_config(key, value)
-                    .await
-                    .map_err(Into::into);
-                job::schedule_resync(self).await?;
-                ret
             }
             _ => {
                 self.sql.set_raw_config(key, value).await?;
-                Ok(())
             }
         }
+        Ok(())
     }
 
     pub async fn set_config_bool(&self, key: Config, value: bool) -> Result<()> {
-        self.set_config(key, if value { Some("1") } else { None })
+        self.set_config(key, if value { Some("1") } else { Some("0") })
             .await?;
         Ok(())
     }
@@ -437,4 +404,17 @@ mod tests {
 
         Ok(())
     }
+
+    /// Regression test for https://github.com/deltachat/deltachat-core-rust/issues/3012
+    #[async_std::test]
+    async fn test_set_config_bool() -> Result<()> {
+        let t = TestContext::new().await;
+
+        // We need some config that defaults to true
+        let c = Config::E2eeEnabled;
+        assert_eq!(t.get_config_bool(c).await?, true);
+        t.set_config_bool(c, false).await?;
+        assert_eq!(t.get_config_bool(c).await?, false);
+        Ok(())
+    }
 }

src/configure.rs

@@ -11,21 +11,20 @@ use async_std::task;
 use job::Action;
 use percent_encoding::{utf8_percent_encode, NON_ALPHANUMERIC};
 
-use crate::dc_tools::EmailAddress;
+use crate::config::Config;
+use crate::constants::{DC_LP_AUTH_FLAGS, DC_LP_AUTH_NORMAL, DC_LP_AUTH_OAUTH2};
+use crate::context::Context;
+use crate::dc_tools::{time, EmailAddress};
 use crate::imap::Imap;
+use crate::job;
 use crate::login_param::{CertificateChecks, LoginParam, ServerLoginParam, Socks5Config};
-use crate::message::Message;
+use crate::message::{Message, Viewtype};
 use crate::oauth2::dc_get_oauth2_addr;
+use crate::param::Params;
 use crate::provider::{Protocol, Socket, UsernamePattern};
 use crate::smtp::Smtp;
 use crate::stock_str;
 use crate::{chat, e2ee, provider};
-use crate::{config::Config, dc_tools::time};
-use crate::{
-    constants::{Viewtype, DC_LP_AUTH_FLAGS, DC_LP_AUTH_NORMAL, DC_LP_AUTH_OAUTH2},
-    job,
-};
-use crate::{context::Context, param::Params};
 
 use auto_mozilla::moz_autoconfigure;
 use auto_outlook::outlk_autodiscover;
@@ -221,7 +220,9 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
             "checking internal provider-info for offline autoconfig"
         );
 
-        if let Some(provider) = provider::get_provider_info(&param_domain, socks5_enabled).await {
+        if let Some(provider) =
+            provider::get_provider_info(ctx, &param_domain, socks5_enabled).await
+        {
             param.provider = Some(provider);
             match provider.status {
                 provider::Status::Ok | provider::Status::Preparation => {
@@ -441,8 +442,7 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
 
     progress!(ctx, 900);
 
-    let create_mvbox = ctx.get_config_bool(Config::MvboxWatch).await?
-        || ctx.get_config_bool(Config::MvboxMove).await?;
+    let create_mvbox = ctx.should_watch_mvbox().await?;
 
     imap.configure_folders(ctx, create_mvbox).await?;
 
@@ -453,11 +453,8 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
     drop(imap);
 
     progress!(ctx, 910);
-    // configuration success - write back the configured parameters with the
-    // "configured_" prefix; also write the "configured"-flag */
     // the trailing underscore is correct
     param.save_to_database(ctx, "configured_").await?;
-    ctx.sql.set_raw_config_bool("configured", true).await?;
     ctx.set_config(Config::ConfiguredTimestamp, Some(&time().to_string()))
         .await?;
 
@@ -475,6 +472,8 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
     progress!(ctx, 940);
     update_device_chats_handle.await?;
 
+    ctx.sql.set_raw_config_bool("configured", true).await?;
+
     Ok(())
 }
 

src/constants.rs

@@ -4,6 +4,7 @@ use once_cell::sync::Lazy;
 use serde::{Deserialize, Serialize};
 
 use crate::chat::ChatId;
+use crate::contact::ContactId;
 
 pub static DC_VERSION_STR: Lazy<String> = Lazy::new(|| env!("CARGO_PKG_VERSION").to_string());
 
@@ -179,11 +180,11 @@ pub const DC_ELLIPSIS: &str = "[...]";
 /// `char`s), not Unicode Grapheme Clusters.
 pub const DC_DESIRED_TEXT_LEN: usize = 5000;
 
-pub const DC_CONTACT_ID_UNDEFINED: u32 = 0;
-pub const DC_CONTACT_ID_SELF: u32 = 1;
-pub const DC_CONTACT_ID_INFO: u32 = 2;
-pub const DC_CONTACT_ID_DEVICE: u32 = 5;
-pub const DC_CONTACT_ID_LAST_SPECIAL: u32 = 9;
+pub const DC_CONTACT_ID_UNDEFINED: ContactId = ContactId::new(0);
+pub const DC_CONTACT_ID_SELF: ContactId = ContactId::new(1);
+pub const DC_CONTACT_ID_INFO: ContactId = ContactId::new(2);
+pub const DC_CONTACT_ID_DEVICE: ContactId = ContactId::new(5);
+pub const DC_CONTACT_ID_LAST_SPECIAL: ContactId = ContactId::new(9);
 
 // decorative address that is used for DC_CONTACT_ID_DEVICE
 // when an api that returns an email is called.
@@ -230,79 +231,6 @@ pub const DC_FOLDERS_CONFIGURED_VERSION: i32 = 3;
 // can be overwritten by the setting `max_smtp_rcpt_to` in provider-db.
 pub const DEFAULT_MAX_SMTP_RCPT_TO: usize = 50;
 
-#[derive(
-    Debug,
-    Display,
-    Clone,
-    Copy,
-    PartialEq,
-    Eq,
-    FromPrimitive,
-    ToPrimitive,
-    FromSql,
-    ToSql,
-    Serialize,
-    Deserialize,
-)]
-#[repr(u32)]
-pub enum Viewtype {
-    Unknown = 0,
-
-    /// Text message.
-    /// The text of the message is set using dc_msg_set_text()
-    /// and retrieved with dc_msg_get_text().
-    Text = 10,
-
-    /// Image message.
-    /// If the image is an animated GIF, the type DC_MSG_GIF should be used.
-    /// File, width and height are set via dc_msg_set_file(), dc_msg_set_dimension
-    /// and retrieved via dc_msg_set_file(), dc_msg_set_dimension().
-    Image = 20,
-
-    /// Animated GIF message.
-    /// File, width and height are set via dc_msg_set_file(), dc_msg_set_dimension()
-    /// and retrieved via dc_msg_get_file(), dc_msg_get_width(), dc_msg_get_height().
-    Gif = 21,
-
-    /// Message containing a sticker, similar to image.
-    /// If possible, the ui should display the image without borders in a transparent way.
-    /// A click on a sticker will offer to install the sticker set in some future.
-    Sticker = 23,
-
-    /// Message containing an Audio file.
-    /// File and duration are set via dc_msg_set_file(), dc_msg_set_duration()
-    /// and retrieved via dc_msg_get_file(), dc_msg_get_duration().
-    Audio = 40,
-
-    /// A voice message that was directly recorded by the user.
-    /// For all other audio messages, the type #DC_MSG_AUDIO should be used.
-    /// File and duration are set via dc_msg_set_file(), dc_msg_set_duration()
-    /// and retrieved via dc_msg_get_file(), dc_msg_get_duration()
-    Voice = 41,
-
-    /// Video messages.
-    /// File, width, height and durarion
-    /// are set via dc_msg_set_file(), dc_msg_set_dimension(), dc_msg_set_duration()
-    /// and retrieved via
-    /// dc_msg_get_file(), dc_msg_get_width(),
-    /// dc_msg_get_height(), dc_msg_get_duration().
-    Video = 50,
-
-    /// Message containing any file, eg. a PDF.
-    /// The file is set via dc_msg_set_file()
-    /// and retrieved via dc_msg_get_file().
-    File = 60,
-
-    /// Message is an invitation to a videochat.
-    VideochatInvitation = 70,
-}
-
-impl Default for Viewtype {
-    fn default() -> Self {
-        Viewtype::Unknown
-    }
-}
-
 pub const DC_JOB_DELETE_MSG_ON_IMAP: i32 = 110;
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq, FromPrimitive, ToPrimitive)]
@@ -314,32 +242,9 @@ pub enum KeyType {
 
 #[cfg(test)]
 mod tests {
-    use super::*;
     use num_traits::FromPrimitive;
 
-    #[test]
-    fn test_derive_display_works_as_expected() {
-        assert_eq!(format!("{}", Viewtype::Audio), "Audio");
-    }
-
-    #[test]
-    fn test_viewtype_values() {
-        // values may be written to disk and must not change
-        assert_eq!(Viewtype::Unknown, Viewtype::default());
-        assert_eq!(Viewtype::Unknown, Viewtype::from_i32(0).unwrap());
-        assert_eq!(Viewtype::Text, Viewtype::from_i32(10).unwrap());
-        assert_eq!(Viewtype::Image, Viewtype::from_i32(20).unwrap());
-        assert_eq!(Viewtype::Gif, Viewtype::from_i32(21).unwrap());
-        assert_eq!(Viewtype::Sticker, Viewtype::from_i32(23).unwrap());
-        assert_eq!(Viewtype::Audio, Viewtype::from_i32(40).unwrap());
-        assert_eq!(Viewtype::Voice, Viewtype::from_i32(41).unwrap());
-        assert_eq!(Viewtype::Video, Viewtype::from_i32(50).unwrap());
-        assert_eq!(Viewtype::File, Viewtype::from_i32(60).unwrap());
-        assert_eq!(
-            Viewtype::VideochatInvitation,
-            Viewtype::from_i32(70).unwrap()
-        );
-    }
+    use super::*;
 
     #[test]
     fn test_chattype_values() {

src/contact.rs

@@ -1,12 +1,14 @@
 //! Contacts module
 
 use std::convert::{TryFrom, TryInto};
+use std::fmt;
 
 use anyhow::{bail, ensure, Context as _, Result};
 use async_std::path::PathBuf;
 use deltachat_derive::{FromSql, ToSql};
 use once_cell::sync::Lazy;
 use regex::Regex;
+use serde::{Deserialize, Serialize};
 
 use crate::aheader::EncryptPreference;
 use crate::chat::ChatId;
@@ -27,6 +29,66 @@ use crate::param::{Param, Params};
 use crate::peerstate::{Peerstate, PeerstateVerifiedStatus};
 use crate::{chat, stock_str};
 
+/// Contact ID, including reserved IDs.
+///
+/// Some contact IDs are reserved to identify special contacts.  This
+/// type can represent both the special as well as normal contacts.
+#[derive(
+    Debug, Copy, Clone, Default, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize,
+)]
+pub struct ContactId(u32);
+
+impl ContactId {
+    /// Creates a new [`ContactId`].
+    pub const fn new(id: u32) -> ContactId {
+        ContactId(id)
+    }
+
+    /// Bad evil escape hatch, do not use.
+    pub const fn to_u32(&self) -> u32 {
+        self.0
+    }
+}
+
+impl fmt::Display for ContactId {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "{}", self.0)
+        // if *self == DC_CONTACT_ID_UNDEFINED {
+        //     write!(f, "Contact#Undefined")
+        // } else if *self == DC_CONTACT_ID_SELF {
+        //     write!(f, "Contact#Self")
+        // } else if *self == DC_CONTACT_ID_INFO {
+        //     write!(f, "Contact#Info")
+        // } else if *self == DC_CONTACT_ID_DEVICE {
+        //     write!(f, "Contact#Device")
+        // } else if *self <= DC_CONTACT_ID_LAST_SPECIAL {
+        //     write!(f, "Contact#Special{}", self.0)
+        // } else {
+        //     write!(f, "Contact#{}", self.0)
+        // }
+    }
+}
+
+/// Allow converting [`ContactId`] to an SQLite type.
+impl rusqlite::types::ToSql for ContactId {
+    fn to_sql(&self) -> rusqlite::Result<rusqlite::types::ToSqlOutput> {
+        let val = rusqlite::types::Value::Integer(i64::from(self.0));
+        let out = rusqlite::types::ToSqlOutput::Owned(val);
+        Ok(out)
+    }
+}
+
+/// Allow converting an SQLite integer directly into [`ContactId`].
+impl rusqlite::types::FromSql for ContactId {
+    fn column_result(value: rusqlite::types::ValueRef) -> rusqlite::types::FromSqlResult<Self> {
+        i64::column_result(value).and_then(|val| {
+            val.try_into()
+                .map(ContactId::new)
+                .map_err(|_| rusqlite::types::FromSqlError::OutOfRange(val))
+        })
+    }
+}
+
 /// An object representing a single contact in memory.
 ///
 /// The contact object is not updated.
@@ -48,7 +110,7 @@ pub struct Contact {
     ///   `dc_set_config` using "addr".
     ///
     /// Normal contact IDs are larger than these special ones (larger than DC_CONTACT_ID_LAST_SPECIAL).
-    pub id: u32,
+    pub id: ContactId,
 
     /// Contact name. It is recommended to use `Contact::get_name`,
     /// `Contact::get_display_name` or `Contact::get_name_n_addr` to access this field.
@@ -183,7 +245,7 @@ impl Default for VerifiedStatus {
 }
 
 impl Contact {
-    pub async fn load_from_db(context: &Context, contact_id: u32) -> Result<Self> {
+    pub async fn load_from_db(context: &Context, contact_id: ContactId) -> Result<Self> {
         let mut contact = context
             .sql
             .query_row(
@@ -191,7 +253,7 @@ impl Contact {
                 c.authname, c.param, c.status
                FROM contacts c
               WHERE c.id=?;",
-                paramsv![contact_id as i32],
+                paramsv![contact_id],
                 |row| {
                     let name: String = row.get(0)?;
                     let addr: String = row.get(1)?;
@@ -245,18 +307,18 @@ impl Contact {
     }
 
     /// Check if a contact is blocked.
-    pub async fn is_blocked_load(context: &Context, id: u32) -> Result<bool> {
+    pub async fn is_blocked_load(context: &Context, id: ContactId) -> Result<bool> {
         let blocked = Self::load_from_db(context, id).await?.blocked;
         Ok(blocked)
     }
 
     /// Block the given contact.
-    pub async fn block(context: &Context, id: u32) -> Result<()> {
+    pub async fn block(context: &Context, id: ContactId) -> Result<()> {
         set_block_contact(context, id, true).await
     }
 
     /// Unblock the given contact.
-    pub async fn unblock(context: &Context, id: u32) -> Result<()> {
+    pub async fn unblock(context: &Context, id: ContactId) -> Result<()> {
         set_block_contact(context, id, false).await
     }
 
@@ -269,7 +331,7 @@ impl Contact {
     /// a bunch of addresses.
     ///
     /// May result in a `#DC_EVENT_CONTACTS_CHANGED` event.
-    pub async fn create(context: &Context, name: &str, addr: &str) -> Result<u32> {
+    pub async fn create(context: &Context, name: &str, addr: &str) -> Result<ContactId> {
         let name = improve_single_line_input(name);
         ensure!(!addr.is_empty(), "Cannot create contact with empty address");
 
@@ -292,12 +354,12 @@ impl Contact {
     }
 
     /// Mark messages from a contact as noticed.
-    pub async fn mark_noticed(context: &Context, id: u32) -> Result<()> {
+    pub async fn mark_noticed(context: &Context, id: ContactId) -> Result<()> {
         context
             .sql
             .execute(
                 "UPDATE msgs SET state=? WHERE from_id=? AND state=?;",
-                paramsv![MessageState::InNoticed, id as i32, MessageState::InFresh],
+                paramsv![MessageState::InNoticed, id, MessageState::InFresh],
             )
             .await?;
         Ok(())
@@ -311,17 +373,17 @@ impl Contact {
     /// use `dc_may_be_valid_addr()`.
     pub async fn lookup_id_by_addr(
         context: &Context,
-        addr: impl AsRef<str>,
+        addr: &str,
         min_origin: Origin,
-    ) -> Result<Option<u32>> {
-        if addr.as_ref().is_empty() {
+    ) -> Result<Option<ContactId>> {
+        if addr.is_empty() {
             bail!("lookup_id_by_addr: empty address");
         }
 
-        let addr_normalized = addr_normalize(addr.as_ref());
+        let addr_normalized = addr_normalize(addr);
 
         if let Some(addr_self) = context.get_config(Config::ConfiguredAddr).await? {
-            if addr_cmp(addr_normalized, addr_self) {
+            if addr_cmp(addr_normalized, &addr_self) {
                 return Ok(Some(DC_CONTACT_ID_SELF));
             }
         }
@@ -333,7 +395,7 @@ impl Contact {
             AND id>?2 AND origin>=?3 AND blocked=0;",
                 paramsv![
                     addr_normalized,
-                    DC_CONTACT_ID_LAST_SPECIAL as i32,
+                    DC_CONTACT_ID_LAST_SPECIAL,
                     min_origin as u32,
                 ],
             )
@@ -371,7 +433,7 @@ impl Contact {
         name: &str,
         addr: &str,
         mut origin: Origin,
-    ) -> Result<(u32, Modifier)> {
+    ) -> Result<(ContactId, Modifier)> {
         let mut sth_modified = Modifier::None;
 
         ensure!(!addr.is_empty(), "Can not add_or_lookup empty address");
@@ -383,7 +445,7 @@ impl Contact {
             .await?
             .unwrap_or_default();
 
-        if addr_cmp(&addr, addr_self) {
+        if addr_cmp(&addr, &addr_self) {
             return Ok((DC_CONTACT_ID_SELF, sth_modified));
         }
 
@@ -500,7 +562,7 @@ impl Contact {
                         paramsv![Chattype::Single, isize::try_from(row_id)?]
                     ).await?;
                     if let Some(chat_id) = chat_id {
-                        let contact = Contact::get_by_id(context, row_id as u32).await?;
+                        let contact = Contact::get_by_id(context, ContactId::new(row_id)).await?;
                         let chat_name = contact.get_display_name();
                         match context
                             .sql
@@ -557,7 +619,7 @@ impl Contact {
             }
         }
 
-        Ok((row_id, sth_modified))
+        Ok((ContactId::new(row_id), sth_modified))
     }
 
     /// Add a number of contacts.
@@ -582,7 +644,7 @@ impl Contact {
 
         for (name, addr) in split_address_book(addr_book).into_iter() {
             let (name, addr) = sanitize_name_and_addr(name, addr);
-            let name = normalize_name(name);
+            let name = normalize_name(&name);
             match Contact::add_or_lookup(context, &name, &addr, Origin::AddressBook).await {
                 Err(err) => {
                     warn!(
@@ -617,7 +679,7 @@ impl Contact {
         context: &Context,
         listflags: u32,
         query: Option<impl AsRef<str>>,
-    ) -> Result<Vec<u32>> {
+    ) -> Result<Vec<ContactId>> {
         let self_addr = context
             .get_config(Config::ConfiguredAddr)
             .await?
@@ -644,16 +706,16 @@ impl Contact {
                  ORDER BY LOWER(iif(c.name='',c.authname,c.name)||c.addr),c.id;",
                     paramsv![
                         self_addr,
-                        DC_CONTACT_ID_LAST_SPECIAL as i32,
+                        DC_CONTACT_ID_LAST_SPECIAL,
                         Origin::IncomingReplyTo,
                         s3str_like_cmd,
                         s3str_like_cmd,
                         if flag_verified_only { 0i32 } else { 1i32 },
                     ],
-                    |row| row.get::<_, i32>(0),
+                    |row| row.get::<_, ContactId>(0),
                     |ids| {
                         for id in ids {
-                            ret.push(id? as u32);
+                            ret.push(id?);
                         }
                         Ok(())
                     },
@@ -690,13 +752,13 @@ impl Contact {
                  ORDER BY LOWER(iif(name='',authname,name)||addr),id;",
                     paramsv![
                         self_addr,
-                        DC_CONTACT_ID_LAST_SPECIAL as i32,
+                        DC_CONTACT_ID_LAST_SPECIAL,
                         Origin::IncomingReplyTo
                     ],
-                    |row| row.get::<_, i32>(0),
+                    |row| row.get::<_, ContactId>(0),
                     |ids| {
                         for id in ids {
-                            ret.push(id? as u32);
+                            ret.push(id?);
                         }
                         Ok(())
                     },
@@ -767,7 +829,7 @@ impl Contact {
     }
 
     /// Get blocked contacts.
-    pub async fn get_all_blocked(context: &Context) -> Result<Vec<u32>> {
+    pub async fn get_all_blocked(context: &Context) -> Result<Vec<ContactId>> {
         Contact::update_blocked_mailinglist_contacts(context)
             .await
             .context("cannot update blocked mailinglist contacts")?;
@@ -776,8 +838,8 @@ impl Contact {
             .sql
             .query_map(
                 "SELECT id FROM contacts WHERE id>? AND blocked!=0 ORDER BY LOWER(iif(name='',authname,name)||addr),id;",
-                paramsv![DC_CONTACT_ID_LAST_SPECIAL as i32],
-                |row| row.get::<_, u32>(0),
+                paramsv![DC_CONTACT_ID_LAST_SPECIAL],
+                |row| row.get::<_, ContactId>(0),
                 |ids| {
                     ids.collect::<std::result::Result<Vec<_>, _>>()
                         .map_err(Into::into)
@@ -792,7 +854,7 @@ impl Contact {
     /// This function returns a string explaining the encryption state
     /// of the contact and if the connection is encrypted the
     /// fingerprints of the keys involved.
-    pub async fn get_encrinfo(context: &Context, contact_id: u32) -> Result<String> {
+    pub async fn get_encrinfo(context: &Context, contact_id: ContactId) -> Result<String> {
         ensure!(
             contact_id > DC_CONTACT_ID_LAST_SPECIAL,
             "Can not provide encryption info for special contact"
@@ -861,7 +923,7 @@ impl Contact {
     /// possible as the contact is in use. In this case, the contact can be blocked.
     ///
     /// May result in a `#DC_EVENT_CONTACTS_CHANGED` event.
-    pub async fn delete(context: &Context, contact_id: u32) -> Result<()> {
+    pub async fn delete(context: &Context, contact_id: ContactId) -> Result<()> {
         ensure!(
             contact_id > DC_CONTACT_ID_LAST_SPECIAL,
             "Can not delete special contact"
@@ -871,17 +933,14 @@ impl Contact {
             .sql
             .count(
                 "SELECT COUNT(*) FROM chats_contacts WHERE contact_id=?;",
-                paramsv![contact_id as i32],
+                paramsv![contact_id],
             )
             .await?;
 
         if count_chats == 0 {
             match context
                 .sql
-                .execute(
-                    "DELETE FROM contacts WHERE id=?;",
-                    paramsv![contact_id as i32],
-                )
+                .execute("DELETE FROM contacts WHERE id=?;", paramsv![contact_id])
                 .await
             {
                 Ok(_) => {
@@ -907,7 +966,7 @@ impl Contact {
     /// For contact DC_CONTACT_ID_SELF (1), the function returns sth.
     /// like "Me" in the selected language and the email address
     /// defined by dc_set_config().
-    pub async fn get_by_id(context: &Context, contact_id: u32) -> Result<Contact> {
+    pub async fn get_by_id(context: &Context, contact_id: ContactId) -> Result<Contact> {
         let contact = Contact::load_from_db(context, contact_id).await?;
 
         Ok(contact)
@@ -919,7 +978,7 @@ impl Contact {
             .sql
             .execute(
                 "UPDATE contacts SET param=? WHERE id=?",
-                paramsv![self.param.to_string(), self.id as i32],
+                paramsv![self.param.to_string(), self.id],
             )
             .await?;
         Ok(())
@@ -931,14 +990,14 @@ impl Contact {
             .sql
             .execute(
                 "UPDATE contacts SET status=? WHERE id=?",
-                paramsv![self.status, self.id as i32],
+                paramsv![self.status, self.id],
             )
             .await?;
         Ok(())
     }
 
     /// Get the ID of the contact.
-    pub fn get_id(&self) -> u32 {
+    pub fn get_id(&self) -> ContactId {
         self.id
     }
 
@@ -1065,7 +1124,7 @@ impl Contact {
     pub async fn addr_equals_contact(
         context: &Context,
         addr: &str,
-        contact_id: u32,
+        contact_id: ContactId,
     ) -> Result<bool> {
         if addr.is_empty() {
             return Ok(false);
@@ -1091,13 +1150,13 @@ impl Contact {
             .sql
             .count(
                 "SELECT COUNT(*) FROM contacts WHERE id>?;",
-                paramsv![DC_CONTACT_ID_LAST_SPECIAL as i32],
+                paramsv![DC_CONTACT_ID_LAST_SPECIAL],
             )
             .await?;
         Ok(count)
     }
 
-    pub async fn real_exists_by_id(context: &Context, contact_id: u32) -> Result<bool> {
+    pub async fn real_exists_by_id(context: &Context, contact_id: ContactId) -> Result<bool> {
         if contact_id <= DC_CONTACT_ID_LAST_SPECIAL {
             return Ok(false);
         }
@@ -1106,7 +1165,7 @@ impl Contact {
             .sql
             .exists(
                 "SELECT COUNT(*) FROM contacts WHERE id=?;",
-                paramsv![contact_id as i32],
+                paramsv![contact_id],
             )
             .await?;
         Ok(exists)
@@ -1114,14 +1173,14 @@ impl Contact {
 
     pub async fn scaleup_origin_by_id(
         context: &Context,
-        contact_id: u32,
+        contact_id: ContactId,
         origin: Origin,
     ) -> Result<()> {
         context
             .sql
             .execute(
                 "UPDATE contacts SET origin=? WHERE id=? AND origin<?;",
-                paramsv![origin, contact_id as i32, origin],
+                paramsv![origin, contact_id, origin],
             )
             .await?;
         Ok(())
@@ -1165,7 +1224,11 @@ fn sanitize_name_and_addr(name: &str, addr: &str) -> (String, String) {
     }
 }
 
-async fn set_block_contact(context: &Context, contact_id: u32, new_blocking: bool) -> Result<()> {
+async fn set_block_contact(
+    context: &Context,
+    contact_id: ContactId,
+    new_blocking: bool,
+) -> Result<()> {
     ensure!(
         contact_id > DC_CONTACT_ID_LAST_SPECIAL,
         "Can't block special contact {}",
@@ -1179,7 +1242,7 @@ async fn set_block_contact(context: &Context, contact_id: u32, new_blocking: boo
             .sql
             .execute(
                 "UPDATE contacts SET blocked=? WHERE id=?;",
-                paramsv![new_blocking as i32, contact_id as i32],
+                paramsv![i32::from(new_blocking), contact_id],
             )
             .await?;
 
@@ -1210,7 +1273,8 @@ WHERE type=? AND id IN (
         // also unblock mailinglist
         // if the contact is a mailinglist address explicitly created to allow unblocking
         if !new_blocking && contact.origin == Origin::MailinglistAddress {
-            if let Some((chat_id, _, _)) = chat::get_chat_id_by_grpid(context, contact.addr).await?
+            if let Some((chat_id, _, _)) =
+                chat::get_chat_id_by_grpid(context, &contact.addr).await?
             {
                 chat_id.unblock(context).await?;
             }
@@ -1229,7 +1293,7 @@ WHERE type=? AND id IN (
 /// this typically happens if we see message with our own profile image, sent from another device.
 pub(crate) async fn set_profile_image(
     context: &Context,
-    contact_id: u32,
+    contact_id: ContactId,
     profile_image: &AvatarAction,
     was_encrypted: bool,
 ) -> Result<()> {
@@ -1276,7 +1340,7 @@ pub(crate) async fn set_profile_image(
 /// between Delta Chat devices.
 pub(crate) async fn set_status(
     context: &Context,
-    contact_id: u32,
+    contact_id: ContactId,
     status: String,
     encrypted: bool,
     has_chat_version: bool,
@@ -1302,7 +1366,7 @@ pub(crate) async fn set_status(
 /// Updates last seen timestamp of the contact if it is earlier than the given `timestamp`.
 pub(crate) async fn update_last_seen(
     context: &Context,
-    contact_id: u32,
+    contact_id: ContactId,
     timestamp: i64,
 ) -> Result<()> {
     ensure!(
@@ -1326,8 +1390,8 @@ pub(crate) async fn update_last_seen(
 /// - Trims the resulting string
 ///
 /// Typically, this function is not needed as it is called implicitly by `Contact::add_address_book`.
-pub fn normalize_name(full_name: impl AsRef<str>) -> String {
-    let full_name = full_name.as_ref().trim();
+pub fn normalize_name(full_name: &str) -> String {
+    let full_name = full_name.trim();
     if full_name.is_empty() {
         return full_name.into();
     }
@@ -1371,16 +1435,16 @@ impl Context {
     /// determine whether the specified addr maps to the/a self addr
     pub async fn is_self_addr(&self, addr: &str) -> Result<bool> {
         if let Some(self_addr) = self.get_config(Config::ConfiguredAddr).await? {
-            Ok(addr_cmp(self_addr, addr))
+            Ok(addr_cmp(&self_addr, addr))
         } else {
             Ok(false)
         }
     }
 }
 
-pub fn addr_cmp(addr1: impl AsRef<str>, addr2: impl AsRef<str>) -> bool {
-    let norm1 = addr_normalize(addr1.as_ref()).to_lowercase();
-    let norm2 = addr_normalize(addr2.as_ref()).to_lowercase();
+pub fn addr_cmp(addr1: &str, addr2: &str) -> bool {
+    let norm1 = addr_normalize(addr1).to_lowercase();
+    let norm2 = addr_normalize(addr2).to_lowercase();
 
     norm1 == norm2
 }
@@ -1473,7 +1537,7 @@ mod tests {
             Origin::IncomingReplyTo,
         )
         .await?;
-        assert_ne!(id, 0);
+        assert_ne!(id, ContactId::new(0));
 
         let contact = Contact::load_from_db(&context.ctx, id).await.unwrap();
         assert_eq!(contact.get_name(), "");
@@ -1525,9 +1589,9 @@ mod tests {
         let t = TestContext::new().await;
         assert_eq!(t.is_self_addr("me@me.org").await?, false);
 
-        let addr = t.configure_alice().await;
+        t.configure_addr("you@you.net").await;
         assert_eq!(t.is_self_addr("me@me.org").await?, false);
-        assert_eq!(t.is_self_addr(&addr).await?, true);
+        assert_eq!(t.is_self_addr("you@you.net").await?, true);
 
         Ok(())
     }
@@ -1628,7 +1692,7 @@ mod tests {
 
         // check SELF
         let contact = Contact::load_from_db(&t, DC_CONTACT_ID_SELF).await.unwrap();
-        assert_eq!(DC_CONTACT_ID_SELF, 1);
+        assert_eq!(DC_CONTACT_ID_SELF, ContactId::new(1));
         assert_eq!(contact.get_name(), stock_str::self_msg(&t).await);
         assert_eq!(contact.get_addr(), ""); // we're not configured
         assert!(!contact.is_blocked());
@@ -1890,7 +1954,7 @@ mod tests {
 
         let alice = TestContext::new_alice().await;
 
-        let id = Contact::lookup_id_by_addr(&alice.ctx, "alice@example.com", Origin::Unknown)
+        let id = Contact::lookup_id_by_addr(&alice.ctx, "alice@example.org", Origin::Unknown)
             .await
             .unwrap();
         assert_eq!(id, Some(DC_CONTACT_ID_SELF));
@@ -1915,7 +1979,7 @@ mod tests {
 
         let bob = TestContext::new_bob().await;
         let chat_alice = bob
-            .create_chat_with_contact("Alice", "alice@example.com")
+            .create_chat_with_contact("Alice", "alice@example.org")
             .await;
         send_text_msg(&bob, chat_alice.id, "Hello".to_string()).await?;
         let msg = bob.pop_sent_msg().await;
@@ -1927,7 +1991,7 @@ mod tests {
             "End-to-end encryption preferred.
 Fingerprints:
 
-alice@example.com:
+alice@example.org:
 2E6F A2CB 23B5 32D7 2863
 4B58 64B0 8F61 A9ED 9443
 
@@ -1978,7 +2042,7 @@ CCCB 5AA9 F6E1 141C 9431
 
         // Bob replies.
         let chat = bob
-            .create_chat_with_contact("Alice", "alice@example.com")
+            .create_chat_with_contact("Alice", "alice@example.org")
             .await;
 
         send_text_msg(&bob, chat.id, "Reply".to_string()).await?;
@@ -2029,12 +2093,12 @@ CCCB 5AA9 F6E1 141C 9431
 
         alice1
             .evtracker
-            .get_matching(|e| e == EventType::SelfavatarChanged)
+            .get_matching(|e| matches!(e, EventType::SelfavatarChanged))
             .await;
 
         // Bob sends a message so that Alice can encrypt to him.
         let chat = bob
-            .create_chat_with_contact("Alice", "alice@example.com")
+            .create_chat_with_contact("Alice", "alice@example.org")
             .await;
 
         send_text_msg(&bob, chat.id, "Reply".to_string()).await?;
@@ -2059,7 +2123,7 @@ CCCB 5AA9 F6E1 141C 9431
         assert!(alice2.get_config(Config::Selfavatar).await?.is_some());
         alice2
             .evtracker
-            .get_matching(|e| e == EventType::SelfavatarChanged)
+            .get_matching(|e| matches!(e, EventType::SelfavatarChanged))
             .await;
 
         Ok(())
@@ -2077,14 +2141,14 @@ CCCB 5AA9 F6E1 141C 9431
 
         let mime = br#"Subject: Hello
 Message-ID: message@example.net
-To: Alice <alice@example.com>
+To: Alice <alice@example.org>
 From: Bob <bob@example.net>
 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
 Chat-Version: 1.0
 Date: Sun, 22 Mar 2020 22:37:55 +0000
 
 Hi."#;
-        dc_receive_imf(&alice, mime, "Inbox", 1, false).await?;
+        dc_receive_imf(&alice, mime, "Inbox", false).await?;
         let msg = alice.get_last_msg().await;
 
         let timestamp = msg.get_timestamp();

src/context.rs

@@ -24,7 +24,6 @@ use crate::login_param::LoginParam;
 use crate::message::{self, MessageState, MsgId};
 use crate::quota::QuotaInfo;
 use crate::scheduler::Scheduler;
-use crate::securejoin::Bob;
 use crate::sql::Sql;
 
 #[derive(Clone, Debug)]
@@ -42,13 +41,9 @@ impl Deref for Context {
 
 #[derive(Debug)]
 pub struct InnerContext {
-    /// Database file path
-    pub(crate) dbfile: PathBuf,
     /// Blob directory path
     pub(crate) blobdir: PathBuf,
     pub(crate) sql: Sql,
-    pub(crate) os_name: Option<String>,
-    pub(crate) bob: Bob,
     pub(crate) last_smeared_timestamp: RwLock<i64>,
     pub(crate) running_state: RwLock<RunningState>,
     /// Mutex to avoid generating the key for the user more than once.
@@ -85,7 +80,7 @@ pub struct InnerContext {
 
 #[derive(Debug)]
 pub struct RunningState {
-    pub ongoing_running: bool,
+    ongoing_running: bool,
     shall_stop_ongoing: bool,
     cancel_sender: Option<Sender<()>>,
 }
@@ -107,10 +102,19 @@ pub fn get_info() -> BTreeMap<&'static str, String> {
 }
 
 impl Context {
-    /// Creates new context.
-    pub async fn new(os_name: String, dbfile: PathBuf, id: u32) -> Result<Context> {
-        // pretty_env_logger::try_init_timed().ok();
+    /// Creates new context and opens the database.
+    pub async fn new(dbfile: PathBuf, id: u32) -> Result<Context> {
+        let context = Self::new_closed(dbfile, id).await?;
 
+        // Open the database if is not encrypted.
+        if context.check_passphrase("".to_string()).await? {
+            context.sql.open(&context, "".to_string()).await?;
+        }
+        Ok(context)
+    }
+
+    /// Creates new context without opening the database.
+    pub async fn new_closed(dbfile: PathBuf, id: u32) -> Result<Context> {
         let mut blob_fname = OsString::new();
         blob_fname.push(dbfile.file_name().unwrap_or_default());
         blob_fname.push("-blobs");
@@ -118,11 +122,38 @@ impl Context {
         if !blobdir.exists().await {
             async_std::fs::create_dir_all(&blobdir).await?;
         }
-        Context::with_blobdir(os_name, dbfile, blobdir, id).await
+        let context = Context::with_blobdir(dbfile, blobdir, id).await?;
+        Ok(context)
+    }
+
+    /// Opens the database with the given passphrase.
+    ///
+    /// Returns true if passphrase is correct, false is passphrase is not correct. Fails on other
+    /// errors.
+    pub async fn open(&self, passphrase: String) -> Result<bool> {
+        if self.sql.check_passphrase(passphrase.clone()).await? {
+            self.sql.open(self, passphrase).await?;
+            Ok(true)
+        } else {
+            Ok(false)
+        }
+    }
+
+    /// Returns true if database is open.
+    pub async fn is_open(&self) -> bool {
+        self.sql.is_open().await
+    }
+
+    /// Tests the database passphrase.
+    ///
+    /// Returns true if passphrase is correct.
+    ///
+    /// Fails if database is already open.
+    pub(crate) async fn check_passphrase(&self, passphrase: String) -> Result<bool> {
+        self.sql.check_passphrase(passphrase).await
     }
 
     pub(crate) async fn with_blobdir(
-        os_name: String,
         dbfile: PathBuf,
         blobdir: PathBuf,
         id: u32,
@@ -136,11 +167,8 @@ impl Context {
         let inner = InnerContext {
             id,
             blobdir,
-            dbfile,
-            os_name: Some(os_name),
             running_state: RwLock::new(Default::default()),
-            sql: Sql::new(),
-            bob: Default::default(),
+            sql: Sql::new(dbfile),
             last_smeared_timestamp: RwLock::new(0),
             generating_key_mutex: Mutex::new(()),
             oauth2_mutex: Mutex::new(()),
@@ -158,7 +186,6 @@ impl Context {
         let ctx = Context {
             inner: Arc::new(inner),
         };
-        ctx.sql.open(&ctx, &ctx.dbfile, false).await?;
 
         Ok(ctx)
     }
@@ -196,7 +223,7 @@ impl Context {
 
     /// Returns database file path.
     pub fn get_dbfile(&self) -> &Path {
-        self.dbfile.as_path()
+        self.sql.dbfile.as_path()
     }
 
     /// Returns blob directory path.
@@ -227,7 +254,7 @@ impl Context {
 
     // Ongoing process allocation/free/check
 
-    pub async fn alloc_ongoing(&self) -> Result<Receiver<()>> {
+    pub(crate) async fn alloc_ongoing(&self) -> Result<Receiver<()>> {
         if self.has_ongoing().await {
             bail!("There is already another ongoing process running.");
         }
@@ -243,7 +270,7 @@ impl Context {
         Ok(receiver)
     }
 
-    pub async fn free_ongoing(&self) {
+    pub(crate) async fn free_ongoing(&self) {
         let s_a = &self.running_state;
         let mut s = s_a.write().await;
 
@@ -252,7 +279,7 @@ impl Context {
         s.cancel_sender.take();
     }
 
-    pub async fn has_ongoing(&self) -> bool {
+    pub(crate) async fn has_ongoing(&self) -> bool {
         let s_a = &self.running_state;
         let s = s_a.read().await;
 
@@ -277,7 +304,7 @@ impl Context {
         };
     }
 
-    pub async fn shall_stop_ongoing(&self) -> bool {
+    pub(crate) async fn shall_stop_ongoing(&self) -> bool {
         self.running_state.read().await.shall_stop_ongoing
     }
 
@@ -307,7 +334,6 @@ impl Context {
             .await?
             .unwrap_or_else(|| "unknown".to_string());
         let e2ee_enabled = self.get_config_int(Config::E2eeEnabled).await?;
-        let e2ee_force = self.get_config_int(Config::E2eeForce).await?;
         let mdns_enabled = self.get_config_int(Config::MdnsEnabled).await?;
         let bcc_self = self.get_config_int(Config::BccSelf).await?;
         let send_sync_msgs = self.get_config_int(Config::SendSyncMsgs).await?;
@@ -326,11 +352,9 @@ impl Context {
             Err(err) => format!("<key failure: {}>", err),
         };
 
-        let inbox_watch = self.get_config_int(Config::InboxWatch).await?;
         let sentbox_watch = self.get_config_int(Config::SentboxWatch).await?;
-        let mvbox_watch = self.get_config_int(Config::MvboxWatch).await?;
         let mvbox_move = self.get_config_int(Config::MvboxMove).await?;
-        let sentbox_move = self.get_config_int(Config::SentboxMove).await?;
+        let only_fetch_mvbox = self.get_config_int(Config::OnlyFetchMvbox).await?;
         let folders_configured = self
             .sql
             .get_raw_config_int("folders_configured")
@@ -356,6 +380,13 @@ impl Context {
         res.insert("number_of_contacts", contacts.to_string());
         res.insert("database_dir", self.get_dbfile().display().to_string());
         res.insert("database_version", dbversion.to_string());
+        res.insert(
+            "database_encrypted",
+            self.sql
+                .is_encrypted()
+                .await
+                .map_or_else(|| "closed".to_string(), |b| b.to_string()),
+        );
         res.insert("journal_mode", journal_mode);
         res.insert("blobdir", self.get_blobdir().display().to_string());
         res.insert("display_name", displayname.unwrap_or_else(|| unset.into()));
@@ -385,17 +416,14 @@ impl Context {
                 .await?
                 .to_string(),
         );
-        res.insert("inbox_watch", inbox_watch.to_string());
         res.insert("sentbox_watch", sentbox_watch.to_string());
-        res.insert("mvbox_watch", mvbox_watch.to_string());
         res.insert("mvbox_move", mvbox_move.to_string());
-        res.insert("sentbox_move", sentbox_move.to_string());
+        res.insert("only_fetch_mvbox", only_fetch_mvbox.to_string());
         res.insert("folders_configured", folders_configured.to_string());
         res.insert("configured_sentbox_folder", configured_sentbox_folder);
         res.insert("configured_mvbox_folder", configured_mvbox_folder);
         res.insert("mdns_enabled", mdns_enabled.to_string());
         res.insert("e2ee_enabled", e2ee_enabled.to_string());
-        res.insert("e2ee_force", e2ee_force.to_string());
         res.insert(
             "key_gen_type",
             self.get_config_int(Config::KeyGenType).await?.to_string(),
@@ -583,14 +611,14 @@ impl Context {
         Ok(spam.as_deref() == Some(folder_name))
     }
 
-    pub fn derive_blobdir(dbfile: &PathBuf) -> PathBuf {
+    pub(crate) fn derive_blobdir(dbfile: &PathBuf) -> PathBuf {
         let mut blob_fname = OsString::new();
         blob_fname.push(dbfile.file_name().unwrap_or_default());
         blob_fname.push("-blobs");
         dbfile.with_file_name(blob_fname)
     }
 
-    pub fn derive_walfile(dbfile: &PathBuf) -> PathBuf {
+    pub(crate) fn derive_walfile(dbfile: &PathBuf) -> PathBuf {
         let mut wal_fname = OsString::new();
         wal_fname.push(dbfile.file_name().unwrap_or_default());
         wal_fname.push("-wal");
@@ -642,21 +670,26 @@ mod tests {
     use crate::chat::{
         get_chat_contacts, get_chat_msgs, send_msg, set_muted, Chat, ChatId, MuteDuration,
     };
-    use crate::constants::{Viewtype, DC_CONTACT_ID_SELF};
+    use crate::constants::DC_CONTACT_ID_SELF;
     use crate::dc_receive_imf::dc_receive_imf;
     use crate::dc_tools::dc_create_outgoing_rfc724_mid;
-    use crate::message::Message;
+    use crate::message::{Message, Viewtype};
     use crate::test_utils::TestContext;
+    use anyhow::Context as _;
     use std::time::Duration;
     use strum::IntoEnumIterator;
+    use tempfile::tempdir;
 
     #[async_std::test]
-    async fn test_wrong_db() {
-        let tmp = tempfile::tempdir().unwrap();
+    async fn test_wrong_db() -> Result<()> {
+        let tmp = tempfile::tempdir()?;
         let dbfile = tmp.path().join("db.sqlite");
-        std::fs::write(&dbfile, b"123").unwrap();
-        let res = Context::new("FakeOs".into(), dbfile.into(), 1).await;
-        assert!(res.is_err());
+        std::fs::write(&dbfile, b"123")?;
+        let res = Context::new(dbfile.into(), 1).await?;
+
+        // Broken database is indistinguishable from encrypted one.
+        assert_eq!(res.is_open().await, false);
+        Ok(())
     }
 
     #[async_std::test]
@@ -673,7 +706,7 @@ mod tests {
             .unwrap();
         let msg = format!(
             "From: {}\n\
-             To: alice@example.com\n\
+             To: alice@example.org\n\
              Message-ID: <{}>\n\
              Chat-Version: 1.0\n\
              Date: Sun, 22 Mar 2020 22:37:57 +0000\n\
@@ -683,7 +716,7 @@ mod tests {
             dc_create_outgoing_rfc724_mid(None, contact.get_addr())
         );
         println!("{}", msg);
-        dc_receive_imf(t, msg.as_bytes(), "INBOX", 1, false)
+        dc_receive_imf(t, msg.as_bytes(), "INBOX", false)
             .await
             .unwrap();
     }
@@ -806,9 +839,7 @@ mod tests {
     async fn test_blobdir_exists() {
         let tmp = tempfile::tempdir().unwrap();
         let dbfile = tmp.path().join("db.sqlite");
-        Context::new("FakeOS".into(), dbfile.into(), 1)
-            .await
-            .unwrap();
+        Context::new(dbfile.into(), 1).await.unwrap();
         let blobdir = tmp.path().join("db.sqlite-blobs");
         assert!(blobdir.is_dir());
     }
@@ -819,7 +850,7 @@ mod tests {
         let dbfile = tmp.path().join("db.sqlite");
         let blobdir = tmp.path().join("db.sqlite-blobs");
         std::fs::write(&blobdir, b"123").unwrap();
-        let res = Context::new("FakeOS".into(), dbfile.into(), 1).await;
+        let res = Context::new(dbfile.into(), 1).await;
         assert!(res.is_err());
     }
 
@@ -829,9 +860,7 @@ mod tests {
         let subdir = tmp.path().join("subdir");
         let dbfile = subdir.join("db.sqlite");
         let dbfile2 = dbfile.clone();
-        Context::new("FakeOS".into(), dbfile.into(), 1)
-            .await
-            .unwrap();
+        Context::new(dbfile.into(), 1).await.unwrap();
         assert!(subdir.is_dir());
         assert!(dbfile2.is_file());
     }
@@ -841,7 +870,7 @@ mod tests {
         let tmp = tempfile::tempdir().unwrap();
         let dbfile = tmp.path().join("db.sqlite");
         let blobdir = PathBuf::new();
-        let res = Context::with_blobdir("FakeOS".into(), dbfile.into(), blobdir, 1).await;
+        let res = Context::with_blobdir(dbfile.into(), blobdir, 1).await;
         assert!(res.is_err());
     }
 
@@ -850,7 +879,7 @@ mod tests {
         let tmp = tempfile::tempdir().unwrap();
         let dbfile = tmp.path().join("db.sqlite");
         let blobdir = tmp.path().join("blobs");
-        let res = Context::with_blobdir("FakeOS".into(), dbfile.into(), blobdir.into(), 1).await;
+        let res = Context::with_blobdir(dbfile.into(), blobdir.into(), 1).await;
         assert!(res.is_err());
     }
 
@@ -1012,4 +1041,29 @@ mod tests {
 
         Ok(())
     }
+
+    #[async_std::test]
+    async fn test_check_passphrase() -> Result<()> {
+        let dir = tempdir()?;
+        let dbfile = dir.path().join("db.sqlite");
+
+        let id = 1;
+        let context = Context::new_closed(dbfile.clone().into(), id)
+            .await
+            .context("failed to create context")?;
+        assert_eq!(context.open("foo".to_string()).await?, true);
+        assert_eq!(context.is_open().await, true);
+        drop(context);
+
+        let id = 2;
+        let context = Context::new(dbfile.into(), id)
+            .await
+            .context("failed to create context")?;
+        assert_eq!(context.is_open().await, false);
+        assert_eq!(context.check_passphrase("bar".to_string()).await?, false);
+        assert_eq!(context.open("false".to_string()).await?, false);
+        assert_eq!(context.open("foo".to_string()).await?, true);
+
+        Ok(())
+    }
 }

src/dc_tools.rs

@@ -5,6 +5,7 @@ use core::cmp::{max, min};
 use std::borrow::Cow;
 use std::fmt;
 use std::io::Cursor;
+use std::str::from_utf8;
 use std::str::FromStr;
 use std::time::{Duration, SystemTime};
 
@@ -12,15 +13,18 @@ use async_std::path::{Path, PathBuf};
 use async_std::prelude::*;
 use async_std::{fs, io};
 
-use anyhow::{bail, Error};
+use anyhow::Error;
 use chrono::{Local, TimeZone};
+use mailparse::dateparse;
+use mailparse::headers::Headers;
+use mailparse::MailHeaderMap;
 use rand::{thread_rng, Rng};
 
 use crate::chat::{add_device_msg, add_device_msg_with_importance};
-use crate::constants::{Viewtype, DC_ELLIPSIS, DC_OUTDATED_WARNING_DAYS};
+use crate::constants::{DC_ELLIPSIS, DC_OUTDATED_WARNING_DAYS};
 use crate::context::Context;
 use crate::events::EventType;
-use crate::message::Message;
+use crate::message::{Message, Viewtype};
 use crate::provider::get_provider_update_timestamp;
 use crate::stock_str;
 
@@ -67,7 +71,7 @@ pub(crate) fn dc_gm2local_offset() -> i64 {
     /* returns the offset that must be _added_ to an UTC/GMT-time to create the localtime.
     the function may return negative values. */
     let lt = Local::now();
-    lt.offset().local_minus_utc() as i64
+    i64::from(lt.offset().local_minus_utc())
 }
 
 // timesmearing
@@ -191,44 +195,26 @@ async fn maybe_warn_on_outdated(context: &Context, now: i64, approx_compile_time
 }
 
 /* Message-ID tools */
+
+/// Generate an ID. The generated ID should be as short and as unique as possible:
+/// - short, because it may also used as part of Message-ID headers or in QR codes
+/// - unique as two IDs generated on two devices should not be the same. However, collisions are not world-wide but only by the few contacts.
+/// IDs generated by this function are 66 bit wide and are returned as 11 base64 characters.
+///
+/// Additional information when used as a message-id or group-id:
+/// - for OUTGOING messages this ID is written to the header as `Chat-Group-ID:` and is added to the message ID as Gr.<grpid>.<random>@<random>
+/// - for INCOMING messages, the ID is taken from the Chat-Group-ID-header or from the Message-ID in the In-Reply-To: or References:-Header
+/// - the group-id should be a string with the characters [a-zA-Z0-9\-_]
 pub(crate) fn dc_create_id() -> String {
-    /* generate an id. the generated ID should be as short and as unique as possible:
-    - short, because it may also used as part of Message-ID headers or in QR codes
-    - unique as two IDs generated on two devices should not be the same. However, collisions are not world-wide but only by the few contacts.
-    IDs generated by this function are 66 bit wide and are returned as 11 base64 characters.
-    If possible, RNG of OpenSSL is used.
-
-    Additional information when used as a message-id or group-id:
-    - for OUTGOING messages this ID is written to the header as `Chat-Group-ID:` and is added to the message ID as Gr.<grpid>.<random>@<random>
-    - for INCOMING messages, the ID is taken from the Chat-Group-ID-header or from the Message-ID in the In-Reply-To: or References:-Header
-    - the group-id should be a string with the characters [a-zA-Z0-9\-_] */
-
+    // ThreadRng implements CryptoRng trait and is supposed to be cryptographically secure.
     let mut rng = thread_rng();
-    let buf: [u32; 3] = [rng.gen(), rng.gen(), rng.gen()];
 
-    encode_66bits_as_base64(buf[0usize], buf[1usize], buf[2usize])
-}
+    // Generate 72 random bits.
+    let mut arr = [0u8; 9];
+    rng.fill(&mut arr[..]);
 
-/// Encode 66 bits as a base64 string.
-/// This is useful for ID generating with short strings as we save 5 character
-/// in each id compared to 64 bit hex encoding. For a typical group ID, these
-/// are 10 characters (grpid+msgid):
-///    hex:    64 bit, 4 bits/character, length = 64/4 = 16 characters
-///    base64: 64 bit, 6 bits/character, length = 64/6 = 11 characters (plus 2 additional bits)
-/// Only the lower 2 bits of `fill` are used.
-fn encode_66bits_as_base64(v1: u32, v2: u32, fill: u32) -> String {
-    use byteorder::{BigEndian, WriteBytesExt};
-
-    let mut wrapped_writer = Vec::new();
-    {
-        let mut enc = base64::write::EncoderWriter::new(&mut wrapped_writer, base64::URL_SAFE);
-        enc.write_u32::<BigEndian>(v1).unwrap();
-        enc.write_u32::<BigEndian>(v2).unwrap();
-        enc.write_u8(((fill & 0x3) as u8) << 6).unwrap();
-        enc.finish().unwrap();
-    }
-    assert_eq!(wrapped_writer.pop(), Some(b'A')); // Remove last "A"
-    String::from_utf8(wrapped_writer).unwrap()
+    // Take 11 base64 characters containing 66 random bits.
+    base64::encode(&arr).chars().take(11).collect()
 }
 
 /// Function generates a Message-ID that can be used for a new outgoing message.
@@ -350,63 +336,6 @@ pub async fn dc_delete_files_in_dir(context: &Context, path: impl AsRef<Path>) {
     }
 }
 
-pub(crate) async fn dc_copy_file(
-    context: &Context,
-    src_path: impl AsRef<Path>,
-    dest_path: impl AsRef<Path>,
-) -> bool {
-    let src_abs = dc_get_abs_path(context, &src_path);
-    let mut src_file = match fs::File::open(&src_abs).await {
-        Ok(file) => file,
-        Err(err) => {
-            warn!(
-                context,
-                "failed to open for read '{}': {}",
-                src_abs.display(),
-                err
-            );
-            return false;
-        }
-    };
-
-    let dest_abs = dc_get_abs_path(context, &dest_path);
-    let mut dest_file = match fs::OpenOptions::new()
-        .create_new(true)
-        .write(true)
-        .open(&dest_abs)
-        .await
-    {
-        Ok(file) => file,
-        Err(err) => {
-            warn!(
-                context,
-                "failed to open for write '{}': {}",
-                dest_abs.display(),
-                err
-            );
-            return false;
-        }
-    };
-
-    match io::copy(&mut src_file, &mut dest_file).await {
-        Ok(_) => true,
-        Err(err) => {
-            error!(
-                context,
-                "Cannot copy \"{}\" to \"{}\": {}",
-                src_abs.display(),
-                dest_abs.display(),
-                err
-            );
-            {
-                // Attempt to remove the failed file, swallow errors resulting from that.
-                fs::remove_file(dest_abs).await.ok();
-            }
-            false
-        }
-    }
-}
-
 pub(crate) async fn dc_create_folder(
     context: &Context,
     path: impl AsRef<Path>,
@@ -504,33 +433,6 @@ pub fn dc_open_file_std<P: AsRef<std::path::Path>>(
     }
 }
 
-/// Returns Ok((temp_path, dest_path)) on success. The backup can then be written to temp_path. If the backup succeeded,
-/// it can be renamed to dest_path. This guarantees that the backup is complete.
-pub(crate) async fn get_next_backup_path(
-    folder: impl AsRef<Path>,
-    backup_time: i64,
-) -> Result<(PathBuf, PathBuf), Error> {
-    let folder = PathBuf::from(folder.as_ref());
-    let stem = chrono::NaiveDateTime::from_timestamp(backup_time, 0)
-        // Don't change this file name format, in has_backup() we use string comparison to determine which backup is newer:
-        .format("delta-chat-backup-%Y-%m-%d")
-        .to_string();
-
-    // 64 backup files per day should be enough for everyone
-    for i in 0..64 {
-        let mut tempfile = folder.clone();
-        tempfile.push(format!("{}-{:02}.tar.part", stem, i));
-
-        let mut destfile = folder.clone();
-        destfile.push(format!("{}-{:02}.tar", stem, i));
-
-        if !tempfile.exists().await && !destfile.exists().await {
-            return Ok((tempfile, destfile));
-        }
-    }
-    bail!("could not create backup file, disk full?");
-}
-
 pub(crate) fn time() -> i64 {
     SystemTime::now()
         .duration_since(SystemTime::UNIX_EPOCH)
@@ -632,8 +534,8 @@ impl rusqlite::types::ToSql for EmailAddress {
 /// Makes sure that a user input that is not supposed to contain newlines does not contain newlines.
 pub(crate) fn improve_single_line_input(input: &str) -> String {
     input
-        .replace("\n", " ")
-        .replace("\r", " ")
+        .replace('\n', " ")
+        .replace('\r', " ")
         .trim()
         .to_string()
 }
@@ -670,13 +572,144 @@ pub fn remove_subject_prefix(last_subject: &str) -> String {
         .to_string()
 }
 
+// Types and methods to create hop-info for message-info
+
+fn extract_address_from_receive_header<'a>(header: &'a str, start: &str) -> Option<&'a str> {
+    let header_len = header.len();
+    header.find(start).and_then(|mut begin| {
+        begin += start.len();
+        let end = header
+            .get(begin..)?
+            .find(|c: char| c.is_whitespace())
+            .unwrap_or(header_len);
+        header.get(begin..begin + end)
+    })
+}
+
+pub(crate) fn parse_receive_header(header: &str) -> String {
+    let header = header.replace(&['\r', '\n'][..], "");
+    let mut hop_info = String::from("Hop: ");
+
+    if let Some(from) = extract_address_from_receive_header(&header, "from ") {
+        hop_info += &format!("From: {}; ", from.trim());
+    }
+
+    if let Some(by) = extract_address_from_receive_header(&header, "by ") {
+        hop_info += &format!("By: {}; ", by.trim());
+    }
+
+    if let Ok(date) = dateparse(&header) {
+        // In tests, use the UTC timezone so that the test is reproducible
+        #[cfg(test)]
+        let date_obj = chrono::Utc.timestamp(date, 0);
+        #[cfg(not(test))]
+        let date_obj = Local.timestamp(date, 0);
+
+        hop_info += &format!("Date: {}", date_obj.to_rfc2822());
+    };
+
+    hop_info
+}
+
+/// parses "receive"-headers
+pub(crate) fn parse_receive_headers(headers: &Headers) -> String {
+    headers
+        .get_all_headers("Received")
+        .iter()
+        .rev()
+        .filter_map(|header_map_item| from_utf8(header_map_item.get_value_raw()).ok())
+        .map(parse_receive_header)
+        .collect::<Vec<_>>()
+        .join("\n")
+}
+
 #[cfg(test)]
 mod tests {
     #![allow(clippy::indexing_slicing)]
 
     use super::*;
 
-    use crate::test_utils::TestContext;
+    use crate::{
+        config::Config, dc_receive_imf::dc_receive_imf, message::get_msg_info,
+        test_utils::TestContext,
+    };
+
+    #[test]
+    fn test_parse_receive_headers() {
+        // Test `parse_receive_headers()` with some more-or-less random emails from the test-data
+        let raw = include_bytes!("../test-data/message/mail_with_cc.txt");
+        let expected =
+            "Hop: From: localhost; By: hq5.merlinux.eu; Date: Sat, 14 Sep 2019 17:00:22 +0000\n\
+             Hop: From: hq5.merlinux.eu; By: hq5.merlinux.eu; Date: Sat, 14 Sep 2019 17:00:25 +0000";
+        check_parse_receive_headers(raw, expected);
+
+        let raw = include_bytes!("../test-data/message/wrong-html.eml");
+        let expected =
+            "Hop: From: oxbsltgw18.schlund.de; By: mrelayeu.kundenserver.de; Date: Thu, 06 Aug 2020 16:40:31 +0000\n\
+             Hop: From: mout.kundenserver.de; By: dd37930.kasserver.com; Date: Thu, 06 Aug 2020 16:40:32 +0000";
+        check_parse_receive_headers(raw, expected);
+
+        let raw = include_bytes!("../test-data/message/posteo_ndn.eml");
+        let expected =
+            "Hop: By: mout01.posteo.de; Date: Tue, 09 Jun 2020 18:44:22 +0000\n\
+             Hop: From: mout01.posteo.de; By: mx04.posteo.de; Date: Tue, 09 Jun 2020 18:44:22 +0000\n\
+             Hop: From: mx04.posteo.de; By: mailin06.posteo.de; Date: Tue, 09 Jun 2020 18:44:23 +0000\n\
+             Hop: From: mailin06.posteo.de; By: proxy02.posteo.de; Date: Tue, 09 Jun 2020 18:44:23 +0000\n\
+             Hop: From: proxy02.posteo.de; By: proxy02.posteo.name; Date: Tue, 09 Jun 2020 18:44:23 +0000\n\
+             Hop: From: proxy02.posteo.name; By: dovecot03.posteo.local; Date: Tue, 09 Jun 2020 18:44:24 +0000";
+        check_parse_receive_headers(raw, expected);
+    }
+
+    fn check_parse_receive_headers(raw: &[u8], expected: &str) {
+        let mail = mailparse::parse_mail(raw).unwrap();
+        let hop_info = parse_receive_headers(&mail.get_headers());
+        assert_eq!(hop_info, expected)
+    }
+
+    #[async_std::test]
+    async fn test_parse_receive_headers_integration() {
+        let raw = include_bytes!("../test-data/message/mail_with_cc.txt");
+        let expected = r"State: Fresh
+
+hi
+
+Message-ID: 2dfdbde7@example.org
+
+Hop: From: localhost; By: hq5.merlinux.eu; Date: Sat, 14 Sep 2019 17:00:22 +0000
+Hop: From: hq5.merlinux.eu; By: hq5.merlinux.eu; Date: Sat, 14 Sep 2019 17:00:25 +0000";
+        check_parse_receive_headers_integration(raw, expected).await;
+
+        let raw = include_bytes!("../test-data/message/encrypted_with_received_headers.eml");
+        let expected = "State: Fresh, Encrypted
+
+Re: Message from alice@example.org
+
+hi back\r\n\
+\r\n\
+-- \r\n\
+Sent with my Delta Chat Messenger: https://delta.chat
+
+Message-ID: Mr.adQpEwndXLH.LPDdlFVJ7wG@example.net
+
+Hop: From: [127.0.0.1]; By: mail.example.org; Date: Mon, 27 Dec 2021 11:21:21 +0000
+Hop: From: mout.example.org; By: hq5.example.org; Date: Mon, 27 Dec 2021 11:21:22 +0000
+Hop: From: hq5.example.org; By: hq5.example.org; Date: Mon, 27 Dec 2021 11:21:22 +0000";
+        check_parse_receive_headers_integration(raw, expected).await;
+    }
+
+    async fn check_parse_receive_headers_integration(raw: &[u8], expected: &str) {
+        let t = TestContext::new_alice().await;
+        t.set_config(Config::ShowEmails, Some("2")).await.unwrap();
+        dc_receive_imf(&t, raw, "INBOX", false).await.unwrap();
+        let msg = t.get_last_msg().await;
+        let msg_info = get_msg_info(&t, msg.id).await.unwrap();
+
+        // Ignore the first rows of the msg_info because they contain a
+        // received time that depends on the test time which makes it impossible to
+        // compare with a static string
+        let capped_result = &msg_info[msg_info.find("State").unwrap()..];
+        assert_eq!(expected, capped_result);
+    }
 
     #[test]
     fn test_rust_ftoa() {
@@ -729,26 +762,6 @@ mod tests {
         assert_eq!(buf.len(), 11);
     }
 
-    #[test]
-    fn test_encode_66bits_as_base64() {
-        assert_eq!(
-            encode_66bits_as_base64(0x01234567, 0x89abcdef, 0),
-            "ASNFZ4mrze8"
-        );
-        assert_eq!(
-            encode_66bits_as_base64(0x01234567, 0x89abcdef, 1),
-            "ASNFZ4mrze9"
-        );
-        assert_eq!(
-            encode_66bits_as_base64(0x01234567, 0x89abcdef, 2),
-            "ASNFZ4mrze-"
-        );
-        assert_eq!(
-            encode_66bits_as_base64(0x01234567, 0x89abcdef, 3),
-            "ASNFZ4mrze_"
-        );
-    }
-
     #[test]
     fn test_dc_extract_grpid_from_rfc724_mid() {
         // Should return None if we pass invalid mid
@@ -890,20 +903,7 @@ mod tests {
 
         assert!(dc_file_exist!(context, &abs_path).await);
 
-        assert!(dc_copy_file(context, "$BLOBDIR/foobar", "$BLOBDIR/dada").await);
-
-        // attempting to copy a second time should fail
-        assert!(!dc_copy_file(context, "$BLOBDIR/foobar", "$BLOBDIR/dada").await);
-
-        assert_eq!(dc_get_filebytes(context, "$BLOBDIR/dada").await, 7);
-
-        let buf = dc_read_file(context, "$BLOBDIR/dada").await.unwrap();
-
-        assert_eq!(buf.len(), 7);
-        assert_eq!(&buf, b"content");
-
         assert!(dc_delete_file(context, "$BLOBDIR/foobar").await);
-        assert!(dc_delete_file(context, "$BLOBDIR/dada").await);
         assert!(dc_create_folder(context, "$BLOBDIR/foobar-folder")
             .await
             .is_ok());
@@ -1029,7 +1029,7 @@ mod tests {
         maybe_warn_on_bad_time(&t, timestamp_past, get_provider_update_timestamp()).await;
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
         assert_eq!(chats.len(), 1);
-        let device_chat_id = chats.get_chat_id(0);
+        let device_chat_id = chats.get_chat_id(0).unwrap();
         let msgs = chat::get_chat_msgs(&t, device_chat_id, 0, None)
             .await
             .unwrap();
@@ -1067,7 +1067,7 @@ mod tests {
         .await;
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
         assert_eq!(chats.len(), 1);
-        assert_eq!(device_chat_id, chats.get_chat_id(0));
+        assert_eq!(device_chat_id, chats.get_chat_id(0).unwrap());
         let msgs = chat::get_chat_msgs(&t, device_chat_id, 0, None)
             .await
             .unwrap();
@@ -1099,7 +1099,7 @@ mod tests {
         .await;
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
         assert_eq!(chats.len(), 1);
-        let device_chat_id = chats.get_chat_id(0);
+        let device_chat_id = chats.get_chat_id(0).unwrap();
         let msgs = chat::get_chat_msgs(&t, device_chat_id, 0, None)
             .await
             .unwrap();
@@ -1121,7 +1121,7 @@ mod tests {
         .await;
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
         assert_eq!(chats.len(), 1);
-        let device_chat_id = chats.get_chat_id(0);
+        let device_chat_id = chats.get_chat_id(0).unwrap();
         let msgs = chat::get_chat_msgs(&t, device_chat_id, 0, None)
             .await
             .unwrap();
@@ -1138,7 +1138,7 @@ mod tests {
         .await;
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
         assert_eq!(chats.len(), 1);
-        let device_chat_id = chats.get_chat_id(0);
+        let device_chat_id = chats.get_chat_id(0).unwrap();
         let msgs = chat::get_chat_msgs(&t, device_chat_id, 0, None)
             .await
             .unwrap();

src/dehtml.rs

@@ -36,9 +36,9 @@ impl Dehtml {
             ""
         }
     }
-    fn append_prefix(&self, line_end: impl AsRef<str>) -> String {
+    fn append_prefix(&self, line_end: &str) -> String {
         // line_end is e.g. "\n\n". We add "> " if necessary.
-        line_end.as_ref().to_owned() + self.line_prefix()
+        line_end.to_string() + self.line_prefix()
     }
     fn get_add_text(&self) -> AddText {
         if self.divs_since_quote_div > 0 && self.divs_since_quoted_content_div == 0 {

src/download.rs

@@ -3,14 +3,14 @@
 use anyhow::{anyhow, Result};
 use deltachat_derive::{FromSql, ToSql};
 use serde::{Deserialize, Serialize};
+use std::collections::BTreeMap;
 
 use crate::config::Config;
-use crate::constants::Viewtype;
 use crate::context::Context;
 use crate::dc_tools::time;
 use crate::imap::{Imap, ImapActionResult};
 use crate::job::{self, Action, Job, Status};
-use crate::message::{Message, MsgId};
+use crate::message::{Message, MsgId, Viewtype};
 use crate::mimeparser::{MimeMessage, Part};
 use crate::param::Params;
 use crate::{job_try, stock_str, EventType};
@@ -129,24 +129,48 @@ impl Job {
         }
 
         let msg = job_try!(Message::load_from_db(context, MsgId::new(self.foreign_id)).await);
-        let server_folder = msg.server_folder.unwrap_or_default();
-        match imap
-            .fetch_single_msg(context, &server_folder, msg.server_uid)
-            .await
-        {
-            ImapActionResult::RetryLater | ImapActionResult::Failed => {
-                job_try!(
-                    msg.id
-                        .update_download_state(context, DownloadState::Failure)
-                        .await
-                );
-                Status::Finished(Err(anyhow!("Call download_full() again to try over.")))
-            }
-            ImapActionResult::Success | ImapActionResult::AlreadyDone => {
-                // update_download_state() not needed as receive_imf() already
-                // set the state and emitted the event.
-                Status::Finished(Ok(()))
+        let row = job_try!(
+            context
+                .sql
+                .query_row_optional(
+                    "SELECT uid, folder FROM imap WHERE rfc724_mid=? AND target!=''",
+                    paramsv![msg.rfc724_mid],
+                    |row| {
+                        let server_uid: u32 = row.get(0)?;
+                        let server_folder: String = row.get(1)?;
+                        Ok((server_uid, server_folder))
+                    }
+                )
+                .await
+        );
+
+        if let Some((server_uid, server_folder)) = row {
+            match imap
+                .fetch_single_msg(context, &server_folder, server_uid, msg.rfc724_mid.clone())
+                .await
+            {
+                ImapActionResult::RetryLater | ImapActionResult::Failed => {
+                    job_try!(
+                        msg.id
+                            .update_download_state(context, DownloadState::Failure)
+                            .await
+                    );
+                    Status::Finished(Err(anyhow!("Call download_full() again to try over.")))
+                }
+                ImapActionResult::Success => {
+                    // update_download_state() not needed as receive_imf() already
+                    // set the state and emitted the event.
+                    Status::Finished(Ok(()))
+                }
             }
+        } else {
+            // No IMAP record found, we don't know the UID and folder.
+            job_try!(
+                msg.id
+                    .update_download_state(context, DownloadState::Failure)
+                    .await
+            );
+            Status::Finished(Err(anyhow!("Call download_full() again to try over.")))
         }
     }
 }
@@ -161,6 +185,7 @@ impl Imap {
         context: &Context,
         folder: &str,
         uid: u32,
+        rfc724_mid: String,
     ) -> ImapActionResult {
         if let Some(imapresult) = self
             .prepare_imap_operation_on_msg(context, folder, uid)
@@ -172,14 +197,20 @@ impl Imap {
         // we are connected, and the folder is selected
         info!(context, "Downloading message {}/{} fully...", folder, uid);
 
-        let (_, error_cnt) = self
-            .fetch_many_msgs(context, folder, vec![uid], false, false)
-            .await;
-        if error_cnt > 0 {
-            return ImapActionResult::Failed;
+        let mut uid_message_ids: BTreeMap<u32, String> = BTreeMap::new();
+        uid_message_ids.insert(uid, rfc724_mid);
+        let (last_uid, _received) = match self
+            .fetch_many_msgs(context, folder, vec![uid], &uid_message_ids, false, false)
+            .await
+        {
+            Ok(res) => res,
+            Err(_) => return ImapActionResult::Failed,
+        };
+        if last_uid.is_none() {
+            ImapActionResult::Failed
+        } else {
+            ImapActionResult::Success
         }
-
-        ImapActionResult::Success
     }
 }
 
@@ -223,13 +254,15 @@ impl MimeMessage {
 
 #[cfg(test)]
 mod tests {
-    use super::*;
+    use num_traits::FromPrimitive;
+
     use crate::chat::send_msg;
-    use crate::constants::Viewtype;
     use crate::dc_receive_imf::dc_receive_imf_inner;
     use crate::ephemeral::Timer;
+    use crate::message::Viewtype;
     use crate::test_utils::TestContext;
-    use num_traits::FromPrimitive;
+
+    use super::*;
 
     #[test]
     fn test_downloadstate_values() {
@@ -311,9 +344,9 @@ mod tests {
 
         dc_receive_imf_inner(
             &t,
+            "Mr.12345678901@example.com",
             header.as_bytes(),
             "INBOX",
-            1,
             false,
             Some(100000),
             false,
@@ -329,9 +362,9 @@ mod tests {
 
         dc_receive_imf_inner(
             &t,
+            "Mr.12345678901@example.com",
             format!("{}\n\n100k text...", header).as_bytes(),
             "INBOX",
-            1,
             false,
             None,
             false,
@@ -359,15 +392,15 @@ mod tests {
         // download message from bob partially, this must not change the ephemeral timer
         dc_receive_imf_inner(
             &t,
+            "first@example.org",
             b"From: Bob <bob@example.org>\n\
-                    To: Alice <alice@example.com>\n\
+                    To: Alice <alice@example.org>\n\
                     Chat-Version: 1.0\n\
                     Subject: subject\n\
                     Message-ID: <first@example.org>\n\
                     Date: Sun, 14 Nov 2021 00:10:00 +0000\
                     Content-Type: text/plain",
             "INBOX",
-            1,
             false,
             Some(100000),
             false,

src/e2ee.rs

@@ -2,7 +2,7 @@
 
 use std::collections::HashSet;
 
-use anyhow::{bail, ensure, format_err, Result};
+use anyhow::{bail, format_err, Context as _, Result};
 use mailparse::ParsedMail;
 use num_traits::FromPrimitive;
 
@@ -19,7 +19,6 @@ use crate::pgp;
 #[derive(Debug)]
 pub struct EncryptHelper {
     pub prefer_encrypt: EncryptPreference,
-    force_preference: bool,
     pub addr: String,
     pub public_key: SignedPublicKey,
 }
@@ -29,7 +28,6 @@ impl EncryptHelper {
         let prefer_encrypt =
             EncryptPreference::from_i32(context.get_config_int(Config::E2eeEnabled).await?)
                 .unwrap_or_default();
-        let force_preference = context.get_config_bool(Config::E2eeForce).await?;
         let addr = match context.get_config(Config::ConfiguredAddr).await? {
             None => {
                 bail!("addr not configured!");
@@ -41,7 +39,6 @@ impl EncryptHelper {
 
         Ok(EncryptHelper {
             prefer_encrypt,
-            force_preference,
             addr,
             public_key,
         })
@@ -103,17 +100,11 @@ impl EncryptHelper {
             }
         }
 
-        let want_encrypt = if self.force_preference {
-            // Ignore preferences of others.
-            self.prefer_encrypt == EncryptPreference::Mutual
-        } else {
-            // Count number of recipients, including self.
-            // This does not depend on whether we send a copy to self or not.
-            let recipients_count = peerstates.len() + 1;
-            2 * prefer_encrypt_count > recipients_count
-        };
+        // Count number of recipients, including self.
+        // This does not depend on whether we send a copy to self or not.
+        let recipients_count = peerstates.len() + 1;
 
-        Ok(e2ee_guaranteed || want_encrypt)
+        Ok(e2ee_guaranteed || 2 * prefer_encrypt_count > recipients_count)
     }
 
     /// Tries to encrypt the passed in `mail`.
@@ -130,9 +121,9 @@ impl EncryptHelper {
             .into_iter()
             .filter_map(|(state, addr)| state.map(|s| (s, addr)))
         {
-            let key = peerstate.take_key(min_verified).ok_or_else(|| {
-                format_err!("proper enc-key for {} missing, cannot encrypt", addr)
-            })?;
+            let key = peerstate
+                .take_key(min_verified)
+                .with_context(|| format!("proper enc-key for {} missing, cannot encrypt", addr))?;
             keyring.add(key);
         }
         keyring.add(self.public_key.clone());
@@ -188,7 +179,6 @@ pub async fn try_decrypt(
     // Possibly perform decryption
     let private_keyring: Keyring<SignedSecretKey> = Keyring::new_self(context).await?;
     let mut public_keyring_for_validate: Keyring<SignedPublicKey> = Keyring::new();
-    let mut signatures = HashSet::default();
 
     if let Some(ref mut peerstate) = peerstate {
         peerstate
@@ -201,14 +191,17 @@ pub async fn try_decrypt(
         }
     }
 
-    let out_mail = decrypt_if_autocrypt_message(
+    let (out_mail, signatures) = match decrypt_if_autocrypt_message(
         context,
         mail,
         private_keyring,
         public_keyring_for_validate,
-        &mut signatures,
     )
-    .await?;
+    .await?
+    {
+        Some((out_mail, signatures)) => (Some(out_mail), signatures),
+        None => (None, Default::default()),
+    };
 
     if let Some(mut peerstate) = peerstate {
         // If message is not encrypted and it is not a read receipt, degrade encryption.
@@ -284,8 +277,7 @@ async fn decrypt_if_autocrypt_message(
     mail: &ParsedMail<'_>,
     private_keyring: Keyring<SignedSecretKey>,
     public_keyring_for_validate: Keyring<SignedPublicKey>,
-    ret_valid_signatures: &mut HashSet<Fingerprint>,
-) -> Result<Option<Vec<u8>>> {
+) -> Result<Option<(Vec<u8>, HashSet<Fingerprint>)>> {
     let encrypted_data_part = match get_autocrypt_mime(mail).or_else(|| get_mixed_up_mime(mail)) {
         None => {
             // not an autocrypt mime message, abort and ignore
@@ -299,36 +291,60 @@ async fn decrypt_if_autocrypt_message(
         encrypted_data_part,
         private_keyring,
         public_keyring_for_validate,
-        ret_valid_signatures,
     )
     .await
 }
 
+/// Validates signatures of Multipart/Signed message part, as defined in RFC 1847.
+///
+/// Returns `None` if the part is not a Multipart/Signed part, otherwise retruns the set of key
+/// fingerprints for which there is a valid signature.
+async fn validate_detached_signature(
+    mail: &ParsedMail<'_>,
+    public_keyring_for_validate: &Keyring<SignedPublicKey>,
+) -> Result<Option<(Vec<u8>, HashSet<Fingerprint>)>> {
+    if mail.ctype.mimetype != "multipart/signed" {
+        return Ok(None);
+    }
+
+    if let [first_part, second_part] = &mail.subparts[..] {
+        // First part is the content, second part is the signature.
+        let content = first_part.raw_bytes;
+        let signature = second_part.get_body_raw()?;
+        let ret_valid_signatures =
+            pgp::pk_validate(content, &signature, public_keyring_for_validate).await?;
+
+        Ok(Some((content.to_vec(), ret_valid_signatures)))
+    } else {
+        Ok(None)
+    }
+}
+
 /// Returns Ok(None) if nothing encrypted was found.
 async fn decrypt_part(
     mail: &ParsedMail<'_>,
     private_keyring: Keyring<SignedSecretKey>,
     public_keyring_for_validate: Keyring<SignedPublicKey>,
-    ret_valid_signatures: &mut HashSet<Fingerprint>,
-) -> Result<Option<Vec<u8>>> {
+) -> Result<Option<(Vec<u8>, HashSet<Fingerprint>)>> {
     let data = mail.get_body_raw()?;
 
     if has_decrypted_pgp_armor(&data) {
-        // we should only have one decryption happening
-        ensure!(ret_valid_signatures.is_empty(), "corrupt signatures");
+        let (plain, ret_valid_signatures) =
+            pgp::pk_decrypt(data, private_keyring, &public_keyring_for_validate).await?;
 
-        let plain = pgp::pk_decrypt(
-            data,
-            private_keyring,
-            public_keyring_for_validate,
-            Some(ret_valid_signatures),
-        )
-        .await?;
+        // Check for detached signatures.
+        // If decrypted part is a multipart/signed, then there is a detached signature.
+        let decrypted_part = mailparse::parse_mail(&plain)?;
+        if let Some((content, valid_detached_signatures)) =
+            validate_detached_signature(&decrypted_part, &public_keyring_for_validate).await?
+        {
+            return Ok(Some((content, valid_detached_signatures)));
+        } else {
+            // If the message was wrongly or not signed, still return the plain text.
+            // The caller has to check the signatures then.
 
-        // If the message was wrongly or not signed, still return the plain text.
-        // The caller has to check the signatures then.
-
-        return Ok(Some(plain));
+            return Ok(Some((plain, ret_valid_signatures)));
+        }
     }
 
     Ok(None)
@@ -374,36 +390,34 @@ pub async fn ensure_secret_key_exists(context: &Context) -> Result<String> {
     let self_addr = context
         .get_config(Config::ConfiguredAddr)
         .await?
-        .ok_or_else(|| {
-            format_err!(concat!(
-                "Failed to get self address, ",
-                "cannot ensure secret key if not configured."
-            ))
-        })?;
+        .context(concat!(
+            "Failed to get self address, ",
+            "cannot ensure secret key if not configured."
+        ))?;
     SignedPublicKey::load_self(context).await?;
     Ok(self_addr)
 }
 
 #[cfg(test)]
 mod tests {
-    use super::*;
-
     use crate::chat;
-    use crate::constants::Viewtype;
-    use crate::dc_receive_imf::dc_receive_imf;
-    use crate::message::Message;
+    use crate::message::{Message, Viewtype};
     use crate::param::Param;
     use crate::peerstate::ToSave;
     use crate::test_utils::{bob_keypair, TestContext};
 
+    use super::*;
+
     mod ensure_secret_key_exists {
         use super::*;
 
         #[async_std::test]
         async fn test_prexisting() {
-            let t = TestContext::new().await;
-            let test_addr = t.configure_alice().await;
-            assert_eq!(ensure_secret_key_exists(&t).await.unwrap(), test_addr);
+            let t = TestContext::new_alice().await;
+            assert_eq!(
+                ensure_secret_key_exists(&t).await.unwrap(),
+                "alice@example.org"
+            );
         }
 
         #[async_std::test]
@@ -474,7 +488,7 @@ Sent with my Delta Chat Messenger: https://delta.chat";
         assert!(!msg.was_encrypted());
 
         // Parsing a message is enough to update peerstate
-        let peerstate_alice = Peerstate::from_addr(&bob.ctx, "alice@example.com")
+        let peerstate_alice = Peerstate::from_addr(&bob.ctx, "alice@example.org")
             .await?
             .expect("no peerstate found in the database");
         assert_eq!(peerstate_alice.prefer_encrypt, EncryptPreference::Mutual);
@@ -505,28 +519,28 @@ Sent with my Delta Chat Messenger: https://delta.chat";
 
         let msg = bob.parse_msg(&sent).await;
         assert!(msg.was_encrypted());
-        let peerstate_alice = Peerstate::from_addr(&bob.ctx, "alice@example.com")
+        let peerstate_alice = Peerstate::from_addr(&bob.ctx, "alice@example.org")
             .await?
             .expect("no peerstate found in the database");
         assert_eq!(peerstate_alice.prefer_encrypt, EncryptPreference::Mutual);
 
         // Alice sends plaintext message with Autocrypt header.
         let mut msg = Message::new(Viewtype::Text);
-        msg.param.set_int(Param::ForcePlaintext, 1);
+        msg.force_plaintext();
         chat::prepare_msg(&alice.ctx, chat_alice, &mut msg).await?;
         chat::send_msg(&alice.ctx, chat_alice, &mut msg).await?;
         let sent = alice.pop_sent_msg().await;
 
         let msg = bob.parse_msg(&sent).await;
         assert!(!msg.was_encrypted());
-        let peerstate_alice = Peerstate::from_addr(&bob.ctx, "alice@example.com")
+        let peerstate_alice = Peerstate::from_addr(&bob.ctx, "alice@example.org")
             .await?
             .expect("no peerstate found in the database");
         assert_eq!(peerstate_alice.prefer_encrypt, EncryptPreference::Mutual);
 
         // Alice sends plaintext message without Autocrypt header.
         let mut msg = Message::new(Viewtype::Text);
-        msg.param.set_int(Param::ForcePlaintext, 1);
+        msg.force_plaintext();
         msg.param.set_int(Param::SkipAutocrypt, 1);
         chat::prepare_msg(&alice.ctx, chat_alice, &mut msg).await?;
         chat::send_msg(&alice.ctx, chat_alice, &mut msg).await?;
@@ -534,7 +548,7 @@ Sent with my Delta Chat Messenger: https://delta.chat";
 
         let msg = bob.parse_msg(&sent).await;
         assert!(!msg.was_encrypted());
-        let peerstate_alice = Peerstate::from_addr(&bob.ctx, "alice@example.com")
+        let peerstate_alice = Peerstate::from_addr(&bob.ctx, "alice@example.org")
             .await?
             .expect("no peerstate found in the database");
         assert_eq!(peerstate_alice.prefer_encrypt, EncryptPreference::Reset);
@@ -612,68 +626,4 @@ Sent with my Delta Chat Messenger: https://delta.chat";
 
         Ok(())
     }
-
-    #[async_std::test]
-    async fn test_e2ee_force() -> Result<()> {
-        let alice = TestContext::new_alice().await;
-        let bob = TestContext::new_bob().await;
-
-        let alice_chat = alice.create_chat(&bob).await;
-        let bob_chat = bob.create_chat(&alice).await;
-
-        alice.set_config(Config::ShowEmails, Some("2")).await?;
-        bob.set_config(Config::ShowEmails, Some("2")).await?;
-
-        // Alice does not prefer encryption.
-        alice.set_config(Config::E2eeEnabled, Some("0")).await?;
-        bob.set_config(Config::E2eeEnabled, Some("1")).await?;
-
-        // Alice sends her key to Bob.
-        let sent_msg = alice.send_text(alice_chat.id, "Hi Bob").await;
-        bob.recv_msg(&sent_msg).await;
-        let received_msg = bob.get_last_msg().await;
-        assert!(!received_msg.get_showpadlock());
-
-        // Bob should not encrypt, because Alice does not prefer encryption.
-        let sent_msg = bob
-            .send_text(bob_chat.id, "This should not be encrypted")
-            .await;
-        alice.recv_msg(&sent_msg).await;
-        let received_msg = alice.get_last_msg().await;
-        assert!(!received_msg.get_showpadlock());
-
-        // Bob ignores Alice's preference for no encryption.
-        bob.set_config(Config::E2eeForce, Some("1")).await?;
-        let sent_msg = bob.send_text(bob_chat.id, "This should be encrypted").await;
-        alice.recv_msg(&sent_msg).await;
-        let received_msg = alice.get_last_msg().await;
-        assert!(received_msg.get_showpadlock());
-
-        // Alice switches to MUA without Autocrypt support.
-        dc_receive_imf(
-            &bob,
-            br#"Subject: Hello from MUA
-Message-ID: foobar@example.com
-To: Bob <bob@example.net>
-From: Alice <alice@example.com>
-Content-Type: text/plain; charset=utf-8
-Date: Sun, 14 Mar 2500 00:00:00 +0000
-
-Hello from MUA."#,
-            "INBOX",
-            100,
-            false,
-        )
-        .await?;
-
-        // Bob can't encrypt now because Alice has no key.
-        let sent_msg = bob
-            .send_text(bob_chat.id, "This should not be encrypted again")
-            .await;
-        alice.recv_msg(&sent_msg).await;
-        let received_msg = alice.get_last_msg().await;
-        assert!(!received_msg.get_showpadlock());
-
-        Ok(())
-    }
 }

src/ephemeral.rs

@@ -52,7 +52,7 @@
 //! `MsgsChanged` event is emitted when a message deletion is due, to
 //! make UI reload displayed messages and cause actual deletion.
 //!
-//! Server deletion happens by generating IMAP deletion jobs based on
+//! Server deletion happens by updating the `imap` table based on
 //! the database entries which are expired either according to their
 //! ephemeral message timers or global `delete_server_after` setting.
 
@@ -67,15 +67,16 @@ use serde::{Deserialize, Serialize};
 
 use crate::chat::{send_msg, ChatId};
 use crate::constants::{
-    Viewtype, DC_CHAT_ID_LAST_SPECIAL, DC_CHAT_ID_TRASH, DC_CONTACT_ID_DEVICE, DC_CONTACT_ID_SELF,
+    DC_CHAT_ID_LAST_SPECIAL, DC_CHAT_ID_TRASH, DC_CONTACT_ID_DEVICE, DC_CONTACT_ID_SELF,
 };
+use crate::contact::ContactId;
 use crate::context::Context;
 use crate::dc_tools::time;
 use crate::download::MIN_DELETE_SERVER_AFTER;
 use crate::events::EventType;
-use crate::job;
-use crate::message::{Message, MessageState, MsgId};
+use crate::message::{Message, MessageState, MsgId, Viewtype};
 use crate::mimeparser::SystemMessage;
+use crate::sql;
 use crate::stock_str;
 use std::cmp::max;
 
@@ -213,7 +214,7 @@ impl ChatId {
 pub(crate) async fn stock_ephemeral_timer_changed(
     context: &Context,
     timer: Timer,
-    from_id: u32,
+    from_id: ContactId,
 ) -> String {
     match timer {
         Timer::Disabled => stock_str::msg_ephemeral_timer_disabled(context, from_id).await,
@@ -263,7 +264,7 @@ pub(crate) async fn stock_ephemeral_timer_changed(
 
 impl MsgId {
     /// Returns ephemeral message timer value for the message.
-    pub(crate) async fn ephemeral_timer(self, context: &Context) -> anyhow::Result<Timer> {
+    pub(crate) async fn ephemeral_timer(self, context: &Context) -> Result<Timer> {
         let res = match context
             .sql
             .query_get_value(
@@ -279,7 +280,7 @@ impl MsgId {
     }
 
     /// Starts ephemeral message timer for the message if it is not started yet.
-    pub(crate) async fn start_ephemeral_timer(self, context: &Context) -> anyhow::Result<()> {
+    pub(crate) async fn start_ephemeral_timer(self, context: &Context) -> Result<()> {
         if let Timer::Enabled { duration } = self.ephemeral_timer(context).await? {
             let ephemeral_timestamp = time().saturating_add(duration.into());
 
@@ -298,6 +299,37 @@ impl MsgId {
     }
 }
 
+pub(crate) async fn start_ephemeral_timers_msgids(
+    context: &Context,
+    msg_ids: &[MsgId],
+) -> Result<()> {
+    let msg_ids: Vec<&dyn crate::ToSql> = msg_ids
+        .iter()
+        .map(|msg_id| msg_id as &dyn crate::ToSql)
+        .collect();
+    let now = time();
+    let count = context
+        .sql
+        .execute(
+            format!(
+                "UPDATE msgs SET ephemeral_timestamp = ? + ephemeral_timer
+         WHERE (ephemeral_timestamp == 0 OR ephemeral_timestamp > ? + ephemeral_timer) AND ephemeral_timer > 0
+         AND id IN ({})",
+                sql::repeat_vars(msg_ids.len())?
+            ),
+            rusqlite::params_from_iter(
+                std::iter::once(&now as &dyn crate::ToSql)
+                    .chain(std::iter::once(&now as &dyn crate::ToSql))
+                    .chain(msg_ids),
+            ),
+        )
+        .await?;
+    if count > 0 {
+        schedule_ephemeral_task(context).await;
+    }
+    Ok(())
+}
+
 /// Deletes messages which are expired according to
 /// `delete_device_after` setting or `ephemeral_timestamp` column.
 ///
@@ -434,11 +466,8 @@ pub async fn schedule_ephemeral_task(context: &Context) {
     }
 }
 
-/// Returns ID of any expired message that should be deleted from the server.
-///
-/// It looks up the trash chat too, to find messages that are already
-/// deleted locally, but not deleted on the server.
-pub(crate) async fn load_imap_deletion_msgid(context: &Context) -> anyhow::Result<Option<MsgId>> {
+/// Schedules expired IMAP messages for deletion.
+pub(crate) async fn delete_expired_imap_messages(context: &Context) -> Result<()> {
     let now = time();
 
     let (threshold_timestamp, threshold_timestamp_extended) =
@@ -452,27 +481,20 @@ pub(crate) async fn load_imap_deletion_msgid(context: &Context) -> anyhow::Resul
 
     context
         .sql
-        .query_row_optional(
-            "SELECT id FROM msgs \
-         WHERE ( \
-         ((download_state = 0 AND timestamp < ?) OR (download_state != 0 AND timestamp < ?)) \
-         OR (ephemeral_timestamp != 0 AND ephemeral_timestamp <= ?) \
-         ) \
-         AND server_uid != 0 \
-         AND NOT id IN (SELECT foreign_id FROM jobs WHERE action = ?)
-         LIMIT 1",
-            paramsv![
-                threshold_timestamp,
-                threshold_timestamp_extended,
-                now,
-                job::Action::DeleteMsgOnImap
-            ],
-            |row| {
-                let msg_id: MsgId = row.get(0)?;
-                Ok(msg_id)
-            },
+        .execute(
+            "UPDATE imap
+             SET target=''
+             WHERE rfc724_mid IN (
+               SELECT rfc724_mid FROM msgs
+               WHERE ((download_state = 0 AND timestamp < ?) OR
+                      (download_state != 0 AND timestamp < ?) OR
+                      (ephemeral_timestamp != 0 AND ephemeral_timestamp <= ?))
+             )",
+            paramsv![threshold_timestamp, threshold_timestamp_extended, now],
         )
-        .await
+        .await?;
+
+    Ok(())
 }
 
 /// Start ephemeral timers for seen messages if they are not started
@@ -507,9 +529,6 @@ pub(crate) async fn start_ephemeral_timers(context: &Context) -> Result<()> {
 
 #[cfg(test)]
 mod tests {
-    use crate::param::Params;
-    use async_std::task::sleep;
-
     use super::*;
     use crate::config::Config;
     use crate::dc_receive_imf::dc_receive_imf;
@@ -725,7 +744,7 @@ mod tests {
     /// Test that Alice replying to the chat without a timer at the same time as Bob enables the
     /// timer does not result in disabling the timer on the Bob's side.
     #[async_std::test]
-    async fn test_ephemeral_timer_rollback() -> anyhow::Result<()> {
+    async fn test_ephemeral_timer_rollback() -> Result<()> {
         let alice = TestContext::new_alice().await;
         let bob = TestContext::new_bob().await;
 
@@ -799,14 +818,14 @@ mod tests {
     }
 
     #[async_std::test]
-    async fn test_ephemeral_delete_msgs() {
+    async fn test_ephemeral_delete_msgs() -> Result<()> {
         let t = TestContext::new_alice().await;
         let chat = t.get_self_chat().await;
 
         t.send_text(chat.id, "Saved message, which we delete manually")
             .await;
         let msg = t.get_last_msg_in(chat.id).await;
-        msg.id.delete_from_db(&t).await.unwrap();
+        msg.id.delete_from_db(&t).await?;
         check_msg_was_deleted(&t, &chat, msg.id).await;
 
         chat.id
@@ -817,36 +836,12 @@ mod tests {
             .send_text(chat.id, "Saved message, disappearing after 1s")
             .await;
 
-        sleep(Duration::from_millis(1100)).await;
+        async_std::task::sleep(Duration::from_millis(1100)).await;
 
-        // Check checks that the msg was deleted locally
+        // Check that the msg was deleted locally.
         check_msg_was_deleted(&t, &chat, msg.sender_msg_id).await;
 
-        // Check that the msg will be deleted on the server
-        // First of all, set a server_uid so that DC thinks that it's actually possible to delete
-        t.sql
-            .execute(
-                "UPDATE msgs SET server_uid=1 WHERE id=?",
-                paramsv![msg.sender_msg_id],
-            )
-            .await
-            .unwrap();
-        let job = job::load_imap_deletion_job(&t).await.unwrap();
-        assert_eq!(
-            job,
-            Some(job::Job::new(
-                job::Action::DeleteMsgOnImap,
-                msg.sender_msg_id.to_u32(),
-                Params::new(),
-                0,
-            ))
-        );
-        // Let's assume that executing the job fails on first try and the job is saved to the db
-        job.unwrap().save(&t).await.unwrap();
-
-        // Make sure that we don't get yet another job when loading from db
-        let job2 = job::load_imap_deletion_job(&t).await.unwrap();
-        assert_eq!(job2, None);
+        Ok(())
     }
 
     async fn check_msg_was_deleted(t: &TestContext, chat: &Chat, msg_id: MsgId) {
@@ -861,8 +856,8 @@ mod tests {
 
         // Check that if there is a message left, the text and metadata are gone
         if let Ok(msg) = Message::load_from_db(t, msg_id).await {
-            assert_eq!(msg.from_id, 0);
-            assert_eq!(msg.to_id, 0);
+            assert_eq!(msg.from_id, ContactId::new(0));
+            assert_eq!(msg.to_id, ContactId::new(0));
             assert!(msg.text.is_none_or_empty(), "{:?}", msg.text);
             let rawtxt: Option<String> = t
                 .sql
@@ -874,7 +869,7 @@ mod tests {
     }
 
     #[async_std::test]
-    async fn test_load_imap_deletion_msgid() -> Result<()> {
+    async fn test_delete_expired_imap_messages() -> Result<()> {
         let t = TestContext::new_alice().await;
         const HOUR: i64 = 60 * 60;
         let now = time();
@@ -887,42 +882,98 @@ mod tests {
             (2000, now - 18 * HOUR, now - HOUR),
             (2020, now - 17 * HOUR, now + HOUR),
         ] {
+            let message_id = id.to_string();
             t.sql
-                .execute(
-                    "INSERT INTO msgs (id, server_uid, timestamp, ephemeral_timestamp) VALUES (?,?,?,?);",
-                    paramsv![id, id, timestamp, ephemeral_timestamp],
-                )
-                .await?;
+                   .execute(
+                       "INSERT INTO msgs (id, rfc724_mid, timestamp, ephemeral_timestamp) VALUES (?,?,?,?);",
+                       paramsv![id, message_id, timestamp, ephemeral_timestamp],
+                   )
+                   .await?;
+            t.sql
+                   .execute(
+                       "INSERT INTO imap (rfc724_mid, folder, uid, target) VALUES (?,'INBOX',?, 'INBOX');",
+                       paramsv![message_id, id],
+                   )
+                   .await?;
         }
 
-        assert_eq!(load_imap_deletion_msgid(&t).await?, Some(MsgId::new(2000)));
+        async fn test_marked_for_deletion(context: &Context, id: u32) -> Result<()> {
+            assert_eq!(
+                context
+                    .sql
+                    .count(
+                        "SELECT COUNT(*) FROM imap WHERE target='' AND rfc724_mid=?",
+                        paramsv![id.to_string()],
+                    )
+                    .await?,
+                1
+            );
+            Ok(())
+        }
 
-        MsgId::new(2000).delete_from_db(&t).await?;
-        assert_eq!(load_imap_deletion_msgid(&t).await?, None);
+        async fn remove_uid(context: &Context, id: u32) -> Result<()> {
+            context
+                .sql
+                .execute(
+                    "DELETE FROM imap WHERE rfc724_mid=?",
+                    paramsv![id.to_string()],
+                )
+                .await?;
+            Ok(())
+        }
+
+        // This should mark message 2000 for deletion.
+        delete_expired_imap_messages(&t).await?;
+        test_marked_for_deletion(&t, 2000).await?;
+        remove_uid(&t, 2000).await?;
+        // No other messages are marked for deletion.
+        assert_eq!(
+            t.sql
+                .count("SELECT COUNT(*) FROM imap WHERE target=''", paramsv![],)
+                .await?,
+            0
+        );
 
         t.set_config(Config::DeleteServerAfter, Some(&*(25 * HOUR).to_string()))
             .await?;
-        assert_eq!(load_imap_deletion_msgid(&t).await?, Some(MsgId::new(1000)));
+        delete_expired_imap_messages(&t).await?;
+        test_marked_for_deletion(&t, 1000).await?;
 
         MsgId::new(1000)
             .update_download_state(&t, DownloadState::Available)
             .await?;
-        assert_eq!(load_imap_deletion_msgid(&t).await?, Some(MsgId::new(1000))); // delete downloadable anyway
-
-        MsgId::new(1000).delete_from_db(&t).await?;
-        assert_eq!(load_imap_deletion_msgid(&t).await?, None);
+        t.sql
+            .execute(
+                "UPDATE imap SET target=folder WHERE rfc724_mid='1000'",
+                paramsv![],
+            )
+            .await?;
+        delete_expired_imap_messages(&t).await?;
+        test_marked_for_deletion(&t, 1000).await?; // Delete downloadable anyway.
+        remove_uid(&t, 1000).await?;
 
         t.set_config(Config::DeleteServerAfter, Some(&*(22 * HOUR).to_string()))
             .await?;
-        assert_eq!(load_imap_deletion_msgid(&t).await?, Some(MsgId::new(1010)));
+        delete_expired_imap_messages(&t).await?;
+        test_marked_for_deletion(&t, 1010).await?;
+        t.sql
+            .execute(
+                "UPDATE imap SET target=folder WHERE rfc724_mid='1010'",
+                paramsv![],
+            )
+            .await?;
 
         MsgId::new(1010)
             .update_download_state(&t, DownloadState::Available)
             .await?;
-        assert_eq!(load_imap_deletion_msgid(&t).await?, None); // keep downloadable for now
-
-        MsgId::new(1010).delete_from_db(&t).await?;
-        assert_eq!(load_imap_deletion_msgid(&t).await?, None);
+        delete_expired_imap_messages(&t).await?;
+        // Keep downloadable for now.
+        assert_eq!(
+            t.sql
+                .count("SELECT COUNT(*) FROM imap WHERE target=''", paramsv![],)
+                .await?,
+            0
+        );
 
         Ok(())
     }
@@ -936,7 +987,7 @@ mod tests {
         dc_receive_imf(
             &alice,
             b"From: Bob <bob@example.com>\n\
-                    To: Alice <alice@example.com>\n\
+                    To: Alice <alice@example.org>\n\
                     Chat-Version: 1.0\n\
                     Subject: Subject\n\
                     Message-ID: <first@example.com>\n\
@@ -944,7 +995,6 @@ mod tests {
                     \n\
                     hello\n",
             "INBOX",
-            1,
             false,
         )
         .await?;
@@ -957,7 +1007,7 @@ mod tests {
         dc_receive_imf(
             &alice,
             b"From: Bob <bob@example.com>\n\
-                    To: Alice <alice@example.com>\n\
+                    To: Alice <alice@example.org>\n\
                     Chat-Version: 1.0\n\
                     Subject: Subject\n\
                     Message-ID: <second@example.com>\n\
@@ -966,7 +1016,6 @@ mod tests {
                     \n\
                     second message\n",
             "INBOX",
-            2,
             false,
         )
         .await?;
@@ -994,7 +1043,7 @@ mod tests {
         dc_receive_imf(
             &alice,
             b"From: Bob <bob@example.com>\n\
-                    To: Alice <alice@example.com>\n\
+                    To: Alice <alice@example.org>\n\
                     Chat-Version: 1.0\n\
                     Subject: Subject\n\
                     Message-ID: <third@example.com>\n\
@@ -1004,7 +1053,6 @@ mod tests {
                     \n\
                     > hello\n",
             "INBOX",
-            3,
             false,
         )
         .await?;

src/events.rs

@@ -7,6 +7,7 @@ use async_std::path::PathBuf;
 use strum::EnumProperty;
 
 use crate::chat::ChatId;
+use crate::contact::ContactId;
 use crate::ephemeral::Timer as EphemeralTimer;
 use crate::message::MsgId;
 
@@ -252,7 +253,7 @@ pub enum EventType {
     ///
     /// @param data1 (int) If set, this is the contact_id of an added contact that should be selected.
     #[strum(props(id = "2030"))]
-    ContactsChanged(Option<u32>),
+    ContactsChanged(Option<ContactId>),
 
     /// Location of one or more contact has changed.
     ///
@@ -260,7 +261,7 @@ pub enum EventType {
     ///     If the locations of several contacts have been changed,
     ///     eg. after calling dc_delete_all_locations(), this parameter is set to `None`.
     #[strum(props(id = "2035"))]
-    LocationChanged(Option<u32>),
+    LocationChanged(Option<ContactId>),
 
     /// Inform about the configuration progress started by configure().
     #[strum(props(id = "2041"))]
@@ -304,7 +305,10 @@ pub enum EventType {
     ///     800=vg-member-added-received received, shown as "bob@addr securely joined GROUP", only sent for the verified-group-protocol.
     ///     1000=Protocol finished for this contact.
     #[strum(props(id = "2060"))]
-    SecurejoinInviterProgress { contact_id: u32, progress: usize },
+    SecurejoinInviterProgress {
+        contact_id: ContactId,
+        progress: usize,
+    },
 
     /// Progress information of a secure-join handshake from the view of the joiner
     /// (Bob, the person who scans the QR code).
@@ -315,7 +319,10 @@ pub enum EventType {
     ///     400=vg-/vc-request-with-auth sent, typically shown as "alice@addr verified, introducing myself."
     ///     (Bob has verified alice and waits until Alice does the same for him)
     #[strum(props(id = "2061"))]
-    SecurejoinJoinerProgress { contact_id: u32, progress: usize },
+    SecurejoinJoinerProgress {
+        contact_id: ContactId,
+        progress: usize,
+    },
 
     /// The connectivity to the server changed.
     /// This means that you should refresh the connectivity view
@@ -326,4 +333,7 @@ pub enum EventType {
 
     #[strum(props(id = "2110"))]
     SelfavatarChanged,
+
+    #[strum(props(id = "2120"))]
+    WebxdcStatusUpdate(MsgId),
 }

src/headerdef.rs

@@ -33,6 +33,7 @@ pub enum HeaderDef {
     XMozillaDraftInfo,
 
     ListId,
+    ListPost,
     References,
     InReplyTo,
     Precedence,

src/html.rs

@@ -279,9 +279,9 @@ mod tests {
     use crate::chat;
     use crate::chat::forward_msgs;
     use crate::config::Config;
-    use crate::constants::{Viewtype, DC_CONTACT_ID_SELF};
+    use crate::constants::DC_CONTACT_ID_SELF;
     use crate::dc_receive_imf::dc_receive_imf;
-    use crate::message::MessengerMessage;
+    use crate::message::{MessengerMessage, Viewtype};
     use crate::test_utils::TestContext;
 
     #[async_std::test]
@@ -365,7 +365,7 @@ test some special html-characters as < > and & but also " and &#x
         // however, rust multiline-strings use just `\n`;
         // therefore, we just remove `\r` before comparison.
         assert_eq!(
-            parser.html.replace("\r", ""),
+            parser.html.replace('\r', ""),
             r##"
 <html>
   <p>mime-modified <b>set</b>; simplify is always regarded as lossy.</p>
@@ -379,7 +379,7 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
         let raw = include_bytes!("../test-data/message/text_alt_html.eml");
         let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
         assert_eq!(
-            parser.html.replace("\r", ""), // see comment in test_htmlparse_html()
+            parser.html.replace('\r', ""), // see comment in test_htmlparse_html()
             r##"<html>
   <p>mime-modified <b>set</b>; simplify is always regarded as lossy.</p>
 </html>
@@ -394,7 +394,7 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
         let raw = include_bytes!("../test-data/message/text_alt_plain_html.eml");
         let parser = HtmlMsgParser::from_bytes(&t.ctx, raw).await.unwrap();
         assert_eq!(
-            parser.html.replace("\r", ""), // see comment in test_htmlparse_html()
+            parser.html.replace('\r', ""), // see comment in test_htmlparse_html()
             r##"<html>
   <p>
     this is <b>html</b>
@@ -440,9 +440,7 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
             .create_chat_with_contact("", "sender@testrun.org")
             .await;
         let raw = include_bytes!("../test-data/message/text_alt_plain_html.eml");
-        dc_receive_imf(&alice, raw, "INBOX", 1, false)
-            .await
-            .unwrap();
+        dc_receive_imf(&alice, raw, "INBOX", false).await.unwrap();
         let msg = alice.get_last_msg_in(chat.get_id()).await;
         assert_ne!(msg.get_from_id(), DC_CONTACT_ID_SELF);
         assert_eq!(msg.is_dc_message, MessengerMessage::No);
@@ -468,7 +466,7 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
 
         // bob: check that bob also got the html-part of the forwarded message
         let bob = TestContext::new_bob().await;
-        let chat = bob.create_chat_with_contact("", "alice@example.com").await;
+        let chat = bob.create_chat_with_contact("", "alice@example.org").await;
         bob.recv_msg(&alice.pop_sent_msg().await).await;
         let msg = bob.get_last_msg_in(chat.get_id()).await;
         assert_ne!(msg.get_from_id(), DC_CONTACT_ID_SELF);
@@ -491,9 +489,7 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
             .create_chat_with_contact("", "sender@testrun.org")
             .await;
         let raw = include_bytes!("../test-data/message/text_alt_plain_html.eml");
-        dc_receive_imf(&alice, raw, "INBOX", 1, false)
-            .await
-            .unwrap();
+        dc_receive_imf(&alice, raw, "INBOX", false).await.unwrap();
         let msg = alice.get_last_msg_in(chat.get_id()).await;
 
         // forward the message to saved-messages,
@@ -560,7 +556,6 @@ test some special html-characters as &lt; &gt; and &amp; but also &quot; and &#x
             &t,
             include_bytes!("../test-data/message/cp1252-html.eml"),
             "INBOX",
-            0,
             false,
         )
         .await?;

src/imap/idle.rs

@@ -1,6 +1,6 @@
 use super::Imap;
 
-use anyhow::{bail, format_err, Result};
+use anyhow::{bail, Context as _, Result};
 use async_imap::extensions::idle::IdleResponse;
 use async_std::prelude::*;
 use std::time::{Duration, SystemTime};
@@ -31,7 +31,7 @@ impl Imap {
         let timeout = Duration::from_secs(23 * 60);
         let mut info = Default::default();
 
-        if self.server_sent_unsolicited_exists(context) {
+        if self.server_sent_unsolicited_exists(context)? {
             return Ok(info);
         }
 
@@ -90,8 +90,8 @@ impl Imap {
             let session = handle
                 .done()
                 .timeout(Duration::from_secs(15))
-                .await
-                .map_err(|err| format_err!("IMAP IDLE protocol timed out: {}", err))??;
+                .await?
+                .context("IMAP IDLE protocol timed out")?;
             self.session = Some(Session { inner: session });
         } else {
             warn!(context, "Attempted to idle without a session");
@@ -150,19 +150,18 @@ impl Imap {
                     }
                     if self.config.can_idle {
                         // we only fake-idled because network was gone during IDLE, probably
-                        break InterruptInfo::new(false, None);
+                        break InterruptInfo::new(false);
                     }
                     info!(context, "fake_idle is connected");
                     // we are connected, let's see if fetching messages results
                     // in anything.  If so, we behave as if IDLE had data but
                     // will have already fetched the messages so perform_*_fetch
                     // will not find any new.
-
                     match self.fetch_new_messages(context, &watch_folder, false).await {
                         Ok(res) => {
                             info!(context, "fetch_new_messages returned {:?}", res);
                             if res {
-                                break InterruptInfo::new(false, None);
+                                break InterruptInfo::new(false);
                             }
                         }
                         Err(err) => {

src/imap/scan_folders.rs

@@ -2,15 +2,18 @@ use std::{collections::BTreeMap, time::Instant};
 
 use anyhow::{Context as _, Result};
 
+use crate::config::Config;
 use crate::imap::Imap;
-use crate::{config::Config, log::LogExt};
+use crate::log::LogExt;
 use crate::{context::Context, imap::FolderMeaning};
-use async_std::prelude::*;
+
+use async_std::stream::StreamExt;
 
 use super::{get_folder_meaning, get_folder_meaning_by_name};
 
 impl Imap {
-    pub async fn scan_folders(&mut self, context: &Context) -> Result<()> {
+    /// Returns true if folders were scanned, false if scanning was postponed.
+    pub(crate) async fn scan_folders(&mut self, context: &Context) -> Result<bool> {
         // First of all, debounce to once per minute:
         let mut last_scan = context.last_full_folder_scan.lock().await;
         if let Some(last_scan) = *last_scan {
@@ -20,28 +23,18 @@ impl Imap {
                 .await?;
 
             if elapsed_secs < debounce_secs {
-                return Ok(());
+                return Ok(false);
             }
         }
         info!(context, "Starting full folder scan");
 
         self.prepare(context).await?;
-        let session = self.session.as_mut();
-        let session = session.context("scan_folders(): IMAP No Connection established")?;
-        let folders: Vec<_> = session.list(Some(""), Some("*")).await?.collect().await;
-        let watched_folders = get_watched_folders(context).await;
+        let folders = self.list_folders(context).await?;
+        let watched_folders = get_watched_folders(context).await?;
 
         let mut folder_configs = BTreeMap::new();
 
         for folder in folders {
-            let folder = match folder {
-                Ok(f) => f,
-                Err(e) => {
-                    warn!(context, "Can't get folder: {}", e);
-                    continue;
-                }
-            };
-
             // Gmail labels are not folders and should be skipped. For example,
             // emails appear in the inbox and under "All Mail" as soon as it is
             // received. The code used to wrongly conclude that the email had
@@ -71,15 +64,15 @@ impl Imap {
             // Don't scan folders that are watched anyway
             if !watched_folders.contains(&folder.name().to_string()) && !is_drafts {
                 // Drain leftover unsolicited EXISTS messages
-                self.server_sent_unsolicited_exists(context);
+                self.server_sent_unsolicited_exists(context)?;
 
                 loop {
-                    self.fetch_new_messages(context, folder.name(), false)
+                    self.fetch_move_delete(context, folder.name())
                         .await
                         .ok_or_log_msg(context, "Can't fetch new msgs in scanned folder");
 
                     // If the server sent an unsocicited EXISTS during the fetch, we need to fetch again
-                    if !self.server_sent_unsolicited_exists(context) {
+                    if !self.server_sent_unsolicited_exists(context)? {
                         break;
                     }
                 }
@@ -98,23 +91,41 @@ impl Imap {
         }
 
         last_scan.replace(Instant::now());
-        Ok(())
+        Ok(true)
+    }
+
+    /// Returns the names of all folders on the IMAP server.
+    pub async fn list_folders(
+        self: &mut Imap,
+        context: &Context,
+    ) -> Result<Vec<async_imap::types::Name>> {
+        let session = self.session.as_mut();
+        let session = session.context("No IMAP connection")?;
+        let list = session
+            .list(Some(""), Some("*"))
+            .await?
+            .filter_map(|f| f.ok_or_log_msg(context, "list_folders() can't get folder"));
+        Ok(list.collect().await)
     }
 }
 
-pub(crate) async fn get_watched_folders(context: &Context) -> Vec<String> {
+pub(crate) async fn get_watched_folder_configs(context: &Context) -> Result<Vec<Config>> {
+    let mut res = vec![Config::ConfiguredInboxFolder];
+    if context.get_config_bool(Config::SentboxWatch).await? {
+        res.push(Config::ConfiguredSentboxFolder);
+    }
+    if context.should_watch_mvbox().await? {
+        res.push(Config::ConfiguredMvboxFolder);
+    }
+    Ok(res)
+}
+
+pub(crate) async fn get_watched_folders(context: &Context) -> Result<Vec<String>> {
     let mut res = Vec::new();
-    let folder_watched_configured = &[
-        (Config::SentboxWatch, Config::ConfiguredSentboxFolder),
-        (Config::MvboxWatch, Config::ConfiguredMvboxFolder),
-        (Config::InboxWatch, Config::ConfiguredInboxFolder),
-    ];
-    for (watched, configured) in folder_watched_configured {
-        if context.get_config_bool(*watched).await.unwrap_or_default() {
-            if let Ok(Some(folder)) = context.get_config(*configured).await {
-                res.push(folder);
-            }
+    for folder_config in get_watched_folder_configs(context).await? {
+        if let Some(folder) = context.get_config(folder_config).await? {
+            res.push(folder);
         }
     }
-    res
+    Ok(res)
 }

src/imap/select_folder.rs

@@ -93,7 +93,11 @@ impl Imap {
         // select new folder
         if let Some(folder) = folder {
             if let Some(ref mut session) = &mut self.session {
-                let res = session.select(folder).await;
+                let res = if self.config.can_condstore {
+                    session.select_condstore(folder).await
+                } else {
+                    session.select(folder).await
+                };
 
                 // <https://tools.ietf.org/html/rfc3501#section-6.3.1>
                 // says that if the server reports select failure we are in

src/imex.rs

@@ -16,21 +16,21 @@ use rand::{thread_rng, Rng};
 use crate::blob::BlobObject;
 use crate::chat::{self, delete_and_reset_all_device_msgs, ChatId};
 use crate::config::Config;
-use crate::constants::{Viewtype, DC_CONTACT_ID_SELF};
+use crate::constants::DC_CONTACT_ID_SELF;
 use crate::context::Context;
 use crate::dc_tools::{
-    dc_copy_file, dc_create_folder, dc_delete_file, dc_delete_files_in_dir, dc_get_filesuffix_lc,
-    dc_open_file_std, dc_read_file, dc_write_file, get_next_backup_path, time, EmailAddress,
+    dc_create_folder, dc_delete_file, dc_delete_files_in_dir, dc_get_filesuffix_lc,
+    dc_open_file_std, dc_read_file, dc_write_file, time, EmailAddress,
 };
 use crate::e2ee;
 use crate::events::EventType;
 use crate::key::{self, DcKey, DcSecretKey, SignedPublicKey, SignedSecretKey};
 use crate::log::LogExt;
-use crate::message::{Message, MsgId};
+use crate::message::{Message, MsgId, Viewtype};
 use crate::mimeparser::SystemMessage;
 use crate::param::Param;
 use crate::pgp;
-use crate::sql::{self, Sql};
+use crate::sql;
 use crate::stock_str;
 
 // Name of the database file in the backup.
@@ -41,24 +41,24 @@ const BLOBS_BACKUP_NAME: &str = "blobs_backup";
 #[repr(u32)]
 pub enum ImexMode {
     /// Export all private keys and all public keys of the user to the
-    /// directory given as `param1`.  The default key is written to the files `public-key-default.asc`
+    /// directory given as `path`.  The default key is written to the files `public-key-default.asc`
     /// and `private-key-default.asc`, if there are more keys, they are written to files as
     /// `public-key-<id>.asc` and `private-key-<id>.asc`
     ExportSelfKeys = 1,
 
-    /// Import private keys found in the directory given as `param1`.
+    /// Import private keys found in the directory given as `path`.
     /// The last imported key is made the default keys unless its name contains the string `legacy`.
     /// Public keys are not imported.
     ImportSelfKeys = 2,
 
-    /// Export a backup to the directory given as `param1`.
+    /// Export a backup to the directory given as `path` with the given `passphrase`.
     /// The backup contains all contacts, chats, images and other data and device independent settings.
     /// The backup does not contain device dependent settings as ringtones or LED notification settings.
     /// The name of the backup is typically `delta-chat-<day>.tar`, if more than one backup is create on a day,
     /// the format is `delta-chat-<day>-<number>.tar`
     ExportBackup = 11,
 
-    /// `param1` is the file (not: directory) to import. The file is normally
+    /// `path` is the file (not: directory) to import. The file is normally
     /// created by DC_IMEX_EXPORT_BACKUP and detected by dc_imex_has_backup(). Importing a backup
     /// is only possible as long as the context is not configured or used in another way.
     ImportBackup = 12,
@@ -78,11 +78,16 @@ pub enum ImexMode {
 ///
 /// Only one import-/export-progress can run at the same time.
 /// To cancel an import-/export-progress, drop the future returned by this function.
-pub async fn imex(context: &Context, what: ImexMode, param1: &Path) -> Result<()> {
+pub async fn imex(
+    context: &Context,
+    what: ImexMode,
+    path: &Path,
+    passphrase: Option<String>,
+) -> Result<()> {
     let cancel = context.alloc_ongoing().await?;
 
     let res = async {
-        let success = imex_inner(context, what, param1).await;
+        let success = imex_inner(context, what, path, passphrase).await;
         match success {
             Ok(()) => {
                 info!(context, "IMEX successfully completed");
@@ -115,15 +120,10 @@ async fn cleanup_aborted_imex(context: &Context, what: ImexMode) {
         dc_delete_file(context, context.get_dbfile()).await;
         dc_delete_files_in_dir(context, context.get_blobdir()).await;
     }
-    if what == ImexMode::ExportBackup || what == ImexMode::ImportBackup {
-        if let Err(e) = context.sql.open(context, context.get_dbfile(), false).await {
-            warn!(context, "Re-opening db after imex failed: {}", e);
-        }
-    }
 }
 
 /// Returns the filename of the backup found (otherwise an error)
-pub async fn has_backup(context: &Context, dir_name: &Path) -> Result<String> {
+pub async fn has_backup(_context: &Context, dir_name: &Path) -> Result<String> {
     let mut dir_iter = async_std::fs::read_dir(dir_name).await?;
     let mut newest_backup_name = "".to_string();
     let mut newest_backup_path: Option<PathBuf> = None;
@@ -145,59 +145,6 @@ pub async fn has_backup(context: &Context, dir_name: &Path) -> Result<String> {
         }
     }
 
-    match newest_backup_path {
-        Some(path) => Ok(path.to_string_lossy().into_owned()),
-        None => has_backup_old(context, dir_name).await,
-        // When we decide to remove support for .bak backups, we can replace this with `None => bail!("no backup found in {}", dir_name.display()),`.
-    }
-}
-
-/// Returns the filename of the backup found (otherwise an error)
-pub async fn has_backup_old(context: &Context, dir_name: &Path) -> Result<String> {
-    let mut dir_iter = async_std::fs::read_dir(dir_name).await?;
-    let mut newest_backup_time = 0;
-    let mut newest_backup_name = "".to_string();
-    let mut newest_backup_path: Option<PathBuf> = None;
-    while let Some(dirent) = dir_iter.next().await {
-        if let Ok(dirent) = dirent {
-            let path = dirent.path();
-            let name = dirent.file_name();
-            let name = name.to_string_lossy();
-            if name.starts_with("delta-chat") && name.ends_with(".bak") {
-                let sql = Sql::new();
-                match sql.open(context, &path, true).await {
-                    Ok(_) => {
-                        let curr_backup_time = sql
-                            .get_raw_config_int("backup_time")
-                            .await?
-                            .unwrap_or_default();
-                        if curr_backup_time > newest_backup_time {
-                            newest_backup_path = Some(path);
-                            newest_backup_time = curr_backup_time;
-                        }
-                        info!(context, "backup_time of {} is {}", name, curr_backup_time);
-                        sql.close().await;
-                    }
-                    Err(e) => {
-                        warn!(
-                            context,
-                            "Found backup file {} which could not be opened: {}", name, e
-                        );
-                        // On some Android devices we can't open sql files that are not in our private directory
-                        // (see <https://github.com/deltachat/deltachat-android/issues/1768>). So, compare names
-                        // to still find the newest backup.
-                        let name: String = name.into();
-                        if newest_backup_time == 0
-                            && (newest_backup_name.is_empty() || name > newest_backup_name)
-                        {
-                            newest_backup_path = Some(path);
-                            newest_backup_name = name;
-                        }
-                    }
-                }
-            }
-        }
-    }
     match newest_backup_path {
         Some(path) => Ok(path.to_string_lossy().into_owned()),
         None => bail!("no backup found in {}", dir_name.display()),
@@ -218,7 +165,6 @@ pub async fn initiate_key_transfer(context: &Context) -> Result<String> {
 }
 
 async fn do_initiate_key_transfer(context: &Context) -> Result<String> {
-    let mut msg: Message;
     let setup_code = create_setup_code(context);
     /* this may require a keypair to be created. this may take a second ... */
     let setup_file_content = render_setup_file(context, &setup_code).await?;
@@ -231,14 +177,16 @@ async fn do_initiate_key_transfer(context: &Context) -> Result<String> {
     .await?;
 
     let chat_id = ChatId::create_for_contact(context, DC_CONTACT_ID_SELF).await?;
-    msg = Message::default();
-    msg.viewtype = Viewtype::File;
+    let mut msg = Message {
+        viewtype: Viewtype::File,
+        ..Default::default()
+    };
     msg.param.set(Param::File, setup_file_blob.as_name());
     msg.subject = stock_str::ac_setup_msg_subject(context).await;
     msg.param
         .set(Param::MimeType, "application/autocrypt-setup");
     msg.param.set_cmd(SystemMessage::AutocryptSetupMessage);
-    msg.param.set_int(Param::ForcePlaintext, 1);
+    msg.force_plaintext();
     msg.param.set_int(Param::SkipAutocrypt, 1);
 
     let msg_id = chat::send_msg(context, chat_id, &mut msg).await?;
@@ -289,7 +237,7 @@ pub async fn render_setup_file(context: &Context, passphrase: &str) -> Result<St
 
     let msg_subj = stock_str::ac_setup_msg_subject(context).await;
     let msg_body = stock_str::ac_setup_msg_body(context).await;
-    let msg_body_html = msg_body.replace("\r", "").replace("\n", "<br>");
+    let msg_body_html = msg_body.replace('\r', "").replace('\n', "<br>");
     Ok(format!(
         concat!(
             "<!DOCTYPE html>\r\n",
@@ -449,7 +397,12 @@ fn normalize_setup_code(s: &str) -> String {
     out
 }
 
-async fn imex_inner(context: &Context, what: ImexMode, path: &Path) -> Result<()> {
+async fn imex_inner(
+    context: &Context,
+    what: ImexMode,
+    path: &Path,
+    passphrase: Option<String>,
+) -> Result<()> {
     info!(context, "Import/export dir: {}", path.display());
     ensure!(context.sql.is_open().await, "Database not opened.");
     context.emit_event(EventType::ImexProgress(10));
@@ -467,26 +420,27 @@ async fn imex_inner(context: &Context, what: ImexMode, path: &Path) -> Result<()
         ImexMode::ExportSelfKeys => export_self_keys(context, path).await,
         ImexMode::ImportSelfKeys => import_self_keys(context, path).await,
 
-        ImexMode::ExportBackup => export_backup(context, path).await,
-        // import_backup() will call import_backup_old() if this is an old backup.
-        ImexMode::ImportBackup => import_backup(context, path).await,
+        ImexMode::ExportBackup => {
+            export_backup(context, path, passphrase.unwrap_or_default()).await
+        }
+        ImexMode::ImportBackup => {
+            import_backup(context, path, passphrase.unwrap_or_default()).await?;
+            context.sql.run_migrations(context).await
+        }
     }
 }
 
-/// Import Backup
-async fn import_backup(context: &Context, backup_to_import: &Path) -> Result<()> {
-    if backup_to_import.to_string_lossy().ends_with(".bak") {
-        // Backwards compability
-        return import_backup_old(context, backup_to_import).await;
-    }
-
-    info!(
-        context,
-        "Import \"{}\" to \"{}\".",
-        backup_to_import.display(),
-        context.get_dbfile().display()
-    );
-
+/// Imports backup into the currently open database.
+///
+/// The contents of the currently open database will be lost.
+///
+/// `passphrase` is the passphrase used to open backup database. If backup is unencrypted, pass
+/// empty string here.
+async fn import_backup(
+    context: &Context,
+    backup_to_import: &Path,
+    passphrase: String,
+) -> Result<()> {
     ensure!(
         !context.is_configured().await?,
         "Cannot import backups to accounts in use."
@@ -495,15 +449,19 @@ async fn import_backup(context: &Context, backup_to_import: &Path) -> Result<()>
         !context.scheduler.read().await.is_running(),
         "cannot import backup, IO already running"
     );
-    context.sql.close().await;
-    dc_delete_file(context, context.get_dbfile()).await;
-    ensure!(
-        !context.get_dbfile().exists().await,
-        "Cannot delete old database."
-    );
 
     let backup_file = File::open(backup_to_import).await?;
     let file_size = backup_file.metadata().await?.len();
+    info!(
+        context,
+        "Import \"{}\" ({} bytes) to \"{}\".",
+        backup_to_import.display(),
+        file_size,
+        context.get_dbfile().display()
+    );
+
+    context.sql.config_cache.write().await.clear();
+
     let archive = Archive::new(backup_file);
 
     let mut entries = archive.entries()?;
@@ -522,11 +480,15 @@ async fn import_backup(context: &Context, backup_to_import: &Path) -> Result<()>
         if f.path()?.file_name() == Some(OsStr::new(DBFILE_BACKUP_NAME)) {
             // async_tar can't unpack to a specified file name, so we just unpack to the blobdir and then move the unpacked file.
             f.unpack_in(context.get_blobdir()).await?;
-            fs::rename(
-                context.get_blobdir().join(DBFILE_BACKUP_NAME),
-                context.get_dbfile(),
-            )
-            .await?;
+            let unpacked_database = context.get_blobdir().join(DBFILE_BACKUP_NAME);
+            context
+                .sql
+                .import(&unpacked_database, passphrase.clone())
+                .await
+                .context("cannot import unpacked database")?;
+            fs::remove_file(unpacked_database)
+                .await
+                .context("cannot remove unpacked database")?;
         } else {
             // async_tar will unpack to blobdir/BLOBS_BACKUP_NAME, so we move the file afterwards.
             f.unpack_in(context.get_blobdir()).await?;
@@ -541,136 +503,52 @@ async fn import_backup(context: &Context, backup_to_import: &Path) -> Result<()>
         }
     }
 
-    context
-        .sql
-        .open(context, context.get_dbfile(), false)
-        .await
-        .context("Could not re-open db")?;
-
     delete_and_reset_all_device_msgs(context).await?;
 
     Ok(())
 }
 
-async fn import_backup_old(context: &Context, backup_to_import: &Path) -> Result<()> {
-    info!(
-        context,
-        "Import \"{}\" to \"{}\".",
-        backup_to_import.display(),
-        context.get_dbfile().display()
-    );
-
-    ensure!(
-        !context.is_configured().await?,
-        "Cannot import backups to accounts in use."
-    );
-    ensure!(
-        !context.scheduler.read().await.is_running(),
-        "cannot import backup, IO already running"
-    );
-    context.sql.close().await;
-    dc_delete_file(context, context.get_dbfile()).await;
-    ensure!(
-        !context.get_dbfile().exists().await,
-        "Cannot delete old database."
-    );
-
-    ensure!(
-        dc_copy_file(context, backup_to_import, context.get_dbfile()).await,
-        "could not copy file"
-    );
-    /* error already logged */
-    /* re-open copied database file */
-    context
-        .sql
-        .open(context, context.get_dbfile(), false)
-        .await
-        .context("Could not re-open db")?;
-
-    delete_and_reset_all_device_msgs(context).await?;
-
-    let total_files_cnt = context
-        .sql
-        .count("SELECT COUNT(*) FROM backup_blobs;", paramsv![])
-        .await?;
-    info!(
-        context,
-        "***IMPORT-in-progress: total_files_cnt={:?}", total_files_cnt,
-    );
-
-    // Load IDs only for now, without the file contents, to avoid
-    // consuming too much memory.
-    let file_ids = context
-        .sql
-        .query_map(
-            "SELECT id FROM backup_blobs ORDER BY id",
-            paramsv![],
-            |row| row.get(0),
-            |ids| {
-                ids.collect::<std::result::Result<Vec<i64>, _>>()
-                    .map_err(Into::into)
-            },
-        )
-        .await?;
-
-    let mut all_files_extracted = true;
-    for (processed_files_cnt, file_id) in file_ids.into_iter().enumerate() {
-        // Load a single blob into memory
-        let (file_name, file_blob) = context
-            .sql
-            .query_row(
-                "SELECT file_name, file_content FROM backup_blobs WHERE id = ?",
-                paramsv![file_id],
-                |row| {
-                    let file_name: String = row.get(0)?;
-                    let file_blob: Vec<u8> = row.get(1)?;
-                    Ok((file_name, file_blob))
-                },
-            )
-            .await?;
-
-        if context.shall_stop_ongoing().await {
-            all_files_extracted = false;
-            break;
-        }
-        let mut permille = processed_files_cnt * 1000 / total_files_cnt;
-        if permille < 10 {
-            permille = 10
-        }
-        if permille > 990 {
-            permille = 990
-        }
-        context.emit_event(EventType::ImexProgress(permille));
-        if file_blob.is_empty() {
-            continue;
-        }
-
-        let path_filename = context.get_blobdir().join(file_name);
-        dc_write_file(context, &path_filename, &file_blob).await?;
-    }
-
-    if all_files_extracted {
-        // only delete backup_blobs if all files were successfully extracted
-        context
-            .sql
-            .execute("DROP TABLE backup_blobs;", paramsv![])
-            .await?;
-        context.sql.execute("VACUUM;", paramsv![]).await.ok();
-        Ok(())
-    } else {
-        bail!("received stop signal");
-    }
-}
-
 /*******************************************************************************
  * Export backup
  ******************************************************************************/
-#[allow(unused)]
-async fn export_backup(context: &Context, dir: &Path) -> Result<()> {
+
+/// Returns Ok((temp_db_path, temp_path, dest_path)) on success. Unencrypted database can be
+/// written to temp_db_path. The backup can then be written to temp_path. If the backup succeeded,
+/// it can be renamed to dest_path. This guarantees that the backup is complete.
+async fn get_next_backup_path(
+    folder: &Path,
+    backup_time: i64,
+) -> Result<(PathBuf, PathBuf, PathBuf)> {
+    let folder = PathBuf::from(folder);
+    let stem = chrono::NaiveDateTime::from_timestamp(backup_time, 0)
+        // Don't change this file name format, in has_backup() we use string comparison to determine which backup is newer:
+        .format("delta-chat-backup-%Y-%m-%d")
+        .to_string();
+
+    // 64 backup files per day should be enough for everyone
+    for i in 0..64 {
+        let mut tempdbfile = folder.clone();
+        tempdbfile.push(format!("{}-{:02}.db", stem, i));
+
+        let mut tempfile = folder.clone();
+        tempfile.push(format!("{}-{:02}.tar.part", stem, i));
+
+        let mut destfile = folder.clone();
+        destfile.push(format!("{}-{:02}.tar", stem, i));
+
+        if !tempdbfile.exists().await && !tempfile.exists().await && !destfile.exists().await {
+            return Ok((tempdbfile, tempfile, destfile));
+        }
+    }
+    bail!("could not create backup file, disk full?");
+}
+
+async fn export_backup(context: &Context, dir: &Path, passphrase: String) -> Result<()> {
     // get a fine backup file name (the name includes the date so that multiple backup instances are possible)
     let now = time();
-    let (temp_path, dest_path) = get_next_backup_path(dir, now).await?;
-    let _d = DeleteOnDrop(temp_path.clone());
+    let (temp_db_path, temp_path, dest_path) = get_next_backup_path(dir, now).await?;
+    let _d1 = DeleteOnDrop(temp_db_path.clone());
+    let _d2 = DeleteOnDrop(temp_path.clone());
 
     context
         .sql
@@ -682,16 +560,14 @@ async fn export_backup(context: &Context, dir: &Path) -> Result<()> {
         .sql
         .execute("VACUUM;", paramsv![])
         .await
-        .map_err(|e| warn!(context, "Vacuum failed, exporting anyway {}", e));
+        .map_err(|e| warn!(context, "Vacuum failed, exporting anyway {}", e))
+        .ok();
 
     ensure!(
         !context.scheduler.read().await.is_running(),
         "cannot export backup, IO already running"
     );
 
-    // we close the database during the export
-    context.sql.close().await;
-
     info!(
         context,
         "Backup '{}' to '{}'.",
@@ -699,10 +575,13 @@ async fn export_backup(context: &Context, dir: &Path) -> Result<()> {
         dest_path.display(),
     );
 
-    let res = export_backup_inner(context, &temp_path).await;
+    context
+        .sql
+        .export(&temp_db_path, passphrase)
+        .await
+        .with_context(|| format!("failed to backup plaintext database to {:?}", temp_db_path))?;
 
-    // we re-open the database after export is finished
-    context.sql.open(context, context.get_dbfile(), false).await;
+    let res = export_backup_inner(context, &temp_db_path, &temp_path).await;
 
     match &res {
         Ok(_) => {
@@ -721,18 +600,21 @@ impl Drop for DeleteOnDrop {
     fn drop(&mut self) {
         let file = self.0.clone();
         // Not using dc_delete_file() here because it would send a DeletedBlobFile event
-        async_std::task::block_on(async move { fs::remove_file(file).await.ok() });
+        async_std::task::block_on(fs::remove_file(file)).ok();
     }
 }
 
-async fn export_backup_inner(context: &Context, temp_path: &PathBuf) -> Result<()> {
+async fn export_backup_inner(
+    context: &Context,
+    temp_db_path: &Path,
+    temp_path: &Path,
+) -> Result<()> {
     let file = File::create(temp_path).await?;
 
     let mut builder = async_tar::Builder::new(file);
 
-    // append_path_with_name() wants the source path as the first argument, append_dir_all() wants it as the second argument.
     builder
-        .append_path_with_name(context.get_dbfile(), DBFILE_BACKUP_NAME)
+        .append_path_with_name(temp_db_path, DBFILE_BACKUP_NAME)
         .await?;
 
     let read_dir: Vec<_> = fs::read_dir(context.get_blobdir()).await?.collect().await;
@@ -936,9 +818,7 @@ mod tests {
 
     #[async_std::test]
     async fn test_render_setup_file() {
-        let t = TestContext::new().await;
-
-        t.configure_alice().await;
+        let t = TestContext::new_alice().await;
         let msg = render_setup_file(&t, "hello").await.unwrap();
         println!("{}", &msg);
         // Check some substrings, indicating things got substituted.
@@ -955,11 +835,10 @@ mod tests {
 
     #[async_std::test]
     async fn test_render_setup_file_newline_replace() {
-        let t = TestContext::new().await;
+        let t = TestContext::new_alice().await;
         t.set_stock_translation(StockMessage::AcSetupMsgBody, "hello\r\nthere".to_string())
             .await
             .unwrap();
-        t.configure_alice().await;
         let msg = render_setup_file(&t, "pw").await.unwrap();
         println!("{}", &msg);
         assert!(msg.contains("<p>hello<br>there</p>"));
@@ -1012,20 +891,66 @@ mod tests {
 
     #[async_std::test]
     async fn test_export_and_import_key() {
-        let context = TestContext::new().await;
-        context.configure_alice().await;
+        let context = TestContext::new_alice().await;
         let blobdir = context.ctx.get_blobdir();
-        if let Err(err) = imex(&context.ctx, ImexMode::ExportSelfKeys, blobdir).await {
+        if let Err(err) = imex(&context.ctx, ImexMode::ExportSelfKeys, blobdir, None).await {
             panic!("got error on export: {:?}", err);
         }
 
-        let context2 = TestContext::new().await;
-        context2.configure_alice().await;
-        if let Err(err) = imex(&context2.ctx, ImexMode::ImportSelfKeys, blobdir).await {
+        let context2 = TestContext::new_alice().await;
+        if let Err(err) = imex(&context2.ctx, ImexMode::ImportSelfKeys, blobdir, None).await {
             panic!("got error on import: {:?}", err);
         }
     }
 
+    #[async_std::test]
+    async fn test_export_and_import_backup() -> Result<()> {
+        let backup_dir = tempfile::tempdir().unwrap();
+
+        let context1 = TestContext::new_alice().await;
+        assert!(context1.is_configured().await?);
+
+        let context2 = TestContext::new().await;
+        assert!(!context2.is_configured().await?);
+        assert!(has_backup(&context2, backup_dir.path().as_ref())
+            .await
+            .is_err());
+
+        // export from context1
+        assert!(imex(
+            &context1,
+            ImexMode::ExportBackup,
+            backup_dir.path().as_ref(),
+            None,
+        )
+        .await
+        .is_ok());
+        let _event = context1
+            .evtracker
+            .get_matching(|evt| matches!(evt, EventType::ImexProgress(1000)))
+            .await;
+
+        // import to context2
+        let backup = has_backup(&context2, backup_dir.path().as_ref()).await?;
+        assert!(
+            imex(&context2, ImexMode::ImportBackup, backup.as_ref(), None)
+                .await
+                .is_ok()
+        );
+        let _event = context2
+            .evtracker
+            .get_matching(|evt| matches!(evt, EventType::ImexProgress(1000)))
+            .await;
+
+        assert!(context2.is_configured().await?);
+        assert_eq!(
+            context2.get_config(Config::Addr).await?,
+            Some("alice@example.org".to_string())
+        );
+
+        Ok(())
+    }
+
     #[test]
     fn test_normalize_setup_code() {
         let norm = normalize_setup_code("123422343234423452346234723482349234");

src/job.rs

@@ -3,29 +3,24 @@
 //! This module implements a job queue maintained in the SQLite database
 //! and job types.
 use std::fmt;
-use std::future::Future;
 
-use anyhow::{bail, ensure, format_err, Context as _, Error, Result};
-use async_smtp::smtp::response::{Category, Code, Detail};
+use anyhow::{bail, format_err, Context as _, Error, Result};
 use deltachat_derive::{FromSql, ToSql};
 use rand::{thread_rng, Rng};
 
-use crate::blob::BlobObject;
-use crate::chat::{self, ChatId};
 use crate::config::Config;
-use crate::contact::{normalize_name, Contact, Modifier, Origin};
+use crate::contact::{normalize_name, Contact, ContactId, Modifier, Origin};
 use crate::context::Context;
-use crate::dc_tools::{dc_delete_file, dc_read_file, time};
-use crate::ephemeral::load_imap_deletion_msgid;
+use crate::dc_tools::time;
 use crate::events::EventType;
 use crate::imap::{Imap, ImapActionResult};
 use crate::location;
 use crate::log::LogExt;
-use crate::message::{self, Message, MessageState, MsgId};
+use crate::message::{Message, MsgId};
 use crate::mimefactory::MimeFactory;
 use crate::param::{Param, Params};
 use crate::scheduler::InterruptInfo;
-use crate::smtp::Smtp;
+use crate::smtp::{smtp_send, Smtp};
 use crate::sql;
 
 // results in ~3 weeks for the last backoff timespan
@@ -96,11 +91,6 @@ pub enum Action {
     // this is user initiated so it should have a fairly high priority
     UpdateRecentQuota = 140,
 
-    // Moving message is prioritized lower than deletion so we don't
-    // bother moving message if it is already scheduled for deletion.
-    MoveMsg = 200,
-    DeleteMsgOnImap = 210,
-
     // This job will download partially downloaded messages completely
     // and is added when download_full() is called.
     // Most messages are downloaded automatically on fetch
@@ -115,7 +105,6 @@ pub enum Action {
     MaybeSendLocations = 5005, // low priority ...
     MaybeSendLocationsEnded = 5007,
     SendMdn = 5010,
-    SendMsgToSmtp = 5901, // ... high priority
 }
 
 impl Default for Action {
@@ -133,17 +122,14 @@ impl From<Action> for Thread {
 
             Housekeeping => Thread::Imap,
             FetchExistingMsgs => Thread::Imap,
-            DeleteMsgOnImap => Thread::Imap,
             ResyncFolders => Thread::Imap,
             MarkseenMsgOnImap => Thread::Imap,
-            MoveMsg => Thread::Imap,
             UpdateRecentQuota => Thread::Imap,
             DownloadMsg => Thread::Imap,
 
             MaybeSendLocations => Thread::Smtp,
             MaybeSendLocationsEnded => Thread::Smtp,
             SendMdn => Thread::Smtp,
-            SendMsgToSmtp => Thread::Smtp,
         }
     }
 }
@@ -211,7 +197,7 @@ impl Job {
                     "UPDATE jobs SET desired_timestamp=?, tries=?, param=? WHERE id=?;",
                     paramsv![
                         self.desired_timestamp,
-                        self.tries as i64,
+                        i64::from(self.tries),
                         self.param.to_string(),
                         self.job_id as i32,
                     ],
@@ -234,222 +220,11 @@ impl Job {
         Ok(())
     }
 
-    async fn smtp_send<F, Fut>(
-        &mut self,
-        context: &Context,
-        recipients: Vec<async_smtp::EmailAddress>,
-        message: Vec<u8>,
-        job_id: u32,
-        smtp: &mut Smtp,
-        success_cb: F,
-    ) -> Status
-    where
-        F: FnOnce() -> Fut,
-        Fut: Future<Output = Result<()>>,
-    {
-        // hold the smtp lock during sending of a job and
-        // its ok/error response processing. Note that if a message
-        // was sent we need to mark it in the database ASAP as we
-        // otherwise might send it twice.
-        if std::env::var(crate::DCC_MIME_DEBUG).is_ok() {
-            info!(context, "smtp-sending out mime message:");
-            println!("{}", String::from_utf8_lossy(&message));
-        }
-
-        smtp.connectivity.set_working(context).await;
-
-        let send_result = smtp.send(context, recipients, message, job_id).await;
-        smtp.last_send_error = send_result.as_ref().err().map(|e| e.to_string());
-
-        let status = match send_result {
-            Err(crate::smtp::send::Error::SmtpSend(err)) => {
-                // Remote error, retry later.
-                warn!(context, "SMTP failed to send: {:?}", &err);
-
-                let res = match err {
-                    async_smtp::smtp::error::Error::Permanent(ref response) => {
-                        // Workaround for incorrectly configured servers returning permanent errors
-                        // instead of temporary ones.
-                        let maybe_transient = match response.code {
-                            // Sometimes servers send a permanent error when actually it is a temporary error
-                            // For documentation see <https://tools.ietf.org/html/rfc3463>
-                            Code {
-                                category: Category::MailSystem,
-                                detail: Detail::Zero,
-                                ..
-                            } => {
-                                // Ignore status code 5.5.0, see <https://support.delta.chat/t/every-other-message-gets-stuck/877/2>
-                                // Maybe incorrectly configured Postfix milter with "reject" instead of "tempfail", which returns
-                                // "550 5.5.0 Service unavailable" instead of "451 4.7.1 Service unavailable - try again later".
-                                //
-                                // Other enhanced status codes, such as Postfix
-                                // "550 5.1.1 <foobar@example.org>: Recipient address rejected: User unknown in local recipient table"
-                                // are not ignored.
-                                response.first_word() == Some(&"5.5.0".to_string())
-                            }
-                            _ => false,
-                        };
-
-                        if maybe_transient {
-                            Status::RetryLater
-                        } else {
-                            // If we do not retry, add an info message to the chat.
-                            // Yandex error "554 5.7.1 [2] Message rejected under suspicion of SPAM; https://ya.cc/..."
-                            // should definitely go here, because user has to open the link to
-                            // resume message sending.
-                            Status::Finished(Err(format_err!("Permanent SMTP error: {}", err)))
-                        }
-                    }
-                    async_smtp::smtp::error::Error::Transient(ref response) => {
-                        // We got a transient 4xx response from SMTP server.
-                        // Give some time until the server-side error maybe goes away.
-
-                        if let Some(first_word) = response.first_word() {
-                            if first_word.ends_with(".1.1")
-                                || first_word.ends_with(".1.2")
-                                || first_word.ends_with(".1.3")
-                            {
-                                // Sometimes we receive transient errors that should be permanent.
-                                // Any extended smtp status codes like x.1.1, x.1.2 or x.1.3 that we
-                                // receive as a transient error are misconfigurations of the smtp server.
-                                // See <https://tools.ietf.org/html/rfc3463#section-3.2>
-                                info!(context, "Smtp-job #{} Received extended status code {} for a transient error. This looks like a misconfigured smtp server, let's fail immediatly", self.job_id, first_word);
-                                Status::Finished(Err(format_err!("Permanent SMTP error: {}", err)))
-                            } else {
-                                Status::RetryLater
-                            }
-                        } else {
-                            Status::RetryLater
-                        }
-                    }
-                    _ => {
-                        if smtp.has_maybe_stale_connection().await {
-                            info!(context, "stale connection? immediately reconnecting");
-                            Status::RetryNow
-                        } else {
-                            Status::RetryLater
-                        }
-                    }
-                };
-
-                // this clears last_success info
-                smtp.disconnect().await;
-
-                res
-            }
-            Err(crate::smtp::send::Error::Envelope(err)) => {
-                // Local error, job is invalid, do not retry.
-                smtp.disconnect().await;
-                warn!(context, "SMTP job is invalid: {}", err);
-                Status::Finished(Err(err.into()))
-            }
-            Err(crate::smtp::send::Error::NoTransport) => {
-                // Should never happen.
-                // It does not even make sense to disconnect here.
-                error!(context, "SMTP job failed because SMTP has no transport");
-                Status::Finished(Err(format_err!("SMTP has not transport")))
-            }
-            Err(crate::smtp::send::Error::Other(err)) => {
-                // Local error, job is invalid, do not retry.
-                smtp.disconnect().await;
-                warn!(context, "unable to load job: {}", err);
-                Status::Finished(Err(err))
-            }
-            Ok(()) => {
-                job_try!(success_cb().await);
-                Status::Finished(Ok(()))
-            }
-        };
-
-        if let Status::Finished(Err(err)) = &status {
-            // We couldn't send the message, so mark it as failed
-            let msg_id = MsgId::new(self.foreign_id);
-            message::set_msg_failed(context, msg_id, Some(err.to_string())).await;
-        }
-        status
-    }
-
-    pub(crate) async fn send_msg_to_smtp(&mut self, context: &Context, smtp: &mut Smtp) -> Status {
-        //  SMTP server, if not yet done
-        if let Err(err) = smtp.connect_configured(context).await {
-            warn!(context, "SMTP connection failure: {:?}", err);
-            smtp.last_send_error = Some(format!("SMTP connection failure: {:#}", err));
-            return Status::RetryLater;
-        }
-
-        let filename = job_try!(job_try!(self
-            .param
-            .get_path(Param::File, context)
-            .map_err(|_| format_err!("Can't get filename")))
-        .ok_or_else(|| format_err!("Can't get filename")));
-        let body = job_try!(dc_read_file(context, &filename).await);
-        let recipients = job_try!(self.param.get(Param::Recipients).ok_or_else(|| {
-            warn!(context, "Missing recipients for job {}", self.job_id);
-            format_err!("Missing recipients")
-        }));
-
-        let recipients_list = recipients
-            .split('\x1e')
-            .filter_map(
-                |addr| match async_smtp::EmailAddress::new(addr.to_string()) {
-                    Ok(addr) => Some(addr),
-                    Err(err) => {
-                        warn!(context, "invalid recipient: {} {:?}", addr, err);
-                        None
-                    }
-                },
-            )
-            .collect::<Vec<_>>();
-
-        /* if there is a msg-id and it does not exist in the db, cancel sending.
-        this happends if dc_delete_msgs() was called
-        before the generated mime was sent out */
-        if 0 != self.foreign_id {
-            match message::exists(context, MsgId::new(self.foreign_id)).await {
-                Ok(exists) => {
-                    if !exists {
-                        return Status::Finished(Err(format_err!(
-                            "Not sending Message {} as it was deleted",
-                            self.foreign_id
-                        )));
-                    }
-                }
-                Err(err) => {
-                    warn!(context, "failed to check message existence: {:?}", err);
-                    smtp.last_send_error =
-                        Some(format!("failed to check message existence: {:#}", err));
-                    return Status::RetryLater;
-                }
-            }
-        };
-
-        let foreign_id = self.foreign_id;
-        self.smtp_send(context, recipients_list, body, self.job_id, smtp, || {
-            async move {
-                // smtp success, update db ASAP, then delete smtp file
-                if 0 != foreign_id {
-                    set_delivered(context, MsgId::new(foreign_id)).await?;
-                }
-                // now also delete the generated file
-                dc_delete_file(context, filename).await;
-
-                // finally, create another send-job if there are items to be synced.
-                // triggering sync-job after msg-send-job guarantees, the recipient has grpid etc.
-                // once the sync message arrives.
-                // if there are no items to sync, this function returns fast.
-                context.send_sync_msg().await?;
-
-                Ok(())
-            }
-        })
-        .await
-    }
-
     /// Get `SendMdn` jobs with foreign_id equal to `contact_id` excluding the `job_id` job.
     async fn get_additional_mdn_jobs(
         &self,
         context: &Context,
-        contact_id: u32,
+        contact_id: ContactId,
     ) -> Result<(Vec<u32>, Vec<String>)> {
         // Extract message IDs from job parameters
         let res: Vec<(u32, MsgId)> = context
@@ -496,7 +271,7 @@ impl Job {
             return Status::Finished(Err(format_err!("MDNs are disabled")));
         }
 
-        let contact_id = self.foreign_id;
+        let contact_id = ContactId::new(self.foreign_id);
         let contact = job_try!(Contact::load_from_db(context, contact_id).await);
         if contact.is_blocked() {
             return Status::Finished(Err(format_err!("Contact is blocked")));
@@ -543,157 +318,13 @@ impl Job {
             return Status::RetryLater;
         }
 
-        self.smtp_send(context, recipients, body, self.job_id, smtp, || {
-            async move {
-                // Remove additional SendMdn jobs we have aggregated into this one.
-                kill_ids(context, &additional_job_ids).await?;
-                Ok(())
-            }
-        })
-        .await
-    }
-
-    async fn move_msg(&mut self, context: &Context, imap: &mut Imap) -> Status {
-        if let Err(err) = imap.prepare(context).await {
-            warn!(context, "could not connect: {:?}", err);
-            return Status::RetryLater;
+        let status = smtp_send(context, &recipients, &body, smtp, msg_id, 0).await;
+        if matches!(status, Status::Finished(Ok(_))) {
+            // Remove additional SendMdn jobs we have aggregated into this one.
+            job_try!(kill_ids(context, &additional_job_ids).await);
         }
 
-        let msg = job_try!(Message::load_from_db(context, MsgId::new(self.foreign_id)).await);
-        let server_folder = &job_try!(msg
-            .server_folder
-            .context("Can't move message out of folder if we don't know the current folder"));
-
-        let move_res = msg.id.needs_move(context, server_folder).await;
-        let dest_folder = match move_res {
-            Err(e) => {
-                warn!(context, "could not load dest folder: {}", e);
-                return Status::RetryLater;
-            }
-            Ok(None) => {
-                warn!(
-                    context,
-                    "msg {} does not need to be moved from {}", msg.id, server_folder
-                );
-                return Status::Finished(Ok(()));
-            }
-            Ok(Some(config)) => match context.get_config(config).await {
-                Ok(folder) => folder,
-                Err(err) => {
-                    warn!(context, "failed to load config: {}", err);
-                    return Status::RetryLater;
-                }
-            },
-        };
-
-        if let Some(dest_folder) = dest_folder {
-            match imap
-                .mv(context, server_folder, msg.server_uid, &dest_folder)
-                .await
-            {
-                ImapActionResult::RetryLater => Status::RetryLater,
-                ImapActionResult::Success => {
-                    // Rust-Imap provides no target uid on mv, so just set it to 0, update again when precheck_imf() is called for the moved message
-                    message::update_server_uid(context, &msg.rfc724_mid, &dest_folder, 0).await;
-                    Status::Finished(Ok(()))
-                }
-                ImapActionResult::Failed => {
-                    Status::Finished(Err(format_err!("IMAP action failed")))
-                }
-                ImapActionResult::AlreadyDone => Status::Finished(Ok(())),
-            }
-        } else {
-            Status::Finished(Err(format_err!("No mvbox folder configured")))
-        }
-    }
-
-    /// Deletes a message on the server.
-    ///
-    /// `foreign_id` is a MsgId.
-    ///
-    /// If the message is in the trash chat or hidden, this job
-    /// removes database record, otherwise it only clears the
-    /// `server_uid` column.  If there are no more records pointing to
-    /// the same message on the server, the job actually removes the
-    /// message on the server.
-    async fn delete_msg_on_imap(&mut self, context: &Context, imap: &mut Imap) -> Status {
-        if let Err(err) = imap.prepare(context).await {
-            warn!(context, "could not connect: {:?}", err);
-            return Status::RetryLater;
-        }
-
-        let msg = job_try!(Message::load_from_db(context, MsgId::new(self.foreign_id)).await);
-
-        if !msg.rfc724_mid.is_empty() {
-            let cnt = message::rfc724_mid_cnt(context, &msg.rfc724_mid).await;
-            info!(
-                context,
-                "Running delete job for message {} which has {} entries in the database",
-                &msg.rfc724_mid,
-                cnt
-            );
-            if cnt > 1 {
-                info!(
-                    context,
-                    "The message is deleted from the server when all parts are deleted.",
-                );
-            } else if cnt == 0 {
-                warn!(
-                    context,
-                    "The message {} has no UID on the server to delete", &msg.rfc724_mid
-                );
-            } else {
-                /* if this is the last existing part of the message,
-                we delete the message from the server */
-                let mid = msg.rfc724_mid;
-                let server_folder = msg.server_folder.as_ref().unwrap();
-                let res = if msg.server_uid == 0 {
-                    // Message is already deleted on IMAP server.
-                    ImapActionResult::AlreadyDone
-                } else {
-                    imap.delete_msg(context, &mid, server_folder, msg.server_uid)
-                        .await
-                };
-                match res {
-                    ImapActionResult::AlreadyDone | ImapActionResult::Success => {}
-                    ImapActionResult::RetryLater | ImapActionResult::Failed => {
-                        // If job has failed, for example due to some
-                        // IMAP bug, we postpone it instead of failing
-                        // immediately. This will prevent adding it
-                        // immediately again if user has enabled
-                        // automatic message deletion. Without this,
-                        // we might waste a lot of traffic constantly
-                        // retrying message deletion.
-                        return Status::RetryLater;
-                    }
-                }
-            }
-            if msg.chat_id.is_trash() || msg.hidden {
-                // Messages are stored in trash chat only to keep
-                // their server UID and Message-ID. Once message is
-                // deleted from the server, database record can be
-                // removed as well.
-                //
-                // Hidden messages are similar to trashed, but are
-                // related to some chat. We also delete their
-                // database records.
-                job_try!(msg.id.delete_from_db(context).await)
-            } else {
-                // Remove server UID from the database record.
-                //
-                // We have either just removed the message from the
-                // server, in which case UID is not valid anymore, or
-                // we have more refernces to the same server UID, so
-                // we remove UID to reduce the number of messages
-                // pointing to the corresponding UID. Once the counter
-                // reaches zero, we will remove the message.
-                job_try!(msg.id.unlink(context).await);
-            }
-            Status::Finished(Ok(()))
-        } else {
-            /* eg. device messages have no Message-ID */
-            Status::Finished(Ok(()))
-        }
+        status
     }
 
     /// Read the recipients from old emails sent by the user and add them as contacts.
@@ -734,37 +365,38 @@ impl Job {
         Status::Finished(Ok(()))
     }
 
-    /// Synchronizes UIDs for sentbox, inbox and mvbox, in this order.
-    ///
-    /// If a copy of the message is present in multiple folders, mvbox
-    /// is preferred to inbox, which is in turn preferred to
-    /// sentbox. This is because in the database it is impossible to
-    /// store multiple UIDs for one message, so we prefer to
-    /// automatically delete messages in the folders managed by Delta
-    /// Chat in contrast to the Sent folder, which is normally managed
-    /// by the user via webmail or another email client.
+    /// Synchronizes UIDs for all folders.
     async fn resync_folders(&mut self, context: &Context, imap: &mut Imap) -> Status {
         if let Err(err) = imap.prepare(context).await {
             warn!(context, "could not connect: {:?}", err);
             return Status::RetryLater;
         }
 
-        let sentbox_folder = job_try!(context.get_config(Config::ConfiguredSentboxFolder).await);
-        if let Some(sentbox_folder) = sentbox_folder {
-            job_try!(imap.resync_folder_uids(context, sentbox_folder).await);
+        let all_folders = match imap.list_folders(context).await {
+            Ok(v) => v,
+            Err(e) => {
+                warn!(context, "Listing folders for resync failed: {:#}", e);
+                return Status::RetryLater;
+            }
+        };
+
+        let mut any_failed = false;
+
+        for folder in all_folders {
+            if let Err(e) = imap
+                .resync_folder_uids(context, folder.name().to_string())
+                .await
+            {
+                warn!(context, "{:#}", e);
+                any_failed = true;
+            }
         }
 
-        let inbox_folder = job_try!(context.get_config(Config::ConfiguredInboxFolder).await);
-        if let Some(inbox_folder) = inbox_folder {
-            job_try!(imap.resync_folder_uids(context, inbox_folder).await);
+        if any_failed {
+            Status::RetryLater
+        } else {
+            Status::Finished(Ok(()))
         }
-
-        let mvbox_folder = job_try!(context.get_config(Config::ConfiguredMvboxFolder).await);
-        if let Some(mvbox_folder) = mvbox_folder {
-            job_try!(imap.resync_folder_uids(context, mvbox_folder).await);
-        }
-
-        Status::Finished(Ok(()))
     }
 
     async fn markseen_msg_on_imap(&mut self, context: &Context, imap: &mut Imap) -> Status {
@@ -774,55 +406,58 @@ impl Job {
         }
 
         let msg = job_try!(Message::load_from_db(context, MsgId::new(self.foreign_id)).await);
-
-        let folder = msg.server_folder.as_ref().unwrap();
-
-        let result = if msg.server_uid == 0 {
-            // The message is moved or deleted by us.
-            //
-            // Do not call set_seen with zero UID, as it will return
-            // ImapActionResult::RetryLater, but we do not want to
-            // retry. If the message was moved, we will create another
-            // job to mark the message as seen later. If it was
-            // deleted, there is nothing to do.
-            info!(context, "Can't mark message as seen: No UID");
-            ImapActionResult::Failed
-        } else {
-            imap.set_seen(context, folder, msg.server_uid).await
-        };
-
-        match result {
-            ImapActionResult::RetryLater => Status::RetryLater,
-            ImapActionResult::AlreadyDone => Status::Finished(Ok(())),
-            ImapActionResult::Success | ImapActionResult::Failed => {
-                // XXX the message might just have been moved
-                // we want to send out an MDN anyway
-                // The job will not be retried so locally
-                // there is no risk of double-sending MDNs.
-                //
-                // Read receipts for system messages are never
-                // sent. These messages have no place to display
-                // received read receipt anyway.  And since their text
-                // is locally generated, quoting them is dangerous as
-                // it may contain contact names. E.g., for original
-                // message "Group left by me", a read receipt will
-                // quote "Group left by <name>", and the name can be a
-                // display name stored in address book rather than
-                // the name sent in the From field by the user.
-                if msg.param.get_bool(Param::WantsMdn).unwrap_or_default()
-                    && !msg.is_system_message()
-                {
-                    let mdns_enabled = job_try!(context.get_config_bool(Config::MdnsEnabled).await);
-                    if mdns_enabled {
-                        if let Err(err) = send_mdn(context, &msg).await {
-                            warn!(context, "could not send out mdn for {}: {}", msg.id, err);
-                            return Status::Finished(Err(err));
-                        }
+        let row = job_try!(
+            context
+                .sql
+                .query_row_optional(
+                    "SELECT uid, folder FROM imap
+                    WHERE rfc724_mid=? AND folder=target
+                    ORDER BY uid ASC
+                    LIMIT 1",
+                    paramsv![msg.rfc724_mid],
+                    |row| {
+                        let uid: u32 = row.get(0)?;
+                        let folder: String = row.get(1)?;
+                        Ok((uid, folder))
                     }
+                )
+                .await
+        );
+        if let Some((server_uid, server_folder)) = row {
+            let result = imap.set_seen(context, &server_folder, server_uid).await;
+            match result {
+                ImapActionResult::RetryLater => return Status::RetryLater,
+                ImapActionResult::Success | ImapActionResult::Failed => {}
+            }
+        } else {
+            info!(
+                context,
+                "Can't mark the message {} as seen on IMAP because there is no known UID",
+                msg.rfc724_mid
+            );
+        }
+
+        // XXX we send MDN even in case of failure to mark the messages as seen, e.g. if it was
+        // already deleted on the server by another device. The job will not be retried so locally
+        // there is no risk of double-sending MDNs.
+        //
+        // Read receipts for system messages are never sent. These messages have no place to
+        // display received read receipt anyway.  And since their text is locally generated,
+        // quoting them is dangerous as it may contain contact names. E.g., for original message
+        // "Group left by me", a read receipt will quote "Group left by <name>", and the name can
+        // be a display name stored in address book rather than the name sent in the From field by
+        // the user.
+        if msg.param.get_bool(Param::WantsMdn).unwrap_or_default() && !msg.is_system_message() {
+            let mdns_enabled = job_try!(context.get_config_bool(Config::MdnsEnabled).await);
+            if mdns_enabled {
+                if let Err(err) = send_mdn(context, &msg).await {
+                    warn!(context, "could not send out mdn for {}: {}", msg.id, err);
+                    return Status::Finished(Err(err));
                 }
-                Status::Finished(Ok(()))
             }
         }
+
+        Status::Finished(Ok(()))
     }
 }
 
@@ -837,9 +472,12 @@ pub async fn kill_action(context: &Context, action: Action) -> Result<()> {
 
 /// Remove jobs with specified IDs.
 async fn kill_ids(context: &Context, job_ids: &[u32]) -> Result<()> {
+    if job_ids.is_empty() {
+        return Ok(());
+    }
     let q = format!(
         "DELETE FROM jobs WHERE id IN({})",
-        job_ids.iter().map(|_| "?").collect::<Vec<&str>>().join(",")
+        sql::repeat_vars(job_ids.len())?
     );
     context
         .sql
@@ -859,17 +497,6 @@ pub async fn action_exists(context: &Context, action: Action) -> Result<bool> {
     Ok(exists)
 }
 
-async fn set_delivered(context: &Context, msg_id: MsgId) -> Result<()> {
-    message::update_msg_state(context, msg_id, MessageState::OutDelivered).await;
-    let chat_id: ChatId = context
-        .sql
-        .query_get_value("SELECT chat_id FROM msgs WHERE id=?", paramsv![msg_id])
-        .await?
-        .unwrap_or_default();
-    context.emit_event(EventType::MsgDelivered { chat_id, msg_id });
-    Ok(())
-}
-
 async fn add_all_recipients_as_contacts(context: &Context, imap: &mut Imap, folder: Config) {
     let mailbox = if let Ok(Some(m)) = context.get_config(folder).await {
         m
@@ -888,7 +515,7 @@ async fn add_all_recipients_as_contacts(context: &Context, imap: &mut Imap, fold
                 let display_name_normalized = contact
                     .display_name
                     .as_ref()
-                    .map(normalize_name)
+                    .map(|s| normalize_name(s))
                     .unwrap_or_default();
 
                 match Contact::add_or_lookup(
@@ -915,144 +542,11 @@ async fn add_all_recipients_as_contacts(context: &Context, imap: &mut Imap, fold
     };
 }
 
-/// Constructs a job for sending a message.
-///
-/// Returns `None` if no messages need to be sent out.
-///
-/// In order to be processed, must be `add`ded.
-pub async fn send_msg_job(context: &Context, msg_id: MsgId) -> Result<Option<Job>> {
-    let mut msg = Message::load_from_db(context, msg_id).await?;
-    msg.try_calc_and_set_dimensions(context).await.ok();
-
-    /* create message */
-    let needs_encryption = msg.param.get_bool(Param::GuaranteeE2ee).unwrap_or_default();
-
-    let attach_selfavatar = match chat::shall_attach_selfavatar(context, msg.chat_id).await {
-        Ok(attach_selfavatar) => attach_selfavatar,
-        Err(err) => {
-            warn!(context, "job: cannot get selfavatar-state: {}", err);
-            false
-        }
-    };
-
-    let mimefactory = MimeFactory::from_msg(context, &msg, attach_selfavatar).await?;
-
-    let mut recipients = mimefactory.recipients();
-
-    let from = context
-        .get_config(Config::ConfiguredAddr)
-        .await?
-        .unwrap_or_default();
-    let lowercase_from = from.to_lowercase();
-
-    // Send BCC to self if it is enabled and we are not going to
-    // delete it immediately.
-    if context.get_config_bool(Config::BccSelf).await?
-        && context.get_config_delete_server_after().await? != Some(0)
-        && !recipients
-            .iter()
-            .any(|x| x.to_lowercase() == lowercase_from)
-    {
-        recipients.push(from);
-    }
-
-    if recipients.is_empty() {
-        // may happen eg. for groups with only SELF and bcc_self disabled
-        info!(
-            context,
-            "message {} has no recipient, skipping smtp-send", msg_id
-        );
-        set_delivered(context, msg_id).await?;
-        return Ok(None);
-    }
-
-    let rendered_msg = match mimefactory.render(context).await {
-        Ok(res) => Ok(res),
-        Err(err) => {
-            message::set_msg_failed(context, msg_id, Some(err.to_string())).await;
-            Err(err)
-        }
-    }?;
-
-    if needs_encryption && !rendered_msg.is_encrypted {
-        /* unrecoverable */
-        message::set_msg_failed(
-            context,
-            msg_id,
-            Some("End-to-end-encryption unavailable unexpectedly."),
-        )
-        .await;
-        bail!(
-            "e2e encryption unavailable {} - {:?}",
-            msg_id,
-            needs_encryption
-        );
-    }
-
-    if rendered_msg.is_gossiped {
-        msg.chat_id.set_gossiped_timestamp(context, time()).await?;
-    }
-
-    if 0 != rendered_msg.last_added_location_id {
-        if let Err(err) = location::set_kml_sent_timestamp(context, msg.chat_id, time()).await {
-            error!(context, "Failed to set kml sent_timestamp: {:?}", err);
-        }
-        if !msg.hidden {
-            if let Err(err) =
-                location::set_msg_location_id(context, msg.id, rendered_msg.last_added_location_id)
-                    .await
-            {
-                error!(context, "Failed to set msg_location_id: {:?}", err);
-            }
-        }
-    }
-
-    if let Some(sync_ids) = rendered_msg.sync_ids_to_delete {
-        if let Err(err) = context.delete_sync_ids(sync_ids).await {
-            error!(context, "Failed to delete sync ids: {:?}", err);
-        }
-    }
-
-    if attach_selfavatar {
-        if let Err(err) = msg.chat_id.set_selfavatar_timestamp(context, time()).await {
-            error!(context, "Failed to set selfavatar timestamp: {:?}", err);
-        }
-    }
-
-    if rendered_msg.is_encrypted && !needs_encryption {
-        msg.param.set_int(Param::GuaranteeE2ee, 1);
-        msg.update_param(context).await;
-    }
-
-    ensure!(!recipients.is_empty(), "no recipients for smtp job set");
-    let mut param = Params::new();
-    let bytes = &rendered_msg.message;
-    let blob = BlobObject::create(context, &rendered_msg.rfc724_mid, bytes).await?;
-
-    let recipients = recipients.join("\x1e");
-    param.set(Param::File, blob.as_name());
-    param.set(Param::Recipients, &recipients);
-
-    msg.subject = rendered_msg.subject.clone();
-    msg.update_subject(context).await;
-
-    let job = create(Action::SendMsgToSmtp, msg_id.to_u32(), param, 0)?;
-
-    Ok(Some(job))
-}
-
 pub(crate) enum Connection<'a> {
     Inbox(&'a mut Imap),
     Smtp(&'a mut Smtp),
 }
 
-pub(crate) async fn load_imap_deletion_job(context: &Context) -> Result<Option<Job>> {
-    let res = load_imap_deletion_msgid(context)
-        .await?
-        .map(|msg_id| Job::new(Action::DeleteMsgOnImap, msg_id.to_u32(), Params::new(), 0));
-    Ok(res)
-}
-
 impl<'a> fmt::Display for Connection<'a> {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         match self {
@@ -1155,16 +649,13 @@ async fn perform_job_action(
 
     let try_res = match job.action {
         Action::Unknown => Status::Finished(Err(format_err!("Unknown job id found"))),
-        Action::SendMsgToSmtp => job.send_msg_to_smtp(context, connection.smtp()).await,
         Action::SendMdn => job.send_mdn(context, connection.smtp()).await,
         Action::MaybeSendLocations => location::job_maybe_send_locations(context, job).await,
         Action::MaybeSendLocationsEnded => {
             location::job_maybe_send_locations_ended(context, job).await
         }
-        Action::DeleteMsgOnImap => job.delete_msg_on_imap(context, connection.inbox()).await,
         Action::ResyncFolders => job.resync_folders(context, connection.inbox()).await,
         Action::MarkseenMsgOnImap => job.markseen_msg_on_imap(context, connection.inbox()).await,
-        Action::MoveMsg => job.move_msg(context, connection.inbox()).await,
         Action::FetchExistingMsgs => job.fetch_existing_msgs(context, connection.inbox()).await,
         Action::Housekeeping => {
             sql::housekeeping(context).await.ok_or_log(context);
@@ -1197,7 +688,7 @@ fn get_backoff_time_offset(tries: u32, action: Action) -> i64 {
             if seconds < 1 {
                 seconds = 1;
             }
-            seconds as i64
+            i64::from(seconds)
         }
     }
 }
@@ -1206,7 +697,11 @@ async fn send_mdn(context: &Context, msg: &Message) -> Result<()> {
     let mut param = Params::new();
     param.set(Param::MsgId, msg.id.to_u32().to_string());
 
-    add(context, Job::new(Action::SendMdn, msg.from_id, param, 0)).await?;
+    add(
+        context,
+        Job::new(Action::SendMdn, msg.from_id.to_u32(), param, 0),
+    )
+    .await?;
 
     Ok(())
 }
@@ -1221,16 +716,6 @@ pub(crate) async fn schedule_resync(context: &Context) -> Result<()> {
     Ok(())
 }
 
-/// Creates a job.
-pub fn create(action: Action, foreign_id: u32, param: Params, delay_seconds: i64) -> Result<Job> {
-    ensure!(
-        action != Action::Unknown,
-        "Invalid action passed to job_add"
-    );
-
-    Ok(Job::new(action, foreign_id, param, delay_seconds))
-}
-
 /// Adds a job to the database, scheduling it.
 pub async fn add(context: &Context, job: Job) -> Result<()> {
     let action = job.action;
@@ -1241,26 +726,17 @@ pub async fn add(context: &Context, job: Job) -> Result<()> {
         match action {
             Action::Unknown => unreachable!(),
             Action::Housekeeping
-            | Action::DeleteMsgOnImap
             | Action::ResyncFolders
             | Action::MarkseenMsgOnImap
             | Action::FetchExistingMsgs
-            | Action::MoveMsg
             | Action::UpdateRecentQuota
             | Action::DownloadMsg => {
                 info!(context, "interrupt: imap");
-                context
-                    .interrupt_inbox(InterruptInfo::new(false, None))
-                    .await;
+                context.interrupt_inbox(InterruptInfo::new(false)).await;
             }
-            Action::MaybeSendLocations
-            | Action::MaybeSendLocationsEnded
-            | Action::SendMdn
-            | Action::SendMsgToSmtp => {
+            Action::MaybeSendLocations | Action::MaybeSendLocationsEnded | Action::SendMdn => {
                 info!(context, "interrupt: smtp");
-                context
-                    .interrupt_smtp(InterruptInfo::new(false, None))
-                    .await;
+                context.interrupt_smtp(InterruptInfo::new(false)).await;
             }
         }
     }
@@ -1295,20 +771,9 @@ pub(crate) async fn load_next(
     let query;
     let params;
     let t = time();
-    let m;
     let thread_i = thread as i64;
 
-    if let Some(msg_id) = info.msg_id {
-        query = r#"
-SELECT id, action, foreign_id, param, added_timestamp, desired_timestamp, tries
-FROM jobs
-WHERE thread=? AND foreign_id=?
-ORDER BY action DESC, added_timestamp
-LIMIT 1;
-"#;
-        m = msg_id;
-        params = paramsv![thread_i, m];
-    } else if !info.probe_network {
+    if !info.probe_network {
         // processing for first-try and after backoff-timeouts:
         // process jobs in the order they were added.
         query = r#"
@@ -1378,12 +843,6 @@ LIMIT 1;
         }
         Thread::Imap => {
             if let Some(job) = job {
-                if job.action < Action::DeleteMsgOnImap {
-                    Ok(load_imap_deletion_job(context).await?.or(Some(job)))
-                } else {
-                    Ok(Some(job))
-                }
-            } else if let Some(job) = load_imap_deletion_job(context).await? {
                 Ok(Some(job))
             } else {
                 Ok(load_housekeeping_job(context).await?)
@@ -1409,8 +868,12 @@ mod tests {
                  VALUES (?, ?, ?, ?, ?, ?);",
                 paramsv![
                     now,
-                    Thread::from(Action::MoveMsg),
-                    if valid { Action::MoveMsg as i32 } else { -1 },
+                    Thread::from(Action::DownloadMsg),
+                    if valid {
+                        Action::DownloadMsg as i32
+                    } else {
+                        -1
+                    },
                     foreign_id,
                     Params::new().to_string(),
                     now
@@ -1429,8 +892,8 @@ mod tests {
         insert_job(&t, 1, false).await; // This can not be loaded into Job struct.
         let jobs = load_next(
             &t,
-            Thread::from(Action::MoveMsg),
-            &InterruptInfo::new(false, None),
+            Thread::from(Action::DownloadMsg),
+            &InterruptInfo::new(false),
         )
         .await?;
         // The housekeeping job should be loaded as we didn't run housekeeping in the last day:
@@ -1439,8 +902,8 @@ mod tests {
         insert_job(&t, 1, true).await;
         let jobs = load_next(
             &t,
-            Thread::from(Action::MoveMsg),
-            &InterruptInfo::new(false, None),
+            Thread::from(Action::DownloadMsg),
+            &InterruptInfo::new(false),
         )
         .await?;
         assert!(jobs.is_some());
@@ -1455,8 +918,8 @@ mod tests {
 
         let jobs = load_next(
             &t,
-            Thread::from(Action::MoveMsg),
-            &InterruptInfo::new(false, None),
+            Thread::from(Action::DownloadMsg),
+            &InterruptInfo::new(false),
         )
         .await?;
         assert!(jobs.is_some());

src/key.rs

@@ -4,7 +4,7 @@ use std::collections::BTreeMap;
 use std::fmt;
 use std::io::Cursor;
 
-use anyhow::{format_err, Result};
+use anyhow::{format_err, Context as _, Result};
 use async_trait::async_trait;
 use num_traits::FromPrimitive;
 use pgp::composed::Deserializable;
@@ -50,8 +50,7 @@ pub trait DcKey: Serialize + Deserializable + KeyTrait + Clone {
     /// the ASCII-armored representation.
     fn from_asc(data: &str) -> Result<(Self::KeyType, BTreeMap<String, String>)> {
         let bytes = data.as_bytes();
-        Self::KeyType::from_armor_single(Cursor::new(bytes))
-            .map_err(|err| format_err!("rPGP error: {}", err))
+        Self::KeyType::from_armor_single(Cursor::new(bytes)).context("rPGP error")
     }
 
     /// Load the users' default key from the database.
@@ -202,7 +201,7 @@ async fn generate_keypair(context: &Context) -> Result<KeyPair> {
     let addr = context
         .get_config(Config::ConfiguredAddr)
         .await?
-        .ok_or_else(|| format_err!("No address configured"))?;
+        .context("no address configured")?;
     let addr = EmailAddress::new(&addr)?;
     let _guard = context.generating_key_mutex.lock().await;
 
@@ -289,17 +288,17 @@ pub async fn store_self_keypair(
             paramsv![public_key, secret_key],
         )
         .await
-        .map_err(|err| err.context("failed to remove old use of key"))?;
+        .context("failed to remove old use of key")?;
     if default == KeyPairUse::Default {
         context
             .sql
             .execute("UPDATE keypairs SET is_default=0;", paramsv![])
             .await
-            .map_err(|err| err.context("failed to clear default"))?;
+            .context("failed to clear default")?;
     }
     let is_default = match default {
-        KeyPairUse::Default => true as i32,
-        KeyPairUse::ReadOnly => false as i32,
+        KeyPairUse::Default => i32::from(true),
+        KeyPairUse::ReadOnly => i32::from(false),
     };
 
     let addr = keypair.addr.to_string();
@@ -313,13 +312,13 @@ pub async fn store_self_keypair(
             paramsv![addr, is_default, public_key, secret_key, t],
         )
         .await
-        .map_err(|err| err.context("failed to insert keypair"))?;
+        .context("failed to insert keypair")?;
 
     Ok(())
 }
 
 /// A key fingerprint
-#[derive(Clone, Eq, PartialEq, Hash)]
+#[derive(Clone, Eq, PartialEq, Hash, serde::Serialize, serde::Deserialize)]
 pub struct Fingerprint(Vec<u8>);
 
 impl Fingerprint {
@@ -510,8 +509,7 @@ i8pcjGO+IZffvyZJVRWfVooBJmWWbPB1pueo3tx8w3+fcuzpxz+RLFKaPyqXO+dD
     #[async_std::test]
     async fn test_load_self_existing() {
         let alice = alice_keypair();
-        let t = TestContext::new().await;
-        t.configure_alice().await;
+        let t = TestContext::new_alice().await;
         let pubkey = SignedPublicKey::load_self(&t).await.unwrap();
         assert_eq!(alice.public, pubkey);
         let seckey = SignedSecretKey::load_self(&t).await.unwrap();
@@ -521,7 +519,7 @@ i8pcjGO+IZffvyZJVRWfVooBJmWWbPB1pueo3tx8w3+fcuzpxz+RLFKaPyqXO+dD
     #[async_std::test]
     async fn test_load_self_generate_public() {
         let t = TestContext::new().await;
-        t.set_config(Config::ConfiguredAddr, Some("alice@example.com"))
+        t.set_config(Config::ConfiguredAddr, Some("alice@example.org"))
             .await
             .unwrap();
         let key = SignedPublicKey::load_self(&t).await;
@@ -531,7 +529,7 @@ i8pcjGO+IZffvyZJVRWfVooBJmWWbPB1pueo3tx8w3+fcuzpxz+RLFKaPyqXO+dD
     #[async_std::test]
     async fn test_load_self_generate_secret() {
         let t = TestContext::new().await;
-        t.set_config(Config::ConfiguredAddr, Some("alice@example.com"))
+        t.set_config(Config::ConfiguredAddr, Some("alice@example.org"))
             .await
             .unwrap();
         let key = SignedSecretKey::load_self(&t).await;
@@ -543,7 +541,7 @@ i8pcjGO+IZffvyZJVRWfVooBJmWWbPB1pueo3tx8w3+fcuzpxz+RLFKaPyqXO+dD
         use std::thread;
 
         let t = TestContext::new().await;
-        t.set_config(Config::ConfiguredAddr, Some("alice@example.com"))
+        t.set_config(Config::ConfiguredAddr, Some("alice@example.org"))
             .await
             .unwrap();
         let thr0 = {
@@ -589,27 +587,6 @@ i8pcjGO+IZffvyZJVRWfVooBJmWWbPB1pueo3tx8w3+fcuzpxz+RLFKaPyqXO+dD
         assert_eq!(nrows().await, 1);
     }
 
-    // Convenient way to create a new key if you need one, run with
-    // `cargo test key::tests::gen_key`.
-    // #[test]
-    // fn gen_key() {
-    //     let name = "fiona";
-    //     let keypair = crate::pgp::create_keypair(
-    //         EmailAddress::new(&format!("{}@example.net", name)).unwrap(),
-    //     )
-    //     .unwrap();
-    //     std::fs::write(
-    //         format!("test-data/key/{}-public.asc", name),
-    //         keypair.public.to_base64(),
-    //     )
-    //     .unwrap();
-    //     std::fs::write(
-    //         format!("test-data/key/{}-secret.asc", name),
-    //         keypair.secret.to_base64(),
-    //     )
-    //     .unwrap();
-    // }
-
     #[test]
     fn test_fingerprint_from_str() {
         let res = Fingerprint::new(vec![

src/keyring.rs

@@ -79,8 +79,7 @@ mod tests {
     #[async_std::test]
     async fn test_keyring_load_self() {
         // new_self() implies load_self()
-        let t = TestContext::new().await;
-        t.configure_alice().await;
+        let t = TestContext::new_alice().await;
         let alice = alice_keypair();
 
         let pub_ring: Keyring<SignedPublicKey> = Keyring::new_self(&t).await.unwrap();

src/lib.rs

@@ -7,12 +7,14 @@
     clippy::all,
     clippy::indexing_slicing,
     clippy::wildcard_imports,
-    clippy::needless_borrow
+    clippy::needless_borrow,
+    clippy::cast_lossless
 )]
 #![allow(
     clippy::match_bool,
     clippy::eval_order_dependence,
-    clippy::bool_assert_comparison
+    clippy::bool_assert_comparison,
+    clippy::manual_split_once
 )]
 
 #[macro_use]
@@ -86,6 +88,7 @@ pub mod stock_str;
 mod sync;
 mod token;
 mod update_helper;
+pub mod webxdc;
 #[macro_use]
 mod dehtml;
 mod color;

src/location.rs

@@ -7,12 +7,13 @@ use quick_xml::events::{BytesEnd, BytesStart, BytesText};
 
 use crate::chat::{self, ChatId};
 use crate::config::Config;
-use crate::constants::{Viewtype, DC_CONTACT_ID_SELF};
+use crate::constants::DC_CONTACT_ID_SELF;
+use crate::contact::ContactId;
 use crate::context::Context;
 use crate::dc_tools::time;
 use crate::events::EventType;
 use crate::job::{self, Job};
-use crate::message::{Message, MsgId};
+use crate::message::{Message, MsgId, Viewtype};
 use crate::mimeparser::SystemMessage;
 use crate::param::Params;
 use crate::stock_str;
@@ -25,7 +26,7 @@ pub struct Location {
     pub longitude: f64,
     pub accuracy: f64,
     pub timestamp: i64,
-    pub contact_id: u32,
+    pub contact_id: ContactId,
     pub msg_id: u32,
     pub chat_id: ChatId,
     pub marker: Option<String>,
@@ -101,10 +102,10 @@ impl Kml {
             let val = event.unescape_and_decode(reader).unwrap_or_default();
 
             let val = val
-                .replace("\n", "")
-                .replace("\r", "")
-                .replace("\t", "")
-                .replace(" ", "");
+                .replace('\n', "")
+                .replace('\r', "")
+                .replace('\t', "")
+                .replace(' ', "");
 
             if self.tag.contains(KmlTag::WHEN) && val.len() >= 19 {
                 // YYYY-MM-DDTHH:MM:SSZ
@@ -223,7 +224,7 @@ pub async fn send_locations_to_chat(
             .unwrap_or_default();
     } else if 0 == seconds && is_sending_locations_before {
         let stock_str = stock_str::msg_location_disabled(context).await;
-        chat::add_info_msg(context, chat_id, stock_str, now).await?;
+        chat::add_info_msg(context, chat_id, &stock_str, now).await?;
     }
     context.emit_event(EventType::ChatModified(chat_id));
     if 0 != seconds {
@@ -558,7 +559,7 @@ pub async fn set_msg_location_id(context: &Context, msg_id: MsgId, location_id:
 pub(crate) async fn save(
     context: &Context,
     chat_id: ChatId,
-    contact_id: u32,
+    contact_id: ContactId,
     locations: &[Location],
     independent: bool,
 ) -> Result<Option<u32>> {
@@ -585,12 +586,12 @@ pub(crate) async fn save(
             conn.prepare_cached("SELECT id FROM locations WHERE timestamp=? AND from_id=?")?;
         let mut stmt_insert = conn.prepare_cached(stmt_insert)?;
 
-        let exists = stmt_test.exists(paramsv![timestamp, contact_id as i32])?;
+        let exists = stmt_test.exists(paramsv![timestamp, contact_id])?;
 
         if independent || !exists {
             stmt_insert.execute(paramsv![
                 timestamp,
-                contact_id as i32,
+                contact_id,
                 chat_id,
                 latitude,
                 longitude,
@@ -747,7 +748,7 @@ pub(crate) async fn job_maybe_send_locations_ended(
             );
 
             let stock_str = stock_str::msg_location_disabled(context).await;
-            job_try!(chat::add_info_msg(context, chat_id, stock_str, now).await);
+            job_try!(chat::add_info_msg(context, chat_id, &stock_str, now).await);
             context.emit_event(EventType::ChatModified(chat_id));
         }
     }

src/login_param.rs

@@ -253,7 +253,8 @@ impl LoginParam {
         sql.set_raw_config(key, Some(&self.imap.server)).await?;
 
         let key = format!("{}mail_port", prefix);
-        sql.set_raw_config_int(key, self.imap.port as i32).await?;
+        sql.set_raw_config_int(key, i32::from(self.imap.port))
+            .await?;
 
         let key = format!("{}mail_user", prefix);
         sql.set_raw_config(key, Some(&self.imap.user)).await?;
@@ -273,7 +274,8 @@ impl LoginParam {
         sql.set_raw_config(key, Some(&self.smtp.server)).await?;
 
         let key = format!("{}send_port", prefix);
-        sql.set_raw_config_int(key, self.smtp.port as i32).await?;
+        sql.set_raw_config_int(key, i32::from(self.smtp.port))
+            .await?;
 
         let key = format!("{}send_user", prefix);
         sql.set_raw_config(key, Some(&self.smtp.user)).await?;
@@ -413,7 +415,7 @@ mod tests {
         let t = TestContext::new().await;
 
         let param = LoginParam {
-            addr: "alice@example.com".to_string(),
+            addr: "alice@example.org".to_string(),
             imap: ServerLoginParam {
                 server: "imap.example.com".to_string(),
                 user: "alice".to_string(),
@@ -424,7 +426,7 @@ mod tests {
             },
             smtp: ServerLoginParam {
                 server: "smtp.example.com".to_string(),
-                user: "alice@example.com".to_string(),
+                user: "alice@example.org".to_string(),
                 password: "bar".to_string(),
                 port: 456,
                 security: Socket::Ssl,

src/mimefactory.rs

@@ -2,14 +2,14 @@
 
 use std::convert::TryInto;
 
-use anyhow::{bail, ensure, format_err, Result};
+use anyhow::{bail, ensure, Context as _, Result};
 use chrono::TimeZone;
 use lettre_email::{mime, Address, Header, MimeMultipartType, PartBuilder};
 
 use crate::blob::BlobObject;
-use crate::chat::{self, Chat};
+use crate::chat::Chat;
 use crate::config::Config;
-use crate::constants::{Chattype, Viewtype, DC_FROM_HANDSHAKE};
+use crate::constants::{Chattype, DC_FROM_HANDSHAKE};
 use crate::contact::Contact;
 use crate::context::{get_version_str, Context};
 use crate::dc_tools::IsNoneOrEmpty;
@@ -22,7 +22,7 @@ use crate::ephemeral::Timer as EphemeralTimer;
 use crate::format_flowed::{format_flowed, format_flowed_quote};
 use crate::html::new_html_mimepart;
 use crate::location;
-use crate::message::{self, Message, MsgId};
+use crate::message::{self, Message, MsgId, Viewtype};
 use crate::mimeparser::SystemMessage;
 use crate::param::Param;
 use crate::peerstate::{Peerstate, PeerstateVerifiedStatus};
@@ -81,7 +81,7 @@ pub struct MimeFactory<'a> {
 /// Result of rendering a message, ready to be submitted to a send job.
 #[derive(Debug, Clone)]
 pub struct RenderedEmail {
-    pub message: Vec<u8>,
+    pub message: String,
     // pub envelope: Envelope,
     pub is_encrypted: bool,
     pub is_gossiped: bool,
@@ -154,6 +154,12 @@ impl<'a> MimeFactory<'a> {
 
         if chat.is_self_talk() {
             recipients.push((from_displayname.to_string(), from_addr.to_string()));
+        } else if chat.is_mailing_list() {
+            let list_post = chat
+                .param
+                .get(Param::ListPost)
+                .context("Can't write to mailinglist without ListPost param")?;
+            recipients.push(("".to_string(), list_post.to_string()));
         } else {
             context
                 .sql
@@ -201,7 +207,6 @@ impl<'a> MimeFactory<'a> {
             )
             .await?;
 
-        let default_str = stock_str::status_line(context).await;
         let factory = MimeFactory {
             from_addr,
             from_displayname,
@@ -209,7 +214,7 @@ impl<'a> MimeFactory<'a> {
             selfstatus: context
                 .get_config(Config::Selfstatus)
                 .await?
-                .unwrap_or(default_str),
+                .unwrap_or_default(),
             recipients,
             timestamp: msg.timestamp_sort,
             loaded: Loaded::Message { chat },
@@ -240,11 +245,10 @@ impl<'a> MimeFactory<'a> {
             .get_config(Config::Displayname)
             .await?
             .unwrap_or_default();
-        let default_str = stock_str::status_line(context).await;
         let selfstatus = context
             .get_config(Config::Selfstatus)
             .await?
-            .unwrap_or(default_str);
+            .unwrap_or_default();
         let timestamp = dc_create_smeared_timestamp(context).await;
 
         let res = MimeFactory::<'a> {
@@ -277,7 +281,7 @@ impl<'a> MimeFactory<'a> {
         let self_addr = context
             .get_config(Config::ConfiguredAddr)
             .await?
-            .ok_or_else(|| format_err!("Not configured"))?;
+            .context("not configured")?;
 
         let mut res = Vec::new();
         for (_, addr) in self
@@ -458,49 +462,121 @@ impl<'a> MimeFactory<'a> {
             self.from_addr.clone(),
         );
 
+        let undisclosed_recipients = match &self.loaded {
+            Loaded::Message { chat } => chat.typ == Chattype::Broadcast,
+            Loaded::Mdn { .. } => false,
+        };
+
         let mut to = Vec::new();
-        for (name, addr) in self.recipients.iter() {
-            if name.is_empty() {
-                to.push(Address::new_mailbox(addr.clone()));
-            } else {
-                to.push(Address::new_mailbox_with_name(
-                    name.to_string(),
-                    addr.clone(),
-                ));
+        if undisclosed_recipients {
+            to.push(Address::new_group(
+                "hidden-recipients".to_string(),
+                Vec::new(),
+            ));
+        } else {
+            let email_to_remove =
+                if self.msg.param.get_cmd() == SystemMessage::MemberRemovedFromGroup {
+                    self.msg.param.get(Param::Arg)
+                } else {
+                    None
+                };
+
+            for (name, addr) in self.recipients.iter() {
+                if let Some(email_to_remove) = email_to_remove {
+                    if email_to_remove == addr {
+                        continue;
+                    }
+                }
+
+                if name.is_empty() {
+                    to.push(Address::new_mailbox(addr.clone()));
+                } else {
+                    to.push(Address::new_mailbox_with_name(
+                        name.to_string(),
+                        addr.clone(),
+                    ));
+                }
+            }
+
+            if to.is_empty() {
+                to.push(from.clone());
             }
         }
 
-        if to.is_empty() {
-            to.push(from.clone());
+        // Start with Internet Message Format headers in the order of the standard example
+        // <https://datatracker.ietf.org/doc/html/rfc5322#appendix-A.1.1>.
+        headers
+            .unprotected
+            .push(Header::new_with_value("From".into(), vec![from]).unwrap());
+        if let Some(sender_displayname) = &self.sender_displayname {
+            let sender =
+                Address::new_mailbox_with_name(sender_displayname.clone(), self.from_addr.clone());
+            headers
+                .unprotected
+                .push(Header::new_with_value("Sender".into(), vec![sender]).unwrap());
+        }
+        headers
+            .unprotected
+            .push(Header::new_with_value("To".into(), to).unwrap());
+
+        let subject_str = self.subject_str(context).await?;
+        let encoded_subject = if subject_str
+            .chars()
+            .all(|c| c.is_ascii_alphanumeric() || c == ' ')
+        // We do not use needs_encoding() here because needs_encoding() returns true if the string contains a space
+        // but we do not want to encode all subjects just because they contain a space.
+        {
+            subject_str.clone()
+        } else {
+            encode_words(&subject_str)
+        };
+        headers
+            .protected
+            .push(Header::new("Subject".into(), encoded_subject));
+
+        let date = chrono::Utc
+            .from_local_datetime(&chrono::NaiveDateTime::from_timestamp(self.timestamp, 0))
+            .unwrap()
+            .to_rfc2822();
+        headers.unprotected.push(Header::new("Date".into(), date));
+
+        let rfc724_mid = match self.loaded {
+            Loaded::Message { .. } => self.msg.rfc724_mid.clone(),
+            Loaded::Mdn { .. } => dc_create_outgoing_rfc724_mid(None, &self.from_addr),
+        };
+        let rfc724_mid_headervalue = render_rfc724_mid(&rfc724_mid);
+
+        // Amazon's SMTP servers change the `Message-ID`, just as Outlook's SMTP servers do.
+        // Outlook's servers add an `X-Microsoft-Original-Message-ID` header with the original `Message-ID`,
+        // and when downloading messages we look for this header in order to correctly identify
+        // messages.
+        // Amazon's servers do not add such a header, so we just add it ourselves.
+        if let Some(server) = context.get_config(Config::ConfiguredSendServer).await? {
+            if server.ends_with(".amazonaws.com") {
+                headers.unprotected.push(Header::new(
+                    "X-Microsoft-Original-Message-ID".into(),
+                    rfc724_mid_headervalue.clone(),
+                ))
+            }
         }
 
         headers
             .unprotected
-            .push(Header::new("MIME-Version".into(), "1.0".into()));
+            .push(Header::new("Message-ID".into(), rfc724_mid_headervalue));
 
+        // Reply headers as in <https://datatracker.ietf.org/doc/html/rfc5322#appendix-A.2>.
+        if !self.in_reply_to.is_empty() {
+            headers
+                .unprotected
+                .push(Header::new("In-Reply-To".into(), self.in_reply_to.clone()));
+        }
         if !self.references.is_empty() {
             headers
                 .unprotected
                 .push(Header::new("References".into(), self.references.clone()));
         }
 
-        if !self.in_reply_to.is_empty() {
-            headers
-                .unprotected
-                .push(Header::new("In-Reply-To".into(), self.in_reply_to.clone()));
-        }
-
-        let date = chrono::Utc
-            .from_local_datetime(&chrono::NaiveDateTime::from_timestamp(self.timestamp, 0))
-            .unwrap()
-            .to_rfc2822();
-
-        headers.unprotected.push(Header::new("Date".into(), date));
-
-        headers
-            .unprotected
-            .push(Header::new("Chat-Version".to_string(), "1.0".to_string()));
-
+        // Automatic Response headers <https://www.rfc-editor.org/rfc/rfc3834>
         if let Loaded::Mdn { .. } = self.loaded {
             headers.unprotected.push(Header::new(
                 "Auto-Submitted".to_string(),
@@ -513,6 +589,11 @@ impl<'a> MimeFactory<'a> {
             ));
         }
 
+        // Non-standard headers.
+        headers
+            .unprotected
+            .push(Header::new("Chat-Version".to_string(), "1.0".to_string()));
+
         if self.req_mdn {
             // we use "Chat-Disposition-Notification-To"
             // because replies to "Disposition-Notification-To" are weird in many cases
@@ -527,21 +608,9 @@ impl<'a> MimeFactory<'a> {
         let grpimage = self.grpimage();
         let force_plaintext = self.should_force_plaintext();
         let skip_autocrypt = self.should_skip_autocrypt();
-        let subject_str = self.subject_str(context).await?;
         let e2ee_guaranteed = self.is_e2ee_guaranteed();
         let encrypt_helper = EncryptHelper::new(context).await?;
 
-        let encoded_subject = if subject_str
-            .chars()
-            .all(|c| c.is_ascii_alphanumeric() || c == ' ')
-        // We do not use needs_encoding() here because needs_encoding() returns true if the string contains a space
-        // but we do not want to encode all subjects just because they contain a space.
-        {
-            subject_str.clone()
-        } else {
-            encode_words(&subject_str)
-        };
-
         if !skip_autocrypt {
             // unless determined otherwise we add the Autocrypt header
             let aheader = encrypt_helper.get_aheader().to_string();
@@ -550,15 +619,6 @@ impl<'a> MimeFactory<'a> {
                 .push(Header::new("Autocrypt".into(), aheader));
         }
 
-        headers
-            .protected
-            .push(Header::new("Subject".into(), encoded_subject));
-
-        let rfc724_mid = match self.loaded {
-            Loaded::Message { .. } => self.msg.rfc724_mid.clone(),
-            Loaded::Mdn { .. } => dc_create_outgoing_rfc724_mid(None, &self.from_addr),
-        };
-
         let ephemeral_timer = self.msg.chat_id.get_ephemeral_timer(context).await?;
         if let EphemeralTimer::Enabled { duration } = ephemeral_timer {
             headers.protected.push(Header::new(
@@ -567,36 +627,11 @@ impl<'a> MimeFactory<'a> {
             ));
         }
 
-        headers.unprotected.push(Header::new(
-            "Message-ID".into(),
-            render_rfc724_mid(&rfc724_mid),
-        ));
-
-        let undisclosed_recipients = match &self.loaded {
-            Loaded::Message { chat } => chat.typ == Chattype::Broadcast,
-            Loaded::Mdn { .. } => false,
-        };
-
-        if undisclosed_recipients {
-            headers
-                .unprotected
-                .push(Header::new("To".into(), "hidden-recipients: ;".to_string()));
-        } else {
-            headers
-                .unprotected
-                .push(Header::new_with_value("To".into(), to).unwrap());
-        }
-
+        // MIME header <https://datatracker.ietf.org/doc/html/rfc2045>.
+        // Content-Type
         headers
             .unprotected
-            .push(Header::new_with_value("From".into(), vec![from]).unwrap());
-        if let Some(sender_displayname) = &self.sender_displayname {
-            let sender =
-                Address::new_mailbox_with_name(sender_displayname.clone(), self.from_addr.clone());
-            headers
-                .unprotected
-                .push(Header::new_with_value("Sender".into(), vec![sender]).unwrap());
-        }
+            .push(Header::new("MIME-Version".into(), "1.0".into()));
 
         let mut is_gossiped = false;
 
@@ -623,6 +658,11 @@ impl<'a> MimeFactory<'a> {
                     "Content-Type".to_string(),
                     "multipart/report; report-type=multi-device-sync".to_string(),
                 ))
+            } else if self.msg.param.get_cmd() == SystemMessage::WebxdcStatusUpdate {
+                PartBuilder::new().header((
+                    "Content-Type".to_string(),
+                    "multipart/report; report-type=status-update".to_string(),
+                ))
             } else {
                 PartBuilder::new().message_type(MimeMultipartType::Mixed)
             };
@@ -747,7 +787,7 @@ impl<'a> MimeFactory<'a> {
         } = self;
 
         Ok(RenderedEmail {
-            message: outer_message.build().as_string().into_bytes(),
+            message: outer_message.build().as_string(),
             // envelope: Envelope::new,
             is_encrypted,
             is_gossiped,
@@ -897,7 +937,9 @@ impl<'a> MimeFactory<'a> {
                     "ephemeral-timer-changed".to_string(),
                 ));
             }
-            SystemMessage::LocationOnly | SystemMessage::MultiDeviceSync => {
+            SystemMessage::LocationOnly
+            | SystemMessage::MultiDeviceSync
+            | SystemMessage::WebxdcStatusUpdate => {
                 // This should prevent automatic replies,
                 // such as non-delivery reports.
                 //
@@ -1098,7 +1140,7 @@ impl<'a> MimeFactory<'a> {
         }
 
         // add attachment part
-        if chat::msgtype_has_file(self.msg.viewtype) {
+        if self.msg.viewtype.has_file() {
             if !is_file_size_okay(context, self.msg).await? {
                 bail!(
                     "Message exceeds the recommended {} MB.",
@@ -1134,6 +1176,16 @@ impl<'a> MimeFactory<'a> {
             let ids = self.msg.param.get(Param::Arg2).unwrap_or_default();
             parts.push(context.build_sync_part(json.to_string()).await);
             self.sync_ids_to_delete = Some(ids.to_string());
+        } else if command == SystemMessage::WebxdcStatusUpdate {
+            let json = self.msg.param.get(Param::Arg).unwrap_or_default();
+            parts.push(context.build_status_update_part(json).await);
+        } else if self.msg.viewtype == Viewtype::Webxdc {
+            if let Some(json) = context
+                .render_webxdc_status_update_object(self.msg.id, None)
+                .await?
+            {
+                parts.push(context.build_status_update_part(&json).await);
+            }
         }
 
         if self.attach_selfavatar {
@@ -1265,7 +1317,7 @@ async fn build_body_file(
         .param
         .get_blob(Param::File, context, true)
         .await?
-        .ok_or_else(|| format_err!("msg has no filename"))?;
+        .context("msg has no filename")?;
     let suffix = blob.suffix().unwrap_or("dat");
 
     // Get file name to use for sending.  For privacy purposes, we do
@@ -1293,8 +1345,7 @@ async fn build_body_file(
             "video_{}.{}",
             chrono::Utc
                 .timestamp(msg.timestamp_sort, 0)
-                .format("%Y-%m-%d_%H-%M-%S")
-                .to_string(),
+                .format("%Y-%m-%d_%H-%M-%S"),
             &suffix
         ),
         _ => blob.as_file_name().to_string(),
@@ -1401,18 +1452,22 @@ fn maybe_encode_words(words: &str) -> String {
 
 #[cfg(test)]
 mod tests {
-    use super::*;
+    use async_std::fs::File;
     use async_std::prelude::*;
+    use mailparse::{addrparse_header, MailHeaderMap};
 
     use crate::chat::ChatId;
+    use crate::chat::{
+        self, add_contact_to_chat, create_group_chat, remove_contact_from_chat, send_text_msg,
+        ProtectionStatus,
+    };
     use crate::chatlist::Chatlist;
     use crate::contact::Origin;
     use crate::dc_receive_imf::dc_receive_imf;
     use crate::mimeparser::MimeMessage;
     use crate::test_utils::{get_chat_msg, TestContext};
 
-    use async_std::fs::File;
-
+    use super::*;
     #[test]
     fn test_render_email_address() {
         let display_name = "ä space";
@@ -1509,7 +1564,7 @@ mod tests {
             msg_to_subject_str(
                 b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                 From: Bob <bob@example.com>\n\
-                To: alice@example.com\n\
+                To: alice@example.org\n\
                 Subject: Antw: Chat: hello\n\
                 Message-ID: <2222@example.com>\n\
                 Date: Sun, 22 Mar 2020 22:37:56 +0000\n\
@@ -1524,7 +1579,7 @@ mod tests {
             msg_to_subject_str(
                 b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                 From: Bob <bob@example.com>\n\
-                To: alice@example.com\n\
+                To: alice@example.org\n\
                 Subject: Infos: 42\n\
                 Message-ID: <2222@example.com>\n\
                 Date: Sun, 22 Mar 2020 22:37:56 +0000\n\
@@ -1543,7 +1598,7 @@ mod tests {
             msg_to_subject_str(
                 b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                 From: bob@example.com\n\
-                To: alice@example.com\n\
+                To: alice@example.org\n\
                 Subject: Chat: hello\n\
                 Chat-Version: 1.0\n\
                 Message-ID: <2223@example.com>\n\
@@ -1561,7 +1616,7 @@ mod tests {
         // 3. Send the first message to a new contact
         let t = TestContext::new_alice().await;
 
-        assert_eq!(first_subject_str(t).await, "Message from alice@example.com");
+        assert_eq!(first_subject_str(t).await, "Message from alice@example.org");
 
         let t = TestContext::new_alice().await;
         t.set_config(Config::Displayname, Some("Alice"))
@@ -1576,7 +1631,7 @@ mod tests {
         msg_to_subject_str(
             "Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
             From: bob@example.com\n\
-            To: alice@example.com\n\
+            To: alice@example.org\n\
             Subject: äääää\n\
             Chat-Version: 1.0\n\
             Message-ID: <2893@example.com>\n\
@@ -1590,7 +1645,7 @@ mod tests {
         msg_to_subject_str(
             "Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
             From: bob@example.com\n\
-            To: alice@example.com\n\
+            To: alice@example.org\n\
             Subject: aäääää\n\
             Chat-Version: 1.0\n\
             Message-ID: <2893@example.com>\n\
@@ -1609,7 +1664,7 @@ mod tests {
         dc_receive_imf(
             &t,
             b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
-            From: alice@example.com\n\
+            From: alice@example.org\n\
             To: bob@example.com\n\
             Subject: Hello, Bob\n\
             Chat-Version: 1.0\n\
@@ -1618,7 +1673,6 @@ mod tests {
             \n\
             hello\n",
             "INBOX",
-            1,
             false,
         )
         .await
@@ -1626,7 +1680,7 @@ mod tests {
         let new_msg = incoming_msg_to_reply_msg(
             b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                  From: bob@example.com\n\
-                 To: alice@example.com\n\
+                 To: alice@example.org\n\
                  Subject: message opened\n\
                  Date: Sun, 22 Mar 2020 23:37:57 +0000\n\
                  Chat-Version: 1.0\n\
@@ -1664,7 +1718,7 @@ mod tests {
             let mut new_msg = Message::new(Viewtype::Text);
             new_msg.set_text(Some("Hi".to_string()));
             if let Some(q) = quote {
-                new_msg.set_quote(t, q).await?;
+                new_msg.set_quote(t, Some(q)).await?;
             }
             let sent = t.send_msg(group_id, &mut new_msg).await;
             get_subject(t, sent).await
@@ -1701,7 +1755,7 @@ mod tests {
             format!(
                 "Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                 From: bob@example.com\n\
-                To: alice@example.com\n\
+                To: alice@example.org\n\
                 Subject: Different subject\n\
                 In-Reply-To: {}\n\
                 Message-ID: <2893@example.com>\n\
@@ -1712,7 +1766,6 @@ mod tests {
             )
             .as_bytes(),
             "INBOX",
-            5,
             false,
         )
         .await?;
@@ -1756,7 +1809,7 @@ mod tests {
         mf.subject_str(&t).await.unwrap()
     }
 
-    // In `imf_raw`, From has to be bob@example.com, To has to be alice@example.com
+    // In `imf_raw`, From has to be bob@example.com, To has to be alice@example.org
     async fn msg_to_subject_str(imf_raw: &[u8]) -> String {
         let subject_str = msg_to_subject_str_inner(imf_raw, false, false, false).await;
 
@@ -1817,14 +1870,13 @@ mod tests {
                 &t,
                 b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                     From: Bob <bob@example.com>\n\
-                    To: alice@example.com\n\
+                    To: alice@example.org\n\
                     Subject: Some other, completely unrelated subject\n\
                     Message-ID: <3cl4@example.com>\n\
                     Date: Sun, 22 Mar 2020 22:37:56 +0000\n\
                     \n\
                     Some other, completely unrelated content\n",
                 "INBOX",
-                2,
                 false,
             )
             .await
@@ -1835,7 +1887,7 @@ mod tests {
         }
 
         if reply {
-            new_msg.set_quote(&t, &incoming_msg).await.unwrap();
+            new_msg.set_quote(&t, Some(&incoming_msg)).await.unwrap();
         }
 
         let mf = MimeFactory::from_msg(&t, &new_msg, false).await.unwrap();
@@ -1849,13 +1901,13 @@ mod tests {
             .await
             .unwrap();
 
-        dc_receive_imf(context, imf_raw, "INBOX", 1, false)
+        dc_receive_imf(context, imf_raw, "INBOX", false)
             .await
             .unwrap();
 
         let chats = Chatlist::try_load(context, 0, None, None).await.unwrap();
 
-        let chat_id = chats.get_chat_id(0);
+        let chat_id = chats.get_chat_id(0).unwrap();
         chat_id.accept(context).await.unwrap();
 
         let mut new_msg = Message::new(Viewtype::Text);
@@ -1877,7 +1929,7 @@ mod tests {
         let msg = incoming_msg_to_reply_msg(
             b"Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
                 From: Charlie <charlie@example.com>\n\
-                To: alice@example.com\n\
+                To: alice@example.org\n\
                 Subject: Chat: hello\n\
                 Chat-Version: 1.0\n\
                 Message-ID: <2223@example.com>\n\
@@ -1895,7 +1947,7 @@ mod tests {
 
         let rendered_msg = mimefactory.render(context).await.unwrap();
 
-        let mail = mailparse::parse_mail(&rendered_msg.message).unwrap();
+        let mail = mailparse::parse_mail(rendered_msg.message.as_bytes()).unwrap();
         assert_eq!(
             mail.headers
                 .iter()
@@ -1905,7 +1957,7 @@ mod tests {
             "1.0"
         );
 
-        let _mime_msg = MimeMessage::from_bytes(context, &rendered_msg.message)
+        let _mime_msg = MimeMessage::from_bytes(context, rendered_msg.message.as_bytes())
             .await
             .unwrap();
     }
@@ -1979,8 +2031,9 @@ mod tests {
         let mut msg = Message::new(Viewtype::Text);
         msg.set_text(Some("this is the text!".to_string()));
 
-        let payload = t.send_msg(chat.id, &mut msg).await.payload();
-        let mut payload = payload.splitn(3, "\r\n\r\n");
+        let sent_msg = t.send_msg(chat.id, &mut msg).await;
+        let mut payload = sent_msg.payload().splitn(3, "\r\n\r\n");
+
         let outer = payload.next().unwrap();
         let inner = payload.next().unwrap();
         let body = payload.next().unwrap();
@@ -1998,8 +2051,8 @@ mod tests {
 
         // if another message is sent, that one must not contain the avatar
         // and no artificial multipart/mixed nesting
-        let payload = t.send_msg(chat.id, &mut msg).await.payload();
-        let mut payload = payload.splitn(2, "\r\n\r\n");
+        let sent_msg = t.send_msg(chat.id, &mut msg).await;
+        let mut payload = sent_msg.payload().splitn(2, "\r\n\r\n");
         let outer = payload.next().unwrap();
         let body = payload.next().unwrap();
 
@@ -2016,4 +2069,58 @@ mod tests {
 
         Ok(())
     }
+
+    /// Test that removed member address does not go into the `To:` field.
+    #[async_std::test]
+    async fn test_remove_member_bcc() -> Result<()> {
+        // Alice creates a group with Bob and Claire and then removes Bob.
+        let alice = TestContext::new_alice().await;
+
+        let bob_id = Contact::create(&alice, "Bob", "bob@example.net").await?;
+        let claire_id = Contact::create(&alice, "Claire", "claire@foo.de").await?;
+
+        let alice_chat_id = create_group_chat(&alice, ProtectionStatus::Unprotected, "foo").await?;
+        add_contact_to_chat(&alice, alice_chat_id, bob_id).await?;
+        add_contact_to_chat(&alice, alice_chat_id, claire_id).await?;
+        send_text_msg(&alice, alice_chat_id, "Creating a group".to_string()).await?;
+
+        remove_contact_from_chat(&alice, alice_chat_id, claire_id).await?;
+        let remove = alice.pop_sent_msg().await;
+        let remove_payload = remove.payload();
+        let parsed = mailparse::parse_mail(remove_payload.as_bytes())?;
+        let to = parsed
+            .headers
+            .get_first_header("To")
+            .context("no To: header parsed")?;
+        let to = addrparse_header(to)?;
+        let mailbox = to
+            .extract_single_info()
+            .context("to: field does not contain exactly one address")?;
+        assert_eq!(mailbox.addr, "bob@example.net");
+
+        Ok(())
+    }
+
+    /// Tests that standard IMF header "From:" comes before non-standard "Autocrypt:" header.
+    #[async_std::test]
+    async fn test_from_before_autocrypt() -> Result<()> {
+        // create chat with bob
+        let t = TestContext::new_alice().await;
+        let chat = t.create_chat_with_contact("bob", "bob@example.org").await;
+
+        // send message to bob: that should get multipart/mixed because of the avatar moved to inner header;
+        // make sure, `Subject:` stays in the outer header (imf header)
+        let mut msg = Message::new(Viewtype::Text);
+        msg.set_text(Some("this is the text!".to_string()));
+
+        let sent_msg = t.send_msg(chat.id, &mut msg).await;
+        let payload = sent_msg.payload();
+
+        assert_eq!(payload.match_indices("Autocrypt:").count(), 1);
+        assert_eq!(payload.match_indices("From:").count(), 1);
+
+        assert!(payload.match_indices("From:").next() < payload.match_indices("Autocrypt:").next());
+
+        Ok(())
+    }
 }

src/mimeparser.rs

@@ -2,6 +2,7 @@
 
 use std::collections::{HashMap, HashSet};
 use std::future::Future;
+use std::io::Cursor;
 use std::pin::Pin;
 
 use anyhow::{bail, Result};
@@ -12,10 +13,10 @@ use once_cell::sync::Lazy;
 
 use crate::aheader::Aheader;
 use crate::blob::BlobObject;
-use crate::constants::{Viewtype, DC_DESIRED_TEXT_LEN, DC_ELLIPSIS};
-use crate::contact::addr_normalize;
+use crate::constants::{DC_DESIRED_TEXT_LEN, DC_ELLIPSIS};
+use crate::contact::{addr_normalize, ContactId};
 use crate::context::Context;
-use crate::dc_tools::{dc_get_filemeta, dc_truncate};
+use crate::dc_tools::{dc_get_filemeta, dc_truncate, parse_receive_headers};
 use crate::dehtml::dehtml;
 use crate::e2ee;
 use crate::events::EventType;
@@ -23,7 +24,7 @@ use crate::format_flowed::unformat_flowed;
 use crate::headerdef::{HeaderDef, HeaderDefMap};
 use crate::key::Fingerprint;
 use crate::location;
-use crate::message;
+use crate::message::{self, Viewtype};
 use crate::param::{Param, Params};
 use crate::peerstate::Peerstate;
 use crate::simplify::simplify;
@@ -47,6 +48,7 @@ pub struct MimeMessage {
     /// Addresses are normalized and lowercased:
     pub recipients: Vec<SingleInfo>,
     pub from: Vec<SingleInfo>,
+    pub list_post: Option<String>,
     pub chat_disposition_notification_to: Option<SingleInfo>,
     pub decrypting_failed: bool,
 
@@ -65,6 +67,7 @@ pub struct MimeMessage {
     pub location_kml: Option<location::Kml>,
     pub message_kml: Option<location::Kml>,
     pub(crate) sync_items: Option<SyncItems>,
+    pub(crate) webxdc_status_update: Option<String>,
     pub(crate) user_avatar: Option<AvatarAction>,
     pub(crate) group_avatar: Option<AvatarAction>,
     pub(crate) mdn_reports: Vec<Report>,
@@ -82,6 +85,8 @@ pub struct MimeMessage {
     /// This is non-empty only if the message was actually encrypted.  It is used
     /// for e.g. late-parsing HTML.
     pub decoded_data: Vec<u8>,
+
+    pub(crate) hop_info: String,
 }
 
 #[derive(Debug, PartialEq)]
@@ -132,6 +137,10 @@ pub enum SystemMessage {
     /// Self-sent-message that contains only json used for multi-device-sync;
     /// if possible, we attach that to other messages as for locations.
     MultiDeviceSync = 20,
+
+    // Sync message that contains a json payload
+    // sent to the other webxdc instances
+    WebxdcStatusUpdate = 30,
 }
 
 impl Default for SystemMessage {
@@ -163,10 +172,12 @@ impl MimeMessage {
             .get_header_value(HeaderDef::Date)
             .and_then(|v| mailparse::dateparse(&v).ok())
             .unwrap_or_default();
+        let hop_info = parse_receive_headers(&mail.get_headers());
 
         let mut headers = Default::default();
         let mut recipients = Default::default();
         let mut from = Default::default();
+        let mut list_post = Default::default();
         let mut chat_disposition_notification_to = None;
 
         // Parse IMF headers.
@@ -175,6 +186,7 @@ impl MimeMessage {
             &mut headers,
             &mut recipients,
             &mut from,
+            &mut list_post,
             &mut chat_disposition_notification_to,
             &mail.headers,
         );
@@ -248,6 +260,7 @@ impl MimeMessage {
                             &mut headers,
                             &mut recipients,
                             &mut throwaway_from,
+                            &mut list_post,
                             &mut chat_disposition_notification_to,
                             &decrypted_mail.headers,
                         );
@@ -275,6 +288,7 @@ impl MimeMessage {
             parts: Vec::new(),
             header: headers,
             recipients,
+            list_post,
             from,
             chat_disposition_notification_to,
             decrypting_failed: false,
@@ -288,12 +302,14 @@ impl MimeMessage {
             location_kml: None,
             message_kml: None,
             sync_items: None,
+            webxdc_status_update: None,
             user_avatar: None,
             group_avatar: None,
             failure_report: None,
             footer: None,
             is_mime_modified: false,
             decoded_data: Vec::new(),
+            hop_info,
         };
 
         match partial {
@@ -353,6 +369,16 @@ impl MimeMessage {
             } else if value == "protection-disabled" {
                 self.is_system_message = SystemMessage::ChatProtectionDisabled;
             }
+        } else if self.get_header(HeaderDef::ChatGroupMemberRemoved).is_some() {
+            self.is_system_message = SystemMessage::MemberRemovedFromGroup;
+        } else if self.get_header(HeaderDef::ChatGroupMemberAdded).is_some() {
+            self.is_system_message = SystemMessage::MemberAddedToGroup;
+        } else if self.get_header(HeaderDef::ChatGroupNameChanged).is_some() {
+            self.is_system_message = SystemMessage::GroupNameChanged;
+        } else if let Some(value) = self.get_header(HeaderDef::ChatContent) {
+            if value == "group-avatar-changed" {
+                self.is_system_message = SystemMessage::GroupImageChanged;
+            }
         }
     }
 
@@ -388,16 +414,18 @@ impl MimeMessage {
     #[allow(clippy::indexing_slicing)]
     fn squash_attachment_parts(&mut self) {
         if let [textpart, filepart] = &self.parts[..] {
-            let need_drop = {
-                textpart.typ == Viewtype::Text
-                    && (filepart.typ == Viewtype::Image
-                        || filepart.typ == Viewtype::Gif
-                        || filepart.typ == Viewtype::Sticker
-                        || filepart.typ == Viewtype::Audio
-                        || filepart.typ == Viewtype::Voice
-                        || filepart.typ == Viewtype::Video
-                        || filepart.typ == Viewtype::File)
-            };
+            let need_drop = textpart.typ == Viewtype::Text
+                && match filepart.typ {
+                    Viewtype::Image
+                    | Viewtype::Gif
+                    | Viewtype::Sticker
+                    | Viewtype::Audio
+                    | Viewtype::Voice
+                    | Viewtype::Video
+                    | Viewtype::File
+                    | Viewtype::Webxdc => true,
+                    Viewtype::Unknown | Viewtype::Text | Viewtype::VideochatInvitation => false,
+                };
 
             if need_drop {
                 let mut filepart = self.parts.swap_remove(1);
@@ -529,7 +557,7 @@ impl MimeMessage {
             };
 
             if let Some(ref subject) = self.get_subject() {
-                if !self.has_chat_version() {
+                if !self.has_chat_version() && self.webxdc_status_update.is_none() {
                     part.msg = subject.to_string();
                 }
             }
@@ -828,6 +856,12 @@ impl MimeMessage {
                                     .await?;
                             }
                         }
+                        Some("status-update") => {
+                            if let Some(second) = mail.subparts.get(1) {
+                                self.add_single_part_if_known(context, second, is_related)
+                                    .await?;
+                            }
+                        }
                         Some(_) => {
                             if let Some(first) = mail.subparts.get(0) {
                                 any_part_added = self
@@ -997,8 +1031,14 @@ impl MimeMessage {
         if decoded_data.is_empty() {
             return;
         }
-        // treat location/message kml file attachments specially
-        if filename.ends_with(".kml") {
+        let reader = Cursor::new(decoded_data);
+        let msg_type = if context
+            .is_webxdc_file(filename, reader)
+            .await
+            .unwrap_or(false)
+        {
+            Viewtype::Webxdc
+        } else if filename.ends_with(".kml") {
             // XXX what if somebody sends eg an "location-highlights.kml"
             // attachment unrelated to location streaming?
             if filename.starts_with("location") || filename.starts_with("message") {
@@ -1014,6 +1054,7 @@ impl MimeMessage {
                 }
                 return;
             }
+            msg_type
         } else if filename == "multi-device-sync.json" {
             let serialized = String::from_utf8_lossy(decoded_data)
                 .parse()
@@ -1026,7 +1067,15 @@ impl MimeMessage {
                 })
                 .ok();
             return;
-        }
+        } else if filename == "status-update.json" {
+            let serialized = String::from_utf8_lossy(decoded_data)
+                .parse()
+                .unwrap_or_default();
+            self.webxdc_status_update = Some(serialized);
+            return;
+        } else {
+            msg_type
+        };
 
         /* we have a regular file attachment,
         write decoded data to new blob object */
@@ -1092,16 +1141,16 @@ impl MimeMessage {
         }
     }
 
-    pub fn repl_msg_by_error(&mut self, error_msg: impl AsRef<str>) {
+    pub fn repl_msg_by_error(&mut self, error_msg: &str) {
         self.is_system_message = SystemMessage::Unknown;
         if let Some(part) = self.parts.first_mut() {
             part.typ = Viewtype::Text;
-            part.msg = format!("[{}]", error_msg.as_ref());
+            part.msg = format!("[{}]", error_msg);
             self.parts.truncate(1);
         }
     }
 
-    pub fn get_rfc724_mid(&self) -> Option<String> {
+    pub(crate) fn get_rfc724_mid(&self) -> Option<String> {
         self.get_header(HeaderDef::XMicrosoftOriginalMessageId)
             .or_else(|| self.get_header(HeaderDef::MessageId))
             .and_then(|msgid| parse_message_id(msgid).ok())
@@ -1112,6 +1161,7 @@ impl MimeMessage {
         headers: &mut HashMap<String, String>,
         recipients: &mut Vec<SingleInfo>,
         from: &mut Vec<SingleInfo>,
+        list_post: &mut Option<String>,
         chat_disposition_notification_to: &mut Option<SingleInfo>,
         fields: &[mailparse::MailHeader<'_>],
     ) {
@@ -1142,6 +1192,10 @@ impl MimeMessage {
         if !from_new.is_empty() {
             *from = from_new;
         }
+        let list_post_new = get_list_post(fields);
+        if list_post_new.is_some() {
+            *list_post = list_post_new;
+        }
     }
 
     fn process_report(
@@ -1159,23 +1213,24 @@ impl MimeMessage {
 
         // must be present
         if let Some(_disposition) = report_fields.get_header_value(HeaderDef::Disposition) {
-            if let Some(original_message_id) = report_fields
+            let original_message_id = report_fields
                 .get_header_value(HeaderDef::OriginalMessageId)
-                .and_then(|v| parse_message_id(&v).ok())
-            {
-                let additional_message_ids = report_fields
-                    .get_header_value(HeaderDef::AdditionalMessageIds)
-                    .map_or_else(Vec::new, |v| {
-                        v.split(' ')
-                            .filter_map(|s| parse_message_id(s).ok())
-                            .collect()
-                    });
+                // MS Exchange doesn't add an Original-Message-Id header. Instead, they put
+                // the original message id into the In-Reply-To header:
+                .or_else(|| report.headers.get_header_value(HeaderDef::InReplyTo))
+                .and_then(|v| parse_message_id(&v).ok());
+            let additional_message_ids = report_fields
+                .get_header_value(HeaderDef::AdditionalMessageIds)
+                .map_or_else(Vec::new, |v| {
+                    v.split(' ')
+                        .filter_map(|s| parse_message_id(s).ok())
+                        .collect()
+                });
 
-                return Ok(Some(Report {
-                    original_message_id,
-                    additional_message_ids,
-                }));
-            }
+            return Ok(Some(Report {
+                original_message_id,
+                additional_message_ids,
+            }));
         }
         warn!(
             context,
@@ -1326,13 +1381,15 @@ impl MimeMessage {
     pub async fn handle_reports(
         &self,
         context: &Context,
-        from_id: u32,
+        from_id: ContactId,
         sent_timestamp: i64,
         parts: &[Part],
     ) {
         for report in &self.mdn_reports {
-            for original_message_id in
-                std::iter::once(&report.original_message_id).chain(&report.additional_message_ids)
+            for original_message_id in report
+                .original_message_id
+                .iter()
+                .chain(&report.additional_message_ids)
             {
                 match message::handle_mdn(context, from_id, original_message_id, sent_timestamp)
                     .await
@@ -1437,7 +1494,10 @@ async fn update_gossip_peerstates(
 #[derive(Debug)]
 pub(crate) struct Report {
     /// Original-Message-ID header
-    original_message_id: String,
+    ///
+    /// It MUST be present if the original message has a Message-ID according to RFC 8098.
+    /// In case we can't find it (shouldn't happen), this is None.
+    original_message_id: Option<String>,
     /// Additional-Message-IDs
     additional_message_ids: Vec<String>,
 }
@@ -1627,6 +1687,14 @@ pub(crate) fn get_from(headers: &[MailHeader]) -> Vec<SingleInfo> {
     get_all_addresses_from_header(headers, |header_key| header_key == "from")
 }
 
+/// Returned addresses are normalized and lowercased.
+pub(crate) fn get_list_post(headers: &[MailHeader]) -> Option<String> {
+    get_all_addresses_from_header(headers, |header_key| header_key == "list-post")
+        .into_iter()
+        .next()
+        .map(|s| s.addr)
+}
+
 fn get_all_addresses_from_header<F>(headers: &[MailHeader], pred: F) -> Vec<SingleInfo>
 where
     F: Fn(String) -> bool,
@@ -2346,7 +2414,7 @@ Additional-Message-IDs: <foo@example.com> <foo@example.net>\n\
         assert_eq!(message.mdn_reports.len(), 1);
         assert_eq!(
             message.mdn_reports[0].original_message_id,
-            "foo@example.org"
+            Some("foo@example.org".to_string())
         );
         assert_eq!(
             &message.mdn_reports[0].additional_message_ids,
@@ -2848,7 +2916,6 @@ On 2020-10-25, Bob wrote:
             &t.ctx,
             include_bytes!("../test-data/message/subj_with_multimedia_msg.eml"),
             "INBOX",
-            1,
             false,
         )
         .await
@@ -2997,7 +3064,7 @@ Subject: ...
 
 Some quote.
 "###;
-        dc_receive_imf(&t, raw, "INBOX", 1, false).await?;
+        dc_receive_imf(&t, raw, "INBOX", false).await?;
 
         // Delta Chat generates In-Reply-To with a starting tab when Message-ID is too long.
         let raw = br###"In-Reply-To:
@@ -3014,7 +3081,7 @@ Subject: ...
 Some reply
 "###;
 
-        dc_receive_imf(&t, raw, "INBOX", 2, false).await?;
+        dc_receive_imf(&t, raw, "INBOX", false).await?;
 
         let msg = t.get_last_msg().await;
         assert_eq!(msg.get_text().unwrap(), "Some reply");
@@ -3034,21 +3101,21 @@ Some reply
 Chat-Version: 1.0\n\
 Message-ID: <foobarbaz@example.org>
 To: Bob <bob@example.org>
-From: Alice <alice@example.com>
+From: Alice <alice@example.org>
 Subject: subject
-Chat-Disposition-Notification-To: alice@example.com
+Chat-Disposition-Notification-To: alice@example.org
 
 Message.
 "###;
 
         // Bob receives message.
-        dc_receive_imf(&bob, raw, "INBOX", 1, false).await?;
+        dc_receive_imf(&bob, raw, "INBOX", false).await?;
         let msg = bob.get_last_msg().await;
         // Message is incoming.
         assert!(msg.param.get_bool(Param::WantsMdn).unwrap());
 
         // Alice receives copy-to-self.
-        dc_receive_imf(&alice, raw, "INBOX", 1, false).await?;
+        dc_receive_imf(&alice, raw, "INBOX", false).await?;
         let msg = alice.get_last_msg().await;
         // Message is outgoing, don't send read receipt to self.
         assert!(msg.param.get_bool(Param::WantsMdn).is_none());
@@ -3064,18 +3131,17 @@ Message.
         dc_receive_imf(
             &alice,
             "Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
-                 From: alice@example.com\n\
+                 From: alice@example.org\n\
                  To: bob@example.net\n\
                  Subject: foo\n\
                  Message-ID: first@example.com\n\
                  Chat-Version: 1.0\n\
-                 Chat-Disposition-Notification-To: alice@example.com\n\
+                 Chat-Disposition-Notification-To: alice@example.org\n\
                  Date: Sun, 22 Mar 2020 22:37:57 +0000\n\
                  \n\
                  hello\n"
                 .as_bytes(),
             "INBOX",
-            1,
             false,
         )
         .await?;
@@ -3087,8 +3153,8 @@ Message.
         dc_receive_imf(
             &alice,
                 "Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n\
-                 From: alice@example.com\n\
-                 To: alice@example.com\n\
+                 From: alice@example.org\n\
+                 To: alice@example.org\n\
                  Subject: message opened\n\
                  Date: Sun, 22 Mar 2020 23:37:57 +0000\n\
                  Chat-Version: 1.0\n\
@@ -3114,7 +3180,6 @@ Message.
                  --SNIPP--"
             .as_bytes(),
             "INBOX",
-            2,
             false,
         )
         .await?;
@@ -3125,4 +3190,40 @@ Message.
 
         Ok(())
     }
+
+    /// Test parsing of MDN sent by MS Exchange.
+    ///
+    /// It does not have required Original-Message-ID field, so it is useless, but we want to
+    /// recognize it as MDN nevertheless to avoid displaying it in the chat as normal message.
+    #[async_std::test]
+    async fn test_ms_exchange_mdn() -> Result<()> {
+        let t = TestContext::new_alice().await;
+        t.set_config(Config::ShowEmails, Some("2")).await?;
+
+        let original =
+            include_bytes!("../test-data/message/ms_exchange_report_original_message.eml");
+        dc_receive_imf(&t, original, "INBOX", false).await?;
+        let original_msg_id = t.get_last_msg().await.id;
+
+        // 1. Test mimeparser directly
+        let mdn =
+            include_bytes!("../test-data/message/ms_exchange_report_disposition_notification.eml");
+        let mimeparser = MimeMessage::from_bytes(&t.ctx, mdn).await?;
+        assert_eq!(mimeparser.mdn_reports.len(), 1);
+        assert_eq!(
+            mimeparser.mdn_reports[0].original_message_id.as_deref(),
+            Some("d5904dc344eeb5deaf9bb44603f0c716@posteo.de")
+        );
+        assert!(mimeparser.mdn_reports[0].additional_message_ids.is_empty());
+
+        // 2. Test that marking the original msg as read works
+        dc_receive_imf(&t, mdn, "INBOX", false).await?;
+
+        assert_eq!(
+            original_msg_id.get_state(&t).await?,
+            MessageState::OutMdnRcvd
+        );
+
+        Ok(())
+    }
 }

src/oauth2.rs

@@ -41,6 +41,7 @@ struct Oauth2 {
 
 /// OAuth 2 Access Token Response
 #[derive(Debug, Deserialize)]
+#[allow(dead_code)]
 struct Response {
     // Should always be there according to: <https://www.oauth.com/oauth2-servers/access-tokens/access-token-response/>
     // but previous code handled its abscense.
@@ -58,7 +59,7 @@ pub async fn dc_get_oauth2_url(
     redirect_uri: &str,
 ) -> Result<Option<String>> {
     let socks5_enabled = context.get_config_bool(Config::Socks5Enabled).await?;
-    if let Some(oauth2) = Oauth2::from_address(addr, socks5_enabled).await {
+    if let Some(oauth2) = Oauth2::from_address(context, addr, socks5_enabled).await {
         context
             .sql
             .set_raw_config("oauth2_pending_redirect_uri", Some(redirect_uri))
@@ -79,7 +80,7 @@ pub async fn dc_get_oauth2_access_token(
     regenerate: bool,
 ) -> Result<Option<String>> {
     let socks5_enabled = context.get_config_bool(Config::Socks5Enabled).await?;
-    if let Some(oauth2) = Oauth2::from_address(addr, socks5_enabled).await {
+    if let Some(oauth2) = Oauth2::from_address(context, addr, socks5_enabled).await {
         let lock = context.oauth2_mutex.lock().await;
 
         // read generated token
@@ -225,7 +226,7 @@ pub async fn dc_get_oauth2_addr(
     code: &str,
 ) -> Result<Option<String>> {
     let socks5_enabled = context.get_config_bool(Config::Socks5Enabled).await?;
-    let oauth2 = match Oauth2::from_address(addr, socks5_enabled).await {
+    let oauth2 = match Oauth2::from_address(context, addr, socks5_enabled).await {
         Some(o) => o,
         None => return Ok(None),
     };
@@ -253,13 +254,13 @@ pub async fn dc_get_oauth2_addr(
 }
 
 impl Oauth2 {
-    async fn from_address(addr: &str, skip_mx: bool) -> Option<Self> {
+    async fn from_address(context: &Context, addr: &str, skip_mx: bool) -> Option<Self> {
         let addr_normalized = normalize_addr(addr);
         if let Some(domain) = addr_normalized
             .find('@')
             .map(|index| addr_normalized.split_at(index + 1).1)
         {
-            if let Some(oauth2_authorizer) = provider::get_provider_info(domain, skip_mx)
+            if let Some(oauth2_authorizer) = provider::get_provider_info(context, domain, skip_mx)
                 .await
                 .and_then(|provider| provider.oauth2_authorizer.as_ref())
             {
@@ -356,32 +357,39 @@ mod tests {
 
     #[async_std::test]
     async fn test_oauth_from_address() {
+        let t = TestContext::new().await;
         assert_eq!(
-            Oauth2::from_address("hello@gmail.com", false).await,
+            Oauth2::from_address(&t, "hello@gmail.com", false).await,
             Some(OAUTH2_GMAIL)
         );
         assert_eq!(
-            Oauth2::from_address("hello@googlemail.com", false).await,
+            Oauth2::from_address(&t, "hello@googlemail.com", false).await,
             Some(OAUTH2_GMAIL)
         );
         assert_eq!(
-            Oauth2::from_address("hello@yandex.com", false).await,
+            Oauth2::from_address(&t, "hello@yandex.com", false).await,
             Some(OAUTH2_YANDEX)
         );
         assert_eq!(
-            Oauth2::from_address("hello@yandex.ru", false).await,
+            Oauth2::from_address(&t, "hello@yandex.ru", false).await,
             Some(OAUTH2_YANDEX)
         );
-
-        assert_eq!(Oauth2::from_address("hello@web.de", false).await, None);
+        assert_eq!(Oauth2::from_address(&t, "hello@web.de", false).await, None);
     }
 
     #[async_std::test]
     async fn test_oauth_from_mx() {
+        // youtube staff seems to use "google workspace with oauth2", figures this out by MX lookup
+        let t = TestContext::new().await;
         assert_eq!(
-            Oauth2::from_address("hello@google.com", false).await,
+            Oauth2::from_address(&t, "hello@youtube.com", false).await,
             Some(OAUTH2_GMAIL)
         );
+        // without MX lookup, we would not know as youtube.com is not in our provider-db
+        assert_eq!(
+            Oauth2::from_address(&t, "hello@youtube.com", true).await,
+            None
+        );
     }
 
     #[async_std::test]

src/param.rs

@@ -110,8 +110,8 @@ pub enum Param {
     /// For Jobs
     AlsoMove = b'M',
 
-    /// For Jobs: space-separated list of message recipients
-    Recipients = b'R',
+    /// For MDN-sending job
+    MsgId = b'I',
 
     /// For Groups
     ///
@@ -136,8 +136,17 @@ pub enum Param {
     /// For Chats
     Devicetalk = b'D',
 
-    /// For MDN-sending job
-    MsgId = b'I',
+    /// For Chats: If this is a mailing list chat, contains the List-Post address.
+    /// None if there simply is no `List-Post` header in the mailing list.
+    /// Some("") if the mailing list is using multiple different List-Post headers.
+    ///
+    /// The List-Post address is the email address where the user can write to in order to
+    /// post something to the mailing list.
+    ListPost = b'p',
+
+    /// For Contacts: If this is the List-Post address of a mailing list, contains
+    /// the List-Id of the mailing list (which is also used as the group id of the chat).
+    ListId = b's',
 
     /// For Contacts: timestamp of status (aka signature or footer) update.
     StatusTimestamp = b'j',
@@ -159,6 +168,12 @@ pub enum Param {
 
     /// For Chats: timestamp of protection settings update.
     ProtectionSettingsTimestamp = b'L',
+
+    /// For Webxdc Message Instances: Current summary
+    WebxdcSummary = b'N',
+
+    /// For Webxdc Message Instances: timestamp of summary update.
+    WebxdcSummaryTimestamp = b'Q',
 }
 
 /// An object for handling key=value parameter lists.
@@ -191,6 +206,11 @@ impl fmt::Display for Params {
 impl str::FromStr for Params {
     type Err = Error;
 
+    /// Parse a raw string to Param.
+    ///
+    /// Silently ignore unknown keys:
+    /// they may come from a downgrade (when a shortly new version adds a key)
+    /// or from an upgrade (when a key is dropped but was used in the past)
     fn from_str(s: &str) -> std::result::Result<Self, Self::Err> {
         let mut inner = BTreeMap::new();
         let mut lines = s.lines().peekable();
@@ -210,8 +230,6 @@ impl str::FromStr for Params {
 
                 if let Some(key) = key.as_bytes().first().and_then(|key| Param::from_u8(*key)) {
                     inner.insert(key, value);
-                } else {
-                    bail!("Unknown key: {}", key);
                 }
             } else {
                 bail!("Not a key-value pair: {:?}", line);
@@ -414,10 +432,12 @@ impl<'a> ParamsFile<'a> {
 mod tests {
     use super::*;
 
+    use anyhow::Result;
     use async_std::fs;
     use async_std::path::Path;
 
     use crate::test_utils::TestContext;
+    use std::str::FromStr;
 
     #[test]
     fn test_dc_param() {
@@ -520,4 +540,14 @@ mod tests {
         assert!(p.get_path(Param::File, &t).unwrap().is_none());
         assert!(p.get_blob(Param::File, &t, false).await.unwrap().is_none());
     }
+
+    #[async_std::test]
+    async fn test_params_unknown_key() -> Result<()> {
+        // 'Z' is used as a key that is known to be unused; these keys should be ignored silently by definition.
+        let p = Params::from_str("w=12\nZ=13\nh=14")?;
+        assert_eq!(p.len(), 2);
+        assert_eq!(p.get(Param::Width), Some("12"));
+        assert_eq!(p.get(Param::Height), Some("14"));
+        Ok(())
+    }
 }

src/peerstate.rs

@@ -277,7 +277,7 @@ impl Peerstate {
 
                 let msg = stock_str::contact_setup_changed(context, self.addr.clone()).await;
 
-                chat::add_info_msg(context, chat_id, msg, timestamp).await?;
+                chat::add_info_msg(context, chat_id, &msg, timestamp).await?;
                 context.emit_event(EventType::ChatModified(chat_id));
             } else {
                 bail!("contact with peerstate.addr {:?} not found", &self.addr);
@@ -379,10 +379,9 @@ impl Peerstate {
     pub fn peek_key(&self, min_verified: PeerstateVerifiedStatus) -> Option<&SignedPublicKey> {
         match min_verified {
             PeerstateVerifiedStatus::BidirectVerified => self.verified_key.as_ref(),
-            PeerstateVerifiedStatus::Unverified => self
-                .public_key
-                .as_ref()
-                .or_else(|| self.gossip_key.as_ref()),
+            PeerstateVerifiedStatus::Unverified => {
+                self.public_key.as_ref().or(self.gossip_key.as_ref())
+            }
         }
     }
 

src/pgp.rs

@@ -4,11 +4,11 @@ use std::collections::{BTreeMap, HashSet};
 use std::io;
 use std::io::Cursor;
 
-use anyhow::{bail, ensure, format_err, Result};
+use anyhow::{bail, ensure, format_err, Context as _, Result};
 use pgp::armor::BlockType;
 use pgp::composed::{
     Deserializable, KeyType as PgpKeyType, Message, SecretKeyParamsBuilder, SignedPublicKey,
-    SignedPublicSubKey, SignedSecretKey, SubkeyParamsBuilder,
+    SignedPublicSubKey, SignedSecretKey, StandaloneSignature, SubkeyParamsBuilder,
 };
 use pgp::crypto::{HashAlgorithm, SymmetricKeyAlgorithm};
 use pgp::types::{
@@ -248,7 +248,7 @@ pub async fn pk_encrypt(
         let pkeys: Vec<SignedPublicKeyOrSubkey> = public_keys_for_encryption
             .keys()
             .iter()
-            .filter_map(|key| select_pk_for_encryption(key))
+            .filter_map(select_pk_for_encryption)
             .collect();
         let pkeys_refs: Vec<&SignedPublicKeyOrSubkey> = pkeys.iter().collect();
 
@@ -277,16 +277,17 @@ pub async fn pk_encrypt(
 /// Receiver private keys are provided in
 /// `private_keys_for_decryption`.
 ///
-/// If `ret_signature_fingerprints` is not `None`, stores fingerprints
+/// Returns decrypted message and fingerprints
 /// of all keys from the `public_keys_for_validation` keyring that
 /// have valid signatures there.
 #[allow(clippy::implicit_hasher)]
 pub async fn pk_decrypt(
     ctext: Vec<u8>,
     private_keys_for_decryption: Keyring<SignedSecretKey>,
-    public_keys_for_validation: Keyring<SignedPublicKey>,
-    ret_signature_fingerprints: Option<&mut HashSet<Fingerprint>>,
-) -> Result<Vec<u8>> {
+    public_keys_for_validation: &Keyring<SignedPublicKey>,
+) -> Result<(Vec<u8>, HashSet<Fingerprint>)> {
+    let mut ret_signature_fingerprints: HashSet<Fingerprint> = Default::default();
+
     let msgs = async_std::task::spawn_blocking(move || {
         let cursor = Cursor::new(ctext);
         let (msg, _) = Message::from_armor_single(cursor)?;
@@ -308,33 +309,54 @@ pub async fn pk_decrypt(
             None => bail!("The decrypted message is empty"),
         };
 
-        if let Some(ret_signature_fingerprints) = ret_signature_fingerprints {
-            if !public_keys_for_validation.is_empty() {
-                let fingerprints = async_std::task::spawn_blocking(move || {
-                    let pkeys = public_keys_for_validation.keys();
+        if !public_keys_for_validation.is_empty() {
+            let pkeys = public_keys_for_validation.keys();
 
-                    let mut fingerprints: Vec<Fingerprint> = Vec::new();
-                    if let signed_msg @ pgp::composed::Message::Signed { .. } = msg {
-                        for pkey in pkeys {
-                            if signed_msg.verify(&pkey.primary_key).is_ok() {
-                                let fp = DcKey::fingerprint(pkey);
-                                fingerprints.push(fp);
-                            }
-                        }
+            let mut fingerprints: Vec<Fingerprint> = Vec::new();
+            if let signed_msg @ pgp::composed::Message::Signed { .. } = msg {
+                for pkey in pkeys {
+                    if signed_msg.verify(&pkey.primary_key).is_ok() {
+                        let fp = DcKey::fingerprint(pkey);
+                        fingerprints.push(fp);
                     }
-                    fingerprints
-                })
-                .await;
-
-                ret_signature_fingerprints.extend(fingerprints);
+                }
             }
+
+            ret_signature_fingerprints.extend(fingerprints);
         }
-        Ok(content)
+        Ok((content, ret_signature_fingerprints))
     } else {
         bail!("No valid messages found");
     }
 }
 
+/// Validates detached signature.
+pub async fn pk_validate(
+    content: &[u8],
+    signature: &[u8],
+    public_keys_for_validation: &Keyring<SignedPublicKey>,
+) -> Result<HashSet<Fingerprint>> {
+    let mut ret: HashSet<Fingerprint> = Default::default();
+
+    let standalone_signature = StandaloneSignature::from_armor_single(Cursor::new(signature))?.0;
+    let pkeys = public_keys_for_validation.keys();
+
+    // Remove trailing CRLF before the delimiter.
+    // According to RFC 3156 it is considered to be part of the MIME delimiter for the purpose of
+    // OpenPGP signature calculation.
+    let content = content
+        .get(..content.len().saturating_sub(2))
+        .context("index is out of range")?;
+
+    for pkey in pkeys {
+        if standalone_signature.verify(pkey, content).is_ok() {
+            let fp = DcKey::fingerprint(pkey);
+            ret.insert(fp);
+        }
+    }
+    Ok(ret)
+}
+
 /// Symmetric encryption.
 pub async fn symm_encrypt(passphrase: &str, plain: &[u8]) -> Result<String> {
     let lit_msg = Message::new_literal_bytes("", plain);
@@ -492,15 +514,12 @@ mod tests {
         decrypt_keyring.add(KEYS.alice_secret.clone());
         let mut sig_check_keyring: Keyring<SignedPublicKey> = Keyring::new();
         sig_check_keyring.add(KEYS.alice_public.clone());
-        let mut valid_signatures: HashSet<Fingerprint> = Default::default();
-        let plain = pk_decrypt(
+        let (plain, valid_signatures) = pk_decrypt(
             CTEXT_SIGNED.as_bytes().to_vec(),
             decrypt_keyring,
-            sig_check_keyring,
-            Some(&mut valid_signatures),
+            &sig_check_keyring,
         )
         .await
-        .map_err(|err| println!("{:?}", err))
         .unwrap();
         assert_eq!(plain, CLEARTEXT);
         assert_eq!(valid_signatures.len(), 1);
@@ -510,15 +529,12 @@ mod tests {
         decrypt_keyring.add(KEYS.bob_secret.clone());
         let mut sig_check_keyring = Keyring::new();
         sig_check_keyring.add(KEYS.alice_public.clone());
-        let mut valid_signatures: HashSet<Fingerprint> = Default::default();
-        let plain = pk_decrypt(
+        let (plain, valid_signatures) = pk_decrypt(
             CTEXT_SIGNED.as_bytes().to_vec(),
             decrypt_keyring,
-            sig_check_keyring,
-            Some(&mut valid_signatures),
+            &sig_check_keyring,
         )
         .await
-        .map_err(|err| println!("{:?}", err))
         .unwrap();
         assert_eq!(plain, CLEARTEXT);
         assert_eq!(valid_signatures.len(), 1);
@@ -529,15 +545,10 @@ mod tests {
         let mut keyring = Keyring::new();
         keyring.add(KEYS.alice_secret.clone());
         let empty_keyring = Keyring::new();
-        let mut valid_signatures: HashSet<Fingerprint> = Default::default();
-        let plain = pk_decrypt(
-            CTEXT_SIGNED.as_bytes().to_vec(),
-            keyring,
-            empty_keyring,
-            Some(&mut valid_signatures),
-        )
-        .await
-        .unwrap();
+        let (plain, valid_signatures) =
+            pk_decrypt(CTEXT_SIGNED.as_bytes().to_vec(), keyring, &empty_keyring)
+                .await
+                .unwrap();
         assert_eq!(plain, CLEARTEXT);
         assert_eq!(valid_signatures.len(), 0);
     }
@@ -549,12 +560,10 @@ mod tests {
         decrypt_keyring.add(KEYS.bob_secret.clone());
         let mut sig_check_keyring = Keyring::new();
         sig_check_keyring.add(KEYS.bob_public.clone());
-        let mut valid_signatures: HashSet<Fingerprint> = Default::default();
-        let plain = pk_decrypt(
+        let (plain, valid_signatures) = pk_decrypt(
             CTEXT_SIGNED.as_bytes().to_vec(),
             decrypt_keyring,
-            sig_check_keyring,
-            Some(&mut valid_signatures),
+            &sig_check_keyring,
         )
         .await
         .unwrap();
@@ -567,34 +576,14 @@ mod tests {
         let mut decrypt_keyring = Keyring::new();
         decrypt_keyring.add(KEYS.bob_secret.clone());
         let sig_check_keyring = Keyring::new();
-        let mut valid_signatures: HashSet<Fingerprint> = Default::default();
-        let plain = pk_decrypt(
+        let (plain, valid_signatures) = pk_decrypt(
             CTEXT_UNSIGNED.as_bytes().to_vec(),
             decrypt_keyring,
-            sig_check_keyring,
-            Some(&mut valid_signatures),
+            &sig_check_keyring,
         )
         .await
         .unwrap();
         assert_eq!(plain, CLEARTEXT);
         assert_eq!(valid_signatures.len(), 0);
     }
-
-    #[async_std::test]
-    async fn test_decrypt_signed_no_sigret() {
-        // Check decrypting signed cyphertext without providing the HashSet for signatures.
-        let mut decrypt_keyring = Keyring::new();
-        decrypt_keyring.add(KEYS.bob_secret.clone());
-        let mut sig_check_keyring = Keyring::new();
-        sig_check_keyring.add(KEYS.alice_public.clone());
-        let plain = pk_decrypt(
-            CTEXT_SIGNED.as_bytes().to_vec(),
-            decrypt_keyring,
-            sig_check_keyring,
-            None,
-        )
-        .await
-        .unwrap();
-        assert_eq!(plain, CLEARTEXT);
-    }
 }

src/plaintext.rs

@@ -41,7 +41,7 @@ impl PlainText {
             // as <http://example.org> cannot be handled correctly
             // (they would become &lt;http://example.org&gt; where the trailing &gt; would become a valid url part).
             // to avoid double encoding, we escape our html-entities by \r that must not be used in the string elsewhere.
-            let line = line.to_string().replace("\r", "");
+            let line = line.to_string().replace('\r', "");
 
             let mut line = LINKIFY_MAIL_RE
                 .replace_all(&*line, "\rLTa href=\rQUOTmailto:$1\rQUOT\rGT$1\rLT/a\rGT")

src/provider.rs

@@ -3,8 +3,10 @@
 mod data;
 
 use crate::config::Config;
+use crate::context::Context;
 use crate::provider::data::{PROVIDER_DATA, PROVIDER_IDS, PROVIDER_UPDATED};
-use async_std_resolver::resolver_from_system_conf;
+use anyhow::Result;
+use async_std_resolver::{config, resolver, resolver_from_system_conf, AsyncStdResolver};
 use chrono::{NaiveDateTime, NaiveTime};
 
 #[derive(Debug, Display, Copy, Clone, PartialEq, FromPrimitive, ToPrimitive)]
@@ -81,6 +83,23 @@ pub struct Provider {
     pub oauth2_authorizer: Option<Oauth2Authorizer>,
 }
 
+/// Get resolver to query MX records.
+///
+/// We first try resolver_from_system_conf() which reads the system's resolver from `/etc/resolv.conf`.
+/// This does not work at least on some Androids, therefore we use use ResolverConfig::default()
+/// which default eg. to google's 8.8.8.8 or 8.8.4.4 as a fallback.
+async fn get_resolver() -> Result<AsyncStdResolver> {
+    if let Ok(resolver) = resolver_from_system_conf().await {
+        return Ok(resolver);
+    }
+    let resolver = resolver(
+        config::ResolverConfig::default(),
+        config::ResolverOpts::default(),
+    )
+    .await?;
+    Ok(resolver)
+}
+
 /// Returns provider for the given domain.
 ///
 /// This function looks up domain in offline database first. If not
@@ -89,15 +108,19 @@ pub struct Provider {
 ///
 /// For compatibility, email address can be passed to this function
 /// instead of the domain.
-pub async fn get_provider_info(domain: &str, skip_mx: bool) -> Option<&'static Provider> {
-    let domain = domain.rsplitn(2, '@').next()?;
+pub async fn get_provider_info(
+    context: &Context,
+    domain: &str,
+    skip_mx: bool,
+) -> Option<&'static Provider> {
+    let domain = domain.rsplit('@').next()?;
 
     if let Some(provider) = get_provider_by_domain(domain) {
         return Some(provider);
     }
 
     if !skip_mx {
-        if let Some(provider) = get_provider_by_mx(domain).await {
+        if let Some(provider) = get_provider_by_mx(context, domain).await {
             return Some(provider);
         }
     }
@@ -117,8 +140,8 @@ pub fn get_provider_by_domain(domain: &str) -> Option<&'static Provider> {
 /// Finds a provider based on MX record for the given domain.
 ///
 /// For security reasons, only Gmail can be configured this way.
-pub async fn get_provider_by_mx(domain: &str) -> Option<&'static Provider> {
-    if let Ok(resolver) = resolver_from_system_conf().await {
+pub async fn get_provider_by_mx(context: &Context, domain: &str) -> Option<&'static Provider> {
+    if let Ok(resolver) = get_resolver().await {
         let mut fqdn: String = domain.to_string();
         if !fqdn.ends_with('.') {
             fqdn.push('.');
@@ -143,6 +166,8 @@ pub async fn get_provider_by_mx(domain: &str) -> Option<&'static Provider> {
                 }
             }
         }
+    } else {
+        warn!(context, "cannot get a resolver to check MX records.");
     }
 
     None
@@ -169,6 +194,7 @@ mod tests {
 
     use super::*;
     use crate::dc_tools::time;
+    use crate::test_utils::TestContext;
     use chrono::NaiveDate;
 
     #[test]
@@ -218,12 +244,13 @@ mod tests {
 
     #[async_std::test]
     async fn test_get_provider_info() {
-        assert!(get_provider_info("", false).await.is_none());
-        assert!(get_provider_info("google.com", false).await.unwrap().id == "gmail");
+        let t = TestContext::new().await;
+        assert!(get_provider_info(&t, "", false).await.is_none());
+        assert!(get_provider_info(&t, "google.com", false).await.unwrap().id == "gmail");
 
         // get_provider_info() accepts email addresses for backwards compatibility
         assert!(
-            get_provider_info("example@google.com", false)
+            get_provider_info(&t, "example@google.com", false)
                 .await
                 .unwrap()
                 .id
@@ -242,4 +269,10 @@ mod tests {
         assert!(get_provider_update_timestamp() <= time());
         assert!(get_provider_update_timestamp() > timestamp_past);
     }
+
+    #[async_std::test]
+    async fn test_get_resolver() -> Result<()> {
+        assert!(get_resolver().await.is_ok());
+        Ok(())
+    }
 }

src/provider/data.rs

@@ -282,61 +282,23 @@ static P_DISROOT: Lazy<Provider> = Lazy::new(|| Provider {
     before_login_hint: "",
     after_login_hint: "",
     overview_page: "https://providers.delta.chat/disroot",
-    server: vec![],
-    config_defaults: None,
-    strict_tls: true,
-    max_smtp_rcpt_to: None,
-    oauth2_authorizer: None,
-});
-
-// dubby.org.md: dubby.org
-static P_DUBBY_ORG: Lazy<Provider> = Lazy::new(|| Provider {
-    id: "dubby.org",
-    status: Status::Ok,
-    before_login_hint: "",
-    after_login_hint: "",
-    overview_page: "https://providers.delta.chat/dubby-org",
     server: vec![
         Server {
             protocol: Imap,
             socket: Ssl,
-            hostname: "dubby.org",
+            hostname: "disroot.org",
             port: 993,
             username_pattern: Email,
         },
         Server {
             protocol: Smtp,
             socket: Starttls,
-            hostname: "dubby.org",
+            hostname: "disroot.org",
             port: 587,
             username_pattern: Email,
         },
-        Server {
-            protocol: Smtp,
-            socket: Ssl,
-            hostname: "dubby.org",
-            port: 465,
-            username_pattern: Email,
-        },
     ],
-    config_defaults: Some(vec![
-        ConfigDefault {
-            key: Config::BccSelf,
-            value: "1",
-        },
-        ConfigDefault {
-            key: Config::SentboxWatch,
-            value: "0",
-        },
-        ConfigDefault {
-            key: Config::MvboxWatch,
-            value: "0",
-        },
-        ConfigDefault {
-            key: Config::MvboxMove,
-            value: "0",
-        },
-    ]),
+    config_defaults: None,
     strict_tls: true,
     max_smtp_rcpt_to: None,
     oauth2_authorizer: None,
@@ -404,7 +366,7 @@ static P_EXAMPLE_COM: Lazy<Provider> = Lazy::new(|| {
 }
 });
 
-// fastmail.md: fastmail.com
+// fastmail.md: 123mail.org, 150mail.com, 150ml.com, 16mail.com, 2-mail.com, 4email.net, 50mail.com, airpost.net, allmail.net, bestmail.us, cluemail.com, elitemail.org, emailcorner.net, emailengine.net, emailengine.org, emailgroups.net, emailplus.org, emailuser.net, eml.cc, f-m.fm, fast-email.com, fast-mail.org, fastem.com, fastemail.us, fastemailer.com, fastest.cc, fastimap.com, fastmail.cn, fastmail.co.uk, fastmail.com, fastmail.com.au, fastmail.de, fastmail.es, fastmail.fm, fastmail.fr, fastmail.im, fastmail.in, fastmail.jp, fastmail.mx, fastmail.net, fastmail.nl, fastmail.org, fastmail.se, fastmail.to, fastmail.tw, fastmail.uk, fastmail.us, fastmailbox.net, fastmessaging.com, fea.st, fmail.co.uk, fmailbox.com, fmgirl.com, fmguy.com, ftml.net, h-mail.us, hailmail.net, imap-mail.com, imap.cc, imapmail.org, inoutbox.com, internet-e-mail.com, internet-mail.org, internetemails.net, internetmailing.net, jetemail.net, justemail.net, letterboxes.org, mail-central.com, mail-page.com, mailandftp.com, mailas.com, mailbolt.com, mailc.net, mailcan.com, mailforce.net, mailftp.com, mailhaven.com, mailingaddress.org, mailite.com, mailmight.com, mailnew.com, mailsent.net, mailservice.ms, mailup.net, mailworks.org, ml1.net, mm.st, myfastmail.com, mymacmail.com, nospammail.net, ownmail.net, petml.com, postinbox.com, postpro.net, proinbox.com, promessage.com, realemail.net, reallyfast.biz, reallyfast.info, rushpost.com, sent.as, sent.at, sent.com, speedpost.net, speedymail.org, ssl-mail.com, swift-mail.com, the-fastest.net, the-quickest.com, theinternetemail.com, veryfast.biz, veryspeedy.net, warpmail.net, xsmail.com, yepmail.net, your-mail.com
 static P_FASTMAIL: Lazy<Provider> = Lazy::new(|| Provider {
     id: "fastmail",
     status: Status::Preparation,
@@ -427,13 +389,6 @@ static P_FASTMAIL: Lazy<Provider> = Lazy::new(|| Provider {
             port: 465,
             username_pattern: Email,
         },
-        Server {
-            protocol: Smtp,
-            socket: Starttls,
-            hostname: "smtp.fastmail.com",
-            port: 587,
-            username_pattern: Email,
-        },
     ],
     config_defaults: None,
     strict_tls: true,
@@ -475,10 +430,6 @@ static P_FIVE_CHAT: Lazy<Provider> = Lazy::new(|| Provider {
             key: Config::SentboxWatch,
             value: "0",
         },
-        ConfigDefault {
-            key: Config::MvboxWatch,
-            value: "0",
-        },
         ConfigDefault {
             key: Config::MvboxMove,
             value: "0",
@@ -668,6 +619,35 @@ static P_ICLOUD: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
+// infomaniak.com.md: ik.me
+static P_INFOMANIAK_COM: Lazy<Provider> = Lazy::new(|| Provider {
+    id: "infomaniak.com",
+    status: Status::Ok,
+    before_login_hint: "",
+    after_login_hint: "",
+    overview_page: "https://providers.delta.chat/infomaniak-com",
+    server: vec![
+        Server {
+            protocol: Imap,
+            socket: Ssl,
+            hostname: "mail.infomaniak.com",
+            port: 993,
+            username_pattern: Email,
+        },
+        Server {
+            protocol: Smtp,
+            socket: Ssl,
+            hostname: "mail.infomaniak.com",
+            port: 465,
+            username_pattern: Email,
+        },
+    ],
+    config_defaults: None,
+    strict_tls: true,
+    max_smtp_rcpt_to: Some(10),
+    oauth2_authorizer: None,
+});
+
 // kolst.com.md: kolst.com
 static P_KOLST_COM: Lazy<Provider> = Lazy::new(|| Provider {
     id: "kolst.com",
@@ -696,12 +676,41 @@ static P_KONTENT_COM: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
+// mail.de.md: mail.de
+static P_MAIL_DE: Lazy<Provider> = Lazy::new(|| Provider {
+    id: "mail.de",
+    status: Status::Ok,
+    before_login_hint: "",
+    after_login_hint: "",
+    overview_page: "https://providers.delta.chat/mail-de",
+    server: vec![
+        Server {
+            protocol: Imap,
+            socket: Ssl,
+            hostname: "imap.mail.de",
+            port: 993,
+            username_pattern: Email,
+        },
+        Server {
+            protocol: Smtp,
+            socket: Ssl,
+            hostname: "smtp.mail.de",
+            port: 465,
+            username_pattern: Email,
+        },
+    ],
+    config_defaults: None,
+    strict_tls: true,
+    max_smtp_rcpt_to: None,
+    oauth2_authorizer: None,
+});
+
 // mail.ru.md: mail.ru, inbox.ru, internet.ru, bk.ru, list.ru
 static P_MAIL_RU: Lazy<Provider> = Lazy::new(|| {
     Provider {
     id: "mail.ru",
-    status: Status::Ok,
-    before_login_hint: "Не рекомендуется использовать mail.ru, потому что он разряжает вашу батарею быстрее, чем другие провайдеры.",
+    status: Status::Preparation,
+    before_login_hint: "Вам необходимо сгенерировать \"пароль для внешнего приложения\" в веб-интерфейсе mail.ru, чтобы mail.ru работал с Delta Chat.",
     after_login_hint: "",
     overview_page: "https://providers.delta.chat/mail-ru",
     server: vec![
@@ -751,7 +760,22 @@ static P_MAILBOX_ORG: Lazy<Provider> = Lazy::new(|| Provider {
     before_login_hint: "",
     after_login_hint: "",
     overview_page: "https://providers.delta.chat/mailbox-org",
-    server: vec![],
+    server: vec![
+        Server {
+            protocol: Imap,
+            socket: Ssl,
+            hostname: "imap.mailbox.org",
+            port: 993,
+            username_pattern: Email,
+        },
+        Server {
+            protocol: Smtp,
+            socket: Ssl,
+            hostname: "smtp.mailbox.org",
+            port: 465,
+            username_pattern: Email,
+        },
+    ],
     config_defaults: None,
     strict_tls: true,
     max_smtp_rcpt_to: None,
@@ -823,10 +847,6 @@ static P_NAUTA_CU: Lazy<Provider> = Lazy::new(|| Provider {
             key: Config::SentboxWatch,
             value: "0",
         },
-        ConfigDefault {
-            key: Config::MvboxWatch,
-            value: "0",
-        },
         ConfigDefault {
             key: Config::MvboxMove,
             value: "0",
@@ -878,7 +898,7 @@ static P_NAVER: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
-// outlook.com.md: hotmail.com, outlook.com, office365.com, outlook.com.tr, live.com
+// outlook.com.md: hotmail.com, outlook.com, office365.com, outlook.com.tr, live.com, outlook.de
 static P_OUTLOOK_COM: Lazy<Provider> = Lazy::new(|| Provider {
     id: "outlook.com",
     status: Status::Ok,
@@ -907,7 +927,7 @@ static P_OUTLOOK_COM: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
-// posteo.md: posteo.de, posteo.af, posteo.at, posteo.be, posteo.ch, posteo.cl, posteo.co, posteo.co.uk, posteo.com.br, posteo.cr, posteo.cz, posteo.dk, posteo.ee, posteo.es, posteo.eu, posteo.fi, posteo.gl, posteo.gr, posteo.hn, posteo.hr, posteo.hu, posteo.ie, posteo.in, posteo.is, posteo.it, posteo.jp, posteo.la, posteo.li, posteo.lt, posteo.lu, posteo.me, posteo.mx, posteo.my, posteo.net, posteo.nl, posteo.no, posteo.nz, posteo.org, posteo.pe, posteo.pl, posteo.pm, posteo.pt, posteo.ro, posteo.ru, posteo.se, posteo.sg, posteo.si, posteo.tn, posteo.uk, posteo.us
+// posteo.md: posteo.de, posteo.af, posteo.at, posteo.be, posteo.ca, posteo.ch, posteo.cl, posteo.co, posteo.co.uk, posteo.com.br, posteo.cr, posteo.cz, posteo.dk, posteo.ee, posteo.es, posteo.eu, posteo.fi, posteo.gl, posteo.gr, posteo.hn, posteo.hr, posteo.hu, posteo.ie, posteo.in, posteo.is, posteo.it, posteo.jp, posteo.la, posteo.li, posteo.lt, posteo.lu, posteo.me, posteo.mx, posteo.my, posteo.net, posteo.nl, posteo.no, posteo.nz, posteo.org, posteo.pe, posteo.pl, posteo.pm, posteo.pt, posteo.ro, posteo.ru, posteo.se, posteo.sg, posteo.si, posteo.tn, posteo.uk, posteo.us
 static P_POSTEO: Lazy<Provider> = Lazy::new(|| Provider {
     id: "posteo",
     status: Status::Ok,
@@ -936,7 +956,7 @@ static P_POSTEO: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
-// protonmail.md: protonmail.com, protonmail.ch
+// protonmail.md: protonmail.com, protonmail.ch, pm.me
 static P_PROTONMAIL: Lazy<Provider> = Lazy::new(|| {
     Provider {
     id: "protonmail",
@@ -979,7 +999,22 @@ static P_RISEUP_NET: Lazy<Provider> = Lazy::new(|| Provider {
     before_login_hint: "",
     after_login_hint: "",
     overview_page: "https://providers.delta.chat/riseup-net",
-    server: vec![],
+    server: vec![
+        Server {
+            protocol: Imap,
+            socket: Ssl,
+            hostname: "mail.riseup.net",
+            port: 993,
+            username_pattern: Email,
+        },
+        Server {
+            protocol: Smtp,
+            socket: Ssl,
+            hostname: "mail.riseup.net",
+            port: 465,
+            username_pattern: Email,
+        },
+    ],
     config_defaults: None,
     strict_tls: true,
     max_smtp_rcpt_to: None,
@@ -1036,7 +1071,22 @@ static P_SYSTEMLI_ORG: Lazy<Provider> = Lazy::new(|| Provider {
     before_login_hint: "",
     after_login_hint: "",
     overview_page: "https://providers.delta.chat/systemli-org",
-    server: vec![],
+    server: vec![
+        Server {
+            protocol: Imap,
+            socket: Ssl,
+            hostname: "mail.systemli.org",
+            port: 993,
+            username_pattern: Email,
+        },
+        Server {
+            protocol: Smtp,
+            socket: Ssl,
+            hostname: "mail.systemli.org",
+            port: 465,
+            username_pattern: Email,
+        },
+    ],
     config_defaults: None,
     strict_tls: true,
     max_smtp_rcpt_to: None,
@@ -1101,10 +1151,6 @@ static P_TESTRUN: Lazy<Provider> = Lazy::new(|| Provider {
             key: Config::SentboxWatch,
             value: "0",
         },
-        ConfigDefault {
-            key: Config::MvboxWatch,
-            value: "0",
-        },
         ConfigDefault {
             key: Config::MvboxMove,
             value: "0",
@@ -1437,13 +1483,128 @@ pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>>
         ("comcast.net", &*P_COMCAST),
         ("dismail.de", &*P_DISMAIL_DE),
         ("disroot.org", &*P_DISROOT),
-        ("dubby.org", &*P_DUBBY_ORG),
         ("e.email", &*P_E_EMAIL),
         ("espiv.net", &*P_ESPIV_NET),
         ("example.com", &*P_EXAMPLE_COM),
         ("example.org", &*P_EXAMPLE_COM),
         ("example.net", &*P_EXAMPLE_COM),
+        ("123mail.org", &*P_FASTMAIL),
+        ("150mail.com", &*P_FASTMAIL),
+        ("150ml.com", &*P_FASTMAIL),
+        ("16mail.com", &*P_FASTMAIL),
+        ("2-mail.com", &*P_FASTMAIL),
+        ("4email.net", &*P_FASTMAIL),
+        ("50mail.com", &*P_FASTMAIL),
+        ("airpost.net", &*P_FASTMAIL),
+        ("allmail.net", &*P_FASTMAIL),
+        ("bestmail.us", &*P_FASTMAIL),
+        ("cluemail.com", &*P_FASTMAIL),
+        ("elitemail.org", &*P_FASTMAIL),
+        ("emailcorner.net", &*P_FASTMAIL),
+        ("emailengine.net", &*P_FASTMAIL),
+        ("emailengine.org", &*P_FASTMAIL),
+        ("emailgroups.net", &*P_FASTMAIL),
+        ("emailplus.org", &*P_FASTMAIL),
+        ("emailuser.net", &*P_FASTMAIL),
+        ("eml.cc", &*P_FASTMAIL),
+        ("f-m.fm", &*P_FASTMAIL),
+        ("fast-email.com", &*P_FASTMAIL),
+        ("fast-mail.org", &*P_FASTMAIL),
+        ("fastem.com", &*P_FASTMAIL),
+        ("fastemail.us", &*P_FASTMAIL),
+        ("fastemailer.com", &*P_FASTMAIL),
+        ("fastest.cc", &*P_FASTMAIL),
+        ("fastimap.com", &*P_FASTMAIL),
+        ("fastmail.cn", &*P_FASTMAIL),
+        ("fastmail.co.uk", &*P_FASTMAIL),
         ("fastmail.com", &*P_FASTMAIL),
+        ("fastmail.com.au", &*P_FASTMAIL),
+        ("fastmail.de", &*P_FASTMAIL),
+        ("fastmail.es", &*P_FASTMAIL),
+        ("fastmail.fm", &*P_FASTMAIL),
+        ("fastmail.fr", &*P_FASTMAIL),
+        ("fastmail.im", &*P_FASTMAIL),
+        ("fastmail.in", &*P_FASTMAIL),
+        ("fastmail.jp", &*P_FASTMAIL),
+        ("fastmail.mx", &*P_FASTMAIL),
+        ("fastmail.net", &*P_FASTMAIL),
+        ("fastmail.nl", &*P_FASTMAIL),
+        ("fastmail.org", &*P_FASTMAIL),
+        ("fastmail.se", &*P_FASTMAIL),
+        ("fastmail.to", &*P_FASTMAIL),
+        ("fastmail.tw", &*P_FASTMAIL),
+        ("fastmail.uk", &*P_FASTMAIL),
+        ("fastmail.us", &*P_FASTMAIL),
+        ("fastmailbox.net", &*P_FASTMAIL),
+        ("fastmessaging.com", &*P_FASTMAIL),
+        ("fea.st", &*P_FASTMAIL),
+        ("fmail.co.uk", &*P_FASTMAIL),
+        ("fmailbox.com", &*P_FASTMAIL),
+        ("fmgirl.com", &*P_FASTMAIL),
+        ("fmguy.com", &*P_FASTMAIL),
+        ("ftml.net", &*P_FASTMAIL),
+        ("h-mail.us", &*P_FASTMAIL),
+        ("hailmail.net", &*P_FASTMAIL),
+        ("imap-mail.com", &*P_FASTMAIL),
+        ("imap.cc", &*P_FASTMAIL),
+        ("imapmail.org", &*P_FASTMAIL),
+        ("inoutbox.com", &*P_FASTMAIL),
+        ("internet-e-mail.com", &*P_FASTMAIL),
+        ("internet-mail.org", &*P_FASTMAIL),
+        ("internetemails.net", &*P_FASTMAIL),
+        ("internetmailing.net", &*P_FASTMAIL),
+        ("jetemail.net", &*P_FASTMAIL),
+        ("justemail.net", &*P_FASTMAIL),
+        ("letterboxes.org", &*P_FASTMAIL),
+        ("mail-central.com", &*P_FASTMAIL),
+        ("mail-page.com", &*P_FASTMAIL),
+        ("mailandftp.com", &*P_FASTMAIL),
+        ("mailas.com", &*P_FASTMAIL),
+        ("mailbolt.com", &*P_FASTMAIL),
+        ("mailc.net", &*P_FASTMAIL),
+        ("mailcan.com", &*P_FASTMAIL),
+        ("mailforce.net", &*P_FASTMAIL),
+        ("mailftp.com", &*P_FASTMAIL),
+        ("mailhaven.com", &*P_FASTMAIL),
+        ("mailingaddress.org", &*P_FASTMAIL),
+        ("mailite.com", &*P_FASTMAIL),
+        ("mailmight.com", &*P_FASTMAIL),
+        ("mailnew.com", &*P_FASTMAIL),
+        ("mailsent.net", &*P_FASTMAIL),
+        ("mailservice.ms", &*P_FASTMAIL),
+        ("mailup.net", &*P_FASTMAIL),
+        ("mailworks.org", &*P_FASTMAIL),
+        ("ml1.net", &*P_FASTMAIL),
+        ("mm.st", &*P_FASTMAIL),
+        ("myfastmail.com", &*P_FASTMAIL),
+        ("mymacmail.com", &*P_FASTMAIL),
+        ("nospammail.net", &*P_FASTMAIL),
+        ("ownmail.net", &*P_FASTMAIL),
+        ("petml.com", &*P_FASTMAIL),
+        ("postinbox.com", &*P_FASTMAIL),
+        ("postpro.net", &*P_FASTMAIL),
+        ("proinbox.com", &*P_FASTMAIL),
+        ("promessage.com", &*P_FASTMAIL),
+        ("realemail.net", &*P_FASTMAIL),
+        ("reallyfast.biz", &*P_FASTMAIL),
+        ("reallyfast.info", &*P_FASTMAIL),
+        ("rushpost.com", &*P_FASTMAIL),
+        ("sent.as", &*P_FASTMAIL),
+        ("sent.at", &*P_FASTMAIL),
+        ("sent.com", &*P_FASTMAIL),
+        ("speedpost.net", &*P_FASTMAIL),
+        ("speedymail.org", &*P_FASTMAIL),
+        ("ssl-mail.com", &*P_FASTMAIL),
+        ("swift-mail.com", &*P_FASTMAIL),
+        ("the-fastest.net", &*P_FASTMAIL),
+        ("the-quickest.com", &*P_FASTMAIL),
+        ("theinternetemail.com", &*P_FASTMAIL),
+        ("veryfast.biz", &*P_FASTMAIL),
+        ("veryspeedy.net", &*P_FASTMAIL),
+        ("warpmail.net", &*P_FASTMAIL),
+        ("xsmail.com", &*P_FASTMAIL),
+        ("yepmail.net", &*P_FASTMAIL),
+        ("your-mail.com", &*P_FASTMAIL),
         ("firemail.at", &*P_FIREMAIL_DE),
         ("firemail.de", &*P_FIREMAIL_DE),
         ("five.chat", &*P_FIVE_CHAT),
@@ -1467,8 +1628,10 @@ pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>>
         ("icloud.com", &*P_ICLOUD),
         ("me.com", &*P_ICLOUD),
         ("mac.com", &*P_ICLOUD),
+        ("ik.me", &*P_INFOMANIAK_COM),
         ("kolst.com", &*P_KOLST_COM),
         ("kontent.com", &*P_KONTENT_COM),
+        ("mail.de", &*P_MAIL_DE),
         ("mail.ru", &*P_MAIL_RU),
         ("inbox.ru", &*P_MAIL_RU),
         ("internet.ru", &*P_MAIL_RU),
@@ -1485,10 +1648,12 @@ pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>>
         ("office365.com", &*P_OUTLOOK_COM),
         ("outlook.com.tr", &*P_OUTLOOK_COM),
         ("live.com", &*P_OUTLOOK_COM),
+        ("outlook.de", &*P_OUTLOOK_COM),
         ("posteo.de", &*P_POSTEO),
         ("posteo.af", &*P_POSTEO),
         ("posteo.at", &*P_POSTEO),
         ("posteo.be", &*P_POSTEO),
+        ("posteo.ca", &*P_POSTEO),
         ("posteo.ch", &*P_POSTEO),
         ("posteo.cl", &*P_POSTEO),
         ("posteo.co", &*P_POSTEO),
@@ -1537,6 +1702,7 @@ pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>>
         ("posteo.us", &*P_POSTEO),
         ("protonmail.com", &*P_PROTONMAIL),
         ("protonmail.ch", &*P_PROTONMAIL),
+        ("pm.me", &*P_PROTONMAIL),
         ("qq.com", &*P_QQ),
         ("foxmail.com", &*P_QQ),
         ("riseup.net", &*P_RISEUP_NET),
@@ -1633,7 +1799,6 @@ pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> =
         ("comcast", &*P_COMCAST),
         ("dismail.de", &*P_DISMAIL_DE),
         ("disroot", &*P_DISROOT),
-        ("dubby.org", &*P_DUBBY_ORG),
         ("e.email", &*P_E_EMAIL),
         ("espiv.net", &*P_ESPIV_NET),
         ("example.com", &*P_EXAMPLE_COM),
@@ -1648,8 +1813,10 @@ pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> =
         ("i.ua", &*P_I_UA),
         ("i3.net", &*P_I3_NET),
         ("icloud", &*P_ICLOUD),
+        ("infomaniak.com", &*P_INFOMANIAK_COM),
         ("kolst.com", &*P_KOLST_COM),
         ("kontent.com", &*P_KONTENT_COM),
+        ("mail.de", &*P_MAIL_DE),
         ("mail.ru", &*P_MAIL_RU),
         ("mail2tor", &*P_MAIL2TOR),
         ("mailbox.org", &*P_MAILBOX_ORG),
@@ -1686,4 +1853,4 @@ pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> =
 });
 
 pub static PROVIDER_UPDATED: Lazy<chrono::NaiveDate> =
-    Lazy::new(|| chrono::NaiveDate::from_ymd(2021, 9, 29));
+    Lazy::new(|| chrono::NaiveDate::from_ymd(2022, 1, 31));

src/qr.rs

@@ -9,7 +9,7 @@ use std::collections::BTreeMap;
 use crate::chat::{self, get_chat_id_by_grpid, ChatIdBlocked};
 use crate::config::Config;
 use crate::constants::Blocked;
-use crate::contact::{addr_normalize, may_be_valid_addr, Contact, Origin};
+use crate::contact::{addr_normalize, may_be_valid_addr, Contact, ContactId, Origin};
 use crate::context::Context;
 use crate::dc_tools::time;
 use crate::key::Fingerprint;
@@ -30,7 +30,7 @@ const HTTPS_SCHEME: &str = "https://";
 #[derive(Debug, Clone, PartialEq)]
 pub enum Qr {
     AskVerifyContact {
-        contact_id: u32,
+        contact_id: ContactId,
         fingerprint: Fingerprint,
         invitenumber: String,
         authcode: String,
@@ -38,16 +38,16 @@ pub enum Qr {
     AskVerifyGroup {
         grpname: String,
         grpid: String,
-        contact_id: u32,
+        contact_id: ContactId,
         fingerprint: Fingerprint,
         invitenumber: String,
         authcode: String,
     },
     FprOk {
-        contact_id: u32,
+        contact_id: ContactId,
     },
     FprMismatch {
-        contact_id: Option<u32>,
+        contact_id: Option<ContactId>,
     },
     FprWithoutAddr {
         fingerprint: String,
@@ -60,7 +60,7 @@ pub enum Qr {
         instance_pattern: String,
     },
     Addr {
-        contact_id: u32,
+        contact_id: ContactId,
     },
     Url {
         url: String,
@@ -69,7 +69,7 @@ pub enum Qr {
         text: String,
     },
     WithdrawVerifyContact {
-        contact_id: u32,
+        contact_id: ContactId,
         fingerprint: Fingerprint,
         invitenumber: String,
         authcode: String,
@@ -77,13 +77,13 @@ pub enum Qr {
     WithdrawVerifyGroup {
         grpname: String,
         grpid: String,
-        contact_id: u32,
+        contact_id: ContactId,
         fingerprint: Fingerprint,
         invitenumber: String,
         authcode: String,
     },
     ReviveVerifyContact {
-        contact_id: u32,
+        contact_id: ContactId,
         fingerprint: Fingerprint,
         invitenumber: String,
         authcode: String,
@@ -91,7 +91,7 @@ pub enum Qr {
     ReviveVerifyGroup {
         grpname: String,
         grpid: String,
-        contact_id: u32,
+        contact_id: ContactId,
         fingerprint: Fingerprint,
         invitenumber: String,
         authcode: String,
@@ -173,7 +173,7 @@ async fn decode_openpgp(context: &Context, qr: &str) -> Result<Qr> {
     };
 
     let name = if let Some(encoded_name) = param.get("n") {
-        let encoded_name = encoded_name.replace("+", "%20"); // sometimes spaces are encoded as `+`
+        let encoded_name = encoded_name.replace('+', "%20"); // sometimes spaces are encoded as `+`
         match percent_decode_str(&encoded_name).decode_utf8() {
             Ok(name) => name.to_string(),
             Err(err) => bail!("Invalid name: {}", err),
@@ -188,7 +188,7 @@ async fn decode_openpgp(context: &Context, qr: &str) -> Result<Qr> {
 
     let grpname = if grpid.is_some() {
         if let Some(encoded_name) = param.get("g") {
-            let encoded_name = encoded_name.replace("+", "%20"); // sometimes spaces are encoded as `+`
+            let encoded_name = encoded_name.replace('+', "%20"); // sometimes spaces are encoded as `+`
             match percent_decode_str(&encoded_name).decode_utf8() {
                 Ok(name) => Some(name.to_string()),
                 Err(err) => bail!("Invalid group name: {}", err),
@@ -282,7 +282,7 @@ async fn decode_openpgp(context: &Context, qr: &str) -> Result<Qr> {
             chat::add_info_msg(
                 context,
                 chat.id,
-                format!("{} verified.", peerstate.addr),
+                &format!("{} verified.", peerstate.addr),
                 time(),
             )
             .await?;
@@ -304,14 +304,14 @@ async fn decode_openpgp(context: &Context, qr: &str) -> Result<Qr> {
 fn decode_account(qr: &str) -> Result<Qr> {
     let payload = qr
         .get(DCACCOUNT_SCHEME.len()..)
-        .ok_or_else(|| format_err!("Invalid DCACCOUNT payload"))?;
+        .context("invalid DCACCOUNT payload")?;
     let url =
         url::Url::parse(payload).with_context(|| format!("Invalid account URL: {:?}", payload))?;
     if url.scheme() == "http" || url.scheme() == "https" {
         Ok(Qr::Account {
             domain: url
                 .host_str()
-                .ok_or_else(|| format_err!("Can't extract WebRTC instance domain"))?
+                .context("can't extract WebRTC instance domain")?
                 .to_string(),
         })
     } else {
@@ -323,7 +323,7 @@ fn decode_account(qr: &str) -> Result<Qr> {
 fn decode_webrtc_instance(_context: &Context, qr: &str) -> Result<Qr> {
     let payload = qr
         .get(DCWEBRTC_SCHEME.len()..)
-        .ok_or_else(|| format_err!("Invalid DCWEBRTC payload"))?;
+        .context("invalid DCWEBRTC payload")?;
 
     let (_type, url) = Message::parse_webrtc_instance(payload);
     let url =
@@ -333,7 +333,7 @@ fn decode_webrtc_instance(_context: &Context, qr: &str) -> Result<Qr> {
         Ok(Qr::WebrtcInstance {
             domain: url
                 .host_str()
-                .ok_or_else(|| format_err!("Can't extract WebRTC instance domain"))?
+                .context("can't extract WebRTC instance domain")?
                 .to_string(),
             instance_pattern: payload.to_string(),
         })
@@ -424,7 +424,7 @@ pub async fn set_config_from_qr(context: &Context, qr: &str) -> Result<()> {
             grpid,
             ..
         } => {
-            let chat_id = get_chat_id_by_grpid(context, grpid)
+            let chat_id = get_chat_id_by_grpid(context, &grpid)
                 .await?
                 .map(|(chat_id, _protected, _blocked)| chat_id);
             token::save(
@@ -711,7 +711,7 @@ mod tests {
             ..
         } = qr
         {
-            assert_ne!(contact_id, 0);
+            assert_ne!(contact_id, ContactId::new(0));
             assert_eq!(grpname, "test ? test !");
         } else {
             bail!("Wrong QR code type");
@@ -729,7 +729,7 @@ mod tests {
             ..
         } = qr
         {
-            assert_ne!(contact_id, 0);
+            assert_ne!(contact_id, ContactId::new(0));
             assert_eq!(grpname, "test ? test !");
 
             let contact = Contact::get_by_id(&ctx.ctx, contact_id).await?;
@@ -751,7 +751,7 @@ mod tests {
         ).await?;
 
         if let Qr::AskVerifyContact { contact_id, .. } = qr {
-            assert_ne!(contact_id, 0);
+            assert_ne!(contact_id, ContactId::new(0));
         } else {
             bail!("Wrong QR code type");
         }
@@ -792,12 +792,12 @@ mod tests {
     async fn test_decode_openpgp_fingerprint() -> Result<()> {
         let ctx = TestContext::new().await;
 
-        let alice_contact_id = Contact::create(&ctx, "Alice", "alice@example.com")
+        let alice_contact_id = Contact::create(&ctx, "Alice", "alice@example.org")
             .await
             .context("failed to create contact")?;
         let pub_key = alice_keypair().public;
         let peerstate = Peerstate {
-            addr: "alice@example.com".to_string(),
+            addr: "alice@example.org".to_string(),
             last_seen: 1,
             last_seen_autocrypt: 1,
             prefer_encrypt: EncryptPreference::Mutual,
@@ -818,7 +818,7 @@ mod tests {
 
         let qr = check_qr(
             &ctx.ctx,
-            "OPENPGP4FPR:1234567890123456789012345678901234567890#a=alice@example.com",
+            "OPENPGP4FPR:1234567890123456789012345678901234567890#a=alice@example.org",
         )
         .await?;
         if let Qr::FprMismatch { contact_id, .. } = qr {
@@ -829,7 +829,7 @@ mod tests {
 
         let qr = check_qr(
             &ctx.ctx,
-            &format!("OPENPGP4FPR:{}#a=alice@example.com", pub_key.fingerprint()),
+            &format!("OPENPGP4FPR:{}#a=alice@example.org", pub_key.fingerprint()),
         )
         .await?;
         if let Qr::FprOk { contact_id, .. } = qr {

src/qr_code_generator.rs

@@ -89,21 +89,23 @@ fn inner_generate_secure_join_qr_code(
     let mut w = tagger::new(&mut svg);
 
     w.elem("svg", |d| {
-        d.attr("xmlns", "http://www.w3.org/2000/svg")
-            .attr("viewBox", format_args!("0 0 {} {}", width, height));
-    })
+        d.attr("xmlns", "http://www.w3.org/2000/svg")?;
+        d.attr("viewBox", format_args!("0 0 {} {}", width, height))?;
+        Ok(())
+    })?
     .build(|w| {
         // White Background apears like a card
         w.single("rect", |d| {
-            d.attr("x", card_border_size)
-                .attr("y", card_border_size)
-                .attr("rx", card_roundness)
-                .attr("stroke", "#c6c6c6")
-                .attr("stroke-width", card_border_size)
-                .attr("width", width - (card_border_size * 2.0))
-                .attr("height", height - (card_border_size * 2.0))
-                .attr("style", "fill:#f2f2f2");
-        });
+            d.attr("x", card_border_size)?;
+            d.attr("y", card_border_size)?;
+            d.attr("rx", card_roundness)?;
+            d.attr("stroke", "#c6c6c6")?;
+            d.attr("stroke-width", card_border_size)?;
+            d.attr("width", width - (card_border_size * 2.0))?;
+            d.attr("height", height - (card_border_size * 2.0))?;
+            d.attr("style", "fill:#f2f2f2")?;
+            Ok(())
+        })?;
         // Qrcode
         w.elem("g", |d| {
             d.attr(
@@ -113,12 +115,12 @@ fn inner_generate_secure_join_qr_code(
                     (width - qr_code_size) / 2.0,
                     ((height - qr_code_size) / 2.0) - qr_translate_up
                 ),
-            );
+            )
             // If the qr code should be in the wrong place,
             // we could also translate and scale the points in the path already,
             // but that would make the resulting svg way bigger in size and might bring up rounding issues,
             // so better avoid doing it manually if possible
-        })
+        })?
         .build(|w| {
             w.single("path", |d| {
                 let mut path_data = String::with_capacity(0);
@@ -132,16 +134,16 @@ fn inner_generate_secure_join_qr_code(
                     }
                 }
 
-                d.attr("style", "fill:#000000")
-                    .attr("d", path_data)
-                    .attr("transform", format!("scale({})", scale));
-            });
-        });
+                d.attr("style", "fill:#000000")?;
+                d.attr("d", path_data)?;
+                d.attr("transform", format!("scale({})", scale))
+            })
+        })?;
 
         // Text
         const BIG_TEXT_CHARS_PER_LINE: usize = 32;
         const SMALL_TEXT_CHARS_PER_LINE: usize = 38;
-        let chars_per_line = if qrcode_description.len() > SMALL_TEXT_CHARS_PER_LINE*2 {
+        let chars_per_line = if qrcode_description.len() > SMALL_TEXT_CHARS_PER_LINE * 2 {
             SMALL_TEXT_CHARS_PER_LINE
         } else {
             BIG_TEXT_CHARS_PER_LINE
@@ -152,27 +154,27 @@ fn inner_generate_secure_join_qr_code(
         } else {
             (19.0, -10.0)
         };
-        for (count, line) in lines.split('\n').enumerate()
-        {
+        for (count, line) in lines.split('\n').enumerate() {
             w.elem("text", |d| {
-                d.attr("y", (count as f32 * (text_font_size * 1.2)) + text_y_pos + text_y_shift)
-                    .attr("x", width / 2.0)
-                    .attr("text-anchor", "middle")
-                    .attr(
-                        "style",
-                        format!(
-                            "font-family:sans-serif;\
+                d.attr(
+                    "y",
+                    (count as f32 * (text_font_size * 1.2)) + text_y_pos + text_y_shift,
+                )?;
+                d.attr("x", width / 2.0)?;
+                d.attr("text-anchor", "middle")?;
+                d.attr(
+                    "style",
+                    format!(
+                        "font-family:sans-serif;\
                         font-weight:bold;\
                         font-size:{}px;\
                         fill:#000000;\
                         stroke:none",
-                            text_font_size
-                        ),
-                    );
-            })
-            .build(|w| {
-                w.put_raw(line);
-            });
+                        text_font_size
+                    ),
+                )
+            })?
+            .build(|w| w.put_raw(line))?;
         }
         // contact avatar in middle of qrcode
         const LOGO_SIZE: f32 = 94.4;
@@ -183,68 +185,64 @@ fn inner_generate_secure_join_qr_code(
             ((height - qr_code_size) / 2.0) - qr_translate_up + logo_position_in_qr;
 
         w.single("circle", |d| {
-            d.attr("cx", logo_position_x + HALF_LOGO_SIZE)
-                .attr("cy", logo_position_y + HALF_LOGO_SIZE)
-                .attr("r", HALF_LOGO_SIZE + avatar_border_size)
-                .attr("style", "fill:#f2f2f2");
-        });
+            d.attr("cx", logo_position_x + HALF_LOGO_SIZE)?;
+            d.attr("cy", logo_position_y + HALF_LOGO_SIZE)?;
+            d.attr("r", HALF_LOGO_SIZE + avatar_border_size)?;
+            d.attr("style", "fill:#f2f2f2")
+        })?;
 
         if let Some(img) = avatar {
-            w.elem("defs", |_| {}).build(|w| {
-                w.elem("clipPath", |d| {
-                    d.attr("id", "avatar-cut");
-                })
-                .build(|w| {
-                    w.single("circle", |d| {
-                        d.attr("cx", logo_position_x + HALF_LOGO_SIZE)
-                            .attr("cy", logo_position_y + HALF_LOGO_SIZE)
-                            .attr("r", HALF_LOGO_SIZE);
-                    });
-                });
-            });
+            w.elem("defs", tagger::no_attr())?.build(|w| {
+                w.elem("clipPath", |d| d.attr("id", "avatar-cut"))?
+                    .build(|w| {
+                        w.single("circle", |d| {
+                            d.attr("cx", logo_position_x + HALF_LOGO_SIZE)?;
+                            d.attr("cy", logo_position_y + HALF_LOGO_SIZE)?;
+                            d.attr("r", HALF_LOGO_SIZE)
+                        })
+                    })
+            })?;
 
             w.single("image", |d| {
-                d.attr("x", logo_position_x)
-                    .attr("y", logo_position_y)
-                    .attr("width", HALF_LOGO_SIZE * 2.0)
-                    .attr("height", HALF_LOGO_SIZE * 2.0)
-                    .attr("preserveAspectRatio", "none")
-                    .attr("clip-path", "url(#avatar-cut)")
-                    .attr(
-                        "href" /*might need xlink:href instead if it doesn't work on older devices?*/, 
-                        format!("data:image/jpeg;base64,{}", base64::encode(img)),
-                    );
-            });
+                d.attr("x", logo_position_x)?;
+                d.attr("y", logo_position_y)?;
+                d.attr("width", HALF_LOGO_SIZE * 2.0)?;
+                d.attr("height", HALF_LOGO_SIZE * 2.0)?;
+                d.attr("preserveAspectRatio", "none")?;
+                d.attr("clip-path", "url(#avatar-cut)")?;
+                d.attr(
+                    "href", /*might need xlink:href instead if it doesn't work on older devices?*/
+                    format!("data:image/jpeg;base64,{}", base64::encode(img)),
+                )
+            })?;
         } else {
             w.single("circle", |d| {
-                d.attr("cx", logo_position_x + HALF_LOGO_SIZE)
-                    .attr("cy", logo_position_y + HALF_LOGO_SIZE)
-                    .attr("r", HALF_LOGO_SIZE)
-                    .attr("style", format!("fill:{}", &color));
-            });
+                d.attr("cx", logo_position_x + HALF_LOGO_SIZE)?;
+                d.attr("cy", logo_position_y + HALF_LOGO_SIZE)?;
+                d.attr("r", HALF_LOGO_SIZE)?;
+                d.attr("style", format!("fill:{}", &color))
+            })?;
 
             let avatar_font_size = LOGO_SIZE * 0.65;
             let font_offset = avatar_font_size * 0.1;
             w.elem("text", |d| {
-                d.attr("y", logo_position_y + HALF_LOGO_SIZE + font_offset)
-                    .attr("x", logo_position_x + HALF_LOGO_SIZE)
-                    .attr("text-anchor", "middle")
-                    .attr("dominant-baseline", "central")
-                    .attr("alignment-baseline", "middle")
-                    .attr(
-                        "style",
-                        format!(
-                            "font-family:sans-serif;\
+                d.attr("y", logo_position_y + HALF_LOGO_SIZE + font_offset)?;
+                d.attr("x", logo_position_x + HALF_LOGO_SIZE)?;
+                d.attr("text-anchor", "middle")?;
+                d.attr("dominant-baseline", "central")?;
+                d.attr("alignment-baseline", "middle")?;
+                d.attr(
+                    "style",
+                    format!(
+                        "font-family:sans-serif;\
                             font-weight:400;\
                             font-size:{}px;\
                             fill:#ffffff;",
-                            avatar_font_size
-                        ),
-                    );
-            })
-            .build(|w| {
-                w.put_raw(avatar_letter.to_uppercase());
-            });
+                        avatar_font_size
+                    ),
+                )
+            })?
+            .build(|w| w.put_raw(avatar_letter.to_uppercase()))?;
         }
 
         // Footer logo
@@ -258,12 +256,10 @@ fn inner_generate_secure_join_qr_code(
                     (width - FOOTER_WIDTH) / 2.0,
                     height - logo_offset - FOOTER_HEIGHT - text_y_shift
                 ),
-            );
-        })
-        .build(|w| {
-            w.put_raw(include_str!("../assets/qrcode_logo_footer.svg"));
-        });
-    });
+            )
+        })?
+        .build(|w| w.put_raw(include_str!("../assets/qrcode_logo_footer.svg")))
+    })?;
 
     Ok(svg)
 }

src/quota.rs

@@ -1,18 +1,17 @@
 //! # Support for IMAP QUOTA extension.
 
-use anyhow::{anyhow, Result};
+use anyhow::{anyhow, Context as _, Result};
 use async_imap::types::{Quota, QuotaResource};
 use std::collections::BTreeMap;
 
 use crate::chat::add_device_msg_with_importance;
 use crate::config::Config;
-use crate::constants::Viewtype;
 use crate::context::Context;
 use crate::dc_tools::time;
 use crate::imap::scan_folders::get_watched_folders;
 use crate::imap::Imap;
 use crate::job::{Action, Status};
-use crate::message::Message;
+use crate::message::{Message, Viewtype};
 use crate::param::Params;
 use crate::{job, stock_str, EventType};
 
@@ -64,7 +63,7 @@ async fn get_unique_quota_roots_and_usage(
                     .iter()
                     .find(|q| &q.root_name == quota_root_name)
                     .cloned()
-                    .ok_or_else(|| anyhow!("quota_root should have a quota"))?;
+                    .context("quota_root should have a quota")?;
                 // replace old quotas, because between fetching quotaroots for folders,
                 // messages could be recieved and so the usage could have been changed
                 *unique_quota_roots
@@ -96,7 +95,7 @@ fn get_highest_usage<'t>(
         }
     }
 
-    highest.ok_or_else(|| anyhow!("no quota_resource found, this is unexpected"))
+    highest.context("no quota_resource found, this is unexpected")
 }
 
 /// Checks if a quota warning is needed.
@@ -137,7 +136,7 @@ impl Context {
         }
 
         let quota = if imap.can_check_quota() {
-            let folders = get_watched_folders(self).await;
+            let folders = get_watched_folders(self).await?;
             get_unique_quota_roots_and_usage(folders, imap).await
         } else {
             Err(anyhow!(stock_str::not_supported_by_provider(self).await))

src/scheduler.rs

@@ -1,4 +1,4 @@
-use anyhow::{bail, Result};
+use anyhow::{bail, Context as _, Result};
 use async_std::prelude::*;
 use async_std::{
     channel::{self, Receiver, Sender},
@@ -8,10 +8,11 @@ use async_std::{
 use crate::config::Config;
 use crate::context::Context;
 use crate::dc_tools::maybe_add_time_based_warnings;
+use crate::ephemeral::delete_expired_imap_messages;
 use crate::imap::Imap;
 use crate::job::{self, Thread};
-use crate::message::MsgId;
-use crate::smtp::Smtp;
+use crate::log::LogExt;
+use crate::smtp::{send_smtp_messages, Smtp};
 
 use self::connectivity::ConnectivityStore;
 
@@ -82,11 +83,15 @@ async fn inbox_loop(ctx: Context, started: Sender<()>, inbox_handlers: ImapConne
         let mut jobs_loaded = 0;
         let mut info = InterruptInfo::default();
         loop {
-            match job::load_next(&ctx, Thread::Imap, &info)
-                .await
-                .ok()
-                .flatten()
-            {
+            let job = match job::load_next(&ctx, Thread::Imap, &info).await {
+                Err(err) => {
+                    error!(ctx, "Failed loading job from the database: {:#}.", err);
+                    None
+                }
+                Ok(job) => job,
+            };
+
+            match job {
                 Some(job) if jobs_loaded <= 20 => {
                     jobs_loaded += 1;
                     job::perform_job(&ctx, job::Connection::Inbox(&mut connection), job).await;
@@ -95,41 +100,15 @@ async fn inbox_loop(ctx: Context, started: Sender<()>, inbox_handlers: ImapConne
                 Some(job) => {
                     // Let the fetch run, but return back to the job afterwards.
                     jobs_loaded = 0;
-                    if ctx
-                        .get_config_bool(Config::InboxWatch)
-                        .await
-                        .unwrap_or_default()
-                    {
-                        info!(ctx, "postponing imap-job {} to run fetch...", job);
-                        fetch(&ctx, &mut connection).await;
-                    }
+                    info!(ctx, "postponing imap-job {} to run fetch...", job);
+                    fetch(&ctx, &mut connection).await;
                 }
                 None => {
                     jobs_loaded = 0;
 
-                    // Expunge folder if needed, e.g. if some jobs have
-                    // deleted messages on the server.
-                    if let Err(err) = connection.maybe_close_folder(&ctx).await {
-                        warn!(ctx, "failed to close folder: {:?}", err);
-                    }
-
                     maybe_add_time_based_warnings(&ctx).await;
 
-                    info = if ctx
-                        .get_config_bool(Config::InboxWatch)
-                        .await
-                        .unwrap_or_default()
-                    {
-                        fetch_idle(&ctx, &mut connection, Config::ConfiguredInboxFolder).await
-                    } else {
-                        if let Err(err) = connection.scan_folders(&ctx).await {
-                            warn!(ctx, "{}", err);
-                            connection.connectivity.set_err(&ctx, err).await;
-                        } else {
-                            connection.connectivity.set_not_configured(&ctx).await;
-                        }
-                        connection.fake_idle(&ctx, None).await
-                    };
+                    info = fetch_idle(&ctx, &mut connection, Config::ConfiguredInboxFolder).await;
                 }
             }
         }
@@ -157,7 +136,7 @@ async fn fetch(ctx: &Context, connection: &mut Imap) {
             }
 
             // fetch
-            if let Err(err) = connection.fetch(ctx, &watch_folder).await {
+            if let Err(err) = connection.fetch_move_delete(ctx, &watch_folder).await {
                 connection.trigger_reconnect(ctx).await;
                 warn!(ctx, "{:#}", err);
             }
@@ -183,22 +162,57 @@ async fn fetch_idle(ctx: &Context, connection: &mut Imap, folder: Config) -> Int
                 return connection.fake_idle(ctx, Some(watch_folder)).await;
             }
 
-            // fetch
-            if let Err(err) = connection.fetch(ctx, &watch_folder).await {
-                connection.trigger_reconnect(ctx).await;
+            // Mark expired messages for deletion.
+            if let Err(err) = delete_expired_imap_messages(ctx)
+                .await
+                .context("delete_expired_imap_messages failed")
+            {
                 warn!(ctx, "{:#}", err);
-                return InterruptInfo::new(false, None);
             }
 
+            // Fetch the watched folder.
+            if let Err(err) = connection.fetch_move_delete(ctx, &watch_folder).await {
+                connection.trigger_reconnect(ctx).await;
+                warn!(ctx, "{:#}", err);
+                return InterruptInfo::new(false);
+            }
+
+            // Scan additional folders only after finishing fetching the watched folder.
+            //
+            // On iOS the application has strictly limited time to work in background, so we may not
+            // be able to scan all folders before time is up if there are many of them.
             if folder == Config::ConfiguredInboxFolder {
                 // Only scan on the Inbox thread in order to prevent parallel scans, which might lead to duplicate messages
-                if let Err(err) = connection.scan_folders(ctx).await {
-                    // Don't reconnect, if there is a problem with the connection we will realize this when IDLEing
-                    // but maybe just one folder can't be selected or something
-                    warn!(ctx, "{}", err);
+                match connection.scan_folders(ctx).await {
+                    Err(err) => {
+                        // Don't reconnect, if there is a problem with the connection we will realize this when IDLEing
+                        // but maybe just one folder can't be selected or something
+                        warn!(ctx, "{}", err);
+                    }
+                    Ok(true) => {
+                        // Fetch the watched folder again in case scanning other folder moved messages
+                        // there.
+                        //
+                        // In most cases this will select the watched folder and return because there are
+                        // no new messages. We want to select the watched folder anyway before going IDLE
+                        // there, so this does not take additional protocol round-trip.
+                        if let Err(err) = connection.fetch_move_delete(ctx, &watch_folder).await {
+                            connection.trigger_reconnect(ctx).await;
+                            warn!(ctx, "{:#}", err);
+                            return InterruptInfo::new(false);
+                        }
+                    }
+                    Ok(false) => {}
                 }
             }
 
+            // Synchronize Seen flags.
+            connection
+                .sync_seen_flags(ctx, &watch_folder)
+                .await
+                .context("sync_seen_flags")
+                .ok_or_log(ctx);
+
             connection.connectivity.set_connected(ctx).await;
 
             // idle
@@ -208,7 +222,7 @@ async fn fetch_idle(ctx: &Context, connection: &mut Imap, folder: Config) -> Int
                     Err(err) => {
                         connection.trigger_reconnect(ctx).await;
                         warn!(ctx, "{}", err);
-                        InterruptInfo::new(false, None)
+                        InterruptInfo::new(false)
                     }
                 }
             } else {
@@ -293,17 +307,25 @@ async fn smtp_loop(ctx: Context, started: Sender<()>, smtp_handlers: SmtpConnect
 
         let mut interrupt_info = Default::default();
         loop {
-            match job::load_next(&ctx, Thread::Smtp, &interrupt_info)
-                .await
-                .ok()
-                .flatten()
-            {
+            let job = match job::load_next(&ctx, Thread::Smtp, &interrupt_info).await {
+                Err(err) => {
+                    error!(ctx, "Failed loading job from the database: {:#}.", err);
+                    None
+                }
+                Ok(job) => job,
+            };
+
+            match job {
                 Some(job) => {
                     info!(ctx, "executing smtp job");
                     job::perform_job(&ctx, job::Connection::Smtp(&mut connection), job).await;
                     interrupt_info = Default::default();
                 }
                 None => {
+                    if let Err(err) = send_smtp_messages(&ctx, &mut connection).await {
+                        warn!(ctx, "send_smtp_messages failed: {:#}", err);
+                    }
+
                     // Fake Idle
                     info!(ctx, "smtp fake idle - started");
                     match &connection.last_send_error {
@@ -353,7 +375,7 @@ impl Scheduler {
             }))
         };
 
-        if ctx.get_config_bool(Config::MvboxWatch).await? {
+        if ctx.should_watch_mvbox().await? {
             let ctx = ctx.clone();
             mvbox_handle = Some(task::spawn(async move {
                 simple_imap_loop(
@@ -437,10 +459,10 @@ impl Scheduler {
             return;
         }
 
-        self.interrupt_inbox(InterruptInfo::new(true, None))
-            .join(self.interrupt_mvbox(InterruptInfo::new(true, None)))
-            .join(self.interrupt_sentbox(InterruptInfo::new(true, None)))
-            .join(self.interrupt_smtp(InterruptInfo::new(true, None)))
+        self.interrupt_inbox(InterruptInfo::new(true))
+            .join(self.interrupt_mvbox(InterruptInfo::new(true)))
+            .join(self.interrupt_sentbox(InterruptInfo::new(true)))
+            .join(self.interrupt_smtp(InterruptInfo::new(true)))
             .await;
     }
 
@@ -449,10 +471,10 @@ impl Scheduler {
             return;
         }
 
-        self.interrupt_inbox(InterruptInfo::new(false, None))
-            .join(self.interrupt_mvbox(InterruptInfo::new(false, None)))
-            .join(self.interrupt_sentbox(InterruptInfo::new(false, None)))
-            .join(self.interrupt_smtp(InterruptInfo::new(false, None)))
+        self.interrupt_inbox(InterruptInfo::new(false))
+            .join(self.interrupt_mvbox(InterruptInfo::new(false)))
+            .join(self.interrupt_sentbox(InterruptInfo::new(false)))
+            .join(self.interrupt_smtp(InterruptInfo::new(false)))
             .await;
     }
 
@@ -682,14 +704,10 @@ struct ImapConnectionHandlers {
 #[derive(Default, Debug)]
 pub struct InterruptInfo {
     pub probe_network: bool,
-    pub msg_id: Option<MsgId>,
 }
 
 impl InterruptInfo {
-    pub fn new(probe_network: bool, msg_id: Option<MsgId>) -> Self {
-        Self {
-            probe_network,
-            msg_id,
-        }
+    pub fn new(probe_network: bool) -> Self {
+        Self { probe_network }
     }
 }

src/scheduler/connectivity.rs

@@ -5,6 +5,7 @@ use async_std::sync::{Mutex, RwLockReadGuard};
 
 use crate::dc_tools::time;
 use crate::events::EventType;
+use crate::imap::scan_folders::get_watched_folder_configs;
 use crate::quota::{
     QUOTA_ERROR_THRESHOLD_PERCENTAGE, QUOTA_MAX_AGE_SECONDS, QUOTA_WARN_THRESHOLD_PERCENTAGE,
 };
@@ -362,17 +363,14 @@ impl Context {
                 [
                     (
                         Config::ConfiguredInboxFolder,
-                        Config::InboxWatch,
                         inbox.state.connectivity.clone(),
                     ),
                     (
                         Config::ConfiguredMvboxFolder,
-                        Config::MvboxWatch,
                         mvbox.state.connectivity.clone(),
                     ),
                     (
                         Config::ConfiguredSentboxFolder,
-                        Config::SentboxWatch,
                         sentbox.state.connectivity.clone(),
                     ),
                 ],
@@ -391,12 +389,12 @@ impl Context {
         //                               - "Sent": Connected
         // =============================================================================================
 
+        let watched_folders = get_watched_folder_configs(self).await?;
         ret += &format!("<h3>{}</h3><ul>", stock_str::incoming_messages(self).await);
-        for (folder, watch, state) in &folders_states {
-            let w = self.get_config(*watch).await.ok_or_log(self);
-
+        for (folder, state) in &folders_states {
             let mut folder_added = false;
-            if w.flatten() == Some("1".to_string()) {
+
+            if watched_folders.contains(folder) {
                 let f = self.get_config(*folder).await.ok_or_log(self).flatten();
 
                 if let Some(foldername) = f {

src/securejoin.rs

@@ -3,21 +3,20 @@
 use std::convert::TryFrom;
 
 use anyhow::{bail, Context as _, Error, Result};
-use async_std::sync::Mutex;
 use percent_encoding::{utf8_percent_encode, AsciiSet, NON_ALPHANUMERIC};
 
 use crate::aheader::EncryptPreference;
-use crate::chat::{self, is_contact_in_chat, Chat, ChatId, ChatIdBlocked, ProtectionStatus};
+use crate::chat::{self, Chat, ChatId, ChatIdBlocked};
 use crate::config::Config;
-use crate::constants::{Blocked, Chattype, Viewtype, DC_CONTACT_ID_LAST_SPECIAL};
-use crate::contact::{Contact, Origin, VerifiedStatus};
+use crate::constants::{Blocked, DC_CONTACT_ID_LAST_SPECIAL};
+use crate::contact::{Contact, ContactId, Origin, VerifiedStatus};
 use crate::context::Context;
 use crate::dc_tools::time;
 use crate::e2ee::ensure_secret_key_exists;
 use crate::events::EventType;
 use crate::headerdef::HeaderDef;
 use crate::key::{DcKey, Fingerprint, SignedPublicKey};
-use crate::message::Message;
+use crate::message::{Message, Viewtype};
 use crate::mimeparser::{MimeMessage, SystemMessage};
 use crate::param::Param;
 use crate::peerstate::{Peerstate, PeerstateKeyType, PeerstateVerifiedStatus, ToSave};
@@ -25,28 +24,16 @@ use crate::qr::check_qr;
 use crate::stock_str;
 use crate::token;
 
+mod bob;
 mod bobstate;
 mod qrinvite;
 
 use crate::token::Namespace;
-use bobstate::{BobHandshakeStage, BobState, BobStateHandle};
+use bobstate::BobState;
 use qrinvite::QrInvite;
 
 pub const NON_ALPHANUMERIC_WITHOUT_DOT: &AsciiSet = &NON_ALPHANUMERIC.remove(b'.');
 
-macro_rules! joiner_progress {
-    ($context:tt, $contact_id:expr, $progress:expr) => {
-        assert!(
-            $progress >= 0 && $progress <= 1000,
-            "value in range 0..1000 expected with: 0=error, 1..999=progress, 1000=success"
-        );
-        $context.emit_event($crate::events::EventType::SecurejoinJoinerProgress {
-            contact_id: $contact_id,
-            progress: $progress,
-        });
-    };
-}
-
 macro_rules! inviter_progress {
     ($context:tt, $contact_id:expr, $progress:expr) => {
         assert!(
@@ -60,100 +47,6 @@ macro_rules! inviter_progress {
     };
 }
 
-/// State for setup-contact/secure-join protocol joiner's side, aka Bob's side.
-///
-/// The setup-contact protocol needs to carry state for both the inviter (Alice) and the
-/// joiner/invitee (Bob).  For Alice this state is minimal and in the `tokens` table in the
-/// database.  For Bob this state is only carried live on the [`Context`] in this struct.
-#[derive(Debug, Default)]
-pub(crate) struct Bob {
-    inner: Mutex<Option<BobState>>,
-}
-
-/// Return value for [`Bob::start_protocol`].
-///
-/// This indicates which protocol variant was started and provides the required information
-/// about it.
-enum StartedProtocolVariant {
-    /// The setup-contact protocol, to verify a contact.
-    SetupContact,
-    /// The secure-join protocol, to join a group.
-    SecureJoin {
-        group_id: String,
-        group_name: String,
-    },
-}
-
-impl Bob {
-    /// Starts the securejoin protocol with the QR `invite`.
-    ///
-    /// This will try to start the securejoin protocol for the given QR `invite`.  If it
-    /// succeeded the protocol state will be tracked in `self`.
-    ///
-    /// This function takes care of starting the "ongoing" mechanism if required and
-    /// handling errors while starting the protocol.
-    ///
-    /// # Returns
-    ///
-    /// If the started protocol is joining a group the returned struct contains information
-    /// about the group and ongoing process.
-    async fn start_protocol(
-        &self,
-        context: &Context,
-        invite: QrInvite,
-    ) -> Result<StartedProtocolVariant, JoinError> {
-        let mut guard = self.inner.lock().await;
-        if guard.is_some() {
-            warn!(context, "The new securejoin will replace the ongoing one.");
-            *guard = None;
-        }
-        let variant = match invite {
-            QrInvite::Group {
-                ref grpid,
-                ref name,
-                ..
-            } => StartedProtocolVariant::SecureJoin {
-                group_id: grpid.clone(),
-                group_name: name.clone(),
-            },
-            _ => StartedProtocolVariant::SetupContact,
-        };
-        match BobState::start_protocol(context, invite).await {
-            Ok((state, stage)) => {
-                if matches!(stage, BobHandshakeStage::RequestWithAuthSent) {
-                    joiner_progress!(context, state.invite().contact_id(), 400);
-                }
-                *guard = Some(state);
-                Ok(variant)
-            }
-            Err(err) => {
-                if let StartedProtocolVariant::SecureJoin { .. } = variant {
-                    context.free_ongoing().await;
-                }
-                Err(err)
-            }
-        }
-    }
-
-    /// Returns a handle to the [`BobState`] of the handshake.
-    ///
-    /// If there currently isn't a handshake running this will return `None`.  Otherwise
-    /// this will return a handle to the current [`BobState`].  This handle allows
-    /// processing an incoming message and allows terminating the handshake.
-    ///
-    /// The handle contains an exclusive lock, which is held until the handle is dropped.
-    /// This guarantees all state and state changes are correct and allows safely
-    /// terminating the handshake without worrying about concurrency.
-    async fn state(&self, context: &Context) -> Option<BobStateHandle<'_>> {
-        let guard = self.inner.lock().await;
-        let ret = BobStateHandle::from_guard(guard);
-        if ret.is_none() {
-            info!(context, "No active BobState found for securejoin handshake");
-        }
-        ret
-    }
-}
-
 /// Generates a Secure Join QR code.
 ///
 /// With `group` set to `None` this generates a setup-contact QR code, with `group` set to a
@@ -280,7 +173,7 @@ pub enum JoinError {
 pub async fn dc_join_securejoin(context: &Context, qr: &str) -> Result<ChatId, JoinError> {
     securejoin(context, qr).await.map_err(|err| {
         warn!(context, "Fatal joiner error: {:#}", err);
-        // This is a modal operation, the user has context on what failed.
+        // The user just scanned this QR code so has context on what failed.
         error!(context, "QR process failed");
         err
     })
@@ -297,46 +190,7 @@ async fn securejoin(context: &Context, qr: &str) -> Result<ChatId, JoinError> {
 
     let invite = QrInvite::try_from(qr_scan)?;
 
-    match context.bob.start_protocol(context, invite.clone()).await? {
-        StartedProtocolVariant::SetupContact => {
-            // for a one-to-one-chat, the chat is already known, return the chat-id,
-            // the verification runs in background
-            let chat_id = ChatId::create_for_contact(context, invite.contact_id())
-                .await
-                .map_err(JoinError::UnknownContact)?;
-            Ok(chat_id)
-        }
-        StartedProtocolVariant::SecureJoin {
-            group_id,
-            group_name,
-        } => {
-            // for a group-join, also create the chat soon and let the verification run in background.
-            // however, the group will become usable only when the protocol has finished.
-            let contact_id = invite.contact_id();
-            let chat_id = if let Some((chat_id, _protected, _blocked)) =
-                chat::get_chat_id_by_grpid(context, &group_id).await?
-            {
-                chat_id.unblock(context).await?;
-                chat_id
-            } else {
-                ChatId::create_multiuser_record(
-                    context,
-                    Chattype::Group,
-                    group_id,
-                    group_name,
-                    Blocked::Not,
-                    ProtectionStatus::Unprotected, // protection is added later as needed
-                )
-                .await?
-            };
-            if !is_contact_in_chat(context, chat_id, contact_id).await? {
-                chat::add_to_chat_contacts_table(context, chat_id, contact_id).await?;
-            }
-            let msg = stock_str::secure_join_started(context, contact_id).await;
-            chat::add_info_msg(context, chat_id, msg, time()).await?;
-            Ok(chat_id)
-        }
-    }
+    bob::start_protocol(context, invite).await
 }
 
 /// Error when failing to send a protocol handshake message.
@@ -351,7 +205,7 @@ pub struct SendMsgError(#[from] anyhow::Error);
 /// Bob's handshake messages are sent in `BobState::send_handshake_message()`.
 async fn send_alice_handshake_msg(
     context: &Context,
-    contact_id: u32,
+    contact_id: ContactId,
     step: &str,
     fingerprint: Option<Fingerprint>,
 ) -> Result<(), SendMsgError> {
@@ -379,7 +233,7 @@ async fn send_alice_handshake_msg(
 }
 
 /// Get an unblocked chat that can be used for info messages.
-async fn info_chat_id(context: &Context, contact_id: u32) -> Result<ChatId> {
+async fn info_chat_id(context: &Context, contact_id: ContactId) -> Result<ChatId> {
     let chat_id_blocked = ChatIdBlocked::get_for_contact(context, contact_id, Blocked::Not).await?;
     Ok(chat_id_blocked.id)
 }
@@ -387,7 +241,7 @@ async fn info_chat_id(context: &Context, contact_id: u32) -> Result<ChatId> {
 async fn fingerprint_equals_sender(
     context: &Context,
     fingerprint: &Fingerprint,
-    contact_id: u32,
+    contact_id: ContactId,
 ) -> Result<bool, Error> {
     let contact = Contact::load_from_db(context, contact_id).await?;
     let peerstate = match Peerstate::from_addr(context, contact.get_addr()).await {
@@ -441,9 +295,8 @@ pub(crate) enum HandshakeMessage {
 
 /// Handle incoming secure-join handshake.
 ///
-/// This function will update the securejoin state in [`InnerContext::bob`] and also
-/// terminate the ongoing process using [`Context::stop_ongoing`] as required by the
-/// protocol.
+/// This function will update the securejoin state in the database as the protocol
+/// progresses.
 ///
 /// A message which results in [`Err`] will be hidden from the user but not deleted, it may
 /// be a valid message for something else we are not aware off.  E.g. it could be part of a
@@ -451,13 +304,11 @@ pub(crate) enum HandshakeMessage {
 ///
 /// When `handle_securejoin_handshake()` is called, the message is not yet filed in the
 /// database; this is done by `receive_imf()` later on as needed.
-///
-/// [`InnerContext::bob`]: crate::context::InnerContext::bob
 #[allow(clippy::indexing_slicing)]
 pub(crate) async fn handle_securejoin_handshake(
     context: &Context,
     mime_message: &MimeMessage,
-    contact_id: u32,
+    contact_id: ContactId,
 ) -> Result<HandshakeMessage> {
     if contact_id <= DC_CONTACT_ID_LAST_SPECIAL {
         return Err(Error::msg("Can not be called with special contact ID"));
@@ -520,38 +371,7 @@ pub(crate) async fn handle_securejoin_handshake(
             ====             Bob - the joiner's side             =====
             ====   Step 4 in "Setup verified contact" protocol   =====
             ========================================================*/
-            match context.bob.state(context).await {
-                Some(mut bobstate) => match bobstate.handle_message(context, mime_message).await {
-                    Some(BobHandshakeStage::Terminated(why)) => {
-                        could_not_establish_secure_connection(
-                            context,
-                            contact_id,
-                            bobstate.chat_id(context).await?,
-                            why,
-                        )
-                        .await?;
-                        Ok(HandshakeMessage::Done)
-                    }
-                    Some(_stage) => {
-                        if join_vg {
-                            // the message reads "Alice replied, waiting for being added to the group…";
-                            // show it only on secure-join and not on setup-contact therefore.
-                            let msg = stock_str::secure_join_replies(context, contact_id).await;
-                            chat::add_info_msg(
-                                context,
-                                bobstate.chat_id(context).await?,
-                                msg,
-                                time(),
-                            )
-                            .await?;
-                        }
-                        joiner_progress!(context, bobstate.invite().contact_id(), 400);
-                        Ok(HandshakeMessage::Done)
-                    }
-                    None => Ok(HandshakeMessage::Ignore),
-                },
-                None => Ok(HandshakeMessage::Ignore),
-            }
+            bob::handle_auth_required(context, mime_message).await
         }
         "vg-request-with-auth" | "vc-request-with-auth" => {
             /*==========================================================
@@ -682,44 +502,14 @@ pub(crate) async fn handle_securejoin_handshake(
             ====             Bob - the joiner's side             ====
             ====   Step 7 in "Setup verified contact" protocol   ====
             =======================================================*/
-            info!(context, "matched vc-contact-confirm step");
-            let retval = if join_vg {
-                HandshakeMessage::Propagate
-            } else {
-                HandshakeMessage::Ignore
-            };
-            match context.bob.state(context).await {
-                Some(mut bobstate) => match bobstate.handle_message(context, mime_message).await {
-                    Some(BobHandshakeStage::Terminated(why)) => {
-                        could_not_establish_secure_connection(
-                            context,
-                            contact_id,
-                            bobstate.chat_id(context).await?,
-                            why,
-                        )
-                        .await?;
-                        Ok(HandshakeMessage::Done)
-                    }
-                    Some(BobHandshakeStage::Completed) => {
-                        // Can only be BobHandshakeStage::Completed
-                        secure_connection_established(
-                            context,
-                            contact_id,
-                            bobstate.chat_id(context).await?,
-                        )
-                        .await?;
-                        Ok(retval)
-                    }
-                    Some(_) => {
-                        warn!(
-                            context,
-                            "Impossible state returned from handling handshake message"
-                        );
-                        Ok(retval)
-                    }
-                    None => Ok(retval),
+            match BobState::from_db(&context.sql).await? {
+                Some(bobstate) => {
+                    bob::handle_contact_confirm(context, bobstate, mime_message).await
+                }
+                None => match join_vg {
+                    true => Ok(HandshakeMessage::Propagate),
+                    false => Ok(HandshakeMessage::Ignore),
                 },
-                None => Ok(retval),
             }
         }
         "vg-member-added-received" | "vc-contact-confirm-received" => {
@@ -741,7 +531,7 @@ pub(crate) async fn handle_securejoin_handshake(
                         .get_header(HeaderDef::SecureJoinGroup)
                         .map(|s| s.as_str())
                         .unwrap_or_else(|| "");
-                    if let Err(err) = chat::get_chat_id_by_grpid(context, &field_grpid).await {
+                    if let Err(err) = chat::get_chat_id_by_grpid(context, field_grpid).await {
                         warn!(context, "Failed to lookup chat_id from grpid: {}", err);
                         return Err(
                             err.context(format!("Chat for group {} not found", &field_grpid))
@@ -781,7 +571,7 @@ pub(crate) async fn handle_securejoin_handshake(
 pub(crate) async fn observe_securejoin_on_other_device(
     context: &Context,
     mime_message: &MimeMessage,
-    contact_id: u32,
+    contact_id: ContactId,
 ) -> Result<HandshakeMessage> {
     if contact_id <= DC_CONTACT_ID_LAST_SPECIAL {
         return Err(Error::msg("Can not be called with special contact ID"));
@@ -846,32 +636,24 @@ pub(crate) async fn observe_securejoin_on_other_device(
 
 async fn secure_connection_established(
     context: &Context,
-    contact_id: u32,
+    contact_id: ContactId,
     chat_id: ChatId,
 ) -> Result<(), Error> {
     let contact = Contact::get_by_id(context, contact_id).await?;
-    let msg = stock_str::contact_verified(context, contact.get_name_n_addr()).await;
-    chat::add_info_msg(context, chat_id, msg, time()).await?;
+    let msg = stock_str::contact_verified(context, &contact).await;
+    chat::add_info_msg(context, chat_id, &msg, time()).await?;
     context.emit_event(EventType::ChatModified(chat_id));
     Ok(())
 }
 
 async fn could_not_establish_secure_connection(
     context: &Context,
-    contact_id: u32,
+    contact_id: ContactId,
     chat_id: ChatId,
     details: &str,
 ) -> Result<(), Error> {
-    let contact = Contact::get_by_id(context, contact_id).await;
-    let msg = stock_str::contact_not_verified(
-        context,
-        if let Ok(ref contact) = contact {
-            contact.get_addr()
-        } else {
-            "?"
-        },
-    )
-    .await;
+    let contact = Contact::get_by_id(context, contact_id).await?;
+    let msg = stock_str::contact_not_verified(context, &contact).await;
     chat::add_info_msg(context, chat_id, &msg, time()).await?;
     warn!(
         context,
@@ -936,46 +718,43 @@ fn encrypted_and_signed(
 mod tests {
     use super::*;
 
-    use async_std::prelude::*;
-
     use crate::chat;
     use crate::chat::ProtectionStatus;
     use crate::chatlist::Chatlist;
     use crate::constants::Chattype;
-    use crate::events::Event;
     use crate::peerstate::Peerstate;
-    use crate::test_utils::TestContext;
-    use std::time::Duration;
+    use crate::test_utils::{TestContext, TestContextManager};
 
     #[async_std::test]
-    async fn test_setup_contact() -> Result<()> {
-        let alice = TestContext::new_alice().await;
-        let bob = TestContext::new_bob().await;
-        assert_eq!(Chatlist::try_load(&alice, 0, None, None).await?.len(), 0);
-        assert_eq!(Chatlist::try_load(&bob, 0, None, None).await?.len(), 0);
-
-        // Setup JoinerProgress sinks.
-        let (joiner_progress_tx, joiner_progress_rx) = async_std::channel::bounded(100);
-        bob.add_event_sink(move |event: Event| {
-            let joiner_progress_tx = joiner_progress_tx.clone();
-            async move {
-                if let EventType::SecurejoinJoinerProgress { .. } = event.typ {
-                    joiner_progress_tx.try_send(event).unwrap();
-                }
-            }
-        })
-        .await;
+    async fn test_setup_contact() {
+        let mut tcm = TestContextManager::new().await;
+        let alice = tcm.alice().await;
+        let bob = tcm.bob().await;
+        assert_eq!(
+            Chatlist::try_load(&alice, 0, None, None)
+                .await
+                .unwrap()
+                .len(),
+            0
+        );
+        assert_eq!(
+            Chatlist::try_load(&bob, 0, None, None).await.unwrap().len(),
+            0
+        );
 
         // Step 1: Generate QR-code, ChatId(0) indicates setup-contact
-        let qr = dc_get_securejoin_qr(&alice.ctx, None).await?;
+        let qr = dc_get_securejoin_qr(&alice.ctx, None).await.unwrap();
 
         // Step 2: Bob scans QR-code, sends vc-request
-        dc_join_securejoin(&bob.ctx, &qr).await?;
-        assert_eq!(Chatlist::try_load(&bob, 0, None, None).await?.len(), 1);
+        dc_join_securejoin(&bob.ctx, &qr).await.unwrap();
+        assert_eq!(
+            Chatlist::try_load(&bob, 0, None, None).await.unwrap().len(),
+            1
+        );
 
         let sent = bob.pop_sent_msg().await;
         assert!(!bob.ctx.has_ongoing().await);
-        assert_eq!(sent.recipient(), "alice@example.com".parse().unwrap());
+        assert_eq!(sent.recipient(), "alice@example.org".parse().unwrap());
         let msg = alice.parse_msg(&sent).await;
         assert!(!msg.was_encrypted());
         assert_eq!(msg.get_header(HeaderDef::SecureJoin).unwrap(), "vc-request");
@@ -983,7 +762,13 @@ mod tests {
 
         // Step 3: Alice receives vc-request, sends vc-auth-required
         alice.recv_msg(&sent).await;
-        assert_eq!(Chatlist::try_load(&alice, 0, None, None).await?.len(), 1);
+        assert_eq!(
+            Chatlist::try_load(&alice, 0, None, None)
+                .await
+                .unwrap()
+                .len(),
+            1
+        );
 
         let sent = alice.pop_sent_msg().await;
         let msg = bob.parse_msg(&sent).await;
@@ -997,28 +782,24 @@ mod tests {
         bob.recv_msg(&sent).await;
 
         // Check Bob emitted the JoinerProgress event.
-        {
-            let evt = joiner_progress_rx
-                .recv()
-                .timeout(Duration::from_secs(10))
-                .await
-                .expect("timeout waiting for JoinerProgress event")
-                .expect("missing JoinerProgress event");
-            match evt.typ {
-                EventType::SecurejoinJoinerProgress {
-                    contact_id,
-                    progress,
-                } => {
-                    let alice_contact_id =
-                        Contact::lookup_id_by_addr(&bob.ctx, "alice@example.com", Origin::Unknown)
-                            .await
-                            .expect("Error looking up contact")
-                            .expect("Contact not found");
-                    assert_eq!(contact_id, alice_contact_id);
-                    assert_eq!(progress, 400);
-                }
-                _ => panic!("Wrong event type"),
+        let event = bob
+            .evtracker
+            .get_matching(|evt| matches!(evt, EventType::SecurejoinJoinerProgress { .. }))
+            .await;
+        match event {
+            EventType::SecurejoinJoinerProgress {
+                contact_id,
+                progress,
+            } => {
+                let alice_contact_id =
+                    Contact::lookup_id_by_addr(&bob.ctx, "alice@example.org", Origin::Unknown)
+                        .await
+                        .expect("Error looking up contact")
+                        .expect("Contact not found");
+                assert_eq!(contact_id, alice_contact_id);
+                assert_eq!(progress, 400);
             }
+            _ => unreachable!(),
         }
 
         // Check Bob sent the right message.
@@ -1049,21 +830,30 @@ mod tests {
             .await
             .unwrap();
         assert_eq!(
-            contact_bob.is_verified(&alice.ctx).await?,
+            contact_bob.is_verified(&alice.ctx).await.unwrap(),
             VerifiedStatus::Unverified
         );
 
         // Step 5+6: Alice receives vc-request-with-auth, sends vc-contact-confirm
         alice.recv_msg(&sent).await;
         assert_eq!(
-            contact_bob.is_verified(&alice.ctx).await?,
+            contact_bob.is_verified(&alice.ctx).await.unwrap(),
             VerifiedStatus::BidirectVerified
         );
 
         // exactly one one-to-one chat should be visible for both now
         // (check this before calling alice.create_chat() explicitly below)
-        assert_eq!(Chatlist::try_load(&alice, 0, None, None).await?.len(), 1);
-        assert_eq!(Chatlist::try_load(&bob, 0, None, None).await?.len(), 1);
+        assert_eq!(
+            Chatlist::try_load(&alice, 0, None, None)
+                .await
+                .unwrap()
+                .len(),
+            1
+        );
+        assert_eq!(
+            Chatlist::try_load(&bob, 0, None, None).await.unwrap().len(),
+            1
+        );
 
         // Check Alice got the verified message in her 1:1 chat.
         {
@@ -1095,7 +885,7 @@ mod tests {
 
         // Bob should not yet have Alice verified
         let contact_alice_id =
-            Contact::lookup_id_by_addr(&bob.ctx, "alice@example.com", Origin::Unknown)
+            Contact::lookup_id_by_addr(&bob.ctx, "alice@example.org", Origin::Unknown)
                 .await
                 .expect("Error looking up contact")
                 .expect("Contact not found");
@@ -1103,14 +893,14 @@ mod tests {
             .await
             .unwrap();
         assert_eq!(
-            contact_bob.is_verified(&bob.ctx).await?,
+            contact_bob.is_verified(&bob.ctx).await.unwrap(),
             VerifiedStatus::Unverified
         );
 
         // Step 7: Bob receives vc-contact-confirm, sends vc-contact-confirm-received
         bob.recv_msg(&sent).await;
         assert_eq!(
-            contact_alice.is_verified(&bob.ctx).await?,
+            contact_alice.is_verified(&bob.ctx).await.unwrap(),
             VerifiedStatus::BidirectVerified
         );
 
@@ -1130,7 +920,7 @@ mod tests {
             let msg = Message::load_from_db(&bob.ctx, msg_id).await.unwrap();
             assert!(msg.is_info());
             let text = msg.get_text().unwrap();
-            assert!(text.contains("alice@example.com verified"));
+            assert!(text.contains("alice@example.org verified"));
         }
 
         // Check Bob sent the final message
@@ -1141,7 +931,6 @@ mod tests {
             msg.get_header(HeaderDef::SecureJoin).unwrap(),
             "vc-contact-confirm-received"
         );
-        Ok(())
     }
 
     #[async_std::test]
@@ -1153,25 +942,14 @@ mod tests {
 
     #[async_std::test]
     async fn test_setup_contact_bob_knows_alice() -> Result<()> {
-        let alice = TestContext::new_alice().await;
-        let bob = TestContext::new_bob().await;
-
-        // Setup JoinerProgress sinks.
-        let (joiner_progress_tx, joiner_progress_rx) = async_std::channel::bounded(100);
-        bob.add_event_sink(move |event: Event| {
-            let joiner_progress_tx = joiner_progress_tx.clone();
-            async move {
-                if let EventType::SecurejoinJoinerProgress { .. } = event.typ {
-                    joiner_progress_tx.try_send(event).unwrap();
-                }
-            }
-        })
-        .await;
+        let mut tcm = TestContextManager::new().await;
+        let alice = tcm.alice().await;
+        let bob = tcm.bob().await;
 
         // Ensure Bob knows Alice_FP
         let alice_pubkey = SignedPublicKey::load_self(&alice.ctx).await?;
         let peerstate = Peerstate {
-            addr: "alice@example.com".into(),
+            addr: "alice@example.org".into(),
             last_seen: 10,
             last_seen_autocrypt: 10,
             prefer_encrypt: EncryptPreference::Mutual,
@@ -1194,30 +972,25 @@ mod tests {
         dc_join_securejoin(&bob.ctx, &qr).await.unwrap();
 
         // Check Bob emitted the JoinerProgress event.
-        {
-            let evt = joiner_progress_rx
-                .recv()
-                .timeout(Duration::from_secs(10))
-                .await
-                .expect("timeout waiting for JoinerProgress event")
-                .expect("missing JoinerProgress event");
-            match evt.typ {
-                EventType::SecurejoinJoinerProgress {
-                    contact_id,
-                    progress,
-                } => {
-                    let alice_contact_id =
-                        Contact::lookup_id_by_addr(&bob.ctx, "alice@example.com", Origin::Unknown)
-                            .await
-                            .expect("Error looking up contact")
-                            .expect("Contact not found");
-                    assert_eq!(contact_id, alice_contact_id);
-                    assert_eq!(progress, 400);
-                }
-                _ => panic!("Wrong event type"),
+        let event = bob
+            .evtracker
+            .get_matching(|evt| matches!(evt, EventType::SecurejoinJoinerProgress { .. }))
+            .await;
+        match event {
+            EventType::SecurejoinJoinerProgress {
+                contact_id,
+                progress,
+            } => {
+                let alice_contact_id =
+                    Contact::lookup_id_by_addr(&bob.ctx, "alice@example.org", Origin::Unknown)
+                        .await
+                        .expect("Error looking up contact")
+                        .expect("Contact not found");
+                assert_eq!(contact_id, alice_contact_id);
+                assert_eq!(progress, 400);
             }
+            _ => unreachable!(),
         }
-        assert!(!bob.ctx.has_ongoing().await);
 
         // Check Bob sent the right handshake message.
         let sent = bob.pop_sent_msg().await;
@@ -1265,7 +1038,7 @@ mod tests {
 
         // Bob should not yet have Alice verified
         let contact_alice_id =
-            Contact::lookup_id_by_addr(&bob.ctx, "alice@example.com", Origin::Unknown)
+            Contact::lookup_id_by_addr(&bob.ctx, "alice@example.org", Origin::Unknown)
                 .await
                 .expect("Error looking up contact")
                 .expect("Contact not found");
@@ -1294,8 +1067,9 @@ mod tests {
 
     #[async_std::test]
     async fn test_setup_contact_concurrent_calls() -> Result<()> {
-        let alice = TestContext::new_alice().await;
-        let bob = TestContext::new_bob().await;
+        let mut tcm = TestContextManager::new().await;
+        let alice = tcm.alice().await;
+        let bob = tcm.bob().await;
 
         // do a scan that is not working as claire is never responding
         let qr_stale = "OPENPGP4FPR:1234567890123456789012345678901234567890#a=claire%40foo.de&n=&i=12345678901&s=23456789012";
@@ -1317,35 +1091,26 @@ mod tests {
             .pop_sent_msg()
             .await
             .payload()
-            .contains("alice@example.com"));
+            .contains("alice@example.org"));
 
         Ok(())
     }
 
     #[async_std::test]
     async fn test_secure_join() -> Result<()> {
-        let alice = TestContext::new_alice().await;
-        let bob = TestContext::new_bob().await;
+        let mut tcm = TestContextManager::new().await;
+        let alice = tcm.alice().await;
+        let bob = tcm.bob().await;
+
+        // We start with empty chatlists.
         assert_eq!(Chatlist::try_load(&alice, 0, None, None).await?.len(), 0);
         assert_eq!(Chatlist::try_load(&bob, 0, None, None).await?.len(), 0);
 
-        // Setup JoinerProgress sinks.
-        let (joiner_progress_tx, joiner_progress_rx) = async_std::channel::bounded(100);
-        bob.add_event_sink(move |event: Event| {
-            let joiner_progress_tx = joiner_progress_tx.clone();
-            async move {
-                if let EventType::SecurejoinJoinerProgress { .. } = event.typ {
-                    joiner_progress_tx.try_send(event).unwrap();
-                }
-            }
-        })
-        .await;
-
-        let chatid =
+        let alice_chatid =
             chat::create_group_chat(&alice.ctx, ProtectionStatus::Protected, "the chat").await?;
 
         // Step 1: Generate QR-code, secure-join implied by chatid
-        let qr = dc_get_securejoin_qr(&alice.ctx, Some(chatid))
+        let qr = dc_get_securejoin_qr(&alice.ctx, Some(alice_chatid))
             .await
             .unwrap();
 
@@ -1354,7 +1119,7 @@ mod tests {
         assert_eq!(Chatlist::try_load(&bob, 0, None, None).await?.len(), 1);
 
         let sent = bob.pop_sent_msg().await;
-        assert_eq!(sent.recipient(), "alice@example.com".parse().unwrap());
+        assert_eq!(sent.recipient(), "alice@example.org".parse().unwrap());
         let msg = alice.parse_msg(&sent).await;
         assert!(!msg.was_encrypted());
         assert_eq!(msg.get_header(HeaderDef::SecureJoin).unwrap(), "vg-request");
@@ -1376,28 +1141,24 @@ mod tests {
         let sent = bob.pop_sent_msg().await;
 
         // Check Bob emitted the JoinerProgress event.
-        {
-            let evt = joiner_progress_rx
-                .recv()
-                .timeout(Duration::from_secs(10))
-                .await
-                .expect("timeout waiting for JoinerProgress event")
-                .expect("missing JoinerProgress event");
-            match evt.typ {
-                EventType::SecurejoinJoinerProgress {
-                    contact_id,
-                    progress,
-                } => {
-                    let alice_contact_id =
-                        Contact::lookup_id_by_addr(&bob.ctx, "alice@example.com", Origin::Unknown)
-                            .await
-                            .expect("Error looking up contact")
-                            .expect("Contact not found");
-                    assert_eq!(contact_id, alice_contact_id);
-                    assert_eq!(progress, 400);
-                }
-                _ => panic!("Wrong event type"),
+        let event = bob
+            .evtracker
+            .get_matching(|evt| matches!(evt, EventType::SecurejoinJoinerProgress { .. }))
+            .await;
+        match event {
+            EventType::SecurejoinJoinerProgress {
+                contact_id,
+                progress,
+            } => {
+                let alice_contact_id =
+                    Contact::lookup_id_by_addr(&bob.ctx, "alice@example.org", Origin::Unknown)
+                        .await
+                        .expect("Error looking up contact")
+                        .expect("Contact not found");
+                assert_eq!(contact_id, alice_contact_id);
+                assert_eq!(progress, 400);
             }
+            _ => unreachable!(),
         }
 
         // Check Bob sent the right handshake message.
@@ -1440,9 +1201,38 @@ mod tests {
             "vg-member-added"
         );
 
+        {
+            // Now Alice's chat with Bob should still be hidden, the verified message should
+            // appear in the group chat.
+
+            let chat = alice
+                .get_chat(&bob)
+                .await
+                .expect("Alice has no 1:1 chat with bob");
+            assert_eq!(
+                chat.blocked,
+                Blocked::Yes,
+                "Alice's 1:1 chat with Bob is not hidden"
+            );
+            let msg_id = chat::get_chat_msgs(&alice.ctx, alice_chatid, 0x1, None)
+                .await
+                .unwrap()
+                .into_iter()
+                .filter_map(|item| match item {
+                    chat::ChatItem::Message { msg_id } => Some(msg_id),
+                    _ => None,
+                })
+                .min()
+                .expect("No messages in Alice's group chat");
+            let msg = Message::load_from_db(&alice.ctx, msg_id).await.unwrap();
+            assert!(msg.is_info());
+            let text = msg.get_text().unwrap();
+            assert!(text.contains("bob@example.net verified"));
+        }
+
         // Bob should not yet have Alice verified
         let contact_alice_id =
-            Contact::lookup_id_by_addr(&bob.ctx, "alice@example.com", Origin::Unknown)
+            Contact::lookup_id_by_addr(&bob.ctx, "alice@example.org", Origin::Unknown)
                 .await
                 .expect("Error looking up contact")
                 .expect("Contact not found");
@@ -1454,10 +1244,53 @@ mod tests {
 
         // Step 7: Bob receives vg-member-added, sends vg-member-added-received
         bob.recv_msg(&sent).await;
-        assert_eq!(
-            contact_alice.is_verified(&bob.ctx).await?,
-            VerifiedStatus::BidirectVerified
-        );
+        {
+            // Bob has Alice verified, message shows up in the group chat.
+            assert_eq!(
+                contact_alice.is_verified(&bob.ctx).await?,
+                VerifiedStatus::BidirectVerified
+            );
+            let chat = bob
+                .get_chat(&alice)
+                .await
+                .expect("Bob has no 1:1 chat with Alice");
+            assert_eq!(
+                chat.blocked,
+                Blocked::Yes,
+                "Bob's 1:1 chat with Alice is not hidden"
+            );
+            for item in chat::get_chat_msgs(&bob.ctx, bob_chatid, 0x1, None)
+                .await
+                .unwrap()
+            {
+                if let chat::ChatItem::Message { msg_id } = item {
+                    let msg = Message::load_from_db(&bob.ctx, msg_id).await.unwrap();
+                    let text = msg.get_text().unwrap();
+                    println!("msg {} text: {}", msg_id, text);
+                }
+            }
+            let mut msg_iter = chat::get_chat_msgs(&bob.ctx, bob_chatid, 0x1, None)
+                .await
+                .unwrap()
+                .into_iter();
+            loop {
+                match msg_iter.next() {
+                    Some(chat::ChatItem::Message { msg_id }) => {
+                        let msg = Message::load_from_db(&bob.ctx, msg_id).await.unwrap();
+                        let text = msg.get_text().unwrap();
+                        match text.contains("alice@example.org verified") {
+                            true => {
+                                assert!(msg.is_info());
+                                break;
+                            }
+                            false => continue,
+                        }
+                    }
+                    Some(_) => continue,
+                    None => panic!("Verified message not found in Bob's group chat"),
+                }
+            }
+        }
 
         let sent = bob.pop_sent_msg().await;
         let msg = alice.parse_msg(&sent).await;

src/securejoin/bob.rs

@@ -0,0 +1,257 @@
+//! Bob's side of SecureJoin handling.
+//!
+//! This are some helper functions around [`BobState`] which augment the state changes with
+//! the required user interactions.
+
+use anyhow::Result;
+
+use crate::chat::{is_contact_in_chat, ChatId, ProtectionStatus};
+use crate::constants::{Blocked, Chattype};
+use crate::contact::Contact;
+use crate::context::Context;
+use crate::dc_tools::time;
+use crate::events::EventType;
+use crate::mimeparser::MimeMessage;
+use crate::{chat, stock_str};
+
+use super::bobstate::{BobHandshakeStage, BobState};
+use super::qrinvite::QrInvite;
+use super::{HandshakeMessage, JoinError};
+
+/// Starts the securejoin protocol with the QR `invite`.
+///
+/// This will try to start the securejoin protocol for the given QR `invite`.  If it
+/// succeeded the protocol state will be tracked in `self`.
+///
+/// This function takes care of handling multiple concurrent joins and handling errors while
+/// starting the protocol.
+///
+/// # Returns
+///
+/// The [`ChatId`] of the created chat is returned, for a SetupContact QR this is the 1:1
+/// chat with Alice, for a SecureJoin QR this is the group chat.
+pub(super) async fn start_protocol(
+    context: &Context,
+    invite: QrInvite,
+) -> Result<ChatId, JoinError> {
+    // A 1:1 chat is needed to send messages to Alice.  When joining a group this chat is
+    // hidden, if a user starts sending messages in it it will be unhidden in
+    // dc_receive_imf.
+    let hidden = match invite {
+        QrInvite::Contact { .. } => Blocked::Not,
+        QrInvite::Group { .. } => Blocked::Yes,
+    };
+    let chat_id = ChatId::create_for_contact_with_blocked(context, invite.contact_id(), hidden)
+        .await
+        .map_err(JoinError::UnknownContact)?;
+
+    // Now start the protocol and initialise the state
+    let (state, stage, aborted_states) =
+        BobState::start_protocol(context, invite.clone(), chat_id).await?;
+    for state in aborted_states {
+        error!(context, "Aborting previously unfinished QR Join process.");
+        state.notify_aborted(context, "new QR scanned").await?;
+        state.emit_progress(context, JoinerProgress::Error);
+    }
+    if matches!(stage, BobHandshakeStage::RequestWithAuthSent) {
+        state.emit_progress(context, JoinerProgress::RequestWithAuthSent);
+    }
+    match invite {
+        QrInvite::Group { .. } => {
+            // For a secure-join we need to create the group and add the contact.  The group will
+            // only become usable once the protocol is finished.
+            // TODO: how does this group become usable?
+            let group_chat_id = state.joining_chat_id(context).await?;
+            if !is_contact_in_chat(context, group_chat_id, invite.contact_id()).await? {
+                chat::add_to_chat_contacts_table(context, group_chat_id, invite.contact_id())
+                    .await?;
+            }
+            let msg = stock_str::secure_join_started(context, invite.contact_id()).await;
+            chat::add_info_msg(context, group_chat_id, &msg, time()).await?;
+            Ok(group_chat_id)
+        }
+        QrInvite::Contact { .. } => {
+            // For setup-contact the BobState already ensured the 1:1 chat exists because it
+            // uses it to send the handshake messages.
+            Ok(state.alice_chat())
+        }
+    }
+}
+
+/// Handles `vc-auth-required` and `vg-auth-required` handshake messages.
+///
+/// # Bob - the joiner's side
+/// ## Step 4 in the "Setup Contact protocol"
+pub(super) async fn handle_auth_required(
+    context: &Context,
+    message: &MimeMessage,
+) -> Result<HandshakeMessage> {
+    match BobState::from_db(&context.sql).await? {
+        Some(mut bobstate) => match bobstate.handle_message(context, message).await? {
+            Some(BobHandshakeStage::Terminated(why)) => {
+                bobstate.notify_aborted(context, why).await?;
+                Ok(HandshakeMessage::Done)
+            }
+            Some(_stage) => {
+                if bobstate.is_join_group() {
+                    // The message reads "Alice replied, waiting to be added to the group…",
+                    // so only show it on secure-join and not on setup-contact.
+                    let contact_id = bobstate.invite().contact_id();
+                    let msg = stock_str::secure_join_replies(context, contact_id).await;
+                    let chat_id = bobstate.joining_chat_id(context).await?;
+                    chat::add_info_msg(context, chat_id, &msg, time()).await?;
+                }
+                bobstate.emit_progress(context, JoinerProgress::RequestWithAuthSent);
+                Ok(HandshakeMessage::Done)
+            }
+            None => Ok(HandshakeMessage::Ignore),
+        },
+        None => Ok(HandshakeMessage::Ignore),
+    }
+}
+
+/// Handles `vc-contact-confirm` and `vg-member-added` handshake messages.
+///
+/// # Bob - the joiner's side
+/// ## Step 4 in the "Setup Contact protocol"
+pub(super) async fn handle_contact_confirm(
+    context: &Context,
+    mut bobstate: BobState,
+    message: &MimeMessage,
+) -> Result<HandshakeMessage> {
+    let retval = if bobstate.is_join_group() {
+        HandshakeMessage::Propagate
+    } else {
+        HandshakeMessage::Ignore
+    };
+    match bobstate.handle_message(context, message).await? {
+        Some(BobHandshakeStage::Terminated(why)) => {
+            bobstate.notify_aborted(context, why).await?;
+            Ok(HandshakeMessage::Done)
+        }
+        Some(BobHandshakeStage::Completed) => {
+            // Note this goes to the 1:1 chat, as when joining a group we implicitly also
+            // verify both contacts (this could be a bug/security issue, see
+            // e.g. https://github.com/deltachat/deltachat-core-rust/issues/1177).
+            bobstate.notify_peer_verified(context).await?;
+            Ok(retval)
+        }
+        Some(_) => {
+            warn!(
+                context,
+                "Impossible state returned from handling handshake message"
+            );
+            Ok(retval)
+        }
+        None => Ok(retval),
+    }
+}
+
+/// Private implementations for user interactions about this [`BobState`].
+impl BobState {
+    fn is_join_group(&self) -> bool {
+        match self.invite() {
+            QrInvite::Contact { .. } => false,
+            QrInvite::Group { .. } => true,
+        }
+    }
+
+    fn emit_progress(&self, context: &Context, progress: JoinerProgress) {
+        let contact_id = self.invite().contact_id();
+        context.emit_event(EventType::SecurejoinJoinerProgress {
+            contact_id,
+            progress: progress.into(),
+        });
+    }
+
+    /// Returns the [`ChatId`] of the chat being joined.
+    ///
+    /// This is the chat in which you want to notify the user as well.
+    ///
+    /// When joining a group this is the [`ChatId`] of the group chat, when verifying a
+    /// contact this is the [`ChatId`] of the 1:1 chat.  The 1:1 chat is assumed to exist
+    /// because a [`BobState`] can not exist without, the group chat will be created if it
+    /// does not yet exist.
+    async fn joining_chat_id(&self, context: &Context) -> Result<ChatId> {
+        match self.invite() {
+            QrInvite::Contact { .. } => Ok(self.alice_chat()),
+            QrInvite::Group {
+                ref grpid,
+                ref name,
+                ..
+            } => {
+                let group_chat_id = match chat::get_chat_id_by_grpid(context, grpid).await? {
+                    Some((chat_id, _protected, _blocked)) => {
+                        chat_id.unblock(context).await?;
+                        chat_id
+                    }
+                    None => {
+                        ChatId::create_multiuser_record(
+                            context,
+                            Chattype::Group,
+                            grpid,
+                            name,
+                            Blocked::Not,
+                            ProtectionStatus::Unprotected, // protection is added later as needed
+                            None,
+                        )
+                        .await?
+                    }
+                };
+                Ok(group_chat_id)
+            }
+        }
+    }
+
+    /// Notifies the user that the SecureJoin was aborted.
+    ///
+    /// This creates an info message in the chat being joined.
+    async fn notify_aborted(&self, context: &Context, why: &str) -> Result<()> {
+        let contact = Contact::get_by_id(context, self.invite().contact_id()).await?;
+        let msg = stock_str::contact_not_verified(context, &contact).await;
+        let chat_id = self.joining_chat_id(context).await?;
+        chat::add_info_msg(context, chat_id, &msg, time()).await?;
+        warn!(
+            context,
+            "StockMessage::ContactNotVerified posted to joining chat ({})", why
+        );
+        Ok(())
+    }
+
+    /// Notifies the user that the SecureJoin peer is verified.
+    ///
+    /// This creates an info message in the chat being joined.
+    async fn notify_peer_verified(&self, context: &Context) -> Result<()> {
+        let contact = Contact::get_by_id(context, self.invite().contact_id()).await?;
+        let msg = stock_str::contact_verified(context, &contact).await;
+        let chat_id = self.joining_chat_id(context).await?;
+        chat::add_info_msg(context, chat_id, &msg, time()).await?;
+        context.emit_event(EventType::ChatModified(chat_id));
+        Ok(())
+    }
+}
+
+/// Progress updates for [`EventType::SecurejoinJoinerProgress`].
+///
+/// This has an `From<JoinerProgress> for usize` impl yielding numbers between 0 and a 1000
+/// which can be shown as a progress bar.
+enum JoinerProgress {
+    /// An error occurred.
+    Error,
+    /// vg-vc-request-with-auth sent.
+    ///
+    /// Typically shows as "alice@addr verified, introducing myself."
+    RequestWithAuthSent,
+    // /// Completed securejoin.
+    // Succeeded,
+}
+
+impl From<JoinerProgress> for usize {
+    fn from(progress: JoinerProgress) -> Self {
+        match progress {
+            JoinerProgress::Error => 0,
+            JoinerProgress::RequestWithAuthSent => 400,
+            // JoinerProgress::Succeeded => 1000,
+        }
+    }
+}

src/securejoin/bobstate.rs

@@ -5,22 +5,21 @@
 //! provides all the information to its driver so it can perform the correct interactions.
 //!
 //! The [`BobState`] is only directly used to initially create it when starting the
-//! protocol.  Afterwards it must be stored in a mutex and the [`BobStateHandle`] should be
-//! used to work with the state.
+//! protocol.
 
-use anyhow::{bail, Error, Result};
-use async_std::sync::MutexGuard;
+use anyhow::{Error, Result};
+use rusqlite::Connection;
 
 use crate::chat::{self, ChatId};
-use crate::constants::{Blocked, Viewtype};
 use crate::contact::{Contact, Origin};
 use crate::context::Context;
 use crate::events::EventType;
 use crate::headerdef::HeaderDef;
 use crate::key::{DcKey, SignedPublicKey};
-use crate::message::Message;
+use crate::message::{Message, Viewtype};
 use crate::mimeparser::{MimeMessage, SystemMessage};
 use crate::param::Param;
+use crate::sql::Sql;
 
 use super::qrinvite::QrInvite;
 use super::{
@@ -46,123 +45,14 @@ pub enum BobHandshakeStage {
     Terminated(&'static str),
 }
 
-/// A handle to work with the [`BobState`] of Bob's securejoin protocol.
+/// The securejoin state kept while Bob is joining.
 ///
-/// This handle can only be created for when an underlying [`BobState`] exists.  It keeps
-/// open a lock which guarantees unique access to the state and this struct must be dropped
-/// to return the lock.
-pub struct BobStateHandle<'a> {
-    guard: MutexGuard<'a, Option<BobState>>,
-    bobstate: BobState,
-    clear_state_on_drop: bool,
-}
-
-impl<'a> BobStateHandle<'a> {
-    /// Creates a new instance, upholding the guarantee that [`BobState`] must exist.
-    pub fn from_guard(mut guard: MutexGuard<'a, Option<BobState>>) -> Option<Self> {
-        guard.take().map(|bobstate| Self {
-            guard,
-            bobstate,
-            clear_state_on_drop: false,
-        })
-    }
-
-    /// Returns the [`ChatId`] of the group chat to join or the 1:1 chat with Alice.
-    pub async fn chat_id(&self, context: &Context) -> Result<ChatId> {
-        match self.bobstate.invite {
-            QrInvite::Group { ref grpid, .. } => {
-                if let Some((chat_id, _, _)) = chat::get_chat_id_by_grpid(context, &grpid).await? {
-                    Ok(chat_id)
-                } else {
-                    bail!("chat not found")
-                }
-            }
-            QrInvite::Contact { .. } => Ok(self.bobstate.chat_id),
-        }
-    }
-
-    /// Returns a reference to the [`QrInvite`] of the joiner process.
-    pub fn invite(&self) -> &QrInvite {
-        &self.bobstate.invite
-    }
-
-    /// Handles the given message for the securejoin handshake for Bob.
-    ///
-    /// This proxies to [`BobState::handle_message`] and makes sure to clear the state when
-    /// the protocol state is terminal.  It returns `Some` if the message successfully
-    /// advanced the state of the protocol state machine, `None` otherwise.
-    pub async fn handle_message(
-        &mut self,
-        context: &Context,
-        mime_message: &MimeMessage,
-    ) -> Option<BobHandshakeStage> {
-        info!(context, "Handling securejoin message for BobStateHandle");
-        match self.bobstate.handle_message(context, mime_message).await {
-            Ok(Some(stage)) => {
-                if matches!(
-                    stage,
-                    BobHandshakeStage::Completed | BobHandshakeStage::Terminated(_)
-                ) {
-                    self.finish_protocol(context).await;
-                }
-                Some(stage)
-            }
-            Ok(None) => None,
-            Err(err) => {
-                warn!(
-                    context,
-                    "Error handling handshake message, aborting handshake: {}", err
-                );
-                self.finish_protocol(context).await;
-                None
-            }
-        }
-    }
-
-    /// Marks the bob handshake as finished.
-    ///
-    /// This will clear the state on [`InnerContext::bob`] once this handle is dropped,
-    /// allowing a new handshake to be started from [`Bob`].
-    ///
-    /// Note that the state is only cleared on Drop since otherwise the invariant that the
-    /// state is always consistent is violated.  However the "ongoing" process is released
-    /// here a little bit earlier as this requires access to the Context, which we do not
-    /// have on Drop (Drop can not run asynchronous code).  Stopping the "ongoing" process
-    /// will release [`securejoin`](super::securejoin) which in turn will finally free the
-    /// ongoing process using [`Context::free_ongoing`].
-    ///
-    /// [`InnerContext::bob`]: crate::context::InnerContext::bob
-    /// [`Bob`]: super::Bob
-    async fn finish_protocol(&mut self, context: &Context) {
-        info!(context, "Finishing securejoin handshake protocol for Bob");
-        self.clear_state_on_drop = true;
-        if let QrInvite::Group { .. } = self.bobstate.invite {
-            context.stop_ongoing().await;
-        }
-    }
-}
-
-impl<'a> Drop for BobStateHandle<'a> {
-    fn drop(&mut self) {
-        if self.clear_state_on_drop {
-            // The Option should already be empty because we take it out in the ctor,
-            // however the typesystem doesn't guarantee this so do it again anyway.
-            self.guard.take();
-        } else {
-            // Make sure to put back the BobState into the Option of the Mutex, it was taken
-            // out by the constructor.
-            self.guard.replace(self.bobstate.clone());
-        }
-    }
-}
-
-/// The securejoin state kept in-memory while Bob is joining.
+/// This is stored in the database and loaded from there using [`BobState::from_db`].  To
+/// create a new one use [`BobState::start_protocol`].
 ///
-/// This is currently stored in [`Bob`] which is stored on the [`Context`], thus Bob can
-/// only run one securejoin joiner protocol at a time.
-///
-/// This purposefully has nothing optional, the state is always fully valid.  See
-/// [`Bob::state`] to get access to this state.
+/// This purposefully has nothing optional, the state is always fully valid.  However once a
+/// terminal state is reached in [`BobState::next`] the entry in the database will already
+/// have been deleted.
 ///
 /// # Conducting the securejoin handshake
 ///
@@ -177,6 +67,8 @@ impl<'a> Drop for BobStateHandle<'a> {
 /// [`Bob::state`]: super::Bob::state
 #[derive(Debug, Clone)]
 pub struct BobState {
+    /// Database primary key.
+    id: i64,
     /// The QR Invite code.
     invite: QrInvite,
     /// The next expected message from Alice.
@@ -188,39 +80,120 @@ pub struct BobState {
 impl BobState {
     /// Starts the securejoin protocol and creates a new [`BobState`].
     ///
+    /// The `chat_id` needs to be the ID of the 1:1 chat with Alice, this chat will be used
+    /// to exchange the SecureJoin handshake messages as well as for showing error messages.
+    ///
     /// # Bob - the joiner's side
     /// ## Step 2 in the "Setup Contact protocol", section 2.1 of countermitm 0.10.0
+    ///
+    /// This currently aborts any other securejoin process if any did not yet complete.  The
+    /// ChatIds of the relevant 1:1 chat of any aborted handshakes are returned so that you
+    /// can report the aboreted handshake in the chat.  (Yes, there can only ever be one
+    /// ChatId in that Vec, the database doesn't care though.)
     pub async fn start_protocol(
         context: &Context,
         invite: QrInvite,
-    ) -> Result<(Self, BobHandshakeStage), JoinError> {
-        let chat_id =
-            ChatId::create_for_contact_with_blocked(context, invite.contact_id(), Blocked::Yes)
-                .await
-                .map_err(JoinError::UnknownContact)?;
-        if fingerprint_equals_sender(context, invite.fingerprint(), invite.contact_id()).await? {
-            // The scanned fingerprint matches Alice's key, we can proceed to step 4b.
-            info!(context, "Taking securejoin protocol shortcut");
-            let state = Self {
-                invite,
-                next: SecureJoinStep::ContactConfirm,
-                chat_id,
+        chat_id: ChatId,
+    ) -> Result<(Self, BobHandshakeStage, Vec<Self>), JoinError> {
+        let (stage, next) =
+            if fingerprint_equals_sender(context, invite.fingerprint(), invite.contact_id()).await?
+            {
+                // The scanned fingerprint matches Alice's key, we can proceed to step 4b.
+                info!(context, "Taking securejoin protocol shortcut");
+                send_handshake_message(context, &invite, chat_id, BobHandshakeMsg::RequestWithAuth)
+                    .await?;
+                (
+                    BobHandshakeStage::RequestWithAuthSent,
+                    SecureJoinStep::ContactConfirm,
+                )
+            } else {
+                send_handshake_message(context, &invite, chat_id, BobHandshakeMsg::Request).await?;
+                (BobHandshakeStage::RequestSent, SecureJoinStep::AuthRequired)
             };
-            state
-                .send_handshake_message(context, BobHandshakeMsg::RequestWithAuth)
-                .await?;
-            Ok((state, BobHandshakeStage::RequestWithAuthSent))
-        } else {
-            let state = Self {
-                invite,
-                next: SecureJoinStep::AuthRequired,
-                chat_id,
-            };
-            state
-                .send_handshake_message(context, BobHandshakeMsg::Request)
-                .await?;
-            Ok((state, BobHandshakeStage::RequestSent))
-        }
+        let (id, aborted_states) =
+            Self::insert_new_db_entry(context, next, invite.clone(), chat_id).await?;
+        let state = Self {
+            id,
+            invite,
+            next,
+            chat_id,
+        };
+        Ok((state, stage, aborted_states))
+    }
+
+    /// Inserts a new entry in the bobstate table, deleting all previous entries.
+    ///
+    /// Returns the ID of the newly inserted entry and all the aborted states.
+    async fn insert_new_db_entry(
+        context: &Context,
+        next: SecureJoinStep,
+        invite: QrInvite,
+        chat_id: ChatId,
+    ) -> Result<(i64, Vec<Self>)> {
+        context
+            .sql
+            .transaction(move |transaction| {
+                // We need to start a write transaction right away, so that we have the
+                // database locked and no one else can write to this table while we read the
+                // rows that we will delete.  So start with a dummy UPDATE.
+                transaction.execute(
+                    r#"UPDATE bobstate SET next_step=?;"#,
+                    params![SecureJoinStep::Terminated],
+                )?;
+                let mut stmt = transaction.prepare("SELECT id FROM bobstate;")?;
+                let mut aborted = Vec::new();
+                for id in stmt.query_map(params![], |row| row.get::<_, i64>(0))? {
+                    let id = id?;
+                    let state = BobState::from_db_id(transaction, id)?;
+                    aborted.push(state);
+                }
+
+                // Finally delete everything and insert new row.
+                transaction.execute("DELETE FROM bobstate;", params![])?;
+                transaction.execute(
+                    "INSERT INTO bobstate (invite, next_step, chat_id) VALUES (?, ?, ?);",
+                    params![invite, next, chat_id],
+                )?;
+                let id = transaction.last_insert_rowid();
+                Ok((id, aborted))
+            })
+            .await
+    }
+
+    /// Load [`BobState`] from the database.
+    pub async fn from_db(sql: &Sql) -> Result<Option<Self>> {
+        // Because of how Self::start_protocol() updates the database we are currently
+        // guaranteed to only have one row.
+        sql.query_row_optional(
+            "SELECT id, invite, next_step, chat_id FROM bobstate;",
+            paramsv![],
+            |row| {
+                let s = BobState {
+                    id: row.get(0)?,
+                    invite: row.get(1)?,
+                    next: row.get(2)?,
+                    chat_id: row.get(3)?,
+                };
+                Ok(s)
+            },
+        )
+        .await
+    }
+
+    fn from_db_id(connection: &Connection, id: i64) -> rusqlite::Result<Self> {
+        connection.query_row(
+            "SELECT invite, next_step, chat_id FROM bobstate WHERE id=?;",
+            params![id],
+            |row| {
+                let s = BobState {
+                    id,
+                    invite: row.get(0)?,
+                    next: row.get(1)?,
+                    chat_id: row.get(2)?,
+                };
+                Ok(s)
+            },
+        )
     }
 
     /// Returns the [`QrInvite`] used to create this [`BobState`].
@@ -228,20 +201,45 @@ impl BobState {
         &self.invite
     }
 
+    /// Returns the [`ChatId`] of the 1:1 chat with the inviter (Alice).
+    pub fn alice_chat(&self) -> ChatId {
+        self.chat_id
+    }
+
+    /// Updates the [`BobState::next`] field in memory and the database.
+    ///
+    /// If the next state is a terminal state it will remove this [`BobState`] from the
+    /// database.
+    ///
+    /// If a user scanned a new QR code after this [`BobState`] was loaded this update will
+    /// fail currently because starting a new joiner process currently kills any previously
+    /// running processes.  This is a limitation which will go away in the future.
+    async fn update_next(&mut self, sql: &Sql, next: SecureJoinStep) -> Result<()> {
+        // TODO: write test verifying how this would fail.
+        match next {
+            SecureJoinStep::AuthRequired | SecureJoinStep::ContactConfirm => {
+                sql.execute(
+                    "UPDATE bobstate SET next_step=? WHERE id=?;",
+                    paramsv![next, self.id],
+                )
+                .await?;
+            }
+            SecureJoinStep::Terminated | SecureJoinStep::Completed => {
+                sql.execute("DELETE FROM bobstate WHERE id=?;", paramsv!(self.id))
+                    .await?;
+            }
+        }
+        self.next = next;
+        Ok(())
+    }
+
     /// Handles the given message for the securejoin handshake for Bob.
     ///
     /// If the message was not used for this handshake `None` is returned, otherwise the new
     /// stage is returned.  Once [`BobHandshakeStage::Completed`] or
     /// [`BobHandshakeStage::Terminated`] are reached this [`BobState`] should be destroyed,
     /// further calling it will just result in the messages being unused by this handshake.
-    ///
-    /// # Errors
-    ///
-    /// Under normal operation this should never return an error, regardless of what kind of
-    /// message it is called with.  Any errors therefore should be treated as fatal internal
-    /// errors and this entire [`BobState`] should be thrown away as the state machine can
-    /// no longer be considered consistent.
-    async fn handle_message(
+    pub async fn handle_message(
         &mut self,
         context: &Context,
         mime_message: &MimeMessage,
@@ -304,17 +302,20 @@ impl BobState {
             } else {
                 "Required encryption missing"
             };
-            self.next = SecureJoinStep::Terminated;
+            self.update_next(&context.sql, SecureJoinStep::Terminated)
+                .await?;
             return Ok(Some(BobHandshakeStage::Terminated(reason)));
         }
         if !fingerprint_equals_sender(context, self.invite.fingerprint(), self.invite.contact_id())
             .await?
         {
-            self.next = SecureJoinStep::Terminated;
+            self.update_next(&context.sql, SecureJoinStep::Terminated)
+                .await?;
             return Ok(Some(BobHandshakeStage::Terminated("Fingerprint mismatch")));
         }
         info!(context, "Fingerprint verified.",);
-        self.next = SecureJoinStep::ContactConfirm;
+        self.update_next(&context.sql, SecureJoinStep::ContactConfirm)
+            .await?;
         self.send_handshake_message(context, BobHandshakeMsg::RequestWithAuth)
             .await?;
         Ok(Some(BobHandshakeStage::RequestWithAuthSent))
@@ -362,7 +363,8 @@ impl BobState {
         if vg_expect_encrypted
             && !encrypted_and_signed(context, mime_message, Some(self.invite.fingerprint()))
         {
-            self.next = SecureJoinStep::Terminated;
+            self.update_next(&context.sql, SecureJoinStep::Terminated)
+                .await?;
             return Ok(Some(BobHandshakeStage::Terminated(
                 "Contact confirm message not encrypted",
             )));
@@ -394,7 +396,8 @@ impl BobState {
             // This is not an error affecting the protocol outcome.
             .ok();
 
-        self.next = SecureJoinStep::Completed;
+        self.update_next(&context.sql, SecureJoinStep::Completed)
+            .await?;
         Ok(Some(BobHandshakeStage::Completed))
     }
 
@@ -406,48 +409,60 @@ impl BobState {
         context: &Context,
         step: BobHandshakeMsg,
     ) -> Result<(), SendMsgError> {
-        let mut msg = Message {
-            viewtype: Viewtype::Text,
-            text: Some(step.body_text(&self.invite)),
-            hidden: true,
-            ..Default::default()
-        };
-        msg.param.set_cmd(SystemMessage::SecurejoinMessage);
-
-        // Sends the step in Secure-Join header.
-        msg.param
-            .set(Param::Arg, step.securejoin_header(&self.invite));
-
-        match step {
-            BobHandshakeMsg::Request => {
-                // Sends the Secure-Join-Invitenumber header in mimefactory.rs.
-                msg.param.set(Param::Arg2, self.invite.invitenumber());
-                msg.param.set_int(Param::ForcePlaintext, 1);
-            }
-            BobHandshakeMsg::RequestWithAuth => {
-                // Sends the Secure-Join-Auth header in mimefactory.rs.
-                msg.param.set(Param::Arg2, self.invite.authcode());
-                msg.param.set_int(Param::GuaranteeE2ee, 1);
-            }
-            BobHandshakeMsg::ContactConfirmReceived => {
-                msg.param.set_int(Param::GuaranteeE2ee, 1);
-            }
-        };
-
-        // Sends our own fingerprint in the Secure-Join-Fingerprint header.
-        let bob_fp = SignedPublicKey::load_self(context).await?.fingerprint();
-        msg.param.set(Param::Arg3, bob_fp.hex());
-
-        // Sends the grpid in the Secure-Join-Group header.
-        if let QrInvite::Group { ref grpid, .. } = self.invite {
-            msg.param.set(Param::Arg4, grpid);
-        }
-
-        chat::send_msg(context, self.chat_id, &mut msg).await?;
-        Ok(())
+        send_handshake_message(context, &self.invite, self.chat_id, step).await
     }
 }
 
+/// Sends the requested handshake message to Alice.
+///
+/// Same as [`BobState::send_handshake_message`] but this variation allows us to send this
+/// message before we create the state in [`BobState::start_protocol`].
+async fn send_handshake_message(
+    context: &Context,
+    invite: &QrInvite,
+    chat_id: ChatId,
+    step: BobHandshakeMsg,
+) -> Result<(), SendMsgError> {
+    let mut msg = Message {
+        viewtype: Viewtype::Text,
+        text: Some(step.body_text(invite)),
+        hidden: true,
+        ..Default::default()
+    };
+    msg.param.set_cmd(SystemMessage::SecurejoinMessage);
+
+    // Sends the step in Secure-Join header.
+    msg.param.set(Param::Arg, step.securejoin_header(invite));
+
+    match step {
+        BobHandshakeMsg::Request => {
+            // Sends the Secure-Join-Invitenumber header in mimefactory.rs.
+            msg.param.set(Param::Arg2, invite.invitenumber());
+            msg.force_plaintext();
+        }
+        BobHandshakeMsg::RequestWithAuth => {
+            // Sends the Secure-Join-Auth header in mimefactory.rs.
+            msg.param.set(Param::Arg2, invite.authcode());
+            msg.param.set_int(Param::GuaranteeE2ee, 1);
+        }
+        BobHandshakeMsg::ContactConfirmReceived => {
+            msg.param.set_int(Param::GuaranteeE2ee, 1);
+        }
+    };
+
+    // Sends our own fingerprint in the Secure-Join-Fingerprint header.
+    let bob_fp = SignedPublicKey::load_self(context).await?.fingerprint();
+    msg.param.set(Param::Arg3, bob_fp.hex());
+
+    // Sends the grpid in the Secure-Join-Group header.
+    if let QrInvite::Group { ref grpid, .. } = invite {
+        msg.param.set(Param::Arg4, grpid);
+    }
+
+    chat::send_msg(context, chat_id, &mut msg).await?;
+    Ok(())
+}
+
 /// Identifies the SecureJoin handshake messages Bob can send.
 enum BobHandshakeMsg {
     /// vc-request or vg-request
@@ -492,8 +507,8 @@ impl BobHandshakeMsg {
 }
 
 /// The next message expected by [`BobState`] in the setup-contact/secure-join protocol.
-#[derive(Debug, Clone, PartialEq)]
-enum SecureJoinStep {
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum SecureJoinStep {
     /// Expecting the auth-required message.
     ///
     /// This corresponds to the `vc-auth-required` or `vg-auth-required` message of step 3d.
@@ -533,3 +548,29 @@ impl SecureJoinStep {
         }
     }
 }
+
+impl rusqlite::types::ToSql for SecureJoinStep {
+    fn to_sql(&self) -> rusqlite::Result<rusqlite::types::ToSqlOutput<'_>> {
+        let num = match &self {
+            SecureJoinStep::AuthRequired => 0,
+            SecureJoinStep::ContactConfirm => 1,
+            SecureJoinStep::Terminated => 2,
+            SecureJoinStep::Completed => 3,
+        };
+        let val = rusqlite::types::Value::Integer(num);
+        let out = rusqlite::types::ToSqlOutput::Owned(val);
+        Ok(out)
+    }
+}
+
+impl rusqlite::types::FromSql for SecureJoinStep {
+    fn column_result(value: rusqlite::types::ValueRef<'_>) -> rusqlite::types::FromSqlResult<Self> {
+        i64::column_result(value).and_then(|val| match val {
+            0 => Ok(SecureJoinStep::AuthRequired),
+            1 => Ok(SecureJoinStep::ContactConfirm),
+            2 => Ok(SecureJoinStep::Terminated),
+            3 => Ok(SecureJoinStep::Completed),
+            _ => Err(rusqlite::types::FromSqlError::OutOfRange(val)),
+        })
+    }
+}

src/securejoin/qrinvite.rs

@@ -8,22 +8,23 @@ use std::convert::TryFrom;
 
 use anyhow::{bail, Error, Result};
 
+use crate::contact::ContactId;
 use crate::key::Fingerprint;
 use crate::qr::Qr;
 
 /// Represents the data from a QR-code scan.
 ///
 /// There are methods to conveniently access fields present in both variants.
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
 pub enum QrInvite {
     Contact {
-        contact_id: u32,
+        contact_id: ContactId,
         fingerprint: Fingerprint,
         invitenumber: String,
         authcode: String,
     },
     Group {
-        contact_id: u32,
+        contact_id: ContactId,
         fingerprint: Fingerprint,
         name: String,
         grpid: String,
@@ -37,7 +38,7 @@ impl QrInvite {
     ///
     /// The actual QR-code contains a URL-encoded email address, but upon scanning this is
     /// translated to a contact ID.
-    pub fn contact_id(&self) -> u32 {
+    pub fn contact_id(&self) -> ContactId {
         match self {
             Self::Contact { contact_id, .. } | Self::Group { contact_id, .. } => *contact_id,
         }
@@ -100,3 +101,22 @@ impl TryFrom<Qr> for QrInvite {
         }
     }
 }
+
+impl rusqlite::types::ToSql for QrInvite {
+    fn to_sql(&self) -> rusqlite::Result<rusqlite::types::ToSqlOutput<'_>> {
+        let json = serde_json::to_string(self)
+            .map_err(|err| rusqlite::Error::ToSqlConversionFailure(Box::new(err)))?;
+        let val = rusqlite::types::Value::Text(json);
+        let out = rusqlite::types::ToSqlOutput::Owned(val);
+        Ok(out)
+    }
+}
+
+impl rusqlite::types::FromSql for QrInvite {
+    fn column_result(value: rusqlite::types::ValueRef<'_>) -> rusqlite::types::FromSqlResult<Self> {
+        String::column_result(value).and_then(|val| {
+            serde_json::from_str(&val)
+                .map_err(|err| rusqlite::types::FromSqlError::Other(Box::new(err)))
+        })
+    }
+}

src/simplify.rs

@@ -240,7 +240,7 @@ fn render_message(lines: &[&str], is_cut_at_end: bool) -> String {
         ret += " [...]";
     }
     // redo escaping done by escape_message_footer_marks()
-    ret.replace("\u{200B}", "")
+    ret.replace('\u{200B}', "")
 }
 
 /// Returns true if the line contains only whitespace.

src/smtp.rs

@@ -4,14 +4,18 @@ pub mod send;
 
 use std::time::{Duration, SystemTime};
 
+use anyhow::{bail, format_err, Context as _, Result};
 use async_smtp::smtp::client::net::ClientTlsParameters;
-use async_smtp::{error, smtp, EmailAddress, ServerAddress};
+use async_smtp::smtp::response::{Category, Code, Detail};
+use async_smtp::{smtp, EmailAddress, ServerAddress};
 
 use crate::constants::DC_LP_AUTH_OAUTH2;
 use crate::events::EventType;
+use crate::job::Status;
 use crate::login_param::{
     dc_build_tls, CertificateChecks, LoginParam, ServerLoginParam, Socks5Config,
 };
+use crate::message::{self, MsgId};
 use crate::oauth2::dc_get_oauth2_access_token;
 use crate::provider::Socket;
 use crate::{context::Context, scheduler::connectivity::ConnectivityStore};
@@ -19,28 +23,6 @@ use crate::{context::Context, scheduler::connectivity::ConnectivityStore};
 /// SMTP write and read timeout in seconds.
 const SMTP_TIMEOUT: u64 = 30;
 
-#[derive(Debug, thiserror::Error)]
-pub enum Error {
-    #[error("Bad parameters")]
-    BadParameters,
-    #[error("Invalid login address {address}: {error}")]
-    InvalidLoginAddress {
-        address: String,
-        #[source]
-        error: error::Error,
-    },
-    #[error("SMTP failed to connect: {0}")]
-    ConnectionFailure(#[source] smtp::error::Error),
-    #[error("SMTP oauth2 error {address}")]
-    Oauth2 { address: String },
-    #[error("TLS error {0}")]
-    Tls(#[from] async_native_tls::Error),
-    #[error("{0}")]
-    Other(#[from] anyhow::Error),
-}
-
-pub type Result<T> = std::result::Result<T, Error>;
-
 #[derive(Default)]
 pub(crate) struct Smtp {
     transport: Option<smtp::SmtpTransport>,
@@ -131,14 +113,11 @@ impl Smtp {
         }
 
         if lp.server.is_empty() || lp.port == 0 {
-            return Err(Error::BadParameters);
+            bail!("bad connection parameters");
         }
 
-        let from =
-            EmailAddress::new(addr.to_string()).map_err(|err| Error::InvalidLoginAddress {
-                address: addr.to_string(),
-                error: err,
-            })?;
+        let from = EmailAddress::new(addr.to_string())
+            .with_context(|| format!("invalid login address {}", addr))?;
 
         self.from = Some(from);
 
@@ -159,9 +138,7 @@ impl Smtp {
             let send_pw = &lp.password;
             let access_token = dc_get_oauth2_access_token(context, addr, send_pw, false).await?;
             if access_token.is_none() {
-                return Err(Error::Oauth2 {
-                    address: addr.to_string(),
-                });
+                bail!("SMTP OAuth 2 error {}", addr);
             }
             let user = &lp.user;
             (
@@ -205,9 +182,7 @@ impl Smtp {
         }
 
         let mut trans = client.into_transport();
-        if let Err(err) = trans.connect().await {
-            return Err(Error::ConnectionFailure(err));
-        }
+        trans.connect().await.context("SMTP failed to connect")?;
 
         self.transport = Some(trans);
         self.last_success = Some(SystemTime::now());
@@ -220,3 +195,324 @@ impl Smtp {
         Ok(())
     }
 }
+
+/// Tries to send a message.
+///
+/// Returns Status::Finished if sending the message should not be retried anymore,
+/// Status::RetryLater if sending should be postponed and Status::RetryNow if it is suspected that
+/// temporary failure is caused by stale connection, in which case a second attempt to send the
+/// same message may be done immediately.
+pub(crate) async fn smtp_send(
+    context: &Context,
+    recipients: &[async_smtp::EmailAddress],
+    message: &str,
+    smtp: &mut Smtp,
+    msg_id: MsgId,
+    rowid: i64,
+) -> Status {
+    if std::env::var(crate::DCC_MIME_DEBUG).is_ok() {
+        info!(context, "smtp-sending out mime message:");
+        println!("{}", message);
+    }
+
+    smtp.connectivity.set_working(context).await;
+
+    let send_result = smtp
+        .send(context, recipients, message.as_bytes(), rowid)
+        .await;
+    smtp.last_send_error = send_result.as_ref().err().map(|e| e.to_string());
+
+    let status = match send_result {
+        Err(crate::smtp::send::Error::SmtpSend(err)) => {
+            // Remote error, retry later.
+            info!(context, "SMTP failed to send: {:?}", &err);
+
+            let res = match err {
+                async_smtp::smtp::error::Error::Permanent(ref response) => {
+                    // Workaround for incorrectly configured servers returning permanent errors
+                    // instead of temporary ones.
+                    let maybe_transient = match response.code {
+                        // Sometimes servers send a permanent error when actually it is a temporary error
+                        // For documentation see <https://tools.ietf.org/html/rfc3463>
+                        Code {
+                            category: Category::MailSystem,
+                            detail: Detail::Zero,
+                            ..
+                        } => {
+                            // Ignore status code 5.5.0, see <https://support.delta.chat/t/every-other-message-gets-stuck/877/2>
+                            // Maybe incorrectly configured Postfix milter with "reject" instead of "tempfail", which returns
+                            // "550 5.5.0 Service unavailable" instead of "451 4.7.1 Service unavailable - try again later".
+                            //
+                            // Other enhanced status codes, such as Postfix
+                            // "550 5.1.1 <foobar@example.org>: Recipient address rejected: User unknown in local recipient table"
+                            // are not ignored.
+                            response.first_word() == Some("5.5.0")
+                        }
+                        _ => false,
+                    };
+
+                    if maybe_transient {
+                        info!(context, "Permanent error that is likely to actually be transient, postponing retry for later");
+                        Status::RetryLater
+                    } else {
+                        info!(context, "Permanent error, message sending failed");
+                        // If we do not retry, add an info message to the chat.
+                        // Yandex error "554 5.7.1 [2] Message rejected under suspicion of SPAM; https://ya.cc/..."
+                        // should definitely go here, because user has to open the link to
+                        // resume message sending.
+                        Status::Finished(Err(format_err!("Permanent SMTP error: {}", err)))
+                    }
+                }
+                async_smtp::smtp::error::Error::Transient(ref response) => {
+                    // We got a transient 4xx response from SMTP server.
+                    // Give some time until the server-side error maybe goes away.
+
+                    if let Some(first_word) = response.first_word() {
+                        if first_word.ends_with(".1.1")
+                            || first_word.ends_with(".1.2")
+                            || first_word.ends_with(".1.3")
+                        {
+                            // Sometimes we receive transient errors that should be permanent.
+                            // Any extended smtp status codes like x.1.1, x.1.2 or x.1.3 that we
+                            // receive as a transient error are misconfigurations of the smtp server.
+                            // See <https://tools.ietf.org/html/rfc3463#section-3.2>
+                            info!(context, "Received extended status code {} for a transient error. This looks like a misconfigured smtp server, let's fail immediatly", first_word);
+                            Status::Finished(Err(format_err!("Permanent SMTP error: {}", err)))
+                        } else {
+                            info!(
+                                context,
+                                "Transient error with status code {}, postponing retry for later",
+                                first_word
+                            );
+                            Status::RetryLater
+                        }
+                    } else {
+                        info!(
+                            context,
+                            "Transient error without status code, postponing retry for later"
+                        );
+                        Status::RetryLater
+                    }
+                }
+                _ => {
+                    info!(
+                        context,
+                        "Message sending failed without error returned by the server"
+                    );
+                    if smtp.has_maybe_stale_connection().await {
+                        info!(context, "Connection is probably stale, retry immediately");
+                        Status::RetryNow
+                    } else {
+                        info!(context, "Connection is not stale, retry later");
+                        Status::RetryLater
+                    }
+                }
+            };
+
+            // this clears last_success info
+            info!(context, "Failed to send message over SMTP, disconnecting");
+            smtp.disconnect().await;
+
+            res
+        }
+        Err(crate::smtp::send::Error::Envelope(err)) => {
+            // Local error, job is invalid, do not retry.
+            smtp.disconnect().await;
+            warn!(context, "SMTP job is invalid: {}", err);
+            Status::Finished(Err(err.into()))
+        }
+        Err(crate::smtp::send::Error::NoTransport) => {
+            // Should never happen.
+            // It does not even make sense to disconnect here.
+            error!(context, "SMTP job failed because SMTP has no transport");
+            Status::Finished(Err(format_err!("SMTP has not transport")))
+        }
+        Err(crate::smtp::send::Error::Other(err)) => {
+            // Local error, job is invalid, do not retry.
+            smtp.disconnect().await;
+            warn!(context, "unable to load job: {}", err);
+            Status::Finished(Err(err))
+        }
+        Ok(()) => Status::Finished(Ok(())),
+    };
+
+    if let Status::Finished(Err(err)) = &status {
+        // We couldn't send the message, so mark it as failed
+        message::set_msg_failed(context, msg_id, Some(err.to_string())).await;
+    }
+    status
+}
+
+/// Sends message identified by `smtp` table rowid over SMTP connection.
+///
+/// Removes row if the message should not be retried, otherwise increments retry count.
+pub(crate) async fn send_msg_to_smtp(
+    context: &Context,
+    smtp: &mut Smtp,
+    rowid: i64,
+) -> anyhow::Result<()> {
+    if let Err(err) = smtp
+        .connect_configured(context)
+        .await
+        .context("SMTP connection failure")
+    {
+        smtp.last_send_error = Some(format!("{:#}", err));
+        return Err(err);
+    }
+
+    // Increase retry count as soon as we have an SMTP connection. This ensures that the message is
+    // eventually removed from the queue by exceeding retry limit even in case of an error that
+    // keeps happening early in the message sending code, e.g. failure to read the message from the
+    // database.
+    context
+        .sql
+        .execute(
+            "UPDATE smtp SET retries=retries+1 WHERE id=?",
+            paramsv![rowid],
+        )
+        .await
+        .context("failed to update retries count")?;
+
+    let (body, recipients, msg_id, retries) = context
+        .sql
+        .query_row(
+            "SELECT mime, recipients, msg_id, retries FROM smtp WHERE id=?",
+            paramsv![rowid],
+            |row| {
+                let mime: String = row.get(0)?;
+                let recipients: String = row.get(1)?;
+                let msg_id: MsgId = row.get(2)?;
+                let retries: i64 = row.get(3)?;
+                Ok((mime, recipients, msg_id, retries))
+            },
+        )
+        .await?;
+    if retries > 6 {
+        message::set_msg_failed(
+            context,
+            msg_id,
+            Some("Number of retries exceeded the limit."),
+        )
+        .await;
+        context
+            .sql
+            .execute("DELETE FROM smtp WHERE id=?", paramsv![rowid])
+            .await
+            .context("failed to remove message with exceeded retry limit from smtp table")?;
+        bail!("Number of retries exceeded the limit");
+    }
+    info!(
+        context,
+        "Retry number {} to send message {} over SMTP", retries, msg_id
+    );
+
+    let recipients_list = recipients
+        .split(' ')
+        .filter_map(
+            |addr| match async_smtp::EmailAddress::new(addr.to_string()) {
+                Ok(addr) => Some(addr),
+                Err(err) => {
+                    warn!(context, "invalid recipient: {} {:?}", addr, err);
+                    None
+                }
+            },
+        )
+        .collect::<Vec<_>>();
+
+    // If there is a msg-id and it does not exist in the db, cancel sending. this happens if
+    // dc_delete_msgs() was called before the generated mime was sent out.
+    if !message::exists(context, msg_id)
+        .await
+        .with_context(|| format!("failed to check message {} existence", msg_id))?
+    {
+        info!(
+            context,
+            "Sending of message {} was cancelled by the user.", msg_id
+        );
+        return Ok(());
+    }
+
+    let status = match smtp_send(
+        context,
+        &recipients_list,
+        body.as_str(),
+        smtp,
+        msg_id,
+        rowid,
+    )
+    .await
+    {
+        Status::RetryNow => {
+            // Do a single retry immediately without increasing retry counter in case of stale
+            // connection.
+            info!(context, "Doing immediate retry to send message.");
+
+            // smtp_send just closed stale SMTP connection, reconnect and try again.
+            if let Err(err) = smtp
+                .connect_configured(context)
+                .await
+                .context("failed to reopen stale SMTP connection")
+            {
+                smtp.last_send_error = Some(format!("{:#}", err));
+                return Err(err);
+            }
+
+            smtp_send(
+                context,
+                &recipients_list,
+                body.as_str(),
+                smtp,
+                msg_id,
+                rowid,
+            )
+            .await
+        }
+        status => status,
+    };
+    match status {
+        Status::Finished(res) => {
+            context
+                .sql
+                .execute("DELETE FROM smtp WHERE id=?", paramsv![rowid])
+                .await?;
+            if res.is_ok() {
+                msg_id.set_delivered(context).await?;
+            }
+            res
+        }
+        Status::RetryNow | Status::RetryLater => Err(format_err!("Retry")),
+    }
+}
+
+/// Tries to send all messages currently in `smtp` table.
+///
+/// Logs and ignores SMTP errors to ensure that a single SMTP message constantly failing to be sent
+/// does not block other messages in the queue from being sent.
+pub(crate) async fn send_smtp_messages(
+    context: &Context,
+    connection: &mut Smtp,
+) -> anyhow::Result<()> {
+    context.send_sync_msg().await?; // Add sync message to the end of the queue if needed.
+    let rowids = context
+        .sql
+        .query_map(
+            "SELECT id FROM smtp ORDER BY id ASC",
+            paramsv![],
+            |row| {
+                let rowid: i64 = row.get(0)?;
+                Ok(rowid)
+            },
+            |rowids| {
+                rowids
+                    .collect::<std::result::Result<Vec<_>, _>>()
+                    .map_err(Into::into)
+            },
+        )
+        .await?;
+    for rowid in rowids {
+        if let Err(err) = send_msg_to_smtp(context, connection, rowid).await {
+            info!(context, "Failed to send message over SMTP: {:#}.", err);
+        }
+    }
+    Ok(())
+}

src/smtp/send.rs

@@ -28,9 +28,9 @@ impl Smtp {
     pub async fn send(
         &mut self,
         context: &Context,
-        recipients: Vec<EmailAddress>,
-        message: Vec<u8>,
-        job_id: u32,
+        recipients: &[EmailAddress],
+        message: &[u8],
+        rowid: i64,
     ) -> Result<()> {
         let message_len_bytes = message.len();
 
@@ -41,7 +41,7 @@ impl Smtp {
             }
         }
 
-        for recipients_chunk in recipients.chunks(chunk_size).into_iter() {
+        for recipients_chunk in recipients.chunks(chunk_size) {
             let recipients_display = recipients_chunk
                 .iter()
                 .map(|x| x.as_ref())
@@ -52,8 +52,8 @@ impl Smtp {
                 .map_err(Error::Envelope)?;
             let mail = SendableEmail::new(
                 envelope,
-                format!("{}", job_id), // only used for internal logging
-                &message,
+                rowid.to_string(), // only used for internal logging
+                message,
             );
 
             if let Some(ref mut transport) = self.transport {

src/sql.rs

@@ -3,22 +3,23 @@
 use async_std::path::Path;
 use async_std::sync::RwLock;
 
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
 use std::convert::TryFrom;
 use std::time::Duration;
 
-use anyhow::{bail, format_err, Context as _, Result};
+use anyhow::{bail, Context as _, Result};
+use async_std::path::PathBuf;
 use async_std::prelude::*;
-use rusqlite::OpenFlags;
+use rusqlite::{config::DbConfig, Connection, OpenFlags};
 
 use crate::blob::BlobObject;
 use crate::chat::{add_device_msg, update_device_icon, update_saved_messages_icon};
 use crate::config::Config;
-use crate::constants::{Viewtype, DC_CHAT_ID_TRASH};
+use crate::constants::DC_CHAT_ID_TRASH;
 use crate::context::Context;
 use crate::dc_tools::{dc_delete_file, time};
 use crate::ephemeral::start_ephemeral_timers;
-use crate::message::Message;
+use crate::message::{Message, Viewtype};
 use crate::param::{Param, Params};
 use crate::peerstate::{deduplicate_peerstates, Peerstate};
 use crate::stock_str;
@@ -38,20 +39,53 @@ mod migrations;
 /// A wrapper around the underlying Sqlite3 object.
 #[derive(Debug)]
 pub struct Sql {
-    pool: RwLock<Option<r2d2::Pool<r2d2_sqlite::SqliteConnectionManager>>>,
-}
+    /// Database file path
+    pub(crate) dbfile: PathBuf,
 
-impl Default for Sql {
-    fn default() -> Self {
-        Self {
-            pool: RwLock::new(None),
-        }
-    }
+    pool: RwLock<Option<r2d2::Pool<r2d2_sqlite::SqliteConnectionManager>>>,
+
+    /// None if the database is not open, true if it is open with passphrase and false if it is
+    /// open without a passphrase.
+    is_encrypted: RwLock<Option<bool>>,
+
+    pub(crate) config_cache: RwLock<HashMap<String, Option<String>>>,
 }
 
 impl Sql {
-    pub fn new() -> Sql {
-        Self::default()
+    pub fn new(dbfile: PathBuf) -> Sql {
+        Self {
+            dbfile,
+            pool: Default::default(),
+            is_encrypted: Default::default(),
+            config_cache: Default::default(),
+        }
+    }
+
+    /// Tests SQLCipher passphrase.
+    ///
+    /// Returns true if passphrase is correct, i.e. the database is new or can be unlocked with
+    /// this passphrase, and false if the database is already encrypted with another passphrase or
+    /// corrupted.
+    ///
+    /// Fails if database is already open.
+    pub async fn check_passphrase(&self, passphrase: String) -> Result<bool> {
+        if self.is_open().await {
+            bail!("Database is already opened.");
+        }
+
+        // Hold the lock to prevent other thread from opening the database.
+        let _lock = self.pool.write().await;
+
+        // Test that the key is correct using a single connection.
+        let connection = Connection::open(&self.dbfile)?;
+        connection
+            .pragma_update(None, "key", &passphrase)
+            .context("failed to set PRAGMA key")?;
+        let key_is_correct = connection
+            .query_row("SELECT count(*) FROM sqlite_master", [], |_row| Ok(()))
+            .is_ok();
+
+        Ok(key_is_correct)
     }
 
     /// Checks if there is currently a connection to the underlying Sqlite database.
@@ -59,37 +93,98 @@ impl Sql {
         self.pool.read().await.is_some()
     }
 
+    /// Returns true if the database is encrypted.
+    ///
+    /// If database is not open, returns `None`.
+    pub(crate) async fn is_encrypted(&self) -> Option<bool> {
+        *self.is_encrypted.read().await
+    }
+
     /// Closes all underlying Sqlite connections.
-    pub async fn close(&self) {
+    async fn close(&self) {
         let _ = self.pool.write().await.take();
         // drop closes the connection
     }
 
-    pub fn new_pool(
+    /// Exports the database to a separate file with the given passphrase.
+    ///
+    /// Set passphrase to empty string to export the database unencrypted.
+    pub(crate) async fn export(&self, path: &Path, passphrase: String) -> Result<()> {
+        let path_str = path
+            .to_str()
+            .with_context(|| format!("path {:?} is not valid unicode", path))?;
+        let conn = self.get_conn().await?;
+        conn.execute(
+            "ATTACH DATABASE ? AS backup KEY ?",
+            paramsv![path_str, passphrase],
+        )
+        .context("failed to attach backup database")?;
+        let res = conn
+            .query_row("SELECT sqlcipher_export('backup')", [], |_row| Ok(()))
+            .context("failed to export to attached backup database");
+        conn.execute("DETACH DATABASE backup", [])
+            .context("failed to detach backup database")?;
+        res?;
+        Ok(())
+    }
+
+    /// Imports the database from a separate file with the given passphrase.
+    pub(crate) async fn import(&self, path: &Path, passphrase: String) -> Result<()> {
+        let path_str = path
+            .to_str()
+            .with_context(|| format!("path {:?} is not valid unicode", path))?;
+        let conn = self.get_conn().await?;
+
+        // Reset the database without reopening it. We don't want to reopen the database because we
+        // don't have main database passphrase at this point.
+        // See <https://sqlite.org/c3ref/c_dbconfig_enable_fkey.html> for documentation.
+        // Without resetting import may fail due to existing tables.
+        conn.set_db_config(DbConfig::SQLITE_DBCONFIG_RESET_DATABASE, true)
+            .context("failed to set SQLITE_DBCONFIG_RESET_DATABASE")?;
+        conn.execute("VACUUM", [])
+            .context("failed to vacuum the database")?;
+        conn.set_db_config(DbConfig::SQLITE_DBCONFIG_RESET_DATABASE, false)
+            .context("failed to unset SQLITE_DBCONFIG_RESET_DATABASE")?;
+
+        conn.execute(
+            "ATTACH DATABASE ? AS backup KEY ?",
+            paramsv![path_str, passphrase],
+        )
+        .context("failed to attach backup database")?;
+        let res = conn
+            .query_row("SELECT sqlcipher_export('main', 'backup')", [], |_row| {
+                Ok(())
+            })
+            .context("failed to import from attached backup database");
+        conn.execute("DETACH DATABASE backup", [])
+            .context("failed to detach backup database")?;
+        res?;
+        Ok(())
+    }
+
+    fn new_pool(
         dbfile: &Path,
-        readonly: bool,
-    ) -> anyhow::Result<r2d2::Pool<r2d2_sqlite::SqliteConnectionManager>> {
+        passphrase: String,
+    ) -> Result<r2d2::Pool<r2d2_sqlite::SqliteConnectionManager>> {
         let mut open_flags = OpenFlags::SQLITE_OPEN_NO_MUTEX;
-        if readonly {
-            open_flags.insert(OpenFlags::SQLITE_OPEN_READ_ONLY);
-        } else {
-            open_flags.insert(OpenFlags::SQLITE_OPEN_READ_WRITE);
-            open_flags.insert(OpenFlags::SQLITE_OPEN_CREATE);
-        }
+        open_flags.insert(OpenFlags::SQLITE_OPEN_READ_WRITE);
+        open_flags.insert(OpenFlags::SQLITE_OPEN_CREATE);
 
         // this actually creates min_idle database handles just now.
         // therefore, with_init() must not try to modify the database as otherwise
         // we easily get busy-errors (eg. table-creation, journal_mode etc. should be done on only one handle)
         let mgr = r2d2_sqlite::SqliteConnectionManager::file(dbfile)
             .with_flags(open_flags)
-            .with_init(|c| {
+            .with_init(move |c| {
                 c.execute_batch(&format!(
-                    "PRAGMA secure_delete=on;
+                    "PRAGMA cipher_memory_security = OFF; -- Too slow on Android
+                     PRAGMA secure_delete=on;
                      PRAGMA busy_timeout = {};
                      PRAGMA temp_store=memory; -- Avoid SQLITE_IOERR_GETTEMPPATH errors on Android
                      ",
                     Duration::from_secs(10).as_millis()
                 ))?;
+                c.pragma_update(None, "key", passphrase.clone())?;
                 Ok(())
             });
 
@@ -102,105 +197,125 @@ impl Sql {
         Ok(pool)
     }
 
+    async fn try_open(&self, context: &Context, dbfile: &Path, passphrase: String) -> Result<()> {
+        *self.pool.write().await = Some(Self::new_pool(dbfile, passphrase.to_string())?);
+
+        {
+            let conn = self.get_conn().await?;
+
+            // Try to enable auto_vacuum. This will only be
+            // applied if the database is new or after successful
+            // VACUUM, which usually happens before backup export.
+            // When auto_vacuum is INCREMENTAL, it is possible to
+            // use PRAGMA incremental_vacuum to return unused
+            // database pages to the filesystem.
+            conn.pragma_update(None, "auto_vacuum", &"INCREMENTAL".to_string())?;
+
+            // journal_mode is persisted, it is sufficient to change it only for one handle.
+            conn.pragma_update(None, "journal_mode", &"WAL".to_string())?;
+
+            // Default synchronous=FULL is much slower. NORMAL is sufficient for WAL mode.
+            conn.pragma_update(None, "synchronous", &"NORMAL".to_string())?;
+        }
+
+        self.run_migrations(context).await?;
+
+        Ok(())
+    }
+
+    pub async fn run_migrations(&self, context: &Context) -> Result<()> {
+        // (1) update low-level database structure.
+        // this should be done before updates that use high-level objects that
+        // rely themselves on the low-level structure.
+
+        let (recalc_fingerprints, update_icons, disable_server_delete, recode_avatar) =
+            migrations::run(context, self)
+                .await
+                .context("failed to run migrations")?;
+
+        // (2) updates that require high-level objects
+        // the structure is complete now and all objects are usable
+
+        if recalc_fingerprints {
+            info!(context, "[migration] recalc fingerprints");
+            let addrs = self
+                .query_map(
+                    "SELECT addr FROM acpeerstates;",
+                    paramsv![],
+                    |row| row.get::<_, String>(0),
+                    |addrs| {
+                        addrs
+                            .collect::<std::result::Result<Vec<_>, _>>()
+                            .map_err(Into::into)
+                    },
+                )
+                .await?;
+            for addr in &addrs {
+                if let Some(ref mut peerstate) = Peerstate::from_addr(context, addr).await? {
+                    peerstate.recalc_fingerprint();
+                    peerstate.save_to_db(self, false).await?;
+                }
+            }
+        }
+
+        if update_icons {
+            update_saved_messages_icon(context).await?;
+            update_device_icon(context).await?;
+        }
+
+        if disable_server_delete {
+            // We now always watch all folders and delete messages there if delete_server is enabled.
+            // So, for people who have delete_server enabled, disable it and add a hint to the devicechat:
+            if context.get_config_delete_server_after().await?.is_some() {
+                let mut msg = Message::new(Viewtype::Text);
+                msg.text = Some(stock_str::delete_server_turned_off(context).await);
+                add_device_msg(context, None, Some(&mut msg)).await?;
+                context
+                    .set_config(Config::DeleteServerAfter, Some("0"))
+                    .await?;
+            }
+        }
+
+        if recode_avatar {
+            if let Some(avatar) = context.get_config(Config::Selfavatar).await? {
+                let mut blob = BlobObject::new_from_path(context, avatar.as_ref()).await?;
+                match blob.recode_to_avatar_size(context).await {
+                    Ok(()) => {
+                        context
+                            .set_config(Config::Selfavatar, Some(&avatar))
+                            .await?
+                    }
+                    Err(e) => {
+                        warn!(context, "Migrations can't recode avatar, removing. {:#}", e);
+                        context.set_config(Config::Selfavatar, None).await?
+                    }
+                }
+            }
+        }
+
+        Ok(())
+    }
+
     /// Opens the provided database and runs any necessary migrations.
     /// If a database is already open, this will return an error.
-    pub async fn open(
-        &self,
-        context: &Context,
-        dbfile: &Path,
-        readonly: bool,
-    ) -> anyhow::Result<()> {
+    pub async fn open(&self, context: &Context, passphrase: String) -> Result<()> {
         if self.is_open().await {
             error!(
                 context,
-                "Cannot open, database \"{:?}\" already opened.", dbfile,
+                "Cannot open, database \"{:?}\" already opened.", self.dbfile,
             );
             bail!("SQL database is already opened.");
         }
 
-        *self.pool.write().await = Some(Self::new_pool(dbfile, readonly)?);
-
-        if !readonly {
-            {
-                let conn = self.get_conn().await?;
-                // journal_mode is persisted, it is sufficient to change it only for one handle.
-                conn.pragma_update(None, "journal_mode", &"WAL".to_string())?;
-
-                // Default synchronous=FULL is much slower. NORMAL is sufficient for WAL mode.
-                conn.pragma_update(None, "synchronous", &"NORMAL".to_string())?;
-            }
-
-            // (1) update low-level database structure.
-            // this should be done before updates that use high-level objects that
-            // rely themselves on the low-level structure.
-
-            let (recalc_fingerprints, update_icons, disable_server_delete, recode_avatar) =
-                migrations::run(context, self).await?;
-
-            // (2) updates that require high-level objects
-            // the structure is complete now and all objects are usable
-
-            if recalc_fingerprints {
-                info!(context, "[migration] recalc fingerprints");
-                let addrs = self
-                    .query_map(
-                        "SELECT addr FROM acpeerstates;",
-                        paramsv![],
-                        |row| row.get::<_, String>(0),
-                        |addrs| {
-                            addrs
-                                .collect::<std::result::Result<Vec<_>, _>>()
-                                .map_err(Into::into)
-                        },
-                    )
-                    .await?;
-                for addr in &addrs {
-                    if let Some(ref mut peerstate) = Peerstate::from_addr(context, addr).await? {
-                        peerstate.recalc_fingerprint();
-                        peerstate.save_to_db(self, false).await?;
-                    }
-                }
-            }
-
-            if update_icons {
-                update_saved_messages_icon(context).await?;
-                update_device_icon(context).await?;
-            }
-
-            if disable_server_delete {
-                // We now always watch all folders and delete messages there if delete_server is enabled.
-                // So, for people who have delete_server enabled, disable it and add a hint to the devicechat:
-                if context.get_config_delete_server_after().await?.is_some() {
-                    let mut msg = Message::new(Viewtype::Text);
-                    msg.text = Some(stock_str::delete_server_turned_off(context).await);
-                    add_device_msg(context, None, Some(&mut msg)).await?;
-                    context
-                        .set_config(Config::DeleteServerAfter, Some("0"))
-                        .await?;
-                }
-            }
-
-            if recode_avatar {
-                if let Some(avatar) = context.get_config(Config::Selfavatar).await? {
-                    let mut blob = BlobObject::new_from_path(context, avatar.as_ref()).await?;
-                    match blob.recode_to_avatar_size(context).await {
-                        Ok(()) => {
-                            context
-                                .set_config(Config::Selfavatar, Some(&avatar))
-                                .await?
-                        }
-                        Err(e) => {
-                            warn!(context, "Migrations can't recode avatar, removing. {:#}", e);
-                            context.set_config(Config::Selfavatar, None).await?
-                        }
-                    }
-                }
-            }
+        let passphrase_nonempty = !passphrase.is_empty();
+        if let Err(err) = self.try_open(context, &self.dbfile, passphrase).await {
+            self.close().await;
+            Err(err)
+        } else {
+            info!(context, "Opened database {:?}.", self.dbfile);
+            *self.is_encrypted.write().await = Some(passphrase_nonempty);
+            Ok(())
         }
-
-        info!(context, "Opened database {:?}.", dbfile);
-
-        Ok(())
     }
 
     /// Execute the given query, returning the number of affected rows.
@@ -219,10 +334,10 @@ impl Sql {
         &self,
         query: impl AsRef<str>,
         params: impl rusqlite::Params,
-    ) -> anyhow::Result<usize> {
+    ) -> Result<i64> {
         let conn = self.get_conn().await?;
         conn.execute(query.as_ref(), params)?;
-        Ok(usize::try_from(conn.last_insert_rowid())?)
+        Ok(conn.last_insert_rowid())
     }
 
     /// Prepares and executes the statement and maps a function over the resulting rows.
@@ -251,9 +366,7 @@ impl Sql {
         &self,
     ) -> Result<r2d2::PooledConnection<r2d2_sqlite::SqliteConnectionManager>> {
         let lock = self.pool.read().await;
-        let pool = lock
-            .as_ref()
-            .ok_or_else(|| format_err!("No SQL connection"))?;
+        let pool = lock.as_ref().context("no SQL connection")?;
         let conn = pool.get()?;
 
         Ok(conn)
@@ -387,6 +500,8 @@ impl Sql {
     /// will already have been logged.
     pub async fn set_raw_config(&self, key: impl AsRef<str>, value: Option<&str>) -> Result<()> {
         let key = key.as_ref();
+
+        let mut lock = self.config_cache.write().await;
         if let Some(value) = value {
             let exists = self
                 .exists(
@@ -412,12 +527,23 @@ impl Sql {
             self.execute("DELETE FROM config WHERE keyname=?;", paramsv![key])
                 .await?;
         }
+        lock.insert(key.to_string(), value.map(|s| s.to_string()));
+        drop(lock);
 
         Ok(())
     }
 
     /// Get configuration options from the database.
     pub async fn get_raw_config(&self, key: impl AsRef<str>) -> Result<Option<String>> {
+        let lock = self.config_cache.read().await;
+        let cached = lock.get(key.as_ref()).cloned();
+        drop(lock);
+
+        if let Some(c) = cached {
+            return Ok(c);
+        }
+
+        let mut lock = self.config_cache.write().await;
         let value = self
             .query_get_value(
                 "SELECT value FROM config WHERE keyname=?;",
@@ -425,6 +551,8 @@ impl Sql {
             )
             .await
             .context(format!("failed to fetch raw config: {}", key.as_ref()))?;
+        lock.insert(key.as_ref().to_string(), value.clone());
+        drop(lock);
 
         Ok(value)
     }
@@ -463,6 +591,11 @@ impl Sql {
             .await
             .map(|s| s.and_then(|r| r.parse().ok()))
     }
+
+    #[cfg(feature = "internals")]
+    pub fn config_cache(&self) -> &RwLock<HashMap<String, Option<String>>> {
+        &self.config_cache
+    }
 }
 
 pub async fn housekeeping(context: &Context) -> Result<()> {
@@ -604,6 +737,16 @@ pub async fn housekeeping(context: &Context) -> Result<()> {
 
     context.schedule_quota_update().await?;
 
+    // Try to clear the freelist to free some space on the disk. This
+    // only works if auto_vacuum is enabled.
+    if let Err(err) = context
+        .sql
+        .execute("PRAGMA incremental_vacuum", paramsv![])
+        .await
+    {
+        warn!(context, "Failed to run incremental vacuum: {}", err);
+    }
+
     if let Err(e) = context
         .set_config(Config::LastHousekeeping, Some(&time().to_string()))
         .await
@@ -666,21 +809,37 @@ async fn maybe_add_from_param(
 /// have a server UID.
 async fn prune_tombstones(sql: &Sql) -> Result<()> {
     sql.execute(
-        "DELETE FROM msgs \
-         WHERE (chat_id = ? OR hidden) \
-         AND server_uid = 0",
+        "DELETE FROM msgs
+         WHERE chat_id=?
+         AND NOT EXISTS (
+         SELECT * FROM imap WHERE msgs.rfc724_mid=rfc724_mid AND target!=''
+         )",
         paramsv![DC_CHAT_ID_TRASH],
     )
     .await?;
     Ok(())
 }
 
+/// Helper function to return comma-separated sequence of `?` chars.
+///
+/// Use this together with [`rusqlite::ParamsFromIter`] to use dynamically generated
+/// parameter lists.
+pub fn repeat_vars(count: usize) -> Result<String> {
+    if count == 0 {
+        bail!("Must have at least one repeat variable");
+    }
+    let mut s = "?,".repeat(count);
+    s.pop(); // Remove trailing comma
+    Ok(s)
+}
+
 #[cfg(test)]
-mod test {
+mod tests {
+    use async_std::channel;
     use async_std::fs::File;
 
     use crate::config::Config;
-    use crate::{test_utils::TestContext, Event, EventType};
+    use crate::{test_utils::TestContext, EventType};
 
     use super::*;
 
@@ -725,6 +884,22 @@ mod test {
         assert!(!t.ctx.sql.col_exists("foobar", "foobar").await.unwrap());
     }
 
+    /// Tests that auto_vacuum is enabled for new databases.
+    #[async_std::test]
+    async fn test_auto_vacuum() -> Result<()> {
+        let t = TestContext::new().await;
+
+        let conn = t.sql.get_conn().await?;
+        let auto_vacuum = conn.pragma_query_value(None, "auto_vacuum", |row| {
+            let auto_vacuum: i32 = row.get(0)?;
+            Ok(auto_vacuum)
+        })?;
+
+        // auto_vacuum=2 is the same as auto_vacuum=INCREMENTAL
+        assert_eq!(auto_vacuum, 2);
+        Ok(())
+    }
+
     #[async_std::test]
     async fn test_housekeeping_db_closed() {
         let t = TestContext::new().await;
@@ -741,7 +916,20 @@ mod test {
             .await
             .unwrap();
 
-        t.add_event_sink(move |event: Event| async move {
+        let (event_sink, event_source) = channel::unbounded();
+        t.add_event_sender(event_sink).await;
+
+        let a = t.get_config(Config::Selfavatar).await.unwrap().unwrap();
+        assert_eq!(avatar_bytes, &async_std::fs::read(&a).await.unwrap()[..]);
+
+        t.sql.close().await;
+        housekeeping(&t).await.unwrap_err(); // housekeeping should fail as the db is closed
+        t.sql.open(&t, "".to_string()).await.unwrap();
+
+        let a = t.get_config(Config::Selfavatar).await.unwrap().unwrap();
+        assert_eq!(avatar_bytes, &async_std::fs::read(&a).await.unwrap()[..]);
+
+        while let Ok(event) = event_source.try_recv() {
             match event.typ {
                 EventType::Info(s) => assert!(
                     !s.contains("Keeping new unreferenced file"),
@@ -751,18 +939,24 @@ mod test {
                 EventType::Error(s) => panic!("{}", s),
                 _ => {}
             }
-        })
-        .await;
+        }
+    }
 
-        let a = t.get_config(Config::Selfavatar).await.unwrap().unwrap();
-        assert_eq!(avatar_bytes, &async_std::fs::read(&a).await.unwrap()[..]);
+    /// Regression test for a bug where housekeeping deleted drafts since their
+    /// `hidden` flag is set.
+    #[async_std::test]
+    async fn test_housekeeping_dont_delete_drafts() {
+        let t = TestContext::new_alice().await;
 
-        t.sql.close().await;
-        housekeeping(&t).await.unwrap_err(); // housekeeping should fail as the db is closed
-        t.sql.open(&t, t.get_dbfile(), false).await.unwrap();
+        let chat = t.create_chat_with_contact("bob", "bob@example.com").await;
+        let mut new_draft = Message::new(Viewtype::Text);
+        new_draft.set_text(Some("This is my draft".to_string()));
+        chat.id.set_draft(&t, Some(&mut new_draft)).await.unwrap();
 
-        let a = t.get_config(Config::Selfavatar).await.unwrap().unwrap();
-        assert_eq!(avatar_bytes, &async_std::fs::read(&a).await.unwrap()[..]);
+        housekeeping(&t).await.unwrap();
+
+        let loaded_draft = chat.id.get_draft(&t).await.unwrap();
+        assert_eq!(loaded_draft.unwrap().text.unwrap(), "This is my draft");
     }
 
     /// Regression test.
@@ -787,14 +981,14 @@ mod test {
         // Create a separate empty database for testing.
         let dir = tempdir()?;
         let dbfile = dir.path().join("testdb.sqlite");
-        let sql = Sql::new();
+        let sql = Sql::new(dbfile.into());
 
         // Create database with all the tables.
-        sql.open(&t, dbfile.as_ref(), false).await.unwrap();
+        sql.open(&t, "".to_string()).await.unwrap();
         sql.close().await;
 
         // Reopen the database
-        sql.open(&t, dbfile.as_ref(), false).await?;
+        sql.open(&t, "".to_string()).await?;
         sql.execute(
             "INSERT INTO config (keyname, value) VALUES (?, ?);",
             paramsv!("foo", "bar"),
@@ -824,20 +1018,59 @@ mod test {
         assert!(!disable_server_delete);
         assert!(!recode_avatar);
 
-        info!(&t, "test_migration_flags: XXX");
+        info!(&t, "test_migration_flags: XXX END MARKER");
 
         loop {
-            if let EventType::Info(info) = t.evtracker.recv().await.unwrap() {
-                assert!(
-                    !info.contains("[migration]"),
-                    "Migrations were run twice, you probably forgot to update the db version"
-                );
-                if info.contains("test_migration_flags: XXX") {
-                    break;
+            let evt = t
+                .evtracker
+                .get_matching(|evt| matches!(evt, EventType::Info(_)))
+                .await;
+            match evt {
+                EventType::Info(msg) => {
+                    assert!(
+                        !msg.contains("[migration]"),
+                        "Migrations were run twice, you probably forgot to update the db version"
+                    );
+                    if msg.contains("test_migration_flags: XXX END MARKER") {
+                        break;
+                    }
                 }
+                _ => unreachable!(),
             }
         }
 
         Ok(())
     }
+
+    #[async_std::test]
+    async fn test_check_passphrase() -> Result<()> {
+        use tempfile::tempdir;
+
+        // The context is used only for logging.
+        let t = TestContext::new().await;
+
+        // Create a separate empty database for testing.
+        let dir = tempdir()?;
+        let dbfile = dir.path().join("testdb.sqlite");
+        let sql = Sql::new(dbfile.clone().into());
+
+        sql.check_passphrase("foo".to_string()).await?;
+        sql.open(&t, "foo".to_string())
+            .await
+            .context("failed to open the database first time")?;
+        sql.close().await;
+
+        // Reopen the database
+        let sql = Sql::new(dbfile.into());
+
+        // Test that we can't open encrypted database without a passphrase.
+        assert!(sql.open(&t, "".to_string()).await.is_err());
+
+        // Now open the database with passpharse, it should succeed.
+        sql.check_passphrase("foo".to_string()).await?;
+        sql.open(&t, "foo".to_string())
+            .await
+            .context("failed to open the database second time")?;
+        Ok(())
+    }
 }

src/sql/migrations.rs

@@ -1,6 +1,6 @@
 //! Migrations module.
 
-use anyhow::Result;
+use anyhow::{Context as _, Result};
 
 use crate::config::Config;
 use crate::constants::ShowEmails;
@@ -19,7 +19,11 @@ pub async fn run(context: &Context, sql: &Sql) -> Result<(bool, bool, bool, bool
     let mut exists_before_update = false;
     let mut dbversion_before_update = DBVERSION;
 
-    if !sql.table_exists("config").await? {
+    if !sql
+        .table_exists("config")
+        .await
+        .context("failed to check if config table exists")?
+    {
         info!(context, "First time init: creating tables",);
         sql.transaction(move |transaction| {
             transaction.execute_batch(TABLES)?;
@@ -32,6 +36,13 @@ pub async fn run(context: &Context, sql: &Sql) -> Result<(bool, bool, bool, bool
             Ok(())
         })
         .await?;
+
+        let mut lock = context.sql.config_cache.write().await;
+        lock.insert(
+            VERSION_CFG.to_string(),
+            Some(format!("{}", dbversion_before_update)),
+        );
+        drop(lock);
     } else {
         exists_before_update = true;
         dbversion_before_update = sql
@@ -497,6 +508,106 @@ item TEXT DEFAULT '');"#,
         )
         .await?;
     }
+    if dbversion < 81 {
+        info!(context, "[migration] v81");
+        sql.execute_migration("ALTER TABLE msgs ADD COLUMN hop_info TEXT;", 81)
+            .await?;
+    }
+    if dbversion < 82 {
+        info!(context, "[migration] v82");
+        sql.execute_migration(
+            r#"CREATE TABLE imap (
+id INTEGER PRIMARY KEY AUTOINCREMENT,
+rfc724_mid TEXT DEFAULT '', -- Message-ID header
+folder TEXT DEFAULT '', -- IMAP folder
+target TEXT DEFAULT '', -- Destination folder, empty to delete.
+uid INTEGER DEFAULT 0, -- UID
+uidvalidity INTEGER DEFAULT 0,
+UNIQUE (folder, uid, uidvalidity)
+);
+CREATE INDEX imap_folder ON imap(folder);
+CREATE INDEX imap_messageid ON imap(rfc724_mid);
+
+INSERT INTO imap
+(rfc724_mid, folder, target, uid, uidvalidity)
+SELECT
+rfc724_mid,
+server_folder AS folder,
+server_folder AS target,
+server_uid AS uid,
+(SELECT uidvalidity FROM imap_sync WHERE folder=server_folder) AS uidvalidity
+FROM msgs
+WHERE server_uid>0
+ON CONFLICT (folder, uid, uidvalidity)
+DO UPDATE SET rfc724_mid=excluded.rfc724_mid,
+              target=excluded.target;
+"#,
+            82,
+        )
+        .await?;
+    }
+    if dbversion < 83 {
+        info!(context, "[migration] v83");
+        sql.execute_migration(
+            "ALTER TABLE imap_sync
+             ADD COLUMN modseq -- Highest modification sequence
+             INTEGER DEFAULT 0",
+            83,
+        )
+        .await?;
+    }
+    if dbversion < 84 {
+        info!(context, "[migration] v84");
+        sql.execute_migration(
+            r#"CREATE TABLE msgs_status_updates (
+id INTEGER PRIMARY KEY AUTOINCREMENT,
+msg_id INTEGER,
+update_item TEXT DEFAULT '',
+update_item_read INTEGER DEFAULT 0);
+CREATE INDEX msgs_status_updates_index1 ON msgs_status_updates (msg_id);"#,
+            84,
+        )
+        .await?;
+    }
+    if dbversion < 85 {
+        info!(context, "[migration] v85");
+        sql.execute_migration(
+            r#"CREATE TABLE smtp (
+id INTEGER PRIMARY KEY,
+rfc724_mid TEXT NOT NULL,          -- Message-ID
+mime TEXT NOT NULL,                -- SMTP payload
+msg_id INTEGER NOT NULL,           -- ID of the message in `msgs` table
+recipients TEXT NOT NULL,          -- List of recipients separated by space
+retries INTEGER NOT NULL DEFAULT 0 -- Number of failed attempts to send the messsage
+);
+CREATE INDEX smtp_messageid ON imap(rfc724_mid);
+"#,
+            85,
+        )
+        .await?;
+    }
+    if dbversion < 86 {
+        info!(context, "[migration] v86");
+        sql.execute_migration(
+            r#"CREATE TABLE bobstate (
+                   id INTEGER PRIMARY KEY AUTOINCREMENT,
+                   invite TEXT NOT NULL,
+                   next_step INTEGER NOT NULL,
+                   chat_id INTEGER NOT NULL
+            );"#,
+            86,
+        )
+        .await?;
+    }
+    if dbversion < 87 {
+        info!(context, "[migration] v87");
+        // the index is used to speed up delete_expired_messages()
+        sql.execute_migration(
+            "CREATE INDEX IF NOT EXISTS msgs_index8 ON msgs (ephemeral_timestamp);",
+            87,
+        )
+        .await?;
+    }
 
     Ok((
         recalc_fingerprints,
@@ -524,7 +635,12 @@ impl Sql {
 
             Ok(())
         })
-        .await?;
+        .await
+        .with_context(|| format!("execute_migration failed for version {}", version))?;
+
+        let mut lock = self.config_cache.write().await;
+        lock.insert(VERSION_CFG.to_string(), Some(format!("{}", version)));
+        drop(lock);
 
         Ok(())
     }

src/sql/tables.sql

@@ -168,6 +168,14 @@ CREATE TABLE tokens (
     timestamp INTEGER DEFAULT 0
 );
 
+-- The currently running securejoin protocols, joiner-side.
+-- CREATE TABLE bobstate (
+--     id INTEGER PRIMARY KEY AUTOINCREMENT,
+--     invite TEXT NOT NULL,
+--     next_step INTEGER NOT NULL,
+--     chat_id INTEGER NOT NULL
+-- );
+
 CREATE TABLE locations (
     id INTEGER PRIMARY KEY AUTOINCREMENT,
     latitude REAL DEFAULT 0.0,

src/stock_str.rs

@@ -10,11 +10,11 @@ use strum_macros::EnumProperty;
 use crate::blob::BlobObject;
 use crate::chat::{self, Chat, ChatId, ProtectionStatus};
 use crate::config::Config;
-use crate::constants::{Viewtype, DC_CONTACT_ID_SELF};
-use crate::contact::{Contact, Origin};
+use crate::constants::DC_CONTACT_ID_SELF;
+use crate::contact::{Contact, ContactId, Origin};
 use crate::context::Context;
 use crate::dc_tools::dc_timestamp_to_str;
-use crate::message::Message;
+use crate::message::{Message, Viewtype};
 use crate::param::Param;
 use humansize::{file_size_opts, FileSize};
 
@@ -53,9 +53,6 @@ pub enum StockMessage {
     #[strum(props(fallback = "File"))]
     File = 12,
 
-    #[strum(props(fallback = "Sent with my Delta Chat Messenger: https://delta.chat"))]
-    StatusLine = 13,
-
     #[strum(props(fallback = "Group name changed from \"%1$s\" to \"%2$s\"."))]
     MsgGrpName = 15,
 
@@ -392,7 +389,7 @@ trait StockStringMods: AsRef<str> + Sized {
     fn action_by_contact<'a>(
         self,
         context: &'a Context,
-        contact_id: u32,
+        contact_id: ContactId,
     ) -> Pin<Box<dyn Future<Output = String> + Send + 'a>>
     where
         Self: Send + 'a,
@@ -455,17 +452,12 @@ pub(crate) async fn file(context: &Context) -> String {
     translated(context, StockMessage::File).await
 }
 
-/// Stock string: `Sent with my Delta Chat Messenger: https://delta.chat`.
-pub(crate) async fn status_line(context: &Context) -> String {
-    translated(context, StockMessage::StatusLine).await
-}
-
 /// Stock string: `Group name changed from "%1$s" to "%2$s".`.
 pub(crate) async fn msg_grp_name(
     context: &Context,
     from_group: impl AsRef<str>,
     to_group: impl AsRef<str>,
-    by_contact: u32,
+    by_contact: ContactId,
 ) -> String {
     translated(context, StockMessage::MsgGrpName)
         .await
@@ -476,7 +468,7 @@ pub(crate) async fn msg_grp_name(
 }
 
 /// Stock string: `Group image changed.`.
-pub(crate) async fn msg_grp_img_changed(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn msg_grp_img_changed(context: &Context, by_contact: ContactId) -> String {
     translated(context, StockMessage::MsgGrpImgChanged)
         .await
         .action_by_contact(context, by_contact)
@@ -490,7 +482,7 @@ pub(crate) async fn msg_grp_img_changed(context: &Context, by_contact: u32) -> S
 pub(crate) async fn msg_add_member(
     context: &Context,
     added_member_addr: impl AsRef<str>,
-    by_contact: u32,
+    by_contact: ContactId,
 ) -> String {
     let addr = added_member_addr.as_ref();
     let who = match Contact::lookup_id_by_addr(context, addr, Origin::Unknown).await {
@@ -514,7 +506,7 @@ pub(crate) async fn msg_add_member(
 pub(crate) async fn msg_del_member(
     context: &Context,
     removed_member_addr: impl AsRef<str>,
-    by_contact: u32,
+    by_contact: ContactId,
 ) -> String {
     let addr = removed_member_addr.as_ref();
     let who = match Contact::lookup_id_by_addr(context, addr, Origin::Unknown).await {
@@ -532,7 +524,7 @@ pub(crate) async fn msg_del_member(
 }
 
 /// Stock string: `Group left.`.
-pub(crate) async fn msg_group_left(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn msg_group_left(context: &Context, by_contact: ContactId) -> String {
     translated(context, StockMessage::MsgGroupLeft)
         .await
         .action_by_contact(context, by_contact)
@@ -582,7 +574,7 @@ pub(crate) async fn read_rcpt_mail_body(context: &Context, message: impl AsRef<s
 }
 
 /// Stock string: `Group image deleted.`.
-pub(crate) async fn msg_grp_img_deleted(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn msg_grp_img_deleted(context: &Context, by_contact: ContactId) -> String {
     translated(context, StockMessage::MsgGrpImgDeleted)
         .await
         .action_by_contact(context, by_contact)
@@ -595,7 +587,10 @@ pub(crate) async fn e2e_preferred(context: &Context) -> String {
 }
 
 /// Stock string: `%1$s invited you to join this group. Waiting for the device of %2$s to reply…`.
-pub(crate) async fn secure_join_started(context: &Context, inviter_contact_id: u32) -> String {
+pub(crate) async fn secure_join_started(
+    context: &Context,
+    inviter_contact_id: ContactId,
+) -> String {
     if let Ok(contact) = Contact::get_by_id(context, inviter_contact_id).await {
         translated(context, StockMessage::SecureJoinStarted)
             .await
@@ -610,7 +605,7 @@ pub(crate) async fn secure_join_started(context: &Context, inviter_contact_id: u
 }
 
 /// Stock string: `%1$s replied, waiting for being added to the group…`.
-pub(crate) async fn secure_join_replies(context: &Context, contact_id: u32) -> String {
+pub(crate) async fn secure_join_replies(context: &Context, contact_id: ContactId) -> String {
     if let Ok(contact) = Contact::get_by_id(context, contact_id).await {
         translated(context, StockMessage::SecureJoinReplies)
             .await
@@ -644,20 +639,19 @@ pub(crate) async fn secure_join_group_qr_description(context: &Context, chat: &C
 }
 
 /// Stock string: `%1$s verified.`.
-pub(crate) async fn contact_verified(context: &Context, contact_addr: impl AsRef<str>) -> String {
+pub(crate) async fn contact_verified(context: &Context, contact: &Contact) -> String {
+    let addr = contact.get_name_n_addr();
     translated(context, StockMessage::ContactVerified)
         .await
-        .replace1(contact_addr)
+        .replace1(addr)
 }
 
 /// Stock string: `Cannot verify %1$s`.
-pub(crate) async fn contact_not_verified(
-    context: &Context,
-    contact_addr: impl AsRef<str>,
-) -> String {
+pub(crate) async fn contact_not_verified(context: &Context, contact: &Contact) -> String {
+    let addr = contact.get_name_n_addr();
     translated(context, StockMessage::ContactNotVerified)
         .await
-        .replace1(contact_addr)
+        .replace1(addr)
 }
 
 /// Stock string: `Changed setup for %1$s`.
@@ -727,7 +721,7 @@ pub(crate) async fn msg_location_enabled(context: &Context) -> String {
 }
 
 /// Stock string: `Location streaming enabled by ...`.
-pub(crate) async fn msg_location_enabled_by(context: &Context, contact: u32) -> String {
+pub(crate) async fn msg_location_enabled_by(context: &Context, contact: ContactId) -> String {
     translated(context, StockMessage::MsgLocationEnabled)
         .await
         .action_by_contact(context, contact)
@@ -793,7 +787,10 @@ pub(crate) async fn failed_sending_to(context: &Context, name: impl AsRef<str>)
 }
 
 /// Stock string: `Message deletion timer is disabled.`.
-pub(crate) async fn msg_ephemeral_timer_disabled(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn msg_ephemeral_timer_disabled(
+    context: &Context,
+    by_contact: ContactId,
+) -> String {
     translated(context, StockMessage::MsgEphemeralTimerDisabled)
         .await
         .action_by_contact(context, by_contact)
@@ -804,7 +801,7 @@ pub(crate) async fn msg_ephemeral_timer_disabled(context: &Context, by_contact:
 pub(crate) async fn msg_ephemeral_timer_enabled(
     context: &Context,
     timer: impl AsRef<str>,
-    by_contact: u32,
+    by_contact: ContactId,
 ) -> String {
     translated(context, StockMessage::MsgEphemeralTimerEnabled)
         .await
@@ -814,7 +811,7 @@ pub(crate) async fn msg_ephemeral_timer_enabled(
 }
 
 /// Stock string: `Message deletion timer is set to 1 minute.`.
-pub(crate) async fn msg_ephemeral_timer_minute(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn msg_ephemeral_timer_minute(context: &Context, by_contact: ContactId) -> String {
     translated(context, StockMessage::MsgEphemeralTimerMinute)
         .await
         .action_by_contact(context, by_contact)
@@ -822,7 +819,7 @@ pub(crate) async fn msg_ephemeral_timer_minute(context: &Context, by_contact: u3
 }
 
 /// Stock string: `Message deletion timer is set to 1 hour.`.
-pub(crate) async fn msg_ephemeral_timer_hour(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn msg_ephemeral_timer_hour(context: &Context, by_contact: ContactId) -> String {
     translated(context, StockMessage::MsgEphemeralTimerHour)
         .await
         .action_by_contact(context, by_contact)
@@ -830,7 +827,7 @@ pub(crate) async fn msg_ephemeral_timer_hour(context: &Context, by_contact: u32)
 }
 
 /// Stock string: `Message deletion timer is set to 1 day.`.
-pub(crate) async fn msg_ephemeral_timer_day(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn msg_ephemeral_timer_day(context: &Context, by_contact: ContactId) -> String {
     translated(context, StockMessage::MsgEphemeralTimerDay)
         .await
         .action_by_contact(context, by_contact)
@@ -838,7 +835,7 @@ pub(crate) async fn msg_ephemeral_timer_day(context: &Context, by_contact: u32)
 }
 
 /// Stock string: `Message deletion timer is set to 1 week.`.
-pub(crate) async fn msg_ephemeral_timer_week(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn msg_ephemeral_timer_week(context: &Context, by_contact: ContactId) -> String {
     translated(context, StockMessage::MsgEphemeralTimerWeek)
         .await
         .action_by_contact(context, by_contact)
@@ -883,7 +880,7 @@ pub(crate) async fn error_no_network(context: &Context) -> String {
 }
 
 /// Stock string: `Chat protection enabled.`.
-pub(crate) async fn protection_enabled(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn protection_enabled(context: &Context, by_contact: ContactId) -> String {
     translated(context, StockMessage::ProtectionEnabled)
         .await
         .action_by_contact(context, by_contact)
@@ -891,7 +888,7 @@ pub(crate) async fn protection_enabled(context: &Context, by_contact: u32) -> St
 }
 
 /// Stock string: `Chat protection disabled.`.
-pub(crate) async fn protection_disabled(context: &Context, by_contact: u32) -> String {
+pub(crate) async fn protection_disabled(context: &Context, by_contact: ContactId) -> String {
     translated(context, StockMessage::ProtectionDisabled)
         .await
         .action_by_contact(context, by_contact)
@@ -917,7 +914,7 @@ pub(crate) async fn delete_server_turned_off(context: &Context) -> String {
 pub(crate) async fn msg_ephemeral_timer_minutes(
     context: &Context,
     minutes: impl AsRef<str>,
-    by_contact: u32,
+    by_contact: ContactId,
 ) -> String {
     translated(context, StockMessage::MsgEphemeralTimerMinutes)
         .await
@@ -930,7 +927,7 @@ pub(crate) async fn msg_ephemeral_timer_minutes(
 pub(crate) async fn msg_ephemeral_timer_hours(
     context: &Context,
     hours: impl AsRef<str>,
-    by_contact: u32,
+    by_contact: ContactId,
 ) -> String {
     translated(context, StockMessage::MsgEphemeralTimerHours)
         .await
@@ -943,7 +940,7 @@ pub(crate) async fn msg_ephemeral_timer_hours(
 pub(crate) async fn msg_ephemeral_timer_days(
     context: &Context,
     days: impl AsRef<str>,
-    by_contact: u32,
+    by_contact: ContactId,
 ) -> String {
     translated(context, StockMessage::MsgEphemeralTimerDays)
         .await
@@ -956,7 +953,7 @@ pub(crate) async fn msg_ephemeral_timer_days(
 pub(crate) async fn msg_ephemeral_timer_weeks(
     context: &Context,
     weeks: impl AsRef<str>,
-    by_contact: u32,
+    by_contact: ContactId,
 ) -> String {
     translated(context, StockMessage::MsgEphemeralTimerWeeks)
         .await
@@ -1113,7 +1110,7 @@ impl Context {
     pub(crate) async fn stock_protection_msg(
         &self,
         protect: ProtectionStatus,
-        from_id: u32,
+        from_id: ContactId,
     ) -> String {
         match protect {
             ProtectionStatus::Unprotected => protection_enabled(self, from_id).await,
@@ -1205,8 +1202,15 @@ mod tests {
     #[async_std::test]
     async fn test_stock_string_repl_str() {
         let t = TestContext::new().await;
+        let contact_id = Contact::create(&t.ctx, "Someone", "someone@example.org")
+            .await
+            .unwrap();
+        let contact = Contact::load_from_db(&t.ctx, contact_id).await.unwrap();
         // uses %1$s substitution
-        assert_eq!(contact_verified(&t, "Foo").await, "Foo verified.");
+        assert_eq!(
+            contact_verified(&t, &contact).await,
+            "Someone (someone@example.org) verified."
+        );
         // We have no string using %1$d to test...
     }
 
@@ -1229,20 +1233,20 @@ mod tests {
     async fn test_stock_system_msg_add_member_by_me() {
         let t = TestContext::new().await;
         assert_eq!(
-            msg_add_member(&t, "alice@example.com", DC_CONTACT_ID_SELF).await,
-            "Member alice@example.com added by me."
+            msg_add_member(&t, "alice@example.org", DC_CONTACT_ID_SELF).await,
+            "Member alice@example.org added by me."
         )
     }
 
     #[async_std::test]
     async fn test_stock_system_msg_add_member_by_me_with_displayname() {
         let t = TestContext::new().await;
-        Contact::create(&t, "Alice", "alice@example.com")
+        Contact::create(&t, "Alice", "alice@example.org")
             .await
             .expect("failed to create contact");
         assert_eq!(
-            msg_add_member(&t, "alice@example.com", DC_CONTACT_ID_SELF).await,
-            "Member Alice (alice@example.com) added by me."
+            msg_add_member(&t, "alice@example.org", DC_CONTACT_ID_SELF).await,
+            "Member Alice (alice@example.org) added by me."
         );
     }
 
@@ -1250,7 +1254,7 @@ mod tests {
     async fn test_stock_system_msg_add_member_by_other_with_displayname() {
         let t = TestContext::new().await;
         let contact_id = {
-            Contact::create(&t, "Alice", "alice@example.com")
+            Contact::create(&t, "Alice", "alice@example.org")
                 .await
                 .expect("Failed to create contact Alice");
             Contact::create(&t, "Bob", "bob@example.com")
@@ -1258,8 +1262,8 @@ mod tests {
                 .expect("failed to create bob")
         };
         assert_eq!(
-            msg_add_member(&t, "alice@example.com", contact_id,).await,
-            "Member Alice (alice@example.com) added by Bob (bob@example.com)."
+            msg_add_member(&t, "alice@example.org", contact_id,).await,
+            "Member Alice (alice@example.org) added by Bob (bob@example.com)."
         );
     }
 
@@ -1288,11 +1292,13 @@ mod tests {
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
         assert_eq!(chats.len(), 2);
 
-        let chat0 = Chat::load_from_db(&t, chats.get_chat_id(0)).await.unwrap();
+        let chat0 = Chat::load_from_db(&t, chats.get_chat_id(0).unwrap())
+            .await
+            .unwrap();
         let (self_talk_id, device_chat_id) = if chat0.is_self_talk() {
-            (chats.get_chat_id(0), chats.get_chat_id(1))
+            (chats.get_chat_id(0).unwrap(), chats.get_chat_id(1).unwrap())
         } else {
-            (chats.get_chat_id(1), chats.get_chat_id(0))
+            (chats.get_chat_id(1).unwrap(), chats.get_chat_id(0).unwrap())
         };
 
         // delete self-talk first; this adds a message to device-chat about how self-talk can be restored

src/summary.rs

@@ -1,11 +1,11 @@
 //! # Message summary for chatlist.
 
 use crate::chat::Chat;
-use crate::constants::{Chattype, Viewtype, DC_CONTACT_ID_SELF};
+use crate::constants::{Chattype, DC_CONTACT_ID_SELF};
 use crate::contact::Contact;
 use crate::context::Context;
 use crate::dc_tools::dc_truncate;
-use crate::message::{Message, MessageState};
+use crate::message::{Message, MessageState, Viewtype};
 use crate::mimeparser::SystemMessage;
 use crate::param::Param;
 use crate::stock_str;
@@ -137,7 +137,14 @@ impl Message {
                 append_text = false;
                 stock_str::videochat_invitation(context).await
             }
-            _ => {
+            Viewtype::Webxdc => {
+                append_text = true;
+                self.get_webxdc_info(context)
+                    .await
+                    .map(|info| info.name)
+                    .unwrap_or_else(|_| "ErrWebxdcName".to_string())
+            }
+            Viewtype::Text | Viewtype::Unknown => {
                 if self.param.get_cmd() != SystemMessage::LocationOnly {
                     "".to_string()
                 } else {

src/sync.rs

@@ -2,10 +2,10 @@
 
 use crate::chat::{Chat, ChatId};
 use crate::config::Config;
-use crate::constants::{Blocked, Viewtype, DC_CONTACT_ID_SELF};
+use crate::constants::{Blocked, DC_CONTACT_ID_SELF};
 use crate::context::Context;
 use crate::dc_tools::time;
-use crate::message::{Message, MsgId};
+use crate::message::{Message, MsgId, Viewtype};
 use crate::mimeparser::SystemMessage;
 use crate::param::Param;
 use crate::sync::SyncData::{AddQrToken, DeleteQrToken};

src/test_utils.rs

@@ -2,76 +2,161 @@
 //!
 //! This private module is only compiled for test runs.
 
+use std::collections::BTreeMap;
 use std::ops::Deref;
-use std::str::FromStr;
+use std::panic;
+use std::thread;
 use std::time::{Duration, Instant};
-use std::{collections::BTreeMap, panic};
-use std::{fmt, thread};
 
 use ansi_term::Color;
-use async_std::channel::Receiver;
-use async_std::path::PathBuf;
+use async_std::channel::{self, Receiver, Sender};
+use async_std::prelude::*;
 use async_std::sync::{Arc, RwLock};
-use async_std::{channel, pin::Pin};
-use async_std::{future::Future, task};
+use async_std::task;
 use chat::ChatItem;
 use once_cell::sync::Lazy;
+use rand::Rng;
 use tempfile::{tempdir, TempDir};
 
 use crate::chat::{self, Chat, ChatId};
 use crate::chatlist::Chatlist;
 use crate::config::Config;
 use crate::constants::Chattype;
-use crate::constants::{Viewtype, DC_CONTACT_ID_SELF, DC_MSG_ID_DAYMARKER, DC_MSG_ID_MARKER1};
-use crate::contact::{Contact, Origin};
+use crate::constants::{DC_CONTACT_ID_SELF, DC_MSG_ID_DAYMARKER, DC_MSG_ID_MARKER1};
+use crate::contact::{Contact, Modifier, Origin};
 use crate::context::Context;
 use crate::dc_receive_imf::dc_receive_imf;
 use crate::dc_tools::EmailAddress;
 use crate::events::{Event, EventType};
-use crate::job::Action;
-use crate::key::{self, DcKey};
-use crate::message::{update_msg_state, Message, MessageState, MsgId};
+use crate::key::{self, DcKey, KeyPair, KeyPairUse};
+use crate::message::{update_msg_state, Message, MessageState, MsgId, Viewtype};
 use crate::mimeparser::MimeMessage;
-use crate::param::{Param, Params};
 
 #[allow(non_upper_case_globals)]
 pub const AVATAR_900x900_BYTES: &[u8] = include_bytes!("../test-data/image/avatar900x900.png");
 
-type EventSink =
-    dyn Fn(Event) -> Pin<Box<dyn Future<Output = ()> + Send + 'static>> + Send + Sync + 'static;
-
 /// Map of [`Context::id`] to names for [`TestContext`]s.
 static CONTEXT_NAMES: Lazy<std::sync::RwLock<BTreeMap<u32, String>>> =
     Lazy::new(|| std::sync::RwLock::new(BTreeMap::new()));
 
+pub struct TestContextManager {
+    log_tx: Sender<Event>,
+    _log_sink: LogSink,
+}
+
+impl TestContextManager {
+    pub async fn new() -> Self {
+        let (log_tx, _log_sink) = LogSink::create();
+        Self { log_tx, _log_sink }
+    }
+
+    pub async fn alice(&mut self) -> TestContext {
+        TestContext::builder()
+            .configure_alice()
+            .with_log_sink(self.log_tx.clone())
+            .build()
+            .await
+    }
+
+    pub async fn bob(&mut self) -> TestContext {
+        TestContext::builder()
+            .configure_bob()
+            .with_log_sink(self.log_tx.clone())
+            .build()
+            .await
+    }
+}
+
+#[derive(Debug, Clone, Default)]
+pub struct TestContextBuilder {
+    key_pair: Option<KeyPair>,
+    log_sink: Option<Sender<Event>>,
+}
+
+impl TestContextBuilder {
+    /// Configures as alice@example.org with fixed secret key.
+    ///
+    /// This is a shortcut for `.with_key_pair(alice_keypair()).
+    pub fn configure_alice(self) -> Self {
+        self.with_key_pair(alice_keypair())
+    }
+
+    /// Configures as bob@example.net with fixed secret key.
+    ///
+    /// This is a shortcut for `.with_key_pair(bob_keypair()).
+    pub fn configure_bob(self) -> Self {
+        self.with_key_pair(bob_keypair())
+    }
+
+    /// Configures the new [`TestContext`] with the provided [`KeyPair`].
+    ///
+    /// This will extract the email address from the key and configure the context with the
+    /// given identity.
+    pub fn with_key_pair(mut self, key_pair: KeyPair) -> Self {
+        self.key_pair = Some(key_pair);
+        self
+    }
+
+    /// Attaches a [`LogSink`] to this [`TestContext`].
+    ///
+    /// This is useful when using multiple [`TestContext`] instances in one test: it allows
+    /// using a single [`LogSink`] for both contexts.  This shows the log messages in
+    /// sequence as they occurred rather than all messages from each context in a single
+    /// block.
+    pub fn with_log_sink(mut self, sink: Sender<Event>) -> Self {
+        self.log_sink = Some(sink);
+        self
+    }
+
+    /// Builds the [`TestContext`].
+    pub async fn build(self) -> TestContext {
+        let name = self.key_pair.as_ref().map(|key| key.addr.local.clone());
+
+        let test_context = TestContext::new_internal(name, self.log_sink).await;
+
+        if let Some(key_pair) = self.key_pair {
+            test_context
+                .configure_addr(&key_pair.addr.to_string())
+                .await;
+            key::store_self_keypair(&test_context, &key_pair, KeyPairUse::Default)
+                .await
+                .expect("Failed to save key");
+        }
+        test_context
+    }
+}
+
 /// A Context and temporary directory.
 ///
 /// The temporary directory can be used to store the SQLite database,
 /// see e.g. [test_context] which does this.
-pub(crate) struct TestContext {
+#[derive(Debug)]
+pub struct TestContext {
     pub ctx: Context,
     pub dir: TempDir,
-    pub evtracker: EvTracker,
-    /// Counter for fake IMAP UIDs in [recv_msg], for private use in that function only.
-    recv_idx: RwLock<u32>,
-    /// Functions to call for events received.
-    event_sinks: Arc<RwLock<Vec<Box<EventSink>>>>,
+    pub evtracker: EventTracker,
+    /// Channels which should receive events from this context.
+    event_senders: Arc<RwLock<Vec<Sender<Event>>>>,
     /// Receives panics from sinks ("sink" means "event handler" here)
-    poison_receiver: channel::Receiver<String>,
-}
-
-impl fmt::Debug for TestContext {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        f.debug_struct("TestContext")
-            .field("ctx", &self.ctx)
-            .field("dir", &self.dir)
-            .field("recv_idx", &self.recv_idx)
-            .field("event_sinks", &String::from("Vec<EventSink>"))
-            .finish()
-    }
+    poison_receiver: Receiver<String>,
+    /// Reference to implicit [`LogSink`] so it is dropped together with the context.
+    ///
+    /// Only used if no explicit `log_sender` is passed into [`TestContext::new_internal`]
+    /// (which is assumed to be the sending end of a [`LogSink`]).
+    ///
+    /// This is a convenience in case only a single [`TestContext`] is used to avoid dealing
+    /// with [`LogSink`].  Never read, thus "dead code", since the only purpose is to
+    /// control when Drop is invoked.
+    #[allow(dead_code)]
+    log_sink: Option<LogSink>,
 }
 
 impl TestContext {
+    /// Returns the builder to have more control over creating the context.
+    pub fn builder() -> TestContextBuilder {
+        TestContextBuilder::default()
+    }
+
     /// Creates a new [`TestContext`].
     ///
     /// The [Context] will be created and have an SQLite database named "db.sqlite" in the
@@ -80,18 +165,33 @@ impl TestContext {
     ///
     /// [Context]: crate::context::Context
     pub async fn new() -> Self {
-        Self::new_named(None).await
+        Self::new_internal(None, None).await
     }
 
-    /// Creates a new [`TestContext`] with a set name used in event logging.
-    pub async fn with_name(name: impl Into<String>) -> Self {
-        Self::new_named(Some(name.into())).await
+    /// Creates a new configured [`TestContext`].
+    ///
+    /// This is a shortcut which automatically calls [`TestContext::configure_alice`] after
+    /// creating the context.
+    pub async fn new_alice() -> Self {
+        Self::builder().configure_alice().build().await
     }
 
-    async fn new_named(name: Option<String>) -> Self {
-        use rand::Rng;
-        pretty_env_logger::try_init().ok();
+    /// Creates a new configured [`TestContext`].
+    ///
+    /// This is a shortcut which configures bob@example.net with a fixed key.
+    pub async fn new_bob() -> Self {
+        Self::builder().configure_bob().build().await
+    }
 
+    /// Internal constructor.
+    ///
+    /// `name` is used to identify this context in e.g. log output.  This is useful mostly
+    /// when you have multiple [`TestContext`]s in a test.
+    ///
+    /// `log_sender` is assumed to be the sender for a [`LogSink`].  If not supplied a new
+    /// [`LogSink`] will be created so that events are logged to this test when the
+    /// [`TestContext`] is dropped.
+    async fn new_internal(name: Option<String>, log_sender: Option<Sender<Event>>) -> Self {
         let dir = tempdir().unwrap();
         let dbfile = dir.path().join("db.sqlite");
         let id = rand::thread_rng().gen();
@@ -99,18 +199,26 @@ impl TestContext {
             let mut context_names = CONTEXT_NAMES.write().unwrap();
             context_names.insert(id, name);
         }
-        let ctx = Context::new("FakeOS".into(), dbfile.into(), id)
+        let ctx = Context::new(dbfile.into(), id)
             .await
             .expect("failed to create context");
 
         let events = ctx.get_event_emitter();
 
-        let event_sinks: Arc<RwLock<Vec<Box<EventSink>>>> = Arc::new(RwLock::new(Vec::new()));
-        let sinks = Arc::clone(&event_sinks);
-        let (poison_sender, poison_receiver) = channel::bounded(1);
-        let (evtracker_sender, evtracker_receiver) = channel::unbounded();
+        let (log_sender, log_sink) = match log_sender {
+            Some(sender) => (sender, None),
+            None => {
+                let (sender, sink) = LogSink::create();
+                (sender, Some(sink))
+            }
+        };
 
-        async_std::task::spawn(async move {
+        let (evtracker_sender, evtracker_receiver) = channel::unbounded();
+        let event_senders = Arc::new(RwLock::new(vec![log_sender, evtracker_sender]));
+        let senders = Arc::clone(&event_senders);
+        let (poison_sender, poison_receiver) = channel::bounded(1);
+
+        task::spawn(async move {
             // Make sure that the test fails if there is a panic on this thread here
             // (but not if there is a panic on another thread)
             let looptask_id = task::current().id();
@@ -126,50 +234,26 @@ impl TestContext {
 
             while let Some(event) = events.recv().await {
                 {
-                    log::debug!("{:?}", event);
-                    let sinks = sinks.read().await;
-                    for sink in sinks.iter() {
-                        sink(event.clone()).await;
+                    let sinks = senders.read().await;
+                    for sender in sinks.iter() {
+                        // Don't block because someone wanted to use a oneshot receiver, use
+                        // an unbounded channel if you want all events.
+                        sender.try_send(event.clone()).ok();
                     }
                 }
-                receive_event(&event);
-                evtracker_sender.send(event.typ).await.ok();
             }
         });
 
         Self {
             ctx,
             dir,
-            evtracker: EvTracker(evtracker_receiver),
-            recv_idx: RwLock::new(0),
-            event_sinks,
+            evtracker: EventTracker(evtracker_receiver),
+            event_senders,
             poison_receiver,
+            log_sink,
         }
     }
 
-    /// Creates a new configured [`TestContext`].
-    ///
-    /// This is a shortcut which automatically calls [`TestContext::configure_alice`] after
-    /// creating the context.
-    pub async fn new_alice() -> Self {
-        let t = Self::with_name("alice").await;
-        t.configure_alice().await;
-        t
-    }
-
-    /// Creates a new configured [`TestContext`].
-    ///
-    /// This is a shortcut which configures bob@example.net with a fixed key.
-    pub async fn new_bob() -> Self {
-        let t = Self::with_name("bob").await;
-        let keypair = bob_keypair();
-        t.configure_addr(&keypair.addr.to_string()).await;
-        key::store_self_keypair(&t, &keypair, key::KeyPairUse::Default)
-            .await
-            .expect("Failed to save Bob's key");
-        t
-    }
-
     /// Sets a name for this [`TestContext`] if one isn't yet set.
     ///
     /// This will show up in events logged in the test output.
@@ -180,32 +264,12 @@ impl TestContext {
             .or_insert_with(|| name.into());
     }
 
-    /// Add a new callback which will receive events.
+    /// Adds a new [`Event`]s sender.
     ///
-    /// The test context runs an async task receiving all events from the [`Context`], which
-    /// are logged to stdout.  This allows you to register additional callbacks which will
-    /// receive all events in case your tests need to watch for a specific event.
-    pub async fn add_event_sink<F, R>(&self, sink: F)
-    where
-        // Aka `F: EventSink` but type aliases are not allowed.
-        F: Fn(Event) -> R + Send + Sync + 'static,
-        R: Future<Output = ()> + Send + 'static,
-    {
-        let mut sinks = self.event_sinks.write().await;
-        sinks.push(Box::new(move |evt| Box::pin(sink(evt))));
-    }
-
-    /// Configure with alice@example.com.
-    ///
-    /// The context will be fake-configured as the alice user, with a pre-generated secret
-    /// key.  The email address of the user is returned as a string.
-    pub async fn configure_alice(&self) -> String {
-        let keypair = alice_keypair();
-        self.configure_addr(&keypair.addr.to_string()).await;
-        key::store_self_keypair(&self.ctx, &keypair, key::KeyPairUse::Default)
-            .await
-            .expect("Failed to save Alice's key");
-        keypair.addr.to_string()
+    /// Once added, all events emitted by this context will be sent to this channel.  This
+    /// is useful if you need to wait for events or make assertions on them.
+    pub async fn add_event_sender(&self, sink: Sender<Event>) {
+        self.event_senders.write().await.push(sink)
     }
 
     /// Configure as a given email address.
@@ -235,27 +299,27 @@ impl TestContext {
     /// Panics if there is no message or on any error.
     pub async fn pop_sent_msg(&self) -> SentMessage {
         let start = Instant::now();
-        let (rowid, foreign_id, raw_params) = loop {
+        let (rowid, msg_id, payload, recipients) = loop {
             let row = self
                 .ctx
                 .sql
-                .query_row(
+                .query_row_optional(
                     r#"
-                    SELECT id, foreign_id, param
-                      FROM jobs
-                     WHERE action=?
-                  ORDER BY desired_timestamp DESC;
-                "#,
-                    paramsv![Action::SendMsgToSmtp],
+                    SELECT id, msg_id, mime, recipients
+                    FROM smtp
+                    ORDER BY id DESC"#,
+                    paramsv![],
                     |row| {
-                        let id: u32 = row.get(0)?;
-                        let foreign_id: u32 = row.get(1)?;
-                        let param: String = row.get(2)?;
-                        Ok((id, foreign_id, param))
+                        let rowid: i64 = row.get(0)?;
+                        let msg_id: MsgId = row.get(1)?;
+                        let mime: String = row.get(2)?;
+                        let recipients: String = row.get(3)?;
+                        Ok((rowid, msg_id, mime, recipients))
                     },
                 )
-                .await;
-            if let Ok(row) = row {
+                .await
+                .expect("query_row_optional failed");
+            if let Some(row) = row {
                 break row;
             }
             if start.elapsed() < Duration::from_secs(3) {
@@ -264,24 +328,18 @@ impl TestContext {
                 panic!("no sent message found in jobs table");
             }
         };
-        let id = MsgId::new(foreign_id);
-        let params = Params::from_str(&raw_params).unwrap();
-        let blob_path = params
-            .get_blob(Param::File, &self.ctx, false)
-            .await
-            .expect("failed to parse blob from param")
-            .expect("no Param::File found in Params")
-            .to_abs_path();
         self.ctx
             .sql
             .execute("DELETE FROM jobs WHERE id=?;", paramsv![rowid])
             .await
             .expect("failed to remove job");
-        update_msg_state(&self.ctx, id, MessageState::OutDelivered).await;
+        update_msg_state(&self.ctx, msg_id, MessageState::OutDelivered)
+            .await
+            .expect("failed to update message state");
         SentMessage {
-            params,
-            blob_path,
-            sender_msg_id: id,
+            payload,
+            sender_msg_id: msg_id,
+            recipients,
         }
     }
 
@@ -302,13 +360,11 @@ impl TestContext {
     ///
     /// Receives a message using the `dc_receive_imf()` pipeline.
     pub async fn recv_msg(&self, msg: &SentMessage) {
-        let mut idx = self.recv_idx.write().await;
-        *idx += 1;
         let received_msg =
             "Received: (Postfix, from userid 1000); Mon, 4 Dec 2006 14:51:39 +0100 (CET)\n"
                 .to_owned()
-                + &msg.payload();
-        dc_receive_imf(&self.ctx, received_msg.as_bytes(), "INBOX", *idx, false)
+                + msg.payload();
+        dc_receive_imf(&self.ctx, received_msg.as_bytes(), "INBOX", false)
             .await
             .unwrap();
     }
@@ -343,31 +399,56 @@ impl TestContext {
             .expect("failed to load msg")
     }
 
+    /// Returns the [`Contact`] for the other [`TestContext`], creating it if necessary.
+    pub async fn add_or_lookup_contact(&self, other: &TestContext) -> Contact {
+        let name = other
+            .ctx
+            .get_config(Config::Displayname)
+            .await
+            .unwrap_or_default()
+            .unwrap_or_default();
+        let addr = other
+            .ctx
+            .get_config(Config::ConfiguredAddr)
+            .await
+            .unwrap()
+            .unwrap();
+        // MailinglistAddress is the lowest allowed origin, we'd prefer to not modify the
+        // origin when creating this contact.
+        let (contact_id, modified) =
+            Contact::add_or_lookup(self, &name, &addr, Origin::MailinglistAddress)
+                .await
+                .unwrap();
+        match modified {
+            Modifier::None => (),
+            Modifier::Modified => warn!(&self.ctx, "Contact {} modified by TestContext", &addr),
+            Modifier::Created => warn!(&self.ctx, "Contact {} created by TestContext", &addr),
+        }
+        Contact::load_from_db(&self.ctx, contact_id).await.unwrap()
+    }
+
+    /// Returns 1:1 [`Chat`] with another account, if it exists.
+    ///
+    /// This first creates a contact using the configured details on the other account, then
+    /// creates a 1:1 chat with this contact.
+    pub async fn get_chat(&self, other: &TestContext) -> Option<Chat> {
+        let contact = self.add_or_lookup_contact(other).await;
+        match ChatId::lookup_by_contact(&self.ctx, contact.id)
+            .await
+            .unwrap()
+        {
+            Some(id) => Some(Chat::load_from_db(&self.ctx, id).await.unwrap()),
+            None => None,
+        }
+    }
+
     /// Creates or returns an existing 1:1 [`Chat`] with another account.
     ///
     /// This first creates a contact using the configured details on the other account, then
     /// creates a 1:1 chat with this contact.
     pub async fn create_chat(&self, other: &TestContext) -> Chat {
-        let (contact_id, _modified) = Contact::add_or_lookup(
-            self,
-            &other
-                .ctx
-                .get_config(Config::Displayname)
-                .await
-                .unwrap_or_default()
-                .unwrap_or_default(),
-            &other
-                .ctx
-                .get_config(Config::ConfiguredAddr)
-                .await
-                .unwrap()
-                .unwrap(),
-            Origin::ManuallyCreated,
-        )
-        .await
-        .unwrap();
-
-        let chat_id = ChatId::create_for_contact(self, contact_id).await.unwrap();
+        let contact = self.add_or_lookup_contact(other).await;
+        let chat_id = ChatId::create_for_contact(self, contact.id).await.unwrap();
         Chat::load_from_db(self, chat_id).await.unwrap()
     }
 
@@ -514,14 +595,46 @@ impl Drop for TestContext {
     }
 }
 
+/// A receiver of [`Event`]s which will log the events to the captured test stdout.
+///
+/// Tests redirect the stdout of the test thread and capture this, showing the captured
+/// stdout if the test fails.  This means printing log messages must be done on the thread
+/// of the test itself and not from a spawned task.
+///
+/// This sink achieves this by printing the events, in the order received, at the time it is
+/// dropped.  Thus to use you must only make sure this sink is dropped in the test itself.
+///
+/// To use this create an instance using [`LogSink::create`] and then use the
+/// [`TestContextBuilder::with_log_sink`].
+#[derive(Debug)]
+pub struct LogSink {
+    events: Receiver<Event>,
+}
+
+impl LogSink {
+    /// Creates a new [`LogSink`] and returns the attached event sink.
+    pub fn create() -> (Sender<Event>, Self) {
+        let (tx, rx) = channel::unbounded();
+        (tx, Self { events: rx })
+    }
+}
+
+impl Drop for LogSink {
+    fn drop(&mut self) {
+        while let Ok(event) = self.events.try_recv() {
+            print_event(&event);
+        }
+    }
+}
+
 /// A raw message as it was scheduled to be sent.
 ///
 /// This is a raw message, probably in the shape DC was planning to send it but not having
 /// passed through a SMTP-IMAP pipeline.
 #[derive(Debug, Clone)]
 pub struct SentMessage {
-    params: Params,
-    blob_path: PathBuf,
+    payload: String,
+    recipients: String,
     pub sender_msg_id: MsgId,
 }
 
@@ -530,33 +643,34 @@ impl SentMessage {
     ///
     /// If there are multiple recipients this is just a random one, so is not very useful.
     pub fn recipient(&self) -> EmailAddress {
-        let raw = self
-            .params
-            .get(Param::Recipients)
-            .expect("no recipients in params");
-        let rcpt = raw.split(' ').next().expect("no recipient found");
+        let rcpt = self
+            .recipients
+            .split(' ')
+            .next()
+            .expect("no recipient found");
         rcpt.parse().expect("failed to parse email address")
     }
 
     /// The raw message payload.
-    pub fn payload(&self) -> String {
-        std::fs::read_to_string(&self.blob_path).unwrap()
+    pub fn payload(&self) -> &str {
+        &self.payload
     }
 }
 
-/// Load a pre-generated keypair for alice@example.com from disk.
+/// Load a pre-generated keypair for alice@example.org from disk.
 ///
 /// This saves CPU cycles by avoiding having to generate a key.
 ///
 /// The keypair was created using the crate::key::tests::gen_key test.
-pub fn alice_keypair() -> key::KeyPair {
-    let addr = EmailAddress::new("alice@example.com").unwrap();
-    let public =
-        key::SignedPublicKey::from_base64(include_str!("../test-data/key/alice-public.asc"))
-            .unwrap();
-    let secret =
-        key::SignedSecretKey::from_base64(include_str!("../test-data/key/alice-secret.asc"))
-            .unwrap();
+pub fn alice_keypair() -> KeyPair {
+    let addr = EmailAddress::new("alice@example.org").unwrap();
+
+    let public = key::SignedPublicKey::from_asc(include_str!("../test-data/key/alice-public.asc"))
+        .unwrap()
+        .0;
+    let secret = key::SignedSecretKey::from_asc(include_str!("../test-data/key/alice-secret.asc"))
+        .unwrap()
+        .0;
     key::KeyPair {
         addr,
         public,
@@ -567,12 +681,14 @@ pub fn alice_keypair() -> key::KeyPair {
 /// Load a pre-generated keypair for bob@example.net from disk.
 ///
 /// Like [alice_keypair] but a different key and identity.
-pub fn bob_keypair() -> key::KeyPair {
+pub fn bob_keypair() -> KeyPair {
     let addr = EmailAddress::new("bob@example.net").unwrap();
-    let public =
-        key::SignedPublicKey::from_base64(include_str!("../test-data/key/bob-public.asc")).unwrap();
-    let secret =
-        key::SignedSecretKey::from_base64(include_str!("../test-data/key/bob-secret.asc")).unwrap();
+    let public = key::SignedPublicKey::from_asc(include_str!("../test-data/key/bob-public.asc"))
+        .unwrap()
+        .0;
+    let secret = key::SignedSecretKey::from_asc(include_str!("../test-data/key/bob-secret.asc"))
+        .unwrap()
+        .0;
     key::KeyPair {
         addr,
         public,
@@ -580,40 +696,43 @@ pub fn bob_keypair() -> key::KeyPair {
     }
 }
 
-pub struct EvTracker(Receiver<EventType>);
+/// Utility to help wait for and retrieve events.
+///
+/// This buffers the events in order they are emitted.  This allows consuming events in
+/// order while looking for the right events using the provided methods.
+///
+/// The methods only return [`EventType`] rather than the full [`Event`] since it can only
+/// be attached to a single [`TestContext`] and therefore the context is already known as
+/// you will be accessing it as [`TestContext::evtracker`].
+#[derive(Debug)]
+pub struct EventTracker(Receiver<Event>);
 
-impl EvTracker {
-    pub async fn get_info_contains(&self, s: &str) -> EventType {
-        loop {
-            let event = self.0.recv().await.unwrap();
-            if let EventType::Info(i) = &event {
-                if i.contains(s) {
-                    return event;
+impl EventTracker {
+    /// Consumes emitted events returning the first matching one.
+    ///
+    /// If no matching events are ready this will wait for new events to arrive and time out
+    /// after 10 seconds.
+    pub async fn get_matching<F: Fn(&EventType) -> bool>(&self, event_matcher: F) -> EventType {
+        async move {
+            loop {
+                let event = self.0.recv().await.unwrap();
+                if event_matcher(&event.typ) {
+                    return event.typ;
                 }
             }
         }
+        .timeout(Duration::from_secs(10))
+        .await
+        .expect("timeout waiting for event match")
     }
 
-    pub async fn get_matching<F: Fn(EventType) -> bool>(&self, event_matcher: F) -> EventType {
-        const TIMEOUT: Duration = Duration::from_secs(20);
-
-        loop {
-            let event = async_std::future::timeout(TIMEOUT, self.recv())
-                .await
-                .unwrap()
-                .unwrap();
-
-            if event_matcher(event.clone()) {
-                return event;
-            }
-        }
-    }
-}
-
-impl Deref for EvTracker {
-    type Target = Receiver<EventType>;
-    fn deref(&self) -> &Self::Target {
-        &self.0
+    /// Consumes events looking for an [`EventType::Info`] with substring matching.
+    pub async fn get_info_contains(&self, s: &str) -> EventType {
+        self.get_matching(|evt| match evt {
+            EventType::Info(ref msg) => msg.contains(s),
+            _ => false,
+        })
+        .await
     }
 }
 
@@ -640,7 +759,7 @@ pub(crate) async fn get_chat_msg(
 /// Pretty-print an event to stdout
 ///
 /// Done during tests this is captured by `cargo test` and associated with the test itself.
-fn receive_event(event: &Event) {
+fn print_event(event: &Event) {
     let green = Color::Green.normal();
     let yellow = Color::Yellow.normal();
     let red = Color::Red.normal();
@@ -732,7 +851,7 @@ async fn log_msg(context: &Context, prefix: impl AsRef<str>, msg: &Message) {
         &contact_name,
         contact_id,
         msgtext.unwrap_or_default(),
-        if msg.get_from_id() == 1u32 {
+        if msg.get_from_id() == DC_CONTACT_ID_SELF {
             ""
         } else if msg.get_state() == MessageState::InSeen {
             "[SEEN]"
@@ -759,3 +878,36 @@ async fn log_msg(context: &Context, prefix: impl AsRef<str>, msg: &Message) {
         statestr,
     );
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    // The following three tests demonstrate, when made to fail, the log output being
+    // directed to the correct test output.
+
+    #[async_std::test]
+    async fn test_with_alice() {
+        let alice = TestContext::builder().configure_alice().build().await;
+        alice.ctx.emit_event(EventType::Info("hello".into()));
+        // panic!("Alice fails");
+    }
+
+    #[async_std::test]
+    async fn test_with_bob() {
+        let bob = TestContext::builder().configure_bob().build().await;
+        bob.ctx.emit_event(EventType::Info("there".into()));
+        // panic!("Bob fails");
+    }
+
+    #[async_std::test]
+    async fn test_with_both() {
+        let mut tcm = TestContextManager::new().await;
+        let alice = tcm.alice().await;
+        let bob = tcm.bob().await;
+
+        alice.ctx.emit_event(EventType::Info("hello".into()));
+        bob.ctx.emit_event(EventType::Info("there".into()));
+        // panic!("Both fail");
+    }
+}

src/update_helper.rs

@@ -1,7 +1,7 @@
 //! # Functions to update timestamps.
 
 use crate::chat::{Chat, ChatId};
-use crate::contact::Contact;
+use crate::contact::{Contact, ContactId};
 use crate::context::Context;
 use crate::param::{Param, Params};
 use anyhow::Result;
@@ -12,7 +12,7 @@ impl Context {
     /// (if we have a ContactId type at some point, the function should go there)
     pub(crate) async fn update_contacts_timestamp(
         &self,
-        contact_id: u32,
+        contact_id: ContactId,
         scope: Param,
         new_timestamp: i64,
     ) -> Result<bool> {
@@ -92,7 +92,7 @@ mod tests {
         dc_receive_imf(
             &t,
             b"From: Bob Authname <bob@example.org>\n\
-                 To: alice@example.com\n\
+                 To: alice@example.org\n\
                  Subject: updated subject\n\
                  Message-ID: <msg2@example.org>\n\
                  Chat-Version: 1.0\n\
@@ -100,14 +100,13 @@ mod tests {
                  \n\
                  second message\n",
             "INBOX",
-            1,
             false,
         )
         .await?;
         dc_receive_imf(
             &t,
             b"From: Bob Authname <bob@example.org>\n\
-                 To: alice@example.com\n\
+                 To: alice@example.org\n\
                  Subject: original subject\n\
                  Message-ID: <msg1@example.org>\n\
                  Chat-Version: 1.0\n\
@@ -115,7 +114,6 @@ mod tests {
                  \n\
                  first message\n",
             "INBOX",
-            2,
             false,
         )
         .await?;
@@ -137,7 +135,7 @@ mod tests {
         dc_receive_imf(
             &t,
             b"From: Bob Authname <bob@example.org>\n\
-                 To: alice@example.com\n\
+                 To: alice@example.org\n\
                  Message-ID: <msg1@example.org>\n\
                  Chat-Version: 1.0\n\
                  Chat-Group-ID: abcde\n\
@@ -146,7 +144,6 @@ mod tests {
                  \n\
                  first message\n",
             "INBOX",
-            1,
             false,
         )
         .await?;
@@ -157,7 +154,7 @@ mod tests {
         dc_receive_imf(
             &t,
             b"From: Bob Authname <bob@example.org>\n\
-                 To: alice@example.com\n\
+                 To: alice@example.org\n\
                  Message-ID: <msg3@example.org>\n\
                  Chat-Version: 1.0\n\
                  Chat-Group-ID: abcde\n\
@@ -167,14 +164,13 @@ mod tests {
                  \n\
                  third message\n",
             "INBOX",
-            2,
             false,
         )
         .await?;
         dc_receive_imf(
             &t,
             b"From: Bob Authname <bob@example.org>\n\
-                 To: alice@example.com\n\
+                 To: alice@example.org\n\
                  Message-ID: <msg2@example.org>\n\
                  Chat-Version: 1.0\n\
                  Chat-Group-ID: abcde\n\
@@ -184,7 +180,6 @@ mod tests {
                  \n\
                  second message\n",
             "INBOX",
-            3,
             false,
         )
         .await?;

standards.md

@@ -12,9 +12,11 @@ Filename encoding                | Encoded Words ([RFC 2047](https://tools.ietf.
 Identify server folders          | IMAP LIST Extension ([RFC 6154](https://tools.ietf.org/html/rfc6154))
 Push                             | IMAP IDLE ([RFC 2177](https://tools.ietf.org/html/rfc2177))
 Quota                            | IMAP QUOTA extension ([RFC 2087](https://tools.ietf.org/html/rfc2087))
+Seen status synchronization      | IMAP CONDSTORE extension ([RFC 7162](https://tools.ietf.org/html/rfc7162))
 Authorization                    | OAuth2 ([RFC 6749](https://tools.ietf.org/html/rfc6749))
 End-to-end encryption            | [Autocrypt Level 1](https://autocrypt.org/level1.html), OpenPGP ([RFC 4880](https://tools.ietf.org/html/rfc4880)), Security Multiparts for MIME ([RFC 1847](https://tools.ietf.org/html/rfc1847)) and [“Mixed Up” Encryption repairing](https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html)
-Configuration assistance         | [Autoconfigure](https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration) and [Autodiscover](https://technet.microsoft.com/library/bb124251(v=exchg.150).aspx)
+Header encryption                | [Protected Headers for Cryptographic E-mail](https://datatracker.ietf.org/doc/draft-autocrypt-lamps-protected-headers/)
+Configuration assistance         | [Autoconfigure](https://web.archive.org/web/20210402044801/https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration) and [Autodiscover](https://technet.microsoft.com/library/bb124251(v=exchg.150).aspx)
 Messenger functions              | [Chat-over-Email](https://github.com/deltachat/deltachat-core-rust/blob/master/spec.md#chat-mail-specification)
 Detect mailing list              | List-Id ([RFC 2919](https://tools.ietf.org/html/rfc2919)) and Precedence ([RFC 3834](https://tools.ietf.org/html/rfc3834))
 User and chat colors             | [XEP-0392](https://xmpp.org/extensions/xep-0392.html): Consistent Color Generation

test-data/key/alice-public.asc

@@ -1 +1,13 @@
-mDMEXlh13RYJKwYBBAHaRw8BAQdAzfVIAleCXMJrq8VeLlEVof6ITCviMktKjmcBKAu4m5C0GUFsaWNlIDxhbGljZUBleGFtcGxlLm9yZz6IkAQTFggAOBYhBC5vossjtTLXKGNLWGSwj2Gp7ZRDBQJeWHXdAhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEGSwj2Gp7ZRDE3oA/i4MCyDMTsjWqDZoQwX/A/GoTO2/V0wKPhjJJy/8m2pMAPkBjOnGOtx2SZpQvJGTa9h804RY6iDrRuI8A/8tEEXAA7g4BF5Ydd0SCisGAQQBl1UBBQEBB0AG7cjWy2SFAU8KnltlubVW67rFiyfp01JrRe6Xqy22HQMBCAeIeAQYFggAIBYhBC5vossjtTLXKGNLWGSwj2Gp7ZRDBQJeWHXdAhsMAAoJEGSwj2Gp7ZRDLo8BAObE8GnsGVwKzNqCvHeWgJsqhjS3C6gvSlV3tEm9XmF6AQDXucIyVfoBwoyMh2h6cSn/ATn5QJb35pgo+ivp3jsMAg==
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mDMEXlh13RYJKwYBBAHaRw8BAQdAzfVIAleCXMJrq8VeLlEVof6ITCviMktKjmcB
+KAu4m5C0GUFsaWNlIDxhbGljZUBleGFtcGxlLm9yZz6IkAQTFggAOBYhBC5vossj
+tTLXKGNLWGSwj2Gp7ZRDBQJeWHXdAhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheA
+AAoJEGSwj2Gp7ZRDE3oA/i4MCyDMTsjWqDZoQwX/A/GoTO2/V0wKPhjJJy/8m2pM
+APkBjOnGOtx2SZpQvJGTa9h804RY6iDrRuI8A/8tEEXAA7g4BF5Ydd0SCisGAQQB
+l1UBBQEBB0AG7cjWy2SFAU8KnltlubVW67rFiyfp01JrRe6Xqy22HQMBCAeIeAQY
+FggAIBYhBC5vossjtTLXKGNLWGSwj2Gp7ZRDBQJeWHXdAhsMAAoJEGSwj2Gp7ZRD
+Lo8BAObE8GnsGVwKzNqCvHeWgJsqhjS3C6gvSlV3tEm9XmF6AQDXucIyVfoBwoyM
+h2h6cSn/ATn5QJb35pgo+ivp3jsMAg==
+=t/Qq
+-----END PGP PUBLIC KEY BLOCK-----