prepare 1.58 (#2584)

* update changelog for 1.58.0

* bump version to 1.58.0

.circleci/config.yml

@@ -1,77 +0,0 @@
-version: 2.1
-executors:
-  default:
-    docker:
-      - image: filecoin/rust:latest
-    working_directory: /mnt/crate
-  doxygen:
-    docker:
-      - image: hrektts/doxygen
-            
-jobs:
-  build_doxygen:
-    executor: doxygen
-    steps:
-      - checkout
-      - run: bash scripts/run-doxygen.sh
-      - run: mkdir -p workspace/c-docs
-      - run: cp -av deltachat-ffi/{html,xml} workspace/c-docs/
-      - persist_to_workspace:
-          root: workspace
-          paths:
-            - c-docs
-
-  remote_python_packaging:
-    machine: true
-    steps:
-      - checkout
-      # the following commands on success produces
-      # workspace/{wheelhouse,py-docs} as artefact directories
-      - run:
-          # building aarch64 packages under qemu is very slow
-          no_output_timeout: 60m
-          command: bash scripts/remote_python_packaging.sh
-      - persist_to_workspace:
-          root: workspace
-          paths:
-            # - c-docs
-            - py-docs
-            - wheelhouse
-
-  upload_docs_wheels:
-    machine: true
-    steps:
-      - checkout
-      - attach_workspace:
-          at: workspace
-      - run: ls -laR workspace
-      - run: scripts/ci_upload.sh workspace/py-docs workspace/wheelhouse workspace/c-docs
-
-workflows:
-  version: 2.1
-
-  test:
-    jobs:
-      - remote_python_packaging:
-          filters:
-            tags:
-              only: /.+/
-            branches:
-              ignore: /.*/
-
-      - build_doxygen:
-          filters:
-            tags:
-              only: /.+/
-            branches:
-              ignore: /.*/
-
-      - upload_docs_wheels:
-          requires:
-             - remote_python_packaging
-             - build_doxygen
-          filters:
-            tags:
-              only: /.+/
-            branches:
-              ignore: /.*/

CHANGELOG.md

@@ -1,5 +1,138 @@
 # Changelog 
 
+
+## 1.58.0
+
+### Fixes
+
+- move WAL file together with database
+  and avoid using data if the database was not closed correctly before #2583
+
+
+## 1.57.0
+
+### API Changes
+
+- breaking change: removed deaddrop chat #2514 #2563
+
+  Contact request chats are not merged into a single virtual
+  "deaddrop" chat anymore. Instead, they are shown in the chatlist the
+  same way as other chats, but sending of messages to them is not
+  allowed and MDNs are not sent automatically until the chat is
+  "accepted" by the user.
+
+  New API:
+  - `dc_chat_is_contact_request()`: returns true if chat is a contact
+    request.  In this case an option to accept the chat via
+    `dc_accept_chat()` should be shown in the UI.
+  - `dc_accept_chat()`: unblock the chat or accept contact request
+  - `dc_block_chat()`: block the chat, currently works only for mailing
+    lists.
+
+  Removed API:
+  - `dc_create_chat_by_msg_id()`: deprecated 2021-02-07 in favor of
+    `dc_decide_on_contact_request()`
+  - `dc_marknoticed_contact()`: deprecated 2021-02-07 in favor of
+    `dc_decide_on_contact_request()`
+  - `dc_decide_on_contact_request()`: this call requires a message ID
+    from deaddrop chat as input. As deaddrop chat is removed, this
+    call can't be used anymore.
+  - `dc_msg_get_real_chat_id()`: use `dc_msg_get_chat_id()` instead, the
+    only difference between these calls was in handling of deaddrop
+    chat
+  - removed `DC_CHAT_ID_DEADDROP` and `DC_STR_DEADDROP` constants
+
+- breaking change: removed `DC_EVENT_ERROR_NETWORK` and `DC_STR_SERVER_RESPONSE`
+  Instead, there is a new api `dc_get_connectivity()`
+  and `dc_get_connectivity_html()`;
+  `DC_EVENT_CONNECTIVITY_CHANGED` is emitted on changes
+
+- breaking change: removed `dc_accounts_import_account()`
+  Instead you need to add an account and call `dc_imex(DC_IMEX_IMPORT_BACKUP)`
+  on its context
+
+- update account api, 2 new methods:
+  `int dc_all_work_done (dc_context_t* context);`
+  `int dc_accounts_all_work_done (dc_accounts_t* accounts);`
+
+- add api to check if a message was `Auto-Submitted`
+  cffi: `int dc_msg_is_bot (const dc_msg_t* msg);`
+  python: `Message.is_bot()`
+
+- `dc_context_t* dc_accounts_get_selected_account (dc_accounts_t* accounts);`
+  now returns `NULL` if there is no selected account
+
+- added `dc_accounts_maybe_network_lost()` for systems core cannot find out
+  connectivity loss on its own (eg. iOS) #2550
+
+### Added
+- use Auto-Submitted: auto-generated header to identify bots #2502
+- allow sending stickers via repl tool
+- chat: make `get_msg_cnt()` and `get_fresh_msg_cnt()` work for deaddrop chat #2493
+- withdraw/revive own qr-codes #2512
+- add Connectivity view (a better api for getting the connection status) #2319 #2549 #2542
+
+### Changes
+- updated spec: new `Chat-User-Avatar` usage, `Chat-Content: sticker`, structure, copyright year #2480
+- update documentation #2548 #2561 #2569
+- breaking: `Accounts::create` does not also create an default account anymore #2500
+- remove "forwarded" from stickers, as the primary way of getting stickers
+  is by asking a bot and then forwarding them currently #2526
+- mimeparser: use mailparse to parse RFC 2231 filenames #2543
+- allow email addresses without dot in the domain part #2112
+- allow installing lib and include under different prefixes #2558
+- remove counter from name provided by `DC_CHAT_ID_ARCHIVED_LINK` #2566
+- improve tests #2487 #2491 #2497
+- refactorings #2492 #2503 #2504 #2506 #2515 #2520 #2567 #2575 #2577 #2579
+- improve ci #2494
+- update provider-database #2565
+
+### Removed
+- remove `dc_accounts_import_account()` api #2521
+- remove `DC_EVENT_ERROR_NETWORK` and `DC_STR_SERVER_RESPONSE` #2319
+
+### Fixes
+- allow stickers with gif-images #2481
+- fix database migration #2486
+- do not count hidden messages in get_msg_cnt(). #2493
+- improve drafts detection #2489
+- fix panic when removing last, selected account from account manager #2500
+- set_draft's message-changed-event returns now draft's msg id instead of 0 #2304
+- avoid hiding outgoing classic emails #2505
+- fixes for message timestamps #2517
+- do not process names, avatars, location XMLs, message signature etc.
+  for duplicate messages #2513
+- fix `can_send` for users not in group #2479
+- fix receiving events for accounts added by `dc_accounts_add_account()` #2559
+- fix which chats messages are assigned to #2465
+- fix: don't create chats when MDNs are received #2578
+
+
+## 1.56.0
+
+- fix downscaling images #2469
+
+- fix outgoing messages popping up in selfchat #2456
+
+- securejoin: display error reason if there is any #2470
+
+- do not allow deleting contacts with ongoing chats #2458
+
+- fix: ignore drafts folder when scanning #2454
+
+- fix: scan folders also when inbox is not watched #2446
+
+- more robust In-Reply-To parsing #2182
+
+- update dependencies #2441 #2438 #2439 #2440 #2447 #2448 #2449 #2452 #2453 #2460 #2464 #2466
+
+- update provider-database #2471
+
+- refactorings #2459 #2457
+
+- improve tests and ci #2445 #2450 #2451
+
+
 ## 1.55.0
 
 - fix panic when receiving some HTML messages #2434

CMakeLists.txt

@@ -8,7 +8,11 @@ add_custom_command(
 	"target/release/libdeltachat.a"
 	"target/release/libdeltachat.so"
 	"target/release/pkgconfig/deltachat.pc"
-	COMMAND PREFIX=${CMAKE_INSTALL_PREFIX} ${CARGO} build --release --no-default-features
+        COMMAND
+        PREFIX=${CMAKE_INSTALL_PREFIX}
+        LIBDIR=${CMAKE_INSTALL_FULL_LIBDIR}
+        INCLUDEDIR=${CMAKE_INSTALL_FULL_INCLUDEDIR}
+        ${CARGO} build --release --no-default-features
 
 	# Build in `deltachat-ffi` directory instead of using
 	# `--package deltachat_ffi` to avoid feature resolver version

Cargo.lock

@@ -138,9 +138,9 @@ dependencies = [
 
 [[package]]
 name = "anyhow"
-version = "1.0.40"
+version = "1.0.42"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "28b2cd92db5cbd74e8e5028f7e27dd7aa3090e89e4f2a197cc7c8dfb69c7063b"
+checksum = "595d3cfa7a60d4555cb5067b99f07142a08ea778de5cf993f7b75c7d8fabc486"
 
 [[package]]
 name = "arrayvec"
@@ -323,19 +323,18 @@ dependencies = [
 
 [[package]]
 name = "async-smtp"
-version = "0.3.4"
-source = "git+https://github.com/async-email/async-smtp?rev=2275fd8d13e39b2c58d6605c786ff06ff9e05708#2275fd8d13e39b2c58d6605c786ff06ff9e05708"
+version = "0.4.0"
+source = "git+https://github.com/async-email/async-smtp?rev=c8800625f7cf29f437143ac7e720ac2730a0962f#c8800625f7cf29f437143ac7e720ac2730a0962f"
 dependencies = [
  "async-native-tls",
  "async-std",
  "async-trait",
- "base64 0.12.3",
+ "base64 0.13.0",
  "bufstream",
- "fast_chemail",
  "hostname 0.1.5",
  "log",
  "nom 5.1.2",
- "pin-project 0.4.27",
+ "pin-project 1.0.5",
  "pin-utils",
  "serde",
  "serde_derive",
@@ -637,27 +636,6 @@ version = "1.4.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610"
 
-[[package]]
-name = "bzip2"
-version = "0.3.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "42b7c3cbf0fa9c1b82308d57191728ca0256cb821220f4e2fd410a72ade26e3b"
-dependencies = [
- "bzip2-sys",
- "libc",
-]
-
-[[package]]
-name = "bzip2-sys"
-version = "0.1.10+1.0.8"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "17fa3d1ac1ca21c5c4e36a97f3c3eb25084576f6fc47bf0139c1123434216c6c"
-dependencies = [
- "cc",
- "libc",
- "pkg-config",
-]
-
 [[package]]
 name = "cache-padded"
 version = "1.1.1"
@@ -847,9 +825,9 @@ checksum = "ea221b5284a47e40033bf9b66f35f984ec0ea2931eb03505246cd27a963f981b"
 
 [[package]]
 name = "cpufeatures"
-version = "0.1.1"
+version = "0.1.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dec1028182c380cc45a2e2c5ec841134f2dfd0f8f5f0a5bcd68004f81b5efdf4"
+checksum = "66c99696f6c9dd7f35d486b9d04d7e6e202aa3e8c40d553f2fdf5e7e0c6a71ef"
 dependencies = [
  "libc",
 ]
@@ -886,7 +864,7 @@ dependencies = [
  "clap",
  "criterion-plot",
  "csv",
- "itertools 0.10.0",
+ "itertools 0.10.1",
  "lazy_static",
  "num-traits",
  "oorandom",
@@ -1129,7 +1107,7 @@ dependencies = [
 
 [[package]]
 name = "deltachat"
-version = "1.55.0"
+version = "1.58.0"
 dependencies = [
  "ansi_term",
  "anyhow",
@@ -1144,7 +1122,6 @@ dependencies = [
  "base64 0.13.0",
  "bitflags",
  "byteorder",
- "charset",
  "chrono",
  "criterion",
  "deltachat_derive",
@@ -1157,7 +1134,7 @@ dependencies = [
  "hex",
  "image",
  "indexmap",
- "itertools 0.10.0",
+ "itertools 0.10.1",
  "kamadak-exif",
  "lettre_email",
  "libc",
@@ -1196,7 +1173,6 @@ dependencies = [
  "toml",
  "url",
  "uuid",
- "zip",
 ]
 
 [[package]]
@@ -1209,7 +1185,7 @@ dependencies = [
 
 [[package]]
 name = "deltachat_ffi"
-version = "1.55.0"
+version = "1.58.0"
 dependencies = [
  "anyhow",
  "async-std",
@@ -1610,9 +1586,9 @@ dependencies = [
 
 [[package]]
 name = "futures"
-version = "0.3.15"
+version = "0.3.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0e7e43a803dae2fa37c1f6a8fe121e1f7bf9548b4dfc0522a42f34145dadfc27"
+checksum = "1adc00f486adfc9ce99f77d717836f0c5aa84965eb0b4f051f4e83f7cab53f8b"
 dependencies = [
  "futures-channel",
  "futures-core",
@@ -1625,9 +1601,9 @@ dependencies = [
 
 [[package]]
 name = "futures-channel"
-version = "0.3.15"
+version = "0.3.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e682a68b29a882df0545c143dc3646daefe80ba479bcdede94d5a703de2871e2"
+checksum = "74ed2411805f6e4e3d9bc904c95d5d423b89b3b25dc0250aa74729de20629ff9"
 dependencies = [
  "futures-core",
  "futures-sink",
@@ -1635,15 +1611,15 @@ dependencies = [
 
 [[package]]
 name = "futures-core"
-version = "0.3.15"
+version = "0.3.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0402f765d8a89a26043b889b26ce3c4679d268fa6bb22cd7c6aad98340e179d1"
+checksum = "af51b1b4a7fdff033703db39de8802c673eb91855f2e0d47dcf3bf2c0ef01f99"
 
 [[package]]
 name = "futures-executor"
-version = "0.3.15"
+version = "0.3.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "badaa6a909fac9e7236d0620a2f57f7664640c56575b71a7552fbd68deafab79"
+checksum = "4d0d535a57b87e1ae31437b892713aee90cd2d7b0ee48727cd11fc72ef54761c"
 dependencies = [
  "futures-core",
  "futures-task",
@@ -1652,15 +1628,15 @@ dependencies = [
 
 [[package]]
 name = "futures-io"
-version = "0.3.15"
+version = "0.3.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "acc499defb3b348f8d8f3f66415835a9131856ff7714bf10dadfc4ec4bdb29a1"
+checksum = "0b0e06c393068f3a6ef246c75cdca793d6a46347e75286933e5e75fd2fd11582"
 
 [[package]]
 name = "futures-lite"
-version = "1.11.3"
+version = "1.12.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b4481d0cd0de1d204a4fa55e7d45f07b1d958abcb06714b3446438e2eff695fb"
+checksum = "7694489acd39452c77daa48516b894c153f192c3578d5a839b62c58099fcbf48"
 dependencies = [
  "fastrand",
  "futures-core",
@@ -1673,9 +1649,9 @@ dependencies = [
 
 [[package]]
 name = "futures-macro"
-version = "0.3.15"
+version = "0.3.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a4c40298486cdf52cc00cd6d6987892ba502c7656a16a4192a9992b1ccedd121"
+checksum = "c54913bae956fb8df7f4dc6fc90362aa72e69148e3f39041fbe8742d21e0ac57"
 dependencies = [
  "autocfg 1.0.1",
  "proc-macro-hack",
@@ -1686,21 +1662,21 @@ dependencies = [
 
 [[package]]
 name = "futures-sink"
-version = "0.3.15"
+version = "0.3.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a57bead0ceff0d6dde8f465ecd96c9338121bb7717d3e7b108059531870c4282"
+checksum = "c0f30aaa67363d119812743aa5f33c201a7a66329f97d1a887022971feea4b53"
 
 [[package]]
 name = "futures-task"
-version = "0.3.15"
+version = "0.3.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8a16bef9fc1a4dddb5bee51c989e3fbba26569cbb0e31f5b303c184e3dd33dae"
+checksum = "bbe54a98670017f3be909561f6ad13e810d9a51f3f061b902062ca3da80799f2"
 
 [[package]]
 name = "futures-util"
-version = "0.3.15"
+version = "0.3.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "feb5c238d27e2bf94ffdfd27b2c29e3df4a68c4193bb6427384259e2bf191967"
+checksum = "67eb846bfd58e44a8481a00049e82c43e0ccb5d61f8dc071057cb19249dd4d78"
 dependencies = [
  "autocfg 1.0.1",
  "futures-channel",
@@ -1794,12 +1770,6 @@ version = "1.7.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "62aca2aba2d62b4a7f5b33f3712cb1b0692779a56fb510499d5c0aa594daeaf3"
 
-[[package]]
-name = "hashbrown"
-version = "0.9.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d7afe4a420e3fe79967a00898cc1f4db7c8a49a9333a29f8a4bd76a253d5cd04"
-
 [[package]]
 name = "hashbrown"
 version = "0.11.2"
@@ -1815,7 +1785,7 @@ version = "0.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7249a3129cbc1ffccd74857f81464a323a152173cdb134e0fd81bc803b29facf"
 dependencies = [
- "hashbrown 0.11.2",
+ "hashbrown",
 ]
 
 [[package]]
@@ -1998,12 +1968,12 @@ dependencies = [
 
 [[package]]
 name = "indexmap"
-version = "1.6.2"
+version = "1.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "824845a0bf897a9042383849b02c1bc219c2383772efcd5c6f9766fa4b81aef3"
+checksum = "bc633605454125dec4b66843673f01c7df2b89479b32e0ed634e43a91cff62a5"
 dependencies = [
  "autocfg 1.0.1",
- "hashbrown 0.9.1",
+ "hashbrown",
 ]
 
 [[package]]
@@ -2050,9 +2020,9 @@ dependencies = [
 
 [[package]]
 name = "itertools"
-version = "0.10.0"
+version = "0.10.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "37d572918e350e82412fe766d24b15e6682fb2ed2bbe018280caa810397cb319"
+checksum = "69ddb889f9d0d08a67338271fa9b62996bc788c7796a5c18cf057420aaed5eaf"
 dependencies = [
  "either",
 ]
@@ -2150,9 +2120,9 @@ dependencies = [
 
 [[package]]
 name = "libc"
-version = "0.2.95"
+version = "0.2.97"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "789da6d93f1b866ffe175afc5322a4d76c038605a1c3319bb57b06967ca98a36"
+checksum = "12b8adadd720df158f4d70dfe7ccc6adb0472d7c55ca83445f6a5ab3e36f8fb6"
 
 [[package]]
 name = "libm"
@@ -2207,9 +2177,9 @@ dependencies = [
 
 [[package]]
 name = "mailparse"
-version = "0.13.4"
+version = "0.13.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "62db73ff1a42b0e3a8858cf0d5c183bdfc23491f7294ae4a8200c83577457386"
+checksum = "c06f526fc13a50f46a3689a6f438cb833c59817c898bb40a3954f341ddf74ce1"
 dependencies = [
  "base64 0.13.0",
  "charset",
@@ -2471,9 +2441,9 @@ checksum = "1a5b3dd1c072ee7963717671d1ca129f1048fda25edea6b752bfc71ac8854170"
 
 [[package]]
 name = "once_cell"
-version = "1.7.2"
+version = "1.8.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "af8b08b04175473088b46763e51ee54da5f9a164bc162f615b91bc179dbf15a3"
+checksum = "692fcb63b64b1758029e0a96ee63e049ce8c5948587f2f7208df04625e5f6b56"
 
 [[package]]
 name = "oorandom"
@@ -3385,9 +3355,9 @@ dependencies = [
 
 [[package]]
 name = "sha-1"
-version = "0.9.6"
+version = "0.9.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8c4cfa741c5832d0ef7fab46cabed29c2aae926db0b11bb2069edd8db5e64e16"
+checksum = "1a0c8611594e2ab4ebbf06ec7cbbf0a99450b8570e96cbf5188b5d5f6ef18d81"
 dependencies = [
  "block-buffer",
  "cfg-if 1.0.0",
@@ -3615,15 +3585,15 @@ checksum = "6446ced80d6c486436db5c078dde11a9f73d42b57fb273121e160b84f63d894c"
 
 [[package]]
 name = "strum"
-version = "0.20.0"
+version = "0.21.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7318c509b5ba57f18533982607f24070a55d353e90d4cae30c467cdb2ad5ac5c"
+checksum = "aaf86bbcfd1fa9670b7a129f64fc0c9fcbbfe4f1bc4210e9e98fe71ffc12cde2"
 
 [[package]]
 name = "strum_macros"
-version = "0.20.1"
+version = "0.21.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ee8bc6b87a5112aeeab1f4a9f7ab634fe6cbefc4850006df31267f4cfb9e3149"
+checksum = "d06aaeeee809dbc59eb4556183dd927df67db1540de5be8d3ec0b6636358a5ec"
 dependencies = [
  "heck",
  "proc-macro2",
@@ -3658,9 +3628,9 @@ dependencies = [
 
 [[package]]
 name = "syn"
-version = "1.0.72"
+version = "1.0.74"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a1e8cdbefb79a9a5a65e0db8b47b723ee907b7c7f8496c76a1770b5c310bab82"
+checksum = "1873d832550d4588c3dbc20f01361ab00bfe741048f71e3fecf145a7cc18b29c"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -4234,17 +4204,3 @@ dependencies = [
  "syn",
  "synstructure",
 ]
-
-[[package]]
-name = "zip"
-version = "0.5.12"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9c83dc9b784d252127720168abd71ea82bf8c3d96b17dc565b5e2a02854f2b27"
-dependencies = [
- "byteorder",
- "bzip2",
- "crc32fast",
- "flate2",
- "thiserror",
- "time 0.1.44",
-]

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat"
-version = "1.55.0"
+version = "1.58.0"
 authors = ["Delta Chat Developers (ML) <delta@codespeak.net>"]
 edition = "2018"
 license = "MPL-2.0"
@@ -15,10 +15,10 @@ lto = true
 deltachat_derive = { path = "./deltachat_derive" }
 
 ansi_term = { version = "0.12.1", optional = true }
-anyhow = "1.0.40"
+anyhow = "1.0.42"
 async-imap = "0.5.0"
 async-native-tls = { version = "0.3.3" }
-async-smtp = { git = "https://github.com/async-email/async-smtp", rev="2275fd8d13e39b2c58d6605c786ff06ff9e05708" }
+async-smtp = { git = "https://github.com/async-email/async-smtp", rev="c8800625f7cf29f437143ac7e720ac2730a0962f" }
 async-std-resolver = "0.20.3"
 async-std = { version = "~1.9.0", features = ["unstable"] }
 async-tar = "0.3.0"
@@ -27,27 +27,26 @@ backtrace = "0.3.59"
 base64 = "0.13"
 bitflags = "1.1.0"
 byteorder = "1.3.1"
-charset = "0.1"
 chrono = "0.4.6"
 dirs = { version = "3.0.2", optional=true }
 email = { git = "https://github.com/deltachat/rust-email", branch = "master" }
 encoded-words = { git = "https://github.com/async-email/encoded-words", branch="master" }
 escaper = "0.1.1"
-futures = "0.3.15"
+futures = "0.3.16"
 hex = "0.4.0"
 image = { version = "0.23.5", default-features=false, features = ["gif", "jpeg", "ico", "png", "pnm", "webp", "bmp"] }
-indexmap = "1.3.0"
-itertools = "0.10.0"
+indexmap = "1.7.0"
+itertools = "0.10.1"
 kamadak-exif = "0.5"
 lettre_email = { git = "https://github.com/deltachat/lettre", branch = "master" }
-libc = "0.2.95"
+libc = "0.2.97"
 log = {version = "0.4.8", optional = true }
-mailparse = "0.13.4"
+mailparse = "0.13.5"
 native-tls = "0.2.3"
 num_cpus = "1.13.0"
 num-derive = "0.3.0"
 num-traits = "0.2.6"
-once_cell = "1.4.1"
+once_cell = "1.8.0"
 percent-encoding = "2.0"
 pgp = { version = "0.7.0", default-features = false }
 pretty_env_logger = { version = "0.4.0", optional = true }
@@ -62,24 +61,23 @@ rustyline = { version = "8.2.0", optional = true }
 sanitize-filename = "0.3.0"
 serde_json = "1.0"
 serde = { version = "1.0", features = ["derive"] }
-sha-1 = "0.9.6"
+sha-1 = "0.9.7"
 sha2 = "0.9.5"
 smallvec = "1.0.0"
 stop-token = "0.2.0"
-strum = "0.20.0"
-strum_macros = "0.20.1"
+strum = "0.21.0"
+strum_macros = "0.21.1"
 surf = { version = "2.0.0-alpha.4", default-features = false, features = ["h1-client"] }
 thiserror = "1.0.25"
 toml = "0.5.6"
 url = "2.2.2"
 uuid = { version = "0.8", features = ["serde", "v4"] }
-zip = "0.5.12"
 
 [dev-dependencies]
 ansi_term = "0.12.0"
 async-std = { version = "1.9.0", features = ["unstable", "attributes"] }
 criterion = "0.3"
-futures-lite = "1.7.0"
+futures-lite = "1.12.0"
 log = "0.4.11"
 pretty_assertions = "0.7.2"
 pretty_env_logger = "0.4.0"
@@ -116,7 +114,7 @@ name = "search_msgs"
 harness = false
 
 [features]
-default = []
+default = ["vendored"]
 internals = []
 repl = ["internals", "rustyline", "log", "pretty_env_logger", "ansi_term", "dirs"]
 vendored = ["async-native-tls/vendored", "async-smtp/native-tls-vendored", "rusqlite/bundled"]

README.md

@@ -3,7 +3,6 @@
 > Deltachat-core written in Rust 
 
 [![Rust CI](https://github.com/deltachat/deltachat-core-rust/actions/workflows/ci.yml/badge.svg)](https://github.com/deltachat/deltachat-core-rust/actions/workflows/ci.yml)
-[![CircleCI](https://circleci.com/gh/deltachat/deltachat-core-rust.svg?style=shield)](https://circleci.com/gh/deltachat/deltachat-core-rust/)
 
 ## Installing Rust and Cargo
 
@@ -80,6 +79,16 @@ For more commands type:
 > help
 ```
 
+## Installing libdeltachat system wide
+
+```
+$ git clone https://github.com/deltachat/deltachat-core-rust.git
+$ cd deltachat-core-rust
+$ cmake -B build . -DCMAKE_INSTALL_PREFIX=/usr
+$ cmake --build build
+$ sudo cmake --install build
+```
+
 ## Development
 
 ```sh

deltachat-ffi/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "deltachat_ffi"
-version = "1.55.0"
+version = "1.58.0"
 description = "Deltachat FFI"
 authors = ["Delta Chat Developers (ML) <delta@codespeak.net>"]
 edition = "2018"
@@ -21,7 +21,7 @@ human-panic = "1.0.1"
 num-traits = "0.2.6"
 serde_json = "1.0"
 async-std = "1.9.0"
-anyhow = "1.0.40"
+anyhow = "1.0.42"
 thiserror = "1.0.25"
 rand = "0.7.3"
 

deltachat-ffi/build.rs

@@ -23,6 +23,8 @@ fn main() {
         version = env::var("CARGO_PKG_VERSION").unwrap(),
         libs_priv = libs_priv,
         prefix = env::var("PREFIX").unwrap_or_else(|_| "/usr/local".to_string()),
+        libdir = env::var("LIBDIR").unwrap_or_else(|_| "/usr/local/lib".to_string()),
+        includedir = env::var("INCLUDEDIR").unwrap_or_else(|_| "/usr/local/include".to_string()),
     );
 
     fs::create_dir_all(target_path.join("pkgconfig")).unwrap();

deltachat-ffi/deltachat.h

@@ -305,7 +305,7 @@ char*           dc_get_blobdir               (const dc_context_t* context);
  *                    DC_SHOW_EMAILS_ACCEPTED_CONTACTS (1)=
  *                    also show all mails of confirmed contacts,
  *                    DC_SHOW_EMAILS_ALL (2)=
- *                    also show mails of unconfirmed contacts in the deaddrop.
+ *                    also show mails of unconfirmed contacts.
  * - `key_gen_type` = DC_KEY_GEN_DEFAULT (0)=
  *                    generate recommended key type (default),
  *                    DC_KEY_GEN_RSA2048 (1)=
@@ -340,7 +340,9 @@ char*           dc_get_blobdir               (const dc_context_t* context);
  *                    https://github.com/cracker0dks/basicwebrtc which some UIs have native support for.
  *                    The type `jitsi:` may be handled by external apps.
  *                    If no type is prefixed, the videochat is handled completely in a browser.
- * - `bot`          = Set to "1" if this is a bot. E.g. prevents adding the "Device messages" and "Saved messages" chats.
+ * - `bot`          = Set to "1" if this is a bot.
+ *                    Prevents adding the "Device messages" and "Saved messages" chats,
+ *                    adds Auto-Submitted header to outgoing messages.
  * - `fetch_existing_msgs` = 1=fetch most recent existing messages on configure (default),
  *                    0=do not fetch existing messages on configure.
  *                    In both cases, existing recipients are added to the contact database.
@@ -461,11 +463,66 @@ char*           dc_get_info                  (const dc_context_t* context);
 char*           dc_get_oauth2_url            (dc_context_t* context, const char* addr, const char* redirect_uri);
 
 
+#define DC_CONNECTIVITY_NOT_CONNECTED        1000
+#define DC_CONNECTIVITY_CONNECTING           2000
+#define DC_CONNECTIVITY_WORKING              3000
+#define DC_CONNECTIVITY_CONNECTED            4000
+
+
+/**
+ * Get the current connectivity, i.e. whether the device is connected to the IMAP server.
+ * One of:
+ * - DC_CONNECTIVITY_NOT_CONNECTED (1000-1999): Show e.g. the string "Not connected" or a red dot
+ * - DC_CONNECTIVITY_CONNECTING (2000-2999): Show e.g. the string "Connecting…" or a yellow dot
+ * - DC_CONNECTIVITY_WORKING (3000-3999): Show e.g. the string "Getting new messages" or a spinning wheel
+ * - DC_CONNECTIVITY_CONNECTED (>=4000): Show e.g. the string "Connected" or a green dot
+ *
+ * We don't use exact values but ranges here so that we can split up
+ * states into multiple states in the future.
+ *
+ * Meant as a rough overview that can be shown 
+ * e.g. in the title of the main screen.
+ *
+ * If the connectivity changes, a #DC_EVENT_CONNECTIVITY_CHANGED will be emitted.
+ *
+ * @memberof dc_context_t
+ * @param context The context object.
+ * @return The current connectivity.
+ */
+int             dc_get_connectivity          (dc_context_t* context);
+
+
+/**
+ * Get an overview of the current connectivity, and possibly more statistics.
+ * Meant to give the user more insight about the current status than
+ * the basic connectivity info returned by dc_get_connectivity(); show this
+ * e.g., if the user taps on said basic connectivity info.
+ *
+ * If this page changes, a #DC_EVENT_CONNECTIVITY_CHANGED will be emitted.
+ *
+ * This comes as an HTML from the core so that we can easily improve it
+ * and the improvement instantly reaches all UIs.
+ *
+ * @memberof dc_context_t
+ * @param context The context object.
+ * @return An HTML page with some info about the current connectivity and status.
+ */
+char*           dc_get_connectivity_html     (dc_context_t* context);
+
+
+/**
+ * Standalone version of dc_accounts_all_work_done().
+ * Only used by the python tests.
+ */
+int             dc_all_work_done             (dc_context_t* context);
+
+
 // connect
 
 /**
  * Configure a context.
- * During configuration IO must not be started, if needed stop IO using dc_stop_io() first.
+ * During configuration IO must not be started,
+ * if needed stop IO using dc_accounts_stop_io() or dc_stop_io() first.
  * If the context is already configured,
  * this function will try to change the configuration.
  *
@@ -625,12 +682,6 @@ int             dc_preconfigure_keypair        (dc_context_t* context, const cha
  *
  * By default, the function adds some special entries to the list.
  * These special entries can be identified by the ID returned by dc_chatlist_get_chat_id():
- * - DC_CHAT_ID_DEADDROP (1) - this special chat is present if there are
- *   messages from addresses that have no relationship to the configured account.
- *   The last of these messages is represented by DC_CHAT_ID_DEADDROP and you can retrieve details
- *   about it with dc_chatlist_get_msg_id(). Typically, the UI asks the user "Do you want to chat with NAME?"
- *   and offers the options "Start chat", "Block" or "Not now".
- *   Call dc_decide_on_contact_request() when the user selected one of these options.
  * - DC_CHAT_ID_ARCHIVED_LINK (6) - this special chat is present if the user has
  *   archived _any_ chat using dc_set_chat_visibility(). The UI should show a link as
  *   "Show archived chats", if the user clicks this item, the UI should show a
@@ -648,10 +699,10 @@ int             dc_preconfigure_keypair        (dc_context_t* context, const cha
  *       the pseudo-chat DC_CHAT_ID_ARCHIVED_LINK is added if there are _any_ archived
  *       chats
  *     - the flag DC_GCL_FOR_FORWARDING sorts "Saved messages" to the top of the chatlist
- *       and hides the "Device chat" and the deaddrop.
+ *       and hides the "Device chat" and contact requests.
  *       typically used on forwarding, may be combined with DC_GCL_NO_SPECIALS
  *       to also hide the archive link.
- *     - if the flag DC_GCL_NO_SPECIALS is set, deaddrop and archive link are not added
+ *     - if the flag DC_GCL_NO_SPECIALS is set, archive link is not added
  *       to the list (may be used e.g. for selecting chats on forwarding, the flag is
  *       not needed when DC_GCL_ARCHIVED_ONLY is already set)
  *     - if the flag DC_GCL_ADD_ALLDONE_HINT is set, DC_CHAT_ID_ALLDONE_HINT
@@ -671,42 +722,12 @@ dc_chatlist_t*  dc_get_chatlist              (dc_context_t* context, int flags,
 
 // handle chats
 
-/**
- * Create a normal chat or a group chat by a messages ID that comes typically
- * from the deaddrop, DC_CHAT_ID_DEADDROP (1).
- *
- * If the given message ID already belongs to a normal chat or to a group chat,
- * the chat ID of this chat is returned and no new chat is created.
- * If a new chat is created, the given message ID is moved to this chat, however,
- * there may be more messages moved to the chat from the deaddrop. To get the
- * chat messages, use dc_get_chat_msgs().
- *
- * If the user is asked before creation, he should be
- * asked whether he wants to chat with the _contact_ belonging to the message;
- * the group names may be really weird when taken from the subject of implicit
- * groups and this may look confusing.
- *
- * Moreover, this function also scales up the origin of the contact belonging
- * to the message and, depending on the contacts origin, messages from the
- * same group may be shown or not - so, all in all, it is fine to show the
- * contact name only.
- *
- * @deprecated Deprecated 2021-02-07, use dc_decide_on_contact_request() instead
- * @memberof dc_context_t
- * @param context The context object as returned from dc_context_new().
- * @param msg_id The message ID to create the chat for.
- * @return The created or reused chat ID on success. 0 on errors.
- */
-uint32_t        dc_create_chat_by_msg_id     (dc_context_t* context, uint32_t msg_id);
-
-
 /**
  * Create a normal chat with a single user.  To create group chats,
  * see dc_create_group_chat().
  *
  * If a chat already exists, this ID is returned, otherwise a new chat is created;
- * this new chat may already contain messages, e.g. from the deaddrop, to get the
- * chat messages, use dc_get_chat_msgs().
+ * to get the chat messages, use dc_get_chat_msgs().
  *
  * @memberof dc_context_t
  * @param context The context object as returned from dc_context_new().
@@ -1078,7 +1099,7 @@ int             dc_estimate_deletion_cnt    (dc_context_t* context, int from_ser
  * or badge counters eg. on the app-icon.
  * The list is already sorted and starts with the most recent fresh message.
  *
- * Messages belonging to muted chats or to the deaddrop are not returned;
+ * Messages belonging to muted chats or to the contact requests are not returned;
  * these messages should not be notified
  * and also badge counters should not include these messages.
  *
@@ -1104,8 +1125,7 @@ dc_array_t*     dc_get_fresh_msgs            (dc_context_t* context);
  *
  * @memberof dc_context_t
  * @param context The context object as returned from dc_context_new().
- * @param chat_id The chat ID of which all messages should be marked as being noticed
- *     (this also works for the virtual chat ID DC_CHAT_ID_DEADDROP).
+ * @param chat_id The chat ID of which all messages should be marked as being noticed.
  */
 void            dc_marknoticed_chat          (dc_context_t* context, uint32_t chat_id);
 
@@ -1214,6 +1234,31 @@ void            dc_set_chat_visibility       (dc_context_t* context, uint32_t ch
  */
 void            dc_delete_chat               (dc_context_t* context, uint32_t chat_id);
 
+/**
+ * Block a chat.
+ *
+ * Blocking 1:1 chats blocks the corresponding contact. Blocking
+ * mailing lists creates a pseudo-contact in the list of blocked
+ * contacts, so blocked mailing lists can be discovered and unblocked
+ * the same way as the contacts.  Blocking group chats deletes the
+ * chat without blocking any contacts, so it may pop up again later.
+ *
+ * @memberof dc_context_t
+ * @param context The context object as returned from dc_context_new().
+ * @param chat_id The ID of the chat to block.
+ */
+void            dc_block_chat                (dc_context_t* context, uint32_t chat_id);
+
+/**
+ * Accept a contact request chat.
+ *
+ * Use it to accept "contact request" chats as indicated by dc_chat_is_contact_request().
+ *
+ * @memberof dc_context_t
+ * @param context The context object as returned from dc_context_new().
+ * @param chat_id The ID of the chat to accept.
+ */
+void            dc_accept_chat               (dc_context_t* context, uint32_t chat_id);
 
 /**
  * Get contact IDs belonging to a chat.
@@ -1225,8 +1270,6 @@ void            dc_delete_chat               (dc_context_t* context, uint32_t ch
  *   explicitly as it may happen that oneself gets removed from a still existing
  *   group
  *
- * - for the deaddrop, the list is empty
- *
  * - for mailing lists, the behavior is not documented currently, we will decide on that later.
  *   for now, the UI should not show the list for mailing lists.
  *   (we do not know all members and there is not always a global mailing list address,
@@ -1574,25 +1617,6 @@ void            dc_delete_msgs               (dc_context_t* context, const uint3
 void            dc_forward_msgs              (dc_context_t* context, const uint32_t* msg_ids, int msg_cnt, uint32_t chat_id);
 
 
-/**
- * Mark all messages sent by the given contact as _noticed_.
- * This function is typically used to ignore a user in the deaddrop temporarily ("Not now" button).
- *
- * The contact is expected to belong to the deaddrop;
- * only one #DC_EVENT_MSGS_NOTICED with chat_id=DC_CHAT_ID_DEADDROP may be emitted.
- *
- * See also dc_marknoticed_chat() and dc_markseen_msgs()
- *
- * @deprecated Deprecated 2021-02-07, use dc_decide_on_contact_request() if the user just hit "Not now" on a button in the deaddrop,
- *      dc_marknoticed_chat() if the user has entered a chat
- *      and dc_markseen_msgs() if the user actually _saw_ a message.
- * @memberof dc_context_t
- * @param context The context object.
- * @param contact_id The contact ID of which all messages should be marked as noticed.
- */
-void            dc_marknoticed_contact       (dc_context_t* context, uint32_t contact_id);
-
-
 /**
  * Mark messages as presented to the user.
  * Typically, UIs call this function on scrolling through the chatlist,
@@ -1604,12 +1628,12 @@ void            dc_marknoticed_contact       (dc_context_t* context, uint32_t co
  *   (if dc_set_config()-options `mdns_enabled` is set)
  *   and the internal state is changed to DC_STATE_IN_SEEN to reflect these actions.
  *
- * - For the deaddrop, no IMAP or MNDs is done
- *   and the internal change is not changed therefore.
+ * - For contact requests, no IMAP or MDNs is done
+ *   and the internal state is not changed therefore.
  *   See also dc_marknoticed_chat().
  *
  * Moreover, timer is started for incoming ephemeral messages.
- * This also happens for messages in the deaddrop.
+ * This also happens for contact requests chats.
  *
  * One #DC_EVENT_MSGS_NOTICED event is emitted per modified chat.
  *
@@ -1636,53 +1660,6 @@ void            dc_markseen_msgs             (dc_context_t* context, const uint3
 dc_msg_t*       dc_get_msg                   (dc_context_t* context, uint32_t msg_id);
 
 
-#define DC_DECISION_START_CHAT 0
-#define DC_DECISION_BLOCK      1
-#define DC_DECISION_NOT_NOW    2
-
-
-/**
- * Call this when the user decided about a deaddrop message ("Do you want to chat with NAME?").
- *
- * Possible decisions are:
- * - DC_DECISION_START_CHAT (0)
- *   - This will create a new chat and return the chat id.
- * - DC_DECISION_BLOCK (1)
- *   - This will block the sender.
- *   - When a new message from the sender arrives,
- *     that will not result in a new contact request.
- *   - The blocked sender will be returned by dc_get_blocked_contacts()
- *     typically, the UI offers an option to unblock senders from there.
- * - DC_DECISION_NOT_NOW (2)
- *   - This will mark all messages from this sender as noticed.
- *   - That the contact request is removed from the chat list.
- *   - When a new message from the sender arrives,
- *     a new contact request with the new message will pop up in the chatlist.
- *   - The contact request stays available in the explicit deaddrop.
- *   - If the contact request is already noticed, nothing happens.
- *
- * If the message belongs to a mailing list,
- * the function makes sure that all messages
- * from the mailing list are blocked or marked as noticed.
- *
- * The user should be asked whether they want to chat with the _contact_ belonging to the message;
- * the group names may be really weird when taken from the subject of implicit (= ad-hoc)
- * groups and this may look confusing. Moreover, this function also scales up the origin of the contact.
- *
- * If the chat belongs to a mailing list, you can also ask
- * "Would you like to read MAILING LIST NAME?"
- * (use dc_msg_get_real_chat_id() to get the chat-id for the contact request
- * and then dc_chat_is_mailing_list(), dc_chat_get_name() and so on)
- *
- * @memberof dc_context_t
- * @param context The context object.
- * @param msg_id ID of Message to decide on.
- * @param decision One of the DC_DECISION_* values.
- * @return The chat id of the created chat, if any.
- */
-uint32_t        dc_decide_on_contact_request (dc_context_t* context, uint32_t msg_id, int decision);
-
-
 // handle contacts
 
 /**
@@ -1876,7 +1853,8 @@ dc_contact_t*   dc_get_contact               (dc_context_t* context, uint32_t co
 
 /**
  * Import/export things.
- * During backup import/export IO must not be started, if needed stop IO using dc_stop_io() first.
+ * During backup import/export IO must not be started,
+ * if needed stop IO using dc_accounts_stop_io() or dc_stop_io() first.
  * What to do is defined by the _what_ parameter which may be one of the following:
  *
  * - **DC_IMEX_EXPORT_BACKUP** (11) - Export a backup to the directory given as `param1`.
@@ -2074,27 +2052,80 @@ void            dc_stop_ongoing_process      (dc_context_t* context);
 #define         DC_QR_TEXT                   330 // text1=text
 #define         DC_QR_URL                    332 // text1=URL
 #define         DC_QR_ERROR                  400 // text1=error string
+#define         DC_QR_WITHDRAW_VERIFYCONTACT 500
+#define         DC_QR_WITHDRAW_VERIFYGROUP   502 // text1=groupname
+#define         DC_QR_REVIVE_VERIFYCONTACT   510
+#define         DC_QR_REVIVE_VERIFYGROUP     512 // text1=groupname
 
 /**
  * Check a scanned QR code.
- * The function should be called after a QR code is scanned.
  * The function takes the raw text scanned and checks what can be done with it.
  *
+ * The UI is supposed to show the result to the user.
+ * In case there are further actions possible,
+ * the UI has to ask the user before doing further steps.
+ *
  * The QR code state is returned in dc_lot_t::state as:
  *
- * - DC_QR_ASK_VERIFYCONTACT with dc_lot_t::id=Contact ID
- * - DC_QR_ASK_VERIFYGROUP withdc_lot_t::text1=Group name
- * - DC_QR_FPR_OK with dc_lot_t::id=Contact ID
- * - DC_QR_FPR_MISMATCH with dc_lot_t::id=Contact ID
+ * - DC_QR_ASK_VERIFYCONTACT with dc_lot_t::id=Contact ID:
+ *   ask whether to verify the contact;
+ *   if so, start the protocol with dc_join_securejoin().
+ *
+ * - DC_QR_ASK_VERIFYGROUP withdc_lot_t::text1=Group name:
+ *   ask whether to join the group;
+ *   if so, start the protocol with dc_join_securejoin().
+ *
+ * - DC_QR_FPR_OK with dc_lot_t::id=Contact ID:
+ *   contact fingerprint verified,
+ *   ask the user if they want to start chatting;
+ *   if so, call dc_create_chat_by_contact_id().
+ *
+ * - DC_QR_FPR_MISMATCH with dc_lot_t::id=Contact ID:
+ *   scanned fingerprint does not match last seen fingerprint.
+ *
  * - DC_QR_FPR_WITHOUT_ADDR with dc_lot_t::test1=Formatted fingerprint
- * - DC_QR_ACCOUNT allows creation of an account, dc_lot_t::text1=domain
- * - DC_QR_WEBRTC_INSTANCE - a shared webrtc-instance
- *   that will be set if dc_set_config_from_qr() is called with the qr-code,
- *   dc_lot_t::text1=domain could be used to ask the user
- * - DC_QR_ADDR with dc_lot_t::id=Contact ID
- * - DC_QR_TEXT with dc_lot_t::text1=Text
- * - DC_QR_URL with dc_lot_t::text1=URL
- * - DC_QR_ERROR with dc_lot_t::text1=Error string
+ *   the scanned QR code contains a fingerprint but no email address;
+ *   suggest the user to establish an encrypted connection first.
+ *
+ * - DC_QR_ACCOUNT dc_lot_t::text1=domain:
+ *   ask the user if they want to create an account on the given domain,
+ *   if so, call dc_set_config_from_qr() and then dc_configure().
+ *
+ * - DC_QR_WEBRTC_INSTANCE with dc_lot_t::text1=domain:
+ *   ask the user if they want to use the given service for video chats;
+ *   if so, call dc_set_config_from_qr().
+ *
+ * - DC_QR_ADDR with dc_lot_t::id=Contact ID:
+ *   email-address scanned,
+ *   ask the user if they want to start chatting;
+ *   if so, call dc_create_chat_by_contact_id()
+ *
+ * - DC_QR_TEXT with dc_lot_t::text1=Text:
+ *   Text scanned,
+ *   ask the user eg. if they want copy to clipboard.
+ *
+ * - DC_QR_URL with dc_lot_t::text1=URL:
+ *   URL scanned,
+ *   ask the user eg. if they want to open a browser or copy to clipboard.
+ *
+ * - DC_QR_ERROR with dc_lot_t::text1=Error string:
+ *   show the error to the user.
+ *
+ * - DC_QR_WITHDRAW_VERIFYCONTACT:
+ *   ask the user if they want to withdraw the their own qr-code;
+ *   if so, call dc_set_config_from_qr().
+ *
+ * - DC_QR_WITHDRAW_VERIFYGROUP with text1=groupname:
+ *   ask the user if they want to withdraw the group-invite code;
+ *   if so, call dc_set_config_from_qr().
+ *
+ * - DC_QR_REVIVE_VERIFYCONTACT:
+ *   ask the user if they want to revive their withdrawn qr-code;
+ *   if so, call dc_set_config_from_qr().
+ *
+ * - DC_QR_REVIVE_VERIFYGROUP with text1=groupname:
+ *   ask the user if they want to revive the withdrawn group-invite code;
+ *   if so, call dc_set_config_from_qr().
  *
  * @memberof dc_context_t
  * @param context The context object.
@@ -2357,7 +2388,7 @@ void dc_str_unref (char* str);
  * The account manager takes an directory
  * where all context-databases are placed in.
  * To add a context to the account manager,
- * use dc_accounts_add_account(), dc_accounts_import_account or dc_accounts_migrate_account().
+ * use dc_accounts_add_account() or dc_accounts_migrate_account().
  * All account information are persisted.
  * To remove a context from the account manager,
  * use dc_accounts_remove_account().
@@ -2402,21 +2433,6 @@ void           dc_accounts_unref                (dc_accounts_t* accounts);
 uint32_t       dc_accounts_add_account          (dc_accounts_t* accounts);
 
 
-/**
- * Import a tarfile-backup to the account manager.
- * On success, a new account is added to the account-manager,
- * with all the data provided by the backup-file.
- * Moreover, the newly created account will be the selected one.
- *
- * @memberof dc_accounts_t
- * @param accounts Account manager as created by dc_accounts_new().
- * @param tarfile Backup as created by dc_imex().
- * @return Account-id, use dc_accounts_get_account() to get the context object.
- *     On errors, 0 is returned.
- */
-uint32_t       dc_accounts_import_account       (dc_accounts_t* accounts, const char* tarfile);
-
-
 /**
  * Migrate independent accounts into accounts managed by the account manager.
  * This will _move_ the database-file and all blob-files to the directory managed
@@ -2486,6 +2502,7 @@ dc_context_t*  dc_accounts_get_account          (dc_accounts_t* accounts, uint32
  *     unmanaged account-context as created by dc_context_new().
  *     Once you do no longer need the context-object, you have to call dc_context_unref() on it,
  *     which, however, will not close the account but only decrease a reference counter.
+ *     If there is no selected account, NULL is returned.
  */
 dc_context_t*  dc_accounts_get_selected_account (dc_accounts_t* accounts);
 
@@ -2501,6 +2518,23 @@ dc_context_t*  dc_accounts_get_selected_account (dc_accounts_t* accounts);
 int            dc_accounts_select_account       (dc_accounts_t* accounts, uint32_t account_id);
 
 
+/**
+ * This is meant especially for iOS, because iOS needs to tell the system when its background work is done.
+ *
+ * iOS can:
+ * - call dc_start_io() (in case IO was not running)
+ * - call dc_maybe_network()
+ * - while dc_accounts_all_work_done() returns false:
+ *   -  Wait for #DC_EVENT_CONNECTIVITY_CHANGED
+ *
+ * @memberof dc_accounts_t
+ * @param accounts Account manager as created by dc_accounts_new().
+ * @return Whether all accounts finished their background work.
+ *      #DC_EVENT_CONNECTIVITY_CHANGED will be sent when this turns to true.
+ */
+int            dc_accounts_all_work_done        (dc_accounts_t* accounts);
+
+
 /**
  * Start job and IMAP/SMTP tasks for all accounts managed by the account manager.
  * If IO is already running, nothing happens.
@@ -2536,6 +2570,21 @@ void           dc_accounts_stop_io              (dc_accounts_t* accounts);
 void           dc_accounts_maybe_network        (dc_accounts_t* accounts);
 
 
+/**
+ * This function can be called when there is a hint that the network is lost.
+ * This is similar to dc_accounts_maybe_network(), however,
+ * it does not retry job processing.
+ *
+ * dc_accounts_maybe_network_lost() is needed only on systems
+ * where the core cannot find out the connectivity loss on its own, eg. iOS.
+ * The function is not needed on Android, MacOS, Windows or Linux.
+ *
+ * @memberof dc_accounts_t
+ * @param accounts Account manager as created by dc_accounts_new().
+ */
+void           dc_accounts_maybe_network_lost    (dc_accounts_t* accounts);
+
+
 /**
  * Create the event emitter that is used to receive events.
  *
@@ -2755,15 +2804,9 @@ int              dc_array_search_id          (const dc_array_t* array, uint32_t
  * and for each messages that is scrolled into view, dc_get_msg() is called then.
  *
  * Why no listflags?
- * Without listflags, dc_get_chatlist() adds the deaddrop
- * and the archive "link" automatically as needed.
+ * Without listflags, dc_get_chatlist() adds
+ * the archive "link" automatically as needed.
  * The UI can just render these items differently then.
- * Although the deaddrop link is currently always the first entry
- * and only present on new messages,
- * there is the rough idea that it can be optionally always present
- * and sorted into the list by date.
- * Rendering the deaddrop in the described way
- * would not add extra work in the UI then.
  */
 
 
@@ -2904,7 +2947,6 @@ char*            dc_chat_get_info_json       (dc_context_t* context, size_t chat
  */
 
 
-#define         DC_CHAT_ID_DEADDROP          1 // virtual chat showing all messages belonging to chats flagged with chats.blocked=2
 #define         DC_CHAT_ID_TRASH             3 // messages that should be deleted get this chat_id; the messages are deleted from the working thread later then. This is also needed as rfc724_mid should be preset as long as the message is not deleted on the server (otherwise it is downloaded again)
 #define         DC_CHAT_ID_ARCHIVED_LINK     6 // only an indicator in a chatlist
 #define         DC_CHAT_ID_ALLDONE_HINT      7 // only an indicator in a chatlist
@@ -2931,7 +2973,6 @@ void            dc_chat_unref                (dc_chat_t* chat);
  * Get chat ID. The chat ID is the ID under which the chat is filed in the database.
  *
  * Special IDs:
- * - DC_CHAT_ID_DEADDROP         (1) - Virtual chat containing messages which senders are not confirmed by the user.
  * - DC_CHAT_ID_ARCHIVED_LINK    (6) - A link at the end of the chatlist, if present the UI should show the button "Archived chats"-
  *
  * "Normal" chat IDs are larger than these special IDs (larger than DC_CHAT_ID_LAST_SPECIAL).
@@ -3023,6 +3064,25 @@ uint32_t        dc_chat_get_color            (const dc_chat_t* chat);
 int             dc_chat_get_visibility       (const dc_chat_t* chat);
 
 
+/**
+ * Check if a chat is a contact request chat.
+ *
+ * UI should display such chats with a [New] badge in the chatlist.
+ *
+ * When such chat is opened, user should be presented with a set of
+ * options instead of the message composition area, for example:
+ * - Accept chat (dc_accept_chat())
+ * - Block chat (dc_block_chat())
+ * - Delete chat (dc_delete_chat())
+ *
+ * @memberof dc_chat_t
+ * @param chat The chat object.
+ * @return 1=chat is a contact request chat
+ *     0=chat is not a contact request chat
+ */
+int             dc_chat_is_contact_request   (const dc_chat_t* chat);
+
+
 /**
  * Check if a group chat is still unpromoted.
  *
@@ -3076,7 +3136,7 @@ int             dc_chat_is_device_talk       (const dc_chat_t* chat);
 
 /**
  * Check if messages can be sent to a given chat.
- * This is not true e.g. for the deaddrop or for the device-talk, cmp. dc_chat_is_device_talk().
+ * This is not true e.g. for contact requests or for the device-talk, cmp. dc_chat_is_device_talk().
  *
  * Calling dc_send_msg() for these chats will fail
  * and the UI may decide to hide input controls therefore.
@@ -3220,9 +3280,6 @@ uint32_t        dc_msg_get_from_id            (const dc_msg_t* msg);
 /**
  * Get the ID of chat the message belongs to.
  * To get details about the chat, pass the returned ID to dc_get_chat().
- * If a message is still in the deaddrop, the ID DC_CHAT_ID_DEADDROP is returned
- * although internally another ID is used.
- * (to get that internal id, use dc_msg_get_real_chat_id())
  *
  * @memberof dc_msg_t
  * @param msg The message object.
@@ -3231,19 +3288,6 @@ uint32_t        dc_msg_get_from_id            (const dc_msg_t* msg);
 uint32_t        dc_msg_get_chat_id            (const dc_msg_t* msg);
 
 
-/**
- * Get the ID of chat the message belongs to.
- * To get details about the chat, pass the returned ID to dc_get_chat().
- * In contrast to dc_msg_get_chat_id(), this function returns the chat-id also
- * for messages in the deaddrop.
- *
- * @memberof dc_msg_t
- * @param msg The message object.
- * @return The ID of the chat the message belongs to, 0 on errors.
- */
-uint32_t        dc_msg_get_real_chat_id       (const dc_msg_t* msg);
-
-
 /**
  * Get the type of the message.
  *
@@ -3493,6 +3537,16 @@ int             dc_msg_get_duration           (const dc_msg_t* msg);
  */
 int             dc_msg_get_showpadlock        (const dc_msg_t* msg);
 
+/**
+ * Check if incoming message is a bot message, i.e. automatically submitted.
+ *
+ * Return value for outgoing messages is unspecified.
+ *
+ * @memberof dc_msg_t
+ * @param msg The message object.
+ * @return 1=message is submitted automatically, 0=message is not automatically submitted.
+ */
+int             dc_msg_is_bot                 (const dc_msg_t* msg); 
 
 /**
  * Get ephemeral timer duration for message.
@@ -4902,26 +4956,6 @@ void dc_event_unref(dc_event_t* event);
 #define DC_EVENT_ERROR                    400
 
 
-/**
- * An action cannot be performed because there is no network available.
- *
- * The library will typically try over after a some time
- * and when dc_maybe_network() is called.
- *
- * Network errors should be reported to users in a non-disturbing way,
- * however, as network errors may come in a sequence,
- * it is not useful to raise each an every error to the user.
- *
- * Moreover, if the UI detects that the device is offline,
- * it is probably more useful to report this to the user
- * instead of the string from data2.
- *
- * @param data1 0
- * @param data2 (char*) Error string, always set, never NULL.
- */
-#define DC_EVENT_ERROR_NETWORK            401
-
-
 /**
  * An action cannot be performed because the user is not in the group.
  * Reported e.g. after a call to
@@ -5106,6 +5140,18 @@ void dc_event_unref(dc_event_t* event);
  */
 #define DC_EVENT_SECUREJOIN_JOINER_PROGRESS       2061
 
+
+/**
+ * The connectivity to the server changed.
+ * This means that you should refresh the connectivity view
+ * and possibly the connectivtiy HTML; see dc_get_connectivity() and
+ * dc_get_connectivity_html() for details.
+ *
+ * @param data1 0
+ * @param data2 0
+ */
+#define DC_EVENT_CONNECTIVITY_CHANGED             2100
+
 /**
  * @}
  */
@@ -5274,11 +5320,6 @@ void dc_event_unref(dc_event_t* event);
 /// Used in summaries.
 #define DC_STR_VOICEMESSAGE               7
 
-/// "Contact requests"
-///
-/// Used as the name for the corresponding chat.
-#define DC_STR_DEADDROP                   8
-
 /// "Image"
 ///
 /// Used in summaries.
@@ -5433,13 +5474,6 @@ void dc_event_unref(dc_event_t* event);
 /// - %1$s will be replaced by the failing login name
 #define DC_STR_CANNOT_LOGIN               60
 
-/// "Could not connect to %1$s: %2$s"
-///
-/// Used in error strings.
-/// - %1$s will be replaced by the failing server
-/// - %2$s by a the error message as returned from the server
-#define DC_STR_SERVER_RESPONSE            61
-
 /// "%1$s by %2$s"
 ///
 /// Used to concretize actions,

deltachat-ffi/deltachat.pc.in

@@ -1,6 +1,6 @@
 prefix={prefix}
-libdir=${{prefix}}/lib
-includedir=${{prefix}}/include
+libdir={libdir}
+includedir={includedir}
 
 Name: {name}
 Description: {description}

deltachat-ffi/src/lib.rs

@@ -256,6 +256,38 @@ fn render_info(
     Ok(res)
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn dc_get_connectivity(context: *const dc_context_t) -> libc::c_int {
+    if context.is_null() {
+        eprintln!("ignoring careless call to dc_get_connectivity()");
+        return 0;
+    }
+    let ctx = &*context;
+    block_on(async move { ctx.get_connectivity().await as u32 as libc::c_int })
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn dc_get_connectivity_html(
+    context: *const dc_context_t,
+) -> *mut libc::c_char {
+    if context.is_null() {
+        eprintln!("ignoring careless call to dc_get_connectivity_html()");
+        return "".strdup();
+    }
+    let ctx = &*context;
+    block_on(async move { ctx.get_connectivity_html().await.strdup() })
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn dc_all_work_done(context: *mut dc_context_t) -> libc::c_int {
+    if context.is_null() {
+        eprintln!("ignoring careless call to dc_all_work_done()");
+        return 0;
+    }
+    let ctx = &*context;
+    block_on(async move { ctx.all_work_done().await as libc::c_int })
+}
+
 #[no_mangle]
 pub unsafe extern "C" fn dc_get_oauth2_url(
     context: *mut dc_context_t,
@@ -368,7 +400,7 @@ pub unsafe extern "C" fn dc_event_get_data1_int(event: *mut dc_event_t) -> libc:
         | EventType::DeletedBlobFile(_)
         | EventType::Warning(_)
         | EventType::Error(_)
-        | EventType::ErrorNetwork(_)
+        | EventType::ConnectivityChanged
         | EventType::ErrorSelfNotInGroup(_) => 0,
         EventType::MsgsChanged { chat_id, .. }
         | EventType::IncomingMsg { chat_id, .. }
@@ -411,7 +443,6 @@ pub unsafe extern "C" fn dc_event_get_data2_int(event: *mut dc_event_t) -> libc:
         | EventType::DeletedBlobFile(_)
         | EventType::Warning(_)
         | EventType::Error(_)
-        | EventType::ErrorNetwork(_)
         | EventType::ErrorSelfNotInGroup(_)
         | EventType::ContactsChanged(_)
         | EventType::LocationChanged(_)
@@ -419,6 +450,7 @@ pub unsafe extern "C" fn dc_event_get_data2_int(event: *mut dc_event_t) -> libc:
         | EventType::ImexProgress(_)
         | EventType::ImexFileWritten(_)
         | EventType::MsgsNoticed(_)
+        | EventType::ConnectivityChanged
         | EventType::ChatModified(_) => 0,
         EventType::MsgsChanged { msg_id, .. }
         | EventType::IncomingMsg { msg_id, .. }
@@ -451,7 +483,6 @@ pub unsafe extern "C" fn dc_event_get_data2_str(event: *mut dc_event_t) -> *mut
         | EventType::DeletedBlobFile(msg)
         | EventType::Warning(msg)
         | EventType::Error(msg)
-        | EventType::ErrorNetwork(msg)
         | EventType::ErrorSelfNotInGroup(msg) => {
             let data2 = msg.to_c_string().unwrap_or_default();
             data2.into_raw()
@@ -468,6 +499,7 @@ pub unsafe extern "C" fn dc_event_get_data2_str(event: *mut dc_event_t) -> *mut
         | EventType::ImexProgress(_)
         | EventType::SecurejoinInviterProgress { .. }
         | EventType::SecurejoinJoinerProgress { .. }
+        | EventType::ConnectivityChanged
         | EventType::ChatEphemeralTimerModified { .. } => ptr::null_mut(),
         EventType::ConfigureProgress { comment, .. } => {
             if let Some(comment) = comment {
@@ -610,23 +642,6 @@ pub unsafe extern "C" fn dc_get_chatlist(
     })
 }
 
-#[no_mangle]
-pub unsafe extern "C" fn dc_create_chat_by_msg_id(context: *mut dc_context_t, msg_id: u32) -> u32 {
-    if context.is_null() {
-        eprintln!("ignoring careless call to dc_create_chat_by_msg_id()");
-        return 0;
-    }
-    let ctx = &*context;
-
-    block_on(async move {
-        chat::create_by_msg_id(&ctx, MsgId::new(msg_id))
-            .await
-            .log_err(ctx, "Failed to create chat from msg_id")
-            .map(|id| id.to_u32())
-            .unwrap_or(0)
-    })
-}
-
 #[no_mangle]
 pub unsafe extern "C" fn dc_create_chat_by_contact_id(
     context: *mut dc_context_t,
@@ -1143,8 +1158,39 @@ pub unsafe extern "C" fn dc_delete_chat(context: *mut dc_context_t, chat_id: u32
         ChatId::new(chat_id)
             .delete(&ctx)
             .await
-            .log_err(ctx, "Failed chat delete")
-            .unwrap_or(())
+            .ok_or_log_msg(ctx, "Failed chat delete");
+    })
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn dc_block_chat(context: *mut dc_context_t, chat_id: u32) {
+    if context.is_null() {
+        eprintln!("ignoring careless call to dc_block_chat()");
+        return;
+    }
+    let ctx = &*context;
+
+    block_on(async move {
+        ChatId::new(chat_id)
+            .block(&ctx)
+            .await
+            .ok_or_log_msg(ctx, "Failed chat block");
+    })
+}
+
+#[no_mangle]
+pub unsafe extern "C" fn dc_accept_chat(context: *mut dc_context_t, chat_id: u32) {
+    if context.is_null() {
+        eprintln!("ignoring careless call to dc_accept_chat()");
+        return;
+    }
+    let ctx = &*context;
+
+    block_on(async move {
+        ChatId::new(chat_id)
+            .accept(&ctx)
+            .await
+            .ok_or_log_msg(ctx, "Failed chat accept");
     })
 }
 
@@ -1537,17 +1583,6 @@ pub unsafe extern "C" fn dc_forward_msgs(
     })
 }
 
-#[no_mangle]
-pub unsafe extern "C" fn dc_marknoticed_contact(context: *mut dc_context_t, contact_id: u32) {
-    if context.is_null() {
-        eprintln!("ignoring careless call to dc_marknoticed_contact()");
-        return;
-    }
-    let ctx = &*context;
-
-    block_on(Contact::mark_noticed(&ctx, contact_id))
-}
-
 #[no_mangle]
 pub unsafe extern "C" fn dc_markseen_msgs(
     context: *mut dc_context_t,
@@ -1737,9 +1772,13 @@ pub unsafe extern "C" fn dc_block_contact(
     let ctx = &*context;
     block_on(async move {
         if block == 0 {
-            Contact::unblock(&ctx, contact_id).await;
+            Contact::unblock(&ctx, contact_id)
+                .await
+                .ok_or_log_msg(&ctx, "Can't unblock contact");
         } else {
-            Contact::block(&ctx, contact_id).await;
+            Contact::block(&ctx, contact_id)
+                .await
+                .ok_or_log_msg(&ctx, "Can't block contact");
         }
     });
 }
@@ -2314,11 +2353,14 @@ pub unsafe extern "C" fn dc_chatlist_get_msg_id(
         return 0;
     }
     let ffi_list = &*chatlist;
-    ffi_list
-        .list
-        .get_msg_id(index as usize)
-        .map(|msg_id| msg_id.to_u32())
-        .unwrap_or(0)
+    let ctx = &*ffi_list.context;
+    match ffi_list.list.get_msg_id(index as usize) {
+        Ok(msg_id) => msg_id.map_or(0, |msg_id| msg_id.to_u32()),
+        Err(err) => {
+            warn!(ctx, "get_msg_id failed: {}", err);
+            0
+        }
+    }
 }
 
 #[no_mangle]
@@ -2344,7 +2386,9 @@ pub unsafe extern "C" fn dc_chatlist_get_summary(
         let lot = ffi_list
             .list
             .get_summary(&ctx, index as usize, maybe_chat)
-            .await;
+            .await
+            .log_err(ctx, "get_summary failed")
+            .unwrap_or_default();
         Box::into_raw(Box::new(lot))
     })
 }
@@ -2360,9 +2404,16 @@ pub unsafe extern "C" fn dc_chatlist_get_summary2(
         return ptr::null_mut();
     }
     let ctx = &*context;
+    let msg_id = if msg_id == 0 {
+        None
+    } else {
+        Some(MsgId::new(msg_id))
+    };
     block_on(async move {
-        let lot =
-            Chatlist::get_summary2(&ctx, ChatId::new(chat_id), MsgId::new(msg_id), None).await;
+        let lot = Chatlist::get_summary2(&ctx, ChatId::new(chat_id), msg_id, None)
+            .await
+            .log_err(ctx, "get_summary2 failed")
+            .unwrap_or_default();
         Box::into_raw(Box::new(lot))
     })
 }
@@ -2482,6 +2533,16 @@ pub unsafe extern "C" fn dc_chat_get_visibility(chat: *mut dc_chat_t) -> libc::c
     }
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn dc_chat_is_contact_request(chat: *mut dc_chat_t) -> libc::c_int {
+    if chat.is_null() {
+        eprintln!("ignoring careless call to dc_chat_is_contact_request()");
+        return 0;
+    }
+    let ffi_chat = &*chat;
+    ffi_chat.chat.is_contact_request() as libc::c_int
+}
+
 #[no_mangle]
 pub unsafe extern "C" fn dc_chat_is_unpromoted(chat: *mut dc_chat_t) -> libc::c_int {
     if chat.is_null() {
@@ -2519,7 +2580,8 @@ pub unsafe extern "C" fn dc_chat_can_send(chat: *mut dc_chat_t) -> libc::c_int {
         return 0;
     }
     let ffi_chat = &*chat;
-    ffi_chat.chat.can_send() as libc::c_int
+    let cxt = &*ffi_chat.context;
+    block_on(ffi_chat.chat.can_send(cxt)) as libc::c_int
 }
 
 #[no_mangle]
@@ -2682,16 +2744,6 @@ pub unsafe extern "C" fn dc_msg_get_chat_id(msg: *mut dc_msg_t) -> u32 {
     ffi_msg.message.get_chat_id().to_u32()
 }
 
-#[no_mangle]
-pub unsafe extern "C" fn dc_msg_get_real_chat_id(msg: *mut dc_msg_t) -> u32 {
-    if msg.is_null() {
-        eprintln!("ignoring careless call to dc_msg_get_real_chat_id()");
-        return 0;
-    }
-    let ffi_msg = &*msg;
-    ffi_msg.message.get_real_chat_id().to_u32()
-}
-
 #[no_mangle]
 pub unsafe extern "C" fn dc_msg_get_viewtype(msg: *mut dc_msg_t) -> libc::c_int {
     if msg.is_null() {
@@ -2857,6 +2909,16 @@ pub unsafe extern "C" fn dc_msg_get_showpadlock(msg: *mut dc_msg_t) -> libc::c_i
     ffi_msg.message.get_showpadlock() as libc::c_int
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn dc_msg_is_bot(msg: *mut dc_msg_t) -> libc::c_int {
+    if msg.is_null() {
+        eprintln!("ignoring careless call to dc_msg_is_bot()");
+        return 0;
+    }
+    let ffi_msg = &*msg;
+    ffi_msg.message.is_bot() as libc::c_int
+}
+
 #[no_mangle]
 pub unsafe extern "C" fn dc_msg_get_ephemeral_timer(msg: *mut dc_msg_t) -> u32 {
     if msg.is_null() {
@@ -3037,32 +3099,6 @@ pub unsafe extern "C" fn dc_msg_get_videochat_url(msg: *mut dc_msg_t) -> *mut li
         .strdup()
 }
 
-#[no_mangle]
-pub unsafe extern "C" fn dc_decide_on_contact_request(
-    context: *mut dc_context_t,
-    msg_id: u32,
-    decision: libc::c_int,
-) -> u32 {
-    if context.is_null() || msg_id <= constants::DC_MSG_ID_LAST_SPECIAL as u32 {
-        eprintln!("ignoring careless call to dc_decide_on_contact_request()");
-    }
-    let ctx = &*context;
-
-    match from_prim(decision) {
-        None => {
-            warn!(ctx, "{} is not a valid decision, ignoring", decision);
-            0
-        }
-        Some(d) => block_on(message::decide_on_contact_request(
-            ctx,
-            MsgId::new(msg_id),
-            d,
-        ))
-        .unwrap_or_default()
-        .to_u32(),
-    }
-}
-
 #[no_mangle]
 pub unsafe extern "C" fn dc_msg_get_videochat_type(msg: *mut dc_msg_t) -> libc::c_int {
     if msg.is_null() {
@@ -3691,8 +3727,9 @@ pub unsafe extern "C" fn dc_accounts_get_selected_account(
     }
 
     let accounts = &*accounts;
-    let ctx = block_on(accounts.get_selected_account());
-    Box::into_raw(Box::new(ctx))
+    block_on(accounts.get_selected_account())
+        .map(|ctx| Box::into_raw(Box::new(ctx)))
+        .unwrap_or_else(std::ptr::null_mut)
 }
 
 #[no_mangle]
@@ -3773,20 +3810,13 @@ pub unsafe extern "C" fn dc_accounts_get_all(accounts: *mut dc_accounts_t) -> *m
 }
 
 #[no_mangle]
-pub unsafe extern "C" fn dc_accounts_import_account(
-    accounts: *mut dc_accounts_t,
-    file: *const libc::c_char,
-) -> u32 {
-    if accounts.is_null() || file.is_null() {
-        eprintln!("ignoring careless call to dc_accounts_import_account()");
+pub unsafe extern "C" fn dc_accounts_all_work_done(accounts: *mut dc_accounts_t) -> libc::c_int {
+    if accounts.is_null() {
+        eprintln!("ignoring careless call to dc_accounts_all_work_done()");
         return 0;
     }
-
     let accounts = &*accounts;
-    let file = to_string_lossy(file);
-    block_on(accounts.import_account(async_std::path::PathBuf::from(file)))
-        .map(|_| 1)
-        .unwrap_or_else(|_| 0)
+    block_on(async move { accounts.all_work_done().await as libc::c_int })
 }
 
 #[no_mangle]
@@ -3814,7 +3844,7 @@ pub unsafe extern "C" fn dc_accounts_stop_io(accounts: *mut dc_accounts_t) {
 #[no_mangle]
 pub unsafe extern "C" fn dc_accounts_maybe_network(accounts: *mut dc_accounts_t) {
     if accounts.is_null() {
-        eprintln!("ignoring careless call to dc_accounts_mabye_network()");
+        eprintln!("ignoring careless call to dc_accounts_maybe_network()");
         return;
     }
 
@@ -3822,6 +3852,17 @@ pub unsafe extern "C" fn dc_accounts_maybe_network(accounts: *mut dc_accounts_t)
     block_on(accounts.maybe_network());
 }
 
+#[no_mangle]
+pub unsafe extern "C" fn dc_accounts_maybe_network_lost(accounts: *mut dc_accounts_t) {
+    if accounts.is_null() {
+        eprintln!("ignoring careless call to dc_accounts_maybe_network_lost()");
+        return;
+    }
+
+    let accounts = &*accounts;
+    block_on(accounts.maybe_network_lost());
+}
+
 pub type dc_accounts_event_emitter_t = deltachat::accounts::EventEmitter;
 
 #[no_mangle]

deltachat-ffi/src/string.rs

@@ -170,15 +170,20 @@ pub(crate) trait Strdup {
     unsafe fn strdup(&self) -> *mut libc::c_char;
 }
 
-impl<T: AsRef<str>> Strdup for T {
+impl Strdup for str {
     unsafe fn strdup(&self) -> *mut libc::c_char {
-        let tmp = CString::new_lossy(self.as_ref());
+        let tmp = CString::new_lossy(self);
         dc_strdup(tmp.as_ptr())
     }
 }
 
-// We can not implement for AsRef<OsStr> because we already implement
-// AsRev<str> and this conflicts.  So implement for Path directly.
+impl Strdup for String {
+    unsafe fn strdup(&self) -> *mut libc::c_char {
+        let s: &str = self;
+        s.strdup()
+    }
+}
+
 impl Strdup for std::path::Path {
     unsafe fn strdup(&self) -> *mut libc::c_char {
         let tmp = self.to_c_string().unwrap_or_else(|_| CString::default());
@@ -186,6 +191,13 @@ impl Strdup for std::path::Path {
     }
 }
 
+impl Strdup for [u8] {
+    unsafe fn strdup(&self) -> *mut libc::c_char {
+        let tmp = CString::new_lossy(self);
+        dc_strdup(tmp.as_ptr())
+    }
+}
+
 /// Convenience methods to turn optional strings into C strings.
 ///
 /// This is the same as the [Strdup] trait but a different trait name

deltachat_derive/Cargo.toml

@@ -9,5 +9,5 @@ license = "MPL-2.0"
 proc-macro = true
 
 [dependencies]
-syn = "1.0.72"
+syn = "1.0.74"
 quote = "1.0.2"

examples/repl/cmdline.rs

@@ -13,13 +13,11 @@ use deltachat::contact::*;
 use deltachat::context::*;
 use deltachat::dc_receive_imf::*;
 use deltachat::dc_tools::*;
-use deltachat::error::Error;
-use deltachat::export_chat::export_chat_to_zip;
 use deltachat::imex::*;
 use deltachat::location;
 use deltachat::log::LogExt;
 use deltachat::lot::LotState;
-use deltachat::message::{self, ContactRequestDecision, Message, MessageState, MsgId};
+use deltachat::message::{self, Message, MessageState, MsgId};
 use deltachat::peerstate::*;
 use deltachat::qr::*;
 use deltachat::sql;
@@ -353,6 +351,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                  configure\n\
                  connect\n\
                  disconnect\n\
+                 connectivity\n\
                  maybenetwork\n\
                  housekeeping\n\
                  help imex (Import/Export)\n\
@@ -374,6 +373,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                  getlocations [<contact-id>]\n\
                  send <text>\n\
                  sendimage <file> [<text>]\n\
+                 sendsticker <file> [<text>]\n\
                  sendfile <file> [<text>]\n\
                  sendhtml <file for html-part> [<text for plain-part>]\n\
                  videochat\n\
@@ -389,11 +389,8 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                  protect <chat-id>\n\
                  unprotect <chat-id>\n\
                  delchat <chat-id>\n\
-                 export-chat <chat-id> <destination-file>\n\
-                 ===========================Contact requests==\n\
-                 decidestartchat <msg-id>\n\
-                 decideblock <msg-id>\n\
-                 decidenotnow <msg-id>\n\
+                 accept <chat-id>\n\
+                 decline <chat-id>\n\
                  ===========================Message commands==\n\
                  listmsgs <query>\n\
                  msginfo <msg-id>\n\
@@ -513,6 +510,14 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
         "info" => {
             println!("{:#?}", context.get_info().await);
         }
+        "connectivity" => {
+            let file = dirs::home_dir()
+                .unwrap_or_default()
+                .join("connectivity.html");
+            let html = context.get_connectivity_html().await;
+            fs::write(&file, html)?;
+            println!("Report written to: {:#?}", file);
+        }
         "maybenetwork" => {
             context.maybe_network().await;
         }
@@ -540,7 +545,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                 for i in (0..cnt).rev() {
                     let chat = Chat::load_from_db(&context, chatlist.get_chat_id(i)).await?;
                     println!(
-                        "{}#{}: {} [{} fresh] {}{}{}",
+                        "{}#{}: {} [{} fresh] {}{}{}{}",
                         chat_prefix(&chat),
                         chat.get_id(),
                         chat.get_name(),
@@ -552,8 +557,13 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
                             ChatVisibility::Pinned => "📌",
                         },
                         if chat.is_protected() { "🛡️" } else { "" },
+                        if chat.is_contact_request() {
+                            "🆕"
+                        } else {
+                            ""
+                        },
                     );
-                    let lot = chatlist.get_summary(&context, i, Some(&chat)).await;
+                    let lot = chatlist.get_summary(&context, i, Some(&chat)).await?;
                     let statestr = if chat.visibility == ChatVisibility::Archived {
                         " [Archived]"
                     } else {
@@ -682,35 +692,6 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
 
             println!("Single#{} created successfully.", chat_id,);
         }
-        "decidestartchat" | "createchatbymsg" => {
-            ensure!(!arg1.is_empty(), "Argument <msg-id> missing");
-            let msg_id = MsgId::new(arg1.parse()?);
-            match message::decide_on_contact_request(
-                &context,
-                msg_id,
-                ContactRequestDecision::StartChat,
-            )
-            .await
-            {
-                Some(chat_id) => {
-                    let chat = Chat::load_from_db(&context, chat_id).await?;
-                    println!("{}#{} created successfully.", chat_prefix(&chat), chat_id);
-                }
-                None => println!("Cannot crate chat."),
-            }
-        }
-        "decidenotnow" => {
-            ensure!(!arg1.is_empty(), "Argument <msg-id> missing");
-            let msg_id = MsgId::new(arg1.parse()?);
-            message::decide_on_contact_request(&context, msg_id, ContactRequestDecision::NotNow)
-                .await;
-        }
-        "decideblock" => {
-            ensure!(!arg1.is_empty(), "Argument <msg-id> missing");
-            let msg_id = MsgId::new(arg1.parse()?);
-            message::decide_on_contact_request(&context, msg_id, ContactRequestDecision::Block)
-                .await;
-        }
         "creategroup" => {
             ensure!(!arg1.is_empty(), "Argument <name> missing.");
             let chat_id =
@@ -868,12 +849,14 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
             ensure!(sel_chat.is_some(), "No chat selected.");
             chat::send_text_msg(&context, sel_chat.as_ref().unwrap().get_id(), "".into()).await?;
         }
-        "sendimage" | "sendfile" => {
+        "sendimage" | "sendsticker" | "sendfile" => {
             ensure!(sel_chat.is_some(), "No chat selected.");
             ensure!(!arg1.is_empty(), "No file given.");
 
             let mut msg = Message::new(if arg0 == "sendimage" {
                 Viewtype::Image
+            } else if arg0 == "sendsticker" {
+                Viewtype::Sticker
             } else {
                 Viewtype::File
             });
@@ -1025,12 +1008,15 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
             let chat_id = ChatId::new(arg1.parse()?);
             chat_id.delete(&context).await?;
         }
-        "export-chat" => {
+        "accept" => {
             ensure!(!arg1.is_empty(), "Argument <chat-id> missing.");
-            ensure!(!arg2.is_empty(), "Argument <destination file> missing.");
             let chat_id = ChatId::new(arg1.parse()?);
-            // todo check if path is valid (dest dir exists) and ends in .zip
-            export_chat_to_zip(&context, chat_id, arg2).await;
+            chat_id.accept(&context).await?;
+        }
+        "blockchat" => {
+            ensure!(!arg1.is_empty(), "Argument <chat-id> missing.");
+            let chat_id = ChatId::new(arg1.parse()?);
+            chat_id.block(&context).await?;
         }
         "msginfo" => {
             ensure!(!arg1.is_empty(), "Argument <msg-id> missing.");
@@ -1144,12 +1130,12 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
         "block" => {
             ensure!(!arg1.is_empty(), "Argument <contact-id> missing.");
             let contact_id = arg1.parse()?;
-            Contact::block(&context, contact_id).await;
+            Contact::block(&context, contact_id).await?;
         }
         "unblock" => {
             ensure!(!arg1.is_empty(), "Argument <contact-id> missing.");
             let contact_id = arg1.parse()?;
-            Contact::unblock(&context, contact_id).await;
+            Contact::unblock(&context, contact_id).await?;
         }
         "listblocked" => {
             let contacts = Contact::get_all_blocked(&context).await?;

examples/repl/main.rs

@@ -57,9 +57,6 @@ fn receive_event(event: EventType) {
         EventType::Error(msg) => {
             error!("{}", msg);
         }
-        EventType::ErrorNetwork(msg) => {
-            error!("[NETWORK] msg={}", msg);
-        }
         EventType::ErrorSelfNotInGroup(msg) => {
             error!("[SELF_NOT_IN_GROUP] {}", msg);
         }
@@ -157,7 +154,7 @@ const IMEX_COMMANDS: [&str; 12] = [
     "stop",
 ];
 
-const DB_COMMANDS: [&str; 9] = [
+const DB_COMMANDS: [&str; 10] = [
     "info",
     "set",
     "get",
@@ -165,18 +162,16 @@ const DB_COMMANDS: [&str; 9] = [
     "configure",
     "connect",
     "disconnect",
+    "connectivity",
     "maybenetwork",
     "housekeeping",
 ];
 
-const CHAT_COMMANDS: [&str; 35] = [
+const CHAT_COMMANDS: [&str; 33] = [
     "listchats",
     "listarchived",
     "chat",
     "createchat",
-    "decidestartchat",
-    "decideblock",
-    "decidenotnow",
     "creategroup",
     "createverified",
     "addmember",
@@ -204,7 +199,8 @@ const CHAT_COMMANDS: [&str; 35] = [
     "protect",
     "unprotect",
     "delchat",
-    "export-chat",
+    "accept",
+    "blockchat",
 ];
 const MESSAGE_COMMANDS: [&str; 6] = [
     "listmsgs",

examples/simple.rs

@@ -19,7 +19,7 @@ fn cb(event: EventType) {
         EventType::Warning(msg) => {
             log::warn!("{}", msg);
         }
-        EventType::Error(msg) | EventType::ErrorNetwork(msg) => {
+        EventType::Error(msg) => {
             log::error!("{}", msg);
         }
         event => {
@@ -86,7 +86,7 @@ async fn main() {
     let chats = Chatlist::try_load(&ctx, 0, None, None).await.unwrap();
 
     for i in 0..chats.len() {
-        let msg = Message::load_from_db(&ctx, chats.get_msg_id(i).unwrap())
+        let msg = Message::load_from_db(&ctx, chats.get_msg_id(i).unwrap().unwrap())
             .await
             .unwrap();
         log::info!("[{}] msg: {:?}", i, msg);

python/README.rst

@@ -58,12 +58,13 @@ end-to-end tests that require accounts on real e-mail servers.
 running "live" tests with temporary accounts
 ---------------------------------------------
 
-If you want to run live functional tests you can set ``DCC_NEW_TMP_EMAIL``::
+If you want to run live functional tests you can set ``DCC_NEW_TMP_EMAIL`` to a URL that creates e-mail accounts.  Most developers use https://testrun.org URLS created and managed by [mailadm](https://mailadm.readthedocs.io/en/latest/).
 
-    export DCC_NEW_TMP_EMAIL=https://testrun.org/new_email?t=1h_4w4r8h7y9nmcdsy
+Please feel free to contact us through a github issue or by e-mail and we'll send you a URL that you can then use for functional tests like this:
 
-With this, pytest runs create ephemeral e-mail accounts on the http://testrun.org server.
-These accounts exists for one 1hour and then are removed completely.
+    export DCC_NEW_TMP_EMAIL=<URL you got from us>
+
+With this account-creation setting, pytest runs create ephemeral e-mail accounts on the http://testrun.org server.  These accounts exists only for one hour and then are removed completely.
 One hour is enough to invoke pytest and run all offline and online tests:
 
     pytest

python/src/deltachat/_build.py

@@ -150,6 +150,7 @@ def extract_defines(flags):
                 | DC_PROVIDER
                 | DC_KEY_GEN
                 | DC_IMEX
+                | DC_CONNECTIVITY
             )         # End of prefix matching
             _[\w_]+   # Match the suffix, e.g. _RSA2048 in DC_KEY_GEN_RSA2048
         )             # Close the capturing group, this contains

python/src/deltachat/account.py

@@ -330,9 +330,6 @@ class Account(object):
         """ Create a 1:1 chat with Account, Contact or e-mail address. """
         return self.create_contact(obj).create_chat()
 
-    def _create_chat_by_message_id(self, msg_id):
-        return Chat(self, lib.dc_create_chat_by_msg_id(self._dc_context, msg_id))
-
     def create_group_chat(self, name, contacts=None, verified=False):
         """ create a new group chat object.
 
@@ -367,9 +364,6 @@ class Account(object):
             chatlist.append(Chat(self, chat_id))
         return chatlist
 
-    def get_deaddrop_chat(self):
-        return Chat(self, const.DC_CHAT_ID_DEADDROP)
-
     def get_device_chat(self):
         return Contact(self, const.DC_CONTACT_ID_DEVICE).create_chat()
 
@@ -574,6 +568,15 @@ class Account(object):
         """ Stop ongoing securejoin, configuration or other core jobs. """
         lib.dc_stop_ongoing_process(self._dc_context)
 
+    def get_connectivity(self):
+        return lib.dc_get_connectivity(self._dc_context)
+
+    def get_connectivity_html(self):
+        return from_dc_charpointer(lib.dc_get_connectivity_html(self._dc_context))
+
+    def all_work_done(self):
+        return lib.dc_all_work_done(self._dc_context)
+
     def start_io(self):
         """ start this account's IO scheduling (Rust-core async scheduler)
 

python/src/deltachat/chat.py

@@ -50,6 +50,14 @@ class Chat(object):
         """
         lib.dc_delete_chat(self.account._dc_context, self.id)
 
+    def block(self):
+        """Block this chat."""
+        lib.dc_block_chat(self.account._dc_context, self.id)
+
+    def accept(self):
+        """Accept this contact request chat."""
+        lib.dc_accept_chat(self.account._dc_context, self.id)
+
     # ------  chat status/metadata API ------------------------------
 
     def is_group(self):
@@ -59,13 +67,6 @@ class Chat(object):
         """
         return lib.dc_chat_get_type(self._dc_chat) == const.DC_CHAT_TYPE_GROUP
 
-    def is_deaddrop(self):
-        """ return true if this chat is a deaddrop chat.
-
-        :returns: True if chat is the deaddrop chat, False otherwise.
-        """
-        return self.id == const.DC_CHAT_ID_DEADDROP
-
     def is_muted(self):
         """ return true if this chat is muted.
 
@@ -73,6 +74,13 @@ class Chat(object):
         """
         return lib.dc_chat_is_muted(self._dc_chat)
 
+    def is_contact_request(self):
+        """ return True if this chat is a contact request chat.
+
+        :returns: True if chat is a contact request chat, False otherwise.
+        """
+        return lib.dc_chat_is_contact_request(self._dc_chat)
+
     def is_promoted(self):
         """ return True if this chat is promoted, i.e.
         the member contacts are aware of their membership,
@@ -84,7 +92,7 @@ class Chat(object):
 
     def can_send(self):
         """Check if messages can be sent to a give chat.
-        This is not true eg. for the deaddrop or for the device-talk
+        This is not true eg. for the contact requests or for the device-talk
 
         :returns: True if the chat is writable, False otherwise
         """

python/src/deltachat/direct_imap.py

@@ -251,7 +251,16 @@ class DirectImap:
             return res
 
     def append(self, folder, msg):
+        """Upload a message to *folder*.
+        Trailing whitespace or a linebreak at the beginning will be removed automatically.
+        """
         if msg.startswith("\n"):
             msg = msg[1:]
         msg = '\n'.join([s.lstrip() for s in msg.splitlines()])
         self.conn.append(folder, msg)
+
+    def get_uid_by_message_id(self, message_id):
+        msgs = self.conn.search(['HEADER', 'MESSAGE-ID', message_id])
+        if len(msgs) == 0:
+            raise Exception("Did not find message " + message_id + ", maybe you forgot to select the correct folder?")
+        return msgs[0]

python/src/deltachat/events.py

@@ -111,6 +111,33 @@ class FFIEventTracker:
             if m is not None:
                 return m.groups()
 
+    def wait_for_connectivity(self, connectivity):
+        """Wait for the specified connectivity.
+        This only works reliably if the connectivity doesn't change
+        again too quickly, otherwise we might miss it."""
+        while 1:
+            if self.account.get_connectivity() == connectivity:
+                return
+            self.get_matching("DC_EVENT_CONNECTIVITY_CHANGED")
+
+    def wait_for_connectivity_change(self, previous, expected_next):
+        """Wait until the connectivity changes to `expected_next`.
+        Fails the test if it changes to something else."""
+        while 1:
+            current = self.account.get_connectivity()
+            if current == expected_next:
+                return
+            elif current != previous:
+                raise Exception("Expected connectivity " + str(expected_next) + " but got " + str(current))
+
+            self.get_matching("DC_EVENT_CONNECTIVITY_CHANGED")
+
+    def wait_for_all_work_done(self):
+        while 1:
+            if self.account.all_work_done():
+                return
+            self.get_matching("DC_EVENT_CONNECTIVITY_CHANGED")
+
     def ensure_event_not_queued(self, event_name_regex):
         __tracebackhide__ = True
         rex = re.compile("(?:{}).*".format(event_name_regex))

python/src/deltachat/hookspec.py

@@ -43,7 +43,7 @@ class PerAccount:
 
     @account_hookspec
     def ac_incoming_message(self, message):
-        """ Called on any incoming message (to deaddrop or chat). """
+        """ Called on any incoming message (both existing chats and contact requests). """
 
     @account_hookspec
     def ac_outgoing_message(self, message):

python/src/deltachat/message.py

@@ -61,16 +61,13 @@ class Message(object):
     def create_chat(self):
         """ create or get an existing chat (group) object for this message.
 
-        If the message is a deaddrop contact request
+        If the message is a contact request
         the sender will become an accepted contact.
 
         :returns: a :class:`deltachat.chat.Chat` object.
         """
-        from .chat import Chat
-        chat_id = lib.dc_create_chat_by_msg_id(self.account._dc_context, self.id)
-        ctx = self.account._dc_context
-        self._dc_msg = ffi.gc(lib.dc_get_msg(ctx, self.id), lib.dc_msg_unref)
-        return Chat(self.account, chat_id)
+        self.chat.accept()
+        return self.chat
 
     @props.with_doc
     def id(self):
@@ -141,6 +138,10 @@ class Message(object):
         """ return True if this message was encrypted. """
         return bool(lib.dc_msg_get_showpadlock(self._dc_msg))
 
+    def is_bot(self):
+        """ return True if this message is submitted automatically. """
+        return bool(lib.dc_msg_is_bot(self._dc_msg))
+
     def is_forwarded(self):
         """ return True if this message was forwarded. """
         return bool(lib.dc_msg_is_forwarded(self._dc_msg))

python/tests/test_account.py

@@ -909,12 +909,11 @@ class TestOnlineAccount:
         msg_in = ac2.get_message_by_id(msg_out.id)
         assert msg_in.text == "message2"
 
-        lp.sec("ac2: check that the message arrive in deaddrop")
+        lp.sec("ac2: check that the message arrived in a chat")
         chat2 = msg_in.chat
         assert msg_in in chat2.get_messages()
         assert not msg_in.is_forwarded()
-        assert chat2.is_deaddrop()
-        assert chat2 == ac2.get_deaddrop_chat()
+        assert chat2.is_contact_request()
 
         lp.sec("ac2: create new chat and forward message to it")
         chat3 = ac2.create_group_chat("newgroup")
@@ -979,16 +978,16 @@ class TestOnlineAccount:
         assert not msg2.is_forwarded()
         assert msg2.get_sender_contact().display_name == ac1.get_config("displayname")
 
-        lp.sec("check the message arrived in contact-requests/deaddrop")
+        lp.sec("check the message arrived in contact request chat")
         chat2 = msg2.chat
         assert msg2 in chat2.get_messages()
-        assert chat2.is_deaddrop()
-        assert chat2.count_fresh_messages() == 0
+        assert chat2.is_contact_request()
+        assert chat2.count_fresh_messages() == 1
         assert msg2.time_received >= msg1.time_sent
 
         lp.sec("create new chat with contact and verify it's proper")
         chat2b = msg2.create_chat()
-        assert not chat2b.is_deaddrop()
+        assert not chat2b.is_contact_request()
         assert chat2b.count_fresh_messages() == 1
 
         lp.sec("mark chat as noticed")
@@ -1319,9 +1318,11 @@ class TestOnlineAccount:
         assert not device_chat.can_send()
         assert device_chat.get_draft() is None
 
-    def test_dont_show_emails_in_draft_folder(self, acfactory):
+    def test_dont_show_emails_in_draft_folder(self, acfactory, lp):
         """Most mailboxes have a "Drafts" folder where constantly new emails appear but we don't actually want to show them.
-        So: If it's outgoing AND there is no Received header AND it's not in the sentbox, then ignore the email."""
+        So: If it's outgoing AND there is no Received header AND it's not in the sentbox, then ignore the email.
+
+        If the draft email is sent out later (i.e. moved to "Sent"), it must be shown."""
         ac1 = acfactory.get_online_configuring_account()
         ac1.set_config("show_emails", "2")
         ac1.create_contact("alice@example.com").create_chat()
@@ -1342,7 +1343,7 @@ class TestOnlineAccount:
             Message-ID: <aepiors@example.org>
             Content-Type: text/plain; charset=utf-8
 
-            message in Drafts
+            message in Drafts that is moved to Sent later
         """.format(ac1.get_config("configured_addr")))
         ac1.direct_imap.append("Sent", """
             From: ac1 <{}>
@@ -1355,6 +1356,7 @@ class TestOnlineAccount:
         """.format(ac1.get_config("configured_addr")))
 
         ac1.set_config("scan_all_folders_debounce_secs", "0")
+        lp.sec("All prepared, now let DC find the message")
         ac1.start_io()
 
         msg = ac1._evtracker.wait_next_messages_changed()
@@ -1365,6 +1367,18 @@ class TestOnlineAccount:
         assert msg.text == "subj – message in Sent"
         assert len(msg.chat.get_messages()) == 1
 
+        ac1.stop_io()
+        lp.sec("'Send out' the draft, i.e. move it to the Sent folder, and wait for DC to display it this time")
+        ac1.direct_imap.select_folder("Drafts")
+        uid = ac1.direct_imap.get_uid_by_message_id("aepiors@example.org")
+        ac1.direct_imap.conn.move(uid, "Sent")
+
+        ac1.start_io()
+        msg2 = ac1._evtracker.wait_next_messages_changed()
+
+        assert msg2.text == "subj – message in Drafts that is moved to Sent later"
+        assert len(msg.chat.get_messages()) == 2
+
     def test_prefer_encrypt(self, acfactory, lp):
         """Test quorum rule for encryption preference in 1:1 and group chat."""
         ac1, ac2, ac3 = acfactory.get_many_online_accounts(3)
@@ -1415,6 +1429,33 @@ class TestOnlineAccount:
         # Majority prefers encryption now
         assert msg5.is_encrypted()
 
+    def test_bot(self, acfactory, lp):
+        """Test that bot messages can be identified as such"""
+        ac1, ac2 = acfactory.get_two_online_accounts()
+        ac1.set_config("bot", "0")
+        ac2.set_config("bot", "1")
+
+        lp.sec("ac1: create chat with ac2")
+        chat = acfactory.get_accepted_chat(ac1, ac2)
+
+        lp.sec("sending a message from ac1 to ac2")
+        text1 = "hello"
+        chat.send_text(text1)
+
+        lp.sec("wait for ac2 to receive a message")
+        msg_in = ac2._evtracker.wait_next_incoming_message()
+        assert msg_in.text == text1
+        assert not msg_in.is_bot()
+
+        lp.sec("sending a message from ac2 to ac1")
+        text2 = "reply"
+        msg_in.chat.send_text(text2)
+
+        lp.sec("wait for ac1 to receive a message")
+        msg_in = ac1._evtracker.wait_next_incoming_message()
+        assert msg_in.text == text2
+        assert msg_in.is_bot()
+
     def test_quote_encrypted(self, acfactory, lp):
         """Test that replies to encrypted messages with quotes are encrypted."""
         ac1, ac2 = acfactory.get_two_online_accounts()
@@ -1811,7 +1852,7 @@ class TestOnlineAccount:
 
         lp.sec("ac2: wait for receiving message and avatar from ac1")
         msg2 = ac2._evtracker.wait_next_messages_changed()
-        assert msg2.chat.is_deaddrop()
+        assert msg2.chat.is_contact_request()
         received_path = msg2.get_sender_contact().get_profile_image()
         assert open(received_path, "rb").read() == open(p, "rb").read()
 
@@ -1886,6 +1927,8 @@ class TestOnlineAccount:
         ev = in_list.get(timeout=10)
         assert ev.action == "chat-modified"
         ev = in_list.get(timeout=10)
+        assert ev.action == "chat-modified"
+        ev = in_list.get(timeout=10)
         assert ev.action == "added"
         assert ev.message.get_sender_contact().addr == ac1_addr
         assert ev.contact.addr == "devnull@testrun.org"
@@ -2000,6 +2043,84 @@ class TestOnlineAccount:
         assert msg_back.chat == chat
         assert chat.get_profile_image() is None
 
+    @pytest.mark.parametrize("inbox_watch", ["0", "1"])
+    def test_connectivity(self, acfactory, lp, inbox_watch):
+        ac1, ac2 = acfactory.get_two_online_accounts()
+        ac1.set_config("inbox_watch", inbox_watch)
+        ac1.set_config("scan_all_folders_debounce_secs", "0")
+
+        ac1._evtracker.wait_for_connectivity(const.DC_CONNECTIVITY_CONNECTED)
+
+        lp.sec("Test stop_io() and start_io()")
+        ac1.stop_io()
+        ac1._evtracker.wait_for_connectivity(const.DC_CONNECTIVITY_NOT_CONNECTED)
+
+        ac1.start_io()
+        ac1._evtracker.wait_for_connectivity(const.DC_CONNECTIVITY_CONNECTING)
+        ac1._evtracker.wait_for_connectivity_change(const.DC_CONNECTIVITY_CONNECTING, const.DC_CONNECTIVITY_CONNECTED)
+
+        lp.sec("Test that after calling start_io(), maybe_network() and waiting for `all_work_done()`, " +
+               "all messages are fetched")
+
+        ac1.direct_imap.select_config_folder("inbox")
+        ac1.direct_imap.idle_start()
+        ac2.create_chat(ac1).send_text("Hi")
+
+        ac1.direct_imap.idle_check(terminate=False)
+        ac1.maybe_network()
+
+        ac1._evtracker.wait_for_all_work_done()
+        msgs = ac1.create_chat(ac2).get_messages()
+        assert len(msgs) == 1
+        assert msgs[0].text == "Hi"
+
+        lp.sec("Test that the connectivity changes to WORKING while new messages are fetched")
+
+        ac2.create_chat(ac1).send_text("Hi 2")
+
+        ac1.direct_imap.idle_check(terminate=True)
+        ac1.maybe_network()
+        ac1._evtracker.wait_for_connectivity_change(const.DC_CONNECTIVITY_CONNECTED, const.DC_CONNECTIVITY_WORKING)
+        ac1._evtracker.wait_for_connectivity_change(const.DC_CONNECTIVITY_WORKING, const.DC_CONNECTIVITY_CONNECTED)
+
+        msgs = ac1.create_chat(ac2).get_messages()
+        assert len(msgs) == 2
+        assert msgs[1].text == "Hi 2"
+
+        lp.sec("Test that the connectivity doesn't flicker to WORKING if there are no new messages")
+
+        ac1.maybe_network()
+        while 1:
+            assert ac1.get_connectivity() == const.DC_CONNECTIVITY_CONNECTED
+            if ac1.all_work_done():
+                break
+            ac1._evtracker.get_matching("DC_EVENT_CONNECTIVITY_CHANGED")
+
+        lp.sec("Test that the connectivity doesn't flicker to WORKING if the sender of the message is blocked")
+        ac1.create_contact(ac2).block()
+
+        ac1.direct_imap.select_config_folder("inbox")
+        ac1.direct_imap.idle_start()
+        ac2.create_chat(ac1).send_text("Hi")
+
+        ac1.direct_imap.idle_check(terminate=True)
+        ac1.maybe_network()
+
+        while 1:
+            assert ac1.get_connectivity() == const.DC_CONNECTIVITY_CONNECTED
+            if ac1.all_work_done():
+                break
+            ac1._evtracker.get_matching("DC_EVENT_CONNECTIVITY_CHANGED")
+
+        lp.sec("Test that the connectivity is NOT_CONNECTED if the password is wrong")
+
+        ac1.set_config("configured_mail_pw", "abc")
+        ac1.stop_io()
+        ac1._evtracker.wait_for_connectivity(const.DC_CONNECTIVITY_NOT_CONNECTED)
+        ac1.start_io()
+        ac1._evtracker.wait_for_connectivity(const.DC_CONNECTIVITY_CONNECTING)
+        ac1._evtracker.wait_for_connectivity(const.DC_CONNECTIVITY_NOT_CONNECTED)
+
     def test_fetch_deleted_msg(self, acfactory, lp):
         """This is a regression test: Messages with \\Deleted flag were downloaded again and again,
         hundreds of times, because uid_next was not updated.
@@ -2716,7 +2837,6 @@ class TestOnlineConfigureFails:
         configtracker = ac1.configure()
         configtracker.wait_progress(500)
         configtracker.wait_progress(0)
-        ac1._evtracker.ensure_event_not_queued("DC_EVENT_ERROR_NETWORK")
 
     def test_invalid_user(self, acfactory):
         ac1, configdict = acfactory.get_online_config()
@@ -2724,7 +2844,6 @@ class TestOnlineConfigureFails:
         configtracker = ac1.configure()
         configtracker.wait_progress(500)
         configtracker.wait_progress(0)
-        ac1._evtracker.ensure_event_not_queued("DC_EVENT_ERROR_NETWORK")
 
     def test_invalid_domain(self, acfactory):
         ac1, configdict = acfactory.get_online_config()
@@ -2732,4 +2851,3 @@ class TestOnlineConfigureFails:
         configtracker = ac1.configure()
         configtracker.wait_progress(500)
         configtracker.wait_progress(0)
-        ac1._evtracker.ensure_event_not_queued("DC_EVENT_ERROR_NETWORK")

scripts/README.md

@@ -1,16 +1,13 @@
 # Continuous Integration Scripts for Delta Chat
 
 Continuous Integration, run through [GitHub
-Actions](https://docs.github.com/actions),
-[CircleCI](https://app.circleci.com/) and an own build machine.
+Actions](https://docs.github.com/actions)
+and an own build machine.
 
 ## Description of scripts 
 
 - `../.github/workflows` contains jobs run by GitHub Actions.
 
-- `../.circleci/config.yml` describing the build jobs that are run
-  by CircleCI.
-
 - `remote_tests_python.sh` rsyncs to a build machine and runs
   `run-python-test.sh` remotely on the build machine. 
 

scripts/ci_upload.sh

@@ -1,75 +0,0 @@
-#!/bin/bash
-
-if [ -z "$DEVPI_LOGIN" ] ; then 
-    echo "required: password for 'dc' user on https://m.devpi/net/dc index"
-    exit 0
-fi
-
-set -xe
-
-PYDOCDIR=${1:?directory with python docs}
-WHEELHOUSEDIR=${2:?directory with pre-built wheels}
-DOXYDOCDIR=${3:?directory where doxygen docs to be found}
-SSHTARGET=ci@b1.delta.chat
-
-
-# if CIRCLE_BRANCH is not set we are called for a tag with empty CIRCLE_BRANCH variable. 
-export BRANCH=${CIRCLE_BRANCH:master}
-
-export BUILDDIR=ci_builds/$REPONAME/$BRANCH/${CIRCLE_JOB:?jobname}/${CIRCLE_BUILD_NUM:?circle-build-number}/wheelhouse
-
-
-# python docs to py.delta.chat
-ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null delta@py.delta.chat mkdir -p build/${BRANCH}
-rsync -avz \
-  --delete \
-  -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
-  "$PYDOCDIR/html/" \
-  delta@py.delta.chat:build/${BRANCH}
-
-# C docs to c.delta.chat
-ssh -o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null delta@c.delta.chat mkdir -p build-c/${BRANCH}
-rsync -avz \
-  --delete \
-  -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
-  "$DOXYDOCDIR/html/" \
-  delta@c.delta.chat:build-c/${BRANCH}
-
-echo -----------------------
-echo upload wheels 
-echo -----------------------
-
-# Bundle external shared libraries into the wheels
-
-ssh -o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $SSHTARGET mkdir -p $BUILDDIR 
-scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null scripts/cleanup_devpi_indices.py $SSHTARGET:$BUILDDIR 
-rsync -avz \
-  -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
-  $WHEELHOUSEDIR \
-  $SSHTARGET:$BUILDDIR 
-
-ssh $SSHTARGET <<_HERE
-    set +x -e
-    # make sure all processes exit when ssh dies
-    shopt -s huponexit
-
-    # we rely on the "venv" virtualenv on the remote account to exist 
-    source venv/bin/activate
-    cd $BUILDDIR
-
-    devpi use https://m.devpi.net
-    devpi login dc --password $DEVPI_LOGIN
-
-    N_BRANCH=${BRANCH//[\/]}
-
-    devpi use dc/\$N_BRANCH || {
-        devpi index -c \$N_BRANCH 
-        devpi use dc/\$N_BRANCH
-    }
-    devpi index \$N_BRANCH bases=/root/pypi
-    devpi upload wheelhouse/deltachat*
-
-    # remove devpi non-master dc indices if thy are too old
-    # this script was copied above 
-    python cleanup_devpi_indices.py
-_HERE

scripts/concourse/README.md

@@ -0,0 +1,21 @@
+# Concourse CI pipeline
+
+`docs_wheels.yml` is a pipeline for [Concourse CI](https://concourse-ci.org/)
+that builds C documentation, Python documentation, Python wheels for `x86_64`
+and `aarch64` and Python source packages, and uploads them.
+
+To setup the pipeline run
+```
+fly -t <your-target> set-pipeline -c docs_wheels.yml -p docs_wheels -l secret.yml
+```
+where `secret.yml` contains the following secrets:
+```
+c.delta.chat:
+  private_key: |
+    -----BEGIN RSA PRIVATE KEY-----
+    ...
+    -----END RSA PRIVATE KEY-----
+devpi:
+  login: dc
+  password: ...
+```

scripts/concourse/docs_wheels.yml

@@ -0,0 +1,218 @@
+resources:
+- name: deltachat-core-rust
+  type: git
+  icon: github
+  source:
+    branch: master
+    uri: https://github.com/deltachat/deltachat-core-rust.git
+
+jobs:
+- name: doxygen
+  plan:
+  - get: deltachat-core-rust
+    trigger: true
+
+  # Build Doxygen documentation
+  - task: build-doxygen
+    config:
+      inputs:
+        - name: deltachat-core-rust
+      outputs:
+        - name: c-docs
+      image_resource:
+        source:
+          repository: hrektts/doxygen
+        type: registry-image
+      platform: linux
+      run:
+        path: bash
+        args:
+        - -exc
+        - |
+          cd deltachat-core-rust
+          bash scripts/run-doxygen.sh
+          cd ..
+          cp -av deltachat-core-rust/deltachat-ffi/{html,xml} c-docs/
+
+  - task: upload-c-docs
+    config:
+      inputs:
+        - name: c-docs
+      image_resource:
+        type: registry-image
+        source:
+          repository: alpine
+      platform: linux
+      run:
+        path: sh
+        args:
+        - -ec
+        - |
+          apk add --no-cache rsync openssh-client
+          mkdir -p ~/.ssh
+          chmod 700 ~/.ssh
+          echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
+          chmod 600 ~/.ssh/id_ed25519
+          rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete c-docs/html/ delta@c.delta.chat:build-c/master
+
+- name: python-x86_64
+  plan:
+  - get: deltachat-core-rust
+
+  # Build manylinux image with additional dependencies
+  - task: build-coredeps
+    privileged: true
+    config:
+      inputs:
+        - name: deltachat-core-rust
+      image_resource:
+        source:
+          repository: vito/oci-build-task
+        type: registry-image
+      outputs:
+        - name: coredeps-image
+          path: image
+      params:
+        CONTEXT: deltachat-core-rust/scripts/docker-coredeps
+        UNPACK_ROOTFS: "true"
+      platform: linux
+      caches:
+      - path: cache
+      run:
+        path: build
+
+  # Use built image to build python wheels
+  - task: build-wheels
+    image: coredeps-image
+    config:
+      inputs:
+        - name: deltachat-core-rust
+          path: .
+      outputs:
+        - name: py-docs
+          path: ./python/doc/_build/
+        # Source packages
+        - name: py-dist
+          path: ./python/.docker-tox/dist/
+        # Binary wheels
+        - name: py-wheels
+          path: ./python/.docker-tox/wheelhouse/
+      platform: linux
+      run:
+        path: bash
+        args:
+        - -exc
+        - |
+          scripts/run_all.sh
+
+  # Upload python docs to py.delta.chat
+  - task: upload-py-docs
+    config:
+      inputs:
+        - name: py-docs
+      image_resource:
+        type: registry-image
+        source:
+          repository: alpine
+      platform: linux
+      run:
+        path: sh
+        args:
+        - -ec
+        - |
+          apk add --no-cache rsync openssh-client
+          mkdir -p ~/.ssh
+          chmod 700 ~/.ssh
+          echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
+          chmod 600 ~/.ssh/id_ed25519
+          rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete py-docs/html/ delta@py.delta.chat:build/master
+
+  # Upload x86_64 wheels and source packages
+  - task: upload-wheels
+    config:
+      inputs:
+        - name: py-wheels
+        - name: py-dist
+      image_resource:
+        type: registry-image
+        source:
+          repository: debian
+      platform: linux
+      run:
+        path: sh
+        args:
+        - -ec
+        - |
+          apt-get update -y
+          apt-get install -y --no-install-recommends python3-pip python3-setuptools
+          pip3 install devpi
+          devpi use https://m.devpi.net/dc/master
+          devpi login ((devpi.login)) --password ((devpi.password))
+          devpi upload py-wheels/*manylinux201*
+          devpi upload py-dist/*
+
+- name: python-aarch64
+  plan:
+  - get: deltachat-core-rust
+
+  # Build manylinux image with additional dependencies
+  - task: build-coredeps
+    privileged: true
+    config:
+      inputs:
+        - name: deltachat-core-rust
+      image_resource:
+        source:
+          repository: vito/oci-build-task
+        type: registry-image
+      outputs:
+        - name: coredeps-image
+          path: image
+      params:
+        CONTEXT: deltachat-core-rust/scripts/docker-coredeps-arm64
+        UNPACK_ROOTFS: "true"
+      platform: linux
+      caches:
+      - path: cache
+      run:
+        path: build
+
+  # Use built image to build python wheels
+  - task: build-wheels
+    image: coredeps-image
+    config:
+      inputs:
+        - name: deltachat-core-rust
+          path: .
+      outputs:
+        - name: py-wheels
+          path: ./python/.docker-tox/wheelhouse/
+      platform: linux
+      run:
+        path: bash
+        args:
+        - -exc
+        - |
+          scripts/run_all.sh
+
+  # Upload aarch64 wheels
+  - task: upload-wheels
+    config:
+      inputs:
+        - name: py-wheels
+      image_resource:
+        type: registry-image
+        source:
+          repository: debian
+      platform: linux
+      run:
+        path: sh
+        args:
+        - -ec
+        - |
+          apt-get update -y
+          apt-get install -y --no-install-recommends python3-pip python3-setuptools
+          pip3 install devpi
+          devpi use https://m.devpi.net/dc/master
+          devpi login ((devpi.login)) --password ((devpi.password))
+          devpi upload py-wheels/*manylinux201*

scripts/remote_python_packaging.sh

@@ -1,67 +0,0 @@
-#!/bin/bash 
-
-export BRANCH=${CIRCLE_BRANCH:-master}
-export REPONAME=${CIRCLE_PROJECT_REPONAME:-deltachat-core-rust}
-export SSHTARGET=${SSHTARGET-ci@b1.delta.chat}
-
-export BUILDDIR=ci_builds/$REPONAME/$BRANCH/${CIRCLE_JOB:?jobname}/${CIRCLE_BUILD_NUM:?circle-build-number}
-
-echo "--- Copying files to $SSHTARGET:$BUILDDIR"
-
-set -xe
-
-ssh -oBatchMode=yes -oStrictHostKeyChecking=no  $SSHTARGET mkdir -p "$BUILDDIR"
-git ls-files >.rsynclist 
-# we seem to need .git for setuptools_scm versioning 
-find .git >>.rsynclist
-rsync --delete --files-from=.rsynclist -az ./ "$SSHTARGET:$BUILDDIR"
-rsync --delete --files-from=.rsynclist -az ./ "$SSHTARGET:$BUILDDIR-arm64"
-
-set +x
-
-# we have to create a remote file for the remote-docker run 
-# so we can do a simple ssh command with a TTY
-# so that when our job dies, all container-runs are aborted.
-# sidenote: the circle-ci machinery will kill ongoing jobs
-# if there are new commits and we want to ensure that 
-# everything is terminated/cleaned up and we have no orphaned
-# useless still-running docker-containers consuming resources. 
-
-for arch in "" "-arm64"; do
-
-ssh $SSHTARGET bash -c "cat >${BUILDDIR}${arch}/exec_docker_run" <<_HERE
-    set +x -e
-    shopt -s huponexit
-    cd ${BUILDDIR}${arch}
-    export DCC_NEW_TMP_EMAIL=$DCC_NEW_TMP_EMAIL
-    set -x
-
-    # run everything else inside docker 
-    docker run -e DCC_NEW_TMP_EMAIL \
-       --rm -it -v \$(pwd):/mnt -w /mnt \
-       deltachat/coredeps${arch} scripts/run_all.sh
-
-_HERE
-
-done
-
-echo "--- Running $CIRCLE_JOB remotely"
-
-echo "--- Building aarch64 wheels"
-ssh -o ServerAliveInterval=30 -t $SSHTARGET bash "$BUILDDIR-arm64/exec_docker_run"
-
-echo "--- Building x86_64 wheels"
-ssh -o ServerAliveInterval=30 -t $SSHTARGET bash "$BUILDDIR/exec_docker_run"
-
-mkdir -p workspace 
-
-# Wheels
-for arch in "" "-arm64"; do
-rsync -avz "$SSHTARGET:$BUILDDIR${arch}/python/.docker-tox/wheelhouse/*manylinux201*" workspace/wheelhouse/
-done
-
-# Source packages
-rsync -avz "$SSHTARGET:$BUILDDIR${arch}/python/.docker-tox/dist/*" workspace/wheelhouse/
-
-# Documentation
-rsync -avz "$SSHTARGET:$BUILDDIR/python/doc/_build/" workspace/py-docs

spec.md

@@ -1,6 +1,6 @@
 # chat-mail specification
 
-Version: 0.32.0
+Version: 0.33.0
 Status:  In-progress 
 Format:  [Semantic Line Breaks](https://sembr.org/)
 
@@ -301,9 +301,9 @@ to add a `Chat-Group-Avatar` only on image changes.
 
 A user MAY have a profile-image that MAY be distributed to their contacts.
 To change or set the profile-image,
-the messenger MUST attach an image file to a message
-and MUST add the header `Chat-User-Avatar`
-with the value set to the image name.
+the messenger MUST add the header `Chat-User-Avatar: base64:IMAGEDATA`.
+To bypass limits of headers, it is recommended not to use the outer header
+and to limit the size to 20k.
 
 To remove the profile-image,
 the messenger MUST add the header `Chat-User-Avatar: 0`.
@@ -320,19 +320,14 @@ The messenger SHOULD NOT send an explicit mail to normal MUAs.
     From: sender@domain
     To: rcpt@domain
     Chat-Version: 1.0
-    Chat-User-Avatar: photo.jpg
     Subject: Chat: Hello, ...
     Content-Type: multipart/mixed; boundary="==break=="
 
     --==break==
     Content-Type: text/plain
+    Chat-User-Avatar: base64:AKCgkJi3j4l5kjoldfUAKCgkJi3j4lldfHjgWICwgIEBQY ...
 
     Hello, I've changed my profile image.
-    --==break==
-    Content-Type: image/jpeg
-    Content-Disposition: attachment; filename="photo.jpg"
-
-    AKCgkJi3j4l5kjoldfUAKCgkJi3j4lldfHjgWICwgIEBQYFBA ...
     --==break==--
 
 The image format SHOULD be image/jpeg or image/png.
@@ -342,6 +337,11 @@ in the same message.
 To save data, it is RECOMMENDED to add a `Chat-User-Avatar` header
 only on image changes.
 
+In older specs, the profile-image was sent as an attachment
+and `Chat-User-Avatar:` specified its name.
+However, it turned out that these attachements are kind of unuexpected to users,
+therefore the profile-image go to the header now.
+
 
 # Locations
 
@@ -401,9 +401,41 @@ it is fine if the location is detected on forwarding etc.
     </kml>
 
 
-# Miscellaneous
+# Stickers
 
-Messengers SHOULD use the header `In-Reply-To` as usual.
+Stickers are send as normal images
+with the additional header `Chat-Content: sticker`.
+
+It is discouraged to send stickers together with user generated text,
+however, stickers can be used as a reply to a message
+and also the footer should be set as usual.
+
+    From: alice@example.org
+    To: bob@example.com
+    Chat-Version: 1.0
+    Chat-Content: sticker
+    Message-ID: Mr.12345uvwxyZ.0005@example.org
+    Subject: Message from Alice
+    Content-Type: multipart/mixed; boundary="==break=="
+
+    --==break==
+    Content-Type: text/plain
+
+    -- 
+    Hi there! I am using this new messenger!
+    --==break==
+    Content-Type: image/png
+    Content-Disposition: attachment; filename="sticker.png"
+
+    R0lGODlhpAGkAfe9AP+zd2eQkZhrI//z9v++PMb///+scrdDT3BtbtrZ2f/LQSsREcdIVf9 ...
+    --==break==--
+
+Typical sticker formats are `image/png`, `image/gif` and `image/webp`.
+Animated stickers are supported
+by just using an image format that supports animation.
+
+
+# Voice messages
 
 Messengers SHOULD add a `Chat-Voice-message: 1` header
 if an attached audio file is a voice message.
@@ -417,6 +449,11 @@ This allows the receiver to show the time without knowing the file format.
     Chat-Voice-Message: 1
     Chat-Duration: 10000
 
+
+# Miscellaneous
+
+Messengers SHOULD use the header `In-Reply-To` as usual.
+
 Messengers MAY send and receive Message Disposition Notifications
 (MDNs, [RFC 8098](https://tools.ietf.org/html/rfc8098),
 [RFC 3503](https://tools.ietf.org/html/rfc3503))
@@ -437,4 +474,4 @@ as the sending time of the message as indicated by its Date header,
 or the time of first receipt if that date is in the future or unavailable.
 
 
-Copyright © 2017-2020 Delta Chat contributors.
+Copyright © 2017-2021 Delta Chat contributors.

src/accounts.rs

@@ -1,5 +1,6 @@
 use std::collections::BTreeMap;
 
+use async_std::channel::{Receiver, Sender};
 use async_std::fs;
 use async_std::path::PathBuf;
 use async_std::prelude::*;
@@ -18,6 +19,7 @@ pub struct Accounts {
     dir: PathBuf,
     config: Config,
     accounts: Arc<RwLock<BTreeMap<u32, Context>>>,
+    emitter: EventEmitter,
 }
 
 impl Accounts {
@@ -30,19 +32,13 @@ impl Accounts {
         Accounts::open(dir).await
     }
 
-    /// Creates a new default structure, including a default account.
+    /// Creates a new default structure.
     pub async fn create(os_name: String, dir: &PathBuf) -> Result<()> {
         fs::create_dir_all(dir)
             .await
             .context("failed to create folder")?;
 
-        // create default account
-        let config = Config::new(os_name.clone(), dir).await?;
-        let account_config = config.new_account(dir).await?;
-
-        Context::new(os_name, account_config.dbfile().into(), account_config.id)
-            .await
-            .context("failed to create default account")?;
+        Config::new(os_name.clone(), dir).await?;
 
         Ok(())
     }
@@ -58,10 +54,16 @@ impl Accounts {
         let config = Config::from_file(config_file).await?;
         let accounts = config.load_accounts().await?;
 
+        let emitter = EventEmitter::new();
+        for account in accounts.values() {
+            emitter.add_account(account).await?;
+        }
+
         Ok(Self {
             dir,
             config,
             accounts: Arc::new(RwLock::new(accounts)),
+            emitter,
         })
     }
 
@@ -71,14 +73,9 @@ impl Accounts {
     }
 
     /// Get the currently selected account.
-    pub async fn get_selected_account(&self) -> Context {
+    pub async fn get_selected_account(&self) -> Option<Context> {
         let id = self.config.get_selected_account().await;
-        self.accounts
-            .read()
-            .await
-            .get(&id)
-            .cloned()
-            .expect("inconsistent state")
+        self.accounts.read().await.get(&id).cloned()
     }
 
     /// Select the given account.
@@ -94,6 +91,7 @@ impl Accounts {
         let account_config = self.config.new_account(&self.dir).await?;
 
         let ctx = Context::new(os_name, account_config.dbfile().into(), account_config.id).await?;
+        self.emitter.add_account(&ctx).await?;
         self.accounts.write().await.insert(account_config.id, ctx);
 
         Ok(account_config.id)
@@ -120,6 +118,7 @@ impl Accounts {
     /// Migrate an existing account into this structure.
     pub async fn migrate_account(&self, dbfile: PathBuf) -> Result<u32> {
         let blobdir = Context::derive_blobdir(&dbfile);
+        let walfile = Context::derive_walfile(&dbfile);
 
         ensure!(
             dbfile.exists().await,
@@ -143,6 +142,7 @@ impl Accounts {
 
         let new_dbfile = account_config.dbfile().into();
         let new_blobdir = Context::derive_blobdir(&new_dbfile);
+        let new_walfile = Context::derive_walfile(&new_dbfile);
 
         let res = {
             fs::create_dir_all(&account_config.dir)
@@ -154,6 +154,11 @@ impl Accounts {
             fs::rename(&blobdir, &new_blobdir)
                 .await
                 .context("failed to rename blobdir")?;
+            if walfile.exists().await {
+                fs::rename(&walfile, &new_walfile)
+                    .await
+                    .context("failed to rename walfile")?;
+            }
             Ok(())
         };
 
@@ -190,23 +195,23 @@ impl Accounts {
         self.accounts.read().await.keys().copied().collect()
     }
 
-    /// Import a backup using a new account and selects it.
-    pub async fn import_account(&self, file: PathBuf) -> Result<u32> {
-        let old_id = self.config.get_selected_account().await;
-
-        let id = self.add_account().await?;
-        let ctx = self.get_account(id).await.expect("just added");
-
-        match crate::imex::imex(&ctx, crate::imex::ImexMode::ImportBackup, &file).await {
-            Ok(_) => Ok(id),
-            Err(err) => {
-                // remove temp account
-                self.remove_account(id).await?;
-                // set selection back
-                self.select_account(old_id).await?;
-                Err(err)
+    /// This is meant especially for iOS, because iOS needs to tell the system when its background work is done.
+    ///
+    /// Returns whether all accounts finished their background work.
+    /// DC_EVENT_CONNECTIVITY_CHANGED will be sent when this turns to true.
+    ///
+    /// iOS can:
+    /// - call dc_start_io() (in case IO was not running)
+    /// - call dc_maybe_network()
+    /// - while dc_accounts_all_work_done() returns false:
+    ///   -  Wait for DC_EVENT_CONNECTIVITY_CHANGED
+    pub async fn all_work_done(&self) -> bool {
+        for account in self.accounts.read().await.values() {
+            if !account.all_work_done().await {
+                return false;
             }
         }
+        true
     }
 
     pub async fn start_io(&self) {
@@ -230,32 +235,69 @@ impl Accounts {
         }
     }
 
+    pub async fn maybe_network_lost(&self) {
+        let accounts = &*self.accounts.read().await;
+        for account in accounts.values() {
+            account.maybe_network_lost().await;
+        }
+    }
+
     /// Unified event emitter.
     pub async fn get_event_emitter(&self) -> EventEmitter {
-        let emitters: Vec<_> = self
-            .accounts
-            .read()
-            .await
-            .iter()
-            .map(|(_id, a)| a.get_event_emitter())
-            .collect();
-
-        EventEmitter(futures::stream::select_all(emitters))
+        self.emitter.clone()
     }
 }
 
-#[derive(Debug)]
-pub struct EventEmitter(futures::stream::SelectAll<crate::events::EventEmitter>);
+#[derive(Debug, Clone)]
+pub struct EventEmitter {
+    /// Aggregate stream of events from all accounts.
+    stream: Arc<RwLock<futures::stream::SelectAll<crate::events::EventEmitter>>>,
+
+    /// Sender for the channel where new account emitters will be pushed.
+    sender: Sender<crate::events::EventEmitter>,
+
+    /// Receiver for the channel where new account emitters will be pushed.
+    receiver: Receiver<crate::events::EventEmitter>,
+}
 
 impl EventEmitter {
-    /// Blocking recv of an event. Return `None` if all `Sender`s have been droped.
-    pub fn recv_sync(&mut self) -> Option<Event> {
-        async_std::task::block_on(self.recv())
+    pub fn new() -> Self {
+        let (sender, receiver) = async_std::channel::unbounded();
+        Self {
+            stream: Arc::new(RwLock::new(futures::stream::SelectAll::new())),
+            sender,
+            receiver,
+        }
     }
 
-    /// Async recv of an event. Return `None` if all `Sender`s have been droped.
-    pub async fn recv(&mut self) -> Option<Event> {
-        self.0.next().await
+    /// Blocking recv of an event. Return `None` if all `Sender`s have been droped.
+    pub fn recv_sync(&mut self) -> Option<Event> {
+        async_std::task::block_on(self.recv()).unwrap_or_default()
+    }
+
+    /// Async recv of an event. Return `None` if all `Sender`s have been dropped.
+    pub async fn recv(&mut self) -> Result<Option<Event>> {
+        let mut stream = self.stream.write().await;
+        loop {
+            match futures::future::select(self.receiver.recv(), stream.next()).await {
+                futures::future::Either::Left((emitter, _)) => {
+                    stream.push(emitter?);
+                }
+                futures::future::Either::Right((ev, _)) => return Ok(ev),
+            }
+        }
+    }
+
+    /// Add event emitter of a new account to the aggregate event emitter.
+    pub async fn add_account(&self, context: &Context) -> Result<()> {
+        self.sender.send(context.get_event_emitter()).await?;
+        Ok(())
+    }
+}
+
+impl Default for EventEmitter {
+    fn default() -> Self {
+        Self::new()
     }
 }
 
@@ -266,7 +308,7 @@ impl async_std::stream::Stream for EventEmitter {
         mut self: std::pin::Pin<&mut Self>,
         cx: &mut std::task::Context<'_>,
     ) -> std::task::Poll<Option<Self::Item>> {
-        std::pin::Pin::new(&mut self.0).poll_next(cx)
+        std::pin::Pin::new(&mut self).poll_next(cx)
     }
 }
 
@@ -444,6 +486,8 @@ mod tests {
         let p: PathBuf = dir.path().join("accounts1").into();
 
         let accounts1 = Accounts::new("my_os".into(), p.clone()).await.unwrap();
+        accounts1.add_account().await.unwrap();
+
         let accounts2 = Accounts::open(p).await.unwrap();
 
         assert_eq!(accounts1.accounts.read().await.len(), 1);
@@ -466,7 +510,11 @@ mod tests {
         let p: PathBuf = dir.path().join("accounts").into();
 
         let accounts = Accounts::new("my_os".into(), p.clone()).await.unwrap();
+        assert_eq!(accounts.accounts.read().await.len(), 0);
+        assert_eq!(accounts.config.get_selected_account().await, 0);
 
+        let id = accounts.add_account().await.unwrap();
+        assert_eq!(id, 1);
         assert_eq!(accounts.accounts.read().await.len(), 1);
         assert_eq!(accounts.config.get_selected_account().await, 1);
 
@@ -483,14 +531,35 @@ mod tests {
         assert_eq!(accounts.accounts.read().await.len(), 1);
     }
 
+    #[async_std::test]
+    async fn test_accounts_remove_last() -> Result<()> {
+        let dir = tempfile::tempdir()?;
+        let p: PathBuf = dir.path().join("accounts").into();
+
+        let accounts = Accounts::new("my_os".into(), p.clone()).await?;
+        assert!(accounts.get_selected_account().await.is_none());
+        assert_eq!(accounts.config.get_selected_account().await, 0);
+
+        let id = accounts.add_account().await?;
+        assert!(accounts.get_selected_account().await.is_some());
+        assert_eq!(id, 1);
+        assert_eq!(accounts.accounts.read().await.len(), 1);
+        assert_eq!(accounts.config.get_selected_account().await, id);
+
+        accounts.remove_account(id).await?;
+        assert!(accounts.get_selected_account().await.is_none());
+
+        Ok(())
+    }
+
     #[async_std::test]
     async fn test_migrate_account() {
         let dir = tempfile::tempdir().unwrap();
         let p: PathBuf = dir.path().join("accounts").into();
 
         let accounts = Accounts::new("my_os".into(), p.clone()).await.unwrap();
-        assert_eq!(accounts.accounts.read().await.len(), 1);
-        assert_eq!(accounts.config.get_selected_account().await, 1);
+        assert_eq!(accounts.accounts.read().await.len(), 0);
+        assert_eq!(accounts.config.get_selected_account().await, 0);
 
         let extern_dbfile: PathBuf = dir.path().join("other").into();
         let ctx = Context::new("my_os".into(), extern_dbfile.clone(), 0)
@@ -506,10 +575,10 @@ mod tests {
             .migrate_account(extern_dbfile.clone())
             .await
             .unwrap();
-        assert_eq!(accounts.accounts.read().await.len(), 2);
-        assert_eq!(accounts.config.get_selected_account().await, 2);
+        assert_eq!(accounts.accounts.read().await.len(), 1);
+        assert_eq!(accounts.config.get_selected_account().await, 1);
 
-        let ctx = accounts.get_selected_account().await;
+        let ctx = accounts.get_selected_account().await.unwrap();
         assert_eq!(
             "me@mail.com",
             ctx.get_config(crate::config::Config::Addr)
@@ -527,7 +596,7 @@ mod tests {
 
         let accounts = Accounts::new("my_os".into(), p.clone()).await.unwrap();
 
-        for expected_id in 2..10 {
+        for expected_id in 1..10 {
             let id = accounts.add_account().await.unwrap();
             assert_eq!(id, expected_id);
         }
@@ -537,4 +606,86 @@ mod tests {
             assert_eq!(ids.get(i), Some(&expected_id));
         }
     }
+
+    #[async_std::test]
+    async fn test_accounts_ids_unique_increasing_and_persisted() -> Result<()> {
+        let dir = tempfile::tempdir()?;
+        let p: PathBuf = dir.path().join("accounts").into();
+        let dummy_accounts = 10;
+
+        let (id0, id1, id2) = {
+            let accounts = Accounts::new("my_os".into(), p.clone()).await?;
+            accounts.add_account().await?;
+            let ids = accounts.get_all().await;
+            assert_eq!(ids.len(), 1);
+
+            let id0 = *ids.get(0).unwrap();
+            let ctx = accounts.get_account(id0).await.unwrap();
+            ctx.set_config(crate::config::Config::Addr, Some("one@example.org"))
+                .await?;
+
+            let id1 = accounts.add_account().await?;
+            let ctx = accounts.get_account(id1).await.unwrap();
+            ctx.set_config(crate::config::Config::Addr, Some("two@example.org"))
+                .await?;
+
+            // add and remove some accounts and force a gap (ids must not be reused)
+            for _ in 0..dummy_accounts {
+                let to_delete = accounts.add_account().await?;
+                accounts.remove_account(to_delete).await?;
+            }
+
+            let id2 = accounts.add_account().await?;
+            let ctx = accounts.get_account(id2).await.unwrap();
+            ctx.set_config(crate::config::Config::Addr, Some("three@example.org"))
+                .await?;
+
+            accounts.select_account(id1).await?;
+
+            (id0, id1, id2)
+        };
+        assert!(id0 > 0);
+        assert!(id1 > id0);
+        assert!(id2 > id1 + dummy_accounts);
+
+        let (id0_reopened, id1_reopened, id2_reopened) = {
+            let accounts = Accounts::new("my_os".into(), p.clone()).await?;
+            let ctx = accounts.get_selected_account().await.unwrap();
+            assert_eq!(
+                ctx.get_config(crate::config::Config::Addr).await?,
+                Some("two@example.org".to_string())
+            );
+
+            let ids = accounts.get_all().await;
+            assert_eq!(ids.len(), 3);
+
+            let id0 = *ids.get(0).unwrap();
+            let ctx = accounts.get_account(id0).await.unwrap();
+            assert_eq!(
+                ctx.get_config(crate::config::Config::Addr).await?,
+                Some("one@example.org".to_string())
+            );
+
+            let id1 = *ids.get(1).unwrap();
+            let t = accounts.get_account(id1).await.unwrap();
+            assert_eq!(
+                t.get_config(crate::config::Config::Addr).await?,
+                Some("two@example.org".to_string())
+            );
+
+            let id2 = *ids.get(2).unwrap();
+            let ctx = accounts.get_account(id2).await.unwrap();
+            assert_eq!(
+                ctx.get_config(crate::config::Config::Addr).await?,
+                Some("three@example.org".to_string())
+            );
+
+            (id0, id1, id2)
+        };
+        assert_eq!(id0, id0_reopened);
+        assert_eq!(id1, id1_reopened);
+        assert_eq!(id2, id2_reopened);
+
+        Ok(())
+    }
 }

src/blob.rs

@@ -74,7 +74,7 @@ impl<'a> BlobObject<'a> {
 
         // workaround a bug in async-std
         // (the executor does not handle blocking operation in Drop correctly,
-        // see https://github.com/async-rs/async-std/issues/900 )
+        // see <https://github.com/async-rs/async-std/issues/900>)
         let _ = file.flush().await;
 
         let blob = BlobObject {
@@ -452,7 +452,7 @@ impl<'a> BlobObject<'a> {
             img.write_to(encoded, image::ImageFormat::Jpeg)?;
             Ok(())
         }
-        fn encode_img_exceeds_bytes(
+        fn encoded_img_exceeds_bytes(
             context: &Context,
             img: &DynamicImage,
             max_bytes: Option<usize>,
@@ -477,7 +477,7 @@ impl<'a> BlobObject<'a> {
         let exceeds_width = img.width() > img_wh || img.height() > img_wh;
 
         let do_scale =
-            exceeds_width || encode_img_exceeds_bytes(context, &img, max_bytes, &mut encoded)?;
+            exceeds_width || encoded_img_exceeds_bytes(context, &img, max_bytes, &mut encoded)?;
         let do_rotate = matches!(orientation, Ok(90) | Ok(180) | Ok(270));
 
         if do_scale || do_rotate {
@@ -500,7 +500,7 @@ impl<'a> BlobObject<'a> {
                 loop {
                     let new_img = img.thumbnail(img_wh, img_wh);
 
-                    if encode_img_exceeds_bytes(context, &new_img, max_bytes, &mut encoded)? {
+                    if encoded_img_exceeds_bytes(context, &new_img, max_bytes, &mut encoded)? {
                         if img_wh < 20 {
                             return Err(format_err!(
                                 "Failed to scale image to below {}B",
@@ -511,6 +511,10 @@ impl<'a> BlobObject<'a> {
 
                         img_wh = img_wh * 2 / 3;
                     } else {
+                        if encoded.is_empty() {
+                            encode_img(&new_img, &mut encoded)?;
+                        }
+
                         info!(
                             context,
                             "Final scaled-down image size: {}B ({}px)",
@@ -617,7 +621,8 @@ mod tests {
 
     use super::*;
 
-    use crate::test_utils::TestContext;
+    use crate::{message::Message, test_utils::TestContext};
+    use image::Pixel;
 
     #[async_std::test]
     async fn test_create() {
@@ -915,4 +920,148 @@ mod tests {
         let avatar_cfg = t.get_config(Config::Selfavatar).await.unwrap();
         assert_eq!(avatar_cfg, avatar_blob.to_str().map(|s| s.to_string()));
     }
+
+    #[async_std::test]
+    async fn test_recode_image() {
+        let bytes = include_bytes!("../test-data/image/avatar1000x1000.jpg");
+        // BALANCED_IMAGE_SIZE > 1000, the original image size, so the image is not scaled down:
+        send_image_check_mediaquality(Some("0"), bytes, 1000, 1000, 0, 1000, 1000)
+            .await
+            .unwrap();
+        send_image_check_mediaquality(
+            Some("1"),
+            bytes,
+            1000,
+            1000,
+            0,
+            WORSE_IMAGE_SIZE,
+            WORSE_IMAGE_SIZE,
+        )
+        .await
+        .unwrap();
+
+        // The "-rotated" files are rotated by 270 degrees using the Exif metadata
+        let bytes = include_bytes!("../test-data/image/rectangle2000x1800-rotated.jpg");
+        let img_rotated = send_image_check_mediaquality(
+            Some("0"),
+            bytes,
+            2000,
+            1800,
+            270,
+            BALANCED_IMAGE_SIZE * 1800 / 2000,
+            BALANCED_IMAGE_SIZE,
+        )
+        .await
+        .unwrap();
+        assert_correct_rotation(&img_rotated);
+
+        let mut bytes = vec![];
+        img_rotated
+            .write_to(&mut bytes, image::ImageFormat::Jpeg)
+            .unwrap();
+        let img_rotated = send_image_check_mediaquality(
+            Some("0"),
+            &bytes,
+            BALANCED_IMAGE_SIZE * 1800 / 2000,
+            BALANCED_IMAGE_SIZE,
+            0,
+            BALANCED_IMAGE_SIZE * 1800 / 2000,
+            BALANCED_IMAGE_SIZE,
+        )
+        .await
+        .unwrap();
+        assert_correct_rotation(&img_rotated);
+
+        let img_rotated = send_image_check_mediaquality(
+            Some("1"),
+            &bytes,
+            BALANCED_IMAGE_SIZE * 1800 / 2000,
+            BALANCED_IMAGE_SIZE,
+            0,
+            WORSE_IMAGE_SIZE * 1800 / 2000,
+            WORSE_IMAGE_SIZE,
+        )
+        .await
+        .unwrap();
+        assert_correct_rotation(&img_rotated);
+
+        let bytes = include_bytes!("../test-data/image/rectangle200x180-rotated.jpg");
+        let img_rotated = send_image_check_mediaquality(Some("0"), bytes, 200, 180, 270, 180, 200)
+            .await
+            .unwrap();
+        assert_correct_rotation(&img_rotated);
+
+        let bytes = include_bytes!("../test-data/image/rectangle200x180-rotated.jpg");
+        let img_rotated = send_image_check_mediaquality(Some("1"), bytes, 200, 180, 270, 180, 200)
+            .await
+            .unwrap();
+        assert_correct_rotation(&img_rotated);
+    }
+
+    fn assert_correct_rotation(img: &DynamicImage) {
+        // The test images are black in the bottom left corner after correctly applying
+        // the EXIF orientation
+
+        let [luma] = img.get_pixel(10, 10).to_luma().0;
+        assert_eq!(luma, 255);
+        let [luma] = img.get_pixel(img.width() - 10, 10).to_luma().0;
+        assert_eq!(luma, 255);
+        let [luma] = img
+            .get_pixel(img.width() - 10, img.height() - 10)
+            .to_luma()
+            .0;
+        assert_eq!(luma, 255);
+        let [luma] = img.get_pixel(10, img.height() - 10).to_luma().0;
+        assert_eq!(luma, 0);
+    }
+
+    async fn send_image_check_mediaquality(
+        media_quality_config: Option<&str>,
+        bytes: &[u8],
+        original_width: u32,
+        original_height: u32,
+        orientation: i32,
+        compressed_width: u32,
+        compressed_height: u32,
+    ) -> anyhow::Result<DynamicImage> {
+        let alice = TestContext::new_alice().await;
+        let bob = TestContext::new_bob().await;
+        alice
+            .set_config(Config::MediaQuality, media_quality_config)
+            .await?;
+        let file = alice.get_blobdir().join("file.jpg");
+
+        File::create(&file).await?.write_all(bytes).await?;
+        let img = image::open(&file)?;
+        assert_eq!(img.width(), original_width);
+        assert_eq!(img.height(), original_height);
+
+        let blob = BlobObject::new_from_path(&alice, &file).await?;
+        assert_eq!(blob.get_exif_orientation(&alice).unwrap_or(0), orientation);
+
+        let mut msg = Message::new(Viewtype::Image);
+        msg.set_file(file.to_str().unwrap(), None);
+        let chat = alice.create_chat(&bob).await;
+        let sent = alice.send_msg(chat.id, &mut msg).await;
+        let alice_msg = alice.get_last_msg().await;
+        assert_eq!(alice_msg.get_width() as u32, compressed_width);
+        assert_eq!(alice_msg.get_height() as u32, compressed_height);
+        let img = image::open(alice_msg.get_file(&alice).unwrap())?;
+        assert_eq!(img.width() as u32, compressed_width);
+        assert_eq!(img.height() as u32, compressed_height);
+
+        bob.recv_msg(&sent).await;
+        let bob_msg = bob.get_last_msg().await;
+        assert_eq!(bob_msg.get_width() as u32, compressed_width);
+        assert_eq!(bob_msg.get_height() as u32, compressed_height);
+        let file = bob_msg.get_file(&bob).unwrap();
+
+        let blob = BlobObject::new_from_path(&bob, &file).await?;
+        assert_eq!(blob.get_exif_orientation(&bob).unwrap_or(0), 0);
+
+        let img = image::open(file)?;
+        assert_eq!(img.width() as u32, compressed_width);
+        assert_eq!(img.height() as u32, compressed_height);
+        Ok(img)
+    }
 }

src/chat.rs

@@ -8,19 +8,17 @@ use anyhow::{bail, ensure, format_err, Context as _, Result};
 use async_std::path::{Path, PathBuf};
 use deltachat_derive::{FromSql, ToSql};
 use itertools::Itertools;
-use num_traits::FromPrimitive;
 use serde::{Deserialize, Serialize};
 
 use crate::aheader::EncryptPreference;
 use crate::blob::{BlobError, BlobObject};
-use crate::chatlist::dc_get_archived_cnt;
 use crate::color::str_to_color;
 use crate::config::Config;
 use crate::constants::{
-    Blocked, Chattype, ShowEmails, Viewtype, DC_CHAT_ID_ALLDONE_HINT, DC_CHAT_ID_ARCHIVED_LINK,
-    DC_CHAT_ID_DEADDROP, DC_CHAT_ID_LAST_SPECIAL, DC_CHAT_ID_TRASH, DC_CONTACT_ID_DEVICE,
-    DC_CONTACT_ID_INFO, DC_CONTACT_ID_LAST_SPECIAL, DC_CONTACT_ID_SELF, DC_GCM_ADDDAYMARKER,
-    DC_GCM_INFO_ONLY, DC_RESEND_USER_AVATAR_DAYS,
+    Blocked, Chattype, Viewtype, DC_CHAT_ID_ALLDONE_HINT, DC_CHAT_ID_ARCHIVED_LINK,
+    DC_CHAT_ID_LAST_SPECIAL, DC_CHAT_ID_TRASH, DC_CONTACT_ID_DEVICE, DC_CONTACT_ID_INFO,
+    DC_CONTACT_ID_LAST_SPECIAL, DC_CONTACT_ID_SELF, DC_GCM_ADDDAYMARKER, DC_GCM_INFO_ONLY,
+    DC_RESEND_USER_AVATAR_DAYS,
 };
 use crate::contact::{addr_cmp, Contact, Origin, VerifiedStatus};
 use crate::context::Context;
@@ -33,7 +31,7 @@ use crate::ephemeral::{delete_expired_messages, schedule_ephemeral_task, Timer a
 use crate::events::EventType;
 use crate::html::new_html_mimepart;
 use crate::job::{self, Action};
-use crate::message::{self, InvalidMsgId, Message, MessageState, MsgId};
+use crate::message::{self, Message, MessageState, MsgId};
 use crate::mimeparser::SystemMessage;
 use crate::param::{Param, Params};
 use crate::peerstate::{Peerstate, PeerstateVerifiedStatus};
@@ -114,15 +112,6 @@ impl ChatId {
         (0..=DC_CHAT_ID_LAST_SPECIAL.0).contains(&self.0)
     }
 
-    /// Chat ID which represents the deaddrop chat.
-    ///
-    /// This is a virtual chat showing all messages belonging to chats
-    /// flagged with [Blocked::Deaddrop].  Usually the UI will show
-    /// these messages as contact requests.
-    pub fn is_deaddrop(self) -> bool {
-        self == DC_CHAT_ID_DEADDROP
-    }
-
     /// Chat ID for messages which need to be deleted.
     ///
     /// Messages which should be deleted get this chat ID and are
@@ -181,14 +170,11 @@ impl ChatId {
     ///
     /// This should be used when **a user action** creates a chat 1:1, it ensures the chat
     /// exists and is unblocked and scales the [`Contact`]'s origin.
-    ///
-    /// If a chat was in the deaddrop unblocking is how it becomes a normal chat and it will
-    /// look to the user like the chat was newly created.
     pub async fn create_for_contact(context: &Context, contact_id: u32) -> Result<Self> {
         let chat_id = match ChatIdBlocked::lookup_by_contact(context, contact_id).await? {
             Some(chat) => {
                 if chat.blocked != Blocked::Not {
-                    chat.id.unblock(context).await;
+                    chat.id.unblock(context).await?;
                 }
                 chat.id
             }
@@ -228,23 +214,90 @@ impl ChatId {
         Ok(())
     }
 
-    pub async fn set_blocked(self, context: &Context, new_blocked: Blocked) -> bool {
+    /// Updates chat blocked status.
+    ///
+    /// Returns true if the value was modified.
+    async fn set_blocked(self, context: &Context, new_blocked: Blocked) -> Result<bool> {
         if self.is_special() {
-            warn!(context, "ignoring setting of Block-status for {}", self);
-            return false;
+            bail!("ignoring setting of Block-status for {}", self);
         }
-        context
+        let count = context
             .sql
             .execute(
-                "UPDATE chats SET blocked=? WHERE id=?;",
+                "UPDATE chats SET blocked=?1 WHERE id=?2 AND blocked != ?1",
                 paramsv![new_blocked, self],
             )
-            .await
-            .is_ok()
+            .await?;
+        Ok(count > 0)
     }
 
-    pub async fn unblock(self, context: &Context) {
-        self.set_blocked(context, Blocked::Not).await;
+    /// Blocks the chat as a result of explicit user action.
+    pub async fn block(self, context: &Context) -> Result<()> {
+        let chat = Chat::load_from_db(context, self).await?;
+
+        match chat.typ {
+            Chattype::Undefined => bail!("Can't block chat of undefined chattype"),
+            Chattype::Single => {
+                for contact_id in get_chat_contacts(context, self).await? {
+                    if contact_id != DC_CONTACT_ID_SELF {
+                        info!(
+                            context,
+                            "Blocking the contact {} to block 1:1 chat", contact_id
+                        );
+                        Contact::block(context, contact_id).await?;
+                    }
+                }
+            }
+            Chattype::Group => {
+                info!(context, "Can't block groups yet, deleting the chat");
+                self.delete(context).await?;
+            }
+            Chattype::Mailinglist => {
+                if self.set_blocked(context, Blocked::Manually).await? {
+                    context.emit_event(EventType::ChatModified(self));
+                }
+            }
+        }
+
+        Ok(())
+    }
+
+    /// Unblocks the chat.
+    pub async fn unblock(self, context: &Context) -> Result<()> {
+        self.set_blocked(context, Blocked::Not).await?;
+        Ok(())
+    }
+
+    /// Accept the contact request.
+    ///
+    /// Unblocks the chat and scales up origin of contacts.
+    pub async fn accept(self, context: &Context) -> Result<()> {
+        let chat = Chat::load_from_db(context, self).await?;
+
+        match chat.typ {
+            Chattype::Undefined => bail!("Can't accept chat of undefined chattype"),
+            Chattype::Single | Chattype::Group => {
+                // User has "created a chat" with all these contacts.
+                //
+                // Previously accepting a chat literally created a chat because unaccepted chats
+                // went to "contact requests" list rather than normal chatlist.
+                for contact_id in get_chat_contacts(context, self).await? {
+                    if contact_id != DC_CONTACT_ID_SELF {
+                        Contact::scaleup_origin_by_id(context, contact_id, Origin::CreateChat)
+                            .await;
+                    }
+                }
+            }
+            Chattype::Mailinglist => {
+                // If the message is from a mailing list, the contacts are not counted as "known"
+            }
+        }
+
+        if self.set_blocked(context, Blocked::Not).await? {
+            context.emit_event(EventType::ChatModified(self));
+        }
+
+        Ok(())
     }
 
     /// Sets protection without sending a message.
@@ -453,12 +506,12 @@ impl ChatId {
     /// Sets draft message.
     ///
     /// Passing `None` as message just deletes the draft
-    pub async fn set_draft(self, context: &Context, msg: Option<&mut Message>) -> Result<()> {
+    pub async fn set_draft(self, context: &Context, mut msg: Option<&mut Message>) -> Result<()> {
         if self.is_special() {
             return Ok(());
         }
 
-        let changed = match msg {
+        let changed = match &mut msg {
             None => self.maybe_delete_draft(context).await?,
             Some(msg) => self.set_draft_raw(context, msg).await?,
         };
@@ -466,7 +519,14 @@ impl ChatId {
         if changed {
             context.emit_event(EventType::MsgsChanged {
                 chat_id: self,
-                msg_id: MsgId::new(0),
+                msg_id: if msg.is_some() {
+                    match self.get_draft_msg_id(context).await? {
+                        Some(msg_id) => msg_id,
+                        None => MsgId::new(0),
+                    }
+                } else {
+                    MsgId::new(0)
+                },
             });
         }
 
@@ -538,7 +598,7 @@ impl ChatId {
         }
 
         let chat = Chat::load_from_db(context, self).await?;
-        if !chat.can_send() {
+        if !chat.can_send(context).await {
             bail!("Can't set a draft: Can't send");
         }
 
@@ -576,7 +636,10 @@ impl ChatId {
     pub async fn get_msg_cnt(self, context: &Context) -> Result<usize> {
         let count = context
             .sql
-            .count("SELECT COUNT(*) FROM msgs WHERE chat_id=?", paramsv![self])
+            .count(
+                "SELECT COUNT(*) FROM msgs WHERE hidden=0 AND chat_id=?",
+                paramsv![self],
+            )
             .await?;
         Ok(count as usize)
     }
@@ -597,10 +660,10 @@ impl ChatId {
             .count(
                 "SELECT COUNT(*)
                 FROM msgs
-                WHERE state=10
+                WHERE state=?
                 AND hidden=0
                 AND chat_id=?;",
-                paramsv![self],
+                paramsv![MessageState::InFresh, self],
             )
             .await?;
         Ok(count as usize)
@@ -740,9 +803,7 @@ impl ChatId {
 
 impl std::fmt::Display for ChatId {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        if self.is_deaddrop() {
-            write!(f, "Chat#Deadrop")
-        } else if self.is_trash() {
+        if self.is_trash() {
             write!(f, "Chat#Trash")
         } else if self.is_archived_link() {
             write!(f, "Chat#ArchivedLink")
@@ -829,12 +890,8 @@ impl Chat {
             .await
             .context(format!("Failed loading chat {} from database", chat_id))?;
 
-        if chat.id.is_deaddrop() {
-            chat.name = stock_str::dead_drop(context).await;
-        } else if chat.id.is_archived_link() {
-            let tempname = stock_str::archived_chats(context).await;
-            let cnt = dc_get_archived_cnt(context).await?;
-            chat.name = format!("{} ({})", tempname, cnt);
+        if chat.id.is_archived_link() {
+            chat.name = stock_str::archived_chats(context).await;
         } else {
             if chat.typ == Chattype::Single {
                 let mut chat_name = "Err [Name not found]".to_owned();
@@ -876,8 +933,13 @@ impl Chat {
     }
 
     /// Returns true if user can send messages to this chat.
-    pub fn can_send(&self) -> bool {
-        !self.id.is_special() && !self.is_device_talk() && !self.is_mailing_list()
+    pub async fn can_send(&self, context: &Context) -> bool {
+        !self.id.is_special()
+            && !self.is_device_talk()
+            && !self.is_mailing_list()
+            && !self.is_contact_request()
+            && (self.typ == Chattype::Single
+                || is_contact_in_chat(context, self.id, DC_CONTACT_ID_SELF).await)
     }
 
     pub async fn update_param(&mut self, context: &Context) -> Result<()> {
@@ -977,6 +1039,14 @@ impl Chat {
         self.visibility
     }
 
+    /// Returns true if chat is a contact request.
+    ///
+    /// Messages cannot be sent to such chat and read receipts are not
+    /// sent until the chat is manually unblocked.
+    pub fn is_contact_request(&self) -> bool {
+        self.blocked == Blocked::Request
+    }
+
     pub fn is_unpromoted(&self) -> bool {
         self.param.get_int(Param::Unpromoted).unwrap_or_default() == 1
     }
@@ -1087,7 +1157,7 @@ impl Chat {
                 }
 
                 // the whole list of messages referenced may be huge;
-                // only use the oldest and and the parent message
+                // only use the oldest and the parent message
                 let parent_references = parent_references
                     .find(' ')
                     .and_then(|n| parent_references.get(..n))
@@ -1243,7 +1313,7 @@ impl rusqlite::types::FromSql for ChatVisibility {
                 2 => ChatVisibility::Pinned,
                 1 => ChatVisibility::Archived,
                 0 => ChatVisibility::Normal,
-                // fallback to to Normal for unknown values, may happen eg. on imports created by a newer version.
+                // fallback to Normal for unknown values, may happen eg. on imports created by a newer version.
                 _ => ChatVisibility::Normal,
             }
         })
@@ -1311,58 +1381,12 @@ pub struct ChatInfo {
     /// Ephemeral message timer.
     pub ephemeral_timer: EphemeralTimer,
     // ToDo:
-    // - [ ] deaddrop,
     // - [ ] summary,
     // - [ ] lastUpdated,
     // - [ ] freshMessageCounter,
     // - [ ] email
 }
 
-/// Create a chat from a message ID.
-///
-/// Typically you'd do this for a message ID found in the
-/// [DC_CHAT_ID_DEADDROP] which turns the chat the message belongs to
-/// into a normal chat.  The chat can be a 1:1 chat or a group chat
-/// and all messages belonging to the chat will be moved from the
-/// deaddrop to the normal chat.
-///
-/// In reality the messages already belong to this chat as receive_imf
-/// always creates chat IDs appropriately, so this function really
-/// only unblocks the chat and "scales up" the origin of the contact
-/// the message is from.
-///
-/// If prompting the user before calling this function, they should be
-/// asked whether they want to chat with the **contact** the message
-/// is from and **not** the group name since this can be really weird
-/// and confusing when taken from subject of implicit groups.
-///
-/// # Returns
-///
-/// The "created" chat ID is returned.
-pub async fn create_by_msg_id(context: &Context, msg_id: MsgId) -> Result<ChatId> {
-    let msg = Message::load_from_db(context, msg_id).await?;
-    let chat = Chat::load_from_db(context, msg.chat_id).await?;
-    ensure!(
-        !chat.id.is_special(),
-        "Message can not belong to a special chat"
-    );
-    if chat.blocked != Blocked::Not {
-        chat.id.unblock(context).await;
-
-        // Sending with 0s as data since multiple messages may have changed.
-        context.emit_event(EventType::MsgsChanged {
-            chat_id: ChatId::new(0),
-            msg_id: MsgId::new(0),
-        });
-    }
-
-    // If the message is from a mailing list, the contacts are not counted as "known"
-    if !chat.is_mailing_list() {
-        Contact::scaleup_origin_by_id(context, msg.from_id, Origin::CreateChat).await;
-    }
-    Ok(chat.id)
-}
-
 pub(crate) async fn update_saved_messages_icon(context: &Context) -> Result<()> {
     // if there is no saved-messages chat, there is nothing to update. this is no error.
     if let Some(chat_id) = ChatId::lookup_by_contact(context, DC_CONTACT_ID_SELF).await? {
@@ -1497,6 +1521,7 @@ impl ChatIdBlocked {
             _ => (),
         }
 
+        let created_timestamp = dc_create_smeared_timestamp(context).await;
         let chat_id = context
             .sql
             .transaction(move |transaction| {
@@ -1509,7 +1534,7 @@ impl ChatIdBlocked {
                         chat_name,
                         params.to_string(),
                         create_blocked as u8,
-                        time(),
+                        created_timestamp,
                     ],
                 )?;
                 let chat_id = ChatId::new(
@@ -1635,7 +1660,7 @@ async fn prepare_msg_common(
     chat_id.unarchive(context).await?;
 
     let mut chat = Chat::load_from_db(context, chat_id).await?;
-    ensure!(chat.can_send(), "cannot send to {}", chat_id);
+    ensure!(chat.can_send(context).await, "cannot send to {}", chat_id);
 
     // The OutPreparing state is set by dc_prepare_msg() before it
     // calls this function and the message is left in the OutPreparing
@@ -1684,11 +1709,7 @@ pub async fn send_msg(context: &Context, chat_id: ChatId, msg: &mut Message) ->
         let forwards = msg.param.get(Param::PrepForwards);
         if let Some(forwards) = forwards {
             for forward in forwards.split(' ') {
-                if let Ok(msg_id) = forward
-                    .parse::<u32>()
-                    .map_err(|_| InvalidMsgId)
-                    .map(MsgId::new)
-                {
+                if let Ok(msg_id) = forward.parse::<u32>().map(MsgId::new) {
                     if let Ok(mut msg) = Message::load_from_db(context, msg_id).await {
                         send_msg_inner(context, chat_id, &mut msg).await?;
                     };
@@ -1903,35 +1924,7 @@ pub async fn get_chat_msgs(
         Ok(ret)
     };
 
-    let items = if chat_id.is_deaddrop() {
-        let show_emails = ShowEmails::from_i32(context.get_config_int(Config::ShowEmails).await?)
-            .unwrap_or_default();
-        context
-            .sql
-            .query_map(
-                "SELECT m.id AS id, m.timestamp AS timestamp
-               FROM msgs m
-               LEFT JOIN chats
-                      ON m.chat_id=chats.id
-               LEFT JOIN contacts
-                      ON m.from_id=contacts.id
-              WHERE m.from_id!=1  -- 1=DC_CONTACT_ID_SELF
-                AND m.from_id!=2  -- 2=DC_CONTACT_ID_INFO
-                AND m.hidden=0
-                AND chats.blocked=2
-                AND contacts.blocked=0
-                AND m.msgrmsg>=?
-              ORDER BY m.timestamp,m.id;",
-                paramsv![if show_emails == ShowEmails::All {
-                    0i32
-                } else {
-                    1i32
-                }],
-                process_row,
-                process_rows,
-            )
-            .await?
-    } else if (flags & DC_GCM_INFO_ONLY) != 0 {
+    let items = if (flags & DC_GCM_INFO_ONLY) != 0 {
         context
             .sql
             .query_map(
@@ -1990,31 +1983,6 @@ pub(crate) async fn marknoticed_chat_if_older_than(
 }
 
 pub async fn marknoticed_chat(context: &Context, chat_id: ChatId) -> Result<()> {
-    // for the virtual deaddrop chat-id,
-    // mark all messages that will appear in the deaddrop as noticed
-    if chat_id.is_deaddrop() {
-        if context
-            .sql
-            .execute(
-                "UPDATE msgs
-                        SET state=?1
-                      WHERE state=?2
-                        AND hidden=0
-                        AND chat_id IN (SELECT id FROM chats WHERE blocked=?3);",
-                paramsv![
-                    MessageState::InNoticed,
-                    MessageState::InFresh,
-                    Blocked::Deaddrop
-                ],
-            )
-            .await?
-            > 0
-        {
-            context.emit_event(EventType::MsgsNoticed(chat_id));
-        }
-        return Ok(());
-    }
-
     // "WHERE" below uses the index `(state, hidden, chat_id)`, see get_fresh_msg_cnt() for reasoning
     // the additional SELECT statement may speed up things as no write-blocking is needed.
     let exists = context
@@ -2076,15 +2044,7 @@ pub async fn get_chat_media(
                 },
             ],
             |row| row.get::<_, MsgId>(0),
-            |ids| {
-                let mut ret = Vec::new();
-                for id in ids {
-                    if let Ok(msg_id) = id {
-                        ret.push(msg_id)
-                    }
-                }
-                Ok(ret)
-            },
+            |ids| Ok(ids.flatten().collect()),
         )
         .await?;
     Ok(list)
@@ -2146,13 +2106,6 @@ pub async fn get_chat_contacts(context: &Context, chat_id: ChatId) -> Result<Vec
     // Normal chats do not include SELF.  Group chats do (as it may happen that one is deleted from a
     // groupchat but the chats stays visible, moreover, this makes displaying lists easier)
 
-    if chat_id.is_deaddrop() {
-        return Ok(Vec::new());
-    }
-
-    // we could also create a list for all contacts in the deaddrop by searching contacts belonging to chats with
-    // chats.blocked=2, however, currently this is not needed
-
     let list = context
         .sql
         .query_map(
@@ -2188,7 +2141,12 @@ pub async fn create_group_chat(
             "INSERT INTO chats
         (type, name, grpid, param, created_timestamp)
         VALUES(?, ?, ?, \'U=1\', ?);",
-            paramsv![Chattype::Group, chat_name, grpid, time(),],
+            paramsv![
+                Chattype::Group,
+                chat_name,
+                grpid,
+                dc_create_smeared_timestamp(context).await,
+            ],
         )
         .await?;
 
@@ -2738,7 +2696,7 @@ pub async fn forward_msgs(context: &Context, msg_ids: &[MsgId], chat_id: ChatId)
 
     chat_id.unarchive(context).await?;
     if let Ok(mut chat) = Chat::load_from_db(context, chat_id).await {
-        ensure!(chat.can_send(), "cannot send to {}", chat_id);
+        ensure!(chat.can_send(context).await, "cannot send to {}", chat_id);
         curr_timestamp = dc_create_smeared_timestamps(context, msg_ids.len()).await;
         let ids = context
             .sql
@@ -2765,8 +2723,11 @@ pub async fn forward_msgs(context: &Context, msg_ids: &[MsgId], chat_id: ChatId)
             // we tested a sort of broadcast
             // by not marking own forwarded messages as such,
             // however, this turned out to be to confusing and unclear.
-            msg.param
-                .set_int(Param::Forwarded, src_msg_id.to_u32() as i32);
+
+            if msg.get_viewtype() != Viewtype::Sticker {
+                msg.param
+                    .set_int(Param::Forwarded, src_msg_id.to_u32() as i32);
+            }
 
             msg.param.remove(Param::GuaranteeE2ee);
             msg.param.remove(Param::ForcePlaintext);
@@ -3061,11 +3022,13 @@ pub(crate) async fn add_info_msg(context: &Context, chat_id: ChatId, text: impl
 mod tests {
     use super::*;
 
-    use crate::chatlist::Chatlist;
+    use crate::chatlist::{dc_get_archived_cnt, Chatlist};
     use crate::constants::{DC_GCL_ARCHIVED_ONLY, DC_GCL_NO_SPECIALS};
     use crate::contact::Contact;
     use crate::dc_receive_imf::dc_receive_imf;
     use crate::test_utils::TestContext;
+    use async_std::fs::File;
+    use async_std::prelude::*;
 
     #[async_std::test]
     async fn test_chat_info() {
@@ -3182,24 +3145,11 @@ mod tests {
         assert!(chat.is_self_talk());
         assert!(chat.visibility == ChatVisibility::Normal);
         assert!(!chat.is_device_talk());
-        assert!(chat.can_send());
+        assert!(chat.can_send(&t).await);
         assert_eq!(chat.name, stock_str::saved_messages(&t).await);
         assert!(chat.get_profile_image(&t).await.unwrap().is_some());
     }
 
-    #[async_std::test]
-    async fn test_deaddrop_chat() {
-        let t = TestContext::new().await;
-        let chat = Chat::load_from_db(&t, DC_CHAT_ID_DEADDROP).await.unwrap();
-        assert_eq!(DC_CHAT_ID_DEADDROP.0, 1);
-        assert!(chat.id.is_deaddrop());
-        assert!(!chat.is_self_talk());
-        assert!(chat.visibility == ChatVisibility::Normal);
-        assert!(!chat.is_device_talk());
-        assert!(!chat.can_send());
-        assert_eq!(chat.name, stock_str::dead_drop(&t).await);
-    }
-
     #[async_std::test]
     async fn test_add_device_msg_unlabelled() {
         let t = TestContext::new().await;
@@ -3274,7 +3224,7 @@ mod tests {
         assert_eq!(chat.get_type(), Chattype::Single);
         assert!(chat.is_device_talk());
         assert!(!chat.is_self_talk());
-        assert!(!chat.can_send());
+        assert!(!chat.can_send(&t).await);
 
         assert_eq!(chat.name, stock_str::device_messages(&t).await);
         assert!(chat.get_profile_image(&t).await.unwrap().is_some());
@@ -3880,7 +3830,7 @@ mod tests {
         );
         send_text_msg(&alice, alice_chat_id, "hi!".to_string())
             .await
-            .ok();
+            .unwrap();
         assert_eq!(
             get_chat_msgs(&alice, alice_chat_id, 0, None)
                 .await
@@ -3904,11 +3854,14 @@ mod tests {
         let bob_chat = Chat::load_from_db(&bob, msg.chat_id).await.unwrap();
         assert_eq!(bob_chat.grpid, alice_chat.grpid);
 
+        // Bob accepts contact request.
+        bob_chat.id.unblock(&bob).await.unwrap();
+
         // Bob answers - simulate a normal MUA by not setting `Chat-*`-headers;
         // moreover, Bob's SMTP-server also replaces the `Message-ID:`-header
         send_text_msg(&bob, bob_chat.id, "ho!".to_string())
             .await
-            .ok();
+            .unwrap();
         let msg = bob.pop_sent_msg().await.payload();
         let msg = msg.replace("Message-ID: <Gr.", "Message-ID: <XXX");
         let msg = msg.replace("Chat-", "XXXX-");
@@ -3953,7 +3906,6 @@ mod tests {
         let chats = Chatlist::try_load(&t, 0, None, None).await?;
         assert_eq!(chats.len(), 1);
         assert_eq!(chats.get_chat_id(0), chat.id);
-        assert_ne!(chats.get_chat_id(0), DC_CHAT_ID_DEADDROP);
         assert_eq!(chat.id.get_fresh_msg_cnt(&t).await?, 1);
         assert_eq!(t.get_fresh_msgs().await?.len(), 1);
 
@@ -3979,7 +3931,7 @@ mod tests {
     }
 
     #[async_std::test]
-    async fn test_marknoticed_deaddrop_chat() -> Result<()> {
+    async fn test_contact_request_fresh_messages() -> Result<()> {
         let t = TestContext::new_alice().await;
 
         let chats = Chatlist::try_load(&t, 0, None, None).await?;
@@ -4002,8 +3954,14 @@ mod tests {
 
         let chats = Chatlist::try_load(&t, 0, None, None).await?;
         assert_eq!(chats.len(), 1);
-        assert_eq!(chats.get_chat_id(0), DC_CHAT_ID_DEADDROP);
-        let msgs = get_chat_msgs(&t, DC_CHAT_ID_DEADDROP, 0, None).await?;
+        let chat_id = chats.get_chat_id(0);
+        assert!(Chat::load_from_db(&t, chat_id)
+            .await
+            .unwrap()
+            .is_contact_request());
+        assert_eq!(chat_id.get_msg_cnt(&t).await?, 1);
+        assert_eq!(chat_id.get_fresh_msg_cnt(&t).await?, 1);
+        let msgs = get_chat_msgs(&t, chat_id, 0, None).await?;
         assert_eq!(msgs.len(), 1);
         let msg_id = match msgs.first().unwrap() {
             ChatItem::Message { msg_id } => *msg_id,
@@ -4011,16 +3969,249 @@ mod tests {
         };
         let msg = message::Message::load_from_db(&t, msg_id).await?;
         assert_eq!(msg.state, MessageState::InFresh);
-        assert_eq!(t.get_fresh_msgs().await?.len(), 0); // deaddrop is excluded from global badge
 
-        marknoticed_chat(&t, DC_CHAT_ID_DEADDROP).await?;
+        // Contact requests are excluded from global badge.
+        assert_eq!(t.get_fresh_msgs().await?.len(), 0);
 
         let chats = Chatlist::try_load(&t, 0, None, None).await?;
-        assert_eq!(chats.len(), 0);
+        assert_eq!(chats.len(), 1);
         let msg = message::Message::load_from_db(&t, msg_id).await?;
-        assert_eq!(msg.state, MessageState::InNoticed);
+        assert_eq!(msg.state, MessageState::InFresh);
         assert_eq!(t.get_fresh_msgs().await?.len(), 0);
 
         Ok(())
     }
+
+    #[async_std::test]
+    async fn test_contact_request_archive() -> Result<()> {
+        let t = TestContext::new_alice().await;
+
+        dc_receive_imf(
+            &t,
+            b"From: bob@example.org\n\
+                 To: alice@example.com\n\
+                 Message-ID: <2@example.org>\n\
+                 Chat-Version: 1.0\n\
+                 Date: Sun, 22 Mar 2021 19:37:57 +0000\n\
+                 \n\
+                 hello\n",
+            "INBOX",
+            1,
+            false,
+        )
+        .await?;
+
+        let chats = Chatlist::try_load(&t, 0, None, None).await?;
+        assert_eq!(chats.len(), 1);
+        let chat_id = chats.get_chat_id(0);
+        assert!(Chat::load_from_db(&t, chat_id).await?.is_contact_request());
+        assert_eq!(dc_get_archived_cnt(&t).await?, 0);
+
+        // archive request without accepting or blocking
+        chat_id.set_visibility(&t, ChatVisibility::Archived).await?;
+
+        let chats = Chatlist::try_load(&t, 0, None, None).await?;
+        assert_eq!(chats.len(), 1);
+        let chat_id = chats.get_chat_id(0);
+        assert!(chat_id.is_archived_link());
+        assert_eq!(dc_get_archived_cnt(&t).await?, 1);
+
+        let chats = Chatlist::try_load(&t, DC_GCL_ARCHIVED_ONLY, None, None).await?;
+        assert_eq!(chats.len(), 1);
+        let chat_id = chats.get_chat_id(0);
+        assert!(Chat::load_from_db(&t, chat_id).await?.is_contact_request());
+
+        Ok(())
+    }
+
+    #[async_std::test]
+    async fn test_classic_email_chat() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+
+        // Alice enables receiving classic emails.
+        alice
+            .set_config(Config::ShowEmails, Some("2"))
+            .await
+            .unwrap();
+
+        // Alice receives a classic (non-chat) message from Bob.
+        dc_receive_imf(
+            &alice,
+            b"From: bob@example.org\n\
+                 To: alice@example.com\n\
+                 Message-ID: <1@example.org>\n\
+                 Date: Sun, 22 Mar 2021 19:37:57 +0000\n\
+                 \n\
+                 hello\n",
+            "INBOX",
+            1,
+            false,
+        )
+        .await?;
+
+        let msg = alice.get_last_msg().await;
+        let chat_id = msg.chat_id;
+        assert_eq!(chat_id.get_fresh_msg_cnt(&alice).await?, 1);
+
+        let msgs = get_chat_msgs(&alice, chat_id, 0, None).await?;
+        assert_eq!(msgs.len(), 1);
+
+        // Alice disables receiving classic emails.
+        alice
+            .set_config(Config::ShowEmails, Some("0"))
+            .await
+            .unwrap();
+
+        // Already received classic email should still be in the chat.
+        assert_eq!(chat_id.get_fresh_msg_cnt(&alice).await?, 1);
+
+        let msgs = get_chat_msgs(&alice, chat_id, 0, None).await?;
+        assert_eq!(msgs.len(), 1);
+
+        Ok(())
+    }
+
+    async fn test_sticker(filename: &str, bytes: &[u8], w: i32, h: i32) -> Result<()> {
+        let alice = TestContext::new_alice().await;
+        let bob = TestContext::new_bob().await;
+        let alice_chat = alice.create_chat(&bob).await;
+        let bob_chat = bob.create_chat(&alice).await;
+
+        let file = alice.get_blobdir().join(filename);
+        File::create(&file).await?.write_all(bytes).await?;
+
+        let mut msg = Message::new(Viewtype::Sticker);
+        msg.set_file(file.to_str().unwrap(), None);
+
+        let sent_msg = alice.send_msg(alice_chat.id, &mut msg).await;
+        let mime = sent_msg.payload();
+        assert_eq!(mime.match_indices("Chat-Content: sticker").count(), 1);
+
+        bob.recv_msg(&sent_msg).await;
+        let msg = bob.get_last_msg().await;
+        assert_eq!(msg.chat_id, bob_chat.id);
+        assert_eq!(msg.get_viewtype(), Viewtype::Sticker);
+        assert_eq!(msg.get_filename(), Some(filename.to_string()));
+        assert_eq!(msg.get_width(), w);
+        assert_eq!(msg.get_height(), h);
+        assert!(msg.get_filebytes(&bob).await > 250);
+
+        Ok(())
+    }
+
+    #[async_std::test]
+    async fn test_sticker_png() -> Result<()> {
+        test_sticker(
+            "sticker.png",
+            include_bytes!("../test-data/image/avatar64x64.png"),
+            64,
+            64,
+        )
+        .await
+    }
+
+    #[async_std::test]
+    async fn test_sticker_jpeg() -> Result<()> {
+        test_sticker(
+            "sticker.jpg",
+            include_bytes!("../test-data/image/avatar1000x1000.jpg"),
+            1000,
+            1000,
+        )
+        .await
+    }
+
+    #[async_std::test]
+    async fn test_sticker_gif() -> Result<()> {
+        test_sticker(
+            "sticker.gif",
+            include_bytes!("../test-data/image/image100x50.gif"),
+            100,
+            50,
+        )
+        .await
+    }
+
+    #[async_std::test]
+    async fn test_sticker_forward() -> Result<()> {
+        // create chats
+        let alice = TestContext::new_alice().await;
+        let bob = TestContext::new_bob().await;
+        let alice_chat = alice.create_chat(&bob).await;
+        let bob_chat = bob.create_chat(&alice).await;
+
+        // create sticker
+        let file_name = "sticker.jpg";
+        let bytes = include_bytes!("../test-data/image/avatar1000x1000.jpg");
+        let file = alice.get_blobdir().join(file_name);
+        File::create(&file).await?.write_all(bytes).await?;
+        let mut msg = Message::new(Viewtype::Sticker);
+        msg.set_file(file.to_str().unwrap(), None);
+
+        // send sticker to bob
+        let sent_msg = alice.send_msg(alice_chat.get_id(), &mut msg).await;
+        bob.recv_msg(&sent_msg).await;
+        let msg = bob.get_last_msg().await;
+
+        // forward said sticker to alice
+        forward_msgs(&bob, &[msg.id], bob_chat.get_id()).await?;
+        let forwarded_msg = bob.pop_sent_msg().await;
+        alice.recv_msg(&forwarded_msg).await;
+
+        // retrieve forwarded sticker which should not have forwarded-flag
+        let msg = alice.get_last_msg().await;
+
+        assert!(!msg.is_forwarded());
+        Ok(())
+    }
+
+    #[async_std::test]
+    async fn test_forward() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+        let bob = TestContext::new_bob().await;
+        let alice_chat = alice.create_chat(&bob).await;
+        let bob_chat = bob.create_chat(&alice).await;
+
+        let mut msg = Message::new(Viewtype::Text);
+        msg.set_text(Some("Hi Bob".to_owned()));
+        let sent_msg = alice.send_msg(alice_chat.get_id(), &mut msg).await;
+        bob.recv_msg(&sent_msg).await;
+        let msg = bob.get_last_msg().await;
+
+        forward_msgs(&bob, &[msg.id], bob_chat.get_id()).await?;
+
+        let forwarded_msg = bob.pop_sent_msg().await;
+        alice.recv_msg(&forwarded_msg).await;
+
+        let msg = alice.get_last_msg().await;
+        assert!(msg.get_text().unwrap() == "Hi Bob");
+        assert!(msg.is_forwarded());
+        Ok(())
+    }
+
+    #[async_std::test]
+    async fn test_can_send_group() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+        let bob = Contact::create(&alice, "", "bob@f.br").await?;
+        let chat_id = ChatId::create_for_contact(&alice, bob).await?;
+        let chat = Chat::load_from_db(&alice, chat_id).await?;
+        assert!(chat.can_send(&alice).await);
+        let chat_id = create_group_chat(&alice, ProtectionStatus::Unprotected, "foo").await?;
+        assert_eq!(
+            Chat::load_from_db(&alice, chat_id)
+                .await?
+                .can_send(&alice)
+                .await,
+            true
+        );
+        remove_contact_from_chat(&alice, chat_id, DC_CONTACT_ID_SELF).await?;
+        assert_eq!(
+            Chat::load_from_db(&alice, chat_id)
+                .await?
+                .can_send(&alice)
+                .await,
+            false
+        );
+        Ok(())
+    }
 }

src/chatlist.rs

@@ -4,9 +4,9 @@ use anyhow::{bail, ensure, Result};
 
 use crate::chat::{update_special_chat_names, Chat, ChatId, ChatVisibility};
 use crate::constants::{
-    Chattype, DC_CHAT_ID_ALLDONE_HINT, DC_CHAT_ID_ARCHIVED_LINK, DC_CHAT_ID_DEADDROP,
-    DC_CONTACT_ID_DEVICE, DC_CONTACT_ID_SELF, DC_CONTACT_ID_UNDEFINED, DC_GCL_ADD_ALLDONE_HINT,
-    DC_GCL_ARCHIVED_ONLY, DC_GCL_FOR_FORWARDING, DC_GCL_NO_SPECIALS,
+    Blocked, Chattype, DC_CHAT_ID_ALLDONE_HINT, DC_CHAT_ID_ARCHIVED_LINK, DC_CONTACT_ID_DEVICE,
+    DC_CONTACT_ID_SELF, DC_CONTACT_ID_UNDEFINED, DC_GCL_ADD_ALLDONE_HINT, DC_GCL_ARCHIVED_ONLY,
+    DC_GCL_FOR_FORWARDING, DC_GCL_NO_SPECIALS,
 };
 use crate::contact::Contact;
 use crate::context::Context;
@@ -34,15 +34,12 @@ use crate::stock_str;
 /// and for each messages that is scrolled into view, dc_get_msg() is called then.
 ///
 /// Why no listflags?
-/// Without listflags, dc_get_chatlist() adds the deaddrop and the archive "link" automatically as needed.
-/// The UI can just render these items differently then. Although the deaddrop link is currently always the
-/// first entry and only present on new messages, there is the rough idea that it can be optionally always
-/// present and sorted into the list by date. Rendering the deaddrop in the described way
-/// would not add extra work in the UI then.
+/// Without listflags, dc_get_chatlist() adds the archive "link" automatically as needed.
+/// The UI can just render these items differently then.
 #[derive(Debug)]
 pub struct Chatlist {
     /// Stores pairs of `chat_id, message_id`
-    ids: Vec<(ChatId, MsgId)>,
+    ids: Vec<(ChatId, Option<MsgId>)>,
 }
 
 impl Chatlist {
@@ -58,12 +55,6 @@ impl Chatlist {
     ///
     /// By default, the function adds some special entries to the list.
     /// These special entries can be identified by the ID returned by chatlist.get_chat_id():
-    /// - DC_CHAT_ID_DEADDROP (1) - this special chat is present if there are
-    ///   messages from addresses that have no relationship to the configured account.
-    ///   The last of these messages is represented by DC_CHAT_ID_DEADDROP and you can retrieve details
-    ///   about it with chatlist.get_msg_id(). Typically, the UI asks the user "Do you want to chat with NAME?"
-    ///   and offers the options "Start chat", "Block" and "Not now";
-    ///   The decision should be passed to dc_decide_on_contact_request().
     /// - DC_CHAT_ID_ARCHIVED_LINK (6) - this special chat is present if the user has
     ///   archived *any* chat using dc_set_chat_visibility(). The UI should show a link as
     ///   "Show archived chats", if the user clicks this item, the UI should show a
@@ -79,9 +70,9 @@ impl Chatlist {
     ///   the pseudo-chat DC_CHAT_ID_ARCHIVED_LINK is added if there are *any* archived
     ///   chats
     /// - the flag DC_GCL_FOR_FORWARDING sorts "Saved messages" to the top of the chatlist
-    ///   and hides the device-chat,
+    ///   and hides the device-chat and contact requests
     ///   typically used on forwarding, may be combined with DC_GCL_NO_SPECIALS
-    /// - if the flag DC_GCL_NO_SPECIALS is set, deaddrop and archive link are not added
+    /// - if the flag DC_GCL_NO_SPECIALS is set, archive link is not added
     ///   to the list (may be used eg. for selecting chats on forwarding, the flag is
     ///   not needed when DC_GCL_ARCHIVED_ONLY is already set)
     /// - if the flag DC_GCL_ADD_ALLDONE_HINT is set, DC_CHAT_ID_ALLDONE_HINT
@@ -111,7 +102,7 @@ impl Chatlist {
 
         let process_row = |row: &rusqlite::Row| {
             let chat_id: ChatId = row.get(0)?;
-            let msg_id: MsgId = row.get(1).unwrap_or_default();
+            let msg_id: Option<MsgId> = row.get(1)?;
             Ok((chat_id, msg_id))
         };
 
@@ -136,13 +127,8 @@ impl Chatlist {
         //   timestamp
         // - the list starts with the newest chats
         //
-        // nb: the query currently shows messages from blocked
-        // contacts in groups.  however, for normal-groups, this is
-        // okay as the message is also returned by dc_get_chat_msgs()
-        // (otherwise it would be hard to follow conversations, wa and
-        // tg do the same) for the deaddrop, however, they should
-        // really be hidden, however, _currently_ the deaddrop is not
-        // shown at all permanent in the chatlist.
+        // The query shows messages from blocked contacts in
+        // groups. Otherwise it would be hard to follow conversations.
         let mut ids = if let Some(query_contact_id) = query_contact_id {
             // show chats shared with a given contact
             context.sql.query_map(
@@ -157,7 +143,7 @@ impl Chatlist {
                                   AND (hidden=0 OR state=?1)
                                   ORDER BY timestamp DESC, id DESC LIMIT 1)
                  WHERE c.id>9
-                   AND c.blocked=0
+                   AND c.blocked!=1
                    AND c.id IN(SELECT chat_id FROM chats_contacts WHERE contact_id=?2)
                  GROUP BY c.id
                  ORDER BY c.archived=?3 DESC, IFNULL(m.timestamp,c.created_timestamp) DESC, m.id DESC;",
@@ -184,7 +170,7 @@ impl Chatlist {
                                   AND (hidden=0 OR state=?)
                                   ORDER BY timestamp DESC, id DESC LIMIT 1)
                  WHERE c.id>9
-                   AND c.blocked=0
+                   AND c.blocked!=1
                    AND c.archived=1
                  GROUP BY c.id
                  ORDER BY IFNULL(m.timestamp,c.created_timestamp) DESC, m.id DESC;",
@@ -218,7 +204,7 @@ impl Chatlist {
                                   AND (hidden=0 OR state=?1)
                                   ORDER BY timestamp DESC, id DESC LIMIT 1)
                  WHERE c.id>9 AND c.id!=?2
-                   AND c.blocked=0
+                   AND c.blocked!=1
                    AND c.name LIKE ?3
                  GROUP BY c.id
                  ORDER BY IFNULL(m.timestamp,c.created_timestamp) DESC, m.id DESC;",
@@ -236,7 +222,7 @@ impl Chatlist {
             } else {
                 ChatId::new(0)
             };
-            let mut ids = context.sql.query_map(
+            let ids = context.sql.query_map(
                 "SELECT c.id, m.id
                  FROM chats c
                  LEFT JOIN msgs m
@@ -248,22 +234,15 @@ impl Chatlist {
                                   AND (hidden=0 OR state=?1)
                                   ORDER BY timestamp DESC, id DESC LIMIT 1)
                  WHERE c.id>9 AND c.id!=?2
-                   AND c.blocked=0
-                   AND NOT c.archived=?3
+                   AND (c.blocked=0 OR (c.blocked=2 AND NOT ?3))
+                   AND NOT c.archived=?4
                  GROUP BY c.id
-                 ORDER BY c.id=?4 DESC, c.archived=?5 DESC, IFNULL(m.timestamp,c.created_timestamp) DESC, m.id DESC;",
-                paramsv![MessageState::OutDraft, skip_id, ChatVisibility::Archived, sort_id_up, ChatVisibility::Pinned],
+                 ORDER BY c.id=?5 DESC, c.archived=?6 DESC, IFNULL(m.timestamp,c.created_timestamp) DESC, m.id DESC;",
+                paramsv![MessageState::OutDraft, skip_id, flag_for_forwarding, ChatVisibility::Archived, sort_id_up, ChatVisibility::Pinned],
                 process_row,
                 process_rows,
             ).await?;
             if !flag_no_specials {
-                if let Some(last_deaddrop_fresh_msg_id) =
-                    get_last_deaddrop_fresh_msg(context).await?
-                {
-                    if !flag_for_forwarding {
-                        ids.insert(0, (DC_CHAT_ID_DEADDROP, last_deaddrop_fresh_msg_id));
-                    }
-                }
                 add_archived_link_item = true;
             }
             ids
@@ -271,9 +250,9 @@ impl Chatlist {
 
         if add_archived_link_item && dc_get_archived_cnt(context).await? > 0 {
             if ids.is_empty() && flag_add_alldone_hint {
-                ids.push((DC_CHAT_ID_ALLDONE_HINT, MsgId::new(0)));
+                ids.push((DC_CHAT_ID_ALLDONE_HINT, None));
             }
-            ids.push((DC_CHAT_ID_ARCHIVED_LINK, MsgId::new(0)));
+            ids.push((DC_CHAT_ID_ARCHIVED_LINK, None));
         }
 
         Ok(Chatlist { ids })
@@ -302,7 +281,7 @@ impl Chatlist {
     /// Get a single message ID of a chatlist.
     ///
     /// To get the message object from the message ID, use dc_get_msg().
-    pub fn get_msg_id(&self, index: usize) -> Result<MsgId> {
+    pub fn get_msg_id(&self, index: usize) -> Result<Option<MsgId>> {
         match self.ids.get(index) {
             Some((_chat_id, msg_id)) => Ok(*msg_id),
             None => bail!("Chatlist index out of range"),
@@ -323,18 +302,19 @@ impl Chatlist {
     /// - dc_lot_t::timestamp: the timestamp of the message.  0 if not applicable.
     /// - dc_lot_t::state: The state of the message as one of the DC_STATE_* constants (see #dc_msg_get_state()).
     //    0 if not applicable.
-    pub async fn get_summary(&self, context: &Context, index: usize, chat: Option<&Chat>) -> Lot {
+    pub async fn get_summary(
+        &self,
+        context: &Context,
+        index: usize,
+        chat: Option<&Chat>,
+    ) -> Result<Lot> {
         // The summary is created by the chat, not by the last message.
         // This is because we may want to display drafts here or stuff as
         // "is typing".
         // Also, sth. as "No messages" would not work if the summary comes from a message.
         let (chat_id, lastmsg_id) = match self.ids.get(index) {
             Some(ids) => ids,
-            None => {
-                let mut ret = Lot::new();
-                ret.text2 = Some("ErrBadChatlistIndex".to_string());
-                return Lot::new();
-            }
+            None => bail!("Chatlist index out of range"),
         };
 
         Chatlist::get_summary2(context, *chat_id, *lastmsg_id, chat).await
@@ -343,50 +323,50 @@ impl Chatlist {
     pub async fn get_summary2(
         context: &Context,
         chat_id: ChatId,
-        lastmsg_id: MsgId,
+        lastmsg_id: Option<MsgId>,
         chat: Option<&Chat>,
-    ) -> Lot {
+    ) -> Result<Lot> {
         let mut ret = Lot::new();
 
         let chat_loaded: Chat;
         let chat = if let Some(chat) = chat {
             chat
-        } else if let Ok(chat) = Chat::load_from_db(context, chat_id).await {
+        } else {
+            let chat = Chat::load_from_db(context, chat_id).await?;
             chat_loaded = chat;
             &chat_loaded
-        } else {
-            return ret;
         };
 
-        let (lastmsg, lastcontact) =
-            if let Ok(lastmsg) = Message::load_from_db(context, lastmsg_id).await {
-                if lastmsg.from_id == DC_CONTACT_ID_SELF {
-                    (Some(lastmsg), None)
-                } else {
-                    match chat.typ {
-                        Chattype::Group | Chattype::Mailinglist => {
-                            let lastcontact =
-                                Contact::load_from_db(context, lastmsg.from_id).await.ok();
-                            (Some(lastmsg), lastcontact)
-                        }
-                        Chattype::Single | Chattype::Undefined => (Some(lastmsg), None),
-                    }
-                }
+        let (lastmsg, lastcontact) = if let Some(lastmsg_id) = lastmsg_id {
+            let lastmsg = Message::load_from_db(context, lastmsg_id).await?;
+            if lastmsg.from_id == DC_CONTACT_ID_SELF {
+                (Some(lastmsg), None)
             } else {
-                (None, None)
-            };
+                match chat.typ {
+                    Chattype::Group | Chattype::Mailinglist => {
+                        let lastcontact =
+                            Contact::load_from_db(context, lastmsg.from_id).await.ok();
+                        (Some(lastmsg), lastcontact)
+                    }
+                    Chattype::Single | Chattype::Undefined => (Some(lastmsg), None),
+                }
+            }
+        } else {
+            (None, None)
+        };
 
         if chat.id.is_archived_link() {
             ret.text2 = None;
-        } else if lastmsg.is_none() || lastmsg.as_ref().unwrap().from_id == DC_CONTACT_ID_UNDEFINED
+        } else if let Some(mut lastmsg) =
+            lastmsg.filter(|msg| msg.from_id != DC_CONTACT_ID_UNDEFINED)
         {
-            ret.text2 = Some(stock_str::no_messages(context).await);
-        } else {
-            ret.fill(&mut lastmsg.unwrap(), chat, lastcontact.as_ref(), context)
+            ret.fill(&mut lastmsg, chat, lastcontact.as_ref(), context)
                 .await;
+        } else {
+            ret.text2 = Some(stock_str::no_messages(context).await);
         }
 
-        ret
+        Ok(ret)
     }
 
     pub fn get_index_for_id(&self, id: ChatId) -> Option<usize> {
@@ -399,35 +379,13 @@ pub async fn dc_get_archived_cnt(context: &Context) -> Result<usize> {
     let count = context
         .sql
         .count(
-            "SELECT COUNT(*) FROM chats WHERE blocked=0 AND archived=1;",
-            paramsv![],
+            "SELECT COUNT(*) FROM chats WHERE blocked!=? AND archived=?;",
+            paramsv![Blocked::Manually, ChatVisibility::Archived],
         )
         .await?;
     Ok(count)
 }
 
-async fn get_last_deaddrop_fresh_msg(context: &Context) -> Result<Option<MsgId>> {
-    // We have an index over the state-column, this should be
-    // sufficient as there are typically only few fresh messages.
-    let id = context
-        .sql
-        .query_get_value(
-            concat!(
-                "SELECT m.id",
-                " FROM msgs m",
-                " LEFT JOIN chats c",
-                "        ON c.id=m.chat_id",
-                " WHERE m.state=10",
-                "   AND m.hidden=0",
-                "   AND c.blocked=2",
-                " ORDER BY m.timestamp DESC, m.id DESC;"
-            ),
-            paramsv![],
-        )
-        .await?;
-    Ok(id)
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -435,8 +393,6 @@ mod tests {
     use crate::chat::{create_group_chat, get_chat_contacts, ProtectionStatus};
     use crate::constants::Viewtype;
     use crate::dc_receive_imf::dc_receive_imf;
-    use crate::message;
-    use crate::message::ContactRequestDecision;
     use crate::stock_str::StockMessage;
     use crate::test_utils::TestContext;
 
@@ -546,7 +502,7 @@ mod tests {
     async fn test_search_single_chat() -> anyhow::Result<()> {
         let t = TestContext::new_alice().await;
 
-        // receive a one-to-one-message, accept contact request
+        // receive a one-to-one-message
         dc_receive_imf(
             &t,
             b"From: Bob Authname <bob@example.org>\n\
@@ -564,15 +520,13 @@ mod tests {
         .await?;
 
         let chats = Chatlist::try_load(&t, 0, Some("Bob Authname"), None).await?;
-        assert_eq!(chats.len(), 0);
+        // Contact request should be searchable
+        assert_eq!(chats.len(), 1);
 
         let msg = t.get_last_msg().await;
-        assert_eq!(msg.get_chat_id(), DC_CHAT_ID_DEADDROP);
+        let chat_id = msg.get_chat_id();
+        chat_id.accept(&t).await.unwrap();
 
-        let chat_id =
-            message::decide_on_contact_request(&t, msg.get_id(), ContactRequestDecision::StartChat)
-                .await
-                .unwrap();
         let contacts = get_chat_contacts(&t, chat_id).await?;
         let contact_id = *contacts.first().unwrap();
         let chat = Chat::load_from_db(&t, chat_id).await?;
@@ -610,7 +564,7 @@ mod tests {
     async fn test_search_single_chat_without_authname() -> anyhow::Result<()> {
         let t = TestContext::new_alice().await;
 
-        // receive a one-to-one-message without authname set, accept contact request
+        // receive a one-to-one-message without authname set
         dc_receive_imf(
             &t,
             b"From: bob@example.org\n\
@@ -628,10 +582,8 @@ mod tests {
         .await?;
 
         let msg = t.get_last_msg().await;
-        let chat_id =
-            message::decide_on_contact_request(&t, msg.get_id(), ContactRequestDecision::StartChat)
-                .await
-                .unwrap();
+        let chat_id = msg.get_chat_id();
+        chat_id.accept(&t).await.unwrap();
         let contacts = get_chat_contacts(&t, chat_id).await?;
         let contact_id = *contacts.first().unwrap();
         let chat = Chat::load_from_db(&t, chat_id).await?;
@@ -684,7 +636,7 @@ mod tests {
         chat_id1.set_draft(&t, Some(&mut msg)).await.unwrap();
 
         let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
-        let summary = chats.get_summary(&t, 0, None).await;
+        let summary = chats.get_summary(&t, 0, None).await.unwrap();
         assert_eq!(summary.get_text2().unwrap(), "foo: bar test"); // the linebreak should be removed from summary
     }
 }

src/color.rs

@@ -42,7 +42,7 @@ mod tests {
     #[test]
     fn test_str_to_angle() {
         // Test against test vectors from
-        // https://xmpp.org/extensions/xep-0392.html#testvectors-fullrange-no-cvd
+        // <https://xmpp.org/extensions/xep-0392.html#testvectors-fullrange-no-cvd>
         assert!((str_to_angle("Romeo") - 327.255249).abs() < 1e-6);
         assert!((str_to_angle("juliet@capulet.lit") - 209.410400).abs() < 1e-6);
         assert!((str_to_angle("😺") - 331.199341).abs() < 1e-6);

src/config.rs

@@ -18,7 +18,18 @@ use crate::stock_str;
 
 /// The available configuration keys.
 #[derive(
-    Debug, Clone, Copy, PartialEq, Eq, Display, EnumString, AsRefStr, EnumIter, EnumProperty,
+    Debug,
+    Clone,
+    Copy,
+    PartialEq,
+    Eq,
+    Display,
+    EnumString,
+    AsRefStr,
+    EnumIter,
+    EnumProperty,
+    PartialOrd,
+    Ord,
 )]
 #[strum(serialize_all = "snake_case")]
 pub enum Config {

src/configure.rs

@@ -345,10 +345,8 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
     progress!(ctx, 600);
 
     // Configure IMAP
-    let (_s, r) = async_std::channel::bounded(1);
-    let mut imap = Imap::new(r);
 
-    let mut imap_configured = false;
+    let mut imap: Option<Imap> = None;
     let imap_servers: Vec<&ServerParams> = servers
         .iter()
         .filter(|params| params.protocol == Protocol::Imap)
@@ -361,18 +359,9 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
         param.imap.port = imap_server.port;
         param.imap.security = imap_server.socket;
 
-        match try_imap_one_param(
-            ctx,
-            &param.imap,
-            &param.addr,
-            oauth2,
-            provider_strict_tls,
-            &mut imap,
-        )
-        .await
-        {
-            Ok(_) => {
-                imap_configured = true;
+        match try_imap_one_param(ctx, &param.imap, &param.addr, oauth2, provider_strict_tls).await {
+            Ok(configured_imap) => {
+                imap = Some(configured_imap);
                 break;
             }
             Err(e) => errors.push(e),
@@ -382,9 +371,10 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
             600 + (800 - 600) * (1 + imap_server_index) / imap_servers_count
         );
     }
-    if !imap_configured {
-        bail!(nicer_configuration_error(ctx, errors).await);
-    }
+    let mut imap = match imap {
+        Some(imap) => imap,
+        None => bail!(nicer_configuration_error(ctx, errors).await),
+    };
 
     progress!(ctx, 850);
 
@@ -463,7 +453,7 @@ async fn get_autoconfig(
 
     if let Ok(res) = moz_autoconfigure(
         ctx,
-        // the doc does not mention `emailaddress=`, however, Thunderbird adds it, see https://releases.mozilla.org/pub/thunderbird/ ,  which makes some sense
+        // the doc does not mention `emailaddress=`, however, Thunderbird adds it, see <https://releases.mozilla.org/pub/thunderbird/>,  which makes some sense
         &format!(
             "https://{}/.well-known/autoconfig/mail/config-v1.1.xml?emailaddress={}",
             &param_domain, &param_addr_urlencoded
@@ -520,26 +510,38 @@ async fn try_imap_one_param(
     addr: &str,
     oauth2: bool,
     provider_strict_tls: bool,
-    imap: &mut Imap,
-) -> Result<(), ConfigurationError> {
+) -> Result<Imap, ConfigurationError> {
     let inf = format!(
         "imap: {}@{}:{} security={} certificate_checks={} oauth2={}",
         param.user, param.server, param.port, param.security, param.certificate_checks, oauth2
     );
     info!(context, "Trying: {}", inf);
 
-    if let Err(err) = imap
-        .connect(context, param, addr, oauth2, provider_strict_tls)
-        .await
-    {
-        info!(context, "failure: {}", err);
-        Err(ConfigurationError {
-            config: inf,
-            msg: err.to_string(),
-        })
-    } else {
-        info!(context, "success: {}", inf);
-        Ok(())
+    let (_s, r) = async_std::channel::bounded(1);
+
+    let mut imap = match Imap::new(param, addr, oauth2, provider_strict_tls, r).await {
+        Err(err) => {
+            info!(context, "failure: {}", err);
+            return Err(ConfigurationError {
+                config: inf,
+                msg: err.to_string(),
+            });
+        }
+        Ok(imap) => imap,
+    };
+
+    match imap.connect(context).await {
+        Err(err) => {
+            info!(context, "failure: {}", err);
+            Err(ConfigurationError {
+                config: inf,
+                msg: err.to_string(),
+            })
+        }
+        Ok(()) => {
+            info!(context, "success: {}", inf);
+            Ok(imap)
+        }
     }
 }
 
@@ -616,10 +618,10 @@ pub enum Error {
     },
 
     #[error("Failed to get URL: {0}")]
-    ReadUrlError(#[from] self::read_url::Error),
+    ReadUrl(#[from] self::read_url::Error),
 
     #[error("Number of redirection is exceeded")]
-    RedirectionError,
+    Redirection,
 }
 
 #[cfg(test)]

src/configure/auto_mozilla.rs

@@ -1,6 +1,6 @@
 //! # Thunderbird's Autoconfiguration implementation
 //!
-//! Documentation: https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration
+//! Documentation: <https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration>
 use quick_xml::events::{BytesStart, Event};
 
 use std::io::BufRead;

src/configure/auto_outlook.rs

@@ -15,27 +15,27 @@ use super::{Error, ServerParams};
 
 /// Result of parsing a single `Protocol` tag.
 ///
-/// https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/protocol-pox
+/// <https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/protocol-pox>
 #[derive(Debug)]
 struct ProtocolTag {
     /// Server type, such as "IMAP", "SMTP" or "POP3".
     ///
-    /// https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/type-pox
+    /// <https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/type-pox>
     pub typ: String,
 
     /// Server identifier, hostname or IP address for IMAP and SMTP.
     ///
-    /// https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/server-pox
+    /// <https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/server-pox>
     pub server: String,
 
     /// Network port.
     ///
-    /// https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/port-pox
+    /// <https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/port-pox>
     pub port: u16,
 
     /// Whether connection should be secure, "on" or "off", default is "on".
     ///
-    /// https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/ssl-pox
+    /// <https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/ssl-pox>
     pub ssl: bool,
 }
 
@@ -203,7 +203,7 @@ pub(crate) async fn outlk_autodiscover(
             }
         }
     }
-    Err(Error::RedirectionError)
+    Err(Error::Redirection)
 }
 
 #[cfg(test)]

src/configure/server_params.rs

@@ -25,16 +25,20 @@ pub(crate) struct ServerParams {
 }
 
 impl ServerParams {
-    fn expand_usernames(mut self, addr: &str) -> Vec<ServerParams> {
+    fn expand_usernames(self, addr: &str) -> Vec<ServerParams> {
         let mut res = Vec::new();
 
         if self.username.is_empty() {
-            self.username = addr.to_string();
-            res.push(self.clone());
+            res.push(Self {
+                username: addr.to_string(),
+                ..self.clone()
+            });
 
             if let Some(at) = addr.find('@') {
-                self.username = addr.split_at(at).0.to_string();
-                res.push(self);
+                res.push(Self {
+                    username: addr.split_at(at).0.to_string(),
+                    ..self
+                });
             }
         } else {
             res.push(self)
@@ -42,24 +46,28 @@ impl ServerParams {
         res
     }
 
-    fn expand_hostnames(mut self, param_domain: &str) -> Vec<ServerParams> {
-        let mut res = Vec::new();
+    fn expand_hostnames(self, param_domain: &str) -> Vec<ServerParams> {
         if self.hostname.is_empty() {
-            self.hostname = param_domain.to_string();
-            res.push(self.clone());
-
-            self.hostname = match self.protocol {
-                Protocol::Imap => "imap.".to_string() + param_domain,
-                Protocol::Smtp => "smtp.".to_string() + param_domain,
-            };
-            res.push(self.clone());
-
-            self.hostname = "mail.".to_string() + param_domain;
-            res.push(self);
+            vec![
+                Self {
+                    hostname: param_domain.to_string(),
+                    ..self.clone()
+                },
+                Self {
+                    hostname: match self.protocol {
+                        Protocol::Imap => "imap.".to_string() + param_domain,
+                        Protocol::Smtp => "smtp.".to_string() + param_domain,
+                    },
+                    ..self.clone()
+                },
+                Self {
+                    hostname: "mail.".to_string() + param_domain,
+                    ..self
+                },
+            ]
         } else {
-            res.push(self);
+            vec![self]
         }
-        res
     }
 
     fn expand_ports(mut self) -> Vec<ServerParams> {
@@ -78,39 +86,47 @@ impl ServerParams {
             }
         }
 
-        let mut res = Vec::new();
         if self.port == 0 {
             // Neither port nor security is set.
             //
             // Try common secure combinations.
 
-            // Try STARTTLS
-            self.socket = Socket::Starttls;
-            self.port = match self.protocol {
-                Protocol::Imap => 143,
-                Protocol::Smtp => 587,
-            };
-            res.push(self.clone());
-
-            // Try TLS
-            self.socket = Socket::Ssl;
-            self.port = match self.protocol {
-                Protocol::Imap => 993,
-                Protocol::Smtp => 465,
-            };
-            res.push(self);
+            vec![
+                // Try STARTTLS
+                Self {
+                    socket: Socket::Starttls,
+                    port: match self.protocol {
+                        Protocol::Imap => 143,
+                        Protocol::Smtp => 587,
+                    },
+                    ..self.clone()
+                },
+                // Try TLS
+                Self {
+                    socket: Socket::Ssl,
+                    port: match self.protocol {
+                        Protocol::Imap => 993,
+                        Protocol::Smtp => 465,
+                    },
+                    ..self
+                },
+            ]
         } else if self.socket == Socket::Automatic {
-            // Try TLS over user-provided port.
-            self.socket = Socket::Ssl;
-            res.push(self.clone());
-
-            // Try STARTTLS over user-provided port.
-            self.socket = Socket::Starttls;
-            res.push(self);
+            vec![
+                // Try TLS over user-provided port.
+                Self {
+                    socket: Socket::Ssl,
+                    ..self.clone()
+                },
+                // Try STARTTLS over user-provided port.
+                Self {
+                    socket: Socket::Starttls,
+                    ..self
+                },
+            ]
         } else {
-            res.push(self);
+            vec![self]
         }
-        res
     }
 }
 

src/constants.rs

@@ -25,7 +25,7 @@ pub static DC_VERSION_STR: Lazy<String> = Lazy::new(|| env!("CARGO_PKG_VERSION")
 pub enum Blocked {
     Not = 0,
     Manually = 1,
-    Deaddrop = 2,
+    Request = 2,
 }
 
 impl Default for Blocked {
@@ -123,8 +123,6 @@ pub const DC_RESEND_USER_AVATAR_DAYS: i64 = 14;
 // do not use too small value that will annoy users checking for nonexistant updates.
 pub const DC_OUTDATED_WARNING_DAYS: i64 = 365;
 
-/// virtual chat showing all messages belonging to chats flagged with chats.blocked=2
-pub const DC_CHAT_ID_DEADDROP: ChatId = ChatId::new(1);
 /// messages that should be deleted get this chat_id; the messages are deleted from the working thread later then. This is also needed as rfc724_mid should be preset as long as the message is not deleted on the server (otherwise it is downloaded again)
 pub const DC_CHAT_ID_TRASH: ChatId = ChatId::new(3);
 /// only an indicator in a chatlist
@@ -173,7 +171,7 @@ pub const DC_ELLIPSE: &str = "[...]";
 /// to keep bubbles and chat flow usable,
 /// and to avoid problems with controls using very long texts,
 /// we limit the text length to DC_DESIRED_TEXT_LEN.
-/// if the text is longer, the full text can be retrieved usind has_html()/get_html().
+/// if the text is longer, the full text can be retrieved using has_html()/get_html().
 ///
 /// we are using a bit less than DC_MAX_GET_TEXT_LEN to avoid cutting twice
 /// (a bit less as truncation may not be exact and ellipses may be added).
@@ -404,7 +402,7 @@ mod tests {
         assert_eq!(Blocked::Not, Blocked::default());
         assert_eq!(Blocked::Not, Blocked::from_i32(0).unwrap());
         assert_eq!(Blocked::Manually, Blocked::from_i32(1).unwrap());
-        assert_eq!(Blocked::Deaddrop, Blocked::from_i32(2).unwrap());
+        assert_eq!(Blocked::Request, Blocked::from_i32(2).unwrap());
     }
 
     #[test]

src/contact.rs

@@ -14,8 +14,8 @@ use crate::chat::ChatId;
 use crate::color::str_to_color;
 use crate::config::Config;
 use crate::constants::{
-    Blocked, Chattype, DC_CHAT_ID_DEADDROP, DC_CONTACT_ID_DEVICE, DC_CONTACT_ID_DEVICE_ADDR,
-    DC_CONTACT_ID_LAST_SPECIAL, DC_CONTACT_ID_SELF, DC_GCL_ADD_SELF, DC_GCL_VERIFIED_ONLY,
+    Blocked, Chattype, DC_CONTACT_ID_DEVICE, DC_CONTACT_ID_DEVICE_ADDR, DC_CONTACT_ID_LAST_SPECIAL,
+    DC_CONTACT_ID_SELF, DC_GCL_ADD_SELF, DC_GCL_VERIFIED_ONLY,
 };
 use crate::context::Context;
 use crate::dc_tools::{dc_get_abs_path, improve_single_line_input, EmailAddress};
@@ -236,13 +236,13 @@ impl Contact {
     }
 
     /// Block the given contact.
-    pub async fn block(context: &Context, id: u32) {
-        set_block_contact(context, id, true).await;
+    pub async fn block(context: &Context, id: u32) -> Result<()> {
+        set_block_contact(context, id, true).await
     }
 
     /// Unblock the given contact.
-    pub async fn unblock(context: &Context, id: u32) {
-        set_block_contact(context, id, false).await;
+    pub async fn unblock(context: &Context, id: u32) -> Result<()> {
+        set_block_contact(context, id, false).await
     }
 
     /// Add a single contact as a result of an _explicit_ user action.
@@ -270,27 +270,22 @@ impl Contact {
             }
         }
         if blocked {
-            Contact::unblock(context, contact_id).await;
+            Contact::unblock(context, contact_id).await?;
         }
 
         Ok(contact_id)
     }
 
     /// Mark messages from a contact as noticed.
-    /// The contact is expected to belong to the deaddrop,
-    /// therefore, DC_EVENT_MSGS_NOTICED(DC_CHAT_ID_DEADDROP) is emitted.
-    pub async fn mark_noticed(context: &Context, id: u32) {
-        if context
+    pub async fn mark_noticed(context: &Context, id: u32) -> Result<()> {
+        context
             .sql
             .execute(
                 "UPDATE msgs SET state=? WHERE from_id=? AND state=?;",
                 paramsv![MessageState::InNoticed, id as i32, MessageState::InFresh],
             )
-            .await
-            .is_ok()
-        {
-            context.emit_event(EventType::MsgsNoticed(DC_CHAT_ID_DEADDROP));
-        }
+            .await?;
+        Ok(())
     }
 
     /// Check if an e-mail address belongs to a known and unblocked contact.
@@ -860,7 +855,7 @@ impl Contact {
             "Can not delete special contact"
         );
 
-        let count_contacts = context
+        let count_chats = context
             .sql
             .count(
                 "SELECT COUNT(*) FROM chats_contacts WHERE contact_id=?;",
@@ -868,19 +863,7 @@ impl Contact {
             )
             .await?;
 
-        let count_msgs = if count_contacts > 0 {
-            context
-                .sql
-                .count(
-                    "SELECT COUNT(*) FROM msgs WHERE from_id=? OR to_id=?;",
-                    paramsv![contact_id as i32, contact_id as i32],
-                )
-                .await?
-        } else {
-            0
-        };
-
-        if count_msgs == 0 {
+        if count_chats == 0 {
             match context
                 .sql
                 .execute(
@@ -902,9 +885,9 @@ impl Contact {
 
         info!(
             context,
-            "could not delete contact {}, there are {} messages with it", contact_id, count_msgs
+            "could not delete contact {}, there are {} chats with it", contact_id, count_chats
         );
-        bail!("Could not delete contact with messages in it");
+        bail!("Could not delete contact with ongoing chats");
     }
 
     /// Get a single contact object.  For a list, see eg. dc_get_contacts().
@@ -1174,56 +1157,58 @@ fn sanitize_name_and_addr(name: &str, addr: &str) -> (String, String) {
     }
 }
 
-async fn set_block_contact(context: &Context, contact_id: u32, new_blocking: bool) {
-    if contact_id <= DC_CONTACT_ID_LAST_SPECIAL {
-        return;
-    }
+async fn set_block_contact(context: &Context, contact_id: u32, new_blocking: bool) -> Result<()> {
+    ensure!(
+        contact_id > DC_CONTACT_ID_LAST_SPECIAL,
+        "Can't block special contact {}",
+        contact_id
+    );
 
-    if let Ok(contact) = Contact::load_from_db(context, contact_id).await {
-        if contact.blocked != new_blocking
-            && context
-                .sql
-                .execute(
-                    "UPDATE contacts SET blocked=? WHERE id=?;",
-                    paramsv![new_blocking as i32, contact_id as i32],
-                )
-                .await
-                .is_ok()
-        {
-            // also (un)block all chats with _only_ this contact - we do not delete them to allow a
-            // non-destructive blocking->unblocking.
-            // (Maybe, beside normal chats (type=100) we should also block group chats with only this user.
-            // However, I'm not sure about this point; it may be confusing if the user wants to add other people;
-            // this would result in recreating the same group...)
-            if context
-                .sql
-                .execute(
-                    r#"
+    let contact = Contact::load_from_db(context, contact_id).await?;
+
+    if contact.blocked != new_blocking {
+        context
+            .sql
+            .execute(
+                "UPDATE contacts SET blocked=? WHERE id=?;",
+                paramsv![new_blocking as i32, contact_id as i32],
+            )
+            .await?;
+
+        // also (un)block all chats with _only_ this contact - we do not delete them to allow a
+        // non-destructive blocking->unblocking.
+        // (Maybe, beside normal chats (type=100) we should also block group chats with only this user.
+        // However, I'm not sure about this point; it may be confusing if the user wants to add other people;
+        // this would result in recreating the same group...)
+        if context
+            .sql
+            .execute(
+                r#"
 UPDATE chats
 SET blocked=?
 WHERE type=? AND id IN (
   SELECT chat_id FROM chats_contacts WHERE contact_id=?
 );
 "#,
-                    paramsv![new_blocking, Chattype::Single, contact_id],
-                )
-                .await
-                .is_ok()
-            {
-                Contact::mark_noticed(context, contact_id).await;
-                context.emit_event(EventType::ContactsChanged(Some(contact_id)));
-            }
+                paramsv![new_blocking, Chattype::Single, contact_id],
+            )
+            .await
+            .is_ok()
+        {
+            Contact::mark_noticed(context, contact_id).await?;
+            context.emit_event(EventType::ContactsChanged(Some(contact_id)));
+        }
 
-            // also unblock mailinglist
-            // if the contact is a mailinglist address explicitly created to allow unblocking
-            if !new_blocking && contact.origin == Origin::MailinglistAddress {
-                if let Ok((chat_id, _, _)) = chat::get_chat_id_by_grpid(context, contact.addr).await
-                {
-                    chat_id.set_blocked(context, Blocked::Not).await;
-                }
+        // also unblock mailinglist
+        // if the contact is a mailinglist address explicitly created to allow unblocking
+        if !new_blocking && contact.origin == Origin::MailinglistAddress {
+            if let Ok((chat_id, _, _)) = chat::get_chat_id_by_grpid(context, contact.addr).await {
+                chat_id.unblock(context).await?;
             }
         }
     }
+
+    Ok(())
 }
 
 /// Set profile image for a contact.
@@ -1400,12 +1385,12 @@ mod tests {
         assert_eq!(may_be_valid_addr("user@domain.tld"), true);
         assert_eq!(may_be_valid_addr("uuu"), false);
         assert_eq!(may_be_valid_addr("dd.tt"), false);
-        assert_eq!(may_be_valid_addr("tt.dd@uu"), false);
-        assert_eq!(may_be_valid_addr("u@d"), false);
-        assert_eq!(may_be_valid_addr("u@d."), false);
-        assert_eq!(may_be_valid_addr("u@d.t"), false);
+        assert_eq!(may_be_valid_addr("tt.dd@uu"), true);
+        assert_eq!(may_be_valid_addr("u@d"), true);
+        assert_eq!(may_be_valid_addr("u@d."), true);
+        assert_eq!(may_be_valid_addr("u@d.t"), true);
         assert_eq!(may_be_valid_addr("u@d.tt"), true);
-        assert_eq!(may_be_valid_addr("u@.tt"), false);
+        assert_eq!(may_be_valid_addr("u@.tt"), true);
         assert_eq!(may_be_valid_addr("@d.tt"), false);
         assert_eq!(may_be_valid_addr("<da@d.tt"), false);
         assert_eq!(may_be_valid_addr("sk <@d.tt>"), false);
@@ -1618,6 +1603,34 @@ mod tests {
         assert!(!contact.is_blocked());
     }
 
+    #[async_std::test]
+    async fn test_delete() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+
+        assert!(Contact::delete(&alice, DC_CONTACT_ID_SELF).await.is_err());
+
+        // Create Bob contact
+        let (contact_id, _) =
+            Contact::add_or_lookup(&alice, "Bob", "bob@example.net", Origin::ManuallyCreated)
+                .await
+                .unwrap();
+
+        let chat = alice
+            .create_chat_with_contact("Bob", "bob@example.net")
+            .await;
+
+        // Can't delete a contact with ongoing chats.
+        assert!(Contact::delete(&alice, contact_id).await.is_err());
+
+        // Delete chat.
+        chat.get_id().delete(&alice).await?;
+
+        // Can delete contact now.
+        Contact::delete(&alice, contact_id).await?;
+
+        Ok(())
+    }
+
     #[async_std::test]
     async fn test_remote_authnames() {
         let t = TestContext::new().await;
@@ -1821,9 +1834,6 @@ mod tests {
         assert!(Contact::create(&t, "", "dskjfdslk@sadklj.dk>")
             .await
             .is_err());
-        assert!(Contact::create(&t, "", "dskjf@dslk@sadkljdk")
-            .await
-            .is_err());
         assert!(Contact::create(&t, "", "dskjf dslk@d.e").await.is_err());
         assert!(Contact::create(&t, "", "<dskjf dslk@sadklj.dk")
             .await

src/context.rs

@@ -161,7 +161,9 @@ impl Context {
 
         {
             let l = &mut *self.inner.scheduler.write().await;
-            l.start(self.clone()).await;
+            if let Err(err) = l.start(self.clone()).await {
+                error!(self, "Failed to start IO: {}", err)
+            }
         }
     }
 
@@ -277,8 +279,8 @@ impl Context {
         let l2 = LoginParam::from_database(self, "configured_").await?;
         let displayname = self.get_config(Config::Displayname).await?;
         let chats = get_chat_cnt(self).await? as usize;
-        let real_msgs = message::get_real_msg_cnt(self).await as usize;
-        let deaddrop_msgs = message::get_deaddrop_msg_cnt(self).await as usize;
+        let unblocked_msgs = message::get_unblocked_msg_cnt(self).await as usize;
+        let request_msgs = message::get_request_msg_cnt(self).await as usize;
         let contacts = Contact::get_real_cnt(self).await? as usize;
         let is_configured = self.get_config_int(Config::Configured).await?;
         let dbversion = self
@@ -334,8 +336,8 @@ impl Context {
         // insert values
         res.insert("bot", self.get_config_int(Config::Bot).await?.to_string());
         res.insert("number_of_chats", chats.to_string());
-        res.insert("number_of_chat_messages", real_msgs.to_string());
-        res.insert("messages_in_contact_requests", deaddrop_msgs.to_string());
+        res.insert("number_of_chat_messages", unblocked_msgs.to_string());
+        res.insert("messages_in_contact_requests", request_msgs.to_string());
         res.insert("number_of_contacts", contacts.to_string());
         res.insert("database_dir", self.get_dbfile().display().to_string());
         res.insert("database_version", dbversion.to_string());
@@ -420,7 +422,7 @@ impl Context {
         Ok(res)
     }
 
-    /// Get a list of fresh, unmuted messages in any chat but deaddrop.
+    /// Get a list of fresh, unmuted messages in unblocked chats.
     ///
     /// The list starts with the most recent message
     /// and is typically used to show notifications.
@@ -538,19 +540,16 @@ impl Context {
 
     pub async fn is_sentbox(&self, folder_name: &str) -> Result<bool> {
         let sentbox = self.get_config(Config::ConfiguredSentboxFolder).await?;
-
         Ok(sentbox.as_deref() == Some(folder_name))
     }
 
     pub async fn is_mvbox(&self, folder_name: &str) -> Result<bool> {
         let mvbox = self.get_config(Config::ConfiguredMvboxFolder).await?;
-
         Ok(mvbox.as_deref() == Some(folder_name))
     }
 
     pub async fn is_spam_folder(&self, folder_name: &str) -> Result<bool> {
         let spam = self.get_config(Config::ConfiguredSpamFolder).await?;
-
         Ok(spam.as_deref() == Some(folder_name))
     }
 
@@ -560,6 +559,13 @@ impl Context {
         blob_fname.push("-blobs");
         dbfile.with_file_name(blob_fname)
     }
+
+    pub fn derive_walfile(dbfile: &PathBuf) -> PathBuf {
+        let mut wal_fname = OsString::new();
+        wal_fname.push(dbfile.file_name().unwrap_or_default());
+        wal_fname.push("-wal");
+        dbfile.with_file_name(wal_fname)
+    }
 }
 
 impl InnerContext {

src/dc_tools.rs

@@ -608,19 +608,8 @@ impl FromStr for EmailAddress {
                 if local.is_empty() {
                     return err("empty string is not valid for local part");
                 }
-                if domain.len() <= 3 {
-                    return err("domain is too short");
-                }
-                let dot = domain.find('.');
-                match dot {
-                    None => {
-                        return err("invalid domain");
-                    }
-                    Some(dot_idx) => {
-                        if dot_idx >= domain.len() - 2 {
-                            return err("invalid domain");
-                        }
-                    }
+                if domain.is_empty() {
+                    return err("missing domain after '@'");
                 }
                 Ok(EmailAddress {
                     local: (*local).to_string(),
@@ -666,7 +655,7 @@ pub fn remove_subject_prefix(last_subject: &str) -> String {
         0
     } else {
         // "Antw:" is the longest abbreviation in
-        // https://en.wikipedia.org/wiki/List_of_email_subject_abbreviations#Abbreviations_in_other_languages,
+        // <https://en.wikipedia.org/wiki/List_of_email_subject_abbreviations#Abbreviations_in_other_languages>,
         // so look at the first _5_ characters:
         match last_subject.chars().take(5).position(|c| c == ':') {
             Some(prefix_end) => prefix_end + 1,
@@ -825,12 +814,19 @@ mod tests {
                 domain: "domain.tld".into(),
             }
         );
+        assert_eq!(
+            "user@localhost".parse::<EmailAddress>().unwrap(),
+            EmailAddress {
+                local: "user".into(),
+                domain: "localhost".into()
+            }
+        );
         assert_eq!("uuu".parse::<EmailAddress>().is_ok(), false);
         assert_eq!("dd.tt".parse::<EmailAddress>().is_ok(), false);
-        assert_eq!("tt.dd@uu".parse::<EmailAddress>().is_ok(), false);
-        assert_eq!("u@d".parse::<EmailAddress>().is_ok(), false);
-        assert_eq!("u@d.".parse::<EmailAddress>().is_ok(), false);
-        assert_eq!("u@d.t".parse::<EmailAddress>().is_ok(), false);
+        assert!("tt.dd@uu".parse::<EmailAddress>().is_ok());
+        assert!("u@d".parse::<EmailAddress>().is_ok());
+        assert!("u@d.".parse::<EmailAddress>().is_ok());
+        assert!("u@d.t".parse::<EmailAddress>().is_ok());
         assert_eq!(
             "u@d.tt".parse::<EmailAddress>().unwrap(),
             EmailAddress {
@@ -838,7 +834,7 @@ mod tests {
                 domain: "d.tt".into(),
             }
         );
-        assert_eq!("u@tt".parse::<EmailAddress>().is_ok(), false);
+        assert!("u@tt".parse::<EmailAddress>().is_ok());
         assert_eq!("@d.tt".parse::<EmailAddress>().is_ok(), false);
     }
 

src/events.rs

@@ -185,21 +185,6 @@ pub enum EventType {
     #[strum(props(id = "400"))]
     Error(String),
 
-    /// An action cannot be performed because there is no network available.
-    ///
-    /// The library will typically try over after a some time
-    /// and when dc_maybe_network() is called.
-    ///
-    /// Network errors should be reported to users in a non-disturbing way,
-    /// however, as network errors may come in a sequence,
-    /// it is not useful to raise each an every error to the user.
-    ///
-    /// Moreover, if the UI detects that the device is offline,
-    /// it is probably more useful to report this to the user
-    /// instead of the string from data2.
-    #[strum(props(id = "401"))]
-    ErrorNetwork(String),
-
     /// An action cannot be performed because the user is not in the group.
     /// Reported eg. after a call to
     /// dc_set_chat_name(), dc_set_chat_profile_image(),
@@ -330,4 +315,11 @@ pub enum EventType {
     ///     (Bob has verified alice and waits until Alice does the same for him)
     #[strum(props(id = "2061"))]
     SecurejoinJoinerProgress { contact_id: u32, progress: usize },
+
+    /// The connectivity to the server changed.
+    /// This means that you should refresh the connectivity view
+    /// and possibly the connectivtiy HTML; see dc_get_connectivity() and
+    /// dc_get_connectivity_html() for details.
+    #[strum(props(id = "2100"))]
+    ConnectivityChanged,
 }

src/export_chat.rs

@@ -1,339 +0,0 @@
-//! Export chats module
-//!
-//! ## Export Format
-//! The format of an exported chat is a zip file with the following structure:
-//! ```text
-//! ├── blobs/          # all files that are referenced by the chat
-//! ├── msg_info/
-//! │   └── [msg_id].txt # message info
-//! ├── msg_source/
-//! │   └── [msg_id].eml # email sourcecode of messages if availible¹
-//! └── chat.json        # chat info, messages and message authors
-//! ```
-//! ##### ¹ Saving Mime header
-//! To save the mime header you need to have the config option [`SaveMimeHeaders`] enabled.
-//! This option saves the mime headers on future messages. Normaly the original email source code is discarded to save space.
-//! You can use the repl tool to do this job:
-//! ```sh
-//! $ cargo run --example repl --features=repl /path/to/account/db.sqlite
-//! > set save_mime_headers 1
-//! ```
-//! [`SaveMimeHeaders`]: ../config/enum.Config.html#variant.SaveMimeHeaders
-
-use crate::chat::*;
-use crate::constants::Viewtype;
-use crate::constants::DC_GCM_ADDDAYMARKER;
-use crate::contact::*;
-use crate::context::Context;
-// use crate::error::Error;
-use crate::dc_tools::time;
-use crate::message::*;
-use std::collections::HashMap;
-use std::fs::File;
-use std::io::prelude::*;
-use std::path::Path;
-use zip::write::FileOptions;
-
-use crate::location::Location;
-use serde::Serialize;
-
-#[derive(Debug)]
-struct ExportChatResult {
-    chat_json: String,
-    // locations_geo_json: String,
-    message_ids: Vec<MsgId>,
-    referenced_blobs: Vec<String>,
-}
-
-pub async fn export_chat_to_zip(context: &Context, chat_id: ChatId, filename: &str) {
-    let res = export_chat_data(&context, chat_id).await;
-    let destination = std::path::Path::new(filename);
-    let pack_res = pack_exported_chat(&context, res, destination).await;
-    match &pack_res {
-        Ok(()) => println!("Exported chat successfully to {}", filename),
-        Err(err) => println!("Error {:?}", err),
-    };
-}
-
-async fn pack_exported_chat(
-    context: &Context,
-    artifact: ExportChatResult,
-    destination: &Path,
-) -> zip::result::ZipResult<()> {
-    let file = std::fs::File::create(&destination).unwrap();
-
-    let mut zip = zip::ZipWriter::new(file);
-
-    zip.start_file("chat.json", Default::default())?;
-    zip.write_all(artifact.chat_json.as_bytes())?;
-
-    zip.add_directory("blobs/", Default::default())?;
-
-    let options = FileOptions::default();
-    for blob_name in artifact.referenced_blobs {
-        let path = context.get_blobdir().join(&blob_name);
-
-        // println!("adding file {:?} as {:?} ...", path, &blob_name);
-        zip.start_file(format!("blobs/{}", &blob_name), options)?;
-        let mut f = File::open(path)?;
-
-        let mut buffer = Vec::new();
-        f.read_to_end(&mut buffer)?;
-        zip.write_all(&*buffer)?;
-        buffer.clear();
-    }
-
-    zip.add_directory("msg_info/", Default::default())?;
-    zip.add_directory("msg_source/", Default::default())?;
-    for id in artifact.message_ids {
-        zip.start_file(format!("msg_info/{}.txt", id.to_u32()), options)?;
-        zip.write_all((get_msg_info(&context, id).await).as_bytes())?;
-        if let Some(mime_headers) = get_mime_headers(&context, id).await {
-            zip.start_file(format!("msg_source/{}.eml", id.to_u32()), options)?;
-            zip.write_all((mime_headers).as_bytes())?;
-        }
-    }
-
-    zip.finish()?;
-    Ok(())
-}
-
-#[derive(Serialize)]
-struct ChatJSON {
-    chat_json_version: u8,
-    export_timestamp: i64,
-    name: String,
-    color: String,
-    profile_img: Option<String>,
-    contacts: HashMap<u32, ContactJSON>,
-    referenced_external_messages:Vec<ChatItemJSON>,
-    messages: Vec<ChatItemJSON>,
-    locations: Vec<Location>,
-}
-
-#[derive(Serialize)]
-struct ContactJSON {
-    name: String,
-    email: String,
-    color: String,
-    profile_img: Option<String>,
-}
-
-#[derive(Serialize)]
-struct FileReference {
-    name: String,
-    filesize: u64,
-    mime: String,
-    path: String,
-}
-
-#[derive(Serialize)]
-struct Qoute {
-    quoted_text: String,
-    message_id: Option<u32>,
-}
-
-#[derive(Serialize)]
-#[serde(tag = "type")]
-enum ChatItemJSON {
-    Message {
-        id: u32,
-        author_id: u32, // from_id
-        view_type: Viewtype,
-        timestamp_sort: i64,
-        timestamp_sent: i64,
-        timestamp_rcvd: i64,
-        text: Option<String>,
-        attachment: Option<FileReference>,
-        location_id: Option<u32>,
-        is_info_message: bool,
-        show_padlock: bool,
-        state: MessageState,
-        is_forwarded: bool,
-        quote: Option<Qoute>
-    },
-    MessageError {
-        id: u32,
-        error: String,
-    },
-    DayMarker {
-        timestamp: i64,
-    },
-}
-
-impl ChatItemJSON {
-    pub async fn from_message(message: &Message, context: &Context) -> ChatItemJSON {
-        let msg_id = message.get_id();
-        ChatItemJSON::Message {
-            id: msg_id.to_u32(),
-            author_id: message.get_from_id(), // from_id
-            view_type: message.get_viewtype(),
-            timestamp_sort: message.timestamp_sort,
-            timestamp_sent: message.timestamp_sent,
-            timestamp_rcvd: message.timestamp_rcvd,
-            text: message.get_text(),
-            attachment: match message.get_file(context) {
-                Some(file) => Some(FileReference {
-                    name: message.get_filename().unwrap_or_else(|| "".to_owned()),
-                    filesize: message.get_filebytes(context).await,
-                    mime: message.get_filemime().unwrap_or_else(|| "".to_owned()),
-                    path: format!(
-                        "blobs/{}",
-                        file.file_name()
-                            .unwrap_or_else(|| std::ffi::OsStr::new(""))
-                            .to_str()
-                            .unwrap()
-                    ),
-                }),
-                None => None,
-            },
-            location_id: match message.has_location() {
-                true => Some(message.location_id),
-                false => None,
-            },
-            is_info_message: message.is_info(),
-            show_padlock: message.get_showpadlock(),
-            state: message.get_state(),
-            is_forwarded: message.is_forwarded(),
-            quote: match message.quoted_text() {
-                Some(text) => match message.quoted_message(&context).await {
-                    Ok(Some(msg)) => Some(Qoute {
-                        quoted_text: text,
-                        message_id: Some(msg.get_id().to_u32())
-                    }),
-                    Err(_) | Ok(None) => Some(Qoute {
-                        quoted_text: text,
-                        message_id: None
-                    })
-                }
-                None => None
-            }
-        }
-    }
-}
-
-async fn export_chat_data(context: &Context, chat_id: ChatId) -> ExportChatResult {
-    let mut blobs = Vec::new();
-    let mut chat_author_ids = Vec::new();
-    // message_ids var is used for writing message info to files
-    let mut message_ids: Vec<MsgId> = Vec::new();
-    let mut message_json: Vec<ChatItemJSON> = Vec::new();
-    let mut referenced_external_messages: Vec<ChatItemJSON> = Vec::new();
-
-    for item in get_chat_msgs(context, chat_id, DC_GCM_ADDDAYMARKER, None).await {
-        if let Some(json_item) = match item {
-            ChatItem::Message { msg_id } => match Message::load_from_db(context, msg_id).await {
-                Ok(message) => {
-                    let filename = message.get_filename();
-                    if let Some(file) = filename {
-                        // push referenced blobs (attachments)
-                        blobs.push(file);
-                    }
-                    message_ids.push(message.id);
-                    // populate contactid list
-                    chat_author_ids.push(message.from_id);
-
-                    if let Ok(Some(ex_msg)) = message.quoted_message(&context).await {
-                        if ex_msg.get_chat_id() != chat_id {
-                            // if external add it to the file
-                            referenced_external_messages.push(ChatItemJSON::from_message(&ex_msg, &context).await)
-                            // contacts don't need to be referenced, because these should only be private replies
-                        }
-                    }
-
-                    Some(ChatItemJSON::from_message(&message, &context).await)
-                }
-                Err(error_message) => Some(ChatItemJSON::MessageError {
-                    id: msg_id.to_u32(),
-                    error: error_message.to_string(),
-                }),
-            },
-            ChatItem::DayMarker { timestamp } => Some(ChatItemJSON::DayMarker { timestamp }),
-            ChatItem::Marker1 => None,
-        } {
-            message_json.push(json_item)
-        }
-    }
-
-    // deduplicate contact list and load the contacts
-    chat_author_ids.sort();
-    chat_author_ids.dedup();
-    // load information about the authors
-    let mut chat_authors: HashMap<u32, ContactJSON> = HashMap::new();
-    chat_authors.insert(
-        0,
-        ContactJSON {
-            name: "Err: Contact not found".to_owned(),
-            email: "error@localhost".to_owned(),
-            profile_img: None,
-            color: "grey".to_owned(),
-        },
-    );
-    for author_id in chat_author_ids {
-        let contact = Contact::get_by_id(context, author_id).await;
-        if let Ok(c) = contact {
-            let profile_img_path: String;
-            if let Some(path) = c.get_profile_image(context).await {
-                profile_img_path = path
-                    .file_name()
-                    .unwrap_or_else(|| std::ffi::OsStr::new(""))
-                    .to_str()
-                    .unwrap()
-                    .to_owned();
-                // push referenced blobs (avatars)
-                blobs.push(profile_img_path.clone());
-            } else {
-                profile_img_path = "".to_owned();
-            }
-            chat_authors.insert(
-                author_id,
-                ContactJSON {
-                    name: c.get_display_name().to_owned(),
-                    email: c.get_addr().to_owned(),
-                    profile_img: match profile_img_path != "" {
-                        true => Some(profile_img_path),
-                        false => None,
-                    },
-                    color: format!("{:#}", c.get_color()), // TODO
-                },
-            );
-        }
-    }
-
-    // Load information about the chat
-    let chat: Chat = Chat::load_from_db(context, chat_id).await.unwrap();
-    let chat_avatar = match chat.get_profile_image(context).await {
-        Some(img) => {
-            let path = img
-                .file_name()
-                .unwrap_or_else(|| std::ffi::OsStr::new(""))
-                .to_str()
-                .unwrap()
-                .to_owned();
-            blobs.push(path.clone());
-            Some(format!("blobs/{}", path))
-        }
-        None => None,
-    };
-
-    let chat_json = ChatJSON {
-        chat_json_version: 1,
-        export_timestamp: time(),
-        name: chat.get_name().to_owned(),
-        color: format!("{:#}", chat.get_color(&context).await),
-        profile_img: chat_avatar,
-        contacts: chat_authors,
-        referenced_external_messages,
-        messages: message_json,
-        locations: crate::location::get_range(&context, chat_id, 0, 0, crate::dc_tools::time())
-            .await,
-    };
-
-    blobs.sort();
-    blobs.dedup();
-    ExportChatResult {
-        chat_json: serde_json::to_string(&chat_json).unwrap(),
-        message_ids,
-        referenced_blobs: blobs,
-    }
-}

src/format_flowed.rs

@@ -104,13 +104,13 @@ pub fn unformat_flowed(text: &str, delsp: bool) -> String {
 
     for line in text.split('\n') {
         // Revert space-stuffing
-        let line = line.strip_prefix(" ").unwrap_or(line);
+        let line = line.strip_prefix(' ').unwrap_or(line);
 
         if !skip_newline {
             result.push('\n');
         }
 
-        if let Some(line) = line.strip_suffix(" ") {
+        if let Some(line) = line.strip_suffix(' ') {
             // Flowed line
             result += line;
             if !delsp {

src/headerdef.rs

@@ -25,6 +25,12 @@ pub enum HeaderDef {
     /// we need to check that header as well.
     XMicrosoftOriginalMessageId,
 
+    /// Thunderbird header used to store Draft information.
+    ///
+    /// Thunderbird 78.11.0 does not set \Draft flag on messages saved as "Template", but sets this
+    /// header, so it can be used to ignore such messages.
+    XMozillaDraftInfo,
+
     ListId,
     References,
     InReplyTo,

src/html.rs

@@ -248,7 +248,7 @@ impl MsgId {
         let rawmime = message::get_mime_headers(context, self).await?;
 
         if !rawmime.is_empty() {
-            match HtmlMsgParser::from_bytes(context, rawmime.as_bytes()).await {
+            match HtmlMsgParser::from_bytes(context, &rawmime).await {
                 Err(err) => {
                     warn!(context, "get_html: parser error: {}", err);
                     Ok(None)
@@ -424,10 +424,10 @@ test some special html-characters as < > and & but also " and &#x
     }
 
     #[async_std::test]
-    async fn test_get_html_empty() {
+    async fn test_get_html_invalid_msgid() {
         let t = TestContext::new().await;
         let msg_id = MsgId::new(100);
-        assert!(msg_id.get_html(&t).await.unwrap().is_none())
+        assert!(msg_id.get_html(&t).await.is_err())
     }
 
     #[async_std::test]
@@ -550,4 +550,26 @@ test some special html-characters as < > and & but also " and &#x
         let html = msg.get_id().get_html(&bob).await.unwrap().unwrap();
         assert!(html.contains("<b>html</b> text"));
     }
+
+    #[async_std::test]
+    async fn test_cp1252_html() -> Result<()> {
+        let t = TestContext::new_alice().await;
+        t.set_config(Config::ShowEmails, Some("2")).await?;
+        dc_receive_imf(
+            &t,
+            include_bytes!("../test-data/message/cp1252-html.eml"),
+            "INBOX",
+            0,
+            false,
+        )
+        .await?;
+        let msg = t.get_last_msg().await;
+        assert_eq!(msg.viewtype, Viewtype::Text);
+        assert!(msg.text.as_ref().unwrap().contains("foo bar ä ö ü ß"));
+        assert!(msg.has_html());
+        let html = msg.get_id().get_html(&t).await?.unwrap();
+        println!("{}", html);
+        assert!(html.contains("foo bar ä ö ü ß"));
+        Ok(())
+    }
 }

src/imap.rs

@@ -8,14 +8,12 @@ use std::{cmp, cmp::max, collections::BTreeMap};
 use anyhow::{bail, format_err, Context as _, Result};
 use async_imap::{
     error::Result as ImapResult,
-    types::{Capability, Fetch, Flag, Mailbox, Name, NameAttribute},
+    types::{Fetch, Flag, Mailbox, Name, NameAttribute, UnsolicitedResponse},
 };
 use async_std::channel::Receiver;
 use async_std::prelude::*;
 use num_traits::FromPrimitive;
 
-use crate::chat;
-use crate::config::Config;
 use crate::constants::{
     Chattype, ShowEmails, Viewtype, DC_FETCH_EXISTING_MSGS_COUNT, DC_FOLDERS_CONFIGURED_VERSION,
     DC_LP_AUTH_OAUTH2,
@@ -36,6 +34,8 @@ use crate::param::Params;
 use crate::provider::Socket;
 use crate::scheduler::InterruptInfo;
 use crate::stock_str;
+use crate::{chat, constants::DC_CONTACT_ID_SELF};
+use crate::{config::Config, scheduler::connectivity::ConnectivityStore};
 
 mod client;
 mod idle;
@@ -92,6 +92,12 @@ pub struct Imap {
     interrupt: Option<stop_token::StopSource>,
     should_reconnect: bool,
     login_failed_once: bool,
+
+    /// True if CAPABILITY command was run successfully once and config.can_* contain correct
+    /// values.
+    capabilities_determined: bool,
+
+    pub(crate) connectivity: ConnectivityStore,
 }
 
 #[derive(Debug)]
@@ -111,14 +117,27 @@ impl async_imap::Authenticator for OAuth2 {
     }
 }
 
-#[derive(Debug, PartialEq)]
+#[derive(Debug, PartialEq, Clone, Copy)]
 enum FolderMeaning {
     Unknown,
     Spam,
-    SentObjects,
+    Sent,
+    Drafts,
     Other,
 }
 
+impl FolderMeaning {
+    fn to_config(self) -> Option<Config> {
+        match self {
+            FolderMeaning::Unknown => None,
+            FolderMeaning::Spam => Some(Config::ConfiguredSpamFolder),
+            FolderMeaning::Sent => Some(Config::ConfiguredSentboxFolder),
+            FolderMeaning::Drafts => None,
+            FolderMeaning::Other => None,
+        }
+    }
+}
+
 #[derive(Debug)]
 struct ImapConfig {
     pub addr: String,
@@ -128,70 +147,106 @@ struct ImapConfig {
     pub selected_folder: Option<String>,
     pub selected_mailbox: Option<Mailbox>,
     pub selected_folder_needs_expunge: bool,
+
     pub can_idle: bool,
 
     /// True if the server has MOVE capability as defined in
-    /// https://tools.ietf.org/html/rfc6851
+    /// <https://tools.ietf.org/html/rfc6851>
     pub can_move: bool,
 }
 
-impl Default for ImapConfig {
-    fn default() -> Self {
-        ImapConfig {
-            addr: "".into(),
-            lp: Default::default(),
-            strict_tls: false,
-            oauth2: false,
+impl Imap {
+    /// Creates new disconnected IMAP client using the specific login parameters.
+    ///
+    /// `addr` is used to renew token if OAuth2 authentication is used.
+    pub async fn new(
+        lp: &ServerLoginParam,
+        addr: &str,
+        oauth2: bool,
+        provider_strict_tls: bool,
+        idle_interrupt: Receiver<InterruptInfo>,
+    ) -> Result<Self> {
+        if lp.server.is_empty() || lp.user.is_empty() || lp.password.is_empty() {
+            bail!("Incomplete IMAP connection parameters");
+        }
+
+        let strict_tls = match lp.certificate_checks {
+            CertificateChecks::Automatic => provider_strict_tls,
+            CertificateChecks::Strict => true,
+            CertificateChecks::AcceptInvalidCertificates
+            | CertificateChecks::AcceptInvalidCertificates2 => false,
+        };
+        let config = ImapConfig {
+            addr: addr.to_string(),
+            lp: lp.clone(),
+            strict_tls,
+            oauth2,
             selected_folder: None,
             selected_mailbox: None,
             selected_folder_needs_expunge: false,
             can_idle: false,
             can_move: false,
-        }
-    }
-}
+        };
 
-impl Imap {
-    pub fn new(idle_interrupt: Receiver<InterruptInfo>) -> Self {
-        Imap {
+        let imap = Imap {
             idle_interrupt,
-            config: Default::default(),
-            session: Default::default(),
-            connected: Default::default(),
-            interrupt: Default::default(),
-            should_reconnect: Default::default(),
-            login_failed_once: Default::default(),
+            config,
+            session: None,
+            connected: false,
+            interrupt: None,
+            should_reconnect: false,
+            login_failed_once: false,
+            connectivity: Default::default(),
+            capabilities_determined: false,
+        };
+
+        Ok(imap)
+    }
+
+    /// Creates new disconnected IMAP client using configured parameters.
+    pub async fn new_configured(
+        context: &Context,
+        idle_interrupt: Receiver<InterruptInfo>,
+    ) -> Result<Self> {
+        if !context.is_configured().await? {
+            bail!("IMAP Connect without configured params");
         }
-    }
 
-    pub fn is_connected(&self) -> bool {
-        self.connected
-    }
+        let param = LoginParam::from_database(context, "configured_").await?;
+        // the trailing underscore is correct
 
-    pub fn should_reconnect(&self) -> bool {
-        self.should_reconnect
-    }
-
-    pub fn trigger_reconnect(&mut self) {
-        self.should_reconnect = true;
+        let imap = Self::new(
+            &param.imap,
+            &param.addr,
+            param.server_flags & DC_LP_AUTH_OAUTH2 != 0,
+            param.provider.map_or(false, |provider| provider.strict_tls),
+            idle_interrupt,
+        )
+        .await?;
+        Ok(imap)
     }
 
     /// Connects or reconnects if needed.
     ///
-    /// It is safe to call this function if already connected, actions
-    /// are performed only as needed.
-    async fn try_setup_handle(&mut self, context: &Context) -> Result<()> {
+    /// It is safe to call this function if already connected, actions are performed only as needed.
+    ///
+    /// Calling this function is not enough to perform IMAP operations. Use [`Imap::prepare`]
+    /// instead if you are going to actually use connection rather than trying connection
+    /// parameters.
+    pub async fn connect(&mut self, context: &Context) -> Result<()> {
         if self.config.lp.server.is_empty() {
             bail!("IMAP operation attempted while it is torn down");
         }
 
         if self.should_reconnect() {
-            self.unsetup_handle(context).await;
+            self.disconnect(context).await;
             self.should_reconnect = false;
         } else if self.is_connected() {
             return Ok(());
         }
 
+        self.connectivity.set_connecting(context).await;
+
         let oauth2 = self.config.oauth2;
 
         let connection_res: ImapResult<Client> = if self.config.lp.security == Socket::Starttls
@@ -256,6 +311,10 @@ impl Imap {
                 self.connected = true;
                 self.session = Some(session);
                 self.login_failed_once = false;
+                emit_event!(
+                    context,
+                    EventType::ImapConnected(format!("IMAP-LOGIN as {}", self.config.lp.user))
+                );
                 Ok(())
             }
 
@@ -286,26 +345,52 @@ impl Imap {
                     self.login_failed_once = true;
                 }
 
-                self.trigger_reconnect();
+                self.trigger_reconnect(context).await;
                 Err(format_err!("{}\n\n{}", message, err))
             }
         }
     }
 
-    /// Connects or reconnects if not already connected.
-    ///
-    /// This function emits network error if it fails.  It should not
-    /// be used during configuration to avoid showing failed attempt
-    /// errors to the user.
-    async fn setup_handle(&mut self, context: &Context) -> Result<()> {
-        let res = self.try_setup_handle(context).await;
-        if let Err(ref err) = res {
-            emit_event!(context, EventType::ErrorNetwork(err.to_string()));
+    /// Determine server capabilities if not done yet.
+    async fn determine_capabilities(&mut self) -> Result<()> {
+        if self.capabilities_determined {
+            return Ok(());
+        }
+
+        match &mut self.session {
+            Some(ref mut session) => match session.capabilities().await {
+                Ok(caps) => {
+                    self.config.can_idle = caps.has_str("IDLE");
+                    self.config.can_move = caps.has_str("MOVE");
+                    self.capabilities_determined = true;
+                    Ok(())
+                }
+                Err(err) => {
+                    bail!("CAPABILITY command error: {}", err);
+                }
+            },
+            None => {
+                bail!("Can't determine server capabilities because connection was not established")
+            }
         }
-        res
     }
 
-    async fn unsetup_handle(&mut self, context: &Context) {
+    /// Prepare for IMAP operation.
+    ///
+    /// Ensure that IMAP client is connected, folders are created and IMAP capabilities are
+    /// determined.
+    pub async fn prepare(&mut self, context: &Context) -> Result<()> {
+        if let Err(err) = self.connect(context).await {
+            self.connectivity.set_err(context, &err).await;
+            return Err(err);
+        }
+
+        self.ensure_configured_folders(context, true).await?;
+        self.determine_capabilities().await?;
+        Ok(())
+    }
+
+    async fn disconnect(&mut self, context: &Context) {
         // Close folder if messages should be expunged
         if let Err(err) = self.close_folder(context).await {
             warn!(context, "failed to close folder: {:?}", err);
@@ -318,139 +403,22 @@ impl Imap {
             }
         }
         self.connected = false;
+        self.capabilities_determined = false;
         self.config.selected_folder = None;
         self.config.selected_mailbox = None;
     }
 
-    async fn free_connect_params(&mut self) {
-        let mut cfg = &mut self.config;
-
-        cfg.addr = "".into();
-        cfg.lp = Default::default();
-
-        cfg.can_idle = false;
-        cfg.can_move = false;
+    pub fn is_connected(&self) -> bool {
+        self.connected
     }
 
-    /// Connects to IMAP account using already-configured parameters.
-    ///
-    /// Emits network error if connection fails.
-    pub async fn connect_configured(&mut self, context: &Context) -> Result<()> {
-        if self.is_connected() && !self.should_reconnect() {
-            return Ok(());
-        }
-        if !context.is_configured().await? {
-            bail!("IMAP Connect without configured params");
-        }
-
-        let param = LoginParam::from_database(context, "configured_").await?;
-        // the trailing underscore is correct
-
-        if let Err(err) = self
-            .connect(
-                context,
-                &param.imap,
-                &param.addr,
-                param.server_flags & DC_LP_AUTH_OAUTH2 != 0,
-                param.provider.map_or(false, |provider| provider.strict_tls),
-            )
-            .await
-        {
-            bail!("IMAP Connection Failed with params {}: {}", param, err);
-        } else {
-            self.ensure_configured_folders(context, true).await
-        }
+    pub fn should_reconnect(&self) -> bool {
+        self.should_reconnect
     }
 
-    /// Tries connecting to imap account using the specific login parameters.
-    ///
-    /// `addr` is used to renew token if OAuth2 authentication is used.
-    ///
-    /// Does not emit network errors, can be used to try various
-    /// parameters during autoconfiguration.
-    pub async fn connect(
-        &mut self,
-        context: &Context,
-        lp: &ServerLoginParam,
-        addr: &str,
-        oauth2: bool,
-        provider_strict_tls: bool,
-    ) -> Result<()> {
-        if lp.server.is_empty() || lp.user.is_empty() || lp.password.is_empty() {
-            bail!("Incomplete IMAP connection parameters");
-        }
-
-        {
-            let mut config = &mut self.config;
-            config.addr = addr.to_string();
-            config.lp = lp.clone();
-            config.strict_tls = match lp.certificate_checks {
-                CertificateChecks::Automatic => provider_strict_tls,
-                CertificateChecks::Strict => true,
-                CertificateChecks::AcceptInvalidCertificates
-                | CertificateChecks::AcceptInvalidCertificates2 => false,
-            };
-            config.oauth2 = oauth2;
-        }
-
-        if let Err(err) = self.try_setup_handle(context).await {
-            warn!(context, "try_setup_handle: {}", err);
-            self.free_connect_params().await;
-            return Err(err);
-        }
-
-        let teardown = match &mut self.session {
-            Some(ref mut session) => match session.capabilities().await {
-                Ok(caps) => {
-                    if !context.sql.is_open().await {
-                        warn!(context, "IMAP-LOGIN as {} ok but ABORTING", lp.user,);
-                        true
-                    } else {
-                        let can_idle = caps.has_str("IDLE");
-                        let can_move = caps.has_str("MOVE");
-                        let caps_list = caps.iter().fold(String::new(), |s, c| {
-                            if let Capability::Atom(x) = c {
-                                s + &format!(" {}", x)
-                            } else {
-                                s + &format!(" {:?}", c)
-                            }
-                        });
-
-                        self.config.can_idle = can_idle;
-                        self.config.can_move = can_move;
-                        self.connected = true;
-                        emit_event!(
-                            context,
-                            EventType::ImapConnected(format!(
-                                "IMAP-LOGIN as {}, capabilities: {}",
-                                lp.user, caps_list,
-                            ))
-                        );
-                        false
-                    }
-                }
-                Err(err) => {
-                    info!(context, "CAPABILITY command error: {}", err);
-                    true
-                }
-            },
-            None => true,
-        };
-
-        if teardown {
-            self.disconnect(context).await;
-
-            warn!(
-                context,
-                "IMAP disconnected immediately after connecting due to error"
-            );
-        }
-        Ok(())
-    }
-
-    pub async fn disconnect(&mut self, context: &Context) {
-        self.unsetup_handle(context).await;
-        self.free_connect_params().await;
+    pub async fn trigger_reconnect(&mut self, context: &Context) {
+        self.connectivity.set_connecting(context).await;
+        self.should_reconnect = true;
     }
 
     pub async fn fetch(&mut self, context: &Context, watch_folder: &str) -> Result<()> {
@@ -458,7 +426,7 @@ impl Imap {
             // probably shutdown
             bail!("IMAP operation attempted while it is torn down");
         }
-        self.setup_handle(context).await?;
+        self.prepare(context).await?;
 
         while self
             .fetch_new_messages(context, &watch_folder, false)
@@ -696,6 +664,7 @@ impl Imap {
                 current_uid,
                 &headers,
                 &msg_id,
+                msg.flags(),
                 folder,
                 show_emails,
             )
@@ -709,6 +678,10 @@ impl Imap {
             }
         }
 
+        if !uids.is_empty() {
+            self.connectivity.set_working(context).await;
+        }
+
         let (largest_uid_processed, error_cnt) = self
             .fetch_many_msgs(context, folder, uids, fetch_existing_msgs)
             .await;
@@ -815,7 +788,7 @@ impl Imap {
         // at least one UID, even if last_seen_uid+1 is past
         // the last UID in the mailbox.  It happens because
         // uid:* is interpreted the same way as *:uid.
-        // See https://tools.ietf.org/html/rfc3501#page-61 for
+        // See <https://tools.ietf.org/html/rfc3501#page-61> for
         // standard reference. Therefore, sometimes we receive
         // already seen messages and have to filter them out.
         let new_msgs = msgs.split_off(&uid_next);
@@ -876,7 +849,7 @@ impl Imap {
 
         if self.session.is_none() {
             // we could not get a valid imap session, this should be retried
-            self.trigger_reconnect();
+            self.trigger_reconnect(context).await;
             warn!(context, "Could not get IMAP session");
             return (None, server_uids.len());
         }
@@ -974,10 +947,6 @@ impl Imap {
         (last_uid, read_errors)
     }
 
-    pub async fn can_move(&self) -> bool {
-        self.config.can_move
-    }
-
     pub async fn mv(
         &mut self,
         context: &Context,
@@ -1002,7 +971,7 @@ impl Imap {
         let set = format!("{}", uid);
         let display_folder_id = format!("{}/{}", folder, uid);
 
-        if self.can_move().await {
+        if self.config.can_move {
             if let Some(ref mut session) = &mut self.session {
                 match session.uid_mv(&set, &dest_folder).await {
                     Ok(_) => {
@@ -1128,7 +1097,7 @@ impl Imap {
             // TODO: make INBOX/SENT/MVBOX perform the jobs on their
             // respective folders to avoid select_folder network traffic
             // and the involved error states
-            if let Err(err) = self.connect_configured(context).await {
+            if let Err(err) = self.prepare(context).await {
                 warn!(context, "prepare_imap_op failed: {}", err);
                 return Some(ImapActionResult::RetryLater);
             }
@@ -1300,9 +1269,8 @@ impl Imap {
 
             let mut delimiter = ".".to_string();
             let mut delimiter_is_default = true;
-            let mut sentbox_folder = None;
-            let mut spam_folder = None;
             let mut mvbox_folder = None;
+            let mut folder_configs = BTreeMap::new();
             let mut fallback_folder = get_fallback_folder(&delimiter);
 
             while let Some(folder) = folders.next().await {
@@ -1321,31 +1289,26 @@ impl Imap {
                 let folder_meaning = get_folder_meaning(&folder);
                 let folder_name_meaning = get_folder_meaning_by_name(folder.name());
                 if folder.name() == "DeltaChat" {
-                    // Always takes precendent
+                    // Always takes precedence
                     mvbox_folder = Some(folder.name().to_string());
                 } else if folder.name() == fallback_folder {
-                    // only set iff none has been already set
+                    // only set if none has been already set
                     if mvbox_folder.is_none() {
                         mvbox_folder = Some(folder.name().to_string());
                     }
-                } else if folder_meaning == FolderMeaning::SentObjects {
-                    // Always takes precedent
-                    sentbox_folder = Some(folder.name().to_string());
-                } else if folder_meaning == FolderMeaning::Spam {
-                    spam_folder = Some(folder.name().to_string());
-                } else if folder_name_meaning == FolderMeaning::SentObjects {
-                    // only set iff none has been already set
-                    if sentbox_folder.is_none() {
-                        sentbox_folder = Some(folder.name().to_string());
-                    }
-                } else if folder_name_meaning == FolderMeaning::Spam && spam_folder.is_none() {
-                    spam_folder = Some(folder.name().to_string());
+                } else if let Some(config) = folder_meaning.to_config() {
+                    // Always takes precedence
+                    folder_configs.insert(config, folder.name().to_string());
+                } else if let Some(config) = folder_name_meaning.to_config() {
+                    // only set if none has been already set
+                    folder_configs
+                        .entry(config)
+                        .or_insert_with(|| folder.name().to_string());
                 }
             }
             drop(folders);
 
             info!(context, "Using \"{}\" as folder-delimiter.", delimiter);
-            info!(context, "sentbox folder is {:?}", sentbox_folder);
 
             if mvbox_folder.is_none() && create_mvbox {
                 info!(context, "Creating MVBOX-folder \"DeltaChat\"...",);
@@ -1393,15 +1356,8 @@ impl Imap {
                     .set_config(Config::ConfiguredMvboxFolder, Some(mvbox_folder))
                     .await?;
             }
-            if let Some(ref sentbox_folder) = sentbox_folder {
-                context
-                    .set_config(Config::ConfiguredSentboxFolder, Some(sentbox_folder))
-                    .await?;
-            }
-            if let Some(ref spam_folder) = spam_folder {
-                context
-                    .set_config(Config::ConfiguredSpamFolder, Some(spam_folder))
-                    .await?;
+            for (config, name) in folder_configs {
+                context.set_config(config, Some(&name)).await?;
             }
             context
                 .sql
@@ -1411,6 +1367,31 @@ impl Imap {
         info!(context, "FINISHED configuring IMAP-folders.");
         Ok(())
     }
+
+    /// Return whether the server sent an unsolicited EXISTS response.
+    /// Drains all responses from `session.unsolicited_responses` in the process.
+    /// If this returns `true`, this means that new emails arrived and you should
+    /// fetch again, even if you just fetched.
+    fn server_sent_unsolicited_exists(&self, context: &Context) -> bool {
+        let session = match &self.session {
+            Some(s) => s,
+            None => return false,
+        };
+        let mut unsolicited_exists = false;
+        while let Ok(response) = session.unsolicited_responses.try_recv() {
+            match response {
+                UnsolicitedResponse::Exists(_) => {
+                    info!(
+                        context,
+                        "Need to fetch again, got unsolicited EXISTS {:?}", response
+                    );
+                    unsolicited_exists = true;
+                }
+                _ => info!(context, "ignoring unsolicited response {:?}", response),
+            }
+        }
+        unsolicited_exists
+    }
 }
 
 /// Try to get the folder meaning by the name of the folder only used if the server does not support XLIST.
@@ -1420,7 +1401,7 @@ impl Imap {
 // CAVE: if possible, take care not to add a name here that is "sent" in one language
 // but sth. different in others - a hard job.
 fn get_folder_meaning_by_name(folder_name: &str) -> FolderMeaning {
-    // source: https://stackoverflow.com/questions/2185391/localized-gmail-imap-folders
+    // source: <https://stackoverflow.com/questions/2185391/localized-gmail-imap-folders>
     const SENT_NAMES: &[&str] = &[
         "sent",
         "sentmail",
@@ -1474,29 +1455,50 @@ fn get_folder_meaning_by_name(folder_name: &str) -> FolderMeaning {
         "迷惑メール",
         "스팸",
     ];
+    const DRAFT_NAMES: &[&str] = &[
+        "Drafts",
+        "Kladder",
+        "Entw?rfe",
+        "Borradores",
+        "Brouillons",
+        "Bozze",
+        "Concepten",
+        "Wersje robocze",
+        "Rascunhos",
+        "Entwürfe",
+        "Koncepty",
+        "Kopie robocze",
+        "Taslaklar",
+        "Utkast",
+        "Πρόχειρα",
+        "Черновики",
+        "下書き",
+        "草稿",
+        "임시보관함",
+    ];
     let lower = folder_name.to_lowercase();
 
     if SENT_NAMES.iter().any(|s| s.to_lowercase() == lower) {
-        FolderMeaning::SentObjects
+        FolderMeaning::Sent
     } else if SPAM_NAMES.iter().any(|s| s.to_lowercase() == lower) {
         FolderMeaning::Spam
+    } else if DRAFT_NAMES.iter().any(|s| s.to_lowercase() == lower) {
+        FolderMeaning::Drafts
     } else {
         FolderMeaning::Unknown
     }
 }
 
 fn get_folder_meaning(folder_name: &Name) -> FolderMeaning {
-    let special_names = vec!["\\Trash", "\\Drafts"];
-
     for attr in folder_name.attributes() {
         if let NameAttribute::Custom(ref label) = attr {
-            if special_names.iter().any(|s| *s == label) {
-                return FolderMeaning::Other;
-            } else if label == "\\Sent" {
-                return FolderMeaning::SentObjects;
-            } else if label == "\\Spam" || label == "\\Junk" {
-                return FolderMeaning::Spam;
-            }
+            match label.as_ref() {
+                "\\Trash" => return FolderMeaning::Other,
+                "\\Sent" => return FolderMeaning::Sent,
+                "\\Spam" | "\\Junk" => return FolderMeaning::Spam,
+                "\\Drafts" => return FolderMeaning::Drafts,
+                _ => {}
+            };
         }
     }
     FolderMeaning::Unknown
@@ -1614,6 +1616,7 @@ fn prefetch_get_message_id(headers: &[mailparse::MailHeader]) -> Result<String>
 pub(crate) async fn prefetch_should_download(
     context: &Context,
     headers: &[mailparse::MailHeader<'_>],
+    mut flags: impl Iterator<Item = Flag<'_>>,
     show_emails: ShowEmails,
 ) -> Result<bool> {
     let is_chat_message = headers.get_header_value(HeaderDef::ChatVersion).is_some();
@@ -1621,7 +1624,7 @@ pub(crate) async fn prefetch_should_download(
     let is_reply_to_chat_message = parent.is_some();
     if let Some(parent) = &parent {
         let chat = chat::Chat::load_from_db(context, parent.get_chat_id()).await?;
-        if chat.typ == Chattype::Group {
+        if chat.typ == Chattype::Group && !chat.id.is_special() {
             // This might be a group command, like removing a group member.
             // We really need to fetch this to avoid inconsistent group state.
             return Ok(true);
@@ -1651,12 +1654,17 @@ pub(crate) async fn prefetch_should_download(
         .get_header_value(HeaderDef::AutocryptSetupMessage)
         .is_some();
 
-    let (_contact_id, blocked_contact, origin) =
+    let (from_id, blocked_contact, origin) =
         from_field_to_contact_id(context, &mimeparser::get_from(headers), true).await?;
     // prevent_rename=true as this might be a mailing list message and in this case it would be bad if we rename the contact.
     // (prevent_rename is the last argument of from_field_to_contact_id())
-    let accepted_contact = origin.is_known();
 
+    if flags.any(|f| f == Flag::Draft) && from_id == DC_CONTACT_ID_SELF {
+        info!(context, "Ignoring draft message");
+        return Ok(false);
+    }
+
+    let accepted_contact = origin.is_known();
     let show = is_autocrypt_setup_message
         || match show_emails {
             ShowEmails::Off => is_chat_message || is_reply_to_chat_message,
@@ -1675,6 +1683,7 @@ async fn message_needs_processing(
     current_uid: u32,
     headers: &[mailparse::MailHeader<'_>],
     msg_id: &str,
+    flags: impl Iterator<Item = Flag<'_>>,
     folder: &str,
     show_emails: ShowEmails,
 ) -> bool {
@@ -1698,7 +1707,7 @@ async fn message_needs_processing(
     // we do not know the message-id
     // or the message-id is missing (in this case, we create one in the further process)
     // or some other error happened
-    let show = match prefetch_should_download(context, headers, show_emails).await {
+    let show = match prefetch_should_download(context, headers, flags, show_emails).await {
         Ok(show) => show,
         Err(err) => {
             warn!(context, "prefetch_should_download error: {}", err);
@@ -1722,7 +1731,7 @@ fn get_fallback_folder(delimiter: &str) -> String {
 }
 
 /// uid_next is the next unique identifier value from the last time we fetched a folder
-/// See https://tools.ietf.org/html/rfc3501#section-2.3.1.1
+/// See <https://tools.ietf.org/html/rfc3501#section-2.3.1.1>
 /// This function is used to update our uid_next after fetching messages.
 pub(crate) async fn set_uid_next(context: &Context, folder: &str, uid_next: u32) -> Result<()> {
     context
@@ -1737,7 +1746,7 @@ pub(crate) async fn set_uid_next(context: &Context, folder: &str, uid_next: u32)
 }
 
 /// uid_next is the next unique identifier value from the last time we fetched a folder
-/// See https://tools.ietf.org/html/rfc3501#section-2.3.1.1
+/// See <https://tools.ietf.org/html/rfc3501#section-2.3.1.1>
 /// This method returns the uid_next from the last time we fetched messages.
 /// We can compare this to the current uid_next to find out whether there are new messages
 /// and fetch from this value on to get all new messages.
@@ -1798,7 +1807,7 @@ pub async fn get_config_last_seen_uid<S: AsRef<str>>(
 }
 
 /// Builds a list of sequence/uid sets. The returned sets have each no more than around 1000
-/// characters because according to https://tools.ietf.org/html/rfc2683#section-3.2.1.5
+/// characters because according to <https://tools.ietf.org/html/rfc2683#section-3.2.1.5>
 /// command lines should not be much more than 1000 chars (servers should allow at least 8000 chars)
 fn build_sequence_sets(mut uids: Vec<u32>) -> Vec<String> {
     uids.sort_unstable();
@@ -1861,25 +1870,16 @@ mod tests {
     use crate::test_utils::TestContext;
     #[test]
     fn test_get_folder_meaning_by_name() {
-        assert_eq!(
-            get_folder_meaning_by_name("Gesendet"),
-            FolderMeaning::SentObjects
-        );
-        assert_eq!(
-            get_folder_meaning_by_name("GESENDET"),
-            FolderMeaning::SentObjects
-        );
-        assert_eq!(
-            get_folder_meaning_by_name("gesendet"),
-            FolderMeaning::SentObjects
-        );
+        assert_eq!(get_folder_meaning_by_name("Gesendet"), FolderMeaning::Sent);
+        assert_eq!(get_folder_meaning_by_name("GESENDET"), FolderMeaning::Sent);
+        assert_eq!(get_folder_meaning_by_name("gesendet"), FolderMeaning::Sent);
         assert_eq!(
             get_folder_meaning_by_name("Messages envoyés"),
-            FolderMeaning::SentObjects
+            FolderMeaning::Sent
         );
         assert_eq!(
             get_folder_meaning_by_name("mEsSaGes envoyÉs"),
-            FolderMeaning::SentObjects
+            FolderMeaning::Sent
         );
         assert_eq!(get_folder_meaning_by_name("xxx"), FolderMeaning::Unknown);
         assert_eq!(get_folder_meaning_by_name("SPAM"), FolderMeaning::Spam);

src/imap/idle.rs

@@ -2,7 +2,6 @@ use super::Imap;
 
 use anyhow::{bail, format_err, Result};
 use async_imap::extensions::idle::IdleResponse;
-use async_imap::types::UnsolicitedResponse;
 use async_std::prelude::*;
 use std::time::{Duration, SystemTime};
 
@@ -25,31 +24,18 @@ impl Imap {
         if !self.can_idle() {
             bail!("IMAP server does not have IDLE capability");
         }
-        self.setup_handle(context).await?;
+        self.prepare(context).await?;
 
         self.select_folder(context, watch_folder.as_deref()).await?;
 
         let timeout = Duration::from_secs(23 * 60);
         let mut info = Default::default();
 
+        if self.server_sent_unsolicited_exists(context) {
+            return Ok(info);
+        }
+
         if let Some(session) = self.session.take() {
-            // if we have unsolicited responses we directly return
-            let mut unsolicited_exists = false;
-            while let Ok(response) = session.unsolicited_responses.try_recv() {
-                match response {
-                    UnsolicitedResponse::Exists(_) => {
-                        warn!(context, "skip idle, got unsolicited EXISTS {:?}", response);
-                        unsolicited_exists = true;
-                    }
-                    _ => info!(context, "ignoring unsolicited response {:?}", response),
-                }
-            }
-
-            if unsolicited_exists {
-                self.session = Some(session);
-                return Ok(info);
-            }
-
             if let Ok(info) = self.idle_interrupt.try_recv() {
                 info!(context, "skip idle, got interrupt {:?}", info);
                 self.session = Some(session);
@@ -158,7 +144,7 @@ impl Imap {
                     // try to connect with proper login params
                     // (setup_handle_if_needed might not know about them if we
                     // never successfully connected)
-                    if let Err(err) = self.connect_configured(context).await {
+                    if let Err(err) = self.prepare(context).await {
                         warn!(context, "fake_idle: could not connect: {}", err);
                         continue;
                     }
@@ -181,7 +167,7 @@ impl Imap {
                         }
                         Err(err) => {
                             error!(context, "could not fetch from folder: {:#}", err);
-                            self.trigger_reconnect()
+                            self.trigger_reconnect(context).await;
                         }
                     }
                 }

src/imap/scan_folders.rs

@@ -1,13 +1,13 @@
-use std::time::Instant;
+use std::{collections::BTreeMap, time::Instant};
 
 use anyhow::{Context as _, Result};
 
-use crate::config::Config;
-use crate::context::Context;
 use crate::imap::Imap;
+use crate::{config::Config, log::LogExt};
+use crate::{context::Context, imap::FolderMeaning};
 use async_std::prelude::*;
 
-use super::{get_folder_meaning, get_folder_meaning_by_name, FolderMeaning};
+use super::{get_folder_meaning, get_folder_meaning_by_name};
 
 impl Imap {
     pub async fn scan_folders(&mut self, context: &Context) -> Result<()> {
@@ -25,14 +25,13 @@ impl Imap {
         }
         info!(context, "Starting full folder scan");
 
-        self.connect_configured(context).await?;
+        self.prepare(context).await?;
         let session = self.session.as_mut();
         let session = session.context("scan_folders(): IMAP No Connection established")?;
         let folders: Vec<_> = session.list(Some(""), Some("*")).await?.collect().await;
         let watched_folders = get_watched_folders(context).await;
 
-        let mut sentbox_folder = None;
-        let mut spam_folder = None;
+        let mut folder_configs = BTreeMap::new();
 
         for folder in folders {
             let folder = match folder {
@@ -43,38 +42,51 @@ impl Imap {
                 }
             };
 
-            let foldername = folder.name();
             let folder_meaning = get_folder_meaning(&folder);
-            let folder_name_meaning = get_folder_meaning_by_name(foldername);
+            let folder_name_meaning = get_folder_meaning_by_name(folder.name());
 
-            if folder_meaning == FolderMeaning::SentObjects {
-                // Always takes precedent
-                sentbox_folder = Some(folder.name().to_string());
-            } else if folder_meaning == FolderMeaning::Spam {
-                spam_folder = Some(folder.name().to_string());
-            } else if folder_name_meaning == FolderMeaning::SentObjects {
-                // only set iff none has been already set
-                if sentbox_folder.is_none() {
-                    sentbox_folder = Some(folder.name().to_string());
-                }
-            } else if folder_name_meaning == FolderMeaning::Spam && spam_folder.is_none() {
-                spam_folder = Some(folder.name().to_string());
+            if let Some(config) = folder_meaning.to_config() {
+                // Always takes precedence
+                folder_configs.insert(config, folder.name().to_string());
+            } else if let Some(config) = folder_name_meaning.to_config() {
+                // only set if none has been already set
+                folder_configs
+                    .entry(config)
+                    .or_insert_with(|| folder.name().to_string());
             }
 
+            let is_drafts = folder_meaning == FolderMeaning::Drafts
+                || (folder_meaning == FolderMeaning::Unknown
+                    && folder_name_meaning == FolderMeaning::Drafts);
+
             // Don't scan folders that are watched anyway
-            if !watched_folders.contains(&foldername.to_string()) {
-                if let Err(e) = self.fetch_new_messages(context, foldername, false).await {
-                    warn!(context, "Can't fetch new msgs in scanned folder: {:#}", e);
+            if !watched_folders.contains(&folder.name().to_string()) && !is_drafts {
+                // Drain leftover unsolicited EXISTS messages
+                self.server_sent_unsolicited_exists(context);
+
+                loop {
+                    self.fetch_new_messages(context, folder.name(), false)
+                        .await
+                        .ok_or_log_msg(context, "Can't fetch new msgs in scanned folder");
+
+                    // If the server sent an unsocicited EXISTS during the fetch, we need to fetch again
+                    if !self.server_sent_unsolicited_exists(context) {
+                        break;
+                    }
                 }
             }
         }
 
-        context
-            .set_config(Config::ConfiguredSentboxFolder, sentbox_folder.as_deref())
-            .await?;
-        context
-            .set_config(Config::ConfiguredSpamFolder, spam_folder.as_deref())
-            .await?;
+        // We iterate over both folder meanings to make sure that if e.g. the "Sent" folder was deleted,
+        // `ConfiguredSentboxFolder` is set to `None`:
+        for config in &[
+            Config::ConfiguredSentboxFolder,
+            Config::ConfiguredSpamFolder,
+        ] {
+            context
+                .set_config(*config, folder_configs.get(config).map(|s| s.as_str()))
+                .await?;
+        }
 
         last_scan.replace(Instant::now());
         Ok(())

src/imap/select_folder.rs

@@ -26,7 +26,7 @@ impl Imap {
     /// Issues a CLOSE command to expunge selected folder.
     ///
     /// CLOSE is considerably faster than an EXPUNGE, see
-    /// https://tools.ietf.org/html/rfc3501#section-6.4.2
+    /// <https://tools.ietf.org/html/rfc3501#section-6.4.2>
     pub(super) async fn close_folder(&mut self, context: &Context) -> Result<()> {
         if let Some(ref folder) = self.config.selected_folder {
             info!(context, "Expunge messages in \"{}\".", folder);
@@ -37,7 +37,7 @@ impl Imap {
                         info!(context, "close/expunge succeeded");
                     }
                     Err(err) => {
-                        self.trigger_reconnect();
+                        self.trigger_reconnect(context).await;
                         return Err(Error::CloseExpungeFailed(err));
                     }
                 }
@@ -70,7 +70,7 @@ impl Imap {
         if self.session.is_none() {
             self.config.selected_folder = None;
             self.config.selected_folder_needs_expunge = false;
-            self.trigger_reconnect();
+            self.trigger_reconnect(context).await;
             return Err(Error::NoSession);
         }
 
@@ -92,7 +92,7 @@ impl Imap {
             if let Some(ref mut session) = &mut self.session {
                 let res = session.select(folder).await;
 
-                // https://tools.ietf.org/html/rfc3501#section-6.3.1
+                // <https://tools.ietf.org/html/rfc3501#section-6.3.1>
                 // says that if the server reports select failure we are in
                 // authenticated (not-select) state.
 
@@ -103,7 +103,7 @@ impl Imap {
                         Ok(NewlySelected::Yes)
                     }
                     Err(async_imap::error::Error::ConnectionLost) => {
-                        self.trigger_reconnect();
+                        self.trigger_reconnect(context).await;
                         self.config.selected_folder = None;
                         Err(Error::ConnectionLost)
                     }
@@ -112,7 +112,7 @@ impl Imap {
                     }
                     Err(err) => {
                         self.config.selected_folder = None;
-                        self.trigger_reconnect();
+                        self.trigger_reconnect(context).await;
                         Err(Error::Other(err.to_string()))
                     }
                 }

src/imex.rs

@@ -184,7 +184,7 @@ pub async fn has_backup_old(context: &Context, dir_name: &Path) -> Result<String
                             "Found backup file {} which could not be opened: {}", name, e
                         );
                         // On some Android devices we can't open sql files that are not in our private directory
-                        // (see https://github.com/deltachat/deltachat-android/issues/1768). So, compare names
+                        // (see <https://github.com/deltachat/deltachat-android/issues/1768>). So, compare names
                         // to still find the newest backup.
                         let name: String = name.into();
                         if newest_backup_time == 0

src/job.rs

@@ -13,9 +13,8 @@ use itertools::Itertools;
 use rand::{thread_rng, Rng};
 
 use crate::blob::BlobObject;
-use crate::chat::{self, Chat, ChatId, ChatIdBlocked, ChatItem};
+use crate::chat::{self, ChatId};
 use crate::config::Config;
-use crate::constants::{Blocked, Chattype, DC_CHAT_ID_DEADDROP};
 use crate::contact::{normalize_name, Contact, Modifier, Origin};
 use crate::context::Context;
 use crate::dc_tools::{dc_delete_file, dc_read_file, time};
@@ -249,10 +248,14 @@ impl Job {
             info!(context, "smtp-sending out mime message:");
             println!("{}", String::from_utf8_lossy(&message));
         }
+
+        smtp.connectivity.set_working(context).await;
+
         let status = match smtp.send(context, recipients, message, job_id).await {
-            Err(crate::smtp::send::Error::SendError(err)) => {
+            Err(crate::smtp::send::Error::SmtpSend(err)) => {
                 // Remote error, retry later.
-                warn!(context, "SMTP failed to send: {:?}", err);
+                warn!(context, "SMTP failed to send: {:?}", &err);
+                smtp.connectivity.set_err(context, &err).await;
                 self.pending_error = Some(err.to_string());
 
                 let res = match err {
@@ -261,13 +264,13 @@ impl Job {
                         // instead of temporary ones.
                         let maybe_transient = match response.code {
                             // Sometimes servers send a permanent error when actually it is a temporary error
-                            // For documentation see https://tools.ietf.org/html/rfc3463
+                            // For documentation see <https://tools.ietf.org/html/rfc3463>
                             Code {
                                 category: Category::MailSystem,
                                 detail: Detail::Zero,
                                 ..
                             } => {
-                                // Ignore status code 5.5.0, see https://support.delta.chat/t/every-other-message-gets-stuck/877/2
+                                // Ignore status code 5.5.0, see <https://support.delta.chat/t/every-other-message-gets-stuck/877/2>
                                 // Maybe incorrectly configured Postfix milter with "reject" instead of "tempfail", which returns
                                 // "550 5.5.0 Service unavailable" instead of "451 4.7.1 Service unavailable - try again later".
                                 //
@@ -301,7 +304,7 @@ impl Job {
                                 // Sometimes we receive transient errors that should be permanent.
                                 // Any extended smtp status codes like x.1.1, x.1.2 or x.1.3 that we
                                 // receive as a transient error are misconfigurations of the smtp server.
-                                // See https://tools.ietf.org/html/rfc3463#section-3.2
+                                // See <https://tools.ietf.org/html/rfc3463#section-3.2>
                                 info!(context, "Smtp-job #{} Received extended status code {} for a transient error. This looks like a misconfigured smtp server, let's fail immediatly", self.job_id, first_word);
                                 Status::Finished(Err(format_err!("Permanent SMTP error: {}", err)))
                             } else {
@@ -326,7 +329,7 @@ impl Job {
 
                 res
             }
-            Err(crate::smtp::send::Error::EnvelopeError(err)) => {
+            Err(crate::smtp::send::Error::Envelope(err)) => {
                 // Local error, job is invalid, do not retry.
                 smtp.disconnect().await;
                 warn!(context, "SMTP job is invalid: {}", err);
@@ -532,7 +535,7 @@ impl Job {
     }
 
     async fn move_msg(&mut self, context: &Context, imap: &mut Imap) -> Status {
-        if let Err(err) = imap.connect_configured(context).await {
+        if let Err(err) = imap.prepare(context).await {
             warn!(context, "could not connect: {:?}", err);
             return Status::RetryLater;
         }
@@ -587,14 +590,15 @@ impl Job {
 
     /// Deletes a message on the server.
     ///
-    /// foreign_id is a MsgId pointing to a message in the trash chat
-    /// or a hidden message.
+    /// `foreign_id` is a MsgId.
     ///
-    /// This job removes the database record. If there are no more
-    /// records pointing to the same message on the server, the job
-    /// also removes the message on the server.
+    /// If the message is in the trash chat or hidden, this job
+    /// removes database record, otherwise it only clears the
+    /// `server_uid` column.  If there are no more records pointing to
+    /// the same message on the server, the job actually removes the
+    /// message on the server.
     async fn delete_msg_on_imap(&mut self, context: &Context, imap: &mut Imap) -> Status {
-        if let Err(err) = imap.connect_configured(context).await {
+        if let Err(err) = imap.prepare(context).await {
             warn!(context, "could not connect: {:?}", err);
             return Status::RetryLater;
         }
@@ -682,7 +686,7 @@ impl Job {
         if job_try!(context.get_config_bool(Config::Bot).await) {
             return Status::Finished(Ok(())); // Bots don't want those messages
         }
-        if let Err(err) = imap.connect_configured(context).await {
+        if let Err(err) = imap.prepare(context).await {
             warn!(context, "could not connect: {:?}", err);
             return Status::RetryLater;
         }
@@ -707,40 +711,6 @@ impl Job {
             }
         }
 
-        // Make sure that if there now is a chat with a contact (created by an outgoing
-        // message), then group contact requests from this contact should also be unblocked.
-        // See https://github.com/deltachat/deltachat-core-rust/issues/2097.
-        for item in job_try!(chat::get_chat_msgs(context, DC_CHAT_ID_DEADDROP, 0, None).await) {
-            if let ChatItem::Message { msg_id } = item {
-                let msg = match Message::load_from_db(context, msg_id).await {
-                    Err(e) => {
-                        warn!(context, "can't get msg: {:#}", e);
-                        return Status::RetryLater;
-                    }
-                    Ok(m) => m,
-                };
-                let chat = match Chat::load_from_db(context, msg.chat_id).await {
-                    Err(e) => {
-                        warn!(context, "can't get chat: {:#}", e);
-                        return Status::RetryLater;
-                    }
-                    Ok(c) => c,
-                };
-                match chat.typ {
-                    Chattype::Group | Chattype::Mailinglist => {
-                        if let Ok(Some(one_to_one_chat)) =
-                            ChatIdBlocked::lookup_by_contact(context, msg.from_id).await
-                        {
-                            if one_to_one_chat.blocked == Blocked::Not {
-                                chat.id.unblock(context).await;
-                            }
-                        }
-                    }
-                    Chattype::Single | Chattype::Undefined => {}
-                }
-            }
-        }
-
         info!(context, "Done fetching existing messages.");
         Status::Finished(Ok(()))
     }
@@ -755,7 +725,7 @@ impl Job {
     /// Chat in contrast to the Sent folder, which is normally managed
     /// by the user via webmail or another email client.
     async fn resync_folders(&mut self, context: &Context, imap: &mut Imap) -> Status {
-        if let Err(err) = imap.connect_configured(context).await {
+        if let Err(err) = imap.prepare(context).await {
             warn!(context, "could not connect: {:?}", err);
             return Status::RetryLater;
         }
@@ -779,7 +749,7 @@ impl Job {
     }
 
     async fn markseen_msg_on_imap(&mut self, context: &Context, imap: &mut Imap) -> Status {
-        if let Err(err) = imap.connect_configured(context).await {
+        if let Err(err) = imap.prepare(context).await {
             warn!(context, "could not connect: {:?}", err);
             return Status::RetryLater;
         }
@@ -1052,16 +1022,9 @@ pub(crate) enum Connection<'a> {
 }
 
 pub(crate) async fn load_imap_deletion_job(context: &Context) -> Result<Option<Job>> {
-    let res = if let Some(msg_id) = load_imap_deletion_msgid(context).await? {
-        Some(Job::new(
-            Action::DeleteMsgOnImap,
-            msg_id.to_u32(),
-            Params::new(),
-            0,
-        ))
-    } else {
-        None
-    };
+    let res = load_imap_deletion_msgid(context)
+        .await?
+        .map(|msg_id| Job::new(Action::DeleteMsgOnImap, msg_id.to_u32(), Params::new(), 0));
     Ok(res)
 }
 

src/key.rs

@@ -4,46 +4,22 @@ use std::collections::BTreeMap;
 use std::fmt;
 use std::io::Cursor;
 
+use anyhow::{format_err, Result};
 use async_trait::async_trait;
 use num_traits::FromPrimitive;
 use pgp::composed::Deserializable;
 use pgp::ser::Serialize;
 use pgp::types::{KeyTrait, SecretKeyTrait};
-use thiserror::Error;
 
 use crate::config::Config;
 use crate::constants::KeyGenType;
 use crate::context::Context;
-use crate::dc_tools::{time, EmailAddress, InvalidEmailError};
+use crate::dc_tools::{time, EmailAddress};
 
 // Re-export key types
 pub use crate::pgp::KeyPair;
 pub use pgp::composed::{SignedPublicKey, SignedSecretKey};
 
-/// Error type for deltachat key handling.
-#[derive(Debug, thiserror::Error)]
-#[non_exhaustive]
-pub enum Error {
-    #[error("Could not decode base64")]
-    Base64Decode(#[from] base64::DecodeError),
-    #[error("rPGP error: {}", _0)]
-    Pgp(#[from] pgp::errors::Error),
-    #[error("Failed to generate PGP key: {}", _0)]
-    Keygen(#[from] crate::pgp::PgpKeygenError),
-    #[error("Failed to save generated key: {}", _0)]
-    StoreKey(#[from] SaveKeyError),
-    #[error("No address configured")]
-    NoConfiguredAddr,
-    #[error("Configured address is invalid: {}", _0)]
-    InvalidConfiguredAddr(#[from] InvalidEmailError),
-    #[error("no data provided")]
-    Empty,
-    #[error("{0}")]
-    Other(#[from] anyhow::Error),
-}
-
-pub type Result<T> = std::result::Result<T, Error>;
-
 /// Convenience trait for working with keys.
 ///
 /// This trait is implemented for rPGP's [SignedPublicKey] and
@@ -74,7 +50,8 @@ pub trait DcKey: Serialize + Deserializable + KeyTrait + Clone {
     /// the ASCII-armored representation.
     fn from_asc(data: &str) -> Result<(Self::KeyType, BTreeMap<String, String>)> {
         let bytes = data.as_bytes();
-        Self::KeyType::from_armor_single(Cursor::new(bytes)).map_err(Error::Pgp)
+        Self::KeyType::from_armor_single(Cursor::new(bytes))
+            .map_err(|err| format_err!("rPGP error: {}", err))
     }
 
     /// Load the users' default key from the database.
@@ -225,7 +202,7 @@ async fn generate_keypair(context: &Context) -> Result<KeyPair> {
     let addr = context
         .get_config(Config::ConfiguredAddr)
         .await?
-        .ok_or(Error::NoConfiguredAddr)?;
+        .ok_or_else(|| format_err!("No address configured"))?;
     let addr = EmailAddress::new(&addr)?;
     let _guard = context.generating_key_mutex.lock().await;
 
@@ -284,24 +261,6 @@ pub enum KeyPairUse {
     ReadOnly,
 }
 
-/// Error saving a keypair to the database.
-#[derive(Debug, thiserror::Error)]
-#[error("SaveKeyError: {message}")]
-pub struct SaveKeyError {
-    message: String,
-    #[source]
-    cause: anyhow::Error,
-}
-
-impl SaveKeyError {
-    fn new(message: impl Into<String>, cause: impl Into<anyhow::Error>) -> Self {
-        Self {
-            message: message.into(),
-            cause: cause.into(),
-        }
-    }
-}
-
 /// Store the keypair as an owned keypair for addr in the database.
 ///
 /// This will save the keypair as keys for the given address.  The
@@ -318,7 +277,7 @@ pub async fn store_self_keypair(
     context: &Context,
     keypair: &KeyPair,
     default: KeyPairUse,
-) -> std::result::Result<(), SaveKeyError> {
+) -> Result<()> {
     // Everything should really be one transaction, more refactoring
     // is needed for that.
     let public_key = DcKey::to_bytes(&keypair.public);
@@ -330,13 +289,13 @@ pub async fn store_self_keypair(
             paramsv![public_key, secret_key],
         )
         .await
-        .map_err(|err| SaveKeyError::new("failed to remove old use of key", err))?;
+        .map_err(|err| err.context("failed to remove old use of key"))?;
     if default == KeyPairUse::Default {
         context
             .sql
             .execute("UPDATE keypairs SET is_default=0;", paramsv![])
             .await
-            .map_err(|err| SaveKeyError::new("failed to clear default", err))?;
+            .map_err(|err| err.context("failed to clear default"))?;
     }
     let is_default = match default {
         KeyPairUse::Default => true as i32,
@@ -354,7 +313,7 @@ pub async fn store_self_keypair(
             paramsv![addr, is_default, public_key, secret_key, t],
         )
         .await
-        .map_err(|err| SaveKeyError::new("failed to insert keypair", err))?;
+        .map_err(|err| err.context("failed to insert keypair"))?;
 
     Ok(())
 }
@@ -364,10 +323,10 @@ pub async fn store_self_keypair(
 pub struct Fingerprint(Vec<u8>);
 
 impl Fingerprint {
-    pub fn new(v: Vec<u8>) -> std::result::Result<Fingerprint, FingerprintError> {
+    pub fn new(v: Vec<u8>) -> Result<Fingerprint> {
         match v.len() {
             20 => Ok(Fingerprint(v)),
-            _ => Err(FingerprintError::WrongLength),
+            _ => Err(format_err!("Wrong fingerprint length")),
         }
     }
 
@@ -406,7 +365,7 @@ impl fmt::Display for Fingerprint {
 
 /// Parse a human-readable or otherwise formatted fingerprint.
 impl std::str::FromStr for Fingerprint {
-    type Err = FingerprintError;
+    type Err = anyhow::Error;
 
     fn from_str(input: &str) -> std::result::Result<Self, Self::Err> {
         let hex_repr: String = input
@@ -420,21 +379,11 @@ impl std::str::FromStr for Fingerprint {
     }
 }
 
-#[derive(Debug, Error)]
-pub enum FingerprintError {
-    #[error("Invalid hex characters")]
-    NotHex(#[from] hex::FromHexError),
-    #[error("Incorrect fingerprint lengths")]
-    WrongLength,
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
     use crate::test_utils::{alice_keypair, TestContext};
 
-    use std::error::Error;
-
     use async_std::sync::Arc;
     use once_cell::sync::Lazy;
 
@@ -676,13 +625,7 @@ i8pcjGO+IZffvyZJVRWfVooBJmWWbPB1pueo3tx8w3+fcuzpxz+RLFKaPyqXO+dD
             .unwrap();
         assert_eq!(fp, res);
 
-        let err = "1".parse::<Fingerprint>().err().unwrap();
-        match err {
-            FingerprintError::NotHex(_) => (),
-            _ => panic!("Wrong error"),
-        }
-        let src_err = err.source().unwrap().downcast_ref::<hex::FromHexError>();
-        assert_eq!(src_err, Some(&hex::FromHexError::OddLength));
+        assert!("1".parse::<Fingerprint>().is_err());
     }
 
     #[test]

src/keyring.rs

@@ -3,7 +3,7 @@
 use anyhow::Result;
 
 use crate::context::Context;
-use crate::key::{self, DcKey};
+use crate::key::DcKey;
 
 /// An in-memory keyring.
 ///
@@ -27,14 +27,14 @@ where
     }
 
     /// Create a new keyring with the the user's secret key loaded.
-    pub async fn new_self(context: &Context) -> Result<Keyring<T>, key::Error> {
+    pub async fn new_self(context: &Context) -> Result<Keyring<T>> {
         let mut keyring: Keyring<T> = Keyring::new();
         keyring.load_self(context).await?;
         Ok(keyring)
     }
 
     /// Load the user's key into the keyring.
-    pub async fn load_self(&mut self, context: &Context) -> Result<(), key::Error> {
+    pub async fn load_self(&mut self, context: &Context) -> Result<()> {
         self.add(T::load_self(context).await?);
         Ok(())
     }

src/lib.rs

@@ -51,7 +51,6 @@ pub mod contact;
 pub mod context;
 mod e2ee;
 pub mod ephemeral;
-pub mod export_chat;
 mod imap;
 pub mod imex;
 mod scheduler;

src/location.rs

@@ -16,8 +16,9 @@ use crate::message::{Message, MsgId};
 use crate::mimeparser::SystemMessage;
 use crate::param::Params;
 use crate::stock_str;
-use serde::Serialize;
-#[derive(Debug, Clone, Default, Serialize)]
+
+/// Location record
+#[derive(Debug, Clone, Default)]
 pub struct Location {
     pub location_id: u32,
     pub latitude: f64,

src/log.rs

@@ -42,17 +42,6 @@ macro_rules! error {
     }};
 }
 
-#[macro_export]
-macro_rules! error_network {
-    ($ctx:expr, $msg:expr) => {
-        error_network!($ctx, $msg,)
-    };
-    ($ctx:expr, $msg:expr, $($args:expr),* $(,)?) => {{
-        let formatted = format!($msg, $($args),*);
-        emit_event!($ctx, $crate::EventType::ErrorNetwork(formatted));
-    }};
-}
-
 #[macro_export]
 macro_rules! emit_event {
     ($ctx:expr, $event:expr) => {
@@ -76,7 +65,7 @@ where
     /// Once it is, you can add `#[track_caller]` to helper functions that use one of the log helpers here
     /// so that the location of the caller can be seen in the log. (this won't work with the macros,
     /// like warn!(), since the file!() and line!() macros don't work with track_caller)  
-    /// See https://github.com/rust-lang/rust/issues/78840 for progress on this.
+    /// See <https://github.com/rust-lang/rust/issues/78840> for progress on this.
     #[track_caller]
     fn log_err(self, context: &Context, msg: &str) -> Result<T, E> {
         self.log_err_inner(context, Some(msg))

src/lot.rs

@@ -110,6 +110,16 @@ pub enum LotState {
     /// text1=error string
     QrError = 400,
 
+    QrWithdrawVerifyContact = 500,
+
+    /// text1=groupname
+    QrWithdrawVerifyGroup = 502,
+
+    QrReviveVerifyContact = 510,
+
+    /// text1=groupname
+    QrReviveVerifyGroup = 512,
+
     // Message States
     MsgInFresh = 10,
     MsgInNoticed = 13,

src/message.rs

@@ -7,14 +7,14 @@ use anyhow::{ensure, format_err, Result};
 use async_std::path::{Path, PathBuf};
 use deltachat_derive::{FromSql, ToSql};
 use itertools::Itertools;
+use rusqlite::types::ValueRef;
 use serde::{Deserialize, Serialize};
 
 use crate::chat::{self, Chat, ChatId};
 use crate::config::Config;
 use crate::constants::{
-    Blocked, Chattype, VideochatType, Viewtype, DC_CHAT_ID_DEADDROP, DC_CHAT_ID_TRASH,
-    DC_CONTACT_ID_INFO, DC_CONTACT_ID_LAST_SPECIAL, DC_CONTACT_ID_SELF, DC_MAX_GET_INFO_LEN,
-    DC_MAX_GET_TEXT_LEN, DC_MSG_ID_LAST_SPECIAL,
+    Blocked, Chattype, VideochatType, Viewtype, DC_CHAT_ID_TRASH, DC_CONTACT_ID_INFO,
+    DC_CONTACT_ID_SELF, DC_MAX_GET_INFO_LEN, DC_MAX_GET_TEXT_LEN, DC_MSG_ID_LAST_SPECIAL,
 };
 use crate::contact::{Contact, Origin};
 use crate::context::Context;
@@ -27,7 +27,7 @@ use crate::events::EventType;
 use crate::job::{self, Action};
 use crate::log::LogExt;
 use crate::lot::{Lot, LotState, Meaning};
-use crate::mimeparser::{FailureReport, SystemMessage};
+use crate::mimeparser::{parse_message_id, FailureReport, SystemMessage};
 use crate::param::{Param, Params};
 use crate::pgp::split_armored_data;
 use crate::stock_str;
@@ -108,7 +108,7 @@ impl MsgId {
                     Ok(Some(ConfiguredInboxFolder))
                 }
             } else {
-                // Blocked/deaddrop message in the spam folder, leave it there
+                // Blocked or contact request message in the spam folder, leave it there
                 Ok(None)
             };
         }
@@ -235,9 +235,9 @@ impl std::fmt::Display for MsgId {
 impl rusqlite::types::ToSql for MsgId {
     fn to_sql(&self) -> rusqlite::Result<rusqlite::types::ToSqlOutput> {
         if self.0 <= DC_MSG_ID_LAST_SPECIAL {
-            return Err(rusqlite::Error::ToSqlConversionFailure(Box::new(
-                InvalidMsgId,
-            )));
+            return Err(rusqlite::Error::ToSqlConversionFailure(
+                format_err!("Invalid MsgId").into(),
+            ));
         }
         let val = rusqlite::types::Value::Integer(self.0 as i64);
         let out = rusqlite::types::ToSqlOutput::Owned(val);
@@ -259,15 +259,6 @@ impl rusqlite::types::FromSql for MsgId {
     }
 }
 
-/// Message ID was invalid.
-///
-/// This usually occurs when trying to use a message ID of
-/// [DC_MSG_ID_LAST_SPECIAL] or below in a situation where this is not
-/// possible.
-#[derive(Debug, thiserror::Error)]
-#[error("Invalid Message ID.")]
-pub struct InvalidMsgId;
-
 #[derive(
     Debug,
     Copy,
@@ -400,7 +391,9 @@ impl Message {
                     let msg = Message {
                         id: row.get("id")?,
                         rfc724_mid: row.get::<_, String>("rfc724mid")?,
-                        in_reply_to: row.get::<_, Option<String>>("mime_in_reply_to")?,
+                        in_reply_to: row
+                            .get::<_, Option<String>>("mime_in_reply_to")?
+                            .and_then(|in_reply_to| parse_message_id(&in_reply_to).ok()),
                         server_folder: row.get::<_, Option<String>>("server_folder")?,
                         server_uid: row.get("server_uid")?,
                         chat_id: row.get("chat_id")?,
@@ -526,19 +519,8 @@ impl Message {
         self.from_id
     }
 
-    /// get the chat-id,
-    /// if the message is a contact request, the DC_CHAT_ID_DEADDROP is returned.
+    /// Returns the chat ID.
     pub fn get_chat_id(&self) -> ChatId {
-        if self.chat_blocked != Blocked::Not {
-            DC_CHAT_ID_DEADDROP
-        } else {
-            self.chat_id
-        }
-    }
-
-    /// get the chat-id, also when the message is still a contact request.
-    /// DC_CHAT_ID_DEADDROP is never returned.
-    pub fn get_real_chat_id(&self) -> ChatId {
         self.chat_id
     }
 
@@ -599,6 +581,11 @@ impl Message {
         self.param.get_int(Param::GuaranteeE2ee).unwrap_or_default() != 0
     }
 
+    /// Returns true if message is Auto-Submitted.
+    pub fn is_bot(&self) -> bool {
+        self.param.get_bool(Param::Bot).unwrap_or_default()
+    }
+
     pub fn get_ephemeral_timer(&self) -> EphemeralTimer {
         self.ephemeral_timer
     }
@@ -755,7 +742,7 @@ impl Message {
         } else if url.contains("$NOROOM") {
             // there are some usecases where a separate room is not needed to use a service
             // eg. if you let in people manually anyway, see discussion at
-            // https://support.delta.chat/t/videochat-with-webex/1412/4 .
+            // <https://support.delta.chat/t/videochat-with-webex/1412/4>.
             // hacks as hiding the room behind `#` are not reliable, therefore,
             // these services are supported by adding the string `$NOROOM` to the url.
             url.replace("$NOROOM", "")
@@ -960,13 +947,6 @@ impl Message {
     }
 }
 
-#[derive(Display, Debug, FromPrimitive)]
-pub enum ContactRequestDecision {
-    StartChat = 0,
-    Block = 1,
-    NotNow = 2,
-}
-
 #[derive(
     Debug,
     Clone,
@@ -1148,76 +1128,6 @@ impl Lot {
     }
 }
 
-/// Call this when the user decided about a deaddrop message ("Do you want to chat with NAME?").
-///
-/// If the decision is `StartChat`, this will create a new chat and return the chat id.
-/// If the decision is `Block`, this will usually block the sender.
-/// If the decision is `NotNow`, this will usually mark all messages from this sender as read.
-///
-/// If the message belongs to a mailing list, makes sure that all messages from this mailing list are
-/// blocked or marked as noticed.
-///
-/// The user should be asked whether they want to chat with the _contact_ belonging to the message;
-/// the group names may be really weird when taken from the subject of implicit (= ad-hoc)
-/// groups and this may look confusing. Moreover, this function also scales up the origin of the contact.
-///
-/// If the chat belongs to a mailing list, you can also ask
-/// "Would you like to read MAILING LIST NAME in Delta Chat?"
-/// (use `Message.get_real_chat_id()` to get the chat-id for the contact request
-/// and then `Chat.is_mailing_list()`, `Chat.get_name()` and so on)
-pub async fn decide_on_contact_request(
-    context: &Context,
-    msg_id: MsgId,
-    decision: ContactRequestDecision,
-) -> Option<ChatId> {
-    let msg = match Message::load_from_db(context, msg_id).await {
-        Ok(m) => m,
-        Err(e) => {
-            warn!(context, "Can't load message: {}", e);
-            return None;
-        }
-    };
-
-    let chat = match Chat::load_from_db(context, msg.chat_id).await {
-        Ok(c) => c,
-        Err(e) => {
-            warn!(context, "Can't load chat: {}", e);
-            return None;
-        }
-    };
-
-    let mut created_chat_id = None;
-    use ContactRequestDecision::*;
-    match (decision, chat.is_mailing_list()) {
-        (StartChat, _) => match chat::create_by_msg_id(context, msg.id).await {
-            Ok(id) => created_chat_id = Some(id),
-            Err(e) => warn!(context, "decide_on_contact_request error: {}", e),
-        },
-
-        (Block, false) => Contact::block(context, msg.from_id).await,
-        (Block, true) => {
-            if !msg.chat_id.set_blocked(context, Blocked::Manually).await {
-                warn!(context, "Block mailing list failed.")
-            }
-        }
-
-        (NotNow, false) => Contact::mark_noticed(context, msg.from_id).await,
-        (NotNow, true) => {
-            if let Err(e) = chat::marknoticed_chat(context, msg.chat_id).await {
-                warn!(context, "Marknoticed failed: {}", e)
-            }
-        }
-    }
-
-    // Multiple chats may have changed, so send 0s
-    // (performance is not so important because this function is not called very often)
-    context.emit_event(EventType::MsgsChanged {
-        chat_id: ChatId::new(0),
-        msg_id: MsgId::new(0),
-    });
-    created_chat_id
-}
-
 pub async fn get_msg_info(context: &Context, msg_id: MsgId) -> Result<String> {
     let msg = Message::load_from_db(context, msg_id).await?;
     let rawtxt: Option<String> = context
@@ -1362,7 +1272,7 @@ pub fn guess_msgtype_from_suffix(path: &Path) -> Option<(Viewtype, &str)> {
         // before using viewtype other than Viewtype::File,
         // make sure, all target UIs support that type in the context of the used viewer/player.
         // if in doubt, it is better to default to Viewtype::File that passes handing to an external app.
-        // (cmp. https://developer.android.com/guide/topics/media/media-formats )
+        // (cmp. <https://developer.android.com/guide/topics/media/media-formats>)
         "3gp" => (Viewtype::Video, "video/3gpp"),
         "aac" => (Viewtype::Audio, "audio/aac"),
         "avi" => (Viewtype::Video, "video/x-msvideo"),
@@ -1436,18 +1346,25 @@ pub fn guess_msgtype_from_suffix(path: &Path) -> Option<(Viewtype, &str)> {
 /// only if `dc_set_config(context, "save_mime_headers", "1")`
 /// was called before.
 ///
-/// Returns an empty string if there are no headers saved for the given message,
+/// Returns an empty vector if there are no headers saved for the given message,
 /// e.g. because of save_mime_headers is not set
 /// or the message is not incoming.
-pub async fn get_mime_headers(context: &Context, msg_id: MsgId) -> Result<String> {
+pub async fn get_mime_headers(context: &Context, msg_id: MsgId) -> Result<Vec<u8>> {
     let headers = context
         .sql
-        .query_get_value(
+        .query_row(
             "SELECT mime_headers FROM msgs WHERE id=?;",
             paramsv![msg_id],
+            |row| {
+                row.get(0).or_else(|err| match row.get_ref(0)? {
+                    ValueRef::Null => Ok(Vec::new()),
+                    ValueRef::Text(text) => Ok(text.to_vec()),
+                    ValueRef::Blob(blob) => Ok(blob.to_vec()),
+                    ValueRef::Integer(_) | ValueRef::Real(_) => Err(err),
+                })
+            },
         )
-        .await?
-        .unwrap_or_default();
+        .await?;
     Ok(headers)
 }
 
@@ -1499,33 +1416,37 @@ pub async fn markseen_msgs(context: &Context, msg_ids: Vec<MsgId>) -> Result<()>
     }
 
     let conn = context.sql.get_conn().await?;
-    let mut stmt = conn.prepare_cached(concat!(
-        "SELECT",
-        "    m.chat_id AS chat_id,",
-        "    m.state AS state,",
-        "    c.blocked AS blocked",
-        " FROM msgs m LEFT JOIN chats c ON c.id=m.chat_id",
-        " WHERE m.id=? AND m.chat_id>9"
-    ))?;
+    let msgs = async_std::task::spawn_blocking(move || -> Result<_> {
+        let mut stmt = conn.prepare_cached(concat!(
+            "SELECT",
+            "    m.chat_id AS chat_id,",
+            "    m.state AS state,",
+            "    c.blocked AS blocked",
+            " FROM msgs m LEFT JOIN chats c ON c.id=m.chat_id",
+            " WHERE m.id=? AND m.chat_id>9"
+        ))?;
 
-    let mut msgs = Vec::with_capacity(msg_ids.len());
-    for id in msg_ids.into_iter() {
-        let query_res = stmt.query_row(paramsv![id], |row| {
-            Ok((
-                row.get::<_, ChatId>("chat_id")?,
-                row.get::<_, MessageState>("state")?,
-                row.get::<_, Option<Blocked>>("blocked")?
-                    .unwrap_or_default(),
-            ))
-        });
-        if let Err(rusqlite::Error::QueryReturnedNoRows) = query_res {
-            continue;
+        let mut msgs = Vec::with_capacity(msg_ids.len());
+        for id in msg_ids.into_iter() {
+            let query_res = stmt.query_row(paramsv![id], |row| {
+                Ok((
+                    row.get::<_, ChatId>("chat_id")?,
+                    row.get::<_, MessageState>("state")?,
+                    row.get::<_, Option<Blocked>>("blocked")?
+                        .unwrap_or_default(),
+                ))
+            });
+            if let Err(rusqlite::Error::QueryReturnedNoRows) = query_res {
+                continue;
+            }
+            let (chat_id, state, blocked) = query_res.map_err(Into::<anyhow::Error>::into)?;
+            msgs.push((id, chat_id, state, blocked));
         }
-        let (chat_id, state, blocked) = query_res.map_err(Into::<anyhow::Error>::into)?;
-        msgs.push((id, chat_id, state, blocked));
-    }
-    drop(stmt);
-    drop(conn);
+        drop(stmt);
+        drop(conn);
+        Ok(msgs)
+    })
+    .await?;
 
     let mut updated_chat_ids = BTreeMap::new();
 
@@ -1716,13 +1637,9 @@ pub async fn handle_mdn(
     rfc724_mid: &str,
     timestamp_sent: i64,
 ) -> Result<Option<(ChatId, MsgId)>> {
-    if from_id <= DC_CONTACT_ID_LAST_SPECIAL || rfc724_mid.is_empty() {
-        return Ok(None);
-    }
-
     let res = context
         .sql
-        .query_row(
+        .query_row_optional(
             concat!(
                 "SELECT",
                 "    m.id AS msg_id,",
@@ -1743,75 +1660,80 @@ pub async fn handle_mdn(
                 ))
             },
         )
-        .await;
-    if let Err(ref err) = res {
-        info!(context, "Failed to select MDN {:?}", err);
-    }
+        .await?;
 
-    if let Ok((msg_id, chat_id, chat_type, msg_state)) = res {
-        let mut read_by_all = false;
+    let (msg_id, chat_id, chat_type, msg_state) = if let Some(res) = res {
+        res
+    } else {
+        info!(
+            context,
+            "handle_mdn found no message with Message-ID {:?} sent by us in the database",
+            rfc724_mid
+        );
+        return Ok(None);
+    };
 
-        if msg_state == MessageState::OutPreparing
-            || msg_state == MessageState::OutPending
-            || msg_state == MessageState::OutDelivered
-        {
-            let mdn_already_in_table = context
+    let mut read_by_all = false;
+    if msg_state == MessageState::OutPreparing
+        || msg_state == MessageState::OutPending
+        || msg_state == MessageState::OutDelivered
+    {
+        let mdn_already_in_table = context
+            .sql
+            .exists(
+                "SELECT COUNT(*) FROM msgs_mdns WHERE msg_id=? AND contact_id=?;",
+                paramsv![msg_id, from_id as i32,],
+            )
+            .await?;
+
+        if !mdn_already_in_table {
+            context
                 .sql
-                .exists(
-                    "SELECT COUNT(*) FROM msgs_mdns WHERE msg_id=? AND contact_id=?;",
-                    paramsv![msg_id, from_id as i32,],
-                )
-                .await
-                .unwrap_or_default();
-
-            if !mdn_already_in_table {
-                context.sql.execute(
+                .execute(
                     "INSERT INTO msgs_mdns (msg_id, contact_id, timestamp_sent) VALUES (?, ?, ?);",
                     paramsv![msg_id, from_id as i32, timestamp_sent],
                 )
-                    .await
-                           .unwrap_or_default(); // TODO: better error handling
-            }
+                .await?;
+        }
 
-            // Normal chat? that's quite easy.
-            if chat_type == Chattype::Single {
+        // Normal chat? that's quite easy.
+        if chat_type == Chattype::Single {
+            update_msg_state(context, msg_id, MessageState::OutMdnRcvd).await;
+            read_by_all = true;
+        } else {
+            // send event about new state
+            let ist_cnt = context
+                .sql
+                .count(
+                    "SELECT COUNT(*) FROM msgs_mdns WHERE msg_id=?;",
+                    paramsv![msg_id],
+                )
+                .await?;
+
+            // Groupsize:  Min. MDNs
+            // 1 S         n/a
+            // 2 SR        1
+            // 3 SRR       2
+            // 4 SRRR      2
+            // 5 SRRRR     3
+            // 6 SRRRRR    3
+            //
+            // (S=Sender, R=Recipient)
+
+            // for rounding, SELF is already included!
+            let soll_cnt = (chat::get_chat_contact_cnt(context, chat_id).await? + 1) / 2;
+            if ist_cnt >= soll_cnt {
                 update_msg_state(context, msg_id, MessageState::OutMdnRcvd).await;
                 read_by_all = true;
-            } else {
-                // send event about new state
-                let ist_cnt = context
-                    .sql
-                    .count(
-                        "SELECT COUNT(*) FROM msgs_mdns WHERE msg_id=?;",
-                        paramsv![msg_id],
-                    )
-                    .await?;
-
-                // Groupsize:  Min. MDNs
-                // 1 S         n/a
-                // 2 SR        1
-                // 3 SRR       2
-                // 4 SRRR      2
-                // 5 SRRRR     3
-                // 6 SRRRRR    3
-                //
-                // (S=Sender, R=Recipient)
-
-                // for rounding, SELF is already included!
-                let soll_cnt = (chat::get_chat_contact_cnt(context, chat_id).await? + 1) / 2;
-                if ist_cnt >= soll_cnt {
-                    update_msg_state(context, msg_id, MessageState::OutMdnRcvd).await;
-                    read_by_all = true;
-                } // else wait for more receipts
-            }
+            } // else wait for more receipts
         }
-        return if read_by_all {
-            Ok(Some((chat_id, msg_id)))
-        } else {
-            Ok(None)
-        };
     }
-    Ok(None)
+
+    if read_by_all {
+        Ok(Some((chat_id, msg_id)))
+    } else {
+        Ok(None)
+    }
 }
 
 /// Marks a message as failed after an ndn (non-delivery-notification) arrived.
@@ -1897,8 +1819,8 @@ async fn ndn_maybe_add_info_msg(
     Ok(())
 }
 
-/// The number of messages assigned to real chat (!=deaddrop, !=trash)
-pub async fn get_real_msg_cnt(context: &Context) -> usize {
+/// The number of messages assigned to unblocked chats
+pub async fn get_unblocked_msg_cnt(context: &Context) -> usize {
     match context
         .sql
         .count(
@@ -1911,13 +1833,14 @@ pub async fn get_real_msg_cnt(context: &Context) -> usize {
     {
         Ok(res) => res,
         Err(err) => {
-            error!(context, "dc_get_real_msg_cnt() failed. {}", err);
+            error!(context, "dc_get_unblocked_msg_cnt() failed. {}", err);
             0
         }
     }
 }
 
-pub async fn get_deaddrop_msg_cnt(context: &Context) -> usize {
+/// Returns the number of messages in contact request chats.
+pub async fn get_request_msg_cnt(context: &Context) -> usize {
     match context
         .sql
         .count(
@@ -1930,7 +1853,7 @@ pub async fn get_deaddrop_msg_cnt(context: &Context) -> usize {
     {
         Ok(res) => res,
         Err(err) => {
-            error!(context, "dc_get_deaddrop_msg_cnt() failed. {}", err);
+            error!(context, "dc_get_request_msg_cnt() failed. {}", err);
             0
         }
     }
@@ -2089,7 +2012,7 @@ mod tests {
     ];
 
     // These are the same as above, but all messages in Spam stay in Spam
-    const COMBINATIONS_DEADDROP: &[(&str, bool, bool, &str)] = &[
+    const COMBINATIONS_REQUEST: &[(&str, bool, bool, &str)] = &[
         ("INBOX", false, false, "INBOX"),
         ("INBOX", false, true, "INBOX"),
         ("INBOX", true, false, "INBOX"),
@@ -2122,8 +2045,8 @@ mod tests {
     }
 
     #[async_std::test]
-    async fn test_needs_move_incoming_deaddrop() {
-        for (folder, mvbox_move, chat_msg, expected_destination) in COMBINATIONS_DEADDROP {
+    async fn test_needs_move_incoming_request() {
+        for (folder, mvbox_move, chat_msg, expected_destination) in COMBINATIONS_REQUEST {
             check_needs_move_combination(
                 folder,
                 *mvbox_move,
@@ -2672,10 +2595,7 @@ mod tests {
 
         // check chat-id of this message
         let msg = alice.get_last_msg().await;
-        assert!(msg.get_chat_id().is_deaddrop());
-        assert!(msg.get_chat_id().is_special());
-        assert!(!msg.get_real_chat_id().is_deaddrop());
-        assert!(!msg.get_real_chat_id().is_special());
+        assert!(!msg.get_chat_id().is_special());
         assert_eq!(msg.get_text().unwrap(), "hello".to_string());
     }
 
@@ -2736,33 +2656,31 @@ mod tests {
         msg.set_text(Some("this is the text!".to_string()));
 
         // alice sends to bob,
-        // bob does not know alice yet and messages go to bob's deaddrop
         assert_eq!(Chatlist::try_load(&bob, 0, None, None).await?.len(), 0);
         bob.recv_msg(&alice.send_msg(alice_chat.id, &mut msg).await)
             .await;
         let msg1 = bob.get_last_msg().await;
+        let bob_chat_id = msg1.chat_id;
         bob.recv_msg(&alice.send_msg(alice_chat.id, &mut msg).await)
             .await;
         let msg2 = bob.get_last_msg().await;
         assert_eq!(msg1.chat_id, msg2.chat_id);
-        assert_ne!(msg1.chat_id, DC_CHAT_ID_DEADDROP);
         let chats = Chatlist::try_load(&bob, 0, None, None).await?;
         assert_eq!(chats.len(), 1);
-        assert_eq!(chats.get_chat_id(0), DC_CHAT_ID_DEADDROP);
-        let msgs = chat::get_chat_msgs(&bob, DC_CHAT_ID_DEADDROP, 0, None).await?;
+        let msgs = chat::get_chat_msgs(&bob, bob_chat_id, 0, None).await?;
         assert_eq!(msgs.len(), 2);
         assert_eq!(bob.get_fresh_msgs().await?.len(), 0);
 
-        // that has no effect in deaddrop
+        // that has no effect in contact request
         markseen_msgs(&bob, vec![msg1.id, msg2.id]).await?;
 
         assert_eq!(Chatlist::try_load(&bob, 0, None, None).await?.len(), 1);
-        let msgs = chat::get_chat_msgs(&bob, DC_CHAT_ID_DEADDROP, 0, None).await?;
+        let bob_chat = Chat::load_from_db(&bob, bob_chat_id).await?;
+        assert_eq!(bob_chat.blocked, Blocked::Request);
+
+        let msgs = chat::get_chat_msgs(&bob, bob_chat_id, 0, None).await?;
         assert_eq!(msgs.len(), 2);
-        let bob_chat_id =
-            decide_on_contact_request(&bob, msg2.get_id(), ContactRequestDecision::StartChat)
-                .await
-                .unwrap();
+        bob_chat_id.accept(&bob).await.unwrap();
 
         // bob sends to alice,
         // alice knows bob and messages appear in normal chat
@@ -2862,4 +2780,50 @@ mod tests {
 
         Ok(())
     }
+
+    #[async_std::test]
+    async fn test_is_bot() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+
+        // Alice receives a message from Bob the bot.
+        dc_receive_imf(
+            &alice,
+            b"From: Bob <bob@example.com>\n\
+                    To: alice@example.com\n\
+                    Chat-Version: 1.0\n\
+                    Message-ID: <123@example.com>\n\
+                    Auto-Submitted: auto-generated\n\
+                    Date: Fri, 29 Jan 2021 21:37:55 +0000\n\
+                    \n\
+                    hello\n",
+            "INBOX",
+            1,
+            false,
+        )
+        .await?;
+        let msg = alice.get_last_msg().await;
+        assert_eq!(msg.get_text().unwrap(), "hello".to_string());
+        assert!(msg.is_bot());
+
+        // Alice receives a message from Bob who is not the bot anymore.
+        dc_receive_imf(
+            &alice,
+            b"From: Bob <bob@example.com>\n\
+                    To: alice@example.com\n\
+                    Chat-Version: 1.0\n\
+                    Message-ID: <456@example.com>\n\
+                    Date: Fri, 29 Jan 2021 21:37:55 +0000\n\
+                    \n\
+                    hello again\n",
+            "INBOX",
+            2,
+            false,
+        )
+        .await?;
+        let msg = alice.get_last_msg().await;
+        assert_eq!(msg.get_text().unwrap(), "hello again".to_string());
+        assert!(!msg.is_bot());
+
+        Ok(())
+    }
 }

src/mimefactory.rs

@@ -491,6 +491,11 @@ impl<'a> MimeFactory<'a> {
                 "Auto-Submitted".to_string(),
                 "auto-replied".to_string(),
             ));
+        } else if context.get_config_bool(Config::Bot).await? {
+            headers.unprotected.push(Header::new(
+                "Auto-Submitted".to_string(),
+                "auto-generated".to_string(),
+            ));
         }
 
         if self.req_mdn {
@@ -860,7 +865,7 @@ impl<'a> MimeFactory<'a> {
                 // This should prevent automatic replies,
                 // such as non-delivery reports.
                 //
-                // See https://tools.ietf.org/html/rfc3834
+                // See <https://tools.ietf.org/html/rfc3834>
                 //
                 // Adding this header without encryption leaks some
                 // information about the message contents, but it can
@@ -1395,7 +1400,7 @@ mod tests {
             Address::new_mailbox_with_name(display_name.to_string(), addr.to_string())
         );
 
-        // Addresses should not be unnecessarily be encoded, see https://github.com/deltachat/deltachat-core-rust/issues/1575:
+        // Addresses should not be unnecessarily be encoded, see <https://github.com/deltachat/deltachat-core-rust/issues/1575>:
         assert_eq!(s, "a space <x@y.org>");
     }
 
@@ -1801,9 +1806,8 @@ mod tests {
 
         let chats = Chatlist::try_load(context, 0, None, None).await.unwrap();
 
-        let chat_id = chat::create_by_msg_id(context, chats.get_msg_id(0).unwrap())
-            .await
-            .unwrap();
+        let chat_id = chats.get_chat_id(0);
+        chat_id.accept(context).await.unwrap();
 
         let mut new_msg = Message::new(Viewtype::Text);
         new_msg.set_text(Some("Hi".to_string()));
@@ -1859,7 +1863,7 @@ mod tests {
 
     #[test]
     fn test_no_empty_lines_in_header() {
-        // See https://github.com/deltachat/deltachat-core-rust/issues/2118
+        // See <https://github.com/deltachat/deltachat-core-rust/issues/2118>
         let to_tuples = [
             ("Nnnn", "nnn@ttttttttt.de"),
             ("😀 ttttttt", "ttttttt@rrrrrr.net"),

src/mimeparser.rs

@@ -3,12 +3,10 @@ use std::future::Future;
 use std::pin::Pin;
 
 use anyhow::{bail, Result};
-use charset::Charset;
 use deltachat_derive::{FromSql, ToSql};
 use lettre_email::mime::{self, Mime};
 use mailparse::{addrparse_header, DispositionType, MailHeader, MailHeaderMap, SingleInfo};
 use once_cell::sync::Lazy;
-use percent_encoding::percent_decode_str;
 
 use crate::aheader::Aheader;
 use crate::blob::BlobObject;
@@ -198,7 +196,7 @@ impl MimeMessage {
                         }
 
                         // Handle any gossip headers if the mail was encrypted.  See section
-                        // "3.6 Key Gossip" of https://autocrypt.org/autocrypt-spec-1.1.0.pdf
+                        // "3.6 Key Gossip" of <https://autocrypt.org/autocrypt-spec-1.1.0.pdf>
                         // but only if the mail was correctly signed:
                         if !signatures.is_empty() {
                             let gossip_headers =
@@ -220,7 +218,7 @@ impl MimeMessage {
                         let mut throwaway_from = from.clone();
 
                         // We do not want to allow unencrypted subject in encrypted emails because the user might falsely think that the subject is safe.
-                        // See https://github.com/deltachat/deltachat-core-rust/issues/1790.
+                        // See <https://github.com/deltachat/deltachat-core-rust/issues/1790>.
                         headers.remove("subject");
 
                         MimeMessage::merge_headers(
@@ -398,7 +396,7 @@ impl MimeMessage {
             if part.typ == Viewtype::Audio && self.get(HeaderDef::ChatVoiceMessage).is_some() {
                 part.typ = Viewtype::Voice;
             }
-            if part.typ == Viewtype::Image {
+            if part.typ == Viewtype::Image || part.typ == Viewtype::Gif {
                 if let Some(value) = self.get(HeaderDef::ChatContent) {
                     if value == "sticker" {
                         part.typ = Viewtype::Sticker;
@@ -498,6 +496,12 @@ impl MimeMessage {
 
             self.parts.push(part);
         }
+
+        if self.header.contains_key("auto-submitted") {
+            for part in &mut self.parts {
+                part.param.set(Param::Bot, "1");
+            }
+        }
     }
 
     async fn avatar_action_from_header(
@@ -731,7 +735,7 @@ impl MimeMessage {
                 The second body part contains the control information necessary to
                 verify the digital signature." We simply take the first body part and
                 skip the rest.  (see
-                https://k9mail.github.io/2016/11/24/OpenPGP-Considerations-Part-I.html
+                <https://k9mail.github.io/2016/11/24/OpenPGP-Considerations-Part-I.html>
                 for background information why we use encrypted+signed) */
                 if let Some(first) = mail.subparts.get(0) {
                     any_part_added = self
@@ -1463,7 +1467,7 @@ fn get_mime_type(mail: &mailparse::ParsedMail<'_>) -> Result<(Mime, Viewtype)> {
         mime::VIDEO => Viewtype::Video,
         mime::MULTIPART => Viewtype::Unknown,
         mime::MESSAGE => {
-            // Enacapsulated messages, see https://www.w3.org/Protocols/rfc1341/7_3_Message.html
+            // Enacapsulated messages, see <https://www.w3.org/Protocols/rfc1341/7_3_Message.html>
             // Also used as part "message/disposition-notification" of "multipart/report", which, however, will
             // be handled separatedly.
             // I've not seen any messages using this, so we do not attach these parts (maybe they're used to attach replies,
@@ -1504,55 +1508,13 @@ fn get_attachment_filename(
     // `Content-Disposition: ... filename=...`
     let mut desired_filename = ct.params.get("filename").map(|s| s.to_string());
 
-    // try to get file name from
-    // `Content-Disposition: ... filename*0*=... filename*1*=... filename*2*=...`
-    // encoded as CHARSET'LANG'test%2E%70%64%66 (key ends with `*`)
-    // or as "encoded-words" (key does not end with `*`)
     if desired_filename.is_none() {
-        let mut apostrophe_encoded = false;
-        desired_filename = ct
-            .params
-            .iter()
-            .filter(|(key, _value)| key.starts_with("filename*"))
-            .fold(None, |acc, (key, value)| {
-                if key.ends_with('*') {
-                    apostrophe_encoded = true;
-                }
-                if let Some(acc) = acc {
-                    Some(acc + value)
-                } else {
-                    Some(value.to_string())
-                }
-            });
-        if apostrophe_encoded {
-            if let Some(name) = desired_filename {
-                let mut parts = name.splitn(3, '\'');
-                desired_filename =
-                    if let (Some(charset), Some(value)) = (parts.next(), parts.last()) {
-                        let decoded_bytes = percent_decode_str(value);
-                        if charset.to_lowercase() == "utf-8" {
-                            Some(decoded_bytes.decode_utf8_lossy().to_string())
-                        } else {
-                            // encoded_words crate say, latin-1 is not reported; moreover, latin1 is a good default
-                            if let Some(charset) = Charset::for_label(charset.as_bytes())
-                                .or_else(|| Charset::for_label(b"latin1"))
-                            {
-                                let decoded_bytes = decoded_bytes.collect::<Vec<u8>>();
-                                let (utf8_str, _, _) = charset.decode(&*decoded_bytes);
-                                Some(utf8_str.into())
-                            } else {
-                                warn!(context, "latin1 encoding does not exist");
-                                None
-                            }
-                        }
-                    } else {
-                        warn!(context, "apostrophed encoding invalid: {}", name);
-                        // be graceful and just use the original name.
-                        // some MUA, including Delta Chat up to core1.50,
-                        // use `filename*` mistakenly for simple encoded-words without following rfc2231
-                        Some(name)
-                    }
-            }
+        if let Some(name) = ct.params.get("filename*").map(|s| s.to_string()) {
+            // be graceful and just use the original name.
+            // some MUA, including Delta Chat up to core1.50,
+            // use `filename*` mistakenly for simple encoded-words without following rfc2231
+            warn!(context, "apostrophed encoding invalid: {}", name);
+            desired_filename = Some(name);
         }
     }
 
@@ -1914,13 +1876,6 @@ mod tests {
         assert_eq!(filename, Some("Maßnahmen Okt. 2020.html".to_string()))
     }
 
-    #[test]
-    fn test_charset_latin1() {
-        // make sure, latin1 exists under this name
-        // as we're using it as default in get_attachment_filename() for non-utf-8
-        assert!(Charset::for_label(b"latin1").is_some());
-    }
-
     #[test]
     fn test_mailparse_content_type() {
         let ctype =
@@ -2828,7 +2783,7 @@ On 2020-10-25, Bob wrote:
         .unwrap();
 
         let chats = Chatlist::try_load(&t.ctx, 0, None, None).await.unwrap();
-        let msg_id = chats.get_msg_id(0).unwrap();
+        let msg_id = chats.get_msg_id(0).unwrap().unwrap();
         let msg = Message::load_from_db(&t.ctx, msg_id).await.unwrap();
 
         assert_eq!(
@@ -2838,7 +2793,7 @@ On 2020-10-25, Bob wrote:
         assert_eq!(msg.viewtype, Viewtype::Image);
         assert_eq!(msg.error(), None);
         assert_eq!(msg.is_dc_message, MessengerMessage::No);
-        assert_eq!(msg.chat_blocked, Blocked::Deaddrop);
+        assert_eq!(msg.chat_blocked, Blocked::Request);
         assert_eq!(msg.state, MessageState::InFresh);
         assert_eq!(msg.get_filebytes(&t).await, 2115);
         assert!(msg.get_file(&t).is_some());
@@ -2955,4 +2910,46 @@ On 2020-10-25, Bob wrote:
             Some("Mr.6Dx7ITn4w38.n9j7epIcuQI@outlook.com".to_string())
         );
     }
+
+    #[async_std::test]
+    async fn test_long_in_reply_to() -> Result<()> {
+        let t = TestContext::new_alice().await;
+
+        // A message with a long Message-ID.
+        // Long message-IDs are generated by Mailjet.
+        let raw = br###"Date: Thu, 28 Jan 2021 00:26:57 +0000
+Chat-Version: 1.0\n\
+Message-ID: <ABCDEFGH.1234567_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA@mailjet.com>
+To: Bob <bob@example.org>
+From: Alice <alice@example.org>
+Subject: ...
+
+Some quote.
+"###;
+        dc_receive_imf(&t, raw, "INBOX", 1, false).await?;
+
+        // Delta Chat generates In-Reply-To with a starting tab when Message-ID is too long.
+        let raw = br###"In-Reply-To:
+	<ABCDEFGH.1234567_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA@mailjet.com>
+Date: Thu, 28 Jan 2021 00:26:57 +0000
+Chat-Version: 1.0\n\
+Message-ID: <foobar@example.org>
+To: Alice <alice@example.org>
+From: Bob <bob@example.org>
+Subject: ...
+
+> Some quote.
+
+Some reply
+"###;
+
+        dc_receive_imf(&t, raw, "INBOX", 2, false).await?;
+
+        let msg = t.get_last_msg().await;
+        assert_eq!(msg.get_text().unwrap(), "Some reply");
+        let quoted_message = msg.quoted_message(&t).await?.unwrap();
+        assert_eq!(quoted_message.get_text().unwrap(), "Some quote.");
+
+        Ok(())
+    }
 }

src/oauth2.rs

@@ -12,7 +12,7 @@ use crate::provider;
 use crate::provider::Oauth2Authorizer;
 
 const OAUTH2_GMAIL: Oauth2 = Oauth2 {
-    // see https://developers.google.com/identity/protocols/OAuth2InstalledApp
+    // see <https://developers.google.com/identity/protocols/OAuth2InstalledApp>
     client_id: "959970109878-4mvtgf6feshskf7695nfln6002mom908.apps.googleusercontent.com",
     get_code: "https://accounts.google.com/o/oauth2/auth?client_id=$CLIENT_ID&redirect_uri=$REDIRECT_URI&response_type=code&scope=https%3A%2F%2Fmail.google.com%2F%20email&access_type=offline",
     init_token: "https://accounts.google.com/o/oauth2/token?client_id=$CLIENT_ID&redirect_uri=$REDIRECT_URI&code=$CODE&grant_type=authorization_code",
@@ -21,7 +21,7 @@ const OAUTH2_GMAIL: Oauth2 = Oauth2 {
 };
 
 const OAUTH2_YANDEX: Oauth2 = Oauth2 {
-    // see https://tech.yandex.com/oauth/doc/dg/reference/auto-code-client-docpage/
+    // see <https://tech.yandex.com/oauth/doc/dg/reference/auto-code-client-docpage/>
     client_id: "c4d0b6735fc8420a816d7e1303469341",
     get_code: "https://oauth.yandex.com/authorize?client_id=$CLIENT_ID&response_type=code&scope=mail%3Aimap_full%20mail%3Asmtp&force_confirm=true",
     init_token: "https://oauth.yandex.com/token?grant_type=authorization_code&code=$CODE&client_id=$CLIENT_ID&client_secret=58b8c6e94cf44fbe952da8511955dacf",
@@ -41,7 +41,7 @@ struct Oauth2 {
 /// OAuth 2 Access Token Response
 #[derive(Debug, Deserialize)]
 struct Response {
-    // Should always be there according to: https://www.oauth.com/oauth2-servers/access-tokens/access-token-response/
+    // Should always be there according to: <https://www.oauth.com/oauth2-servers/access-tokens/access-token-response/>
     // but previous code handled its abscense.
     access_token: Option<String>,
     token_type: String,
@@ -129,7 +129,7 @@ pub async fn dc_get_oauth2_access_token(
             };
 
         // to allow easier specification of different configurations,
-        // token_url is in GET-method-format, sth. as https://domain?param1=val1&param2=val2 -
+        // token_url is in GET-method-format, sth. as <https://domain?param1=val1&param2=val2> -
         // convert this to POST-format ...
         let mut parts = token_url.splitn(2, '?');
         let post_url = parts.next().unwrap_or_default();

src/param.rs

@@ -60,6 +60,9 @@ pub enum Param {
     /// For Messages
     WantsMdn = b'r',
 
+    /// For Messages: a message with Auto-Submitted header ("bot").
+    Bot = b'b',
+
     /// For Messages: unset or 0=not forwarded,
     /// 1=forwarded from unknown msg_id, >9 forwarded from msg_id
     Forwarded = b'a',

src/peerstate.rs

@@ -267,10 +267,9 @@ impl Peerstate {
                 .query_get_value("SELECT id FROM contacts WHERE addr=?;", paramsv![self.addr])
                 .await?
             {
-                let chat_id =
-                    ChatIdBlocked::get_for_contact(context, contact_id, Blocked::Deaddrop)
-                        .await?
-                        .id;
+                let chat_id = ChatIdBlocked::get_for_contact(context, contact_id, Blocked::Request)
+                    .await?
+                    .id;
 
                 let msg = stock_str::contact_setup_changed(context, self.addr.clone()).await;
 
@@ -493,12 +492,6 @@ impl Peerstate {
     }
 }
 
-impl From<crate::key::FingerprintError> for rusqlite::Error {
-    fn from(_source: crate::key::FingerprintError) -> Self {
-        Self::InvalidColumnType(0, "Invalid fingerprint".into(), rusqlite::types::Type::Text)
-    }
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;

src/pgp.rs

@@ -86,7 +86,7 @@ impl<'a> PublicKeyTrait for SignedPublicKeyOrSubkey<'a> {
     }
 }
 
-/// Split data from PGP Armored Data as defined in https://tools.ietf.org/html/rfc4880#section-6.2.
+/// Split data from PGP Armored Data as defined in <https://tools.ietf.org/html/rfc4880#section-6.2>.
 ///
 /// Returns (type, headers, base64 encoded body).
 pub fn split_armored_data(buf: &[u8]) -> Result<(BlockType, BTreeMap<String, String>, Vec<u8>)> {

src/plaintext.rs

@@ -64,7 +64,7 @@ impl PlainText {
                 // flowed text as of RFC 3676 -
                 // a leading space shall be removed
                 // and is only there to allow > at the beginning of a line that is no quote.
-                line = line.strip_prefix(" ").unwrap_or(&line).to_string();
+                line = line.strip_prefix(' ').unwrap_or(&line).to_string();
                 if is_quote {
                     line = "<em>".to_owned() + &line + "</em>";
                 }

src/provider/data.rs

@@ -8,6 +8,23 @@ use std::collections::HashMap;
 
 use once_cell::sync::Lazy;
 
+// 163.md: 163.com
+static P_163: Lazy<Provider> = Lazy::new(|| {
+    Provider {
+    id: "163",
+    status: Status::Broken,
+    before_login_hint: "163 Mail does not work since it forces the email clients to connect with an IMAP ID, which is currently not the case of Delta Chat.",
+    after_login_hint: "",
+    overview_page: "https://providers.delta.chat/163",
+    server: vec![
+    ],
+    config_defaults: None,
+    strict_tls: true,
+    max_smtp_rcpt_to: None,
+    oauth2_authorizer: None,
+}
+});
+
 // aktivix.org.md: aktivix.org
 static P_AKTIVIX_ORG: Lazy<Provider> = Lazy::new(|| Provider {
     id: "aktivix.org",
@@ -597,8 +614,7 @@ static P_I3_NET: Lazy<Provider> = Lazy::new(|| Provider {
 static P_ICLOUD: Lazy<Provider> = Lazy::new(|| Provider {
     id: "icloud",
     status: Status::Preparation,
-    before_login_hint:
-        "You must create an app-specific password for Delta Chat before you can login.",
+    before_login_hint: "You must create an app-specific password for Delta Chat before login.",
     after_login_hint: "",
     overview_page: "https://providers.delta.chat/icloud",
     server: vec![
@@ -651,7 +667,7 @@ static P_KONTENT_COM: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
-// mail.ru.md: mail.ru, inbox.ru, bk.ru, list.ru
+// mail.ru.md: mail.ru, inbox.ru, internet.ru, bk.ru, list.ru
 static P_MAIL_RU: Lazy<Provider> = Lazy::new(|| {
     Provider {
     id: "mail.ru",
@@ -660,6 +676,8 @@ static P_MAIL_RU: Lazy<Provider> = Lazy::new(|| {
     after_login_hint: "",
     overview_page: "https://providers.delta.chat/mail-ru",
     server: vec![
+        Server { protocol: Imap, socket: Ssl, hostname: "imap.mail.ru", port: 993, username_pattern: Email },
+        Server { protocol: Smtp, socket: Ssl, hostname: "smtp.mail.ru", port: 465, username_pattern: Email },
     ],
     config_defaults: None,
     strict_tls: true,
@@ -773,6 +791,35 @@ static P_NAUTA_CU: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
+// naver.md: naver.com
+static P_NAVER: Lazy<Provider> = Lazy::new(|| Provider {
+    id: "naver",
+    status: Status::Preparation,
+    before_login_hint: "Manually enabling IMAP/SMTP is required.",
+    after_login_hint: "",
+    overview_page: "https://providers.delta.chat/naver",
+    server: vec![
+        Server {
+            protocol: Imap,
+            socket: Ssl,
+            hostname: "imap.naver.com",
+            port: 993,
+            username_pattern: Emaillocalpart,
+        },
+        Server {
+            protocol: Smtp,
+            socket: Starttls,
+            hostname: "smtp.naver.com",
+            port: 587,
+            username_pattern: Email,
+        },
+    ],
+    config_defaults: None,
+    strict_tls: true,
+    max_smtp_rcpt_to: None,
+    oauth2_authorizer: None,
+});
+
 // outlook.com.md: hotmail.com, outlook.com, office365.com, outlook.com.tr, live.com
 static P_OUTLOOK_COM: Lazy<Provider> = Lazy::new(|| Provider {
     id: "outlook.com",
@@ -802,7 +849,7 @@ static P_OUTLOOK_COM: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
-// posteo.md: posteo.de, posteo.af, posteo.at, posteo.be, posteo.ch, posteo.cl, posteo.co, posteo.co.uk, posteo.com.br, posteo.cr, posteo.cz, posteo.dk, posteo.ee, posteo.es, posteo.eu, posteo.fi, posteo.gl, posteo.gr, posteo.hn, posteo.hr, posteo.hu, posteo.ie, posteo.in, posteo.is, posteo.jp, posteo.la, posteo.li, posteo.lt, posteo.lu, posteo.me, posteo.mx, posteo.my, posteo.net, posteo.nl, posteo.no, posteo.nz, posteo.org, posteo.pe, posteo.pl, posteo.pm, posteo.pt, posteo.ro, posteo.ru, posteo.se, posteo.sg, posteo.si, posteo.tn, posteo.uk, posteo.us
+// posteo.md: posteo.de, posteo.af, posteo.at, posteo.be, posteo.ch, posteo.cl, posteo.co, posteo.co.uk, posteo.com.br, posteo.cr, posteo.cz, posteo.dk, posteo.ee, posteo.es, posteo.eu, posteo.fi, posteo.gl, posteo.gr, posteo.hn, posteo.hr, posteo.hu, posteo.ie, posteo.in, posteo.is, posteo.it, posteo.jp, posteo.la, posteo.li, posteo.lt, posteo.lu, posteo.me, posteo.mx, posteo.my, posteo.net, posteo.nl, posteo.no, posteo.nz, posteo.org, posteo.pe, posteo.pl, posteo.pm, posteo.pt, posteo.ro, posteo.ru, posteo.se, posteo.sg, posteo.si, posteo.tn, posteo.uk, posteo.us
 static P_POSTEO: Lazy<Provider> = Lazy::new(|| Provider {
     id: "posteo",
     status: Status::Ok,
@@ -848,6 +895,25 @@ static P_PROTONMAIL: Lazy<Provider> = Lazy::new(|| {
 }
 });
 
+// qq.md: qq.com, foxmail.com
+static P_QQ: Lazy<Provider> = Lazy::new(|| {
+    Provider {
+    id: "qq",
+    status: Status::Preparation,
+    before_login_hint: "Manually enabling IMAP/SMTP and creating an app-specific password for Delta Chat are required.",
+    after_login_hint: "",
+    overview_page: "https://providers.delta.chat/qq",
+    server: vec![
+        Server { protocol: Imap, socket: Ssl, hostname: "imap.qq.com", port: 993, username_pattern: Emaillocalpart },
+        Server { protocol: Smtp, socket: Starttls, hostname: "smtp.qq.com", port: 465, username_pattern: Email },
+    ],
+    config_defaults: None,
+    strict_tls: true,
+    max_smtp_rcpt_to: None,
+    oauth2_authorizer: None,
+}
+});
+
 // riseup.net.md: riseup.net
 static P_RISEUP_NET: Lazy<Provider> = Lazy::new(|| Provider {
     id: "riseup.net",
@@ -876,6 +942,35 @@ static P_ROGERS_COM: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
+// systemausfall.org.md: systemausfall.org, solidaris.me
+static P_SYSTEMAUSFALL_ORG: Lazy<Provider> = Lazy::new(|| Provider {
+    id: "systemausfall.org",
+    status: Status::Ok,
+    before_login_hint: "",
+    after_login_hint: "",
+    overview_page: "https://providers.delta.chat/systemausfall-org",
+    server: vec![
+        Server {
+            protocol: Imap,
+            socket: Ssl,
+            hostname: "mail.systemausfall.org",
+            port: 993,
+            username_pattern: Email,
+        },
+        Server {
+            protocol: Smtp,
+            socket: Ssl,
+            hostname: "mail.systemausfall.org",
+            port: 465,
+            username_pattern: Email,
+        },
+    ],
+    config_defaults: None,
+    strict_tls: true,
+    max_smtp_rcpt_to: None,
+    oauth2_authorizer: None,
+});
+
 // systemli.org.md: systemli.org
 static P_SYSTEMLI_ORG: Lazy<Provider> = Lazy::new(|| Provider {
     id: "systemli.org",
@@ -991,6 +1086,23 @@ static P_TISCALI_IT: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
+// tutanota.md: tutanota.com, tutanota.de, tutamail.com, tuta.io, keemail.me
+static P_TUTANOTA: Lazy<Provider> = Lazy::new(|| {
+    Provider {
+    id: "tutanota",
+    status: Status::Broken,
+    before_login_hint: "Tutanota does not offer the standard IMAP e-mail protocol, so you cannot log in with Delta Chat to Tutanota.",
+    after_login_hint: "",
+    overview_page: "https://providers.delta.chat/tutanota",
+    server: vec![
+    ],
+    config_defaults: None,
+    strict_tls: true,
+    max_smtp_rcpt_to: None,
+    oauth2_authorizer: None,
+}
+});
+
 // ukr.net.md: ukr.net
 static P_UKR_NET: Lazy<Provider> = Lazy::new(|| Provider {
     id: "ukr.net",
@@ -1048,6 +1160,35 @@ static P_VFEMAIL: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
+// vivaldi.md: vivaldi.net
+static P_VIVALDI: Lazy<Provider> = Lazy::new(|| Provider {
+    id: "vivaldi",
+    status: Status::Ok,
+    before_login_hint: "",
+    after_login_hint: "",
+    overview_page: "https://providers.delta.chat/vivaldi",
+    server: vec![
+        Server {
+            protocol: Imap,
+            socket: Starttls,
+            hostname: "imap.vivaldi.net",
+            port: 143,
+            username_pattern: Email,
+        },
+        Server {
+            protocol: Smtp,
+            socket: Starttls,
+            hostname: "smtp.vivaldi.net",
+            port: 587,
+            username_pattern: Email,
+        },
+    ],
+    config_defaults: None,
+    strict_tls: true,
+    max_smtp_rcpt_to: None,
+    oauth2_authorizer: None,
+});
+
 // vodafone.de.md: vodafone.de, vodafonemail.de
 static P_VODAFONE_DE: Lazy<Provider> = Lazy::new(|| Provider {
     id: "vodafone.de",
@@ -1174,8 +1315,38 @@ static P_ZIGGO_NL: Lazy<Provider> = Lazy::new(|| Provider {
     oauth2_authorizer: None,
 });
 
+// zoho.md: zohomail.eu, zoho.com
+static P_ZOHO: Lazy<Provider> = Lazy::new(|| Provider {
+    id: "zoho",
+    status: Status::Preparation,
+    before_login_hint: "To use Zoho Mail, you have to turn on IMAP in the Zoho Mail backend.",
+    after_login_hint: "",
+    overview_page: "https://providers.delta.chat/zoho",
+    server: vec![
+        Server {
+            protocol: Imap,
+            socket: Ssl,
+            hostname: "imap.zoho.eu",
+            port: 993,
+            username_pattern: Email,
+        },
+        Server {
+            protocol: Smtp,
+            socket: Ssl,
+            hostname: "smtp.zoho.eu",
+            port: 465,
+            username_pattern: Email,
+        },
+    ],
+    config_defaults: None,
+    strict_tls: true,
+    max_smtp_rcpt_to: None,
+    oauth2_authorizer: None,
+});
+
 pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>> = Lazy::new(|| {
     [
+        ("163.com", &*P_163),
         ("aktivix.org", &*P_AKTIVIX_ORG),
         ("aol.com", &*P_AOL),
         ("arcor.de", &*P_ARCOR_DE),
@@ -1221,12 +1392,14 @@ pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>>
         ("kontent.com", &*P_KONTENT_COM),
         ("mail.ru", &*P_MAIL_RU),
         ("inbox.ru", &*P_MAIL_RU),
+        ("internet.ru", &*P_MAIL_RU),
         ("bk.ru", &*P_MAIL_RU),
         ("list.ru", &*P_MAIL_RU),
         ("mailbox.org", &*P_MAILBOX_ORG),
         ("secure.mailbox.org", &*P_MAILBOX_ORG),
         ("mailo.com", &*P_MAILO_COM),
         ("nauta.cu", &*P_NAUTA_CU),
+        ("naver.com", &*P_NAVER),
         ("hotmail.com", &*P_OUTLOOK_COM),
         ("outlook.com", &*P_OUTLOOK_COM),
         ("office365.com", &*P_OUTLOOK_COM),
@@ -1256,6 +1429,7 @@ pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>>
         ("posteo.ie", &*P_POSTEO),
         ("posteo.in", &*P_POSTEO),
         ("posteo.is", &*P_POSTEO),
+        ("posteo.it", &*P_POSTEO),
         ("posteo.jp", &*P_POSTEO),
         ("posteo.la", &*P_POSTEO),
         ("posteo.li", &*P_POSTEO),
@@ -1283,16 +1457,26 @@ pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>>
         ("posteo.us", &*P_POSTEO),
         ("protonmail.com", &*P_PROTONMAIL),
         ("protonmail.ch", &*P_PROTONMAIL),
+        ("qq.com", &*P_QQ),
+        ("foxmail.com", &*P_QQ),
         ("riseup.net", &*P_RISEUP_NET),
         ("rogers.com", &*P_ROGERS_COM),
+        ("systemausfall.org", &*P_SYSTEMAUSFALL_ORG),
+        ("solidaris.me", &*P_SYSTEMAUSFALL_ORG),
         ("systemli.org", &*P_SYSTEMLI_ORG),
         ("t-online.de", &*P_T_ONLINE),
         ("magenta.de", &*P_T_ONLINE),
         ("testrun.org", &*P_TESTRUN),
         ("tiscali.it", &*P_TISCALI_IT),
+        ("tutanota.com", &*P_TUTANOTA),
+        ("tutanota.de", &*P_TUTANOTA),
+        ("tutamail.com", &*P_TUTANOTA),
+        ("tuta.io", &*P_TUTANOTA),
+        ("keemail.me", &*P_TUTANOTA),
         ("ukr.net", &*P_UKR_NET),
         ("undernet.uy", &*P_UNDERNET_UY),
         ("vfemail.net", &*P_VFEMAIL),
+        ("vivaldi.net", &*P_VIVALDI),
         ("vodafone.de", &*P_VODAFONE_DE),
         ("vodafonemail.de", &*P_VODAFONE_DE),
         ("web.de", &*P_WEB_DE),
@@ -1345,6 +1529,8 @@ pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>>
         ("ya.ru", &*P_YANDEX_RU),
         ("narod.ru", &*P_YANDEX_RU),
         ("ziggo.nl", &*P_ZIGGO_NL),
+        ("zohomail.eu", &*P_ZOHO),
+        ("zoho.com", &*P_ZOHO),
     ]
     .iter()
     .copied()
@@ -1353,6 +1539,7 @@ pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>>
 
 pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> = Lazy::new(|| {
     [
+        ("163", &*P_163),
         ("aktivix.org", &*P_AKTIVIX_ORG),
         ("aol", &*P_AOL),
         ("arcor.de", &*P_ARCOR_DE),
@@ -1384,23 +1571,29 @@ pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> =
         ("mailbox.org", &*P_MAILBOX_ORG),
         ("mailo.com", &*P_MAILO_COM),
         ("nauta.cu", &*P_NAUTA_CU),
+        ("naver", &*P_NAVER),
         ("outlook.com", &*P_OUTLOOK_COM),
         ("posteo", &*P_POSTEO),
         ("protonmail", &*P_PROTONMAIL),
+        ("qq", &*P_QQ),
         ("riseup.net", &*P_RISEUP_NET),
         ("rogers.com", &*P_ROGERS_COM),
+        ("systemausfall.org", &*P_SYSTEMAUSFALL_ORG),
         ("systemli.org", &*P_SYSTEMLI_ORG),
         ("t-online", &*P_T_ONLINE),
         ("testrun", &*P_TESTRUN),
         ("tiscali.it", &*P_TISCALI_IT),
+        ("tutanota", &*P_TUTANOTA),
         ("ukr.net", &*P_UKR_NET),
         ("undernet.uy", &*P_UNDERNET_UY),
         ("vfemail", &*P_VFEMAIL),
+        ("vivaldi", &*P_VIVALDI),
         ("vodafone.de", &*P_VODAFONE_DE),
         ("web.de", &*P_WEB_DE),
         ("yahoo", &*P_YAHOO),
         ("yandex.ru", &*P_YANDEX_RU),
         ("ziggo.nl", &*P_ZIGGO_NL),
+        ("zoho", &*P_ZOHO),
     ]
     .iter()
     .copied()
@@ -1408,4 +1601,4 @@ pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> =
 });
 
 pub static PROVIDER_UPDATED: Lazy<chrono::NaiveDate> =
-    Lazy::new(|| chrono::NaiveDate::from_ymd(2021, 4, 10));
+    Lazy::new(|| chrono::NaiveDate::from_ymd(2021, 7, 28));

src/qr.rs

@@ -6,7 +6,7 @@ use percent_encoding::percent_decode_str;
 use serde::Deserialize;
 use std::collections::BTreeMap;
 
-use crate::chat::{self, ChatIdBlocked};
+use crate::chat::{self, get_chat_id_by_grpid, ChatIdBlocked};
 use crate::config::Config;
 use crate::constants::Blocked;
 use crate::contact::{addr_normalize, may_be_valid_addr, Contact, Origin};
@@ -16,6 +16,7 @@ use crate::log::LogExt;
 use crate::lot::{Lot, LotState};
 use crate::message::Message;
 use crate::peerstate::Peerstate;
+use crate::token;
 
 const OPENPGP4FPR_SCHEME: &str = "OPENPGP4FPR:"; // yes: uppercase
 const DCACCOUNT_SCHEME: &str = "DCACCOUNT:";
@@ -86,11 +87,7 @@ async fn decode_openpgp(context: &Context, qr: &str) -> Lot {
     };
     let fingerprint: Fingerprint = match fingerprint.parse() {
         Ok(fp) => fp,
-        Err(err) => {
-            return Error::new(err)
-                .context("Failed to parse fingerprint in QR code")
-                .into()
-        }
+        Err(err) => return err.context("Failed to parse fingerprint in QR code").into(),
     };
 
     let param: BTreeMap<&str, &str> = fragment
@@ -159,7 +156,7 @@ async fn decode_openpgp(context: &Context, qr: &str) -> Lot {
                     .map(|(id, _)| id)
                     .unwrap_or_default();
 
-            if let Ok(chat) = ChatIdBlocked::get_for_contact(context, lot.id, Blocked::Deaddrop)
+            if let Ok(chat) = ChatIdBlocked::get_for_contact(context, lot.id, Blocked::Request)
                 .await
                 .log_err(context, "Failed to create (new) chat for contact")
             {
@@ -193,6 +190,25 @@ async fn decode_openpgp(context: &Context, qr: &str) -> Lot {
         lot.fingerprint = Some(fingerprint);
         lot.invitenumber = invitenumber;
         lot.auth = auth;
+
+        // scanning own qr-code offers withdraw/revive instead of secure-join
+        if context.is_self_addr(&addr).await.unwrap_or_default() {
+            if let Some(ref invitenumber) = lot.invitenumber {
+                lot.state =
+                    if token::exists(context, token::Namespace::InviteNumber, &*invitenumber).await
+                    {
+                        if lot.state == LotState::QrAskVerifyContact {
+                            LotState::QrWithdrawVerifyContact
+                        } else {
+                            LotState::QrWithdrawVerifyGroup
+                        }
+                    } else if lot.state == LotState::QrAskVerifyContact {
+                        LotState::QrReviveVerifyContact
+                    } else {
+                        LotState::QrReviveVerifyGroup
+                    }
+            }
+        }
     } else {
         return format_err!("Missing address").into();
     }
@@ -279,7 +295,8 @@ async fn set_account_from_qr(context: &Context, qr: &str) -> Result<(), Error> {
 }
 
 pub async fn set_config_from_qr(context: &Context, qr: &str) -> Result<(), Error> {
-    match check_qr(context, qr).await.state {
+    let lot = check_qr(context, qr).await;
+    match lot.state {
         LotState::QrAccount => set_account_from_qr(context, qr).await,
         LotState::QrWebrtcInstance => {
             let val = decode_webrtc_instance(context, qr).text2;
@@ -288,6 +305,47 @@ pub async fn set_config_from_qr(context: &Context, qr: &str) -> Result<(), Error
                 .await?;
             Ok(())
         }
+        LotState::QrWithdrawVerifyContact | LotState::QrWithdrawVerifyGroup => {
+            token::delete(
+                context,
+                token::Namespace::InviteNumber,
+                lot.invitenumber.unwrap_or_default().as_str(),
+            )
+            .await?;
+            token::delete(
+                context,
+                token::Namespace::Auth,
+                lot.auth.unwrap_or_default().as_str(),
+            )
+            .await?;
+            Ok(())
+        }
+        LotState::QrReviveVerifyContact | LotState::QrReviveVerifyGroup => {
+            let chat_id = if lot.state == LotState::QrReviveVerifyContact {
+                None
+            } else {
+                Some(
+                    get_chat_id_by_grpid(context, &lot.text2.unwrap_or_default())
+                        .await?
+                        .0,
+                )
+            };
+            token::save(
+                context,
+                token::Namespace::InviteNumber,
+                chat_id,
+                &lot.invitenumber.unwrap_or_default(),
+            )
+            .await?;
+            token::save(
+                context,
+                token::Namespace::Auth,
+                chat_id,
+                &lot.auth.unwrap_or_default(),
+            )
+            .await?;
+            Ok(())
+        }
         _ => bail!("qr code does not contain config: {}", qr),
     }
 }
@@ -441,9 +499,12 @@ mod tests {
     use super::*;
 
     use crate::aheader::EncryptPreference;
+    use crate::chat::{create_group_chat, ProtectionStatus};
     use crate::key::DcKey;
     use crate::peerstate::ToSave;
+    use crate::securejoin::dc_get_securejoin_qr;
     use crate::test_utils::{alice_keypair, TestContext};
+    use anyhow::Result;
 
     #[async_std::test]
     async fn test_decode_http() {
@@ -720,6 +781,62 @@ mod tests {
         assert_eq!(res.get_id(), 0);
     }
 
+    #[async_std::test]
+    async fn test_withdraw_verfifycontact() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+        let qr = dc_get_securejoin_qr(&alice, None).await.unwrap();
+
+        // scanning own verfify-contact code offers withdrawing
+        let check = check_qr(&alice, &qr).await;
+        assert_eq!(check.state, LotState::QrWithdrawVerifyContact);
+        assert!(check.text1.is_none());
+        set_config_from_qr(&alice, &qr).await?;
+
+        // scanning withdrawn verfify-contact code offers reviving
+        let check = check_qr(&alice, &qr).await;
+        assert_eq!(check.state, LotState::QrReviveVerifyContact);
+        assert!(check.text1.is_none());
+        set_config_from_qr(&alice, &qr).await?;
+        let check = check_qr(&alice, &qr).await;
+        assert_eq!(check.state, LotState::QrWithdrawVerifyContact);
+
+        // someone else always scans as ask-verify-contact
+        let bob = TestContext::new_bob().await;
+        let check = check_qr(&bob, &qr).await;
+        assert_eq!(check.state, LotState::QrAskVerifyContact);
+        assert!(check.text1.is_none());
+        assert!(set_config_from_qr(&bob, &qr).await.is_err());
+
+        Ok(())
+    }
+
+    #[async_std::test]
+    async fn test_withdraw_verfifygroup() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+        let chat_id = create_group_chat(&alice, ProtectionStatus::Unprotected, "foo").await?;
+        let qr = dc_get_securejoin_qr(&alice, Some(chat_id)).await.unwrap();
+
+        // scanning own verfify-group code offers withdrawing
+        let check = check_qr(&alice, &qr).await;
+        assert_eq!(check.state, LotState::QrWithdrawVerifyGroup);
+        assert_eq!(check.text1, Some("foo".to_string()));
+        set_config_from_qr(&alice, &qr).await?;
+
+        // scanning withdrawn verfify-group code offers reviving
+        let check = check_qr(&alice, &qr).await;
+        assert_eq!(check.state, LotState::QrReviveVerifyGroup);
+        assert_eq!(check.text1, Some("foo".to_string()));
+
+        // someone else always scans as ask-verify-group
+        let bob = TestContext::new_bob().await;
+        let check = check_qr(&bob, &qr).await;
+        assert_eq!(check.state, LotState::QrAskVerifyGroup);
+        assert_eq!(check.text1, Some("foo".to_string()));
+        assert!(set_config_from_qr(&bob, &qr).await.is_err());
+
+        Ok(())
+    }
+
     #[async_std::test]
     async fn test_decode_account() {
         let ctx = TestContext::new().await;

src/scheduler.rs

@@ -1,3 +1,4 @@
+use anyhow::{bail, Result};
 use async_std::prelude::*;
 use async_std::{
     channel::{self, Receiver, Sender},
@@ -12,6 +13,10 @@ use crate::job::{self, Thread};
 use crate::message::MsgId;
 use crate::smtp::Smtp;
 
+use self::connectivity::ConnectivityStore;
+
+pub(crate) mod connectivity;
+
 pub(crate) struct StopToken;
 
 /// Job and connection scheduler.
@@ -34,7 +39,16 @@ pub(crate) enum Scheduler {
 impl Context {
     /// Indicate that the network likely has come back.
     pub async fn maybe_network(&self) {
-        self.scheduler.read().await.maybe_network().await;
+        let lock = self.scheduler.read().await;
+        lock.maybe_network().await;
+        connectivity::idle_interrupted(lock).await;
+    }
+
+    /// Indicate that the network likely is lost.
+    pub async fn maybe_network_lost(&self) {
+        let lock = self.scheduler.read().await;
+        lock.maybe_network_lost().await;
+        connectivity::idle_interrupted(lock).await;
     }
 
     pub(crate) async fn interrupt_inbox(&self, info: InterruptInfo) {
@@ -106,6 +120,9 @@ async fn inbox_loop(ctx: Context, started: Sender<()>, inbox_handlers: ImapConne
                     } else {
                         if let Err(err) = connection.scan_folders(&ctx).await {
                             warn!(ctx, "{}", err);
+                            connection.connectivity.set_err(&ctx, err).await;
+                        } else {
+                            connection.connectivity.set_not_configured(&ctx).await;
                         }
                         connection.fake_idle(&ctx, None).await
                     };
@@ -130,27 +147,25 @@ async fn inbox_loop(ctx: Context, started: Sender<()>, inbox_handlers: ImapConne
 async fn fetch(ctx: &Context, connection: &mut Imap) {
     match ctx.get_config(Config::ConfiguredInboxFolder).await {
         Ok(Some(watch_folder)) => {
-            if let Err(err) = connection.connect_configured(ctx).await {
-                error_network!(ctx, "{}", err);
+            if let Err(err) = connection.prepare(ctx).await {
+                warn!(ctx, "Could not connect: {}", err);
                 return;
             }
 
             // fetch
             if let Err(err) = connection.fetch(ctx, &watch_folder).await {
-                connection.trigger_reconnect();
+                connection.trigger_reconnect(ctx).await;
                 warn!(ctx, "{:#}", err);
             }
         }
         Ok(None) => {
-            warn!(ctx, "Can not fetch inbox folder, not set");
-            connection.fake_idle(ctx, None).await;
+            info!(ctx, "Can not fetch inbox folder, not set");
         }
         Err(err) => {
             warn!(
                 ctx,
                 "Can not fetch inbox folder, failed to get config: {:?}", err
             );
-            connection.fake_idle(ctx, None).await;
         }
     }
 }
@@ -159,15 +174,16 @@ async fn fetch_idle(ctx: &Context, connection: &mut Imap, folder: Config) -> Int
     match ctx.get_config(folder).await {
         Ok(Some(watch_folder)) => {
             // connect and fake idle if unable to connect
-            if let Err(err) = connection.connect_configured(ctx).await {
+            if let Err(err) = connection.prepare(ctx).await {
                 warn!(ctx, "imap connection failed: {}", err);
                 return connection.fake_idle(ctx, Some(watch_folder)).await;
             }
 
             // fetch
             if let Err(err) = connection.fetch(ctx, &watch_folder).await {
-                connection.trigger_reconnect();
+                connection.trigger_reconnect(ctx).await;
                 warn!(ctx, "{:#}", err);
+                return InterruptInfo::new(false, None);
             }
 
             if folder == Config::ConfiguredInboxFolder {
@@ -179,22 +195,25 @@ async fn fetch_idle(ctx: &Context, connection: &mut Imap, folder: Config) -> Int
                 }
             }
 
+            connection.connectivity.set_connected(ctx).await;
+
             // idle
             if connection.can_idle() {
-                connection
-                    .idle(ctx, Some(watch_folder))
-                    .await
-                    .unwrap_or_else(|err| {
-                        connection.trigger_reconnect();
+                match connection.idle(ctx, Some(watch_folder)).await {
+                    Ok(v) => v,
+                    Err(err) => {
+                        connection.trigger_reconnect(ctx).await;
                         warn!(ctx, "{}", err);
                         InterruptInfo::new(false, None)
-                    })
+                    }
+                }
             } else {
                 connection.fake_idle(ctx, Some(watch_folder)).await
             }
         }
         Ok(None) => {
-            warn!(ctx, "Can not watch {} folder, not set", folder);
+            connection.connectivity.set_not_configured(ctx).await;
+            info!(ctx, "Can not watch {} folder, not set", folder);
             connection.fake_idle(ctx, None).await
         }
         Err(err) => {
@@ -279,6 +298,7 @@ async fn smtp_loop(ctx: Context, started: Sender<()>, smtp_handlers: SmtpConnect
                 None => {
                     // Fake Idle
                     info!(ctx, "smtp fake idle - started");
+                    connection.connectivity.set_connected(&ctx).await;
                     interrupt_info = idle_interrupt_receiver.recv().await.unwrap_or_default();
                     info!(ctx, "smtp fake idle - interrupted")
                 }
@@ -301,11 +321,11 @@ async fn smtp_loop(ctx: Context, started: Sender<()>, smtp_handlers: SmtpConnect
 
 impl Scheduler {
     /// Start the scheduler, panics if it is already running.
-    pub async fn start(&mut self, ctx: Context) {
-        let (mvbox, mvbox_handlers) = ImapConnectionState::new();
-        let (sentbox, sentbox_handlers) = ImapConnectionState::new();
+    pub async fn start(&mut self, ctx: Context) -> Result<()> {
+        let (mvbox, mvbox_handlers) = ImapConnectionState::new(&ctx).await?;
+        let (sentbox, sentbox_handlers) = ImapConnectionState::new(&ctx).await?;
         let (smtp, smtp_handlers) = SmtpConnectionState::new();
-        let (inbox, inbox_handlers) = ImapConnectionState::new();
+        let (inbox, inbox_handlers) = ImapConnectionState::new(&ctx).await?;
 
         let (inbox_start_send, inbox_start_recv) = channel::bounded(1);
         let (mvbox_start_send, mvbox_start_recv) = channel::bounded(1);
@@ -321,11 +341,7 @@ impl Scheduler {
             }))
         };
 
-        if ctx
-            .get_config_bool(Config::MvboxWatch)
-            .await
-            .unwrap_or_default()
-        {
+        if ctx.get_config_bool(Config::MvboxWatch).await? {
             let ctx = ctx.clone();
             mvbox_handle = Some(task::spawn(async move {
                 simple_imap_loop(
@@ -341,13 +357,14 @@ impl Scheduler {
                 .send(())
                 .await
                 .expect("mvbox start send, missing receiver");
+            mvbox_handlers
+                .connection
+                .connectivity
+                .set_not_configured(&ctx)
+                .await
         }
 
-        if ctx
-            .get_config_bool(Config::SentboxWatch)
-            .await
-            .unwrap_or_default()
-        {
+        if ctx.get_config_bool(Config::SentboxWatch).await? {
             let ctx = ctx.clone();
             sentbox_handle = Some(task::spawn(async move {
                 simple_imap_loop(
@@ -363,6 +380,11 @@ impl Scheduler {
                 .send(())
                 .await
                 .expect("sentbox start send, missing receiver");
+            sentbox_handlers
+                .connection
+                .connectivity
+                .set_not_configured(&ctx)
+                .await
         }
 
         let smtp_handle = {
@@ -391,10 +413,11 @@ impl Scheduler {
             .try_join(smtp_start_recv.recv())
             .await
         {
-            error!(ctx, "failed to start scheduler: {}", err);
+            bail!("failed to start scheduler: {}", err);
         }
 
         info!(ctx, "scheduler is running");
+        Ok(())
     }
 
     async fn maybe_network(&self) {
@@ -409,6 +432,18 @@ impl Scheduler {
             .await;
     }
 
+    async fn maybe_network_lost(&self) {
+        if !self.is_running() {
+            return;
+        }
+
+        self.interrupt_inbox(InterruptInfo::new(false, None))
+            .join(self.interrupt_mvbox(InterruptInfo::new(false, None)))
+            .join(self.interrupt_sentbox(InterruptInfo::new(false, None)))
+            .join(self.interrupt_smtp(InterruptInfo::new(false, None)))
+            .await;
+    }
+
     async fn interrupt_inbox(&self, info: InterruptInfo) {
         if let Scheduler::Running { ref inbox, .. } = self {
             inbox.interrupt(info).await;
@@ -514,6 +549,8 @@ struct ConnectionState {
     stop_sender: Sender<()>,
     /// Channel to interrupt idle.
     idle_interrupt_sender: Sender<InterruptInfo>,
+    /// Mutex to pass connectivity info between IMAP/SMTP threads and the API
+    connectivity: ConnectivityStore,
 }
 
 impl ConnectionState {
@@ -556,6 +593,7 @@ impl SmtpConnectionState {
             shutdown_receiver,
             stop_sender,
             idle_interrupt_sender,
+            connectivity: handlers.connection.connectivity.clone(),
         };
 
         let conn = SmtpConnectionState { state };
@@ -588,13 +626,13 @@ pub(crate) struct ImapConnectionState {
 
 impl ImapConnectionState {
     /// Construct a new connection.
-    fn new() -> (Self, ImapConnectionHandlers) {
+    async fn new(context: &Context) -> Result<(Self, ImapConnectionHandlers)> {
         let (stop_sender, stop_receiver) = channel::bounded(1);
         let (shutdown_sender, shutdown_receiver) = channel::bounded(1);
         let (idle_interrupt_sender, idle_interrupt_receiver) = channel::bounded(1);
 
         let handlers = ImapConnectionHandlers {
-            connection: Imap::new(idle_interrupt_receiver),
+            connection: Imap::new_configured(context, idle_interrupt_receiver).await?,
             stop_receiver,
             shutdown_sender,
         };
@@ -603,11 +641,12 @@ impl ImapConnectionState {
             shutdown_receiver,
             stop_sender,
             idle_interrupt_sender,
+            connectivity: handlers.connection.connectivity.clone(),
         };
 
         let conn = ImapConnectionState { state };
 
-        (conn, handlers)
+        Ok((conn, handlers))
     }
 
     /// Interrupt any form of idle.

src/scheduler/connectivity.rs

@@ -0,0 +1,397 @@
+use core::fmt;
+use std::{ops::Deref, sync::Arc};
+
+use async_std::sync::{Mutex, RwLockReadGuard};
+
+use crate::events::EventType;
+use crate::{config::Config, scheduler::Scheduler};
+use crate::{context::Context, log::LogExt};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, EnumProperty, PartialOrd, Ord)]
+pub enum Connectivity {
+    NotConnected = 1000,
+    Connecting = 2000,
+    /// Fetching or sending messages
+    Working = 3000,
+    Connected = 4000,
+}
+
+// The order of the connectivities is important: worse connectivities (i.e. those at
+// the top) take priority. This means that e.g. if any folder has an error - usually
+// because there is no internet connection - the connectivity for the whole
+// account will be `Notconnected`.
+#[derive(Debug, Clone, PartialEq, Eq, EnumProperty)]
+enum DetailedConnectivity {
+    Error(String),
+    Uninitialized,
+    Connecting,
+    Working,
+    InterruptingIdle,
+    Connected,
+
+    /// The folder was configured not to be watched or configured_*_folder is not set
+    NotConfigured,
+}
+
+impl Default for DetailedConnectivity {
+    fn default() -> Self {
+        DetailedConnectivity::Uninitialized
+    }
+}
+
+impl DetailedConnectivity {
+    fn to_basic(&self) -> Option<Connectivity> {
+        match self {
+            DetailedConnectivity::Error(_) => Some(Connectivity::NotConnected),
+            DetailedConnectivity::Uninitialized => Some(Connectivity::NotConnected),
+            DetailedConnectivity::Connecting => Some(Connectivity::Connecting),
+            DetailedConnectivity::Working => Some(Connectivity::Working),
+            DetailedConnectivity::InterruptingIdle => Some(Connectivity::Connected),
+            DetailedConnectivity::Connected => Some(Connectivity::Connected),
+
+            // Just don't return a connectivity, probably the folder is configured not to be
+            // watched or there is e.g. no "Sent" folder, so we are not interested in it
+            DetailedConnectivity::NotConfigured => None,
+        }
+    }
+
+    fn to_icon(&self) -> String {
+        match self {
+            DetailedConnectivity::Error(_)
+            | DetailedConnectivity::Uninitialized
+            | DetailedConnectivity::NotConfigured => "<span class=\"red dot\"></span>".to_string(),
+            DetailedConnectivity::Connecting => "<span class=\"yellow dot\"></span>".to_string(),
+            DetailedConnectivity::Working
+            | DetailedConnectivity::InterruptingIdle
+            | DetailedConnectivity::Connected => "<span class=\"green dot\"></span>".to_string(),
+        }
+    }
+
+    fn to_string_imap(&self, _context: &Context) -> String {
+        match self {
+            DetailedConnectivity::Error(e) => format!("Error: {}", e),
+            DetailedConnectivity::Uninitialized => "Not started".to_string(),
+            DetailedConnectivity::Connecting => "Connecting…".to_string(),
+            DetailedConnectivity::Working => "Getting new messages…".to_string(),
+            DetailedConnectivity::InterruptingIdle | DetailedConnectivity::Connected => {
+                "Connected".to_string()
+            }
+            DetailedConnectivity::NotConfigured => "Not configured".to_string(),
+        }
+    }
+
+    fn to_string_smtp(&self, _context: &Context) -> String {
+        match self {
+            DetailedConnectivity::Error(e) => format!("Error: {}", e),
+            DetailedConnectivity::Uninitialized => {
+                "(You did not try to send a message recently)".to_string()
+            }
+            DetailedConnectivity::Connecting => "Connecting…".to_string(),
+            DetailedConnectivity::Working => "Sending…".to_string(),
+
+            // We don't know any more than that the last message was sent successfully;
+            // since sending the last message, connectivity could have changed, which we don't notice
+            // until another message is sent
+            DetailedConnectivity::InterruptingIdle | DetailedConnectivity::Connected => {
+                "Your last message was sent successfully".to_string()
+            }
+            DetailedConnectivity::NotConfigured => "Not configured".to_string(),
+        }
+    }
+
+    fn all_work_done(&self) -> bool {
+        match self {
+            DetailedConnectivity::Error(_) => true,
+            DetailedConnectivity::Uninitialized => false,
+            DetailedConnectivity::Connecting => false,
+            DetailedConnectivity::Working => false,
+            DetailedConnectivity::InterruptingIdle => false,
+            DetailedConnectivity::Connected => true,
+            DetailedConnectivity::NotConfigured => true,
+        }
+    }
+}
+
+#[derive(Clone, Default)]
+pub(crate) struct ConnectivityStore(Arc<Mutex<DetailedConnectivity>>);
+
+impl ConnectivityStore {
+    async fn set(&self, context: &Context, v: DetailedConnectivity) {
+        {
+            *self.0.lock().await = v;
+        }
+        context.emit_event(EventType::ConnectivityChanged);
+    }
+
+    pub(crate) async fn set_err(&self, context: &Context, e: impl ToString) {
+        self.set(context, DetailedConnectivity::Error(e.to_string()))
+            .await;
+    }
+    pub(crate) async fn set_connecting(&self, context: &Context) {
+        self.set(context, DetailedConnectivity::Connecting).await;
+    }
+    pub(crate) async fn set_working(&self, context: &Context) {
+        self.set(context, DetailedConnectivity::Working).await;
+    }
+    pub(crate) async fn set_connected(&self, context: &Context) {
+        self.set(context, DetailedConnectivity::Connected).await;
+    }
+    pub(crate) async fn set_not_configured(&self, context: &Context) {
+        self.set(context, DetailedConnectivity::NotConfigured).await;
+    }
+
+    async fn get_detailed(&self) -> DetailedConnectivity {
+        self.0.lock().await.deref().clone()
+    }
+    async fn get_basic(&self) -> Option<Connectivity> {
+        self.0.lock().await.to_basic()
+    }
+    async fn get_all_work_done(&self) -> bool {
+        self.0.lock().await.all_work_done()
+    }
+}
+
+/// Set all folder states to InterruptingIdle in case they were `Connected` before.
+/// Called during `dc_maybe_network()` to make sure that `dc_accounts_all_work_done()`
+/// returns false immediately after `dc_maybe_network()`.
+pub(crate) async fn idle_interrupted(scheduler: RwLockReadGuard<'_, Scheduler>) {
+    let [inbox, mvbox, sentbox] = match &*scheduler {
+        Scheduler::Running {
+            inbox,
+            mvbox,
+            sentbox,
+            ..
+        } => [
+            inbox.state.connectivity.clone(),
+            mvbox.state.connectivity.clone(),
+            sentbox.state.connectivity.clone(),
+        ],
+        Scheduler::Stopped => return,
+    };
+    drop(scheduler);
+
+    let mut connectivity_lock = inbox.0.lock().await;
+    // For the inbox, we also have to set the connectivity to InterruptingIdle if it was
+    // NotConfigured before: If all folders are NotConfigured, dc_get_connectivity()
+    // returns Connected. But after dc_maybe_network(), dc_get_connectivity() must not
+    // return Connected until DC is completely done with fetching folders; this also
+    // includes scan_folders() which happens on the inbox thread.
+    if *connectivity_lock == DetailedConnectivity::Connected
+        || *connectivity_lock == DetailedConnectivity::NotConfigured
+    {
+        *connectivity_lock = DetailedConnectivity::InterruptingIdle;
+    }
+    drop(connectivity_lock);
+
+    for state in &[&mvbox, &sentbox] {
+        let mut connectivity_lock = state.0.lock().await;
+        if *connectivity_lock == DetailedConnectivity::Connected {
+            *connectivity_lock = DetailedConnectivity::InterruptingIdle;
+        }
+    }
+    // No need to send ConnectivityChanged, the user-facing connectivity doesn't change because
+    // of what we do here.
+}
+
+impl fmt::Debug for ConnectivityStore {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        if let Some(guard) = self.0.try_lock() {
+            write!(f, "ConnectivityStore {:?}", &*guard)
+        } else {
+            write!(f, "ConnectivityStore [LOCKED]")
+        }
+    }
+}
+
+impl Context {
+    /// Get the current connectivity, i.e. whether the device is connected to the IMAP server.
+    /// One of:
+    /// - DC_CONNECTIVITY_NOT_CONNECTED (1000-1999): Show e.g. the string "Not connected" or a red dot
+    /// - DC_CONNECTIVITY_CONNECTING (2000-2999): Show e.g. the string "Connecting…" or a yellow dot
+    /// - DC_CONNECTIVITY_WORKING (3000-3999): Show e.g. the string "Getting new messages" or a spinning wheel
+    /// - DC_CONNECTIVITY_CONNECTED (>=4000): Show e.g. the string "Connected" or a green dot
+    ///
+    /// We don't use exact values but ranges here so that we can split up
+    /// states into multiple states in the future.
+    ///
+    /// Meant as a rough overview that can be shown
+    /// e.g. in the title of the main screen.
+    ///
+    /// If the connectivity changes, a DC_EVENT_CONNECTIVITY_CHANGED will be emitted.
+    pub async fn get_connectivity(&self) -> Connectivity {
+        let lock = self.scheduler.read().await;
+        let stores: Vec<_> = match &*lock {
+            Scheduler::Running {
+                inbox,
+                mvbox,
+                sentbox,
+                ..
+            } => [&inbox.state, &mvbox.state, &sentbox.state]
+                .iter()
+                .map(|state| state.connectivity.clone())
+                .collect(),
+            Scheduler::Stopped => return Connectivity::NotConnected,
+        };
+        drop(lock);
+
+        let mut connectivities = Vec::new();
+        for s in stores {
+            if let Some(connectivity) = s.get_basic().await {
+                connectivities.push(connectivity);
+            }
+        }
+        connectivities
+            .into_iter()
+            .min()
+            .unwrap_or(Connectivity::Connected)
+    }
+
+    /// Get an overview of the current connectivity, and possibly more statistics.
+    /// Meant to give the user more insight about the current status than
+    /// the basic connectivity info returned by dc_get_connectivity(); show this
+    /// e.g., if the user taps on said basic connectivity info.
+    ///
+    /// If this page changes, a DC_EVENT_CONNECTIVITY_CHANGED will be emitted.
+    ///
+    /// This comes as an HTML from the core so that we can easily improve it
+    /// and the improvement instantly reaches all UIs.
+    pub async fn get_connectivity_html(&self) -> String {
+        let mut ret = r#"<!DOCTYPE html>
+            <html>
+            <head>
+                <meta charset="UTF-8" />
+                <meta name="viewport" content="initial-scale=1.0" />
+                <style>
+                    ul {
+                        list-style-type: none;
+                        padding-left: 1em;
+                    }
+
+                    .dot {
+                        height: 0.9em; width: 0.9em;
+                        border-radius: 50%;
+                        display: inline-block;
+                        position: relative; left: -0.1em; top: 0.1em;
+                    }
+                    .red {
+                        background-color: #f33b2d;
+                    }
+                    .green {
+                        background-color: #34c759;
+                    }
+                    .yellow {
+                        background-color: #fdc625;
+                    }
+                </style>
+            </head>
+            <body>"#
+            .to_string();
+
+        let lock = self.scheduler.read().await;
+        let (folders_states, smtp) = match &*lock {
+            Scheduler::Running {
+                inbox,
+                mvbox,
+                sentbox,
+                smtp,
+                ..
+            } => (
+                [
+                    (
+                        Config::ConfiguredInboxFolder,
+                        Config::InboxWatch,
+                        inbox.state.connectivity.clone(),
+                    ),
+                    (
+                        Config::ConfiguredMvboxFolder,
+                        Config::MvboxWatch,
+                        mvbox.state.connectivity.clone(),
+                    ),
+                    (
+                        Config::ConfiguredSentboxFolder,
+                        Config::SentboxWatch,
+                        sentbox.state.connectivity.clone(),
+                    ),
+                ],
+                smtp.state.connectivity.clone(),
+            ),
+            Scheduler::Stopped => {
+                ret += "Not started</body></html>\n";
+                return ret;
+            }
+        };
+        drop(lock);
+
+        ret += "<h3>Incoming messages</h3><ul>";
+        for (folder, watch, state) in &folders_states {
+            let w = self.get_config(*watch).await.ok_or_log(self);
+
+            let mut folder_added = false;
+            if w.flatten() == Some("1".to_string()) {
+                let f = self.get_config(*folder).await.ok_or_log(self).flatten();
+
+                if let Some(foldername) = f {
+                    let detailed = &state.get_detailed().await;
+                    ret += "<li>";
+                    ret += &*detailed.to_icon();
+                    ret += " <b>";
+                    ret += &*escaper::encode_minimal(&foldername);
+                    ret += ":</b> ";
+                    ret += &*escaper::encode_minimal(&*detailed.to_string_imap(self));
+                    ret += "</li>";
+
+                    folder_added = true;
+                }
+            }
+
+            if !folder_added && folder == &Config::ConfiguredInboxFolder {
+                let detailed = &state.get_detailed().await;
+                if let DetailedConnectivity::Error(_) = detailed {
+                    // On the inbox thread, we also do some other things like scan_folders and run jobs
+                    // so, maybe, the inbox is not watched, but something else went wrong
+                    ret += "<li>";
+                    ret += &*detailed.to_icon();
+                    ret += " ";
+                    ret += &*escaper::encode_minimal(&detailed.to_string_imap(self));
+                    ret += "</li>";
+                }
+            }
+        }
+        ret += "</ul>";
+
+        ret += "<h3>Outgoing messages</h3><ul><li>";
+        let detailed = smtp.get_detailed().await;
+        ret += &*detailed.to_icon();
+        ret += " ";
+        ret += &*escaper::encode_minimal(&detailed.to_string_smtp(self));
+        ret += "</li></ul>";
+
+        ret += "</body></html>\n";
+        ret
+    }
+
+    pub async fn all_work_done(&self) -> bool {
+        let lock = self.scheduler.read().await;
+        let stores: Vec<_> = match &*lock {
+            Scheduler::Running {
+                inbox,
+                mvbox,
+                sentbox,
+                smtp,
+                ..
+            } => [&inbox.state, &mvbox.state, &sentbox.state, &smtp.state]
+                .iter()
+                .map(|state| state.connectivity.clone())
+                .collect(),
+            Scheduler::Stopped => return false,
+        };
+        drop(lock);
+
+        for s in &stores {
+            if !s.get_all_work_done().await {
+                return false;
+            }
+        }
+        true
+    }
+}

src/securejoin.rs

@@ -17,7 +17,7 @@ use crate::context::Context;
 use crate::e2ee::ensure_secret_key_exists;
 use crate::events::EventType;
 use crate::headerdef::HeaderDef;
-use crate::key::{self, DcKey, Fingerprint, SignedPublicKey};
+use crate::key::{DcKey, Fingerprint, SignedPublicKey};
 use crate::message::Message;
 use crate::mimeparser::{MimeMessage, SystemMessage};
 use crate::param::Param;
@@ -253,17 +253,17 @@ async fn get_self_fingerprint(context: &Context) -> Option<Fingerprint> {
 
 #[derive(Debug, thiserror::Error)]
 pub enum JoinError {
-    #[error("Unknown QR-code")]
+    #[error("Unknown QR-code: {0}")]
     QrCode(#[from] QrError),
     #[error("A setup-contact/secure-join protocol is already running")]
     AlreadyRunning,
     #[error("An \"ongoing\" process is already running")]
     OngoingRunning,
-    #[error("Failed to send handshake message")]
+    #[error("Failed to send handshake message: {0}")]
     SendMessage(#[from] SendMsgError),
     // Note that this can currently only occur if there is a bug in the QR/Lot code as this
     // is supposed to create a contact for us.
-    #[error("Unknown contact (this is a bug)")]
+    #[error("Unknown contact (this is a bug): {0}")]
     UnknownContact(#[source] anyhow::Error),
     // Note that this can only occur if we failed to create the chat correctly.
     #[error("Ongoing sender dropped (this is a bug)")]
@@ -355,12 +355,6 @@ async fn securejoin(context: &Context, qr: &str) -> Result<ChatId, JoinError> {
 #[error("Failed sending handshake message")]
 pub struct SendMsgError(#[from] anyhow::Error);
 
-impl From<key::Error> for SendMsgError {
-    fn from(source: key::Error) -> Self {
-        Self(anyhow::Error::new(source))
-    }
-}
-
 async fn send_handshake_msg(
     context: &Context,
     contact_chat_id: ChatId,
@@ -507,7 +501,7 @@ pub(crate) async fn handle_securejoin_handshake(
                 )
             })?;
         if chat.blocked != Blocked::Not {
-            chat.id.unblock(context).await;
+            chat.id.unblock(context).await?;
         }
         chat.id
     };
@@ -802,7 +796,7 @@ pub(crate) async fn observe_securejoin_on_other_device(
                 )
             })?;
         if chat.blocked != Blocked::Not {
-            chat.id.unblock(context).await;
+            chat.id.unblock(context).await?;
         }
         chat.id
     };

src/securejoin/qrinvite.rs

@@ -9,7 +9,7 @@ use std::convert::TryFrom;
 
 use anyhow::Result;
 
-use crate::key::{Fingerprint, FingerprintError};
+use crate::key::Fingerprint;
 use crate::lot::{Lot, LotState};
 
 /// Represents the data from a QR-code scan.
@@ -103,8 +103,6 @@ impl TryFrom<Lot> for QrInvite {
 pub enum QrError {
     #[error("Unsupported protocol in QR-code")]
     UnsupportedProtocol,
-    #[error("Failed to read fingerprint")]
-    InvalidFingerprint(#[from] FingerprintError),
     #[error("Missing fingerprint")]
     MissingFingerprint,
     #[error("Missing invitenumber")]

src/simplify.rs

@@ -153,8 +153,8 @@ fn remove_bottom_quote<'a>(lines: &'a [&str]) -> (&'a [&'a str], Option<String>)
         let quoted_text = lines[l_last..first_quoted_line]
             .iter()
             .map(|s| {
-                s.strip_prefix(">")
-                    .map_or(*s, |u| u.strip_prefix(" ").unwrap_or(u))
+                s.strip_prefix('>')
+                    .map_or(*s, |u| u.strip_prefix(' ').unwrap_or(u))
             })
             .join("\n");
         if l_last > 1 && is_empty_line(lines[l_last - 1]) {
@@ -199,8 +199,8 @@ fn remove_top_quote<'a>(lines: &'a [&str]) -> (&'a [&'a str], Option<String>) {
                 lines[first_quoted_line..last_quoted_line + 1]
                     .iter()
                     .map(|s| {
-                        s.strip_prefix(">")
-                            .map_or(*s, |u| u.strip_prefix(" ").unwrap_or(u))
+                        s.strip_prefix('>')
+                            .map_or(*s, |u| u.strip_prefix(' ').unwrap_or(u))
                     })
                     .join("\n"),
             ),
@@ -247,8 +247,8 @@ fn render_message(lines: &[&str], is_cut_at_end: bool) -> String {
 fn is_empty_line(buf: &str) -> bool {
     buf.chars().all(char::is_whitespace)
     // for some time, this checked for `char <= ' '`,
-    // see discussion at: https://github.com/deltachat/deltachat-core-rust/pull/402#discussion_r317062392
-    // and https://github.com/deltachat/deltachat-core-rust/pull/2104/files#r538973613
+    // see discussion at: <https://github.com/deltachat/deltachat-core-rust/pull/402#discussion_r317062392>
+    // and <https://github.com/deltachat/deltachat-core-rust/pull/2104/files#r538973613>
 }
 
 fn is_quoted_headline(buf: &str) -> bool {
@@ -396,7 +396,7 @@ mod tests {
         assert!(!is_cut);
         assert_eq!(footer, None);
 
-        // Nonstandard footer sent by https://siju.es/
+        // Nonstandard footer sent by <https://siju.es/>
         let input = "Message text here\n---Desde mi teléfono con SIJÚ\n\nQuote here".to_string();
         let (plain, _, is_cut, _, footer) = simplify(input.clone(), false);
         assert_eq!(plain, "Message text here [...]");

src/smtp.rs

@@ -8,12 +8,11 @@ use async_smtp::smtp::client::net::ClientTlsParameters;
 use async_smtp::{error, smtp, EmailAddress};
 
 use crate::constants::DC_LP_AUTH_OAUTH2;
-use crate::context::Context;
 use crate::events::EventType;
 use crate::login_param::{dc_build_tls, CertificateChecks, LoginParam, ServerLoginParam};
 use crate::oauth2::dc_get_oauth2_access_token;
 use crate::provider::Socket;
-use crate::stock_str;
+use crate::{context::Context, scheduler::connectivity::ConnectivityStore};
 
 /// SMTP write and read timeout in seconds.
 const SMTP_TIMEOUT: u64 = 30;
@@ -28,12 +27,12 @@ pub enum Error {
         #[source]
         error: error::Error,
     },
-    #[error("SMTP: failed to connect: {0}")]
+    #[error("SMTP failed to connect: {0}")]
     ConnectionFailure(#[source] smtp::error::Error),
-    #[error("SMTP: failed to setup connection {0:?}")]
+    #[error("SMTP failed to setup connection: {0}")]
     ConnectionSetupFailure(#[source] smtp::error::Error),
-    #[error("SMTP: oauth2 error {address}")]
-    Oauth2Error { address: String },
+    #[error("SMTP oauth2 error {address}")]
+    Oauth2 { address: String },
     #[error("TLS error {0}")]
     Tls(#[from] async_native_tls::Error),
     #[error("{0}")]
@@ -53,6 +52,8 @@ pub(crate) struct Smtp {
     /// (eg connect or send succeeded). On initialization and disconnect
     /// it is set to None.
     last_success: Option<SystemTime>,
+
+    pub(crate) connectivity: ConnectivityStore,
 }
 
 impl Smtp {
@@ -97,6 +98,7 @@ impl Smtp {
             return Ok(());
         }
 
+        self.connectivity.set_connecting(context).await;
         let lp = LoginParam::from_database(context, "configured_").await?;
         let res = self
             .connect(
@@ -107,16 +109,10 @@ impl Smtp {
                 lp.provider.map_or(false, |provider| provider.strict_tls),
             )
             .await;
-        if let Err(ref err) = res {
-            let message = stock_str::server_response(
-                context,
-                format!("SMTP {}:{}", lp.smtp.server, lp.smtp.port),
-                err.to_string(),
-            )
-            .await;
 
-            context.emit_event(EventType::ErrorNetwork(message));
-        };
+        if let Err(err) = &res {
+            self.connectivity.set_err(context, err).await;
+        }
         res
     }
 
@@ -163,7 +159,7 @@ impl Smtp {
             let send_pw = &lp.password;
             let access_token = dc_get_oauth2_access_token(context, addr, send_pw, false).await?;
             if access_token.is_none() {
-                return Err(Error::Oauth2Error {
+                return Err(Error::Oauth2 {
                     address: addr.to_string(),
                 });
             }

src/smtp/send.rs

@@ -14,9 +14,9 @@ pub type Result<T> = std::result::Result<T, Error>;
 #[derive(Debug, thiserror::Error)]
 pub enum Error {
     #[error("Envelope error: {}", _0)]
-    EnvelopeError(#[from] async_smtp::error::Error),
+    Envelope(#[from] async_smtp::error::Error),
     #[error("Send error: {}", _0)]
-    SendError(#[from] async_smtp::smtp::error::Error),
+    SmtpSend(#[from] async_smtp::smtp::error::Error),
     #[error("SMTP has no transport")]
     NoTransport,
     #[error("{}", _0)]
@@ -46,8 +46,7 @@ impl Smtp {
             let recipients = recipients_chunk.to_vec();
             let recipients_display = recipients.iter().map(|x| x.to_string()).join(",");
 
-            let envelope =
-                Envelope::new(self.from.clone(), recipients).map_err(Error::EnvelopeError)?;
+            let envelope = Envelope::new(self.from.clone(), recipients).map_err(Error::Envelope)?;
             let mail = SendableEmail::new(
                 envelope,
                 format!("{}", job_id), // only used for internal logging
@@ -60,7 +59,7 @@ impl Smtp {
                 transport
                     .send_with_timeout(mail, Some(&Duration::from_secs(timeout)))
                     .await
-                    .map_err(Error::SendError)?;
+                    .map_err(Error::SmtpSend)?;
 
                 context.emit_event(EventType::SmtpMessageSent(format!(
                     "Message len={} was smtp-sent to {}",

src/sql.rs

@@ -198,7 +198,7 @@ impl Sql {
             }
         }
 
-        info!(context, "Opened {:?}.", dbfile);
+        info!(context, "Opened database {:?}.", dbfile);
 
         Ok(())
     }
@@ -770,7 +770,7 @@ mod test {
     /// existed and `PRAGMA` returned non-empty result.
     ///
     /// Statements were not finalized due to a bug in sqlx:
-    /// https://github.com/launchbadge/sqlx/issues/1147
+    /// <https://github.com/launchbadge/sqlx/issues/1147>
     #[async_std::test]
     async fn test_db_reopen() -> Result<()> {
         use tempfile::tempdir;
@@ -802,4 +802,36 @@ mod test {
 
         Ok(())
     }
+
+    #[async_std::test]
+    async fn test_migration_flags() -> Result<()> {
+        let t = TestContext::new().await;
+        t.evtracker.get_info_contains("Opened database").await;
+
+        // as migrations::run() was already executed on context creation,
+        // another call should not result in any action needed.
+        // this test catches some bugs where dbversion was forgotten to be persisted.
+        let (recalc_fingerprints, update_icons, disable_server_delete, recode_avatar) =
+            migrations::run(&t, &t.sql).await?;
+        assert!(!recalc_fingerprints);
+        assert!(!update_icons);
+        assert!(!disable_server_delete);
+        assert!(!recode_avatar);
+
+        info!(&t, "test_migration_flags: XXX");
+
+        loop {
+            if let EventType::Info(info) = t.evtracker.recv().await.unwrap() {
+                assert!(
+                    !info.contains("[migration]"),
+                    "Migrations were run twice, you probably forgot to update the db version"
+                );
+                if info.contains("test_migration_flags: XXX") {
+                    break;
+                }
+            }
+        }
+
+        Ok(())
+    }
 }

src/sql/migrations.rs

@@ -167,7 +167,7 @@ CREATE TABLE tokens (
 ALTER TABLE acpeerstates ADD COLUMN verified_key;
 ALTER TABLE acpeerstates ADD COLUMN verified_key_fingerprint TEXT DEFAULT '';
 CREATE INDEX acpeerstates_index5 ON acpeerstates (verified_key_fingerprint);"#,
-            38,
+            39,
         )
         .await?;
     }
@@ -454,7 +454,7 @@ paramsv![]
         info!(context, "[migration] v75");
         sql.execute_migration(
             "ALTER TABLE contacts ADD COLUMN status TEXT DEFAULT '';",
-            74,
+            75,
         )
         .await?;
     }
@@ -466,6 +466,14 @@ paramsv![]
     if dbversion < 77 {
         info!(context, "[migration] v77");
         recode_avatar = true;
+        sql.set_db_version(77).await?;
+    }
+    if dbversion < 78 {
+        // move requests to "Archived Chats",
+        // this way, the app looks familiar after the contact request upgrade.
+        info!(context, "[migration] v78");
+        sql.execute_migration("UPDATE chats SET archived=1 WHERE blocked=2;", 78)
+            .await?;
     }
 
     Ok((

src/sql/tables.sql

@@ -72,6 +72,10 @@ CREATE TABLE msgs (
     timestamp_sent INTEGER DEFAULT 0,
     timestamp_rcvd INTEGER DEFAULT 0,
     hidden INTEGER DEFAULT 0,
+    -- mime_headers column actually contains BLOBs, i.e. it may
+    -- contain non-UTF8 MIME messages.  TEXT was a bad choice, but
+    -- thanks to SQLite 3 being dynamically typed, there is no need to
+    -- change column type.
     mime_headers TEXT,
     mime_in_reply_to TEXT,
     mime_references TEXT,

src/stock_str.rs

@@ -39,9 +39,6 @@ pub enum StockMessage {
     #[strum(props(fallback = "Voice message"))]
     VoiceMessage = 7,
 
-    #[strum(props(fallback = "Contact requests"))]
-    DeadDrop = 8,
-
     #[strum(props(fallback = "Image"))]
     Image = 9,
 
@@ -130,9 +127,6 @@ pub enum StockMessage {
     ))]
     CannotLogin = 60,
 
-    #[strum(props(fallback = "Could not connect to %1$s: %2$s"))]
-    ServerResponse = 61,
-
     #[strum(props(fallback = "%1$s by %2$s."))]
     MsgActionByUser = 62,
 
@@ -363,11 +357,6 @@ pub(crate) async fn voice_message(context: &Context) -> String {
     translated(context, StockMessage::VoiceMessage).await
 }
 
-/// Stock string: `Contact requests`.
-pub(crate) async fn dead_drop(context: &Context) -> String {
-    translated(context, StockMessage::DeadDrop).await
-}
-
 /// Stock string: `Image`.
 pub(crate) async fn image(context: &Context) -> String {
     translated(context, StockMessage::Image).await
@@ -583,18 +572,6 @@ pub(crate) async fn cannot_login(context: &Context, user: impl AsRef<str>) -> St
         .replace1(user)
 }
 
-/// Stock string: `Could not connect to %1$s: %2$s`.
-pub(crate) async fn server_response(
-    context: &Context,
-    server: impl AsRef<str>,
-    details: impl AsRef<str>,
-) -> String {
-    translated(context, StockMessage::ServerResponse)
-        .await
-        .replace1(server)
-        .replace2(details)
-}
-
 /// Stock string: `%1$s by %2$s.`.
 pub(crate) async fn msg_action_by_user(
     context: &Context,
@@ -1000,10 +977,7 @@ mod tests {
     #[async_std::test]
     async fn test_stock_string_repl_str2() {
         let t = TestContext::new().await;
-        assert_eq!(
-            server_response(&t, "foo", "bar").await,
-            "Could not connect to foo: bar"
-        );
+        assert_eq!(msg_action_by_user(&t, "foo", "bar").await, "foo by bar.");
     }
 
     #[async_std::test]

src/test_utils.rs

@@ -9,6 +9,7 @@ use std::{collections::BTreeMap, panic};
 use std::{fmt, thread};
 
 use ansi_term::Color;
+use async_std::channel::Receiver;
 use async_std::path::PathBuf;
 use async_std::sync::{Arc, RwLock};
 use async_std::{channel, pin::Pin};
@@ -47,6 +48,7 @@ static CONTEXT_NAMES: Lazy<std::sync::RwLock<BTreeMap<u32, String>>> =
 pub(crate) struct TestContext {
     pub ctx: Context,
     pub dir: TempDir,
+    pub evtracker: EvTracker,
     /// Counter for fake IMAP UIDs in [recv_msg], for private use in that function only.
     recv_idx: RwLock<u32>,
     /// Functions to call for events received.
@@ -103,6 +105,8 @@ impl TestContext {
         let event_sinks: Arc<RwLock<Vec<Box<EventSink>>>> = Arc::new(RwLock::new(Vec::new()));
         let sinks = Arc::clone(&event_sinks);
         let (poison_sender, poison_receiver) = channel::bounded(1);
+        let (evtracker_sender, evtracker_receiver) = channel::unbounded();
+
         async_std::task::spawn(async move {
             // Make sure that the test fails if there is a panic on this thread here:
             let current_id = task::current().id();
@@ -122,13 +126,15 @@ impl TestContext {
                         sink(event.clone()).await;
                     }
                 }
-                receive_event(event);
+                receive_event(&event);
+                evtracker_sender.send(event.typ).await.ok();
             }
         });
 
         Self {
             ctx,
             dir,
+            evtracker: EvTracker(evtracker_receiver),
             recv_idx: RwLock::new(0),
             event_sinks,
             poison_receiver,
@@ -325,7 +331,7 @@ impl TestContext {
         // The chatlist describes what you see when you open DC, a list of chats and in each of them
         // the first words of the last message. To get the last message overall, we look at the chat at the top of the
         // list, which has the index 0.
-        let msg_id = chats.get_msg_id(0).unwrap();
+        let msg_id = chats.get_msg_id(0).unwrap().unwrap();
         Message::load_from_db(&self.ctx, msg_id)
             .await
             .expect("failed to load msg")
@@ -568,6 +574,28 @@ pub fn bob_keypair() -> key::KeyPair {
     }
 }
 
+pub struct EvTracker(Receiver<EventType>);
+
+impl EvTracker {
+    pub async fn get_info_contains(&self, s: &str) -> EventType {
+        loop {
+            let event = self.0.recv().await.unwrap();
+            if let EventType::Info(i) = &event {
+                if i.contains(s) {
+                    return event;
+                }
+            }
+        }
+    }
+}
+
+impl Deref for EvTracker {
+    type Target = Receiver<EventType>;
+    fn deref(&self) -> &Self::Target {
+        &self.0
+    }
+}
+
 /// Gets a specific message from a chat and asserts that the chat has a specific length.
 ///
 /// Panics if the length of the chat is not `asserted_msgs_count` or if the chat item at `index` is not a Message.
@@ -591,19 +619,18 @@ pub(crate) async fn get_chat_msg(
 /// Pretty-print an event to stdout
 ///
 /// Done during tests this is captured by `cargo test` and associated with the test itself.
-fn receive_event(event: Event) {
+fn receive_event(event: &Event) {
     let green = Color::Green.normal();
     let yellow = Color::Yellow.normal();
     let red = Color::Red.normal();
 
-    let msg = match event.typ {
+    let msg = match &event.typ {
         EventType::Info(msg) => format!("INFO: {}", msg),
         EventType::SmtpConnected(msg) => format!("[SMTP_CONNECTED] {}", msg),
         EventType::ImapConnected(msg) => format!("[IMAP_CONNECTED] {}", msg),
         EventType::SmtpMessageSent(msg) => format!("[SMTP_MESSAGE_SENT] {}", msg),
         EventType::Warning(msg) => format!("WARN: {}", yellow.paint(msg)),
         EventType::Error(msg) => format!("ERROR: {}", red.paint(msg)),
-        EventType::ErrorNetwork(msg) => format!("{}", red.paint(format!("[NETWORK] msg={}", msg))),
         EventType::ErrorSelfNotInGroup(msg) => {
             format!("{}", red.paint(format!("[SELF_NOT_IN_GROUP] {}", msg)))
         }

src/token.rs

@@ -28,11 +28,13 @@ impl Default for Namespace {
     }
 }
 
-/// Creates a new token and saves it into the database.
-///
-/// Returns created token.
-pub async fn save(context: &Context, namespace: Namespace, foreign_id: Option<ChatId>) -> String {
-    let token = dc_create_id();
+/// Saves a token to the database.
+pub async fn save(
+    context: &Context,
+    namespace: Namespace,
+    foreign_id: Option<ChatId>,
+    token: &str,
+) -> Result<()> {
     match foreign_id {
         Some(foreign_id) => context
             .sql
@@ -40,21 +42,29 @@ pub async fn save(context: &Context, namespace: Namespace, foreign_id: Option<Ch
                 "INSERT INTO tokens (namespc, foreign_id, token, timestamp) VALUES (?, ?, ?, ?);",
                 paramsv![namespace, foreign_id, token, time()],
             )
-            .await
-            .ok(),
-        None => context
-            .sql
-            .execute(
-                "INSERT INTO tokens (namespc, token, timestamp) VALUES (?, ?, ?);",
-                paramsv![namespace, token, time()],
-            )
-            .await
-            .ok(),
+            .await?,
+        None => {
+            context
+                .sql
+                .execute(
+                    "INSERT INTO tokens (namespc, token, timestamp) VALUES (?, ?, ?);",
+                    paramsv![namespace, token, time()],
+                )
+                .await?
+        }
     };
 
-    token
+    Ok(())
 }
 
+/// Lookup most recently created token for a namespace/chat combination.
+///
+/// As there may be more than one valid token for a chat-id,
+/// (eg. when a qr code token is withdrawn, recreated and revived later),
+/// use lookup() for qr-code creation only;
+/// do not use lookup() to check for token validity.
+///
+/// To check if a given token is valid, use exists().
 pub async fn lookup(
     context: &Context,
     namespace: Namespace,
@@ -65,7 +75,7 @@ pub async fn lookup(
             context
                 .sql
                 .query_get_value(
-                    "SELECT token FROM tokens WHERE namespc=? AND foreign_id=?;",
+                    "SELECT token FROM tokens WHERE namespc=? AND foreign_id=? ORDER BY timestamp DESC LIMIT 1;",
                     paramsv![namespace, chat_id],
                 )
                 .await?
@@ -75,7 +85,7 @@ pub async fn lookup(
             context
                 .sql
                 .query_get_value(
-                    "SELECT token FROM tokens WHERE namespc=? AND foreign_id=0;",
+                    "SELECT token FROM tokens WHERE namespc=? AND foreign_id=0 ORDER BY timestamp DESC LIMIT 1;",
                     paramsv![namespace],
                 )
                 .await?
@@ -93,7 +103,9 @@ pub async fn lookup_or_new(
         return token;
     }
 
-    save(context, namespace, foreign_id).await
+    let token = dc_create_id();
+    save(context, namespace, foreign_id, &token).await.ok();
+    token
 }
 
 pub async fn exists(context: &Context, namespace: Namespace, token: &str) -> bool {
@@ -106,3 +118,14 @@ pub async fn exists(context: &Context, namespace: Namespace, token: &str) -> boo
         .await
         .unwrap_or_default()
 }
+
+pub async fn delete(context: &Context, namespace: Namespace, token: &str) -> Result<()> {
+    context
+        .sql
+        .execute(
+            "DELETE FROM tokens WHERE namespc=? AND token=?;",
+            paramsv![namespace, token],
+        )
+        .await?;
+    Ok(())
+}

standards.md

@@ -13,7 +13,7 @@ Push                             | IMAP IDLE ([RFC 2177](https://tools.ietf.org/
 Authorization                    | OAuth2 ([RFC 6749](https://tools.ietf.org/html/rfc6749))
 End-to-end encryption            | [Autocrypt Level 1](https://autocrypt.org/level1.html), OpenPGP ([RFC 4880](https://tools.ietf.org/html/rfc4880)), Security Multiparts for MIME ([RFC 1847](https://tools.ietf.org/html/rfc1847)) and [“Mixed Up” Encryption repairing](https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html)
 Configuration assistance         | [Autoconfigure](https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration) and [Autodiscover](https://technet.microsoft.com/library/bb124251(v=exchg.150).aspx)
-Messenger functions              | [Chat-over-Email](https://github.com/deltachat/deltachat-core-rust/blob/master/spec.md#chat-over-email-specification)
+Messenger functions              | [Chat-over-Email](https://github.com/deltachat/deltachat-core-rust/blob/master/spec.md#chat-mail-specification)
 Detect mailing list              | List-Id ([RFC 2919](https://tools.ietf.org/html/rfc2919)) and Precedence ([RFC 3834](https://tools.ietf.org/html/rfc3834))
 User and chat colors             | [XEP-0392](https://xmpp.org/extensions/xep-0392.html): Consistent Color Generation
 Send and receive system messages | Multipart/Report Media Type ([RFC 6522](https://tools.ietf.org/html/rfc6522))

test-data/message/cp1252-html.eml

@@ -0,0 +1,18 @@
+Subject: test non-utf8 and dc_get_msg_html()
+To: tunis4 <tunis4@testrun.org>
+From: "B. Petersen" <bpetersen@b44t.com>
+Message-ID: <00007126-1efa-290b-2120-200251f50f23@b44t.com>
+Date: Thu, 27 May 2021 17:33:16 +0200
+MIME-Version: 1.0
+Content-Type: text/plain; charset=windows-1252; format=flowed
+Content-Language: en-US
+Content-Transfer-Encoding: 8bit
+
+foo bar <20> <20> <20> <20>
+
+
+-------- Forwarded Message --------
+Subject: 	Foo
+Date: 	Fri, 21 May 2021 08:42:20 +0000
+
+just to force a "more button"