Compare commits

...

43 Commits

Author SHA1 Message Date
link2xt
03bb92c942 chore(release): prepare for 1.126.1 2023-10-24 14:30:02 +00:00
link2xt
b0da5a54cc chore(cargo): update ahash to make cargo-deny happy 2023-10-23 21:56:14 +00:00
link2xt
44e056e210 ci: build Windows wheels for deltachat-rpc-server 2023-10-23 20:38:58 +00:00
link2xt
48a8680ba4 build: set 755 permission on binaries inside deltachat-rpc-server .whl 2023-10-23 20:20:28 +00:00
link2xt
b73bcc2c22 ci: build macOS wheels for deltachat-rpc-server 2023-10-23 20:20:14 +00:00
link2xt
cff42936aa ci: add missing .exe suffix to windows deltachat-rpc-server paths 2023-10-23 20:19:35 +00:00
link2xt
d3b04004b4 feat(imap): buffer STARTTLS command
Using BufWriter ensures that `STARTTLS` command is sent
as a single packet.

Also refactor the code to ensure we only convert to
Box<dyn SessionStream> in the end.
2023-10-23 16:26:29 +00:00
link2xt
5cd92f10ef ci: build deltachat-rpc-server binaries for aarch64 macOS 2023-10-23 16:02:36 +00:00
link2xt
22a3ab983b refactor: download messages without jobs 2023-10-23 11:52:47 +00:00
link2xt
83d2e6b8b4 fix: do not interrupt IMAP loop from get_connectivity_html()
Android calls get_connectivity_html()
every time connectivity changes, which in turn interrupts
IMAP loop and triggers change from "not connected" to "connecting"
state.

To avoid such infinite loop of IMAP interrupts when
there is not connectivity, update quota only when IMAP
loop is interrupted otherwise. This anyway happens
when a message is received or maybe_network is called.

Also remove outdated comments about `Action::UpdateRecentQuota` job
which does not exist anymore.
2023-10-22 18:48:14 +00:00
link2xt
4e979c5880 build: make source package for deltachat-rpc-server install fixed version 2023-10-22 17:49:03 +00:00
link2xt
71e1089139 fix: do not hardcode version in deltachat-rpc-server source package 2023-10-22 16:46:26 +00:00
link2xt
349c154a99 ci: attempt to fix artifacts download for deltachat-rpc-server
Had to do this step manually for 1.126.0,
hopefully the step will work automatically for 1.127.0.
2023-10-22 15:24:02 +00:00
link2xt
4e08bb7b05 chore(release): prepare for 1.126.0 2023-10-22 14:01:07 +00:00
Asiel Díaz Benítez
934ca6a7d7 api: add send_draft() to JSON-RPC API (#4839) 2023-10-22 13:14:08 +00:00
link2xt
e878caebe3 ci: build Python wheels for deltachat-rpc-server 2023-10-22 13:13:04 +00:00
link2xt
088eda2983 build(scripts/zig-rpc-server.sh): move built binaries to dist/ 2023-10-22 13:13:04 +00:00
link2xt
418cd24979 build: strip release binaries
This significantly reduces binary size for deltachat-rpc-server,
from 42 MiB to 15 MiB in case of aarch64.
2023-10-22 13:13:04 +00:00
link2xt
b4fe9e3eec build: use Zig via ziglang PyPI package
This avoids cluttering workdir with unpacked tarball
and will work automatically once PEP-723 is implemented.
2023-10-22 13:13:04 +00:00
link2xt
c13bbd05cd chore: add dist/ to .gitignore 2023-10-22 13:13:04 +00:00
link2xt
58330fe8b2 ci: remove musl check
It requires installing Zig and slows down CI.
This was not broken for a while and we can do
more frequent core releases to catch problems.
2023-10-22 13:13:04 +00:00
link2xt
680d024b05 docs: document scripts/codespell.sh 2023-10-22 06:53:44 +00:00
link2xt
defcd5764b chore: spellcheck 2023-10-22 06:53:26 +00:00
link2xt
0227bbc305 fix(imap): fallback to STATUS if SELECT did not return UIDNEXT
Winmail Pro Mail Server 5.1.0616 does not return UIDNEXT
in response to SELECT, but returns it when explicitly requested
via STATUS command:

    ? SELECT INBOX
    * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
    * OK [PERMANENTFLAGS (\Draft \Answered \Flagged \Deleted \Seen)] Limited
    * 2 EXISTS
    * 0 RECENT
    * OK [UIDVALIDITY 1697802109] Ok
    ? OK [READ-WRITE] Ok SELECT completed
    ? STATUS INBOX (UIDNEXT)
    * STATUS "INBOX" (UIDNEXT 4)
    ? OK STATUS completed

Previously used FETCH method is reported to fail for some users,
the FETCH command sometimes returns no results.
Besides, there is no guarantee that the message with
the highest sequence number has the highest UID.

In the worst case if STATUS does not return UIDNEXT
in response to explicit request, we fall back to setting
UIDNEXT to 1 instead of returning an error.
2023-10-20 22:29:44 +00:00
link2xt
d05afec289 chore(cargo): update async-imap to fix STATUS command 2023-10-20 22:29:44 +00:00
link2xt
64035d3ecb fix: set soft_heap_limit on SQLite database
This should prevent unlimited growth of memory usage
by SQLite for is page cache.
2023-10-20 05:18:21 +00:00
link2xt
21e0bb28ad build: create source distribution for deltachat-rpc-server 2023-10-19 14:38:03 +00:00
link2xt
c6358169ad fix: s/env/venv/ in scripts/make-python-testenv.sh 2023-10-18 20:10:51 +00:00
Asiel Díaz Benítez
955f4fbb19 add self-address to backup filename (#4820)
close #4816

---------

Co-authored-by: B. Petersen <r10s@b44t.com>
2023-10-18 12:31:32 -04:00
adbenitez
df7c44ae42 fix: wrong type hint 2023-10-17 17:18:25 +02:00
Hocuri
8573649bf7 feat: Make broadcast lists create their own chat (#4644)
feat: Make broadcast lists create their own chat - UIs need to ask for
the name when creating broadcast lists now (see
https://github.com/deltachat/deltachat-android/pull/2653)

That's quite a minimal approach: Add a List-ID header to outgoing
broadcast lists, so that the receiving Delta Chat shows them as a
separate chat, as talked about with @r10s and @hpk42.

Done:
- [x] Fix an existing bug that the chat name isn't updated when the
broadcast/mailing list name changes (I already started this locally)

To be done in other PRs:
- [ ] Right now the receiving side shows "Mailing list" in the subtitle
of such a chat, it would be nicer if it showed "Broadcast list" (or
alternatively, rename "Broadcast list" to "Mailing list", too)
- [ ] The UIs should probably ask for a name before creating the
broadcast list, since it will actually be sent over the wire. (Android
PR: https://github.com/deltachat/deltachat-android/pull/2653)

Fixes https://github.com/deltachat/deltachat-core-rust/issues/4597

BREAKING CHANGE: This means that UIs need to ask for the name when creating a broadcast list, similar to https://github.com/deltachat/deltachat-android/pull/2653.
2023-10-17 10:40:47 +02:00
link2xt
52c46c6dca build: add script to build deltachat-rpc-server wheels 2023-10-16 17:04:55 +00:00
link2xt
54ea3ec5d6 build: workaround OpenSSL crate expecting libatomic to be available 2023-10-15 22:57:46 +00:00
Sebastian Klähn
b239535964 api: allow to filter by unread in chatlist:try_load (#4824)
close #4738
2023-10-14 11:12:53 +02:00
link2xt
da5d844ec4 chore: rustfmt 2023-10-14 04:25:13 +00:00
link2xt
a87635dcf4 chore(release): prepare for 1.125.0 2023-10-14 04:21:48 +00:00
link2xt
e30517e62c refactor: log MDN sending errors 2023-10-14 03:29:20 +00:00
iequidoo
a54f3c4b31 fix: Don't try to send more MDNs if there's a tmp SMTP error (#4534)
If there's a temporary SMTP error, pretend there are no more MDNs to send in send_mdn(). It's
unlikely that other MDNs could be sent successfully in case of connectivity problems. This approach
is simpler and perhaps even better than adding a progressive backoff between MDN sending retries --
MDNs just will be resent after a reconnection, which makes some sense.
2023-10-12 19:19:43 -03:00
iequidoo
bda6cea0ce feat: Make gossip period configurable (#4346)
This is needed to test periodic re-gossiping in existing chats.

Also add a test for verified groups on that even if "member added" message is missed by a device of
newly added member, after re-gossiping Autocrypt keys to the group it successfully learns these keys
and marks other members as verified.
2023-10-12 05:45:20 -03:00
iequidoo
6fece09ed7 test: test_qr_new_group_unblocked(): W/a message reordering on server
There was a recent failure of the test probably as a result of message reordering on the server:
https://github.com/deltachat/deltachat-core-rust/actions/runs/6464605602/job/17549624095?pr=4813.
2023-10-11 04:54:03 -03:00
link2xt
3917c6b2f0 test(deltachat-rpc-client): enable logs in pytest
This makes pytest setup a logger for `logging` module.
2023-10-10 19:20:42 +00:00
link2xt
96a89b5bdc fix: return verifier contacts regardless of their origin
Previously `Origin::AddressBook` was required,
resulting in a lot of
"Could not lookup contact with address ... which introduced ..."
warnings.
2023-10-10 19:20:22 +00:00
link2xt
b55027fe71 fix: set connectivity status to "connected" during fake idle
Set connectivity status to "connected" at the end of connect() call.
Otherwise for servers that do not support IMAP IDLE
connect() is called at the beginning of fake idle
and connectivity stays in "connecting" status most of the time.
2023-10-10 05:37:51 +00:00
54 changed files with 1021 additions and 959 deletions

View File

@@ -39,10 +39,6 @@ jobs:
- name: Check
run: cargo check --workspace --all-targets --all-features
# Check with musl libc target which is used for `deltachat-rpc-server` releases.
- name: Check musl
run: scripts/zig-musl-check.sh
cargo_deny:
name: cargo deny
runs-on: ubuntu-latest

View File

@@ -26,35 +26,17 @@ jobs:
steps:
- uses: actions/checkout@v3
- name: Build
- name: Install ziglang
run: pip install wheel ziglang==0.11.0
- name: Build deltachat-rpc-server binaries
run: sh scripts/zig-rpc-server.sh
- name: Upload x86_64 binary
- name: Upload dist directory with Linux binaries
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-x86_64
path: target/x86_64-unknown-linux-musl/release/deltachat-rpc-server
if-no-files-found: error
- name: Upload i686 binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-i686
path: target/i686-unknown-linux-musl/release/deltachat-rpc-server
if-no-files-found: error
- name: Upload aarch64 binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-aarch64
path: target/aarch64-unknown-linux-musl/release/deltachat-rpc-server
if-no-files-found: error
- name: Upload armv7 binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-armv7
path: target/armv7-unknown-linux-musleabihf/release/deltachat-rpc-server
name: linux
path: dist/
if-no-files-found: error
build_windows:
@@ -92,39 +74,87 @@ jobs:
build_macos:
name: Build deltachat-rpc-server for macOS
strategy:
fail-fast: false
matrix:
include:
- arch: x86_64
- arch: aarch64
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Setup rust target
run: rustup target add x86_64-apple-darwin
run: rustup target add ${{ matrix.arch }}-apple-darwin
- name: Build
run: cargo build --release --package deltachat-rpc-server --target x86_64-apple-darwin --features vendored
run: cargo build --release --package deltachat-rpc-server --target ${{ matrix.arch }}-apple-darwin --features vendored
- name: Upload binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-x86_64-macos
path: target/x86_64-apple-darwin/release/deltachat-rpc-server
name: deltachat-rpc-server-${{ matrix.arch }}-macos
path: target/${{ matrix.arch }}-apple-darwin/release/deltachat-rpc-server
if-no-files-found: error
publish:
name: Upload binaries to the release
name: Build wheels and upload binaries to the release
needs: ["build_linux", "build_windows", "build_macos"]
permissions:
contents: write
runs-on: "ubuntu-latest"
steps:
- name: Download built binaries
uses: "actions/download-artifact@v3"
- uses: actions/checkout@v3
- name: Compose dist/ directory
- name: Download Linux binaries
uses: actions/download-artifact@v3
with:
name: linux
path: dist/
- name: Download win32 binary
uses: actions/download-artifact@v3
with:
name: deltachat-rpc-server-win32.exe
path: deltachat-rpc-server-win32.exe.d
- name: Download win64 binary
uses: actions/download-artifact@v3
with:
name: deltachat-rpc-server-win64.exe
path: deltachat-rpc-server-win64.exe.d
- name: Download macOS binary for x86_64
uses: actions/download-artifact@v3
with:
name: deltachat-rpc-server-x86_64-macos
path: deltachat-rpc-server-x86_64-macos.d
- name: Download macOS binary for aarch64
uses: actions/download-artifact@v3
with:
name: deltachat-rpc-server-aarch64-macos
path: deltachat-rpc-server-aarch64-macos.d
- name: Flatten dist/ directory
run: |
mkdir dist
for x in x86_64 i686 aarch64 armv7 win32.exe win64.exe x86_64-macos; do
mv "deltachat-rpc-server-$x"/* "dist/deltachat-rpc-server-$x"
done
mv deltachat-rpc-server-win32.exe.d/deltachat-rpc-server.exe dist/deltachat-rpc-server-win32.exe
mv deltachat-rpc-server-win64.exe.d/deltachat-rpc-server.exe dist/deltachat-rpc-server-win64.exe
mv deltachat-rpc-server-x86_64-macos.d/deltachat-rpc-server dist/deltachat-rpc-server-x86_64-macos
mv deltachat-rpc-server-aarch64-macos.d/deltachat-rpc-server dist/deltachat-rpc-server-aarch64-macos
# Python 3.11 is needed for tomllib used in scripts/wheel-rpc-server.py
- name: Install python 3.12
uses: actions/setup-python@v4
with:
python-version: 3.12
- name: Install wheel
run: pip install wheel
- name: Build deltachat-rpc-server Python wheels and source package
run: scripts/wheel-rpc-server.py
- name: List downloaded artifacts
run: ls -l dist/

1
.gitignore vendored
View File

@@ -1,6 +1,7 @@
/target
**/*.rs.bk
/build
/dist
# ignore vi temporaries
*~

View File

@@ -1,5 +1,91 @@
# Changelog
## [1.126.1] - 2023-10-24
### Fixes
- Do not hardcode version in deltachat-rpc-server source package.
- Do not interrupt IMAP loop from `get_connectivity_html()`.
### Features / Changes
- imap: Buffer `STARTTLS` command.
### Build system
- Build `deltachat-rpc-server` binary for aarch64 macOS.
- Build `deltachat-rpc-server` wheels for macOS and Windows.
### Refactor
- Remove job queue.
### Miscellaneous Tasks
- cargo: Update `ahash` to make `cargo-deny` happy.
## [1.126.0] - 2023-10-22
### API-Changes
- Allow to filter by unread in `chatlist:try_load` ([#4824](https://github.com/deltachat/deltachat-core-rust/pull/4824)).
- Add `misc_send_draft()` to JSON-RPC API ([#4839](https://github.com/deltachat/deltachat-core-rust/pull/4839)).
### Features / Changes
- [**breaking**] Make broadcast lists create their own chat ([#4644](https://github.com/deltachat/deltachat-core-rust/pull/4644)).
- This means that UIs need to ask for the name when creating a broadcast list, similar to <https://github.com/deltachat/deltachat-android/pull/2653>.
- Add self-address to backup filename ([#4820](https://github.com/deltachat/deltachat-core-rust/pull/4820))
### CI
- Build Python wheels for deltachat-rpc-server.
### Build system
- Strip release binaries.
- Workaround OpenSSL crate expecting libatomic to be available.
### Fixes
- Set `soft_heap_limit` on SQLite database.
- imap: Fallback to `STATUS` if `SELECT` did not return UIDNEXT.
## [1.125.0] - 2023-10-14
### API-Changes
- [**breaking**] deltachat-rpc-client: Replace `asyncio` with threads.
- Validate boolean values passed to `set_config`. Attempts to set values other than `0` and `1` will result in an error.
### CI
- Reduce required Python version for deltachat-rpc-client from 3.8 to 3.7.
### Features / Changes
- Add developer option to disable IDLE.
### Fixes
- `deltachat-rpc-client`: Run `deltachat-rpc-server` in its own process group. This prevents reception of `SIGINT` by the server when the bot is terminated with `^C`.
- python: Don't automatically set the displayname to "bot" when setting log level.
- Don't update `timestamp`, `timestamp_rcvd`, `state` when replacing partially downloaded message ([#4700](https://github.com/deltachat/deltachat-core-rust/pull/4700)).
- Assign encrypted partially downloaded group messages to 1:1 chat ([#4757](https://github.com/deltachat/deltachat-core-rust/pull/4757)).
- Return all contacts from `Contact::get_all` for bots ([#4811](https://github.com/deltachat/deltachat-core-rust/pull/4811)).
- Set connectivity status to "connected" during fake idle.
- Return verifier contacts regardless of their origin.
- Don't try to send more MDNs if there's a temporary SMTP error ([#4534](https://github.com/deltachat/deltachat-core-rust/pull/4534)).
### Refactor
- deltachat-rpc-client: Close stdin instead of sending `SIGTERM`.
- deltachat-rpc-client: Remove print() calls. Standard `logging` package is for logging instead.
### Tests
- deltachat-rpc-client: Enable logs in pytest.
## [1.124.1] - 2023-10-05
### Fixes
@@ -2879,3 +2965,5 @@ https://github.com/deltachat/deltachat-core-rust/pulls?q=is%3Apr+is%3Aclosed
[1.123.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.122.0...v1.123.0
[1.124.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.123.0...v1.124.0
[1.124.1]: https://github.com/deltachat/deltachat-core-rust/compare/v1.124.0...v1.124.1
[1.125.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.124.1...v1.125.0
[1.126.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.125.0...v1.126.0

71
Cargo.lock generated
View File

@@ -43,24 +43,14 @@ dependencies = [
[[package]]
name = "ahash"
version = "0.7.6"
version = "0.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fcb51a0695d8f838b1ee009b3fbf66bda078cd64590202a864a8f3e8c4315c47"
dependencies = [
"getrandom 0.2.10",
"once_cell",
"version_check",
]
[[package]]
name = "ahash"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c99f64d1e06488f620f932677e24bc6e2897582980441ae90a671415bd7ec2f"
checksum = "cd7d5a2cecb58716e47d67d5703a249964b14c7be1ec3cad3affc295b2d1c35d"
dependencies = [
"cfg-if",
"once_cell",
"version_check",
"zerocopy",
]
[[package]]
@@ -221,9 +211,9 @@ dependencies = [
[[package]]
name = "async-imap"
version = "0.9.1"
version = "0.9.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b538b767cbf9c162a6c5795d4b932bd2c20ba10b5a91a94d2b2b6886c1dce6a8"
checksum = "936c1b580be4373b48c9c687e0c79285441664398354df28d0860087cac0c069"
dependencies = [
"async-channel",
"base64 0.21.2",
@@ -1103,7 +1093,7 @@ dependencies = [
[[package]]
name = "deltachat"
version = "1.124.1"
version = "1.126.1"
dependencies = [
"ansi_term",
"anyhow",
@@ -1179,7 +1169,7 @@ dependencies = [
[[package]]
name = "deltachat-jsonrpc"
version = "1.124.1"
version = "1.126.1"
dependencies = [
"anyhow",
"async-channel",
@@ -1203,7 +1193,7 @@ dependencies = [
[[package]]
name = "deltachat-repl"
version = "1.124.1"
version = "1.126.1"
dependencies = [
"ansi_term",
"anyhow",
@@ -1218,7 +1208,7 @@ dependencies = [
[[package]]
name = "deltachat-rpc-server"
version = "1.124.1"
version = "1.126.1"
dependencies = [
"anyhow",
"deltachat",
@@ -1243,7 +1233,7 @@ dependencies = [
[[package]]
name = "deltachat_ffi"
version = "1.124.1"
version = "1.126.1"
dependencies = [
"anyhow",
"deltachat",
@@ -2171,17 +2161,14 @@ name = "hashbrown"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888"
dependencies = [
"ahash 0.7.6",
]
[[package]]
name = "hashbrown"
version = "0.14.0"
version = "0.14.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c6201b9ff9fd90a5a3bac2e56a830d0caa509576f0e503818ee82c181b3437a"
checksum = "f93e7192158dbcda357bdec5fb5788eebf8bbac027f3f33e719d29135ae84156"
dependencies = [
"ahash 0.8.3",
"ahash",
"allocator-api2",
]
@@ -2191,7 +2178,7 @@ version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "312f66718a2d7789ffef4f4b7b213138ed9f1eb3aa1d0d82fc99f88fb3ffd26f"
dependencies = [
"hashbrown 0.14.0",
"hashbrown 0.14.2",
]
[[package]]
@@ -2451,7 +2438,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d5477fe2230a79769d8dc68e0eabf5437907c0457a5614a9e8dddb67f65eb65d"
dependencies = [
"equivalent",
"hashbrown 0.14.0",
"hashbrown 0.14.2",
]
[[package]]
@@ -5235,13 +5222,9 @@ checksum = "22049a19f4a68748a168c0fc439f9516686aa045927ff767eca0a85101fb6e73"
[[package]]
name = "unicode-linebreak"
version = "0.1.4"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c5faade31a542b8b35855fff6e8def199853b2da8da256da52f52f1316ee3137"
dependencies = [
"hashbrown 0.12.3",
"regex",
]
checksum = "3b09c83c3c29d37506a3e260c08c03743a6bb66a9cd432c6934ab501a190571f"
[[package]]
name = "unicode-normalization"
@@ -5810,6 +5793,26 @@ dependencies = [
"syn 1.0.109",
]
[[package]]
name = "zerocopy"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a7af71d8643341260a65f89fa60c0eeaa907f34544d8f6d9b0df72f069b5e74"
dependencies = [
"zerocopy-derive",
]
[[package]]
name = "zerocopy-derive"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9731702e2f0617ad526794ae28fbc6f6ca8849b5ba729666c2a5bc4b6ddee2cd"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.23",
]
[[package]]
name = "zeroize"
version = "1.6.0"

View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat"
version = "1.124.1"
version = "1.126.1"
edition = "2021"
license = "MPL-2.0"
rust-version = "1.65"
@@ -24,6 +24,7 @@ lto = true
panic = 'abort'
opt-level = "z"
codegen-units = 1
strip = true
[patch.crates-io]
quinn-udp = { git = "https://github.com/quinn-rs/quinn", branch="main" }

View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat_ffi"
version = "1.124.1"
version = "1.126.1"
description = "Deltachat FFI"
edition = "2018"
readme = "README.md"

View File

@@ -503,6 +503,9 @@ char* dc_get_blobdir (const dc_context_t* context);
* to not mess up with non-delivery-reports or read-receipts.
* 0=no limit (default).
* Changes affect future messages only.
* - `gossip_period` = How often to gossip Autocrypt keys in chats with multiple recipients, in
* seconds. 2 days by default.
* This is not supposed to be changed by UIs and only used for testing.
* - `ui.*` = All keys prefixed by `ui.` can be used by the user-interfaces for system-specific purposes.
* The prefix should be followed by the system and maybe subsystem,
* e.g. `ui.desktop.foo`, `ui.desktop.linux.bar`, `ui.android.foo`, `ui.dc40.bar`, `ui.bot.simplebot.baz`.
@@ -881,7 +884,8 @@ int dc_preconfigure_keypair (dc_context_t* context, const cha
* - if the flag DC_GCL_ADD_ALLDONE_HINT is set, DC_CHAT_ID_ALLDONE_HINT
* is added as needed.
* @param query_str An optional query for filtering the list. Only chats matching this query
* are returned. Give NULL for no filtering.
* are returned. Give NULL for no filtering. When `is:unread` is contained in the query,
* the chatlist is filtered such that only chats with unread messages show up.
* @param query_id An optional contact ID for filtering the list. Only chats including this contact ID
* are returned. Give 0 for no filtering.
* @return A chatlist as an dc_chatlist_t object.
@@ -1714,24 +1718,12 @@ uint32_t dc_create_group_chat (dc_context_t* context, int protect
* Create a new broadcast list.
*
* Broadcast lists are similar to groups on the sending device,
* however, recipients get the messages in normal one-to-one chats
* and will not be aware of other members.
* however, recipients get the messages in a read-only chat
* and will see who the other members are.
*
* Replies to broadcasts go only to the sender
* and not to all broadcast recipients.
* Moreover, replies will not appear in the broadcast list
* but in the one-to-one chat with the person answering.
*
* The name and the image of the broadcast list is set automatically
* and is visible to the sender only.
* Not asking for these data allows more focused creation
* and we bypass the question who will get which data.
* Also, many users will have at most one broadcast list
* so, a generic name and image is sufficient at the first place.
*
* Later on, however, the name can be changed using dc_set_chat_name().
* The image cannot be changed to have a unique, recognizable icon in the chat lists.
* All in all, this is also what other messengers are doing here.
* For historical reasons, this function does not take a name directly,
* instead you have to set the name using dc_set_chat_name()
* after creating the broadcast list.
*
* @memberof dc_context_t
* @param context The context object.
@@ -2274,8 +2266,7 @@ dc_contact_t* dc_get_contact (dc_context_t* context, uint32_t co
* the backup is not encrypted.
* The backup contains all contacts, chats, images and other data and device independent settings.
* The backup does not contain device dependent settings as ringtones or LED notification settings.
* The name of the backup is typically `delta-chat-<day>.tar`, if more than one backup is create on a day,
* the format is `delta-chat-<day>-<number>.tar`
* The name of the backup is `delta-chat-backup-<day>-<number>-<addr>.tar`.
*
* - **DC_IMEX_IMPORT_BACKUP** (12) - `param1` is the file (not: directory) to import. `param2` is the passphrase.
* The file is normally created by DC_IMEX_EXPORT_BACKUP and detected by dc_imex_has_backup(). Importing a backup
@@ -3959,7 +3950,7 @@ int64_t dc_msg_get_received_timestamp (const dc_msg_t* msg);
* Get the message time used for sorting.
* This function returns the timestamp that is used for sorting the message
* into lists as returned e.g. by dc_get_chat_msgs().
* This may be the reveived time, the sending time or another time.
* This may be the received time, the sending time or another time.
*
* To get the receiving time, use dc_msg_get_received_timestamp().
* To get the sending time, use dc_msg_get_timestamp().

View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat-jsonrpc"
version = "1.124.1"
version = "1.126.1"
description = "DeltaChat JSON-RPC API"
edition = "2021"
default-run = "deltachat-jsonrpc-server"

View File

@@ -812,24 +812,12 @@ impl CommandApi {
/// Create a new broadcast list.
///
/// Broadcast lists are similar to groups on the sending device,
/// however, recipients get the messages in normal one-to-one chats
/// and will not be aware of other members.
/// however, recipients get the messages in a read-only chat
/// and will see who the other members are.
///
/// Replies to broadcasts go only to the sender
/// and not to all broadcast recipients.
/// Moreover, replies will not appear in the broadcast list
/// but in the one-to-one chat with the person answering.
///
/// The name and the image of the broadcast list is set automatically
/// and is visible to the sender only.
/// Not asking for these data allows more focused creation
/// and we bypass the question who will get which data.
/// Also, many users will have at most one broadcast list
/// so, a generic name and image is sufficient at the first place.
///
/// Later on, however, the name can be changed using dc_set_chat_name().
/// The image cannot be changed to have a unique, recognizable icon in the chat lists.
/// All in all, this is also what other messengers are doing here.
/// For historical reasons, this function does not take a name directly,
/// instead you have to set the name using dc_set_chat_name()
/// after creating the broadcast list.
async fn create_broadcast_list(&self, account_id: u32) -> Result<u32> {
let ctx = self.get_context(account_id).await?;
chat::create_broadcast_list(&ctx)
@@ -2065,6 +2053,23 @@ impl CommandApi {
ChatId::new(chat_id).set_draft(&ctx, Some(&mut draft)).await
}
// send the chat's current set draft
async fn misc_send_draft(&self, account_id: u32, chat_id: u32) -> Result<u32> {
let ctx = self.get_context(account_id).await?;
if let Some(draft) = ChatId::new(chat_id).get_draft(&ctx).await? {
let mut draft = draft;
let msg_id = chat::send_msg(&ctx, ChatId::new(chat_id), &mut draft)
.await?
.to_u32();
Ok(msg_id)
} else {
Err(anyhow!(
"chat with id {} doesn't have draft message",
chat_id
))
}
}
}
// Helper functions (to prevent code duplication)

View File

@@ -55,5 +55,5 @@
},
"type": "module",
"types": "dist/deltachat.d.ts",
"version": "1.124.1"
"version": "1.126.1"
}

View File

@@ -148,7 +148,7 @@ describe("online tests", function () {
waitForEvent(dc, "IncomingMsg", accountId1),
]);
dc.rpc.miscSendTextMessage(accountId2, chatId, "super secret message");
// Check if answer arives at A and if it is encrypted
// Check if answer arrives at A and if it is encrypted
await eventPromise2;
const messageId = (

View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat-repl"
version = "1.124.1"
version = "1.126.1"
license = "MPL-2.0"
edition = "2021"

View File

@@ -71,3 +71,6 @@ line-length = 120
[tool.isort]
profile = "black"
[tool.pytest.ini_options]
log_cli = true

View File

@@ -3,7 +3,6 @@ import logging
from typing import (
TYPE_CHECKING,
Callable,
Coroutine,
Dict,
Iterable,
Optional,
@@ -92,7 +91,7 @@ class Client:
"""Process events forever."""
self.run_until(lambda _: False)
def run_until(self, func: Callable[[AttrDict], Union[bool, Coroutine]]) -> AttrDict:
def run_until(self, func: Callable[[AttrDict], bool]) -> AttrDict:
"""Process events until the given callable evaluates to True.
The callable should accept an AttrDict object representing the

View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat-rpc-server"
version = "1.124.1"
version = "1.126.1"
description = "DeltaChat JSON-RPC server"
edition = "2021"
readme = "README.md"

View File

@@ -11,7 +11,6 @@ ignore = [
# when upgrading.
# Please keep this list alphabetically sorted.
skip = [
{ name = "ahash", version = "0.7.6" },
{ name = "base16ct", version = "0.1.1" },
{ name = "base64", version = "<0.21" },
{ name = "bitflags", version = "1.3.2" },

View File

@@ -108,7 +108,7 @@ The most obvious alternative would be to create a new contact with the new addre
#### Upsides:
- With this approach, it's easier to switch to a model where the info about the transition is encoded in the PGP key. Since the key is gossiped, the information about the transition will spread virally.
- (Also, less important: Slightly faster transition: If you send a message to e.g. "Delta Chat Dev", all members of the "sub-group" "delta android" will know of your transition.)
- It's easier to implement (if too many problems turn up, we can still switch to another approach and didn't wast that much development time.)
- It's easier to implement (if too many problems turn up, we can still switch to another approach and didn't waste that much development time.)
[full messages](https://github.com/deltachat/deltachat-core-rust/pull/2896#discussion_r852002161)

View File

@@ -36,7 +36,7 @@ export class Context extends EventEmitter {
}
}
/** Opens a stanalone context (without an account manager)
/** Opens a standalone context (without an account manager)
* automatically starts the event handler */
static open(cwd: string): Context {
const dbFile = join(cwd, 'db.sqlite')

View File

@@ -60,5 +60,5 @@
"test:mocha": "mocha -r esm node/test/test.js --growl --reporter=spec --bail --exit"
},
"types": "node/dist/index.d.ts",
"version": "1.124.1"
"version": "1.126.1"
}

View File

@@ -539,8 +539,6 @@ def test_see_new_verified_member_after_going_online(acfactory, tmp_path, lp):
assert msg_in.text == msg_out.text
assert msg_in.get_sender_contact().addr == ac2_addr
ac1.set_config("bcc_self", "0")
def test_use_new_verified_group_after_going_online(acfactory, tmp_path, lp):
"""Another test for the bug #3836:
@@ -589,4 +587,67 @@ def test_use_new_verified_group_after_going_online(acfactory, tmp_path, lp):
assert msg_in.get_sender_contact().addr == ac2.get_config("addr")
assert msg_in.text == msg_out.text
ac2.set_config("bcc_self", "0")
def test_verified_group_vs_delete_server_after(acfactory, tmp_path, lp):
"""Test for the issue #4346:
- User is added to a verified group.
- First device of the user downloads "member added" from the group.
- First device removes "member added" from the server.
- Some new messages are sent to the group.
- Second device comes online, receives these new messages. The result is a verified group with unverified members.
- First device re-gossips Autocrypt keys to the group.
- Now the seconds device has all members verified.
"""
ac1, ac2 = acfactory.get_online_accounts(2)
ac2_offl = acfactory.new_online_configuring_account(cloned_from=ac2)
for ac in [ac2, ac2_offl]:
ac.set_config("bcc_self", "1")
ac2.set_config("delete_server_after", "1")
ac2.set_config("gossip_period", "0") # Re-gossip in every message
acfactory.bring_accounts_online()
dir = tmp_path / "exportdir"
dir.mkdir()
ac2.export_self_keys(str(dir))
ac2_offl.import_self_keys(str(dir))
ac2_offl.stop_io()
lp.sec("ac1: create verified-group QR, ac2 scans and joins")
chat1 = ac1.create_group_chat("hello", verified=True)
assert chat1.is_protected()
qr = chat1.get_join_qr()
lp.sec("ac2: start QR-code based join-group protocol")
chat2 = ac2.qr_join_chat(qr)
ac1._evtracker.wait_securejoin_inviter_progress(1000)
# Wait for "Member Me (<addr>) added by <addr>." message.
msg_in = ac2._evtracker.wait_next_incoming_message()
assert msg_in.is_system_message()
lp.sec("ac2: waiting for 'member added' to be deleted on the server")
ac2._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_DELETED")
lp.sec("ac1: sending 'hi' to the group")
ac2.set_config("delete_server_after", "0")
chat1.send_text("hi")
lp.sec("ac2_offl: going online, checking the 'hi' message")
ac2_offl.start_io()
msg_in = ac2_offl._evtracker.wait_next_incoming_message()
assert not msg_in.is_system_message()
assert msg_in.text.startswith("[Sender of this message is not verified:")
ac2_offl_ac1_contact = msg_in.get_sender_contact()
assert ac2_offl_ac1_contact.addr == ac1.get_config("addr")
assert not ac2_offl_ac1_contact.is_verified()
chat2_offl = msg_in.chat
assert chat2_offl.is_protected()
lp.sec("ac2: sending message re-gossiping Autocrypt keys")
chat2.send_text("hi2")
lp.sec("ac2_offl: receiving message")
ev = ac2_offl._evtracker.get_matching("DC_EVENT_INCOMING_MSG|DC_EVENT_MSGS_CHANGED")
msg_in = ac2_offl.get_message_by_id(ev.data2)
assert not msg_in.is_system_message()
assert msg_in.text == "hi2"
assert msg_in.chat == chat2_offl
assert msg_in.get_sender_contact().addr == ac2.get_config("addr")
assert ac2_offl_ac1_contact.is_verified()

View File

@@ -1698,12 +1698,10 @@ def test_qr_new_group_unblocked(acfactory, lp):
ac1_new_chat = ac1.create_group_chat("Another group")
ac1_new_chat.add_contact(ac2)
ac1_new_chat.send_text("Hello!")
# Receive "Member added" message.
ac2._evtracker.wait_next_incoming_message()
# Receive "Hello!" message.
ac1_new_chat.send_text("Hello!")
ac2_msg = ac2._evtracker.wait_next_incoming_message()
assert ac2_msg.text == "Hello!"
assert ac2_msg.chat.is_contact_request()

View File

@@ -1 +1 @@
2023-10-05
2023-10-24

View File

@@ -10,6 +10,8 @@ and an own build machine.
- `deny.sh` runs `cargo deny` for all Rust code in the project.
- `codespell.sh` spellchecks the source code using `codespell` tool.
- `../.github/workflows` contains jobs run by GitHub Actions.
- `remote_tests_python.sh` rsyncs to a build machine and runs

View File

@@ -14,4 +14,4 @@ export DCC_RS_DEV="$PWD"
cargo build -p deltachat_ffi --features jsonrpc
tox -c python -e py --devenv venv
env/bin/pip install --upgrade pip
venv/bin/pip install --upgrade pip

197
scripts/wheel-rpc-server.py Executable file
View File

@@ -0,0 +1,197 @@
#!/usr/bin/env python3
"""Build Python wheels for deltachat-rpc-server.
Run scripts/zig-rpc-server.sh first."""
from pathlib import Path
from wheel.wheelfile import WheelFile
import tomllib
import tarfile
from io import BytesIO
def metadata_contents(version):
return f"""Metadata-Version: 2.1
Name: deltachat-rpc-server
Version: {version}
Summary: Delta Chat JSON-RPC server
"""
def build_source_package(version):
filename = f"dist/deltachat-rpc-server-{version}.tar.gz"
with tarfile.open(filename, "w:gz") as pkg:
def pack(name, contents):
contents = contents.encode()
tar_info = tarfile.TarInfo(f"deltachat-rpc-server-{version}/{name}")
tar_info.mode = 0o644
tar_info.size = len(contents)
pkg.addfile(tar_info, BytesIO(contents))
pack("PKG-INFO", metadata_contents(version))
pack(
"pyproject.toml",
f"""[build-system]
requires = ["setuptools==68.2.2", "pip"]
build-backend = "setuptools.build_meta"
[project]
name = "deltachat-rpc-server"
version = "{version}"
[project.scripts]
deltachat-rpc-server = "deltachat_rpc_server:main"
""",
)
pack(
"setup.py",
f"""
import sys
from setuptools import setup, find_packages
from distutils.cmd import Command
from setuptools.command.install import install
from setuptools.command.build import build
import subprocess
import platform
import tempfile
from zipfile import ZipFile
from pathlib import Path
import shutil
class BuildCommand(build):
def run(self):
tmpdir = tempfile.mkdtemp()
subprocess.run(
[
sys.executable,
"-m",
"pip",
"download",
"--no-input",
"--timeout",
"1000",
"--platform",
"musllinux_1_1_" + platform.machine(),
"--only-binary=:all:",
"deltachat-rpc-server=={version}",
],
cwd=tmpdir,
)
wheel_path = next(Path(tmpdir).glob("*.whl"))
with ZipFile(wheel_path, "r") as wheel:
exe_path = wheel.extract("deltachat_rpc_server/deltachat-rpc-server", "src")
Path(exe_path).chmod(0o700)
wheel.extract("deltachat_rpc_server/__init__.py", "src")
shutil.rmtree(tmpdir)
return super().run()
setup(
cmdclass={{"build": BuildCommand}},
package_data={{"deltachat_rpc_server": ["deltachat-rpc-server"]}},
)
""",
)
pack("src/deltachat_rpc_server/__init__.py", "")
def build_wheel(version, binary, tag, windows=False):
filename = f"dist/deltachat_rpc_server-{version}-{tag}.whl"
with WheelFile(filename, "w") as wheel:
wheel.write("LICENSE", "deltachat_rpc_server/LICENSE")
wheel.write("deltachat-rpc-server/README.md", "deltachat_rpc_server/README.md")
if windows:
wheel.writestr(
"deltachat_rpc_server/__init__.py",
"""import os, sys, subprocess
def main():
argv = [os.path.join(os.path.dirname(__file__), "deltachat-rpc-server.exe"), *sys.argv[1:]]
sys.exit(subprocess.call(argv))
""",
)
else:
wheel.writestr(
"deltachat_rpc_server/__init__.py",
"""import os, sys
def main():
argv = [os.path.join(os.path.dirname(__file__), "deltachat-rpc-server"), *sys.argv[1:]]
os.execv(argv[0], argv)
""",
)
Path(binary).chmod(0o755)
wheel.write(
binary,
"deltachat_rpc_server/deltachat-rpc-server.exe"
if windows
else "deltachat_rpc_server/deltachat-rpc-server",
)
wheel.writestr(
f"deltachat_rpc_server-{version}.dist-info/METADATA",
metadata_contents(version),
)
wheel.writestr(
f"deltachat_rpc_server-{version}.dist-info/WHEEL",
"Wheel-Version: 1.0\nRoot-Is-Purelib: false\nTag: {tag}",
)
wheel.writestr(
f"deltachat_rpc_server-{version}.dist-info/entry_points.txt",
"[console_scripts]\ndeltachat-rpc-server = deltachat_rpc_server:main",
)
def main():
with open("deltachat-rpc-server/Cargo.toml", "rb") as f:
cargo_toml = tomllib.load(f)
version = cargo_toml["package"]["version"]
Path("dist").mkdir(exist_ok=True)
build_source_package(version)
build_wheel(
version,
"dist/deltachat-rpc-server-x86_64-linux",
"py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.musllinux_1_1_x86_64",
)
build_wheel(
version,
"dist/deltachat-rpc-server-armv7-linux",
"py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.musllinux_1_1_armv7l",
)
build_wheel(
version,
"dist/deltachat-rpc-server-aarch64-linux",
"py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64",
)
build_wheel(
version,
"dist/deltachat-rpc-server-i686-linux",
"py3-none-manylinux_2_12_i686.manylinux2010_i686.musllinux_1_1_i686",
)
# macOS versions for platform compatibility tags are taken from https://doc.rust-lang.org/rustc/platform-support.html
build_wheel(
version,
"dist/deltachat-rpc-server-x86_64-macos",
"py3-none-macosx_10_7_x86_64",
)
build_wheel(
version,
"dist/deltachat-rpc-server-aarch64-macos",
"py3-none-macosx_11_0_arm64",
)
build_wheel(
version, "dist/deltachat-rpc-server-win32.exe", "py3-none-win32", windows=True
)
build_wheel(
version,
"dist/deltachat-rpc-server-win64.exe",
"py3-none-win_amd64",
windows=True,
)
main()

View File

@@ -1,10 +1,20 @@
#!/usr/bin/env python
# /// pyproject
# [run]
# dependencies = [
# "ziglang==0.11.0"
# ]
# ///
import os
import subprocess
import sys
import os
def flag_filter(flag: str) -> bool:
# Workaround for <https://github.com/sfackler/rust-openssl/issues/2043>.
if flag == "-latomic":
return False
if flag == "-lc":
return False
if flag == "-Wl,-melf_i386":
@@ -24,8 +34,23 @@ def main():
else:
zig_cpu_args = []
# Disable atomics and use locks instead in OpenSSL.
# Zig toolchains do not provide atomics.
# This is a workaround for <https://github.com/deltachat/deltachat-core-rust/issues/4799>
args += ["-DBROKEN_CLANG_ATOMICS"]
subprocess.run(
["zig", "cc", "-target", zig_target, *zig_cpu_args, *args], check=True
[
sys.executable,
"-m",
"ziglang",
"cc",
"-target",
zig_target,
*zig_cpu_args,
*args,
],
check=True,
)

View File

@@ -1,28 +0,0 @@
#!/bin/sh
#
# Run `cargo check` with musl libc.
# This requires `zig` to compile vendored openssl.
set -x
set -e
unset RUSTFLAGS
# Pin Rust version to avoid uncontrolled changes in the compiler and linker flags.
export RUSTUP_TOOLCHAIN=1.72.0
ZIG_VERSION=0.11.0
# Download Zig
rm -fr "$ZIG_VERSION" "zig-linux-x86_64-$ZIG_VERSION.tar.xz"
wget "https://ziglang.org/builds/zig-linux-x86_64-$ZIG_VERSION.tar.xz"
tar xf "zig-linux-x86_64-$ZIG_VERSION.tar.xz"
export PATH="$PWD/zig-linux-x86_64-$ZIG_VERSION:$PATH"
rustup target add x86_64-unknown-linux-musl
CC="$PWD/scripts/zig-cc" \
TARGET_CC="$PWD/scripts/zig-cc" \
CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER="$PWD/scripts/zig-cc" \
LD="$PWD/scripts/zig-cc" \
ZIG_TARGET="x86_64-linux-musl" \
cargo check --release --target x86_64-unknown-linux-musl -p deltachat_ffi --features jsonrpc

View File

@@ -10,14 +10,6 @@ unset RUSTFLAGS
# Pin Rust version to avoid uncontrolled changes in the compiler and linker flags.
export RUSTUP_TOOLCHAIN=1.72.0
ZIG_VERSION=0.11.0
# Download Zig
rm -fr "$ZIG_VERSION" "zig-linux-x86_64-$ZIG_VERSION.tar.xz"
wget "https://ziglang.org/builds/zig-linux-x86_64-$ZIG_VERSION.tar.xz"
tar xf "zig-linux-x86_64-$ZIG_VERSION.tar.xz"
export PATH="$PWD/zig-linux-x86_64-$ZIG_VERSION:$PATH"
rustup target add i686-unknown-linux-musl
CC="$PWD/scripts/zig-cc" \
TARGET_CC="$PWD/scripts/zig-cc" \
@@ -50,3 +42,9 @@ CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER="$PWD/scripts/zig-cc" \
LD="$PWD/scripts/zig-cc" \
ZIG_TARGET="aarch64-linux-musl" \
cargo build --release --target aarch64-unknown-linux-musl -p deltachat-rpc-server --features vendored
mkdir -p dist
cp target/x86_64-unknown-linux-musl/release/deltachat-rpc-server dist/deltachat-rpc-server-x86_64-linux
cp target/i686-unknown-linux-musl/release/deltachat-rpc-server dist/deltachat-rpc-server-i686-linux
cp target/aarch64-unknown-linux-musl/release/deltachat-rpc-server dist/deltachat-rpc-server-aarch64-linux
cp target/armv7-unknown-linux-musleabihf/release/deltachat-rpc-server dist/deltachat-rpc-server-armv7-linux

View File

@@ -43,7 +43,7 @@ the `Subject` header SHOULD be `Message from <sender name>`.
Replies to messages MAY follow the typical `Re:`-format.
The body MAY contain text which MUST have the content type `text/plain`
or `mulipart/alternative` containing `text/plain`.
or `multipart/alternative` containing `text/plain`.
The text MAY be divided into a user-text-part and a footer-part using the
line `-- ` (minus, minus, space, lineend).

View File

@@ -2337,7 +2337,7 @@ pub async fn send_msg_sync(context: &Context, chat_id: ChatId, msg: &mut Message
}
async fn send_msg_inner(context: &Context, chat_id: ChatId, msg: &mut Message) -> Result<MsgId> {
// protect all system messages againts RTLO attacks
// protect all system messages against RTLO attacks
if msg.is_system_message() {
msg.text = strip_rtlo_characters(&msg.text);
}
@@ -6094,22 +6094,40 @@ mod tests {
get_chat_contacts(&alice, chat_bob.id).await?.pop().unwrap(),
)
.await?;
let chat = Chat::load_from_db(&alice, broadcast_id).await?;
assert_eq!(chat.typ, Chattype::Broadcast);
assert_eq!(chat.name, stock_str::broadcast_list(&alice).await);
assert!(!chat.is_self_talk());
set_chat_name(&alice, broadcast_id, "Broadcast list").await?;
{
let chat = Chat::load_from_db(&alice, broadcast_id).await?;
assert_eq!(chat.typ, Chattype::Broadcast);
assert_eq!(chat.name, "Broadcast list");
assert!(!chat.is_self_talk());
send_text_msg(&alice, broadcast_id, "ola!".to_string()).await?;
let msg = alice.get_last_msg().await;
assert_eq!(msg.chat_id, chat.id);
send_text_msg(&alice, broadcast_id, "ola!".to_string()).await?;
let msg = alice.get_last_msg().await;
assert_eq!(msg.chat_id, chat.id);
}
let msg = bob.recv_msg(&alice.pop_sent_msg().await).await;
assert_eq!(msg.get_text(), "ola!");
assert!(!msg.get_showpadlock()); // avoid leaking recipients in encryption data
let chat = Chat::load_from_db(&bob, msg.chat_id).await?;
assert_eq!(chat.typ, Chattype::Single);
assert_eq!(chat.id, chat_bob.id);
assert!(!chat.is_self_talk());
{
let msg = bob.recv_msg(&alice.pop_sent_msg().await).await;
assert_eq!(msg.get_text(), "ola!");
assert_eq!(msg.subject, "Broadcast list");
assert!(!msg.get_showpadlock()); // avoid leaking recipients in encryption data
let chat = Chat::load_from_db(&bob, msg.chat_id).await?;
assert_eq!(chat.typ, Chattype::Mailinglist);
assert_ne!(chat.id, chat_bob.id);
assert_eq!(chat.name, "Broadcast list");
assert!(!chat.is_self_talk());
}
{
// Alice changes the name:
set_chat_name(&alice, broadcast_id, "My great broadcast").await?;
let sent = alice.send_text(broadcast_id, "I changed the title!").await;
let msg = bob.recv_msg(&sent).await;
assert_eq!(msg.subject, "Re: My great broadcast");
let bob_chat = Chat::load_from_db(&bob, msg.chat_id).await?;
assert_eq!(bob_chat.name, "My great broadcast");
}
Ok(())
}

View File

@@ -1,6 +1,7 @@
//! # Chat list module.
use anyhow::{ensure, Context as _, Result};
use once_cell::sync::Lazy;
use crate::chat::{update_special_chat_names, Chat, ChatId, ChatVisibility};
use crate::constants::{
@@ -15,6 +16,10 @@ use crate::stock_str;
use crate::summary::Summary;
use crate::tools::IsNoneOrEmpty;
/// Regex to find out if a query should filter by unread messages.
pub static IS_UNREAD_FILTER: Lazy<regex::Regex> =
Lazy::new(|| regex::Regex::new(r"\bis:unread\b").unwrap());
/// An object representing a single chatlist in memory.
///
/// Chatlist objects contain chat IDs and, if possible, message IDs belonging to them.
@@ -78,7 +83,8 @@ impl Chatlist {
/// - if the flag DC_GCL_ADD_ALLDONE_HINT is set, DC_CHAT_ID_ALLDONE_HINT
/// is added as needed.
/// `query`: An optional query for filtering the list. Only chats matching this query
/// are returned.
/// are returned. When `is:unread` is contained in the query, the chatlist is
/// filtered such that only chats with unread messages show up.
/// `query_contact_id`: An optional contact ID for filtering the list. Only chats including this contact ID
/// are returned.
pub async fn try_load(
@@ -172,8 +178,10 @@ impl Chatlist {
)
.await?
} else if let Some(query) = query {
let query = query.trim().to_string();
ensure!(!query.is_empty(), "missing query");
let mut query = query.trim().to_string();
ensure!(!query.is_empty(), "query mustn't be empty");
let only_unread = IS_UNREAD_FILTER.find(&query).is_some();
query = IS_UNREAD_FILTER.replace(&query, "").trim().to_string();
// allow searching over special names that may change at any time
// when the ui calls set_stock_translation()
@@ -198,9 +206,10 @@ impl Chatlist {
WHERE c.id>9 AND c.id!=?2
AND c.blocked!=1
AND c.name LIKE ?3
AND (NOT ?4 OR EXISTS (SELECT 1 FROM msgs m WHERE m.chat_id = c.id AND m.state == ?5 AND hidden=0))
GROUP BY c.id
ORDER BY IFNULL(m.timestamp,c.created_timestamp) DESC, m.id DESC;",
(MessageState::OutDraft, skip_id, str_like_cmd),
(MessageState::OutDraft, skip_id, str_like_cmd, only_unread, MessageState::InFresh),
process_row,
process_rows,
)
@@ -462,7 +471,8 @@ pub async fn get_last_message_for_chat(
mod tests {
use super::*;
use crate::chat::{
create_group_chat, get_chat_contacts, remove_contact_from_chat, ProtectionStatus,
add_contact_to_chat, create_group_chat, get_chat_contacts, remove_contact_from_chat,
send_text_msg, ProtectionStatus,
};
use crate::message::Viewtype;
use crate::receive_imf::receive_imf;
@@ -471,7 +481,7 @@ mod tests {
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_try_load() {
let t = TestContext::new().await;
let t = TestContext::new_bob().await;
let chat_id1 = create_group_chat(&t, ProtectionStatus::Unprotected, "a chat")
.await
.unwrap();
@@ -510,6 +520,31 @@ mod tests {
let chats = Chatlist::try_load(&t, 0, Some("b"), None).await.unwrap();
assert_eq!(chats.len(), 1);
// receive a message from alice
let alice = TestContext::new_alice().await;
let alice_chat_id = create_group_chat(&alice, ProtectionStatus::Unprotected, "alice chat")
.await
.unwrap();
add_contact_to_chat(
&alice,
alice_chat_id,
Contact::create(&alice, "bob", "bob@example.net")
.await
.unwrap(),
)
.await
.unwrap();
send_text_msg(&alice, alice_chat_id, "hi".into())
.await
.unwrap();
let sent_msg = alice.pop_sent_msg().await;
t.recv_msg(&sent_msg).await;
let chats = Chatlist::try_load(&t, 0, Some("is:unread"), None)
.await
.unwrap();
assert!(chats.len() == 1);
let chats = Chatlist::try_load(&t, DC_GCL_ARCHIVED_ONLY, None, None)
.await
.unwrap();

View File

@@ -318,6 +318,13 @@ pub enum Config {
/// Last message processed by the bot.
LastMsgId,
/// How often to gossip Autocrypt keys in chats with multiple recipients, in seconds. 2 days by
/// default.
///
/// This is not supposed to be changed by UIs and only used for testing.
#[strum(props(default = "172800"))]
GossipPeriod,
}
impl Context {

View File

@@ -26,7 +26,6 @@ use crate::config::Config;
use crate::contact::addr_cmp;
use crate::context::Context;
use crate::imap::Imap;
use crate::job;
use crate::log::LogExt;
use crate::login_param::{CertificateChecks, LoginParam, ServerLoginParam};
use crate::message::{Message, Viewtype};
@@ -466,7 +465,7 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
if configured_addr != param.addr {
// Switched account, all server UIDs we know are invalid
info!(ctx, "Scheduling resync because the address has changed.");
job::schedule_resync(ctx).await?;
ctx.schedule_resync().await?;
}
}

View File

@@ -1265,7 +1265,7 @@ impl Contact {
return Ok(Some(ContactId::SELF));
}
match Contact::lookup_id_by_addr(context, &verifier_addr, Origin::AddressBook).await? {
match Contact::lookup_id_by_addr(context, &verifier_addr, Origin::Unknown).await? {
Some(contact_id) => Ok(Some(contact_id)),
None => {
let addr = &self.addr;

View File

@@ -4,7 +4,7 @@ use std::collections::{BTreeMap, HashMap};
use std::ffi::OsString;
use std::ops::Deref;
use std::path::{Path, PathBuf};
use std::sync::atomic::AtomicBool;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime};
@@ -23,7 +23,7 @@ use crate::key::{DcKey, SignedPublicKey};
use crate::login_param::LoginParam;
use crate::message::{self, MessageState, MsgId};
use crate::quota::QuotaInfo;
use crate::scheduler::SchedulerState;
use crate::scheduler::{InterruptInfo, SchedulerState};
use crate::sql::Sql;
use crate::stock_str::StockStrings;
use crate::timesmearing::SmearedTimestamp;
@@ -38,7 +38,7 @@ use crate::tools::{duration_to_str, time};
///
/// # Examples
///
/// Creating a new unecrypted database:
/// Creating a new unencrypted database:
///
/// ```
/// # let rt = tokio::runtime::Runtime::new().unwrap();
@@ -211,9 +211,6 @@ pub struct InnerContext {
/// Set to `None` if quota was never tried to load.
pub(crate) quota: RwLock<Option<QuotaInfo>>,
/// Set to true if quota update is requested.
pub(crate) quota_update_request: AtomicBool,
/// IMAP UID resync request.
pub(crate) resync_request: AtomicBool,
@@ -384,7 +381,6 @@ impl Context {
scheduler: SchedulerState::new(),
ratelimit: RwLock::new(Ratelimit::new(Duration::new(60, 0), 6.0)), // Allow at least 1 message every 10 seconds + a burst of 6.
quota: RwLock::new(None),
quota_update_request: AtomicBool::new(false),
resync_request: AtomicBool::new(false),
new_msgs_notify,
server_id: RwLock::new(None),
@@ -426,6 +422,16 @@ impl Context {
self.scheduler.maybe_network().await;
}
pub(crate) async fn schedule_resync(&self) -> Result<()> {
self.resync_request.store(true, Ordering::Relaxed);
self.scheduler
.interrupt_inbox(InterruptInfo {
probe_network: false,
})
.await;
Ok(())
}
/// Returns a reference to the underlying SQL instance.
///
/// Warning: this is only here for testing, not part of the public API.
@@ -754,7 +760,6 @@ impl Context {
.await?
.to_string(),
);
res.insert(
"debug_logging",
self.get_config_int(Config::DebugLogging).await?.to_string(),
@@ -763,6 +768,10 @@ impl Context {
"last_msg_id",
self.get_config_int(Config::LastMsgId).await?.to_string(),
);
res.insert(
"gossip_period",
self.get_config_int(Config::GossipPeriod).await?.to_string(),
);
let elapsed = self.creation_time.elapsed();
res.insert("uptime", duration_to_str(elapsed.unwrap_or_default()));

View File

@@ -10,11 +10,11 @@ use serde::{Deserialize, Serialize};
use crate::config::Config;
use crate::context::Context;
use crate::imap::{Imap, ImapActionResult};
use crate::job::{self, Action, Job, Status};
use crate::message::{Message, MsgId, Viewtype};
use crate::mimeparser::{MimeMessage, Part};
use crate::scheduler::InterruptInfo;
use crate::tools::time;
use crate::{job_try, stock_str, EventType};
use crate::{stock_str, EventType};
/// Download limits should not be used below `MIN_DOWNLOAD_LIMIT`.
///
@@ -90,7 +90,14 @@ impl MsgId {
DownloadState::Available | DownloadState::Failure => {
self.update_download_state(context, DownloadState::InProgress)
.await?;
job::add(context, Job::new(Action::DownloadMsg, self.to_u32())).await?;
context
.sql
.execute("INSERT INTO download (msg_id) VALUES (?)", (self,))
.await?;
context
.scheduler
.interrupt_inbox(InterruptInfo::new(false))
.await;
}
}
Ok(())
@@ -124,59 +131,49 @@ impl Message {
}
}
impl Job {
/// Actually download a message.
/// Called in response to `Action::DownloadMsg`.
pub(crate) async fn download_msg(&self, context: &Context, imap: &mut Imap) -> Status {
if let Err(err) = imap.prepare(context).await {
warn!(context, "download: could not connect: {:#}", err);
return Status::RetryNow;
}
/// Actually download a message partially downloaded before.
///
/// Most messages are downloaded automatically on fetch instead.
pub(crate) async fn download_msg(context: &Context, msg_id: MsgId, imap: &mut Imap) -> Result<()> {
imap.prepare(context).await?;
let msg = job_try!(Message::load_from_db(context, MsgId::new(self.foreign_id)).await);
let row = job_try!(
context
.sql
.query_row_optional(
"SELECT uid, folder FROM imap WHERE rfc724_mid=? AND target=folder",
(&msg.rfc724_mid,),
|row| {
let server_uid: u32 = row.get(0)?;
let server_folder: String = row.get(1)?;
Ok((server_uid, server_folder))
}
)
.await
);
let msg = Message::load_from_db(context, msg_id).await?;
let row = context
.sql
.query_row_optional(
"SELECT uid, folder FROM imap WHERE rfc724_mid=? AND target!=''",
(&msg.rfc724_mid,),
|row| {
let server_uid: u32 = row.get(0)?;
let server_folder: String = row.get(1)?;
Ok((server_uid, server_folder))
},
)
.await?;
if let Some((server_uid, server_folder)) = row {
match imap
.fetch_single_msg(context, &server_folder, server_uid, msg.rfc724_mid.clone())
.await
{
ImapActionResult::RetryLater | ImapActionResult::Failed => {
job_try!(
msg.id
.update_download_state(context, DownloadState::Failure)
.await
);
Status::Finished(Err(anyhow!("Call download_full() again to try over.")))
}
ImapActionResult::Success => {
// update_download_state() not needed as receive_imf() already
// set the state and emitted the event.
Status::Finished(Ok(()))
}
}
} else {
// No IMAP record found, we don't know the UID and folder.
job_try!(
if let Some((server_uid, server_folder)) = row {
match imap
.fetch_single_msg(context, &server_folder, server_uid, msg.rfc724_mid.clone())
.await
{
ImapActionResult::RetryLater | ImapActionResult::Failed => {
msg.id
.update_download_state(context, DownloadState::Failure)
.await
);
Status::Finished(Err(anyhow!("Call download_full() again to try over.")))
.await?;
Err(anyhow!("Call download_full() again to try over."))
}
ImapActionResult::Success => {
// update_download_state() not needed as receive_imf() already
// set the state and emitted the event.
Ok(())
}
}
} else {
// No IMAP record found, we don't know the UID and folder.
msg.id
.update_download_state(context, DownloadState::Failure)
.await?;
Err(anyhow!("Call download_full() again to try over."))
}
}

View File

@@ -26,7 +26,6 @@ use crate::contact::{normalize_name, Contact, ContactAddress, ContactId, Modifie
use crate::context::Context;
use crate::events::EventType;
use crate::headerdef::{HeaderDef, HeaderDefMap};
use crate::job;
use crate::login_param::{CertificateChecks, LoginParam, ServerLoginParam};
use crate::message::{self, Message, MessageState, MessengerMessage, MsgId, Viewtype};
use crate::mimeparser;
@@ -82,7 +81,6 @@ const RFC724MID_UID: &str = "(UID BODY.PEEK[HEADER.FIELDS (\
MESSAGE-ID \
X-MICROSOFT-ORIGINAL-MESSAGE-ID\
)])";
const JUST_UID: &str = "(UID)";
const BODY_FULL: &str = "(FLAGS BODY.PEEK[])";
const BODY_PARTIAL: &str = "(FLAGS RFC822.SIZE BODY.PEEK[HEADER])";
@@ -391,6 +389,7 @@ impl Imap {
"IMAP-LOGIN as {}",
self.config.lp.user
)));
self.connectivity.set_connected(context).await;
info!(context, "Successfully logged into IMAP server");
Ok(())
}
@@ -614,7 +613,7 @@ impl Imap {
"The server illegally decreased the uid_next of folder {folder:?} from {old_uid_next} to {uid_next} without changing validity ({new_uid_validity}), resyncing UIDs...",
);
set_uid_next(context, folder, uid_next).await?;
job::schedule_resync(context).await?;
context.schedule_resync().await?;
}
uid_next != old_uid_next // If uid_next changed, there are new emails
} else {
@@ -626,18 +625,6 @@ impl Imap {
// UIDVALIDITY is modified, reset highest seen MODSEQ.
set_modseq(context, folder, 0).await?;
if mailbox.exists == 0 {
info!(context, "Folder {folder:?} is empty.");
// set uid_next=1 for empty folders.
// If we do not do this here, we'll miss the first message
// as we will get in here again and fetch from uid_next then.
// Also, the "fall back to fetching" below would need a non-zero mailbox.exists to work.
set_uid_next(context, folder, 1).await?;
set_uidvalidity(context, folder, new_uid_validity).await?;
return Ok(false);
}
// ============== uid_validity has changed or is being set the first time. ==============
let new_uid_next = match mailbox.uid_next {
@@ -645,25 +632,35 @@ impl Imap {
None => {
warn!(
context,
"IMAP folder {folder:?} has no uid_next, fall back to fetching."
"SELECT response for IMAP folder {folder:?} has no UIDNEXT, fall back to STATUS command."
);
// note that we use fetch by sequence number
// and thus we only need to get exactly the
// last-index message.
let set = format!("{}", mailbox.exists);
let mut list = session
.inner
.fetch(set, JUST_UID)
.await
.context("Error fetching UID")?;
let mut new_last_seen_uid = None;
while let Some(fetch) = list.try_next().await? {
if fetch.message == mailbox.exists && fetch.uid.is_some() {
new_last_seen_uid = fetch.uid;
}
// RFC 3501 says STATUS command SHOULD NOT be used
// on the currently selected mailbox because the same
// information can be obtained by other means,
// such as reading SELECT response.
//
// However, it also says that UIDNEXT is REQUIRED
// in the SELECT response and if we are here,
// it is actually not returned.
//
// In particular, Winmail Pro Mail Server 5.1.0616
// never returns UIDNEXT in SELECT response,
// but responds to "SELECT INBOX (UIDNEXT)" command.
let status = session
.inner
.status(folder, "(UIDNEXT)")
.await
.context("STATUS (UIDNEXT) error for {folder:?}")?;
if let Some(uid_next) = status.uid_next {
uid_next
} else {
warn!(context, "STATUS {folder} (UIDNEXT) did not return UIDNEXT");
// Set UIDNEXT to 1 as a last resort fallback.
1
}
new_last_seen_uid.context("select: failed to fetch")? + 1
}
};
@@ -680,7 +677,7 @@ impl Imap {
.await?;
if old_uid_validity != 0 || old_uid_next != 0 {
job::schedule_resync(context).await?;
context.schedule_resync().await?;
}
info!(
context,

View File

@@ -15,6 +15,7 @@ use crate::net::connect_tcp;
use crate::net::session::SessionStream;
use crate::net::tls::wrap_tls;
use crate::socks::Socks5Config;
use fast_socks5::client::Socks5Stream;
/// IMAP write and read timeout.
pub(crate) const IMAP_TIMEOUT: Duration = Duration::from_secs(30);
@@ -64,6 +65,12 @@ async fn determine_capabilities(
}
impl Client {
fn new(stream: Box<dyn SessionStream>) -> Self {
Self {
inner: ImapClient::new(stream),
}
}
pub(crate) async fn login(self, username: &str, password: &str) -> Result<Session> {
let Client { inner, .. } = self;
let mut session = inner
@@ -98,27 +105,24 @@ impl Client {
let tls_stream = wrap_tls(strict_tls, hostname, tcp_stream).await?;
let buffered_stream = BufWriter::new(tls_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let mut client = ImapClient::new(session_stream);
let mut client = Client::new(session_stream);
let _greeting = client
.read_response()
.await
.context("failed to read greeting")??;
Ok(Client { inner: client })
Ok(client)
}
pub async fn connect_insecure(context: &Context, hostname: &str, port: u16) -> Result<Self> {
let tcp_stream = connect_tcp(context, hostname, port, IMAP_TIMEOUT, false).await?;
let buffered_stream = BufWriter::new(tcp_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let mut client = ImapClient::new(session_stream);
let mut client = Client::new(session_stream);
let _greeting = client
.read_response()
.await
.context("failed to read greeting")??;
Ok(Client { inner: client })
Ok(client)
}
pub async fn connect_starttls(
@@ -130,7 +134,8 @@ impl Client {
let tcp_stream = connect_tcp(context, hostname, port, IMAP_TIMEOUT, strict_tls).await?;
// Run STARTTLS command and convert the client back into a stream.
let mut client = ImapClient::new(tcp_stream);
let buffered_tcp_stream = BufWriter::new(tcp_stream);
let mut client = ImapClient::new(buffered_tcp_stream);
let _greeting = client
.read_response()
.await
@@ -139,7 +144,8 @@ impl Client {
.run_command_and_check_ok("STARTTLS", None)
.await
.context("STARTTLS command failed")?;
let tcp_stream = client.into_inner();
let buffered_tcp_stream = client.into_inner();
let tcp_stream = buffered_tcp_stream.into_inner();
let tls_stream = wrap_tls(strict_tls, hostname, tcp_stream)
.await
@@ -147,9 +153,8 @@ impl Client {
let buffered_stream = BufWriter::new(tls_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let client = ImapClient::new(session_stream);
Ok(Client { inner: client })
let client = Client::new(session_stream);
Ok(client)
}
pub async fn connect_secure_socks5(
@@ -165,13 +170,12 @@ impl Client {
let tls_stream = wrap_tls(strict_tls, domain, socks5_stream).await?;
let buffered_stream = BufWriter::new(tls_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let mut client = ImapClient::new(session_stream);
let mut client = Client::new(session_stream);
let _greeting = client
.read_response()
.await
.context("failed to read greeting")??;
Ok(Client { inner: client })
Ok(client)
}
pub async fn connect_insecure_socks5(
@@ -185,13 +189,12 @@ impl Client {
.await?;
let buffered_stream = BufWriter::new(socks5_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let mut client = ImapClient::new(session_stream);
let mut client = Client::new(session_stream);
let _greeting = client
.read_response()
.await
.context("failed to read greeting")??;
Ok(Client { inner: client })
Ok(client)
}
pub async fn connect_starttls_socks5(
@@ -206,7 +209,8 @@ impl Client {
.await?;
// Run STARTTLS command and convert the client back into a stream.
let mut client = ImapClient::new(socks5_stream);
let buffered_socks5_stream = BufWriter::new(socks5_stream);
let mut client = ImapClient::new(buffered_socks5_stream);
let _greeting = client
.read_response()
.await
@@ -215,15 +219,15 @@ impl Client {
.run_command_and_check_ok("STARTTLS", None)
.await
.context("STARTTLS command failed")?;
let socks5_stream = client.into_inner();
let buffered_socks5_stream = client.into_inner();
let socks5_stream: Socks5Stream<_> = buffered_socks5_stream.into_inner();
let tls_stream = wrap_tls(strict_tls, hostname, socks5_stream)
.await
.context("STARTTLS upgrade failed")?;
let buffered_stream = BufWriter::new(tls_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let client = ImapClient::new(session_stream);
Ok(Client { inner: client })
let client = Client::new(session_stream);
Ok(client)
}
}

View File

@@ -58,8 +58,7 @@ pub enum ImexMode {
/// Export a backup to the directory given as `path` with the given `passphrase`.
/// The backup contains all contacts, chats, images and other data and device independent settings.
/// The backup does not contain device dependent settings as ringtones or LED notification settings.
/// The name of the backup is typically `delta-chat-<day>.tar`, if more than one backup is create on a day,
/// the format is `delta-chat-<day>-<number>.tar`
/// The name of the backup is `delta-chat-backup-<day>-<number>-<addr>.tar`.
ExportBackup = 11,
/// `path` is the file (not: directory) to import. The file is normally
@@ -128,7 +127,7 @@ pub async fn has_backup(_context: &Context, dir_name: &Path) -> Result<String> {
&& (newest_backup_name.is_empty() || name > newest_backup_name)
{
// We just use string comparison to determine which backup is newer.
// This works fine because the filenames have the form ...delta-chat-backup-2020-07-24-00.tar
// This works fine because the filenames have the form `delta-chat-backup-2023-10-18-00-foo@example.com.tar`
newest_backup_path = Some(path);
newest_backup_name = name;
}
@@ -484,7 +483,11 @@ async fn import_backup(
/// Returns Ok((temp_db_path, temp_path, dest_path)) on success. Unencrypted database can be
/// written to temp_db_path. The backup can then be written to temp_path. If the backup succeeded,
/// it can be renamed to dest_path. This guarantees that the backup is complete.
fn get_next_backup_path(folder: &Path, backup_time: i64) -> Result<(PathBuf, PathBuf, PathBuf)> {
fn get_next_backup_path(
folder: &Path,
addr: &str,
backup_time: i64,
) -> Result<(PathBuf, PathBuf, PathBuf)> {
let folder = PathBuf::from(folder);
let stem = chrono::NaiveDateTime::from_timestamp_opt(backup_time, 0)
.context("can't get next backup path")?
@@ -495,13 +498,13 @@ fn get_next_backup_path(folder: &Path, backup_time: i64) -> Result<(PathBuf, Pat
// 64 backup files per day should be enough for everyone
for i in 0..64 {
let mut tempdbfile = folder.clone();
tempdbfile.push(format!("{stem}-{i:02}.db"));
tempdbfile.push(format!("{stem}-{i:02}-{addr}.db"));
let mut tempfile = folder.clone();
tempfile.push(format!("{stem}-{i:02}.tar.part"));
tempfile.push(format!("{stem}-{i:02}-{addr}.tar.part"));
let mut destfile = folder.clone();
destfile.push(format!("{stem}-{i:02}.tar"));
destfile.push(format!("{stem}-{i:02}-{addr}.tar"));
if !tempdbfile.exists() && !tempfile.exists() && !destfile.exists() {
return Ok((tempdbfile, tempfile, destfile));
@@ -516,7 +519,8 @@ fn get_next_backup_path(folder: &Path, backup_time: i64) -> Result<(PathBuf, Pat
async fn export_backup(context: &Context, dir: &Path, passphrase: String) -> Result<()> {
// get a fine backup file name (the name includes the date so that multiple backup instances are possible)
let now = time();
let (temp_db_path, temp_path, dest_path) = get_next_backup_path(dir, now)?;
let self_addr = context.get_primary_self_addr().await?;
let (temp_db_path, temp_path, dest_path) = get_next_backup_path(dir, &self_addr, now)?;
let _d1 = DeleteOnDrop(temp_db_path.clone());
let _d2 = DeleteOnDrop(temp_path.clone());

View File

@@ -1,390 +0,0 @@
//! # Job module.
//!
//! This module implements a job queue maintained in the SQLite database
//! and job types.
#![allow(missing_docs)]
use std::fmt;
use std::sync::atomic::Ordering;
use anyhow::{Context as _, Result};
use deltachat_derive::{FromSql, ToSql};
use rand::{thread_rng, Rng};
use crate::context::Context;
use crate::imap::Imap;
use crate::scheduler::InterruptInfo;
use crate::tools::time;
// results in ~3 weeks for the last backoff timespan
const JOB_RETRIES: u32 = 17;
/// Job try result.
#[derive(Debug, Display)]
pub enum Status {
Finished(Result<()>),
RetryNow,
}
#[macro_export]
macro_rules! job_try {
($expr:expr) => {
match $expr {
std::result::Result::Ok(val) => val,
std::result::Result::Err(err) => {
return $crate::job::Status::Finished(Err(err.into()));
}
}
};
($expr:expr,) => {
$crate::job_try!($expr)
};
}
#[derive(
Debug,
Display,
Copy,
Clone,
PartialEq,
Eq,
PartialOrd,
FromPrimitive,
ToPrimitive,
FromSql,
ToSql,
)]
#[repr(u32)]
pub enum Action {
// This job will download partially downloaded messages completely
// and is added when download_full() is called.
// Most messages are downloaded automatically on fetch
// and do not go through this job.
DownloadMsg = 250,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Job {
pub job_id: u32,
pub action: Action,
pub foreign_id: u32,
pub desired_timestamp: i64,
pub added_timestamp: i64,
pub tries: u32,
}
impl fmt::Display for Job {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "#{}, action {}", self.job_id, self.action)
}
}
impl Job {
pub fn new(action: Action, foreign_id: u32) -> Self {
let timestamp = time();
Self {
job_id: 0,
action,
foreign_id,
desired_timestamp: timestamp,
added_timestamp: timestamp,
tries: 0,
}
}
/// Deletes the job from the database.
async fn delete(self, context: &Context) -> Result<()> {
if self.job_id != 0 {
context
.sql
.execute("DELETE FROM jobs WHERE id=?;", (self.job_id as i32,))
.await?;
}
Ok(())
}
/// Saves the job to the database, creating a new entry if necessary.
///
/// The Job is consumed by this method.
pub(crate) async fn save(self, context: &Context) -> Result<()> {
info!(context, "saving job {:?}", self);
if self.job_id != 0 {
context
.sql
.execute(
"UPDATE jobs SET desired_timestamp=?, tries=? WHERE id=?;",
(
self.desired_timestamp,
i64::from(self.tries),
self.job_id as i32,
),
)
.await?;
} else {
context.sql.execute(
"INSERT INTO jobs (added_timestamp, action, foreign_id, desired_timestamp) VALUES (?,?,?,?);",
(
self.added_timestamp,
self.action,
self.foreign_id,
self.desired_timestamp
)
).await?;
}
Ok(())
}
}
pub(crate) enum Connection<'a> {
Inbox(&'a mut Imap),
}
impl<'a> Connection<'a> {
fn inbox(&mut self) -> &mut Imap {
match self {
Connection::Inbox(imap) => imap,
}
}
}
pub(crate) async fn perform_job(context: &Context, mut connection: Connection<'_>, mut job: Job) {
info!(context, "Job {} started...", &job);
let try_res = match perform_job_action(context, &job, &mut connection, 0).await {
Status::RetryNow => perform_job_action(context, &job, &mut connection, 1).await,
x => x,
};
match try_res {
Status::RetryNow => {
let tries = job.tries + 1;
if tries < JOB_RETRIES {
info!(context, "Increase job {job} tries to {tries}.");
job.tries = tries;
let time_offset = get_backoff_time_offset(tries);
job.desired_timestamp = time() + time_offset;
info!(
context,
"job #{} not succeeded on try #{}, retry in {} seconds.",
job.job_id,
tries,
time_offset
);
job.save(context).await.unwrap_or_else(|err| {
error!(context, "Failed to save job: {err:#}.");
});
} else {
info!(
context,
"Remove job {job} as it exhausted {JOB_RETRIES} retries."
);
job.delete(context).await.unwrap_or_else(|err| {
error!(context, "Failed to delete job: {err:#}.");
});
}
}
Status::Finished(res) => {
if let Err(err) = res {
warn!(context, "Remove job {job} as it failed with error {err:#}.");
} else {
info!(context, "Remove job {job} as it succeeded.");
}
job.delete(context).await.unwrap_or_else(|err| {
error!(context, "failed to delete job: {:#}", err);
});
}
}
}
async fn perform_job_action(
context: &Context,
job: &Job,
connection: &mut Connection<'_>,
tries: u32,
) -> Status {
info!(context, "Begin immediate try {tries} of job {job}.");
let try_res = match job.action {
Action::DownloadMsg => job.download_msg(context, connection.inbox()).await,
};
info!(context, "Finished immediate try {tries} of job {job}.");
try_res
}
fn get_backoff_time_offset(tries: u32) -> i64 {
// Exponential backoff
let n = 2_i32.pow(tries - 1) * 60;
let mut rng = thread_rng();
let r: i32 = rng.gen();
let mut seconds = r % (n + 1);
if seconds < 1 {
seconds = 1;
}
i64::from(seconds)
}
pub(crate) async fn schedule_resync(context: &Context) -> Result<()> {
context.resync_request.store(true, Ordering::Relaxed);
context
.scheduler
.interrupt_inbox(InterruptInfo {
probe_network: false,
})
.await;
Ok(())
}
/// Adds a job to the database, scheduling it.
pub async fn add(context: &Context, job: Job) -> Result<()> {
job.save(context).await.context("failed to save job")?;
info!(context, "Interrupt: IMAP.");
context
.scheduler
.interrupt_inbox(InterruptInfo::new(false))
.await;
Ok(())
}
/// Load jobs from the database.
///
/// The `probe_network` parameter decides how to query
/// jobs, this is tricky and probably wrong currently. Look at the
/// SQL queries for details.
pub(crate) async fn load_next(context: &Context, info: &InterruptInfo) -> Result<Option<Job>> {
info!(context, "Loading job.");
let query;
let params;
let t = time();
if !info.probe_network {
// processing for first-try and after backoff-timeouts:
// process jobs in the order they were added.
query = r#"
SELECT id, action, foreign_id, param, added_timestamp, desired_timestamp, tries
FROM jobs
WHERE desired_timestamp<=?
ORDER BY action DESC, added_timestamp
LIMIT 1;
"#;
params = vec![t];
} else {
// processing after call to dc_maybe_network():
// process _all_ pending jobs that failed before
// in the order of their backoff-times.
query = r#"
SELECT id, action, foreign_id, param, added_timestamp, desired_timestamp, tries
FROM jobs
WHERE tries>0
ORDER BY desired_timestamp, action DESC
LIMIT 1;
"#;
params = vec![];
};
loop {
let job_res = context
.sql
.query_row_optional(query, rusqlite::params_from_iter(params.clone()), |row| {
let job = Job {
job_id: row.get("id")?,
action: row.get("action")?,
foreign_id: row.get("foreign_id")?,
desired_timestamp: row.get("desired_timestamp")?,
added_timestamp: row.get("added_timestamp")?,
tries: row.get("tries")?,
};
Ok(job)
})
.await;
match job_res {
Ok(job) => return Ok(job),
Err(err) => {
// Remove invalid job from the DB
info!(context, "Cleaning up job, because of {err:#}.");
// TODO: improve by only doing a single query
let id = context
.sql
.query_row(query, rusqlite::params_from_iter(params.clone()), |row| {
row.get::<_, i32>(0)
})
.await
.context("failed to retrieve invalid job ID from the database")?;
context
.sql
.execute("DELETE FROM jobs WHERE id=?;", (id,))
.await
.with_context(|| format!("failed to delete invalid job {id}"))?;
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::test_utils::TestContext;
async fn insert_job(context: &Context, foreign_id: i64, valid: bool) {
let now = time();
context
.sql
.execute(
"INSERT INTO jobs
(added_timestamp, action, foreign_id, desired_timestamp)
VALUES (?, ?, ?, ?);",
(
now,
if valid {
Action::DownloadMsg as i32
} else {
-1
},
foreign_id,
now,
),
)
.await
.unwrap();
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_load_next_job_two() -> Result<()> {
// We want to ensure that loading jobs skips over jobs which
// fails to load from the database instead of failing to load
// all jobs.
let t = TestContext::new().await;
insert_job(&t, 1, false).await; // This can not be loaded into Job struct.
let jobs = load_next(&t, &InterruptInfo::new(false)).await?;
assert!(jobs.is_none());
insert_job(&t, 1, true).await;
let jobs = load_next(&t, &InterruptInfo::new(false)).await?;
assert!(jobs.is_some());
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_load_next_job_one() -> Result<()> {
let t = TestContext::new().await;
insert_job(&t, 1, true).await;
let jobs = load_next(&t, &InterruptInfo::new(false)).await?;
assert!(jobs.is_some());
Ok(())
}
}

View File

@@ -65,10 +65,6 @@ mod e2ee;
pub mod ephemeral;
mod imap;
pub mod imex;
pub mod release;
mod scheduler;
#[macro_use]
mod job;
pub mod key;
mod keyring;
pub mod location;
@@ -84,6 +80,8 @@ pub mod provider;
pub mod qr;
pub mod qr_code_generator;
pub mod quota;
pub mod release;
mod scheduler;
pub mod securejoin;
mod simplify;
mod smtp;

View File

@@ -359,9 +359,10 @@ impl<'a> MimeFactory<'a> {
async fn should_do_gossip(&self, context: &Context) -> Result<bool> {
match &self.loaded {
Loaded::Message { chat } => {
// beside key- and member-changes, force re-gossip every 48 hours
// beside key- and member-changes, force a periodic re-gossip.
let gossiped_timestamp = chat.id.get_gossiped_timestamp(context).await?;
if time() > gossiped_timestamp + (2 * 24 * 60 * 60) {
let gossip_period = context.get_config_i64(Config::GossipPeriod).await?;
if time() >= gossiped_timestamp + gossip_period {
Ok(true)
} else {
let cmd = self.msg.param.get_cmd();
@@ -414,7 +415,9 @@ impl<'a> MimeFactory<'a> {
return Ok(self.msg.subject.clone());
}
if chat.typ == Chattype::Group && quoted_msg_subject.is_none_or_empty() {
if (chat.typ == Chattype::Group || chat.typ == Chattype::Broadcast)
&& quoted_msg_subject.is_none_or_empty()
{
let re = if self.in_reply_to.is_empty() {
""
} else {
@@ -423,15 +426,13 @@ impl<'a> MimeFactory<'a> {
return Ok(format!("{}{}", re, chat.name));
}
if chat.typ != Chattype::Broadcast {
let parent_subject = if quoted_msg_subject.is_none_or_empty() {
chat.param.get(Param::LastSubject)
} else {
quoted_msg_subject.as_deref()
};
if let Some(last_subject) = parent_subject {
return Ok(format!("Re: {}", remove_subject_prefix(last_subject)));
}
let parent_subject = if quoted_msg_subject.is_none_or_empty() {
chat.param.get(Param::LastSubject)
} else {
quoted_msg_subject.as_deref()
};
if let Some(last_subject) = parent_subject {
return Ok(format!("Re: {}", remove_subject_prefix(last_subject)));
}
let self_name = &match context.get_config(Config::Displayname).await? {
@@ -593,6 +594,15 @@ impl<'a> MimeFactory<'a> {
));
}
if let Loaded::Message { chat } = &self.loaded {
if chat.typ == Chattype::Broadcast {
headers.protected.push(Header::new(
"List-ID".into(),
format!("{} <{}>", chat.name, chat.grpid),
));
}
}
// Non-standard headers.
headers
.unprotected
@@ -2331,7 +2341,7 @@ mod tests {
// Now Bob can send an encrypted message to Alice.
let mut msg = Message::new(Viewtype::File);
// Long messages are truncated and MimeMessage::decoded_data is set for them. We need
// decoded_data to check presense of the necessary headers.
// decoded_data to check presence of the necessary headers.
msg.set_text("a".repeat(constants::DC_DESIRED_TEXT_LEN + 1));
msg.set_file_from_bytes(&bob, "foo.bar", "content".as_bytes(), None)
.await?;

View File

@@ -123,23 +123,6 @@ pub(crate) enum AvatarAction {
Change(String),
}
#[derive(Debug, PartialEq)]
pub(crate) enum MailinglistType {
/// The message belongs to a mailing list and has a `ListId:`-header
/// that should be used to get a unique id.
ListIdBased,
/// The message belongs to a mailing list, but there is no `ListId:`-header;
/// `Sender:`-header should be used to get a unique id.
/// This method is used by implementations as Majordomo.
/// Note, that the `Sender:` header alone is not sufficient to detect these lists,
/// `get_mailinglist_type()` check additional conditions therefore.
SenderBased,
/// The message does not belong to a mailing list.
None,
}
/// System message type.
#[derive(
Debug, Default, Display, Clone, Copy, PartialEq, Eq, FromPrimitive, ToPrimitive, ToSql, FromSql,
@@ -1340,26 +1323,28 @@ impl MimeMessage {
self.parts.push(part);
}
pub(crate) fn get_mailinglist_type(&self) -> MailinglistType {
if self.get_header(HeaderDef::ListId).is_some() {
return MailinglistType::ListIdBased;
} else if self.get_header(HeaderDef::Sender).is_some() {
pub(crate) fn get_mailinglist_header(&self) -> Option<&str> {
if let Some(list_id) = self.get_header(HeaderDef::ListId) {
// The message belongs to a mailing list and has a `ListId:`-header
// that should be used to get a unique id.
return Some(list_id);
} else if let Some(sender) = self.get_header(HeaderDef::Sender) {
// the `Sender:`-header alone is no indicator for mailing list
// as also used for bot-impersonation via `set_override_sender_name()`
if let Some(precedence) = self.get_header(HeaderDef::Precedence) {
if precedence == "list" || precedence == "bulk" {
return MailinglistType::SenderBased;
// The message belongs to a mailing list, but there is no `ListId:`-header;
// `Sender:`-header is be used to get a unique id.
// This method is used by implementations as Majordomo.
return Some(sender);
}
}
}
MailinglistType::None
None
}
pub(crate) fn is_mailinglist_message(&self) -> bool {
match self.get_mailinglist_type() {
MailinglistType::ListIdBased | MailinglistType::SenderBased => true,
MailinglistType::None => false,
}
self.get_mailinglist_header().is_some()
}
pub fn repl_msg_by_error(&mut self, error_msg: &str) {
@@ -1893,7 +1878,7 @@ fn get_mime_type(
} else {
// Enacapsulated messages, see <https://www.w3.org/Protocols/rfc1341/7_3_Message.html>
// Also used as part "message/disposition-notification" of "multipart/report", which, however, will
// be handled separatedly.
// be handled separately.
// I've not seen any messages using this, so we do not attach these parts (maybe they're used to attach replies,
// which are unwanted at all).
// For now, we skip these parts at all; if desired, we could return DcMimeType::File/DC_MSG_File

View File

@@ -21,7 +21,7 @@ use crate::stock_str;
/// Type of the public key stored inside the peerstate.
#[derive(Debug)]
pub enum PeerstateKeyType {
/// Pubilc key sent in the `Autocrypt-Gossip` header.
/// Public key sent in the `Autocrypt-Gossip` header.
GossipKey,
/// Public key sent in the `Autocrypt` header.

View File

@@ -1,7 +1,6 @@
//! # Support for IMAP QUOTA extension.
use std::collections::BTreeMap;
use std::sync::atomic::Ordering;
use anyhow::{anyhow, Context as _, Result};
use async_imap::types::{Quota, QuotaResource};
@@ -13,7 +12,6 @@ use crate::imap::scan_folders::get_watched_folders;
use crate::imap::session::Session as ImapSession;
use crate::imap::Imap;
use crate::message::{Message, Viewtype};
use crate::scheduler::InterruptInfo;
use crate::tools::time;
use crate::{stock_str, EventType};
@@ -34,17 +32,12 @@ pub const QUOTA_ERROR_THRESHOLD_PERCENTAGE: u64 = 95;
/// providers report bad values and we would then spam the user.
pub const QUOTA_ALLCLEAR_PERCENTAGE: u64 = 75;
/// if recent quota is older,
/// it is re-fetched on dc_get_connectivity_html()
pub const QUOTA_MAX_AGE_SECONDS: i64 = 60;
/// Server quota information with an update timestamp.
#[derive(Debug)]
pub struct QuotaInfo {
/// Recently loaded quota information.
/// set to `Err()` if the provider does not support quota or on other errors,
/// set to `Ok()` for valid quota information.
/// Updated by `Action::UpdateRecentQuota`
pub(crate) recent: Result<BTreeMap<String, Vec<QuotaResource>>>,
/// Timestamp when structure was modified.
@@ -110,18 +103,6 @@ pub fn needs_quota_warning(curr_percentage: u64, warned_at_percentage: u64) -> b
}
impl Context {
// Adds a job to update `quota.recent`
pub(crate) async fn schedule_quota_update(&self) -> Result<()> {
let requested = self.quota_update_request.swap(true, Ordering::Relaxed);
if !requested {
// Quota update was not requested before.
self.scheduler
.interrupt_inbox(InterruptInfo::new(false))
.await;
}
Ok(())
}
/// Updates `quota.recent`, sets `quota.modified` to the current time
/// and emits an event to let the UIs update connectivity view.
///
@@ -130,8 +111,6 @@ impl Context {
/// As the message is added only once, the user is not spammed
/// in case for some providers the quota is always at ~100%
/// and new space is allocated as needed.
///
/// Called in response to `Action::UpdateRecentQuota`.
pub(crate) async fn update_recent_quota(&self, imap: &mut Imap) -> Result<()> {
if let Err(err) = imap.prepare(self).await {
warn!(self, "could not connect: {:#}", err);
@@ -166,9 +145,6 @@ impl Context {
}
}
// Clear the request to update quota.
self.quota_update_request.store(false, Ordering::Relaxed);
*self.quota.write().await = Some(QuotaInfo {
recent: quota,
modified: time(),

View File

@@ -28,9 +28,7 @@ use crate::log::LogExt;
use crate::message::{
self, rfc724_mid_exists, Message, MessageState, MessengerMessage, MsgId, Viewtype,
};
use crate::mimeparser::{
parse_message_ids, AvatarAction, MailinglistType, MimeMessage, SystemMessage,
};
use crate::mimeparser::{parse_message_ids, AvatarAction, MimeMessage, SystemMessage};
use crate::param::{Param, Params};
use crate::peerstate::{Peerstate, PeerstateKeyType, PeerstateVerifiedStatus};
use crate::reaction::{set_msg_reaction, Reaction};
@@ -654,45 +652,23 @@ async fn add_parts(
if chat_id.is_none() {
// check if the message belongs to a mailing list
match mime_parser.get_mailinglist_type() {
MailinglistType::ListIdBased => {
if let Some(list_id) = mime_parser.get_header(HeaderDef::ListId) {
if let Some((new_chat_id, new_chat_id_blocked)) =
create_or_lookup_mailinglist(
context,
allow_creation,
list_id,
mime_parser,
)
.await?
{
chat_id = Some(new_chat_id);
chat_id_blocked = new_chat_id_blocked;
}
}
if let Some(mailinglist_header) = mime_parser.get_mailinglist_header() {
if let Some((new_chat_id, new_chat_id_blocked)) = create_or_lookup_mailinglist(
context,
allow_creation,
mailinglist_header,
mime_parser,
)
.await?
{
chat_id = Some(new_chat_id);
chat_id_blocked = new_chat_id_blocked;
}
MailinglistType::SenderBased => {
if let Some(sender) = mime_parser.get_header(HeaderDef::Sender) {
if let Some((new_chat_id, new_chat_id_blocked)) =
create_or_lookup_mailinglist(
context,
allow_creation,
sender,
mime_parser,
)
.await?
{
chat_id = Some(new_chat_id);
chat_id_blocked = new_chat_id_blocked;
}
}
}
MailinglistType::None => {}
}
}
if let Some(chat_id) = chat_id {
apply_mailinglist_changes(context, mime_parser, chat_id).await?;
apply_mailinglist_changes(context, mime_parser, sent_timestamp, chat_id).await?;
}
// if contact renaming is prevented (for mailinglists and bots),
@@ -1921,6 +1897,8 @@ async fn apply_group_changes(
Ok(better_msg)
}
static LIST_ID_REGEX: Lazy<Regex> = Lazy::new(|| Regex::new(r"^(.+)<(.+)>$").unwrap());
/// Create or lookup a mailing list chat.
///
/// `list_id_header` contains the Id that must be used for the mailing list
@@ -1937,23 +1915,71 @@ async fn create_or_lookup_mailinglist(
list_id_header: &str,
mime_parser: &MimeMessage,
) -> Result<Option<(ChatId, Blocked)>> {
static LIST_ID: Lazy<Regex> = Lazy::new(|| Regex::new(r"^(.+)<(.+)>$").unwrap());
let (mut name, listid) = match LIST_ID.captures(list_id_header) {
Some(cap) => (cap[1].trim().to_string(), cap[2].trim().to_string()),
None => (
"".to_string(),
list_id_header
.trim()
.trim_start_matches('<')
.trim_end_matches('>')
.to_string(),
),
let listid = match LIST_ID_REGEX.captures(list_id_header) {
Some(cap) => cap[2].trim().to_string(),
None => list_id_header
.trim()
.trim_start_matches('<')
.trim_end_matches('>')
.to_string(),
};
if let Some((chat_id, _, blocked)) = chat::get_chat_id_by_grpid(context, &listid).await? {
return Ok(Some((chat_id, blocked)));
}
let name = compute_mailinglist_name(list_id_header, &listid, mime_parser);
if allow_creation {
// list does not exist but should be created
let param = mime_parser.list_post.as_ref().map(|list_post| {
let mut p = Params::new();
p.set(Param::ListPost, list_post);
p.to_string()
});
let is_bot = context.get_config_bool(Config::Bot).await?;
let blocked = if is_bot {
Blocked::Not
} else {
Blocked::Request
};
let chat_id = ChatId::create_multiuser_record(
context,
Chattype::Mailinglist,
&listid,
&name,
blocked,
ProtectionStatus::Unprotected,
param,
)
.await
.with_context(|| {
format!(
"failed to create mailinglist '{}' for grpid={}",
&name, &listid
)
})?;
chat::add_to_chat_contacts_table(context, chat_id, &[ContactId::SELF]).await?;
Ok(Some((chat_id, blocked)))
} else {
info!(context, "Creating list forbidden by caller.");
Ok(None)
}
}
#[allow(clippy::indexing_slicing)]
fn compute_mailinglist_name(
list_id_header: &str,
listid: &str,
mime_parser: &MimeMessage,
) -> String {
let mut name = match LIST_ID_REGEX.captures(list_id_header) {
Some(cap) => cap[1].trim().to_string(),
None => "".to_string(),
};
// for mailchimp lists, the name in `ListId` is just a long number.
// a usable name for these lists is in the `From` header
// and we can detect these lists by a unique `ListId`-suffix.
@@ -1997,50 +2023,14 @@ async fn create_or_lookup_mailinglist(
// 51231231231231231231231232869f58.xing.com -> xing.com
static PREFIX_32_CHARS_HEX: Lazy<Regex> =
Lazy::new(|| Regex::new(r"([0-9a-fA-F]{32})\.(.{6,})").unwrap());
if let Some(cap) = PREFIX_32_CHARS_HEX.captures(&listid) {
if let Some(cap) = PREFIX_32_CHARS_HEX.captures(listid) {
name = cap[2].to_string();
} else {
name = listid.clone();
name = listid.to_string();
}
}
if allow_creation {
// list does not exist but should be created
let param = mime_parser.list_post.as_ref().map(|list_post| {
let mut p = Params::new();
p.set(Param::ListPost, list_post);
p.to_string()
});
let is_bot = context.get_config_bool(Config::Bot).await?;
let blocked = if is_bot {
Blocked::Not
} else {
Blocked::Request
};
let chat_id = ChatId::create_multiuser_record(
context,
Chattype::Mailinglist,
&listid,
&name,
blocked,
ProtectionStatus::Unprotected,
param,
)
.await
.with_context(|| {
format!(
"failed to create mailinglist '{}' for grpid={}",
&name, &listid
)
})?;
chat::add_to_chat_contacts_table(context, chat_id, &[ContactId::SELF]).await?;
Ok(Some((chat_id, blocked)))
} else {
info!(context, "Creating list forbidden by caller.");
Ok(None)
}
strip_rtlo_characters(&name)
}
/// Set ListId param on the contact and ListPost param the chat.
@@ -2049,9 +2039,10 @@ async fn create_or_lookup_mailinglist(
async fn apply_mailinglist_changes(
context: &Context,
mime_parser: &MimeMessage,
sent_timestamp: i64,
chat_id: ChatId,
) -> Result<()> {
let Some(list_post) = &mime_parser.list_post else {
let Some(mailinglist_header) = mime_parser.get_mailinglist_header() else {
return Ok(());
};
@@ -2061,6 +2052,24 @@ async fn apply_mailinglist_changes(
}
let listid = &chat.grpid;
let new_name = compute_mailinglist_name(mailinglist_header, listid, mime_parser);
if chat.name != new_name
&& chat_id
.update_timestamp(context, Param::GroupNameTimestamp, sent_timestamp)
.await?
{
info!(context, "Updating listname for chat {chat_id}.");
context
.sql
.execute("UPDATE chats SET name=? WHERE id=?;", (new_name, chat_id))
.await?;
context.emit_event(EventType::ChatModified(chat_id));
}
let Some(list_post) = &mime_parser.list_post else {
return Ok(());
};
let list_post = match ContactAddress::new(list_post) {
Ok(list_post) => list_post,
Err(err) => {

View File

@@ -13,16 +13,16 @@ use self::connectivity::ConnectivityStore;
use crate::config::Config;
use crate::contact::{ContactId, RecentlySeenLoop};
use crate::context::Context;
use crate::download::download_msg;
use crate::ephemeral::{self, delete_expired_imap_messages};
use crate::events::EventType;
use crate::imap::{FolderMeaning, Imap};
use crate::job;
use crate::location;
use crate::log::LogExt;
use crate::message::MsgId;
use crate::smtp::{send_smtp_messages, Smtp};
use crate::sql;
use crate::tools::time;
use crate::tools::{duration_to_str, maybe_add_time_based_warnings};
use crate::tools::{duration_to_str, maybe_add_time_based_warnings, time};
pub(crate) mod connectivity;
@@ -323,6 +323,37 @@ pub(crate) struct Scheduler {
recently_seen_loop: RecentlySeenLoop,
}
async fn download_msgs(context: &Context, imap: &mut Imap) -> Result<()> {
let msg_ids = context
.sql
.query_map(
"SELECT msg_id FROM download",
(),
|row| {
let msg_id: MsgId = row.get(0)?;
Ok(msg_id)
},
|rowids| {
rowids
.collect::<std::result::Result<Vec<_>, _>>()
.map_err(Into::into)
},
)
.await?;
for msg_id in msg_ids {
if let Err(err) = download_msg(context, msg_id, imap).await {
warn!(context, "Failed to download message {msg_id}: {:#}.", err);
}
context
.sql
.execute("DELETE FROM download WHERE msg_id=?", (msg_id,))
.await?;
}
Ok(())
}
async fn inbox_loop(
ctx: Context,
started: oneshot::Sender<()>,
@@ -344,79 +375,76 @@ async fn inbox_loop(
return;
};
let mut info = InterruptInfo::default();
loop {
let job = match job::load_next(&ctx, &info).await {
Err(err) => {
error!(ctx, "Failed loading job from the database: {:#}.", err);
None
}
Ok(job) => job,
};
{
// Update quota no more than once a minute.
let quota_needs_update = {
let quota = ctx.quota.read().await;
quota
.as_ref()
.filter(|quota| quota.modified + 60 > time())
.is_none()
};
match job {
Some(job) => {
job::perform_job(&ctx, job::Connection::Inbox(&mut connection), job).await;
info = Default::default();
}
None => {
let quota_requested = ctx.quota_update_request.swap(false, Ordering::Relaxed);
if quota_requested {
if let Err(err) = ctx.update_recent_quota(&mut connection).await {
warn!(ctx, "Failed to update quota: {:#}.", err);
}
if quota_needs_update {
if let Err(err) = ctx.update_recent_quota(&mut connection).await {
warn!(ctx, "Failed to update quota: {:#}.", err);
}
let resync_requested = ctx.resync_request.swap(false, Ordering::Relaxed);
if resync_requested {
if let Err(err) = connection.resync_folders(&ctx).await {
warn!(ctx, "Failed to resync folders: {:#}.", err);
ctx.resync_request.store(true, Ordering::Relaxed);
}
}
maybe_add_time_based_warnings(&ctx).await;
match ctx.get_config_i64(Config::LastHousekeeping).await {
Ok(last_housekeeping_time) => {
let next_housekeeping_time =
last_housekeeping_time.saturating_add(60 * 60 * 24);
if next_housekeeping_time <= time() {
sql::housekeeping(&ctx).await.log_err(&ctx).ok();
}
}
Err(err) => {
warn!(ctx, "Failed to get last housekeeping time: {}", err);
}
};
match ctx.get_config_bool(Config::FetchedExistingMsgs).await {
Ok(fetched_existing_msgs) => {
if !fetched_existing_msgs {
// Consider it done even if we fail.
//
// This operation is not critical enough to retry,
// especially if the error is persistent.
if let Err(err) =
ctx.set_config_bool(Config::FetchedExistingMsgs, true).await
{
warn!(ctx, "Can't set Config::FetchedExistingMsgs: {:#}", err);
}
if let Err(err) = connection.fetch_existing_msgs(&ctx).await {
warn!(ctx, "Failed to fetch existing messages: {:#}", err);
connection.trigger_reconnect(&ctx);
}
}
}
Err(err) => {
warn!(ctx, "Can't get Config::FetchedExistingMsgs: {:#}", err);
}
}
info = fetch_idle(&ctx, &mut connection, FolderMeaning::Inbox).await;
}
}
let resync_requested = ctx.resync_request.swap(false, Ordering::Relaxed);
if resync_requested {
if let Err(err) = connection.resync_folders(&ctx).await {
warn!(ctx, "Failed to resync folders: {:#}.", err);
ctx.resync_request.store(true, Ordering::Relaxed);
}
}
maybe_add_time_based_warnings(&ctx).await;
match ctx.get_config_i64(Config::LastHousekeeping).await {
Ok(last_housekeeping_time) => {
let next_housekeeping_time =
last_housekeeping_time.saturating_add(60 * 60 * 24);
if next_housekeeping_time <= time() {
sql::housekeeping(&ctx).await.log_err(&ctx).ok();
}
}
Err(err) => {
warn!(ctx, "Failed to get last housekeeping time: {}", err);
}
};
match ctx.get_config_bool(Config::FetchedExistingMsgs).await {
Ok(fetched_existing_msgs) => {
if !fetched_existing_msgs {
// Consider it done even if we fail.
//
// This operation is not critical enough to retry,
// especially if the error is persistent.
if let Err(err) =
ctx.set_config_bool(Config::FetchedExistingMsgs, true).await
{
warn!(ctx, "Can't set Config::FetchedExistingMsgs: {:#}", err);
}
if let Err(err) = connection.fetch_existing_msgs(&ctx).await {
warn!(ctx, "Failed to fetch existing messages: {:#}", err);
connection.trigger_reconnect(&ctx);
}
}
}
Err(err) => {
warn!(ctx, "Can't get Config::FetchedExistingMsgs: {:#}", err);
}
}
if let Err(err) = download_msgs(&ctx, &mut connection).await {
warn!(ctx, "Failed to download messages: {:#}", err);
}
fetch_idle(&ctx, &mut connection, FolderMeaning::Inbox).await;
}
};

View File

@@ -8,10 +8,7 @@ use tokio::sync::Mutex;
use crate::events::EventType;
use crate::imap::{scan_folders::get_watched_folder_configs, FolderMeaning};
use crate::quota::{
QUOTA_ERROR_THRESHOLD_PERCENTAGE, QUOTA_MAX_AGE_SECONDS, QUOTA_WARN_THRESHOLD_PERCENTAGE,
};
use crate::tools::time;
use crate::quota::{QUOTA_ERROR_THRESHOLD_PERCENTAGE, QUOTA_WARN_THRESHOLD_PERCENTAGE};
use crate::{context::Context, log::LogExt};
use crate::{stock_str, tools};
@@ -472,14 +469,9 @@ impl Context {
ret += format!("<li>{e}</li>").as_str();
}
}
if quota.modified + QUOTA_MAX_AGE_SECONDS < time() {
self.schedule_quota_update().await?;
}
} else {
let not_connected = stock_str::not_connected(self).await;
ret += &format!("<li>{not_connected}</li>");
self.schedule_quota_update().await?;
}
ret += "</ul>";

View File

@@ -673,12 +673,14 @@ pub(crate) async fn send_smtp_messages(context: &Context, connection: &mut Smtp)
/// On failure returns an error without removing any `smtp_mdns` entries, the caller is responsible
/// for removing the corresponding entry to prevent endless loop in case the entry is invalid, e.g.
/// points to non-existent message or contact.
///
/// Returns true on success, false on temporary error.
async fn send_mdn_msg_id(
context: &Context,
msg_id: MsgId,
contact_id: ContactId,
smtp: &mut Smtp,
) -> Result<()> {
) -> Result<bool> {
let contact = Contact::get_by_id(context, contact_id).await?;
if contact.is_blocked() {
return Err(format_err!("Contact is blocked"));
@@ -730,14 +732,14 @@ async fn send_mdn_msg_id(
.execute(&q, rusqlite::params_from_iter(additional_msg_ids))
.await?;
}
Ok(())
Ok(true)
}
SendResult::Retry => {
info!(
context,
"Temporary SMTP failure while sending an MDN for {}", msg_id
);
Ok(())
Ok(false)
}
SendResult::Failure(err) => Err(err),
}
@@ -784,15 +786,20 @@ async fn send_mdn(context: &Context, smtp: &mut Smtp) -> Result<bool> {
.await
.context("failed to update MDN retries count")?;
if let Err(err) = send_mdn_msg_id(context, msg_id, contact_id, smtp).await {
let res = send_mdn_msg_id(context, msg_id, contact_id, smtp).await;
if let Err(ref err) = res {
// If there is an error, for example there is no message corresponding to the msg_id in the
// database, do not try to send this MDN again.
warn!(
context,
"Error sending MDN for {msg_id}, removing it: {err:#}."
);
context
.sql
.execute("DELETE FROM smtp_mdns WHERE msg_id = ?", (msg_id,))
.await?;
Err(err)
} else {
Ok(true)
}
// If there's a temporary error, pretend there are no more MDNs to send. It's unlikely that
// other MDNs could be sent successfully in case of connectivity problems.
res
}

View File

@@ -686,6 +686,7 @@ fn new_connection(path: &Path, passphrase: &str) -> Result<Connection> {
PRAGMA secure_delete=on;
PRAGMA busy_timeout = 0; -- fail immediately
PRAGMA temp_store=memory; -- Avoid SQLITE_IOERR_GETTEMPPATH errors on Android
PRAGMA soft_heap_limit = 8388608; -- 8 MiB limit, same as set in Android SQLiteDatabase.
PRAGMA foreign_keys=on;
",
)?;
@@ -741,8 +742,6 @@ pub async fn housekeeping(context: &Context) -> Result<()> {
warn!(context, "Failed to deduplicate peerstates: {:#}.", err)
}
context.schedule_quota_update().await?;
// Try to clear the freelist to free some space on the disk. This
// only works if auto_vacuum is enabled.
match context

View File

@@ -730,6 +730,15 @@ CREATE INDEX smtp_messageid ON imap(rfc724_mid);
)
.await?;
}
if dbversion < 102 {
sql.execute_migration(
"CREATE TABLE download (
msg_id INTEGER NOT NULL -- id of the message stub in msgs table
)",
102,
)
.await?;
}
let new_version = sql
.get_raw_config_int(VERSION_CFG)

View File

@@ -1550,7 +1550,7 @@ mod tests {
let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
assert_eq!(chats.len(), 0);
// a subsequent call to update_device_chats() must not re-add manally deleted messages or chats
// a subsequent call to update_device_chats() must not re-add manually deleted messages or chats
t.update_device_chats().await.unwrap();
let chats = Chatlist::try_load(&t, 0, None, None).await.unwrap();
assert_eq!(chats.len(), 0);

View File

@@ -268,7 +268,7 @@ pub(crate) fn create_id() -> String {
/// Function generates a Message-ID that can be used for a new outgoing message.
/// - this function is called for all outgoing messages.
/// - the message ID should be globally unique
/// - do not add a counter or any private data as this leaks information unncessarily
/// - do not add a counter or any private data as this leaks information unnecessarily
pub(crate) fn create_outgoing_rfc724_mid(grpid: Option<&str>, from_addr: &str) -> String {
let hostname = from_addr
.find('@')
@@ -700,7 +700,7 @@ pub(crate) fn buf_decompress(buf: &[u8]) -> Result<Vec<u8>> {
}
const RTLO_CHARACTERS: [char; 5] = ['\u{202A}', '\u{202B}', '\u{202C}', '\u{202D}', '\u{202E}'];
/// This method strips all occurances of the RTLO Unicode character.
/// This method strips all occurrences of the RTLO Unicode character.
/// [Why is this needed](https://github.com/deltachat/deltachat-core-rust/issues/3479)?
pub(crate) fn strip_rtlo_characters(input_str: &str) -> String {
input_str.replace(|char| RTLO_CHARACTERS.contains(&char), "")