Compare commits

..

190 Commits

Author SHA1 Message Date
Sebastian Klähn
9ac9a4a214 Add bot contact to every account 2024-10-23 14:22:37 +02:00
link2xt
aa3ef5011b build: nix flake update fenix 2024-10-23 03:31:39 +00:00
link2xt
1d3072c287 build: nix flake update nixpkgs 2024-10-23 03:19:33 +00:00
link2xt
4fb59177fa chore(release): prepare for 1.148.1 2024-10-23 02:37:22 +00:00
link2xt
d841bcb41e Revert "build: nix flake update"
This reverts commit 6f22ce2722.
2024-10-23 02:07:22 +00:00
link2xt
d205bc410b chore(release): prepare for 1.148.0 2024-10-23 00:27:49 +00:00
bjoern
0d573ac037 feat: add delta chat logo to QR codes (#6093)
the chosen error correction allows tolerates about 15% "erroneous
codewords", the logo is of a similar size as the old avatars,
so it should be fine.
2024-10-23 01:43:50 +02:00
link2xt
a55e33fbc7 fix(sql): run PRAGMA incremental_vacuum on a write connection
Otherwise it always fails with SQLITE_READONLY:
```
WARNING src/sql.rs:769: Failed to run incremental vacuum: attempt to write a readonly database: Error code 8: Attempt to write a readonly database.
```
2024-10-22 23:23:57 +00:00
bjoern
839b0e94af api: create QR codes from any data (#6090)
this PR adds a function that can be used to create any QR code, in a raw
form.

this can be used to create add-contact as well as add-second-device QR
codes (eg. `dc_create_qr_svg(dc_get_securejoin_qr())`) - as well as for
other QR codes as proxies.

the disadvantage of the rich-formatted QR codes as created by
`dc_get_securejoin_qr_svg()` and `dc_backup_provider_get_qr_svg()` were:

- they do not look good and cannot interact with UI layout wise (but
also tapping eg. an address is not easily possible)
- esp. text really looks bad. even with
[some](e5dc8fe3d8)
[hacks](https://github.com/deltachat/deltachat-android/pull/2215) it
[stays buggy](https://github.com/deltachat/deltachat-ios/issues/2200);
the bugs mainly come from different SVG implementation, all need their
own quirks
- accessibility is probably bad as well

we thought that time, SVG is a great thing for QR codes, but apart from
basic geometrics, it is not.

so, we avoid text, this also means to avoid putting an avatar in the
middle of the QR code (we can put some generic symbol there, eg.
different ones for add-contact and add-second-device).

while this looks like a degradation, also other messengers use more raw
QR codes. also, we removed many data from the QR code anyway, eg. the
email address is no longer there. that time, sharing QR images was more
a thing, meanwhile we have invite links, that are much better for that
purpose.

in theory, we could also leave the SVG path completely and go for PNG -
which we did not that time as PNG and text looks bad, as the system font
is not easily usable :) but going for PNG would add further challenges
as passing binary data around, and also UI-implemtation-wise, that would
be a larger step. so, let's stay with SVG in a form we know is
compatible.

the old QR code functions are deprecated.
2024-10-22 21:49:45 +02:00
bjoern
f2e600dc55 feat: internal profile names (#6088)
this PR allows setting a "private tag" for a profile, see
https://github.com/deltachat/deltachat-android/pull/3373 for a possible
UI.

currently, the core does not do anything with the tag (so, it could also
be a ui.-config option), however, this may change in the future - it
might bet synced, and become also otherwise useful in core. also, having
this in core is better documentation-wise, as otherwise each UI easily
does its own things :)
2024-10-22 09:43:36 +02:00
bjoern
61fd0d400f notify adding reactions (#6072)
this PR adds an event for reactions received for one's own messages.

this will allow UIs to add notification for these reactions.

**Screenshots** at https://github.com/deltachat/deltachat-ios/pull/2331:

---------

Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
2024-10-21 21:35:03 +02:00
link2xt
7424d06416 refactor(sql): move write mutex into connection pool 2024-10-21 19:14:37 +00:00
link2xt
aa71fbe04c refactor: resultify get_self_fingerprint() 2024-10-21 13:03:58 +00:00
link2xt
c5cadd9991 feat: add in-memory cache for DNS
This adds "stale-while-revalidate" in-memory cache for DNS. Instead of
calling `tokio::net::lookup_host` we use previous result of
`tokio::net::lookup_host` immediately and spawn revalidation task in the
background. This way all lookups after the first successful one return
immediately.

Most of the time results returned by resolvers are the same anyway, but
with this cache we avoid waiting 60 second timeout if DNS request is
lost. Common reason result may be different is round-robin DNS load
balancing and switching from IPv4 to IPv6 network. For round-robin DNS
we don't break load balancing but simply use a different result, and for
IPv6 we anyway likely have a result in persistent cache and can use IPv4
otherwise.

Especially frequent should be the case when you send a message over SMTP
and SMTP connection is stale (older than 60 s), so we open a new one.
With this change new connection will be set up faster as you don't need
to wait for DNS resolution, so message will be sent faster.
2024-10-21 10:46:11 +00:00
Septias
c92554dc1f fix typo 2024-10-21 11:29:55 +02:00
link2xt
94c6d1dea4 fix: call update_connection_history for proxified connections 2024-10-20 18:36:37 +00:00
Hocuri
d27d0ef476 chore: Silence a rust-analyzer false-positive (#6077)
rust-analyzer was showing warnings here because it is always also
building in the Test configuration, and EventType has a

```rust
#[cfg(test)]
Test,
```
variant, which was not matched.
2024-10-20 20:21:32 +02:00
Hocuri
d3f75360fa fix: Resolve warning about default-features, and make it possible to disable vendoring (#6079)
On main, when running `cargo build`, the following warning is emitted:

> warning:
/home/jonathan/deltachat-android/jni/deltachat-core-rust/deltachat-ffi/Cargo.toml:
`default-features` is ignored for deltachat, since `default-features`
was not specified for `workspace.dependencies.deltachat`, this could
become a hard error in the future

This is because when referring to a workspace dependency, it's not
possible to remove features, it's only possible to add features, so that
the `vendored` feature was always enabled with no possibility to disable
it.

This PR restores the wanted behavior of enabling vendoring by default
with the possibility to disable it with "default-features = false".

It fixes `nix build .#python-docs` by not passing
`--no-default-features` when building deltachat with nix.
2024-10-20 18:33:47 +02:00
link2xt
06a6cc48d2 feat(sql): set PRAGMA query_only to avoid writing on read-only connections
Co-authored-by: iequidoo <dgreshilov@gmail.com>
2024-10-20 14:51:46 +00:00
iequidoo
b13f2709be test: Message from old setup preserves contact verification, but breaks 1:1 protection
If a message from an old contact's setup is received, the outdated Autocrypt header isn't applied,
so the contact verification preserves. But the chat protection breaks because the old message is
sorted to the bottom as it mustn't be sorted over the protection info message (which is `InNoticed`
moreover). Would be nice to preserve the chat protection too e.g. add a "protection broken" message,
then the old message and then a new "protection enabled" message, but let's record the current
behaviour first.
2024-10-20 10:05:28 -03:00
Sebastian Klähn
1b824705fd feat: Add realtime advertisement received event (#6043)
Co-authored-by: link2xt <link2xt@testrun.org>
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
2024-10-20 06:27:57 +00:00
link2xt
6f22ce2722 build: nix flake update 2024-10-20 06:01:30 +00:00
link2xt
5e58bf7575 feat: add more context to send_msg errors 2024-10-19 20:52:30 +00:00
link2xt
85d7c1f942 ci: update Rust to 1.82.0 2024-10-19 20:31:30 +00:00
iequidoo
df4fd82140 fix: ChatId::maybe_delete_draft: Don't delete message if it's not a draft anymore (#6053)
Follow-up to 07fa9c35ee.
2024-10-19 11:48:45 -03:00
link2xt
65b970a191 test: fix test_securejoin_after_contact_resetup flakiness 2024-10-18 15:36:21 +00:00
link2xt
5e13b4c736 feat: log when late Autocrypt header is ignored 2024-10-18 15:36:21 +00:00
link2xt
864833d232 fix: increase MAX_SECONDS_TO_LEND_FROM_FUTURE to 30
5 seconds is easy to exhaust
by running securejoin, especially
when it happens in automatic tests.

This may however easily affect bots as well.
2024-10-17 22:22:14 +00:00
link2xt
3d07db6e62 feat: log the logic for (not) doing AEAP 2024-10-17 22:22:14 +00:00
link2xt
9e88764a8a test(test_aeap_flow_verified): do not start ac1new
ac1new is an account that is only used
to get a new address for ac1.
It should not even be started
and run IMAP loop.
2024-10-17 22:21:54 +00:00
link2xt
e70b879182 test: make test_verified_group_member_added_recovery more reliable
To avoid reordering, wait for "member removed" message
to be received before sending "member added".
The test failed at least once
because email server may reorder the messages internally
while delivering.
2024-10-17 15:11:10 +00:00
link2xt
00d296e1ff test(test_aeap_flow_verified): wait for "member added" before sending messages (#6057)
Otherwise instead of "old address"
ac2 may receive "member added",
resulting in this failure:
```
>       assert msg_in_1.text == msg_out.text
E       AssertionError: assert 'Member Me (c...hat.computer.' == 'old address'
E         - old address
E         + Member Me (ci-hfpxxe@***) added by ci-8e7mkr@***.
```
2024-10-17 15:10:55 +00:00
link2xt
e07f1a8b9c docs: fix too_long_first_doc_paragraph clippy lint
This lint is enabled for beta and nightly Rust.
2024-10-17 14:29:03 +00:00
link2xt
02b9085147 feat: prioritize cached results if DNS resolver returns many results
This ensures we do not get stuck trying DNS resolver results
when we have a known to work IP address in the cache
and DNS resolver returns garbage
either because it is a captive portal
or if it maliciously wants to get us stuck
trying a long list of unresponsive IP addresses.

This also limits the number of results we try to 10 overall.
If there are more results, we will retry later
with new resolution results.
2024-10-17 11:55:14 +00:00
link2xt
07fa9c35ee fix: replace old draft with a new one atomically
This prevents creation of multiple drafts per chat.
2024-10-17 11:52:50 +00:00
link2xt
7db7c0aab1 refactor: use HeaderDef constant for Chat-Disposition-Notification-To 2024-10-17 07:10:54 +00:00
link2xt
30b23df816 docs: document MimeFactory.req_mdn 2024-10-17 07:10:54 +00:00
link2xt
4efd0d1ef7 test: always gossip if gossip_period is set to 0
This fixes flakiness of `test_verified_group_vs_delete_server_after`.
2024-10-15 22:55:33 +00:00
link2xt
f14880146a feat(deltachat-repl): built-in QR code printer
Print QR codes with Rust code
instead of depending on external `qrencode`.
2024-10-15 22:55:20 +00:00
link2xt
3a72188548 test(test_qr_setup_contact_svg): stop testing for no display name
It is impossible to set no display name anyway
in Delta Chat Android at least
because we don't want email addresses
in the UI.

This test does not work with long domains
that may get wrapped, so better remove it
instead of trying to prevent wrapping of domains.
2024-10-15 17:35:38 +00:00
link2xt
351f28361d docs: set_protection_for_timestamp_sort does not send messages
It only adds info messages.
2024-10-15 09:14:23 +00:00
link2xt
c5b78741d6 refactor: fix clippy::needless_lifetimes warnings 2024-10-15 09:14:23 +00:00
link2xt
57871bbaf8 refactor(set_protection_for_timestamp_sort): do not log bubbled up errors
Otherwise error may be logged twice.
2024-10-15 09:14:23 +00:00
link2xt
287256693c refactor: fix elided_named_lifetimes warning in beta Rust 2024-10-15 09:14:23 +00:00
iequidoo
d660f55a99 feat: Sort received outgoing message down if it's fresher than all non fresh messages
Received messages shouldn't mingle with just sent ones and appear somewhere in the middle of the
chat, so we go after the newest non fresh message.

But if a received outgoing message is older than some `InSeen` message, better sort the received
message purely by timestamp (this is an heuristic in order not to break the Gmail-like case
simulated by `verified_chats::test_old_message_4()`). We could place the received message just
before that `InSeen` message, but anyway the user may not notice it.

At least this fixes outgoing messages sorting for shared accounts where messages from other devices
should be sorted the same way as incoming ones.
2024-10-14 21:22:11 -03:00
link2xt
f1ca689f99 feat: IMAP COMPRESS support 2024-10-14 14:01:22 +00:00
iequidoo
796b0d7752 refactor: update_msg_state: Don't avoid downgrading OutMdnRcvd to OutDelivered
`OutMdnRcvd` is a "virtual" message state now, only old messages can have this state in the db, so
`update_msg_state()` can be simplified.
2024-10-14 10:24:01 -03:00
link2xt
2ea5c86a5a chore(release): prepare for 1.147.1 2024-10-13 18:40:33 +00:00
iequidoo
50b250cf78 docs(CONTRIBUTING.md): Add a note on deleting/changing db columns 2024-10-13 15:34:15 -03:00
iequidoo
3c03370589 fix: Readd tokens.foreign_id column (#6038)
Otherwise backups exported from the current core and imported in versions < 1.144.0 have QR codes
not working. The breaking change which removed the column is
5a6efdff44.
2024-10-13 15:34:15 -03:00
iequidoo
8f41aed917 fix: Assume file extensions are 32 chars max and don't contain whitespace (#5338)
Before file extensions were also limited to 32 chars, but extra chars in the beginning were just cut
off, e.g. "file.with_lots_of_characters_behind_point_and_double_ending.tar.gz" was considered to
have an extension "d_point_and_double_ending.tar.gz". Better to take only "tar.gz" then.

Also don't include whitespace-containing parts in extensions. File extensions generally don't
contain whitespaces.
2024-10-11 11:04:41 -03:00
link2xt
19be12a25d chore(cargo): upgrade async_zip to 0.0.17 (#6035) 2024-10-11 00:17:40 +00:00
link2xt
6a121b87eb fix: do not emit progress 1000 when configuration is cancelled
There is already code below that emits
progress 0 or 1000 depending on whether
configuration succeeded or failed.

Before this change cancelling resulted
in progress 0 emitted,
immediately followed by progress 1000.
2024-10-10 10:34:58 +00:00
link2xt
420c0ed9b0 build(deltachat-rpc-client): add classifiers for all supported Python versions 2024-10-10 07:18:06 +00:00
link2xt
e05bb03db6 build: build Python 3.13 wheels 2024-10-10 07:18:06 +00:00
link2xt
73fcb97eef ci: update to Python 3.13 2024-10-10 07:18:06 +00:00
iequidoo
8acf391ffe refactor: MsgId::update_download_state: Don't fail if the message doesn't exist anymore
If a race happens and the message disappears, there's just nothing to do and no sense to
fail. Follow-up to 22e5bf8571.
2024-10-08 12:31:41 -03:00
iequidoo
aacea2de25 fix: Reset quota on configured address change (#5908) 2024-10-07 18:04:53 -03:00
iequidoo
b713e8cd94 chore(cargo): bump futures-* from 0.3.30 to 0.3.31
futures-util 0.3.30 is yanked.
2024-10-07 15:33:09 -03:00
link2xt
b7be0b7bf6 chore(release): prepare for 1.147.0 2024-10-05 18:04:17 +00:00
link2xt
2cb8b53256 fix: emit progress 0 if get_backup() fails 2024-10-05 17:58:24 +00:00
link2xt
a592a470cf fix: make backup reception cancellable by stopping ongoing process
This is already documented in JSON-RPC API,
but in fact ongoing process was not allocated.
2024-10-05 17:58:24 +00:00
link2xt
c4d07ab99e fix: smooth progress bar for backup transfer
Before this change progress bar only started
when database is already transferred.
Database is usually the largest file
in the whole transfer, so the transfer appears
to be stuck for the sender.

With this change progress bar
starts for backup export
as soon as connection is received
and counts bytes transferred over the connection
using AsyncWrite wrapper.

Similarly for backup import,
AsyncRead wrapper counts the bytes
received and emits progress events.
2024-10-05 17:58:24 +00:00
link2xt
eddd5a0d25 fix: make it possible to cancel ongoing backup transfer 2024-10-05 17:58:24 +00:00
link2xt
0f43d5d8f4 fix: break out of accept() loop if there is an error transferring backup 2024-10-05 17:58:24 +00:00
link2xt
2e6d3aebae docs(CONTRIBUTING.md): add more SQL advices 2024-10-05 13:09:49 +00:00
link2xt
650995dc41 feat(deltachat-repl): print send-backup QR code to the terminal 2024-10-04 22:53:30 +00:00
link2xt
283a1f1653 fix: skip unconfigured folders in background_fetch()
Otherwise `background_fetch()` fails on unconfigured Mvbox,
which is typical for chatmail accounts,
and does not get to checking QUOTA ever.
2024-10-04 21:54:42 +00:00
link2xt
d33909a054 feat: reuse existing connections in background_fetch() if I/O is started 2024-10-04 21:54:42 +00:00
link2xt
129be3aa27 feat(deltachat-repl): add fetch command to test background_fetch() 2024-10-04 15:52:59 +00:00
link2xt
8a88479d8f fix(query_row_optional): do not treat rows with NULL as missing rows
Instead of treating NULL type error
as absence of the row,
handle NULL values with SQL.
Previously we sometimes
accidentally treated a single column
being NULL as the lack of the whole row.
2024-10-04 14:43:06 +00:00
Hocuri
5711f2fe3a feat: More context for the "Cannot establish guaranteed..." info message (#6022)
The "Cannot establish guaranteed end-to-end encryption with ..." info
message can have lots of causes, and it happened twice to us now that it
took us some time to figure out which one it is.

So, include some more detail in the info message by simply adding the
non-translated error message in parantheses.

If we want to put in some more effort for nicer error messages, we
could:
- Introduce one new translated string "Cannot establish guaranteed
end-to-end encryption with …. Cause: %2$s" or similar (and remove the
old stock string)
- And/Or: Introduce new translated strings for all the possible errors
- And/Or: Maybe reword it in order to account better for the case that
the chat already is marked as g-e2ee, or use a different wording
(because if the chat is marked as g-e2ee then it might be nice to notify
the user that something may have gone wrong, but it's still working,
just that maybe the other side doesn't have us verified now)


![Screenshot_20241003-222245](https://github.com/user-attachments/assets/c064c82e-01ac-4bac-ab11-3c9ac9db5298)
2024-10-04 13:51:06 +02:00
link2xt
46922d4d9d fix: do not attempt to reference info messages
Info messages are added
at the beginning of unpromoted group chats
("Others will only see this group after you sent a first message."),
may be created by WebXDC etc.

They are not sent outside
and have local Message-ID that
is not known to other recipients
so they should be skipped when constructing
In-Reply-To and References.
2024-10-03 21:49:58 +00:00
link2xt
75fe4e106a api!: remove deprecated get_next_media() APIs 2024-10-03 21:04:03 +00:00
iequidoo
7c60ac863e feat: MsgId::get_info(): Report original filename as well 2024-10-03 15:49:03 -03:00
link2xt
fa9bd7f144 chore(release): prepare for 1.146.0 2024-10-03 17:21:42 +00:00
link2xt
22e5bf8571 fix(download_msg): do not fail if the message does not exist anymore
Without this fix IMAP loop may get stuck
trying to download non-existing message over and over
like this:
```
src/imap.rs:372: Logging into IMAP server with LOGIN.
src/imap.rs:388: Successfully logged into IMAP server
src/scheduler.rs:361: Failed to download message Msg#3467: Message Msg#3467 does not exist.
src/scheduler.rs:418: Failed fetch_idle: Failed to download messages: Message Msg#3467 does not exist
```

The whole download operation fails
due to attempt to set the state of non-existing message
to "failed". Now download of the message
will "succeed" if the message does not exist
and we don't try to set its state.
2024-10-03 17:13:53 +00:00
link2xt
c8ba516e83 refactor(decode_ideltachat): construct error message lazily 2024-10-03 15:39:27 +00:00
dependabot[bot]
4b021f509c chore(cargo): bump syn from 2.0.77 to 2.0.79
Bumps [syn](https://github.com/dtolnay/syn) from 2.0.77 to 2.0.79.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/2.0.77...2.0.79)

---
updated-dependencies:
- dependency-name: syn
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-02 17:06:38 -03:00
dependabot[bot]
bd1e06cfa7 Merge pull request #6003 from deltachat/dependabot/cargo/serde-1.0.210 2024-10-02 20:05:43 +00:00
dependabot[bot]
11e5a00366 chore(cargo): bump quick-xml from 0.36.1 to 0.36.2
Bumps [quick-xml](https://github.com/tafia/quick-xml) from 0.36.1 to 0.36.2.
- [Release notes](https://github.com/tafia/quick-xml/releases)
- [Changelog](https://github.com/tafia/quick-xml/blob/master/Changelog.md)
- [Commits](https://github.com/tafia/quick-xml/compare/v0.36.1...v0.36.2)

---
updated-dependencies:
- dependency-name: quick-xml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-02 17:04:25 -03:00
dependabot[bot]
5fdecdcc16 chore(cargo): bump serde from 1.0.209 to 1.0.210
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.209 to 1.0.210.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.209...v1.0.210)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-02 02:32:20 +00:00
dependabot[bot]
77b899813c Merge pull request #6000 from deltachat/dependabot/cargo/rustls-pki-types-1.9.0 2024-10-02 02:30:36 +00:00
dependabot[bot]
7843e0ed29 Merge pull request #6001 from deltachat/dependabot/cargo/hyper-util-0.1.9 2024-10-02 02:29:26 +00:00
dependabot[bot]
a036c86857 Merge pull request #6002 from deltachat/dependabot/cargo/pretty_assertions-1.4.1 2024-10-02 02:28:57 +00:00
dependabot[bot]
e535a6f859 Merge pull request #6012 from deltachat/dependabot/cargo/tempfile-3.13.0 2024-10-02 02:28:00 +00:00
dependabot[bot]
5384d5f75d Merge pull request #6008 from deltachat/dependabot/cargo/libc-0.2.159 2024-10-02 02:27:38 +00:00
dependabot[bot]
c569696fff Merge pull request #6007 from deltachat/dependabot/cargo/bytes-1.7.2 2024-10-02 02:27:00 +00:00
dependabot[bot]
a6732f5a5c Merge pull request #6011 from deltachat/dependabot/cargo/thiserror-1.0.64 2024-10-02 02:26:25 +00:00
dependabot[bot]
9978f89b1b Merge pull request #6005 from deltachat/dependabot/cargo/tokio-stream-0.1.16 2024-10-02 02:26:02 +00:00
dependabot[bot]
dbca15e5ef Merge pull request #6010 from deltachat/dependabot/cargo/anyhow-1.0.89 2024-10-02 02:25:19 +00:00
dependabot[bot]
91649effa6 chore(cargo): bump tempfile from 3.10.1 to 3.13.0
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.10.1 to 3.13.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.10.1...v3.13.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 21:46:25 +00:00
dependabot[bot]
672ff58e3c chore(cargo): bump thiserror from 1.0.63 to 1.0.64
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.63 to 1.0.64.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.63...1.0.64)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 21:46:12 +00:00
dependabot[bot]
a85b7ceb9c chore(cargo): bump anyhow from 1.0.86 to 1.0.89
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.86 to 1.0.89.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.86...1.0.89)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 21:45:56 +00:00
dependabot[bot]
943ec19de4 chore(cargo): bump libc from 0.2.158 to 0.2.159
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.158 to 0.2.159.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Changelog](https://github.com/rust-lang/libc/blob/0.2.159/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.158...0.2.159)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 21:45:11 +00:00
dependabot[bot]
733da91c5c chore(cargo): bump bytes from 1.7.1 to 1.7.2
Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.7.1 to 1.7.2.
- [Release notes](https://github.com/tokio-rs/bytes/releases)
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.7.1...v1.7.2)

---
updated-dependencies:
- dependency-name: bytes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 21:44:50 +00:00
dependabot[bot]
d899cc730a chore(cargo): bump tokio-stream from 0.1.15 to 0.1.16
Bumps [tokio-stream](https://github.com/tokio-rs/tokio) from 0.1.15 to 0.1.16.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-stream-0.1.15...tokio-stream-0.1.16)

---
updated-dependencies:
- dependency-name: tokio-stream
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 21:43:59 +00:00
dependabot[bot]
5872b64265 chore(cargo): bump pretty_assertions from 1.4.0 to 1.4.1
Bumps [pretty_assertions](https://github.com/rust-pretty-assertions/rust-pretty-assertions) from 1.4.0 to 1.4.1.
- [Release notes](https://github.com/rust-pretty-assertions/rust-pretty-assertions/releases)
- [Changelog](https://github.com/rust-pretty-assertions/rust-pretty-assertions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-pretty-assertions/rust-pretty-assertions/compare/v1.4.0...v1.4.1)

---
updated-dependencies:
- dependency-name: pretty_assertions
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 21:42:57 +00:00
dependabot[bot]
5d8035f741 chore(cargo): bump hyper-util from 0.1.7 to 0.1.9
Bumps [hyper-util](https://github.com/hyperium/hyper-util) from 0.1.7 to 0.1.9.
- [Release notes](https://github.com/hyperium/hyper-util/releases)
- [Changelog](https://github.com/hyperium/hyper-util/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper-util/compare/v0.1.7...v0.1.9)

---
updated-dependencies:
- dependency-name: hyper-util
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 21:42:32 +00:00
dependabot[bot]
3d183336f5 chore(cargo): bump rustls-pki-types from 1.8.0 to 1.9.0
Bumps [rustls-pki-types](https://github.com/rustls/pki-types) from 1.8.0 to 1.9.0.
- [Release notes](https://github.com/rustls/pki-types/releases)
- [Commits](https://github.com/rustls/pki-types/compare/v/1.8.0...v/1.9.0)

---
updated-dependencies:
- dependency-name: rustls-pki-types
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 21:42:18 +00:00
WofWca
9c931c22cc refactor: better log message for failed QR scan
It did not interpolate the `{prefix}`,
it just printed it in plain text.
2024-09-30 17:34:54 -03:00
link2xt
78a0d7501b feat: use Rustls instead of native TLS for HTTPS requests
HTTPS requests are used to fetch
remote images in HTML emails,
to fetch autoconfig XML,
to POST requests for `DCACCOUNT:` QR codes
to make OAuth 2 API requests
and to connect to HTTPS proxies.

Rustls is more aggressive than OpenSSL
in deprecating cryptographic algorithms
so we cannot use it for IMAP and SMTP
to avoid breaking compatibility,
but for HTTPS requests listed
above this should not result in problems.

As HTTPS requests use only strict TLS checks,
there is no `strict_tls` argument
in `wrap_rustls` function.

Rustls is already used by iroh,
so this change does not introduce new dependencies.
2024-09-26 22:35:44 +00:00
link2xt
638da904e7 refactor: merge build_tls() function into wrap_tls() 2024-09-26 22:35:44 +00:00
iequidoo
fe0c9958a6 feat: Assign message to ad-hoc group with matching name and members (#5385)
This should fix ad-hoc groups splitting when messages are fetched out of order from different
folders or otherwise reordered, or some messages are missing so that the messages reference chain is
broken, or a member was removed from the thread and readded later, etc. Even if this way two
different threads are merged, it looks acceptable, having many threads with the same name/subject
and members isn't a common use case.
2024-09-26 17:09:11 -03:00
iequidoo
c469fcb435 refactor: Move group name calculation out of create_adhoc_group() 2024-09-26 17:09:11 -03:00
link2xt
02db6bcb8e chore(release): prepare for 1.145.0 2024-09-26 19:22:10 +00:00
link2xt
4b74c9d85f fix: avoid changing delete_server_after default for existing configurations 2024-09-26 19:18:12 +00:00
link2xt
040ac0ffe3 refactor: do not wrap shadowsocks::ProxyClientStream
Updated `shadowsocks` implements `Debug` for the type,
so there is no need to wrap it.
2024-09-26 14:28:08 +00:00
link2xt
bfef129dbf chore: sort dependency list 2024-09-22 18:00:55 +00:00
link2xt
486ea3a358 chore(release): prepare for 1.144.0 2024-09-21 18:53:02 +00:00
link2xt
624ae86913 api!: make QR code type for proxy not specific to SOCKS5 (#5980) 2024-09-21 18:26:07 +00:00
link2xt
b47b96d5d6 chore(cargo): update iroh to 0.25
According to
<https://www.iroh.computer/blog/iroh-0-25-0-custom-protocols-for-all>
gossip now handles updating direct addresses automatically.
2024-09-20 22:56:24 +00:00
link2xt
f6b5c5d150 feat: generate 144-bit group IDs
Instead of generating 72 random bits
and reducing them to 66 bits of Base64 characters,
generate 144 bits (18 bytes)
which is exactly 24 Base64 characters.

This should still be accepted by existing
Delta Chat clients which expect group ID
to be between 11 and 32 characters.

Message-ID creation is also simplified
to not have `Mr.` prefix
and dot in between two IDs.
Now it is a single ID followed by `@localhost`.

Some outdated documentation comments
are removed, e.g. group messages
don't start with `Gr.` already.
2024-09-20 22:38:28 +00:00
link2xt
9cc65c615c feat(smtp): more verbose SMTP connection establishment errors
The greeting is now always read manually,
even for STARTTLS connections,
so the errors returned on failure to read form the stream
are the same regardless of the connection type.
2024-09-20 20:37:47 +00:00
iequidoo
d6845bd5e9 feat: Use IMAP APPEND command to upload sync messages (#5845)
Why:
- With IMAP APPEND we can upload messages directly to the DeltaChat folder (for non-chatmail
  accounts).
- We can set the `\Seen` flag immediately so that if the user has other MUA, it doesn't alert about
  a new message if it's just a sync message (there were several such reports on the support
  forum). Though this also isn't useful for chatmail.
- We don't need SMTP envelope and overall remove some overhead on processing sync messages.
2024-09-20 17:07:45 -03:00
iequidoo
0b908db272 chore(deps): bump async-imap from 0.10.0 to 0.10.1 2024-09-20 17:07:45 -03:00
iequidoo
841ed43f11 feat: Don't put displayname into From/To/Sender if it equals to address (#5983)
If a displayname equals to the address, adding it looks excessive.
Moreover, it's not useful for Delta Chat receiving the message because
`sanitize_name_and_addr()` removes such a displayname anyway. Also now
at least DC Android requires specifying profile name, so there should be
a fallback for users having meaningful addresses to keep the old
behaviour when Core generates `From` w/o the profile name, and this
question has already appeared on the forum.
2024-09-20 15:59:33 -03:00
link2xt
60cd6f56be chore(cargo): update lazy_static to 1.5.0
This removes duplicate `spin` dependency.
2024-09-18 15:31:13 +00:00
link2xt
060fd55249 feat: HTTP(S) tunneling
HTTP proxy is tested with deltachat-repl
against local Privoxy
using
```
> set proxy_url http://127.0.0.1:8118/
> setqr dcaccount:https://nine.testrun.org/new
> configure
> connect
```
2024-09-18 10:52:31 +00:00
link2xt
38c7f7300e Partially revert "test(test-data): remove public keys that can be derived from secret keys" (#5977)
This reverts commit 1caf672904.

Otherwise public key signature is regenerated each time the key is
loaded and test `key::tests::test_load_self_existing` which loads the
key twice fails when two loads happen on different seconds.

Closes #5976
2024-09-18 09:48:01 +00:00
link2xt
f7a705c6da refactor: use KeyPair::new() in create_keypair() 2024-09-16 20:51:16 +00:00
iequidoo
f497e4dd12 docs: Why search_msgs() only looks at the first kilobytes of long messages 2024-09-16 17:14:51 -03:00
iequidoo
0a63083df7 fix: Shorten message text in locally sent messages too (#2281) 2024-09-16 17:14:51 -03:00
iequidoo
5a6efdff44 fix: Save QR code token regardless of whether the group exists (#5954)
Groups promotion to other devices and QR code tokens synchronisation are not synchronised processes,
so there are reasons why a QR code token may arrive earlier than the first group message:
- We are going to upload sync messages via IMAP while group messages are sent by SMTP.
- If sync messages go to the mvbox, they can be fetched earlier than group messages from Inbox.
2024-09-16 16:40:26 -03:00
link2xt
7efb5a269c docs(CONTRIBUTING.md): document how to format SQL statements 2024-09-16 18:11:42 +00:00
link2xt
1caf672904 test(test-data): remove public keys that can be derived from secret keys 2024-09-16 17:00:16 +00:00
link2xt
7743072411 refactor: remove addr from KeyPair 2024-09-16 17:00:16 +00:00
link2xt
c461c4f02e refactor: do not store deprecated addr and is_default into keypairs 2024-09-16 17:00:16 +00:00
iequidoo
5b597f3a95 feat: Don't SMTP-send messages to self-chat if BccSelf is disabled
`chat::create_send_msg_jobs()` already handles `Config::BccSelf` as needed. The only exception is
Autocrypt setup messages. This change unifies the logic for the self-chat and groups only containing
`SELF`.
2024-09-15 23:48:06 -03:00
iequidoo
b69488685f feat: Make resending OutPending messages possible (#5817)
This makes possible to schedule one more sending of the message, the existing jobs are not
cancelled. Otherwise it's complicated to implement bots that resend messages when a new member joins
the group.
2024-09-15 16:27:39 -03:00
link2xt
afb01e3e90 chore: update provider database
This change removes OAuth2 for Gmail
as Delta Chat does not have a working
client ID anymore.
Tests are adjusted to test against Yandex
and MX queries for OAuth2 are always disabled
because they were only used to detect Google Workspace.
2024-09-13 17:58:25 +00:00
link2xt
7ff14dc26b feat: log unexpected message state when resending fails 2024-09-12 05:06:05 +00:00
link2xt
0c33064193 chore(release): prepare for 1.143.0 2024-09-12 01:52:14 +00:00
link2xt
61d77584e8 chore(cargo): update typescript-type-def to 0.5.12
This removes unmaintained proc-macro-error dependency.
2024-09-12 01:35:43 +00:00
link2xt
37ca9d7319 feat: shadowsocks support
This change introduces new config options
`proxy_enabled` and `proxy_url`
that replace `socks5_*`.

Tested with deltachat-repl
by starting it with
`cargo run --locked -p deltachat-repl -- deltachat-db` and running
```
> set proxy_enabled 1
> set proxy_url ss://...
> setqr dcaccount:https://chatmail.example.org/new
> configure
```
2024-09-12 00:22:09 +00:00
iequidoo
2c136f6355 refactor: get_config_bool_opt(): Return None if only default value exists
And also:
- Make it `pub(crate)`.
- Use it in `should_request_mdns()` as using `config_exists()` there isn't correct because the
  latter doesn't look at environment.
2024-09-10 18:10:59 -03:00
iequidoo
52dcc7e350 refactor: Make Context::config_exists() crate-public 2024-09-10 18:10:59 -03:00
iequidoo
ff6488371c feat: Delete messages from a chatmail server immediately by default (#5805) (#5840)
I.e. treat `DeleteServerAfter == None` as "delete at once". But when a backup is exported, set
`DeleteServerAfter` to 0 so that the server decides when to delete messages, in order not to break
the multi-device case. Even if a backup is not aimed for deploying more devices, `DeleteServerAfter`
must be set to 0, otherwise the backup is half-useful because after a restoration the user wouldn't
see new messages deleted by the device after the backup was done. But if the user explicitly set
`DeleteServerAfter`, don't change it when exporting a backup. Anyway even for non-chatmail case the
app should warn the user before a backup export if they have `DeleteServerAfter` enabled.

Also do the same after a backup import. While this isn't reliable as we can crash in between, this
is a problem only for old backups, new backups already have `DeleteServerAfter` set if necessary.

---------

Co-authored-by: Hocuri <hocuri@gmx.de>
2024-09-08 16:53:56 -03:00
link2xt
0782b5abdd ci: update Rust to 1.81.0 2024-09-08 07:08:12 +00:00
link2xt
2e2ba96d75 chore(cargo-deny): silence unmaintained proc-macro-error warning 2024-09-08 06:35:01 +00:00
iequidoo
853e38e054 feat: ChatId::create_for_contact_with_blocked: Don't emit events on no op 2024-09-05 10:46:30 -03:00
iequidoo
418dfbf994 fix: Don't sync QR code token before populating the group (#5935)
Otherwise other devices don't yet know about the group and can't handle the sync message correctly.
2024-09-04 15:18:26 -03:00
dependabot[bot]
533a872118 Merge pull request #5950 from deltachat/dependabot/cargo/quinn-proto-0.11.8 2024-09-04 00:04:55 +00:00
dependabot[bot]
2ae854e8ea chore(cargo): bump quinn-proto from 0.11.3 to 0.11.8
Bumps [quinn-proto](https://github.com/quinn-rs/quinn) from 0.11.3 to 0.11.8.
- [Release notes](https://github.com/quinn-rs/quinn/releases)
- [Commits](https://github.com/quinn-rs/quinn/compare/quinn-proto-0.11.3...quinn-proto-0.11.8)

---
updated-dependencies:
- dependency-name: quinn-proto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-03 20:50:28 +00:00
link2xt
3969383857 Merge tag 'v1.142.12' 2024-09-02 23:58:26 +00:00
link2xt
e4ebb91712 chore(release): prepare for 1.142.12 2024-09-02 23:57:32 +00:00
iequidoo
eb3c1b3c25 fix: Display Config::MdnsEnabled as true by default (#5948) 2024-09-02 23:51:51 +00:00
iequidoo
c257482838 fix: Display Config::MdnsEnabled as true by default (#5948) 2024-09-02 20:50:15 -03:00
link2xt
0a46e64971 fix: use default server list for providers that don't have one
There are providers in the provider database
that do not have servers specified.
For such providers default list should be tried
just like when configuring unknown providers.
2024-09-02 22:57:31 +00:00
iequidoo
845420cf17 test: Alice is (non-)bot on Bob's side after QR contact setup 2024-09-02 18:06:52 -03:00
dependabot[bot]
96ea0db88e Merge pull request #5945 from deltachat/dependabot/cargo/libc-0.2.158 2024-09-01 23:21:12 +00:00
dependabot[bot]
d99c735e12 Merge pull request #5944 from deltachat/dependabot/cargo/async-imap-0.10.0 2024-09-01 22:40:49 +00:00
dependabot[bot]
d48f4100e9 Merge pull request #5943 from deltachat/dependabot/cargo/bytes-1.7.1 2024-09-01 22:39:57 +00:00
dependabot[bot]
7e73d5fdac chore(cargo): bump serde_json from 1.0.122 to 1.0.127
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.122 to 1.0.127.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.122...1.0.127)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 19:27:16 -03:00
dependabot[bot]
152cdfe9bc chore(cargo): bump syn from 2.0.72 to 2.0.77
Bumps [syn](https://github.com/dtolnay/syn) from 2.0.72 to 2.0.77.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](https://github.com/dtolnay/syn/compare/2.0.72...2.0.77)

---
updated-dependencies:
- dependency-name: syn
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 19:16:07 -03:00
dependabot[bot]
a9eedafbcb chore(cargo): bump serde from 1.0.205 to 1.0.209
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.205 to 1.0.209.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.205...v1.0.209)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 19:10:15 -03:00
dependabot[bot]
5baf191483 chore(cargo): bump quote from 1.0.36 to 1.0.37
Bumps [quote](https://github.com/dtolnay/quote) from 1.0.36 to 1.0.37.
- [Release notes](https://github.com/dtolnay/quote/releases)
- [Commits](https://github.com/dtolnay/quote/compare/1.0.36...1.0.37)

---
updated-dependencies:
- dependency-name: quote
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 19:02:34 -03:00
dependabot[bot]
2d2e703884 chore(cargo): bump libc from 0.2.155 to 0.2.158
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.155 to 0.2.158.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Changelog](https://github.com/rust-lang/libc/blob/0.2.158/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.155...0.2.158)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 21:09:15 +00:00
dependabot[bot]
026450ddf3 chore(cargo): bump async-imap from 0.9.7 to 0.10.0
Bumps [async-imap](https://github.com/async-email/async-imap) from 0.9.7 to 0.10.0.
- [Changelog](https://github.com/async-email/async-imap/blob/main/CHANGELOG.md)
- [Commits](https://github.com/async-email/async-imap/compare/v0.9.7...v0.10.0)

---
updated-dependencies:
- dependency-name: async-imap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 21:09:01 +00:00
dependabot[bot]
5646782d23 chore(cargo): bump bytes from 1.5.0 to 1.7.1
Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.5.0 to 1.7.1.
- [Release notes](https://github.com/tokio-rs/bytes/releases)
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.5.0...v1.7.1)

---
updated-dependencies:
- dependency-name: bytes
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-01 21:08:44 +00:00
link2xt
dd1c2e836b feat(securejoin): ignore invalid *-request-with-auth messages silently 2024-09-01 16:10:41 +00:00
link2xt
be73076e9e chore(cargo): replace unmaintained ansi_term with nu-ansi-term 2024-09-01 16:10:30 +00:00
link2xt
9d47be0d8a Merge tag 'v1.142.11' 2024-08-30 22:38:15 +00:00
link2xt
fcf3dbbad4 chore(release): prepare for 1.142.11 2024-08-30 22:37:12 +00:00
link2xt
d344cc3bdd fix: set backward verification when observing vc-contact-confirm or vg-member-added (#5930)
Documentation comment says forward and backward verification is set,
but the code was not doing it.
`vc-contact-confirm` and `vg-member-added` messages
indicate that other device finished securejoin protocol
so we know Bob has our key marked as verified.
2024-08-30 19:51:26 +00:00
link2xt
93e181b2da docs: document that bcc_self is enabled by default
bcc_self has been enabled by default
since core version 1.95.0
by merging
PR <https://github.com/deltachat/deltachat-core-rust/pull/3612>.

However deltachat.h documentation
still incorrectly said that bcc_self is disabled by default.
2024-08-30 19:21:01 +00:00
link2xt
3867808927 chore(cargo): reduce number of duplicate dependencies 2024-08-30 04:24:31 +00:00
link2xt
c7c3b9ca90 feat: replace reqwest with hyper
This change replaces
usage of `reqwest` and `hyper-util`
with custom connection establishment code
so it is done in the same way
as for IMAP and SMTP connections.
This way we control HTTP, IMAP and SMTP
connection establishment
and schedule connection attempts
to resolved IP addresses
in the same way for all 3 protocols.
2024-08-29 23:10:17 +00:00
link2xt
54cfc21e28 Remove old iroh 0.4 2024-08-29 20:59:41 +00:00
link2xt
f01514dba4 fix: start new connections independently of connection failures
With current implementation
every time connection fails
we take the next delay from `delays` iterator.
In the worst case first 4 DNS results
immediately refuse connection
and we start fifth connection attempt
with 1 year timeout,
effectively continuing all remaining
connection attempts without concurrency.

With new implementation
new connection attempts are
added to `connection_attempt_set`
independently of connection failures
and after 10 seconds
we always end up with five
parallel connection attempts
as long as there are enough IP addresses.
2024-08-29 13:24:28 +00:00
link2xt
ee5723416e chore(cargo): update iroh to 0.23.0 2024-08-28 23:15:38 +00:00
link2xt
aab8ef2726 feat: parallelize IMAP and SMTP connection attempts (#5915)
Previously for each connection candidate (essentially host and port
pair) after resolving the host to a list of IPs Delta Chat iterated IP
addresses one by one. Now for IMAP and SMTP we try up to 5 IP addresses
in parallel. We start with one connection and add more connections
later. If some connection fails, e.g. we try to connect to IPv6 on IPv4
network and get "Network is unreachable" (ENETUNREACH) error, we replace
failed connection with another one immediately.

Co-authored-by: Hocuri <hocuri@gmx.de>
2024-08-28 22:00:07 +00:00
link2xt
84c1ffd7cc fix: do not allow quotes with "... wrote:" headers in chat messages 2024-08-28 16:05:03 +00:00
link2xt
273158a337 fix: add Auto-Submitted header in a single place
This ensures we don't add multiple Auto-Submitted headers
when bots send vg-request or vc-request messages.

The change fixes failing
receive_imf::tests::test_bot_accepts_another_group_after_qr_scan
test.
2024-08-27 18:31:07 +00:00
link2xt
099f0e2d18 Merge tag 'v1.142.10' 2024-08-26 18:54:27 +00:00
iequidoo
af77c0c987 feat: Add "Auto-Submitted: auto-replied" header to appropriate SecureJoin messages
I.e. to all messages except "v{c,g}-request" as they sent out on a QR code scanning which is a
manual action and "vg-member-added" as formally this message is auto-submitted, but the member
addition is a result of an explicit user action. Otherwise it would be strange to have the
Auto-Submitted header in "member-added" messages of verified groups only.
2024-08-25 16:19:41 -03:00
link2xt
f912bc78e6 fix(http): set I/O timeout to 1 minute rather than whole request timeout
Before the fix HTTP client
had no connection timeout,
so it only had a chance
to test one IPv6 and one IPv4
address if the first addresses timed out.
Now it can test at least 4 addresses
of each family and more if some addresses
refuse connection rather than time out.
2024-08-25 17:06:34 +00:00
link2xt
137ee9334c feat: always use preloaded DNS results
Otherwise if DNS server returns incorrect results,
we may never try preloaded DNS results.
For example, we may get our first results
from a captive portal.

To test, add `127.0.0.1 example.org`
and try to create an account.
Without this change we only try 127.0.0.1 and fail.
With this change preloaded DNS results are tried as well.
2024-08-25 15:33:18 +00:00
link2xt
36e5e964e5 Merge tag 'v1.142.9' 2024-08-24 21:43:43 +00:00
link2xt
ef12a76a9e chore: update provider database 2024-08-23 13:24:07 +00:00
bjoern
6b3de9d7da recognize t.me proxy qr codes (#5895)
this PR adds the type DC_QR_SOCKS5_PROXY to `dc_check_qr()` for
**supporting telegram proxy QR codes**. if returned, the UI should ask
the user if they want to us the proxy and call
`dc_set_config_from_qr();` afterwards (plus maybe `dc_configure()`).

idea is to improve our proxy story, follow ups may be:

- in UI, - move proxy out of "Account & Password", as a **separate
"Proxy Activity"** (it should stay in "Advanced" for now, however, below
"Server", which might be moved up)

- allow **opening the "Proxy Activity" from the welcome screens**
three-dot-menu (that would also solve a long standing issue that
entering the email address bypasses the proxy

- show proxy usage in the "Connectivity View" and/or add an **icon** to
the main chatlist screen (beside three-dot menu) in case some proxy is
in use; tapping this icon will open the "Proxy Activity"

- the the new "Proxy Activity", add a **share / show proxy QR code**
button. that would generate invite links in the form
`https://i.delta.chat/socks#...` - so that tapping then opens the app.
support for these links need to be added to core then.

- handle a list of proxies in core, offer selection in UI. the list
could be one for all profiles and could be filled eg. by normal invite
links or other channels

---------

Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
2024-08-23 09:49:49 +02:00
link2xt
3599e4be16 fix: save custom username if user entered it 2024-08-23 05:44:28 +00:00
link2xt
8dc844e194 refactor(login_param): use Config:: constants to avoid typos in key names 2024-08-22 00:44:06 +00:00
link2xt
104c60840a test: test that alternative port 443 works 2024-08-22 00:14:57 +00:00
link2xt
f2cb098148 fix: make SMTP server list readable in context info 2024-08-21 13:04:37 +00:00
link2xt
30b998eca3 Merge tag 'v1.142.8' 2024-08-21 12:48:17 +00:00
link2xt
b5133fe8c8 fix: fix loading imap_certificate_checks
Fix a typo in the config name
(by using `Config::` to avoid it)
and make sure we don't panic on unknown values.

Also test that we don't panic on unknown
`configured_imap_certificate_checks` values.
2024-08-21 12:46:12 +00:00
link2xt
08ec133aac refactor: use get_configured_provider() in ConfiguredLoginParam::load() 2024-08-21 07:24:15 +00:00
link2xt
7d7391887a fix: do not ignore legacy configured_{mail,send}_user for known providers 2024-08-21 07:24:15 +00:00
link2xt
e7d4ccffe2 feat: automatic reconfiguration 2024-08-19 16:36:56 +00:00
111 changed files with 7178 additions and 4427 deletions

View File

@@ -24,7 +24,7 @@ jobs:
name: Lint Rust
runs-on: ubuntu-latest
env:
RUSTUP_TOOLCHAIN: 1.80.1
RUSTUP_TOOLCHAIN: 1.82.0
steps:
- uses: actions/checkout@v4
with:
@@ -95,11 +95,11 @@ jobs:
matrix:
include:
- os: ubuntu-latest
rust: 1.80.1
rust: 1.82.0
- os: windows-latest
rust: 1.80.1
rust: 1.82.0
- os: macos-latest
rust: 1.80.1
rust: 1.82.0
# Minimum Supported Rust Version = 1.77.0
- os: ubuntu-latest
@@ -211,9 +211,9 @@ jobs:
include:
# Currently used Rust version.
- os: ubuntu-latest
python: 3.12
python: 3.13
- os: macos-latest
python: 3.12
python: 3.13
# PyPy tests
- os: ubuntu-latest
@@ -263,11 +263,11 @@ jobs:
matrix:
include:
- os: ubuntu-latest
python: 3.12
python: 3.13
- os: macos-latest
python: 3.12
python: 3.13
- os: windows-latest
python: 3.12
python: 3.13
# PyPy tests
- os: ubuntu-latest

View File

@@ -1,5 +1,287 @@
# Changelog
## [1.148.1] - 2024-10-23
### Build system
- Revert "build: nix flake update"
This reverts commit 6f22ce2722b51773d7fbb0d89e4764f963cafd91..
## [1.148.0] - 2024-10-22
### API-Changes
- Create QR codes from any data ([#6090](https://github.com/deltachat/deltachat-core-rust/pull/6090)).
- Add delta chat logo to QR codes ([#6093](https://github.com/deltachat/deltachat-core-rust/pull/6093)).
- Add realtime advertisement received event ([#6043](https://github.com/deltachat/deltachat-core-rust/pull/6043)).
- Notify adding reactions ([#6072](https://github.com/deltachat/deltachat-core-rust/pull/6072))
- Internal profile names ([#6088](https://github.com/deltachat/deltachat-core-rust/pull/6088)).
### Features / Changes
- IMAP COMPRESS support.
- Sort received outgoing message down if it's fresher than all non fresh messages.
- Prioritize cached results if DNS resolver returns many results.
- Add in-memory cache for DNS.
- deltachat-repl: Built-in QR code printer.
- Log the logic for (not) doing AEAP.
- Log when late Autocrypt header is ignored.
- Add more context to `send_msg` errors.
### Fixes
- Replace old draft with a new one atomically.
- ChatId::maybe_delete_draft: Don't delete message if it's not a draft anymore ([#6053](https://github.com/deltachat/deltachat-core-rust/pull/6053)).
- Call update_connection_history for proxified connections.
- sql: Set PRAGMA query_only to avoid writing on read-only connections.
- sql: Run `PRAGMA incremental_vacuum` on a write connection.
- Increase MAX_SECONDS_TO_LEND_FROM_FUTURE to 30.
### Build system
- Nix flake update.
- Resolve warning about default-features, and make it possible to disable vendoring ([#6079](https://github.com/deltachat/deltachat-core-rust/pull/6079)).
- Silence a rust-analyzer false-positive ([#6077](https://github.com/deltachat/deltachat-core-rust/pull/6077)).
### CI
- Update Rust to 1.82.0.
### Documentation
- Set_protection_for_timestamp_sort does not send messages.
- Document MimeFactory.req_mdn.
- Fix `too_long_first_doc_paragraph` clippy lint.
### Refactor
- Update_msg_state: Don't avoid downgrading OutMdnRcvd to OutDelivered.
- Fix elided_named_lifetimes warning.
- set_protection_for_timestamp_sort: Do not log bubbled up errors.
- Fix clippy::needless_lifetimes warnings.
- Use `HeaderDef` constant for Chat-Disposition-Notification-To.
- Resultify get_self_fingerprint().
- sql: Move write mutex into connection pool.
### Tests
- test_qr_setup_contact_svg: Stop testing for no display name.
- Always gossip if gossip_period is set to 0.
- test_aeap_flow_verified: Wait for "member added" before sending messages ([#6057](https://github.com/deltachat/deltachat-core-rust/pull/6057)).
- Make test_verified_group_member_added_recovery more reliable.
- test_aeap_flow_verified: Do not start ac1new.
- Fix `test_securejoin_after_contact_resetup` flakiness.
- Message from old setup preserves contact verification, but breaks 1:1 protection.
## [1.147.1] - 2024-10-13
### Build system
- Build Python 3.13 wheels.
- deltachat-rpc-client: Add classifiers for all supported Python versions.
### CI
- Update to Python 3.13.
### Documentation
- CONTRIBUTING.md: Add a note on deleting/changing db columns.
### Fixes
- Reset quota on configured address change ([#5908](https://github.com/deltachat/deltachat-core-rust/pull/5908)).
- Do not emit progress 1000 when configuration is cancelled.
- Assume file extensions are 32 chars max and don't contain whitespace ([#5338](https://github.com/deltachat/deltachat-core-rust/pull/5338)).
- Readd tokens.foreign_id column ([#6038](https://github.com/deltachat/deltachat-core-rust/pull/6038)).
### Miscellaneous Tasks
- cargo: Bump futures-* from 0.3.30 to 0.3.31.
- cargo: Upgrade async_zip to 0.0.17 ([#6035](https://github.com/deltachat/deltachat-core-rust/pull/6035)).
### Refactor
- MsgId::update_download_state: Don't fail if the message doesn't exist anymore.
## [1.147.0] - 2024-10-05
### API-Changes
- [**breaking**] Remove deprecated get_next_media() APIs.
### Features / Changes
- Reuse existing connections in background_fetch() if I/O is started.
- MsgId::get_info(): Report original filename as well.
- More context for the "Cannot establish guaranteed..." info message ([#6022](https://github.com/deltachat/deltachat-core-rust/pull/6022)).
- deltachat-repl: Add `fetch` command to test `background_fetch()`.
- deltachat-repl: Print send-backup QR code to the terminal.
### Fixes
- Do not attempt to reference info messages.
- query_row_optional: Do not treat rows with NULL as missing rows.
- Skip unconfigured folders in `background_fetch()`.
- Break out of accept() loop if there is an error transferring backup.
- Make it possible to cancel ongoing backup transfer.
- Make backup reception cancellable by stopping ongoing process.
- Smooth progress bar for backup transfer.
- Emit progress 0 if get_backup() fails.
### Documentation
- CONTRIBUTING.md: Add more SQL advices.
## [1.146.0] - 2024-10-03
### Fixes
- download_msg: Do not fail if the message does not exist anymore.
- Better log message for failed QR scan.
### Features / Changes
- Assign message to ad-hoc group with matching name and members ([#5385](https://github.com/deltachat/deltachat-core-rust/pull/5385)).
- Use Rustls instead of native TLS for HTTPS requests.
### Miscellaneous Tasks
- cargo: Bump anyhow from 1.0.86 to 1.0.89.
- cargo: Bump tokio-stream from 0.1.15 to 0.1.16.
- cargo: Bump thiserror from 1.0.63 to 1.0.64.
- cargo: Bump bytes from 1.7.1 to 1.7.2.
- cargo: Bump libc from 0.2.158 to 0.2.159.
- cargo: Bump tempfile from 3.10.1 to 3.13.0.
- cargo: Bump pretty_assertions from 1.4.0 to 1.4.1.
- cargo: Bump hyper-util from 0.1.7 to 0.1.9.
- cargo: Bump rustls-pki-types from 1.8.0 to 1.9.0.
- cargo: Bump quick-xml from 0.36.1 to 0.36.2.
- cargo: Bump serde from 1.0.209 to 1.0.210.
- cargo: Bump syn from 2.0.77 to 2.0.79.
### Refactor
- Move group name calculation out of create_adhoc_group().
- Merge build_tls() function into wrap_tls().
## [1.145.0] - 2024-09-26
### Fixes
- Avoid changing `delete_server_after` default for existing configurations.
### Miscellaneous Tasks
- Sort dependency list.
### Refactor
- Do not wrap shadowsocks::ProxyClientStream.
## [1.144.0] - 2024-09-21
### API-Changes
- [**breaking**] Make QR code type for proxy not specific to SOCKS5 ([#5980](https://github.com/deltachat/deltachat-core-rust/pull/5980)).
`DC_QR_SOCKS5_PROXY` is replaced with `DC_QR_PROXY`.
### Features / Changes
- Make resending OutPending messages possible ([#5817](https://github.com/deltachat/deltachat-core-rust/pull/5817)).
- Don't SMTP-send messages to self-chat if BccSelf is disabled.
- HTTP(S) tunneling.
- Don't put displayname into From/To/Sender if it equals to address ([#5983](https://github.com/deltachat/deltachat-core-rust/pull/5983)).
- Use IMAP APPEND command to upload sync messages ([#5845](https://github.com/deltachat/deltachat-core-rust/pull/5845)).
- Generate 144-bit group IDs.
- smtp: More verbose SMTP connection establishment errors.
- Log unexpected message state when resending fails.
### Fixes
- Save QR code token regardless of whether the group exists ([#5954](https://github.com/deltachat/deltachat-core-rust/pull/5954)).
- Shorten message text in locally sent messages too ([#2281](https://github.com/deltachat/deltachat-core-rust/pull/2281)).
### Documentation
- CONTRIBUTING.md: Document how to format SQL statements.
### Miscellaneous Tasks
- Update provider database.
- cargo: Update iroh to 0.25.
- cargo: Update lazy_static to 1.5.0.
- deps: Bump async-imap from 0.10.0 to 0.10.1.
### Refactor
- Do not store deprecated `addr` and `is_default` into `keypairs`.
- Remove `addr` from KeyPair.
- Use `KeyPair::new()` in `create_keypair()`.
## [1.143.0] - 2024-09-12
### Features / Changes
- Automatic reconfiguration, e.g. switching to implicit TLS if STARTTLS port stops working.
- Always use preloaded DNS results.
- Add "Auto-Submitted: auto-replied" header to appropriate SecureJoin messages.
- Parallelize IMAP and SMTP connection attempts ([#5915](https://github.com/deltachat/deltachat-core-rust/pull/5915)).
- securejoin: Ignore invalid *-request-with-auth messages silently.
- ChatId::create_for_contact_with_blocked: Don't emit events on no op.
- Delete messages from a chatmail server immediately by default ([#5805](https://github.com/deltachat/deltachat-core-rust/pull/5805)) ([#5840](https://github.com/deltachat/deltachat-core-rust/pull/5840)).
- Shadowsocks support.
- Recognize t.me SOCKS5 proxy QR codes ([#5895](https://github.com/deltachat/deltachat-core-rust/pull/5895))
- Remove old iroh 0.4 and support for old `DCBACKUP` QR codes.
### Fixes
- http: Set I/O timeout to 1 minute rather than whole request timeout.
- Add Auto-Submitted header in a single place.
- Do not allow quotes with "... wrote:" headers in chat messages.
- Don't sync QR code token before populating the group ([#5935](https://github.com/deltachat/deltachat-core-rust/pull/5935)).
### Documentation
- Document that `bcc_self` is enabled by default.
### CI
- Update Rust to 1.81.0.
### Miscellaneous Tasks
- Update provider database.
- cargo: Update iroh to 0.23.0.
- cargo: Reduce number of duplicate dependencies.
- cargo: Replace unmaintained ansi_term with nu-ansi-term.
- Replace `reqwest` with direct usage of `hyper`.
### Refactor
- login_param: Use Config:: constants to avoid typos in key names.
- Make Context::config_exists() crate-public.
- Get_config_bool_opt(): Return None if only default value exists.
### Tests
- Test that alternative port 443 works.
- Alice is (non-)bot on Bob's side after QR contact setup.
## [1.142.12] - 2024-09-02
### Fixes
- Display Config::MdnsEnabled as true by default ([#5948](https://github.com/deltachat/deltachat-core-rust/pull/5948)).
## [1.142.11] - 2024-08-30
### Fixes
- Set backward verification when observing vc-contact-confirm or `vg-member-added` ([#5930](https://github.com/deltachat/deltachat-core-rust/pull/5930)).
## [1.142.10] - 2024-08-26
### Fixes
@@ -4792,3 +5074,13 @@ https://github.com/deltachat/deltachat-core-rust/pulls?q=is%3Apr+is%3Aclosed
[1.142.8]: https://github.com/deltachat/deltachat-core-rust/compare/v1.142.7...v1.142.8
[1.142.9]: https://github.com/deltachat/deltachat-core-rust/compare/v1.142.8...v1.142.9
[1.142.10]: https://github.com/deltachat/deltachat-core-rust/compare/v1.142.9..v1.142.10
[1.142.11]: https://github.com/deltachat/deltachat-core-rust/compare/v1.142.10..v1.142.11
[1.142.12]: https://github.com/deltachat/deltachat-core-rust/compare/v1.142.11..v1.142.12
[1.143.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.142.12..v1.143.0
[1.144.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.143.0..v1.144.0
[1.145.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.144.0..v1.145.0
[1.146.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.145.0..v1.146.0
[1.147.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.146.0..v1.147.0
[1.147.1]: https://github.com/deltachat/deltachat-core-rust/compare/v1.147.0..v1.147.1
[1.148.0]: https://github.com/deltachat/deltachat-core-rust/compare/v1.147.1..v1.148.0
[1.148.1]: https://github.com/deltachat/deltachat-core-rust/compare/v1.148.0..v1.148.1

View File

@@ -27,7 +27,7 @@ add_custom_command(
PREFIX=${CMAKE_INSTALL_PREFIX}
LIBDIR=${CMAKE_INSTALL_FULL_LIBDIR}
INCLUDEDIR=${CMAKE_INSTALL_FULL_INCLUDEDIR}
${CARGO} build --target-dir=${CMAKE_BINARY_DIR}/target --release --no-default-features --features jsonrpc
${CARGO} build --target-dir=${CMAKE_BINARY_DIR}/target --release --features jsonrpc
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/deltachat-ffi
)

View File

@@ -32,6 +32,66 @@ on the contributing page: <https://github.com/deltachat/deltachat-core-rust/cont
We format the code using `rustfmt`. Run `cargo fmt` prior to committing the code.
Run `scripts/clippy.sh` to check the code for common mistakes with [Clippy].
### SQL
Multi-line SQL statements should be formatted using string literals,
for example
```
sql.execute(
"CREATE TABLE messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
text TEXT DEFAULT '' NOT NULL -- message text
) STRICT",
)
.await?;
```
Do not use macros like [`concat!`](https://doc.rust-lang.org/std/macro.concat.html)
or [`indoc!](https://docs.rs/indoc).
Do not escape newlines like this:
```
sql.execute(
"CREATE TABLE messages ( \
id INTEGER PRIMARY KEY AUTOINCREMENT, \
text TEXT DEFAULT '' NOT NULL \
) STRICT",
)
.await?;
```
Escaping newlines
is prone to errors like this if space before backslash is missing:
```
"SELECT foo\
FROM bar"
```
Literal above results in `SELECT fooFROM bar` string.
This style also does not allow using `--` comments.
---
Declare new SQL tables with [`STRICT`](https://sqlite.org/stricttables.html) keyword
to make SQLite check column types.
Declare primary keys with [`AUTOINCREMENT`](https://www.sqlite.org/autoinc.html) keyword.
This avoids reuse of the row IDs and can avoid dangerous bugs
like forwarding wrong message because the message was deleted
and another message took its row ID.
Declare all new columns as `NOT NULL`
and set the `DEFAULT` value if it is optional so the column can be skipped in `INSERT` statements.
Dealing with `NULL` values both in SQL and in Rust is tricky and we try to avoid it.
If column is already declared without `NOT NULL`, use `IFNULL` function to provide default value when selecting it.
Use `HAVING COUNT(*) > 0` clause
to [prevent aggregate functions such as `MIN` and `MAX` from returning `NULL`](https://stackoverflow.com/questions/66527856/aggregate-functions-max-etc-return-null-instead-of-no-rows).
Don't delete unused columns too early, but maybe after several months/releases, unused columns are
still used by older versions, so deleting them breaks downgrading the core or importing a backup in
an older version. Also don't change the column type, consider adding a new column with another name
instead. Finally, never change column semantics, this is especially dangerous because the `STRICT`
keyword doesn't help here.
### Commit messages
Commit messages follow the [Conventional Commits] notation.
We use [git-cliff] to generate the changelog from commit messages before the release.

2055
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat"
version = "1.142.10"
version = "1.148.1"
edition = "2021"
license = "MPL-2.0"
rust-version = "1.77"
@@ -41,27 +41,30 @@ ratelimit = { path = "./deltachat-ratelimit" }
anyhow = { workspace = true }
async-broadcast = "0.7.1"
async-channel = { workspace = true }
async-imap = { version = "0.9.7", default-features = false, features = ["runtime-tokio"] }
async-imap = { version = "0.10.2", default-features = false, features = ["runtime-tokio", "compress"] }
async-native-tls = { version = "0.5", default-features = false, features = ["runtime-tokio"] }
async-smtp = { version = "0.9", default-features = false, features = ["runtime-tokio"] }
async_zip = { version = "0.0.12", default-features = false, features = ["deflate", "fs"] }
async_zip = { version = "0.0.17", default-features = false, features = ["deflate", "tokio-fs"] }
base64 = { workspace = true }
brotli = { version = "6", default-features=false, features = ["std"] }
bytes = "1"
chrono = { workspace = true, features = ["alloc", "clock", "std"] }
email = { git = "https://github.com/deltachat/rust-email", branch = "master" }
encoded-words = { git = "https://github.com/async-email/encoded-words", branch = "master" }
escaper = "0.1"
fast-socks5 = "0.9"
fd-lock = "4"
futures = { workspace = true }
futures-lite = { workspace = true }
futures = { workspace = true }
hex = "0.4.0"
hickory-resolver = "0.24"
hickory-resolver = "=0.25.0-alpha.2"
http-body-util = "0.1.2"
humansize = "2"
hyper = "1"
hyper-util = "0.1.9"
image = { version = "0.25.1", default-features=false, features = ["gif", "jpeg", "ico", "png", "pnm", "webp", "bmp"] }
iroh_old = { version = "0.4.2", default-features = false, package = "iroh"}
iroh-net = { version = "0.22.0", default-features = false }
iroh-gossip = { version = "0.22.0", default-features = false, features = ["net"] }
iroh-gossip = { version = "0.25.0", default-features = false, features = ["net"] }
iroh-net = { version = "0.25.0", default-features = false }
kamadak-exif = "0.5.3"
lettre_email = { git = "https://github.com/deltachat/lettre", branch = "master" }
libc = { workspace = true }
@@ -71,47 +74,53 @@ num_cpus = "1.16"
num-derive = "0.4"
num-traits = { workspace = true }
once_cell = { workspace = true }
percent-encoding = "2.3"
parking_lot = "0.12"
percent-encoding = "2.3"
pgp = { version = "0.13.2", default-features = false }
pin-project = "1"
qrcodegen = "1.7.0"
quick-xml = "0.36"
quoted_printable = "0.5"
rand = { workspace = true }
regex = { workspace = true }
reqwest = { version = "0.12.5", features = ["json"] }
rusqlite = { workspace = true, features = ["sqlcipher"] }
rust-hsluv = "0.1"
rustls-pki-types = "1.9.0"
rustls = { version = "0.23.13", default-features = false }
sanitize-filename = { workspace = true }
serde_json = { workspace = true }
serde_urlencoded = "0.7.1"
serde = { workspace = true, features = ["derive"] }
sha-1 = "0.10"
shadowsocks = { version = "1.21.0", default-features = false, features = ["aead-cipher-2022"] }
smallvec = "1.13.2"
strum = "0.26"
strum_macros = "0.26"
tagger = "4.3.4"
textwrap = "0.16.1"
thiserror = { workspace = true }
tokio = { workspace = true, features = ["fs", "rt-multi-thread", "macros"] }
tokio-io-timeout = "1.2.0"
tokio-stream = { version = "0.1.15", features = ["fs"] }
tokio-rustls = { version = "0.26.0", default-features = false }
tokio-stream = { version = "0.1.16", features = ["fs"] }
tokio-tar = { version = "0.3" } # TODO: integrate tokio into async-tar
tokio-util = { workspace = true }
tokio = { workspace = true, features = ["fs", "rt-multi-thread", "macros"] }
toml = "0.8"
url = "2"
uuid = { version = "1", features = ["serde", "v4"] }
webpki-roots = "0.26.6"
[dev-dependencies]
ansi_term = { workspace = true }
anyhow = { workspace = true, features = ["backtrace"] } # Enable `backtrace` feature in tests.
criterion = { version = "0.5.1", features = ["async_tokio"] }
futures-lite = { workspace = true }
log = { workspace = true }
nu-ansi-term = { workspace = true }
pretty_assertions = "1.4.1"
proptest = { version = "1", default-features = false, features = ["std"] }
tempfile = { workspace = true }
testdir = "0.9.0"
tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
pretty_assertions = "1.3.0"
[workspace]
members = [
@@ -156,26 +165,26 @@ harness = false
[workspace.dependencies]
anyhow = "1"
ansi_term = "0.12.1"
async-channel = "2.3.1"
base64 = "0.22"
chrono = { version = "0.4.38", default-features = false }
deltachat-contact-tools = { path = "deltachat-contact-tools" }
deltachat-jsonrpc = { path = "deltachat-jsonrpc" }
deltachat = { path = "." }
deltachat-jsonrpc = { path = "deltachat-jsonrpc", default-features = false }
deltachat = { path = ".", default-features = false }
futures = "0.3.30"
futures-lite = "2.3.0"
libc = "0.2"
log = "0.4"
nu-ansi-term = "0.46"
num-traits = "0.2"
once_cell = "1.18.0"
rand = "0.8"
regex = "1.10"
rusqlite = "0.32"
sanitize-filename = "0.5"
serde_json = "1"
serde = "1.0"
tempfile = "3.10.1"
serde_json = "1"
tempfile = "3.13.0"
thiserror = "1"
# 1.38 is the latest version before `mio` dependency update
@@ -193,9 +202,7 @@ yerpc = "0.6.2"
default = ["vendored"]
internals = []
vendored = [
"async-native-tls/vendored",
"rusqlite/bundled-sqlcipher-vendored-openssl",
"reqwest/native-tls-vendored"
"rusqlite/bundled-sqlcipher-vendored-openssl"
]
[lints.rust]

View File

@@ -0,0 +1,12 @@
<path
style="fill:#ffffff;fill-opacity:1;stroke:none"
d="m 24.015419,1.2870249 c -12.549421,0 -22.7283936,10.1789711 -22.7283936,22.7283931 0,12.549422 10.1789726,22.728395 22.7283936,22.728395 14.337742,-0.342877 9.614352,-4.702705 23.697556,0.969161 -7.545453,-13.001555 -1.082973,-13.32964 -0.969161,-23.697556 0,-12.549422 -10.178973,-22.7283931 -22.728395,-22.7283931 z" />
<path
style="fill:#000000;fill-opacity:1;stroke:none"
d="M 23.982249,5.3106163 C 13.645822,5.4364005 5.2618355,13.92999 5.2618355,24.275753 c 0,10.345764 8.3839865,18.635301 18.7204135,18.509516 9.827724,-0.03951 7.516769,-5.489695 18.380082,-0.443187 -5.950849,-9.296115 0.201753,-10.533667 0.340336,-18.521947 0,-10.345766 -8.383989,-18.6353031 -18.720418,-18.5095187 z" />
<g
style="fill:#ffffff"
transform="scale(1.1342891,0.88160947)">
<path
d="m 21.360141,23.513382 q -1.218487,-1.364705 -3.387392,-3.265543 -2.388233,-2.095797 -3.216804,-3.289913 -0.828571,-1.218486 -0.828571,-2.6563 0,-2.144536 1.998318,-3.363022 1.998317,-1.2428565 5.215121,-1.2428565 3.216804,0 5.605037,1.0966375 2.412603,1.096638 2.412603,3.021846 0,0.92605 -0.584873,1.535293 -0.584874,0.609243 -1.364705,0.609243 -1.121008,0 -2.631931,-1.681511 -1.535292,-1.705881 -2.60756,-2.388233 -1.047898,-0.706722 -2.461343,-0.706722 -1.803359,0 -2.973106,0.804201 -1.145377,0.804201 -1.145377,2.047057 0,1.169747 0.950419,2.193275 0.950419,1.023529 4.898315,3.728568 4.215963,2.899998 5.946213,4.532769 1.75462,1.632772 2.851258,3.972265 1.096638,2.339494 1.096638,4.947055 0,4.581508 -3.241174,8.090749 -3.216804,3.484871 -7.530245,3.484871 -3.923526,0 -6.628566,-2.802519 -2.705039,-2.802518 -2.705039,-7.481506 0,-4.508399 2.973106,-7.530245 2.997477,-3.021846 7.359658,-3.655459 z m 1.072268,1.121008 q -6.994112,1.145377 -6.994112,9.601672 0,4.36218 1.730251,6.774783 1.75462,2.412603 4.069744,2.412603 2.412603,0 3.972265,-2.315124 1.559663,-2.339493 1.559663,-6.311759 0,-5.751255 -4.337811,-10.162175 z" />
</g>

View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat_ffi"
version = "1.142.10"
version = "1.148.1"
description = "Deltachat FFI"
edition = "2018"
readme = "README.md"

View File

@@ -403,11 +403,8 @@ char* dc_get_blobdir (const dc_context_t* context);
* - `send_port` = SMTP-port, guessed if left out
* - `send_security`= SMTP-socket, one of @ref DC_SOCKET, defaults to #DC_SOCKET_AUTO
* - `server_flags` = IMAP-/SMTP-flags as a combination of @ref DC_LP flags, guessed if left out
* - `socks5_enabled` = SOCKS5 enabled
* - `socks5_host` = SOCKS5 proxy server host
* - `socks5_port` = SOCKS5 proxy server port
* - `socks5_user` = SOCKS5 proxy username
* - `socks5_password` = SOCKS5 proxy password
* - `proxy_enabled` = Proxy enabled. Disabled by default.
* - `proxy_url` = Proxy URL. May contain multiple URLs separated by newline, but only the first one is used.
* - `imap_certificate_checks` = how to check IMAP certificates, one of the @ref DC_CERTCK flags, defaults to #DC_CERTCK_AUTO (0)
* - `smtp_certificate_checks` = deprecated option, should be set to the same value as `imap_certificate_checks` but ignored by the new core
* - `displayname` = Own name to use when sending messages. MUAs are allowed to spread this way e.g. using CC, defaults to empty
@@ -422,8 +419,8 @@ char* dc_get_blobdir (const dc_context_t* context);
* - `mdns_enabled` = 0=do not send or request read receipts,
* 1=send and request read receipts
* default=send and request read receipts, only send but not reuqest if `bot` is set
* - `bcc_self` = 0=do not send a copy of outgoing messages to self (default),
* 1=send a copy of outgoing messages to self.
* - `bcc_self` = 0=do not send a copy of outgoing messages to self,
* 1=send a copy of outgoing messages to self (default).
* Sending messages to self is needed for a proper multi-account setup,
* however, on the other hand, may lead to unwanted notifications in non-delta clients.
* - `sentbox_watch`= 1=watch `Sent`-folder for changes,
@@ -525,6 +522,8 @@ char* dc_get_blobdir (const dc_context_t* context);
* In contrast to `dc_set_chat_mute_duration()`,
* fresh message and badge counters are not changed by this setting,
* but should be tuned down where appropriate.
* - `private_tag` = Optional tag as "Work", "Family".
* Meant to help profile owner to differ between profiles with similar names.
* - `ui.*` = All keys prefixed by `ui.` can be used by the user-interfaces for system-specific purposes.
* The prefix should be followed by the system and maybe subsystem,
* e.g. `ui.desktop.foo`, `ui.desktop.linux.bar`, `ui.android.foo`, `ui.dc40.bar`, `ui.bot.simplebot.baz`.
@@ -867,13 +866,10 @@ void dc_maybe_network (dc_context_t* context);
*
* @memberof dc_context_t
* @param context The context as created by dc_context_new().
* @param addr The e-mail address of the user. This must match the
* configured_addr setting of the context as well as the UID of the key.
* @param public_data Ignored, actual public key is extracted from secret_data.
* @param secret_data ASCII armored secret key.
* @return 1 on success, 0 on failure.
*/
int dc_preconfigure_keypair (dc_context_t* context, const char *addr, const char *public_data, const char *secret_data);
int dc_preconfigure_keypair (dc_context_t* context, const char *secret_data);
// handle chatlists
@@ -1553,30 +1549,6 @@ void dc_marknoticed_chat (dc_context_t* context, uint32_t ch
dc_array_t* dc_get_chat_media (dc_context_t* context, uint32_t chat_id, int msg_type, int msg_type2, int msg_type3);
/**
* Search next/previous message based on a given message and a list of types.
* Typically used to implement the "next" and "previous" buttons
* in a gallery or in a media player.
*
* @deprecated Deprecated 2023-10-03, use dc_get_chat_media() and navigate the returned array instead.
* @memberof dc_context_t
* @param context The context object as returned from dc_context_new().
* @param msg_id The ID of the current message from which the next or previous message should be searched.
* @param dir 1=get the next message, -1=get the previous one.
* @param msg_type The message type to search for.
* If 0, the message type from curr_msg_id is used.
* @param msg_type2 Alternative message type to search for. 0 to skip.
* @param msg_type3 Alternative message type to search for. 0 to skip.
* @return Returns the message ID that should be played next.
* The returned message is in the same chat as the given one
* and has one of the given types.
* Typically, this result is passed again to dc_get_next_media()
* later on the next swipe.
* If there is not next/previous message, the function returns 0.
*/
uint32_t dc_get_next_media (dc_context_t* context, uint32_t msg_id, int dir, int msg_type, int msg_type2, int msg_type3);
/**
* Set chat visibility to pinned, archived or normal.
*
@@ -2507,6 +2479,7 @@ void dc_stop_ongoing_process (dc_context_t* context);
#define DC_QR_BACKUP 251
#define DC_QR_BACKUP2 252
#define DC_QR_WEBRTC_INSTANCE 260 // text1=domain, text2=instance pattern
#define DC_QR_PROXY 271 // text1=address (e.g. "127.0.0.1:9050")
#define DC_QR_ADDR 320 // id=contact
#define DC_QR_TEXT 330 // text1=text
#define DC_QR_URL 332 // text1=URL
@@ -2560,6 +2533,10 @@ void dc_stop_ongoing_process (dc_context_t* context);
* ask the user if they want to use the given service for video chats;
* if so, call dc_set_config_from_qr().
*
* - DC_QR_SOCKS5_PROXY with dc_lot_t::text1=host, dc_lot_t::text2=port:
* ask the user if they want to use the given proxy and overwrite the previous one, if any.
* if so, call dc_set_config_from_qr() and restart I/O.
*
* - DC_QR_ADDR with dc_lot_t::id=Contact ID:
* e-mail address scanned, optionally, a draft message could be set in
* dc_lot_t::text1 in which case dc_lot_t::text1_meaning will be DC_TEXT1_DRAFT;
@@ -2634,6 +2611,7 @@ char* dc_get_securejoin_qr (dc_context_t* context, uint32_t ch
* Get QR code image from the QR code text generated by dc_get_securejoin_qr().
* See dc_get_securejoin_qr() for details about the contained QR code.
*
* @deprecated 2024-10 use dc_create_qr_svg(dc_get_securejoin_qr()) instead.
* @memberof dc_context_t
* @param context The context object.
* @param chat_id group-chat-id for secure-join or 0 for setup-contact,
@@ -2814,6 +2792,22 @@ dc_array_t* dc_get_locations (dc_context_t* context, uint32_t cha
void dc_delete_all_locations (dc_context_t* context);
// misc
/**
* Create a QR code from any input data.
*
* The QR code is returned as a square SVG image.
*
* @memberof dc_context_t
* @param payload The content for the QR code.
* @return SVG image with the QR code.
* On errors, an empty string is returned.
* The returned string must be released using dc_str_unref() after usage.
*/
char* dc_create_qr_svg (const char* payload);
/**
* Get last error string.
*
@@ -2902,6 +2896,7 @@ char* dc_backup_provider_get_qr (const dc_backup_provider_t* backup_provider);
* This works like dc_backup_provider_qr() but returns the text of a rendered
* SVG image containing the QR code.
*
* @deprecated 2024-10 use dc_create_qr_svg(dc_backup_provider_get_qr()) instead.
* @memberof dc_backup_provider_t
* @param backup_provider The backup provider object as created by
* dc_backup_provider_new().
@@ -2941,7 +2936,7 @@ void dc_backup_provider_unref (dc_backup_provider_t* backup_provider);
* Gets a backup offered by a dc_backup_provider_t object on another device.
*
* This function is called on a device that scanned the QR code offered by
* dc_backup_sender_qr() or dc_backup_sender_qr_svg(). Typically this is a
* dc_backup_provider_get_qr(). Typically this is a
* different device than that which provides the backup.
*
* This call will block while the backup is being transferred and only
@@ -6056,6 +6051,21 @@ void dc_event_unref(dc_event_t* event);
#define DC_EVENT_REACTIONS_CHANGED 2001
/**
* A reaction to one's own sent message received.
* Typically, the UI will show a notification for that.
*
* In addition to this event, DC_EVENT_REACTIONS_CHANGED is emitted.
*
* @param data1 (int) contact_id ID of the contact sending this reaction.
* @param data2 (int) msg_id + (char*) reaction.
* ID of the message for which a reaction was received in dc_event_get_data2_int(),
* and the reaction as dc_event_get_data2_str().
* string must be passed to dc_str_unref() afterwards.
*/
#define DC_EVENT_INCOMING_REACTION 2002
/**
* There is a fresh message. Typically, the user will show an notification
* when receiving this message.
@@ -6273,7 +6283,7 @@ void dc_event_unref(dc_event_t* event);
/**
* webxdc status update received.
* Webxdc status update received.
* To get the received status update, use dc_get_webxdc_status_updates() with
* `serial` set to the last known update
* (in case of special bots, `status_update_serial` from `data2`
@@ -6308,6 +6318,15 @@ void dc_event_unref(dc_event_t* event);
#define DC_EVENT_WEBXDC_REALTIME_DATA 2150
/**
* Advertisement for ephemeral peer channel communication received.
* This can be used by bots to initiate peer-to-peer communication from their side.
* @param data1 (int) msg_id
* @param data2 0
*/
#define DC_EVENT_WEBXDC_REALTIME_ADVERTISEMENT 2151
/**
* Tells that the Background fetch was completed (or timed out).
*

View File

@@ -30,7 +30,7 @@ use deltachat::ephemeral::Timer as EphemeralTimer;
use deltachat::imex::BackupProvider;
use deltachat::key::preconfigure_keypair;
use deltachat::message::MsgId;
use deltachat::qr_code_generator::{generate_backup_qr, get_securejoin_qr_svg};
use deltachat::qr_code_generator::{create_qr_svg, generate_backup_qr, get_securejoin_qr_svg};
use deltachat::stock_str::StockMessage;
use deltachat::webxdc::StatusUpdateSerial;
use deltachat::*;
@@ -541,6 +541,7 @@ pub unsafe extern "C" fn dc_event_get_id(event: *mut dc_event_t) -> libc::c_int
EventType::ErrorSelfNotInGroup(_) => 410,
EventType::MsgsChanged { .. } => 2000,
EventType::ReactionsChanged { .. } => 2001,
EventType::IncomingReaction { .. } => 2002,
EventType::IncomingMsg { .. } => 2005,
EventType::IncomingMsgBunch { .. } => 2006,
EventType::MsgsNoticed { .. } => 2008,
@@ -563,10 +564,14 @@ pub unsafe extern "C" fn dc_event_get_id(event: *mut dc_event_t) -> libc::c_int
EventType::WebxdcStatusUpdate { .. } => 2120,
EventType::WebxdcInstanceDeleted { .. } => 2121,
EventType::WebxdcRealtimeData { .. } => 2150,
EventType::WebxdcRealtimeAdvertisementReceived { .. } => 2151,
EventType::AccountsBackgroundFetchDone => 2200,
EventType::ChatlistChanged => 2300,
EventType::ChatlistItemChanged { .. } => 2301,
EventType::EventChannelOverflow { .. } => 2400,
#[allow(unreachable_patterns)]
#[cfg(test)]
_ => unreachable!("This is just to silence a rust_analyzer false-positive"),
}
}
@@ -597,6 +602,7 @@ pub unsafe extern "C" fn dc_event_get_data1_int(event: *mut dc_event_t) -> libc:
| EventType::ErrorSelfNotInGroup(_)
| EventType::AccountsBackgroundFetchDone => 0,
EventType::ChatlistChanged => 0,
EventType::IncomingReaction { contact_id, .. } => contact_id.to_u32() as libc::c_int,
EventType::MsgsChanged { chat_id, .. }
| EventType::ReactionsChanged { chat_id, .. }
| EventType::IncomingMsg { chat_id, .. }
@@ -621,11 +627,15 @@ pub unsafe extern "C" fn dc_event_get_data1_int(event: *mut dc_event_t) -> libc:
}
EventType::WebxdcRealtimeData { msg_id, .. }
| EventType::WebxdcStatusUpdate { msg_id, .. }
| EventType::WebxdcRealtimeAdvertisementReceived { msg_id }
| EventType::WebxdcInstanceDeleted { msg_id, .. } => msg_id.to_u32() as libc::c_int,
EventType::ChatlistItemChanged { chat_id } => {
chat_id.unwrap_or_default().to_u32() as libc::c_int
}
EventType::EventChannelOverflow { n } => *n as libc::c_int,
#[allow(unreachable_patterns)]
#[cfg(test)]
_ => unreachable!("This is just to silence a rust_analyzer false-positive"),
}
}
@@ -666,9 +676,11 @@ pub unsafe extern "C" fn dc_event_get_data2_int(event: *mut dc_event_t) -> libc:
| EventType::ChatlistItemChanged { .. }
| EventType::ConfigSynced { .. }
| EventType::ChatModified(_)
| EventType::WebxdcRealtimeAdvertisementReceived { .. }
| EventType::EventChannelOverflow { .. } => 0,
EventType::MsgsChanged { msg_id, .. }
| EventType::ReactionsChanged { msg_id, .. }
| EventType::IncomingReaction { msg_id, .. }
| EventType::IncomingMsg { msg_id, .. }
| EventType::MsgDelivered { msg_id, .. }
| EventType::MsgFailed { msg_id, .. }
@@ -682,6 +694,9 @@ pub unsafe extern "C" fn dc_event_get_data2_int(event: *mut dc_event_t) -> libc:
..
} => status_update_serial.to_u32() as libc::c_int,
EventType::WebxdcRealtimeData { data, .. } => data.len() as libc::c_int,
#[allow(unreachable_patterns)]
#[cfg(test)]
_ => unreachable!("This is just to silence a rust_analyzer false-positive"),
}
}
@@ -733,6 +748,7 @@ pub unsafe extern "C" fn dc_event_get_data2_str(event: *mut dc_event_t) -> *mut
| EventType::IncomingMsgBunch { .. }
| EventType::ChatlistItemChanged { .. }
| EventType::ChatlistChanged
| EventType::WebxdcRealtimeAdvertisementReceived { .. }
| EventType::EventChannelOverflow { .. } => ptr::null_mut(),
EventType::ConfigureProgress { comment, .. } => {
if let Some(comment) = comment {
@@ -754,6 +770,14 @@ pub unsafe extern "C" fn dc_event_get_data2_str(event: *mut dc_event_t) -> *mut
libc::memcpy(ptr, data.as_ptr() as *mut libc::c_void, data.len());
ptr as *mut libc::c_char
}
EventType::IncomingReaction { reaction, .. } => reaction
.as_str()
.to_c_string()
.unwrap_or_default()
.into_raw(),
#[allow(unreachable_patterns)]
#[cfg(test)]
_ => unreachable!("This is just to silence a rust_analyzer false-positive"),
}
}
@@ -835,8 +859,6 @@ pub unsafe extern "C" fn dc_maybe_network(context: *mut dc_context_t) {
#[no_mangle]
pub unsafe extern "C" fn dc_preconfigure_keypair(
context: *mut dc_context_t,
addr: *const libc::c_char,
_public_data: *const libc::c_char,
secret_data: *const libc::c_char,
) -> i32 {
if context.is_null() {
@@ -844,9 +866,8 @@ pub unsafe extern "C" fn dc_preconfigure_keypair(
return 0;
}
let ctx = &*context;
let addr = to_string_lossy(addr);
let secret_data = to_string_lossy(secret_data);
block_on(preconfigure_keypair(ctx, &addr, &secret_data))
block_on(preconfigure_keypair(ctx, &secret_data))
.context("Failed to save keypair")
.log_err(ctx)
.is_ok() as libc::c_int
@@ -1446,48 +1467,6 @@ pub unsafe extern "C" fn dc_get_chat_media(
})
}
#[no_mangle]
#[allow(deprecated)]
pub unsafe extern "C" fn dc_get_next_media(
context: *mut dc_context_t,
msg_id: u32,
dir: libc::c_int,
msg_type: libc::c_int,
or_msg_type2: libc::c_int,
or_msg_type3: libc::c_int,
) -> u32 {
if context.is_null() {
eprintln!("ignoring careless call to dc_get_next_media()");
return 0;
}
let direction = if dir < 0 {
chat::Direction::Backward
} else {
chat::Direction::Forward
};
let ctx = &*context;
let msg_type = from_prim(msg_type).expect(&format!("invalid msg_type = {msg_type}"));
let or_msg_type2 =
from_prim(or_msg_type2).expect(&format!("incorrect or_msg_type2 = {or_msg_type2}"));
let or_msg_type3 =
from_prim(or_msg_type3).expect(&format!("incorrect or_msg_type3 = {or_msg_type3}"));
block_on(async move {
chat::get_next_media(
ctx,
MsgId::new(msg_id),
direction,
msg_type,
or_msg_type2,
or_msg_type3,
)
.await
.map(|msg_id| msg_id.map(|id| id.to_u32()).unwrap_or_default())
.unwrap_or(0)
})
}
#[no_mangle]
pub unsafe extern "C" fn dc_set_chat_visibility(
context: *mut dc_context_t,
@@ -2615,6 +2594,18 @@ pub unsafe extern "C" fn dc_delete_all_locations(context: *mut dc_context_t) {
});
}
#[no_mangle]
pub unsafe extern "C" fn dc_create_qr_svg(payload: *const libc::c_char) -> *mut libc::c_char {
if payload.is_null() {
eprintln!("ignoring careless call to dc_create_qr_svg()");
return "".strdup();
}
create_qr_svg(&to_string_lossy(payload))
.unwrap_or_else(|_| "".to_string())
.strdup()
}
#[no_mangle]
pub unsafe extern "C" fn dc_get_last_error(context: *mut dc_context_t) -> *mut libc::c_char {
if context.is_null() {
@@ -4537,19 +4528,16 @@ pub unsafe extern "C" fn dc_provider_new_from_email_with_dns(
let addr = to_string_lossy(addr);
let ctx = &*context;
let socks5_enabled = block_on(async move {
ctx.get_config_bool(config::Config::Socks5Enabled)
.await
.context("Can't get config")
.log_err(ctx)
});
let proxy_enabled = block_on(ctx.get_config_bool(config::Config::ProxyEnabled))
.context("Can't get config")
.log_err(ctx);
match socks5_enabled {
Ok(socks5_enabled) => {
match proxy_enabled {
Ok(proxy_enabled) => {
match block_on(provider::get_provider_info_by_addr(
ctx,
addr.as_str(),
socks5_enabled,
proxy_enabled,
))
.log_err(ctx)
.unwrap_or_default()

View File

@@ -34,34 +34,34 @@ pub enum Meaning {
}
impl Lot {
pub fn get_text1(&self) -> Option<&str> {
pub fn get_text1(&self) -> Option<Cow<str>> {
match self {
Self::Summary(summary) => match &summary.prefix {
None => None,
Some(SummaryPrefix::Draft(text)) => Some(text),
Some(SummaryPrefix::Username(username)) => Some(username),
Some(SummaryPrefix::Me(text)) => Some(text),
Some(SummaryPrefix::Draft(text)) => Some(Cow::Borrowed(text)),
Some(SummaryPrefix::Username(username)) => Some(Cow::Borrowed(username)),
Some(SummaryPrefix::Me(text)) => Some(Cow::Borrowed(text)),
},
Self::Qr(qr) => match qr {
Qr::AskVerifyContact { .. } => None,
Qr::AskVerifyGroup { grpname, .. } => Some(grpname),
Qr::AskVerifyGroup { grpname, .. } => Some(Cow::Borrowed(grpname)),
Qr::FprOk { .. } => None,
Qr::FprMismatch { .. } => None,
Qr::FprWithoutAddr { fingerprint, .. } => Some(fingerprint),
Qr::Account { domain } => Some(domain),
Qr::Backup { .. } => None,
Qr::FprWithoutAddr { fingerprint, .. } => Some(Cow::Borrowed(fingerprint)),
Qr::Account { domain } => Some(Cow::Borrowed(domain)),
Qr::Backup2 { .. } => None,
Qr::WebrtcInstance { domain, .. } => Some(domain),
Qr::Addr { draft, .. } => draft.as_deref(),
Qr::Url { url } => Some(url),
Qr::Text { text } => Some(text),
Qr::WebrtcInstance { domain, .. } => Some(Cow::Borrowed(domain)),
Qr::Proxy { host, port, .. } => Some(Cow::Owned(format!("{host}:{port}"))),
Qr::Addr { draft, .. } => draft.as_deref().map(Cow::Borrowed),
Qr::Url { url } => Some(Cow::Borrowed(url)),
Qr::Text { text } => Some(Cow::Borrowed(text)),
Qr::WithdrawVerifyContact { .. } => None,
Qr::WithdrawVerifyGroup { grpname, .. } => Some(grpname),
Qr::WithdrawVerifyGroup { grpname, .. } => Some(Cow::Borrowed(grpname)),
Qr::ReviveVerifyContact { .. } => None,
Qr::ReviveVerifyGroup { grpname, .. } => Some(grpname),
Qr::Login { address, .. } => Some(address),
Qr::ReviveVerifyGroup { grpname, .. } => Some(Cow::Borrowed(grpname)),
Qr::Login { address, .. } => Some(Cow::Borrowed(address)),
},
Self::Error(err) => Some(err),
Self::Error(err) => Some(Cow::Borrowed(err)),
}
}
@@ -102,9 +102,9 @@ impl Lot {
Qr::FprMismatch { .. } => LotState::QrFprMismatch,
Qr::FprWithoutAddr { .. } => LotState::QrFprWithoutAddr,
Qr::Account { .. } => LotState::QrAccount,
Qr::Backup { .. } => LotState::QrBackup,
Qr::Backup2 { .. } => LotState::QrBackup2,
Qr::WebrtcInstance { .. } => LotState::QrWebrtcInstance,
Qr::Proxy { .. } => LotState::QrProxy,
Qr::Addr { .. } => LotState::QrAddr,
Qr::Url { .. } => LotState::QrUrl,
Qr::Text { .. } => LotState::QrText,
@@ -128,9 +128,9 @@ impl Lot {
Qr::FprMismatch { contact_id } => contact_id.unwrap_or_default().to_u32(),
Qr::FprWithoutAddr { .. } => Default::default(),
Qr::Account { .. } => Default::default(),
Qr::Backup { .. } => Default::default(),
Qr::Backup2 { .. } => Default::default(),
Qr::WebrtcInstance { .. } => Default::default(),
Qr::Proxy { .. } => Default::default(),
Qr::Addr { contact_id, .. } => contact_id.to_u32(),
Qr::Url { .. } => Default::default(),
Qr::Text { .. } => Default::default(),
@@ -185,6 +185,9 @@ pub enum LotState {
/// text1=domain, text2=instance pattern
QrWebrtcInstance = 260,
/// text1=address, text2=protocol
QrProxy = 271,
/// id=contact
QrAddr = 320,

View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat-jsonrpc"
version = "1.142.10"
version = "1.148.1"
description = "DeltaChat JSON-RPC API"
edition = "2021"
default-run = "deltachat-jsonrpc-server"
@@ -25,7 +25,7 @@ async-channel = { workspace = true }
futures = { workspace = true }
serde_json = { workspace = true }
yerpc = { workspace = true, features = ["anyhow_expose", "openrpc"] }
typescript-type-def = { version = "0.5.8", features = ["json_value"] }
typescript-type-def = { version = "0.5.12", features = ["json_value"] }
tokio = { workspace = true }
sanitize-filename = { workspace = true }
walkdir = "2.5.0"

View File

@@ -321,12 +321,12 @@ impl CommandApi {
) -> Result<Option<ProviderInfo>> {
let ctx = self.get_context(account_id).await?;
let socks5_enabled = ctx
.get_config_bool(deltachat::config::Config::Socks5Enabled)
let proxy_enabled = ctx
.get_config_bool(deltachat::config::Config::ProxyEnabled)
.await?;
let provider_info =
get_provider_info(&ctx, email.split('@').last().unwrap_or(""), socks5_enabled).await;
get_provider_info(&ctx, email.split('@').last().unwrap_or(""), proxy_enabled).await;
Ok(ProviderInfo::from_dc_type(provider_info))
}
@@ -1552,55 +1552,6 @@ impl CommandApi {
Ok(media.iter().map(|msg_id| msg_id.to_u32()).collect())
}
/// Search next/previous message based on a given message and a list of types.
/// Typically used to implement the "next" and "previous" buttons
/// in a gallery or in a media player.
///
/// one combined call for getting chat::get_next_media for both directions
/// the manual chat::get_next_media in only one direction is not exposed by the jsonrpc yet
///
/// Deprecated 2023-10-03, use `get_chat_media` method
/// and navigate the returned array instead.
#[allow(deprecated)]
async fn get_neighboring_chat_media(
&self,
account_id: u32,
msg_id: u32,
message_type: MessageViewtype,
or_message_type2: Option<MessageViewtype>,
or_message_type3: Option<MessageViewtype>,
) -> Result<(Option<u32>, Option<u32>)> {
let ctx = self.get_context(account_id).await?;
let msg_type: Viewtype = message_type.into();
let msg_type2: Viewtype = or_message_type2.map(|v| v.into()).unwrap_or_default();
let msg_type3: Viewtype = or_message_type3.map(|v| v.into()).unwrap_or_default();
let prev = chat::get_next_media(
&ctx,
MsgId::new(msg_id),
chat::Direction::Backward,
msg_type,
msg_type2,
msg_type3,
)
.await?
.map(|id| id.to_u32());
let next = chat::get_next_media(
&ctx,
MsgId::new(msg_id),
chat::Direction::Forward,
msg_type,
msg_type2,
msg_type3,
)
.await?
.map(|id| id.to_u32());
Ok((prev, next))
}
// ---------------------------------------------
// backup
// ---------------------------------------------
@@ -1995,9 +1946,13 @@ impl CommandApi {
async fn send_msg(&self, account_id: u32, chat_id: u32, data: MessageData) -> Result<u32> {
let ctx = self.get_context(account_id).await?;
let mut message = data.create_message(&ctx).await?;
let mut message = data
.create_message(&ctx)
.await
.context("Failed to create message")?;
let msg_id = chat::send_msg(&ctx, ChatId::new(chat_id), &mut message)
.await?
.await
.context("Failed to send created message")?
.to_u32();
Ok(msg_id)
}

View File

@@ -98,6 +98,14 @@ pub enum EventType {
contact_id: u32,
},
/// Incoming reaction, should be notified.
#[serde(rename_all = "camelCase")]
IncomingReaction {
contact_id: u32,
msg_id: u32,
reaction: String,
},
/// There is a fresh message. Typically, the user will show an notification
/// when receiving this message.
///
@@ -244,6 +252,11 @@ pub enum EventType {
#[serde(rename_all = "camelCase")]
WebxdcRealtimeData { msg_id: u32, data: Vec<u8> },
/// Advertisement received over an ephemeral peer channel.
/// This can be used by bots to initiate peer-to-peer communication from their side.
#[serde(rename_all = "camelCase")]
WebxdcRealtimeAdvertisementReceived { msg_id: u32 },
/// Inform that a message containing a webxdc instance has been deleted
#[serde(rename_all = "camelCase")]
WebxdcInstanceDeleted { msg_id: u32 },
@@ -297,6 +310,15 @@ impl From<CoreEventType> for EventType {
msg_id: msg_id.to_u32(),
contact_id: contact_id.to_u32(),
},
CoreEventType::IncomingReaction {
contact_id,
msg_id,
reaction,
} => IncomingReaction {
contact_id: contact_id.to_u32(),
msg_id: msg_id.to_u32(),
reaction: reaction.as_str().to_string(),
},
CoreEventType::IncomingMsg { chat_id, msg_id } => IncomingMsg {
chat_id: chat_id.to_u32(),
msg_id: msg_id.to_u32(),
@@ -373,6 +395,11 @@ impl From<CoreEventType> for EventType {
msg_id: msg_id.to_u32(),
data,
},
CoreEventType::WebxdcRealtimeAdvertisementReceived { msg_id } => {
WebxdcRealtimeAdvertisementReceived {
msg_id: msg_id.to_u32(),
}
}
CoreEventType::WebxdcInstanceDeleted { msg_id } => WebxdcInstanceDeleted {
msg_id: msg_id.to_u32(),
},

View File

@@ -605,16 +605,13 @@ impl MessageData {
message.set_location(latitude, longitude);
}
if let Some(id) = self.quoted_message_id {
let quoted_message = Message::load_from_db(context, MsgId::new(id))
.await
.context("Failed to load quoted message")?;
message
.set_quote(
context,
Some(
&Message::load_from_db(context, MsgId::new(id))
.await
.context("message to quote could not be loaded")?,
),
)
.await?;
.set_quote(context, Some(&quoted_message))
.await
.context("Failed to set quote")?;
} else if let Some(text) = self.quoted_text {
let protect = false;
message.set_quote_text(Some((text, protect)));
@@ -640,7 +637,7 @@ pub struct MessageInfo {
error: Option<String>,
rfc724_mid: String,
server_urls: Vec<String>,
hop_info: Option<String>,
hop_info: String,
}
impl MessageInfo {

View File

@@ -32,9 +32,6 @@ pub enum QrObject {
Account {
domain: String,
},
Backup {
ticket: String,
},
Backup2 {
auth_token: String,
@@ -44,6 +41,11 @@ pub enum QrObject {
domain: String,
instance_pattern: String,
},
Proxy {
url: String,
host: String,
port: u16,
},
Addr {
contact_id: u32,
draft: Option<String>,
@@ -134,9 +136,6 @@ impl From<Qr> for QrObject {
}
Qr::FprWithoutAddr { fingerprint } => QrObject::FprWithoutAddr { fingerprint },
Qr::Account { domain } => QrObject::Account { domain },
Qr::Backup { ticket } => QrObject::Backup {
ticket: ticket.to_string(),
},
Qr::Backup2 {
ref node_addr,
auth_token,
@@ -152,6 +151,7 @@ impl From<Qr> for QrObject {
domain,
instance_pattern,
},
Qr::Proxy { url, host, port } => QrObject::Proxy { url, host, port },
Qr::Addr { contact_id, draft } => {
let contact_id = contact_id.to_u32();
QrObject::Addr { contact_id, draft }

View File

@@ -83,7 +83,7 @@ mod tests {
assert_eq!(result, response.to_owned());
}
{
let request = r#"{"jsonrpc":"2.0","method":"batch_set_config","id":2,"params":[1,{"addr":"","mail_user":"","mail_pw":"","mail_server":"","mail_port":"","mail_security":"","imap_certificate_checks":"","send_user":"","send_pw":"","send_server":"","send_port":"","send_security":"","smtp_certificate_checks":"","socks5_enabled":"0","socks5_host":"","socks5_port":"","socks5_user":"","socks5_password":""}]}"#;
let request = r#"{"jsonrpc":"2.0","method":"batch_set_config","id":2,"params":[1,{"addr":"","mail_user":"","mail_pw":"","mail_server":"","mail_port":"","mail_security":"","imap_certificate_checks":"","send_user":"","send_pw":"","send_server":"","send_port":"","send_security":"","smtp_certificate_checks":""}]}"#;
let response = r#"{"jsonrpc":"2.0","id":2,"result":null}"#;
session.handle_incoming(request).await;
let result = receiver.recv().await?;

View File

@@ -58,5 +58,5 @@
},
"type": "module",
"types": "dist/deltachat.d.ts",
"version": "1.142.10"
"version": "1.148.1"
}

View File

@@ -86,10 +86,7 @@ describe("online tests", function () {
null
);
const chatId = await dc.rpc.createChatByContactId(accountId1, contactId);
const eventPromise = Promise.race([
waitForEvent(dc, "MsgsChanged", accountId2),
waitForEvent(dc, "IncomingMsg", accountId2),
]);
const eventPromise = waitForEvent(dc, "IncomingMsg", accountId2);
await dc.rpc.miscSendTextMessage(accountId1, chatId, "Hello");
const { chatId: chatIdOnAccountB } = await eventPromise;
@@ -119,10 +116,7 @@ describe("online tests", function () {
null
);
const chatId = await dc.rpc.createChatByContactId(accountId1, contactId);
const eventPromise = Promise.race([
waitForEvent(dc, "MsgsChanged", accountId2),
waitForEvent(dc, "IncomingMsg", accountId2),
]);
const eventPromise = waitForEvent(dc, "IncomingMsg", accountId2);
dc.rpc.miscSendTextMessage(accountId1, chatId, "Hello2");
// wait for message from A
console.log("wait for message from A");
@@ -143,10 +137,7 @@ describe("online tests", function () {
);
expect(message.text).equal("Hello2");
// Send message back from B to A
const eventPromise2 = Promise.race([
waitForEvent(dc, "MsgsChanged", accountId1),
waitForEvent(dc, "IncomingMsg", accountId1),
]);
const eventPromise2 = waitForEvent(dc, "IncomingMsg", accountId1);
dc.rpc.miscSendTextMessage(accountId2, chatId, "super secret message");
// Check if answer arrives at A and if it is encrypted
await eventPromise2;

View File

@@ -1,16 +1,17 @@
[package]
name = "deltachat-repl"
version = "1.142.10"
version = "1.148.1"
license = "MPL-2.0"
edition = "2021"
repository = "https://github.com/deltachat/deltachat-core-rust"
[dependencies]
ansi_term = { workspace = true }
anyhow = { workspace = true }
deltachat = { workspace = true, features = ["internals"]}
dirs = "5"
log = { workspace = true }
nu-ansi-term = { workspace = true }
qr2term = "0.3.3"
rusqlite = { workspace = true }
rustyline = "14"
tokio = { workspace = true, features = ["fs", "rt-multi-thread", "macros"] }

View File

@@ -22,6 +22,7 @@ use deltachat::mimeparser::SystemMessage;
use deltachat::peer_channels::{send_webxdc_realtime_advertisement, send_webxdc_realtime_data};
use deltachat::peerstate::*;
use deltachat::qr::*;
use deltachat::qr_code_generator::create_qr_svg;
use deltachat::reaction::send_reaction;
use deltachat::receive_imf::*;
use deltachat::sql;
@@ -355,6 +356,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
configure\n\
connect\n\
disconnect\n\
fetch\n\
connectivity\n\
maybenetwork\n\
housekeeping\n\
@@ -424,6 +426,7 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
checkqr <qr-content>\n\
joinqr <qr-content>\n\
setqr <qr-content>\n\
createqrsvg <qr-content>\n\
providerinfo <addr>\n\
fileinfo <file>\n\
estimatedeletion <seconds>\n\
@@ -486,8 +489,9 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
}
"send-backup" => {
let provider = BackupProvider::prepare(&context).await?;
let qr = provider.qr();
println!("QR code: {}", format_backup(&qr)?);
let qr = format_backup(&provider.qr())?;
println!("QR code: {}", qr);
qr2term::print_qr(qr.as_str())?;
provider.await?;
}
"receive-backup" => {
@@ -1247,12 +1251,19 @@ pub async fn cmdline(context: Context, line: &str, chat_id: &mut ChatId) -> Resu
Err(err) => println!("Cannot set config from QR code: {err:?}"),
}
}
"createqrsvg" => {
ensure!(!arg1.is_empty(), "Argument <qr-content> missing.");
let svg = create_qr_svg(arg1)?;
let file = dirs::home_dir().unwrap_or_default().join("qr.svg");
fs::write(&file, svg).await?;
println!("{file:#?} written.");
}
"providerinfo" => {
ensure!(!arg1.is_empty(), "Argument <addr> missing.");
let socks5_enabled = context
.get_config_bool(config::Config::Socks5Enabled)
let proxy_enabled = context
.get_config_bool(config::Config::ProxyEnabled)
.await?;
match provider::get_provider_info(&context, arg1, socks5_enabled).await {
match provider::get_provider_info(&context, arg1, proxy_enabled).await {
Some(info) => {
println!("Information for provider belonging to {arg1}:");
println!("status: {}", info.status as u32);

View File

@@ -9,10 +9,7 @@
extern crate deltachat;
use std::borrow::Cow::{self, Borrowed, Owned};
use std::io::{self, Write};
use std::process::Command;
use ansi_term::Color;
use anyhow::{bail, Error};
use deltachat::chat::ChatId;
use deltachat::config;
@@ -22,6 +19,7 @@ use deltachat::qr_code_generator::get_securejoin_qr_svg;
use deltachat::securejoin::*;
use deltachat::EventType;
use log::{error, info, warn};
use nu_ansi_term::Color;
use rustyline::completion::{Completer, FilenameCompleter, Pair};
use rustyline::error::ReadlineError;
use rustyline::highlight::{Highlighter, MatchingBracketHighlighter};
@@ -168,7 +166,7 @@ const IMEX_COMMANDS: [&str; 13] = [
"stop",
];
const DB_COMMANDS: [&str; 10] = [
const DB_COMMANDS: [&str; 11] = [
"info",
"set",
"get",
@@ -176,6 +174,7 @@ const DB_COMMANDS: [&str; 10] = [
"configure",
"connect",
"disconnect",
"fetch",
"connectivity",
"maybenetwork",
"housekeeping",
@@ -241,12 +240,13 @@ const CONTACT_COMMANDS: [&str; 9] = [
"unblock",
"listblocked",
];
const MISC_COMMANDS: [&str; 11] = [
const MISC_COMMANDS: [&str; 12] = [
"getqr",
"getqrsvg",
"getbadqr",
"checkqr",
"joinqr",
"createqrsvg",
"fileinfo",
"clear",
"exit",
@@ -417,6 +417,9 @@ async fn handle_cmd(
"disconnect" => {
ctx.stop_io().await;
}
"fetch" => {
ctx.background_fetch().await?;
}
"configure" => {
ctx.configure().await?;
}
@@ -446,12 +449,7 @@ async fn handle_cmd(
qr.replace_range(12..22, "0000000000")
}
println!("{qr}");
let output = Command::new("qrencode")
.args(["-t", "ansiutf8", qr.as_str(), "-o", "-"])
.output()
.expect("failed to execute process");
io::stdout().write_all(&output.stdout).unwrap();
io::stderr().write_all(&output.stderr).unwrap();
qr2term::print_qr(qr.as_str())?;
}
}
"getqrsvg" => {

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "deltachat-rpc-client"
version = "1.142.10"
version = "1.148.1"
description = "Python client for Delta Chat core JSON-RPC interface"
classifiers = [
"Development Status :: 5 - Production/Stable",
@@ -13,10 +13,13 @@ classifiers = [
"Operating System :: POSIX :: Linux",
"Operating System :: MacOS :: MacOS X",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Topic :: Communications :: Chat",
"Topic :: Communications :: Email"
]

View File

@@ -9,18 +9,19 @@ import io
import pathlib
import ssl
from contextlib import contextmanager
from typing import TYPE_CHECKING
from imap_tools import (
AND,
Header,
MailBox,
MailBoxTls,
MailMessage,
MailMessageFlags,
errors,
)
from . import Account, const
if TYPE_CHECKING:
from . import Account
FLAGS = b"FLAGS"
FETCH = b"FETCH"
@@ -35,28 +36,15 @@ class DirectImap:
self.connect()
def connect(self):
# Assume the testing server supports TLS on port 993.
host = self.account.get_config("configured_mail_server")
port = int(self.account.get_config("configured_mail_port"))
security = int(self.account.get_config("configured_mail_security"))
port = 993
user = self.account.get_config("addr")
host = user.rsplit("@")[-1]
pw = self.account.get_config("mail_pw")
if security == const.SocketSecurity.PLAIN:
ssl_context = None
else:
ssl_context = ssl.create_default_context()
# don't check if certificate hostname doesn't match target hostname
ssl_context.check_hostname = False
# don't check if the certificate is trusted by a certificate authority
ssl_context.verify_mode = ssl.CERT_NONE
if security == const.SocketSecurity.STARTTLS:
self.conn = MailBoxTls(host, port, ssl_context=ssl_context)
elif security == const.SocketSecurity.PLAIN or security == const.SocketSecurity.SSL:
self.conn = MailBox(host, port, ssl_context=ssl_context)
self.conn = MailBox(host, port, ssl_context=ssl.create_default_context())
self.conn.login(user, pw)
self.select_folder("INBOX")

View File

@@ -1,4 +1,5 @@
import logging
import time
import pytest
@@ -44,13 +45,6 @@ def test_qr_setup_contact_svg(acfactory) -> None:
_qr_code, svg = alice.get_qr_code_svg()
# Test that email address is in SVG
# when we have no display name.
# Check only the domain name, because
# long address may be split over multiple lines
# and not matched.
assert domain in svg
alice.set_config("displayname", "Alice")
# Test that display name is used
@@ -61,14 +55,21 @@ def test_qr_setup_contact_svg(acfactory) -> None:
@pytest.mark.parametrize("protect", [True, False])
def test_qr_securejoin(acfactory, protect):
alice, bob = acfactory.get_online_accounts(2)
def test_qr_securejoin(acfactory, protect, tmp_path):
alice, bob, fiona = acfactory.get_online_accounts(3)
logging.info("Alice creates a verified group")
alice_chat = alice.create_group("Verified group", protect=protect)
# Setup second device for Alice
# to test observing securejoin protocol.
alice.export_backup(tmp_path)
files = list(tmp_path.glob("*.tar"))
alice2 = acfactory.get_unconfigured_account()
alice2.import_backup(files[0])
logging.info("Alice creates a group")
alice_chat = alice.create_group("Group", protect=protect)
assert alice_chat.get_basic_snapshot().is_protected == protect
logging.info("Bob joins verified group")
logging.info("Bob joins the group")
qr_code = alice_chat.get_qr_code()
bob.secure_join(qr_code)
@@ -97,6 +98,21 @@ def test_qr_securejoin(acfactory, protect):
bob_contact_alice_snapshot = bob_contact_alice.get_snapshot()
assert bob_contact_alice_snapshot.is_verified
# Start second Alice device.
# Alice observes securejoin protocol and verifies Bob on second device.
alice2.start_io()
alice2.wait_for_securejoin_inviter_success()
alice2_contact_bob = alice2.get_contact_by_addr(bob.get_config("addr"))
alice2_contact_bob_snapshot = alice2_contact_bob.get_snapshot()
assert alice2_contact_bob_snapshot.is_verified
# The QR code token is synced, so alice2 must be able to handle join requests.
logging.info("Fiona joins the group via alice2")
alice.stop_io()
fiona.secure_join(qr_code)
alice2.wait_for_securejoin_inviter_success()
fiona.wait_for_securejoin_joiner_success()
def test_qr_securejoin_contact_request(acfactory) -> None:
"""Alice invites Bob to a group when Bob's chat with Alice is in a contact request mode."""
@@ -310,7 +326,6 @@ def test_verified_group_member_added_recovery(acfactory) -> None:
ac3_contact_ac2 = ac3.get_contact_by_addr(ac2.get_config("addr"))
ac3_chat.remove_contact(ac3_contact_ac2)
ac3_chat.add_contact(ac3_contact_ac2)
msg_id = ac2.wait_for_incoming_msg_event().msg_id
message = ac2.get_message_by_id(msg_id)
@@ -320,6 +335,8 @@ def test_verified_group_member_added_recovery(acfactory) -> None:
snapshot = ac1.get_message_by_id(ac1.wait_for_incoming_msg_event().msg_id).get_snapshot()
assert "removed" in snapshot.text
ac3_chat.add_contact(ac3_contact_ac2)
event = ac2.wait_for_incoming_msg_event()
msg_id = event.msg_id
chat_id = event.chat_id
@@ -443,7 +460,10 @@ def test_qr_new_group_unblocked(acfactory):
def test_aeap_flow_verified(acfactory):
"""Test that a new address is added to a contact when it changes its address."""
ac1, ac2, ac1new = acfactory.get_online_accounts(3)
ac1, ac2 = acfactory.get_online_accounts(2)
# ac1new is only used to get a new address.
ac1new = acfactory.new_preconfigured_account()
logging.info("ac1: create verified-group QR, ac2 scans and joins")
chat = ac1.create_group("hello", protect=True)
@@ -452,6 +472,7 @@ def test_aeap_flow_verified(acfactory):
logging.info("ac2: start QR-code based join-group protocol")
ac2.secure_join(qr_code)
ac1.wait_for_securejoin_inviter_success()
ac2.wait_for_securejoin_joiner_success()
logging.info("sending first message")
msg_out = chat.send_text("old address").get_snapshot()
@@ -549,6 +570,7 @@ def test_securejoin_after_contact_resetup(acfactory) -> None:
# ac1 waits for member added message and creates a QR code.
snapshot = ac1.get_message_by_id(ac1.wait_for_incoming_msg_event().msg_id).get_snapshot()
assert snapshot.text == "Member Me ({}) added by {}.".format(ac1.get_config("addr"), ac3.get_config("addr"))
ac1_qr_code = snapshot.chat.get_qr_code()
# ac2 verifies ac1
@@ -563,17 +585,29 @@ def test_securejoin_after_contact_resetup(acfactory) -> None:
# ac1 resetups the account.
ac1 = acfactory.resetup_account(ac1)
# ac1 sends a message to ac2.
ac1_contact_ac2 = ac1.create_contact(ac2.get_config("addr"), "")
ac1_chat_ac2 = ac1_contact_ac2.create_chat()
ac1_chat_ac2.send_text("Hello!")
# Loop sending message from ac1 to ac2
# until ac2 accepts new ac1 key.
#
# This may not happen immediately because resetup of ac1
# rewinds "smeared timestamp" so Date: header for messages
# sent by new ac1 are in the past compared to the last Date:
# header sent by old ac1.
while True:
# ac1 sends a message to ac2.
ac1_contact_ac2 = ac1.create_contact(ac2.get_config("addr"), "")
ac1_chat_ac2 = ac1_contact_ac2.create_chat()
ac1_chat_ac2.send_text("Hello!")
# ac2 receives a message.
snapshot = ac2.get_message_by_id(ac2.wait_for_incoming_msg_event().msg_id).get_snapshot()
assert snapshot.text == "Hello!"
# ac2 receives a message.
snapshot = ac2.get_message_by_id(ac2.wait_for_incoming_msg_event().msg_id).get_snapshot()
assert snapshot.text == "Hello!"
logging.info("ac2 received Hello!")
# ac1 is no longer verified for ac2 as new Autocrypt key is not the same as old verified key.
assert not ac2_contact_ac1.get_snapshot().is_verified
# ac1 is no longer verified for ac2 as new Autocrypt key is not the same as old verified key.
logging.info("ac2 addr={}, ac1 addr={}".format(ac2.get_config("addr"), ac1.get_config("addr")))
if not ac2_contact_ac1.get_snapshot().is_verified:
break
time.sleep(1)
# ac1 goes offline.
ac1.remove()
@@ -635,7 +669,8 @@ def test_withdraw_securejoin_qr(acfactory):
logging.info("Bob scanned withdrawn QR code")
while True:
event = alice.wait_for_event()
if event.kind == EventType.MSGS_CHANGED and event.chat_id != 0:
if (
event.kind == EventType.WARNING
and "Ignoring vg-request-with-auth message because of invalid auth code." in event.msg
):
break
snapshot = alice.get_message_by_id(event.msg_id).get_snapshot()
assert snapshot.text == "Cannot establish guaranteed end-to-end encryption with {}".format(bob.get_config("addr"))

View File

@@ -83,6 +83,26 @@ def test_configure_ip(acfactory) -> None:
account.configure()
def test_configure_alternative_port(acfactory) -> None:
"""Test that configuration with alternative port 443 works."""
account = acfactory.new_preconfigured_account()
account.set_config("mail_port", "443")
account.set_config("send_port", "443")
account.configure()
def test_configure_username(acfactory) -> None:
account = acfactory.new_preconfigured_account()
addr = account.get_config("addr")
account.set_config("mail_user", addr)
account.configure()
assert account.get_config("configured_mail_user") == addr
def test_account(acfactory) -> None:
alice, bob = acfactory.get_online_accounts(2)
@@ -413,7 +433,7 @@ def test_provider_info(rpc) -> None:
assert provider_info["id"] == "gmail"
# Disable MX record resolution.
rpc.set_config(account_id, "socks5_enabled", "1")
rpc.set_config(account_id, "proxy_enabled", "1")
provider_info = rpc.get_provider_info(account_id, "github.com")
assert provider_info is None

View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat-rpc-server"
version = "1.142.10"
version = "1.148.1"
description = "DeltaChat JSON-RPC server"
edition = "2021"
readme = "README.md"

View File

@@ -15,5 +15,5 @@
},
"type": "module",
"types": "index.d.ts",
"version": "1.142.10"
"version": "1.148.1"
}

View File

@@ -1,7 +1,6 @@
[advisories]
ignore = [
"RUSTSEC-2020-0071",
"RUSTSEC-2022-0093",
# Timing attack on RSA.
# Delta Chat does not use RSA for new keys
@@ -10,15 +9,8 @@ ignore = [
# <https://rustsec.org/advisories/RUSTSEC-2023-0071>
"RUSTSEC-2023-0071",
# Unmaintained ansi_term
"RUSTSEC-2021-0139",
# Unmaintained encoding
"RUSTSEC-2021-0153",
# Problem in curve25519-dalek 3.2.0 used by iroh 0.4.
# curve25519-dalek 4.1.3 has the problem fixed.
"RUSTSEC-2024-0344",
]
[bans]
@@ -27,28 +19,10 @@ ignore = [
# when upgrading.
# Please keep this list alphabetically sorted.
skip = [
{ name = "asn1-rs-derive", version = "0.4.0" },
{ name = "asn1-rs-impl", version = "0.1.0" },
{ name = "asn1-rs", version = "0.5.2" },
{ name = "async-channel", version = "1.9.0" },
{ name = "base16ct", version = "0.1.1" },
{ name = "base64", version = "<0.21" },
{ name = "base64", version = "0.21.7" },
{ name = "bitflags", version = "1.3.2" },
{ name = "block-buffer", version = "<0.10" },
{ name = "convert_case", version = "0.4.0" },
{ name = "curve25519-dalek", version = "3.2.0" },
{ name = "darling_core", version = "<0.14" },
{ name = "darling_macro", version = "<0.14" },
{ name = "darling", version = "<0.14" },
{ name = "der_derive", version = "0.6.1" },
{ name = "derive_more", version = "0.99.17" },
{ name = "der-parser", version = "8.2.0" },
{ name = "der", version = "0.6.1" },
{ name = "digest", version = "<0.10" },
{ name = "dlopen2", version = "0.4.1" },
{ name = "ed25519-dalek", version = "1.0.1" },
{ name = "ed25519", version = "1.5.3" },
{ name = "event-listener", version = "2.5.3" },
{ name = "event-listener", version = "4.0.3" },
{ name = "fastrand", version = "1.9.0" },
@@ -58,60 +32,30 @@ skip = [
{ name = "http-body", version = "0.4.6" },
{ name = "http", version = "0.2.12" },
{ name = "hyper", version = "0.14.28" },
{ name = "idna", version = "0.4.0" },
{ name = "netlink-packet-core", version = "0.5.0" },
{ name = "netlink-packet-route", version = "0.15.0" },
{ name = "nix", version = "0.26.4" },
{ name = "oid-registry", version = "0.6.1" },
{ name = "pem-rfc7468", version = "0.6.0" },
{ name = "pem", version = "1.1.1" },
{ name = "pkcs8", version = "0.9.0" },
{ name = "quick-error", version = "<2.0" },
{ name = "rand_chacha", version = "<0.3" },
{ name = "rand_core", version = "<0.6" },
{ name = "rand", version = "<0.8" },
{ name = "rcgen", version = "<0.12.1" },
{ name = "redox_syscall", version = "0.3.5" },
{ name = "regex-automata", version = "0.1.10" },
{ name = "regex-syntax", version = "0.6.29" },
{ name = "ring", version = "0.16.20" },
{ name = "rustls-pemfile", version = "1.0.4" },
{ name = "rustls", version = "0.21.11" },
{ name = "rustls-webpki", version = "0.101.7" },
{ name = "sec1", version = "0.3.0" },
{ name = "sha2", version = "<0.10" },
{ name = "signature", version = "1.6.4" },
{ name = "spin", version = "<0.9.6" },
{ name = "spki", version = "0.6.0" },
{ name = "ssh-encoding", version = "0.1.0" },
{ name = "ssh-key", version = "0.5.1" },
{ name = "strsim", version = "0.10.0" },
{ name = "sync_wrapper", version = "0.1.2" },
{ name = "synstructure", version = "0.12.6" },
{ name = "syn", version = "1.0.109" },
{ name = "system-configuration-sys", version = "0.5.0" },
{ name = "system-configuration", version = "0.5.1" },
{ name = "time", version = "<0.3" },
{ name = "tokio-rustls", version = "0.24.1" },
{ name = "toml_edit", version = "0.21.1" },
{ name = "untrusted", version = "0.7.1" },
{ name = "wasi", version = "<0.11" },
{ name = "webpki-roots", version ="0.25.4" },
{ name = "windows_aarch64_gnullvm", version = "<0.52" },
{ name = "windows_aarch64_msvc", version = "<0.52" },
{ name = "windows-core", version = "<0.54.0" },
{ name = "windows_i686_gnu", version = "<0.52" },
{ name = "windows_i686_msvc", version = "<0.52" },
{ name = "windows-sys", version = "<0.52" },
{ name = "windows-sys", version = "<0.59" },
{ name = "windows-targets", version = "<0.52" },
{ name = "windows", version = "0.32.0" },
{ name = "windows", version = "<0.54.0" },
{ name = "windows_x86_64_gnullvm", version = "<0.52" },
{ name = "windows_x86_64_gnu", version = "<0.52" },
{ name = "windows_x86_64_msvc", version = "<0.52" },
{ name = "winnow", version = "0.5.40" },
{ name = "winreg", version = "0.50.0" },
{ name = "x509-parser", version = "<0.16.0" },
]

24
flake.lock generated
View File

@@ -48,11 +48,11 @@
"rust-analyzer-src": "rust-analyzer-src"
},
"locked": {
"lastModified": 1714112748,
"narHash": "sha256-jq6Cpf/pQH85p+uTwPPrGG8Ky/zUOTwMJ7mcqc5M4So=",
"lastModified": 1729578683,
"narHash": "sha256-h0Wmvrkadbyi3IJXFLPi+QyYjCAKDr2xQ6dLxlQ8cXY=",
"owner": "nix-community",
"repo": "fenix",
"rev": "3ae4b908a795b6a3824d401a0702e11a7157d7e1",
"rev": "d66cda53e8193a878742dcadb5bb75f4df7c3c0a",
"type": "github"
},
"original": {
@@ -166,11 +166,11 @@
},
"nixpkgs_2": {
"locked": {
"lastModified": 1713895582,
"narHash": "sha256-cfh1hi+6muQMbi9acOlju3V1gl8BEaZBXBR9jQfQi4U=",
"lastModified": 1729256560,
"narHash": "sha256-/uilDXvCIEs3C9l73JTACm4quuHUsIHcns1c+cHUJwA=",
"owner": "nixos",
"repo": "nixpkgs",
"rev": "572af610f6151fd41c212f897c71f7056e3fb518",
"rev": "4c2fcb090b1f3e5b47eaa7bd33913b574a11e0a0",
"type": "github"
},
"original": {
@@ -195,11 +195,11 @@
},
"nixpkgs_4": {
"locked": {
"lastModified": 1714076141,
"narHash": "sha256-Drmja/f5MRHZCskS6mvzFqxEaZMeciScCTFxWVLqWEY=",
"lastModified": 1729413321,
"narHash": "sha256-I4tuhRpZFa6Fu6dcH9Dlo5LlH17peT79vx1y1SpeKt0=",
"owner": "nixos",
"repo": "nixpkgs",
"rev": "7bb2ccd8cdc44c91edba16c48d2c8f331fb3d856",
"rev": "1997e4aa514312c1af7e2bda7fad1644e778ff26",
"type": "github"
},
"original": {
@@ -222,11 +222,11 @@
"rust-analyzer-src": {
"flake": false,
"locked": {
"lastModified": 1714031783,
"narHash": "sha256-xS/niQsq1CQPOe4M4jvVPO2cnXS/EIeRG5gIopUbk+Q=",
"lastModified": 1729533545,
"narHash": "sha256-A/AuEWcGwwjpfBCZqWDNNg5GwYrJduzLvlMe+A7xG5U=",
"owner": "rust-lang",
"repo": "rust-analyzer",
"rev": "56bee2ddafa6177b19c631eedc88d43366553223",
"rev": "de2ff17bc513807412d7bbaba1d995a774938583",
"type": "github"
},
"original": {

View File

@@ -50,6 +50,7 @@ module.exports = {
DC_EVENT_IMEX_PROGRESS: 2051,
DC_EVENT_INCOMING_MSG: 2005,
DC_EVENT_INCOMING_MSG_BUNCH: 2006,
DC_EVENT_INCOMING_REACTION: 2002,
DC_EVENT_INFO: 100,
DC_EVENT_LOCATION_CHANGED: 2035,
DC_EVENT_MSGS_CHANGED: 2000,
@@ -67,6 +68,7 @@ module.exports = {
DC_EVENT_SMTP_MESSAGE_SENT: 103,
DC_EVENT_WARNING: 300,
DC_EVENT_WEBXDC_INSTANCE_DELETED: 2121,
DC_EVENT_WEBXDC_REALTIME_ADVERTISEMENT: 2151,
DC_EVENT_WEBXDC_REALTIME_DATA: 2150,
DC_EVENT_WEBXDC_STATUS_UPDATE: 2120,
DC_GCL_ADD_ALLDONE_HINT: 4,
@@ -134,6 +136,7 @@ module.exports = {
DC_QR_FPR_OK: 210,
DC_QR_FPR_WITHOUT_ADDR: 230,
DC_QR_LOGIN: 520,
DC_QR_PROXY: 271,
DC_QR_REVIVE_VERIFYCONTACT: 510,
DC_QR_REVIVE_VERIFYGROUP: 512,
DC_QR_TEXT: 330,

View File

@@ -16,6 +16,7 @@ module.exports = {
410: 'DC_EVENT_ERROR_SELF_NOT_IN_GROUP',
2000: 'DC_EVENT_MSGS_CHANGED',
2001: 'DC_EVENT_REACTIONS_CHANGED',
2002: 'DC_EVENT_INCOMING_REACTION',
2005: 'DC_EVENT_INCOMING_MSG',
2006: 'DC_EVENT_INCOMING_MSG_BUNCH',
2008: 'DC_EVENT_MSGS_NOTICED',
@@ -38,6 +39,7 @@ module.exports = {
2120: 'DC_EVENT_WEBXDC_STATUS_UPDATE',
2121: 'DC_EVENT_WEBXDC_INSTANCE_DELETED',
2150: 'DC_EVENT_WEBXDC_REALTIME_DATA',
2151: 'DC_EVENT_WEBXDC_REALTIME_ADVERTISEMENT',
2200: 'DC_EVENT_ACCOUNTS_BACKGROUND_FETCH_DONE',
2300: 'DC_EVENT_CHATLIST_CHANGED',
2301: 'DC_EVENT_CHATLIST_ITEM_CHANGED',

View File

@@ -50,6 +50,7 @@ export enum C {
DC_EVENT_IMEX_PROGRESS = 2051,
DC_EVENT_INCOMING_MSG = 2005,
DC_EVENT_INCOMING_MSG_BUNCH = 2006,
DC_EVENT_INCOMING_REACTION = 2002,
DC_EVENT_INFO = 100,
DC_EVENT_LOCATION_CHANGED = 2035,
DC_EVENT_MSGS_CHANGED = 2000,
@@ -67,6 +68,7 @@ export enum C {
DC_EVENT_SMTP_MESSAGE_SENT = 103,
DC_EVENT_WARNING = 300,
DC_EVENT_WEBXDC_INSTANCE_DELETED = 2121,
DC_EVENT_WEBXDC_REALTIME_ADVERTISEMENT = 2151,
DC_EVENT_WEBXDC_REALTIME_DATA = 2150,
DC_EVENT_WEBXDC_STATUS_UPDATE = 2120,
DC_GCL_ADD_ALLDONE_HINT = 4,
@@ -134,6 +136,7 @@ export enum C {
DC_QR_FPR_OK = 210,
DC_QR_FPR_WITHOUT_ADDR = 230,
DC_QR_LOGIN = 520,
DC_QR_PROXY = 271,
DC_QR_REVIVE_VERIFYCONTACT = 510,
DC_QR_REVIVE_VERIFYGROUP = 512,
DC_QR_TEXT = 330,
@@ -320,6 +323,7 @@ export const EventId2EventName: { [key: number]: string } = {
410: 'DC_EVENT_ERROR_SELF_NOT_IN_GROUP',
2000: 'DC_EVENT_MSGS_CHANGED',
2001: 'DC_EVENT_REACTIONS_CHANGED',
2002: 'DC_EVENT_INCOMING_REACTION',
2005: 'DC_EVENT_INCOMING_MSG',
2006: 'DC_EVENT_INCOMING_MSG_BUNCH',
2008: 'DC_EVENT_MSGS_NOTICED',
@@ -342,6 +346,7 @@ export const EventId2EventName: { [key: number]: string } = {
2120: 'DC_EVENT_WEBXDC_STATUS_UPDATE',
2121: 'DC_EVENT_WEBXDC_INSTANCE_DELETED',
2150: 'DC_EVENT_WEBXDC_REALTIME_DATA',
2151: 'DC_EVENT_WEBXDC_REALTIME_ADVERTISEMENT',
2200: 'DC_EVENT_ACCOUNTS_BACKGROUND_FETCH_DONE',
2300: 'DC_EVENT_CHATLIST_CHANGED',
2301: 'DC_EVENT_CHATLIST_ITEM_CHANGED',

View File

@@ -475,47 +475,6 @@ export class Context extends EventEmitter {
return binding.dcn_get_msg_html(this.dcn_context, Number(messageId))
}
getNextMediaMessage(
messageId: number,
msgType1: number,
msgType2: number,
msgType3: number
) {
debug(
`getNextMediaMessage ${messageId} ${msgType1} ${msgType2} ${msgType3}`
)
return this._getNextMedia(messageId, 1, msgType1, msgType2, msgType3)
}
getPreviousMediaMessage(
messageId: number,
msgType1: number,
msgType2: number,
msgType3: number
) {
debug(
`getPreviousMediaMessage ${messageId} ${msgType1} ${msgType2} ${msgType3}`
)
return this._getNextMedia(messageId, -1, msgType1, msgType2, msgType3)
}
_getNextMedia(
messageId: number,
dir: number,
msgType1: number,
msgType2: number,
msgType3: number
): number {
return binding.dcn_get_next_media(
this.dcn_context,
Number(messageId),
dir,
msgType1 || 0,
msgType2 || 0,
msgType3 || 0
)
}
getSecurejoinQrCode(chatId: number): string {
debug(`getSecurejoinQrCode ${chatId}`)
return binding.dcn_get_securejoin_qr(this.dcn_context, Number(chatId))

View File

@@ -1053,27 +1053,6 @@ NAPI_METHOD(dcn_get_msg_html) {
NAPI_RETURN_AND_UNREF_STRING(msg_html);
}
NAPI_METHOD(dcn_get_next_media) {
NAPI_ARGV(6);
NAPI_DCN_CONTEXT();
NAPI_ARGV_UINT32(msg_id, 1);
NAPI_ARGV_INT32(dir, 2);
NAPI_ARGV_INT32(msg_type1, 3);
NAPI_ARGV_INT32(msg_type2, 4);
NAPI_ARGV_INT32(msg_type3, 5);
//TRACE("calling..");
uint32_t next_id = dc_get_next_media(dcn_context->dc_context,
msg_id,
dir,
msg_type1,
msg_type2,
msg_type3);
//TRACE("result %d", next_id);
NAPI_RETURN_UINT32(next_id);
}
NAPI_METHOD(dcn_set_chat_visibility) {
NAPI_ARGV(3);
NAPI_DCN_CONTEXT();
@@ -3443,7 +3422,6 @@ NAPI_INIT() {
NAPI_EXPORT_FUNCTION(dcn_get_msg_cnt);
NAPI_EXPORT_FUNCTION(dcn_get_msg_info);
NAPI_EXPORT_FUNCTION(dcn_get_msg_html);
NAPI_EXPORT_FUNCTION(dcn_get_next_media);
NAPI_EXPORT_FUNCTION(dcn_set_chat_visibility);
NAPI_EXPORT_FUNCTION(dcn_get_securejoin_qr);
NAPI_EXPORT_FUNCTION(dcn_get_securejoin_qr_svg);

View File

@@ -271,7 +271,7 @@ describe('Basic offline Tests', function () {
'sync_msgs',
'sentbox_watch',
'show_emails',
'socks5_enabled',
'proxy_enabled',
'sqlite_version',
'uptime',
'used_account_settings',

View File

@@ -55,5 +55,5 @@
"test:mocha": "mocha node/test/test.mjs --growl --reporter=spec --bail --exit"
},
"types": "node/dist/index.d.ts",
"version": "1.142.10"
"version": "1.148.1"
}

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "deltachat"
version = "1.142.10"
version = "1.148.1"
description = "Python bindings for the Delta Chat Core library using CFFI against the Rust-implemented libdeltachat"
readme = "README.rst"
requires-python = ">=3.7"

View File

@@ -194,15 +194,13 @@ class Account:
assert res != ffi.NULL, f"config value not found for: {name!r}"
return from_dc_charpointer(res)
def _preconfigure_keypair(self, addr: str, secret: str) -> None:
def _preconfigure_keypair(self, secret: str) -> None:
"""See dc_preconfigure_keypair() in deltachat.h.
In other words, you don't need this.
"""
res = lib.dc_preconfigure_keypair(
self._dc_context,
as_dc_charpointer(addr),
ffi.NULL,
as_dc_charpointer(secret),
)
if res == 0:

View File

@@ -308,7 +308,7 @@ class Chat:
msg = as_dc_charpointer(text)
msg_id = lib.dc_send_text_msg(self.account._dc_context, self.id, msg)
if msg_id == 0:
raise ValueError("message could not be send, does chat exist?")
raise ValueError("The message could not be sent. Does the chat exist?")
return Message.from_db(self.account, msg_id)
def send_file(self, path, mime_type="application/octet-stream"):

View File

@@ -8,19 +8,19 @@ import io
import pathlib
import ssl
from contextlib import contextmanager
from typing import List
from typing import List, TYPE_CHECKING
from imap_tools import (
AND,
Header,
MailBox,
MailBoxTls,
MailMessage,
MailMessageFlags,
errors,
)
from deltachat import Account, const
if TYPE_CHECKING:
from deltachat import Account
FLAGS = b"FLAGS"
FETCH = b"FETCH"
@@ -28,7 +28,7 @@ ALL = "1:*"
class DirectImap:
def __init__(self, account: Account) -> None:
def __init__(self, account: "Account") -> None:
self.account = account
self.logid = account.get_config("displayname") or id(account)
self._idling = False
@@ -36,27 +36,13 @@ class DirectImap:
def connect(self):
host = self.account.get_config("configured_mail_server")
port = int(self.account.get_config("configured_mail_port"))
security = int(self.account.get_config("configured_mail_security"))
port = 993
user = self.account.get_config("addr")
host = user.rsplit("@")[-1]
pw = self.account.get_config("mail_pw")
if security == const.DC_SOCKET_PLAIN:
ssl_context = None
else:
ssl_context = ssl.create_default_context()
# don't check if certificate hostname doesn't match target hostname
ssl_context.check_hostname = False
# don't check if the certificate is trusted by a certificate authority
ssl_context.verify_mode = ssl.CERT_NONE
if security == const.DC_SOCKET_STARTTLS:
self.conn = MailBoxTls(host, port, ssl_context=ssl_context)
elif security == const.DC_SOCKET_PLAIN or security == const.DC_SOCKET_SSL:
self.conn = MailBox(host, port, ssl_context=ssl_context)
self.conn = MailBox(host, port, ssl_context=ssl.create_default_context())
self.conn.login(user, pw)
self.select_folder("INBOX")

View File

@@ -462,7 +462,7 @@ class ACFactory:
def remove_preconfigured_keys(self) -> None:
self._preconfigured_keys = []
def _preconfigure_key(self, account, addr):
def _preconfigure_key(self, account):
# Only set a preconfigured key if we haven't used it yet for another account.
try:
keyname = self._preconfigured_keys.pop(0)
@@ -471,9 +471,9 @@ class ACFactory:
else:
fname_sec = self.data.read_path(f"key/{keyname}-secret.asc")
if fname_sec:
account._preconfigure_keypair(addr, fname_sec)
account._preconfigure_keypair(fname_sec)
return True
print(f"WARN: could not use preconfigured keys for {addr!r}")
print("WARN: could not use preconfigured keys")
def get_pseudo_configured_account(self, passphrase: Optional[str] = None) -> Account:
# do a pseudo-configured account
@@ -492,7 +492,7 @@ class ACFactory:
"configured": "1",
},
)
self._preconfigure_key(ac, addr)
self._preconfigure_key(ac)
self._acsetup.init_logging(ac)
return ac
@@ -525,9 +525,10 @@ class ACFactory:
configdict.setdefault("mvbox_move", False)
configdict.setdefault("sentbox_watch", False)
configdict.setdefault("sync_msgs", False)
configdict.setdefault("delete_server_after", 0)
ac.update_config(configdict)
self._acsetup._account2config[ac] = configdict
self._preconfigure_key(ac, configdict["addr"])
self._preconfigure_key(ac)
return ac
def wait_configured(self, account) -> None:

View File

@@ -488,10 +488,18 @@ def test_move_sync_msgs(acfactory):
ac1 = acfactory.new_online_configuring_account(bcc_self=True, sync_msgs=True, fix_is_chatmail=True)
acfactory.bring_accounts_online()
ac1.direct_imap.select_folder("DeltaChat")
# Sync messages may also be sent during the configuration.
mvbox_msg_cnt = len(ac1.direct_imap.get_all_messages())
ac1.set_config("displayname", "Alice")
ac1._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
ac1._evtracker.get_matching("DC_EVENT_MSG_DELIVERED")
ac1.set_config("displayname", "Bob")
ac1._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
ac1._evtracker.get_matching("DC_EVENT_MSG_DELIVERED")
ac1.direct_imap.select_folder("Inbox")
assert len(ac1.direct_imap.get_all_messages()) == 0
ac1.direct_imap.select_folder("DeltaChat")
assert len(ac1.direct_imap.get_all_messages()) == mvbox_msg_cnt + 2
def test_forward_messages(acfactory, lp):
@@ -620,7 +628,7 @@ def test_long_group_name(acfactory, lp):
def test_send_self_message(acfactory, lp):
ac1 = acfactory.new_online_configuring_account(mvbox_move=True)
ac1 = acfactory.new_online_configuring_account(mvbox_move=True, bcc_self=True)
acfactory.bring_accounts_online()
lp.sec("ac1: create self chat")
chat = ac1.get_self_contact().create_chat()
@@ -2076,12 +2084,11 @@ def test_send_receive_locations(acfactory, lp):
def test_immediate_autodelete(acfactory, lp):
ac1 = acfactory.new_online_configuring_account()
ac2 = acfactory.new_online_configuring_account()
acfactory.bring_accounts_online()
# "1" means delete immediately, while "0" means do not delete
ac2.set_config("delete_server_after", "1")
acfactory.bring_accounts_online()
lp.sec("ac1: create chat with ac2")
chat1 = ac1.create_chat(ac2)
ac2.create_chat(ac1)

View File

@@ -67,7 +67,7 @@ class TestOfflineAccountBasic:
ac = acfactory.get_unconfigured_account()
alice_secret = data.read_path("key/alice-secret.asc")
assert alice_secret
ac._preconfigure_keypair("alice@example.org", alice_secret)
ac._preconfigure_keypair(alice_secret)
def test_getinfo(self, acfactory):
ac1 = acfactory.get_unconfigured_account()

View File

@@ -1 +1 @@
2024-08-26
2024-10-23

View File

@@ -7,7 +7,7 @@ set -euo pipefail
#
# Avoid using rustup here as it depends on reading /proc/self/exe and
# has problems running under QEMU.
RUST_VERSION=1.80.1
RUST_VERSION=1.82.0
ARCH="$(uname -m)"
test -f "/lib/libc.musl-$ARCH.so.1" && LIBC=musl || LIBC=gnu

View File

@@ -31,6 +31,6 @@ unset CHATMAIL_DOMAIN
# Try to build wheels for a range of interpreters, but don't fail if they are not available.
# E.g. musllinux_1_1 does not have PyPy interpreters as of 2022-07-10
tox --workdir "$TOXWORKDIR" -e py37,py38,py39,py310,py311,py312,pypy37,pypy38,pypy39,pypy310 --skip-missing-interpreters true
tox --workdir "$TOXWORKDIR" -e py37,py38,py39,py310,py311,py312,py313,pypy37,pypy38,pypy39,pypy310 --skip-missing-interpreters true
auditwheel repair "$TOXWORKDIR"/wheelhouse/deltachat* -w "$TOXWORKDIR/wheelhouse"

View File

@@ -6,7 +6,7 @@ set -euo pipefail
export TZ=UTC
# Provider database revision.
REV=05c1b2029da74718e4bdc3799a46e29c4f794dc7
REV=77cbf92a8565fdf1bcaba10fa93c1455c750a1e9
CORE_ROOT="$PWD"
TMP="$(mktemp -d)"

View File

@@ -253,16 +253,16 @@ impl<'a> BlobObject<'a> {
///
/// The extension part will always be lowercased.
fn sanitise_name(name: &str) -> (String, String) {
let mut name = name.to_string();
let mut name = name;
for part in name.rsplit('/') {
if !part.is_empty() {
name = part.to_string();
name = part;
break;
}
}
for part in name.rsplit('\\') {
if !part.is_empty() {
name = part.to_string();
name = part;
break;
}
}
@@ -272,32 +272,39 @@ impl<'a> BlobObject<'a> {
replacement: "",
};
let clean = sanitize_filename::sanitize_with_options(name, opts);
// Let's take the tricky filename
let name = sanitize_filename::sanitize_with_options(name, opts);
// Let's take a tricky filename,
// "file.with_lots_of_characters_behind_point_and_double_ending.tar.gz" as an example.
// Split it into "file" and "with_lots_of_characters_behind_point_and_double_ending.tar.gz":
let mut iter = clean.splitn(2, '.');
let stem: String = iter.next().unwrap_or_default().chars().take(64).collect();
// stem == "file"
let ext_chars = iter.next().unwrap_or_default().chars();
let ext: String = ext_chars
// Assume that the extension is 32 chars maximum.
let ext: String = name
.chars()
.rev()
.take(32)
.take_while(|c| !c.is_whitespace())
.take(33)
.collect::<Vec<_>>()
.iter()
.rev()
.collect();
// ext == "d_point_and_double_ending.tar.gz"
// ext == "nd_point_and_double_ending.tar.gz"
if ext.is_empty() {
(stem, "".to_string())
// Split it into "nd_point_and_double_ending" and "tar.gz":
let mut iter = ext.splitn(2, '.');
iter.next();
let ext = iter.next().unwrap_or_default();
let ext = if ext.is_empty() {
String::new()
} else {
(stem, format!(".{ext}").to_lowercase())
// Return ("file", ".d_point_and_double_ending.tar.gz")
// which is not perfect but acceptable.
}
format!(".{ext}")
// ".tar.gz"
};
let stem = name
.strip_suffix(&ext)
.unwrap_or_default()
.chars()
.take(64)
.collect();
(stem, ext.to_lowercase())
}
/// Checks whether a name is a valid blob name.
@@ -615,7 +622,7 @@ fn exif_orientation(exif: &exif::Exif, context: &Context) -> i32 {
0
}
impl<'a> fmt::Display for BlobObject<'a> {
impl fmt::Display for BlobObject<'_> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "$BLOBDIR/{}", self.name)
}
@@ -666,10 +673,6 @@ impl<'a> BlobDirContents<'a> {
pub(crate) fn iter(&self) -> BlobDirIter<'_> {
BlobDirIter::new(self.context, self.inner.iter())
}
pub(crate) fn len(&self) -> usize {
self.inner.len()
}
}
/// A iterator over all the [`BlobObject`]s in the blobdir.
@@ -967,6 +970,19 @@ mod tests {
assert!(!stem.contains(':'));
assert!(!stem.contains('*'));
assert!(!stem.contains('?'));
let (stem, ext) = BlobObject::sanitise_name(
"file.with_lots_of_characters_behind_point_and_double_ending.tar.gz",
);
assert_eq!(
stem,
"file.with_lots_of_characters_behind_point_and_double_ending"
);
assert_eq!(ext, ".tar.gz");
let (stem, ext) = BlobObject::sanitise_name("a. tar.tar.gz");
assert_eq!(stem, "a. tar");
assert_eq!(ext, ".tar.gz");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]

View File

@@ -46,8 +46,8 @@ use crate::stock_str;
use crate::sync::{self, Sync::*, SyncData};
use crate::tools::{
buf_compress, create_id, create_outgoing_rfc724_mid, create_smeared_timestamp,
create_smeared_timestamps, get_abs_path, gm2local_offset, smeared_time, time, IsNoneOrEmpty,
SystemTime,
create_smeared_timestamps, get_abs_path, gm2local_offset, smeared_time, time,
truncate_msg_text, IsNoneOrEmpty, SystemTime,
};
use crate::webxdc::StatusUpdateSerial;
@@ -279,9 +279,10 @@ impl ChatId {
) -> Result<Self> {
let chat_id = match ChatIdBlocked::lookup_by_contact(context, contact_id).await? {
Some(chat) => {
if create_blocked == Blocked::Not && chat.blocked != Blocked::Not {
chat.id.set_blocked(context, Blocked::Not).await?;
if create_blocked != Blocked::Not || chat.blocked == Blocked::Not {
return Ok(chat.id);
}
chat.id.set_blocked(context, Blocked::Not).await?;
chat.id
}
None => {
@@ -577,7 +578,7 @@ impl ChatId {
Ok(())
}
/// Sets protection and sends or adds a message.
/// Sets protection and adds a message.
///
/// `timestamp_sort` is used as the timestamp of the added message
/// and should be the timestamp of the change happening.
@@ -588,20 +589,16 @@ impl ChatId {
timestamp_sort: i64,
contact_id: Option<ContactId>,
) -> Result<()> {
match self.inner_set_protection(context, protect).await {
Ok(protection_status_modified) => {
if protection_status_modified {
self.add_protection_msg(context, protect, contact_id, timestamp_sort)
.await?;
chatlist_events::emit_chatlist_item_changed(context, self);
}
Ok(())
}
Err(e) => {
error!(context, "Cannot set protection: {e:#}."); // make error user-visible
Err(e)
}
let protection_status_modified = self
.inner_set_protection(context, protect)
.await
.with_context(|| format!("Cannot set protection for {self}"))?;
if protection_status_modified {
self.add_protection_msg(context, protect, contact_id, timestamp_sort)
.await?;
chatlist_events::emit_chatlist_item_changed(context, self);
}
Ok(())
}
/// Sets protection and sends or adds a message.
@@ -615,8 +612,9 @@ impl ChatId {
contact_id: Option<ContactId>,
) -> Result<()> {
let sort_to_bottom = true;
let (received, incoming) = (false, false);
let ts = self
.calc_sort_timestamp(context, timestamp_sent, sort_to_bottom, false)
.calc_sort_timestamp(context, timestamp_sent, sort_to_bottom, received, incoming)
.await?
// Always sort protection messages below `SystemMessage::SecurejoinWait{,Timeout}` ones
// in case of race conditions.
@@ -868,13 +866,14 @@ impl ChatId {
///
/// Returns `true`, if message was deleted, `false` otherwise.
async fn maybe_delete_draft(self, context: &Context) -> Result<bool> {
match self.get_draft_msg_id(context).await? {
Some(msg_id) => {
msg_id.delete_from_db(context).await?;
Ok(true)
}
None => Ok(false),
}
Ok(context
.sql
.execute(
"DELETE FROM msgs WHERE chat_id=? AND state=?",
(self, MessageState::OutDraft),
)
.await?
> 0)
}
/// Set provided message as draft message for specified chat.
@@ -946,12 +945,18 @@ impl ChatId {
}
}
// insert new draft
self.maybe_delete_draft(context).await?;
let row_id = context
.sql
.insert(
"INSERT INTO msgs (
.transaction(|transaction| {
// Delete existing draft if it exists.
transaction.execute(
"DELETE FROM msgs WHERE chat_id=? AND state=?",
(self, MessageState::OutDraft),
)?;
// Insert new draft.
transaction.execute(
"INSERT INTO msgs (
chat_id,
from_id,
timestamp,
@@ -963,19 +968,22 @@ impl ChatId {
hidden,
mime_in_reply_to)
VALUES (?,?,?,?,?,?,?,?,?,?);",
(
self,
ContactId::SELF,
time(),
msg.viewtype,
MessageState::OutDraft,
&msg.text,
message::normalize_text(&msg.text),
msg.param.to_string(),
1,
msg.in_reply_to.as_deref().unwrap_or_default(),
),
)
(
self,
ContactId::SELF,
time(),
msg.viewtype,
MessageState::OutDraft,
&msg.text,
message::normalize_text(&msg.text),
msg.param.to_string(),
1,
msg.in_reply_to.as_deref().unwrap_or_default(),
),
)?;
Ok(transaction.last_insert_rowid())
})
.await?;
msg.id = MsgId::new(row_id.try_into()?);
Ok(true)
@@ -1041,7 +1049,13 @@ impl ChatId {
pub(crate) async fn get_timestamp(self, context: &Context) -> Result<Option<i64>> {
let timestamp = context
.sql
.query_get_value("SELECT MAX(timestamp) FROM msgs WHERE chat_id=?", (self,))
.query_get_value(
"SELECT MAX(timestamp)
FROM msgs
WHERE chat_id=?
HAVING COUNT(*) > 0",
(self,),
)
.await?;
Ok(timestamp)
}
@@ -1227,6 +1241,7 @@ impl ChatId {
AND ((state BETWEEN {} AND {}) OR (state >= {})) \
AND NOT hidden \
AND download_state={} \
AND from_id != {} \
ORDER BY timestamp DESC, id DESC \
LIMIT 1;",
MessageState::InFresh as u32,
@@ -1235,6 +1250,9 @@ impl ChatId {
// Do not reply to not fully downloaded messages. Such a message could be a group chat
// message that we assigned to 1:1 chat.
DownloadState::Done as u32,
// Do not reference info messages, they are not actually sent out
// and have Message-IDs unknown to other chat members.
ContactId::INFO.to_u32(),
);
sql.query_row_optional(&query, (self,), f).await
}
@@ -1246,7 +1264,7 @@ impl ChatId {
) -> Result<Option<(String, String, String)>> {
self.parent_query(
context,
"rfc724_mid, mime_in_reply_to, mime_references",
"rfc724_mid, mime_in_reply_to, IFNULL(mime_references, '')",
state_out_min,
|row: &rusqlite::Row| {
let rfc724_mid: String = row.get(0)?;
@@ -1381,12 +1399,14 @@ impl ChatId {
/// corresponding event in case of a system message (usually the current system time).
/// `always_sort_to_bottom` makes this ajust the returned timestamp up so that the message goes
/// to the chat bottom.
/// `received` -- whether the message is received. Otherwise being sent.
/// `incoming` -- whether the message is incoming.
pub(crate) async fn calc_sort_timestamp(
self,
context: &Context,
message_timestamp: i64,
always_sort_to_bottom: bool,
received: bool,
incoming: bool,
) -> Result<i64> {
let mut sort_timestamp = cmp::min(message_timestamp, smeared_time(context));
@@ -1400,26 +1420,45 @@ impl ChatId {
context
.sql
.query_get_value(
"SELECT MAX(timestamp) FROM msgs WHERE chat_id=? AND state!=?",
"SELECT MAX(timestamp)
FROM msgs
WHERE chat_id=? AND state!=?
HAVING COUNT(*) > 0",
(self, MessageState::OutDraft),
)
.await?
} else if incoming {
// get newest non fresh message for this chat.
// If a user hasn't been online for some time, the Inbox is fetched first and then the
// Sentbox. In order for Inbox and Sent messages to be allowed to mingle, outgoing
// messages are purely sorted by their sent timestamp. NB: The Inbox must be fetched
// first otherwise Inbox messages would be always below old Sentbox messages. We could
// take in the query below only incoming messages, but then new incoming messages would
// mingle with just sent outgoing ones and apear somewhere in the middle of the chat.
} else if received {
// Received messages shouldn't mingle with just sent ones and appear somewhere in the
// middle of the chat, so we go after the newest non fresh message.
//
// But if a received outgoing message is older than some seen message, better sort the
// received message purely by timestamp. We could place it just before that seen
// message, but anyway the user may not notice it.
//
// NB: Received outgoing messages may break sorting of fresh incoming ones, but this
// shouldn't happen frequently. Seen incoming messages don't really break sorting of
// fresh ones, they rather mean that older incoming messages are actually seen as well.
context
.sql
.query_get_value(
"SELECT MAX(timestamp) FROM msgs WHERE chat_id=? AND hidden=0 AND state>?",
(self, MessageState::InFresh),
.query_row_optional(
"SELECT MAX(timestamp), MAX(IIF(state=?,timestamp_sent,0))
FROM msgs
WHERE chat_id=? AND hidden=0 AND state>?
HAVING COUNT(*) > 0",
(MessageState::InSeen, self, MessageState::InFresh),
|row| {
let ts: i64 = row.get(0)?;
let ts_sent_seen: i64 = row.get(1)?;
Ok((ts, ts_sent_seen))
},
)
.await?
.and_then(|(ts, ts_sent_seen)| {
match incoming || ts_sent_seen <= message_timestamp {
true => Some(ts),
false => None,
}
})
} else {
None
};
@@ -1934,11 +1973,13 @@ impl Chat {
msg.param.set_int(Param::AttachGroupImage, 1);
self.param.remove(Param::Unpromoted);
self.update_param(context).await?;
// send_sync_msg() is called (usually) a moment later at send_msg_to_smtp()
// when the group creation message is actually sent through SMTP --
// this makes sure, the other devices are aware of grpid that is used in the sync-message.
// TODO: Remove this compat code needed because Core <= v1.143:
// - doesn't accept synchronization of QR code tokens for unpromoted groups, so we also
// send them when the group is promoted.
// - doesn't sync QR code tokens for unpromoted groups and the group might be created
// before an upgrade.
context
.sync_qr_code_tokens(Some(self.id))
.sync_qr_code_tokens(Some(self.grpid.as_str()))
.await
.log_err(context)
.ok();
@@ -2071,6 +2112,8 @@ impl Chat {
msg.from_id = ContactId::SELF;
msg.rfc724_mid = new_rfc724_mid;
msg.timestamp_sort = timestamp;
let (msg_text, was_truncated) = truncate_msg_text(context, msg.text.clone()).await?;
let mime_modified = new_mime_headers.is_some() | was_truncated;
// add message to the database
if let Some(update_msg_id) = update_msg_id {
@@ -2092,14 +2135,14 @@ impl Chat {
msg.timestamp_sort,
msg.viewtype,
msg.state,
msg.text,
message::normalize_text(&msg.text),
msg_text,
message::normalize_text(&msg_text),
&msg.subject,
msg.param.to_string(),
msg.hidden,
msg.in_reply_to.as_deref().unwrap_or_default(),
new_references,
new_mime_headers.is_some(),
mime_modified,
new_mime_headers.unwrap_or_default(),
location_id as i32,
ephemeral_timer,
@@ -2143,14 +2186,14 @@ impl Chat {
msg.timestamp_sort,
msg.viewtype,
msg.state,
msg.text,
message::normalize_text(&msg.text),
msg_text,
message::normalize_text(&msg_text),
&msg.subject,
msg.param.to_string(),
msg.hidden,
msg.in_reply_to.as_deref().unwrap_or_default(),
new_references,
new_mime_headers.is_some(),
mime_modified,
new_mime_headers.unwrap_or_default(),
location_id as i32,
ephemeral_timer,
@@ -2240,7 +2283,7 @@ pub(crate) async fn sync(context: &Context, id: SyncId, action: SyncAction) -> R
context
.add_sync_item(SyncData::AlterChat { id, action })
.await?;
context.scheduler.interrupt_smtp().await;
context.scheduler.interrupt_inbox().await;
Ok(())
}
@@ -2898,13 +2941,15 @@ async fn prepare_send_msg(
);
message::update_msg_state(context, msg.id, MessageState::OutPending).await?;
}
create_send_msg_jobs(context, msg).await
let row_ids = create_send_msg_jobs(context, msg)
.await
.context("Failed to create send jobs")?;
Ok(row_ids)
}
/// Constructs jobs for sending a message and inserts them into the `smtp` table.
/// Constructs jobs for sending a message and inserts them into the appropriate table.
///
/// Returns row ids if jobs were created or an empty `Vec` otherwise, e.g. when sending to a
/// group with only self and no BCC-to-self configured.
/// Returns row ids if `smtp` table jobs were created or an empty `Vec` otherwise.
///
/// The caller has to interrupt SMTP loop or otherwise process new rows.
pub(crate) async fn create_send_msg_jobs(context: &Context, msg: &mut Message) -> Result<Vec<i64>> {
@@ -2998,12 +3043,6 @@ pub(crate) async fn create_send_msg_jobs(context: &Context, msg: &mut Message) -
}
}
if let Some(sync_ids) = rendered_msg.sync_ids_to_delete {
if let Err(err) = context.delete_sync_ids(sync_ids).await {
error!(context, "Failed to delete sync ids: {err:#}.");
}
}
if attach_selfavatar {
if let Err(err) = msg.chat_id.set_selfavatar_timestamp(context, now).await {
error!(context, "Failed to set selfavatar timestamp: {err:#}.");
@@ -3020,19 +3059,30 @@ pub(crate) async fn create_send_msg_jobs(context: &Context, msg: &mut Message) -
let chunk_size = context.get_max_smtp_rcpt_to().await?;
let trans_fn = |t: &mut rusqlite::Transaction| {
let mut row_ids = Vec::<i64>::new();
for recipients_chunk in recipients.chunks(chunk_size) {
let recipients_chunk = recipients_chunk.join(" ");
let row_id = t.execute(
"INSERT INTO smtp (rfc724_mid, recipients, mime, msg_id) \
VALUES (?1, ?2, ?3, ?4)",
(
&rendered_msg.rfc724_mid,
recipients_chunk,
&rendered_msg.message,
msg.id,
),
if let Some(sync_ids) = rendered_msg.sync_ids_to_delete {
t.execute(
&format!("DELETE FROM multi_device_sync WHERE id IN ({sync_ids})"),
(),
)?;
row_ids.push(row_id.try_into()?);
t.execute(
"INSERT INTO imap_send (mime, msg_id) VALUES (?, ?)",
(&rendered_msg.message, msg.id),
)?;
} else {
for recipients_chunk in recipients.chunks(chunk_size) {
let recipients_chunk = recipients_chunk.join(" ");
let row_id = t.execute(
"INSERT INTO smtp (rfc724_mid, recipients, mime, msg_id) \
VALUES (?1, ?2, ?3, ?4)",
(
&rendered_msg.rfc724_mid,
recipients_chunk,
&rendered_msg.message,
msg.id,
),
)?;
row_ids.push(row_id.try_into()?);
}
}
Ok(row_ids)
};
@@ -3407,65 +3457,6 @@ pub async fn get_chat_media(
Ok(list)
}
/// Indicates the direction over which to iterate.
#[derive(Debug, Clone, PartialEq, Eq)]
#[repr(i32)]
pub enum Direction {
/// Search forward.
Forward = 1,
/// Search backward.
Backward = -1,
}
/// Searches next/previous message based on the given message and list of types.
///
/// Deprecated since 2023-10-03.
#[deprecated(note = "use `get_chat_media` instead")]
pub async fn get_next_media(
context: &Context,
curr_msg_id: MsgId,
direction: Direction,
msg_type: Viewtype,
msg_type2: Viewtype,
msg_type3: Viewtype,
) -> Result<Option<MsgId>> {
let mut ret: Option<MsgId> = None;
if let Ok(msg) = Message::load_from_db(context, curr_msg_id).await {
let list: Vec<MsgId> = get_chat_media(
context,
Some(msg.chat_id),
if msg_type != Viewtype::Unknown {
msg_type
} else {
msg.viewtype
},
msg_type2,
msg_type3,
)
.await?;
for (i, msg_id) in list.iter().enumerate() {
if curr_msg_id == *msg_id {
match direction {
Direction::Forward => {
if i + 1 < list.len() {
ret = list.get(i + 1).copied();
}
}
Direction::Backward => {
if i >= 1 {
ret = list.get(i - 1).copied();
}
}
}
break;
}
}
}
Ok(ret)
}
/// Returns a vector of contact IDs for given chat ID.
pub async fn get_chat_contacts(context: &Context, chat_id: ChatId) -> Result<Vec<ContactId>> {
// Normal chats do not include SELF. Group chats do (as it may happen that one is deleted from a
@@ -3726,17 +3717,13 @@ pub(crate) async fn add_contact_to_chat_ex(
bail!("can not add contact because the account is not part of the group/broadcast");
}
let sync_qr_code_tokens;
if from_handshake && chat.param.get_int(Param::Unpromoted).unwrap_or_default() == 1 {
chat.param.remove(Param::Unpromoted);
chat.update_param(context).await?;
if context
.sync_qr_code_tokens(Some(chat_id))
.await
.log_err(context)
.is_ok()
{
context.scheduler.interrupt_smtp().await;
}
sync_qr_code_tokens = true;
} else {
sync_qr_code_tokens = false;
}
if context.is_self_addr(contact.get_addr()).await? {
@@ -3780,6 +3767,20 @@ pub(crate) async fn add_contact_to_chat_ex(
return Err(e);
}
sync = Nosync;
// TODO: Remove this compat code needed because Core <= v1.143:
// - doesn't accept synchronization of QR code tokens for unpromoted groups, so we also send
// them when the group is promoted.
// - doesn't sync QR code tokens for unpromoted groups and the group might be created before
// an upgrade.
if sync_qr_code_tokens
&& context
.sync_qr_code_tokens(Some(chat.grpid.as_str()))
.await
.log_err(context)
.is_ok()
{
context.scheduler.interrupt_inbox().await;
}
}
context.emit_event(EventType::ChatModified(chat_id));
if sync.into() {
@@ -4254,10 +4255,14 @@ pub async fn resend_msgs(context: &Context, msg_ids: &[MsgId]) -> Result<()> {
msg.update_param(context).await?;
}
match msg.get_state() {
MessageState::OutFailed | MessageState::OutDelivered | MessageState::OutMdnRcvd => {
// `get_state()` may return an outdated `OutPending`, so update anyway.
MessageState::OutPending
| MessageState::OutFailed
| MessageState::OutDelivered
| MessageState::OutMdnRcvd => {
message::update_msg_state(context, msg.id, MessageState::OutPending).await?
}
_ => bail!("unexpected message state"),
msg_state => bail!("Unexpected message state {msg_state}"),
}
context.emit_event(EventType::MsgsChanged {
chat_id: msg.chat_id,
@@ -4377,7 +4382,10 @@ pub async fn add_device_msg_with_importance(
if let Some(last_msg_time) = context
.sql
.query_get_value(
"SELECT MAX(timestamp) FROM msgs WHERE chat_id=?",
"SELECT MAX(timestamp)
FROM msgs
WHERE chat_id=?
HAVING COUNT(*) > 0",
(chat_id,),
)
.await?
@@ -4703,6 +4711,7 @@ mod tests {
use super::*;
use crate::chatlist::get_archived_cnt;
use crate::constants::{DC_GCL_ARCHIVED_ONLY, DC_GCL_NO_SPECIALS};
use crate::headerdef::HeaderDef;
use crate::message::delete_msgs;
use crate::receive_imf::receive_imf;
use crate::test_utils::{sync, TestContext, TestContextManager};
@@ -4858,6 +4867,37 @@ mod tests {
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_only_one_draft_per_chat() -> Result<()> {
let t = TestContext::new_alice().await;
let chat_id = create_group_chat(&t, ProtectionStatus::Unprotected, "abc").await?;
let msgs: Vec<message::Message> = (1..=1000)
.map(|i| {
let mut msg = Message::new(Viewtype::Text);
msg.set_text(i.to_string());
msg
})
.collect();
let mut tasks = Vec::new();
for mut msg in msgs {
let ctx = t.clone();
let task = tokio::spawn(async move {
let ctx = ctx;
chat_id.set_draft(&ctx, Some(&mut msg)).await
});
tasks.push(task);
}
futures::future::join_all(tasks.into_iter()).await;
assert!(chat_id.get_draft(&t).await?.is_some());
chat_id.set_draft(&t, None).await?;
assert!(chat_id.get_draft(&t).await?.is_none());
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_change_quotes_on_reused_message_object() -> Result<()> {
let t = TestContext::new_alice().await;
@@ -6269,11 +6309,10 @@ mod tests {
// Alice has an SMTP-server replacing the `Message-ID:`-header (as done eg. by outlook.com).
let sent_msg = alice.pop_sent_msg().await;
let msg = sent_msg.payload();
assert_eq!(msg.match_indices("Message-ID: <Mr.").count(), 2);
assert_eq!(msg.match_indices("References: <Mr.").count(), 1);
let msg = msg.replace("Message-ID: <Mr.", "Message-ID: <XXX");
assert_eq!(msg.match_indices("Message-ID: <Mr.").count(), 0);
assert_eq!(msg.match_indices("References: <Mr.").count(), 1);
assert_eq!(msg.match_indices("Message-ID: <").count(), 2);
assert_eq!(msg.match_indices("References: <").count(), 1);
let msg = msg.replace("Message-ID: <", "Message-ID: <X.X");
assert_eq!(msg.match_indices("References: <").count(), 1);
// Bob receives this message, he may detect group by `References:`- or `Chat-Group:`-header
receive_imf(&bob, msg.as_bytes(), false).await.unwrap();
@@ -6290,7 +6329,7 @@ mod tests {
send_text_msg(&bob, bob_chat.id, "ho!".to_string()).await?;
let sent_msg = bob.pop_sent_msg().await;
let msg = sent_msg.payload();
let msg = msg.replace("Message-ID: <Mr.", "Message-ID: <XXX");
let msg = msg.replace("Message-ID: <", "Message-ID: <X.X");
let msg = msg.replace("Chat-", "XXXX-");
assert_eq!(msg.match_indices("Chat-").count(), 0);
@@ -6840,8 +6879,29 @@ mod tests {
)
.await?;
let sent2 = alice.pop_sent_msg().await;
resend_msgs(&alice, &[sent1.sender_msg_id]).await?;
let resent_msg_id = sent1.sender_msg_id;
resend_msgs(&alice, &[resent_msg_id]).await?;
assert_eq!(
resent_msg_id.get_state(&alice).await?,
MessageState::OutPending
);
resend_msgs(&alice, &[resent_msg_id]).await?;
// Message can be re-sent multiple times.
assert_eq!(
resent_msg_id.get_state(&alice).await?,
MessageState::OutPending
);
alice.pop_sent_msg().await;
// There's still one more pending SMTP job.
assert_eq!(
resent_msg_id.get_state(&alice).await?,
MessageState::OutPending
);
let sent3 = alice.pop_sent_msg().await;
assert_eq!(
resent_msg_id.get_state(&alice).await?,
MessageState::OutDelivered
);
// Bob receives all messages
let bob = TestContext::new_bob().await;
@@ -7638,4 +7698,29 @@ mod tests {
Ok(())
}
/// Tests that info message is ignored when constructing `In-Reply-To`.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_info_not_referenced() -> Result<()> {
let mut tcm = TestContextManager::new();
let alice = &tcm.alice().await;
let bob = &tcm.bob().await;
let bob_received_message = tcm.send_recv_accept(alice, bob, "Hi!").await;
let bob_chat_id = bob_received_message.chat_id;
add_info_msg(bob, bob_chat_id, "Some info", create_smeared_timestamp(bob)).await?;
// Bob sends a message.
// This message should reference Alice's "Hi!" message and not the info message.
let sent = bob.send_text(bob_chat_id, "Hi hi!").await;
let mime_message = alice.parse_msg(&sent).await;
let in_reply_to = mime_message.get_header(HeaderDef::InReplyTo).unwrap();
assert_eq!(
in_reply_to,
format!("<{}>", bob_received_message.rfc724_mid)
);
Ok(())
}
}

View File

@@ -91,21 +91,44 @@ pub enum Config {
/// Should not be extended in the future, create new config keys instead.
ServerFlags,
/// True if proxy is enabled.
///
/// Can be used to disable proxy without erasing known URLs.
ProxyEnabled,
/// Proxy URL.
///
/// Supported URLs schemes are `http://` (HTTP), `https://` (HTTPS),
/// `socks5://` (SOCKS5) and `ss://` (Shadowsocks).
///
/// May contain multiple URLs separated by newline, in which case the first one is used.
ProxyUrl,
/// True if SOCKS5 is enabled.
///
/// Can be used to disable SOCKS5 without erasing SOCKS5 configuration.
///
/// Deprecated in favor of `ProxyEnabled`.
Socks5Enabled,
/// SOCKS5 proxy server hostname or address.
///
/// Deprecated in favor of `ProxyUrl`.
Socks5Host,
/// SOCKS5 proxy server port.
///
/// Deprecated in favor of `ProxyUrl`.
Socks5Port,
/// SOCKS5 proxy server username.
///
/// Deprecated in favor of `ProxyUrl`.
Socks5User,
/// SOCKS5 proxy server password.
///
/// Deprecated in favor of `ProxyUrl`.
Socks5Password,
/// Own name to use in the `From:` field when sending messages.
@@ -129,6 +152,7 @@ pub enum Config {
/// True if Message Delivery Notifications (read receipts) should
/// be sent and requested.
#[strum(props(default = "1"))]
MdnsEnabled,
/// True if "Sent" folder should be watched for changes.
@@ -173,12 +197,12 @@ pub enum Config {
/// Timer in seconds after which the message is deleted from the
/// server.
///
/// Equals to 0 by default, which means the message is never
/// deleted.
/// 0 means messages are never deleted by Delta Chat.
///
/// Value 1 is treated as "delete at once": messages are deleted
/// immediately, without moving to DeltaChat folder.
#[strum(props(default = "0"))]
///
/// Default is 1 for chatmail accounts before a backup export, 0 otherwise.
DeleteServerAfter,
/// Timer in seconds after which the message is deleted from the
@@ -199,21 +223,32 @@ pub enum Config {
/// The primary email address. Also see `SecondaryAddrs`.
ConfiguredAddr,
/// List of configured IMAP servers as a JSON array.
ConfiguredImapServers,
/// Configured IMAP server hostname.
///
/// This is replaced by `configured_imap_servers` for new configurations.
ConfiguredMailServer,
/// Configured IMAP server port.
///
/// This is replaced by `configured_imap_servers` for new configurations.
ConfiguredMailPort,
/// Configured IMAP server security (e.g. TLS, STARTTLS).
///
/// This is replaced by `configured_imap_servers` for new configurations.
ConfiguredMailSecurity,
/// Configured IMAP server username.
///
/// This is set if user has configured username manually.
ConfiguredMailUser,
/// Configured IMAP server password.
ConfiguredMailPw,
/// Configured IMAP server port.
ConfiguredMailPort,
/// Configured IMAP server security (e.g. TLS, STARTTLS).
ConfiguredMailSecurity,
/// Configured TLS certificate checks.
/// This option is saved on successful configuration
/// and should not be modified manually.
@@ -222,18 +257,32 @@ pub enum Config {
/// but has "IMAP" in the name for backwards compatibility.
ConfiguredImapCertificateChecks,
/// List of configured SMTP servers as a JSON array.
ConfiguredSmtpServers,
/// Configured SMTP server hostname.
///
/// This is replaced by `configured_smtp_servers` for new configurations.
ConfiguredSendServer,
/// Configured SMTP server port.
///
/// This is replaced by `configured_smtp_servers` for new configurations.
ConfiguredSendPort,
/// Configured SMTP server security (e.g. TLS, STARTTLS).
///
/// This is replaced by `configured_smtp_servers` for new configurations.
ConfiguredSendSecurity,
/// Configured SMTP server username.
///
/// This is set if user has configured username manually.
ConfiguredSendUser,
/// Configured SMTP server password.
ConfiguredSendPw,
/// Configured SMTP server port.
ConfiguredSendPort,
/// Deprecated, stored for backwards compatibility.
///
/// ConfiguredImapCertificateChecks is actually used.
@@ -242,9 +291,6 @@ pub enum Config {
/// Whether OAuth 2 is used with configured provider.
ConfiguredServerFlags,
/// Configured SMTP server security (e.g. TLS, STARTTLS).
ConfiguredSendSecurity,
/// Configured folder for incoming messages.
ConfiguredInboxFolder,
@@ -275,6 +321,10 @@ pub enum Config {
/// True if account is muted.
IsMuted,
/// Optional tag as "Work", "Family".
/// Meant to help profile owner to differ between profiles with similar names.
PrivateTag,
/// All secondary self addresses separated by spaces
/// (`addr1@example.org addr2@example.org addr3@example.org`)
SecondaryAddrs,
@@ -415,13 +465,16 @@ impl Config {
}
impl Context {
/// Returns true if configuration value is set for the given key.
pub async fn config_exists(&self, key: Config) -> Result<bool> {
/// Returns true if configuration value is set in the db for the given key.
///
/// NB: Don't use this to check if the key is configured because this doesn't look into
/// environment. The proper use of this function is e.g. checking a key before setting it.
pub(crate) async fn config_exists(&self, key: Config) -> Result<bool> {
Ok(self.sql.get_raw_config(key.as_ref()).await?.is_some())
}
/// Get a configuration key. Returns `None` if no value is set, and no default value found.
pub async fn get_config(&self, key: Config) -> Result<Option<String>> {
/// Get a config key value. Returns `None` if no value is set.
pub(crate) async fn get_config_opt(&self, key: Config) -> Result<Option<String>> {
let env_key = format!("DELTACHAT_{}", key.as_ref().to_uppercase());
if let Ok(value) = env::var(env_key) {
return Ok(Some(value));
@@ -441,19 +494,38 @@ impl Context {
Config::SysConfigKeys => Some(get_config_keys_string()),
_ => self.sql.get_raw_config(key.as_ref()).await?,
};
Ok(value)
}
/// Get a config key value if set, or a default value. Returns `None` if no value exists.
pub async fn get_config(&self, key: Config) -> Result<Option<String>> {
let value = self.get_config_opt(key).await?;
if value.is_some() {
return Ok(value);
}
// Default values
match key {
Config::ConfiguredInboxFolder => Ok(Some("INBOX".to_owned())),
_ => Ok(key.get_str("default").map(|s| s.to_string())),
}
let val = match key {
Config::ConfiguredInboxFolder => Some("INBOX"),
Config::DeleteServerAfter => match Box::pin(self.is_chatmail()).await? {
false => Some("0"),
true => Some("1"),
},
_ => key.get_str("default"),
};
Ok(val.map(|s| s.to_string()))
}
/// Returns Some(T) if a value for the given key exists and was successfully parsed.
/// Returns Some(T) if a value for the given key is set and was successfully parsed.
/// Returns None if could not parse.
pub(crate) async fn get_config_opt_parsed<T: FromStr>(&self, key: Config) -> Result<Option<T>> {
self.get_config_opt(key)
.await
.map(|s: Option<String>| s.and_then(|s| s.parse().ok()))
}
/// Returns Some(T) if a value for the given key exists (incl. default value) and was
/// successfully parsed.
/// Returns None if could not parse.
pub async fn get_config_parsed<T: FromStr>(&self, key: Config) -> Result<Option<T>> {
self.get_config(key)
@@ -481,14 +553,21 @@ impl Context {
Ok(self.get_config_parsed(key).await?.unwrap_or_default())
}
/// Returns boolean configuration value (if any) for the given key.
pub async fn get_config_bool_opt(&self, key: Config) -> Result<Option<bool>> {
Ok(self.get_config_parsed::<i32>(key).await?.map(|x| x != 0))
/// Returns boolean configuration value (if set) for the given key.
pub(crate) async fn get_config_bool_opt(&self, key: Config) -> Result<Option<bool>> {
Ok(self
.get_config_opt_parsed::<i32>(key)
.await?
.map(|x| x != 0))
}
/// Returns boolean configuration value for the given key.
pub async fn get_config_bool(&self, key: Config) -> Result<bool> {
Ok(self.get_config_bool_opt(key).await?.unwrap_or_default())
Ok(self
.get_config_parsed::<i32>(key)
.await?
.map(|x| x != 0)
.unwrap_or_default())
}
/// Returns true if movebox ("DeltaChat" folder) should be watched.
@@ -514,6 +593,12 @@ impl Context {
&& !self.get_config_bool(Config::Bot).await?)
}
/// Returns whether sync messages should be uploaded to the mvbox.
pub(crate) async fn should_move_sync_msgs(&self) -> Result<bool> {
Ok(self.get_config_bool(Config::MvboxMove).await?
|| !self.get_config_bool(Config::IsChatmail).await?)
}
/// Returns whether MDNs should be requested.
pub(crate) async fn should_request_mdns(&self) -> Result<bool> {
match self.get_config_bool_opt(Config::MdnsEnabled).await? {
@@ -524,10 +609,7 @@ impl Context {
/// Returns whether MDNs should be sent.
pub(crate) async fn should_send_mdns(&self) -> Result<bool> {
Ok(self
.get_config_bool_opt(Config::MdnsEnabled)
.await?
.unwrap_or(true))
self.get_config_bool(Config::MdnsEnabled).await
}
/// Gets configured "delete_server_after" value.
@@ -535,11 +617,16 @@ impl Context {
/// `None` means never delete the message, `Some(0)` means delete
/// at once, `Some(x)` means delete after `x` seconds.
pub async fn get_config_delete_server_after(&self) -> Result<Option<i64>> {
match self.get_config_int(Config::DeleteServerAfter).await? {
0 => Ok(None),
1 => Ok(Some(0)),
x => Ok(Some(i64::from(x))),
}
let val = match self
.get_config_parsed::<i64>(Config::DeleteServerAfter)
.await?
.unwrap_or(0)
{
0 => None,
1 => Some(0),
x => Some(x),
};
Ok(val)
}
/// Gets the configured provider, as saved in the `configured_provider` value.
@@ -584,6 +671,7 @@ impl Context {
fn check_config(key: Config, value: Option<&str>) -> Result<()> {
match key {
Config::Socks5Enabled
| Config::ProxyEnabled
| Config::BccSelf
| Config::E2eeEnabled
| Config::MdnsEnabled
@@ -714,7 +802,7 @@ impl Context {
{
return Ok(());
}
self.scheduler.interrupt_smtp().await;
self.scheduler.interrupt_inbox().await;
Ok(())
}
@@ -773,6 +861,8 @@ impl Context {
///
/// This should only be used by test code and during configure.
pub(crate) async fn set_primary_self_addr(&self, primary_new: &str) -> Result<()> {
self.quota.write().await.take();
// add old primary address (if exists) to secondary addresses
let mut secondary_addrs = self.get_all_self_addrs().await?;
// never store a primary address also as a secondary
@@ -785,7 +875,7 @@ impl Context {
self.set_config_internal(Config::ConfiguredAddr, Some(primary_new))
.await?;
self.emit_event(EventType::ConnectivityChanged);
Ok(())
}
@@ -984,9 +1074,15 @@ mod tests {
let t = &TestContext::new_alice().await;
assert!(t.should_request_mdns().await?);
assert!(t.should_send_mdns().await?);
assert!(t.get_config_bool_opt(Config::MdnsEnabled).await?.is_none());
// The setting should be displayed correctly.
assert!(t.get_config_bool(Config::MdnsEnabled).await?);
t.set_config_bool(Config::Bot, true).await?;
assert!(!t.should_request_mdns().await?);
assert!(t.should_send_mdns().await?);
assert!(t.get_config_bool_opt(Config::MdnsEnabled).await?.is_none());
assert!(t.get_config_bool(Config::MdnsEnabled).await?);
Ok(())
}
@@ -1091,7 +1187,7 @@ mod tests {
let status = "Synced via usual message";
alice0.set_config(Config::Selfstatus, Some(status)).await?;
alice0.send_sync_msg().await?;
alice0.pop_sent_msg().await;
alice0.pop_sent_sync_msg().await;
let status1 = "Synced via sync message";
alice1.set_config(Config::Selfstatus, Some(status1)).await?;
tcm.send_recv(alice0, alice1, "hi Alice!").await;
@@ -1115,7 +1211,7 @@ mod tests {
.set_config(Config::Selfavatar, Some(file.to_str().unwrap()))
.await?;
alice0.send_sync_msg().await?;
alice0.pop_sent_msg().await;
alice0.pop_sent_sync_msg().await;
let file = alice1.dir.path().join("avatar.jpg");
let bytes = include_bytes!("../test-data/image/avatar1000x1000.jpg");
tokio::fs::write(&file, bytes).await?;

View File

@@ -11,9 +11,9 @@
mod auto_mozilla;
mod auto_outlook;
mod server_params;
pub(crate) mod server_params;
use anyhow::{bail, ensure, Context as _, Result};
use anyhow::{bail, ensure, format_err, Context as _, Result};
use auto_mozilla::moz_autoconfigure;
use auto_outlook::outlk_autodiscover;
use deltachat_contact_tools::EmailAddress;
@@ -25,14 +25,16 @@ use tokio::task;
use crate::config::{self, Config};
use crate::context::Context;
use crate::imap::{session::Session as ImapSession, Imap};
use crate::imap::Imap;
use crate::log::LogExt;
use crate::login_param::{CertificateChecks, LoginParam, ServerLoginParam};
use crate::login_param::{
ConfiguredCertificateChecks, ConfiguredLoginParam, ConfiguredServerLoginParam,
ConnectionCandidate, EnteredCertificateChecks, EnteredLoginParam,
};
use crate::message::{Message, Viewtype};
use crate::oauth2::get_oauth2_addr;
use crate::provider::{Protocol, Socket, UsernamePattern};
use crate::smtp::Smtp;
use crate::socks::Socks5Config;
use crate::stock_str;
use crate::sync::Sync::*;
use crate::tools::time;
@@ -78,10 +80,7 @@ impl Context {
let res = self
.inner_configure()
.race(cancel_channel.recv().map(|_| {
progress!(self, 0);
Ok(())
}))
.race(cancel_channel.recv().map(|_| Err(format_err!("Cancelled"))))
.await;
self.free_ongoing().await;
@@ -110,16 +109,15 @@ impl Context {
async fn inner_configure(&self) -> Result<()> {
info!(self, "Configure ...");
let mut param = LoginParam::load_candidate_params(self).await?;
let param = EnteredLoginParam::load(self).await?;
let old_addr = self.get_config(Config::ConfiguredAddr).await?;
let success = configure(self, &mut param).await;
let configured_param_res = configure(self, &param).await;
self.set_config_internal(Config::NotifyAboutWrongPw, None)
.await?;
on_configure_completed(self, param, old_addr).await?;
on_configure_completed(self, configured_param_res?, old_addr).await?;
success?;
self.set_config_internal(Config::NotifyAboutWrongPw, Some("1"))
.await?;
Ok(())
@@ -128,7 +126,7 @@ impl Context {
async fn on_configure_completed(
context: &Context,
param: LoginParam,
param: ConfiguredLoginParam,
old_addr: Option<String>,
) -> Result<()> {
if let Some(provider) = param.provider {
@@ -178,19 +176,28 @@ async fn on_configure_completed(
Ok(())
}
async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
progress!(ctx, 1);
/// Retrieves data from autoconfig and provider database
/// to transform user-entered login parameters into complete configuration.
async fn get_configured_param(
ctx: &Context,
param: &EnteredLoginParam,
) -> Result<ConfiguredLoginParam> {
ensure!(!param.addr.is_empty(), "Missing email address.");
let socks5_config = param.socks5_config.clone();
let socks5_enabled = socks5_config.is_some();
ensure!(!param.imap.password.is_empty(), "Missing (IMAP) password.");
let ctx2 = ctx.clone();
let update_device_chats_handle = task::spawn(async move { ctx2.update_device_chats().await });
// SMTP password is an "advanced" setting. If unset, use the same password as for IMAP.
let smtp_password = if param.smtp.password.is_empty() {
param.imap.password.clone()
} else {
param.smtp.password.clone()
};
// Step 1: Load the parameters and check email-address and password
let proxy_config = param.proxy_config.clone();
let proxy_enabled = proxy_config.is_some();
// OAuth is always set either for both IMAP and SMTP or not at all.
if param.imap.oauth2 {
let mut addr = param.addr.clone();
if param.oauth2 {
// the used oauth2 addr may differ, check this.
// if get_oauth2_addr() is not available in the oauth2 implementation, just use the given one.
progress!(ctx, 10);
@@ -199,7 +206,7 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
.and_then(|e| e.parse().ok())
{
info!(ctx, "Authorized address is {}", oauth2_addr);
param.addr = oauth2_addr;
addr = oauth2_addr;
ctx.sql
.set_raw_config("addr", Some(param.addr.as_str()))
.await?;
@@ -211,9 +218,9 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
let parsed = EmailAddress::new(&param.addr).context("Bad email-address")?;
let param_domain = parsed.domain;
// Step 2: Autoconfig
progress!(ctx, 200);
let provider;
let param_autoconfig;
if param.imap.server.is_empty()
&& param.imap.port == 0
@@ -225,77 +232,51 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
&& param.smtp.user.is_empty()
{
// no advanced parameters entered by the user: query provider-database or do Autoconfig
info!(
ctx,
"checking internal provider-info for offline autoconfig"
);
if let Some(provider) =
provider::get_provider_info(ctx, &param_domain, socks5_enabled).await
{
param.provider = Some(provider);
match provider.status {
provider::Status::Ok | provider::Status::Preparation => {
if provider.server.is_empty() {
info!(ctx, "offline autoconfig found, but no servers defined");
param_autoconfig = None;
} else {
info!(ctx, "offline autoconfig found");
let servers = provider
.server
.iter()
.map(|s| ServerParams {
protocol: s.protocol,
socket: s.socket,
hostname: s.hostname.to_string(),
port: s.port,
username: match s.username_pattern {
UsernamePattern::Email => param.addr.to_string(),
UsernamePattern::Emaillocalpart => {
if let Some(at) = param.addr.find('@') {
param.addr.split_at(at).0.to_string()
} else {
param.addr.to_string()
}
}
},
})
.collect();
provider = provider::get_provider_info(ctx, &param_domain, proxy_enabled).await;
if let Some(provider) = provider {
if provider.server.is_empty() {
info!(ctx, "Offline autoconfig found, but no servers defined.");
param_autoconfig = None;
} else {
info!(ctx, "Offline autoconfig found.");
let servers = provider
.server
.iter()
.map(|s| ServerParams {
protocol: s.protocol,
socket: s.socket,
hostname: s.hostname.to_string(),
port: s.port,
username: match s.username_pattern {
UsernamePattern::Email => param.addr.to_string(),
UsernamePattern::Emaillocalpart => {
if let Some(at) = param.addr.find('@') {
param.addr.split_at(at).0.to_string()
} else {
param.addr.to_string()
}
}
},
})
.collect();
param_autoconfig = Some(servers)
}
}
provider::Status::Broken => {
info!(ctx, "offline autoconfig found, provider is broken");
param_autoconfig = None;
}
param_autoconfig = Some(servers)
}
} else {
// Try receiving autoconfig
info!(ctx, "no offline autoconfig found");
info!(ctx, "No offline autoconfig found.");
param_autoconfig = get_autoconfig(ctx, param, &param_domain).await;
}
} else {
provider = None;
param_autoconfig = None;
}
let user_strict_tls = match param.certificate_checks {
CertificateChecks::Automatic => None,
CertificateChecks::Strict => Some(true),
CertificateChecks::AcceptInvalidCertificates
| CertificateChecks::AcceptInvalidCertificates2 => Some(false),
};
let provider_strict_tls = param.provider.map(|provider| provider.opt.strict_tls);
let strict_tls = user_strict_tls.or(provider_strict_tls).unwrap_or(true);
// Do not save `CertificateChecks::Automatic` into `configured_imap_certificate_checks`.
param.certificate_checks = if strict_tls {
CertificateChecks::Strict
} else {
CertificateChecks::AcceptInvalidCertificates
};
progress!(ctx, 500);
let mut servers = param_autoconfig.unwrap_or_default();
@@ -326,107 +307,125 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
let servers = expand_param_vector(servers, &param.addr, &param_domain);
let configured_login_param = ConfiguredLoginParam {
addr,
imap: servers
.iter()
.filter_map(|params| {
let Ok(security) = params.socket.try_into() else {
return None;
};
if params.protocol == Protocol::Imap {
Some(ConfiguredServerLoginParam {
connection: ConnectionCandidate {
host: params.hostname.clone(),
port: params.port,
security,
},
user: params.username.clone(),
})
} else {
None
}
})
.collect(),
imap_user: param.imap.user.clone(),
imap_password: param.imap.password.clone(),
smtp: servers
.iter()
.filter_map(|params| {
let Ok(security) = params.socket.try_into() else {
return None;
};
if params.protocol == Protocol::Smtp {
Some(ConfiguredServerLoginParam {
connection: ConnectionCandidate {
host: params.hostname.clone(),
port: params.port,
security,
},
user: params.username.clone(),
})
} else {
None
}
})
.collect(),
smtp_user: param.smtp.user.clone(),
smtp_password,
proxy_config: param.proxy_config.clone(),
provider,
certificate_checks: match param.certificate_checks {
EnteredCertificateChecks::Automatic => ConfiguredCertificateChecks::Automatic,
EnteredCertificateChecks::Strict => ConfiguredCertificateChecks::Strict,
EnteredCertificateChecks::AcceptInvalidCertificates
| EnteredCertificateChecks::AcceptInvalidCertificates2 => {
ConfiguredCertificateChecks::AcceptInvalidCertificates
}
},
oauth2: param.oauth2,
};
Ok(configured_login_param)
}
async fn configure(ctx: &Context, param: &EnteredLoginParam) -> Result<ConfiguredLoginParam> {
progress!(ctx, 1);
let ctx2 = ctx.clone();
let update_device_chats_handle = task::spawn(async move { ctx2.update_device_chats().await });
let configured_param = get_configured_param(ctx, param).await?;
let strict_tls = configured_param.strict_tls();
progress!(ctx, 550);
// Spawn SMTP configuration task
let mut smtp = Smtp::new();
// to try SMTP while connecting to IMAP.
let context_smtp = ctx.clone();
let mut smtp_param = param.smtp.clone();
let smtp_addr = param.addr.clone();
let smtp_servers: Vec<ServerParams> = servers
.iter()
.filter(|params| params.protocol == Protocol::Smtp)
.cloned()
.collect();
let smtp_param = configured_param.smtp.clone();
let smtp_password = configured_param.smtp_password.clone();
let smtp_addr = configured_param.addr.clone();
let proxy_config = configured_param.proxy_config.clone();
let smtp_config_task = task::spawn(async move {
let mut smtp_configured = false;
let mut errors = Vec::new();
for smtp_server in smtp_servers {
smtp_param.user.clone_from(&smtp_server.username);
smtp_param.server.clone_from(&smtp_server.hostname);
smtp_param.port = smtp_server.port;
smtp_param.security = smtp_server.socket;
let mut smtp = Smtp::new();
smtp.connect(
&context_smtp,
&smtp_param,
&smtp_password,
&proxy_config,
&smtp_addr,
strict_tls,
configured_param.oauth2,
)
.await?;
match try_smtp_one_param(
&context_smtp,
&smtp_param,
&socks5_config,
&smtp_addr,
strict_tls,
&mut smtp,
)
.await
{
Ok(_) => {
smtp_configured = true;
break;
}
Err(e) => errors.push(e),
}
}
if smtp_configured {
Ok(smtp_param)
} else {
Err(errors)
}
Ok::<(), anyhow::Error>(())
});
progress!(ctx, 600);
// Configure IMAP
let mut imap: Option<(Imap, ImapSession)> = None;
let imap_servers: Vec<&ServerParams> = servers
.iter()
.filter(|params| params.protocol == Protocol::Imap)
.collect();
let imap_servers_count = imap_servers.len();
let mut errors = Vec::new();
for (imap_server_index, imap_server) in imap_servers.into_iter().enumerate() {
param.imap.user.clone_from(&imap_server.username);
param.imap.server.clone_from(&imap_server.hostname);
param.imap.port = imap_server.port;
param.imap.security = imap_server.socket;
match try_imap_one_param(
ctx,
&param.imap,
&param.socks5_config,
&param.addr,
strict_tls,
)
.await
{
Ok(configured_imap) => {
imap = Some(configured_imap);
break;
}
Err(e) => errors.push(e),
}
progress!(
ctx,
600 + (800 - 600) * (1 + imap_server_index) / imap_servers_count
);
}
let (mut imap, mut imap_session) = match imap {
Some(imap) => imap,
None => bail!(nicer_configuration_error(ctx, errors).await),
let (_s, r) = async_channel::bounded(1);
let mut imap = Imap::new(
configured_param.imap.clone(),
configured_param.imap_password.clone(),
configured_param.proxy_config.clone(),
&configured_param.addr,
strict_tls,
configured_param.oauth2,
r,
);
let mut imap_session = match imap.connect(ctx).await {
Ok(session) => session,
Err(err) => bail!("{}", nicer_configuration_error(ctx, err.to_string()).await),
};
progress!(ctx, 850);
// Wait for SMTP configuration
match smtp_config_task.await.unwrap() {
Ok(smtp_param) => {
param.smtp = smtp_param;
}
Err(errors) => {
bail!(nicer_configuration_error(ctx, errors).await);
}
}
smtp_config_task.await.unwrap()?;
progress!(ctx, 900);
@@ -474,8 +473,7 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
}
}
// the trailing underscore is correct
param.save_as_configured_params(ctx).await?;
configured_param.save_as_configured_params(ctx).await?;
ctx.set_config_internal(Config::ConfiguredTimestamp, Some(&time().to_string()))
.await?;
@@ -493,7 +491,7 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
ctx.sql.set_raw_config_bool("configured", true).await?;
Ok(())
Ok(configured_param)
}
/// Retrieve available autoconfigurations.
@@ -502,7 +500,7 @@ async fn configure(ctx: &Context, param: &mut LoginParam) -> Result<()> {
/// B. If we have no configuration yet, search configuration in Thunderbird's central database
async fn get_autoconfig(
ctx: &Context,
param: &LoginParam,
param: &EnteredLoginParam,
param_domain: &str,
) -> Option<Vec<ServerParams>> {
let param_addr_urlencoded = utf8_percent_encode(&param.addr, NON_ALPHANUMERIC).to_string();
@@ -573,140 +571,19 @@ async fn get_autoconfig(
None
}
async fn try_imap_one_param(
context: &Context,
param: &ServerLoginParam,
socks5_config: &Option<Socks5Config>,
addr: &str,
strict_tls: bool,
) -> Result<(Imap, ImapSession), ConfigurationError> {
let inf = format!(
"imap: {}@{}:{} security={} strict_tls={} oauth2={} socks5_config={}",
param.user,
param.server,
param.port,
param.security,
strict_tls,
param.oauth2,
if let Some(socks5_config) = socks5_config {
socks5_config.to_string()
} else {
"None".to_string()
}
);
info!(context, "Trying: {}", inf);
let (_s, r) = async_channel::bounded(1);
let mut imap = match Imap::new(param, socks5_config.clone(), addr, strict_tls, r) {
Err(err) => {
info!(context, "failure: {:#}", err);
return Err(ConfigurationError {
config: inf,
msg: format!("{err:#}"),
});
}
Ok(imap) => imap,
};
match imap.connect(context).await {
Err(err) => {
info!(context, "IMAP failure: {err:#}.");
Err(ConfigurationError {
config: inf,
msg: format!("{err:#}"),
})
}
Ok(session) => {
info!(context, "IMAP success: {inf}.");
Ok((imap, session))
}
}
}
async fn try_smtp_one_param(
context: &Context,
param: &ServerLoginParam,
socks5_config: &Option<Socks5Config>,
addr: &str,
strict_tls: bool,
smtp: &mut Smtp,
) -> Result<(), ConfigurationError> {
let inf = format!(
"smtp: {}@{}:{} security={} strict_tls={} oauth2={} socks5_config={}",
param.user,
param.server,
param.port,
param.security,
strict_tls,
param.oauth2,
if let Some(socks5_config) = socks5_config {
socks5_config.to_string()
} else {
"None".to_string()
}
);
info!(context, "Trying: {}", inf);
if let Err(err) = smtp
.connect(context, param, socks5_config, addr, strict_tls)
.await
async fn nicer_configuration_error(context: &Context, e: String) -> String {
if e.to_lowercase().contains("could not resolve")
|| e.to_lowercase().contains("connection attempts")
|| e.to_lowercase()
.contains("temporary failure in name resolution")
|| e.to_lowercase().contains("name or service not known")
|| e.to_lowercase()
.contains("failed to lookup address information")
{
info!(context, "SMTP failure: {err:#}.");
Err(ConfigurationError {
config: inf,
msg: format!("{err:#}"),
})
} else {
info!(context, "SMTP success: {inf}.");
smtp.disconnect();
Ok(())
}
}
/// Failure to connect and login with email client configuration.
#[derive(Debug, thiserror::Error)]
#[error("Trying {config}…\nError: {msg}")]
pub struct ConfigurationError {
/// Tried configuration description.
config: String,
/// Error message.
msg: String,
}
async fn nicer_configuration_error(context: &Context, errors: Vec<ConfigurationError>) -> String {
let first_err = if let Some(f) = errors.first() {
f
} else {
// This means configuration failed but no errors have been captured. This should never
// happen, but if it does, the user will see classic "Error: no error".
return "no error".to_string();
};
if errors.iter().all(|e| {
e.msg.to_lowercase().contains("could not resolve")
|| e.msg.to_lowercase().contains("no dns resolution results")
|| e.msg
.to_lowercase()
.contains("temporary failure in name resolution")
|| e.msg.to_lowercase().contains("name or service not known")
|| e.msg
.to_lowercase()
.contains("failed to lookup address information")
}) {
return stock_str::error_no_network(context).await;
}
if errors.iter().all(|e| e.msg == first_err.msg) {
return first_err.msg.to_string();
}
errors
.iter()
.map(|e| e.to_string())
.collect::<Vec<String>>()
.join("\n\n")
e
}
#[derive(Debug, thiserror::Error)]
@@ -732,7 +609,9 @@ pub enum Error {
mod tests {
#![allow(clippy::indexing_slicing)]
use super::*;
use crate::config::Config;
use crate::login_param::EnteredServerLoginParam;
use crate::test_utils::TestContext;
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -744,4 +623,24 @@ mod tests {
t.set_config(Config::MailPw, Some("123456")).await.unwrap();
assert!(t.configure().await.is_err());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_get_configured_param() -> Result<()> {
let t = &TestContext::new().await;
let entered_param = EnteredLoginParam {
addr: "alice@example.org".to_string(),
imap: EnteredServerLoginParam {
user: "alice@example.net".to_string(),
password: "foobar".to_string(),
..Default::default()
},
..Default::default()
};
let configured_param = get_configured_param(t, &entered_param).await?;
assert_eq!(configured_param.imap_user, "alice@example.net");
assert_eq!(configured_param.smtp_user, "");
Ok(())
}
}

View File

@@ -179,7 +179,9 @@ pub const DC_DESIRED_TEXT_LEN: usize = DC_DESIRED_TEXT_LINE_LEN * DC_DESIRED_TEX
// and may be set together with the username, password etc.
// via dc_set_config() using the key "server_flags".
/// Force OAuth2 authorization. This flag does not skip automatic configuration.
/// Force OAuth2 authorization.
///
/// This flag does not skip automatic configuration.
/// Before calling configure() with DC_LP_AUTH_OAUTH2 set,
/// the user has to confirm access at the URL returned by dc_get_oauth2_url().
pub const DC_LP_AUTH_OAUTH2: i32 = 0x2;

View File

@@ -30,7 +30,6 @@ use crate::context::Context;
use crate::events::EventType;
use crate::key::{load_self_public_key, DcKey, SignedPublicKey};
use crate::log::LogExt;
use crate::login_param::LoginParam;
use crate::message::MessageState;
use crate::mimeparser::AvatarAction;
use crate::param::{Param, Params};
@@ -1191,7 +1190,10 @@ impl Contact {
);
let contact = Contact::get_by_id(context, contact_id).await?;
let loginparam = LoginParam::load_configured_params(context).await?;
let addr = context
.get_config(Config::ConfiguredAddr)
.await?
.unwrap_or_default();
let peerstate = Peerstate::from_addr(context, &contact.addr).await?;
let Some(peerstate) = peerstate.filter(|peerstate| peerstate.peek_key(false).is_some())
@@ -1220,8 +1222,8 @@ impl Contact {
.peek_key(false)
.map(|k| k.fingerprint().to_string())
.unwrap_or_default();
if loginparam.addr < peerstate.addr {
cat_fingerprint(&mut ret, &loginparam.addr, &fingerprint_self, "");
if addr < peerstate.addr {
cat_fingerprint(&mut ret, &addr, &fingerprint_self, "");
cat_fingerprint(
&mut ret,
&peerstate.addr,
@@ -1235,7 +1237,7 @@ impl Contact {
&fingerprint_other_verified,
&fingerprint_other_unverified,
);
cat_fingerprint(&mut ret, &loginparam.addr, &fingerprint_self, "");
cat_fingerprint(&mut ret, &addr, &fingerprint_self, "");
}
Ok(ret)
@@ -2888,7 +2890,7 @@ Hi."#;
bob.recv_msg(&sent_msg).await;
let contact = Contact::get_by_id(&bob, *contacts.first().unwrap()).await?;
let green = ansi_term::Color::Green.normal();
let green = nu_ansi_term::Color::Green.normal();
assert!(
contact.was_seen_recently(),
"{}",

View File

@@ -27,7 +27,7 @@ use crate::download::DownloadState;
use crate::events::{Event, EventEmitter, EventType, Events};
use crate::imap::{FolderMeaning, Imap, ServerMetadata};
use crate::key::{load_self_public_key, load_self_secret_key, DcKey as _};
use crate::login_param::LoginParam;
use crate::login_param::{ConfiguredLoginParam, EnteredLoginParam};
use crate::message::{self, Message, MessageState, MsgId, Viewtype};
use crate::param::{Param, Params};
use crate::peer_channels::Iroh;
@@ -515,8 +515,11 @@ impl Context {
Ok(val)
}
/// Does a background fetch
/// pauses the scheduler and does one imap fetch, then unpauses and returns
/// Does a single round of fetching from IMAP and returns.
///
/// Can be used even if I/O is currently stopped.
/// If I/O is currently stopped, starts a new IMAP connection
/// and fetches from Inbox and DeltaChat folders.
pub async fn background_fetch(&self) -> Result<()> {
if !(self.is_configured().await?) {
return Ok(());
@@ -524,35 +527,63 @@ impl Context {
let address = self.get_primary_self_addr().await?;
let time_start = tools::Time::now();
info!(self, "background_fetch started fetching {address}");
info!(self, "background_fetch started fetching {address}.");
let _pause_guard = self.scheduler.pause(self.clone()).await?;
if self.scheduler.is_running().await {
self.scheduler.maybe_network().await;
// connection
let mut connection = Imap::new_configured(self, channel::bounded(1).1).await?;
let mut session = connection.prepare(self).await?;
// Wait until fetching is finished.
// Ideally we could wait for connectivity change events,
// but sleep loop is good enough.
// fetch imap folders
for folder_meaning in [FolderMeaning::Inbox, FolderMeaning::Mvbox] {
let (_, watch_folder) = convert_folder_meaning(self, folder_meaning).await?;
connection
.fetch_move_delete(self, &mut session, &watch_folder, folder_meaning)
.await?;
}
// First 100 ms sleep in chunks of 10 ms.
for _ in 0..10 {
if self.all_work_done().await {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
}
// update quota (to send warning if full) - but only check it once in a while
if self
.quota_needs_update(DC_BACKGROUND_FETCH_QUOTA_CHECK_RATELIMIT)
.await
{
if let Err(err) = self.update_recent_quota(&mut session).await {
warn!(self, "Failed to update quota: {err:#}.");
// If we are not finished in 100 ms, keep waking up every 100 ms.
while !self.all_work_done().await {
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
}
} else {
// Pause the scheduler to ensure another connection does not start
// while we are fetching on a dedicated connection.
let _pause_guard = self.scheduler.pause(self.clone()).await?;
// Start a new dedicated connection.
let mut connection = Imap::new_configured(self, channel::bounded(1).1).await?;
let mut session = connection.prepare(self).await?;
// Fetch IMAP folders.
// Inbox is fetched before Mvbox because fetching from Inbox
// may result in moving some messages to Mvbox.
for folder_meaning in [FolderMeaning::Inbox, FolderMeaning::Mvbox] {
if let Some((_folder_config, watch_folder)) =
convert_folder_meaning(self, folder_meaning).await?
{
connection
.fetch_move_delete(self, &mut session, &watch_folder, folder_meaning)
.await?;
}
}
// Update quota (to send warning if full) - but only check it once in a while.
if self
.quota_needs_update(DC_BACKGROUND_FETCH_QUOTA_CHECK_RATELIMIT)
.await
{
if let Err(err) = self.update_recent_quota(&mut session).await {
warn!(self, "Failed to update quota: {err:#}.");
}
}
}
info!(
self,
"background_fetch done for {address} took {:?}",
"background_fetch done for {address} took {:?}.",
time_elapsed(&time_start),
);
@@ -715,8 +746,10 @@ impl Context {
/// Returns information about the context as key-value pairs.
pub async fn get_info(&self) -> Result<BTreeMap<&'static str, String>> {
let unset = "0";
let l = LoginParam::load_candidate_params_unchecked(self).await?;
let l2 = LoginParam::load_configured_params(self).await?;
let l = EnteredLoginParam::load(self).await?;
let l2 = ConfiguredLoginParam::load(self)
.await?
.map_or_else(|| "Not configured".to_string(), |param| param.to_string());
let secondary_addrs = self.get_secondary_self_addrs().await?.join(", ");
let displayname = self.get_config(Config::Displayname).await?;
let chats = get_chat_cnt(self).await?;
@@ -724,7 +757,7 @@ impl Context {
let request_msgs = message::get_request_msg_cnt(self).await;
let contacts = Contact::get_real_cnt(self).await?;
let is_configured = self.get_config_int(Config::Configured).await?;
let socks5_enabled = self.get_config_int(Config::Socks5Enabled).await?;
let proxy_enabled = self.get_config_int(Config::ProxyEnabled).await?;
let dbversion = self
.sql
.get_raw_config_int("dbversion")
@@ -805,9 +838,9 @@ impl Context {
.unwrap_or_else(|| "<unset>".to_string()),
);
res.insert("is_configured", is_configured.to_string());
res.insert("socks5_enabled", socks5_enabled.to_string());
res.insert("proxy_enabled", proxy_enabled.to_string());
res.insert("entered_account_settings", l.to_string());
res.insert("used_account_settings", l2.to_string());
res.insert("used_account_settings", l2);
if let Some(server_id) = &*self.server_id.read().await {
res.insert("imap_server_id", format!("{server_id:?}"));
@@ -824,6 +857,12 @@ impl Context {
"is_muted",
self.get_config_bool(Config::IsMuted).await?.to_string(),
);
res.insert(
"private_tag",
self.get_config(Config::PrivateTag)
.await?
.unwrap_or_else(|| "<unset>".to_string()),
);
if let Some(metadata) = &*self.metadata.read().await {
if let Some(comment) = &metadata.comment {
@@ -1265,6 +1304,12 @@ impl Context {
///
/// If `chat_id` is provided this searches only for messages in this chat, if `chat_id`
/// is `None` this searches messages from all chats.
///
/// NB: Wrt the search in long messages which are shown truncated with the "Show Full Message…"
/// button, we only look at the first several kilobytes. Let's not fix this -- one can send a
/// dictionary in the message that matches any reasonable search request, but the user won't see
/// the match because they should tap on "Show Full Message…" for that. Probably such messages
/// would only clutter search results.
pub async fn search_msgs(&self, chat_id: Option<ChatId>, query: &str) -> Result<Vec<MsgId>> {
let real_query = query.trim().to_lowercase();
if real_query.is_empty() {
@@ -1691,6 +1736,8 @@ mod tests {
"server_flags",
"skip_start_messages",
"smtp_certificate_checks",
"proxy_url", // May contain passwords, don't leak it to the logs.
"socks5_enabled", // SOCKS5 options are deprecated.
"socks5_host",
"socks5_port",
"socks5_user",

View File

@@ -313,7 +313,7 @@ pub(crate) async fn get_autocrypt_peerstate(
if let Some(ref mut peerstate) = peerstate {
if addr_cmp(&peerstate.addr, from) {
if allow_change {
peerstate.apply_header(header, message_time);
peerstate.apply_header(context, header, message_time);
peerstate.save_to_db(&context.sql).await?;
} else {
info!(

View File

@@ -98,19 +98,26 @@ impl MsgId {
Ok(())
}
/// Updates the message download state. Returns `Ok` if the message doesn't exist anymore.
pub(crate) async fn update_download_state(
self,
context: &Context,
download_state: DownloadState,
) -> Result<()> {
let msg = Message::load_from_db(context, self).await?;
context
if context
.sql
.execute(
"UPDATE msgs SET download_state=? WHERE id=?;",
(download_state, self),
)
.await?;
.await?
== 0
{
return Ok(());
}
let Some(msg) = Message::load_from_db_optional(context, self).await? else {
return Ok(());
};
context.emit_event(EventType::MsgsChanged {
chat_id: msg.chat_id,
msg_id: self,
@@ -135,7 +142,17 @@ pub(crate) async fn download_msg(
msg_id: MsgId,
session: &mut Session,
) -> Result<()> {
let msg = Message::load_from_db(context, msg_id).await?;
let Some(msg) = Message::load_from_db_optional(context, msg_id).await? else {
// If partially downloaded message was already deleted
// we do not know its Message-ID anymore
// so cannot download it.
//
// Probably the message expired due to `delete_device_after`
// setting or was otherwise removed from the device,
// so we don't want it to reappear anyway.
return Ok(());
};
let row = context
.sql
.query_row_optional(
@@ -312,11 +329,19 @@ mod tests {
DownloadState::InProgress,
DownloadState::Failure,
DownloadState::Done,
DownloadState::Done,
] {
msg_id.update_download_state(&t, *s).await?;
let msg = Message::load_from_db(&t, msg_id).await?;
assert_eq!(msg.download_state(), *s);
}
t.sql
.execute("DELETE FROM msgs WHERE id=?", (msg_id,))
.await?;
// Nothing to do is ok.
msg_id
.update_download_state(&t, DownloadState::Done)
.await?;
Ok(())
}

View File

@@ -69,7 +69,7 @@ use std::num::ParseIntError;
use std::str::FromStr;
use std::time::{Duration, UNIX_EPOCH};
use anyhow::{ensure, Result};
use anyhow::{ensure, Context as _, Result};
use async_channel::Receiver;
use serde::{Deserialize, Serialize};
use tokio::time::timeout;
@@ -176,9 +176,13 @@ impl ChatId {
pub async fn get_ephemeral_timer(self, context: &Context) -> Result<Timer> {
let timer = context
.sql
.query_get_value("SELECT ephemeral_timer FROM chats WHERE id=?;", (self,))
.await?;
Ok(timer.unwrap_or_default())
.query_get_value(
"SELECT IFNULL(ephemeral_timer, 0) FROM chats WHERE id=?",
(self,),
)
.await?
.with_context(|| format!("Chat {self} not found"))?;
Ok(timer)
}
/// Set ephemeral timer value without sending a message.
@@ -509,7 +513,8 @@ async fn next_delete_device_after_timestamp(context: &Context) -> Result<Option<
FROM msgs
WHERE chat_id > ?
AND chat_id != ?
AND chat_id != ?;
AND chat_id != ?
HAVING count(*) > 0
"#,
(DC_CHAT_ID_TRASH, self_chat_id, device_chat_id),
)
@@ -533,7 +538,8 @@ async fn next_expiration_timestamp(context: &Context) -> Option<i64> {
SELECT min(ephemeral_timestamp)
FROM msgs
WHERE ephemeral_timestamp != 0
AND chat_id != ?;
AND chat_id != ?
HAVING count(*) > 0
"#,
(DC_CHAT_ID_TRASH,), // Trash contains already deleted messages, skip them
)
@@ -1410,4 +1416,14 @@ mod tests {
Ok(())
}
/// Tests that `.get_ephemeral_timer()` returns an error for invalid chat ID.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_get_ephemeral_timer_wrong_chat_id() -> Result<()> {
let context = TestContext::new().await;
let chat_id = ChatId::new(12345);
assert!(chat_id.get_ephemeral_timer(&context).await.is_err());
Ok(())
}
}

View File

@@ -8,6 +8,7 @@ use crate::config::Config;
use crate::contact::ContactId;
use crate::ephemeral::Timer as EphemeralTimer;
use crate::message::MsgId;
use crate::reaction::Reaction;
use crate::webxdc::StatusUpdateSerial;
/// Event payload.
@@ -94,6 +95,18 @@ pub enum EventType {
contact_id: ContactId,
},
/// Reactions for the message changed.
IncomingReaction {
/// ID of the contact whose reaction set is changed.
contact_id: ContactId,
/// ID of the message for which reactions were changed.
msg_id: MsgId,
/// The reaction.
reaction: Reaction,
},
/// There is a fresh message. Typically, the user will show an notification
/// when receiving this message.
///
@@ -288,6 +301,13 @@ pub enum EventType {
data: Vec<u8>,
},
/// Advertisement received over an ephemeral peer channel.
/// This can be used by bots to initiate peer-to-peer communication from their side.
WebxdcRealtimeAdvertisementReceived {
/// Message ID of the webxdc instance.
msg_id: MsgId,
},
/// Inform that a message containing a webxdc instance has been deleted.
WebxdcInstanceDeleted {
/// ID of the deleted message.

View File

@@ -32,15 +32,19 @@ use crate::contact::{Contact, ContactId, Modifier, Origin};
use crate::context::Context;
use crate::events::EventType;
use crate::headerdef::{HeaderDef, HeaderDefMap};
use crate::login_param::{LoginParam, ServerLoginParam};
use crate::log::LogExt;
use crate::login_param::{
prioritize_server_login_params, ConfiguredLoginParam, ConfiguredServerLoginParam,
};
use crate::message::{self, Message, MessageState, MessengerMessage, MsgId, Viewtype};
use crate::mimeparser;
use crate::net::proxy::ProxyConfig;
use crate::net::session::SessionStream;
use crate::oauth2::get_oauth2_access_token;
use crate::receive_imf::{
from_field_to_contact_id, get_prefetch_parent_message, receive_imf_inner, ReceivedMsg,
};
use crate::scheduler::connectivity::ConnectivityStore;
use crate::socks::Socks5Config;
use crate::sql;
use crate::stock_str;
use crate::tools::{self, create_id, duration_to_str};
@@ -52,7 +56,7 @@ pub mod scan_folders;
pub mod select_folder;
pub(crate) mod session;
use client::Client;
use client::{determine_capabilities, Client};
use mailparse::SingleInfo;
use session::Session;
@@ -73,12 +77,18 @@ pub(crate) struct Imap {
addr: String,
/// Login parameters.
lp: ServerLoginParam,
lp: Vec<ConfiguredServerLoginParam>,
/// Password.
password: String,
/// Proxy configuration.
proxy_config: Option<ProxyConfig>,
/// SOCKS 5 configuration.
socks5_config: Option<Socks5Config>,
strict_tls: bool,
oauth2: bool,
login_failed_once: bool,
pub(crate) connectivity: ConnectivityStore,
@@ -228,31 +238,29 @@ impl Imap {
///
/// `addr` is used to renew token if OAuth2 authentication is used.
pub fn new(
lp: &ServerLoginParam,
socks5_config: Option<Socks5Config>,
lp: Vec<ConfiguredServerLoginParam>,
password: String,
proxy_config: Option<ProxyConfig>,
addr: &str,
strict_tls: bool,
oauth2: bool,
idle_interrupt_receiver: Receiver<()>,
) -> Result<Self> {
if lp.server.is_empty() || lp.user.is_empty() || lp.password.is_empty() {
bail!("Incomplete IMAP connection parameters");
}
let imap = Imap {
) -> Self {
Imap {
idle_interrupt_receiver,
addr: addr.to_string(),
lp: lp.clone(),
socks5_config,
lp,
password,
proxy_config,
strict_tls,
oauth2,
login_failed_once: false,
connectivity: Default::default(),
conn_last_try: UNIX_EPOCH,
conn_backoff_ms: 0,
// 1 connection per minute + a burst of 2.
ratelimit: Ratelimit::new(Duration::new(120, 0), 2.0),
};
Ok(imap)
}
}
/// Creates new disconnected IMAP client using configured parameters.
@@ -260,18 +268,18 @@ impl Imap {
context: &Context,
idle_interrupt_receiver: Receiver<()>,
) -> Result<Self> {
if !context.is_configured().await? {
bail!("IMAP Connect without configured params");
}
let param = LoginParam::load_configured_params(context).await?;
let param = ConfiguredLoginParam::load(context)
.await?
.context("Not configured")?;
let imap = Self::new(
&param.imap,
param.socks5_config.clone(),
param.imap.clone(),
param.imap_password.clone(),
param.proxy_config.clone(),
&param.addr,
param.strict_tls(),
param.oauth2,
idle_interrupt_receiver,
)?;
);
Ok(imap)
}
@@ -283,10 +291,6 @@ impl Imap {
/// instead if you are going to actually use connection rather than trying connection
/// parameters.
pub(crate) async fn connect(&mut self, context: &Context) -> Result<Session> {
if self.lp.server.is_empty() {
bail!("IMAP operation attempted while it is torn down");
}
let now = tools::Time::now();
let until_can_send = max(
min(self.conn_last_try, now)
@@ -328,91 +332,123 @@ impl Imap {
);
self.conn_backoff_ms = max(BACKOFF_MIN_MS, self.conn_backoff_ms);
let connection_res = Client::connect(
context,
self.lp.server.as_ref(),
self.lp.port,
self.strict_tls,
self.socks5_config.clone(),
self.lp.security,
)
.await;
let client = connection_res?;
self.conn_backoff_ms = BACKOFF_MIN_MS;
self.ratelimit.send();
let imap_user: &str = self.lp.user.as_ref();
let imap_pw: &str = self.lp.password.as_ref();
let oauth2 = self.lp.oauth2;
let login_res = if oauth2 {
info!(context, "Logging into IMAP server with OAuth 2");
let addr: &str = self.addr.as_ref();
let token = get_oauth2_access_token(context, addr, imap_pw, true)
.await?
.context("IMAP could not get OAUTH token")?;
let auth = OAuth2 {
user: imap_user.into(),
access_token: token,
let login_params = prioritize_server_login_params(&context.sql, &self.lp, "imap").await?;
let mut first_error = None;
for lp in login_params {
info!(context, "IMAP trying to connect to {}.", &lp.connection);
let connection_candidate = lp.connection.clone();
let client = match Client::connect(
context,
self.proxy_config.clone(),
self.strict_tls,
connection_candidate,
)
.await
{
Ok(client) => client,
Err(err) => {
warn!(context, "IMAP failed to connect: {err:#}.");
first_error.get_or_insert(err);
continue;
}
};
client.authenticate("XOAUTH2", auth).await
} else {
info!(context, "Logging into IMAP server with LOGIN");
client.login(imap_user, imap_pw).await
};
match login_res {
Ok(session) => {
// Store server ID in the context to display in account info.
let mut lock = context.server_id.write().await;
lock.clone_from(&session.capabilities.server_id);
self.conn_backoff_ms = BACKOFF_MIN_MS;
self.ratelimit.send();
self.login_failed_once = false;
context.emit_event(EventType::ImapConnected(format!(
"IMAP-LOGIN as {}",
self.lp.user
)));
self.connectivity.set_connected(context).await;
info!(context, "Successfully logged into IMAP server");
Ok(session)
}
let imap_user: &str = lp.user.as_ref();
let imap_pw: &str = &self.password;
Err(err) => {
let imap_user = self.lp.user.to_owned();
let message = stock_str::cannot_login(context, &imap_user).await;
let login_res = if self.oauth2 {
info!(context, "Logging into IMAP server with OAuth 2.");
let addr: &str = self.addr.as_ref();
warn!(context, "{} ({:#})", message, err);
let token = get_oauth2_access_token(context, addr, imap_pw, true)
.await?
.context("IMAP could not get OAUTH token")?;
let auth = OAuth2 {
user: imap_user.into(),
access_token: token,
};
client.authenticate("XOAUTH2", auth).await
} else {
info!(context, "Logging into IMAP server with LOGIN.");
client.login(imap_user, imap_pw).await
};
let lock = context.wrong_pw_warning_mutex.lock().await;
if self.login_failed_once
&& err.to_string().to_lowercase().contains("authentication")
&& context.get_config_bool(Config::NotifyAboutWrongPw).await?
{
if let Err(e) = context
.set_config_internal(Config::NotifyAboutWrongPw, None)
.await
{
warn!(context, "{:#}", e);
}
drop(lock);
match login_res {
Ok(mut session) => {
let capabilities = determine_capabilities(&mut session).await?;
let mut msg = Message::new(Viewtype::Text);
msg.text.clone_from(&message);
if let Err(e) =
chat::add_device_msg_with_importance(context, None, Some(&mut msg), true)
let session = if capabilities.can_compress {
info!(context, "Enabling IMAP compression.");
let compressed_session = session
.compress(|s| {
let session_stream: Box<dyn SessionStream> = Box::new(s);
session_stream
})
.await
{
warn!(context, "{:#}", e);
}
} else {
self.login_failed_once = true;
.context("Failed to enable IMAP compression")?;
Session::new(compressed_session, capabilities)
} else {
Session::new(session, capabilities)
};
// Store server ID in the context to display in account info.
let mut lock = context.server_id.write().await;
lock.clone_from(&session.capabilities.server_id);
self.login_failed_once = false;
context.emit_event(EventType::ImapConnected(format!(
"IMAP-LOGIN as {}",
lp.user
)));
self.connectivity.set_connected(context).await;
info!(context, "Successfully logged into IMAP server");
return Ok(session);
}
Err(format_err!("{}\n\n{:#}", message, err))
Err(err) => {
let imap_user = lp.user.to_owned();
let message = stock_str::cannot_login(context, &imap_user).await;
let err_str = err.to_string();
warn!(context, "IMAP failed to login: {err:#}.");
first_error.get_or_insert(format_err!("{message} ({err:#})"));
let lock = context.wrong_pw_warning_mutex.lock().await;
if self.login_failed_once
&& err_str.to_lowercase().contains("authentication")
&& context.get_config_bool(Config::NotifyAboutWrongPw).await?
{
if let Err(e) = context
.set_config_internal(Config::NotifyAboutWrongPw, None)
.await
{
warn!(context, "{e:#}.");
}
drop(lock);
let mut msg = Message::new(Viewtype::Text);
msg.text.clone_from(&message);
if let Err(e) = chat::add_device_msg_with_importance(
context,
None,
Some(&mut msg),
true,
)
.await
{
warn!(context, "Failed to add device message: {e:#}.");
}
} else {
self.login_failed_once = true;
}
}
}
}
Err(first_error.unwrap_or_else(|| format_err!("No IMAP connection candidates provided")))
}
/// Prepare for IMAP operation.
@@ -1025,6 +1061,52 @@ impl Session {
Ok(())
}
/// Uploads sync messages from the `imap_send` table with `\Seen` flag set.
pub(crate) async fn send_sync_msgs(&mut self, context: &Context, folder: &str) -> Result<()> {
context.send_sync_msg().await?;
while let Some((id, mime, msg_id, attempts)) = context
.sql
.query_row_optional(
"SELECT id, mime, msg_id, attempts FROM imap_send ORDER BY id LIMIT 1",
(),
|row| {
let id: i64 = row.get(0)?;
let mime: String = row.get(1)?;
let msg_id: MsgId = row.get(2)?;
let attempts: i64 = row.get(3)?;
Ok((id, mime, msg_id, attempts))
},
)
.await
.context("Failed to SELECT from imap_send")?
{
let res = self
.append(folder, Some("(\\Seen)"), None, mime)
.await
.with_context(|| format!("IMAP APPEND to {folder} failed for {msg_id}"))
.log_err(context);
if res.is_ok() {
msg_id.set_delivered(context).await?;
}
const MAX_ATTEMPTS: i64 = 2;
if res.is_ok() || attempts >= MAX_ATTEMPTS - 1 {
context
.sql
.execute("DELETE FROM imap_send WHERE id=?", (id,))
.await
.context("Failed to delete from imap_send")?;
} else {
context
.sql
.execute("UPDATE imap_send SET attempts=attempts+1 WHERE id=?", (id,))
.await
.context("Failed to update imap_send.attempts")?;
res?;
}
}
Ok(())
}
/// Stores pending `\Seen` flags for messages in `imap_markseen` table.
pub(crate) async fn store_seen_flags_on_imap(&mut self, context: &Context) -> Result<()> {
let rows = context

View File

@@ -25,6 +25,10 @@ pub(crate) struct Capabilities {
/// <https://tools.ietf.org/html/rfc5464>
pub can_metadata: bool,
/// True if the server has COMPRESS=DEFLATE capability as defined in
/// <https://tools.ietf.org/html/rfc4978>
pub can_compress: bool,
/// True if the server supports XDELTAPUSH capability.
/// This capability means setting /private/devicetoken IMAP METADATA
/// on the INBOX results in new mail notifications

View File

@@ -1,22 +1,21 @@
use std::net::SocketAddr;
use std::ops::{Deref, DerefMut};
use anyhow::{bail, format_err, Context as _, Result};
use anyhow::{Context as _, Result};
use async_imap::Client as ImapClient;
use async_imap::Session as ImapSession;
use fast_socks5::client::Socks5Stream;
use tokio::io::BufWriter;
use super::capabilities::Capabilities;
use super::session::Session;
use crate::context::Context;
use crate::login_param::{ConnectionCandidate, ConnectionSecurity};
use crate::net::dns::{lookup_host_with_cache, update_connect_timestamp};
use crate::net::proxy::ProxyConfig;
use crate::net::session::SessionStream;
use crate::net::tls::wrap_tls;
use crate::net::update_connection_history;
use crate::net::{connect_tcp_inner, connect_tls_inner};
use crate::provider::Socket;
use crate::socks::Socks5Config;
use crate::net::{
connect_tcp_inner, connect_tls_inner, run_connection_attempts, update_connection_history,
};
use crate::tools::time;
#[derive(Debug)]
@@ -51,7 +50,7 @@ fn alpn(port: u16) -> &'static [&'static str] {
/// Determine server capabilities.
///
/// If server supports ID capability, send our client ID.
async fn determine_capabilities(
pub(crate) async fn determine_capabilities(
session: &mut ImapSession<Box<dyn SessionStream>>,
) -> Result<Capabilities> {
let caps = session
@@ -69,6 +68,7 @@ async fn determine_capabilities(
can_check_quota: caps.has_str("QUOTA"),
can_condstore: caps.has_str("CONDSTORE"),
can_metadata: caps.has_str("METADATA"),
can_compress: caps.has_str("COMPRESS=DEFLATE"),
can_push: caps.has_str("XDELTAPUSH"),
is_chatmail: caps.has_str("XCHATMAIL"),
server_id,
@@ -83,86 +83,121 @@ impl Client {
}
}
pub(crate) async fn login(self, username: &str, password: &str) -> Result<Session> {
pub(crate) async fn login(
self,
username: &str,
password: &str,
) -> Result<ImapSession<Box<dyn SessionStream>>> {
let Client { inner, .. } = self;
let mut session = inner
let session = inner
.login(username, password)
.await
.map_err(|(err, _client)| err)?;
let capabilities = determine_capabilities(&mut session).await?;
Ok(Session::new(session, capabilities))
Ok(session)
}
pub(crate) async fn authenticate(
self,
auth_type: &str,
authenticator: impl async_imap::Authenticator,
) -> Result<Session> {
) -> Result<ImapSession<Box<dyn SessionStream>>> {
let Client { inner, .. } = self;
let mut session = inner
let session = inner
.authenticate(auth_type, authenticator)
.await
.map_err(|(err, _client)| err)?;
let capabilities = determine_capabilities(&mut session).await?;
Ok(Session::new(session, capabilities))
Ok(session)
}
async fn connection_attempt(
context: Context,
host: String,
security: ConnectionSecurity,
resolved_addr: SocketAddr,
strict_tls: bool,
) -> Result<Self> {
let context = &context;
let host = &host;
info!(
context,
"Attempting IMAP connection to {host} ({resolved_addr})."
);
let res = match security {
ConnectionSecurity::Tls => {
Client::connect_secure(resolved_addr, host, strict_tls).await
}
ConnectionSecurity::Starttls => {
Client::connect_starttls(resolved_addr, host, strict_tls).await
}
ConnectionSecurity::Plain => Client::connect_insecure(resolved_addr).await,
};
match res {
Ok(client) => {
let ip_addr = resolved_addr.ip().to_string();
let port = resolved_addr.port();
let save_cache = match security {
ConnectionSecurity::Tls | ConnectionSecurity::Starttls => strict_tls,
ConnectionSecurity::Plain => false,
};
if save_cache {
update_connect_timestamp(context, host, &ip_addr).await?;
}
update_connection_history(context, "imap", host, port, &ip_addr, time()).await?;
Ok(client)
}
Err(err) => {
warn!(
context,
"Failed to connect to {host} ({resolved_addr}): {err:#}."
);
Err(err)
}
}
}
pub async fn connect(
context: &Context,
host: &str,
port: u16,
proxy_config: Option<ProxyConfig>,
strict_tls: bool,
socks5_config: Option<Socks5Config>,
security: Socket,
candidate: ConnectionCandidate,
) -> Result<Self> {
if let Some(socks5_config) = socks5_config {
let host = &candidate.host;
let port = candidate.port;
let security = candidate.security;
if let Some(proxy_config) = proxy_config {
let client = match security {
Socket::Automatic => bail!("IMAP port security is not configured"),
Socket::Ssl => {
Client::connect_secure_socks5(context, host, port, strict_tls, socks5_config)
ConnectionSecurity::Tls => {
Client::connect_secure_proxy(context, host, port, strict_tls, proxy_config)
.await?
}
Socket::Starttls => {
Client::connect_starttls_socks5(context, host, port, socks5_config, strict_tls)
ConnectionSecurity::Starttls => {
Client::connect_starttls_proxy(context, host, port, proxy_config, strict_tls)
.await?
}
Socket::Plain => {
Client::connect_insecure_socks5(context, host, port, socks5_config).await?
ConnectionSecurity::Plain => {
Client::connect_insecure_proxy(context, host, port, proxy_config).await?
}
};
update_connection_history(context, "imap", host, port, host, time()).await?;
Ok(client)
} else {
let mut first_error = None;
let load_cache =
strict_tls && (security == Socket::Ssl || security == Socket::Starttls);
for resolved_addr in
lookup_host_with_cache(context, host, port, "imap", load_cache).await?
{
let res = match security {
Socket::Automatic => bail!("IMAP port security is not configured"),
Socket::Ssl => Client::connect_secure(resolved_addr, host, strict_tls).await,
Socket::Starttls => {
Client::connect_starttls(resolved_addr, host, strict_tls).await
}
Socket::Plain => Client::connect_insecure(resolved_addr).await,
};
match res {
Ok(client) => {
let ip_addr = resolved_addr.ip().to_string();
if load_cache {
update_connect_timestamp(context, host, &ip_addr).await?;
}
update_connection_history(context, "imap", host, port, &ip_addr, time())
.await?;
return Ok(client);
}
Err(err) => {
warn!(context, "Failed to connect to {resolved_addr}: {err:#}.");
first_error.get_or_insert(err);
}
}
}
Err(first_error.unwrap_or_else(|| format_err!("no DNS resolution results for {host}")))
let load_cache = match security {
ConnectionSecurity::Tls | ConnectionSecurity::Starttls => strict_tls,
ConnectionSecurity::Plain => false,
};
let connection_futures =
lookup_host_with_cache(context, host, port, "imap", load_cache)
.await?
.into_iter()
.map(|resolved_addr| {
let context = context.clone();
let host = host.to_string();
Self::connection_attempt(context, host, security, resolved_addr, strict_tls)
});
run_connection_attempts(connection_futures).await
}
}
@@ -217,17 +252,17 @@ impl Client {
Ok(client)
}
async fn connect_secure_socks5(
async fn connect_secure_proxy(
context: &Context,
domain: &str,
port: u16,
strict_tls: bool,
socks5_config: Socks5Config,
proxy_config: ProxyConfig,
) -> Result<Self> {
let socks5_stream = socks5_config
let proxy_stream = proxy_config
.connect(context, domain, port, strict_tls)
.await?;
let tls_stream = wrap_tls(strict_tls, domain, alpn(port), socks5_stream).await?;
let tls_stream = wrap_tls(strict_tls, domain, alpn(port), proxy_stream).await?;
let buffered_stream = BufWriter::new(tls_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let mut client = Client::new(session_stream);
@@ -238,14 +273,14 @@ impl Client {
Ok(client)
}
async fn connect_insecure_socks5(
async fn connect_insecure_proxy(
context: &Context,
domain: &str,
port: u16,
socks5_config: Socks5Config,
proxy_config: ProxyConfig,
) -> Result<Self> {
let socks5_stream = socks5_config.connect(context, domain, port, false).await?;
let buffered_stream = BufWriter::new(socks5_stream);
let proxy_stream = proxy_config.connect(context, domain, port, false).await?;
let buffered_stream = BufWriter::new(proxy_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let mut client = Client::new(session_stream);
let _greeting = client
@@ -255,20 +290,20 @@ impl Client {
Ok(client)
}
async fn connect_starttls_socks5(
async fn connect_starttls_proxy(
context: &Context,
hostname: &str,
port: u16,
socks5_config: Socks5Config,
proxy_config: ProxyConfig,
strict_tls: bool,
) -> Result<Self> {
let socks5_stream = socks5_config
let proxy_stream = proxy_config
.connect(context, hostname, port, strict_tls)
.await?;
// Run STARTTLS command and convert the client back into a stream.
let buffered_socks5_stream = BufWriter::new(socks5_stream);
let mut client = ImapClient::new(buffered_socks5_stream);
let buffered_proxy_stream = BufWriter::new(proxy_stream);
let mut client = ImapClient::new(buffered_proxy_stream);
let _greeting = client
.read_response()
.await
@@ -277,10 +312,10 @@ impl Client {
.run_command_and_check_ok("STARTTLS", None)
.await
.context("STARTTLS command failed")?;
let buffered_socks5_stream = client.into_inner();
let socks5_stream: Socks5Stream<_> = buffered_socks5_stream.into_inner();
let buffered_proxy_stream = client.into_inner();
let proxy_stream = buffered_proxy_stream.into_inner();
let tls_stream = wrap_tls(strict_tls, hostname, &[], socks5_stream)
let tls_stream = wrap_tls(strict_tls, hostname, &[], proxy_stream)
.await
.context("STARTTLS upgrade failed")?;
let buffered_stream = BufWriter::new(tls_stream);

View File

@@ -2,18 +2,21 @@
use std::ffi::OsStr;
use std::path::{Path, PathBuf};
use std::pin::Pin;
use ::pgp::types::KeyTrait;
use anyhow::{bail, ensure, format_err, Context as _, Result};
use deltachat_contact_tools::EmailAddress;
use futures::TryStreamExt;
use futures_lite::FutureExt;
use pin_project::pin_project;
use tokio::fs::{self, File};
use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
use tokio_tar::Archive;
use crate::blob::BlobDirContents;
use crate::chat::{self, delete_and_reset_all_device_msgs};
use crate::config::Config;
use crate::context::Context;
use crate::e2ee;
use crate::events::EventType;
@@ -177,10 +180,7 @@ async fn set_self_key(context: &Context, armored: &str, set_default: bool) -> Re
info!(context, "No Autocrypt-Prefer-Encrypt header.");
};
let self_addr = context.get_primary_self_addr().await?;
let addr = EmailAddress::new(&self_addr)?;
let keypair = pgp::KeyPair {
addr,
public: public_key,
secret: private_key,
};
@@ -215,7 +215,7 @@ async fn imex_inner(
path.display()
);
ensure!(context.sql.is_open().await, "Database not opened.");
context.emit_event(EventType::ImexProgress(10));
context.emit_event(EventType::ImexProgress(1));
if what == ImexMode::ExportBackup || what == ImexMode::ExportSelfKeys {
// before we export anything, make sure the private key exists
@@ -297,12 +297,71 @@ pub(crate) async fn import_backup_stream<R: tokio::io::AsyncRead + Unpin>(
.0
}
/// Reader that emits progress events as bytes are read from it.
#[pin_project]
struct ProgressReader<R> {
/// Wrapped reader.
#[pin]
inner: R,
/// Number of bytes successfully read from the internal reader.
read: usize,
/// Total size of the backup .tar file expected to be read from the reader.
/// Used to calculate the progress.
file_size: usize,
/// Last progress emitted to avoid emitting the same progress value twice.
last_progress: usize,
/// Context for emitting progress events.
context: Context,
}
impl<R> ProgressReader<R> {
fn new(r: R, context: Context, file_size: u64) -> Self {
Self {
inner: r,
read: 0,
file_size: file_size as usize,
last_progress: 1,
context,
}
}
}
impl<R> AsyncRead for ProgressReader<R>
where
R: AsyncRead,
{
fn poll_read(
self: Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
buf: &mut ReadBuf<'_>,
) -> std::task::Poll<std::io::Result<()>> {
let this = self.project();
let before = buf.filled().len();
let res = this.inner.poll_read(cx, buf);
if let std::task::Poll::Ready(Ok(())) = res {
*this.read = this.read.saturating_add(buf.filled().len() - before);
let progress = std::cmp::min(1000 * *this.read / *this.file_size, 999);
if progress > *this.last_progress {
this.context.emit_event(EventType::ImexProgress(progress));
*this.last_progress = progress;
}
}
res
}
}
async fn import_backup_stream_inner<R: tokio::io::AsyncRead + Unpin>(
context: &Context,
backup_file: R,
file_size: u64,
passphrase: String,
) -> (Result<()>,) {
let backup_file = ProgressReader::new(backup_file, context.clone(), file_size);
let mut archive = Archive::new(backup_file);
let mut entries = match archive.entries() {
@@ -310,29 +369,12 @@ async fn import_backup_stream_inner<R: tokio::io::AsyncRead + Unpin>(
Err(e) => return (Err(e).context("Failed to get archive entries"),),
};
let mut blobs = Vec::new();
// We already emitted ImexProgress(10) above
let mut last_progress = 10;
const PROGRESS_MIGRATIONS: u128 = 999;
let mut total_size: u64 = 0;
let mut res: Result<()> = loop {
let mut f = match entries.try_next().await {
Ok(Some(f)) => f,
Ok(None) => break Ok(()),
Err(e) => break Err(e).context("Failed to get next entry"),
};
total_size += match f.header().entry_size() {
Ok(size) => size,
Err(e) => break Err(e).context("Failed to get entry size"),
};
let max = PROGRESS_MIGRATIONS - 1;
let progress = std::cmp::min(
max * u128::from(total_size) / std::cmp::max(u128::from(file_size), 1),
max,
);
if progress > last_progress {
context.emit_event(EventType::ImexProgress(progress as usize));
last_progress = progress;
}
let path = match f.path() {
Ok(path) => path.to_path_buf(),
@@ -373,13 +415,16 @@ async fn import_backup_stream_inner<R: tokio::io::AsyncRead + Unpin>(
.await
.context("cannot import unpacked database");
}
if res.is_ok() {
res = adjust_delete_server_after(context).await;
}
fs::remove_file(unpacked_database)
.await
.context("cannot remove unpacked database")
.log_err(context)
.ok();
if res.is_ok() {
context.emit_event(EventType::ImexProgress(PROGRESS_MIGRATIONS as usize));
context.emit_event(EventType::ImexProgress(999));
res = context.sql.run_migrations(context).await;
}
if res.is_ok() {
@@ -452,7 +497,14 @@ async fn export_backup(context: &Context, dir: &Path, passphrase: String) -> Res
let file = File::create(&temp_path).await?;
let blobdir = BlobDirContents::new(context).await?;
export_backup_stream(context, &temp_db_path, blobdir, file)
let mut file_size = 0;
file_size += temp_db_path.metadata()?.len();
for blob in blobdir.iter() {
file_size += blob.to_abs_path().metadata()?.len()
}
export_backup_stream(context, &temp_db_path, blobdir, file, file_size)
.await
.context("Exporting backup to file failed")?;
fs::rename(temp_path, &dest_path).await?;
@@ -460,33 +512,99 @@ async fn export_backup(context: &Context, dir: &Path, passphrase: String) -> Res
Ok(())
}
/// Writer that emits progress events as bytes are written into it.
#[pin_project]
struct ProgressWriter<W> {
/// Wrapped writer.
#[pin]
inner: W,
/// Number of bytes successfully written into the internal writer.
written: usize,
/// Total size of the backup .tar file expected to be written into the writer.
/// Used to calculate the progress.
file_size: usize,
/// Last progress emitted to avoid emitting the same progress value twice.
last_progress: usize,
/// Context for emitting progress events.
context: Context,
}
impl<W> ProgressWriter<W> {
fn new(w: W, context: Context, file_size: u64) -> Self {
Self {
inner: w,
written: 0,
file_size: file_size as usize,
last_progress: 1,
context,
}
}
}
impl<W> AsyncWrite for ProgressWriter<W>
where
W: AsyncWrite,
{
fn poll_write(
self: Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
buf: &[u8],
) -> std::task::Poll<Result<usize, std::io::Error>> {
let this = self.project();
let res = this.inner.poll_write(cx, buf);
if let std::task::Poll::Ready(Ok(written)) = res {
*this.written = this.written.saturating_add(written);
let progress = std::cmp::min(1000 * *this.written / *this.file_size, 999);
if progress > *this.last_progress {
this.context.emit_event(EventType::ImexProgress(progress));
*this.last_progress = progress;
}
}
res
}
fn poll_flush(
self: Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Result<(), std::io::Error>> {
self.project().inner.poll_flush(cx)
}
fn poll_shutdown(
self: Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Result<(), std::io::Error>> {
self.project().inner.poll_shutdown(cx)
}
}
/// Exports the database and blobs into a stream.
pub(crate) async fn export_backup_stream<'a, W>(
context: &'a Context,
temp_db_path: &Path,
blobdir: BlobDirContents<'a>,
writer: W,
file_size: u64,
) -> Result<()>
where
W: tokio::io::AsyncWrite + tokio::io::AsyncWriteExt + Unpin + Send + 'static,
{
let writer = ProgressWriter::new(writer, context.clone(), file_size);
let mut builder = tokio_tar::Builder::new(writer);
builder
.append_path_with_name(temp_db_path, DBFILE_BACKUP_NAME)
.await?;
let mut last_progress = 10;
for (i, blob) in blobdir.iter().enumerate() {
for blob in blobdir.iter() {
let mut file = File::open(blob.to_abs_path()).await?;
let path_in_archive = PathBuf::from(BLOBS_BACKUP_NAME).join(blob.as_name());
builder.append_file(path_in_archive, &mut file).await?;
let progress = std::cmp::min(1000 * i / blobdir.len(), 999);
if progress > last_progress {
context.emit_event(EventType::ImexProgress(progress));
last_progress = progress;
}
}
builder.finish().await?;
@@ -677,6 +795,7 @@ async fn export_database(
.to_str()
.with_context(|| format!("path {} is not valid unicode", dest.display()))?;
adjust_delete_server_after(context).await?;
context
.sql
.set_raw_config_int("backup_time", timestamp)
@@ -706,6 +825,19 @@ async fn export_database(
.await
}
/// Sets `Config::DeleteServerAfter` to "never" if needed so that new messages are present on the
/// server after a backup restoration or available for all devices in multi-device case.
/// NB: Calling this after a backup import isn't reliable as we can crash in between, but this is a
/// problem only for old backups, new backups already have `DeleteServerAfter` set if necessary.
async fn adjust_delete_server_after(context: &Context) -> Result<()> {
if context.is_chatmail().await? && !context.config_exists(Config::DeleteServerAfter).await? {
context
.set_config(Config::DeleteServerAfter, Some("0"))
.await?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use std::time::Duration;
@@ -891,6 +1023,49 @@ mod tests {
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_export_import_chatmail_backup() -> Result<()> {
let backup_dir = tempfile::tempdir().unwrap();
let context1 = &TestContext::new_alice().await;
// Check that the setting is displayed correctly.
assert_eq!(
context1.get_config(Config::DeleteServerAfter).await?,
Some("0".to_string())
);
context1.set_config_bool(Config::IsChatmail, true).await?;
assert_eq!(
context1.get_config(Config::DeleteServerAfter).await?,
Some("1".to_string())
);
assert_eq!(context1.get_config_delete_server_after().await?, Some(0));
imex(context1, ImexMode::ExportBackup, backup_dir.path(), None).await?;
let _event = context1
.evtracker
.get_matching(|evt| matches!(evt, EventType::ImexProgress(1000)))
.await;
let context2 = &TestContext::new().await;
let backup = has_backup(context2, backup_dir.path()).await?;
imex(context2, ImexMode::ImportBackup, backup.as_ref(), None).await?;
let _event = context2
.evtracker
.get_matching(|evt| matches!(evt, EventType::ImexProgress(1000)))
.await;
assert!(context2.is_configured().await?);
assert!(context2.is_chatmail().await?);
for ctx in [context1, context2] {
assert_eq!(
ctx.get_config(Config::DeleteServerAfter).await?,
Some("0".to_string())
);
assert_eq!(ctx.get_config_delete_server_after().await?, None);
}
Ok(())
}
/// This is a regression test for
/// https://github.com/deltachat/deltachat-android/issues/2263
/// where the config cache wasn't reset properly after a backup.

View File

@@ -31,36 +31,25 @@ use std::pin::Pin;
use std::sync::Arc;
use std::task::Poll;
use anyhow::{anyhow, bail, ensure, format_err, Context as _, Result};
use futures_lite::StreamExt;
use anyhow::{bail, format_err, Context as _, Result};
use futures_lite::FutureExt;
use iroh_net::relay::RelayMode;
use iroh_net::Endpoint;
use iroh_old;
use iroh_old::blobs::Collection;
use iroh_old::get::DataStream;
use iroh_old::progress::ProgressEmitter;
use iroh_old::provider::Ticket;
use tokio::fs::{self, File};
use tokio::io::{self, AsyncWriteExt, BufWriter};
use tokio::sync::broadcast::error::RecvError;
use tokio::sync::{broadcast, Mutex};
use tokio::task::{JoinHandle, JoinSet};
use tokio_stream::wrappers::ReadDirStream;
use tokio::fs;
use tokio::task::JoinHandle;
use tokio_util::sync::CancellationToken;
use crate::chat::{add_device_msg, delete_and_reset_all_device_msgs};
use crate::chat::add_device_msg;
use crate::context::Context;
use crate::imex::BlobDirContents;
use crate::message::{Message, Viewtype};
use crate::qr::{self, Qr};
use crate::qr::Qr;
use crate::stock_str::backup_transfer_msg_body;
use crate::tools::{create_id, time, TempPathGuard};
use crate::EventType;
use super::{export_backup_stream, export_database, import_backup_stream, DBFILE_BACKUP_NAME};
const MAX_CONCURRENT_DIALS: u8 = 16;
/// ALPN protocol identifier for the backup transfer protocol.
const BACKUP_ALPN: &[u8] = b"/deltachat/backup";
@@ -109,7 +98,7 @@ impl BackupProvider {
let endpoint = Endpoint::builder()
.alpns(vec![BACKUP_ALPN.to_vec()])
.relay_mode(relay_mode)
.bind(0)
.bind()
.await?;
let node_addr = endpoint.node_addr().await?;
@@ -120,6 +109,7 @@ impl BackupProvider {
.get_blobdir()
.parent()
.context("Context dir not found")?;
let dbfile = context_dir.join(DBFILE_BACKUP_NAME);
if fs::metadata(&dbfile).await.is_ok() {
fs::remove_file(&dbfile).await?;
@@ -135,7 +125,6 @@ impl BackupProvider {
export_database(context, &dbfile, passphrase, time())
.await
.context("Database export failed")?;
context.emit_event(EventType::ImexProgress(300));
let drop_token = CancellationToken::new();
let handle = {
@@ -189,6 +178,7 @@ impl BackupProvider {
}
info!(context, "Received valid backup authentication token.");
context.emit_event(EventType::ImexProgress(1));
let blobdir = BlobDirContents::new(&context).await?;
@@ -200,7 +190,7 @@ impl BackupProvider {
send_stream.write_all(&file_size.to_be_bytes()).await?;
export_backup_stream(&context, &dbfile, blobdir, send_stream)
export_backup_stream(&context, &dbfile, blobdir, send_stream, file_size)
.await
.context("Failed to write backup into QUIC stream")?;
info!(context, "Finished writing backup into QUIC stream.");
@@ -232,12 +222,31 @@ impl BackupProvider {
conn = endpoint.accept() => {
if let Some(conn) = conn {
let conn = match conn.accept() {
Ok(conn) => conn,
Err(err) => {
warn!(context, "Failed to accept iroh connection: {err:#}.");
continue;
}
};
// Got a new in-progress connection.
let context = context.clone();
let auth_token = auth_token.clone();
let dbfile = dbfile.clone();
if let Err(err) = Self::handle_connection(context.clone(), conn, auth_token, dbfile).await {
if let Err(err) = Self::handle_connection(context.clone(), conn, auth_token, dbfile).race(
async {
cancel_token.recv().await.ok();
Err(format_err!("Backup transfer cancelled"))
}
).race(
async {
drop_token.cancelled().await;
Err(format_err!("Backup provider dropped"))
}
).await {
warn!(context, "Error while handling backup connection: {err:#}.");
context.emit_event(EventType::ImexProgress(0));
break;
} else {
info!(context, "Backup transfer finished successfully.");
break;
@@ -247,10 +256,12 @@ impl BackupProvider {
}
},
_ = cancel_token.recv() => {
info!(context, "Backup transfer cancelled by the user, stopping accept loop.");
context.emit_event(EventType::ImexProgress(0));
break;
}
_ = drop_token.cancelled() => {
info!(context, "Backup transfer cancelled by dropping the provider, stopping accept loop.");
context.emit_event(EventType::ImexProgress(0));
break;
}
@@ -279,33 +290,6 @@ impl Future for BackupProvider {
}
}
/// Retrieves backup from a legacy backup provider using iroh 0.4.
pub async fn get_legacy_backup(context: &Context, qr: Qr) -> Result<()> {
ensure!(
matches!(qr, Qr::Backup { .. }),
"QR code for backup must be of type DCBACKUP"
);
ensure!(
!context.is_configured().await?,
"Cannot import backups to accounts in use."
);
// Acquire global "ongoing" mutex.
let cancel_token = context.alloc_ongoing().await?;
let _guard = context.scheduler.pause(context.clone()).await;
info!(
context,
"Running get_backup for {}",
qr::format_backup(&qr)?
);
let res = tokio::select! {
biased;
res = get_backup_inner(context, qr) => res,
_ = cancel_token.recv() => Err(format_err!("cancelled")),
};
context.free_ongoing().await;
res
}
pub async fn get_backup2(
context: &Context,
node_addr: iroh_net::NodeAddr,
@@ -313,7 +297,7 @@ pub async fn get_backup2(
) -> Result<()> {
let relay_mode = RelayMode::Disabled;
let endpoint = Endpoint::builder().relay_mode(relay_mode).bind(0).await?;
let endpoint = Endpoint::builder().relay_mode(relay_mode).bind().await?;
let conn = endpoint.connect(node_addr, BACKUP_ALPN).await?;
let (mut send_stream, mut recv_stream) = conn.open_bi().await?;
@@ -335,9 +319,13 @@ pub async fn get_backup2(
// Send an acknowledgement, but ignore the errors.
// We have imported backup successfully already.
send_stream.write_all(b".").await.ok();
send_stream.finish().await.ok();
send_stream.finish().ok();
info!(context, "Sent backup reception acknowledgment.");
// Wait for the peer to acknowledge reception of the acknowledgement
// before closing the connection.
_ = send_stream.stopped().await;
Ok(())
}
@@ -349,202 +337,33 @@ pub async fn get_backup2(
///
/// This is a long running operation which will return only when completed.
///
/// Using [`Qr`] as argument is a bit odd as it only accepts specific variants of it. It
/// does avoid having [`iroh_old::provider::Ticket`] in the primary API however, without
/// Using [`Qr`] as argument is a bit odd as it only accepts specific variant of it. It
/// does avoid having [`iroh_net::NodeAddr`] in the primary API however, without
/// having to revert to untyped bytes.
pub async fn get_backup(context: &Context, qr: Qr) -> Result<()> {
match qr {
Qr::Backup { .. } => get_legacy_backup(context, qr).await?,
Qr::Backup2 {
node_addr,
auth_token,
} => get_backup2(context, node_addr, auth_token).await?,
_ => bail!("QR code for backup must be of type DCBACKUP or DCBACKUP2"),
} => {
let cancel_token = context.alloc_ongoing().await?;
let res = get_backup2(context, node_addr, auth_token)
.race(async {
cancel_token.recv().await.ok();
Err(format_err!("Backup reception cancelled"))
})
.await;
if res.is_err() {
context.emit_event(EventType::ImexProgress(0));
}
context.free_ongoing().await;
res?;
}
_ => bail!("QR code for backup must be of type DCBACKUP2"),
}
Ok(())
}
async fn get_backup_inner(context: &Context, qr: Qr) -> Result<()> {
let ticket = match qr {
Qr::Backup { ticket } => ticket,
_ => bail!("QR code for backup must be of type DCBACKUP"),
};
match transfer_from_provider(context, &ticket).await {
Ok(()) => {
context.sql.run_migrations(context).await?;
delete_and_reset_all_device_msgs(context).await?;
context.emit_event(ReceiveProgress::Completed.into());
Ok(())
}
Err(err) => {
// Clean up any blobs we already wrote.
let readdir = fs::read_dir(context.get_blobdir()).await?;
let mut readdir = ReadDirStream::new(readdir);
while let Some(dirent) = readdir.next().await {
if let Ok(dirent) = dirent {
fs::remove_file(dirent.path()).await.ok();
}
}
context.emit_event(ReceiveProgress::Failed.into());
Err(err)
}
}
}
async fn transfer_from_provider(context: &Context, ticket: &Ticket) -> Result<()> {
let progress = ProgressEmitter::new(0, ReceiveProgress::max_blob_progress());
spawn_progress_proxy(context.clone(), progress.subscribe());
let on_connected = || {
context.emit_event(ReceiveProgress::Connected.into());
async { Ok(()) }
};
let on_collection = |collection: &Collection| {
context.emit_event(ReceiveProgress::CollectionReceived.into());
progress.set_total(collection.total_blobs_size());
async { Ok(()) }
};
let jobs = Mutex::new(JoinSet::default());
let on_blob =
|hash, reader, name| on_blob(context, &progress, &jobs, ticket, hash, reader, name);
// Perform the transfer.
let keylog = false; // Do not enable rustls SSLKEYLOGFILE env var functionality
let stats = iroh_old::get::run_ticket(
ticket,
keylog,
MAX_CONCURRENT_DIALS,
on_connected,
on_collection,
on_blob,
)
.await?;
let mut jobs = jobs.lock().await;
while let Some(job) = jobs.join_next().await {
job.context("job failed")?;
}
drop(progress);
info!(
context,
"Backup transfer finished, transfer rate was {} Mbps.",
stats.mbits()
);
Ok(())
}
/// Get callback when a blob is received from the provider.
///
/// This writes the blobs to the blobdir. If the blob is the database it will import it to
/// the database of the current [`Context`].
async fn on_blob(
context: &Context,
progress: &ProgressEmitter,
jobs: &Mutex<JoinSet<()>>,
ticket: &Ticket,
_hash: iroh_old::Hash,
mut reader: DataStream,
name: String,
) -> Result<DataStream> {
ensure!(!name.is_empty(), "Received a nameless blob");
let path = if name.starts_with("db/") {
let context_dir = context
.get_blobdir()
.parent()
.ok_or_else(|| anyhow!("Context dir not found"))?;
let dbfile = context_dir.join(DBFILE_BACKUP_NAME);
if fs::metadata(&dbfile).await.is_ok() {
fs::remove_file(&dbfile).await?;
warn!(context, "Previous database export deleted");
}
dbfile
} else {
ensure!(name.starts_with("blob/"), "malformatted blob name");
let blobname = name.rsplit('/').next().context("malformatted blob name")?;
context.get_blobdir().join(blobname)
};
let mut wrapped_reader = progress.wrap_async_read(&mut reader);
let file = File::create(&path).await?;
let mut file = BufWriter::with_capacity(128 * 1024, file);
io::copy(&mut wrapped_reader, &mut file).await?;
file.flush().await?;
if name.starts_with("db/") {
let context = context.clone();
let token = ticket.token().to_string();
jobs.lock().await.spawn(async move {
if let Err(err) = context.sql.import(&path, token).await {
error!(context, "cannot import database: {:#?}", err);
}
if let Err(err) = fs::remove_file(&path).await {
error!(
context,
"failed to delete database import file '{}': {:#?}",
path.display(),
err,
);
}
});
}
Ok(reader)
}
/// Spawns a task proxying progress events.
///
/// This spawns a tokio task which receives events from the [`ProgressEmitter`] and sends
/// them to the context. The task finishes when the emitter is dropped.
///
/// This could be done directly in the emitter by making it less generic.
fn spawn_progress_proxy(context: Context, mut rx: broadcast::Receiver<u16>) {
tokio::spawn(async move {
loop {
match rx.recv().await {
Ok(step) => context.emit_event(ReceiveProgress::BlobProgress(step).into()),
Err(RecvError::Closed) => break,
Err(RecvError::Lagged(_)) => continue,
}
}
});
}
/// Create [`EventType::ImexProgress`] events using readable names.
///
/// Plus you get warnings if you don't use all variants.
#[derive(Debug)]
enum ReceiveProgress {
Connected,
CollectionReceived,
/// A value between 0 and 85 interpreted as a percentage.
///
/// Other values are already used by the other variants of this enum.
BlobProgress(u16),
Completed,
Failed,
}
impl ReceiveProgress {
/// The maximum value for [`ReceiveProgress::BlobProgress`].
///
/// This only exists to keep this magic value local in this type.
fn max_blob_progress() -> u16 {
85
}
}
impl From<ReceiveProgress> for EventType {
fn from(source: ReceiveProgress) -> Self {
let val = match source {
ReceiveProgress::Connected => 50,
ReceiveProgress::CollectionReceived => 100,
ReceiveProgress::BlobProgress(val) => 100 + 10 * val,
ReceiveProgress::Completed => 1000,
ReceiveProgress::Failed => 0,
};
EventType::ImexProgress(val.into())
}
}
#[cfg(test)]
mod tests {
use std::time::Duration;

View File

@@ -244,7 +244,7 @@ async fn generate_keypair(context: &Context) -> Result<KeyPair> {
let _guard = context.generating_key_mutex.lock().await;
// Check if the key appeared while we were waiting on the lock.
match load_keypair(context, &addr).await? {
match load_keypair(context).await? {
Some(key_pair) => Ok(key_pair),
None => {
let start = tools::Time::now();
@@ -266,10 +266,7 @@ async fn generate_keypair(context: &Context) -> Result<KeyPair> {
}
}
pub(crate) async fn load_keypair(
context: &Context,
addr: &EmailAddress,
) -> Result<Option<KeyPair>> {
pub(crate) async fn load_keypair(context: &Context) -> Result<Option<KeyPair>> {
let res = context
.sql
.query_row_optional(
@@ -287,7 +284,6 @@ pub(crate) async fn load_keypair(
Ok(if let Some((pub_bytes, sec_bytes)) = res {
Some(KeyPair {
addr: addr.clone(),
public: SignedPublicKey::from_slice(&pub_bytes)?,
secret: SignedSecretKey::from_slice(&sec_bytes)?,
})
@@ -337,17 +333,11 @@ pub(crate) async fn store_self_keypair(
KeyPairUse::ReadOnly => false,
};
// `addr` and `is_default` written for compatibility with older versions,
// until new cores are rolled out everywhere.
// otherwise "add second device" or "backup" may break.
// moreover, this allows downgrades to the previous version.
// writing of `addr` and `is_default` can be removed ~ 2024-08
let addr = keypair.addr.to_string();
transaction
.execute(
"INSERT OR REPLACE INTO keypairs (public_key, private_key, addr, is_default)
VALUES (?,?,?,?)",
(&public_key, &secret_key, addr, is_default),
"INSERT OR REPLACE INTO keypairs (public_key, private_key)
VALUES (?,?)",
(&public_key, &secret_key),
)
.context("Failed to insert keypair")?;
@@ -377,15 +367,10 @@ pub(crate) async fn store_self_keypair(
/// This API is used for testing purposes
/// to avoid generating the key in tests.
/// Use import/export APIs instead.
pub async fn preconfigure_keypair(context: &Context, addr: &str, secret_data: &str) -> Result<()> {
let addr = EmailAddress::new(addr)?;
pub async fn preconfigure_keypair(context: &Context, secret_data: &str) -> Result<()> {
let secret = SignedSecretKey::from_asc(secret_data)?.0;
let public = secret.split_public_key()?;
let keypair = KeyPair {
addr,
public,
secret,
};
let keypair = KeyPair { public, secret };
store_self_keypair(context, &keypair, KeyPairUse::Default).await?;
Ok(())
}

View File

@@ -84,7 +84,6 @@ mod scheduler;
pub mod securejoin;
mod simplify;
mod smtp;
mod socks;
pub mod stock_str;
mod sync;
mod timesmearing;

File diff suppressed because it is too large Load Diff

View File

@@ -148,21 +148,6 @@ impl MsgId {
Ok(())
}
/// Deletes a message, corresponding MDNs and unsent SMTP messages from the database.
pub(crate) async fn delete_from_db(self, context: &Context) -> Result<()> {
context
.sql
.transaction(move |transaction| {
transaction.execute("DELETE FROM smtp WHERE msg_id=?", (self,))?;
transaction.execute("DELETE FROM msgs_mdns WHERE msg_id=?", (self,))?;
transaction.execute("DELETE FROM msgs_status_updates WHERE msg_id=?", (self,))?;
transaction.execute("DELETE FROM msgs WHERE id=?", (self,))?;
Ok(())
})
.await?;
Ok(())
}
pub(crate) async fn set_delivered(self, context: &Context) -> Result<()> {
update_msg_state(context, self, MessageState::OutDelivered).await?;
let chat_id: ChatId = context
@@ -219,11 +204,13 @@ impl MsgId {
}
/// Returns information about hops of a message, used for message info
pub async fn hop_info(self, context: &Context) -> Result<Option<String>> {
context
pub async fn hop_info(self, context: &Context) -> Result<String> {
let hop_info = context
.sql
.query_get_value("SELECT hop_info FROM msgs WHERE id=?", (self,))
.await
.query_get_value("SELECT IFNULL(hop_info, '') FROM msgs WHERE id=?", (self,))
.await?
.with_context(|| format!("Message {self} not found"))?;
Ok(hop_info)
}
/// Returns detailed message information in a multi-line text form.
@@ -328,7 +315,12 @@ impl MsgId {
if let Some(path) = msg.get_file(context) {
let bytes = get_filebytes(context, &path).await?;
ret += &format!("\nFile: {}, {} bytes\n", path.display(), bytes);
ret += &format!(
"\nFile: {}, name: {}, {} bytes\n",
path.display(),
msg.get_filename().unwrap_or_default(),
bytes
);
}
if msg.viewtype != Viewtype::Text {
@@ -361,7 +353,11 @@ impl MsgId {
let hop_info = self.hop_info(context).await?;
ret += "\n\n";
ret += &hop_info.unwrap_or_else(|| "No Hop Info".to_owned());
if hop_info.is_empty() {
ret += "No Hop Info";
} else {
ret += &hop_info;
}
Ok(ret)
}
@@ -1815,8 +1811,8 @@ pub(crate) async fn update_msg_state(
context
.sql
.execute(
&format!("UPDATE msgs SET state=?1 {error_subst} WHERE id=?2 AND (?1!=?3 OR state<?3)"),
(state, msg_id, MessageState::OutDelivered),
&format!("UPDATE msgs SET state=? {error_subst} WHERE id=?"),
(state, msg_id),
)
.await?;
Ok(())
@@ -1904,6 +1900,7 @@ pub async fn get_request_msg_cnt(context: &Context) -> usize {
/// Estimates the number of messages that will be deleted
/// by the options `delete_device_after` or `delete_server_after`.
///
/// This is typically used to show the estimated impact to the user
/// before actually enabling deletion of old messages.
///
@@ -1993,7 +1990,9 @@ pub(crate) async fn rfc724_mid_exists_ex(
.query_row_optional(
&("SELECT id, timestamp_sent, MIN(".to_string()
+ expr
+ ") FROM msgs WHERE rfc724_mid=? ORDER BY timestamp_sent DESC"),
+ ") FROM msgs WHERE rfc724_mid=?
HAVING COUNT(*) > 0 -- Prevent MIN(expr) from returning NULL when there are no rows.
ORDER BY timestamp_sent DESC"),
(rfc724_mid,),
|row| {
let msg_id: MsgId = row.get(0)?;
@@ -2357,6 +2356,25 @@ mod tests {
assert_eq!(quoted_msg.get_text(), msg2.quoted_text().unwrap());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_no_quote() {
let mut tcm = TestContextManager::new();
let alice = &tcm.alice().await;
let bob = &tcm.bob().await;
tcm.send_recv_accept(alice, bob, "Hi!").await;
let msg = tcm
.send_recv(
alice,
bob,
"On 2024-08-28, Alice wrote:\n> A quote.\nNot really.",
)
.await;
assert!(msg.quoted_text().is_none());
assert!(msg.quoted_message(bob).await.unwrap().is_none());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_unencrypted_quote_encrypted_message() -> Result<()> {
let mut tcm = TestContextManager::new();

View File

@@ -82,7 +82,10 @@ pub struct MimeFactory {
/// as needed.
references: String,
/// True if the message requests Message Disposition Notification
/// using `Chat-Disposition-Notification-To` header.
req_mdn: bool,
last_added_location_id: Option<u32>,
/// If the created mime-structure contains sync-items,
@@ -104,10 +107,8 @@ pub struct RenderedEmail {
pub is_gossiped: bool,
pub last_added_location_id: Option<u32>,
/// A comma-separated string of sync-IDs that are used by the rendered email
/// and must be deleted once the message is actually queued for sending
/// (deletion must be done by `delete_sync_ids()`).
/// If the rendered email is not queued for sending, the IDs must not be deleted.
/// A comma-separated string of sync-IDs that are used by the rendered email and must be deleted
/// from `multi_device_sync` once the message is actually queued for sending.
pub sync_ids_to_delete: Option<String>,
/// Message ID (Message in the sense of Email)
@@ -117,6 +118,13 @@ pub struct RenderedEmail {
pub subject: String,
}
fn new_address_with_name(name: &str, address: String) -> Address {
match name == address {
true => Address::new_mailbox(address),
false => Address::new_mailbox_with_name(name.to_string(), address),
}
}
impl MimeFactory {
pub async fn from_msg(context: &Context, msg: Message) -> Result<MimeFactory> {
let chat = Chat::load_from_db(context, msg.chat_id).await?;
@@ -143,7 +151,9 @@ impl MimeFactory {
let mut req_mdn = false;
if chat.is_self_talk() {
recipients.push((from_displayname.to_string(), from_addr.to_string()));
if msg.param.get_cmd() == SystemMessage::AutocryptSetupMessage {
recipients.push((from_displayname.to_string(), from_addr.to_string()));
}
} else if chat.is_mailing_list() {
let list_post = chat
.param
@@ -194,7 +204,8 @@ impl MimeFactory {
let (in_reply_to, references) = context
.sql
.query_row(
"SELECT mime_in_reply_to, mime_references FROM msgs WHERE id=?",
"SELECT mime_in_reply_to, IFNULL(mime_references, '')
FROM msgs WHERE id=?",
(msg.id,),
|row| {
let in_reply_to: String = row.get(0)?;
@@ -344,7 +355,11 @@ impl MimeFactory {
// beside key- and member-changes, force a periodic re-gossip.
let gossiped_timestamp = chat.id.get_gossiped_timestamp(context).await?;
let gossip_period = context.get_config_i64(Config::GossipPeriod).await?;
if time() >= gossiped_timestamp + gossip_period {
// `gossip_period == 0` is a special case for testing,
// enabling gossip in every message.
// Othewise "smeared timestamps" may result in the condition
// to fail even if the clock is monotonic.
if gossip_period == 0 || time() >= gossiped_timestamp + gossip_period {
Ok(true)
} else {
Ok(false)
@@ -472,10 +487,7 @@ impl MimeFactory {
pub async fn render(mut self, context: &Context) -> Result<RenderedEmail> {
let mut headers = Vec::<Header>::new();
let from = Address::new_mailbox_with_name(
self.from_displayname.to_string(),
self.from_addr.clone(),
);
let from = new_address_with_name(&self.from_displayname, self.from_addr.clone());
let undisclosed_recipients = match &self.loaded {
Loaded::Message { chat, .. } => chat.typ == Chattype::Broadcast,
@@ -510,10 +522,7 @@ impl MimeFactory {
if name.is_empty() {
to.push(Address::new_mailbox(addr.clone()));
} else {
to.push(Address::new_mailbox_with_name(
name.to_string(),
addr.clone(),
));
to.push(new_address_with_name(name, addr.clone()));
}
}
@@ -528,8 +537,7 @@ impl MimeFactory {
headers.push(from_header.clone());
if let Some(sender_displayname) = &self.sender_displayname {
let sender =
Address::new_mailbox_with_name(sender_displayname.clone(), self.from_addr.clone());
let sender = new_address_with_name(sender_displayname, self.from_addr.clone());
headers.push(Header::new_with_value("Sender".into(), vec![sender]).unwrap());
}
headers.push(Header::new_with_value("To".into(), to.clone()).unwrap());
@@ -579,6 +587,16 @@ impl MimeFactory {
"Auto-Submitted".to_string(),
"auto-generated".to_string(),
));
} else if let Loaded::Message { msg, .. } = &self.loaded {
if msg.param.get_cmd() == SystemMessage::SecurejoinMessage {
let step = msg.param.get(Param::Arg).unwrap_or_default();
if step != "vg-request" && step != "vc-request" {
headers.push(Header::new(
"Auto-Submitted".to_string(),
"auto-replied".to_string(),
));
}
}
}
if let Loaded::Message { chat, .. } = &self.loaded {
@@ -599,7 +617,9 @@ impl MimeFactory {
// because replies to "Disposition-Notification-To" are weird in many cases
// eg. are just freetext and/or do not follow any standard.
headers.push(Header::new(
"Chat-Disposition-Notification-To".into(),
HeaderDef::ChatDispositionNotificationTo
.get_headername()
.to_string(),
self.from_addr.clone(),
));
}
@@ -1664,10 +1684,7 @@ mod tests {
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == ' '));
let s = format!(
"{}",
Address::new_mailbox_with_name(display_name.to_string(), addr.to_string())
);
let s = format!("{}", new_address_with_name(display_name, addr.to_string()));
println!("{s}");
@@ -1684,15 +1701,19 @@ mod tests {
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == ' '));
let s = format!(
"{}",
Address::new_mailbox_with_name(display_name.to_string(), addr.to_string())
);
let s = format!("{}", new_address_with_name(display_name, addr.to_string()));
// Addresses should not be unnecessarily be encoded, see <https://github.com/deltachat/deltachat-core-rust/issues/1575>:
assert_eq!(s, "a space <x@y.org>");
}
#[test]
fn test_render_email_address_duplicated_as_name() {
let addr = "x@y.org";
let s = format!("{}", new_address_with_name(addr, addr.to_string()));
assert_eq!(s, "<x@y.org>");
}
#[test]
fn test_render_rfc724_mid() {
assert_eq!(
@@ -2234,7 +2255,7 @@ mod tests {
if name.is_empty() {
Address::new_mailbox(addr.to_string())
} else {
Address::new_mailbox_with_name(name.to_string(), addr.to_string())
new_address_with_name(name, addr.to_string())
}
})
.collect();

View File

@@ -17,7 +17,7 @@ use crate::aheader::{Aheader, EncryptPreference};
use crate::blob::BlobObject;
use crate::chat::{add_info_msg, ChatId};
use crate::config::Config;
use crate::constants::{self, Chattype, DC_DESIRED_TEXT_LINES, DC_DESIRED_TEXT_LINE_LEN};
use crate::constants::{self, Chattype};
use crate::contact::{Contact, ContactId, Origin};
use crate::context::Context;
use crate::decrypt::{
@@ -34,7 +34,7 @@ use crate::peerstate::Peerstate;
use crate::simplify::{simplify, SimplifiedText};
use crate::sync::SyncItems;
use crate::tools::{
create_smeared_timestamp, get_filemeta, parse_receive_headers, smeared_time, truncate_by_lines,
create_smeared_timestamp, get_filemeta, parse_receive_headers, smeared_time, truncate_msg_text,
validate_id,
};
use crate::{chatlist_events, location, stock_str, tools};
@@ -1179,22 +1179,11 @@ impl MimeMessage {
(simplified_txt, top_quote)
};
let is_bot = context.get_config_bool(Config::Bot).await?;
let simplified_txt = if is_bot {
simplified_txt
} else {
// Truncate text if it has too many lines
let (simplified_txt, was_truncated) = truncate_by_lines(
simplified_txt,
DC_DESIRED_TEXT_LINES,
DC_DESIRED_TEXT_LINE_LEN,
);
if was_truncated {
self.is_mime_modified = was_truncated;
}
simplified_txt
};
let (simplified_txt, was_truncated) =
truncate_msg_text(context, simplified_txt).await?;
if was_truncated {
self.is_mime_modified = was_truncated;
}
if !simplified_txt.is_empty() || simplified_quote.is_some() {
let mut part = Part {
@@ -3609,6 +3598,17 @@ On 2020-10-25, Bob wrote:
assert!(mimemsg.parts[0].msg.len() <= DC_DESIRED_TEXT_LEN + DC_ELLIPSIS.len());
}
{
let chat = t.get_self_chat().await;
t.send_text(chat.id, &long_txt).await;
let msg = t.get_last_msg_in(chat.id).await;
assert!(msg.has_html());
assert!(
msg.text.matches("just repeated").count() <= DC_DESIRED_TEXT_LEN / REPEAT_TXT.len()
);
assert!(msg.text.len() <= DC_DESIRED_TEXT_LEN + DC_ELLIPSIS.len());
}
t.set_config(Config::Bot, Some("1")).await?;
{

View File

@@ -1,4 +1,5 @@
//! # Common network utilities.
use std::future::Future;
use std::net::SocketAddr;
use std::pin::Pin;
use std::time::Duration;
@@ -6,14 +7,17 @@ use std::time::Duration;
use anyhow::{format_err, Context as _, Result};
use async_native_tls::TlsStream;
use tokio::net::TcpStream;
use tokio::task::JoinSet;
use tokio::time::timeout;
use tokio_io_timeout::TimeoutStream;
use crate::context::Context;
use crate::sql::Sql;
use crate::tools::time;
pub(crate) mod dns;
pub(crate) mod http;
pub(crate) mod proxy;
pub(crate) mod session;
pub(crate) mod tls;
@@ -43,6 +47,14 @@ pub(crate) async fn prune_connection_history(context: &Context) -> Result<()> {
Ok(())
}
/// Update the timestamp of the last successfull connection
/// to the given `host` and `port`
/// with the given application protocol `alpn`.
///
/// `addr` is the string representation of IP address.
/// If connection is made over a proxy which does
/// its own DNS resolution,
/// `addr` should be the same as `host`.
pub(crate) async fn update_connection_history(
context: &Context,
alpn: &str,
@@ -64,21 +76,22 @@ pub(crate) async fn update_connection_history(
Ok(())
}
/// Returns timestamp of the most recent successful connection
/// to the host and port for given protocol.
pub(crate) async fn load_connection_timestamp(
context: &Context,
sql: &Sql,
alpn: &str,
host: &str,
port: u16,
addr: &str,
addr: Option<&str>,
) -> Result<Option<i64>> {
let timestamp = context
.sql
let timestamp = sql
.query_get_value(
"SELECT timestamp FROM connection_history
WHERE host = ?
AND port = ?
AND alpn = ?
AND addr = ?",
AND addr = IFNULL(?, addr)",
(host, port, alpn, addr),
)
.await?;
@@ -121,6 +134,96 @@ pub(crate) async fn connect_tls_inner(
Ok(tls_stream)
}
/// Runs connection attempt futures.
///
/// Accepts iterator of connection attempt futures
/// and runs them until one of them succeeds
/// or all of them fail.
///
/// If all connection attempts fail, returns the first error.
///
/// This functions starts with one connection attempt and maintains
/// up to five parallel connection attempts if connecting takes time.
pub(crate) async fn run_connection_attempts<O, I, F>(mut futures: I) -> Result<O>
where
I: Iterator<Item = F>,
F: Future<Output = Result<O>> + Send + 'static,
O: Send + 'static,
{
let mut connection_attempt_set = JoinSet::new();
// Start additional connection attempts after 300 ms, 1 s, 5 s and 10 s.
// This way we can have up to 5 parallel connection attempts at the same time.
let mut delay_set = JoinSet::new();
for delay in [
Duration::from_millis(300),
Duration::from_secs(1),
Duration::from_secs(5),
Duration::from_secs(10),
] {
delay_set.spawn(tokio::time::sleep(delay));
}
let mut first_error = None;
let res = loop {
if let Some(fut) = futures.next() {
connection_attempt_set.spawn(fut);
}
tokio::select! {
biased;
res = connection_attempt_set.join_next() => {
match res {
Some(res) => {
match res.context("Failed to join task") {
Ok(Ok(conn)) => {
// Successfully connected.
break Ok(conn);
}
Ok(Err(err)) => {
// Some connection attempt failed.
first_error.get_or_insert(err);
}
Err(err) => {
break Err(err);
}
}
}
None => {
// Out of connection attempts.
//
// Break out of the loop and return error.
break Err(
first_error.unwrap_or_else(|| format_err!("No connection attempts were made"))
);
}
}
},
_ = delay_set.join_next(), if !delay_set.is_empty() => {
// Delay expired.
//
// Don't do anything other than pushing
// another connection attempt into `connection_attempt_set`.
}
}
};
// Abort remaining connection attempts and free resources
// such as OS sockets and `Context` references
// held by connection attempt tasks.
//
// `delay_set` contains just `sleep` tasks
// so no need to await futures there,
// it is enough that futures are aborted
// when the set is dropped.
connection_attempt_set.shutdown().await;
res
}
/// If `load_cache` is true, may use cached DNS results.
/// Because the cache may be poisoned with incorrect results by networks hijacking DNS requests,
/// this option should only be used when connection is authenticated,
@@ -133,22 +236,9 @@ pub(crate) async fn connect_tcp(
port: u16,
load_cache: bool,
) -> Result<Pin<Box<TimeoutStream<TcpStream>>>> {
let mut first_error = None;
for resolved_addr in lookup_host_with_cache(context, host, port, "", load_cache).await? {
match connect_tcp_inner(resolved_addr).await {
Ok(stream) => {
return Ok(stream);
}
Err(err) => {
warn!(
context,
"Failed to connect to {}: {:#}.", resolved_addr, err
);
first_error.get_or_insert(err);
}
}
}
Err(first_error.unwrap_or_else(|| format_err!("no DNS resolution results for {host}")))
let connection_futures = lookup_host_with_cache(context, host, port, "", load_cache)
.await?
.into_iter()
.map(connect_tcp_inner);
run_connection_attempts(connection_futures).await
}

View File

@@ -1,4 +1,44 @@
//! DNS resolution and cache.
//!
//! DNS cache in Delta Chat has two layers:
//! in-memory cache and persistent `dns_cache` SQL table.
//!
//! In-memory cache is using a "stale-while-revalidate" strategy.
//! If there is a cached value, it is returned immediately
//! and revalidation task is started in the background
//! to replace old cached IP addresses with new ones.
//! If there is no cached value yet,
//! lookup only finishes when `lookup_host` returns first results.
//! In-memory cache is shared between all accounts
//! and is never stored on the disk.
//! It can be thought of as an extension
//! of the system resolver.
//!
//! Persistent `dns_cache` SQL table is used to collect
//! all IP addresses ever seen for the hostname
//! together with the timestamp
//! of the last time IP address has been seen.
//! Note that this timestamp reflects the time
//! IP address was returned by the in-memory cache
//! rather than the underlying system resolver.
//! Unused entries are removed after 30 days
//! (`CACHE_TTL` constant) to avoid having
//! old non-working IP addresses in the cache indefinitely.
//!
//! When Delta Chat needs an IP address for the host,
//! it queries in-memory cache for the next result
//! and merges the list of IP addresses
//! with the list of IP addresses from persistent cache.
//! Resulting list is constructed
//! by taking the first two results from the resolver
//! followed up by persistent cache results
//! and terminated by the rest of resolver results.
//!
//! Persistent cache results are sorted
//! by the time of the most recent successful connection
//! using the result. For results that have never been
//! used for successful connection timestamp of
//! retrieving them from in-memory cache is used.
use anyhow::{Context as _, Result};
use std::collections::HashMap;
@@ -42,33 +82,110 @@ pub(crate) async fn prune_dns_cache(context: &Context) -> Result<()> {
Ok(())
}
/// Looks up the hostname and updates DNS cache
/// on success.
/// Map from hostname to IP addresses.
///
/// NOTE: sync RwLock is used, so it must not be held across `.await`
/// to avoid deadlocks.
/// See
/// <https://docs.rs/tokio/1.40.0/tokio/sync/struct.Mutex.html#which-kind-of-mutex-should-you-use>
/// and
/// <https://stackoverflow.com/questions/63712823/why-do-i-get-a-deadlock-when-using-tokio-with-a-stdsyncmutex>.
static LOOKUP_HOST_CACHE: Lazy<parking_lot::RwLock<HashMap<String, Vec<IpAddr>>>> =
Lazy::new(Default::default);
/// Wrapper for `lookup_host` that returns IP addresses.
async fn lookup_ips(host: impl tokio::net::ToSocketAddrs) -> Result<impl Iterator<Item = IpAddr>> {
Ok(lookup_host(host)
.await
.context("DNS lookup failure")?
.map(|addr| addr.ip()))
}
async fn lookup_host_with_memory_cache(
context: &Context,
hostname: &str,
port: u16,
) -> Result<Vec<IpAddr>> {
let stale_result = {
let rwlock_read_guard = LOOKUP_HOST_CACHE.read();
rwlock_read_guard.get(hostname).cloned()
};
if let Some(stale_result) = stale_result {
// Revalidate the cache in the background.
{
let context = context.clone();
let hostname = hostname.to_string();
tokio::spawn(async move {
match lookup_ips((hostname.clone(), port)).await {
Ok(res) => {
LOOKUP_HOST_CACHE.write().insert(hostname, res.collect());
}
Err(err) => {
warn!(
context,
"Failed to revalidate results for {hostname:?}: {err:#}."
);
}
}
});
}
info!(
context,
"Using memory-cached DNS resolution for {hostname}."
);
Ok(stale_result)
} else {
info!(
context,
"No memory-cached DNS resolution for {hostname} available, waiting for the resolver."
);
let res: Vec<IpAddr> = lookup_ips((hostname, port)).await?.collect();
// Insert initial result into the cache.
//
// There may already be a result from a parallel
// task stored, overwriting it is not a problem.
LOOKUP_HOST_CACHE
.write()
.insert(hostname.to_string(), res.clone());
Ok(res)
}
}
/// Looks up the hostname and updates
/// persistent DNS cache on success.
async fn lookup_host_and_update_cache(
context: &Context,
hostname: &str,
port: u16,
now: i64,
) -> Result<Vec<SocketAddr>> {
let res: Vec<SocketAddr> = timeout(super::TIMEOUT, lookup_host((hostname, port)))
.await
.context("DNS lookup timeout")?
.context("DNS lookup failure")?
.collect();
let res: Vec<IpAddr> = timeout(
super::TIMEOUT,
lookup_host_with_memory_cache(context, hostname, port),
)
.await
.context("DNS lookup timeout")?
.context("DNS lookup with memory cache failure")?;
for addr in &res {
let ip_string = addr.ip().to_string();
for ip in &res {
let ip_string = ip.to_string();
if ip_string == hostname {
// IP address resolved into itself, not interesting to cache.
continue;
}
info!(context, "Resolved {hostname}:{port} into {addr}.");
info!(context, "Resolved {hostname} into {ip}.");
// Update the cache.
update_cache(context, hostname, &ip_string, now).await?;
}
let res = res
.into_iter()
.map(|ip| SocketAddr::new(ip, port))
.collect();
Ok(res)
}
@@ -108,6 +225,10 @@ pub(crate) async fn update_connect_timestamp(
Ok(())
}
/// Preloaded DNS results that can be used in case of DNS server failures.
///
/// See <https://support.delta.chat/t/no-dns-resolution-result/2778> and
/// <https://github.com/deltachat/deltachat-core-rust/issues/4920> for reasons.
static DNS_PRELOAD: Lazy<HashMap<&'static str, Vec<IpAddr>>> = Lazy::new(|| {
HashMap::from([
(
@@ -501,21 +622,6 @@ static DNS_PRELOAD: Lazy<HashMap<&'static str, Vec<IpAddr>>> = Lazy::new(|| {
])
});
/// Load hardcoded cache if everything else fails.
///
/// See <https://support.delta.chat/t/no-dns-resolution-result/2778> and
/// <https://github.com/deltachat/deltachat-core-rust/issues/4920> for reasons.
///
/// In the future we may pre-resolve all provider database addresses
/// and build them in.
fn load_hardcoded_cache(hostname: &str, port: u16) -> Vec<SocketAddr> {
if let Some(ips) = DNS_PRELOAD.get(hostname) {
ips.iter().map(|ip| SocketAddr::new(*ip, port)).collect()
} else {
Vec::new()
}
}
async fn lookup_cache(
context: &Context,
host: &str,
@@ -574,11 +680,16 @@ async fn sort_by_connection_timestamp(
alpn: &str,
host: &str,
) -> Result<Vec<SocketAddr>> {
let mut res: Vec<(Option<i64>, SocketAddr)> = Vec::new();
let mut res: Vec<(Option<i64>, SocketAddr)> = Vec::with_capacity(input.len());
for addr in input {
let timestamp =
load_connection_timestamp(context, alpn, host, addr.port(), &addr.ip().to_string())
.await?;
let timestamp = load_connection_timestamp(
&context.sql,
alpn,
host,
addr.port(),
Some(&addr.ip().to_string()),
)
.await?;
res.push((timestamp, addr));
}
res.sort_by_key(|(ts, _addr)| std::cmp::Reverse(*ts));
@@ -603,9 +714,14 @@ pub(crate) async fn lookup_host_with_cache(
load_cache: bool,
) -> Result<Vec<SocketAddr>> {
let now = time();
let mut resolved_addrs = match lookup_host_and_update_cache(context, hostname, port, now).await
{
Ok(res) => res,
let resolved_addrs = match lookup_host_and_update_cache(context, hostname, port, now).await {
Ok(res) => {
if alpn.is_empty() {
res
} else {
sort_by_connection_timestamp(context, res, alpn, hostname).await?
}
}
Err(err) => {
warn!(
context,
@@ -614,24 +730,43 @@ pub(crate) async fn lookup_host_with_cache(
Vec::new()
}
};
if !alpn.is_empty() {
resolved_addrs =
sort_by_connection_timestamp(context, resolved_addrs, alpn, hostname).await?;
}
if load_cache {
for addr in lookup_cache(context, hostname, port, alpn, now).await? {
if !resolved_addrs.contains(&addr) {
resolved_addrs.push(addr);
let mut cache = lookup_cache(context, hostname, port, alpn, now).await?;
if let Some(ips) = DNS_PRELOAD.get(hostname) {
for ip in ips {
let addr = SocketAddr::new(*ip, port);
if !cache.contains(&addr) {
cache.push(addr);
}
}
}
if resolved_addrs.is_empty() {
return Ok(load_hardcoded_cache(hostname, port));
Ok(merge_with_cache(resolved_addrs, cache))
} else {
Ok(resolved_addrs)
}
}
/// Merges results received from DNS with cached results.
///
/// At most 10 results are returned.
fn merge_with_cache(
mut resolved_addrs: Vec<SocketAddr>,
cache: Vec<SocketAddr>,
) -> Vec<SocketAddr> {
let rest = resolved_addrs.split_off(std::cmp::min(resolved_addrs.len(), 2));
for addr in cache.into_iter().chain(rest.into_iter()) {
if !resolved_addrs.contains(&addr) {
resolved_addrs.push(addr);
if resolved_addrs.len() >= 10 {
break;
}
}
}
Ok(resolved_addrs)
resolved_addrs
}
#[cfg(test)]
@@ -867,4 +1002,131 @@ mod tests {
],
);
}
#[test]
fn test_merge_with_cache() {
let first_addr = IpAddr::V4(Ipv4Addr::new(192, 168, 1, 1));
let second_addr = IpAddr::V4(Ipv4Addr::new(192, 168, 1, 2));
// If there is no cache, just return resolved addresses.
{
let resolved_addrs = vec![
SocketAddr::new(first_addr, 993),
SocketAddr::new(second_addr, 993),
];
let cache = vec![];
assert_eq!(
merge_with_cache(resolved_addrs.clone(), cache),
resolved_addrs
);
}
// If cache contains address that is not in resolution results,
// it is inserted in the merged result.
{
let resolved_addrs = vec![SocketAddr::new(first_addr, 993)];
let cache = vec![SocketAddr::new(second_addr, 993)];
assert_eq!(
merge_with_cache(resolved_addrs, cache),
vec![
SocketAddr::new(first_addr, 993),
SocketAddr::new(second_addr, 993),
]
);
}
// If cache contains address that is already in resolution results,
// it is not duplicated.
{
let resolved_addrs = vec![
SocketAddr::new(first_addr, 993),
SocketAddr::new(second_addr, 993),
];
let cache = vec![SocketAddr::new(second_addr, 993)];
assert_eq!(
merge_with_cache(resolved_addrs, cache),
vec![
SocketAddr::new(first_addr, 993),
SocketAddr::new(second_addr, 993),
]
);
}
// If DNS resolvers returns a lot of results,
// we should try cached results before going through all
// the resolver results.
{
let resolved_addrs = vec![
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 2)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 3)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 4)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 5)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 6)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 7)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 8)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 9)), 993),
];
let cache = vec![SocketAddr::new(second_addr, 993)];
assert_eq!(
merge_with_cache(resolved_addrs, cache),
vec![
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 2)), 993),
SocketAddr::new(second_addr, 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 3)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 4)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 5)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 6)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 7)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 8)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 9)), 993),
]
);
}
// Even if cache already contains all the incorrect results
// that resolver returns, this should not result in them being sorted to the top.
// Cache has known to work result returned first,
// so we should try it after the second result.
{
let resolved_addrs = vec![
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 2)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 3)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 4)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 5)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 6)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 7)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 8)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 9)), 993),
];
let cache = vec![
SocketAddr::new(second_addr, 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 9)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 8)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 7)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 6)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 5)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 4)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 3)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 2)), 993),
];
assert_eq!(
merge_with_cache(resolved_addrs, cache),
vec![
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 2)), 993),
SocketAddr::new(second_addr, 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 9)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 8)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 7)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 6)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 5)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 4)), 993),
SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 3)), 993),
]
);
}
}
}

View File

@@ -1,21 +1,16 @@
//! # HTTP module.
use std::sync::Arc;
use anyhow::{anyhow, Result};
use anyhow::{anyhow, bail, Context as _, Result};
use bytes::Bytes;
use http_body_util::BodyExt;
use hyper_util::rt::TokioIo;
use mime::Mime;
use once_cell::sync::Lazy;
use serde::Serialize;
use crate::context::Context;
use crate::net::lookup_host_with_cache;
use crate::socks::Socks5Config;
static LETSENCRYPT_ROOT: Lazy<reqwest::tls::Certificate> = Lazy::new(|| {
reqwest::tls::Certificate::from_der(include_bytes!(
"../../assets/root-certificates/letsencrypt/isrgrootx1.der"
))
.unwrap()
});
use crate::net::proxy::ProxyConfig;
use crate::net::session::SessionStream;
use crate::net::tls::wrap_rustls;
/// HTTP(S) GET response.
#[derive(Debug)]
@@ -32,48 +27,94 @@ pub struct Response {
/// Retrieves the text contents of URL using HTTP GET request.
pub async fn read_url(context: &Context, url: &str) -> Result<String> {
Ok(read_url_inner(context, url).await?.text().await?)
let response = read_url_blob(context, url).await?;
let text = String::from_utf8_lossy(&response.blob);
Ok(text.to_string())
}
async fn get_http_sender<B>(
context: &Context,
parsed_url: hyper::Uri,
) -> Result<hyper::client::conn::http1::SendRequest<B>>
where
B: hyper::body::Body + 'static + Send,
B::Data: Send,
B::Error: Into<Box<dyn std::error::Error + Send + Sync>>,
{
let scheme = parsed_url.scheme_str().context("URL has no scheme")?;
let host = parsed_url.host().context("URL has no host")?;
let proxy_config_opt = ProxyConfig::load(context).await?;
let stream: Box<dyn SessionStream> = match scheme {
"http" => {
let port = parsed_url.port_u16().unwrap_or(80);
// It is safe to use cached IP addresses
// for HTTPS URLs, but for HTTP URLs
// better resolve from scratch each time to prevent
// cache poisoning attacks from having lasting effects.
let load_cache = false;
if let Some(proxy_config) = proxy_config_opt {
let proxy_stream = proxy_config
.connect(context, host, port, load_cache)
.await?;
Box::new(proxy_stream)
} else {
let tcp_stream = crate::net::connect_tcp(context, host, port, load_cache).await?;
Box::new(tcp_stream)
}
}
"https" => {
let port = parsed_url.port_u16().unwrap_or(443);
let load_cache = true;
if let Some(proxy_config) = proxy_config_opt {
let proxy_stream = proxy_config
.connect(context, host, port, load_cache)
.await?;
let tls_stream = wrap_rustls(host, &[], proxy_stream).await?;
Box::new(tls_stream)
} else {
let tcp_stream = crate::net::connect_tcp(context, host, port, load_cache).await?;
let tls_stream = wrap_rustls(host, &[], tcp_stream).await?;
Box::new(tls_stream)
}
}
_ => bail!("Unknown URL scheme"),
};
let io = TokioIo::new(stream);
let (sender, conn) = hyper::client::conn::http1::handshake(io).await?;
tokio::task::spawn(conn);
Ok(sender)
}
/// Retrieves the binary contents of URL using HTTP GET request.
pub async fn read_url_blob(context: &Context, url: &str) -> Result<Response> {
let response = read_url_inner(context, url).await?;
let content_type = response
.headers()
.get(reqwest::header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse::<Mime>().ok());
let mimetype = content_type
.as_ref()
.map(|mime| mime.essence_str().to_string());
let encoding = content_type.as_ref().and_then(|mime| {
mime.get_param(mime::CHARSET)
.map(|charset| charset.as_str().to_string())
});
let blob: Vec<u8> = response.bytes().await?.into();
Ok(Response {
blob,
mimetype,
encoding,
})
}
async fn read_url_inner(context: &Context, url: &str) -> Result<reqwest::Response> {
// It is safe to use cached IP addresses
// for HTTPS URLs, but for HTTP URLs
// better resolve from scratch each time to prevent
// cache poisoning attacks from having lasting effects.
let load_cache = url.starts_with("https://");
let client = get_client(context, load_cache).await?;
let mut url = url.to_string();
// Follow up to 10 http-redirects
for _i in 0..10 {
let response = client.get(&url).send().await?;
let parsed_url = url
.parse::<hyper::Uri>()
.with_context(|| format!("Failed to parse URL {url:?}"))?;
let mut sender = get_http_sender(context, parsed_url.clone()).await?;
let authority = parsed_url
.authority()
.context("URL has no authority")?
.clone();
let req = hyper::Request::builder()
.uri(parsed_url.path())
.header(hyper::header::HOST, authority.as_str())
.body(http_body_util::Empty::<Bytes>::new())?;
let response = sender.send_request(req).await?;
if response.status().is_redirection() {
let headers = response.headers();
let header = headers
let header = response
.headers()
.get_all("location")
.iter()
.last()
@@ -84,72 +125,119 @@ async fn read_url_inner(context: &Context, url: &str) -> Result<reqwest::Respons
continue;
}
return Ok(response);
let content_type = response
.headers()
.get("content-type")
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse::<Mime>().ok());
let mimetype = content_type
.as_ref()
.map(|mime| mime.essence_str().to_string());
let encoding = content_type.as_ref().and_then(|mime| {
mime.get_param(mime::CHARSET)
.map(|charset| charset.as_str().to_string())
});
let body = response.collect().await?.to_bytes();
let blob: Vec<u8> = body.to_vec();
return Ok(Response {
blob,
mimetype,
encoding,
});
}
Err(anyhow!("Followed 10 redirections"))
}
struct CustomResolver {
context: Context,
/// Whether to return cached results or not.
/// If resolver can be used for URLs
/// without TLS, e.g. HTTP URLs from HTML email,
/// this must be false. If TLS is used
/// and certificate hostnames are checked,
/// it is safe to load cache.
load_cache: bool,
}
impl CustomResolver {
fn new(context: Context, load_cache: bool) -> Self {
Self {
context,
load_cache,
}
/// Sends an empty POST request to the URL.
///
/// Returns response text and whether request was successful or not.
///
/// Does not follow redirects.
pub(crate) async fn post_empty(context: &Context, url: &str) -> Result<(String, bool)> {
let parsed_url = url
.parse::<hyper::Uri>()
.with_context(|| format!("Failed to parse URL {url:?}"))?;
let scheme = parsed_url.scheme_str().context("URL has no scheme")?;
if scheme != "https" {
bail!("POST requests to non-HTTPS URLs are not allowed");
}
let mut sender = get_http_sender(context, parsed_url.clone()).await?;
let authority = parsed_url
.authority()
.context("URL has no authority")?
.clone();
let req = hyper::Request::post(parsed_url.path())
.header(hyper::header::HOST, authority.as_str())
.body(http_body_util::Empty::<Bytes>::new())?;
let response = sender.send_request(req).await?;
let response_status = response.status();
let body = response.collect().await?.to_bytes();
let text = String::from_utf8_lossy(&body);
let response_text = text.to_string();
Ok((response_text, response_status.is_success()))
}
impl reqwest::dns::Resolve for CustomResolver {
fn resolve(&self, hostname: reqwest::dns::Name) -> reqwest::dns::Resolving {
let context = self.context.clone();
let load_cache = self.load_cache;
Box::pin(async move {
let port = 443; // Actual port does not matter.
let socket_addrs =
lookup_host_with_cache(&context, hostname.as_str(), port, "", load_cache).await;
match socket_addrs {
Ok(socket_addrs) => {
let addrs: reqwest::dns::Addrs = Box::new(socket_addrs.into_iter());
Ok(addrs)
}
Err(err) => Err(err.into()),
}
})
/// Posts string to the given URL.
///
/// Returns true if successful HTTP response code was returned.
///
/// Does not follow redirects.
#[allow(dead_code)]
pub(crate) async fn post_string(context: &Context, url: &str, body: String) -> Result<bool> {
let parsed_url = url
.parse::<hyper::Uri>()
.with_context(|| format!("Failed to parse URL {url:?}"))?;
let scheme = parsed_url.scheme_str().context("URL has no scheme")?;
if scheme != "https" {
bail!("POST requests to non-HTTPS URLs are not allowed");
}
let mut sender = get_http_sender(context, parsed_url.clone()).await?;
let authority = parsed_url
.authority()
.context("URL has no authority")?
.clone();
let request = hyper::Request::post(parsed_url.path())
.header(hyper::header::HOST, authority.as_str())
.body(body)?;
let response = sender.send_request(request).await?;
Ok(response.status().is_success())
}
pub(crate) async fn get_client(context: &Context, load_cache: bool) -> Result<reqwest::Client> {
let socks5_config = Socks5Config::from_database(&context.sql).await?;
let resolver = Arc::new(CustomResolver::new(context.clone(), load_cache));
/// Sends a POST request with x-www-form-urlencoded data.
///
/// Does not follow redirects.
pub(crate) async fn post_form<T: Serialize + ?Sized>(
context: &Context,
url: &str,
form: &T,
) -> Result<Bytes> {
let parsed_url = url
.parse::<hyper::Uri>()
.with_context(|| format!("Failed to parse URL {url:?}"))?;
let scheme = parsed_url.scheme_str().context("URL has no scheme")?;
if scheme != "https" {
bail!("POST requests to non-HTTPS URLs are not allowed");
}
let builder = reqwest::ClientBuilder::new()
.timeout(super::TIMEOUT)
.add_root_certificate(LETSENCRYPT_ROOT.clone())
.dns_resolver(resolver);
let builder = if let Some(socks5_config) = socks5_config {
let proxy = reqwest::Proxy::all(socks5_config.to_url())?;
builder.proxy(proxy)
} else {
// Disable usage of "system" proxy configured via environment variables.
// It is enabled by default in `reqwest`, see
// <https://docs.rs/reqwest/0.11.14/reqwest/struct.ClientBuilder.html#method.no_proxy>
// for documentation.
builder.no_proxy()
};
Ok(builder.build()?)
let encoded_body = serde_urlencoded::to_string(form).context("Failed to encode data")?;
let mut sender = get_http_sender(context, parsed_url.clone()).await?;
let authority = parsed_url
.authority()
.context("URL has no authority")?
.clone();
let request = hyper::Request::post(parsed_url.path())
.header(hyper::header::HOST, authority.as_str())
.header("content-type", "application/x-www-form-urlencoded")
.body(encoded_body)?;
let response = sender.send_request(request).await?;
let bytes = response.collect().await?.to_bytes();
Ok(bytes)
}

655
src/net/proxy.rs Normal file
View File

@@ -0,0 +1,655 @@
//! # Proxy support.
//!
//! Delta Chat supports HTTP(S) CONNECT, SOCKS5 and Shadowsocks protocols.
use std::fmt;
use std::pin::Pin;
use anyhow::{bail, format_err, Context as _, Result};
use base64::Engine;
use bytes::{BufMut, BytesMut};
use fast_socks5::client::Socks5Stream;
use fast_socks5::util::target_addr::ToTargetAddr;
use fast_socks5::AuthenticationMethod;
use fast_socks5::Socks5Command;
use percent_encoding::{percent_encode, NON_ALPHANUMERIC};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio_io_timeout::TimeoutStream;
use url::Url;
use crate::config::Config;
use crate::context::Context;
use crate::net::connect_tcp;
use crate::net::session::SessionStream;
use crate::net::tls::wrap_rustls;
use crate::sql::Sql;
/// Default SOCKS5 port according to [RFC 1928](https://tools.ietf.org/html/rfc1928).
pub const DEFAULT_SOCKS_PORT: u16 = 1080;
#[derive(Debug, Clone)]
pub struct ShadowsocksConfig {
pub server_config: shadowsocks::config::ServerConfig,
}
impl PartialEq for ShadowsocksConfig {
fn eq(&self, other: &Self) -> bool {
self.server_config.to_url() == other.server_config.to_url()
}
}
impl Eq for ShadowsocksConfig {}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct HttpConfig {
/// HTTP proxy host.
pub host: String,
/// HTTP proxy port.
pub port: u16,
/// Username and password for basic authentication.
///
/// If set, `Proxy-Authorization` header is sent.
pub user_password: Option<(String, String)>,
}
impl HttpConfig {
fn from_url(url: Url) -> Result<Self> {
let host = url
.host_str()
.context("HTTP proxy URL has no host")?
.to_string();
let port = url
.port_or_known_default()
.context("HTTP(S) URLs are guaranteed to return Some port")?;
let user_password = if let Some(password) = url.password() {
let username = percent_encoding::percent_decode_str(url.username())
.decode_utf8()
.context("HTTP(S) proxy username is not a valid UTF-8")?
.to_string();
let password = percent_encoding::percent_decode_str(password)
.decode_utf8()
.context("HTTP(S) proxy password is not a valid UTF-8")?
.to_string();
Some((username, password))
} else {
None
};
let http_config = HttpConfig {
host,
port,
user_password,
};
Ok(http_config)
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Socks5Config {
pub host: String,
pub port: u16,
pub user_password: Option<(String, String)>,
}
impl Socks5Config {
async fn connect(
&self,
context: &Context,
target_host: &str,
target_port: u16,
load_dns_cache: bool,
) -> Result<Socks5Stream<Pin<Box<TimeoutStream<TcpStream>>>>> {
let tcp_stream = connect_tcp(context, &self.host, self.port, load_dns_cache)
.await
.context("Failed to connect to SOCKS5 proxy")?;
let authentication_method = if let Some((username, password)) = self.user_password.as_ref()
{
Some(AuthenticationMethod::Password {
username: username.into(),
password: password.into(),
})
} else {
None
};
let mut socks_stream =
Socks5Stream::use_stream(tcp_stream, authentication_method, Default::default()).await?;
let target_addr = (target_host, target_port).to_target_addr()?;
socks_stream
.request(Socks5Command::TCPConnect, target_addr)
.await?;
Ok(socks_stream)
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ProxyConfig {
// HTTP proxy.
Http(HttpConfig),
// HTTPS proxy.
Https(HttpConfig),
// SOCKS5 proxy.
Socks5(Socks5Config),
// Shadowsocks proxy.
Shadowsocks(ShadowsocksConfig),
}
/// Constructs HTTP/1.1 `CONNECT` request for HTTP(S) proxy.
fn http_connect_request(host: &str, port: u16, auth: Option<(&str, &str)>) -> String {
// According to <https://datatracker.ietf.org/doc/html/rfc7230#section-5.4>
// clients MUST send `Host:` header in HTTP/1.1 requests,
// so repeat the host there.
let mut res = format!("CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}:{port}\r\n");
if let Some((username, password)) = auth {
res += "Proxy-Authorization: Basic ";
res += &base64::engine::general_purpose::STANDARD.encode(format!("{username}:{password}"));
res += "\r\n";
}
res += "\r\n";
res
}
/// Sends HTTP/1.1 `CONNECT` request over given connection
/// to establish an HTTP tunnel.
///
/// Returns the same connection back so actual data can be tunneled over it.
async fn http_tunnel<T>(mut conn: T, host: &str, port: u16, auth: Option<(&str, &str)>) -> Result<T>
where
T: AsyncReadExt + AsyncWriteExt + Unpin,
{
// Send HTTP/1.1 CONNECT request.
let request = http_connect_request(host, port, auth);
conn.write_all(request.as_bytes()).await?;
let mut buffer = BytesMut::with_capacity(4096);
let res = loop {
if !buffer.has_remaining_mut() {
bail!("CONNECT response exceeded buffer size");
}
let n = conn.read_buf(&mut buffer).await?;
if n == 0 {
bail!("Unexpected end of CONNECT response");
}
let res = &buffer[..];
if res.ends_with(b"\r\n\r\n") {
// End of response is not reached, read more.
break res;
}
};
// Normally response looks like
// `HTTP/1.1 200 Connection established\r\n\r\n`.
if !res.starts_with(b"HTTP/") {
bail!("Unexpected HTTP CONNECT response: {res:?}");
}
// HTTP-version followed by space has fixed length
// according to RFC 7230:
// <https://datatracker.ietf.org/doc/html/rfc7230#section-3.1.2>
//
// Normally status line starts with `HTTP/1.1 `.
// We only care about 3-digit status code.
let status_code = res
.get(9..12)
.context("HTTP status line does not contain a status code")?;
// Interpert status code according to
// <https://datatracker.ietf.org/doc/html/rfc7231#section-6>.
if status_code == b"407" {
Err(format_err!("Proxy Authentication Required"))
} else if status_code.starts_with(b"2") {
// Success.
Ok(conn)
} else {
Err(format_err!(
"Failed to establish HTTP CONNECT tunnel: {res:?}"
))
}
}
impl ProxyConfig {
/// Creates a new proxy configuration by parsing given proxy URL.
fn from_url(url: &str) -> Result<Self> {
let url = Url::parse(url).context("Cannot parse proxy URL")?;
match url.scheme() {
"http" => {
let http_config = HttpConfig::from_url(url)?;
Ok(Self::Http(http_config))
}
"https" => {
let https_config = HttpConfig::from_url(url)?;
Ok(Self::Https(https_config))
}
"ss" => {
let server_config = shadowsocks::config::ServerConfig::from_url(url.as_str())?;
let shadowsocks_config = ShadowsocksConfig { server_config };
Ok(Self::Shadowsocks(shadowsocks_config))
}
// Because of `curl` convention,
// `socks5` URL scheme may be expected to resolve domain names locally
// with `socks5h` URL scheme meaning that hostnames are passed to the proxy.
// Resolving hostnames locally is not supported
// in Delta Chat when using a proxy
// to prevent DNS leaks.
// Because of this we do not distinguish
// between `socks5` and `socks5h`.
"socks5" => {
let host = url
.host_str()
.context("socks5 URL has no host")?
.to_string();
let port = url.port().unwrap_or(DEFAULT_SOCKS_PORT);
let user_password = if let Some(password) = url.password() {
let username = percent_encoding::percent_decode_str(url.username())
.decode_utf8()
.context("SOCKS5 username is not a valid UTF-8")?
.to_string();
let password = percent_encoding::percent_decode_str(password)
.decode_utf8()
.context("SOCKS5 password is not a valid UTF-8")?
.to_string();
Some((username, password))
} else {
None
};
let socks5_config = Socks5Config {
host,
port,
user_password,
};
Ok(Self::Socks5(socks5_config))
}
scheme => Err(format_err!("Unknown URL scheme {scheme:?}")),
}
}
/// Migrates legacy `socks5_host`, `socks5_port`, `socks5_user` and `socks5_password`
/// config into `proxy_url` if `proxy_url` is unset or empty.
///
/// Unsets `socks5_host`, `socks5_port`, `socks5_user` and `socks5_password` in any case.
async fn migrate_socks_config(sql: &Sql) -> Result<()> {
if sql.get_raw_config("proxy_url").await?.is_none() {
// Load legacy SOCKS5 settings.
if let Some(host) = sql
.get_raw_config("socks5_host")
.await?
.filter(|s| !s.is_empty())
{
let port: u16 = sql
.get_raw_config_int("socks5_port")
.await?
.unwrap_or(DEFAULT_SOCKS_PORT.into()) as u16;
let user = sql.get_raw_config("socks5_user").await?.unwrap_or_default();
let pass = sql
.get_raw_config("socks5_password")
.await?
.unwrap_or_default();
let mut proxy_url = "socks5://".to_string();
if !pass.is_empty() {
proxy_url += &percent_encode(user.as_bytes(), NON_ALPHANUMERIC).to_string();
proxy_url += ":";
proxy_url += &percent_encode(pass.as_bytes(), NON_ALPHANUMERIC).to_string();
proxy_url += "@";
};
proxy_url += &host;
proxy_url += ":";
proxy_url += &port.to_string();
sql.set_raw_config("proxy_url", Some(&proxy_url)).await?;
} else {
sql.set_raw_config("proxy_url", Some("")).await?;
}
let socks5_enabled = sql.get_raw_config("socks5_enabled").await?;
sql.set_raw_config("proxy_enabled", socks5_enabled.as_deref())
.await?;
}
sql.set_raw_config("socks5_enabled", None).await?;
sql.set_raw_config("socks5_host", None).await?;
sql.set_raw_config("socks5_port", None).await?;
sql.set_raw_config("socks5_user", None).await?;
sql.set_raw_config("socks5_password", None).await?;
Ok(())
}
/// Reads proxy configuration from the database.
pub async fn load(context: &Context) -> Result<Option<Self>> {
Self::migrate_socks_config(&context.sql)
.await
.context("Failed to migrate legacy SOCKS config")?;
let enabled = context.get_config_bool(Config::ProxyEnabled).await?;
if !enabled {
return Ok(None);
}
let proxy_url = context
.get_config(Config::ProxyUrl)
.await?
.unwrap_or_default();
let proxy_url = proxy_url
.split_once('\n')
.map_or(proxy_url.clone(), |(first_url, _rest)| {
first_url.to_string()
});
let proxy_config = Self::from_url(&proxy_url).context("Failed to parse proxy URL")?;
Ok(Some(proxy_config))
}
/// If `load_dns_cache` is true, loads cached DNS resolution results.
/// Use this only if the connection is going to be protected with TLS checks.
pub async fn connect(
&self,
context: &Context,
target_host: &str,
target_port: u16,
load_dns_cache: bool,
) -> Result<Box<dyn SessionStream>> {
match self {
ProxyConfig::Http(http_config) => {
let load_cache = false;
let tcp_stream = crate::net::connect_tcp(
context,
&http_config.host,
http_config.port,
load_cache,
)
.await?;
let auth = if let Some((username, password)) = &http_config.user_password {
Some((username.as_str(), password.as_str()))
} else {
None
};
let tunnel_stream = http_tunnel(tcp_stream, target_host, target_port, auth).await?;
Ok(Box::new(tunnel_stream))
}
ProxyConfig::Https(https_config) => {
let load_cache = true;
let tcp_stream = crate::net::connect_tcp(
context,
&https_config.host,
https_config.port,
load_cache,
)
.await?;
let tls_stream = wrap_rustls(&https_config.host, &[], tcp_stream).await?;
let auth = if let Some((username, password)) = &https_config.user_password {
Some((username.as_str(), password.as_str()))
} else {
None
};
let tunnel_stream = http_tunnel(tls_stream, target_host, target_port, auth).await?;
Ok(Box::new(tunnel_stream))
}
ProxyConfig::Socks5(socks5_config) => {
let socks5_stream = socks5_config
.connect(context, target_host, target_port, load_dns_cache)
.await?;
Ok(Box::new(socks5_stream))
}
ProxyConfig::Shadowsocks(ShadowsocksConfig { server_config }) => {
let shadowsocks_context = shadowsocks::context::Context::new_shared(
shadowsocks::config::ServerType::Local,
);
let tcp_stream = {
let server_addr = server_config.addr();
let host = server_addr.host();
let port = server_addr.port();
connect_tcp(context, &host, port, load_dns_cache)
.await
.context("Failed to connect to Shadowsocks proxy")?
};
let shadowsocks_stream = shadowsocks::ProxyClientStream::from_stream(
shadowsocks_context,
tcp_stream,
server_config,
(target_host.to_string(), target_port),
);
Ok(Box::new(shadowsocks_stream))
}
}
}
}
impl fmt::Display for Socks5Config {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(
f,
"host:{},port:{},user_password:{}",
self.host,
self.port,
if let Some(user_password) = self.user_password.clone() {
format!("user: {}, password: ***", user_password.0)
} else {
"user: None".to_string()
}
)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::Config;
use crate::test_utils::TestContext;
#[test]
fn test_socks5_url() {
let proxy_config = ProxyConfig::from_url("socks5://127.0.0.1:9050").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Socks5(Socks5Config {
host: "127.0.0.1".to_string(),
port: 9050,
user_password: None
})
);
let proxy_config = ProxyConfig::from_url("socks5://foo:bar@127.0.0.1:9150").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Socks5(Socks5Config {
host: "127.0.0.1".to_string(),
port: 9150,
user_password: Some(("foo".to_string(), "bar".to_string()))
})
);
let proxy_config = ProxyConfig::from_url("socks5://%66oo:b%61r@127.0.0.1:9150").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Socks5(Socks5Config {
host: "127.0.0.1".to_string(),
port: 9150,
user_password: Some(("foo".to_string(), "bar".to_string()))
})
);
let proxy_config = ProxyConfig::from_url("socks5://127.0.0.1:80").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Socks5(Socks5Config {
host: "127.0.0.1".to_string(),
port: 80,
user_password: None
})
);
let proxy_config = ProxyConfig::from_url("socks5://127.0.0.1").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Socks5(Socks5Config {
host: "127.0.0.1".to_string(),
port: 1080,
user_password: None
})
);
let proxy_config = ProxyConfig::from_url("socks5://127.0.0.1:1080").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Socks5(Socks5Config {
host: "127.0.0.1".to_string(),
port: 1080,
user_password: None
})
);
}
#[test]
fn test_http_url() {
let proxy_config = ProxyConfig::from_url("http://127.0.0.1").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Http(HttpConfig {
host: "127.0.0.1".to_string(),
port: 80,
user_password: None
})
);
let proxy_config = ProxyConfig::from_url("http://127.0.0.1:80").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Http(HttpConfig {
host: "127.0.0.1".to_string(),
port: 80,
user_password: None
})
);
let proxy_config = ProxyConfig::from_url("http://127.0.0.1:443").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Http(HttpConfig {
host: "127.0.0.1".to_string(),
port: 443,
user_password: None
})
);
}
#[test]
fn test_https_url() {
let proxy_config = ProxyConfig::from_url("https://127.0.0.1").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Https(HttpConfig {
host: "127.0.0.1".to_string(),
port: 443,
user_password: None
})
);
let proxy_config = ProxyConfig::from_url("https://127.0.0.1:80").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Https(HttpConfig {
host: "127.0.0.1".to_string(),
port: 80,
user_password: None
})
);
let proxy_config = ProxyConfig::from_url("https://127.0.0.1:443").unwrap();
assert_eq!(
proxy_config,
ProxyConfig::Https(HttpConfig {
host: "127.0.0.1".to_string(),
port: 443,
user_password: None
})
);
}
#[test]
fn test_http_connect_request() {
assert_eq!(http_connect_request("example.org", 143, Some(("aladdin", "opensesame"))), "CONNECT example.org:143 HTTP/1.1\r\nHost: example.org:143\r\nProxy-Authorization: Basic YWxhZGRpbjpvcGVuc2VzYW1l\r\n\r\n");
assert_eq!(
http_connect_request("example.net", 587, None),
"CONNECT example.net:587 HTTP/1.1\r\nHost: example.net:587\r\n\r\n"
);
}
#[test]
fn test_shadowsocks_url() {
// Example URL from <https://shadowsocks.org/doc/sip002.html>.
let proxy_config =
ProxyConfig::from_url("ss://YWVzLTEyOC1nY206dGVzdA@192.168.100.1:8888#Example1")
.unwrap();
assert!(matches!(proxy_config, ProxyConfig::Shadowsocks(_)));
}
#[test]
fn test_invalid_proxy_url() {
assert!(ProxyConfig::from_url("foobar://127.0.0.1:9050").is_err());
assert!(ProxyConfig::from_url("abc").is_err());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_socks5_migration() -> Result<()> {
let t = TestContext::new().await;
// Test that config is migrated on attempt to load even if disabled.
t.set_config(Config::Socks5Host, Some("127.0.0.1")).await?;
t.set_config(Config::Socks5Port, Some("9050")).await?;
let proxy_config = ProxyConfig::load(&t).await?;
// Even though proxy is not enabled, config should be migrated.
assert_eq!(proxy_config, None);
assert_eq!(
t.get_config(Config::ProxyUrl).await?.unwrap(),
"socks5://127.0.0.1:9050"
);
Ok(())
}
// Test SOCKS5 setting migration if proxy was never configured.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_socks5_migration_unconfigured() -> Result<()> {
let t = TestContext::new().await;
// Try to load config to trigger migration.
assert_eq!(ProxyConfig::load(&t).await?, None);
assert_eq!(t.get_config(Config::ProxyEnabled).await?, None);
assert_eq!(
t.get_config(Config::ProxyUrl).await?.unwrap(),
String::new()
);
Ok(())
}
// Test SOCKS5 setting migration if SOCKS5 host is empty.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_socks5_migration_empty() -> Result<()> {
let t = TestContext::new().await;
t.set_config(Config::Socks5Host, Some("")).await?;
// Try to load config to trigger migration.
assert_eq!(ProxyConfig::load(&t).await?, None);
assert_eq!(t.get_config(Config::ProxyEnabled).await?, None);
assert_eq!(
t.get_config(Config::ProxyUrl).await?.unwrap(),
String::new()
);
Ok(())
}
}

View File

@@ -1,4 +1,3 @@
use async_native_tls::TlsStream;
use fast_socks5::client::Socks5Stream;
use std::pin::Pin;
use std::time::Duration;
@@ -17,11 +16,16 @@ impl SessionStream for Box<dyn SessionStream> {
self.as_mut().set_read_timeout(timeout);
}
}
impl<T: SessionStream> SessionStream for TlsStream<T> {
impl<T: SessionStream> SessionStream for async_native_tls::TlsStream<T> {
fn set_read_timeout(&mut self, timeout: Option<Duration>) {
self.get_mut().set_read_timeout(timeout);
}
}
impl<T: SessionStream> SessionStream for tokio_rustls::client::TlsStream<T> {
fn set_read_timeout(&mut self, timeout: Option<Duration>) {
self.get_mut().0.set_read_timeout(timeout);
}
}
impl<T: SessionStream> SessionStream for BufStream<T> {
fn set_read_timeout(&mut self, timeout: Option<Duration>) {
self.get_mut().set_read_timeout(timeout);
@@ -44,6 +48,16 @@ impl<T: SessionStream> SessionStream for Socks5Stream<T> {
self.get_socket_mut().set_read_timeout(timeout)
}
}
impl<T: SessionStream> SessionStream for shadowsocks::ProxyClientStream<T> {
fn set_read_timeout(&mut self, timeout: Option<Duration>) {
self.get_mut().set_read_timeout(timeout)
}
}
impl<T: SessionStream> SessionStream for async_imap::DeflateStream<T> {
fn set_read_timeout(&mut self, timeout: Option<Duration>) {
self.get_mut().set_read_timeout(timeout)
}
}
/// Session stream with a read buffer.
pub(crate) trait SessionBufStream: SessionStream + AsyncBufRead {}

View File

@@ -1,4 +1,5 @@
//! TLS support.
use std::sync::Arc;
use anyhow::Result;
use async_native_tls::{Certificate, Protocol, TlsConnector, TlsStream};
@@ -14,41 +15,42 @@ static LETSENCRYPT_ROOT: Lazy<Certificate> = Lazy::new(|| {
.unwrap()
});
pub fn build_tls(strict_tls: bool, alpns: &[&str]) -> TlsConnector {
let tls_builder = TlsConnector::new()
.min_protocol_version(Some(Protocol::Tlsv12))
.request_alpns(alpns)
.add_root_certificate(LETSENCRYPT_ROOT.clone());
if strict_tls {
tls_builder
} else {
tls_builder
.danger_accept_invalid_hostnames(true)
.danger_accept_invalid_certs(true)
}
}
pub async fn wrap_tls<T: AsyncRead + AsyncWrite + Unpin>(
strict_tls: bool,
hostname: &str,
alpn: &[&str],
stream: T,
) -> Result<TlsStream<T>> {
let tls = build_tls(strict_tls, alpn);
let tls_builder = TlsConnector::new()
.min_protocol_version(Some(Protocol::Tlsv12))
.request_alpns(alpn)
.add_root_certificate(LETSENCRYPT_ROOT.clone());
let tls = if strict_tls {
tls_builder
} else {
tls_builder
.danger_accept_invalid_hostnames(true)
.danger_accept_invalid_certs(true)
};
let tls_stream = tls.connect(hostname, stream).await?;
Ok(tls_stream)
}
#[cfg(test)]
mod tests {
use super::*;
pub async fn wrap_rustls<T: AsyncRead + AsyncWrite + Unpin>(
hostname: &str,
alpn: &[&str],
stream: T,
) -> Result<tokio_rustls::client::TlsStream<T>> {
let mut root_cert_store = rustls::RootCertStore::empty();
root_cert_store.extend(webpki_roots::TLS_SERVER_ROOTS.iter().cloned());
#[test]
fn test_build_tls() {
// we are using some additional root certificates.
// make sure, they do not break construction of TlsConnector
let _ = build_tls(true, &[]);
let _ = build_tls(false, &[]);
}
let mut config = rustls::ClientConfig::builder()
.with_root_certificates(root_cert_store)
.with_no_client_auth();
config.alpn_protocols = alpn.iter().map(|s| s.as_bytes().to_vec()).collect();
let tls = tokio_rustls::TlsConnector::from(Arc::new(config));
let name = rustls_pki_types::ServerName::try_from(hostname)?.to_owned();
let tls_stream = tls.connect(name, stream).await?;
Ok(tls_stream)
}

View File

@@ -2,12 +2,13 @@
use std::collections::HashMap;
use anyhow::Result;
use anyhow::{Context as _, Result};
use percent_encoding::{utf8_percent_encode, NON_ALPHANUMERIC};
use serde::Deserialize;
use crate::config::Config;
use crate::context::Context;
use crate::net::http::post_form;
use crate::net::read_url_blob;
use crate::provider;
use crate::provider::Oauth2Authorizer;
use crate::tools::time;
@@ -60,8 +61,7 @@ pub async fn get_oauth2_url(
addr: &str,
redirect_uri: &str,
) -> Result<Option<String>> {
let socks5_enabled = context.get_config_bool(Config::Socks5Enabled).await?;
if let Some(oauth2) = Oauth2::from_address(context, addr, socks5_enabled).await {
if let Some(oauth2) = Oauth2::from_address(context, addr).await {
context
.sql
.set_raw_config("oauth2_pending_redirect_uri", Some(redirect_uri))
@@ -81,8 +81,7 @@ pub(crate) async fn get_oauth2_access_token(
code: &str,
regenerate: bool,
) -> Result<Option<String>> {
let socks5_enabled = context.get_config_bool(Config::Socks5Enabled).await?;
if let Some(oauth2) = Oauth2::from_address(context, addr, socks5_enabled).await {
if let Some(oauth2) = Oauth2::from_address(context, addr).await {
let lock = context.oauth2_mutex.lock().await;
// read generated token
@@ -159,25 +158,19 @@ pub(crate) async fn get_oauth2_access_token(
// ... and POST
// All OAuth URLs are hardcoded HTTPS URLs,
// so it is safe to load DNS cache.
let load_cache = true;
let client = crate::net::http::get_client(context, load_cache).await?;
let response: Response = match client.post(post_url).form(&post_param).send().await {
Ok(resp) => match resp.json().await {
let response: Response = match post_form(context, post_url, &post_param).await {
Ok(resp) => match serde_json::from_slice(&resp) {
Ok(response) => response,
Err(err) => {
warn!(
context,
"Failed to parse OAuth2 JSON response from {}: error: {}", token_url, err
"Failed to parse OAuth2 JSON response from {token_url}: {err:#}."
);
return Ok(None);
}
},
Err(err) => {
warn!(context, "Error calling OAuth2 at {}: {:?}", token_url, err);
warn!(context, "Error calling OAuth2 at {token_url}: {err:#}.");
return Ok(None);
}
};
@@ -236,8 +229,7 @@ pub(crate) async fn get_oauth2_addr(
addr: &str,
code: &str,
) -> Result<Option<String>> {
let socks5_enabled = context.get_config_bool(Config::Socks5Enabled).await?;
let oauth2 = match Oauth2::from_address(context, addr, socks5_enabled).await {
let oauth2 = match Oauth2::from_address(context, addr).await {
Some(o) => o,
None => return Ok(None),
};
@@ -246,11 +238,20 @@ pub(crate) async fn get_oauth2_addr(
}
if let Some(access_token) = get_oauth2_access_token(context, addr, code, false).await? {
let addr_out = oauth2.get_addr(context, &access_token).await;
let addr_out = match oauth2.get_addr(context, &access_token).await {
Ok(addr) => addr,
Err(err) => {
warn!(context, "Error getting addr: {err:#}.");
None
}
};
if addr_out.is_none() {
// regenerate
if let Some(access_token) = get_oauth2_access_token(context, addr, code, true).await? {
Ok(oauth2.get_addr(context, &access_token).await)
Ok(oauth2
.get_addr(context, &access_token)
.await
.unwrap_or_default())
} else {
Ok(None)
}
@@ -263,8 +264,9 @@ pub(crate) async fn get_oauth2_addr(
}
impl Oauth2 {
async fn from_address(context: &Context, addr: &str, skip_mx: bool) -> Option<Self> {
async fn from_address(context: &Context, addr: &str) -> Option<Self> {
let addr_normalized = normalize_addr(addr);
let skip_mx = true;
if let Some(domain) = addr_normalized
.find('@')
.map(|index| addr_normalized.split_at(index + 1).1)
@@ -282,7 +284,7 @@ impl Oauth2 {
None
}
async fn get_addr(&self, context: &Context, access_token: &str) -> Option<String> {
async fn get_addr(&self, context: &Context, access_token: &str) -> Result<Option<String>> {
let userinfo_url = self.get_userinfo.unwrap_or("");
let userinfo_url = replace_in_uri(userinfo_url, "$ACCESS_TOKEN", access_token);
@@ -294,44 +296,21 @@ impl Oauth2 {
// "picture": "https://lh4.googleusercontent.com/-Gj5jh_9R0BY/AAAAAAAAAAI/AAAAAAAAAAA/IAjtjfjtjNA/photo.jpg"
// }
// All OAuth URLs are hardcoded HTTPS URLs,
// so it is safe to load DNS cache.
let load_cache = true;
let client = match crate::net::http::get_client(context, load_cache).await {
Ok(cl) => cl,
Err(err) => {
warn!(context, "failed to get HTTP client: {}", err);
return None;
}
};
let response = match client.get(userinfo_url).send().await {
Ok(response) => response,
Err(err) => {
warn!(context, "failed to get userinfo: {}", err);
return None;
}
};
let response: Result<HashMap<String, serde_json::Value>, _> = response.json().await;
let parsed = match response {
Ok(parsed) => parsed,
Err(err) => {
warn!(context, "Error getting userinfo: {}", err);
return None;
}
};
let response = read_url_blob(context, &userinfo_url).await?;
let parsed: HashMap<String, serde_json::Value> =
serde_json::from_slice(&response.blob).context("Error getting userinfo")?;
// CAVE: serde_json::Value.as_str() removes the quotes of json-strings
// but serde_json::Value.to_string() does not!
if let Some(addr) = parsed.get("email") {
if let Some(s) = addr.as_str() {
Some(s.to_string())
Ok(Some(s.to_string()))
} else {
warn!(context, "E-mail in userinfo is not a string: {}", addr);
None
Ok(None)
}
} else {
warn!(context, "E-mail missing in userinfo.");
None
Ok(None)
}
}
}
@@ -385,38 +364,20 @@ mod tests {
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_oauth_from_address() {
let t = TestContext::new().await;
assert_eq!(
Oauth2::from_address(&t, "hello@gmail.com", false).await,
Some(OAUTH2_GMAIL)
);
assert_eq!(
Oauth2::from_address(&t, "hello@googlemail.com", false).await,
Some(OAUTH2_GMAIL)
);
assert_eq!(
Oauth2::from_address(&t, "hello@yandex.com", false).await,
Some(OAUTH2_YANDEX)
);
assert_eq!(
Oauth2::from_address(&t, "hello@yandex.ru", false).await,
Some(OAUTH2_YANDEX)
);
assert_eq!(Oauth2::from_address(&t, "hello@web.de", false).await, None);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_oauth_from_mx() {
// youtube staff seems to use "google workspace with oauth2", figures this out by MX lookup
let t = TestContext::new().await;
// Delta Chat does not have working Gmail client ID anymore.
assert_eq!(Oauth2::from_address(&t, "hello@gmail.com").await, None);
assert_eq!(Oauth2::from_address(&t, "hello@googlemail.com").await, None);
assert_eq!(
Oauth2::from_address(&t, "hello@youtube.com", false).await,
Some(OAUTH2_GMAIL)
Oauth2::from_address(&t, "hello@yandex.com").await,
Some(OAUTH2_YANDEX)
);
// without MX lookup, we would not know as youtube.com is not in our provider-db
assert_eq!(
Oauth2::from_address(&t, "hello@youtube.com", true).await,
None
Oauth2::from_address(&t, "hello@yandex.ru").await,
Some(OAUTH2_YANDEX)
);
assert_eq!(Oauth2::from_address(&t, "hello@web.de").await, None);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -432,11 +393,11 @@ mod tests {
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_get_oauth2_url() {
let ctx = TestContext::new().await;
let addr = "dignifiedquire@gmail.com";
let addr = "example@yandex.com";
let redirect_uri = "chat.delta:/com.b44t.messenger";
let res = get_oauth2_url(&ctx.ctx, addr, redirect_uri).await.unwrap();
assert_eq!(res, Some("https://accounts.google.com/o/oauth2/auth?client_id=959970109878%2D4mvtgf6feshskf7695nfln6002mom908%2Eapps%2Egoogleusercontent%2Ecom&redirect_uri=chat%2Edelta%3A%2Fcom%2Eb44t%2Emessenger&response_type=code&scope=https%3A%2F%2Fmail.google.com%2F%20email&access_type=offline".into()));
assert_eq!(res, Some("https://oauth.yandex.com/authorize?client_id=c4d0b6735fc8420a816d7e1303469341&response_type=code&scope=mail%3Aimap_full%20mail%3Asmtp&force_confirm=true".into()));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]

View File

@@ -253,7 +253,7 @@ impl Context {
.secret_key(secret_key)
.alpns(vec![GOSSIP_ALPN.to_vec()])
.relay_mode(relay_mode)
.bind(0)
.bind()
.await?;
// create gossip
@@ -265,7 +265,6 @@ impl Context {
// Shuts down on deltachat shutdown
tokio::spawn(endpoint_loop(context, endpoint.clone(), gossip.clone()));
tokio::spawn(gossip_direct_address_loop(endpoint.clone(), gossip.clone()));
Ok(Iroh {
endpoint,
@@ -285,15 +284,6 @@ impl Context {
}
}
/// Loop to update direct addresses of the gossip.
async fn gossip_direct_address_loop(endpoint: Endpoint, gossip: Gossip) -> Result<()> {
let mut stream = endpoint.direct_addresses();
while let Some(addrs) = stream.next().await {
gossip.update_direct_addresses(&addrs)?;
}
Ok(())
}
/// Cache a peers [NodeId] for one topic.
pub(crate) async fn iroh_add_peer_for_topic(
ctx: &Context,
@@ -442,6 +432,13 @@ pub(crate) async fn create_iroh_header(
async fn endpoint_loop(context: Context, endpoint: Endpoint, gossip: Gossip) {
while let Some(conn) = endpoint.accept().await {
let conn = match conn.accept() {
Ok(conn) => conn,
Err(err) => {
warn!(context, "Failed to accept iroh connection: {err:#}.");
continue;
}
};
info!(context, "IROH_REALTIME: accepting iroh connection");
let gossip = gossip.clone();
let context = context.clone();
@@ -579,6 +576,13 @@ mod tests {
.unwrap();
bob.recv_msg_trash(&alice.pop_sent_msg().await).await;
loop {
let event = bob.evtracker.recv().await.unwrap();
if let EventType::WebxdcRealtimeAdvertisementReceived { msg_id } = event.typ {
assert!(msg_id == alice_webxdc.id);
break;
}
}
let bob_iroh = bob.get_or_try_init_peer_channel().await.unwrap();
// Bob adds alice to gossip peers.

View File

@@ -343,7 +343,7 @@ impl Peerstate {
}
/// Updates peerstate according to the given `Autocrypt` header.
pub fn apply_header(&mut self, header: &Aheader, message_time: i64) {
pub fn apply_header(&mut self, context: &Context, header: &Aheader, message_time: i64) {
if !addr_cmp(&self.addr, &header.addr) {
return;
}
@@ -362,6 +362,13 @@ impl Peerstate {
self.public_key = Some(header.public_key.clone());
self.recalc_fingerprint();
}
} else {
warn!(
context,
"Ignoring outdated Autocrypt header because message_time={} < last_seen={}.",
message_time,
self.last_seen
);
}
}
@@ -766,23 +773,65 @@ pub(crate) async fn maybe_do_aeap_transition(
// If the from addr is different from the peerstate address we know,
// we may want to do an AEAP transition.
if !addr_cmp(&peerstate.addr, &mime_parser.from.addr)
// Check if it's a chat message; we do this to avoid
// some accidental transitions if someone writes from multiple
// addresses with an MUA.
&& mime_parser.has_chat_version()
// Check if the message is encrypted and signed correctly. If it's not encrypted, it's
// probably from a new contact sharing the same key.
&& !mime_parser.signatures.is_empty()
// Check if the From: address was also in the signed part of the email.
// Without this check, an attacker could replay a message from Alice
// to Bob. Then Bob's device would do an AEAP transition from Alice's
// to the attacker's address, allowing for easier phishing.
&& mime_parser.from_is_signed
// DC avoids sending messages with the same timestamp, that's why `>` is here unlike in
// `Peerstate::apply_header()`.
&& info.message_time > peerstate.last_seen
{
if !addr_cmp(&peerstate.addr, &mime_parser.from.addr) {
// Check if it's a chat message; we do this to avoid
// some accidental transitions if someone writes from multiple
// addresses with an MUA.
if !mime_parser.has_chat_version() {
info!(
context,
"Not doing AEAP from {} to {} because the message is not a chat message.",
&peerstate.addr,
&mime_parser.from.addr
);
return Ok(());
}
// Check if the message is encrypted and signed correctly. If it's not encrypted, it's
// probably from a new contact sharing the same key.
if mime_parser.signatures.is_empty() {
info!(
context,
"Not doing AEAP from {} to {} because the message is not encrypted and signed.",
&peerstate.addr,
&mime_parser.from.addr
);
return Ok(());
}
// Check if the From: address was also in the signed part of the email.
// Without this check, an attacker could replay a message from Alice
// to Bob. Then Bob's device would do an AEAP transition from Alice's
// to the attacker's address, allowing for easier phishing.
if !mime_parser.from_is_signed {
info!(
context,
"Not doing AEAP from {} to {} because From: is not signed.",
&peerstate.addr,
&mime_parser.from.addr
);
return Ok(());
}
// DC avoids sending messages with the same timestamp, that's why messages
// with equal timestamps are ignored here unlike in `Peerstate::apply_header()`.
if info.message_time <= peerstate.last_seen {
info!(
context,
"Not doing AEAP from {} to {} because {} < {}.",
&peerstate.addr,
&mime_parser.from.addr,
info.message_time,
peerstate.last_seen
);
return Ok(());
}
info!(
context,
"Doing AEAP transition from {} to {}.", &peerstate.addr, &mime_parser.from.addr
);
let info = &mut mime_parser.decryption_info;
let peerstate = info.peerstate.as_mut().context("no peerstate??")?;
// Add info messages to chats with this (verified) contact
@@ -800,7 +849,7 @@ pub(crate) async fn maybe_do_aeap_transition(
let header = info.autocrypt_header.as_ref().context(
"Internal error: Tried to do an AEAP transition without an autocrypt header??",
)?;
peerstate.apply_header(header, info.message_time);
peerstate.apply_header(context, header, info.message_time);
peerstate
.save_to_db_ex(&context.sql, Some(&old_addr))
@@ -979,6 +1028,8 @@ mod tests {
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_peerstate_degrade_reordering() {
let ctx = crate::test_utils::TestContext::new().await;
let addr = "example@example.org";
let pub_key = alice_keypair().public;
let header = Aheader::new(addr.to_string(), pub_key, EncryptPreference::Mutual);
@@ -1003,7 +1054,7 @@ mod tests {
fingerprint_changed: false,
};
peerstate.apply_header(&header, 100);
peerstate.apply_header(&ctx, &header, 100);
assert_eq!(peerstate.prefer_encrypt, EncryptPreference::Mutual);
peerstate.degrade_encryption(300);
@@ -1011,11 +1062,11 @@ mod tests {
// This has message time 200, while encryption was degraded at timestamp 300.
// Because of reordering, header should not be applied.
peerstate.apply_header(&header, 200);
peerstate.apply_header(&ctx, &header, 200);
assert_eq!(peerstate.prefer_encrypt, EncryptPreference::Reset);
// Same header will be applied in the future.
peerstate.apply_header(&header, 300);
peerstate.apply_header(&ctx, &header, 300);
assert_eq!(peerstate.prefer_encrypt, EncryptPreference::Mutual);
}
}

View File

@@ -14,9 +14,7 @@ use pgp::composed::{
use pgp::crypto::ecc_curve::ECCCurve;
use pgp::crypto::hash::HashAlgorithm;
use pgp::crypto::sym::SymmetricKeyAlgorithm;
use pgp::types::{
CompressionAlgorithm, KeyTrait, Mpi, PublicKeyTrait, SecretKeyTrait, StringToKey,
};
use pgp::types::{CompressionAlgorithm, KeyTrait, Mpi, PublicKeyTrait, StringToKey};
use rand::{thread_rng, CryptoRng, Rng};
use tokio::runtime::Handle;
@@ -43,7 +41,7 @@ enum SignedPublicKeyOrSubkey<'a> {
Subkey(&'a SignedPublicSubKey),
}
impl<'a> KeyTrait for SignedPublicKeyOrSubkey<'a> {
impl KeyTrait for SignedPublicKeyOrSubkey<'_> {
fn fingerprint(&self) -> Vec<u8> {
match self {
Self::Key(k) => k.fingerprint(),
@@ -66,7 +64,7 @@ impl<'a> KeyTrait for SignedPublicKeyOrSubkey<'a> {
}
}
impl<'a> PublicKeyTrait for SignedPublicKeyOrSubkey<'a> {
impl PublicKeyTrait for SignedPublicKeyOrSubkey<'_> {
fn verify_signature(
&self,
hash: HashAlgorithm,
@@ -135,9 +133,6 @@ pub fn split_armored_data(buf: &[u8]) -> Result<(BlockType, BTreeMap<String, Str
/// keys together as they are one unit.
#[derive(Debug, Clone, Eq, PartialEq)]
pub struct KeyPair {
/// Email address.
pub addr: EmailAddress,
/// Public key.
pub public: SignedPublicKey,
@@ -145,6 +140,18 @@ pub struct KeyPair {
pub secret: SignedSecretKey,
}
impl KeyPair {
/// Creates new keypair from a secret key.
///
/// Public key is split off the secret key.
pub fn new(secret: SignedSecretKey) -> Result<Self> {
use crate::key::DcSecretKey;
let public = secret.split_public_key()?;
Ok(Self { public, secret })
}
}
/// Create a new key pair.
///
/// Both secret and public key consist of signing primary key and encryption subkey
@@ -201,19 +208,12 @@ pub(crate) fn create_keypair(addr: EmailAddress, keygen_type: KeyGenType) -> Res
.verify()
.context("invalid secret key generated")?;
let public_key = secret_key
.public_key()
.sign(&secret_key, || "".into())
.context("failed to sign public key")?;
public_key
let key_pair = KeyPair::new(secret_key)?;
key_pair
.public
.verify()
.context("invalid public key generated")?;
Ok(KeyPair {
addr,
public: public_key,
secret: secret_key,
})
Ok(key_pair)
}
/// Select public key or subkey to use for encryption.

View File

@@ -1,6 +1,6 @@
//! [Provider database](https://providers.delta.chat/) module.
mod data;
pub(crate) mod data;
use anyhow::Result;
use deltachat_contact_tools::EmailAddress;

View File

@@ -509,6 +509,8 @@ static P_FREENET_DE: Provider = Provider {
overview_page: "https://providers.delta.chat/freenet-de",
server: &[
Server { protocol: Imap, socket: Ssl, hostname: "mx.freenet.de", port: 993, username_pattern: Email },
Server { protocol: Imap, socket: Starttls, hostname: "mx.freenet.de", port: 143, username_pattern: Email },
Server { protocol: Smtp, socket: Ssl, hostname: "mx.freenet.de", port: 465, username_pattern: Email },
Server { protocol: Smtp, socket: Starttls, hostname: "mx.freenet.de", port: 587, username_pattern: Email },
],
opt: ProviderOptions::new(),
@@ -532,7 +534,7 @@ static P_GMAIL: Provider = Provider {
..ProviderOptions::new()
},
config_defaults: None,
oauth2_authorizer: Some(Oauth2Authorizer::Gmail),
oauth2_authorizer: None,
};
// gmx.net.md: gmx.net, gmx.de, gmx.at, gmx.ch, gmx.org, gmx.eu, gmx.info, gmx.biz, gmx.com
@@ -874,6 +876,20 @@ static P_MEHL_CLOUD: Provider = Provider {
after_login_hint: "",
overview_page: "https://providers.delta.chat/mehl-cloud",
server: &[
Server {
protocol: Imap,
socket: Ssl,
hostname: "mehl.cloud",
port: 443,
username_pattern: Email,
},
Server {
protocol: Smtp,
socket: Ssl,
hostname: "mehl.cloud",
port: 443,
username_pattern: Email,
},
Server {
protocol: Imap,
socket: Ssl,
@@ -924,6 +940,41 @@ static P_MEHL_STORE: Provider = Provider {
oauth2_authorizer: None,
};
// migadu.md: migadu.com
static P_MIGADU: Provider = Provider {
id: "migadu",
status: Status::Ok,
before_login_hint: "",
after_login_hint: "",
overview_page: "https://providers.delta.chat/migadu",
server: &[
Server {
protocol: Imap,
socket: Ssl,
hostname: "imap.migadu.com",
port: 993,
username_pattern: Email,
},
Server {
protocol: Smtp,
socket: Ssl,
hostname: "smtp.migadu.com",
port: 465,
username_pattern: Email,
},
Server {
protocol: Smtp,
socket: Starttls,
hostname: "smtp.migadu.com",
port: 587,
username_pattern: Email,
},
],
opt: ProviderOptions::new(),
config_defaults: None,
oauth2_authorizer: None,
};
// nauta.cu.md: nauta.cu
static P_NAUTA_CU: Provider = Provider {
id: "nauta.cu",
@@ -1009,6 +1060,20 @@ static P_NINE_TESTRUN_ORG: Provider = Provider {
after_login_hint: "",
overview_page: "https://providers.delta.chat/nine-testrun-org",
server: &[
Server {
protocol: Imap,
socket: Ssl,
hostname: "nine.testrun.org",
port: 443,
username_pattern: Email,
},
Server {
protocol: Smtp,
socket: Ssl,
hostname: "nine.testrun.org",
port: 443,
username_pattern: Email,
},
Server {
protocol: Imap,
socket: Ssl,
@@ -1037,20 +1102,6 @@ static P_NINE_TESTRUN_ORG: Provider = Provider {
port: 587,
username_pattern: Email,
},
Server {
protocol: Imap,
socket: Ssl,
hostname: "nine.testrun.org",
port: 443,
username_pattern: Email,
},
Server {
protocol: Smtp,
socket: Ssl,
hostname: "nine.testrun.org",
port: 443,
username_pattern: Email,
},
],
opt: ProviderOptions::new(),
config_defaults: Some(&[ConfigDefault {
@@ -1144,7 +1195,7 @@ static P_OUVATON_COOP: Provider = Provider {
oauth2_authorizer: None,
};
// posteo.md: posteo.de, posteo.af, posteo.at, posteo.be, posteo.ca, posteo.ch, posteo.cl, posteo.co, posteo.co.uk, posteo.com.br, posteo.cr, posteo.cz, posteo.dk, posteo.ee, posteo.es, posteo.eu, posteo.fi, posteo.gl, posteo.gr, posteo.hn, posteo.hr, posteo.hu, posteo.ie, posteo.in, posteo.is, posteo.it, posteo.jp, posteo.la, posteo.li, posteo.lt, posteo.lu, posteo.me, posteo.mx, posteo.my, posteo.net, posteo.nl, posteo.no, posteo.nz, posteo.org, posteo.pe, posteo.pl, posteo.pm, posteo.pt, posteo.ro, posteo.ru, posteo.se, posteo.sg, posteo.si, posteo.tn, posteo.uk, posteo.us
// posteo.md: posteo.de, posteo.af, posteo.at, posteo.be, posteo.ca, posteo.ch, posteo.cl, posteo.co, posteo.co.uk, posteo.com, posteo.com.br, posteo.cr, posteo.cz, posteo.dk, posteo.ee, posteo.es, posteo.eu, posteo.fi, posteo.gl, posteo.gr, posteo.hn, posteo.hr, posteo.hu, posteo.ie, posteo.in, posteo.is, posteo.it, posteo.jp, posteo.la, posteo.li, posteo.lt, posteo.lu, posteo.me, posteo.mx, posteo.my, posteo.net, posteo.nl, posteo.no, posteo.nz, posteo.org, posteo.pe, posteo.pl, posteo.pm, posteo.pt, posteo.ro, posteo.ru, posteo.se, posteo.sg, posteo.si, posteo.tn, posteo.uk, posteo.us
static P_POSTEO: Provider = Provider {
id: "posteo",
status: Status::Ok,
@@ -1518,11 +1569,26 @@ static P_TUTANOTA: Provider = Provider {
// ukr.net.md: ukr.net
static P_UKR_NET: Provider = Provider {
id: "ukr.net",
status: Status::Ok,
before_login_hint: "",
status: Status::Preparation,
before_login_hint: "You must allow IMAP access to your account before you can login.",
after_login_hint: "",
overview_page: "https://providers.delta.chat/ukr-net",
server: &[],
server: &[
Server {
protocol: Imap,
socket: Ssl,
hostname: "imap.ukr.net",
port: 993,
username_pattern: Email,
},
Server {
protocol: Smtp,
socket: Ssl,
hostname: "smtp.ukr.net",
port: 465,
username_pattern: Email,
},
],
opt: ProviderOptions::new(),
config_defaults: None,
oauth2_authorizer: None,
@@ -1804,7 +1870,7 @@ static P_ZOHO: Provider = Provider {
oauth2_authorizer: None,
};
pub(crate) static PROVIDER_DATA: [(&str, &Provider); 531] = [
pub(crate) static PROVIDER_DATA: [(&str, &Provider); 533] = [
("163.com", &P_163),
("aktivix.org", &P_AKTIVIX_ORG),
("aliyun.com", &P_ALIYUN),
@@ -2177,6 +2243,7 @@ pub(crate) static PROVIDER_DATA: [(&str, &Provider); 531] = [
("ente.quest", &P_MEHL_STORE),
("ente.cfd", &P_MEHL_STORE),
("nein.jetzt", &P_MEHL_STORE),
("migadu.com", &P_MIGADU),
("nauta.cu", &P_NAUTA_CU),
("naver.com", &P_NAVER),
("nine.testrun.org", &P_NINE_TESTRUN_ORG),
@@ -2197,6 +2264,7 @@ pub(crate) static PROVIDER_DATA: [(&str, &Provider); 531] = [
("posteo.cl", &P_POSTEO),
("posteo.co", &P_POSTEO),
("posteo.co.uk", &P_POSTEO),
("posteo.com", &P_POSTEO),
("posteo.com.br", &P_POSTEO),
("posteo.cr", &P_POSTEO),
("posteo.cz", &P_POSTEO),
@@ -2379,6 +2447,7 @@ pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> =
("mailo.com", &P_MAILO_COM),
("mehl.cloud", &P_MEHL_CLOUD),
("mehl.store", &P_MEHL_STORE),
("migadu", &P_MIGADU),
("nauta.cu", &P_NAUTA_CU),
("naver", &P_NAVER),
("nine.testrun.org", &P_NINE_TESTRUN_ORG),
@@ -2417,4 +2486,4 @@ pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> =
});
pub static _PROVIDER_UPDATED: Lazy<chrono::NaiveDate> =
Lazy::new(|| chrono::NaiveDate::from_ymd_opt(2024, 8, 14).unwrap());
Lazy::new(|| chrono::NaiveDate::from_ymd_opt(2024, 9, 13).unwrap());

View File

@@ -61,16 +61,13 @@ impl PushSubscriber {
return Ok(());
};
let load_cache = true;
let response = http::get_client(context, load_cache)
.await?
.post("https://notifications.delta.chat/register")
.body(format!("{{\"token\":\"{token}\"}}"))
.send()
.await?;
let response_status = response.status();
if response_status.is_success() {
if http::post_string(
context,
"https://notifications.delta.chat/register",
format!("{{\"token\":\"{token}\"}}"),
)
.await?
{
state.heartbeat_subscribed = true;
}
Ok(())

437
src/qr.rs
View File

@@ -7,11 +7,11 @@ use anyhow::{anyhow, bail, ensure, Context as _, Result};
pub use dclogin_scheme::LoginOptions;
use deltachat_contact_tools::{addr_normalize, may_be_valid_addr, ContactAddress};
use once_cell::sync::Lazy;
use percent_encoding::percent_decode_str;
use percent_encoding::{percent_decode_str, percent_encode, NON_ALPHANUMERIC};
use serde::Deserialize;
use self::dclogin_scheme::configure_from_login_qr;
use crate::chat::{get_chat_id_by_grpid, ChatIdBlocked};
use crate::chat::ChatIdBlocked;
use crate::config::Config;
use crate::constants::Blocked;
use crate::contact::{Contact, ContactId, Origin};
@@ -19,10 +19,11 @@ use crate::context::Context;
use crate::events::EventType;
use crate::key::Fingerprint;
use crate::message::Message;
use crate::net::http::post_empty;
use crate::net::proxy::DEFAULT_SOCKS_PORT;
use crate::peerstate::Peerstate;
use crate::token;
use crate::tools::validate_id;
use iroh_old as iroh;
const OPENPGP4FPR_SCHEME: &str = "OPENPGP4FPR:"; // yes: uppercase
const IDELTACHAT_SCHEME: &str = "https://i.delta.chat/#";
@@ -30,15 +31,13 @@ const IDELTACHAT_NOSLASH_SCHEME: &str = "https://i.delta.chat#";
const DCACCOUNT_SCHEME: &str = "DCACCOUNT:";
pub(super) const DCLOGIN_SCHEME: &str = "DCLOGIN:";
const DCWEBRTC_SCHEME: &str = "DCWEBRTC:";
const TG_SOCKS_SCHEME: &str = "https://t.me/socks";
const MAILTO_SCHEME: &str = "mailto:";
const MATMSG_SCHEME: &str = "MATMSG:";
const VCARD_SCHEME: &str = "BEGIN:VCARD";
const SMTP_SCHEME: &str = "SMTP:";
const HTTP_SCHEME: &str = "http://";
const HTTPS_SCHEME: &str = "https://";
/// Legacy backup transfer based on iroh 0.4.
pub(crate) const DCBACKUP_SCHEME: &str = "DCBACKUP:";
const SHADOWSOCKS_SCHEME: &str = "ss://";
/// Backup transfer based on iroh-net.
pub(crate) const DCBACKUP2_SCHEME: &str = "DCBACKUP2:";
@@ -110,20 +109,6 @@ pub enum Qr {
domain: String,
},
/// Provides a backup that can be retrieved using legacy iroh 0.4.
///
/// This contains all the data needed to connect to a device and download a backup from
/// it to configure the receiving device with the same account.
Backup {
/// Printable version of the provider information.
///
/// This is the printable version of a `sendme` ticket, which contains all the
/// information to connect to and authenticate a backup provider.
///
/// The format is somewhat opaque, but `sendme` can deserialise this.
ticket: iroh::provider::Ticket,
},
/// Provides a backup that can be retrieved using iroh-net based backup transfer protocol.
Backup2 {
/// Iroh node address.
@@ -142,6 +127,28 @@ pub enum Qr {
instance_pattern: String,
},
/// Ask the user if they want to use the given proxy.
///
/// Note that HTTP(S) URLs without a path
/// and query parameters are treated as HTTP(S) proxy URL.
/// UI may want to still offer to open the URL
/// in the browser if QR code contents
/// starts with `http://` or `https://`
/// and the QR code was not scanned from
/// the proxy configuration screen.
Proxy {
/// Proxy URL.
///
/// This is the URL that is going to be added.
url: String,
/// Host extracted from the URL to display in the UI.
host: String,
/// Port extracted from the URL to display in the UI.
port: u16,
},
/// Contact address is scanned.
///
/// Optionally, a draft message could be provided.
@@ -277,8 +284,10 @@ pub async fn check_qr(context: &Context, qr: &str) -> Result<Qr> {
dclogin_scheme::decode_login(qr)?
} else if starts_with_ignore_case(qr, DCWEBRTC_SCHEME) {
decode_webrtc_instance(context, qr)?
} else if starts_with_ignore_case(qr, DCBACKUP_SCHEME) {
decode_backup(qr)?
} else if starts_with_ignore_case(qr, TG_SOCKS_SCHEME) {
decode_tg_socks_proxy(context, qr)?
} else if qr.starts_with(SHADOWSOCKS_SCHEME) {
decode_shadowsocks_proxy(qr)?
} else if starts_with_ignore_case(qr, DCBACKUP2_SCHEME) {
decode_backup2(qr)?
} else if qr.starts_with(MAILTO_SCHEME) {
@@ -289,9 +298,44 @@ pub async fn check_qr(context: &Context, qr: &str) -> Result<Qr> {
decode_matmsg(context, qr).await?
} else if qr.starts_with(VCARD_SCHEME) {
decode_vcard(context, qr).await?
} else if qr.starts_with(HTTP_SCHEME) || qr.starts_with(HTTPS_SCHEME) {
Qr::Url {
url: qr.to_string(),
} else if let Ok(url) = url::Url::parse(qr) {
match url.scheme() {
"socks5" => Qr::Proxy {
url: qr.to_string(),
host: url.host_str().context("URL has no host")?.to_string(),
port: url.port().unwrap_or(DEFAULT_SOCKS_PORT),
},
"http" | "https" => {
// Parsing with a non-standard scheme
// is a hack to work around the `url` crate bug
// <https://github.com/servo/rust-url/issues/957>.
let url = if let Some(rest) = qr.strip_prefix("http://") {
url::Url::parse(&format!("foobarbaz://{rest}"))?
} else if let Some(rest) = qr.strip_prefix("https://") {
url::Url::parse(&format!("foobarbaz://{rest}"))?
} else {
// Should not happen.
url
};
if url.port().is_none() | (url.path() != "") | url.query().is_some() {
// URL without a port, with a path or query cannot be a proxy URL.
Qr::Url {
url: qr.to_string(),
}
} else {
Qr::Proxy {
url: qr.to_string(),
host: url.host_str().context("URL has no host")?.to_string(),
port: url
.port_or_known_default()
.context("HTTP(S) URLs are guaranteed to return Some port")?,
}
}
}
_ => Qr::Url {
url: qr.to_string(),
},
}
} else {
Qr::Text {
@@ -301,7 +345,7 @@ pub async fn check_qr(context: &Context, qr: &str) -> Result<Qr> {
Ok(qrcode)
}
/// Formats the text of the [`Qr::Backup`] variant.
/// Formats the text of the [`Qr::Backup2`] variant.
///
/// This is the inverse of [`check_qr`] for that variant only.
///
@@ -309,7 +353,6 @@ pub async fn check_qr(context: &Context, qr: &str) -> Result<Qr> {
/// into `FromStr`.
pub fn format_backup(qr: &Qr) -> Result<String> {
match qr {
Qr::Backup { ref ticket } => Ok(format!("{DCBACKUP_SCHEME}{ticket}")),
Qr::Backup2 {
ref node_addr,
ref auth_token,
@@ -496,7 +539,7 @@ async fn decode_ideltachat(context: &Context, prefix: &str, qr: &str) -> Result<
let qr = qr.replacen('&', "#", 1);
decode_openpgp(context, &qr)
.await
.context("failed to decode {prefix} QR code")
.with_context(|| format!("failed to decode {prefix} QR code"))
}
/// scheme: `DCACCOUNT:https://example.org/new_email?t=1w_7wDjgjelxeX884x96v3`
@@ -539,16 +582,55 @@ fn decode_webrtc_instance(_context: &Context, qr: &str) -> Result<Qr> {
}
}
/// Decodes a [`DCBACKUP_SCHEME`] QR code.
///
/// The format of this scheme is `DCBACKUP:<encoded ticket>`. The encoding is the
/// [`iroh::provider::Ticket`]'s `Display` impl.
fn decode_backup(qr: &str) -> Result<Qr> {
let payload = qr
.strip_prefix(DCBACKUP_SCHEME)
.ok_or_else(|| anyhow!("invalid DCBACKUP scheme"))?;
let ticket: iroh::provider::Ticket = payload.parse().context("invalid DCBACKUP payload")?;
Ok(Qr::Backup { ticket })
/// scheme: `https://t.me/socks?server=foo&port=123` or `https://t.me/socks?server=1.2.3.4&port=123`
fn decode_tg_socks_proxy(_context: &Context, qr: &str) -> Result<Qr> {
let url = url::Url::parse(qr).context("Invalid t.me/socks url")?;
let mut host: Option<String> = None;
let mut port: u16 = DEFAULT_SOCKS_PORT;
let mut user: Option<String> = None;
let mut pass: Option<String> = None;
for (key, value) in url.query_pairs() {
if key == "server" {
host = Some(value.to_string());
} else if key == "port" {
port = value.parse().unwrap_or(DEFAULT_SOCKS_PORT);
} else if key == "user" {
user = Some(value.to_string());
} else if key == "pass" {
pass = Some(value.to_string());
}
}
let Some(host) = host else {
bail!("Bad t.me/socks url: {:?}", url);
};
let mut url = "socks5://".to_string();
if let Some(pass) = pass {
url += &percent_encode(user.unwrap_or_default().as_bytes(), NON_ALPHANUMERIC).to_string();
url += ":";
url += &percent_encode(pass.as_bytes(), NON_ALPHANUMERIC).to_string();
url += "@";
};
url += &host;
url += ":";
url += &port.to_string();
Ok(Qr::Proxy { url, host, port })
}
/// Decodes `ss://` URLs for Shadowsocks proxies.
fn decode_shadowsocks_proxy(qr: &str) -> Result<Qr> {
let server_config = shadowsocks::config::ServerConfig::from_url(qr)?;
let addr = server_config.addr();
let host = addr.host().to_string();
let port = addr.port();
Ok(Qr::Proxy {
url: qr.to_string(),
host,
port,
})
}
/// Decodes a [`DCBACKUP2_SCHEME`] QR code.
@@ -594,21 +676,8 @@ async fn set_account_from_qr(context: &Context, qr: &str) -> Result<()> {
bail!("DCACCOUNT QR codes must use HTTPS scheme");
}
// As only HTTPS is used, it is safe to load DNS cache.
let load_cache = true;
let response = crate::net::http::get_client(context, load_cache)
.await?
.post(url_str)
.send()
.await?;
let response_status = response.status();
let response_text = response
.text()
.await
.context("Cannot create account, request failed: empty response")?;
if response_status.is_success() {
let (response_text, response_success) = post_empty(context, url_str).await?;
if response_success {
let CreateAccountSuccessResponse { password, email } = serde_json::from_str(&response_text)
.with_context(|| {
format!("Cannot create account, response is malformed:\n{response_text:?}")
@@ -649,6 +718,23 @@ pub async fn set_config_from_qr(context: &Context, qr: &str) -> Result<()> {
.set_config_internal(Config::WebrtcInstance, Some(&instance_pattern))
.await?;
}
Qr::Proxy { url, .. } => {
let old_proxy_url_value = context
.get_config(Config::ProxyUrl)
.await?
.unwrap_or_default();
let proxy_urls: Vec<&str> = std::iter::once(url.as_str())
.chain(
old_proxy_url_value
.split('\n')
.filter(|s| !s.is_empty() && *s != url),
)
.collect();
context
.set_config(Config::ProxyUrl, Some(&proxy_urls.join("\n")))
.await?;
context.set_config_bool(Config::ProxyEnabled, true).await?;
}
Qr::WithdrawVerifyContact {
invitenumber,
authcode,
@@ -679,7 +765,7 @@ pub async fn set_config_from_qr(context: &Context, qr: &str) -> Result<()> {
token::save(context, token::Namespace::InviteNumber, None, &invitenumber).await?;
token::save(context, token::Namespace::Auth, None, &authcode).await?;
context.sync_qr_code_tokens(None).await?;
context.scheduler.interrupt_smtp().await;
context.scheduler.interrupt_inbox().await;
}
Qr::ReviveVerifyGroup {
invitenumber,
@@ -687,19 +773,16 @@ pub async fn set_config_from_qr(context: &Context, qr: &str) -> Result<()> {
grpid,
..
} => {
let chat_id = get_chat_id_by_grpid(context, &grpid)
.await?
.map(|(chat_id, _protected, _blocked)| chat_id);
token::save(
context,
token::Namespace::InviteNumber,
chat_id,
Some(&grpid),
&invitenumber,
)
.await?;
token::save(context, token::Namespace::Auth, chat_id, &authcode).await?;
context.sync_qr_code_tokens(chat_id).await?;
context.scheduler.interrupt_smtp().await;
token::save(context, token::Namespace::Auth, Some(&grpid), &authcode).await?;
context.sync_qr_code_tokens(Some(&grpid)).await?;
context.scheduler.interrupt_inbox().await;
}
Qr::Login { address, options } => {
configure_from_login_qr(context, &address, options).await?
@@ -870,6 +953,7 @@ mod tests {
use super::*;
use crate::aheader::EncryptPreference;
use crate::chat::{create_group_chat, ProtectionStatus};
use crate::config::Config;
use crate::key::DcKey;
use crate::securejoin::get_securejoin_qr;
use crate::test_utils::{alice_keypair, TestContext};
@@ -878,11 +962,38 @@ mod tests {
async fn test_decode_http() -> Result<()> {
let ctx = TestContext::new().await;
let qr = check_qr(&ctx.ctx, "http://www.hello.com:80").await?;
assert_eq!(
qr,
Qr::Proxy {
url: "http://www.hello.com:80".to_string(),
host: "www.hello.com".to_string(),
port: 80
}
);
// If it has no explicit port, then it is not a proxy.
let qr = check_qr(&ctx.ctx, "http://www.hello.com").await?;
assert_eq!(
qr,
Qr::Url {
url: "http://www.hello.com".to_string()
url: "http://www.hello.com".to_string(),
}
);
// If it has a path, then it is not a proxy.
let qr = check_qr(&ctx.ctx, "http://www.hello.com/").await?;
assert_eq!(
qr,
Qr::Url {
url: "http://www.hello.com/".to_string(),
}
);
let qr = check_qr(&ctx.ctx, "http://www.hello.com/hello").await?;
assert_eq!(
qr,
Qr::Url {
url: "http://www.hello.com/hello".to_string(),
}
);
@@ -893,11 +1004,38 @@ mod tests {
async fn test_decode_https() -> Result<()> {
let ctx = TestContext::new().await;
let qr = check_qr(&ctx.ctx, "https://www.hello.com:443").await?;
assert_eq!(
qr,
Qr::Proxy {
url: "https://www.hello.com:443".to_string(),
host: "www.hello.com".to_string(),
port: 443
}
);
// If it has no explicit port, then it is not a proxy.
let qr = check_qr(&ctx.ctx, "https://www.hello.com").await?;
assert_eq!(
qr,
Qr::Url {
url: "https://www.hello.com".to_string()
url: "https://www.hello.com".to_string(),
}
);
// If it has a path, then it is not a proxy.
let qr = check_qr(&ctx.ctx, "https://www.hello.com/").await?;
assert_eq!(
qr,
Qr::Url {
url: "https://www.hello.com/".to_string(),
}
);
let qr = check_qr(&ctx.ctx, "https://www.hello.com/hello").await?;
assert_eq!(
qr,
Qr::Url {
url: "https://www.hello.com/hello".to_string(),
}
);
@@ -1478,6 +1616,69 @@ mod tests {
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_decode_tg_socks_proxy() -> Result<()> {
let t = TestContext::new().await;
let qr = check_qr(&t, "https://t.me/socks?server=84.53.239.95&port=4145").await?;
assert_eq!(
qr,
Qr::Proxy {
url: "socks5://84.53.239.95:4145".to_string(),
host: "84.53.239.95".to_string(),
port: 4145,
}
);
let qr = check_qr(&t, "https://t.me/socks?server=foo.bar&port=123").await?;
assert_eq!(
qr,
Qr::Proxy {
url: "socks5://foo.bar:123".to_string(),
host: "foo.bar".to_string(),
port: 123,
}
);
let qr = check_qr(&t, "https://t.me/socks?server=foo.baz").await?;
assert_eq!(
qr,
Qr::Proxy {
url: "socks5://foo.baz:1080".to_string(),
host: "foo.baz".to_string(),
port: 1080,
}
);
let qr = check_qr(
&t,
"https://t.me/socks?server=foo.baz&port=12345&user=ada&pass=ms%21%2F%24",
)
.await?;
assert_eq!(
qr,
Qr::Proxy {
url: "socks5://ada:ms%21%2F%24@foo.baz:12345".to_string(),
host: "foo.baz".to_string(),
port: 12345,
}
);
// wrong domain results in Qr:Url instead of Qr::Socks5Proxy
let qr = check_qr(&t, "https://not.me/socks?noserver=84.53.239.95&port=4145").await?;
assert_eq!(
qr,
Qr::Url {
url: "https://not.me/socks?noserver=84.53.239.95&port=4145".to_string()
}
);
let qr = check_qr(&t, "https://t.me/socks?noserver=84.53.239.95&port=4145").await;
assert!(qr.is_err());
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_decode_account_bad_scheme() {
let ctx = TestContext::new().await;
@@ -1498,7 +1699,7 @@ mod tests {
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_set_config_from_qr() -> Result<()> {
async fn test_set_webrtc_instance_config_from_qr() -> Result<()> {
let ctx = TestContext::new().await;
assert!(ctx.ctx.get_config(Config::WebrtcInstance).await?.is_none());
@@ -1507,10 +1708,6 @@ mod tests {
assert!(res.is_err());
assert!(ctx.ctx.get_config(Config::WebrtcInstance).await?.is_none());
let res = set_config_from_qr(&ctx.ctx, "https://no.qr").await;
assert!(res.is_err());
assert!(ctx.ctx.get_config(Config::WebrtcInstance).await?.is_none());
let res = set_config_from_qr(&ctx.ctx, "dcwebrtc:https://example.org/").await;
assert!(res.is_ok());
assert_eq!(
@@ -1528,4 +1725,106 @@ mod tests {
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_set_proxy_config_from_qr() -> Result<()> {
let t = TestContext::new().await;
assert_eq!(t.get_config_bool(Config::ProxyEnabled).await?, false);
let res = set_config_from_qr(&t, "https://t.me/socks?server=foo&port=666").await;
assert!(res.is_ok());
assert_eq!(t.get_config_bool(Config::ProxyEnabled).await?, true);
assert_eq!(
t.get_config(Config::ProxyUrl).await?,
Some("socks5://foo:666".to_string())
);
// Test URL without port.
let res = set_config_from_qr(&t, "https://t.me/socks?server=1.2.3.4").await;
assert!(res.is_ok());
assert_eq!(t.get_config_bool(Config::ProxyEnabled).await?, true);
assert_eq!(
t.get_config(Config::ProxyUrl).await?,
Some("socks5://1.2.3.4:1080\nsocks5://foo:666".to_string())
);
// make sure, user&password are set when specified in the URL
// Password is an URL-encoded "x&%$X".
let res =
set_config_from_qr(&t, "https://t.me/socks?server=jau&user=Da&pass=x%26%25%24X").await;
assert!(res.is_ok());
assert_eq!(
t.get_config(Config::ProxyUrl).await?,
Some(
"socks5://Da:x%26%25%24X@jau:1080\nsocks5://1.2.3.4:1080\nsocks5://foo:666"
.to_string()
)
);
// Scanning existing proxy brings it to the top in the list.
let res = set_config_from_qr(&t, "https://t.me/socks?server=foo&port=666").await;
assert!(res.is_ok());
assert_eq!(t.get_config_bool(Config::ProxyEnabled).await?, true);
assert_eq!(
t.get_config(Config::ProxyUrl).await?,
Some(
"socks5://foo:666\nsocks5://Da:x%26%25%24X@jau:1080\nsocks5://1.2.3.4:1080"
.to_string()
)
);
set_config_from_qr(
&t,
"ss://YWVzLTEyOC1nY206dGVzdA@192.168.100.1:8888#Example1",
)
.await?;
assert_eq!(
t.get_config(Config::ProxyUrl).await?,
Some(
"ss://YWVzLTEyOC1nY206dGVzdA@192.168.100.1:8888#Example1\nsocks5://foo:666\nsocks5://Da:x%26%25%24X@jau:1080\nsocks5://1.2.3.4:1080"
.to_string()
)
);
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_decode_shadowsocks() -> Result<()> {
let ctx = TestContext::new().await;
let qr = check_qr(
&ctx.ctx,
"ss://YWVzLTEyOC1nY206dGVzdA@192.168.100.1:8888#Example1",
)
.await?;
assert_eq!(
qr,
Qr::Proxy {
url: "ss://YWVzLTEyOC1nY206dGVzdA@192.168.100.1:8888#Example1".to_string(),
host: "192.168.100.1".to_string(),
port: 8888,
}
);
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_decode_socks5() -> Result<()> {
let ctx = TestContext::new().await;
let qr = check_qr(&ctx.ctx, "socks5://127.0.0.1:9050").await?;
assert_eq!(
qr,
Qr::Proxy {
url: "socks5://127.0.0.1:9050".to_string(),
host: "127.0.0.1".to_string(),
port: 9050,
}
);
Ok(())
}
}

View File

@@ -8,7 +8,7 @@ use num_traits::cast::ToPrimitive;
use super::{Qr, DCLOGIN_SCHEME};
use crate::config::Config;
use crate::context::Context;
use crate::login_param::CertificateChecks;
use crate::login_param::EnteredCertificateChecks;
use crate::provider::Socket;
/// Options for `dclogin:` scheme.
@@ -55,7 +55,7 @@ pub enum LoginOptions {
smtp_security: Option<Socket>,
/// Certificate checks.
certificate_checks: Option<CertificateChecks>,
certificate_checks: Option<EnteredCertificateChecks>,
},
}
@@ -72,7 +72,7 @@ pub(super) fn decode_login(qr: &str) -> Result<Qr> {
.unwrap_or(url_without_scheme);
let addr = payload
.split(|c| c == '?' || c == '/')
.split(['?', '/'])
.next()
.context("invalid DCLOGIN payload E3")?;
@@ -146,11 +146,12 @@ fn parse_socket_security(security: Option<&String>) -> Result<Option<Socket>> {
fn parse_certificate_checks(
certificate_checks: Option<&String>,
) -> Result<Option<CertificateChecks>> {
) -> Result<Option<EnteredCertificateChecks>> {
Ok(match certificate_checks.map(|s| s.as_str()) {
Some("0") => Some(CertificateChecks::Automatic),
Some("1") => Some(CertificateChecks::Strict),
Some("3") => Some(CertificateChecks::AcceptInvalidCertificates),
Some("0") => Some(EnteredCertificateChecks::Automatic),
Some("1") => Some(EnteredCertificateChecks::Strict),
Some("2") => Some(EnteredCertificateChecks::AcceptInvalidCertificates),
Some("3") => Some(EnteredCertificateChecks::AcceptInvalidCertificates2),
Some(other) => bail!("Unknown certificatecheck level: {}", other),
None => None,
})
@@ -263,7 +264,7 @@ mod test {
use anyhow::bail;
use super::{decode_login, LoginOptions};
use crate::{login_param::CertificateChecks, provider::Socket, qr::Qr};
use crate::{login_param::EnteredCertificateChecks, provider::Socket, qr::Qr};
macro_rules! login_options_just_pw {
($pw: expr) => {
@@ -386,7 +387,7 @@ mod test {
smtp_username: Some("max@host.tld".to_owned()),
smtp_password: Some("3242HS".to_owned()),
smtp_security: Some(Socket::Plain),
certificate_checks: Some(CertificateChecks::Strict),
certificate_checks: Some(EnteredCertificateChecks::Strict),
}
);
} else {

View File

@@ -14,6 +14,77 @@ use crate::qr::{self, Qr};
use crate::securejoin;
use crate::stock_str::{self, backup_transfer_qr};
/// Create a QR code from any input data.
pub fn create_qr_svg(qrcode_content: &str) -> Result<String> {
let all_size = 512.0;
let qr_code_size = 416.0;
let logo_size = 96.0;
let qr = QrCode::encode_text(qrcode_content, QrCodeEcc::Medium)?;
let mut svg = String::with_capacity(28000);
let mut w = tagger::new(&mut svg);
w.elem("svg", |d| {
d.attr("xmlns", "http://www.w3.org/2000/svg")?;
d.attr("viewBox", format_args!("0 0 {all_size} {all_size}"))?;
d.attr("xmlns:xlink", "http://www.w3.org/1999/xlink")?; // required for enabling xlink:href on browsers
Ok(())
})?
.build(|w| {
// background
w.single("rect", |d| {
d.attr("x", 0)?;
d.attr("y", 0)?;
d.attr("width", all_size)?;
d.attr("height", all_size)?;
d.attr("style", "fill:#ffffff")?;
Ok(())
})?;
// QR code
w.elem("g", |d| {
d.attr(
"transform",
format!(
"translate({},{})",
(all_size - qr_code_size) / 2.0,
((all_size - qr_code_size) / 2.0)
),
)
})?
.build(|w| {
w.single("path", |d| {
let mut path_data = String::with_capacity(0);
let scale = qr_code_size / qr.size() as f32;
for y in 0..qr.size() {
for x in 0..qr.size() {
if qr.get_module(x, y) {
path_data += &format!("M{x},{y}h1v1h-1z");
}
}
}
d.attr("style", "fill:#000000")?;
d.attr("d", path_data)?;
d.attr("transform", format!("scale({scale})"))
})
})?;
w.elem("g", |d| {
d.attr(
"transform",
format!(
"translate({},{}) scale(2)", // data in qr_overlay_delta.svg-part are 48 x 48, scaling by 2 results in desired logo_size of 96
(all_size - logo_size) / 2.0,
(all_size - logo_size) / 2.0
),
)
})?
.build(|w| w.put_raw_escapable(include_str!("../assets/qr_overlay_delta.svg-part")))
})?;
Ok(svg)
}
/// Returns SVG of the QR code to join the group or verify contact.
///
/// If `chat_id` is `None`, returns verification QR code.
@@ -304,6 +375,14 @@ mod tests {
use super::*;
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_qr_svg() -> Result<()> {
let svg = create_qr_svg("this is a test QR code \" < > &")?;
assert!(svg.contains("<svg"));
assert!(svg.contains("</svg>"));
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_svg_escaping() {
let svg = inner_generate_secure_join_qr_code(

View File

@@ -25,6 +25,7 @@ pub const QUOTA_ERROR_THRESHOLD_PERCENTAGE: u64 = 95;
/// if quota is below this value (again),
/// QuotaExceeding is cleared.
///
/// This value should be a bit below QUOTA_WARN_THRESHOLD_PERCENTAGE to
/// avoid jittering and lots of warnings when quota is exactly at the warning threshold.
///
@@ -73,7 +74,7 @@ async fn get_unique_quota_roots_and_usage(
fn get_highest_usage<'t>(
unique_quota_roots: &'t BTreeMap<String, Vec<QuotaResource>>,
) -> Result<(u64, &'t String, &QuotaResource)> {
) -> Result<(u64, &'t String, &'t QuotaResource)> {
let mut highest: Option<(u64, &'t String, &QuotaResource)> = None;
for (name, resources) in unique_quota_roots {
for r in resources {
@@ -197,7 +198,7 @@ mod tests {
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_quota_needs_update() {
async fn test_quota_needs_update() -> Result<()> {
let mut tcm = TestContextManager::new();
let t = &tcm.unconfigured().await;
const TIMEOUT: u64 = 60;
@@ -214,5 +215,14 @@ mod tests {
modified: tools::Time::now(),
});
assert!(!t.quota_needs_update(TIMEOUT).await);
t.evtracker.clear_events();
t.set_primary_self_addr("new@addr").await?;
assert!(t.quota.read().await.is_none());
t.evtracker
.get_matching(|evt| matches!(evt, EventType::ConnectivityChanged))
.await;
assert!(t.quota_needs_update(TIMEOUT).await);
Ok(())
}
}

View File

@@ -19,6 +19,7 @@ use std::collections::BTreeMap;
use std::fmt;
use anyhow::Result;
use serde::{Deserialize, Serialize};
use crate::chat::{send_msg, Chat, ChatId};
use crate::chatlist_events;
@@ -31,7 +32,7 @@ use crate::param::Param;
/// A single reaction consisting of multiple emoji sequences.
///
/// It is guaranteed to have all emojis sorted and deduplicated inside.
#[derive(Debug, Default, Clone)]
#[derive(Debug, Default, Clone, Deserialize, Eq, PartialEq, Serialize)]
pub struct Reaction {
/// Canonical representation of reaction as a string of space-separated emojis.
reaction: String,
@@ -173,7 +174,7 @@ async fn set_msg_id_reaction(
chat_id: ChatId,
contact_id: ContactId,
timestamp: i64,
reaction: Reaction,
reaction: &Reaction,
) -> Result<()> {
if reaction.is_empty() {
// Simply remove the record instead of setting it to empty string.
@@ -244,7 +245,7 @@ pub async fn send_reaction(context: &Context, msg_id: MsgId, reaction: &str) ->
msg.chat_id,
ContactId::SELF,
reaction_msg.timestamp_sort,
reaction,
&reaction,
)
.await?;
Ok(reaction_msg_id)
@@ -275,16 +276,28 @@ pub(crate) async fn set_msg_reaction(
contact_id: ContactId,
timestamp: i64,
reaction: Reaction,
is_incoming_fresh: bool,
) -> Result<()> {
if let Some((msg_id, _)) = rfc724_mid_exists(context, in_reply_to).await? {
set_msg_id_reaction(context, msg_id, chat_id, contact_id, timestamp, reaction).await
set_msg_id_reaction(context, msg_id, chat_id, contact_id, timestamp, &reaction).await?;
if is_incoming_fresh
&& !reaction.is_empty()
&& msg_id.get_state(context).await?.is_outgoing()
{
context.emit_event(EventType::IncomingReaction {
contact_id,
msg_id,
reaction,
});
}
} else {
info!(
context,
"Can't assign reaction to unknown message with Message-ID {}", in_reply_to
);
Ok(())
}
Ok(())
}
/// Get our own reaction for a given message.
@@ -563,6 +576,38 @@ Here's my footer -- bob@example.net"
Ok(())
}
async fn expect_incoming_reactions_event(
t: &TestContext,
expected_msg_id: MsgId,
expected_contact_id: ContactId,
expected_reaction: &str,
) -> Result<()> {
let event = t
.evtracker
.get_matching(|evt| matches!(evt, EventType::IncomingReaction { .. }))
.await;
match event {
EventType::IncomingReaction {
msg_id,
contact_id,
reaction,
} => {
assert_eq!(msg_id, expected_msg_id);
assert_eq!(contact_id, expected_contact_id);
assert_eq!(reaction, Reaction::from(expected_reaction));
}
_ => unreachable!(),
}
Ok(())
}
async fn has_incoming_reactions_event(t: &TestContext) -> bool {
t.evtracker
.get_matching_opt(t, |evt| matches!(evt, EventType::IncomingReaction { .. }))
.await
.is_some()
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_send_reaction() -> Result<()> {
let alice = TestContext::new_alice().await;
@@ -593,6 +638,7 @@ Here's my footer -- bob@example.net"
send_reaction(&bob, bob_msg.id, "👍").await.unwrap();
expect_reactions_changed_event(&bob, bob_msg.chat_id, bob_msg.id, ContactId::SELF).await?;
assert!(!has_incoming_reactions_event(&bob).await);
assert_eq!(get_chat_msgs(&bob, bob_msg.chat_id).await?.len(), 2);
let bob_reaction_msg = bob.pop_sent_msg().await;
@@ -610,6 +656,7 @@ Here's my footer -- bob@example.net"
assert_eq!(bob_reaction.as_str(), "👍");
expect_reactions_changed_event(&alice, chat_alice.id, alice_msg.sender_msg_id, *bob_id)
.await?;
expect_incoming_reactions_event(&alice, alice_msg.sender_msg_id, *bob_id, "👍").await?;
// Alice reacts to own message.
send_reaction(&alice, alice_msg.sender_msg_id, "👍 😀")
@@ -650,6 +697,7 @@ Here's my footer -- bob@example.net"
send_reaction(&bob, bob_msg1.id, "👍").await?;
let bob_send_reaction = bob.pop_sent_msg().await;
alice.recv_msg_trash(&bob_send_reaction).await;
assert!(has_incoming_reactions_event(&alice).await);
let chatlist = Chatlist::try_load(&bob, 0, None, None).await?;
let summary = chatlist.get_summary(&bob, 0, None).await?;
@@ -665,6 +713,7 @@ Here's my footer -- bob@example.net"
send_reaction(&alice, alice_msg1.sender_msg_id, "🍿").await?;
let alice_send_reaction = alice.pop_sent_msg().await;
bob.recv_msg_opt(&alice_send_reaction).await;
assert!(!has_incoming_reactions_event(&bob).await);
assert_summary(&alice, "You reacted 🍿 to \"Party?\"").await;
assert_summary(&bob, "ALICE reacted 🍿 to \"Party?\"").await;

View File

@@ -35,7 +35,7 @@ use crate::peerstate::Peerstate;
use crate::reaction::{set_msg_reaction, Reaction};
use crate::securejoin::{self, handle_securejoin_handshake, observe_securejoin_on_other_device};
use crate::simplify;
use crate::sql;
use crate::sql::{self, params_iter};
use crate::stock_str;
use crate::sync::Sync::*;
use crate::tools::{self, buf_compress, remove_subject_prefix};
@@ -1255,11 +1255,13 @@ async fn add_parts(
let in_fresh = state == MessageState::InFresh;
let sort_to_bottom = false;
let received = true;
let sort_timestamp = chat_id
.calc_sort_timestamp(
context,
mime_parser.timestamp_sent,
sort_to_bottom,
received,
mime_parser.incoming,
)
.await?;
@@ -1444,6 +1446,9 @@ async fn add_parts(
Ok(node_addr) => {
info!(context, "Adding iroh peer with address {node_addr:?}.");
let instance_id = parent.context("Failed to get parent message")?.id;
context.emit_event(EventType::WebxdcRealtimeAdvertisementReceived {
msg_id: instance_id,
});
if let Some(topic) = get_iroh_topic_for_msg(context, instance_id).await? {
let node_id = node_addr.node_id;
let relay_server = node_addr.relay_url().map(|relay| relay.as_str());
@@ -1468,6 +1473,7 @@ async fn add_parts(
for part in &mime_parser.parts {
if part.is_reaction {
let reaction_str = simplify::remove_footers(part.msg.as_str());
let is_incoming_fresh = mime_parser.incoming && !seen && !fetching_existing_messages;
set_msg_reaction(
context,
mime_in_reply_to,
@@ -1475,6 +1481,7 @@ async fn add_parts(
from_id,
sort_timestamp,
Reaction::from(reaction_str.as_str()),
is_incoming_fresh,
)
.await?;
}
@@ -1827,22 +1834,82 @@ async fn lookup_chat_or_create_adhoc_group(
// Try to assign to a chat based on In-Reply-To/References.
lookup_chat_by_reply(context, mime_parser, parent, to_ids, from_id).await?
{
Ok(Some((new_chat_id, new_chat_id_blocked)))
} else if allow_creation {
// Try to create an ad hoc group.
create_adhoc_group(
context,
mime_parser,
create_blocked,
from_id,
to_ids,
is_partial_download,
)
.await
.context("Could not create ad hoc group")
} else {
Ok(None)
return Ok(Some((new_chat_id, new_chat_id_blocked)));
}
// Partial download may be an encrypted message with protected Subject header. We do not want to
// create a group with "..." or "Encrypted message" as a subject. The same is for undecipherable
// messages. Instead, assign the message to 1:1 chat with the sender.
if is_partial_download {
info!(
context,
"Ad-hoc group cannot be created from partial download."
);
return Ok(None);
}
if mime_parser.decrypting_failed {
warn!(
context,
"Not creating ad-hoc group for message that cannot be decrypted."
);
return Ok(None);
}
let grpname = mime_parser
.get_subject()
.map(|s| remove_subject_prefix(&s))
.unwrap_or_else(|| "👥📧".to_string());
let mut contact_ids = Vec::with_capacity(to_ids.len() + 1);
contact_ids.extend(to_ids);
if !contact_ids.contains(&from_id) {
contact_ids.push(from_id);
}
if let Some((chat_id, blocked)) = context
.sql
.query_row_optional(
&format!(
"SELECT c.id, c.blocked
FROM chats c INNER JOIN msgs m ON c.id=m.chat_id
WHERE m.hidden=0 AND c.grpid='' AND c.name=?
AND (SELECT COUNT(*) FROM chats_contacts
WHERE chat_id=c.id)=?
AND (SELECT COUNT(*) FROM chats_contacts
WHERE chat_id=c.id
AND contact_id NOT IN ({}))=0
ORDER BY m.timestamp DESC",
sql::repeat_vars(contact_ids.len()),
),
rusqlite::params_from_iter(
params_iter(&[&grpname])
.chain(params_iter(&[contact_ids.len()]))
.chain(params_iter(&contact_ids)),
),
|row| {
let id: ChatId = row.get(0)?;
let blocked: Blocked = row.get(1)?;
Ok((id, blocked))
},
)
.await?
{
info!(
context,
"Assigning message to ad-hoc group {chat_id} with matching name and members."
);
return Ok(Some((chat_id, blocked)));
}
if !allow_creation {
return Ok(None);
}
create_adhoc_group(
context,
mime_parser,
create_blocked,
from_id,
to_ids,
&grpname,
)
.await
.context("Could not create ad hoc group")
}
/// If this method returns true, the message shall be assigned to the 1:1 chat with the sender.
@@ -2512,19 +2579,8 @@ async fn create_adhoc_group(
create_blocked: Blocked,
from_id: ContactId,
to_ids: &[ContactId],
is_partial_download: bool,
grpname: &str,
) -> Result<Option<(ChatId, Blocked)>> {
if is_partial_download {
// Partial download may be an encrypted message with protected Subject header.
//
// We do not want to create a group with "..." or "Encrypted message" as a subject.
info!(
context,
"Ad-hoc group cannot be created from partial download."
);
return Ok(None);
}
let mut member_ids: Vec<ContactId> = to_ids.to_vec();
if !member_ids.contains(&(from_id)) {
member_ids.push(from_id);
@@ -2536,22 +2592,6 @@ async fn create_adhoc_group(
if mime_parser.is_mailinglist_message() {
return Ok(None);
}
if mime_parser.decrypting_failed {
// Do not create a new ad-hoc group if the message cannot be
// decrypted.
//
// The subject may be encrypted and contain a placeholder such
// as "...". It can also be a COI group, with encrypted
// Chat-Group-ID and incompatible Message-ID format.
//
// Instead, assign the message to 1:1 chat with the sender.
warn!(
context,
"Not creating ad-hoc group for message that cannot be decrypted."
);
return Ok(None);
}
if mime_parser
.get_header(HeaderDef::ChatGroupMemberRemoved)
.is_some()
@@ -2566,16 +2606,11 @@ async fn create_adhoc_group(
return Ok(None);
}
let grpname = mime_parser
.get_subject()
.map(|s| remove_subject_prefix(&s))
.unwrap_or_else(|| "👥📧".to_string());
let new_chat_id: ChatId = ChatId::create_multiuser_record(
context,
Chattype::Group,
"", // Ad hoc groups have no ID.
&grpname,
grpname,
create_blocked,
ProtectionStatus::Unprotected,
None,

View File

@@ -204,6 +204,71 @@ async fn test_adhoc_group_show_all() {
assert_eq!(chat::get_chat_contacts(&t, chat_id).await.unwrap().len(), 3);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_adhoc_groups_merge() -> Result<()> {
let mut tcm = TestContextManager::new();
let alice = &tcm.alice().await;
receive_imf(
alice,
b"From: bob@example.net\n\
To: alice@example.org, claire@example.com\n\
Message-ID: <1111@example.net>\n\
Date: Sun, 22 Mar 2020 22:37:57 +0000\n\
Subject: New thread\n\
\n\
The first of us should create a thread as discussed\n",
false,
)
.await?;
receive_imf(
alice,
b"From: alice@example.org\n\
To: bob@example.net, claire@example.com\n\
Message-ID: <2222@example.org>\n\
Date: Sun, 22 Mar 2020 22:37:58 +0000\n\
Subject: New thread\n\
\n\
The first of us should create a thread as discussed\n",
false,
)
.await?;
let chats = Chatlist::try_load(alice, 0, None, None).await?;
assert_eq!(chats.len(), 1);
let chat_id = chats.get_chat_id(0)?;
assert_eq!(chat_id.get_msg_cnt(alice).await?, 2);
// If member list doesn't match, threads aren't merged.
receive_imf(
alice,
b"From: bob@example.net\n\
To: alice@example.org, claire@example.com, fiona@example.net\n\
Message-ID: <3333@example.net>\n\
Date: Sun, 22 Mar 2020 22:37:57 +0000\n\
Subject: New thread\n\
\n\
This is another thread, with Fiona\n",
false,
)
.await?;
let chats = Chatlist::try_load(alice, 0, None, None).await?;
assert_eq!(chats.len(), 2);
receive_imf(
alice,
b"From: bob@example.net\n\
To: alice@example.org, fiona@example.net\n\
Message-ID: <4444@example.net>\n\
Date: Sun, 22 Mar 2020 22:37:57 +0000\n\
Subject: New thread\n\
\n\
This is yet another thread, with Fiona and 0 Claires\n",
false,
)
.await?;
let chats = Chatlist::try_load(alice, 0, None, None).await?;
assert_eq!(chats.len(), 3);
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_read_receipt_and_unarchive() -> Result<()> {
// create alice's account
@@ -2107,6 +2172,19 @@ async fn test_no_unencrypted_name_in_self_chat() -> Result<()> {
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_no_smtp_job_for_self_chat() -> Result<()> {
let mut tcm = TestContextManager::new();
let bob = &tcm.bob().await;
bob.set_config_bool(Config::BccSelf, false).await?;
let chat_id = bob.get_self_chat().await.id;
let mut msg = Message::new(Viewtype::Text);
msg.text = "Happy birthday to me".to_string();
chat::send_msg(bob, chat_id, &mut msg).await?;
assert!(bob.pop_sent_msg_opt(Duration::ZERO).await.is_none());
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_outgoing_classic_mail_creates_chat() {
let alice = TestContext::new_alice().await;
@@ -3085,20 +3163,21 @@ Reply from different address
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_long_and_duplicated_filenames() -> Result<()> {
async fn test_weird_and_duplicated_filenames() -> Result<()> {
let mut tcm = TestContextManager::new();
let alice = tcm.alice().await;
let bob = tcm.bob().await;
for filename_sent in &[
"foo.bar very long file name test baz.tar.gz",
"foobarabababababababbababababverylongfilenametestbaz.tar.gz",
"foo.barabababababababbababababverylongfilenametestbaz.tar.gz",
"fooo...tar.gz",
"foo. .tar.gz",
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.tar.gz",
"a.tar.gz",
"a.tar.gz",
"a.a..a.a.a.a.tar.gz",
"a. tar.tar.gz",
] {
let attachment = alice.blobdir.join(filename_sent);
let content = format!("File content of {filename_sent}");
@@ -4732,6 +4811,36 @@ async fn test_protected_group_add_remove_member_missing_key() -> Result<()> {
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_older_message_from_2nd_device() -> Result<()> {
let mut tcm = TestContextManager::new();
let alice = &tcm.alice().await;
let chat_id = alice
.create_chat_with_contact("", "bob@example.net")
.await
.id;
alice.send_text(chat_id, "We share this account").await;
let received = receive_imf(
alice,
b"From: alice@example.org\n\
To: bob@example.net\n\
Message-ID: <1234-2-4@example.org>\n\
Date: Sat, 07 Dec 1970 19:00:26 +0000\n\
\n\
I'm Alice too\n",
true,
)
.await?
.unwrap();
alice
.golden_test_chat(
received.chat_id,
"receive_imf_older_message_from_2nd_device",
)
.await;
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_dont_create_adhoc_group_on_member_removal() -> Result<()> {
let mut tcm = TestContextManager::new();

View File

@@ -433,36 +433,33 @@ async fn inbox_loop(
}
/// Convert folder meaning
/// used internally by [fetch_idle] and [Context::background_fetch]
/// used internally by [fetch_idle] and [Context::background_fetch].
///
/// Returns folder configuration key and folder name
/// if such folder is configured, `Ok(None)` otherwise.
pub async fn convert_folder_meaning(
ctx: &Context,
folder_meaning: FolderMeaning,
) -> Result<(Config, String)> {
) -> Result<Option<(Config, String)>> {
let folder_config = match folder_meaning.to_config() {
Some(c) => c,
None => {
bail!("Bad folder meaning: {}", folder_meaning);
// Such folder cannot be configured,
// e.g. a `FolderMeaning::Spam` folder.
return Ok(None);
}
};
let folder = match ctx.get_config(folder_config).await {
Ok(folder) => folder,
Err(err) => {
bail!(
"Can not watch {} folder, failed to retrieve config: {:#}",
folder_config,
err
);
}
};
let folder = ctx
.get_config(folder_config)
.await
.with_context(|| format!("Failed to retrieve {folder_config} folder"))?;
let watch_folder = if let Some(watch_folder) = folder {
watch_folder
if let Some(watch_folder) = folder {
Ok(Some((folder_config, watch_folder)))
} else {
bail!("Can not watch {} folder, not set", folder_config);
};
Ok((folder_config, watch_folder))
Ok(None)
}
}
async fn inbox_fetch_idle(ctx: &Context, imap: &mut Imap, mut session: Session) -> Result<Session> {
@@ -554,20 +551,32 @@ async fn fetch_idle(
mut session: Session,
folder_meaning: FolderMeaning,
) -> Result<Session> {
let (folder_config, watch_folder) = match convert_folder_meaning(ctx, folder_meaning).await {
Ok(meaning) => meaning,
Err(err) => {
// Warning instead of error because the folder may not be configured.
// For example, this happens if the server does not have Sent folder
// but watching Sent folder is enabled.
warn!(ctx, "Error converting IMAP Folder name: {err:#}.");
connection.connectivity.set_not_configured(ctx).await;
connection.idle_interrupt_receiver.recv().await.ok();
return Err(err);
}
let Some((folder_config, watch_folder)) = convert_folder_meaning(ctx, folder_meaning).await?
else {
// The folder is not configured.
// For example, this happens if the server does not have Sent folder
// but watching Sent folder is enabled.
connection.connectivity.set_not_configured(ctx).await;
connection.idle_interrupt_receiver.recv().await.ok();
bail!("Cannot fetch folder {folder_meaning} because it is not configured");
};
if folder_config == Config::ConfiguredInboxFolder {
let mvbox;
let syncbox = match ctx.should_move_sync_msgs().await? {
false => &watch_folder,
true => {
mvbox = ctx.get_config(Config::ConfiguredMvboxFolder).await?;
mvbox.as_deref().unwrap_or(&watch_folder)
}
};
session
.send_sync_msgs(ctx, syncbox)
.await
.context("fetch_idle: send_sync_msgs")
.log_err(ctx)
.ok();
session
.store_seen_flags_on_imap(ctx)
.await

View File

@@ -1,13 +1,13 @@
//! Implementation of [SecureJoin protocols](https://securejoin.delta.chat/).
use anyhow::{bail, Context as _, Error, Result};
use anyhow::{ensure, Context as _, Error, Result};
use percent_encoding::{utf8_percent_encode, AsciiSet, NON_ALPHANUMERIC};
use crate::aheader::EncryptPreference;
use crate::chat::{self, Chat, ChatId, ChatIdBlocked, ProtectionStatus};
use crate::chat::{self, get_chat_id_by_grpid, Chat, ChatId, ChatIdBlocked, ProtectionStatus};
use crate::chatlist_events;
use crate::config::Config;
use crate::constants::Blocked;
use crate::constants::{Blocked, Chattype};
use crate::contact::{Contact, ContactId, Origin};
use crate::context::Context;
use crate::e2ee::ensure_secret_key_exists;
@@ -60,45 +60,51 @@ pub async fn get_securejoin_qr(context: &Context, group: Option<ChatId>) -> Resu
ensure_secret_key_exists(context).await.ok();
// invitenumber will be used to allow starting the handshake,
// auth will be used to verify the fingerprint
let sync_token = token::lookup(context, Namespace::InviteNumber, group)
let chat = match group {
Some(id) => {
let chat = Chat::load_from_db(context, id).await?;
ensure!(
chat.typ == Chattype::Group,
"Can't generate SecureJoin QR code for 1:1 chat {id}"
);
ensure!(
!chat.grpid.is_empty(),
"Can't generate SecureJoin QR code for ad-hoc group {id}"
);
Some(chat)
}
None => None,
};
let grpid = chat.as_ref().map(|c| c.grpid.as_str());
let sync_token = token::lookup(context, Namespace::InviteNumber, grpid)
.await?
.is_none();
let invitenumber = token::lookup_or_new(context, Namespace::InviteNumber, group).await?;
let auth = token::lookup_or_new(context, Namespace::Auth, group).await?;
// invitenumber will be used to allow starting the handshake,
// auth will be used to verify the fingerprint
let invitenumber = token::lookup_or_new(context, Namespace::InviteNumber, grpid).await?;
let auth = token::lookup_or_new(context, Namespace::Auth, grpid).await?;
let self_addr = context.get_primary_self_addr().await?;
let self_name = context
.get_config(Config::Displayname)
.await?
.unwrap_or_default();
let fingerprint: Fingerprint = match get_self_fingerprint(context).await {
Some(fp) => fp,
None => {
bail!("No fingerprint, cannot generate QR code.");
}
};
let fingerprint = get_self_fingerprint(context).await?;
let self_addr_urlencoded =
utf8_percent_encode(&self_addr, NON_ALPHANUMERIC_WITHOUT_DOT).to_string();
let self_name_urlencoded =
utf8_percent_encode(&self_name, NON_ALPHANUMERIC_WITHOUT_DOT).to_string();
let qr = if let Some(group) = group {
let qr = if let Some(chat) = chat {
// parameters used: a=g=x=i=s=
let chat = Chat::load_from_db(context, group).await?;
if chat.grpid.is_empty() {
bail!(
"can't generate securejoin QR code for ad-hoc group {}",
group
);
}
let group_name = chat.get_name();
let group_name_urlencoded = utf8_percent_encode(group_name, NON_ALPHANUMERIC).to_string();
if sync_token {
context.sync_qr_code_tokens(Some(chat.id)).await?;
context.scheduler.interrupt_smtp().await;
context
.sync_qr_code_tokens(Some(chat.grpid.as_str()))
.await?;
context.scheduler.interrupt_inbox().await;
}
format!(
"OPENPGP4FPR:{}#a={}&g={}&x={}&i={}&s={}",
@@ -113,7 +119,7 @@ pub async fn get_securejoin_qr(context: &Context, group: Option<ChatId>) -> Resu
// parameters used: a=n=i=s=
if sync_token {
context.sync_qr_code_tokens(None).await?;
context.scheduler.interrupt_smtp().await;
context.scheduler.interrupt_inbox().await;
}
format!(
"OPENPGP4FPR:{}#a={}&n={}&i={}&s={}",
@@ -129,14 +135,11 @@ pub async fn get_securejoin_qr(context: &Context, group: Option<ChatId>) -> Resu
Ok(qr)
}
async fn get_self_fingerprint(context: &Context) -> Option<Fingerprint> {
match load_self_public_key(context).await {
Ok(key) => Some(key.fingerprint()),
Err(_) => {
warn!(context, "get_self_fingerprint(): failed to load key");
None
}
}
async fn get_self_fingerprint(context: &Context) -> Result<Fingerprint> {
let key = load_self_public_key(context)
.await
.context("Failed to load key")?;
Ok(key.fingerprint())
}
/// Take a scanned QR-code and do the setup-contact/join-group/invite handshake.
@@ -369,62 +372,54 @@ pub(crate) async fn handle_securejoin_handshake(
==========================================================*/
// verify that Secure-Join-Fingerprint:-header matches the fingerprint of Bob
let fingerprint: Fingerprint =
match mime_message.get_header(HeaderDef::SecureJoinFingerprint) {
Some(fp) => fp.parse()?,
None => {
could_not_establish_secure_connection(
context,
contact_id,
info_chat_id(context, contact_id).await?,
"Fingerprint not provided.",
)
.await?;
return Ok(HandshakeMessage::Ignore);
}
};
if !encrypted_and_signed(context, mime_message, Some(&fingerprint)) {
could_not_establish_secure_connection(
let Some(fp) = mime_message.get_header(HeaderDef::SecureJoinFingerprint) else {
warn!(
context,
contact_id,
info_chat_id(context, contact_id).await?,
"Auth not encrypted.",
)
.await?;
"Ignoring {step} message because fingerprint is not provided."
);
return Ok(HandshakeMessage::Ignore);
};
let fingerprint: Fingerprint = fp.parse()?;
if !encrypted_and_signed(context, mime_message, &fingerprint) {
warn!(
context,
"Ignoring {step} message because the message is not encrypted."
);
return Ok(HandshakeMessage::Ignore);
}
if !verify_sender_by_fingerprint(context, &fingerprint, contact_id).await? {
could_not_establish_secure_connection(
warn!(
context,
contact_id,
info_chat_id(context, contact_id).await?,
"Fingerprint mismatch on inviter-side.",
)
.await?;
"Ignoring {step} message because of fingerprint mismatch."
);
return Ok(HandshakeMessage::Ignore);
}
info!(context, "Fingerprint verified.",);
// verify that the `Secure-Join-Auth:`-header matches the secret written to the QR code
let Some(auth) = mime_message.get_header(HeaderDef::SecureJoinAuth) else {
could_not_establish_secure_connection(
warn!(
context,
contact_id,
info_chat_id(context, contact_id).await?,
"Auth not provided.",
)
.await?;
"Ignoring {step} message because of missing auth code."
);
return Ok(HandshakeMessage::Ignore);
};
let Some(group_chat_id) = token::auth_chat_id(context, auth).await? else {
could_not_establish_secure_connection(
let Some(grpid) = token::auth_foreign_key(context, auth).await? else {
warn!(
context,
contact_id,
info_chat_id(context, contact_id).await?,
"Auth invalid.",
)
.await?;
"Ignoring {step} message because of invalid auth code."
);
return Ok(HandshakeMessage::Ignore);
};
let group_chat_id = match grpid.as_str() {
"" => None,
id => {
let Some((chat_id, ..)) = get_chat_id_by_grpid(context, id).await? else {
warn!(context, "Ignoring {step} message: unknown grpid {id}.",);
return Ok(HandshakeMessage::Ignore);
};
Some(chat_id)
}
};
let contact_addr = Contact::get_by_id(context, contact_id)
.await?
@@ -439,13 +434,10 @@ pub(crate) async fn handle_securejoin_handshake(
)
.await?;
if !fingerprint_found {
could_not_establish_secure_connection(
warn!(
context,
contact_id,
info_chat_id(context, contact_id).await?,
"Fingerprint mismatch on inviter-side.",
)
.await?;
"Ignoring {step} message because of the failure to find matching peerstate."
);
return Ok(HandshakeMessage::Ignore);
}
contact_id.regossip_keys(context).await?;
@@ -453,7 +445,20 @@ pub(crate) async fn handle_securejoin_handshake(
info!(context, "Auth verified.",);
context.emit_event(EventType::ContactsChanged(Some(contact_id)));
inviter_progress(context, contact_id, 600);
if group_chat_id.is_unset() {
if let Some(group_chat_id) = group_chat_id {
// Join group.
secure_connection_established(
context,
contact_id,
group_chat_id,
mime_message.timestamp_sent,
)
.await?;
chat::add_contact_to_chat_ex(context, Nosync, group_chat_id, contact_id, true)
.await?;
inviter_progress(context, contact_id, 800);
inviter_progress(context, contact_id, 1000);
} else {
// Setup verified contact.
secure_connection_established(
context,
@@ -467,19 +472,6 @@ pub(crate) async fn handle_securejoin_handshake(
.context("failed sending vc-contact-confirm message")?;
inviter_progress(context, contact_id, 1000);
} else {
// Join group.
secure_connection_established(
context,
contact_id,
group_chat_id,
mime_message.timestamp_sent,
)
.await?;
chat::add_contact_to_chat_ex(context, Nosync, group_chat_id, contact_id, true)
.await?;
inviter_progress(context, contact_id, 800);
inviter_progress(context, contact_id, 1000);
}
Ok(HandshakeMessage::Ignore) // "Done" would delete the message and break multi-device (the key from Autocrypt-header is needed)
}
@@ -577,11 +569,7 @@ pub(crate) async fn observe_securejoin_on_other_device(
return Ok(HandshakeMessage::Ignore);
};
if !encrypted_and_signed(
context,
mime_message,
get_self_fingerprint(context).await.as_ref(),
) {
if !encrypted_and_signed(context, mime_message, &get_self_fingerprint(context).await?) {
could_not_establish_secure_connection(
context,
contact_id,
@@ -637,6 +625,10 @@ pub(crate) async fn observe_securejoin_on_other_device(
return Ok(HandshakeMessage::Ignore);
};
peerstate.set_verified(key.clone(), fingerprint, addr)?;
if matches!(step, "vg-member-added" | "vc-contact-confirm") {
peerstate.backward_verified_key_id =
Some(context.get_config_i64(Config::KeyId).await?).filter(|&id| id > 0);
}
peerstate.prefer_encrypt = EncryptPreference::Mutual;
peerstate.save_to_db(&context.sql).await?;
@@ -692,7 +684,10 @@ async fn could_not_establish_secure_connection(
details: &str,
) -> Result<()> {
let contact = Contact::get_by_id(context, contact_id).await?;
let msg = stock_str::contact_not_verified(context, &contact).await;
let mut msg = stock_str::contact_not_verified(context, &contact).await;
msg += " (";
msg += details;
msg += ")";
chat::add_info_msg(context, chat_id, &msg, time()).await?;
warn!(
context,
@@ -733,24 +728,19 @@ async fn mark_peer_as_verified(
fn encrypted_and_signed(
context: &Context,
mimeparser: &MimeMessage,
expected_fingerprint: Option<&Fingerprint>,
expected_fingerprint: &Fingerprint,
) -> bool {
if !mimeparser.was_encrypted() {
warn!(context, "Message not encrypted.",);
false
} else if let Some(expected_fingerprint) = expected_fingerprint {
if !mimeparser.signatures.contains(expected_fingerprint) {
warn!(
context,
"Message does not match expected fingerprint {}.", expected_fingerprint,
);
false
} else {
true
}
} else {
warn!(context, "Fingerprint for comparison missing.");
} else if !mimeparser.signatures.contains(expected_fingerprint) {
warn!(
context,
"Message does not match expected fingerprint {}.", expected_fingerprint,
);
false
} else {
true
}
}
@@ -777,6 +767,7 @@ mod tests {
CheckProtectionTimestamp,
WrongAliceGossip,
SecurejoinWaitTimeout,
AliceIsBot,
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -799,6 +790,11 @@ mod tests {
test_setup_contact_ex(SetupContactCase::SecurejoinWaitTimeout).await
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_setup_contact_alice_is_bot() {
test_setup_contact_ex(SetupContactCase::AliceIsBot).await
}
async fn test_setup_contact_ex(case: SetupContactCase) {
let mut tcm = TestContextManager::new();
let alice = tcm.alice().await;
@@ -807,13 +803,19 @@ mod tests {
bob.set_config(Config::Displayname, Some("Bob Examplenet"))
.await
.unwrap();
alice
.set_config(Config::VerifiedOneOnOneChats, Some("1"))
.await
.unwrap();
bob.set_config(Config::VerifiedOneOnOneChats, Some("1"))
.await
.unwrap();
let alice_auto_submitted_hdr;
match case {
SetupContactCase::AliceIsBot => {
alice.set_config_bool(Config::Bot, true).await.unwrap();
alice_auto_submitted_hdr = "Auto-Submitted: auto-generated";
}
_ => alice_auto_submitted_hdr = "Auto-Submitted: auto-replied",
};
for t in [&alice, &bob] {
t.set_config_bool(Config::VerifiedOneOnOneChats, true)
.await
.unwrap();
}
assert_eq!(
Chatlist::try_load(&alice, 0, None, None)
@@ -849,6 +851,7 @@ mod tests {
assert!(!msg.was_encrypted());
assert_eq!(msg.get_header(HeaderDef::SecureJoin).unwrap(), "vc-request");
assert!(msg.get_header(HeaderDef::SecureJoinInvitenumber).is_some());
assert!(msg.get_header(HeaderDef::AutoSubmitted).is_none());
// Step 3: Alice receives vc-request, sends vc-auth-required
alice.recv_msg_trash(&sent).await;
@@ -861,6 +864,7 @@ mod tests {
);
let sent = alice.pop_sent_msg().await;
assert!(sent.payload.contains(alice_auto_submitted_hdr));
assert!(!sent.payload.contains("Alice Exampleorg"));
let msg = bob.parse_msg(&sent).await;
assert!(msg.was_encrypted());
@@ -907,6 +911,7 @@ mod tests {
// Check Bob sent the right message.
let sent = bob.pop_sent_msg().await;
assert!(sent.payload.contains("Auto-Submitted: auto-replied"));
assert!(!sent.payload.contains("Bob Examplenet"));
let mut msg = alice.parse_msg(&sent).await;
let vc_request_with_auth_ts_sent = msg
@@ -972,6 +977,7 @@ mod tests {
.await
.unwrap();
assert_eq!(contact_bob.get_authname(), "Bob Examplenet");
assert_eq!(contact_bob.is_bot(), false);
// exactly one one-to-one chat should be visible for both now
// (check this before calling alice.create_chat() explicitly below)
@@ -1011,6 +1017,7 @@ mod tests {
// Check Alice sent the right message to Bob.
let sent = alice.pop_sent_msg().await;
assert!(sent.payload.contains(alice_auto_submitted_hdr));
assert!(!sent.payload.contains("Alice Exampleorg"));
let msg = bob.parse_msg(&sent).await;
assert!(msg.was_encrypted());
@@ -1029,6 +1036,7 @@ mod tests {
.await
.unwrap();
assert_eq!(contact_alice.get_authname(), "Alice Exampleorg");
assert_eq!(contact_alice.is_bot(), case == SetupContactCase::AliceIsBot);
if case != SetupContactCase::SecurejoinWaitTimeout {
// Later we check that the timeout message isn't added to the already protected chat.
@@ -1243,6 +1251,7 @@ mod tests {
assert!(!msg.was_encrypted());
assert_eq!(msg.get_header(HeaderDef::SecureJoin).unwrap(), "vg-request");
assert!(msg.get_header(HeaderDef::SecureJoinInvitenumber).is_some());
assert!(msg.get_header(HeaderDef::AutoSubmitted).is_none());
// Old Delta Chat core sent `Secure-Join-Group` header in `vg-request`,
// but it was only used by Alice in `vg-request-with-auth`.
@@ -1256,6 +1265,7 @@ mod tests {
alice.recv_msg_trash(&sent).await;
let sent = alice.pop_sent_msg().await;
assert!(sent.payload.contains("Auto-Submitted: auto-replied"));
let msg = bob.parse_msg(&sent).await;
assert!(msg.was_encrypted());
assert_eq!(
@@ -1289,6 +1299,7 @@ mod tests {
}
// Check Bob sent the right handshake message.
assert!(sent.payload.contains("Auto-Submitted: auto-replied"));
let msg = alice.parse_msg(&sent).await;
assert!(msg.was_encrypted());
assert_eq!(
@@ -1321,6 +1332,10 @@ mod tests {
msg.get_header(HeaderDef::SecureJoin).unwrap(),
"vg-member-added"
);
// Formally this message is auto-submitted, but as the member addition is a result of an
// explicit user action, the Auto-Submitted header shouldn't be present. Otherwise it would
// be strange to have it in "member-added" messages of verified groups only.
assert!(msg.get_header(HeaderDef::AutoSubmitted).is_none());
{
// Now Alice's chat with Bob should still be hidden, the verified message should
@@ -1447,13 +1462,11 @@ First thread."#;
let mut tcm = TestContextManager::new();
let alice = tcm.alice().await;
let bob = tcm.bob().await;
alice
.set_config(Config::VerifiedOneOnOneChats, Some("1"))
.await
.unwrap();
bob.set_config(Config::VerifiedOneOnOneChats, Some("1"))
.await
.unwrap();
for t in [&alice, &bob] {
t.set_config_bool(Config::VerifiedOneOnOneChats, true)
.await
.unwrap();
}
let qr = get_securejoin_qr(&alice.ctx, None).await.unwrap();
join_securejoin(&bob.ctx, &qr).await.unwrap();

View File

@@ -47,7 +47,7 @@ pub(super) async fn start_protocol(context: &Context, invite: QrInvite) -> Resul
BobState::start_protocol(context, invite.clone(), chat_id).await?;
for state in aborted_states {
error!(context, "Aborting previously unfinished QR Join process.");
state.notify_aborted(context, "new QR scanned").await?;
state.notify_aborted(context, "New QR code scanned").await?;
state.emit_progress(context, JoinerProgress::Error);
}
if matches!(stage, BobHandshakeStage::RequestWithAuthSent) {
@@ -73,8 +73,9 @@ pub(super) async fn start_protocol(context: &Context, invite: QrInvite) -> Resul
// Calculate the sort timestamp before checking the chat protection status so that if we
// race with its change, we don't add our message below the protection message.
let sort_to_bottom = true;
let (received, incoming) = (false, false);
let ts_sort = chat_id
.calc_sort_timestamp(context, 0, sort_to_bottom, false)
.calc_sort_timestamp(context, 0, sort_to_bottom, received, incoming)
.await?;
if chat_id.is_protected(context).await? == ProtectionStatus::Unprotected {
let ts_start = time();
@@ -194,7 +195,10 @@ impl BobState {
/// This creates an info message in the chat being joined.
async fn notify_aborted(&self, context: &Context, why: &str) -> Result<()> {
let contact = Contact::get_by_id(context, self.invite().contact_id()).await?;
let msg = stock_str::contact_not_verified(context, &contact).await;
let mut msg = stock_str::contact_not_verified(context, &contact).await;
msg += " (";
msg += why;
msg += ")";
let chat_id = self.joining_chat_id(context).await?;
chat::add_info_msg(context, chat_id, &msg, time()).await?;
warn!(

View File

@@ -270,7 +270,7 @@ impl BobState {
context,
"Bob Step 4 - handling {{vc,vg}}-auth-required message."
);
if !encrypted_and_signed(context, mime_message, Some(self.invite.fingerprint())) {
if !encrypted_and_signed(context, mime_message, self.invite.fingerprint()) {
let reason = if mime_message.was_encrypted() {
"Valid signature missing"
} else {

View File

@@ -117,7 +117,7 @@ pub(crate) fn simplify(mut input: String, is_chat_message: bool) -> SimplifiedTe
let lines = split_lines(&input);
let (lines, is_forwarded) = skip_forward_header(&lines);
let (lines, mut top_quote) = remove_top_quote(lines);
let (lines, mut top_quote) = remove_top_quote(lines, is_chat_message);
let original_lines = &lines;
let (lines, footer_lines) = remove_message_footer(lines);
let footer = footer_lines.map(|footer_lines| render_message(footer_lines, false));
@@ -210,7 +210,10 @@ fn remove_bottom_quote<'a>(lines: &'a [&str]) -> (&'a [&'a str], Option<String>)
}
#[allow(clippy::indexing_slicing)]
fn remove_top_quote<'a>(lines: &'a [&str]) -> (&'a [&'a str], Option<String>) {
fn remove_top_quote<'a>(
lines: &'a [&str],
is_chat_message: bool,
) -> (&'a [&'a str], Option<String>) {
let mut first_quoted_line = 0;
let mut last_quoted_line = None;
let mut has_quoted_headline = false;
@@ -220,7 +223,11 @@ fn remove_top_quote<'a>(lines: &'a [&str]) -> (&'a [&'a str], Option<String>) {
first_quoted_line = l;
}
last_quoted_line = Some(l)
} else if is_quoted_headline(line) && !has_quoted_headline && last_quoted_line.is_none() {
} else if !is_chat_message
&& is_quoted_headline(line)
&& !has_quoted_headline
&& last_quoted_line.is_none()
{
has_quoted_headline = true
} else {
/* non-quoting line found */
@@ -396,17 +403,34 @@ mod tests {
#[test]
fn test_remove_top_quote() {
let (lines, top_quote) = remove_top_quote(&["> first", "> second"]);
let (lines, top_quote) = remove_top_quote(&["> first", "> second"], true);
assert!(lines.is_empty());
assert_eq!(top_quote.unwrap(), "first\nsecond");
let (lines, top_quote) = remove_top_quote(&["> first", "> second", "not a quote"]);
let (lines, top_quote) = remove_top_quote(&["> first", "> second", "not a quote"], true);
assert_eq!(lines, &["not a quote"]);
assert_eq!(top_quote.unwrap(), "first\nsecond");
let (lines, top_quote) = remove_top_quote(&["not a quote", "> first", "> second"]);
let (lines, top_quote) = remove_top_quote(&["not a quote", "> first", "> second"], true);
assert_eq!(lines, &["not a quote", "> first", "> second"]);
assert!(top_quote.is_none());
let (lines, top_quote) = remove_top_quote(
&["On 2024-08-28, Bob wrote:", "> quote", "not a quote"],
false,
);
assert_eq!(lines, &["not a quote"]);
assert_eq!(top_quote.unwrap(), "quote");
let (lines, top_quote) = remove_top_quote(
&["On 2024-08-28, Bob wrote:", "> quote", "not a quote"],
true,
);
assert_eq!(
lines,
&["On 2024-08-28, Bob wrote:", "> quote", "not a quote"]
);
assert!(top_quote.is_none());
}
#[test]

View File

@@ -5,7 +5,7 @@ pub mod send;
use anyhow::{bail, format_err, Context as _, Error, Result};
use async_smtp::response::{Category, Code, Detail};
use async_smtp::{self as smtp, EmailAddress, SmtpTransport};
use async_smtp::{EmailAddress, SmtpTransport};
use tokio::task;
use crate::chat::{add_info_msg_with_cmd, ChatId};
@@ -13,14 +13,14 @@ use crate::config::Config;
use crate::contact::{Contact, ContactId};
use crate::context::Context;
use crate::events::EventType;
use crate::login_param::{LoginParam, ServerLoginParam};
use crate::login_param::prioritize_server_login_params;
use crate::login_param::{ConfiguredLoginParam, ConfiguredServerLoginParam};
use crate::message::Message;
use crate::message::{self, MsgId};
use crate::mimefactory::MimeFactory;
use crate::net::proxy::ProxyConfig;
use crate::net::session::SessionBufStream;
use crate::oauth2::get_oauth2_access_token;
use crate::scheduler::connectivity::ConnectivityStore;
use crate::socks::Socks5Config;
use crate::sql;
use crate::stock_str::unencrypted_email;
use crate::tools::{self, time_elapsed};
@@ -88,96 +88,76 @@ impl Smtp {
}
self.connectivity.set_connecting(context).await;
let lp = LoginParam::load_configured_params(context).await?;
let lp = ConfiguredLoginParam::load(context)
.await?
.context("Not configured")?;
self.connect(
context,
&lp.smtp,
&lp.socks5_config,
&lp.smtp_password,
&lp.proxy_config,
&lp.addr,
lp.strict_tls(),
lp.oauth2,
)
.await
}
/// Connect using the provided login params.
#[allow(clippy::too_many_arguments)]
pub async fn connect(
&mut self,
context: &Context,
lp: &ServerLoginParam,
socks5_config: &Option<Socks5Config>,
login_params: &[ConfiguredServerLoginParam],
password: &str,
proxy_config: &Option<ProxyConfig>,
addr: &str,
strict_tls: bool,
oauth2: bool,
) -> Result<()> {
if self.is_connected() {
warn!(context, "SMTP already connected.");
return Ok(());
}
if lp.server.is_empty() || lp.port == 0 {
bail!("bad connection parameters");
}
let from = EmailAddress::new(addr.to_string())
.with_context(|| format!("invalid login address {addr}"))?;
.with_context(|| format!("Invalid address {addr:?}"))?;
self.from = Some(from);
let domain = &lp.server;
let port = lp.port;
let session_stream = connect::connect_stream(
context,
domain,
port,
strict_tls,
socks5_config.clone(),
lp.security,
)
.await?;
let client = smtp::SmtpClient::new().smtp_utf8(true).without_greeting();
let mut transport = SmtpTransport::new(client, session_stream).await?;
// Authenticate.
{
let (creds, mechanism) = if lp.oauth2 {
// oauth2
let send_pw = &lp.password;
let access_token = get_oauth2_access_token(context, addr, send_pw, false).await?;
if access_token.is_none() {
bail!("SMTP OAuth 2 error {}", addr);
let login_params =
prioritize_server_login_params(&context.sql, login_params, "smtp").await?;
for lp in login_params {
info!(context, "SMTP trying to connect to {}.", &lp.connection);
let transport = match connect::connect_and_auth(
context,
proxy_config,
strict_tls,
lp.connection.clone(),
oauth2,
addr,
&lp.user,
password,
)
.await
{
Ok(transport) => transport,
Err(err) => {
warn!(context, "SMTP failed to connect and authenticate: {err:#}.");
continue;
}
let user = &lp.user;
(
smtp::authentication::Credentials::new(
user.to_string(),
access_token.unwrap_or_default(),
),
vec![smtp::authentication::Mechanism::Xoauth2],
)
} else {
// plain
let user = lp.user.clone();
let pw = lp.password.clone();
(
smtp::authentication::Credentials::new(user, pw),
vec![
smtp::authentication::Mechanism::Plain,
smtp::authentication::Mechanism::Login,
],
)
};
transport.try_login(&creds, &mechanism).await?;
self.transport = Some(transport);
self.last_success = Some(tools::Time::now());
context.emit_event(EventType::SmtpConnected(format!(
"SMTP-LOGIN as {} ok",
lp.user,
)));
return Ok(());
}
self.transport = Some(transport);
self.last_success = Some(tools::Time::now());
context.emit_event(EventType::SmtpConnected(format!(
"SMTP-LOGIN as {} ok",
lp.user,
)));
Ok(())
Err(format_err!("SMTP failed to connect"))
}
}
@@ -505,7 +485,6 @@ pub(crate) async fn send_smtp_messages(context: &Context, connection: &mut Smtp)
let ratelimited = if context.ratelimit.read().await.can_send() {
// add status updates and sync messages to end of sending queue
context.flush_status_updates().await?;
context.send_sync_msg().await?;
false
} else {
true

View File

@@ -2,18 +2,20 @@
use std::net::SocketAddr;
use anyhow::{bail, format_err, Context as _, Result};
use anyhow::{bail, Context as _, Result};
use async_smtp::{SmtpClient, SmtpTransport};
use tokio::io::BufStream;
use tokio::io::{AsyncBufRead, AsyncWrite, BufStream};
use crate::context::Context;
use crate::login_param::{ConnectionCandidate, ConnectionSecurity};
use crate::net::dns::{lookup_host_with_cache, update_connect_timestamp};
use crate::net::proxy::ProxyConfig;
use crate::net::session::SessionBufStream;
use crate::net::tls::wrap_tls;
use crate::net::update_connection_history;
use crate::net::{connect_tcp_inner, connect_tls_inner};
use crate::provider::Socket;
use crate::socks::Socks5Config;
use crate::net::{
connect_tcp_inner, connect_tls_inner, run_connection_attempts, update_connection_history,
};
use crate::oauth2::get_oauth2_access_token;
use crate::tools::time;
/// Converts port number to ALPN list.
@@ -26,6 +28,115 @@ fn alpn(port: u16) -> &'static [&'static str] {
}
}
// Constructs a new SMTP transport
// over a stream with already skipped SMTP greeting.
async fn new_smtp_transport<S: AsyncBufRead + AsyncWrite + Unpin>(
stream: S,
) -> Result<SmtpTransport<S>> {
// We always read the greeting manually to unify
// the cases of STARTTLS where the greeting is
// sent outside the encrypted channel and implicit TLS
// where the greeting is sent after establishing TLS channel.
let client = SmtpClient::new().smtp_utf8(true).without_greeting();
let transport = SmtpTransport::new(client, stream)
.await
.context("Failed to send EHLO command")?;
Ok(transport)
}
#[allow(clippy::too_many_arguments)]
pub(crate) async fn connect_and_auth(
context: &Context,
proxy_config: &Option<ProxyConfig>,
strict_tls: bool,
candidate: ConnectionCandidate,
oauth2: bool,
addr: &str,
user: &str,
password: &str,
) -> Result<SmtpTransport<Box<dyn SessionBufStream>>> {
let session_stream = connect_stream(context, proxy_config.clone(), strict_tls, candidate)
.await
.context("SMTP failed to connect")?;
let mut transport = new_smtp_transport(session_stream).await?;
// Authenticate.
let (creds, mechanism) = if oauth2 {
// oauth2
let access_token = get_oauth2_access_token(context, addr, password, false)
.await
.context("SMTP failed to get OAUTH2 access token")?;
if access_token.is_none() {
bail!("SMTP OAuth 2 error {}", addr);
}
(
async_smtp::authentication::Credentials::new(
user.to_string(),
access_token.unwrap_or_default(),
),
vec![async_smtp::authentication::Mechanism::Xoauth2],
)
} else {
// plain
(
async_smtp::authentication::Credentials::new(user.to_string(), password.to_string()),
vec![
async_smtp::authentication::Mechanism::Plain,
async_smtp::authentication::Mechanism::Login,
],
)
};
transport
.try_login(&creds, &mechanism)
.await
.context("SMTP failed to login")?;
Ok(transport)
}
async fn connection_attempt(
context: Context,
host: String,
security: ConnectionSecurity,
resolved_addr: SocketAddr,
strict_tls: bool,
) -> Result<Box<dyn SessionBufStream>> {
let context = &context;
let host = &host;
info!(
context,
"Attempting SMTP connection to {host} ({resolved_addr})."
);
let res = match security {
ConnectionSecurity::Tls => connect_secure(resolved_addr, host, strict_tls).await,
ConnectionSecurity::Starttls => connect_starttls(resolved_addr, host, strict_tls).await,
ConnectionSecurity::Plain => connect_insecure(resolved_addr).await,
};
match res {
Ok(stream) => {
let ip_addr = resolved_addr.ip().to_string();
let port = resolved_addr.port();
let save_cache = match security {
ConnectionSecurity::Tls | ConnectionSecurity::Starttls => strict_tls,
ConnectionSecurity::Plain => false,
};
if save_cache {
update_connect_timestamp(context, host, &ip_addr).await?;
}
update_connection_history(context, "smtp", host, port, &ip_addr, time()).await?;
Ok(stream)
}
Err(err) => {
warn!(
context,
"Failed to connect to {host} ({resolved_addr}): {err:#}."
);
Err(err)
}
}
}
/// Returns TLS, STARTTLS or plaintext connection
/// using SOCKS5 or direct connection depending on the given configuration.
///
@@ -34,59 +145,46 @@ fn alpn(port: u16) -> &'static [&'static str] {
/// does not send welcome message over TLS connection
/// after establishing it, welcome message is always ignored
/// to unify the result regardless of whether TLS or STARTTLS is used.
pub(crate) async fn connect_stream(
async fn connect_stream(
context: &Context,
host: &str,
port: u16,
proxy_config: Option<ProxyConfig>,
strict_tls: bool,
socks5_config: Option<Socks5Config>,
security: Socket,
candidate: ConnectionCandidate,
) -> Result<Box<dyn SessionBufStream>> {
if let Some(socks5_config) = socks5_config {
let host = &candidate.host;
let port = candidate.port;
let security = candidate.security;
if let Some(proxy_config) = proxy_config {
let stream = match security {
Socket::Automatic => bail!("SMTP port security is not configured"),
Socket::Ssl => {
connect_secure_socks5(context, host, port, strict_tls, socks5_config.clone())
ConnectionSecurity::Tls => {
connect_secure_proxy(context, host, port, strict_tls, proxy_config.clone()).await?
}
ConnectionSecurity::Starttls => {
connect_starttls_proxy(context, host, port, strict_tls, proxy_config.clone())
.await?
}
Socket::Starttls => {
connect_starttls_socks5(context, host, port, strict_tls, socks5_config.clone())
.await?
}
Socket::Plain => {
connect_insecure_socks5(context, host, port, socks5_config.clone()).await?
ConnectionSecurity::Plain => {
connect_insecure_proxy(context, host, port, proxy_config.clone()).await?
}
};
update_connection_history(context, "smtp", host, port, host, time()).await?;
Ok(stream)
} else {
let mut first_error = None;
let load_cache = strict_tls && (security == Socket::Ssl || security == Socket::Starttls);
let load_cache = match security {
ConnectionSecurity::Tls | ConnectionSecurity::Starttls => strict_tls,
ConnectionSecurity::Plain => false,
};
for resolved_addr in lookup_host_with_cache(context, host, port, "smtp", load_cache).await?
{
let res = match security {
Socket::Automatic => bail!("SMTP port security is not configured"),
Socket::Ssl => connect_secure(resolved_addr, host, strict_tls).await,
Socket::Starttls => connect_starttls(resolved_addr, host, strict_tls).await,
Socket::Plain => connect_insecure(resolved_addr).await,
};
match res {
Ok(stream) => {
let ip_addr = resolved_addr.ip().to_string();
if load_cache {
update_connect_timestamp(context, host, &ip_addr).await?;
}
update_connection_history(context, "smtp", host, port, &ip_addr, time())
.await?;
return Ok(stream);
}
Err(err) => {
warn!(context, "Failed to connect to {resolved_addr}: {err:#}.");
first_error.get_or_insert(err);
}
}
}
Err(first_error.unwrap_or_else(|| format_err!("no DNS resolution results for {host}")))
let connection_futures = lookup_host_with_cache(context, host, port, "smtp", load_cache)
.await?
.into_iter()
.map(|resolved_addr| {
let context = context.clone();
let host = host.to_string();
connection_attempt(context, host, security, resolved_addr, strict_tls)
});
run_connection_attempts(connection_futures).await
}
}
@@ -100,51 +198,55 @@ async fn skip_smtp_greeting<R: tokio::io::AsyncBufReadExt + Unpin>(stream: &mut
let mut line = String::with_capacity(512);
loop {
line.clear();
let read = stream.read_line(&mut line).await?;
let read = stream
.read_line(&mut line)
.await
.context("Failed to read from stream while waiting for SMTP greeting")?;
if read == 0 {
bail!("Unexpected EOF while reading SMTP greeting.");
bail!("Unexpected EOF while reading SMTP greeting");
}
if line.starts_with("220-") {
continue;
} else if line.starts_with("220 ") {
return Ok(());
} else {
bail!("Unexpected greeting: {line:?}.");
bail!("Unexpected greeting: {line:?}");
}
}
}
async fn connect_secure_socks5(
async fn connect_secure_proxy(
context: &Context,
hostname: &str,
port: u16,
strict_tls: bool,
socks5_config: Socks5Config,
proxy_config: ProxyConfig,
) -> Result<Box<dyn SessionBufStream>> {
let socks5_stream = socks5_config
let proxy_stream = proxy_config
.connect(context, hostname, port, strict_tls)
.await?;
let tls_stream = wrap_tls(strict_tls, hostname, alpn(port), socks5_stream).await?;
let tls_stream = wrap_tls(strict_tls, hostname, alpn(port), proxy_stream).await?;
let mut buffered_stream = BufStream::new(tls_stream);
skip_smtp_greeting(&mut buffered_stream).await?;
let session_stream: Box<dyn SessionBufStream> = Box::new(buffered_stream);
Ok(session_stream)
}
async fn connect_starttls_socks5(
async fn connect_starttls_proxy(
context: &Context,
hostname: &str,
port: u16,
strict_tls: bool,
socks5_config: Socks5Config,
proxy_config: ProxyConfig,
) -> Result<Box<dyn SessionBufStream>> {
let socks5_stream = socks5_config
let proxy_stream = proxy_config
.connect(context, hostname, port, strict_tls)
.await?;
// Run STARTTLS command and convert the client back into a stream.
let client = SmtpClient::new().smtp_utf8(true);
let transport = SmtpTransport::new(client, BufStream::new(socks5_stream)).await?;
let mut buffered_stream = BufStream::new(proxy_stream);
skip_smtp_greeting(&mut buffered_stream).await?;
let transport = new_smtp_transport(buffered_stream).await?;
let tcp_stream = transport.starttls().await?.into_inner();
let tls_stream = wrap_tls(strict_tls, hostname, &[], tcp_stream)
.await
@@ -154,16 +256,14 @@ async fn connect_starttls_socks5(
Ok(session_stream)
}
async fn connect_insecure_socks5(
async fn connect_insecure_proxy(
context: &Context,
hostname: &str,
port: u16,
socks5_config: Socks5Config,
proxy_config: ProxyConfig,
) -> Result<Box<dyn SessionBufStream>> {
let socks5_stream = socks5_config
.connect(context, hostname, port, false)
.await?;
let mut buffered_stream = BufStream::new(socks5_stream);
let proxy_stream = proxy_config.connect(context, hostname, port, false).await?;
let mut buffered_stream = BufStream::new(proxy_stream);
skip_smtp_greeting(&mut buffered_stream).await?;
let session_stream: Box<dyn SessionBufStream> = Box::new(buffered_stream);
Ok(session_stream)
@@ -189,8 +289,9 @@ async fn connect_starttls(
let tcp_stream = connect_tcp_inner(addr).await?;
// Run STARTTLS command and convert the client back into a stream.
let client = async_smtp::SmtpClient::new().smtp_utf8(true);
let transport = async_smtp::SmtpTransport::new(client, BufStream::new(tcp_stream)).await?;
let mut buffered_stream = BufStream::new(tcp_stream);
skip_smtp_greeting(&mut buffered_stream).await?;
let transport = new_smtp_transport(buffered_stream).await?;
let tcp_stream = transport.starttls().await?.into_inner();
let tls_stream = wrap_tls(strict_tls, host, &[], tcp_stream)
.await

View File

@@ -1,148 +0,0 @@
//! # SOCKS5 support.
use std::fmt;
use std::pin::Pin;
use anyhow::Result;
use fast_socks5::client::{Config, Socks5Stream};
use fast_socks5::util::target_addr::ToTargetAddr;
use fast_socks5::AuthenticationMethod;
use fast_socks5::Socks5Command;
use percent_encoding::{utf8_percent_encode, NON_ALPHANUMERIC};
use tokio::net::TcpStream;
use tokio_io_timeout::TimeoutStream;
use crate::context::Context;
use crate::net::connect_tcp;
use crate::sql::Sql;
#[derive(Default, Debug, Clone, PartialEq, Eq)]
pub struct Socks5Config {
pub host: String,
pub port: u16,
pub user_password: Option<(String, String)>,
}
impl Socks5Config {
/// Reads SOCKS5 configuration from the database.
pub async fn from_database(sql: &Sql) -> Result<Option<Self>> {
let enabled = sql.get_raw_config_bool("socks5_enabled").await?;
if enabled {
let host = sql.get_raw_config("socks5_host").await?.unwrap_or_default();
let port: u16 = sql
.get_raw_config_int("socks5_port")
.await?
.unwrap_or_default() as u16;
let user = sql.get_raw_config("socks5_user").await?.unwrap_or_default();
let password = sql
.get_raw_config("socks5_password")
.await?
.unwrap_or_default();
let socks5_config = Self {
host,
port,
user_password: if !user.is_empty() {
Some((user, password))
} else {
None
},
};
Ok(Some(socks5_config))
} else {
Ok(None)
}
}
/// Converts SOCKS5 configuration into URL.
pub fn to_url(&self) -> String {
// `socks5h` means that hostname is resolved into address by the proxy
// and DNS requests should not leak.
let mut url = "socks5h://".to_string();
if let Some((username, password)) = &self.user_password {
let username_urlencoded = utf8_percent_encode(username, NON_ALPHANUMERIC).to_string();
let password_urlencoded = utf8_percent_encode(password, NON_ALPHANUMERIC).to_string();
url += &format!("{username_urlencoded}:{password_urlencoded}@");
}
url += &format!("{}:{}", self.host, self.port);
url
}
/// If `load_dns_cache` is true, loads cached DNS resolution results.
/// Use this only if the connection is going to be protected with TLS checks.
pub async fn connect(
&self,
context: &Context,
target_host: &str,
target_port: u16,
load_dns_cache: bool,
) -> Result<Socks5Stream<Pin<Box<TimeoutStream<TcpStream>>>>> {
let tcp_stream = connect_tcp(context, &self.host, self.port, load_dns_cache).await?;
let authentication_method = if let Some((username, password)) = self.user_password.as_ref()
{
Some(AuthenticationMethod::Password {
username: username.into(),
password: password.into(),
})
} else {
None
};
let mut socks_stream =
Socks5Stream::use_stream(tcp_stream, authentication_method, Config::default()).await?;
let target_addr = (target_host, target_port).to_target_addr()?;
socks_stream
.request(Socks5Command::TCPConnect, target_addr)
.await?;
Ok(socks_stream)
}
}
impl fmt::Display for Socks5Config {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(
f,
"host:{},port:{},user_password:{}",
self.host,
self.port,
if let Some(user_password) = self.user_password.clone() {
format!("user: {}, password: ***", user_password.0)
} else {
"user: None".to_string()
}
)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_socks5h_url() {
let config = Socks5Config {
host: "127.0.0.1".to_string(),
port: 9050,
user_password: None,
};
assert_eq!(config.to_url(), "socks5h://127.0.0.1:9050");
let config = Socks5Config {
host: "example.org".to_string(),
port: 1080,
user_password: Some(("root".to_string(), "toor".to_string())),
};
assert_eq!(config.to_url(), "socks5h://root:toor@example.org:1080");
let config = Socks5Config {
host: "example.org".to_string(),
port: 1080,
user_password: Some(("root".to_string(), "foo/?\\@".to_string())),
};
assert_eq!(
config.to_url(),
"socks5h://root:foo%2F%3F%5C%40@example.org:1080"
);
}
}

View File

@@ -5,7 +5,7 @@ use std::path::{Path, PathBuf};
use anyhow::{bail, Context as _, Result};
use rusqlite::{config::DbConfig, types::ValueRef, Connection, OpenFlags, Row};
use tokio::sync::{Mutex, MutexGuard, RwLock};
use tokio::sync::RwLock;
use crate::blob::BlobObject;
use crate::chat::{self, add_device_msg, update_device_icon, update_saved_messages_icon};
@@ -60,11 +60,6 @@ pub struct Sql {
/// Database file path
pub(crate) dbfile: PathBuf,
/// Write transactions mutex.
///
/// See [`Self::write_lock`].
write_mtx: Mutex<()>,
/// SQL connection pool.
pool: RwLock<Option<Pool>>,
@@ -81,7 +76,6 @@ impl Sql {
pub fn new(dbfile: PathBuf) -> Sql {
Self {
dbfile,
write_mtx: Mutex::new(()),
pool: Default::default(),
is_encrypted: Default::default(),
config_cache: Default::default(),
@@ -147,7 +141,8 @@ impl Sql {
let mut config_cache = self.config_cache.write().await;
config_cache.clear();
self.call_write(move |conn| {
let query_only = false;
self.call(query_only, move |conn| {
// Check that backup passphrase is correct before resetting our database.
conn.execute("ATTACH DATABASE ? AS backup KEY ?", (path_str, passphrase))
.context("failed to attach backup database")?;
@@ -325,7 +320,8 @@ impl Sql {
let mut lock = self.pool.write().await;
let pool = lock.take().context("SQL connection pool is not open")?;
let conn = pool.get().await?;
let query_only = false;
let conn = pool.get(query_only).await?;
if !passphrase.is_empty() {
conn.pragma_update(None, "rekey", passphrase.clone())
.context("Failed to set PRAGMA rekey")?;
@@ -337,59 +333,20 @@ impl Sql {
Ok(())
}
/// Locks the write transactions mutex in order to make sure that there never are
/// multiple write transactions at once.
/// Allocates a connection and calls `function` with the connection.
///
/// Doing the locking ourselves instead of relying on SQLite has these reasons:
///
/// - SQLite's locking mechanism is non-async, blocking a thread
/// - SQLite's locking mechanism just sleeps in a loop, which is really inefficient
///
/// ---
///
/// More considerations on alternatives to the current approach:
///
/// We use [DEFERRED](https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions) transactions.
///
/// In order to never get concurrency issues, we could make all transactions IMMEDIATE,
/// but this would mean that there can never be two simultaneous transactions.
///
/// Read transactions can simply be made DEFERRED to run in parallel w/o any drawbacks.
///
/// DEFERRED write transactions without doing the locking ourselves would have these drawbacks:
///
/// 1. As mentioned above, SQLite's locking mechanism is non-async and sleeps in a loop.
/// 2. If there are other write transactions, we block the db connection until
/// upgraded. If some reader comes then, it has to get the next, less used connection with a
/// worse per-connection page cache (SQLite allows one write and any number of reads in parallel).
/// 3. If a transaction is blocked for more than `busy_timeout`, it fails with SQLITE_BUSY.
/// 4. If upon a successful upgrade to a write transaction the db has been modified,
/// the transaction has to be rolled back and retried, which means extra work in terms of
/// CPU/battery.
///
/// The only pro of making write transactions DEFERRED w/o the external locking would be some
/// parallelism between them.
///
/// Another option would be to make write transactions IMMEDIATE, also
/// w/o the external locking. But then cons 1. - 3. above would still be valid.
pub async fn write_lock(&self) -> MutexGuard<'_, ()> {
self.write_mtx.lock().await
}
/// Allocates a connection and calls `function` with the connection. If `function` does write
/// queries,
/// - either first take a lock using `write_lock()`
/// - or use `call_write()` instead.
/// If `query_only` is true, allocates read-only connection,
/// otherwise allocates write connection.
///
/// Returns the result of the function.
async fn call<'a, F, R>(&'a self, function: F) -> Result<R>
async fn call<'a, F, R>(&'a self, query_only: bool, function: F) -> Result<R>
where
F: 'a + FnOnce(&mut Connection) -> Result<R> + Send,
R: Send + 'static,
{
let lock = self.pool.read().await;
let pool = lock.as_ref().context("no SQL connection")?;
let mut conn = pool.get().await?;
let mut conn = pool.get(query_only).await?;
let res = tokio::task::block_in_place(move || function(&mut conn))?;
Ok(res)
}
@@ -403,8 +360,8 @@ impl Sql {
F: 'a + FnOnce(&mut Connection) -> Result<R> + Send,
R: Send + 'static,
{
let _lock = self.write_lock().await;
self.call(function).await
let query_only = false;
self.call(query_only, function).await
}
/// Execute `query` assuming it is a write query, returning the number of affected rows.
@@ -444,7 +401,8 @@ impl Sql {
G: Send + FnMut(rusqlite::MappedRows<F>) -> Result<H>,
H: Send + 'static,
{
self.call(move |conn| {
let query_only = true;
self.call(query_only, move |conn| {
let mut stmt = conn.prepare(sql)?;
let res = stmt.query_map(params, f)?;
g(res)
@@ -476,7 +434,8 @@ impl Sql {
F: FnOnce(&rusqlite::Row) -> rusqlite::Result<T> + Send,
T: Send + 'static,
{
self.call(move |conn| {
let query_only = true;
self.call(query_only, move |conn| {
let res = conn.query_row(query, params, f)?;
Ok(res)
})
@@ -512,7 +471,8 @@ impl Sql {
/// Query the database if the requested table already exists.
pub async fn table_exists(&self, name: &str) -> Result<bool> {
self.call(move |conn| {
let query_only = true;
self.call(query_only, move |conn| {
let mut exists = false;
conn.pragma(None, "table_info", name.to_string(), |_row| {
// will only be executed if the info was found
@@ -527,7 +487,8 @@ impl Sql {
/// Check if a column exists in a given table.
pub async fn col_exists(&self, table_name: &str, col_name: &str) -> Result<bool> {
self.call(move |conn| {
let query_only = true;
self.call(query_only, move |conn| {
let mut exists = false;
// `PRAGMA table_info` returns one row per column,
// each row containing 0=cid, 1=name, 2=type, 3=notnull, 4=dflt_value
@@ -555,18 +516,19 @@ impl Sql {
F: Send + FnOnce(&rusqlite::Row) -> rusqlite::Result<T>,
T: Send + 'static,
{
self.call(move |conn| match conn.query_row(sql.as_ref(), params, f) {
Ok(res) => Ok(Some(res)),
Err(rusqlite::Error::QueryReturnedNoRows) => Ok(None),
Err(rusqlite::Error::InvalidColumnType(_, _, rusqlite::types::Type::Null)) => Ok(None),
Err(err) => Err(err.into()),
let query_only = true;
self.call(query_only, move |conn| {
match conn.query_row(sql.as_ref(), params, f) {
Ok(res) => Ok(Some(res)),
Err(rusqlite::Error::QueryReturnedNoRows) => Ok(None),
Err(err) => Err(err.into()),
}
})
.await
}
/// Executes a query which is expected to return one row and one
/// column. If the query does not return a value or returns SQL
/// `NULL`, returns `Ok(None)`.
/// column. If the query does not return any rows, returns `Ok(None)`.
pub async fn query_get_value<T>(
&self,
query: &str,
@@ -719,6 +681,36 @@ fn new_connection(path: &Path, passphrase: &str) -> Result<Connection> {
Ok(conn)
}
// Tries to clear the freelist to free some space on the disk.
//
// This only works if auto_vacuum is enabled.
async fn incremental_vacuum(context: &Context) -> Result<()> {
context
.sql
.call_write(move |conn| {
let mut stmt = conn
.prepare("PRAGMA incremental_vacuum")
.context("Failed to prepare incremental_vacuum statement")?;
// It is important to step the statement until it returns no more rows.
// Otherwise it will not free as many pages as it can:
// <https://stackoverflow.com/questions/53746807/sqlite-incremental-vacuum-removing-only-one-free-page>.
let mut rows = stmt
.query(())
.context("Failed to run incremental_vacuum statement")?;
let mut row_count = 0;
while let Some(_row) = rows
.next()
.context("Failed to step incremental_vacuum statement")?
{
row_count += 1;
}
info!(context, "Incremental vacuum freed {row_count} pages.");
Ok(())
})
.await
}
/// Cleanup the account to restore some storage and optimize the database.
pub async fn housekeeping(context: &Context) -> Result<()> {
// Setting `Config::LastHousekeeping` at the beginning avoids endless loops when things do not
@@ -751,24 +743,8 @@ pub async fn housekeeping(context: &Context) -> Result<()> {
);
}
// Try to clear the freelist to free some space on the disk. This
// only works if auto_vacuum is enabled.
match context
.sql
.query_row_optional("PRAGMA incremental_vacuum", (), |_row| Ok(()))
.await
{
Err(err) => {
warn!(context, "Failed to run incremental vacuum: {err:#}.");
}
Ok(Some(())) => {
// Incremental vacuum returns a zero-column result if it did anything.
info!(context, "Successfully ran incremental vacuum.");
}
Ok(None) => {
// Incremental vacuum returned `SQLITE_DONE` immediately,
// there were no pages to remove.
}
if let Err(err) = incremental_vacuum(context).await {
warn!(context, "Failed to run incremental vacuum: {err:#}.");
}
context
@@ -1094,9 +1070,10 @@ mod tests {
async fn test_auto_vacuum() -> Result<()> {
let t = TestContext::new().await;
let query_only = true;
let auto_vacuum = t
.sql
.call(|conn| {
.call(query_only, |conn| {
let auto_vacuum = conn.pragma_query_value(None, "auto_vacuum", |row| {
let auto_vacuum: i32 = row.get(0)?;
Ok(auto_vacuum)
@@ -1322,8 +1299,9 @@ mod tests {
{
let lock = sql.pool.read().await;
let pool = lock.as_ref().unwrap();
let conn1 = pool.get().await?;
let conn2 = pool.get().await?;
let query_only = true;
let conn1 = pool.get(query_only).await?;
let conn2 = pool.get(query_only).await?;
conn1
.query_row("SELECT count(*) FROM sqlite_master", [], |_row| Ok(()))
.unwrap();
@@ -1348,4 +1326,72 @@ mod tests {
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_query_only() -> Result<()> {
let t = TestContext::new().await;
// `query_row` does not acquire write lock
// and operates on read-only connection.
// Using it to `INSERT` should fail.
let res = t
.sql
.query_row(
"INSERT INTO config (keyname, value) VALUES (?, ?) RETURNING 1",
("xyz", "ijk"),
|row| {
let res: u32 = row.get(0)?;
Ok(res)
},
)
.await;
assert!(res.is_err());
// If you want to `INSERT` and get value via `RETURNING`,
// use `call_write` or `transaction`.
let res: Result<u32> = t
.sql
.call_write(|conn| {
let val = conn.query_row(
"INSERT INTO config (keyname, value) VALUES (?, ?) RETURNING 2",
("foo", "bar"),
|row| {
let res: u32 = row.get(0)?;
Ok(res)
},
)?;
Ok(val)
})
.await;
assert_eq!(res.unwrap(), 2);
let res = t
.sql
.transaction(|t| {
let val = t.query_row(
"INSERT INTO config (keyname, value) VALUES (?, ?) RETURNING 3",
("abc", "def"),
|row| {
let res: u32 = row.get(0)?;
Ok(res)
},
)?;
Ok(val)
})
.await;
assert_eq!(res.unwrap(), 3);
Ok(())
}
/// Tests that incremental_vacuum does not fail.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_incremental_vacuum() -> Result<()> {
let t = TestContext::new().await;
incremental_vacuum(&t).await?;
Ok(())
}
}

View File

@@ -971,6 +971,114 @@ CREATE INDEX msgs_status_updates_index2 ON msgs_status_updates (uid);
.await?;
}
inc_and_check(&mut migration_version, 118)?;
if dbversion < migration_version {
sql.execute_migration(
"CREATE TABLE tokens_new (
id INTEGER PRIMARY KEY,
namespc INTEGER DEFAULT 0,
foreign_key TEXT DEFAULT '',
token TEXT DEFAULT '',
timestamp INTEGER DEFAULT 0
) STRICT;
INSERT INTO tokens_new
SELECT t.id, t.namespc, IFNULL(c.grpid, ''), t.token, t.timestamp
FROM tokens t LEFT JOIN chats c ON t.foreign_id=c.id;
DROP TABLE tokens;
ALTER TABLE tokens_new RENAME TO tokens;",
migration_version,
)
.await?;
}
inc_and_check(&mut migration_version, 119)?;
if dbversion < migration_version {
sql.execute_migration(
"CREATE TABLE imap_send (
id INTEGER PRIMARY KEY AUTOINCREMENT,
mime TEXT NOT NULL, -- Message content
msg_id INTEGER NOT NULL, -- ID of the message in the `msgs` table
attempts INTEGER NOT NULL DEFAULT 0 -- Number of failed attempts to send the message
)",
migration_version,
)
.await?;
}
inc_and_check(&mut migration_version, 120)?;
if dbversion < migration_version {
// Core 1.143.0 changed the default for `delete_server_after`
// to delete immediately (`1`) for chatmail accounts that don't have multidevice
// and updating to `0` when backup is exported.
//
// Since we don't know if existing configurations
// are multidevice, we set `delete_server_after` for them
// to the old default of `0`, so only new configurations are
// affected by the default change.
//
// `INSERT OR IGNORE` works
// because `keyname` was made UNIQUE in migration 106.
sql.execute_migration(
"INSERT OR IGNORE INTO config (keyname, value)
SELECT 'delete_server_after', '0'
FROM config WHERE keyname='configured'
",
migration_version,
)
.await?;
}
inc_and_check(&mut migration_version, 121)?;
if dbversion < migration_version {
sql.execute_migration(
"CREATE INDEX chats_index4 ON chats (name)",
migration_version,
)
.await?;
}
inc_and_check(&mut migration_version, 122)?;
if dbversion < migration_version {
sql.execute_migration(
"ALTER TABLE tokens ADD COLUMN foreign_id INTEGER NOT NULL DEFAULT 0",
migration_version,
)
.await?;
}
inc_and_check(&mut migration_version, 123)?;
if dbversion < migration_version {
// Add FOREIGN KEY(msg_id).
sql.execute_migration(
"CREATE TABLE new_msgs_status_updates (
id INTEGER PRIMARY KEY AUTOINCREMENT,
msg_id INTEGER,
update_item TEXT DEFAULT '',
uid TEXT UNIQUE,
FOREIGN KEY(msg_id) REFERENCES msgs(id) ON DELETE CASCADE
);
INSERT OR IGNORE INTO new_msgs_status_updates SELECT
id, msg_id, update_item, uid
FROM msgs_status_updates;
DROP TABLE msgs_status_updates;
ALTER TABLE new_msgs_status_updates RENAME TO msgs_status_updates;
CREATE INDEX msgs_status_updates_index1 ON msgs_status_updates (msg_id);
CREATE INDEX msgs_status_updates_index2 ON msgs_status_updates (uid);
",
migration_version,
)
.await?;
}
inc_and_check(&mut migration_version, 124)?;
if dbversion < migration_version {
sql.execute_migration(
"INSERT INTO contacts (name, addr, origin, authname, is_bot) VALUES ('xstore', 'xstore@testrun.org', 67108864, '', true);",
migration_version,
)
.await?;
}
let new_version = sql
.get_raw_config_int(VERSION_CFG)
.await?
@@ -1034,3 +1142,27 @@ impl Sql {
self.set_db_version_in_cache(version).await
}
}
#[cfg(test)]
mod tests {
use crate::{contact, test_utils::TestContext};
use anyhow::Result;
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_name() -> Result<()> {
let bob = TestContext::new_bob().await;
let xstore_id = contact::Contact::lookup_id_by_addr(
&bob,
"xstore@testrun.org",
contact::Origin::Unknown,
)
.await?
.expect("Expect xstore contact");
let xstore = contact::Contact::get_by_id(&bob, xstore_id).await?;
assert_eq!(xstore.get_name(), "xstore");
assert_eq!(xstore.get_addr(), "xstore@testrun.org");
assert_eq!(xstore.origin, contact::Origin::ManuallyCreated);
assert_eq!(xstore.is_bot(), true);
Ok(())
}
}

Some files were not shown because too many files have changed in this diff Show More