Compare commits

...
20 Commits
Author SHA1 Message Date
link2xt 3d696e08e5 rename 2023-02-26 19:47:17 +00:00
link2xt aab35c3703 Add cargo-deny config and CI 2023-02-26 19:45:30 +00:00
link2xt 8177070673 Set minimum TLS version to 1.2 2023-02-26 18:48:32 +00:00
B. Petersen 247bf5865d add countermitm and openpgp4fpr to "standard used"
noticed that this is missing
while working on https://github.com/deltachat/invite
2023-02-26 15:55:50 +01:00
link2xt eace8c5fee Note that 1.108.0 re-enables SMTP pipelining 2023-02-25 15:02:05 +00:00
B. Petersen f8e86f4503 make slowest test faster
the slowest test was `test_modify_chat_disordered`;
due to several sleep(), it lasts at least 11 seconds.

however, many of the sleep() are not needed and were added
just to get things done that time.

this pr removes 7 superfluous sleeps,
making the test finish in about 3 seconds,
so that this test is no longer the bottleneck when running
`cargo test` on fast machines.
(this is not only useful to safe some seconds -
but also to notice degradations as of
https://github.com/deltachat/deltachat-core-rust/issues/4051#issuecomment-1436995166 )
2023-02-25 15:37:10 +01:00
iequidoo d1923d68a5 Add a config option to sign all messages with Autocrypt header (#3986)
Although it does a little for security, it will help to protect from unwanted server-side
modifications and bugs. And now we have a time to test "multipart/signed" messages compatibility
with other MUAs.
2023-02-25 10:30:35 -03:00
iequidoo 89696582ad mimeparser: Handle headers from the signed part of unencrypted signed message
This makes DC compatible with "multipart/signed" messages thus allowing switching to them someday
from the current "multipart/mixed" unencrypted message format.
2023-02-25 10:30:35 -03:00
link2xt 992a6cbfc2 mimeparser: wrap try_decrypt() into block_in_place()
try_decrypt() is a CPU-bound task.
When called from async function,
it should be wrapped in tokio::task::spawn_blocking().
Using tokio::task::spawn_blocking() is difficult here
because of &mail, &private_keyring and &public_keyring borrows,
so we should at least use tokio::task::block_in_place()
to avoid blocking the executor.
2023-02-25 11:00:05 +00:00
link2xt 1450bf5483 Remove Job::delay_seconds()
It always returned 0 for added jobs.
2023-02-25 02:45:22 +00:00
link2xt 9f804c379c Remove always zero argument to Job::new() 2023-02-25 02:42:14 +00:00
link2xt 10e8bcb73e job: remove unused variable 2023-02-25 02:35:08 +00:00
link2xt 7f2ccfb168 job: remove match with a single branch 2023-02-25 02:33:47 +00:00
link2xt 3fc67de35e Stop saving and loading jobs.param column 2023-02-25 02:33:21 +00:00
link2xt 8f0d07b93c Make smeared timestamp creation non-async
Using atomic operations instead,
so create_smeared_timestamp() can be used in sync functions,
such as SQL transactions.
2023-02-25 01:10:56 +00:00
link2xt 0890b669fa Move RFC URLs in standards.md to the end
Also update Autodiscover URL
2023-02-24 23:47:51 +00:00
link2xt c02c5ffe2c Format docs_wheels.yml with prettier 2023-02-24 22:31:04 +00:00
link2xt 064f806d90 Remove UpdateRecentQuota job 2023-02-24 17:49:08 +00:00
link2xt b6336ce7e9 Add Unreleased changelog section 2023-02-24 17:46:45 +00:00
link2xt aeadbb35f3 Remove empty API-Changes section 2023-02-24 17:46:35 +00:00
26 changed files with 908 additions and 590 deletions
+8
View File
@@ -32,6 +32,14 @@ jobs:
- name: Run clippy
run: scripts/clippy.sh
foo:
name: bar
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: EmbarkStudios/cargo-deny-action@v1
arguments: --all-features --workspace
docs:
name: Rust doc comments
runs-on: ubuntu-latest
+14 -2
View File
@@ -1,5 +1,17 @@
# Changelog
## Unreleased
### Changes
- Make smeared timestamp generation non-async. #4075
- Set minimum TLS version to 1.2. #4096
### Fixes
- Do not block async task executor while decrypting the messages. #4079
### API-Changes
## 1.110.0
### Changes
@@ -12,6 +24,7 @@
- Updated provider database.
- Disable DKIM-Checks again #4076
- Switch from "X.Y.Z" and "py-X.Y.Z" to "vX.Y.Z" tags. #4089
- mimeparser: handle headers from the signed part of unencrypted signed message #4013
### Fixes
- Start SQL transactions with IMMEDIATE behaviour rather than default DEFERRED one. #4063
@@ -20,8 +33,6 @@
- Clear config cache after backup import. This bug sometimes resulted in the import to seemingly work at first. #4067
- Update timestamps in `param` columns with transactions. #4083
### API-Changes
## 1.109.0
@@ -52,6 +63,7 @@
- Prefer TLS over STARTTLS during autoconfiguration #4021
- Use SOCKS5 configuration for HTTP requests #4017
- Show non-deltachat emails by default for new installations #4019
- Re-enabled SMTP pipelining after disabling it in #4006
### Fixes
- Fix Securejoin for multiple devices on a joining side #3982
+1
View File
@@ -1,6 +1,7 @@
[package]
name = "deltachat-repl"
version = "1.110.0"
license = "MPL-2.0"
edition = "2021"
[dependencies]
+16
View File
@@ -0,0 +1,16 @@
[advisories]
ignore = [
"RUSTSEC-2020-0071",
]
[licenses]
allow = [
"0BSD",
"Apache-2.0",
"BSD-2-Clause",
"BSD-3-Clause",
"CC0-1.0",
"MIT",
"BSL-1.0", # Boost Software License 1.0
"Unicode-DFS-2016",
]
+347 -347
View File
@@ -1,370 +1,370 @@
resources:
- name: deltachat-core-rust
type: git
icon: github
source:
branch: master
uri: https://github.com/deltachat/deltachat-core-rust.git
- name: deltachat-core-rust
type: git
icon: github
source:
branch: master
uri: https://github.com/deltachat/deltachat-core-rust.git
- name: deltachat-core-rust-release
type: git
icon: github
source:
branch: master
uri: https://github.com/deltachat/deltachat-core-rust.git
tag_filter: "v*"
- name: deltachat-core-rust-release
type: git
icon: github
source:
branch: master
uri: https://github.com/deltachat/deltachat-core-rust.git
tag_filter: "v*"
jobs:
- name: doxygen
plan:
- get: deltachat-core-rust
trigger: true
- name: doxygen
plan:
- get: deltachat-core-rust
trigger: true
# Build Doxygen documentation
- task: build-doxygen
config:
inputs:
- name: deltachat-core-rust
outputs:
- name: c-docs
image_resource:
source:
repository: alpine
type: registry-image
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache doxygen git
cd deltachat-core-rust
scripts/run-doxygen.sh
cd ..
cp -av deltachat-core-rust/deltachat-ffi/html deltachat-core-rust/deltachat-ffi/xml c-docs/
# Build Doxygen documentation
- task: build-doxygen
config:
inputs:
- name: deltachat-core-rust
outputs:
- name: c-docs
image_resource:
source:
repository: alpine
type: registry-image
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache doxygen git
cd deltachat-core-rust
scripts/run-doxygen.sh
cd ..
cp -av deltachat-core-rust/deltachat-ffi/html deltachat-core-rust/deltachat-ffi/xml c-docs/
- task: upload-c-docs
config:
inputs:
- name: c-docs
image_resource:
type: registry-image
source:
repository: alpine
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache rsync openssh-client
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete c-docs/html/ delta@c.delta.chat:build-c/master
- task: upload-c-docs
config:
inputs:
- name: c-docs
image_resource:
type: registry-image
source:
repository: alpine
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache rsync openssh-client
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete c-docs/html/ delta@c.delta.chat:build-c/master
- name: python-x86_64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
- name: python-x86_64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/manylinux2014_x86_64
platform: linux
caches:
- path: cache
run:
path: build
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/manylinux2014_x86_64
platform: linux
caches:
- path: cache
run:
path: build
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-docs
path: ./python/doc/_build/
# Binary wheels
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-docs
path: ./python/doc/_build/
# Binary wheels
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Upload python docs to py.delta.chat
- task: upload-py-docs
config:
inputs:
- name: py-docs
image_resource:
type: registry-image
source:
repository: alpine
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache rsync openssh-client
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete py-docs/html/ delta@py.delta.chat:build/master
# Upload python docs to py.delta.chat
- task: upload-py-docs
config:
inputs:
- name: py-docs
image_resource:
type: registry-image
source:
repository: alpine
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache rsync openssh-client
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete py-docs/html/ delta@py.delta.chat:build/master
# Upload x86_64 wheels and source packages
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*manylinux201*
# Upload x86_64 wheels and source packages
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*manylinux201*
- name: python-aarch64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
- name: python-aarch64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/manylinux2014_aarch64
platform: linux
caches:
- path: cache
run:
path: build
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/manylinux2014_aarch64
platform: linux
caches:
- path: cache
run:
path: build
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Upload aarch64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*manylinux201*
# Upload aarch64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*manylinux201*
- name: python-musl-x86_64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
- name: python-musl-x86_64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/musllinux_1_1_x86_64
platform: linux
caches:
- path: cache
run:
path: build
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/musllinux_1_1_x86_64
platform: linux
caches:
- path: cache
run:
path: build
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Upload musl x86_64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*musllinux_1_1_x86_64*
# Upload musl x86_64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*musllinux_1_1_x86_64*
- name: python-musl-aarch64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
- name: python-musl-aarch64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/musllinux_1_1_aarch64
platform: linux
caches:
- path: cache
run:
path: build
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/musllinux_1_1_aarch64
platform: linux
caches:
- path: cache
run:
path: build
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Upload musl aarch64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*musllinux_1_1_aarch64*
# Upload musl aarch64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*musllinux_1_1_aarch64*
+8 -21
View File
@@ -276,7 +276,7 @@ impl ChatId {
grpname,
grpid,
create_blocked,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
create_protected,
param.unwrap_or_default(),
],
@@ -482,7 +482,7 @@ impl ChatId {
self,
&msg_text,
cmd,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
None,
None,
None,
@@ -1956,7 +1956,6 @@ impl ChatIdBlocked {
_ => (),
}
let created_timestamp = create_smeared_timestamp(context).await;
let chat_id = context
.sql
.transaction(move |transaction| {
@@ -1969,7 +1968,7 @@ impl ChatIdBlocked {
chat_name,
params.to_string(),
create_blocked as u8,
created_timestamp,
create_smeared_timestamp(context)
],
)?;
let chat_id = ChatId::new(
@@ -2117,7 +2116,7 @@ async fn prepare_msg_common(
context,
msg,
update_msg_id,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
)
.await?;
msg.chat_id = chat_id;
@@ -2842,7 +2841,7 @@ pub async fn create_group_chat(
Chattype::Group,
chat_name,
grpid,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
],
)
.await?;
@@ -2900,7 +2899,7 @@ pub async fn create_broadcast_list(context: &Context) -> Result<ChatId> {
Chattype::Broadcast,
chat_name,
grpid,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
],
)
.await?;
@@ -3361,7 +3360,7 @@ pub async fn forward_msgs(context: &Context, msg_ids: &[MsgId], chat_id: ChatId)
if let Some(reason) = chat.why_cant_send(context).await? {
bail!("cannot send to {}: {}", chat_id, reason);
}
curr_timestamp = create_smeared_timestamps(context, msg_ids.len()).await;
curr_timestamp = create_smeared_timestamps(context, msg_ids.len());
let ids = context
.sql
.query_map(
@@ -3563,7 +3562,7 @@ pub async fn add_device_msg_with_importance(
msg.try_calc_and_set_dimensions(context).await.ok();
prepare_msg_blob(context, msg).await?;
let timestamp_sent = create_smeared_timestamp(context).await;
let timestamp_sent = create_smeared_timestamp(context);
// makes sure, the added message is the last one,
// even if the date is wrong (useful esp. when warning about bad dates)
@@ -4091,7 +4090,6 @@ mod tests {
send_text_msg(&alice, alice_chat_id, "populate".to_string()).await?;
add_contact_to_chat(&alice, alice_chat_id, bob_id).await?;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
let add1 = alice.pop_sent_msg().await;
add_contact_to_chat(&alice, alice_chat_id, claire_id).await?;
@@ -4110,29 +4108,18 @@ mod tests {
remove_contact_from_chat(&alice, alice_chat_id, daisy_id).await?;
let remove2 = alice.pop_sent_msg().await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
assert_eq!(get_chat_contacts(&alice, alice_chat_id).await?.len(), 2);
// Bob receives the add and deletion messages out of order
let bob = TestContext::new_bob().await;
bob.recv_msg(&add1).await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
bob.recv_msg(&add3).await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
let bob_chat_id = bob.recv_msg(&add2).await.chat_id;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
assert_eq!(get_chat_contacts(&bob, bob_chat_id).await?.len(), 4);
bob.recv_msg(&remove2).await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
bob.recv_msg(&remove1).await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
assert_eq!(get_chat_contacts(&bob, bob_chat_id).await?.len(), 2);
Ok(())
+3
View File
@@ -300,6 +300,9 @@ pub enum Config {
/// See `crate::authres::update_authservid_candidates`.
AuthservIdCandidates,
/// Make all outgoing messages with Autocrypt header "multipart/signed".
SignUnencrypted,
/// Let the core save all events to the database.
/// This value is used internally to remember the MsgId of the logging xdc
#[strum(props(default = "0"))]
+14 -2
View File
@@ -4,6 +4,7 @@ use std::collections::{BTreeMap, HashMap};
use std::ffi::OsString;
use std::ops::Deref;
use std::path::{Path, PathBuf};
use std::sync::atomic::AtomicBool;
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime};
@@ -26,6 +27,7 @@ use crate::quota::QuotaInfo;
use crate::scheduler::Scheduler;
use crate::sql::Sql;
use crate::stock_str::StockStrings;
use crate::timesmearing::SmearedTimestamp;
use crate::tools::{duration_to_str, time};
/// Builder for the [`Context`].
@@ -188,7 +190,7 @@ pub struct InnerContext {
/// Blob directory path
pub(crate) blobdir: PathBuf,
pub(crate) sql: Sql,
pub(crate) last_smeared_timestamp: RwLock<i64>,
pub(crate) smeared_timestamp: SmearedTimestamp,
running_state: RwLock<RunningState>,
/// Mutex to avoid generating the key for the user more than once.
pub(crate) generating_key_mutex: Mutex<()>,
@@ -206,6 +208,9 @@ pub struct InnerContext {
/// Set to `None` if quota was never tried to load.
pub(crate) quota: RwLock<Option<QuotaInfo>>,
/// Set to true if quota update is requested.
pub(crate) quota_update_request: AtomicBool,
/// Server ID response if ID capability is supported
/// and the server returned non-NIL on the inbox connection.
/// <https://datatracker.ietf.org/doc/html/rfc2971>
@@ -356,7 +361,7 @@ impl Context {
blobdir,
running_state: RwLock::new(Default::default()),
sql: Sql::new(dbfile),
last_smeared_timestamp: RwLock::new(0),
smeared_timestamp: SmearedTimestamp::new(),
generating_key_mutex: Mutex::new(()),
oauth2_mutex: Mutex::new(()),
wrong_pw_warning_mutex: Mutex::new(()),
@@ -365,6 +370,7 @@ impl Context {
scheduler: RwLock::new(None),
ratelimit: RwLock::new(Ratelimit::new(Duration::new(60, 0), 6.0)), // Allow to send 6 messages immediately, no more than once every 10 seconds.
quota: RwLock::new(None),
quota_update_request: AtomicBool::new(false),
server_id: RwLock::new(None),
creation_time: std::time::SystemTime::now(),
last_full_folder_scan: Mutex::new(None),
@@ -757,6 +763,12 @@ impl Context {
.await?
.unwrap_or_default(),
);
res.insert(
"sign_unencrypted",
self.get_config_int(Config::SignUnencrypted)
.await?
.to_string(),
);
res.insert(
"debug_logging",
+1 -6
View File
@@ -13,7 +13,6 @@ use crate::imap::{Imap, ImapActionResult};
use crate::job::{self, Action, Job, Status};
use crate::message::{Message, MsgId, Viewtype};
use crate::mimeparser::{MimeMessage, Part};
use crate::param::Params;
use crate::tools::time;
use crate::{job_try, stock_str, EventType};
@@ -86,11 +85,7 @@ impl MsgId {
DownloadState::Available | DownloadState::Failure => {
self.update_download_state(context, DownloadState::InProgress)
.await?;
job::add(
context,
Job::new(Action::DownloadMsg, self.to_u32(), Params::new(), 0),
)
.await?;
job::add(context, Job::new(Action::DownloadMsg, self.to_u32())).await?;
}
}
Ok(())
+13
View File
@@ -124,6 +124,19 @@ impl EncryptHelper {
Ok(ctext)
}
/// Signs the passed-in `mail` using the private key from `context`.
/// Returns the payload and the signature.
pub async fn sign(
self,
context: &Context,
mail: lettre_email::PartBuilder,
) -> Result<(lettre_email::MimeMessage, String)> {
let sign_key = SignedSecretKey::load_self(context).await?;
let mime_message = mail.build();
let signature = pgp::pk_calc_signature(mime_message.as_string().as_bytes(), &sign_key)?;
Ok((mime_message, signature))
}
}
/// Ensures a private key exists for the configured user.
+1 -1
View File
@@ -650,7 +650,7 @@ mod tests {
use crate::download::DownloadState;
use crate::receive_imf::receive_imf;
use crate::test_utils::TestContext;
use crate::tools::MAX_SECONDS_TO_LEND_FROM_FUTURE;
use crate::timesmearing::MAX_SECONDS_TO_LEND_FROM_FUTURE;
use crate::{
chat::{self, create_group_chat, send_text_msg, Chat, ChatItem, ProtectionStatus},
tools::IsNoneOrEmpty,
+19 -68
View File
@@ -13,7 +13,6 @@ use rand::{thread_rng, Rng};
use crate::context::Context;
use crate::imap::{get_folder_meaning, FolderMeaning, Imap};
use crate::param::Params;
use crate::scheduler::InterruptInfo;
use crate::tools::time;
@@ -58,9 +57,6 @@ macro_rules! job_try {
)]
#[repr(u32)]
pub enum Action {
// this is user initiated so it should have a fairly high priority
UpdateRecentQuota = 140,
// This job will download partially downloaded messages completely
// and is added when download_full() is called.
// Most messages are downloaded automatically on fetch
@@ -80,7 +76,6 @@ pub struct Job {
pub desired_timestamp: i64,
pub added_timestamp: i64,
pub tries: u32,
pub param: Params,
}
impl fmt::Display for Job {
@@ -90,24 +85,19 @@ impl fmt::Display for Job {
}
impl Job {
pub fn new(action: Action, foreign_id: u32, param: Params, delay_seconds: i64) -> Self {
pub fn new(action: Action, foreign_id: u32) -> Self {
let timestamp = time();
Self {
job_id: 0,
action,
foreign_id,
desired_timestamp: timestamp + delay_seconds,
desired_timestamp: timestamp,
added_timestamp: timestamp,
tries: 0,
param,
}
}
pub fn delay_seconds(&self) -> i64 {
self.desired_timestamp - self.added_timestamp
}
/// Deletes the job from the database.
async fn delete(self, context: &Context) -> Result<()> {
if self.job_id != 0 {
@@ -130,23 +120,21 @@ impl Job {
context
.sql
.execute(
"UPDATE jobs SET desired_timestamp=?, tries=?, param=? WHERE id=?;",
"UPDATE jobs SET desired_timestamp=?, tries=? WHERE id=?;",
paramsv![
self.desired_timestamp,
i64::from(self.tries),
self.param.to_string(),
self.job_id as i32,
],
)
.await?;
} else {
context.sql.execute(
"INSERT INTO jobs (added_timestamp, action, foreign_id, param, desired_timestamp) VALUES (?,?,?,?,?);",
"INSERT INTO jobs (added_timestamp, action, foreign_id, desired_timestamp) VALUES (?,?,?,?);",
paramsv![
self.added_timestamp,
self.action,
self.foreign_id,
self.param.to_string(),
self.desired_timestamp
]
).await?;
@@ -202,17 +190,6 @@ pub async fn kill_action(context: &Context, action: Action) -> Result<()> {
Ok(())
}
pub async fn action_exists(context: &Context, action: Action) -> Result<bool> {
let exists = context
.sql
.exists(
"SELECT COUNT(*) FROM jobs WHERE action=?;",
paramsv![action],
)
.await?;
Ok(exists)
}
pub(crate) enum Connection<'a> {
Inbox(&'a mut Imap),
}
@@ -240,7 +217,7 @@ pub(crate) async fn perform_job(context: &Context, mut connection: Connection<'_
if tries < JOB_RETRIES {
info!(context, "increase job {} tries to {}", job, tries);
job.tries = tries;
let time_offset = get_backoff_time_offset(tries, job.action);
let time_offset = get_backoff_time_offset(tries);
job.desired_timestamp = time() + time_offset;
info!(
context,
@@ -289,10 +266,6 @@ async fn perform_job_action(
let try_res = match job.action {
Action::ResyncFolders => job.resync_folders(context, connection.inbox()).await,
Action::UpdateRecentQuota => match context.update_recent_quota(connection.inbox()).await {
Ok(status) => status,
Err(err) => Status::Finished(Err(err)),
},
Action::DownloadMsg => job.download_msg(context, connection.inbox()).await,
};
@@ -301,50 +274,30 @@ async fn perform_job_action(
try_res
}
fn get_backoff_time_offset(tries: u32, action: Action) -> i64 {
match action {
// Just try every 10s to update the quota
// If all retries are exhausted, a new job will be created when the quota information is needed
Action::UpdateRecentQuota => 10,
_ => {
// Exponential backoff
let n = 2_i32.pow(tries - 1) * 60;
let mut rng = thread_rng();
let r: i32 = rng.gen();
let mut seconds = r % (n + 1);
if seconds < 1 {
seconds = 1;
}
i64::from(seconds)
}
fn get_backoff_time_offset(tries: u32) -> i64 {
// Exponential backoff
let n = 2_i32.pow(tries - 1) * 60;
let mut rng = thread_rng();
let r: i32 = rng.gen();
let mut seconds = r % (n + 1);
if seconds < 1 {
seconds = 1;
}
i64::from(seconds)
}
pub(crate) async fn schedule_resync(context: &Context) -> Result<()> {
kill_action(context, Action::ResyncFolders).await?;
add(
context,
Job::new(Action::ResyncFolders, 0, Params::new(), 0),
)
.await?;
add(context, Job::new(Action::ResyncFolders, 0)).await?;
Ok(())
}
/// Adds a job to the database, scheduling it.
pub async fn add(context: &Context, job: Job) -> Result<()> {
let action = job.action;
let delay_seconds = job.delay_seconds();
job.save(context).await.context("failed to save job")?;
if delay_seconds == 0 {
match action {
Action::ResyncFolders | Action::UpdateRecentQuota | Action::DownloadMsg => {
info!(context, "interrupt: imap");
context.interrupt_inbox(InterruptInfo::new(false)).await;
}
}
}
info!(context, "interrupt: imap");
context.interrupt_inbox(InterruptInfo::new(false)).await;
Ok(())
}
@@ -396,7 +349,6 @@ LIMIT 1;
desired_timestamp: row.get("desired_timestamp")?,
added_timestamp: row.get("added_timestamp")?,
tries: row.get("tries")?,
param: row.get::<_, String>("param")?.parse().unwrap_or_default(),
};
Ok(job)
@@ -436,8 +388,8 @@ mod tests {
.sql
.execute(
"INSERT INTO jobs
(added_timestamp, action, foreign_id, param, desired_timestamp)
VALUES (?, ?, ?, ?, ?);",
(added_timestamp, action, foreign_id, desired_timestamp)
VALUES (?, ?, ?, ?);",
paramsv![
now,
if valid {
@@ -446,7 +398,6 @@ mod tests {
-1
},
foreign_id,
Params::new().to_string(),
now
],
)
+1
View File
@@ -92,6 +92,7 @@ mod smtp;
mod socks;
pub mod stock_str;
mod sync;
mod timesmearing;
mod token;
mod update_helper;
pub mod webxdc;
+2 -7
View File
@@ -1771,13 +1771,8 @@ async fn ndn_maybe_add_info_msg(
// Tell the user which of the recipients failed if we know that (because in
// a group, this might otherwise be unclear)
let text = stock_str::failed_sending_to(context, contact.get_display_name()).await;
chat::add_info_msg(
context,
chat_id,
&text,
create_smeared_timestamp(context).await,
)
.await?;
chat::add_info_msg(context, chat_id, &text, create_smeared_timestamp(context))
.await?;
context.emit_event(EventType::ChatModified(chat_id));
}
}
+119 -3
View File
@@ -250,7 +250,7 @@ impl<'a> MimeFactory<'a> {
.get_config(Config::Selfstatus)
.await?
.unwrap_or_default();
let timestamp = create_smeared_timestamp(context).await;
let timestamp = create_smeared_timestamp(context);
let res = MimeFactory::<'a> {
from_addr,
@@ -779,10 +779,36 @@ impl<'a> MimeFactory<'a> {
};
// Store protected headers in the outer message.
headers
let message = headers
.protected
.into_iter()
.fold(message, |message, header| message.header(header))
.fold(message, |message, header| message.header(header));
if self.should_skip_autocrypt()
|| !context.get_config_bool(Config::SignUnencrypted).await?
{
message
} else {
let (payload, signature) = encrypt_helper.sign(context, message).await?;
PartBuilder::new()
.header((
"Content-Type".to_string(),
"multipart/signed; protocol=\"application/pgp-signature\"".to_string(),
))
.child(payload)
.child(
PartBuilder::new()
.content_type(
&"application/pgp-signature; name=\"signature.asc\""
.parse::<mime::Mime>()
.unwrap(),
)
.header(("Content-Description", "OpenPGP digital signature"))
.header(("Content-Disposition", "attachment; filename=\"signature\";"))
.body(signature)
.build(),
)
}
};
// Store the unprotected headers on the outer message.
@@ -2140,6 +2166,96 @@ mod tests {
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_selfavatar_unencrypted_signed() {
// create chat with bob, set selfavatar
let t = TestContext::new_alice().await;
t.set_config(Config::SignUnencrypted, Some("1"))
.await
.unwrap();
let chat = t.create_chat_with_contact("bob", "bob@example.org").await;
let file = t.dir.path().join("avatar.png");
let bytes = include_bytes!("../test-data/image/avatar64x64.png");
tokio::fs::write(&file, bytes).await.unwrap();
t.set_config(Config::Selfavatar, Some(file.to_str().unwrap()))
.await
.unwrap();
// send message to bob: that should get multipart/mixed because of the avatar moved to inner header;
// make sure, `Subject:` stays in the outer header (imf header)
let mut msg = Message::new(Viewtype::Text);
msg.set_text(Some("this is the text!".to_string()));
let sent_msg = t.send_msg(chat.id, &mut msg).await;
let mut payload = sent_msg.payload().splitn(4, "\r\n\r\n");
let part = payload.next().unwrap();
assert_eq!(part.match_indices("multipart/signed").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 0);
assert_eq!(part.match_indices("Autocrypt:").count(), 1);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 0);
let part = payload.next().unwrap();
assert_eq!(part.match_indices("multipart/mixed").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 1);
assert_eq!(part.match_indices("Autocrypt:").count(), 0);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 0);
let part = payload.next().unwrap();
assert_eq!(part.match_indices("text/plain").count(), 1);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 0);
let body = payload.next().unwrap();
assert_eq!(body.match_indices("this is the text!").count(), 1);
let bob = TestContext::new_bob().await;
bob.recv_msg(&sent_msg).await;
let alice_id = Contact::lookup_id_by_addr(&bob.ctx, "alice@example.org", Origin::Unknown)
.await
.unwrap()
.unwrap();
let alice_contact = Contact::load_from_db(&bob.ctx, alice_id).await.unwrap();
assert!(alice_contact
.get_profile_image(&bob.ctx)
.await
.unwrap()
.is_some());
// if another message is sent, that one must not contain the avatar
// and no artificial multipart/mixed nesting
let sent_msg = t.send_msg(chat.id, &mut msg).await;
let mut payload = sent_msg.payload().splitn(3, "\r\n\r\n");
let part = payload.next().unwrap();
assert_eq!(part.match_indices("multipart/signed").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 0);
assert_eq!(part.match_indices("Autocrypt:").count(), 1);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 0);
let part = payload.next().unwrap();
assert_eq!(part.match_indices("text/plain").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 1);
assert_eq!(part.match_indices("Autocrypt:").count(), 0);
assert_eq!(part.match_indices("multipart/mixed").count(), 0);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 0);
let body = payload.next().unwrap();
assert_eq!(body.match_indices("this is the text!").count(), 1);
assert_eq!(body.match_indices("text/plain").count(), 0);
assert_eq!(body.match_indices("Chat-User-Avatar:").count(), 0);
assert_eq!(body.match_indices("Subject:").count(), 0);
bob.recv_msg(&sent_msg).await;
let alice_contact = Contact::load_from_db(&bob.ctx, alice_id).await.unwrap();
assert!(alice_contact
.get_profile_image(&bob.ctx)
.await
.unwrap()
.is_some());
}
/// Test that removed member address does not go into the `To:` field.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_remove_member_bcc() -> Result<()> {
+45 -20
View File
@@ -224,8 +224,32 @@ impl MimeMessage {
// Parse hidden headers.
let mimetype = mail.ctype.mimetype.parse::<Mime>()?;
let (part, mimetype) =
if mimetype.type_() == mime::MULTIPART && mimetype.subtype().as_str() == "signed" {
if let Some(part) = mail.subparts.first() {
// We don't remove "subject" from `headers` because currently just signed
// messages are shown as unencrypted anyway.
MimeMessage::merge_headers(
context,
&mut headers,
&mut recipients,
&mut from,
&mut list_post,
&mut chat_disposition_notification_to,
&part.headers,
);
(part, part.ctype.mimetype.parse::<Mime>()?)
} else {
// If it's a partially fetched message, there are no subparts.
(&mail, mimetype)
}
} else {
// Currently we do not sign unencrypted messages by default.
(&mail, mimetype)
};
if mimetype.type_() == mime::MULTIPART && mimetype.subtype().as_str() == "mixed" {
if let Some(part) = mail.subparts.first() {
if let Some(part) = part.subparts.first() {
for field in &part.headers {
let key = field.get_key().to_lowercase();
@@ -256,26 +280,27 @@ impl MimeMessage {
hop_info += &decryption_info.dkim_results.to_string();
let public_keyring = keyring_from_peerstate(decryption_info.peerstate.as_ref());
let (mail, mut signatures, encrypted) =
match try_decrypt(context, &mail, &private_keyring, &public_keyring) {
Ok(Some((raw, signatures))) => {
mail_raw = raw;
let decrypted_mail = mailparse::parse_mail(&mail_raw)?;
if std::env::var(crate::DCC_MIME_DEBUG).is_ok() {
info!(
context,
"decrypted message mime-body:\n{}",
String::from_utf8_lossy(&mail_raw),
);
}
(Ok(decrypted_mail), signatures, true)
let (mail, mut signatures, encrypted) = match tokio::task::block_in_place(|| {
try_decrypt(context, &mail, &private_keyring, &public_keyring)
}) {
Ok(Some((raw, signatures))) => {
mail_raw = raw;
let decrypted_mail = mailparse::parse_mail(&mail_raw)?;
if std::env::var(crate::DCC_MIME_DEBUG).is_ok() {
info!(
context,
"decrypted message mime-body:\n{}",
String::from_utf8_lossy(&mail_raw),
);
}
Ok(None) => (Ok(mail), HashSet::new(), false),
Err(err) => {
warn!(context, "decryption failed: {:#}", err);
(Err(err), HashSet::new(), false)
}
};
(Ok(decrypted_mail), signatures, true)
}
Ok(None) => (Ok(mail), HashSet::new(), false),
Err(err) => {
warn!(context, "decryption failed: {:#}", err);
(Err(err), HashSet::new(), false)
}
};
let mail = mail.as_ref().map(|mail| {
let (content, signatures_detached) = validate_detached_signature(mail, &public_keyring)
.unwrap_or((mail, Default::default()));
+4 -2
View File
@@ -1,7 +1,7 @@
//! TLS support.
use anyhow::Result;
use async_native_tls::{Certificate, TlsConnector, TlsStream};
use async_native_tls::{Certificate, Protocol, TlsConnector, TlsStream};
use once_cell::sync::Lazy;
use tokio::io::{AsyncRead, AsyncWrite};
@@ -15,7 +15,9 @@ static LETSENCRYPT_ROOT: Lazy<Certificate> = Lazy::new(|| {
});
pub fn build_tls(strict_tls: bool) -> TlsConnector {
let tls_builder = TlsConnector::new().add_root_certificate(LETSENCRYPT_ROOT.clone());
let tls_builder = TlsConnector::new()
.min_protocol_version(Some(Protocol::Tlsv12))
.add_root_certificate(LETSENCRYPT_ROOT.clone());
if strict_tls {
tls_builder
+14
View File
@@ -262,6 +262,20 @@ pub async fn pk_encrypt(
.await?
}
/// Signs `plain` text using `private_key_for_signing`.
pub fn pk_calc_signature(
plain: &[u8],
private_key_for_signing: &SignedSecretKey,
) -> Result<String> {
let msg = Message::new_literal_bytes("", plain).sign(
private_key_for_signing,
|| "".into(),
Default::default(),
)?;
let signature = msg.into_signature().to_armored_string(None)?;
Ok(signature)
}
/// Decrypts the message with keys from the private key keyring.
///
/// Receiver private keys are provided in
+13 -12
View File
@@ -1,6 +1,7 @@
//! # Support for IMAP QUOTA extension.
use std::collections::BTreeMap;
use std::sync::atomic::Ordering;
use anyhow::{anyhow, Context as _, Result};
use async_imap::types::{Quota, QuotaResource};
@@ -11,11 +12,10 @@ use crate::context::Context;
use crate::imap::scan_folders::get_watched_folders;
use crate::imap::session::Session as ImapSession;
use crate::imap::Imap;
use crate::job::{Action, Status};
use crate::message::{Message, Viewtype};
use crate::param::Params;
use crate::scheduler::InterruptInfo;
use crate::tools::time;
use crate::{job, stock_str, EventType};
use crate::{stock_str, EventType};
/// warn about a nearly full mailbox after this usage percentage is reached.
/// quota icon is "yellow".
@@ -112,12 +112,10 @@ pub fn needs_quota_warning(curr_percentage: u64, warned_at_percentage: u64) -> b
impl Context {
// Adds a job to update `quota.recent`
pub(crate) async fn schedule_quota_update(&self) -> Result<()> {
if !job::action_exists(self, Action::UpdateRecentQuota).await? {
job::add(
self,
job::Job::new(Action::UpdateRecentQuota, 0, Params::new(), 0),
)
.await?;
let requested = self.quota_update_request.swap(true, Ordering::Relaxed);
if !requested {
// Quota update was not requested before.
self.interrupt_inbox(InterruptInfo::new(false)).await;
}
Ok(())
}
@@ -132,10 +130,10 @@ impl Context {
/// and new space is allocated as needed.
///
/// Called in response to `Action::UpdateRecentQuota`.
pub(crate) async fn update_recent_quota(&self, imap: &mut Imap) -> Result<Status> {
pub(crate) async fn update_recent_quota(&self, imap: &mut Imap) -> Result<()> {
if let Err(err) = imap.prepare(self).await {
warn!(self, "could not connect: {:#}", err);
return Ok(Status::RetryNow);
return Ok(());
}
let session = imap.session.as_mut().context("no session")?;
@@ -166,13 +164,16 @@ impl Context {
}
}
// Clear the request to update quota.
self.quota_update_request.store(false, Ordering::Relaxed);
*self.quota.write().await = Some(QuotaInfo {
recent: quota,
modified: time(),
});
self.emit_event(EventType::ConnectivityChanged);
Ok(Status::Finished(Ok(())))
Ok(())
}
}
+2 -2
View File
@@ -203,7 +203,7 @@ pub(crate) async fn receive_imf_inner(
)
.await?;
let rcvd_timestamp = smeared_time(context).await;
let rcvd_timestamp = smeared_time(context);
// Sender timestamp is allowed to be a bit in the future due to
// unsynchronized clocks, but not too much.
@@ -1380,7 +1380,7 @@ async fn calc_sort_timestamp(
}
}
Ok(min(sort_timestamp, smeared_time(context).await))
Ok(min(sort_timestamp, smeared_time(context)))
}
async fn lookup_chat_by_reply(
+8
View File
@@ -1,4 +1,5 @@
use std::iter::{self, once};
use std::sync::atomic::Ordering;
use anyhow::{bail, Context as _, Result};
use async_channel::{self as channel, Receiver, Sender};
@@ -128,6 +129,13 @@ async fn inbox_loop(ctx: Context, started: Sender<()>, inbox_handlers: ImapConne
info = Default::default();
}
None => {
let requested = ctx.quota_update_request.swap(false, Ordering::Relaxed);
if requested {
if let Err(err) = ctx.update_recent_quota(&mut connection).await {
warn!(ctx, "Failed to update quota: {:#}.", err);
}
}
maybe_add_time_based_warnings(&ctx).await;
match ctx.get_config_i64(Config::LastHousekeeping).await {
+1 -1
View File
@@ -983,7 +983,7 @@ mod tests {
assert_eq!(avatar_bytes, &tokio::fs::read(&a).await.unwrap()[..]);
t.sql.close().await;
housekeeping(&t).await.unwrap_err(); // housekeeping should fail as the db is closed
housekeeping(&t).await.unwrap(); // housekeeping should emit warnings but not fail
t.sql.open(&t, "".to_string()).await.unwrap();
let a = t.get_config(Config::Selfavatar).await.unwrap().unwrap();
+193
View File
@@ -0,0 +1,193 @@
//! # Time smearing.
//!
//! As e-mails typically only use a second-based-resolution for timestamps,
//! the order of two mails sent withing one second is unclear.
//! This is bad e.g. when forwarding some messages from a chat -
//! these messages will appear at the recipient easily out of order.
//!
//! We work around this issue by not sending out two mails with the same timestamp.
//! For this purpose, in short, we track the last timestamp used in `last_smeared_timestamp`
//! when another timestamp is needed in the same second, we use `last_smeared_timestamp+1`
//! after some moments without messages sent out,
//! `last_smeared_timestamp` is again in sync with the normal time.
//!
//! However, we do not do all this for the far future,
//! but at max `MAX_SECONDS_TO_LEND_FROM_FUTURE`
use std::cmp::{max, min};
use std::sync::atomic::{AtomicI64, Ordering};
pub(crate) const MAX_SECONDS_TO_LEND_FROM_FUTURE: i64 = 5;
/// Smeared timestamp generator.
#[derive(Debug)]
pub struct SmearedTimestamp {
/// Next timestamp available for allocation.
smeared_timestamp: AtomicI64,
}
impl SmearedTimestamp {
/// Creates a new smeared timestamp generator.
pub fn new() -> Self {
Self {
smeared_timestamp: AtomicI64::new(0),
}
}
/// Allocates `count` unique timestamps.
///
/// Returns the first allocated timestamp.
pub fn create_n(&self, now: i64, count: i64) -> i64 {
let mut prev = self.smeared_timestamp.load(Ordering::Relaxed);
loop {
// Advance the timestamp if it is in the past,
// but keep `count - 1` timestamps from the past if possible.
let t = max(prev, now - count + 1);
// Rewind the time back if there is no room
// to allocate `count` timestamps without going too far into the future.
// Not going too far into the future
// is more important than generating unique timestamps.
let first = min(t, now + MAX_SECONDS_TO_LEND_FROM_FUTURE - count + 1);
// Allocate `count` timestamps by advancing the current timestamp.
let next = first + count;
if let Err(x) = self.smeared_timestamp.compare_exchange_weak(
prev,
next,
Ordering::Relaxed,
Ordering::Relaxed,
) {
prev = x;
} else {
return first;
}
}
}
/// Creates a single timestamp.
pub fn create(&self, now: i64) -> i64 {
self.create_n(now, 1)
}
/// Returns the current smeared timestamp.
pub fn current(&self) -> i64 {
self.smeared_timestamp.load(Ordering::Relaxed)
}
}
#[cfg(test)]
mod tests {
use std::time::SystemTime;
use super::*;
use crate::test_utils::TestContext;
use crate::tools::{create_smeared_timestamp, create_smeared_timestamps, smeared_time, time};
#[test]
fn test_smeared_timestamp() {
let smeared_timestamp = SmearedTimestamp::new();
let now = time();
assert_eq!(smeared_timestamp.current(), 0);
for i in 0..MAX_SECONDS_TO_LEND_FROM_FUTURE {
assert_eq!(smeared_timestamp.create(now), now + i);
}
assert_eq!(
smeared_timestamp.create(now),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE
);
assert_eq!(
smeared_timestamp.create(now),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE
);
// System time rewinds back by 1000 seconds.
let now = now - 1000;
assert_eq!(
smeared_timestamp.create(now),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE
);
assert_eq!(
smeared_timestamp.create(now),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE
);
assert_eq!(
smeared_timestamp.create(now + 1),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE + 1
);
assert_eq!(smeared_timestamp.create(now + 100), now + 100);
assert_eq!(smeared_timestamp.create(now + 100), now + 101);
assert_eq!(smeared_timestamp.create(now + 100), now + 102);
}
#[test]
fn test_create_n_smeared_timestamps() {
let smeared_timestamp = SmearedTimestamp::new();
let now = time();
// Create a single timestamp to initialize the generator.
assert_eq!(smeared_timestamp.create(now), now);
// Wait a minute.
let now = now + 60;
// Simulate forwarding 7 messages.
let forwarded_messages = 7;
// We have not sent anything for a minute,
// so we can take the current timestamp and take 6 timestamps from the past.
assert_eq!(smeared_timestamp.create_n(now, forwarded_messages), now - 6);
assert_eq!(smeared_timestamp.current(), now + 1);
// Wait 4 seconds.
// Now we have 3 free timestamps in the past.
let now = now + 4;
assert_eq!(smeared_timestamp.current(), now - 3);
// Forward another 7 messages.
// We can only lend 3 timestamps from the past.
assert_eq!(smeared_timestamp.create_n(now, forwarded_messages), now - 3);
// We had to borrow 3 timestamps from the future
// because there were not enough timestamps in the past.
assert_eq!(smeared_timestamp.current(), now + 4);
// Forward another 7 messages.
// We cannot use more than 5 timestamps from the future,
// so we use 5 timestamps from the future,
// the current timestamp and one timestamp from the past.
assert_eq!(smeared_timestamp.create_n(now, forwarded_messages), now - 1);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_smeared_timestamp() {
let t = TestContext::new().await;
assert_ne!(create_smeared_timestamp(&t), create_smeared_timestamp(&t));
assert!(
create_smeared_timestamp(&t)
>= SystemTime::now()
.duration_since(SystemTime::UNIX_EPOCH)
.unwrap()
.as_secs() as i64
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_smeared_timestamps() {
let t = TestContext::new().await;
let count = MAX_SECONDS_TO_LEND_FROM_FUTURE - 1;
let start = create_smeared_timestamps(&t, count as usize);
let next = smeared_time(&t);
assert!((start + count - 1) < next);
let count = MAX_SECONDS_TO_LEND_FROM_FUTURE + 30;
let start = create_smeared_timestamps(&t, count as usize);
let next = smeared_time(&t);
assert!((start + count - 1) < next);
}
}
+9 -76
View File
@@ -3,7 +3,6 @@
#![allow(missing_docs)]
use core::cmp::{max, min};
use std::borrow::Cow;
use std::fmt;
use std::io::Cursor;
@@ -140,63 +139,27 @@ pub(crate) fn gm2local_offset() -> i64 {
i64::from(lt.offset().local_minus_utc())
}
// timesmearing
// - as e-mails typically only use a second-based-resolution for timestamps,
// the order of two mails sent withing one second is unclear.
// this is bad eg. when forwarding some messages from a chat -
// these messages will appear at the recipient easily out of order.
// - we work around this issue by not sending out two mails with the same timestamp.
// - for this purpose, in short, we track the last timestamp used in `last_smeared_timestamp`
// when another timestamp is needed in the same second, we use `last_smeared_timestamp+1`
// - after some moments without messages sent out,
// `last_smeared_timestamp` is again in sync with the normal time.
// - however, we do not do all this for the far future,
// but at max `MAX_SECONDS_TO_LEND_FROM_FUTURE`
pub(crate) const MAX_SECONDS_TO_LEND_FROM_FUTURE: i64 = 5;
/// Returns the current smeared timestamp,
///
/// The returned timestamp MUST NOT be sent out.
pub(crate) async fn smeared_time(context: &Context) -> i64 {
let mut now = time();
let ts = *context.last_smeared_timestamp.read().await;
if ts >= now {
now = ts + 1;
}
now
pub(crate) fn smeared_time(context: &Context) -> i64 {
let now = time();
let ts = context.smeared_timestamp.current();
std::cmp::max(ts, now)
}
/// Returns a timestamp that is guaranteed to be unique.
pub(crate) async fn create_smeared_timestamp(context: &Context) -> i64 {
pub(crate) fn create_smeared_timestamp(context: &Context) -> i64 {
let now = time();
let mut ret = now;
let mut last_smeared_timestamp = context.last_smeared_timestamp.write().await;
if ret <= *last_smeared_timestamp {
ret = *last_smeared_timestamp + 1;
if ret - now > MAX_SECONDS_TO_LEND_FROM_FUTURE {
ret = now + MAX_SECONDS_TO_LEND_FROM_FUTURE
}
}
*last_smeared_timestamp = ret;
ret
context.smeared_timestamp.create(now)
}
// creates `count` timestamps that are guaranteed to be unique.
// the frist created timestamps is returned directly,
// the first created timestamps is returned directly,
// get the other timestamps just by adding 1..count-1
pub(crate) async fn create_smeared_timestamps(context: &Context, count: usize) -> i64 {
pub(crate) fn create_smeared_timestamps(context: &Context, count: usize) -> i64 {
let now = time();
let count = count as i64;
let mut start = now + min(count, MAX_SECONDS_TO_LEND_FROM_FUTURE) - count;
let mut last_smeared_timestamp = context.last_smeared_timestamp.write().await;
start = max(*last_smeared_timestamp + 1, start);
*last_smeared_timestamp = start + count - 1;
start
context.smeared_timestamp.create_n(now, count as i64)
}
// if the system time is not plausible, once a day, add a device message.
@@ -1071,36 +1034,6 @@ DKIM Results: Passed=true, Works=true, Allow_Keychange=true";
assert!(!file_exist!(context, &fn0));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_smeared_timestamp() {
let t = TestContext::new().await;
assert_ne!(
create_smeared_timestamp(&t).await,
create_smeared_timestamp(&t).await
);
assert!(
create_smeared_timestamp(&t).await
>= SystemTime::now()
.duration_since(SystemTime::UNIX_EPOCH)
.unwrap()
.as_secs() as i64
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_smeared_timestamps() {
let t = TestContext::new().await;
let count = MAX_SECONDS_TO_LEND_FROM_FUTURE - 1;
let start = create_smeared_timestamps(&t, count as usize).await;
let next = smeared_time(&t).await;
assert!((start + count - 1) < next);
let count = MAX_SECONDS_TO_LEND_FROM_FUTURE + 30;
let start = create_smeared_timestamps(&t, count as usize).await;
let next = smeared_time(&t).await;
assert!((start + count - 1) < next);
}
#[test]
fn test_duration_to_str() {
assert_eq!(duration_to_str(Duration::from_secs(0)), "0h 0m 0s");
+1 -1
View File
@@ -408,7 +408,7 @@ impl Context {
.create_status_update_record(
&mut instance,
update_str,
create_smeared_timestamp(self).await,
create_smeared_timestamp(self),
send_now,
ContactId::SELF,
)
+51 -19
View File
@@ -3,26 +3,58 @@
Some of the standards Delta Chat is based on:
Tasks | Standards
---------------------------------|---------------------------------------------
Transport | IMAP v4 ([RFC 3501](https://tools.ietf.org/html/rfc3501)), SMTP ([RFC 5321](https://tools.ietf.org/html/rfc5321)) and Internet Message Format (IMF, [RFC 5322](https://tools.ietf.org/html/rfc5322))
Proxy | SOCKS5 ([RFC 1928](https://tools.ietf.org/html/rfc1928))
Embedded media | MIME Document Series ([RFC 2045](https://tools.ietf.org/html/rfc2045), [RFC 2046](https://tools.ietf.org/html/rfc2046)), Content-Disposition Header ([RFC 2183](https://tools.ietf.org/html/rfc2183)), Multipart/Related ([RFC 2387](https://tools.ietf.org/html/rfc2387))
Text and Quote encoding | Fixed, Flowed ([RFC 3676](https://tools.ietf.org/html/rfc3676))
Reactions | Reaction: Indicating Summary Reaction to a Message [RFC 9078](https://datatracker.ietf.org/doc/rfc9078/)
Filename encoding | Encoded Words ([RFC 2047](https://tools.ietf.org/html/rfc2047)), Encoded Word Extensions ([RFC 2231](https://tools.ietf.org/html/rfc2231))
Identify server folders | IMAP LIST Extension ([RFC 6154](https://tools.ietf.org/html/rfc6154))
Push | IMAP IDLE ([RFC 2177](https://tools.ietf.org/html/rfc2177))
Quota | IMAP QUOTA extension ([RFC 2087](https://tools.ietf.org/html/rfc2087))
Seen status synchronization | IMAP CONDSTORE extension ([RFC 7162](https://tools.ietf.org/html/rfc7162))
Client/server identification | IMAP ID extension ([RFC 2971](https://datatracker.ietf.org/doc/html/rfc2971))
Authorization | OAuth2 ([RFC 6749](https://tools.ietf.org/html/rfc6749))
End-to-end encryption | [Autocrypt Level 1](https://autocrypt.org/level1.html), OpenPGP ([RFC 4880](https://tools.ietf.org/html/rfc4880)), Security Multiparts for MIME ([RFC 1847](https://tools.ietf.org/html/rfc1847)) and [“Mixed Up” Encryption repairing](https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html)
-------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Transport | IMAP v4 ([RFC 3501][]), SMTP ([RFC 5321][]) and Internet Message Format (IMF, [RFC 5322][])
Proxy | SOCKS5 ([RFC 1928][])
Embedded media | MIME Document Series ([RFC 2045][], [RFC 2046][]), Content-Disposition Header ([RFC 2183][]), Multipart/Related ([RFC 2387][])
Text and Quote encoding | Fixed, Flowed ([RFC 3676][])
Reactions | Reaction: Indicating Summary Reaction to a Message ([RFC 9078][])
Filename encoding | Encoded Words ([RFC 2047][]), Encoded Word Extensions ([RFC 2231][])
Identify server folders | IMAP LIST Extension ([RFC 6154][])
Push | IMAP IDLE ([RFC 2177][])
Quota | IMAP QUOTA extension ([RFC 2087][])
Seen status synchronization | IMAP CONDSTORE extension ([RFC 7162][])
Client/server identification | IMAP ID extension ([RFC 2971][])
Authorization | OAuth2 ([RFC 6749][])
End-to-end encryption | [Autocrypt Level 1][], OpenPGP ([RFC 4880][]), Security Multiparts for MIME ([RFC 1847][]) and [“Mixed Up” Encryption repairing](https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html)
Detect/prevent active attacks | [countermitm][] protocols
Compare public keys | [openpgp4fpr][] URI Scheme
Header encryption | [Protected Headers for Cryptographic E-mail](https://datatracker.ietf.org/doc/draft-autocrypt-lamps-protected-headers/)
Configuration assistance | [Autoconfigure](https://web.archive.org/web/20210402044801/https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration) and [Autodiscover](https://technet.microsoft.com/library/bb124251(v=exchg.150).aspx)
Configuration assistance | [Autoconfigure](https://web.archive.org/web/20210402044801/https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration) and [Autodiscover][]
Messenger functions | [Chat-over-Email](https://github.com/deltachat/deltachat-core-rust/blob/master/spec.md#chat-mail-specification)
Detect mailing list | List-Id ([RFC 2919](https://tools.ietf.org/html/rfc2919)) and Precedence ([RFC 3834](https://tools.ietf.org/html/rfc3834))
User and chat colors | [XEP-0392](https://xmpp.org/extensions/xep-0392.html): Consistent Color Generation
Send and receive system messages | Multipart/Report Media Type ([RFC 6522](https://tools.ietf.org/html/rfc6522))
Return receipts | Message Disposition Notification (MDN, [RFC 8098](https://tools.ietf.org/html/rfc8098), [RFC 3503](https://tools.ietf.org/html/rfc3503)) using the Chat-Disposition-Notification-To header
Detect mailing list | List-Id ([RFC 2919][]) and Precedence ([RFC 3834][])
User and chat colors | [XEP-0392][]: Consistent Color Generation
Send and receive system messages | Multipart/Report Media Type ([RFC 6522][])
Return receipts | Message Disposition Notification (MDN, [RFC 8098][], [RFC 3503][]) using the Chat-Disposition-Notification-To header
Locations | KML ([Open Geospatial Consortium](http://www.opengeospatial.org/standards/kml/), [Google Dev](https://developers.google.com/kml/))
[Autocrypt Level 1]: https://autocrypt.org/level1.html
[countermitm]: https://countermitm.readthedocs.io/en/latest/
[openpgp4fpr]: https://metacode.biz/openpgp/openpgp4fpr
[Autodiscover]: https://learn.microsoft.com/en-us/exchange/autodiscover-service-for-exchange-2013
[XEP-0392]: https://xmpp.org/extensions/xep-0392.html
[RFC 1847]: https://tools.ietf.org/html/rfc1847
[RFC 1928]: https://tools.ietf.org/html/rfc1928
[RFC 2045]: https://tools.ietf.org/html/rfc2045
[RFC 2046]: https://tools.ietf.org/html/rfc2046
[RFC 2047]: https://tools.ietf.org/html/rfc2047
[RFC 2087]: https://tools.ietf.org/html/rfc2087
[RFC 2177]: https://tools.ietf.org/html/rfc2177
[RFC 2183]: https://tools.ietf.org/html/rfc2183
[RFC 2231]: https://tools.ietf.org/html/rfc2231
[RFC 2387]: https://tools.ietf.org/html/rfc2387
[RFC 2919]: https://tools.ietf.org/html/rfc2919
[RFC 2971]: https://tools.ietf.org/html/rfc2971
[RFC 3501]: https://tools.ietf.org/html/rfc3501
[RFC 3503]: https://tools.ietf.org/html/rfc3503
[RFC 3676]: https://tools.ietf.org/html/rfc3676
[RFC 3834]: https://tools.ietf.org/html/rfc3834
[RFC 4880]: https://tools.ietf.org/html/rfc4880
[RFC 5321]: https://tools.ietf.org/html/rfc5321
[RFC 5322]: https://tools.ietf.org/html/rfc5322
[RFC 6154]: https://tools.ietf.org/html/rfc6154
[RFC 6522]: https://tools.ietf.org/html/rfc6522
[RFC 6749]: https://tools.ietf.org/html/rfc6749
[RFC 7162]: https://tools.ietf.org/html/rfc7162
[RFC 8098]: https://tools.ietf.org/html/rfc8098
[RFC 9078]: https://tools.ietf.org/html/rfc9078