Mirrors/chatmail-core
1
0
Fork 0
mirror of https://github.com/chatmail/core.git synced 2026-05-04 13:56:30 +03:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into flub/send-backup

Browse Source
This commit is contained in:
Floris Bruynooghe
2023-03-02 11:35:17 +01:00
parent 9d88ef069e 4e166b1b4a
commit e3014a349c
72 changed files with 2256 additions and 1383 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
.github/workflows/ci.yml vendored
View File

@@ -1,5 +1,11 @@
name: Rust CI
# Cancel previously started workflow runs
# when the branch is updated.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
on:
pull_request:
push:
@@ -26,6 +32,23 @@ jobs:
- name: Run clippy
run: scripts/clippy.sh
cargo_deny:
name: cargo deny
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: EmbarkStudios/cargo-deny-action@v1
with:
arguments: --all-features --workspace
provider_database:
name: Check provider database
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Check provider database
run: scripts/update-provider-database.sh
docs:
name: Rust doc comments
runs-on: ubuntu-latest

98
.github/workflows/deltachat-rpc-server.yml vendored
View File

@@ -2,21 +2,105 @@
name: Build deltachat-rpc-server binaries
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
on:
workflow_dispatch:
jobs:
build_server:
name: Build deltachat-rpc-server
# Build a version statically linked against musl libc
# to avoid problems with glibc version incompatibility.
build_static_linux:
name: Build deltachat-rpc-server for Linux
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- name: Setup rust target
run: rustup target add x86_64-unknown-linux-musl
- name: Install musl-gcc
run: sudo apt install musl-tools
- name: Build
env:
RUSTFLAGS: "-C link-arg=-s"
run: cargo build --release --target x86_64-unknown-linux-musl -p deltachat-rpc-server --features vendored
- name: Upload binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-x86_64
path: target/x86_64-unknown-linux-musl/release/deltachat-rpc-server
if-no-files-found: error
build_aarch64_linux:
name: Cross-compile deltachat-rpc-server for aarch64 Linux
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- name: Build
run: sh scripts/aarch64-unknown-linux-musl.sh
- name: Upload binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-aarch64
path: target/aarch64-unknown-linux-musl/release/deltachat-rpc-server
if-no-files-found: error
build_android:
name: Cross-compile deltachat-rpc-server for Android (armeabi-v7a, arm64-v8a, x86 and x86_64)
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: nttld/setup-ndk@v1
id: setup-ndk
with:
ndk-version: r21d
- name: Build
env:
ANDROID_NDK_ROOT: ${{ steps.setup-ndk.outputs.ndk-path }}
run: sh scripts/android-rpc-server.sh
- name: Upload binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-android-armv7
path: target/armv7-linux-androideabi/release/deltachat-rpc-server
if-no-files-found: error
- name: Upload binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-android-aarch64
path: target/aarch64-linux-android/release/deltachat-rpc-server
if-no-files-found: error
- name: Upload binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-android-i686
path: target/i686-linux-android/release/deltachat-rpc-server
if-no-files-found: error
- name: Upload binary
uses: actions/upload-artifact@v3
with:
name: deltachat-rpc-server-android-x86_64
path: target/x86_64-linux-android/release/deltachat-rpc-server
if-no-files-found: error
build_windows:
name: Build deltachat-rpc-server for Windows
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-22.04
artifact: gnu-linux-x86_64
path: deltachat-rpc-server
target: x86_64-unknown-linux-gnu
- os: windows-latest
artifact: win32.exe
path: deltachat-rpc-server.exe

9
.github/workflows/node-tests.yml vendored
View File

@@ -1,11 +1,16 @@
name: "node.js tests"
# Cancel previously started workflow runs
# when the branch is updated.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
on:
pull_request:
push:
branches:
- master
- staging
- trying
jobs:
tests:

24
CHANGELOG.md
View File

@@ -2,6 +2,23 @@
## Unreleased
### Changes
- Make smeared timestamp generation non-async. #4075
- Set minimum TLS version to 1.2. #4096
- Run `cargo-deny` in CI. #4101
- Check provider database with CI. #4099
- Switch to DEFERRED transactions #4100
- Ability to send backup over network and QR code to setup second device #4007
### Fixes
- Do not block async task executor while decrypting the messages. #4079
### API-Changes
- jsonrpc: add more advanced API to send a message. #4097
- jsonrpc: add get webxdc blob API `getWebxdcBlob` #4070
## 1.110.0
### Changes
- use transaction in `Contact::add_or_lookup()` #4059
- Organize the connection pool as a stack rather than a queue to ensure that
@@ -11,13 +28,15 @@
- Remove `Sql.get_conn()` interface in favor of `.call()` and `.transaction()`. #4055
- Updated provider database.
- Disable DKIM-Checks again #4076
- ability to send backup over network and QR code to setup second device #4007
- Switch from "X.Y.Z" and "py-X.Y.Z" to "vX.Y.Z" tags. #4089
- mimeparser: handle headers from the signed part of unencrypted signed message #4013
### Fixes
- Start SQL transactions with IMMEDIATE behaviour rather than default DEFERRED one. #4063
- Fix a problem with Gmail where (auto-)deleted messages would get archived instead of deleted.
Move them to the Trash folder for Gmail which auto-deletes trashed messages in 30 days #3972
- Clear config cache after backup import. This bug sometimes resulted in the import to seemingly work at first. #4067
- Update timestamps in `param` columns with transactions. #4083
### API-Changes
@@ -51,6 +70,7 @@
- Prefer TLS over STARTTLS during autoconfiguration #4021
- Use SOCKS5 configuration for HTTP requests #4017
- Show non-deltachat emails by default for new installations #4019
- Re-enabled SMTP pipelining after disabling it in #4006
### Fixes
- Fix Securejoin for multiple devices on a joining side #3982
@@ -438,7 +458,7 @@
- Auto accept contact requests if `Config::Bot` is set for a client #3567
- Don't prepend the subject to chat messages in mailinglists
- fix `set_core_version.py` script to also update version in `deltachat-jsonrpc/typescript/package.json` #3585
- Reject webxcd-updates from contacts who are not group members #3568
- Reject webxdc-updates from contacts who are not group members #3568
## 1.93.0

982
Cargo.lock generated
View File

File diff suppressed because it is too large Load Diff

2
Cargo.toml
View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat"
version = "1.109.0"
version = "1.110.0"
edition = "2021"
license = "MPL-2.0"
rust-version = "1.63"

2
deltachat-ffi/Cargo.toml
View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat_ffi"
version = "1.109.0"
version = "1.110.0"
description = "Deltachat FFI"
edition = "2018"
readme = "README.md"

35
deltachat-ffi/src/lib.rs
View File

@@ -4697,33 +4697,22 @@ mod jsonrpc {
return ptr::null_mut();
}
let cmd_api =
deltachat_jsonrpc::api::CommandApi::from_arc((*account_manager).inner.clone());
let account_manager = &*account_manager;
let events = block_on(account_manager.read()).get_event_emitter();
let cmd_api = deltachat_jsonrpc::api::CommandApi::from_arc(account_manager.inner.clone());
let (request_handle, receiver) = RpcClient::new();
let request_handle2 = request_handle.clone();
let handle = RpcSession::new(request_handle, cmd_api);
let handle = RpcSession::new(request_handle.clone(), cmd_api);
let events = block_on({
async {
let am = (*account_manager).inner.clone();
let ev = am.read().await.get_event_emitter();
drop(am);
ev
}
});
let event_thread = spawn({
async move {
while let Some(event) = events.recv().await {
let event = event_to_json_rpc_notification(event);
request_handle2
.send_notification("event", Some(event))
.await?;
}
let res: Result<(), anyhow::Error> = Ok(());
res
let event_thread = spawn(async move {
while let Some(event) = events.recv().await {
let event = event_to_json_rpc_notification(event);
request_handle
.send_notification("event", Some(event))
.await?;
}
let res: Result<(), anyhow::Error> = Ok(());
res
});
let instance = dc_jsonrpc_instance_t {

7
deltachat-jsonrpc/Cargo.toml
View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat-jsonrpc"
version = "1.109.0"
version = "1.110.0"
description = "DeltaChat JSON-RPC API"
edition = "2021"
default-run = "deltachat-jsonrpc-server"
@@ -21,14 +21,15 @@ log = "0.4"
async-channel = { version = "1.8.0" }
futures = { version = "0.3.26" }
serde_json = "1.0.91"
yerpc = { version = "^0.4.0", features = ["anyhow_expose"] }
yerpc = { version = "0.4.3", features = ["anyhow_expose"] }
typescript-type-def = { version = "0.5.5", features = ["json_value"] }
tokio = { version = "1.25.0" }
sanitize-filename = "0.4"
walkdir = "2.3.2"
base64 = "0.21"
# optional dependencies
axum = { version = "0.5.9", optional = true, features = ["ws"] }
axum = { version = "0.6.6", optional = true, features = ["ws"] }
env_logger = { version = "0.10.0", optional = true }
[dev-dependencies]

60
deltachat-jsonrpc/src/api/mod.rs
View File

@@ -41,6 +41,7 @@ use types::account::Account;
use types::chat::FullChat;
use types::chat_list::ChatListEntry;
use types::contact::ContactObject;
use types::message::MessageData;
use types::message::MessageObject;
use types::provider_info::ProviderInfo;
use types::webxdc::WebxdcMessageInfo;
@@ -1525,6 +1526,23 @@ impl CommandApi {
WebxdcMessageInfo::get_for_message(&ctx, MsgId::new(instance_msg_id)).await
}
/// Get blob encoded as base64 from a webxdc message
///
/// path is the path of the file within webxdc archive
async fn get_webxdc_blob(
&self,
account_id: u32,
instance_msg_id: u32,
path: String,
) -> Result<String> {
let ctx = self.get_context(account_id).await?;
let message = Message::load_from_db(&ctx, MsgId::new(instance_msg_id)).await?;
let blob = message.get_webxdc_blob(&ctx, &path).await?;
use base64::{engine::general_purpose, Engine as _};
Ok(general_purpose::STANDARD_NO_PAD.encode(blob))
}
/// Forward messages to another chat.
///
/// All types of messages can be forwarded,
@@ -1574,6 +1592,48 @@ impl CommandApi {
Ok(message_id.to_u32())
}
async fn send_msg(&self, account_id: u32, chat_id: u32, data: MessageData) -> Result<u32> {
let ctx = self.get_context(account_id).await?;
let mut message = Message::new(if let Some(viewtype) = data.viewtype {
viewtype.into()
} else if data.file.is_some() {
Viewtype::File
} else {
Viewtype::Text
});
if data.text.is_some() {
message.set_text(data.text);
}
if data.html.is_some() {
message.set_html(data.html);
}
if data.override_sender_name.is_some() {
message.set_override_sender_name(data.override_sender_name);
}
if let Some(file) = data.file {
message.set_file(file, None);
}
if let Some((latitude, longitude)) = data.location {
message.set_location(latitude, longitude);
}
if let Some(id) = data.quoted_message_id {
message
.set_quote(
&ctx,
Some(
&Message::load_from_db(&ctx, MsgId::new(id))
.await
.context("message to quote could not be loaded")?,
),
)
.await?;
}
let msg_id = chat::send_msg(&ctx, ChatId::new(chat_id), &mut message)
.await?
.to_u32();
Ok(msg_id)
}
// ---------------------------------------------
// functions for the composer
// the composer is the message input field

12
deltachat-jsonrpc/src/api/types/message.rs
View File

@@ -502,3 +502,15 @@ impl From<ChatItem> for JSONRPCMessageListItem {
}
}
}
#[derive(Deserialize, TypeDef)]
#[serde(rename_all = "camelCase")]
pub struct MessageData {
pub text: Option<String>,
pub html: Option<String>,
pub viewtype: Option<MessageViewtype>,
pub file: Option<String>,
pub location: Option<(f64, f64)>,
pub override_sender_name: Option<String>,
pub quoted_message_id: Option<u32>,
}

10
deltachat-jsonrpc/typescript/package.json
View File

@@ -3,7 +3,7 @@
"dependencies": {
"@deltachat/tiny-emitter": "3.0.0",
"isomorphic-ws": "^4.0.1",
"yerpc": "^0.3.3"
"yerpc": "^0.4.3"
},
"devDependencies": {
"@types/chai": "^4.2.21",
@@ -26,8 +26,8 @@
},
"exports": {
".": {
"require": "./dist/deltachat.cjs",
"import": "./dist/deltachat.js"
"import": "./dist/deltachat.js",
"require": "./dist/deltachat.cjs"
}
},
"license": "MPL-2.0",
@@ -36,8 +36,8 @@
"scripts": {
"build": "run-s generate-bindings extract-constants build:tsc build:bundle build:cjs",
"build:bundle": "esbuild --format=esm --bundle dist/deltachat.js --outfile=dist/deltachat.bundle.js",
"build:tsc": "tsc",
"build:cjs": "esbuild --format=cjs --bundle --packages=external dist/deltachat.js --outfile=dist/deltachat.cjs",
"build:tsc": "tsc",
"docs": "typedoc --out docs deltachat.ts",
"example": "run-s build example:build example:start",
"example:build": "esbuild --bundle dist/example/example.js --outfile=dist/example.bundle.js",
@@ -55,5 +55,5 @@
},
"type": "module",
"types": "dist/deltachat.d.ts",
"version": "1.109.0"
"version": "1.110.0"
}

3
deltachat-repl/Cargo.toml
View File

@@ -1,6 +1,7 @@
[package]
name = "deltachat-repl"
version = "1.109.0"
version = "1.110.0"
license = "MPL-2.0"
edition = "2021"
[dependencies]

42
deltachat-rpc-client/src/deltachat_rpc_client/chat.py
View File

@@ -3,7 +3,7 @@ from dataclasses import dataclass
from typing import TYPE_CHECKING, Dict, List, Optional, Tuple, Union
from ._utils import AttrDict
from .const import ChatVisibility
from .const import ChatVisibility, ViewType
from .contact import Contact
from .message import Message
@@ -108,15 +108,27 @@ class Chat:
async def send_message(
self,
text: Optional[str] = None,
html: Optional[str] = None,
viewtype: Optional[ViewType] = None,
file: Optional[str] = None,
location: Optional[Tuple[float, float]] = None,
override_sender_name: Optional[str] = None,
quoted_msg: Optional[Union[int, Message]] = None,
) -> Message:
"""Send a message and return the resulting Message instance."""
if isinstance(quoted_msg, Message):
quoted_msg = quoted_msg.id
msg_id, _ = await self._rpc.misc_send_msg(self.account.id, self.id, text, file, location, quoted_msg)
draft = {
"text": text,
"html": html,
"viewtype": viewtype,
"file": file,
"location": location,
"overrideSenderName": override_sender_name,
"quotedMsg": quoted_msg,
}
msg_id = await self._rpc.send_msg(self.account.id, self.id, draft)
return Message(self.account, msg_id)
async def send_text(self, text: str) -> Message:
@@ -182,19 +194,23 @@ class Chat:
"""Add contacts to this group."""
for cnt in contact:
if isinstance(cnt, str):
cnt = (await self.account.create_contact(cnt)).id
contact_id = (await self.account.create_contact(cnt)).id
elif not isinstance(cnt, int):
cnt = cnt.id
await self._rpc.add_contact_to_chat(self.account.id, self.id, cnt)
contact_id = cnt.id
else:
contact_id = cnt
await self._rpc.add_contact_to_chat(self.account.id, self.id, contact_id)
async def remove_contact(self, *contact: Union[int, str, Contact]) -> None:
"""Remove members from this group."""
for cnt in contact:
if isinstance(cnt, str):
cnt = (await self.account.create_contact(cnt)).id
contact_id = (await self.account.create_contact(cnt)).id
elif not isinstance(cnt, int):
cnt = cnt.id
await self._rpc.remove_contact_from_chat(self.account.id, self.id, cnt)
contact_id = cnt.id
else:
contact_id = cnt
await self._rpc.remove_contact_from_chat(self.account.id, self.id, contact_id)
async def get_contacts(self) -> List[Contact]:
"""Get the contacts belonging to this chat.
@@ -230,9 +246,9 @@ class Chat:
locations = []
contacts: Dict[int, Contact] = {}
for loc in result:
loc = AttrDict(loc)
loc["chat"] = self
loc["contact"] = contacts.setdefault(loc.contact_id, Contact(self.account, loc.contact_id))
loc["message"] = Message(self.account, loc.msg_id)
locations.append(loc)
location = AttrDict(loc)
location["chat"] = self
location["contact"] = contacts.setdefault(location.contact_id, Contact(self.account, location.contact_id))
location["message"] = Message(self.account, location.msg_id)
locations.append(location)
return locations

2
deltachat-rpc-server/Cargo.toml
View File

@@ -1,6 +1,6 @@
[package]
name = "deltachat-rpc-server"
version = "1.109.0"
version = "1.110.0"
description = "DeltaChat JSON-RPC server"
edition = "2021"
readme = "README.md"

16
deny.toml Normal file
View File

@@ -0,0 +1,16 @@
[advisories]
ignore = [
"RUSTSEC-2020-0071",
]
[licenses]
allow = [
"0BSD",
"Apache-2.0",
"BSD-2-Clause",
"BSD-3-Clause",
"CC0-1.0",
"MIT",
"BSL-1.0", # Boost Software License 1.0
"Unicode-DFS-2016",
]

2
package.json
View File

@@ -60,5 +60,5 @@
"test:mocha": "mocha -r esm node/test/test.js --growl --reporter=spec --bail --exit"
},
"types": "node/dist/index.d.ts",
"version": "1.109.0"
"version": "1.110.0"
}

6
python/pyproject.toml
View File

@@ -44,14 +44,14 @@ deltachat = [
[tool.setuptools_scm]
root = ".."
tag_regex = '^(?P<prefix>py-)?(?P<version>[^\+]+)(?P<suffix>.*)?$'
git_describe_command = "git describe --dirty --tags --long --match py-*.*"
tag_regex = '^(?P<prefix>v)?(?P<version>[^\+]+)(?P<suffix>.*)?$'
git_describe_command = "git describe --dirty --tags --long --match v*.*"
[tool.black]
line-length = 120
[tool.ruff]
select = ["E", "F", "W", "YTT", "C4", "ISC", "ICN", "TID", "DTZ", "PLC", "PLE", "PLW", "PIE", "COM", "UP004", "UP010", "UP031", "UP032"]
select = ["E", "F", "W", "YTT", "C4", "ISC", "ICN", "TID", "DTZ", "PLC", "PLE", "PLW", "PIE", "COM", "UP004", "UP010", "UP031", "UP032", "ANN204"]
line-length = 120
[tool.isort]

14
python/src/deltachat/account.py
View File

@@ -121,7 +121,7 @@ class Account:
"""re-enable logging."""
self._logging = True
def __repr__(self):
def __repr__(self) -> str:
return f"<Account path={self.db_path}>"
# def __del__(self):
@@ -284,9 +284,9 @@ class Account:
:returns: :class:`deltachat.contact.Contact` instance.
"""
(name, addr) = self.get_contact_addr_and_name(obj, name)
name = as_dc_charpointer(name)
addr = as_dc_charpointer(addr)
contact_id = lib.dc_create_contact(self._dc_context, name, addr)
name_c = as_dc_charpointer(name)
addr_c = as_dc_charpointer(addr)
contact_id = lib.dc_create_contact(self._dc_context, name_c, addr_c)
return Contact(self, contact_id)
def get_contact(self, obj) -> Optional[Contact]:
@@ -363,12 +363,12 @@ class Account:
:returns: list of :class:`deltachat.contact.Contact` objects.
"""
flags = 0
query = as_dc_charpointer(query)
query_c = as_dc_charpointer(query)
if only_verified:
flags |= const.DC_GCL_VERIFIED_ONLY
if with_self:
flags |= const.DC_GCL_ADD_SELF
dc_array = ffi.gc(lib.dc_get_contacts(self._dc_context, flags, query), lib.dc_array_unref)
dc_array = ffi.gc(lib.dc_get_contacts(self._dc_context, flags, query_c), lib.dc_array_unref)
return list(iter_array(dc_array, lambda x: Contact(self, x)))
def get_fresh_messages(self) -> Generator[Message, None, None]:
@@ -767,7 +767,7 @@ class Account:
class ScannedQRCode:
def __init__(self, dc_lot):
def __init__(self, dc_lot) -> None:
self._dc_lot = dc_lot
def is_ask_verifycontact(self):

16
python/src/deltachat/chat.py
View File

@@ -24,7 +24,7 @@ class Chat:
You obtain instances of it through :class:`deltachat.account.Account`.
"""
def __init__(self, account, id) -> None:
def __init__(self, account, id: int) -> None:
from .account import Account
assert isinstance(account, Account), repr(account)
@@ -162,8 +162,8 @@ class Chat:
:param name: as a unicode string.
:returns: True on success, False otherwise
"""
name = as_dc_charpointer(name)
return bool(lib.dc_set_chat_name(self.account._dc_context, self.id, name))
name_c = as_dc_charpointer(name)
return bool(lib.dc_set_chat_name(self.account._dc_context, self.id, name_c))
def get_color(self):
"""return the color of the chat.
@@ -532,13 +532,13 @@ class Chat:
# ------ location streaming API ------------------------------
def is_sending_locations(self):
def is_sending_locations(self) -> bool:
"""return True if this chat has location-sending enabled currently.
:returns: True if location sending is enabled.
"""
return lib.dc_is_sending_locations_to_chat(self.account._dc_context, self.id)
return bool(lib.dc_is_sending_locations_to_chat(self.account._dc_context, self.id))
def enable_sending_locations(self, seconds):
def enable_sending_locations(self, seconds) -> None:
"""enable sending locations for this chat.
all subsequent messages will carry a location with them.
@@ -572,7 +572,7 @@ class Chat:
class Location:
def __init__(self, latitude, longitude, accuracy, timestamp, marker):
def __init__(self, latitude, longitude, accuracy, timestamp, marker) -> None:
assert isinstance(timestamp, datetime)
self.latitude = latitude
self.longitude = longitude
@@ -580,5 +580,5 @@ class Location:
self.timestamp = timestamp
self.marker = marker
def __eq__(self, other):
def __eq__(self, other) -> bool:
return self.__dict__ == other.__dict__

6
python/src/deltachat/contact.py
View File

@@ -15,7 +15,7 @@ class Contact:
You obtain instances of it through :class:`deltachat.account.Account`.
"""
def __init__(self, account, id):
def __init__(self, account, id) -> None:
from .account import Account
assert isinstance(account, Account), repr(account)
@@ -27,10 +27,10 @@ class Contact:
return False
return self.account._dc_context == other.account._dc_context and self.id == other.id
def __ne__(self, other):
def __ne__(self, other) -> bool:
return not self == other
def __repr__(self):
def __repr__(self) -> str:
return f"<Contact id={self.id} addr={self.addr} dc_context={self.account._dc_context}>"
@property

2
python/src/deltachat/direct_imap.py
View File

@@ -191,7 +191,7 @@ class DirectImap:
class IdleManager:
def __init__(self, direct_imap):
def __init__(self, direct_imap) -> None:
self.direct_imap = direct_imap
self.log = direct_imap.account.log
# fetch latest messages before starting idle so that it only

9
python/src/deltachat/events.py
View File

@@ -25,12 +25,12 @@ def get_dc_event_name(integer, _DC_EVENTNAME_MAP={}):
class FFIEvent:
def __init__(self, name: str, data1, data2):
def __init__(self, name: str, data1, data2) -> None:
self.name = name
self.data1 = data1
self.data2 = data2
def __str__(self):
def __str__(self) -> str:
if self.name == "DC_EVENT_INFO":
return f"INFO {self.data2}"
if self.name == "DC_EVENT_WARNING":
@@ -84,7 +84,10 @@ class FFIEventLogger:
class FFIEventTracker:
def __init__(self, account, timeout=None):
account: Account
_event_queue: Queue
def __init__(self, account: Account, timeout=None) -> None:
self.account = account
self._timeout = timeout
self._event_queue = Queue()

4
python/src/deltachat/message.py
View File

@@ -19,7 +19,7 @@ class Message:
:class:`deltachat.chat.Chat`.
"""
def __init__(self, account, dc_msg):
def __init__(self, account, dc_msg) -> None:
self.account = account
assert isinstance(self.account._dc_context, ffi.CData)
assert isinstance(dc_msg, ffi.CData)
@@ -33,7 +33,7 @@ class Message:
return False
return self.account == other.account and self.id == other.id
def __repr__(self):
def __repr__(self) -> str:
c = self.get_sender_contact()
typ = "outgoing" if self.is_outgoing() else "incoming"
return (

4
python/src/deltachat/reactions.py
View File

@@ -10,14 +10,14 @@ class Reactions:
You obtain instances of it through :class:`deltachat.message.Message`.
"""
def __init__(self, account, dc_reactions):
def __init__(self, account, dc_reactions) -> None:
assert isinstance(account._dc_context, ffi.CData)
assert isinstance(dc_reactions, ffi.CData)
assert dc_reactions != ffi.NULL
self.account = account
self._dc_reactions = dc_reactions
def __repr__(self):
def __repr__(self) -> str:
return f"<Reactions dc_reactions={self._dc_reactions}>"
@classmethod

18
python/src/deltachat/testplugin.py
View File

@@ -9,7 +9,7 @@ import threading
import time
import weakref
from queue import Queue
from typing import Callable, List, Optional
from typing import Callable, List, Optional, Dict, Set
import pytest
import requests
@@ -65,8 +65,8 @@ def pytest_configure(config):
# Additionally make the acfactory use a logging/no-logging default.
class LoggingAspect:
def __init__(self):
self._accounts = weakref.WeakSet()
def __init__(self) -> None:
self._accounts: weakref.WeakSet = weakref.WeakSet()
@deltachat.global_hookimpl
def dc_account_init(self, account):
@@ -143,10 +143,12 @@ def testprocess(request):
class TestProcess:
"""A pytest session-scoped instance to help with managing "live" account configurations."""
def __init__(self, pytestconfig):
_addr2files: Dict[str, Dict[pathlib.Path, bytes]]
def __init__(self, pytestconfig) -> None:
self.pytestconfig = pytestconfig
self._addr2files = {}
self._configlist = []
self._configlist: List[Dict[str, str]] = []
def get_liveconfig_producer(self):
"""provide live account configs, cached on a per-test-process scope
@@ -277,10 +279,10 @@ class ACSetup:
_configured_events: Queue
def __init__(self, testprocess, init_time):
def __init__(self, testprocess, init_time) -> None:
self._configured_events = Queue()
self._account2state = {}
self._imap_cleaned = set()
self._account2state: Dict[Account, str] = {}
self._imap_cleaned: Set[str] = set()
self.testprocess = testprocess
self.init_time = init_time

20
python/src/deltachat/tracker.py
View File

@@ -1,19 +1,25 @@
from queue import Queue
from threading import Event
from typing import List, TYPE_CHECKING
from .hookspec import Global, account_hookimpl
if TYPE_CHECKING:
from .events import FFIEvent
class ImexFailed(RuntimeError):
"""Exception for signalling that import/export operations failed."""
class ImexTracker:
def __init__(self):
_imex_events: Queue
def __init__(self) -> None:
self._imex_events = Queue()
@account_hookimpl
def ac_process_ffi_event(self, ffi_event):
def ac_process_ffi_event(self, ffi_event: "FFIEvent") -> None:
if ffi_event.name == "DC_EVENT_IMEX_PROGRESS":
self._imex_events.put(ffi_event.data1)
elif ffi_event.name == "DC_EVENT_IMEX_FILE_WRITTEN":
@@ -50,7 +56,13 @@ class ConfigureFailed(RuntimeError):
class ConfigureTracker:
ConfigureFailed = ConfigureFailed
def __init__(self, account):
_configure_events: Queue
_smtp_finished: Event
_imap_finished: Event
_ffi_events: List["FFIEvent"]
_progress: Queue
def __init__(self, account) -> None:
self.account = account
self._configure_events = Queue()
self._smtp_finished = Event()
@@ -60,7 +72,7 @@ class ConfigureTracker:
self._gm = Global._get_plugin_manager()
@account_hookimpl
def ac_process_ffi_event(self, ffi_event):
def ac_process_ffi_event(self, ffi_event: "FFIEvent") -> None:
self._ffi_events.append(ffi_event)
if ffi_event.name == "DC_EVENT_SMTP_CONNECTED":
self._smtp_finished.set()

4
python/tests/stress_test_db.py
View File

@@ -77,7 +77,7 @@ class ReportType:
class AutoReplier:
def __init__(self, account, log, num_send, num_bigfiles, report_func):
def __init__(self, account, log, num_send, num_bigfiles, report_func) -> None:
self.account = account
self._log = log
self.report_func = report_func
@@ -90,7 +90,7 @@ class AutoReplier:
self._thread.setDaemon(True)
self._thread.start()
def log(self, message):
def log(self, message) -> None:
self._log(f"{self.addr} {message}")
def thread_stats(self):

25
python/tests/test_1_online.py
View File

@@ -1604,7 +1604,7 @@ def test_add_remove_member_remote_events(acfactory, lp):
in_list = queue.Queue()
class EventHolder:
def __init__(self, **kwargs):
def __init__(self, **kwargs) -> None:
self.__dict__.update(kwargs)
class InPlugin:
@@ -1987,13 +1987,16 @@ def test_delete_multiple_messages(acfactory, lp):
lp.sec("ac2: deleting all messages except third")
assert len(to_delete) == len(texts) - 1
ac2.delete_messages(to_delete)
ac2._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_DELETED")
ac2._evtracker.get_info_contains("close/expunge succeeded")
lp.sec("ac2: test that only one message is left")
ac2.direct_imap.select_config_folder("inbox")
assert len(ac2.direct_imap.get_all_messages()) == 1
while 1:
ac2._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_DELETED")
ac2._evtracker.get_info_contains("close/expunge succeeded")
ac2.direct_imap.select_config_folder("inbox")
nr_msgs = len(ac2.direct_imap.get_all_messages())
assert nr_msgs > 0
if nr_msgs == 1:
break
def test_trash_multiple_messages(acfactory, lp):
@@ -2017,11 +2020,15 @@ def test_trash_multiple_messages(acfactory, lp):
lp.sec("ac2: deleting all messages except second")
assert len(to_delete) == len(texts) - 1
ac2.delete_messages(to_delete)
ac2._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
lp.sec("ac2: test that only one message is left")
ac2.direct_imap.select_config_folder("inbox")
assert len(ac2.direct_imap.get_all_messages()) == 1
while 1:
ac2._evtracker.get_matching("DC_EVENT_IMAP_MESSAGE_MOVED")
ac2.direct_imap.select_config_folder("inbox")
nr_msgs = len(ac2.direct_imap.get_all_messages())
assert nr_msgs > 0
if nr_msgs == 1:
break
def test_configure_error_msgs_wrong_pw(acfactory):

2
scripts/README.md
View File

@@ -20,6 +20,8 @@ and an own build machine.
- `run_all.sh` builds Python wheels
- `aarch64-unknown-linux-musl.sh` cross-compiles static `deltachat-rpc-server` for aarch64
## Triggering runs on the build machine locally (fast!)
There is experimental support for triggering a remote Python or Rust test run

18
scripts/aarch64-unknown-linux-musl.sh Executable file
View File

@@ -0,0 +1,18 @@
#!/bin/sh
#
# Build statically linked deltachat-rpc-server for aarch64-unknown-linux-musl.
set -x
set -e
# Download Zig
rm -fr zig-linux-x86_64-0.10.1 zig-linux-x86_64-0.10.1.tar.xz
wget https://ziglang.org/download/0.10.1/zig-linux-x86_64-0.10.1.tar.xz
tar xf zig-linux-x86_64-0.10.1.tar.xz
export PATH="$PATH:$PWD/zig-linux-x86_64-0.10.1"
cargo install cargo-zigbuild
rustup target add aarch64-unknown-linux-musl
cargo zigbuild --release --target aarch64-unknown-linux-musl -p deltachat-rpc-server --features vendored

44
scripts/android-rpc-server.sh Executable file
View File

@@ -0,0 +1,44 @@
#!/bin/sh
# Build deltachat-rpc-server for Android.
set -e
test -n "$ANDROID_NDK_ROOT" || exit 1
RUSTUP_TOOLCHAIN="1.64.0"
rustup install "$RUSTUP_TOOLCHAIN"
rustup target add armv7-linux-androideabi aarch64-linux-android i686-linux-android x86_64-linux-android --toolchain "$RUSTUP_TOOLCHAIN"
KERNEL="$(uname -s | tr '[:upper:]' '[:lower:]')"
ARCH="$(uname -m)"
NDK_HOST_TAG="$KERNEL-$ARCH"
TOOLCHAIN="$ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/$NDK_HOST_TAG"
export PATH="$PATH:$TOOLCHAIN/bin/"
PACKAGE="deltachat-rpc-server"
export CARGO_PROFILE_RELEASE_LTO=on
CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_LINKER="$TOOLCHAIN/bin/armv7a-linux-androideabi16-clang" \
CFLAGS=-D__ANDROID_API__=16 \
TARGET_CC=armv7a-linux-androideabi16-clang \
TARGET_AR=llvm-ar \
cargo "+$RUSTUP_TOOLCHAIN" rustc --release --target armv7-linux-androideabi -p $PACKAGE
CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER="$TOOLCHAIN/bin/aarch64-linux-android21-clang" \
CFLAGS=-D__ANDROID_API__=21 \
TARGET_CC=aarch64-linux-android21-clang \
TARGET_AR=llvm-ar \
cargo "+$RUSTUP_TOOLCHAIN" rustc --release --target aarch64-linux-android -p $PACKAGE
CARGO_TARGET_I686_LINUX_ANDROID_LINKER="$TOOLCHAIN/bin/i686-linux-android16-clang" \
CFLAGS=-D__ANDROID_API__=16 \
TARGET_CC=i686-linux-android16-clang \
TARGET_AR=llvm-ar \
cargo "+$RUSTUP_TOOLCHAIN" rustc --release --target i686-linux-android -p $PACKAGE
CARGO_TARGET_X86_64_LINUX_ANDROID_LINKER="$TOOLCHAIN/bin/x86_64-linux-android21-clang" \
CFLAGS=-D__ANDROID_API__=21 \
TARGET_CC=x86_64-linux-android21-clang \
TARGET_AR=llvm-ar \
cargo "+$RUSTUP_TOOLCHAIN" rustc --release --target x86_64-linux-android -p $PACKAGE

694
scripts/concourse/docs_wheels.yml
View File

@@ -1,370 +1,370 @@
resources:
- name: deltachat-core-rust
type: git
icon: github
source:
branch: master
uri: https://github.com/deltachat/deltachat-core-rust.git
- name: deltachat-core-rust
type: git
icon: github
source:
branch: master
uri: https://github.com/deltachat/deltachat-core-rust.git
- name: deltachat-core-rust-release
type: git
icon: github
source:
branch: master
uri: https://github.com/deltachat/deltachat-core-rust.git
tag_filter: "py-*"
- name: deltachat-core-rust-release
type: git
icon: github
source:
branch: master
uri: https://github.com/deltachat/deltachat-core-rust.git
tag_filter: "v*"
jobs:
- name: doxygen
plan:
- get: deltachat-core-rust
trigger: true
- name: doxygen
plan:
- get: deltachat-core-rust
trigger: true
# Build Doxygen documentation
- task: build-doxygen
config:
inputs:
- name: deltachat-core-rust
outputs:
- name: c-docs
image_resource:
source:
repository: alpine
type: registry-image
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache doxygen git
cd deltachat-core-rust
scripts/run-doxygen.sh
cd ..
cp -av deltachat-core-rust/deltachat-ffi/html deltachat-core-rust/deltachat-ffi/xml c-docs/
# Build Doxygen documentation
- task: build-doxygen
config:
inputs:
- name: deltachat-core-rust
outputs:
- name: c-docs
image_resource:
source:
repository: alpine
type: registry-image
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache doxygen git
cd deltachat-core-rust
scripts/run-doxygen.sh
cd ..
cp -av deltachat-core-rust/deltachat-ffi/html deltachat-core-rust/deltachat-ffi/xml c-docs/
- task: upload-c-docs
config:
inputs:
- name: c-docs
image_resource:
type: registry-image
source:
repository: alpine
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache rsync openssh-client
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete c-docs/html/ delta@c.delta.chat:build-c/master
- task: upload-c-docs
config:
inputs:
- name: c-docs
image_resource:
type: registry-image
source:
repository: alpine
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache rsync openssh-client
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete c-docs/html/ delta@c.delta.chat:build-c/master
- name: python-x86_64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
- name: python-x86_64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/manylinux2014_x86_64
platform: linux
caches:
- path: cache
run:
path: build
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/manylinux2014_x86_64
platform: linux
caches:
- path: cache
run:
path: build
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-docs
path: ./python/doc/_build/
# Binary wheels
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-docs
path: ./python/doc/_build/
# Binary wheels
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Upload python docs to py.delta.chat
- task: upload-py-docs
config:
inputs:
- name: py-docs
image_resource:
type: registry-image
source:
repository: alpine
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache rsync openssh-client
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete py-docs/html/ delta@py.delta.chat:build/master
# Upload python docs to py.delta.chat
- task: upload-py-docs
config:
inputs:
- name: py-docs
image_resource:
type: registry-image
source:
repository: alpine
platform: linux
run:
path: sh
args:
- -ec
- |
apk add --no-cache rsync openssh-client
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "(("c.delta.chat".private_key))" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
rsync -e "ssh -o StrictHostKeyChecking=no" -avz --delete py-docs/html/ delta@py.delta.chat:build/master
# Upload x86_64 wheels and source packages
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*manylinux201*
# Upload x86_64 wheels and source packages
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*manylinux201*
- name: python-aarch64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
- name: python-aarch64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/manylinux2014_aarch64
platform: linux
caches:
- path: cache
run:
path: build
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/manylinux2014_aarch64
platform: linux
caches:
- path: cache
run:
path: build
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Upload aarch64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*manylinux201*
# Upload aarch64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*manylinux201*
- name: python-musl-x86_64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
- name: python-musl-x86_64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/musllinux_1_1_x86_64
platform: linux
caches:
- path: cache
run:
path: build
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/musllinux_1_1_x86_64
platform: linux
caches:
- path: cache
run:
path: build
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Upload musl x86_64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*musllinux_1_1_x86_64*
# Upload musl x86_64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*musllinux_1_1_x86_64*
- name: python-musl-aarch64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
- name: python-musl-aarch64
plan:
- get: deltachat-core-rust
- get: deltachat-core-rust-release
trigger: true
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/musllinux_1_1_aarch64
platform: linux
caches:
- path: cache
run:
path: build
# Build manylinux image with additional dependencies
- task: build-coredeps
privileged: true
config:
inputs:
# Building the latest, not tagged coredeps
- name: deltachat-core-rust
image_resource:
source:
repository: concourse/oci-build-task
type: registry-image
outputs:
- name: coredeps-image
path: image
params:
CONTEXT: deltachat-core-rust/scripts/coredeps
UNPACK_ROOTFS: "true"
BUILD_ARG_BASEIMAGE: quay.io/pypa/musllinux_1_1_aarch64
platform: linux
caches:
- path: cache
run:
path: build
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Use built image to build python wheels
- task: build-wheels
image: coredeps-image
config:
inputs:
- name: deltachat-core-rust-release
path: .
outputs:
- name: py-wheels
path: ./python/.docker-tox/wheelhouse/
platform: linux
run:
path: bash
args:
- -exc
- |
scripts/run_all.sh
# Upload musl aarch64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*musllinux_1_1_aarch64*
# Upload musl aarch64 wheels
- task: upload-wheels
config:
inputs:
- name: py-wheels
image_resource:
type: registry-image
source:
repository: debian
platform: linux
run:
path: sh
args:
- -ec
- |
apt-get update -y
apt-get install -y --no-install-recommends python3-pip python3-setuptools
pip3 install devpi
devpi use https://m.devpi.net/dc/master
devpi login ((devpi.login)) --password ((devpi.password))
devpi upload py-wheels/*musllinux_1_1_aarch64*

92
src/provider/update.py → scripts/create-provider-data-rs.py
View File

@@ -11,15 +11,17 @@ out_domains = ""
out_ids = ""
domains_set = set()
def camel(name):
words = name.split("_")
return "".join(w.capitalize() for i, w in enumerate(words))
def cleanstr(s):
s = s.strip()
s = s.replace("\n", " ")
s = s.replace("\\", "\\\\")
s = s.replace("\"", "\\\"")
s = s.replace('"', '\\"')
return s
@@ -64,7 +66,13 @@ def process_config_defaults(data):
config_defaults = data.get("config_defaults", "")
for key in config_defaults:
value = str(config_defaults[key])
defaults += " ConfigDefault { key: Config::" + camel(key) + ", value: \"" + value + "\" },\n"
defaults += (
" ConfigDefault { key: Config::"
+ camel(key)
+ ', value: "'
+ value
+ '" },\n'
)
defaults += " ])"
return defaults
@@ -88,11 +96,11 @@ def process_data(data, file):
raise TypeError("domain used twice: " + domain)
domains_set.add(domain)
domains += " (\"" + domain + "\", &*" + file2varname(file) + "),\n"
domains += ' ("' + domain + '", &*' + file2varname(file) + "),\n"
comment += domain + ", "
ids = ""
ids += " (\"" + file2id(file) + "\", &*" + file2varname(file) + "),\n"
ids += ' ("' + file2id(file) + '", &*' + file2varname(file) + "),\n"
server = ""
has_imap = False
@@ -120,8 +128,19 @@ def process_data(data, file):
if username_pattern != "EMAIL" and username_pattern != "EMAILLOCALPART":
raise TypeError("bad username pattern")
server += (" Server { protocol: " + protocol.capitalize() + ", socket: " + socket.capitalize() + ", hostname: \""
+ hostname + "\", port: " + str(port) + ", username_pattern: " + username_pattern.capitalize() + " },\n")
server += (
" Server { protocol: "
+ protocol.capitalize()
+ ", socket: "
+ socket.capitalize()
+ ', hostname: "'
+ hostname
+ '", port: '
+ str(port)
+ ", username_pattern: "
+ username_pattern.capitalize()
+ " },\n"
)
opt = process_opt(data)
config_defaults = process_config_defaults(data)
@@ -133,12 +152,16 @@ def process_data(data, file):
before_login_hint = cleanstr(data.get("before_login_hint", ""))
after_login_hint = cleanstr(data.get("after_login_hint", ""))
if (not has_imap and not has_smtp) or (has_imap and has_smtp):
provider += "static " + file2varname(file) + ": Lazy<Provider> = Lazy::new(|| Provider {\n"
provider += " id: \"" + file2id(file) + "\",\n"
provider += (
"static "
+ file2varname(file)
+ ": Lazy<Provider> = Lazy::new(|| Provider {\n"
)
provider += ' id: "' + file2id(file) + '",\n'
provider += " status: Status::" + status.capitalize() + ",\n"
provider += " before_login_hint: \"" + before_login_hint + "\",\n"
provider += " after_login_hint: \"" + after_login_hint + "\",\n"
provider += " overview_page: \"" + file2url(file) + "\",\n"
provider += ' before_login_hint: "' + before_login_hint + '",\n'
provider += ' after_login_hint: "' + after_login_hint + '",\n'
provider += ' overview_page: "' + file2url(file) + '",\n'
provider += " server: vec![\n" + server + " ],\n"
provider += " opt: " + opt + ",\n"
provider += " config_defaults: " + config_defaults + ",\n"
@@ -148,7 +171,9 @@ def process_data(data, file):
raise TypeError("SMTP and IMAP must be specified together or left out both")
if status != "OK" and before_login_hint == "":
raise TypeError("status PREPARATION or BROKEN requires before_login_hint: " + file)
raise TypeError(
"status PREPARATION or BROKEN requires before_login_hint: " + file
)
# finally, add the provider
global out_all, out_domains, out_ids
@@ -172,7 +197,7 @@ def process_file(file):
def process_dir(dir):
print("processing directory: {}".format(dir), file=sys.stderr)
files = sorted(f for f in dir.iterdir() if f.suffix == '.md')
files = sorted(f for f in dir.iterdir() if f.suffix == ".md")
for f in files:
process_file(f)
@@ -181,28 +206,41 @@ if __name__ == "__main__":
if len(sys.argv) < 2:
raise SystemExit("usage: update.py DIR_WITH_MD_FILES > data.rs")
out_all = ("// file generated by src/provider/update.py\n\n"
"use crate::provider::Protocol::*;\n"
"use crate::provider::Socket::*;\n"
"use crate::provider::UsernamePattern::*;\n"
"use crate::provider::{\n"
" Config, ConfigDefault, Oauth2Authorizer, Provider, ProviderOptions, Server, Status,\n"
"};\n"
"use std::collections::HashMap;\n\n"
"use once_cell::sync::Lazy;\n\n")
out_all = (
"// file generated by src/provider/update.py\n\n"
"use crate::provider::Protocol::*;\n"
"use crate::provider::Socket::*;\n"
"use crate::provider::UsernamePattern::*;\n"
"use crate::provider::{\n"
" Config, ConfigDefault, Oauth2Authorizer, Provider, ProviderOptions, Server, Status,\n"
"};\n"
"use std::collections::HashMap;\n\n"
"use once_cell::sync::Lazy;\n\n"
)
process_dir(Path(sys.argv[1]))
out_all += "pub(crate) static PROVIDER_DATA: Lazy<HashMap<&'static str, &'static Provider>> = Lazy::new(|| [\n"
out_all += out_domains;
out_all += out_domains
out_all += "].iter().copied().collect());\n\n"
out_all += "pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> = Lazy::new(|| [\n"
out_all += out_ids;
out_all += out_ids
out_all += "].iter().copied().collect());\n\n"
now = datetime.datetime.utcnow()
out_all += "pub static PROVIDER_UPDATED: Lazy<chrono::NaiveDate> = "\
"Lazy::new(|| chrono::NaiveDate::from_ymd_opt("+str(now.year)+", "+str(now.month)+", "+str(now.day)+").unwrap());\n"
if len(sys.argv) < 3:
now = datetime.datetime.utcnow()
else:
now = datetime.datetime.fromisoformat(sys.argv[2])
out_all += (
"pub static PROVIDER_UPDATED: Lazy<chrono::NaiveDate> = "
"Lazy::new(|| chrono::NaiveDate::from_ymd_opt("
+ str(now.year)
+ ", "
+ str(now.month)
+ ", "
+ str(now.day)
+ ").unwrap());\n"
)
print(out_all)

6
scripts/set_core_version.py
View File

@@ -115,10 +115,8 @@ def main():
print("after commit, on master make sure to: ")
print("")
print(f" git tag -a {newversion}")
print(f" git push origin {newversion}")
print(f" git tag -a py-{newversion}")
print(f" git push origin py-{newversion}")
print(f" git tag -a v{newversion}")
print(f" git push origin v{newversion}")
print("")

22
scripts/update-provider-database.sh Executable file
View File

@@ -0,0 +1,22 @@
#!/usr/bin/env bash
# Updates provider database.
# Returns 1 if the database is changed, 0 otherwise.
set -euo pipefail
export TZ=UTC
# Provider database revision.
REV=3c8f7e846c915a183dc44536fb5480d1f25d7c42
CORE_ROOT="$PWD"
TMP="$(mktemp -d)"
git clone --filter=blob:none https://github.com/deltachat/provider-db.git "$TMP"
cd "$TMP"
git checkout "$REV"
DATE=$(git show -s --format=%cs)
"$CORE_ROOT"/scripts/create-provider-data-rs.py "$TMP/_providers" "$DATE" >"$CORE_ROOT/src/provider/data.rs"
rustfmt "$CORE_ROOT/src/provider/data.rs"
rm -fr "$TMP"
cd "$CORE_ROOT"
test -z "$(git status --porcelain src/provider/data.rs)"

3
src/accounts.rs
View File

@@ -476,10 +476,13 @@ impl Config {
struct AccountConfig {
/// Unique id.
pub id: u32,
/// Root directory for all data for this account.
///
/// The path is relative to the account manager directory.
pub dir: std::path::PathBuf,
/// Universally unique account identifier.
pub uuid: Uuid,
}

32
src/chat.rs
View File

@@ -276,7 +276,7 @@ impl ChatId {
grpname,
grpid,
create_blocked,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
create_protected,
param.unwrap_or_default(),
],
@@ -482,7 +482,7 @@ impl ChatId {
self,
&msg_text,
cmd,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
None,
None,
None,
@@ -1881,7 +1881,10 @@ pub(crate) async fn update_special_chat_names(context: &Context) -> Result<()> {
/// [`Deref`]: std::ops::Deref
#[derive(Debug)]
pub(crate) struct ChatIdBlocked {
/// Chat ID.
pub id: ChatId,
/// Whether the chat is blocked, unblocked or a contact request.
pub blocked: Blocked,
}
@@ -1953,7 +1956,6 @@ impl ChatIdBlocked {
_ => (),
}
let created_timestamp = create_smeared_timestamp(context).await;
let chat_id = context
.sql
.transaction(move |transaction| {
@@ -1966,7 +1968,7 @@ impl ChatIdBlocked {
chat_name,
params.to_string(),
create_blocked as u8,
created_timestamp,
create_smeared_timestamp(context)
],
)?;
let chat_id = ChatId::new(
@@ -2114,7 +2116,7 @@ async fn prepare_msg_common(
context,
msg,
update_msg_id,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
)
.await?;
msg.chat_id = chat_id;
@@ -2839,7 +2841,7 @@ pub async fn create_group_chat(
Chattype::Group,
chat_name,
grpid,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
],
)
.await?;
@@ -2897,7 +2899,7 @@ pub async fn create_broadcast_list(context: &Context) -> Result<ChatId> {
Chattype::Broadcast,
chat_name,
grpid,
create_smeared_timestamp(context).await,
create_smeared_timestamp(context),
],
)
.await?;
@@ -3358,7 +3360,7 @@ pub async fn forward_msgs(context: &Context, msg_ids: &[MsgId], chat_id: ChatId)
if let Some(reason) = chat.why_cant_send(context).await? {
bail!("cannot send to {}: {}", chat_id, reason);
}
curr_timestamp = create_smeared_timestamps(context, msg_ids.len()).await;
curr_timestamp = create_smeared_timestamps(context, msg_ids.len());
let ids = context
.sql
.query_map(
@@ -3560,7 +3562,7 @@ pub async fn add_device_msg_with_importance(
msg.try_calc_and_set_dimensions(context).await.ok();
prepare_msg_blob(context, msg).await?;
let timestamp_sent = create_smeared_timestamp(context).await;
let timestamp_sent = create_smeared_timestamp(context);
// makes sure, the added message is the last one,
// even if the date is wrong (useful esp. when warning about bad dates)
@@ -4088,7 +4090,6 @@ mod tests {
send_text_msg(&alice, alice_chat_id, "populate".to_string()).await?;
add_contact_to_chat(&alice, alice_chat_id, bob_id).await?;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
let add1 = alice.pop_sent_msg().await;
add_contact_to_chat(&alice, alice_chat_id, claire_id).await?;
@@ -4107,29 +4108,18 @@ mod tests {
remove_contact_from_chat(&alice, alice_chat_id, daisy_id).await?;
let remove2 = alice.pop_sent_msg().await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
assert_eq!(get_chat_contacts(&alice, alice_chat_id).await?.len(), 2);
// Bob receives the add and deletion messages out of order
let bob = TestContext::new_bob().await;
bob.recv_msg(&add1).await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
bob.recv_msg(&add3).await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
let bob_chat_id = bob.recv_msg(&add2).await.chat_id;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
assert_eq!(get_chat_contacts(&bob, bob_chat_id).await?.len(), 4);
bob.recv_msg(&remove2).await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
bob.recv_msg(&remove1).await;
tokio::time::sleep(std::time::Duration::from_millis(1100)).await;
assert_eq!(get_chat_contacts(&bob, bob_chat_id).await?.len(), 2);
Ok(())

3
src/config.rs
View File

@@ -300,6 +300,9 @@ pub enum Config {
/// See `crate::authres::update_authservid_candidates`.
AuthservIdCandidates,
/// Make all outgoing messages with Autocrypt header "multipart/signed".
SignUnencrypted,
/// Let the core save all events to the database.
/// This value is used internally to remember the MsgId of the logging xdc
#[strum(props(default = "0"))]

4
src/configure.rs
View File

@@ -646,10 +646,14 @@ async fn try_smtp_one_param(
}
}
/// Failure to connect and login with email client configuration.
#[derive(Debug, thiserror::Error)]
#[error("Trying {config}…\nError: {msg}")]
pub struct ConfigurationError {
/// Tried configuration description.
config: String,
/// Error message.
msg: String,
}

6
src/constants.rs
View File

@@ -190,11 +190,11 @@ pub const DC_LP_AUTH_NORMAL: i32 = 0x4;
pub const DC_LP_AUTH_FLAGS: i32 = DC_LP_AUTH_OAUTH2 | DC_LP_AUTH_NORMAL;
/// How many existing messages shall be fetched after configuration.
pub const DC_FETCH_EXISTING_MSGS_COUNT: i64 = 100;
pub(crate) const DC_FETCH_EXISTING_MSGS_COUNT: i64 = 100;
// max. width/height of an avatar
pub const BALANCED_AVATAR_SIZE: u32 = 256;
pub const WORSE_AVATAR_SIZE: u32 = 128;
pub(crate) const BALANCED_AVATAR_SIZE: u32 = 256;
pub(crate) const WORSE_AVATAR_SIZE: u32 = 128;
// max. width/height of images
pub const BALANCED_IMAGE_SIZE: u32 = 1280;

20
src/context.rs
View File

@@ -4,6 +4,7 @@ use std::collections::{BTreeMap, HashMap};
use std::ffi::OsString;
use std::ops::Deref;
use std::path::{Path, PathBuf};
use std::sync::atomic::AtomicBool;
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime};
@@ -27,6 +28,7 @@ use crate::quota::QuotaInfo;
use crate::scheduler::Scheduler;
use crate::sql::Sql;
use crate::stock_str::StockStrings;
use crate::timesmearing::SmearedTimestamp;
use crate::tools::{duration_to_str, time};
/// Builder for the [`Context`].
@@ -189,7 +191,7 @@ pub struct InnerContext {
/// Blob directory path
pub(crate) blobdir: PathBuf,
pub(crate) sql: Sql,
pub(crate) last_smeared_timestamp: RwLock<i64>,
pub(crate) smeared_timestamp: SmearedTimestamp,
/// The global "ongoing" process state.
///
/// This is a global mutex-like state for operations which should be modal in the
@@ -211,6 +213,12 @@ pub struct InnerContext {
/// Set to `None` if quota was never tried to load.
pub(crate) quota: RwLock<Option<QuotaInfo>>,
/// Set to true if quota update is requested.
pub(crate) quota_update_request: AtomicBool,
/// IMAP UID resync request.
pub(crate) resync_request: AtomicBool,
/// Server ID response if ID capability is supported
/// and the server returned non-NIL on the inbox connection.
/// <https://datatracker.ietf.org/doc/html/rfc2971>
@@ -369,7 +377,7 @@ impl Context {
blobdir,
running_state: RwLock::new(Default::default()),
sql: Sql::new(dbfile),
last_smeared_timestamp: RwLock::new(0),
smeared_timestamp: SmearedTimestamp::new(),
generating_key_mutex: Mutex::new(()),
oauth2_mutex: Mutex::new(()),
wrong_pw_warning_mutex: Mutex::new(()),
@@ -378,6 +386,8 @@ impl Context {
scheduler: RwLock::new(None),
ratelimit: RwLock::new(Ratelimit::new(Duration::new(60, 0), 6.0)), // Allow to send 6 messages immediately, no more than once every 10 seconds.
quota: RwLock::new(None),
quota_update_request: AtomicBool::new(false),
resync_request: AtomicBool::new(false),
server_id: RwLock::new(None),
creation_time: std::time::SystemTime::now(),
last_full_folder_scan: Mutex::new(None),
@@ -789,6 +799,12 @@ impl Context {
.await?
.unwrap_or_default(),
);
res.insert(
"sign_unencrypted",
self.get_config_int(Config::SignUnencrypted)
.await?
.to_string(),
);
res.insert(
"debug_logging",

7
src/download.rs
View File

@@ -13,7 +13,6 @@ use crate::imap::{Imap, ImapActionResult};
use crate::job::{self, Action, Job, Status};
use crate::message::{Message, MsgId, Viewtype};
use crate::mimeparser::{MimeMessage, Part};
use crate::param::Params;
use crate::tools::time;
use crate::{job_try, stock_str, EventType};
@@ -86,11 +85,7 @@ impl MsgId {
DownloadState::Available | DownloadState::Failure => {
self.update_download_state(context, DownloadState::InProgress)
.await?;
job::add(
context,
Job::new(Action::DownloadMsg, self.to_u32(), Params::new(), 0),
)
.await?;
job::add(context, Job::new(Action::DownloadMsg, self.to_u32())).await?;
}
}
Ok(())

13
src/e2ee.rs
View File

@@ -124,6 +124,19 @@ impl EncryptHelper {
Ok(ctext)
}
/// Signs the passed-in `mail` using the private key from `context`.
/// Returns the payload and the signature.
pub async fn sign(
self,
context: &Context,
mail: lettre_email::PartBuilder,
) -> Result<(lettre_email::MimeMessage, String)> {
let sign_key = SignedSecretKey::load_self(context).await?;
let mime_message = mail.build();
let signature = pgp::pk_calc_signature(mime_message.as_string().as_bytes(), &sign_key)?;
Ok((mime_message, signature))
}
}
/// Ensures a private key exists for the configured user.

2
src/ephemeral.rs
View File

@@ -650,7 +650,7 @@ mod tests {
use crate::download::DownloadState;
use crate::receive_imf::receive_imf;
use crate::test_utils::TestContext;
use crate::tools::MAX_SECONDS_TO_LEND_FROM_FUTURE;
use crate::timesmearing::MAX_SECONDS_TO_LEND_FROM_FUTURE;
use crate::{
chat::{self, create_group_chat, send_text_msg, Chat, ChatItem, ProtectionStatus},
tools::IsNoneOrEmpty,

23
src/imap.rs
View File

@@ -116,6 +116,8 @@ impl async_imap::Authenticator for OAuth2 {
#[derive(Debug, Display, PartialEq, Eq, Clone, Copy)]
pub enum FolderMeaning {
Unknown,
/// Spam folder.
Spam,
Inbox,
Mvbox,
@@ -149,8 +151,11 @@ impl FolderMeaning {
#[derive(Debug)]
struct ImapConfig {
/// Email address.
pub addr: String,
pub lp: ServerLoginParam,
/// SOCKS 5 configuration.
pub socks5_config: Option<Socks5Config>,
pub strict_tls: bool,
}
@@ -899,6 +904,24 @@ impl Imap {
info!(context, "Done fetching existing messages.");
Ok(())
}
/// Synchronizes UIDs for all folders.
pub(crate) async fn resync_folders(&mut self, context: &Context) -> Result<()> {
self.prepare(context).await?;
let all_folders = self
.list_folders(context)
.await
.context("listing folders for resync")?;
for folder in all_folders {
let folder_meaning = get_folder_meaning(&folder);
if folder_meaning != FolderMeaning::Virtual {
self.resync_folder_uids(context, folder.name(), folder_meaning)
.await?;
}
}
Ok(())
}
}
impl Session {

16
src/imap/client.rs
View File

@@ -11,9 +11,9 @@ use tokio::io::BufWriter;
use super::capabilities::Capabilities;
use super::session::Session;
use crate::context::Context;
use crate::login_param::build_tls;
use crate::net::connect_tcp;
use crate::net::session::SessionStream;
use crate::net::tls::wrap_tls;
use crate::socks::Socks5Config;
/// IMAP write and read timeout.
@@ -95,8 +95,7 @@ impl Client {
strict_tls: bool,
) -> Result<Self> {
let tcp_stream = connect_tcp(context, hostname, port, IMAP_TIMEOUT, strict_tls).await?;
let tls = build_tls(strict_tls);
let tls_stream = tls.connect(hostname, tcp_stream).await?;
let tls_stream = wrap_tls(strict_tls, hostname, tcp_stream).await?;
let buffered_stream = BufWriter::new(tls_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let mut client = ImapClient::new(session_stream);
@@ -142,9 +141,7 @@ impl Client {
.context("STARTTLS command failed")?;
let tcp_stream = client.into_inner();
let tls = build_tls(strict_tls);
let tls_stream = tls
.connect(hostname, tcp_stream)
let tls_stream = wrap_tls(strict_tls, hostname, tcp_stream)
.await
.context("STARTTLS upgrade failed")?;
@@ -165,8 +162,7 @@ impl Client {
let socks5_stream = socks5_config
.connect(context, domain, port, IMAP_TIMEOUT, strict_tls)
.await?;
let tls = build_tls(strict_tls);
let tls_stream = tls.connect(domain, socks5_stream).await?;
let tls_stream = wrap_tls(strict_tls, domain, socks5_stream).await?;
let buffered_stream = BufWriter::new(tls_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let mut client = ImapClient::new(session_stream);
@@ -221,9 +217,7 @@ impl Client {
.context("STARTTLS command failed")?;
let socks5_stream = client.into_inner();
let tls = build_tls(strict_tls);
let tls_stream = tls
.connect(hostname, socks5_stream)
let tls_stream = wrap_tls(strict_tls, hostname, socks5_stream)
.await
.context("STARTTLS upgrade failed")?;
let buffered_stream = BufWriter::new(tls_stream);

2
src/imap/select_folder.rs
View File

@@ -1,3 +1,5 @@
//! # IMAP folder selection module.
use anyhow::Context as _;
use super::session::Session as ImapSession;

2
src/imex.rs
View File

@@ -760,7 +760,7 @@ async fn export_database(context: &Context, dest: &Path, passphrase: String) ->
sql::housekeeping(context).await.ok_or_log(context);
context
.sql
.call(|conn| {
.call_write(|conn| {
conn.execute("VACUUM;", params![])
.map_err(|err| warn!(context, "Vacuum failed, exporting anyway {err}"))
.ok();

150
src/job.rs
View File

@@ -6,14 +6,14 @@
#![allow(missing_docs)]
use std::fmt;
use std::sync::atomic::Ordering;
use anyhow::{Context as _, Result};
use deltachat_derive::{FromSql, ToSql};
use rand::{thread_rng, Rng};
use crate::context::Context;
use crate::imap::{get_folder_meaning, FolderMeaning, Imap};
use crate::param::Params;
use crate::imap::Imap;
use crate::scheduler::InterruptInfo;
use crate::tools::time;
@@ -25,7 +25,6 @@ const JOB_RETRIES: u32 = 17;
pub enum Status {
Finished(Result<()>),
RetryNow,
RetryLater,
}
#[macro_export]
@@ -58,18 +57,11 @@ macro_rules! job_try {
)]
#[repr(u32)]
pub enum Action {
// this is user initiated so it should have a fairly high priority
UpdateRecentQuota = 140,
// This job will download partially downloaded messages completely
// and is added when download_full() is called.
// Most messages are downloaded automatically on fetch
// and do not go through this job.
DownloadMsg = 250,
// UID synchronization is high-priority to make sure correct UIDs
// are used by message moving/deletion.
ResyncFolders = 300,
}
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -80,7 +72,6 @@ pub struct Job {
pub desired_timestamp: i64,
pub added_timestamp: i64,
pub tries: u32,
pub param: Params,
}
impl fmt::Display for Job {
@@ -90,24 +81,19 @@ impl fmt::Display for Job {
}
impl Job {
pub fn new(action: Action, foreign_id: u32, param: Params, delay_seconds: i64) -> Self {
pub fn new(action: Action, foreign_id: u32) -> Self {
let timestamp = time();
Self {
job_id: 0,
action,
foreign_id,
desired_timestamp: timestamp + delay_seconds,
desired_timestamp: timestamp,
added_timestamp: timestamp,
tries: 0,
param,
}
}
pub fn delay_seconds(&self) -> i64 {
self.desired_timestamp - self.added_timestamp
}
/// Deletes the job from the database.
async fn delete(self, context: &Context) -> Result<()> {
if self.job_id != 0 {
@@ -130,23 +116,21 @@ impl Job {
context
.sql
.execute(
"UPDATE jobs SET desired_timestamp=?, tries=?, param=? WHERE id=?;",
"UPDATE jobs SET desired_timestamp=?, tries=? WHERE id=?;",
paramsv![
self.desired_timestamp,
i64::from(self.tries),
self.param.to_string(),
self.job_id as i32,
],
)
.await?;
} else {
context.sql.execute(
"INSERT INTO jobs (added_timestamp, action, foreign_id, param, desired_timestamp) VALUES (?,?,?,?,?);",
"INSERT INTO jobs (added_timestamp, action, foreign_id, desired_timestamp) VALUES (?,?,?,?);",
paramsv![
self.added_timestamp,
self.action,
self.foreign_id,
self.param.to_string(),
self.desired_timestamp
]
).await?;
@@ -154,63 +138,6 @@ impl Job {
Ok(())
}
/// Synchronizes UIDs for all folders.
async fn resync_folders(&mut self, context: &Context, imap: &mut Imap) -> Status {
if let Err(err) = imap.prepare(context).await {
warn!(context, "could not connect: {:#}", err);
return Status::RetryLater;
}
let all_folders = match imap.list_folders(context).await {
Ok(v) => v,
Err(e) => {
warn!(context, "Listing folders for resync failed: {:#}", e);
return Status::RetryLater;
}
};
let mut any_failed = false;
for folder in all_folders {
let folder_meaning = get_folder_meaning(&folder);
if folder_meaning == FolderMeaning::Virtual {
continue;
}
if let Err(e) = imap
.resync_folder_uids(context, folder.name(), folder_meaning)
.await
{
warn!(context, "{:#}", e);
any_failed = true;
}
}
if any_failed {
Status::RetryLater
} else {
Status::Finished(Ok(()))
}
}
}
/// Delete all pending jobs with the given action.
pub async fn kill_action(context: &Context, action: Action) -> Result<()> {
context
.sql
.execute("DELETE FROM jobs WHERE action=?;", paramsv![action])
.await?;
Ok(())
}
pub async fn action_exists(context: &Context, action: Action) -> Result<bool> {
let exists = context
.sql
.exists(
"SELECT COUNT(*) FROM jobs WHERE action=?;",
paramsv![action],
)
.await?;
Ok(exists)
}
pub(crate) enum Connection<'a> {
@@ -234,13 +161,13 @@ pub(crate) async fn perform_job(context: &Context, mut connection: Connection<'_
};
match try_res {
Status::RetryNow | Status::RetryLater => {
Status::RetryNow => {
let tries = job.tries + 1;
if tries < JOB_RETRIES {
info!(context, "increase job {} tries to {}", job, tries);
job.tries = tries;
let time_offset = get_backoff_time_offset(tries, job.action);
let time_offset = get_backoff_time_offset(tries);
job.desired_timestamp = time() + time_offset;
info!(
context,
@@ -288,11 +215,6 @@ async fn perform_job_action(
info!(context, "begin immediate try {} of job {}", tries, job);
let try_res = match job.action {
Action::ResyncFolders => job.resync_folders(context, connection.inbox()).await,
Action::UpdateRecentQuota => match context.update_recent_quota(connection.inbox()).await {
Ok(status) => status,
Err(err) => Status::Finished(Err(err)),
},
Action::DownloadMsg => job.download_msg(context, connection.inbox()).await,
};
@@ -301,50 +223,34 @@ async fn perform_job_action(
try_res
}
fn get_backoff_time_offset(tries: u32, action: Action) -> i64 {
match action {
// Just try every 10s to update the quota
// If all retries are exhausted, a new job will be created when the quota information is needed
Action::UpdateRecentQuota => 10,
_ => {
// Exponential backoff
let n = 2_i32.pow(tries - 1) * 60;
let mut rng = thread_rng();
let r: i32 = rng.gen();
let mut seconds = r % (n + 1);
if seconds < 1 {
seconds = 1;
}
i64::from(seconds)
}
fn get_backoff_time_offset(tries: u32) -> i64 {
// Exponential backoff
let n = 2_i32.pow(tries - 1) * 60;
let mut rng = thread_rng();
let r: i32 = rng.gen();
let mut seconds = r % (n + 1);
if seconds < 1 {
seconds = 1;
}
i64::from(seconds)
}
pub(crate) async fn schedule_resync(context: &Context) -> Result<()> {
kill_action(context, Action::ResyncFolders).await?;
add(
context,
Job::new(Action::ResyncFolders, 0, Params::new(), 0),
)
.await?;
context.resync_request.store(true, Ordering::Relaxed);
context
.interrupt_inbox(InterruptInfo {
probe_network: false,
})
.await;
Ok(())
}
/// Adds a job to the database, scheduling it.
pub async fn add(context: &Context, job: Job) -> Result<()> {
let action = job.action;
let delay_seconds = job.delay_seconds();
job.save(context).await.context("failed to save job")?;
if delay_seconds == 0 {
match action {
Action::ResyncFolders | Action::UpdateRecentQuota | Action::DownloadMsg => {
info!(context, "interrupt: imap");
context.interrupt_inbox(InterruptInfo::new(false)).await;
}
}
}
info!(context, "interrupt: imap");
context.interrupt_inbox(InterruptInfo::new(false)).await;
Ok(())
}
@@ -396,7 +302,6 @@ LIMIT 1;
desired_timestamp: row.get("desired_timestamp")?,
added_timestamp: row.get("added_timestamp")?,
tries: row.get("tries")?,
param: row.get::<_, String>("param")?.parse().unwrap_or_default(),
};
Ok(job)
@@ -436,8 +341,8 @@ mod tests {
.sql
.execute(
"INSERT INTO jobs
(added_timestamp, action, foreign_id, param, desired_timestamp)
VALUES (?, ?, ?, ?, ?);",
(added_timestamp, action, foreign_id, desired_timestamp)
VALUES (?, ?, ?, ?);",
paramsv![
now,
if valid {
@@ -446,7 +351,6 @@ mod tests {
-1
},
foreign_id,
Params::new().to_string(),
now
],
)

1
src/lib.rs
View File

@@ -92,6 +92,7 @@ mod smtp;
mod socks;
pub mod stock_str;
mod sync;
mod timesmearing;
mod token;
mod update_helper;
pub mod webxdc;

2
src/location.rs
View File

@@ -603,7 +603,7 @@ pub(crate) async fn save(
context
.sql
.call(|conn| {
.call_write(|conn| {
let mut stmt_test = conn
.prepare_cached("SELECT id FROM locations WHERE timestamp=? AND from_id=?")?;
let mut stmt_insert = conn.prepare_cached(stmt_insert)?;

33
src/login_param.rs
View File

@@ -3,8 +3,6 @@
use std::fmt;
use anyhow::{ensure, Result};
use async_native_tls::Certificate;
use once_cell::sync::Lazy;
use crate::constants::{DC_LP_AUTH_FLAGS, DC_LP_AUTH_NORMAL, DC_LP_AUTH_OAUTH2};
use crate::provider::{get_provider_by_id, Provider};
@@ -306,28 +304,6 @@ fn unset_empty(s: &str) -> &str {
}
}
// this certificate is missing on older android devices (eg. lg with android6 from 2017)
// certificate downloaded from https://letsencrypt.org/certificates/
static LETSENCRYPT_ROOT: Lazy<Certificate> = Lazy::new(|| {
Certificate::from_der(include_bytes!(
"../assets/root-certificates/letsencrypt/isrgrootx1.der"
))
.unwrap()
});
pub fn build_tls(strict_tls: bool) -> async_native_tls::TlsConnector {
let tls_builder =
async_native_tls::TlsConnector::new().add_root_certificate(LETSENCRYPT_ROOT.clone());
if strict_tls {
tls_builder
} else {
tls_builder
.danger_accept_invalid_hostnames(true)
.danger_accept_invalid_certs(true)
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -378,13 +354,4 @@ mod tests {
assert_eq!(param, loaded);
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_build_tls() -> Result<()> {
// we are using some additional root certificates.
// make sure, they do not break construction of TlsConnector
let _ = build_tls(true);
let _ = build_tls(false);
Ok(())
}
}

9
src/message.rs
View File

@@ -1771,13 +1771,8 @@ async fn ndn_maybe_add_info_msg(
// Tell the user which of the recipients failed if we know that (because in
// a group, this might otherwise be unclear)
let text = stock_str::failed_sending_to(context, contact.get_display_name()).await;
chat::add_info_msg(
context,
chat_id,
&text,
create_smeared_timestamp(context).await,
)
.await?;
chat::add_info_msg(context, chat_id, &text, create_smeared_timestamp(context))
.await?;
context.emit_event(EventType::ChatModified(chat_id));
}
}

122
src/mimefactory.rs
View File

@@ -250,7 +250,7 @@ impl<'a> MimeFactory<'a> {
.get_config(Config::Selfstatus)
.await?
.unwrap_or_default();
let timestamp = create_smeared_timestamp(context).await;
let timestamp = create_smeared_timestamp(context);
let res = MimeFactory::<'a> {
from_addr,
@@ -779,10 +779,36 @@ impl<'a> MimeFactory<'a> {
};
// Store protected headers in the outer message.
headers
let message = headers
.protected
.into_iter()
.fold(message, |message, header| message.header(header))
.fold(message, |message, header| message.header(header));
if self.should_skip_autocrypt()
|| !context.get_config_bool(Config::SignUnencrypted).await?
{
message
} else {
let (payload, signature) = encrypt_helper.sign(context, message).await?;
PartBuilder::new()
.header((
"Content-Type".to_string(),
"multipart/signed; protocol=\"application/pgp-signature\"".to_string(),
))
.child(payload)
.child(
PartBuilder::new()
.content_type(
&"application/pgp-signature; name=\"signature.asc\""
.parse::<mime::Mime>()
.unwrap(),
)
.header(("Content-Description", "OpenPGP digital signature"))
.header(("Content-Disposition", "attachment; filename=\"signature\";"))
.body(signature)
.build(),
)
}
};
// Store the unprotected headers on the outer message.
@@ -2140,6 +2166,96 @@ mod tests {
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_selfavatar_unencrypted_signed() {
// create chat with bob, set selfavatar
let t = TestContext::new_alice().await;
t.set_config(Config::SignUnencrypted, Some("1"))
.await
.unwrap();
let chat = t.create_chat_with_contact("bob", "bob@example.org").await;
let file = t.dir.path().join("avatar.png");
let bytes = include_bytes!("../test-data/image/avatar64x64.png");
tokio::fs::write(&file, bytes).await.unwrap();
t.set_config(Config::Selfavatar, Some(file.to_str().unwrap()))
.await
.unwrap();
// send message to bob: that should get multipart/mixed because of the avatar moved to inner header;
// make sure, `Subject:` stays in the outer header (imf header)
let mut msg = Message::new(Viewtype::Text);
msg.set_text(Some("this is the text!".to_string()));
let sent_msg = t.send_msg(chat.id, &mut msg).await;
let mut payload = sent_msg.payload().splitn(4, "\r\n\r\n");
let part = payload.next().unwrap();
assert_eq!(part.match_indices("multipart/signed").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 0);
assert_eq!(part.match_indices("Autocrypt:").count(), 1);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 0);
let part = payload.next().unwrap();
assert_eq!(part.match_indices("multipart/mixed").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 1);
assert_eq!(part.match_indices("Autocrypt:").count(), 0);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 0);
let part = payload.next().unwrap();
assert_eq!(part.match_indices("text/plain").count(), 1);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 0);
let body = payload.next().unwrap();
assert_eq!(body.match_indices("this is the text!").count(), 1);
let bob = TestContext::new_bob().await;
bob.recv_msg(&sent_msg).await;
let alice_id = Contact::lookup_id_by_addr(&bob.ctx, "alice@example.org", Origin::Unknown)
.await
.unwrap()
.unwrap();
let alice_contact = Contact::load_from_db(&bob.ctx, alice_id).await.unwrap();
assert!(alice_contact
.get_profile_image(&bob.ctx)
.await
.unwrap()
.is_some());
// if another message is sent, that one must not contain the avatar
// and no artificial multipart/mixed nesting
let sent_msg = t.send_msg(chat.id, &mut msg).await;
let mut payload = sent_msg.payload().splitn(3, "\r\n\r\n");
let part = payload.next().unwrap();
assert_eq!(part.match_indices("multipart/signed").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 0);
assert_eq!(part.match_indices("Autocrypt:").count(), 1);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 0);
let part = payload.next().unwrap();
assert_eq!(part.match_indices("text/plain").count(), 1);
assert_eq!(part.match_indices("Subject:").count(), 1);
assert_eq!(part.match_indices("Autocrypt:").count(), 0);
assert_eq!(part.match_indices("multipart/mixed").count(), 0);
assert_eq!(part.match_indices("Chat-User-Avatar:").count(), 0);
let body = payload.next().unwrap();
assert_eq!(body.match_indices("this is the text!").count(), 1);
assert_eq!(body.match_indices("text/plain").count(), 0);
assert_eq!(body.match_indices("Chat-User-Avatar:").count(), 0);
assert_eq!(body.match_indices("Subject:").count(), 0);
bob.recv_msg(&sent_msg).await;
let alice_contact = Contact::load_from_db(&bob.ctx, alice_id).await.unwrap();
assert!(alice_contact
.get_profile_image(&bob.ctx)
.await
.unwrap()
.is_some());
}
/// Test that removed member address does not go into the `To:` field.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_remove_member_bcc() -> Result<()> {

65
src/mimeparser.rs
View File

@@ -224,8 +224,32 @@ impl MimeMessage {
// Parse hidden headers.
let mimetype = mail.ctype.mimetype.parse::<Mime>()?;
let (part, mimetype) =
if mimetype.type_() == mime::MULTIPART && mimetype.subtype().as_str() == "signed" {
if let Some(part) = mail.subparts.first() {
// We don't remove "subject" from `headers` because currently just signed
// messages are shown as unencrypted anyway.
MimeMessage::merge_headers(
context,
&mut headers,
&mut recipients,
&mut from,
&mut list_post,
&mut chat_disposition_notification_to,
&part.headers,
);
(part, part.ctype.mimetype.parse::<Mime>()?)
} else {
// If it's a partially fetched message, there are no subparts.
(&mail, mimetype)
}
} else {
// Currently we do not sign unencrypted messages by default.
(&mail, mimetype)
};
if mimetype.type_() == mime::MULTIPART && mimetype.subtype().as_str() == "mixed" {
if let Some(part) = mail.subparts.first() {
if let Some(part) = part.subparts.first() {
for field in &part.headers {
let key = field.get_key().to_lowercase();
@@ -256,26 +280,27 @@ impl MimeMessage {
hop_info += &decryption_info.dkim_results.to_string();
let public_keyring = keyring_from_peerstate(decryption_info.peerstate.as_ref());
let (mail, mut signatures, encrypted) =
match try_decrypt(context, &mail, &private_keyring, &public_keyring) {
Ok(Some((raw, signatures))) => {
mail_raw = raw;
let decrypted_mail = mailparse::parse_mail(&mail_raw)?;
if std::env::var(crate::DCC_MIME_DEBUG).is_ok() {
info!(
context,
"decrypted message mime-body:\n{}",
String::from_utf8_lossy(&mail_raw),
);
}
(Ok(decrypted_mail), signatures, true)
let (mail, mut signatures, encrypted) = match tokio::task::block_in_place(|| {
try_decrypt(context, &mail, &private_keyring, &public_keyring)
}) {
Ok(Some((raw, signatures))) => {
mail_raw = raw;
let decrypted_mail = mailparse::parse_mail(&mail_raw)?;
if std::env::var(crate::DCC_MIME_DEBUG).is_ok() {
info!(
context,
"decrypted message mime-body:\n{}",
String::from_utf8_lossy(&mail_raw),
);
}
Ok(None) => (Ok(mail), HashSet::new(), false),
Err(err) => {
warn!(context, "decryption failed: {:#}", err);
(Err(err), HashSet::new(), false)
}
};
(Ok(decrypted_mail), signatures, true)
}
Ok(None) => (Ok(mail), HashSet::new(), false),
Err(err) => {
warn!(context, "decryption failed: {:#}", err);
(Err(err), HashSet::new(), false)
}
};
let mail = mail.as_ref().map(|mail| {
let (content, signatures_detached) = validate_detached_signature(mail, &public_keyring)
.unwrap_or((mail, Default::default()));

1
src/net.rs
View File

@@ -13,6 +13,7 @@ use crate::context::Context;
use crate::tools::time;
pub(crate) mod session;
pub(crate) mod tls;
async fn connect_tcp_inner(addr: SocketAddr, timeout_val: Duration) -> Result<TcpStream> {
let tcp_stream = timeout(timeout_val, TcpStream::connect(addr))

52
src/net/tls.rs Normal file
View File

@@ -0,0 +1,52 @@
//! TLS support.
use anyhow::Result;
use async_native_tls::{Certificate, Protocol, TlsConnector, TlsStream};
use once_cell::sync::Lazy;
use tokio::io::{AsyncRead, AsyncWrite};
// this certificate is missing on older android devices (eg. lg with android6 from 2017)
// certificate downloaded from https://letsencrypt.org/certificates/
static LETSENCRYPT_ROOT: Lazy<Certificate> = Lazy::new(|| {
Certificate::from_der(include_bytes!(
"../../assets/root-certificates/letsencrypt/isrgrootx1.der"
))
.unwrap()
});
pub fn build_tls(strict_tls: bool) -> TlsConnector {
let tls_builder = TlsConnector::new()
.min_protocol_version(Some(Protocol::Tlsv12))
.add_root_certificate(LETSENCRYPT_ROOT.clone());
if strict_tls {
tls_builder
} else {
tls_builder
.danger_accept_invalid_hostnames(true)
.danger_accept_invalid_certs(true)
}
}
pub async fn wrap_tls<T: AsyncRead + AsyncWrite + Unpin>(
strict_tls: bool,
hostname: &str,
stream: T,
) -> Result<TlsStream<T>> {
let tls = build_tls(strict_tls);
let tls_stream = tls.connect(hostname, stream).await?;
Ok(tls_stream)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_build_tls() {
// we are using some additional root certificates.
// make sure, they do not break construction of TlsConnector
let _ = build_tls(true);
let _ = build_tls(false);
}
}

14
src/pgp.rs
View File

@@ -262,6 +262,20 @@ pub async fn pk_encrypt(
.await?
}
/// Signs `plain` text using `private_key_for_signing`.
pub fn pk_calc_signature(
plain: &[u8],
private_key_for_signing: &SignedSecretKey,
) -> Result<String> {
let msg = Message::new_literal_bytes("", plain).sign(
private_key_for_signing,
|| "".into(),
Default::default(),
)?;
let signature = msg.into_signature().to_armored_string(None)?;
Ok(signature)
}
/// Decrypts the message with keys from the private key keyring.
///
/// Receiver private keys are provided in

2
src/provider/data.rs
View File

@@ -1952,4 +1952,4 @@ pub(crate) static PROVIDER_IDS: Lazy<HashMap<&'static str, &'static Provider>> =
});
pub static PROVIDER_UPDATED: Lazy<chrono::NaiveDate> =
Lazy::new(|| chrono::NaiveDate::from_ymd_opt(2023, 2, 21).unwrap());
Lazy::new(|| chrono::NaiveDate::from_ymd_opt(2023, 2, 20).unwrap());

4
src/qr.rs
View File

@@ -515,11 +515,15 @@ fn decode_backup(qr: &str) -> Result<Qr> {
#[derive(Debug, Deserialize)]
struct CreateAccountSuccessResponse {
/// Email address.
email: String,
/// Password.
password: String,
}
#[derive(Debug, Deserialize)]
struct CreateAccountErrorResponse {
/// Reason for the failure to create account returned by the server.
reason: String,
}

25
src/quota.rs
View File

@@ -1,6 +1,7 @@
//! # Support for IMAP QUOTA extension.
use std::collections::BTreeMap;
use std::sync::atomic::Ordering;
use anyhow::{anyhow, Context as _, Result};
use async_imap::types::{Quota, QuotaResource};
@@ -11,11 +12,10 @@ use crate::context::Context;
use crate::imap::scan_folders::get_watched_folders;
use crate::imap::session::Session as ImapSession;
use crate::imap::Imap;
use crate::job::{Action, Status};
use crate::message::{Message, Viewtype};
use crate::param::Params;
use crate::scheduler::InterruptInfo;
use crate::tools::time;
use crate::{job, stock_str, EventType};
use crate::{stock_str, EventType};
/// warn about a nearly full mailbox after this usage percentage is reached.
/// quota icon is "yellow".
@@ -112,12 +112,10 @@ pub fn needs_quota_warning(curr_percentage: u64, warned_at_percentage: u64) -> b
impl Context {
// Adds a job to update `quota.recent`
pub(crate) async fn schedule_quota_update(&self) -> Result<()> {
if !job::action_exists(self, Action::UpdateRecentQuota).await? {
job::add(
self,
job::Job::new(Action::UpdateRecentQuota, 0, Params::new(), 0),
)
.await?;
let requested = self.quota_update_request.swap(true, Ordering::Relaxed);
if !requested {
// Quota update was not requested before.
self.interrupt_inbox(InterruptInfo::new(false)).await;
}
Ok(())
}
@@ -132,10 +130,10 @@ impl Context {
/// and new space is allocated as needed.
///
/// Called in response to `Action::UpdateRecentQuota`.
pub(crate) async fn update_recent_quota(&self, imap: &mut Imap) -> Result<Status> {
pub(crate) async fn update_recent_quota(&self, imap: &mut Imap) -> Result<()> {
if let Err(err) = imap.prepare(self).await {
warn!(self, "could not connect: {:#}", err);
return Ok(Status::RetryNow);
return Ok(());
}
let session = imap.session.as_mut().context("no session")?;
@@ -166,13 +164,16 @@ impl Context {
}
}
// Clear the request to update quota.
self.quota_update_request.store(false, Ordering::Relaxed);
*self.quota.write().await = Some(QuotaInfo {
recent: quota,
modified: time(),
});
self.emit_event(EventType::ConnectivityChanged);
Ok(Status::Finished(Ok(())))
Ok(())
}
}

95
src/receive_imf.rs
View File

@@ -203,7 +203,7 @@ pub(crate) async fn receive_imf_inner(
)
.await?;
let rcvd_timestamp = smeared_time(context).await;
let rcvd_timestamp = smeared_time(context);
// Sender timestamp is allowed to be a bit in the future due to
// unsynchronized clocks, but not too much.
@@ -1149,7 +1149,10 @@ async fn add_parts(
// also change `MsgId::trash()` and `delete_expired_messages()`
let trash = chat_id.is_trash() || (is_location_kml && msg.is_empty());
let row_id = context.sql.insert(
let row_id = context
.sql
.call_write(|conn| {
let mut stmt = conn.prepare_cached(
r#"
INSERT INTO msgs
(
@@ -1179,47 +1182,51 @@ SET rfc724_mid=excluded.rfc724_mid, chat_id=excluded.chat_id,
bytes=excluded.bytes, mime_headers=excluded.mime_headers, mime_in_reply_to=excluded.mime_in_reply_to,
mime_references=excluded.mime_references, mime_modified=excluded.mime_modified, error=excluded.error, ephemeral_timer=excluded.ephemeral_timer,
ephemeral_timestamp=excluded.ephemeral_timestamp, download_state=excluded.download_state, hop_info=excluded.hop_info
"#,
paramsv![
replace_msg_id,
rfc724_mid,
if trash { DC_CHAT_ID_TRASH } else { chat_id },
if trash { ContactId::UNDEFINED } else { from_id },
if trash { ContactId::UNDEFINED } else { to_id },
sort_timestamp,
sent_timestamp,
rcvd_timestamp,
typ,
state,
is_dc_message,
if trash { "" } else { msg },
if trash { "" } else { &subject },
// txt_raw might contain invalid utf8
if trash { "" } else { &txt_raw },
if trash {
"".to_string()
} else {
param.to_string()
},
part.bytes as isize,
if (save_mime_headers || mime_modified) && !trash {
mime_headers.clone()
} else {
Vec::new()
},
mime_in_reply_to,
mime_references,
mime_modified,
part.error.as_deref().unwrap_or_default(),
ephemeral_timer,
ephemeral_timestamp,
if is_partial_download.is_some() {
DownloadState::Available
} else {
DownloadState::Done
},
mime_parser.hop_info
]).await?;
"#)?;
stmt.execute(params![
replace_msg_id,
rfc724_mid,
if trash { DC_CHAT_ID_TRASH } else { chat_id },
if trash { ContactId::UNDEFINED } else { from_id },
if trash { ContactId::UNDEFINED } else { to_id },
sort_timestamp,
sent_timestamp,
rcvd_timestamp,
typ,
state,
is_dc_message,
if trash { "" } else { msg },
if trash { "" } else { &subject },
// txt_raw might contain invalid utf8
if trash { "" } else { &txt_raw },
if trash {
"".to_string()
} else {
param.to_string()
},
part.bytes as isize,
if (save_mime_headers || mime_modified) && !trash {
mime_headers.clone()
} else {
Vec::new()
},
mime_in_reply_to,
mime_references,
mime_modified,
part.error.as_deref().unwrap_or_default(),
ephemeral_timer,
ephemeral_timestamp,
if is_partial_download.is_some() {
DownloadState::Available
} else {
DownloadState::Done
},
mime_parser.hop_info
])?;
let row_id = conn.last_insert_rowid();
Ok(row_id)
})
.await?;
// We only replace placeholder with a first part,
// afterwards insert additional parts.
@@ -1373,7 +1380,7 @@ async fn calc_sort_timestamp(
}
}
Ok(min(sort_timestamp, smeared_time(context).await))
Ok(min(sort_timestamp, smeared_time(context)))
}
async fn lookup_chat_by_reply(

70
src/receive_imf/tests.rs
View File

@@ -2133,6 +2133,76 @@ Original signature updated",
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_ignore_old_status_updates() -> Result<()> {
let t = TestContext::new_alice().await;
let bob_id = Contact::add_or_lookup(
&t,
"",
ContactAddress::new("bob@example.net")?,
Origin::AddressBook,
)
.await?
.0;
receive_imf(
&t,
b"From: Bob <bob@example.net>
To: Alice <alice@example.org>
Message-ID: <2@example.org>
Date: Wed, 22 Feb 2023 20:00:00 +0000
body
--
sig wednesday",
false,
)
.await?;
let chat_id = t.get_last_msg().await.chat_id;
let bob = Contact::load_from_db(&t, bob_id).await?;
assert_eq!(bob.get_status(), "sig wednesday");
assert_eq!(get_chat_msgs(&t, chat_id).await?.len(), 1);
receive_imf(
&t,
b"From: Bob <bob@example.net>
To: Alice <alice@example.org>
Message-ID: <1@example.org>
Date: Tue, 21 Feb 2023 20:00:00 +0000
body
--
sig tuesday",
false,
)
.await?;
let bob = Contact::load_from_db(&t, bob_id).await?;
assert_eq!(bob.get_status(), "sig wednesday");
assert_eq!(get_chat_msgs(&t, chat_id).await?.len(), 2);
receive_imf(
&t,
b"From: Bob <bob@example.net>
To: Alice <alice@example.org>
Message-ID: <3@example.org>
Date: Thu, 23 Feb 2023 20:00:00 +0000
body
--
sig thursday",
false,
)
.await?;
let bob = Contact::load_from_db(&t, bob_id).await?;
assert_eq!(bob.get_status(), "sig thursday");
assert_eq!(get_chat_msgs(&t, chat_id).await?.len(), 3);
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_chat_assignment_private_classical_reply() {
for outgoing_is_classical in &[true, false] {

16
src/scheduler.rs
View File

@@ -1,4 +1,5 @@
use std::iter::{self, once};
use std::sync::atomic::Ordering;
use anyhow::{bail, Context as _, Result};
use async_channel::{self as channel, Receiver, Sender};
@@ -128,6 +129,21 @@ async fn inbox_loop(ctx: Context, started: Sender<()>, inbox_handlers: ImapConne
info = Default::default();
}
None => {
let quota_requested = ctx.quota_update_request.swap(false, Ordering::Relaxed);
if quota_requested {
if let Err(err) = ctx.update_recent_quota(&mut connection).await {
warn!(ctx, "Failed to update quota: {:#}.", err);
}
}
let resync_requested = ctx.resync_request.swap(false, Ordering::Relaxed);
if resync_requested {
if let Err(err) = connection.resync_folders(&ctx).await {
warn!(ctx, "Failed to resync folders: {:#}.", err);
ctx.resync_request.store(true, Ordering::Relaxed);
}
}
maybe_add_time_based_warnings(&ctx).await;
match ctx.get_config_i64(Config::LastHousekeeping).await {

17
src/smtp.rs
View File

@@ -13,12 +13,13 @@ use tokio::task;
use crate::config::Config;
use crate::contact::{Contact, ContactId};
use crate::events::EventType;
use crate::login_param::{build_tls, CertificateChecks, LoginParam, ServerLoginParam};
use crate::login_param::{CertificateChecks, LoginParam, ServerLoginParam};
use crate::message::Message;
use crate::message::{self, MsgId};
use crate::mimefactory::MimeFactory;
use crate::net::connect_tcp;
use crate::net::session::SessionStream;
use crate::net::tls::wrap_tls;
use crate::oauth2::get_oauth2_access_token;
use crate::provider::Socket;
use crate::socks::Socks5Config;
@@ -119,8 +120,7 @@ impl Smtp {
let socks5_stream = socks5_config
.connect(context, hostname, port, SMTP_TIMEOUT, strict_tls)
.await?;
let tls = build_tls(strict_tls);
let tls_stream = tls.connect(hostname, socks5_stream).await?;
let tls_stream = wrap_tls(strict_tls, hostname, socks5_stream).await?;
let buffered_stream = BufWriter::new(tls_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let client = smtp::SmtpClient::new().smtp_utf8(true);
@@ -144,9 +144,7 @@ impl Smtp {
let client = smtp::SmtpClient::new().smtp_utf8(true);
let transport = SmtpTransport::new(client, socks5_stream).await?;
let tcp_stream = transport.starttls().await?;
let tls = build_tls(strict_tls);
let tls_stream = tls
.connect(hostname, tcp_stream)
let tls_stream = wrap_tls(strict_tls, hostname, tcp_stream)
.await
.context("STARTTLS upgrade failed")?;
let buffered_stream = BufWriter::new(tls_stream);
@@ -181,8 +179,7 @@ impl Smtp {
strict_tls: bool,
) -> Result<SmtpTransport<Box<dyn SessionStream>>> {
let tcp_stream = connect_tcp(context, hostname, port, SMTP_TIMEOUT, false).await?;
let tls = build_tls(strict_tls);
let tls_stream = tls.connect(hostname, tcp_stream).await?;
let tls_stream = wrap_tls(strict_tls, hostname, tcp_stream).await?;
let buffered_stream = BufWriter::new(tls_stream);
let session_stream: Box<dyn SessionStream> = Box::new(buffered_stream);
let client = smtp::SmtpClient::new().smtp_utf8(true);
@@ -203,9 +200,7 @@ impl Smtp {
let client = smtp::SmtpClient::new().smtp_utf8(true);
let transport = SmtpTransport::new(client, tcp_stream).await?;
let tcp_stream = transport.starttls().await?;
let tls = build_tls(strict_tls);
let tls_stream = tls
.connect(hostname, tcp_stream)
let tls_stream = wrap_tls(strict_tls, hostname, tcp_stream)
.await
.context("STARTTLS upgrade failed")?;
let buffered_stream = BufWriter::new(tls_stream);

81
src/sql.rs
View File

@@ -5,8 +5,8 @@ use std::convert::TryFrom;
use std::path::{Path, PathBuf};
use anyhow::{bail, Context as _, Result};
use rusqlite::{self, config::DbConfig, Connection, OpenFlags, TransactionBehavior};
use tokio::sync::RwLock;
use rusqlite::{self, config::DbConfig, Connection, OpenFlags};
use tokio::sync::{Mutex, MutexGuard, RwLock};
use crate::blob::BlobObject;
use crate::chat::{add_device_msg, update_device_icon, update_saved_messages_icon};
@@ -56,6 +56,11 @@ pub struct Sql {
/// Database file path
pub(crate) dbfile: PathBuf,
/// Write transaction mutex.
///
/// See [`Self::write_lock`].
write_mtx: Mutex<()>,
/// SQL connection pool.
pool: RwLock<Option<Pool>>,
@@ -72,6 +77,7 @@ impl Sql {
pub fn new(dbfile: PathBuf) -> Sql {
Self {
dbfile,
write_mtx: Mutex::new(()),
pool: Default::default(),
is_encrypted: Default::default(),
config_cache: Default::default(),
@@ -130,7 +136,7 @@ impl Sql {
.with_context(|| format!("path {path:?} is not valid unicode"))?
.to_string();
let res = self
.call(move |conn| {
.call_write(move |conn| {
// Check that backup passphrase is correct before resetting our database.
conn.execute(
"ATTACH DATABASE ? AS backup KEY ?",
@@ -299,10 +305,40 @@ impl Sql {
}
}
/// Allocates a connection and calls given function with the connection.
/// Locks the write transactions mutex.
/// We do not make all transactions
/// [IMMEDIATE](https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions)
/// for more parallelism -- at least read transactions can be made DEFERRED to run in parallel
/// w/o any drawbacks. But if we make write transactions DEFERRED also w/o any external locking,
/// then they are upgraded from read to write ones on the first write statement. This has some
/// drawbacks:
/// - If there are other write transactions, we block the thread and the db connection until
/// upgraded. Also if some reader comes then, it has to get next, less used connection with a
/// worse per-connection page cache.
/// - If a transaction is blocked for more than busy_timeout, it fails with SQLITE_BUSY.
/// - Configuring busy_timeout is not the best way to manage transaction timeouts, we would
/// prefer it to be integrated with Rust/tokio asyncs. Moreover, SQLite implements waiting
/// using sleeps.
/// - If upon a successful upgrade to a write transaction the db has been modified by another
/// one, the transaction has to be rolled back and retried. It is an extra work in terms of
/// CPU/battery.
/// - Maybe minor, but we lose some fairness in servicing write transactions, i.e. we service
/// them in the order of the first write statement, not in the order they come.
/// The only pro of making write transactions DEFERRED w/o the external locking is some
/// parallelism between them. Also we have an option to make write transactions IMMEDIATE, also
/// w/o the external locking. But then the most of cons above are still valid. Instead, if we
/// perform all write transactions under an async mutex, the only cons is losing some
/// parallelism for write transactions.
pub async fn write_lock(&self) -> MutexGuard<'_, ()> {
self.write_mtx.lock().await
}
/// Allocates a connection and calls `function` with the connection. If `function` does write
/// queries, either a lock must be taken first using `write_lock()` or `call_write()` used
/// instead.
///
/// Returns the result of the function.
pub async fn call<'a, F, R>(&'a self, function: F) -> Result<R>
async fn call<'a, F, R>(&'a self, function: F) -> Result<R>
where
F: 'a + FnOnce(&mut Connection) -> Result<R> + Send,
R: Send + 'static,
@@ -314,13 +350,26 @@ impl Sql {
Ok(res)
}
/// Execute the given query, returning the number of affected rows.
/// Allocates a connection and calls given function, assuming it does write queries, with the
/// connection.
///
/// Returns the result of the function.
pub async fn call_write<'a, F, R>(&'a self, function: F) -> Result<R>
where
F: 'a + FnOnce(&mut Connection) -> Result<R> + Send,
R: Send + 'static,
{
let _lock = self.write_lock().await;
self.call(function).await
}
/// Execute `query` assuming it is a write query, returning the number of affected rows.
pub async fn execute(
&self,
query: &str,
params: impl rusqlite::Params + Send,
) -> Result<usize> {
self.call(move |conn| {
self.call_write(move |conn| {
let res = conn.execute(query, params)?;
Ok(res)
})
@@ -329,7 +378,7 @@ impl Sql {
/// Executes the given query, returning the last inserted row ID.
pub async fn insert(&self, query: &str, params: impl rusqlite::Params + Send) -> Result<i64> {
self.call(move |conn| {
self.call_write(move |conn| {
conn.execute(query, params)?;
Ok(conn.last_insert_rowid())
})
@@ -390,23 +439,17 @@ impl Sql {
.await
}
/// Execute the function inside a transaction.
/// Execute the function inside a transaction assuming that it does write queries.
///
/// If the function returns an error, the transaction will be rolled back. If it does not return an
/// error, the transaction will be committed.
///
/// Transactions started use IMMEDIATE behavior
/// rather than default DEFERRED behavior
/// to avoid "database is busy" errors
/// which may happen when DEFERRED transaction
/// is attempted to be promoted to a write transaction.
pub async fn transaction<G, H>(&self, callback: G) -> Result<H>
where
H: Send + 'static,
G: Send + FnOnce(&mut rusqlite::Transaction<'_>) -> Result<H>,
{
self.call(move |conn| {
let mut transaction = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
self.call_write(move |conn| {
let mut transaction = conn.transaction()?;
let ret = callback(&mut transaction);
match ret {
@@ -617,7 +660,7 @@ fn new_connection(path: &Path, passphrase: &str) -> Result<Connection> {
conn.execute_batch(
"PRAGMA cipher_memory_security = OFF; -- Too slow on Android
PRAGMA secure_delete=on;
PRAGMA busy_timeout = 60000; -- 60 seconds
PRAGMA busy_timeout = 0; -- fail immediately
PRAGMA temp_store=memory; -- Avoid SQLITE_IOERR_GETTEMPPATH errors on Android
PRAGMA foreign_keys=on;
",
@@ -983,7 +1026,7 @@ mod tests {
assert_eq!(avatar_bytes, &tokio::fs::read(&a).await.unwrap()[..]);
t.sql.close().await;
housekeeping(&t).await.unwrap_err(); // housekeeping should fail as the db is closed
housekeeping(&t).await.unwrap(); // housekeeping should emit warnings but not fail
t.sql.open(&t, "".to_string()).await.unwrap();
let a = t.get_config(Config::Selfavatar).await.unwrap().unwrap();

193
src/timesmearing.rs Normal file
View File

@@ -0,0 +1,193 @@
//! # Time smearing.
//!
//! As e-mails typically only use a second-based-resolution for timestamps,
//! the order of two mails sent withing one second is unclear.
//! This is bad e.g. when forwarding some messages from a chat -
//! these messages will appear at the recipient easily out of order.
//!
//! We work around this issue by not sending out two mails with the same timestamp.
//! For this purpose, in short, we track the last timestamp used in `last_smeared_timestamp`
//! when another timestamp is needed in the same second, we use `last_smeared_timestamp+1`
//! after some moments without messages sent out,
//! `last_smeared_timestamp` is again in sync with the normal time.
//!
//! However, we do not do all this for the far future,
//! but at max `MAX_SECONDS_TO_LEND_FROM_FUTURE`
use std::cmp::{max, min};
use std::sync::atomic::{AtomicI64, Ordering};
pub(crate) const MAX_SECONDS_TO_LEND_FROM_FUTURE: i64 = 5;
/// Smeared timestamp generator.
#[derive(Debug)]
pub struct SmearedTimestamp {
/// Next timestamp available for allocation.
smeared_timestamp: AtomicI64,
}
impl SmearedTimestamp {
/// Creates a new smeared timestamp generator.
pub fn new() -> Self {
Self {
smeared_timestamp: AtomicI64::new(0),
}
}
/// Allocates `count` unique timestamps.
///
/// Returns the first allocated timestamp.
pub fn create_n(&self, now: i64, count: i64) -> i64 {
let mut prev = self.smeared_timestamp.load(Ordering::Relaxed);
loop {
// Advance the timestamp if it is in the past,
// but keep `count - 1` timestamps from the past if possible.
let t = max(prev, now - count + 1);
// Rewind the time back if there is no room
// to allocate `count` timestamps without going too far into the future.
// Not going too far into the future
// is more important than generating unique timestamps.
let first = min(t, now + MAX_SECONDS_TO_LEND_FROM_FUTURE - count + 1);
// Allocate `count` timestamps by advancing the current timestamp.
let next = first + count;
if let Err(x) = self.smeared_timestamp.compare_exchange_weak(
prev,
next,
Ordering::Relaxed,
Ordering::Relaxed,
) {
prev = x;
} else {
return first;
}
}
}
/// Creates a single timestamp.
pub fn create(&self, now: i64) -> i64 {
self.create_n(now, 1)
}
/// Returns the current smeared timestamp.
pub fn current(&self) -> i64 {
self.smeared_timestamp.load(Ordering::Relaxed)
}
}
#[cfg(test)]
mod tests {
use std::time::SystemTime;
use super::*;
use crate::test_utils::TestContext;
use crate::tools::{create_smeared_timestamp, create_smeared_timestamps, smeared_time, time};
#[test]
fn test_smeared_timestamp() {
let smeared_timestamp = SmearedTimestamp::new();
let now = time();
assert_eq!(smeared_timestamp.current(), 0);
for i in 0..MAX_SECONDS_TO_LEND_FROM_FUTURE {
assert_eq!(smeared_timestamp.create(now), now + i);
}
assert_eq!(
smeared_timestamp.create(now),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE
);
assert_eq!(
smeared_timestamp.create(now),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE
);
// System time rewinds back by 1000 seconds.
let now = now - 1000;
assert_eq!(
smeared_timestamp.create(now),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE
);
assert_eq!(
smeared_timestamp.create(now),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE
);
assert_eq!(
smeared_timestamp.create(now + 1),
now + MAX_SECONDS_TO_LEND_FROM_FUTURE + 1
);
assert_eq!(smeared_timestamp.create(now + 100), now + 100);
assert_eq!(smeared_timestamp.create(now + 100), now + 101);
assert_eq!(smeared_timestamp.create(now + 100), now + 102);
}
#[test]
fn test_create_n_smeared_timestamps() {
let smeared_timestamp = SmearedTimestamp::new();
let now = time();
// Create a single timestamp to initialize the generator.
assert_eq!(smeared_timestamp.create(now), now);
// Wait a minute.
let now = now + 60;
// Simulate forwarding 7 messages.
let forwarded_messages = 7;
// We have not sent anything for a minute,
// so we can take the current timestamp and take 6 timestamps from the past.
assert_eq!(smeared_timestamp.create_n(now, forwarded_messages), now - 6);
assert_eq!(smeared_timestamp.current(), now + 1);
// Wait 4 seconds.
// Now we have 3 free timestamps in the past.
let now = now + 4;
assert_eq!(smeared_timestamp.current(), now - 3);
// Forward another 7 messages.
// We can only lend 3 timestamps from the past.
assert_eq!(smeared_timestamp.create_n(now, forwarded_messages), now - 3);
// We had to borrow 3 timestamps from the future
// because there were not enough timestamps in the past.
assert_eq!(smeared_timestamp.current(), now + 4);
// Forward another 7 messages.
// We cannot use more than 5 timestamps from the future,
// so we use 5 timestamps from the future,
// the current timestamp and one timestamp from the past.
assert_eq!(smeared_timestamp.create_n(now, forwarded_messages), now - 1);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_smeared_timestamp() {
let t = TestContext::new().await;
assert_ne!(create_smeared_timestamp(&t), create_smeared_timestamp(&t));
assert!(
create_smeared_timestamp(&t)
>= SystemTime::now()
.duration_since(SystemTime::UNIX_EPOCH)
.unwrap()
.as_secs() as i64
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_smeared_timestamps() {
let t = TestContext::new().await;
let count = MAX_SECONDS_TO_LEND_FROM_FUTURE - 1;
let start = create_smeared_timestamps(&t, count as usize);
let next = smeared_time(&t);
assert!((start + count - 1) < next);
let count = MAX_SECONDS_TO_LEND_FROM_FUTURE + 30;
let start = create_smeared_timestamps(&t, count as usize);
let next = smeared_time(&t);
assert!((start + count - 1) < next);
}
}

87
src/tools.rs
View File

@@ -3,7 +3,6 @@
#![allow(missing_docs)]
use core::cmp::{max, min};
use std::borrow::Cow;
use std::fmt;
use std::io::Cursor;
@@ -140,63 +139,27 @@ pub(crate) fn gm2local_offset() -> i64 {
i64::from(lt.offset().local_minus_utc())
}
// timesmearing
// - as e-mails typically only use a second-based-resolution for timestamps,
// the order of two mails sent withing one second is unclear.
// this is bad eg. when forwarding some messages from a chat -
// these messages will appear at the recipient easily out of order.
// - we work around this issue by not sending out two mails with the same timestamp.
// - for this purpose, in short, we track the last timestamp used in `last_smeared_timestamp`
// when another timestamp is needed in the same second, we use `last_smeared_timestamp+1`
// - after some moments without messages sent out,
// `last_smeared_timestamp` is again in sync with the normal time.
// - however, we do not do all this for the far future,
// but at max `MAX_SECONDS_TO_LEND_FROM_FUTURE`
pub(crate) const MAX_SECONDS_TO_LEND_FROM_FUTURE: i64 = 5;
/// Returns the current smeared timestamp,
///
/// The returned timestamp MUST NOT be sent out.
pub(crate) async fn smeared_time(context: &Context) -> i64 {
let mut now = time();
let ts = *context.last_smeared_timestamp.read().await;
if ts >= now {
now = ts + 1;
}
now
pub(crate) fn smeared_time(context: &Context) -> i64 {
let now = time();
let ts = context.smeared_timestamp.current();
std::cmp::max(ts, now)
}
/// Returns a timestamp that is guaranteed to be unique.
pub(crate) async fn create_smeared_timestamp(context: &Context) -> i64 {
pub(crate) fn create_smeared_timestamp(context: &Context) -> i64 {
let now = time();
let mut ret = now;
let mut last_smeared_timestamp = context.last_smeared_timestamp.write().await;
if ret <= *last_smeared_timestamp {
ret = *last_smeared_timestamp + 1;
if ret - now > MAX_SECONDS_TO_LEND_FROM_FUTURE {
ret = now + MAX_SECONDS_TO_LEND_FROM_FUTURE
}
}
*last_smeared_timestamp = ret;
ret
context.smeared_timestamp.create(now)
}
// creates `count` timestamps that are guaranteed to be unique.
// the frist created timestamps is returned directly,
// the first created timestamps is returned directly,
// get the other timestamps just by adding 1..count-1
pub(crate) async fn create_smeared_timestamps(context: &Context, count: usize) -> i64 {
pub(crate) fn create_smeared_timestamps(context: &Context, count: usize) -> i64 {
let now = time();
let count = count as i64;
let mut start = now + min(count, MAX_SECONDS_TO_LEND_FROM_FUTURE) - count;
let mut last_smeared_timestamp = context.last_smeared_timestamp.write().await;
start = max(*last_smeared_timestamp + 1, start);
*last_smeared_timestamp = start + count - 1;
start
context.smeared_timestamp.create_n(now, count as i64)
}
// if the system time is not plausible, once a day, add a device message.
@@ -592,6 +555,8 @@ pub(crate) fn improve_single_line_input(input: &str) -> String {
}
pub(crate) trait IsNoneOrEmpty<T> {
/// Returns true if an Option does not contain a string
/// or contains an empty string.
fn is_none_or_empty(&self) -> bool;
}
impl<T> IsNoneOrEmpty<T> for Option<T>
@@ -1069,36 +1034,6 @@ DKIM Results: Passed=true, Works=true, Allow_Keychange=true";
assert!(!file_exist!(context, &fn0));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_smeared_timestamp() {
let t = TestContext::new().await;
assert_ne!(
create_smeared_timestamp(&t).await,
create_smeared_timestamp(&t).await
);
assert!(
create_smeared_timestamp(&t).await
>= SystemTime::now()
.duration_since(SystemTime::UNIX_EPOCH)
.unwrap()
.as_secs() as i64
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_create_smeared_timestamps() {
let t = TestContext::new().await;
let count = MAX_SECONDS_TO_LEND_FROM_FUTURE - 1;
let start = create_smeared_timestamps(&t, count as usize).await;
let next = smeared_time(&t).await;
assert!((start + count - 1) < next);
let count = MAX_SECONDS_TO_LEND_FROM_FUTURE + 30;
let start = create_smeared_timestamps(&t, count as usize).await;
let next = smeared_time(&t).await;
assert!((start + count - 1) < next);
}
#[test]
fn test_duration_to_str() {
assert_eq!(duration_to_str(Duration::from_secs(0)), "0h 0m 0s");

55
src/update_helper.rs
View File

@@ -2,8 +2,8 @@
use anyhow::Result;
use crate::chat::{Chat, ChatId};
use crate::contact::{Contact, ContactId};
use crate::chat::ChatId;
use crate::contact::ContactId;
use crate::context::Context;
use crate::param::{Param, Params};
@@ -17,12 +17,26 @@ impl Context {
scope: Param,
new_timestamp: i64,
) -> Result<bool> {
let mut contact = Contact::load_from_db(self, contact_id).await?;
if contact.param.update_timestamp(scope, new_timestamp)? {
contact.update_param(self).await?;
return Ok(true);
}
Ok(false)
self.sql
.transaction(|transaction| {
let mut param: Params = transaction.query_row(
"SELECT param FROM contacts WHERE id=?",
[contact_id],
|row| {
let param: String = row.get(0)?;
Ok(param.parse().unwrap_or_default())
},
)?;
let update = param.update_timestamp(scope, new_timestamp)?;
if update {
transaction.execute(
"UPDATE contacts SET param=? WHERE id=?",
params![param.to_string(), contact_id],
)?;
}
Ok(update)
})
.await
}
}
@@ -35,12 +49,24 @@ impl ChatId {
scope: Param,
new_timestamp: i64,
) -> Result<bool> {
let mut chat = Chat::load_from_db(context, *self).await?;
if chat.param.update_timestamp(scope, new_timestamp)? {
chat.update_param(context).await?;
return Ok(true);
}
Ok(false)
context
.sql
.transaction(|transaction| {
let mut param: Params =
transaction.query_row("SELECT param FROM chats WHERE id=?", [self], |row| {
let param: String = row.get(0)?;
Ok(param.parse().unwrap_or_default())
})?;
let update = param.update_timestamp(scope, new_timestamp)?;
if update {
transaction.execute(
"UPDATE chats SET param=? WHERE id=?",
params![param.to_string(), self],
)?;
}
Ok(update)
})
.await
}
}
@@ -60,6 +86,7 @@ impl Params {
#[cfg(test)]
mod tests {
use super::*;
use crate::chat::Chat;
use crate::receive_imf::receive_imf;
use crate::test_utils::TestContext;
use crate::tools::time;

7
src/webxdc.rs
View File

@@ -408,7 +408,7 @@ impl Context {
.create_status_update_record(
&mut instance,
update_str,
create_smeared_timestamp(self).await,
create_smeared_timestamp(self),
send_now,
ContactId::SELF,
)
@@ -431,6 +431,7 @@ impl Context {
async fn pop_smtp_status_update(
&self,
) -> Result<Option<(MsgId, StatusUpdateSerial, StatusUpdateSerial, String)>> {
let _lock = self.sql.write_lock().await;
let res = self
.sql
.query_row_optional(
@@ -670,8 +671,10 @@ impl Message {
Ok(archive)
}
/// Return file form inside an archive.
/// Return file from inside an archive.
/// Currently, this works only if the message is an webxdc instance.
///
/// `name` is the filename within the archive, e.g. `index.html`.
pub async fn get_webxdc_blob(&self, context: &Context, name: &str) -> Result<Vec<u8>> {
ensure!(self.viewtype == Viewtype::Webxdc, "No webxdc instance.");

70
standards.md
View File

@@ -3,26 +3,58 @@
Some of the standards Delta Chat is based on:
Tasks | Standards
---------------------------------|---------------------------------------------
Transport | IMAP v4 ([RFC 3501](https://tools.ietf.org/html/rfc3501)), SMTP ([RFC 5321](https://tools.ietf.org/html/rfc5321)) and Internet Message Format (IMF, [RFC 5322](https://tools.ietf.org/html/rfc5322))
Proxy | SOCKS5 ([RFC 1928](https://tools.ietf.org/html/rfc1928))
Embedded media | MIME Document Series ([RFC 2045](https://tools.ietf.org/html/rfc2045), [RFC 2046](https://tools.ietf.org/html/rfc2046)), Content-Disposition Header ([RFC 2183](https://tools.ietf.org/html/rfc2183)), Multipart/Related ([RFC 2387](https://tools.ietf.org/html/rfc2387))
Text and Quote encoding | Fixed, Flowed ([RFC 3676](https://tools.ietf.org/html/rfc3676))
Reactions | Reaction: Indicating Summary Reaction to a Message [RFC 9078](https://datatracker.ietf.org/doc/rfc9078/)
Filename encoding | Encoded Words ([RFC 2047](https://tools.ietf.org/html/rfc2047)), Encoded Word Extensions ([RFC 2231](https://tools.ietf.org/html/rfc2231))
Identify server folders | IMAP LIST Extension ([RFC 6154](https://tools.ietf.org/html/rfc6154))
Push | IMAP IDLE ([RFC 2177](https://tools.ietf.org/html/rfc2177))
Quota | IMAP QUOTA extension ([RFC 2087](https://tools.ietf.org/html/rfc2087))
Seen status synchronization | IMAP CONDSTORE extension ([RFC 7162](https://tools.ietf.org/html/rfc7162))
Client/server identification | IMAP ID extension ([RFC 2971](https://datatracker.ietf.org/doc/html/rfc2971))
Authorization | OAuth2 ([RFC 6749](https://tools.ietf.org/html/rfc6749))
End-to-end encryption | [Autocrypt Level 1](https://autocrypt.org/level1.html), OpenPGP ([RFC 4880](https://tools.ietf.org/html/rfc4880)), Security Multiparts for MIME ([RFC 1847](https://tools.ietf.org/html/rfc1847)) and [“Mixed Up” Encryption repairing](https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html)
-------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Transport | IMAP v4 ([RFC 3501][]), SMTP ([RFC 5321][]) and Internet Message Format (IMF, [RFC 5322][])
Proxy | SOCKS5 ([RFC 1928][])
Embedded media | MIME Document Series ([RFC 2045][], [RFC 2046][]), Content-Disposition Header ([RFC 2183][]), Multipart/Related ([RFC 2387][])
Text and Quote encoding | Fixed, Flowed ([RFC 3676][])
Reactions | Reaction: Indicating Summary Reaction to a Message ([RFC 9078][])
Filename encoding | Encoded Words ([RFC 2047][]), Encoded Word Extensions ([RFC 2231][])
Identify server folders | IMAP LIST Extension ([RFC 6154][])
Push | IMAP IDLE ([RFC 2177][])
Quota | IMAP QUOTA extension ([RFC 2087][])
Seen status synchronization | IMAP CONDSTORE extension ([RFC 7162][])
Client/server identification | IMAP ID extension ([RFC 2971][])
Authorization | OAuth2 ([RFC 6749][])
End-to-end encryption | [Autocrypt Level 1][], OpenPGP ([RFC 4880][]), Security Multiparts for MIME ([RFC 1847][]) and [“Mixed Up” Encryption repairing](https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html)
Detect/prevent active attacks | [countermitm][] protocols
Compare public keys | [openpgp4fpr][] URI Scheme
Header encryption | [Protected Headers for Cryptographic E-mail](https://datatracker.ietf.org/doc/draft-autocrypt-lamps-protected-headers/)
Configuration assistance | [Autoconfigure](https://web.archive.org/web/20210402044801/https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration) and [Autodiscover](https://technet.microsoft.com/library/bb124251(v=exchg.150).aspx)
Configuration assistance | [Autoconfigure](https://web.archive.org/web/20210402044801/https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration) and [Autodiscover][]
Messenger functions | [Chat-over-Email](https://github.com/deltachat/deltachat-core-rust/blob/master/spec.md#chat-mail-specification)
Detect mailing list | List-Id ([RFC 2919](https://tools.ietf.org/html/rfc2919)) and Precedence ([RFC 3834](https://tools.ietf.org/html/rfc3834))
User and chat colors | [XEP-0392](https://xmpp.org/extensions/xep-0392.html): Consistent Color Generation
Send and receive system messages | Multipart/Report Media Type ([RFC 6522](https://tools.ietf.org/html/rfc6522))
Return receipts | Message Disposition Notification (MDN, [RFC 8098](https://tools.ietf.org/html/rfc8098), [RFC 3503](https://tools.ietf.org/html/rfc3503)) using the Chat-Disposition-Notification-To header
Detect mailing list | List-Id ([RFC 2919][]) and Precedence ([RFC 3834][])
User and chat colors | [XEP-0392][]: Consistent Color Generation
Send and receive system messages | Multipart/Report Media Type ([RFC 6522][])
Return receipts | Message Disposition Notification (MDN, [RFC 8098][], [RFC 3503][]) using the Chat-Disposition-Notification-To header
Locations | KML ([Open Geospatial Consortium](http://www.opengeospatial.org/standards/kml/), [Google Dev](https://developers.google.com/kml/))
[Autocrypt Level 1]: https://autocrypt.org/level1.html
[countermitm]: https://countermitm.readthedocs.io/en/latest/
[openpgp4fpr]: https://metacode.biz/openpgp/openpgp4fpr
[Autodiscover]: https://learn.microsoft.com/en-us/exchange/autodiscover-service-for-exchange-2013
[XEP-0392]: https://xmpp.org/extensions/xep-0392.html
[RFC 1847]: https://tools.ietf.org/html/rfc1847
[RFC 1928]: https://tools.ietf.org/html/rfc1928
[RFC 2045]: https://tools.ietf.org/html/rfc2045
[RFC 2046]: https://tools.ietf.org/html/rfc2046
[RFC 2047]: https://tools.ietf.org/html/rfc2047
[RFC 2087]: https://tools.ietf.org/html/rfc2087
[RFC 2177]: https://tools.ietf.org/html/rfc2177
[RFC 2183]: https://tools.ietf.org/html/rfc2183
[RFC 2231]: https://tools.ietf.org/html/rfc2231
[RFC 2387]: https://tools.ietf.org/html/rfc2387
[RFC 2919]: https://tools.ietf.org/html/rfc2919
[RFC 2971]: https://tools.ietf.org/html/rfc2971
[RFC 3501]: https://tools.ietf.org/html/rfc3501
[RFC 3503]: https://tools.ietf.org/html/rfc3503
[RFC 3676]: https://tools.ietf.org/html/rfc3676
[RFC 3834]: https://tools.ietf.org/html/rfc3834
[RFC 4880]: https://tools.ietf.org/html/rfc4880
[RFC 5321]: https://tools.ietf.org/html/rfc5321
[RFC 5322]: https://tools.ietf.org/html/rfc5322
[RFC 6154]: https://tools.ietf.org/html/rfc6154
[RFC 6522]: https://tools.ietf.org/html/rfc6522
[RFC 6749]: https://tools.ietf.org/html/rfc6749
[RFC 7162]: https://tools.ietf.org/html/rfc7162
[RFC 8098]: https://tools.ietf.org/html/rfc8098
[RFC 9078]: https://tools.ietf.org/html/rfc9078