8b58b16cb5
I used some AI to draft a first version of this, and then reworked it. This is one of multiple possibilities to fix https://github.com/chatmail/core/issues/8041: For bots, the IncomingMsg event is not emitted when the pre-message arrives, only when the post-message arrives. Also, post-messages are downloaded immediately, not after all the small messages are downloaded. The `get_next_msgs()` API is deprecated. Instead, bots need to listen to the IncomingMsg event in order to be notified about new events. Is this acceptable for bots? THE PROBLEM THAT WAS SOLVED BY THIS: With pre-messages, it's hard for bots to wait for the message to be fully downloaded and then process it. Up until now, bots used get_next_msgs() to query the unprocessed messages, then set last_msg_id after processing a message to that they won't process it again. But this will now also return messages that were not fully downloaded. ALTERNATIVES: In the following, I will explain the alternatives, and for why it's not so easy to just make the `get_next_msgs()` API work. If it's not understandable, I'm happy to elaborate more. Core can't just completely ignore the pre-message for two reasons: - If a post-message containing a Webxdc arrives later, and some webxdc updates arrive in the meantime, then these updates will be lost. - The post-message doesn't contain the text (reasoning was to avoid duplicate text for people who didn't upgrade yet during the 2.43.0 rollout) There are multiple solutions: - Add the message as hidden in the database when the pre-message arrives. - When the post-message arrives and we want to make it available for bots, we need to update the msg_id because of how the `get_next_msgs()` API works. This means that we need to update all webxdc updates that reference this msg_id. - Alternatively, we could make webxdc's reference the rfc724_mid instead of the msg_id, so that we don't need stable msg_ids anymore - Alternatively, we could deprecate `get_next_msgs()`, and ask bots to use plain events for message processing again. It's not that bad; worst case, the bot crashes and then forgets to react to some messages, but the user will just try again. And if some message makes the bot crash, then it might actually be good not to try and process it again. - Store the pre-message text and `PostMsgMetadata` (or alternatively, the whole mime) in a new database table. Wait with processing it until the post-message arrives. Additionally, the logic that small messages are downloaded before post-messages should be disabled for bots, in order to prevent reordering.
The chatmail core library implements low-level network and encryption protocols,
integrated by many chat bots and higher level applications,
allowing to securely participate in the globally scaled e-mail server network.
We provide reproducibly-built deltachat-rpc-server static binaries
that offer a stdio-based high-level JSON-RPC API for instant messaging purposes.
The following protocols are handled without requiring API users to know much about them:
-
secure TLS setup with DNS caching and shadowsocks/proxy support
-
safe and interoperable MIME parsing and MIME building.
-
security-audited end-to-end encryption with rPGP and Autocrypt and SecureJoin protocols
-
ephemeral Peer-to-Peer networking using Iroh for multi-device setup and webxdc realtime data.
-
a simulation- and real-world tested P2P group membership protocol without requiring server state.
Installing Rust and Cargo
To download and install the official compiler for the Rust programming language, and the Cargo package manager, run the command in your user environment:
$ curl https://sh.rustup.rs -sSf | sh
On Windows, you may need to also install Perl to be able to compile deltachat-core.
Using the CLI client
Compile and run the command line utility, using cargo:
$ cargo run --locked -p deltachat-repl -- ~/profile-db
where ~/profile-db is the database file. The utility will create it if it does not exist.
Optionally, install deltachat-repl binary with
$ cargo install --locked --path deltachat-repl/
and run as
$ deltachat-repl ~/profile-db
Configure your account (if not already configured):
Chatmail is awaiting your commands.
> set addr your@email.org
> set mail_pw yourpassword
> configure
Connect to your mail server (if already configured):
> connect
Export your public key to a vCard file:
> make-vcard my.vcard 1
Create contacts by address or vCard file:
> addcontact yourfriends@email.org
> import-vcard key-contact.vcard
List contacts:
> listcontacts
Contact#Contact#11: key-contact@email.org <key-contact@email.org>
Contact#Contact#Self: Me √ <your@email.org>
2 key contacts.
Contact#Contact#10: yourfriends@email.org <yourfriends@email.org>
1 address contacts.
Create a chat with your friend and send a message:
> createchat 10
Single#Chat#12 created successfully.
> chat 12
Selecting chat Chat#12
Single#Chat#12: yourfriends@email.org [yourfriends@email.org] Icon: profile-db-blobs/4138c52e5bc1c576cda7dd44d088c07.png
0 messages.
81.252µs to create this list, 123.625µs to mark all messages as noticed.
> send hi
List messages when inside a chat:
> chat
For more commands type:
> help
Installing libdeltachat system wide
$ git clone https://github.com/chatmail/core.git
$ cd deltachat-core-rust
$ cmake -B build . -DCMAKE_INSTALL_PREFIX=/usr
$ cmake --build build
$ sudo cmake --install build
Development
# run tests
$ cargo test --all
# build c-ffi
$ cargo build -p deltachat_ffi --release
Debugging environment variables
-
DCC_MIME_DEBUG: if set outgoing and incoming message will be printed -
RUST_LOG=async_imap=trace,async_smtp=trace: enable IMAP and SMTP tracing in addition to info messages.
Expensive tests
Some tests are expensive and marked with #[ignore], to run these
use the --ignored argument to the test binary (not to cargo itself):
$ cargo test -- --ignored
Fuzzing
Install cargo-bolero with
$ cargo install cargo-bolero
Run fuzzing tests with
$ cd fuzz
$ cargo bolero test fuzz_mailparse -s NONE
Corpus is created at fuzz/fuzz_targets/corpus,
you can add initial inputs there.
For fuzz_mailparse target corpus can be populated with
../test-data/message/*.eml.
Features
vendored: When using Openssl for TLS, this bundles a vendored version.
Update Provider Data
To add the updates from the
provider-db to the core,
check line REV= inside ./scripts/update-provider-database.sh
and then run the script.
Language bindings and frontend projects
Language bindings are available for:
- C [📂 source | 📚 docs]
- -> libdeltachat is going to be deprecated and only exists because Android, iOS and Ubuntu Touch are still using it. If you build a new project, then please use the jsonrpc api instead.
- JS: [📂 source | 📦 npm | 📚 docs]
- Python [📂 source | 📦 pypi | 📚 docs]
- Go [📂 source]
- Java and Swift (contained in the Android/iOS repos)
The following "frontend" projects make use of the Rust-library or its language bindings: