This changes the JSON-RPC APIs to get a QR code from the backup
provider to block. It means once you have a (blocking) call to
provide_backup() you can call get_backup_qr() or get_backup_qr_svg()
and they will block until the QR code is available.
Calling get_backup_qr() or get_backup_qr_svg() when there is no backup
provider will immediately error.
This adds a result extension trait to explicitly set the last error,
which *should* be the default for the FFI. Currently not touching all
APIs since that's potentially disruptive and we're close to a release.
The logging story is messy, as described in the doc comment. We
should further clean this up and tidy up these APIs so it's more
obvious to people how to do the right thing.
This makes the BackupProvider automatically invoke pause-io while it
is needed.
It needed to make the guard independent from the Context lifetime to
make this work. Which is a bit sad.
To handle backups the UIs have to make sure they do stop the IO
scheduler and also don't accidentally restart it while working on it.
Since they have to call start_io from a bunch of locations this can be
a bit difficult to manage.
This introduces a mechanism for the core to pause IO for some time,
which is used by the imex function. It interacts well with other
calls to dc_start_io() and dc_stop_io() making sure that when resumed
the scheduler will be running or not as the latest calls to them.
This was a little more invasive then hoped due to the scheduler. The
additional abstraction of the scheduler on the context seems a nice
improvement though.
async-smtp does not implement read buffering anymore
and expects library user to implement it.
To implement read buffer, we wrap streams into BufStream
instead of BufWriter.
The fact that `PRAGMA incremental_vacuum` may return a zero-column
SQLITE_ROW result is documented in `sqlite3_data_count()` documentation:
<https://www.sqlite.org/c3ref/data_count.html>
Previously successful auto_vacuum worked,
but resulted in a "Failed to run incremental vacuum" log.
With existing approach of constructing
the SQL query dynamically I get errors like this:
ephemeral.rs:575: update failed: too many SQL variables: Error code 1: SQL error or missing database
In my case it is trying to delete 143658 messages.
This is the result of importing a Desktop backup
and enabling device auto-deletion on the phone.
Current SQLite limit is 32766 variables
as stated in <https://www.sqlite.org/limits.html>.
This way we can test some recently added config options that we don't want to expose in UI like
DeleteToTrash or SignUnencrypted. Note that persistent config options like DeleteToTrash should
remain anyway because they allow fine-grained (per-account) control. Having them matters for tests
also.
This fixes use-after-free in case dc_context_unref() is called
while the background process spawned by dc_configure() is still
running.
Corresponding regression test in Python can be run with
`pytest tests/test_1_online.py::test_configure_unref`.
dc_configure() spawns a background configuration process.
It should increase the number of context references
so even if we unref the context, it is not dropped
until the end of the configuration process.
Currently running the test
with `pytest tests/test_1_online.py::test_configure_unref`
results in segmentation fault.
`<meta name="color-scheme" content="light dark" />` is a hint to the browsers
that the page can be rendered in light as well as in dark mode
and that the browser can apply corresponding defaults.
as we do not add css colors on our own,
this is sufficient for letting generated html messasge being rendered
in dark mode.
cmp. https://drafts.csswg.org/css-color-adjust/#color-scheme-propcloses#4146
standard footers meanwhile go the "contact status",
so they are no longer a reason to trigger "full message view".
this was already discussed when the HTML view was introduced at #2125
however, forgotten to change when the "contact status" was added at #2218
this pr will result in a cleaner chat view
with less "Show Full Message..." buttons
and will also save some storage
(in fact, i came over that when reviewing #4129 )
the contrast was decreased at
https://github.com/deltachat/deltachat-core-rust/pull/4136 ,
there were also suggestions to fix it there,
but it was probably just forgotten :)
this pr increases the contrast
between code background and code font again to the level it was before
and also to what similar themes are doing.
We are currently using libsqlite3-sys 0.25.2,
corresponding to SQLcipher 4.5.2
and SQLite 3.39.2.
SQLite supports ALTER TABLE DROP COLUMN since version 3.35.0,
and it has received critical database corruption bugfixes in 3.35.5.
There have been no fixes to it between SQLite 3.36.0 and 3.41.0,
so it appears stable now.
Explicit `prepare_msg` corresponding to `dc_prepare_msg`
is intended only for the case where the message is prepared
before the file is ready. It is not indented for calling
right before send_msg and requires that file path in the blobdir.
With explicit `prepare_msg` removed
all the tests still pass but there is no requirement
that the file is put into blobdir beforehand.
This adds more configuration to cargo-deny so that the output is not a
giant list of warnings hiding new entries. Instead the config now
lists the things that would emit warnings. It also adds -Dwarnings to
the CI job so that new warnings will be cleaned up: they can be added
to the config easily to fix the warnings.
Desktop does use this as it allows reading QR codes as text from the
clipboard as well as copying the QR text to the clipboard instead of
showing the QR code.
We do not make all transactions
[IMMEDIATE](https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions)
for more parallelism -- at least read transactions can be made DEFERRED to run in parallel
w/o any drawbacks. But if we make write transactions DEFERRED also w/o any external locking,
then they are upgraded from read to write ones on the first write statement. This has some
drawbacks:
- If there are other write transactions, we block the thread and the db connection until
upgraded. Also if some reader comes then, it has to get next, less used connection with a
worse per-connection page cache.
- If a transaction is blocked for more than busy_timeout, it fails with SQLITE_BUSY.
- Configuring busy_timeout is not the best way to manage transaction timeouts, we would
prefer it to be integrated with Rust/tokio asyncs. Moreover, SQLite implements waiting
using sleeps.
- If upon a successful upgrade to a write transaction the db has been modified by another
one, the transaction has to be rolled back and retried. It is an extra work in terms of
CPU/battery.
- Maybe minor, but we lose some fairness in servicing write transactions, i.e. we service
them in the order of the first write statement, not in the order they come.
The only pro of making write transactions DEFERRED w/o the external locking is some
parallelism between them. Also we have an option to make write transactions IMMEDIATE, also
w/o the external locking. But then the most of cons above are still valid. Instead, if we
perform all write transactions under an async mutex, the only cons is losing some
parallelism for write transactions.
scripts/update-provider-database.sh checks out the provider database
and updates data.rs file,
making it easier to update.
CI uses this script to check that checked in data.rs
corresponds to the provider database repository.
the slowest test was `test_modify_chat_disordered`;
due to several sleep(), it lasts at least 11 seconds.
however, many of the sleep() are not needed and were added
just to get things done that time.
this pr removes 7 superfluous sleeps,
making the test finish in about 3 seconds,
so that this test is no longer the bottleneck when running
`cargo test` on fast machines.
(this is not only useful to safe some seconds -
but also to notice degradations as of
https://github.com/deltachat/deltachat-core-rust/issues/4051#issuecomment-1436995166 )
Although it does a little for security, it will help to protect from unwanted server-side
modifications and bugs. And now we have a time to test "multipart/signed" messages compatibility
with other MUAs.
This makes DC compatible with "multipart/signed" messages thus allowing switching to them someday
from the current "multipart/mixed" unencrypted message format.
try_decrypt() is a CPU-bound task.
When called from async function,
it should be wrapped in tokio::task::spawn_blocking().
Using tokio::task::spawn_blocking() is difficult here
because of &mail, &private_keyring and &public_keyring borrows,
so we should at least use tokio::task::block_in_place()
to avoid blocking the executor.
Previously we used tags like "1.109.0" and "py-1.109.0".
Since Python bindings are always released
at the same time as the core,
it is easier to use a single tag for them.
"v" prefix is added to make matching by "v*" easier
in CI and scripts.
the speedup of #4065 is larger than measured first
running `cargo test`, first we thought there is only a slight speedup
from 13.5 seconds to 12.5 seconds on a m1pro.
however, there is one test that does 11 seconds of sleep() (test_modify_chat_disordered)
as this test is not run very first, this slows down things overall.
skipping this test,
speedup is from 13.5 seconds to 9.5 seconds -
28% faster - and this is something we should mention in the changelog :)
(this pr does not remove or change the slow test.
i think, due to the number of cores, this is not needed until someone
has a machine where the other tests run in 2 seconds or so)
Latest versions 0.0.248 and 0.0.249 report false positive:
src/deltachat_rpc_client/client.py:105:9: RET503 [*] Missing explicit `return` at the end of function able to return
.call() interface is safer because it ensures
that blocking operations on SQL connection
are called within tokio::task::block_in_place().
Previously some code called blocking operations
in async context, e.g. add_parts() in receive_imf module.
The underlying implementation of .call()
can later be replaced with an implementation
that does not require block_in_place(),
e.g. a worker pool,
without changing the code using the .call() interface.
IIRC, this was written this way back when we didn't have config caching,
in order to save database accesses by getting the config outside the
for loop.
Gmail archives messages marked as `\Deleted` by default if those messages aren't in the Trash. But
if move them to the Trash instead, they will be auto-deleted in 30 days.
With the introduction of transactions in Contact::add_or_lookup(),
python tests sometimes fail to create contacts with the folowing error:
Cannot create contact: add_or_lookup: database is locked: Error code 5: The database file is locked
`PRAGMA busy_timeout=60000` does not affect
this case as the error is returned before 60 seconds pass.
DEFERRED transactions with write operations need
to be retried from scratch
if they are rolled back
due to a write operation on another connection.
Using IMMEDIATE transactions for writing
is an attempt to fix this problem
without a retry loop.
If we later need DEFERRED transactions,
e.g. for reading a snapshot without locking the database,
we may introduce another function for this.
When connection pool is organized as a stack,
it always returns most recently used connection.
Because each connection has its own page cache,
using the connection with fresh cache improves performance.
I commented out `oauth2::tests::test_oauth_from_mx`
because it requires network connection,
turned off the Wi-Fi and ran the tests.
Before the change, with a queue:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 56.424 s ± 4.515 s [User: 183.181 s, System: 128.156 s]
Range (min … max): 52.123 s … 68.193 s 10 runs
```
With a stack:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 29.887 s ± 1.377 s [User: 101.226 s, System: 45.573 s]
Range (min … max): 26.591 s … 31.010 s 10 runs
```
On version 1.107.1:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 43.658 s ± 1.079 s [User: 202.582 s, System: 50.723 s]
Range (min … max): 41.531 s … 45.170 s 10 runs
```
This also removes BackupProvider::join in favour of implementing
Future directly. I wondered about implementing a FusedFutre to make
this a little safer but it would introduce a dependency on the futures
crate in deltachat-ffi which did not exist yet, so I didn't do that.
the axum update broke the websocket server, because yerpc still uses a the old 5.9 version.
So I needed to downgrade the dependency until https://github.com/deltachat/yerpc/pull/35 is merged.
I want to retry the kaiOS client experiment, for this I need a working websocket server binary.
We no longer need that in the transfer case, that would give very
weird results. This also means there is nothing imex-specific about
this anymore so move it to blobs.rs
In execute_migration transaction first update the version, and only
then execute the rest of the migration. This ensures that transaction
is immediately recognized as write transaction and there is no need
to promote it from read transaction to write transaction later, e.g.
in the case of "DROP TABLE IF EXISTS" that is a read only operation if
the table does not exist. Promoting a read transaction to write
transaction may result in an error if database is locked.
* jsonrpc: `get_messages` now returns a map with `MessageLoadResult`
instead of failing completely if one of the requested messages could not be loaded.
* add pr number to changelog
* format errors with causes instead of debug output
also for chatlistitemfetchresult
get_chat_msgs() function is split into new get_chat_msgs() without flags
and get_chat_msgs_ex() which accepts booleans instead of bitflags.
FFI call dc_get_chat_msgs() is still using bitflags for compatibility.
JSON-RPC calls get_message_ids() and get_message_list_items()
accept booleans instead of bitflags now.
This adds functionality to send and receive a backup over the network
using a QR code.
The sender or provider prepares the backup, sets up a server that
waits for clients. It provides a ticket in the form of a QR code
which contains connection and authentication information.
The receiver uses the QR code to connect to the provider and fetches
backup, restoring it locally.
actions-rs/clippy-check runs clippy with --message-format=json option
and converts the output into annotations.
This makes clippy output unreadable, it is JSON
and you cannot quickly find the line number to fix.
Annotations in the code review view look nice,
but on large PRs they are less usable because you need
to scroll the whole page to find all the annotations.
There was the following bug:
- Bob has two devices, the second is offline.
- Alice creates a verified group and sends a QR invitation to Bob.
- Bob joins the group.
- Bob's second devices goes online, but sees a contact request instead of the verified group.
- The "member added" message is not a system message but a plain text message.
- Sending a message fails as the key is missing -- message info says "proper enc-key for <Alice>
missing, cannot encrypt".
This fixes the case with multiple devices on the joining side: if we observe such a message from
another device, we mark an inviter as verified and an accepted contact thus causing a subsequent
"vg-member-added" message -- in case of a verified group -- to create it properly.
DNS cache is used as a fallback if TCP connection
to all IP addresses returned by DNS failed.
If strict TLS checks are disabled,
DNS cache results are stored, but not used.
GitHub Pull Request: <https://github.com/deltachat/deltachat-core-rust/pull/3970>
This bug was introduced by 3cf78749df - there are three visibility
states: Archived, Pinned and Normal - in the database, this "visibility" is named historically
"archived" ... the original code has an AND archived=1 therefore.
It seems .abort() does not work on the recently seen loop
in some cases, e.g. if it is busy looping in a separate thread.
In my case after account reconfiguration recently seen loop
kept running and issuing warnings about closed interrupt channel.
Exit from recently seen loop on errors to avoid using 100% CPU
in such cases.
Derive Debug, PartialEq and Eq for Peerstate,
so `verifier` is included in Debug output and compared.
Store verifier as empty string
instead of NULL in the database.
- Return Result from set_verified() so that it can't be missed.
- Pass Fingerprint to set_verified() by value to avoid cloning it inside. This optimises out an
extra clone() if we already have a value that can be moved at the caller side. However, this may
add an extra clone() if set_verified() fails, but let's not optimise the fail scenario.
- Alice has two devices, the second is offline.
- Alice creates a verified group and sends a QR invitation to Bob.
- Bob joins the group and sends a message there. Alice sees it.
- Alice's second devices goes online, but doesn't see Bob in the group.
- If bcc_self is set, gossip headers must be added despite of the number of group members.
- If another device observes Secure-Join, instead of looking for Secure-Join-Fingerprint in
"vg-member-added"/"vc-contact-confirm" messages it must use keys from Autocrypt-Gossip headers as
described in the Countermitm doc
(https://countermitm.readthedocs.io/en/latest/new.html#joining-a-verified-group-secure-join).
This setting is true by default and causes
Windows build to cancel when Linux fails
due to flaky test and vice versa.
Cancelled test then has to be restarted
from scratch even though it was not going
to fail.
This allows to distinguish exceptions,
such as database errors, from invalid user input.
For example, if the From: field of the message
does not look like an email address, the mail
should be ignored. But if there is a database
failure while writing a new contact for the address,
this error should be bubbled up.
* Print chats after a test failed again
E.g.
```
========== Chats of bob: ==========
Single#Chat#10: alice@example.org [alice@example.org]
--------------------------------------------------------------------------------
Msg#10: (Contact#Contact#10): hellooo [FRESH]
Msg#11: (Contact#Contact#10): hellooo without mailing list [FRESH]
--------------------------------------------------------------------------------
========== Chats of alice: ==========
Single#Chat#10: bob@example.net [bob@example.net]
--------------------------------------------------------------------------------
Msg#10: Me (Contact#Contact#Self): hellooo √
Msg#11: Me (Contact#Contact#Self): hellooo without mailing list √
--------------------------------------------------------------------------------
```
I found this very useful sometimes, so, let's try to re-introduce it (it
was removed in #3449)
* Add failing test for TestContext::drop()
* Do not panic in TestContext::drop() if runtime is dropped
Co-authored-by: link2xt <link2xt@testrun.org>
this is a followup to #3918
I went with option "C" from my comment:
https://github.com/deltachat/deltachat-core-rust/pull/3918#issuecomment-1371224339
- Archive link is (still) very different from a normal chat, so most of the options would be empty / not relevant
- avatar path is not needed as desktop renders it differently anyway,
it's not really a chat like saved messages or device messages where it made more sense
for the core to supply the icon, vs. using the svg directly.
- translating the string in the coreas stock-string is more annoying than doing it from the ui, especially when
this special pseudo chat has different rendering anyway so also no need to provide a name property
* let marknoticed_chat() work for DC_CHAT_ID_ARCHIVED_LINK
* fix test_util::get_last_msg() - the first position may be the archive-link if 'adding specials' is allowed
* add a test for the archived-link message counter
* update CHANGELOG
* move 'archived link' betweeen pinned and normal cahts or above normal chats
* add icon for 'archived chats' link
* let get_fresh_msg_cnt() work for DC_CHAT_ID_ARCHIVED_LINK
* move 'archived link' topmost
* use less noticeable archived-icon
* slightly smaller archived icon
* update CHANGELOG
There are at least two user reports that fetching existing messages
sometimes results in infinite loop of retrying it. Account is working
if set up from the backup, but never starts working if set up
from scratch.
This change improves error reporting, but also sets FetchedExistingMsgs
before actually trying to do it. This way if the operation fails,
connection is reestablished, but fetching existing messages is not
retried again over and over.
async-imap does not do its own buffering, but calls flush() after
sending each command. Using BufWriter reduces the number of write()
system calls used to send a single command.
Note that BufWriter is set up on top of TLS streams, because
we can't guarantee that TLS libraries flush the stream before
waiting for response.
This makes it possible to fuzz test the functions
without exposing the module interface in the deltachat core
interface.
Also ensure that format_flowed will not grow a dependency
on deltachat core types.
If we move the detached signatures validation code out of try_decrypt(), we don't need to convert an
already parsed signed message part to Vec and then parse it back. Also this simplifies the
try_decrypt() semantics and return type. It can't make a good coffee anyway.
Bumping MSRV from 1.61.0 to 1.63.0, because `arbitrary` crate requires
it and fuzzing crates depend on it, at least by default. We still use
1.64.0 as our default rust toolchain.
Services like Lacre [1] on Disroot and Inbound Encryption on Posteo [2]
offer to encrypt all incoming messages with the provided OpenPGP
public key. Resulting messages are encrypted, but not end-to-end encrypted
and not signed by the sender, therefore should not have a padlock displayed.
However, such encrypted and unsigned message is also not an indication
of an error on ongoing attack, so we shoud not report this as a problem
to the user.
[1] https://lacre.io/
[2] https://posteo.de/en/help/how-do-i-activate-inbound-encryption-with-my-public-pgp-key
This way we don't need a separate code path for signatures validation for unencrypted
messages. Also, now we degrade encryption only if there are no valid signatures, so the code for
upgrading encryption back isn't needed.
Note that if the message is encrypted, we don't check whether it's signed with an attached key
currently, otherwise a massive refactoring of the code is needed because for encrypted messages a
signature is checked and discarded first now.
* do not `SELECT *` on old tables to fill new ones
the old table may contain deprecrated columns for whatever reason;
as a result the query fails as the statement tries to insert eg.
16 columns into 12 colums
(concrete error for acpeerstate that have several deprecated columns)
* update CHANGELOG
IMAP capabilities and selected folder are IMAP session,
not IMAP client property.
Moving most operations into IMAP session structure
removes the need to constantly check whether IMAP session exists
and reduces number of invalid states, e.g. when a folder is selected but
there is no connection.
Capabilities are determined immediately after logging in,
so there is no need for `capabilities_determined` flag anymore.
Capabilities of the server are always known if there is a session.
`should_reconnect` flag and `disconnect()` function are removed: we
drop the session on error. Even though RFC 3501 says that a client
SHOULD NOT close the connection without a LOGOUT, it is more reliable
to always just drop the connection, especially after an error.
* Because both only make problems with mailing lists, it's easiest to just disable them. If we want, we can make them work properly with mailing lists one day and re-enable them, but this needs some further thoughts.
Part of #3701
* Use load_from_db() in more tests
* clippy
* Changelog
* Downgrade warning to info, improve message
* Use lifetimes instead of cloning
When a batch of messages is moved from Inbox to DeltaChat folder with a single MOVE command, their
UIDs may be reordered (e.g. Gmail is known for that) which leads to that messages are processed by
receive_imf in the wrong order. But the INTERNALDATE attribute is preserved during a MOVE according
to RFC3501. So, use it for sorting fetched messages.
Both reversed and original order do not make much sense
for the bot. Ideally bots should have their own key
to get the list of fresh messages in the order of IDs.
This esp. speeds up receive_imf a bit when we recreate the member list (recreate_member_list == true).
It's a preparation for https://github.com/deltachat/deltachat-core-rust/issues/3768, which will be a one-line-fix, but recreate the member list more often, so that we want to optimize this case a bit.
It also adds a benchmark for this case. It's not that easy to make the benchmark non-flaky, but by closing all other programs and locking the CPU to 1.5GHz it worked. It is consistently a few percent faster than ./without-optim:
```
Receive messages/Receive 100 Chat-Group-Member-{Added|Removed} messages
time: [52.257 ms 52.569 ms 52.941 ms]
change: [-3.5301% -2.6181% -1.6697%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
```
It's a w/a for "Space added before long group names after MIME serialization/deserialization"
issue. DC itself never creates group names with leading/trailing whitespace, so it can be safely
removed. On the sender side there's no trim() because group names anyway go through
improve_single_line_input(). And I believe we should send the exact name we have in our db. Also
there's no check for leading/trailing whitespace because there may be existing user databases with
group names having such whitespaces.
Since switch to async we don't have spurious "database is busy"
errors anymore. Since an error is irrecoverable in most cases,
we can skip the message. The cost of this is we may
accidentally skip a correct message if I/O fails, but
the advantage is that we are guaranteed to never confuse
irrecoverable error with recoverable one and get stuck in
infinite loop redownloading the same message over and over.
That's a bug which @Simon-Laux and probably also @hpk42 had, where one malformed incoming (Spam-) mail blocked the receiving of all emails coming after it.
The problem was that from_field_to_contact_id() returned ContactId::UNDEFINED, and then lookup_by_contact() returned Err.
* Treat multiple From addresses as if there was no From: addr
* changelog
* Don't send invalid emails through the whole receive_imf pipeline
Instead, directly create a trash entry for them.
* Don't create trash entries for randomly generated Message-Id's
* clippy
* fix typo
Co-authored-by: link2xt <link2xt@testrun.org>
If we fetch messages out of order, then f.e. reactions don't work because if we process a reaction not yet having the corresponding message processed, the reaction is thrown away.
* allow deleting referenced contacts in UI
we are quite often getting requests of users
who want to get rid of some contact in the "new chat" list.
there is already a "delete" option,
but it does not work for referenced contacts -
however, it is not obvious for users that a contact is in use,
esp. of some mailing list or larger chat, old contacts, whatever.
this pr revives an old idea [^1] of "soft deleting" referenced contacts -
this way, the user can remove the annoying entry
without the need to understand complicated things
and finally saying that deletion is impossible :)
once the contact is reused, it will reappear,
however, this is already explained in the confirmation dialog of the UIs.
technically, this pr was simpler as expected as we already have
a Origin::Hidden, that is just reused here.
[^1]: https://github.com/deltachat/deltachat-core/pull/542
* update rust doccomment
* update changelog
* avoid races on contact deletion
chats may be created between checking for "no chats" and contact deletion.
this is prevented by putting the statement into an EXCLUSIVE transaction.
* fix failing python test
Note that `IMAP IDLE protocol timed out` was previously
added to the wrong error: not the timeout error (first `?`)
but actual error happened during IDLE, such as a network error.
* Add DC_EVENT_INCOMING_MSG event
* Fix lots of compile errors
* Docs
* Changelog
* Fix python tests
Adding DC_EVENT_INCOMING_MSG_BUNCH made the python tests fail because they use `get_matching("DC_EVENT_INCOMING_MSG")`, which also matches DC_EVENT_INCOMING_MSG_BUNCH, so the tests got confused.
This fixes `get_matching()` to only match whole event names.
* Also fix test_ac_setup_message_twice()
The built regex was ^EVENT_NAME1|EVENT_NAME2$, which becomes parsed as
"^EVENT_NAME1" OR "EVENT_NAME2$". Introduce a group (parentheses) to fix
this.
* desktop will use DC_EVENT_INCOMING_MSG_BUNCH,
so I would not call it experimental anymore
* add generated node constants
* msg_ids in the event as Vec<u32>
number[] in js land
this is way more convinient than a json encoded string.
* Apply suggestions from code review
Co-authored-by: bjoern <r10s@b44t.com>
Co-authored-by: Simon Laux <mobile.info@simonlaux.de>
Co-authored-by: Simon Laux <Simon-Laux@users.noreply.github.com>
Co-authored-by: bjoern <r10s@b44t.com>
Seems like consume_events() didn't work properly, i.e. in some cases it
didn't see the latest events and failed to consume them. So, the
IncomingMsg event from receiving DC_MAILINGLIST stayed in the events
channel, which made this fail:
```rust
// Check that no notification is displayed for blocked mailing list message.
while let Ok(event) = t.evtracker.try_recv() {
assert!(!matches!(event.typ, EventType::IncomingMsg { .. }));
}
```
Fix it by explicitly waiting for the first IncomingMsg event.
The problem was that a message without Autocrypt key or with a wrong
signature resets peerstate regardless of what DKIM check says. I
inserted sleep(1.1) to make sure reset always happens and make the bug
reproducible, then fixed it by forbidding reset if DKIM check fails.
https://github.com/deltachat/deltachat-core-rust/pull/3731
The way to create a Context is now rather burdensome, users have to
create and import a bunch of things just to get a Context. So let's
introduce a builder.
Notice that the builder can only produce an open context, if the
context can not be opened it is dropped. This is on purpose, the
Context itself can become RAII again at some point by doing this.
Only the FFI needs to have the concept of an open and a closed
Context.
Optimised debug builds result in an extremely slow code-build-test
cycle. The reason we do this is because we have a few tests which end
up overflowing the stack. This increases the stack instead.
mio 0.8.5 does not use libc epoll_create1() function on Android anymore,
so it will be possible to revert e29b6f9974
in the next Android release with the new core.
Fix#3507
Note that this is not intended for a release at this point! We first have to test whether it runs stable enough. If we want to make a release while we are not confident enough in authres-checking, then we have to disable it.
BTW, most of the 3000 new lines are in `test_data/messages/dkimchecks...`, not the actual code
da3a4b94 adds the results to the Message info. It currently does this by adding them to `hop_info`. Maybe we should rename `hop_info` to `extra_info` or something; this has the disadvantage that we can't rename the sql column name though.
Follow-ups for this could be:
- In `update_authservid_candidates()`: Implement the rest of the algorithm @hpk42 and me thought about. What's missing is remembering how sure we are that these are the right authserv-ids. Esp., when receiving a message sent from another account at the same domain, we can be quite sure that the authserv-ids in there are the ones of our email server. This will make authres-checking work with buzon.uy, disroot.org, yandex.ru, mailo.com, and riseup.net.
- Think about how we present this to the user - e.g. currently the only change is that we don't accept key changes, which will mean that the small lock on the message is not shown.
- And it will mean that we can fully enable AEAP, after revisiting the security implications of this, and assuming everyone (esp. @link2xt who pointed out the problems in the first place) feels comfortable with it.
* let search_msgs() return unaccepted requests
unaccepted chat requests are shown in the chatlist,
it should be returned by search_msgs() an by the other search functions as well.
form the view of the user, the search acts like a filter,
so there is no reason to hide things additionally.
also, the user may remember a word in a chat request,
maybe even an archived one (there is no need to accept a request before archiving)
that one wants to search later on.
* test searching for unaccepted requests
* simplyfy expression; `c.blocked!=1` is also what is used in similar statements
This way no temporary rows are created and it is easier to maintain
because UPDATE statement is right below the INSERT statement,
unlike `merge_messages` function which is easy to forget about.
* jsonrpc: typescript client: export constants
under `C` enum,
similar to how its exported from `deltachat-node`
* add pr number to changelog
* fix tests
* fix changelog entry position
* set timeout for node ci tests to 10min
set timeout for node ci tests to 10min for the test step,
macOS takes 12min for the whole workflow with cached core build,
so 10min just for the test step should be plenty.
* don't forget to set the limit on windows, too
* jsonrpc in cffi also sends events now
* add pr id to changelog
* jsonrpc: new format for events and better typescript autocompletion (#3663)
* jsonrpc: new format for events and better typescript autocompletion
* adjust doc comments
Very small PR; Motivation: Easier navigation using Go-To-definition.
Because, using go-to-definition of rust-analyzer on parse() doesn't take you to the actual parse() implementation but its trait definiton. On the other hand, it's very easy to find EmailAddress::new().
remove function `messageListGetMessageIds()`,
it is replaced by `getMessageIds()` and `getMessageListEntries()`
the latter returns a new `MessageListItem` type,
which is the now prefered way of using the message list.
All contexts created by the same account manager
share stock string translations. Setting translation on
a single context automatically sets translations for all other
accounts, so it is enough to set translations on the active account.
* jsonrpc js client: ci upload and new name
* Update jsonrpc-client-npm-package.yml
* Update jsonrpc-client-npm-package.yml
* Update jsonrpc-client-npm-package.yml
* change details message
* make sure to generate dist directory
* add method for desktop to get notification relevant information for a message
* add pr number to changelog
* rename MessageNotificationData to MessageNotificationInfo
* enable `BccSelf` by default
enabling `BccSelf` improves user experience as
it is easier to set up another device
and ppl will also see "all" messages in other user agents directly.
for uncounted user problems, after diving into the issue,
the resulting device was "turn on BccSelf".
disabled `BccSelf` was probably the the number one single reason
of user problems.
main drawback of the change are potentially double notifications
when using a shared account and having another mail app on the same device.
however, we meanwhile do not recommend shared accounts at all,
the issue is also fixable by the other mail apps (as done by K-9)
and could be even regarded as a feature (you can decide which app to use for ansering).
but at the end the drawback is probably much smaller than the issues reported above.
* adapt tests to `BccSelf` enabled
* update CHANGELOG
Instead of emitting single MsgsChanged event
with zero chat and msg IDs, emit one event per message.
Also emit WebxdcInstanceDeleted event if expired message
contains a webxdc.
"IMAP folder and UID information" is no longer stored
in the `msgs` table since creation of the `imap` table.
As a result `msgs` table entries do not contain UID information in the
first place.
This commit updates documentation to reflect this change and also
points to `prune_tombstones()` procedure which actually deletes `msgs`
rows.
* truncate incoming messages by lines,
because many linebreaks seem to cause the chat open delay on deltachat-ios
* run cargo fmt
* remove DC_DESIRED_TEXT_LINES_THRESHOLD
and use Strings instead of Cow<str>
* remove usage of clippy::indexing_slicing in truncate_by_lines (#3596)
* adjust comments
* Fix truncate_by_lines tests
* Reword indexing/slicing error
* Remove unnecessary conditional
* Fix a typo in the comment
Co-authored-by: link2xt <link2xt@testrun.org>
update node constants, this was forgotten in #3592
not a big deal as these are generated on building node,
so this is just a simple commit of these files after running `npm run build`
* add more functions, see changelog for details
* add pr number to changelog
* clarify doc comment
* clarify usage of BasicChat
and adjust properties acordingly
r10s is right it should only contain what we need of the expensive calls
* fix doc typos
* run cargo fmt
* jsonrpc: add connectivity functions
* fix typo
* fix typo
* Add get_contact_encryption_info and get_connectivity_html
Fix get_connectivity_html and get_encrinfo futures not being Send. See https://github.com/rust-lang/rust/issues/101650 for more information.
Co-authored-by: jikstra <jikstra@disroot.org>
* Update CHANGELOG
* Update typescript files
* remove todo from changelog
Co-authored-by: jikstra <jikstra@disroot.org>
* add request_internet_access manifest option
* test request_internet_access
* update CHANGELOG
* force warning when internet access is enabled
if internet access is enabled,
show a warning instead of the normal summary
(the internet access is currently mainly to test out integrations
as maps for video chat; the summary is dispensable in the cases currently)
* adapt json-rpc's WebxdcMessageInfo
* get_chat_media() from any chat similar to search_msgs()
* do not return hidden media
this fixes the issue that drafts
and other hidden messages pop up in the gallery.
* add a test for get_chat_media()
* use None instead of ChatId::new(0)
* clarify scope of 'any' in get_chat_media()
* adapt json rpc to changed get_chat_media()
* jsonrpc: chat_get_media turn chat_id into option
and also still allow `0` for dev convenience
(though I'm not totally sure if thats the right decision)
* cargo fmt
Co-authored-by: Simon Laux <mobile.info@simonlaux.de>
Python bindings expect all functions defined in deltachat.h
to be available, even if there is no high-level interface.
scripts/run-python-test.sh doesn't work without this.
* integrate json-rpc repo
https://github.com/deltachat/deltachat-jsonrpc
* get target dir from cargo
* fix clippy
* use node 16 in ci
use `npm i` instead of `npm ci`
try fix ci script
and fix a doc comment
* fix get_provider_info docs
* refactor function name
* fix formatting
make test pass
fix clippy
* update .gitignore
* change now returns event names as id
directly, no conversion method or number ids anymore
also longer timeout for requesting test accounts from mailadm
* fix compile after rebase
* add json api to cffi and expose it in dc node
* add some files to npm ignore
that don't need to be in the npm package
* add jsonrpc crate to set_core_version
* add jsonrpc feature flag
* call a jsonrpc function in segfault example
* break loop on empty response
* fix closing segfault
thanks again to link2xt for figguring this out
* activate other tests again
* remove selectAccount from highlevel client
* put jsonrpc stuff in own module
* disable jsonrpc by default
* add @deltachat/jsonrpc-client
to make sure its dependencies are installed, too
whwn installing dc-node
* commit types.ts
that dc-node has everything it needs to provide @deltachat/jsonrpc-client
without an extra ts compile step
* improve naming
* Changes for tokio compat, upgrade to yerpc 0.3
This also changes the webserver binary to use axum in place of tide.
* Improvements to typescript package
* Improve docs.
* improve docs, fix example
* Fix CFFI for JSON-RPC changes
* use stable toolchain not 1.56.0
* fix ci
* try to fix ci
* remove emtpy file
allow unused code for new_from_arc
* expose anyhow errors
feature name was wrong
* use multi-threaded runtime in JSON-RPC webserver
* improve test setup and code style
* don't wait for IO on webserver start
* Bump yerpc to 0.3.1 with fix for axum server
* update todo document
remove specific api stuff for now,
we now have the an incremental aproach on moving
not the all at-once effort I though it would be
* remove debug logs
* changelog entry about the jsonrpc
* Fix method name casings and cleanups
* Improve JSON-RPC CI, no need to build things multiple times
* Naming consistency: Use DeltaChat not Deltachat
* Improve documentation
* fix docs
* adress dig's comments
- description in cargo.toml
- impl From<EventType> for EventTypeName
- rename `CommandApi::new_from_arc` -> `CommandApi::from_arc`
- pre-allocate if we know the entry count already
- remove unused enumerate
- remove unused serde attribute comment
- rename `FullChat::from_dc_chat_id` -> `FullChat::try_from_dc_chat_id`
* make it more idiomatic:
rename `ContactObject::from_dc_contact -> `ContactObject::try_from_dc_contact`
* apply link2xt's suggestions:
- unref jsonrpc_instance in same thread it was created in
- increase `max_queue_size` from 1 to 1000
* reintroduce segfault test script
* remove unneeded context
thanks to link2xt for pointing that out
* Update deltachat-ffi/deltachat.h
Co-authored-by: bjoern <r10s@b44t.com>
* Update deltachat-ffi/deltachat.h
Co-authored-by: bjoern <r10s@b44t.com>
* make sure to use dc_str_unref instead of free
on cstrings returned/owned by rust
* Increase online test timeouts for CI
* fix the typos
thanks to ralphtheninja for finding them
* restore same configure behaviour as desktop:
make configure restart io with the old configuration if it had one on error
* found another segfault:
this time in batch_set_config
* remove print from test
* make dcn_json_rpc_request return undefined instead of not returning
this might have been the cause for the second segfault
* add set_config_from_qr to jsonrpc
* add `add_device_message` to jsonrpc
* jsonrpc: add `get_fresh_msgs` and `get_fresh_msg_cnt`
* jsonrpc: add dm_chat_contact to ChatListItemFetchResult
* add webxdc methods to jsonrpc:
- `webxdc_send_status_update`
- `webxdc_get_status_updates`
- `message_get_webxdc_info`
* add `chat_get_media` to jsonrpc
also add viewtype wrapper enum and use it in `MessageObject`,
additionally to using it in `chat_get_media`
* use camelCase in all js object properties
* Add check_qr function to jsonrpc
* Fixed clippy errors and formatting
* Fixed formatting
* fix changelog ordering after rebase
* fix compile after merging in master branch
Co-authored-by: Simon Laux <mobile.info@simonlaux.de>
Co-authored-by: Simon Laux <Simon-Laux@users.noreply.github.com>
Co-authored-by: bjoern <r10s@b44t.com>
Co-authored-by: flipsimon <28535045+flipsimon@users.noreply.github.com>
* improve error handling for account setup from qrcode
closes#3192
* replace issue id with pr id in changelog
* Update src/qr.rs
Co-authored-by: bjoern <r10s@b44t.com>
* show response when it's invalid in success case, too
* fix changelog entry
Co-authored-by: bjoern <r10s@b44t.com>
We install `tox` via `pipx` in `Dockerfile`, there is no need to
configure path to `python3.7`.
Also remove use of `pushd`/`popd` and switch from `bash` to `/bin/sh`.
#3491 introduced a bug that your address is only replaced in the first group you write to, which was rather hard to fix. In order to be able to release something, we agreed to revert it and instead only replace the contacts in verified groups (and in broadcast lists, if the signing key is verified).
Highlights:
* Revert "Only do the AEAP transition in the chat where it happened"
This reverts commit 22f4cd7b79.
* Only do the transition for verified groups (and broadcast lists)
To be exact, only do the transition if the signing key fingerpring is
verified. And only do it in verified groups and broadcast lists
* Slightly adapt string to this change
* Changelog
musllinux images miss PyPy interpreters,
we want to skip building PyPy wheels for musl
instead of failing the build.
Also remove workaround from CI scripts.
Wheels are now published to PyPI, recommend it instead of devpi. We
build the wheels only for releases anyway.
Suggest using tox instead of listing all the pytest dependencies to
avoid keeping them up to date in the readme.
We no longer publish `docker/coredeps` images, they cannot be
pulled from a container registry anymore.
Troubleshooting section is outdated, because vsyscall emulation is
only needed for manylinux2010 images, not manylinux2014 or
musllinux_1_1 images.
Mention Podman as an alternative to Docker.
Link to https://py.delta.chat/ instead of only examples.
Remove note about wheels for Mac and Windows. Nobody requests them
anyway.
* save webxdc-updates for not yet downloaded messages, that are probably webxdc instances then
* test webxdc updates received while instance is not yet downloaded
* keep msg_id on downloading messages
keeping msg_id on downloading messages
has the advantage that webxdc updates and other references to the msg_id
can be processed as usual.
if a message expands to multiple msg_id,
the last one is kept,
however, this does not affect webxdc at all.
(alternatives may be to update `msgs_status_updates`
but that seems more complicated and even less elegant,
another alternative would be to use different keys (eg. `rfc274_mid`),
but that also seems not to be much easier and would waste space as well.
also both alternatives would need adaption for other foreign keys)
* update CHANGELOG
* do not emit WebxdcStatusUpdate event in case the message is not yet downloaded
* move DELETE/UPDATE to an transaction
* make merge_msg_id() a little less confusing
* use some webxdc-update-param from placeholder
(the placeholder may be updated,
the just downloaded messages is not)
* more precise function name
* test not directly downloading status updates
* test not directly downloading mdn
* do not `SELECT timestamp` if not used
ordering is by `id` since #2364, selecting `timestamp` is not needed.
(came over this when keeping `id` on downloading in #3487 -
had in mind there was sth. special with ids ...
however, the assumption of #2364 is even more true with #3487 -
before, new (and then maybe much larger) ids were inserted
and could result in wrong search result ordering)
* remove another unused `SELECT timestamp`
Remove unused docker-doxygen Dockerfile.
Switch scripts/run-doxygen.sh from bash to sh.
Use docker.io/alpine image instead of unsupported hrektts/doxygen
It happened multiple times now that I wanted to quickly execute a test, but because of a warning that had become an error, it didn't execute.
This turns warnings into warnings again; our CI will fail if there is a warning, anyway (because of RUSTFLAGS: -Dwarnings)
* ignore status/footer updates from mailinglist messages
mailinglist software often modified existing footers
or adds footers on their own.
therefore,therefore, these footers often do not reflect the status/footer set by the user.
* test status footers not updated from mailinglists
some mailinglists have their name in square brackets prepended to the subject.
we pick up that name and remove the square brackets,
as these do not look good as the chat name in delta chat.
if a mailing list has a sequence of square brackets, we use all of them
(this seems to be okayish, at least i do not know of any complains).
however, the removal of square brackets was not nice for sequences,
resulting in `ml topic] [sub topic`.
this pr removes the square brackets only for the first name
and leave the other ones as is.
* more flexible render_webxdc_status_update_object()
* delay webxdc updates when ratelimit is reached
* inject updates messages to the SMTP loop as needed
this avoids starting several tasks
and allows sending updates out after a restart of the app.
* use mutex to prevent race conditions in status updates
* check ratelimiter only before the sending loop; it won't change until messages are actually sent out
* fix typo
* prefer standard type declaration over turbofish syntax
* use UNIQUE and ON CONFLICT for query smtp_status_updates
* combine DELETE+SELECT to one atomic statement
* as all operations on smtp_status_updates are now atomic, a mutex is no longer needed
* test DELETE+RETURNING statement
* simplify calls to can_send()
* comment about ratelimit boolean in send_smtp_messages()
This makes quotes created by user display properly in other MUAs like
Thunderbird and not start with space in MUAs that don't support format=flowed.
To distinguish user-created quotes from top-quote inserted by Delta
Chat, a newline is inserted if there is no top-quote and the first
line starts with ">".
Nested quotes are not supported, e.g. line starting with "> >" will
start with ">" on the next line if wrapped.
* do not use ratelimiter for bots
i tried some other approaches as having optional ratelimiter
or handle `can_send()` for bots differently,
but all that results in _far_ more code and/or indirections -
esp. as the "bot" config can change and is also persisted -
and the ratelimiter is created
at a point where the database is not yet available ...
of course, all that could be refactored as well,
but this two-liner also does the job :)
* update CHANGELOG
This allows account manager to construct a single event channel and
inject it into all created contexts instead of aggregating events from
separate event emitters.
This makes sure that under normal circumstances the LoginParam struct
is always fully validated, ensure future use does not have to be
careful with this.
The brittle handling of `server_flags` is also abstraced away from
users of it and is now handled entirely internally, as the flags is
really only a boolean a lot of the flag parsing complexity is removed.
The OAuth2 flag is moved into the ServerLoginParam struct as it really
belongs in there.
New ratelimiter module counts number of sent messages and calculates
the time until more messages can be sent.
Rate limiter is currently applied only to MDNs. Other messages are
sent without rate limiting even if quota is exceeded, but MDNs are not
sent until ratelimiter allows sending again.
* force a reason when calling `set_msg_failed()`
the string is displayed to the user,
so even _some_ context as "NDN without further details"
is better than an empty string.
* make clippy happy
* add CHANGELOG entry
* clarify webxdc reference wrt info-messages
* add from_id parameter to add_info_msg_with_cmd()
* flag webxdc-info-messages as such
* set from_id to sender for webxdc-info-messages
* test additional webxdc info properties
* do not add series of similar info messages
instead, if on adding the last info message
is already from the same webxdc and sender,
just update the text
* test cleanup of webxdc info messages series
* update changelog
* make clippy happy
there is no real complexity in the args,
so allowing one more arg is probably fine.
if really wanted, we can refactor the function in another pr;
this pr is already complex enough :)
* use cleaner function names and comments
* clarify CHANGELOG
Not completely sure it's worth it since some other dependencies still
depend on it. Anyway, proc macros are said to be bad for compile times, I just typed out what the proc macro generates and it's only 8 more lines, and we're already doing it this way in e.g. action_by_contact() and collect_texts_recursive() (the latter needs the boxed future both for the trait and for recursion).
* do not wipe info for drafts
* drafts and received webxdc-instances, resent or not, do not trigger info-messages
* test that resending a webxdc does not not add legacy info-messages
* make clippy happy
* update CHANGELOG
bcc_self-updates are not received
due to the normal prevention of not even downloading these messages.
there is no extra defense of that on webxdc-level;
status-updates do not even have a "global-unique" id
(they have a local id and the message where they're wrappted into have the
"global-unique" Message-Id)
I added this poison_sender and poison_receiver stuff back when we had event listeners (which were called "sinks", too, but anyway), i.e. user-definable closures that were run in the events loop. This was to make sure that the test fails if the closure panics. But since we don't have them anymore and this code isn't supposed to panic anyway:
```rust
while let Some(event) = events.recv().await {
for sender in senders.read().await.iter() {
sender.try_send(event.clone()).ok();
}
}
```
* test contact name changes applied everywhere
this test is failing on current master,
a changed authname is set to `contact.authname` but not cached at `chat.name`;
resulting in `dc_chat_get_name()` returning a name
undiscoverable by `dc_get_chatlist(name)`.
* fix: update chat.name on contact.authname changes
* do read contact.display_name from database only of chat.name is empty
* update CHANGELOG
This ensures that no invalid states are possible,
like the one where cancel channel exists, but
no ongoing process is running, or the one where
the ongoing process is not allocated, but
should not be stopped.
mimeparser now handles try_decrypt() errors instead of simply logging
them. If try_decrypt() returns an error, a single message bubble
with an error is added to the chat.
The case when encrypted part is found in a non-standard MIME structure
is not treated as an encryption failure anymore. Instead, encrypted
MIME part is presented as a file to the user, so they can download the
part and decrypt it manually.
Because try_decrypt() errors are handled by mimeparser now,
try_decrypt() was fixed to avoid trying to load private_keyring if the
message is not encrypted. In tests the context receiving message
usually does not have self address configured, so loading private
keyring via Keyring::new_self() fails together with the try_decrypt().
Skipping of all [Gmail] folders was introduced to avoid scanning
virtual "[Gmail]/All Mail" folder. However, it also skips Sent and
Spam folders which reside inside [Gmail]. As a result
configured_sentbox_folder becomes unset after folder scan, making it
impossible to watch Sent folder, and Spam folder is never scanned.
This change makes Delta Chat identify virtual Gmail folders by their
flags, so virtual folders are skipped while Sent and Spam folders are
still scanned.
* don't start workflow on py-*tag
* move node.js tests to separate action
* node.js CI: move PR ID and tags to environment variables
* node.js CI: small bracket mistake
* delete prebuilds.tar.gz before packaging
* use node/README.md as npm README
* Rename aeap-mvp.rst to aeap-mvp.md
* Update aeap-mvp.md
* Apply suggestions from hpk's review
Co-authored-by: holger krekel <holger@merlinux.eu>
* Additions to holger's review
* Decide on TODOs
* Drop the "If we are going to assign a message to a chat, but the sender is not a member of this chat" condition again
Co-authored-by: holger krekel <holger@merlinux.eu>
The problem was in the handle_fingerprint_change() function which
attempted to add a warning about fingerprint change to all chats with
the contact.
This failed because of the SQL query failing to find the contact for
self in the `contacts` table. So the warning was not added, but at the
same time the whole decryption process failed.
The fix is to skip handle_fingerprint_change() for self addreses.
If there are no MOVE/DELETE operations pending, the folder
should not be SELECTed.
When `only_fetch_mvbox` is enabled, `fetch_new_messages` skips
most folders, but `move_delete_messages` always selects the
folder even if there are no operations pending. Selecting
all folders wastes time during folder scan and turns
recent messages into non-recent.
* make upload to previews fail if it's executed on a tag not a PR
* node-package.yml: use tag in uploaded file name
* rename file to node-§tag.tar.gz before upload
* get rid of bash error?
in fact, `get_chat_msgs()` with `marker1before` is not used on iOS,
however, iOS still needs the special chat-id.
in #3274 we did not check this possibility.
it may be changed, of course, however, that would requore some refactorings
in an anyway complicated area and is probably not worth the effort.
Hold the same write lock while checking if ongoing
process is already allocated and while allocating it.
Otherwise it is possible for two parallel processes
running alloc_ongoing() to decide that no ongoing
process is allocated and allocate two ongoing processes.
* create same contact-colors when addresses differ in upper-/lowercase
this leaves group-colors based on group names as is,
so, "MY GROUP" creates a different color than "my group",
as these names are better synced and also not an ID in this sense,
this is probably fine here.
(also when looking at the examples from
https://xmpp.org/extensions/xep-0392.html#testvectors-fullrange-no-cvd ,
case-sensistifity for group names seems to be fine)
* add a test for upper-/lowercase in group names
* update CHANGELOG
Google Workspace has an option "Append footer" which appends standard
footer defined by administrator to all outgoing messages. However,
there is no plain text part in encrypted messages sent by Delta Chat,
so Google Workspace turn the message into multipart/mixed MIME, where
the first part is an empty plaintext part with a footer and the second
part is the original encrypted message.
This commit makes Delta Chat attempt to repair such messages,
similarly to how it already repairs "Mixed Up" MIME structure in
`get_mixed_up_mime`.
don't ignore core sourcefiles,
prevented npm installation on architectures with no prebuild
don't run lint checks on windows
github actions don't like double quotes apparently
minimize node.js CI
update ubuntu runner to 22.04
README: update link to node bindings source
simplify link in readme
node: fix crash with invalid account id
(throw error when getContext failed)
fix typo in readme
remove node specific changelog
change prebuild machine back to ubuntu 18.04
move package.json to root level to include rust source in npm package
change path in m1 patch
github action to upload to download.delta.chat/node/ on tag
try build with ubuntu 20.04
Update node/README.md
try building it with newer ubuntu because it wants glibc 2.33
fix path for prebuildify script
throw error when instanciating a wrapper class on `null`
(Context, Message, Chat, ChatList and so on)
try fix selecting the right cache
to fix the strange glibc bug
also revert back ubuntu version to 18.04
also bump package.json version with release script
fix paths since we moved around package.json
github action: fix path
document npm release - it's so much easier now!
there are no PR checks to post to if this action is executed on a tag
github action: fix artifact names
fix paths? wtf do I know, it's 3AM and I'm drunk
fix syntax error
don't upload preview if action is run on tag
is the tag ID an empty string or null?
node-package github action is done so far
also include scripts in package
only publish docs on push to master branch
actually bump package.json version in set_core_version script
prettify package.json
fix test - we don't really need to assert that
remove unnecessary ls statement from github action
adjust scripts to new location of deltachat-core-rust
adjust dc-node readme to repo change
mention old repository
migrate github actions, try out if they work
fix path to node docs in SSH github action
passing mailadm token to node tests
hopefully fixing the download paths for the artifacts
fixing download paths, this time for real
post upload link to details
fix scp command
forgot to remove platform_status dict
fixing paths in the github action
add github action to delete node preview builds when PR is closed
move environment variable to yaml
remove git trash
github action to release to npm
use different action which also works with branches for testing
we don't want to publish to NPM through the CI
see what lint issues windows has
Scheduler has no Stopped state anymore. If Scheduler exists, it is
always started. Scheduler is stopped via Scheduler.stop(), which
consumes Scheduler and cannot fail.
Unlike jobs which are executed before sending normal messages, MDNs
from `smtp_mdns` table are sent after sending messages from `smtp`
table. This way normal messages have higher priority than MDNs.
There are no SMTP jobs anymore. All jobs are IMAP jobs, so
`jobs.thread` column is not used anymore.
.expect() may panic, which is probably not what we want here.
it seems better to bubble up the error (as we are doing in the other cases)
(i was checking some .expect usages after we had a similar issue at #3264)
* tagger.put_raw() has changes sematics and escapes strings on its own now
an explicit escaping leads to double escaping and to wrong display.
this should also improve lenght calculation,
as a quote and other specials counts as 1 character and not as 4-6.
* test encoding of generated qr-code-svg
* streamline function argument wording
use qrcode_description instead of raw_qrcode_description -
there is nothing "raw" in the argument,
it is a string as used throughout the app.
i was wondering mainly about the whole line
"Failed to start IO: scheduler is already stopped".
i think it has to be "already started" and not "already stopped".
Hold scheduler lock during the whole procedure of scheduler starting
and stopping. This ensures that two processes can't get two read locks
in parallel and start loops or send the stop signal twice.
Also remove shutdown channels: it is enough to wait
for the loop handle without receiving a shutdown signal
from the end of the loop.
- "full message view" not needed because of footers that go to contact status #4151
- Pick up system's light/dark mode in generated message HTML #4150
- Support non-persistent configuration with DELTACHAT_* env
- Print deltachat-repl errors with causes. #4166
- Increase MSRV to 1.64. #4167
- Core takes care of stopping and re-starting IO itself where needed,
e.g. during backup creation. It is no longer needed to call
dc_stop_io(). dc_start_io() can now be called at any time without
harm. #4138
- More accurate maybe_add_bcc_self device message text #4175
### Fixes
- Fix segmentation fault if `dc_context_unref()` is called during
background process spawned by `dc_configure()` or `dc_imex()`
or `dc_jsonrpc_instance_t` is unreferenced
during handling the JSON-RPC request. #4153
- Delete expired messages using multiple SQL requests. #4158
- Do not emit "Failed to run incremental vacuum" warnings on success. #4160
- Ability to send backup over network and QR code to setup second device #4007
- Disable buffering during STARTTLS setup. #4190
## [1.111.0] - 2023-03-05
### Changes
- Make smeared timestamp generation non-async. #4075
- Set minimum TLS version to 1.2. #4096
- Run `cargo-deny` in CI. #4101
- Check provider database with CI. #4099
- Switch to DEFERRED transactions #4100
### Fixes
- Do not block async task executor while decrypting the messages. #4079
- Housekeeping: delete the blobs backup dir #4123
### API-Changes
- jsonrpc: add more advanced API to send a message. #4097
- jsonrpc: add get webxdc blob API `getWebxdcBlob`#4070
## 1.110.0
### Changes
- use transaction in `Contact::add_or_lookup()`#4059
- Organize the connection pool as a stack rather than a queue to ensure that
connection page cache is reused more often.
This speeds up tests by 28%, real usage will have lower speedup. #4065
- Use transaction in `update_blocked_mailinglist_contacts`. #4058
- Remove `Sql.get_conn()` interface in favor of `.call()` and `.transaction()`. #4055
- Updated provider database.
- Disable DKIM-Checks again #4076
- Switch from "X.Y.Z" and "py-X.Y.Z" to "vX.Y.Z" tags. #4089
- mimeparser: handle headers from the signed part of unencrypted signed message #4013
### Fixes
- Start SQL transactions with IMMEDIATE behaviour rather than default DEFERRED one. #4063
- Fix a problem with Gmail where (auto-)deleted messages would get archived instead of deleted.
Move them to the Trash folder for Gmail which auto-deletes trashed messages in 30 days #3972
- Clear config cache after backup import. This bug sometimes resulted in the import to seemingly work at first. #4067
- Update timestamps in `param` columns with transactions. #4083
### API-Changes
## 1.109.0
### Changes
- deltachat-rpc-client: use `dataclass` for `Account`, `Chat`, `Contact` and `Message`#4042
### Fixes
- deltachat-rpc-server: do not block stdin while processing the request. #4041
deltachat-rpc-server now reads the next request as soon as previous request handler is spawned.
- Enable `auto_vacuum` on all SQL connections. #2955
- Replace `r2d2` connection pool with an own implementation. #4050#4053#4043#4061
This change improves reliability
by closing all database connections immediately when the context is closed.
### API-Changes
- Remove `MimeMessage::from_bytes()` public interface. #4033
- BREAKING Types: jsonrpc: `get_messages` now returns a map with `MessageLoadResult` instead of failing completely if one of the requested messages could not be loaded. #4038
- Add `dc_msg_set_subject()`. C-FFI #4057
- Mark python bindings as supporting typing according to PEP 561 #4045
## 1.108.0
### Changes
- Use read/write timeouts instead of per-command timeouts for SMTP #3985
- Cache DNS results for SMTP connections #3985
- Prefer TLS over STARTTLS during autoconfiguration #4021
- Use SOCKS5 configuration for HTTP requests #4017
- Show non-deltachat emails by default for new installations #4019
- Re-enabled SMTP pipelining after disabling it in #4006
### Fixes
- Fix Securejoin for multiple devices on a joining side #3982
- python: handle NULL value returned from `dc_get_msg()`#4020
Account.`get_message_by_id` may return `None` in this case.
### API-Changes
- Remove bitflags from `get_chat_msgs()` interface #4022
C interface is not changed.
Rust and JSON-RPC API have `flags` integer argument
replaced with two boolean flags `info_only` and `add_daymarker`.
- jsonrpc: add API to check if the message is sent by a bot #3877
## 1.107.1
### Changes
- Log server security (TLS/STARTTLS/plain) type #4005
### Fixes
- Disable SMTP pipelining #4006
## 1.107.0
### Changes
- Pipeline SMTP commands #3924
- Cache DNS results for IMAP connections #3970
### Fixes
- Securejoin: Fix adding and handling Autocrypt-Gossip headers #3914
- fix verifier-by addr was empty string instead of None #3961
- Emit DC_EVENT_MSGS_CHANGED for DC_CHAT_ID_ARCHIVED_LINK when the number of archived chats with
unread messages increases #3959
- Fix Peerstate comparison #3962
- Log SOCKS5 configuration for IMAP like already done for SMTP #3964
- Fix SOCKS5 usage for IMAP #3965
- Exit from recently seen loop on interrupt channel errors to avoid busy looping #3966
### API-Changes
- jsonrpc: add verified-by information to `Contact`-Object
- Remove `attach_selfavatar` config #3951
### Changes
- add debug logging support for webxdcs #3296
## 1.106.0
### Changes
- Only send IncomingMsgBunch if there are more than 0 new messages #3941
### Fixes
- fix: only send contact changed event for recently seen if it is relevant (not too old to matter) #3938
- Immediately save `accounts.toml` if it was modified by a migration from absolute paths to relative paths #3943
- Do not treat invalid email addresses as an exception #3942
- Add timeouts to HTTP requests #3948
## 1.105.0
### Changes
- Validate signatures in try_decrypt() even if the message isn't encrypted #3859
- Don't parse the message again after detached signatures validation #3862
- Move format=flowed support to a separate crate #3869
- cargo: bump quick-xml from 0.23.0 to 0.26.0 #3722
- Add fuzzing tests #3853
- Add mappings for some file types to Viewtype / MIME type #3881
- Buffer IMAP client writes #3888
- move `DC_CHAT_ID_ARCHIVED_LINK` to the top of chat lists
and make `dc_get_fresh_msg_cnt()` work for `DC_CHAT_ID_ARCHIVED_LINK`#3918
- make `dc_marknoticed_chat()` work for `DC_CHAT_ID_ARCHIVED_LINK`#3919
- Update provider database
### API-Changes
- jsonrpc: add python API for webxdc updates #3872
- jsonrpc: add fresh message count to ChatListItemFetchResult::ArchiveLink
- Add ffi functions to retrieve `verified by` information #3786
- resultify `Message::get_filebytes()`#3925
### Fixes
- Do not add an error if the message is encrypted but not signed #3860
- Do not strip leading spaces from message lines #3867
- Fix uncaught exception in JSON-RPC tests #3884
- Fix STARTTLS connection and add a test for it #3907
- Trigger reconnection when failing to fetch existing messages #3911
- Do not retry fetching existing messages after failure, prevents infinite reconnection loop #3913
- Ensure format=flowed formatting is always reversible on the receiver side #3880
## 1.104.0
### Changes
- Don't use deprecated `chrono` functions #3798
- Document accounts manager #3837
- If a classical-email-user sends an email to a group and adds new recipients,
add the new recipients as group members #3781
- Remove `pytest-async` plugin #3846
- Only send the message about ephemeral timer change if the chat is promoted #3847
- Use relative paths in `accounts.toml`#3838
### Fixes
- Set read/write timeouts for IMAP over SOCKS5 #3833
- Treat attached PGP keys as peer keys with mutual encryption preference #3832
- fix migration of old databases #3842
- Fix cargo clippy and doc errors after Rust update to 1.66 #3850
- Don't send GroupNameChanged message if the group name doesn't change in terms of
`improve_single_line_input()`#3852
- Prefer encryption for the peer if the message is encrypted or signed with the known key #3849
## 1.103.0
### Changes
- Disable Autocrypt & Authres-checking for mailing lists,
because they don't work well with mailing lists #3765
- Refactor: Remove the remaining AsRef<str>#3669
- Add more logging to `fetch_many_msgs` and refactor it #3811
- Small speedup #3780
- Log the reason when the message cannot be sent to the chat #3810
- Add IMAP server ID line to the context info only when it is known #3814
- Remove autogenerated typescript files #3815
- Move functions that require an IMAP session from `Imap` to `Session`
to reduce the number of code paths where IMAP session may not exist.
Drop connection on error instead of trying to disconnect,
potentially preventing IMAP task from getting stuck. #3812
### API-Changes
- Add Python API to send reactions #3762
- jsonrpc: add message errors to MessageObject #3788
- jsonrpc: Add async Python client #3734
### Fixes
- Make sure malformed messages will never block receiving further messages anymore #3771
- strip leading/trailing whitespace from "Chat-Group-Name{,-Changed}:" headers content #3650
- Assume all Thunderbird users prefer encryption #3774
- refactor peerstate handling to ensure no duplicate peerstates #3776
- Fetch messages in order of their INTERNALDATE (fixes reactions for Gmail f.e.) #3789
- python: do not pass NULL to ffi.gc if the context can't be created #3818
- Add read/write timeouts to IMAP sockets #3820
- Add connection timeout to IMAP sockets #3828
- Disable read timeout during IMAP IDLE #3826
- Bots automatically accept mailing lists #3831
## 1.102.0
### Changes
- If an email has multiple From addresses, handle this as if there was
no From address, to prevent from forgery attacks. Also, improve
handling of emails with invalid From addresses in general #3667
### API-Changes
### Fixes
- fix detection of "All mail", "Trash", "Junk" etc folders. #3760
- fetch messages sequentially to fix reactions on partially downloaded messages #3688
- Fix a bug where one malformed message blocked receiving any further messages #3769
## 1.101.0
### Changes
- add `configured_inbox_folder` to account info #3748
-`dc_delete_contact()` hides contacts if referenced #3751
- emit "contacts changed" event when the contact is no longer "seen recently" #3703
- do not allow peerstate reset if DKIM check failed #3731
## 1.98.0
### API-Changes
- jsonrpc: typescript client: export constants under `C` enum, similar to how its exported from `deltachat-node`#3681
- added reactions support #3644
- jsonrpc: reactions: added reactions to `Message` type and the `sendReaction()` method #3686
### Changes
- simplify `UPSERT` queries #3676
### Fixes
## 1.97.0
### API-Changes
- jsonrpc: add function: #3641, #3645, #3653
-`getChatContacts()`
-`createGroupChat()`
-`createBroadcastList()`
-`setChatName()`
-`setChatProfileImage()`
-`downloadFullMessage()`
-`lookupContactIdByAddr()`
-`sendVideochatInvitation()`
-`searchMessages()`
-`messageIdsToSearchResults()`
-`setChatVisibility()`
-`getChatEphemeralTimer()`
-`setChatEphemeralTimer()`
-`getLocations()`
-`getAccountFileSize()`
-`estimateAutoDeletionCount()`
-`setStockStrings()`
-`exportSelfKeys()`
-`importSelfKeys()`
-`sendSticker()`
-`changeContactName()`
-`deleteContact()`
-`joinSecurejoin()`
-`stopIoForAllAccounts()`
-`startIoForAllAccounts()`
-`startIo()`
-`stopIo()`
-`exportBackup()`
-`importBackup()`
-`getMessageHtml()`#3671
-`miscGetStickerFolder` and `miscGetStickers`#3672
- breaking: jsonrpc: remove function `messageListGetMessageIds()`, it is replaced by `getMessageIds()` and `getMessageListItems()` the latter returns a new `MessageListItem` type, which is the now preferred way of using the message list.
- jsonrpc: add type: #3641, #3645
-`MessageSearchResult`
-`Location`
- jsonrpc: add `viewType` to quoted message(`MessageQuote` type) in `Message` object type #3651
### Changes
- Look at Authentication-Results. Don't accept Autocrypt key changes
if they come with negative authentiation results while this contact
sent emails with positive authentication results in the past. #3583
- jsonrpc in cffi also sends events now #3662
- jsonrpc: new format for events and better typescript autocompletion
- Join all "[migration] vXX" log messages into one
### Fixes
- share stock string translations across accounts created by the same account manager #3640
- suppress welcome device messages after account import #3642
- fix unix timestamp used for daymarker #3660
## 1.96.0
### Changes
- jsonrpc js client:
- Change package name from `deltachat-jsonrpc-client` to `@deltachat/jsonrpc-client`
- remove relative file dependency to it from `deltachat-node` (because it did not work anyway and broke the nix build of desktop)
- ci: add github ci action to upload it to our download server automatically on release
## 1.95.0
### API-Changes
- jsonrpc: add `mailingListAddress` property to `FullChat`#3607
- over jsonrpc built with napi.rs: \[[📂 source](https://github.com/deltachat/napi-jsonrpc) | [📦 npm](https://www.npmjs.com/package/@deltachat/napi-jsonrpc)\]
[^1]: Out of date / unmaintained, if you like those languages feel free to start maintaining them. If you have questions we'll help you, please ask in the issues.
This crate provides a [JSON-RPC 2.0](https://www.jsonrpc.org/specification) interface to DeltaChat.
The JSON-RPC API is exposed in two fashions:
* A executable that exposes the JSON-RPC API through a [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) server running on localhost.
* The JSON-RPC API can also be called through the [C FFI](../deltachat-ffi). The C FFI needs to be built with the `jsonrpc` feature. It will then expose the functions `dc_jsonrpc_init`, `dc_jsonrpc_request`, `dc_jsonrpc_next_response` and `dc_jsonrpc_unref`. See the docs in the [header file](../deltachat-ffi/deltachat.h) for details.
We also include a JavaScript and TypeScript client for the JSON-RPC API. The source for this is in the [`typescript`](typescript) folder. The client can easily be used with the WebSocket server to build DeltaChat apps for web browsers or Node.js. See the [examples](typescript/example) for details.
## Usage
#### Running the WebSocket server
From within this folder, you can start the WebSocket server with the following command:
```sh
cargo run --features webserver
```
If you want to use the server in a production setup, first build it in release mode:
```sh
cargo build --features webserver --release
```
You will then find the `deltachat-jsonrpc-server` executable in your `target/release` folder.
The executable currently does not support any command-line arguments. By default, once started it will accept WebSocket connections on `ws://localhost:20808/ws`. It will store the persistent configuration and databases in a `./accounts` folder relative to the directory from where it is started.
The server can be configured with environment variables:
|variable|default|description|
|-|-|-|
|`DC_PORT`|`20808`|port to listen on|
|`DC_ACCOUNTS_PATH`|`./accounts`|path to storage directory|
If you are targeting other architectures (like KaiOS or Android), the webserver binary can be cross-compiled easily with [rust-cross](https://github.com/cross-rs/cross):
The package includes a JavaScript/TypeScript client which is partially auto-generated through the JSON-RPC library used by this crate ([yerpc](https://github.com/Frando/yerpc/)). Find the source in the [`typescript`](typescript) folder.
To use it locally, first install the dependencies and compile the TypeScript code to JavaScript:
```sh
cd typescript
npm install
npm run build
```
The JavaScript client is not yet published on NPM (but will likely be soon). Currently, it is recommended to vendor the bundled build. After running `npm run build` as documented above, there will be a file `dist/deltachat.bundle.js`. This is an ESM module containing all dependencies. Copy this file to your project and import the DeltaChat class.
```typescript
import{DeltaChat}from'./deltachat.bundle.js'
constdc=newDeltaChat('ws://localhost:20808/ws')
constaccounts=awaitdc.rpc.getAllAccounts()
console.log('accounts',accounts)
```
A script is included to build autogenerated documentation, which includes all RPC methods:
```sh
cd typescript
npm run docs
```
Then open the [`typescript/docs`](typescript/docs) folder in a web browser.
## Development
#### Running the example app
We include a small demo web application that talks to the WebSocket server. It can be used for testing. Feel invited to expand this.
```sh
cd typescript
npm run build
npm run example:build
npm run example:start
```
Then, open [`http://localhost:8080/example.html`](http://localhost:8080/example.html) in a web browser.
Run `npm run example:dev` to live-rebuild the example app when files changes.
### Testing
The crate includes both a basic Rust smoke test and more featureful integration tests that use the TypeScript client.
#### Rust tests
To run the Rust test, use this command:
```
cargo test
```
#### TypeScript tests
```
cd typescript
npm run test
```
This will build the `deltachat-jsonrpc-server` binary and then run a test suite against the WebSocket server.
The test suite includes some tests that need online connectivity and a way to create test email accounts. To run these tests, talk to DeltaChat developers to get a token for the `testrun.org` service, or use a local instance of [`mailadm`](https://github.com/deltachat/docker-mailadm).
Then, set the `DCC_NEW_TMP_EMAIL` environment variable to your mailadm token before running the tests.
```
DCC_NEW_TMP_EMAIL=https://testrun.org/new_email?t=yourtoken npm run test
```
#### Test Coverage
Running `npm run test` will report test coverage. For the coverage to be accurate the online tests need to be run.
> If you are offline and want to see the coverage results anyway (even though they are inaccurate), you can bypass the errors of the online tests by setting the `COVERAGE_OFFLINE=1` environment variable.
A summary of the coverage will be reported in the terminal after the test run. Open `coverage/index.html` in a web browser for a detailed report.
- [ ] different test type to simulate two devices: to test autocrypt_initiate_key_transfer & autocrypt_continue_key_transfer
## MVP - Websocket server&client
For kaiOS and other experiments, like a deltachat "web" over network from an android phone.
- [ ] coverage for a majority of the API
- [ ] Blobs served
- [ ] Blob upload (for attachments, setting profile-picture, importing backup and so on)
- [ ] other way blobs can be addressed when using websocket vs. jsonrpc over dc-node
- [ ] Web push API? At least some kind of notification hook closure this lib can accept.
### Other Ideas for the Websocket server
- [ ] make sure there can only be one connection at a time to the ws
- why? , it could give problems if its commanded from multiple connections
- [ ] encrypted connection?
- [ ] authenticated connection?
- [ ] Look into unit-testing for the proc macros?
- [ ] proc macro taking over doc comments to generated typescript file
## Desktop Apis
Incomplete todo for desktop api porting, just some remainders for points that might need more work:
- [ ] manual start/stop io functions in the api for context and accounts, so "not syncing all accounts" can still be done in desktop -> webserver should then not do start io on all accounts by default
"CAN NOT RUN COVERAGE correctly: Missing DCC_NEW_TMP_EMAIL environment variable!\n\n",
"You can set COVERAGE_OFFLINE=1 to circumvent this check and skip the online tests, but those coverage results will be wrong, because some functions can only be tested in the online test"
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
