Any control information from the message
should only be downloaded when the message
is fully downloaded to avoid processing it twice.
Besides, "partial" messages may actually be full messages
with an error that are only processed as partial
to add a message bubble allowing to download the message later.
In-Reply-To may refer to non-call message
as we do not control the sender.
It may also happen that call message
was received by older version and processed
as text, in which case correct In-Reply-To
appears to be referring to the text message.
Emitting an `AccountsItemChanged` event is needed for UIs to know when `Contact::get_color()` starts
returning the true color for `SELF`. Before, an address-based color was returned for a new account
which was changing to a fingerprint-based color e.g. on app restart. Now the self-color is our
favorite gray until own keypair is generated or imported e.g. via ASM.
It really depends on the screen,
but on Android phones with higher DPI
than laptop screen the contrast
looks a bit too low.
It should not be increased too high
because of clipping.
I have not updated Python interpreters for legacy bindings
in scripts/run_all.sh because I have not checked
if manylinux images already exist and work for building wheels.
Legacy Python bindings are deprecated
and we don't publish new releases for them.
Old Delta Chat clients don't provide timestamps for added and removed members in messages, so at
least implicit member changes may be ignored as it's not clear if they are newer than explicit
member changes from modern clients. Lost messages aren't so frequent anyway, and overall
compatibility with old versions may be limited already.
Determinate Systems GitHub action
installs Nix version from Determinate Systems
and Determinate Nix Installer has
dropped support for installing upstream Nix:
https://determinate.systems/blog/installer-dropping-upstream/
This commit switches to upstream Nix
to avoid accidentally depending on any features
of Determinate Nix.
With newer Iroh it is possible to obtain
own node address before home relay is selected
and accidentally send own address without relay URL.
It took me some time to debug why Iroh 0.92.0
did not work with iroh-relay 0.92.0
so I'm adding these assertions even
while we still use Iroh 0.35.0.
I have logs from a user where messages are prefetched for long minutes, and while it's not a problem
on its own, we can't rely that the connection overlives such a period, so make
`fetch_new_messages()` prefetch (and then actually download) messages in batches of 500 messages.
Regarding the `@deprecated` addition:
it's not clear to me why the `listAccounts` function exists,
why `getAllAccounts` gets special treatment.
It has been there ever since the introduction of JSON-RPC.
Maybe it's because of the existence of `getContextEvents`,
which has to do with accounts.
But hopefully the new doc on `getContextEvents`
compensates for this.
this PR clarifies some events, also updating to recent changes.
see changed code for details
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
In messages with text/plain, text/html and text/calendar parts
within a multipart/alternative, text/calendar part is currently ignored,
text/plain is displayed and HTML is available via HTML API.
Ignoring `receive_imf_inner()` errors, i.e. silently skipping messages on failures, leads to bugs
never fixed. As for temporary I/O errors, ignoring them leads to lost messages, in this case it's
better to bubble up the error and get the IMAP loop stuck. However if there's some logic error, it's
better to show it to the user so that it's more likely reported, and continue receiving messages. To
distinguish these cases, on error, try adding the message as partially downloaded with the error set
to `msgs.error`, this way the user also can retry downloading the message to finally see it if the
problem is fixed.
DKIM-Signatures apply to the last headers, so start from the last header and take a valid one,
i.e. skip headers having unknown critical attributes, etc. Though this means that Autocrypt header
must be "oversigned" to guarantee that a not DKIM-signed header isn't taken, still start from the
last header for consistency with processing other headers. This isn't a security issue anyway.
Some Delta Chat clients (Desktop, for example)
do `leave_webxdc_realtime`
regardless of whether we've ever joined a realtime channel
in the first place. Such as when closing a WebXDC window.
This might result in unexpected and suspicious firewall warnings.
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Actually this leads to fetching messages only from watched folders and Spam. Motivation:
- At least Gmail has virtual folders which aren't correctly detected as such, e.g. "Sent".
- At least Gmail has many virtual folders and scanning all of them takes significant time, 5-6 secs
in median for me. This slows down receiving new messages and consumes battery.
- Delta Chat shouldn't fetch messages from folders potentially created by other apps for their own
purposes. NB: All compatible Delta Chat forks should use the "DeltaChat" folder as mvbox.
- Fetching from folders that aren't watched, e.g. from "Sent", may lead to message ordering issues.
SDP offer and answer contain newlines.
Without the fix these newlines are not encoded at all
and break the header into multiple headers
or even prevent parsing of the following headers.
Quoting @adbenitez:
> I have been using the SecurejoinInviterProgress event to show a
welcome message when user scan the QR/link of the bot (== starts a chat
with the bot)
> but this have a big problem: in that event all you know is that a
contact completed the secure-join process, you don't know if it was via
certain 1:1 invite link or a group invitation, then a group-invite bot
would send you a help message in 1:1 every time you join a group with it
Since it's easy enough to add this information to the
SecurejoinInviterProgress event, I wrote a PR to do so.
This is a preparation for expiring authentication tokens.
If we make authentication token expire,
we need to generate new authentication tokens each time
QR code screen is opened in the UI,
so authentication token is fresh.
We however don't want to completely invalidate
old authentication codes at the same time,
e.g. they should still be valid for joining groups,
just not result in a verification on the inviter side.
Since a group now can have a lot of authentication tokens,
it is easy to lose track of them
without any way to remove them
as they are not displayed anywhere in the UI.
As a solution, we now remove all
tokens corresponding to a group ID
when one token is withdrawn,
or all non-group tokens
when a single non-group token is withdrawn.
"Reset QR code" option already present
in the UI which works by resetting
current QR code will work without any UI changes,
but will now result in invalidation
of all previously created QR codes and invite links.
- sync declined calls from callee to caller, as usual in all larger
messengers
- introduce the call states "Missed call", "Declined call" and
"Cancelled all" ("Ended call" is gone)
- allow calling end_call()/accept_call() for already ended/accepted
calls, in practise, handling all cornercases is tricky in UI - and the
state needs anyways to be tracked.
- track and show the call duration
the duration calculation depends on local time, but it is displayed only
coarse and is not needed for any state. this can be improved as needed,
timestamps of the corresponding messages are probably better at some
point. or ending device sends its view of the time around. but for the
first throw, it seems good enough
if we finally want that set of states, it can be exposed to a json-info
in a subsequent call, so that the UI can render it more nicely. fallback
strings as follows will stay for now to make adaption in other UI easy,
and for debugging:
<img width="320" alt="IMG_0154"
src="https://github.com/user-attachments/assets/09a89bfb-66f4-4184-b05c-e8040b96cf44"
/>
successor of https://github.com/chatmail/core/pull/6650
this PR uses the initial "call messages" (that has a separate viewtype
since #7174) to show all call status.
this is what most other messengers are doing as well. additional "info
messages" after a call are no longer needed.
on the wire, as we cannot pickpack on visible info messages, we use
hidden messages, similar to eg. webxdc status updates.
in future PR, it is planned to allow getting call state as a json, so
that UI can render nicely. it is then decided if we want to translate
the strings in the core.
<img width="320" alt="IMG_0150"
src="https://github.com/user-attachments/assets/41ee3fa3-8be4-42c3-8dd9-d20f49881650"
/>
successor of https://github.com/chatmail/core/pull/6650
a dedicated viewtype allows the UI to show a more advanced UI, but even
when using the defaults,
it has the advantage that incoming/outgoing and the date are directly
visible.
successor of https://github.com/chatmail/core/pull/6650
The setting is already removed from the UIs,
but users who had it disabled previously have
no way to enable it. After this change
encryption is effectively always preferred.
UIs now display green checkmark in a profile
if the contact is verified.
Chats with key-contacts cannot become unprotected,
so there is no need to check 1:1 chat.
3 months were proven to be too short some years ago, after that issue,
we went far up to 12 months.
however, 12 months were considered too long after recent discussions :)
so, 6 months seems to be a good compromise.
the warning is still repeated every months and the text is unchanged.
advantage is still that this approach does not require network or
opt-in, and catches really all lazy updaters with few effort, cmp
https://github.com/deltachat/deltachat-desktop/issues/5422
Now that the previous commit avoids creating incorrect reverse verification chains, we can do
this. Sure, existing users' dbs aready have verification chains ending with "unknown" roots, but at
least for new users updating `verifier_id` to a known verifier makes sense.
If this happens, mark the contact as verified by an unknown contact instead. This avoids introducing
incorrect reverse chains: if the verifier itself has an unknown verifier, it may be `contact_id`
actually (directly or indirectly) on the other device (which is needed for getting "verified by
unknown contact" in the first place).
Also verify not yet verified contacts w/o setting a verifier for them (in the db it's stored as
`verifier_id=id` though) because we don't know who verified them for another device.
While testing the previous commit i understood that it's better to try giving different colors to
groups, particularly if their names are equal so that they visually differ, and at the same time
preserve the color if the group is renamed. Using `grpid` solves this. So let groups change colors
once and forever.
We can't just fail on an invalid chat name because the user would lose the work already done in the
UI like selecting members. Sometimes happens to me when i put space into name.
Follow-up to https://github.com/chatmail/core/pull/7125: We now have a
mix of non-async (parking_lot) and async (tokio) Mutexes used for the
connectivity. We can just use non-async Mutexes, because we don't
attempt to hold them over an await point. I also tested that we get a
compiler error if we do try to hold one over an await point (rather than
just deadlocking/blocking the executor on runtime).
Not 100% sure about using the parking_lot rather than std Mutex, because
since https://github.com/rust-lang/rust/issues/93740, parking_lot
doesn't have a lot of advantages anymore. But as long as iroh depends on
it, we might as well use it ourselves.
`get_connectivity()` is expected to return immediately, not when the scheduler finishes updating its
state in `start_io()/stop_io()/pause_io()`, otherwise it causes app non-responsiveness.
Instead of read-locking `SchedulerState::inner`, store the `ConnectivityStore` collection in
`Context` and fetch it from there in `get_connectivity()`. Update it every time we release a write
lock on `SchedulerState::inner`.
If the contact is already introduced by someone,
usually by adding to a verified group,
it should not be reverified because of another
chat message is a verified group.
This usually results is verification loops
and is not meaningful because the verifier
likely got this same contact introduced
in the same group.
We haven't dropped verified groups yet, so we need to do smth with messages that can't be verified
yet which often occurs because of messages reordering, particularly in large groups. Apart from the
reported case #7059, i had other direct reports that sometimes messages can't be verified for
various reasons, but when the reason is already fixed, it should be possible to re-download failed
messages and see them.
Also remove the code replacing the message text with a verification error from
`apply_group_changes()` as `add_parts()` already does this.
This doesn't fix anything in UIs currently because they don't call `get_securejoin_qr()` for
unencrypted groups, but it's still better to log an error which will be shown in this case.
In order to debug some test failures wrt broadcast channels, I need to
read the log of some tests that have an alice0 and an alice1.
It's currently not possible to tell whether a line was logged by alice0
or alice1, so, I would like to change that with this PR.
Edit: Turns out that there are more tests that call their profiles
"alice1"/"alice2" (or "alice"/"alice2") than "alice0"/"alice1". So, I
changed the logging to count "alice", "alice2", "alice3", ….
Not sure whether I should adapt old tests; it might save someone some
confusion in the future, but I might also make a mistake and make it so
that the test doesn't properly test anymore.
Chat was only loaded to avoid removing GuaranteeE2ee
for protected chats, but resending a message
in protected chat is guaranteed to be encrypted anyway.
Otherwise if the message is loaded by the UI
after GuaranteeE2ee is reset but before SMTP queue item
is created, the message may appear as unencrypted
even if it was actually resent as encrypted.
Encrypted message may create unencrypted groups
if the message does not have a Chat-Group-ID.
This can happen if v1 client sends an encrypted
message to opportunistically encrypted ad hoc group.
In this case `from_id` corresponds to the key-contact,
but we should add address-contact of the sender
to the member list.
The icon is mainly used to identify unencrypted chats
in the chatlist where encrypted and unencrypted chats are mixed.
It is used for group chats rather than only for 1:1 chats
with address-contacts.
We don't want to prefer returning verified contacts because e.g. if a bot was reinstalled and its
key changed, it may not be verified, and we don't want to bring the user to the old chat if they
click on the bot email address. But trying to return accepted contacts increases security and
doesn't break the described scenario.
If an address-contact and a key-contact were seen at exactly the same time, that doesn't necessarily
mean that it's a random event, it might occur because some code updates contacts this way in some
scenario. While this is unlikely, prefer to look up the key-contact.
Work around possible checkpoint starvations (there were cases reported when a WAL file is bigger
than 200M) and also make sure we truncate the WAL periodically. Auto-checkponting does not normally
truncate the WAL (unless the `journal_size_limit` pragma is set), see
https://www.sqlite.org/wal.html.
- It's obvious that addresses are needed to add contacts, so `_by_address_list` looks excessive.
- The function only looks up `SELF` by address.
- The name was confusing because there's also `lookup_key_contacts_by_address_list()` that actually
looks up key contacts by addresses (not only `SELF`).
- Added import-vcard and make-vcard commands to deltachat-repl. These
wrap the corresponding import_vcard() and make_vcard() operations from
src/contact.rs. Each command takes a file path argument specifying the
location of the vCard file to be read or written. The make-vcard command
takes one or more IDs to write to the file.
- Documented commands in "Using the CLI client" section of README.md.
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
The info will look like:
```
debug_assertions=On - DO NOT RELEASE THIS BUILD
```
or:
```
debug_assertions=Off
```
I tested that this actually works when compiling on Android.
<details><summary>This is how it looked before the second
commit</summary>
<p>
The deltachat_core_version line in the info will look like:
```
deltachat_core_version=v2.5.0 [debug build]
```
or:
```
deltachat_core_version=v2.5.0 [release build]
```
I tested that this actually works when compiling on Android.
</p>
</details>
This adds a test for https://github.com/chatmail/core/pull/7032/.
The crash happened if you received a message that has the from contact
also in the "To: " and "Chat-Group-Member-Fpr: " headers. Not sure how
it happened that such a message was created; I managed to create one in
a test, but needed to access some internals of MimeFactory for that.
I then saved the email that is sent by Alice into the test-data
directory, and then made a test that Bob doesn't crash if he receives
this email. And as a sanity-check, also test that Bob marks Fiona as
verified after receiving this email.
Sort recipients by `add_timestamp DESC` so that if the group is large and there are multiple SMTP
messages, a newly added member receives the member addition message earlier and has gossiped keys of
other members (otherwise the new member may receive messages from other members earlier and fail to
verify them).
This PR adds a test to reproduce a bug raised by @adbenitez that peer
channels break when the resend feature is used.
---------
Co-authored-by: iequidoo <dgreshilov@gmail.com>
This should fix https://github.com/chatmail/core/issues/7030; @r10s if
you could test whether it fixes your problem
The crash happened if you received a message that has the from contact
also in the "To: " and "Chat-Group-Member-Fpr: " headers. Not sure how
it happened that such a message was created.
Previously, chat protection was only removed if the chat became an
email-chat because the key-contact has a different email, but not if the
chat became an email-chat for a different reason.
Therefore, it could happen that the migration produced a protected,
unencrypted chat, where you couldn't write any messages.
I tested that applying this fix actually fixes the bug I had.
tokio-io-timeout 1.2.0 used previously
did not reset the timeout when returning
timeout error. This resulted
in infinite loop spamming
the log with messages that look like this and using 100% CPU:
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Read error on stream 192.168.1.20:993 after reading 9118 and writing 1036 bytes: timed out.
Normally these messages should be separated by at least 1 minute timeout.
The reason for infinite loop is not figured out yet,
but this change should at least fix 100% CPU usage.
See <https://github.com/sfackler/tokio-io-timeout/issues/13>
for the bugreport and
<https://github.com/sfackler/tokio-io-timeout/pull/14>
for the bugfix.
Socket may lose peer address when it is disconnected from
the server side. In this case debug_assert! failed
for me when running the core in debug mode on desktop.
To avoid the case of peer_addr not being available,
we now store it when LoggingStream is created.
this PR adds a info message "messages are end-to-end-encrypted" also for
chats created by eg. vcards. by the removal of lock icons, this is a
good place to hint for that in addition; this is also what eg. whatsapp
and others are doing
the wording itself is tweaked at
https://github.com/deltachat/deltachat-android/pull/3817 (and there is
also the rough idea to make the message a little more outstanding, by
some more dedicated colors)
~~did not test in practise, if this leads to double "e2ee info messages"
on secure join, tests look good, however.~~ EDIT: did lots of practise
tests meanwhile :)
most of the changes in this PR are about test ...
ftr, in another PR, after 2.0 reeases, there could probably quite some
code cleanup wrt set-protection, protection-disabled etc.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
- After the revisions to support key contacts, the 'listcontacts'
command in the repl only lists key-contacts. A separate flag now
has to be passed to Contact::get_all() to list address contacts.
This makes it difficult to run the example in README.md because we
need to see the new contact so we can get its ID for the next
command in the example, that creates a chat by ID.
- Revised 'listcontacts' command to make a second call to
Contact::get_all() with the DC_GCL_ADDRESS flag, then print the
e-mail contacts after the key contacts.
- Revised configuration example in top-level README.md to reflect
current command output.
fixes#7011
Delta Chat always adds protected headers to the inner encrypted or signed message, so if a protected
header is only present in the outer part, it should be ignored because it's probably added by the
server or somebody else. The exceptions are Subject and List-ID because there are known cases when
they are only present in the outer message part.
Also treat any Chat-* headers as protected. This fixes e.g. a case when the server injects a
"Chat-Version" IMF header tricking Delta Chat into thinking that it's a chat message.
Also handle "Auto-Submitted" and "Autocrypt-Setup-Message" as protected headers on the receiver
side, this was apparently forgotten.
Headers are normally added at the top of the message,
e.g. when forwarding new `Received` headers are
added at the top.
When headers are protected with DKIM-Signature
and oversigning is not used,
forged headers may be added on top
so headers from the top are generally less trustworthy.
This is tested with `test_take_last_header`,
but so far last header was only preferred
for known headers. This change extends
preference of the last header to all headers.
Connection sometimes fails while processing FETCH
responses. In this case `fetch_new_messages` exits early
and does not advance next expected UID even if
some messages were processed.
This results in prefetching the same messages
after reconnection and log messages
similar to
"Not moving the message ab05c85a-e191-4fd2-a951-9972bc7e167f@localhost that we have seen before.".
With this change we advance next expected UID
even if `fetch_new_messages` returns a network error.
A donation request device message is added if >= 100 messages have been sent and delivered. The
condition is checked every 30 days since the first message is sent. The message is added only once.
Follow-up for https://github.com/chatmail/core/pull/6992
Since we're printing the hint to stderr now, rather than stdout (as per
link2xt's suggestion), it was too noisy. Also, it was printed once for
every test account rather than once per test.
Now, it integrates nicely with rust's hint to enable a backtrace:
```
stderr ───
thread 'chat::chat_tests::test_broadcasts_name_and_avatar' panicked at src/chat/chat_tests.rs:2757:5:
assertion failed: `(left == right)`
Diff < left / right > :
<true
>false
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: If you want to examine the database files, set environment variable DELTACHAT_SAVE_TMP_DB=1
FAIL [ 0.265s] deltachat chat::chat_tests::test_broadcast
```
Add `logged_debug_assert` macro logging a warning if a condition is not satisfied, before invoking
`debug_assert!`, and use this macro where `Context` is accessible (i.e. don't change function
signatures for now).
Follow-up to 0359481ba4.
We have some debug assertions already, but we also want the corresponding errors in the release
configuration so that it's not less reliable than non-optimized one. This doesn't change any
function signatures, only debug assertions in functions returning `Result` are replaced.
Co-authored-by: l <link2xt@testrun.org>
Part of #6884.
The channel owner will not be notified in any way that you left, they
will only see that there is one member less.
For the translated stock strings, this is what we agreed on in the
group:
- Add a new "Leave Channel" stock string (will need to be done in UIs)
- Reword the existing "Are you sure you want to leave this group?"
string to "Are you sure you want to leave?" (the options are "Cancel"
and "Leave Group" / "Leave Channel", so it's clear what you are leaving)
(will need to be done in the deltachat-android repo, other UIs will pick
it up automatically)
- Reword the existing "You left the group." string to "You left". (done
here, I will adapt the strings in deltachat-android, too)
I adapted DC Android by pushing
6df2740884
to https://github.com/deltachat/deltachat-android/pull/3783.
---------
Co-authored-by: l <link2xt@testrun.org>
I had it a few times now that I wanted to examine the database in order
to debug a test failure. Setting this environment variable makes this
easy in the future.
New public API `set_accounts_order` allows setting the order of accounts.
The account order is stored as a list of account IDs in `accounts.toml`
under a new `accounts_order: Vec<u32>` field.
BREAKING CHANGE: messages with invalid images, images of unknown size,
huge images, will have Viewtype::File
After changing the logic of Viewtype selection, I had to fix 3 old tests
that used invalid Base64 image data.
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Non-members can't modify the member list (incl. adding themselves), modify an ephemeral timer, so
they shouldn't be able to change the group name or avatar, just for consistency. Even if messages
are reordered and a group name change from a new member arrives before its addition, the new group
name will be applied on a receipt of the next message following the addition message because
Chat-Group-Name-Timestamp increases. While Delta Chat groups aimed for chatting with trusted
contacts, accepting group changes from everyone knowing Chat-Group-Id means that if any of the past
members have the key compromised, the group should be recreated which looks impractical.
There's a comment in `tools` that tells to use `tools::Time` instead of `Instant` because on Android
the latter doesn't advance in the deep sleep mode. The only place except `migrations` where
`Instant` is used is tests, but we don't run CI on Android. It's unlikely that Delta Chat goes to
the deep sleep while executing migrations, but still possible, so let's use `tools::Time` as
everywhere else.
Save:
- (old contact id) -> (new contact id) mapping.
- The message id starting from which all messages are already migrated.
Run the migration from `housekeeping()` for at least 500 ms and for >= 1000 messages per run.
We don't want images having unsupported format or corrupted ones to be sent as `Image` and appear in
the "Images" tab in UIs because they can't be displayed correctly.
If a user attaches an image as `File`, we should send the original filename. And vice versa, if it's
`Image` originally, we mustn't reveal the filename.
The filename used for sending is now also saved to the db, so all the sender's devices will display
the same filename in the message info.
Otherwise unsupported and corrupted images are displayed in the "Images" tab in UIs and that looks
as a Delta Chat bug. This should be a rare case though, so log it as error and let the user know
that metadata isn't removed from the image at least.
`Viewtype::Sticker` has special meaning: the file should be an image having fully transparent
pixels. But "tgs" (Telegram animated sticker) is a compressed JSON and isn't recognized by Core as
image.
Older versions of Delta Chat ignore the message if it contains a
ChatGroupId header. ("older versions" means all versions without #6901,
i.e.currently released versions)
This means that without this PR, broadcast channel messages sent from
current main don't arrive at a device running latest released DC.
Part of #6884.
In https://github.com/chatmail/core/pull/6901, I unfortunately forgot to
document the API change when squash-merging, so, I'm doing this with the
PR here.
The API change is breaking because not adapting to the new channel types
would lead to errors.
Part of #6884
----
- [x] Add new chat type `InBroadcastChannel` and `OutBroadcastChannel`
for incoming / outgoing channels, where the former is similar to a
`Mailinglist` and the latter is similar to a `Broadcast` (which is
removed)
- Consideration for naming: `InChannel`/`OutChannel` (without
"broadcast") would be shorter, but less greppable because we already
have a lot of occurences of `channel` in the code. Consistently calling
them `BcChannel`/`bc_channel` in the code would be both short and
greppable, but a bit arcane when reading it at first. Opinions are
welcome; if I hear none, I'll keep with `BroadcastChannel`.
- [x] api: Add create_broadcast_channel(), deprecate
create_broadcast_list() (or `create_channel()` / `create_bc_channel()`
if we decide to switch)
- Adjust code comments to match the new behavior.
- [x] Ask Desktop developers what they use `is_broadcast` field for, and
whether it should be true for both outgoing & incoming channels (or look
it up myself)
- I added `is_out_broadcast_channel`, and deprecated `is_broadcast`, for
now
- [x] When the user changes the broadcast channel name, immediately show
this change on receiving devices
- [x] Allow to change brodacast channel avatar, and immediately apply it
on the receiving device
- [x] Make it possible to block InBroadcastChannel
- [x] Make it possible to set the avatar of an OutgoingChannel, and
apply it on the receiving side
- [x] DECIDE whether we still want to use the broadcast icon as the
default icon or whether we want to use the letter-in-a-circle
- We decided to use the letter-in-a-circle for now, because it's easier
to implement, and I need to stay in the time plan
- [x] chat.rs: Return an error if the user tries to modify a
`InBroadcastChannel`
- [x] Add automated regression tests
- [x] Grep for `broadcast` and see whether there is any other work I
need to do
- [x] Bug: Don't show `~` in front of the sender's same in broadcast
lists
----
Note that I removed the following guard:
```rust
if !new_chat_contacts.contains(&ContactId::SELF) {
warn!(
context,
"Received group avatar update for group chat {} we are not a member of.", chat.id
);
} else if !new_chat_contacts.contains(&from_id) {
warn!(
context,
"Contact {from_id} attempts to modify group chat {} avatar without being a member.",
chat.id,
);
} else [...]
```
i.e. with this change, non-members will be able to modify the avatar.
Things were slightly easier this way, and I think that this is in line
with non-members being able to modify the group name and memberlist
(they need to know the Group-Chat-Id, anyway), but I can also change it
back.
- The implementation of listverified was removed in commit
37dc1f5ca0, but it still shows up in
the help and in the auto-complete grammar.
- Removed listverified where it still appears.
closes#6971
We need this because it's not clear whether Android should switch to
JsonRPC for everything, because of concerns that JsonRPC might be a lot
slower than the CFFI (although we still need to measure that).
SQL migration to key contacts generates a lot of events,
and they are dropped in desktop logs because it does
not read the events fast enough.
This at least reduces the number of dropped messages.
This change introduces a new type of contacts
identified by their public key fingerprint
rather than an e-mail address.
Encrypted chats now stay encrypted
and unencrypted chats stay unencrypted.
For example, 1:1 chats with key-contacts
are encrypted and 1:1 chats with address-contacts
are unencrypted.
Groups that have a group ID are encrypted
and can only contain key-contacts
while groups that don't have a group ID ("adhoc groups")
are unencrypted and can only contain address-contacts.
JSON-RPC API `reset_contact_encryption` is removed.
Python API `Contact.reset_encryption` is removed.
"Group tracking plugin" in legacy Python API was removed because it
relied on parsing email addresses from system messages with regexps.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Co-authored-by: B. Petersen <r10s@b44t.com>
Display name is rarely needed for debugging,
so there is no need to include it in the logs.
Display name is even already listed in `skip_from_get_info`,
but the test only allowed the values to be skipped
without checking that they are always skipped.
Before this PR, ConfiguredAddr (which will be used to store the primary
transport) would have been changed when adding a new transport. Doesn't
matter yet because it's not possible yet to have multiple transports.
But I wanted to fix this bug already so that I'm not suprised by it
later.
Citing @link2xt:
> RFC examples sometimes don't escape commas, but there is errata that fixes some of them.
Also this unescapes commas in all fields. This can lead to, say, an email address with commas, but
anyway the caller should check parsed `VcardContact`'s fields for correctness.
OpenSSL is vendored, but because of rusqlite feature
transitively enabling vendoring feature.
This change makes vendoring explicit
even if we disable SQLCipher in the future.
New `fuzz` profile is added
because cargo-bolero now requires it and uses
by default, while `--release` option is removed.
Instructions for running AFL are removed from the README
because it requires some system reconfiguration
and I did not test it this time.
The main reason for this change is the app picker
that Delta Chat clients use, which utilizes
the `fetch_url` function.
Sometimes we get an error from the server,
but we have no way to figure out that it's an error,
other than inspecting the body, which we don't (and shouldn't) do.
This results in us attempting to send webxdc apps
that are not even valid .zip files.
Another, arguably even worse thing is that
we also put the error responses to the cache,
so it's not easy to recover from such an error.
So, let's just return an error if the response code
is not a successful response code.
If we receive a message from non-verified contact
in a non-protected chat with a Chat-Verified header,
there is no need to upgrade the chat
to verified and display an error.
If it was an attack, an attacker could
just not send the Chat-Verified header.
Most of the time, however, it is just
message reordering.
The test was still WIP but got merged together with the fix. I suggest
to keep the fix in main and add the test in a follow-up RP. The test
should suffice becaues I tested it manually.
When an invalid webxdc is set as draft, json-rpc's `get_draft` fails,
because `get_webxdc_info` which it calls, fails because the zip reader
can not read a non-zip file. With this change, any error occurring in
`get_webxdc_info` is ignored and the None-variant is returned instead. I
also added a test, that setting invalid xdcs is draft is fine core-wise
and checked that the input field stays responsive when a fake.xdc
produced like in #6826 is added to draft
close#6826
closes#6873 , see there for reasoning.
tested that on iOS already, works like a charm - and was much easier
than expected as @iequidoo already updated `timestamp_rcvd` on status
updates in https://github.com/chatmail/core/pull/5388
~~a test is missing, ordering is not tested at all, will check if that
is doable reasonably easy~~ EDIT: added a test
Fix https://github.com/chatmail/core/issues/6621; I also tested on
Android that the webxdc self-addr actually stays the same when staging a
draft and then sending.
Follow-up to https://github.com/chatmail/core/pull/6704; #6704 made sure
that the webxdc self-addr doesn't change when creating a message and
then sending it. This PR here makes sure that the rfc724_mid (which is
needed to compute the self-addr) is saved when setting a draft, so that
it's loaded properly after a call to get_draft().
cc @adbenitez @r10s @Septias
FuturesUnordered is likely buggy and iroh previously switched
to JoinSet in <https://github.com/n0-computer/iroh/pull/1647>.
We also have reports with logs of background_fetch getting
stuck so apparently task cancellation after timeout does not work
as intended with FuturesUnordered.
Set `rfc724_mid` in `Message::new()`, `Message::new_text()`, and
`Message::default()` instead of when sending the message. This way the
rfc724 mid can be read in the draft stage which makes it more consistent
for bots. Tests had to be adjusted to create multiple messages to get
unique mid, otherwise core would not send the messages out.
this PR scaled avatars using the Triangle-filter,
resulting in often better image quality and smaller files (5%).
it comes at high costs,
therefore, we do not do that unconditionally for each image sent, see
comment in the code
and https://github.com/chatmail/core/pull/6815
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Revert the biggest part of https://github.com/chatmail/core/pull/6722/
in order to fix#6816. Reopens
https://github.com/chatmail/core/issues/6706.
Rationale for reverting instead of fixing is that it's not trivial to
implement "if the chat is encrypted, can_send() returns true": When
sending a message, in order to check whether to encrypt, we load all
peerstates and check whether all of them can be encrypted to
(`should_encrypt()`). We could do this in `can_send()`, but this would
make it quite slow for groups. With multi-transport, the ways of
checking whether to encrypt will be different, so in order not to do
unnecessary work now, this PR just revert parts of
[https://github.com/chatmail/core/pull/6722/](https://github.com/chatmail/core/pull/6817#),
so that we can make things work nicely when multi-transport is merged.
As a quick mitigation, we could increase the timeout from 15s to
something like 1 minute or 1 day: Long enough that usually securejoin
will finish before, but short enough that it's possible to send to old
chats that had a failed securejoin long in the past.
Move all `configured_*` parameters into a new SQL table `transports`.
All `configured_*` parameters are deprecated; the only exception is
`configured_addr`, which is used to store the address of the primary
transport. Currently, there can only ever be one primary transport (i.e.
the `transports` table only ever has one row); this PR is not supposed
to change DC's behavior in any meaningful way.
This is a preparation for mt.
---------
Co-authored-by: l <link2xt@testrun.org>
In the `test` cfg, introduce `MimeMessage::headers_removed` hash set and `header_exists()` function
returning whether the header exists in any part of the parsed message. `get_header()` shouldn't be
used in tests for checking absense of headers because it returns `None` for removed ("ignored")
headers.
This change simplifies
updating the gossip timestamps
when we receive a message
because we only need to know
the keys received in Autocrypt-Gossip
header and which chat the message is
assigned to.
We no longer need to iterate
over the member list.
This is a preparation
for PGP contacts
and member lists that contain
key fingerprints rather than
email addresses.
This change also removes encryption preference
from Autocrypt-Gossip header.
It SHOULD NOT be gossiped
according to the Autocrypt specification
and we ignore encryption preference anyway
since 1.157.0.
test_gossip_optimization is removed
because it relied on a per-chat gossip_timestamp.
Broadcast lists are encrypted since 1.159.0,
but Autocrypt-Gossip was not disabled.
As Autocrypt-Gossip contains the email address
and the key of the recipient, it should
not be sent to broadcast lists.
this PR moves now advanced/unsupported ASM strings to core, removing
work from translations, esp. as another hint is added which would
require retranslations. it is better to have that just in english, it is
a nerd feature anyways.
moverover, this PR removes special rendering of ASM in the summary,
which might be confusion, but mainly it is now unneeded, dead code
i'll do another android PR that will point to "Add Second Device"
already on ASM generation EDIT: done at
https://github.com/deltachat/deltachat-android/pull/3726
targets https://github.com/deltachat/deltachat-desktop/issues/4946
it was all the time questionable if not encrypting broadcast lists
rules the issue that recipients may know each other cryptographically.
however, meanwhile with chatmail, unncrypted broadcasts are no longer possible,
and we actively broke workflows eg. from this teacher:
https://support.delta.chat/t/broadcast-funktioniert-nach-update-nicht-meht/3694
this basically reverts commit
7e5907daf2
which was that time added last-minute and without lots discussions :)
let the students get their homework again :)
Primary key is usually used for certification.
It is possible to make a certification- and encryption-
capable key with RSA, but RFC 9580 says
that implementations SHOULD NOT generate RSA keys.
The test works most of the time, but essentially tests that splitting
the public key from a private key
generates the same result.
However, it fails if two signatures are generated
at different seconds.
Closes#6762
fetch_existing option is not enabled in existing clients
and does not work with encrypted messages
without importing the key into a newely created account.
next android/desktop/ios releases won't have the "Show Classic Emails"
option for chatmail.
to avoid issues with user that have set sth else than "All", we ignore
the option alltogether for chatmail profiles.
ftr, i do not expect ppl having that option changed for chatmail much,
it does not make much sense. so this PR is mainly to save our limited
support resources :) (usecase: "look, i am using chatmail to sign up at
SERVICE, but for security reasons i set show=all only when i reset my
password" :)
one could also do that in a migration, however, (a) migrations always
come with some risk, even the easiest ones, and (b) the show_emails
option is subject to change or disappear anyways, subsequent changes are
easier in code than in additional or removed migrations, and (c) it is
really only one line, that does not add much with complexity
Otherwise if an invite link is generated and the group is renamed then before the promotion, the
joined member will have the group name from the invite link, not the new one.
for tuning down email address everywhere, that bit is missing in core.
it was never useful, as it was never shown on the receivers side. and
for the sender side, the context the qr code is opened is clear
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
Without tox any python `scripts/make-python-env.sh` does not run. Maybe
at some point we can even generate the environment for testing with
`venvHook` like functionality. We could also add `python3` instead of
fixing it to 3.11, but I don't know which version core expects
`self.accounts.read().await.get_all()` acquires a read lock
and does not release it until the end of `for` loop.
After that, a writer may get into the queue,
e.g. because of the concurrent `add_account` call.
In this case `let context_option = self.accounts.read().await.get_account(id);`
tries to acquire another read lock and deadlocks
because tokio RwLock is write-preferring and will not
give another read lock while there is a writer in the queue.
At the same time, writer never gets a write lock
because the first read lock is not released.
The fix is to get a single read lock
for the whole `get_all_accounts()` call.
This is described in <https://docs.rs/tokio/1.44.1/tokio/sync/struct.RwLock.html#method.read>:
"Note that under the priority policy of RwLock, read locks are not
granted until prior write locks, to prevent starvation. Therefore
deadlock may occur if a read lock is held by the current task, a write
lock attempt is made, and then a subsequent read lock attempt is made by
the current task."
instead of showing addresses in info message, provide an API to get the
contact-id.
UI can then make the info message tappable and open the contact profile
in scope
the corresponding iOS PR - incl. **screencast** - is at
https://github.com/deltachat/deltachat-ios/pull/2652 ; jsonrpc can come
in a subsequent PR when things are settled on android/ios
the number of parameters in `add_info_msg_with_cmd` gets bigger and
bigger, however, i did not want to refactor this in this PR. it is also
not really adding complexity
closes#6702
---------
Co-authored-by: link2xt <link2xt@testrun.org>
Co-authored-by: Hocuri <hocuri@gmx.de>
otherwise, by tuning down the email addresses,
one does not really has and idea who is SELF.
maybe the dialog is the only way at the end to get the transport
adresses of contacts,
this is unclear atm, this PR fixes the issue at hand
Targets https://github.com/chatmail/core/issues/6636
Right now the error message is:
> Error: Delta Chat is already running. To use Delta Chat, you must
first close the existing Delta Chat process, or restart your device.
>
> (accounts.lock lock file is already locked)
other suggestions welcome!
vc-request is an unencrypted message
that Bob sends when he does not have Alice's key.
It also does not contain
Bob's avatar and name,
so the contact has only the email address
at this point and it is too early
to show it.
The test is mostly testing that groups can be matched
even if Message-ID is replaced.
Delta Chat no longer places group ID into Message-ID
or References, so the test is not
testing anything other than the ability
to match groups based on References header.
This API allows to explicitly set
a name of the contact
instead of trying to create a new contact
with the same address.
Not all contacts are identified
by the email address
and we are going to introduce
contacts identified by their keys.
Four new APIs `add_transport_from_qr()`, `add_transport()`,
`list_transports()`, `delete_transport()`, as described in the draft at
"API".
The `add_tranport*()` APIs automatically stops and starts I/O; for
`configure()` the stopping and starting is done in the JsonRPC bindings,
which is not where things like this should be done I think, the bindings
should just translate the APIs.
This also completely disables AEAP for now.
I won't be available for a week, but if you want to merge this already,
feel free to just commit all review suggestions and squash-merge.
Found and fixed a bug while investigating
https://github.com/chatmail/core/issues/6656. It's not the same bug,
though.
Steps to reproduce this bug:
- Create a new profile
- Transfer it to a second device
- Send a message from the first device
- -> It will never arrive on the second device, instead a warning will
be printed that you are using DC on multiple devices.
The bug was that the key wasn't created before the backup transfer, so
that the second device then created its own key instead of using the
same key as the first device.
In order to regression-test, this PR now changes `clone()` to use "Add
second device" instead of exporting and importing a backup. Exporting
and importing a backup has enough tests already.
This PR also adds an unrelated test `test_selfavatar_sync()`.
The bug was introduced by https://github.com/chatmail/core/pull/6574 in
v1.156.0
WebSocket support is not used
and is not maintained. It still uses
outdated axum 0.7 version
and does not have any authentication.
Delta Chat Desktop has a new browser target
that implements WebSocket support on top
of stdio server, supports blobs
and is tested in CI.
When it is possible to test that no unhidden contact
is creating by looking through the contact list
or get the contact ID as the from_id of received message,
do it to avoid acidentally creating a contact
or changing its origin before testing.
async-compression 0.4.21 fixes a bug in the encoder
where it did not flush all the internal state sometimes,
resulting in IMAP APPEND command timing out
waiting for response when uploading large sync messages.
See <https://github.com/Nullus157/async-compression/pull/333>
for details.
this PR deletes all known messages belonging to a chat when the chat is
deleted.
this may not be an exhaustive list as a client might not know all
message-ids (eg. when using different times for "delete from device").
in this case, other devices may know more IDs. otherwise, the chatmail
server will eventually clean up at some point. for non-chatmail, this is
up to the user then.
the deletion sql commands were inspired by
[`delete_msgs_ex`](https://github.com/chatmail/core/blob/main/src/message.rs#L1743)
(in fact, [a first
try](https://github.com/chatmail/core/compare/r10s/clear-chat-on-delete)
was adapting that part, however, that seems less performant as lots of
sql commands are needed)
successor of #5007
SecureJoin and importing a vCard are the primary
ways we want to support for creating contacts.
Typing in an email address and relying on Autocrypt
results in sending the first message unencrypted
and we want to clearly separate unencrypted and encrypted
chats in the future.
To make the tests more stable, we set up test contacts
with vCards as this always immediately
results in creating a single encrypted chat
and this is not going to change.
up to now, avatars were restricted to 20k, 256x256 pixel.
resulting in often poor avatar quality.
(the limitation comes from times
where unencrypted outlook inner headers were a thing;
this has changed meanwhile)
to increase quality while keeping a reasonable small size,
for "balanced quality",
we increase the allowed width to 512x512 and the allowed size to 60k.
for "worse quality", things stay unchanged at 128x128 pixel and 20k.
Currently broadcast lists creation is synced across devices. Groups creation, in some means, too --
on a group promotion by a first message. Otoh, messages deletion is synced now as well. So, for
feature-completeness, sync chats deletion too.
Say in the README that cargo-bolero@0.8.0
should be installed as newer versions
do not work currently.
Update `mailparse` to 0.16.0
because this is what Delta Chat currently uses.
I also added fuzzer outputs to .gitignore
Add "Chat-Group-Name-Timestamp" message header and use the last-write-wins logic when updating group
names (similar to group member timestamps). Note that if the "Chat-Group-Name-Changed" header is
absent though, we don't add a system message (`MsgGrpNameChangedBy`) because we don't want to blame
anyone.
Replacing default key
when a profile is already part of
verified groups results in
`[The message was sent with non-verified encryption. See 'Info' for more details]`
messages for other users.
It is still possible
to import the default key before
Delta Chat generates the key.
When there is a broken group (which might happen with multi-transport),
people want to leave it.
The problem is that every "Group left" message notifies all other
members and pops up the chat, so that other members also want to leave
the group.
This PR makes it so that "Group left" messages don't create a
notification, don't cause a number-in-a-cirle badge counter on the chat,
and don't sort up the chat in the chatlist.
If a group is deleted, then the group won't pop up when someone leaves
it; this worked fine already before this PR, and there also is a test
for it.
Instead of being trashed, the message containing a reaction remains in the chat, hidden and InFresh. When the chat is opened, it will be marked as Seen on the server, so that a second device removes the notifications for the reaction.
Close https://github.com/deltachat/deltachat-core-rust/issues/6210.
Also, this adds a benchmark.
Now that we are deduplicating everywhere, we can get rid of some code.
The old python bindings did not get an optional `name` parameter because
they are deprecated anyway, but it would be easy to add it.
> _greetings from the ice of the deutsche bahn 🚂🚃🚃🚃 always a pleasure to
see how well delta chat meanwhile performs in bad networks :)_
this PR adds an API to request other chat members to replace the message
text of an already sent message. scope is mainly to fix typos. this
feature is known from whatsapp, telegram, signal, and is
[requested](https://support.delta.chat/t/retract-edit-sent-messages/1918)
[since](https://support.delta.chat/t/edit-messages-in-delta-chat/899)
[years](https://github.com/deltachat/deltachat-android/issues/198).
technically, a message with an
[`Obsoletes:`](https://datatracker.ietf.org/doc/html/rfc2076#section-3.6)
header is sent out.
```
From: alice@nine
To: bob@nine
Message-ID: 2000@nine
In-Reply-To: 1000@nine
Obsoletes: 1000@nine
Edited: this is the new text
```
the body is the new text, prefixed by the static text `Edited:` (which
is not a header). the latter is to make the message appear more nicely
in Non-Delta-MUA. save for the `In-Reply-To` header. the `Edited:`
prefix is removed by Delta Chat on receiving.
headers should be protected and moved to e2ee part as usual.
corrected message text is flagged, and UI should show this state, in
practise as "Edited" beside the date.
in case, the original message is not found, nothing happens and the
correction message is trashes (assuming the original was deleted).
question: is the `Obsoletes:` header a good choice? i _thought_ there is
some more specifica RFC, but i cannot find sth. in any case, it should
be an header that is not used otherwise by MUA, to make sure no wanted
messages disappear.
what is NOT done and out of scope:
- optimise if messages are not yet sent out. this is doable, but
introduces quite some cornercaes and may not be worth the effort
- replaces images or other attachments. this is also a bit cornercasy
and beyond "typo fixing", and better be handled by "delete for me and
others" (which may come soon, having the idea now, it seems easy :)
- get edit history in any way. not sure if this is worth the effort,
remember, as being a private messenger, we assume trust among chat
members. it is also questionable wrt privacy, seized devices etc.
- add text where nothing was before; again, scope is "typo fixing",
better avoid cornercases
- saved messages are not edited (this is anyway questionable)
- quoted texts, that are used for the case the original message is
deleted, are not updated
- edits are ignored when the original message is not there yet (out of
order, not yet downloaded)
- message status indicator does not show if edits are sent out or not -
similar to reactions, webxdc updates, sync messages. signal has the same
issue :) still, connectivity should show if there are messages pending
<img width="366" alt="Screenshot 2025-02-17 at 17 25 02"
src="https://github.com/user-attachments/assets/a4a53996-438b-47ef-9004-2c9062eea5d7"
/>
corresponding iOS branch (no PR yet):
https://github.com/deltachat/deltachat-ios/compare/main...r10s/edit-messages
---------
Co-authored-by: l <link2xt@testrun.org>
This makes it so that files will be deduplicated when using the JsonRPC
API.
@nicodh and @WofWca you know the Desktop code and how it is using the
API, so, you can probably tell me whether this is a good way of changing
the JsonRPC code - feel free to push changes directly to this PR here!
This PR here changes the existing functions instead of creating new
ones; we can alternatively create new ones if it allows for a smoother
transition.
This brings a few changes:
- If you pass a file that is already in the blobdir, it will be renamed
to `<hash>.<extension>` immediately (previously, the filename on the
disk stayed the same)
- If you pass a file that's not in the blobdir yet, it will be copied to
the blobdir immediately (previously, it was copied to the blobdir later,
when sending)
- If you create a file and then pass it to `create_message()`, it's
better to directly create it in the blobdir, since it doesn't need to be
copied.
- You must not write to the files after they were passed to core,
because otherwise, the hash will be wrong. So, if Desktop recodes videos
or so, then the video file mustn't just be overwritten. What you can do
instead is write the recoded video to a file with a random name in the
blobdir and then create a new message with the new attachment. If
needed, we can also create a JsonRPC for `set_file_and_deduplicate()`
that replaces the file on an existing message.
In order to test whether everything still works, the desktop issue has a
list of things to test:
https://github.com/deltachat/deltachat-desktop/issues/4498
Core issue: #6265
---------
Co-authored-by: l <link2xt@testrun.org>
This hopefully does not trigger Apple spam filter
which seems to pass thorugh Message-IDs with hyphens.
It is not confirmed, but better blend in
by mimicking existing popular email
where possible.
With iOS and Desktop copying the file to a to a temp file with the name
of `get_filename()`, it should be sanitized first.
The PR can be reviewed commit-by-commit or all at once.
This task is not guaranteed to be cancellation-safe
and provides a stop token for safe cancellation instead.
We should always cancel the task properly
and not by racing against another future.
Otherwise following call to Handle.done()
may work on IMAP session that is in the middle
of response, for example.
this converts old QR codes to the new format, in an hacky, but simple
way, see #6518 for more details and for code snippet
then QR code change is esp. bad as ppl will have different versions for
some days at least, weakening overall UX, esp. of first-time-users that
may come to delta because of praised, seamless multidevice ... :)
i tested in https://github.com/deltachat/deltachat-ios/pull/2595 that
this actually fixes the problem, and there is no deeper issue with
changed chashes or so - seemed not to be the case, at least, with this
hack, core accepts QR codes from the released 1.52-and-before series
this hack gives user time to update, it can be removed after some months
(we can also remove the old BACKUP qr code alltogether then)
we should still not wait too long with the PR as there are already
versions out with the "new/bad" QR code (and growing, as new iOS
installations will get the new format, one cannot revert a version, only
pause rollout)
---------
Co-authored-by: link2xt <link2xt@testrun.org>
This change adds a test and updates mailparse from 0.15.0 to 0.16.0.
mailparse 0.16.0 includes a fix for the bug
that resulted in CRLF being included at the end of the body.
Workaround for the bug in the `pk_validate` function is also removed.
APNS tokens never expire unless
the user uninstalls the application.
Because of this in most cases
the token remains valid forever
and chatmail server never removes the token
even if it is unencrypted
or the user has removed Delta Chat profile
from the device but not the whole application.
We want to modify chatmail servers
to remember the last time the token was stored
and remove them after some time.
Before we do this, we need to modify
the client to store the device token
each time so the server knows which tokens are used
and can update their timestamps.
QR code format changed because `NodeAddr` serialization
changed between iroh 0.29.0 and iroh 0.30.0.
We have already released iroh 0.30.0,
so the test at least makes change we don't break compatibility again.
There was a bug that file extensions were removed when recoding an
avatar. The problem was that `recode_avatar` used `name` to check for
the extension, but some functions passed an empty string.
There even were two tests from before the decision to keep the
extensions that tested for the faulty behavior.
These were the last places in the `deltachat` crate where files were
stored without deduplication. The CFFI python bindings are the last
thing that's still missing.
Deduplicate:
- In the REPL
- In `store_from_base64()`, which writes avatars received in headers
- In a few tests
- The saved messages, broadcast, device, archive icons
- The autocrypt setup message
1-2 more PRs, and we can get rid of `BlobObject::create`,
`sanitise_name()`, and some others
When receiving messages, blobs will be deduplicated with the new
function `create_and_deduplicate_from_bytes()`. For sending files, this
adds a new function `set_file_and_deduplicate()` instead of
deduplicating by default.
This is for
https://github.com/deltachat/deltachat-core-rust/issues/6265; read the
issue description there for more details.
TODO:
- [x] Set files as read-only
- [x] Don't do a write when the file is already identical
- [x] The first 32 chars or so of the 64-character hash are enough. I
calculated that if 10b people (i.e. all of humanity) use DC, and each of
them has 200k distinct blob files (I have 4k in my day-to-day account),
and we used 20 chars, then the expected value for the number of name
collisions would be ~0.0002 (and the probability that there is a least
one name collision is lower than that) [^1]. I added 12 more characters
to be on the super safe side, but this wouldn't be necessary and I could
also make it 20 instead of 32.
- Not 100% sure whether that's necessary at all - it would mainly be
necessary if we might hit a length limit on some file systems (the
blobdir is usually sth like
`accounts/2ff9fc096d2f46b6832b24a1ed99c0d6/dc.db-blobs` (53 chars), plus
64 chars for the filename would be 117).
- [x] "touch" the files to prevent them from being deleted
- [x] TODOs in the code
For later PRs:
- Replace `BlobObject::create(…)` with
`BlobObject::create_and_deduplicate(…)` in order to deduplicate
everytime core creates a file
- Modify JsonRPC to deduplicate blob files
- Possibly rename BlobObject.name to BlobObject.file in order to prevent
confusion (because `name` usually means "user-visible-name", not "name
of the file on disk").
[^1]: Calculated with both https://printfn.github.io/fend/ and
https://www.geogebra.org/calculator, both of which came to the same
result
([1](https://github.com/user-attachments/assets/bbb62550-3781-48b5-88b1-ba0e29c28c0d),
[2](https://github.com/user-attachments/assets/82171212-b797-4117-a39f-0e132eac7252))
---------
Co-authored-by: l <link2xt@testrun.org>
- This [is said to lead improve compilation
speed](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html#Assorted-Tricks)
- When grepping for a function invocation, this makes it easy to see whether it's from a test or "real" code
- We're calling the files e.g. `chat_tests.rs` instead of `tests.rs` for the same reason why we moved `imap/mod.rs` to `imap.rs`: Otherwise, your editor always shows you that you're in the file `tests.rs` and you don't know which one.
This is only moving mimeparser and chat tests, because these were the
biggest files; we can move more files in subsequent PRs if we like it.
Fix https://github.com/deltachat/deltachat-core-rust/issues/6433
I at first only wanted to do it any outgoing messages, but @link2xt was
concerned that this may accidentally enable bcc_self, e.g. in the
following case:
- you send out a message
- it's deleted, e.g. via ephemeral messages
- Someone forwards this outgoing message to you again, e.g. via a
mailing list.
If we create an unpromoted group,
add a member there and then remove it
before we promote a group, there is no need to
add such member to the list of past members
and send the address of this member to the group
when it is promoted.
> _took quite some time until i found the time to finish this PR and to
find a time window that does not disturb other developments too much,
but here we are:_
this PR enables UI to improve "Saved messages" hugely, bringing it on
WhatsApp's "Starred Messages" or Telegram's "Saved Messages" level. with
this PR, UIs can add the following functionality with few effort ([~100
loc on iOS](https://github.com/deltachat/deltachat-ios/pull/2527)):
- add a "Save" button in the messages context menu, allowing to save a
message
- show directly in the chat if a message was saved, eg. by a little star
★
- in "Saved Messages", show the message in its context - so with author,
avatar, date and keep incoming/outgoing state
- in "Saved Messages", a button to go to the original message can be
added
- if the original message was deleted, one can still go to the original
chat
these features were often requested, in the forum, but also in many
one-to-one discussions, recently at the global gathering.
moreover, in contrast to the old method with "forward to saved", no
traffic is wasted - the messages are saved locally, and only a small
sync messages is sent around
this is how it looks out on iOS:
<img width="260" alt="Screenshot 2025-01-17 at 00 02 51"
src="https://github.com/user-attachments/assets/902741b6-167f-4af1-9f9a-bd0439dd20d0"
/> <img width="353" alt="Screenshot 2025-01-17 at 00 05
33"
src="https://github.com/user-attachments/assets/97eceb79-6e43-4a89-9792-2e077e7141cd"
/>
technically, still a copy is done from the original message (with
already now deduplicated blobs), so that things work nicely with
deletion modes; eg. you can save an important message and preserve it
from deletion that way.
jsonrpc can be done in a subsequent PR, i was implementing the UI on iOS
where that was not needed (and most API were part of message object that
is not in use in jsonrpc atm)
@hpk42 the forward issue we discussed earller that day is already solved
(first implementation did not had an explict save_msgs() but was using
forward_msgs(SELF) as saving - with the disadvantage that forwarding to
SELF is not working, eg. if one wants the old behaviour) acutally, that
made the PR a lot nicer, as now very few existing code is changed
<details>
<summary>previous considerations and abandoned things</summary>
while working on this PR, there was also the idea to just set a flag
“starred” in the message table and not copy anything. however, while
tempting, that would result in more complexity, questions and drawbacks
in UI:
- delete-message, delete-chat, auto-deletion, clear-chat would raise
questions - what do do with the “Starred”? having a copy in “Saved
Messages” does not raise this question
- newly saved messages appear naturally as “new” in “Saved Messages”,
simply setting a flag would show them somewhere in between - unless we
do additional effort
- “Saved Messages” already has its place in the UI - and allows to
_directly_ save things there as well - not easily doable with “starring”
- one would need to re-do many things that already exist in “Saved
Messages”, at least in core
- one idea to solve some of the issues could be to have “Starred” as
well as “Saved Messages” - however, that would irritate user, one would
remember exactly what was done with a message, and understand the fine
differences
whatsapp does this “starred”, btw, so when original is deleted, starred
is deleted as well. Telegram does things similar to us, Signal does
nothing. Whatsapp has a per-chat view for starred messages - if needed,
we could do sth. like that as well, however, let’s first see how far the
“all view” goes (in contrast to whatsapp, we have profiles for
separation as well)
for the wording: “saving” is what we’re doing here, this is much more on
point as “starring” - which is more the idea of a “bookmark”, and i
think, whatsapp uses this wording to not raise false expectations
(still, i know of ppl that were quite upset that a “starred” message was
deleted when eg. the chat was cleared to save some memory)
wrt webxdc app updates: options that come into mind were: _empty_ (as
now), _snapshot_ (copy all updates) or _shortcut_ (always open
original). i am not sure what the best solution is, the easiest was
_empty_, so i went for that, as it is (a) obvious, and what is already
done with forwarding and (b) the original is easy to open now (in
contrast to forwarding).
so, might totally be that we need or want to tweak things here, but i
would leave that outside the first iteration, things are not worsened in
that area
wrt reactions: as things are detached, similar to webxdc updates, we do
not not to show the original reactions - that way, one can use reactions
for private markers (telegram is selling that feature :)
to the icon: a disk or a bookmark might be other options, but the star
is nicer and also know from other apps - and anyways a but vague UX
wise. so i think, it is fine
finally, known issue is that if a message was saved that does not exist
on another device, it does not get there. i think, that issue is a weak
one, and can be ignored mostly, most times, user will save messages soon
after receiving, and if on some devices auto-deletion is done, it is
maybe not even expected to have suddenly another copy there
</details>
EDIT: once this is merged, detailed issues about what to do should be
filed for android/desktop (however, they do not have urgency to adapt,
things will continue working as is)
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
This partially restores the fix from c9cf2b7f2e
that was removed during the addition of new group consistency at de63527d94
but only for "Member added" messages.
Multiple "Member added" messages happen
when the same QR code is processed multiple times
by multiple devices.
Some migrations change the `config` table, but they don't update the
cache. While this wasn't the cause for
https://github.com/deltachat/deltachat-core-rust/issues/6432, it might
have caused a similar bug, so, let's clear the config cache after every
migration.
Users report that in a setup with Android (1.50.4 from F-Droid) and Desktop (1.48.0 x86_64 .deb
release) and chatmail account `bcc_self` was reverted to 0 on Android, resulting in messages sent
from Android not appearing on Desktop. This might happen because of the bug in migration #127, it
doesn't handle `delete_server_after` > 1. Existing chatmail configurations having
`delete_server_after` != 1 ("delete at once") should get `bcc_self` enabled, they may be multidevice
configurations:
- Before migration #127, `delete_server_after` was set to 0 upon a backup export, but
then `bcc_self` is enabled instead (whose default is changed to 0 for chatmail).
- The user might set `delete_server_after` to a value other than 0 or 1 when that was
possible in UIs.
So let's add another migration fixing this. But still don't check `is_chatmail` for simplicity.
Using `repeat_vars()` to generate SQL statements led to some of them having more than
`SQLITE_MAX_VARIABLE_NUMBER` parameters and thus failing, so let's get rid of this pattern. But
let's not optimise for now and just repeat executing an SQL statement in a loop, all the places
where `repeat_vars()` is used seem not performance-critical and containing functions execute other
SQL statements in loops. If needed, performance can be improved by preparing a statement and
executing it in a loop. An exception is `lookup_chat_or_create_adhoc_group()` where `repeat_vars()`
can't be replaced with a query loop, there we need to replace the `SELECT` query with a read
transaction creating a temporary table which is used to perform the SELECT query then.
Before, `Chat::why_cant_send()` just returned `CantSendReason` after the first unsuccessful check
allowing to handle the result and finally send the message if the condition is acceptable in which
case the remaining checks are not done. This didn't result in any bugs, but to make the code more
robust let's add a functional parameter to filter failed checks without early return.
This implements new group consistency algorithm described in
<https://github.com/deltachat/deltachat-core-rust/issues/6401>
New `Chat-Group-Member-Timestamps` header is added
to send timestamps of member additions and removals.
Member is part of the chat if its addition timestamp
is greater or equal to the removal timestamp.
Python 3.7 is not supported on GitHub Actions ubuntu-latest runner:
https://github.com/actions/setup-python/issues/962
Python 3.7 has reached EOL more than 1 year ago anyway,
so not worth the effort supporting it.
This test keeps failing on macOS CI,
capturing events like `DC_EVENT_ACCOUNTS_ITEM_CHANGED`
before FailPlugin is setup.
These CI runners likely get less resources
because there is a limited number of them,
and this triggers this race condition.
Race is fixed by setting up fail plugin
before starting to capture events.
Currently Delta Chat puts self address in the To field
to avoid the To field being empty.
There is a plan to put empty `hidden-recipients`
group there, this fix prepares the receiver for such messages.
New Delta Chat is going to send self-sent messages
with undisclosed recipients instead of placing self into the `To` field.
To avoid assigning broadcast list messages to Saved Messages chat,
we should check the mailing list headers
before attempting to assign to Saved Messages.
First of all, chatmail servers normally forbid to send unencrypted mail, so if we know the peer's
key, we should encrypt to it. Chatmail setups have `E2eeEnabled=1` by default and this isn't
possible to change in UIs, so this change fixes the chatmail case. Additionally, for chatmail, if a
peer has `EncryptPreference::Reset`, let's handle it as `EncryptPreference::NoPreference` for the
reason above. Still, if `E2eeEnabled` is 0 for a chatmail setup somehow, e.g. the user set it via
environment, let's assume that the user knows what they do and ignore `IsChatmail` flag.
NB:
- If we don't know the peer's key, we should try to send an unencrypted message as before for a
chatmail setup.
- This change doesn't remove the "majority rule", but now the majority with
`EncryptPreference::NoPreference` can't disable encryption if the local preference is `Mutual`. To
disable encryption, some peer should have a missing peerstate or, for the non-chatmail case, the
majority should have `EncryptPreference::Reset`.
Change `BccSelf` default to 0 for chatmail configurations and enable it upon a backup export. As for
`DeleteServerAfter` who was set to 0 upon a backup export before, make its default dependent on
`BccSelf` for chatmail. We don't need `BccSelf` for chatmail by default because we assume
single-device use. Also `BccSelf` is needed for "classic" email accounts even if `DeleteServerAfter`
is set to "immediately" to detect that a message was sent if SMTP server is slow to respond and
connection is lost before receiving the status line which isn't a problem for chatmail servers.
Even if UIs don't call `Message::force_sticker()`, they don't want conversions of `Sticker` to
`Image` if it's obviously not an image, particularly, has non-image extension. Also UIs don't want
conversions of `Sticker` to anything other than `Image`, so let's keep the `Sticker` viewtype in
this case.
In multi-device case `vg-request-with-auth` left on IMAP may result in situation when Bob joins the
group, then leaves it, then second Alice device comes online and processes `vg-request-with-auth`
again and adds Bob back. So we should IMAP-delete `vg-request-with-auth`. Another device will know
the Bob's key from Autocrypt-Gossip. It's not a problem if Alice loses state (restores from an old
backup) or goes offline for long before sending `vg-member-added`, anyway it may not be delivered by
the server, rather Bob should retry sending SecureJoin messages as he is a part which wants to join,
so let's not solve this for now.
Add a `create` param to `select_with_uidvalidity()` instead of always trying to create the folder
and return `Ok(false)` from it if the folder doesn't exist and shouldn't be created, and handle this
in `store_seen_flags_on_imap()` by just removing "jobs" from the `imap_markseen` table. Also don't
create the folder in other code paths where it's not necessary.
This is to avoid revalidating HTTP cache too frequently (and have many parallel revalidation tasks)
if revalidation fails or the HTTP request takes some time. The stale period >= 1 hour, so 1 more
minute won't be a problem.
It was called from `receive_imf` when an outgoing message is received. But
`Imap::fetch_new_messages()` already calls `chat::mark_old_messages_as_noticed()` which does the job
better (per-message).
SQL statements fail if the number of variables
exceeds `SQLITE_LIMIT_VARIABLE_NUMBER`.
Remaining repeat_vars() calls are difficult to replace
and use arrays passed from the UI,
e.g. forwarded message IDs or marked as seen IDs.
UIs may want to display smth like "Transferring..." after "Establishing connection between
devices..." on nonzero progress. Before, progress on the receiver side was starting with 2 after
receiving enough data.
It only contained proxy
that some users ran in Termux
to look at IMAP traffic.
The same can be achieved with `socat`, e.g.:
socat -v TCP-LISTEN:9999,bind=127.0.0.1 OPENSSL:nine.testrun.org:993
This fixes the HTML display of messages containing forwarded messages. Before, forwarded messages
weren't rendered in HTML and if a forwarded message is long and therefore truncated in the chat, it
could only be seen in the "Message Info". In #4462 it was suggested to display "Show Full
Message..." for each truncated message part and save to `msgs.mime_headers` only the corresponding
part, but this is a quite huge change and refactoring and also it may be good that currently we save
the full message structure to `msgs.mime_headers`, so i'd suggest not to change this for now.
Otherwise the "Show Full Message..." button appears somewhere in the middle of the multipart
message, e.g. after a text in the first message bubble, but before a text in the second
bubble. Moreover, if the second/n-th bubble's text is shortened (ends with "[...]"), the user should
scroll up to click on "Show Full Message..." which doesn't look reasonable. Scrolling down looks
more acceptable (e.g. if the first bubble's text is shortened in a multipart message).
I'd even suggest to show somehow that message bubbles belong to the same multipart message, e.g. add
"[↵]" to the text of all bubbles except the last one, but let's discuss this first.
without the prefix,
it looks as if it is part of the Message-ID,
esp. if Message-ID is longer,
a break on different delimiters may look exactly the same.
see #6329 for some screenshots that initially confused me :)
A user reported to me that after they left a group, they were implicitly readded, but there's no any
readdition message, so currently it looks in the chat like leaving it has no effect, just new
messages continue to arrive. The readdition probably happened because some member didn't receive the
user's self-removal message, anyway, at least there must be a message that the user is readded, even
if it isn't known by whom.
A NDN may arrive days after the message is sent when it's already impossible to tell which message
wasn't delivered looking at the "Failed to send" info message, so it only clutters the chat and
makes the user think they tried to send some message recently which isn't true. Moreover, the info
message duplicates the info already displayed in the error message behind the exclamation mark and
info messages do not point to the message that is failed to be sent.
Moreover it works rarely because `mimeparser.rs` only parses recipients from `x-failed-recipients`,
so it likely only works for Gmail. Postfix does not add this `X-Failed-Recipients` header. Let's
remove this parsing too. Thanks to @link2xt for pointing this out.
This PR:
- Moves the note about the false positive to the end of the test output,
where it is more likely to be noticed
- Also notes in test_modify_chat_disordered() and
test_setup_contact_*(), in addition to the existing note in
test_was_seen_recently()
This fixes HTML display of truncated (long) sent messages ("Show full message" in UIs). Before,
incorrect HTML was stored (with missing line breaks etc.) for them. Now stored plaintext is
formatted to HTML upon calling `MsgId::get_html()` and this results in the same HTML as on a
receiver side.
Also cleaned up test_connectivity()
which tested that state does not flicker to WORKING
when there are no messages to be fetched.
The state is expected to flicker to WORKING
when checking for new messages,
so the tests were outdated since
change 3b0b2379b8
The benchmark function (e.g. `recv_all_emails()`) is executed multiple
times on the same context. During the second iteration, all the emails
were already in the database, so, receiving them again failed.
This PR fixes that by passing in a second `iteration` counter that is
different for every invocation of the benchmark function.
This better reflects that this state means
we just connected and there may me work to do.
This state is converted to DC_CONNECTIVITY_WORKING
instead of DC_CONNECTIVITY_CONNECTED state now.
Before this change when IMAP connected
to the server, it switched
from DC_CONNECTIVITY_NOT_CONNECTED
to DC_CONNECTIVITY_CONNECTING,
then to DC_CONNECTIVITY_CONNECTED (actually preparing)
then to DC_CONNECTIVITY_WORKING
and then to DC_CONNECTIVITY_CONNECTED again (actually idle).
On fast connections this resulted in flickering "Connected"
string in the status bar right before "Updating..."
and on slow connections this "Connected" state
before "Updating..." lasted for a while
leaving the user to wonder if there are no new messages
or if Delta Chat will still switch to "Updating..."
before going into "Connected" state again.
Already apply rust beta (1.84) clippy suggestions now, before they let
CI fail in 6 weeks.
The newly used functions are available since 1.70, our MSRV is 1.77, so
we can use them.
A sync message for accepting or blocking a 1:1 chat may arrive before the first message from the
contact, when it does not exist yet. This frequently happens in non-chatmail accounts that have
moving to the DeltaChat folder disabled because Delta Chat unconditionally uploads sync messages to
the DeltaChat folder. Let's create a hidden contact in this case and a 1:1 chat for it.
this partly reverts experimental #3516
that allowed any .xdc sent to "Saved Messages" to request internet.
this helped on pushing map integration forward.
meanwhile, however, we have that map integration (#5461 and #5678),
that implies `info.internet_access` being set.
experimental `manifest.request_internet_access` is no longer needed therefore.
future will tell, if we revive the option at some point or
go for more intrations ('sending' is discussed often :) -
but currently it is not needed.
with this PR, when an `.xdc` with `request_integration = map` in the
manifest is added to the "Saved Messages" chat, it is used _locally_ as
an replacement for the shipped maps.xdc (other devices will see the
`.xdc` but not use it)
this allows easy development and adapting the map to use services that
work better in some area.
there are lots of known discussions and ideas about adding more barriers
of safety. however, after internal discussions, we decided to move
forward and also to allow internet, if requested by an integration (as
discussed at
https://github.com/deltachat/deltachat-core-rust/pull/3516).
the gist is to ease development and to make users who want to adapt,
actionable _now_, without making things too hard and adding too high
barriers or stressing our own resources/power too much.
note, that things are still experimental and will be the next time -
without the corresponding switch being enabled, nothing will work at
all, so we can be quite relaxed here :)
for android/ios, things will work directly. for desktop, allow_internet
needs to be accepted unconditionally from core. for the future, we might
add a question before using an integration and/or add signing. or sth.
completely different - but for now, the thing is to get started.
nb: "integration" field in the webxdc-info is experimental as well and
should not be used in UIs at all currently, it may vanish again and is
there mainly for simplicity of the code; therefore, no need to document
that.
successor of https://github.com/deltachat/deltachat-core-rust/pull/5461
this is how it looks like currently - again, please note that all that
is an experiment!
<img width=320
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/f659c891-f46a-4e28-9d0a-b6783d69be8d>
<img width=320
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/54549b3c-a894-4568-9e27-d5f1caea2d22>
... when going out of experimental, there are loots of ideas, eg.
changing "Start" to "integrate"
following `RELEASE.md`, after merging, the following is needed:
6. Tag the release: `git tag --annotate v1.151.2`.
7. Push the release tag: `git push origin v1.151.2`.
8. Create a GitHub release: `gh release create v1.151.2 --notes ''`.
this PR adds the `href` from `update.href` to the IncomingWebxdcNotify
event (DC_EVENT_INCOMING_WEBXDC_NOTIFY in cffi)
purpose is to add a "Start" button to the notifications that allow
starting the app immediately with the given href
using long options make things less mystical, it is clearer what
happens.
(apart from that i also disallowed `git -a` on my machine in general, as
`git commit -a` is considered harmful. as my approach to disallow that
is a bit greedy and disallows `-a` just for any git commands, this is
the only place where i regularly struggle :)
Text parts are using quoted-printable encoding
which takes care of wrapping long lines,
so using format=flowed is unnecessary.
This improves compatibility with receivers
which do not support format=flowed.
Receiving format=flowed messages is still possible, receiver side of
Delta Chat is unchanged.
- **feat: add `AccountsChanged` and `AccountsItemChanged` events**
- **emit event and add tests**
closes#6106
TODO:
- [x] test receiving synced config from second device
- [x] bug: investigate how to delay the configuration event until it is
actually configured - because desktop gets the event but still shows
account as if it was unconfigured, maybe event is emitted before the
value is written to the database?
- [x] update node bindings constants
this PR removes most usages of the `descr` parameter.
- to avoid noise in different branches etc. (as annoying on similar, at
a first glance simple changes), i left the external API stable
- also, the effort to do a database migration seems to be over the top,
so the column is left and set to empty strings on future updates - maybe
we can recycle the column at some point ;)
closes#6245
add `update.href` property option to update objects send via
`Context::send_webxdc_status_update()`.
when set together with `update.info`,
UI can implement the info message as a link that is passed to the webxdc
via `window.location.href`.
for that purpose, UI will read the link back from
`Message::get_webxdc_href()`.
Practically,
this allows e.g. an calendar.xdc
to emits clickable update messages
opening the calendar at the correct date.
closes#6219
documentation at https://github.com/webxdc/website/pull/90
it may be handy for an xdc to have only one list of all adresses, or
there may just be bugs.
in any case, do not notify SELF, e.g. in a multi-device setup; we're
also not doing this for other messages.
this is also a preparation for having an option to notify ALL.
I found the old documentation rather hard to understand. The new doc
string:
- uses whole sentences, leaving less space for misinterpretation
- explicitly mentions that it can happen that there is no
webxdc-info-message
- is clearly structured using bullet points.
this PR adds support for the property `update.notify` to notify about
changes in `update.info` or `update.summary`. the property can be set to
an array of addresses [^1]
core emits then the event `IncomingWebxdcNotify`, resulting in all UIs
to display a system notification, maybe even via PUSH.
for using the existing `update.info` and `update.summary`: the message
is no secret and should be visible to all group members as usual, to not
break the UX of having same group messages on all devices of all users -
as known already from the normal messages.
also, that way, there is no question what happens if user have disabled
notifications as the change is presented in the chat as well
doc counterpart at https://github.com/webxdc/website/pull/90closes#6217
[^1]: addresses come in either via the payload as currently or as an
explicit sender in the future - this does not affect this PR. same for
translations, see discussions at #6217 and #6097
---------
Co-authored-by: adb <asieldbenitez@gmail.com>
Co-authored-by: l <link2xt@testrun.org>
this PR adds the address to be used by the UI for
`window.webxdc.selfAddr` to webxdc-info. UIs need to be changed
accordingly and must not use configured_addr any longer.
the address is created by sha256(private-key + rfc724_mid) , which
results in different addresses for each webxdc, without the option to
find out the real address of the user.
this also returns the same address for a multi-device-setup - sending
totally random self address around might be an alternative, however
would require connectivity (both devices may be offline on first start).
for existing app, after the change, there will be a new user, resulting
eg. in a new highscore, otherwise, things should be mostly fine. this
assumption is also important as we might change the thing another time
when it comes to multi-transport.
ftr, addresses look like
`0f187e3f420748b03e3da76543e9a84ecff822687ce7e94f250c04c7c50398bc` now
when this is merged, we need to adapt #6230 and file issues for all UI
to use `info.selfAddr`
closes#6216
`receive_imf() is only used in tests and the REPL, which enables the
"internals" feature. This PR marks it as such, so that it's clear not
only from the comment that this function is not used for anything else.
If a message partially downloaded before is already IMAP-seen upon a full download, it should be
updated to `InSeen`. OTOH if it's not IMAP-seen, but already `InNoticed` locally, its state should
be preserved. So we take the maximum of two states.
This fixes sending MDNs for big messages when they are downloaded and really seen. Otherwise MDNs
are not sent for big encrypted messages because they "don't want MDN" until downloaded.
Add a test on what happens currently when apps call `markseen_msgs()` for not downloaded encrypted
messages. Such messages are marked as seen, but MDNs aren't sent for them. Also currently when such
a message is downloaded, it remains `InSeen` despite the full content hasn't yet been seen by the
user.
As discussed in #5467 we want to use `i.delta.chat` in QR codes in favor
of `OPENPGP4FPR:` scheme. This PR does the replacement in
`get_securejoin_qr` which is used in `get_securejoin_qr_svg`.
close#5467
This is needed for iOS (https://github.com/deltachat/deltachat-ios/pull/2393), see comment in the code. An alternative would be
to add an API `invalidate_config_cache()` or to do nothing and just
assume that things will be fine.
There is a change in behavior for the case
when name is the same as the suffix
(`name_len` == `namespc_len`),
but normally `files_in_use` should not contain empty filenames.
The message may be deleted while chatlist item is loading.
In this case displaying "No messages" is better than failing.
Ideally loading of the chatlist item
should happen in 1 database transaction and
always return some message if chat is not empty,
but this requires large refactoring.
this PR changes `DC_CERTCK_ACCEPT_*` to the same values in cffi as rust
does. and regards the same values as deprecated afterwards
there is some confusion about what is deprecated and what not, see
https://github.com/deltachat/deltachat-android/issues/3408
iOS needs to be adapted as it was following the docs in the CFFI before,
same desktop. both need to be graceful on reading and strict on writing.
~~**this PR is considered harmful,** so we should not merge that in
during 1.48 release, there is no urgency, things are fine (wondering if
it isn't even worth the effort, however, having different values and
deprecations is a call for trouble in the future ...)~~
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
due to async processing,
it may happen getConnectivityHtml() is called from UI before startIO() is actually called.
eg. on iOS, we may delay startIo() if another process is still processing a PUSH notification -
when during this time, the connectivity view is opened,
it is weird if a big error "CONTACT THE DEVELOPERS!11!!!" is shown :)
also, there is not really a function is_connected(),
for $reasons, as this turned out to be flacky,
so it is not even easy to check the state before calling getConnectivityHtml()
it is not worth in doing too much special,
we are talking about rare situaton,
also, the connectivity view gets updated some moments later.
Before I was getting
```
error: attribute 'targetPlatforms' missing
at /nix/store/dyzl40h25l04565n90psbhzgnc5vp2xr-source/pkgs/build-support/rust/build-rust-package/default.nix:162:7:
161| meta.platforms or lib.platforms.all
162| rustc.targetPlatforms;
| ^
163| };
```
This was probably an upstream issues as discussed in here
https://discourse.nixos.org/t/error-attribute-targetplatforms-missing-after-updating-inputs/54494
After this update it is fixed.
80 characters are a bit limited in practise ...
On Mon, 3 Jan, 2022 at 8:34 PM "Anonymous The Mighty" <anonymous@example.com> wrote:
... already breaks the limit. it is good to allow up to 40 additional characters
for name + email address.
allowing any length, however, may catch too much,
as the line could also be a normal paragraph with important content,
so 120 characters seems reasonable.
the idea of adding more complexity here would probably lead only to, well more complexity -
things can anyways go wrong -
and, we have the "show full message..." button for exactly that purpose,
so that the user can access everything as original.
so, if things go wrong sometimes,
this is expected and fine.
There were many cases in which "member added/removed" messages were added to chats even if they
actually do nothing because a member is already added or removed. But primarily this fixes a
scenario when Alice has several devices and shares an invite link somewhere, and both their devices
handle the SecureJoin and issue `ChatGroupMemberAdded` messages so all other members see a
duplicated group member addition.
This change adds support for receiving
Autocrypt header in the protected part of encrypted message.
Autocrypt header is now also allowed in mailing lists.
Previously Autocrypt header was rejected when
List-Post header was present,
but the check for the address being equal to the From: address
is sufficient.
New experimental `protect_autocrypt` config is disabled
by default because Delta Chat with reception
support should be released first on all platforms.
Over the past years, it happend two times that a user came to me worried
about a false-positive "Cannot login as ***. Please check if the e-mail
address and the password are correct." message.
I'm not sure why this happened, but this PR makes the logic for
showing this notification stricter:
- Before: The notification is shown if connection fails two times in a
row, and the second error contains the word "authentication".
- Now: The notification is shown if the connection fails two times in a
row, and _both_ error messages contain the word "authentication".
The second commit just renames `login_failed_once` to
`authentication_failed_once` in order to reflect this change.
Delta Chat for Android does not support Android 4 anymore,
so there is no reason to keep using unsupported NDK.
r27 is the latest LTS version of Android NDK.
Tested:
- `nix build .#deltachat-rpc-server-arm64-v8a-android`
- `nix build .#deltachat-rpc-server-armv6l-linux`
`nix build .#deltachat-rpc-server-x86_64-android`
and
`nix build .#deltachat-rpc-server-x86-android`
still fail, but we do not build it in CI.
Explicit check for `-----BEGIN PGP MESSAGE-----` is unnecessary
and not sufficient to ensure that the message is valid.
We have already checked the MIME type,
so ASCII-armored OpenPGP message should be inside.
If it's not, decryption will fail anyway.
close#2338
Concat error messages when receiving new ndns.
This PR adds a newline followed by the new NDN error to the error text.
Maybe we should use something more prominent like
```
-----------------------------------------------------------------------
```
or more newlines, but I'm not sure. This maybe has to be tested on a
real device to see what works best.
3f9242a saves name from all QR codes to `name` (i.e. manually edited name), but for SecureJoin QR
codes the name should be saved to `authname` because such QR codes are generated by the
inviter. Other QR codes may be generated locally and not only by Delta Chat, so the name from them
mustn't go to `authname` and be revealed to the network or other contacts.
If the number of retries for message is exceeded,
do not fail when marking it as failed if the message does not exist.
Otherwise we may never delete the message from SMTP queue
because corresponding msg_id is not valid anymore.
The server should decode the URL and according to RFC 3986
query parameters may or may not be URL-encoded,
but at some servers don't decode the dot correctly.
`@` is decoded correctly by autoconfig.murena.io
This may indicate that there was a new \Seen flag
that we don't want to skip.
Also don't drain unsolicited responses while scanning folders. Now we
only drain unsolicited responses right before IDLE and always redo the
whole fetch cycle if there have been some. Some message in the scanned
folder may not be fetched that would be previously fetched otherwise,
but it will be picked up on the next folder scan.
Just a small refactoring. Instead of rebinding res all the time just use
`and` and `and_then`how they are inteded to be used. Improves code
readability imo.
This adds a function to `Message`:
```rust
pub fn new_text(text: String) -> Self {
Message {
viewtype: Viewtype::Text,
text,
..Default::default()
}
}
```
I keep expecting that a function like this must exist and being
surprised that it doesn't.
Open question is whether it should be `pub` or `pub(crate)` - I made it
`pub` for now because it may be useful for others and we currently we
aren't thinking about the Rust API that much, anyway, but I can make it
`pub(crate)`, too (then it can't be used in deltachat-jsonrpc and
deltachat-repl).
I replaced some usages of Message::new(Viewtype::Text), but not all yet,
I'm going to do this in a follow-up, which will remove another around 65
LOC.
Right now, when there is an SMTP connection error, the connectivity view
will always show "Error: SMTP connection failure: SMTP failed to
connect".
Instead, I just used the same method that is used in imap connect()
already.
0a63083df7 (fix: Shorten message text in locally sent messages too)
sets `msgs.mime_modified` for long outgoing messages, but forgets to save full message text.
I.e. add the "Messages are guaranteed to be end-to-end encrypted from now on." message and mark the
chat as protected again because no user action is required in this case. There are a couple of
problems though:
- If the program crashes earlier than the protection is restored, the chat remains
protection-broken. But this problem already exists because `ChatId::set_protection()` is never
retried.
- If multiple old unverified messages are received, protection messages added in between don't
annihilate, so they clutter the chat.
Let's always set `Config::NotifyAboutWrongPw` before saving configuration, better if a wrong
password notification is shown once more than not shown at all. It shouldn't be a big problem
because reconfiguration is a manual action and isn't done frequently.
Also for the same reason reset `Config::NotifyAboutWrongPw` only after a successful addition of the
appropriate device message.
Otherwise it always fails with SQLITE_READONLY:
```
WARNING src/sql.rs:769: Failed to run incremental vacuum: attempt to write a readonly database: Error code 8: Attempt to write a readonly database.
```
this PR adds a function that can be used to create any QR code, in a raw
form.
this can be used to create add-contact as well as add-second-device QR
codes (eg. `dc_create_qr_svg(dc_get_securejoin_qr())`) - as well as for
other QR codes as proxies.
the disadvantage of the rich-formatted QR codes as created by
`dc_get_securejoin_qr_svg()` and `dc_backup_provider_get_qr_svg()` were:
- they do not look good and cannot interact with UI layout wise (but
also tapping eg. an address is not easily possible)
- esp. text really looks bad. even with
[some](e5dc8fe3d8)
[hacks](https://github.com/deltachat/deltachat-android/pull/2215) it
[stays buggy](https://github.com/deltachat/deltachat-ios/issues/2200);
the bugs mainly come from different SVG implementation, all need their
own quirks
- accessibility is probably bad as well
we thought that time, SVG is a great thing for QR codes, but apart from
basic geometrics, it is not.
so, we avoid text, this also means to avoid putting an avatar in the
middle of the QR code (we can put some generic symbol there, eg.
different ones for add-contact and add-second-device).
while this looks like a degradation, also other messengers use more raw
QR codes. also, we removed many data from the QR code anyway, eg. the
email address is no longer there. that time, sharing QR images was more
a thing, meanwhile we have invite links, that are much better for that
purpose.
in theory, we could also leave the SVG path completely and go for PNG -
which we did not that time as PNG and text looks bad, as the system font
is not easily usable :) but going for PNG would add further challenges
as passing binary data around, and also UI-implemtation-wise, that would
be a larger step. so, let's stay with SVG in a form we know is
compatible.
the old QR code functions are deprecated.
this PR allows setting a "private tag" for a profile, see
https://github.com/deltachat/deltachat-android/pull/3373 for a possible
UI.
currently, the core does not do anything with the tag (so, it could also
be a ui.-config option), however, this may change in the future - it
might bet synced, and become also otherwise useful in core. also, having
this in core is better documentation-wise, as otherwise each UI easily
does its own things :)
This adds "stale-while-revalidate" in-memory cache for DNS. Instead of
calling `tokio::net::lookup_host` we use previous result of
`tokio::net::lookup_host` immediately and spawn revalidation task in the
background. This way all lookups after the first successful one return
immediately.
Most of the time results returned by resolvers are the same anyway, but
with this cache we avoid waiting 60 second timeout if DNS request is
lost. Common reason result may be different is round-robin DNS load
balancing and switching from IPv4 to IPv6 network. For round-robin DNS
we don't break load balancing but simply use a different result, and for
IPv6 we anyway likely have a result in persistent cache and can use IPv4
otherwise.
Especially frequent should be the case when you send a message over SMTP
and SMTP connection is stale (older than 60 s), so we open a new one.
With this change new connection will be set up faster as you don't need
to wait for DNS resolution, so message will be sent faster.
rust-analyzer was showing warnings here because it is always also
building in the Test configuration, and EventType has a
```rust
#[cfg(test)]
Test,
```
variant, which was not matched.
On main, when running `cargo build`, the following warning is emitted:
> warning:
/home/jonathan/deltachat-android/jni/deltachat-core-rust/deltachat-ffi/Cargo.toml:
`default-features` is ignored for deltachat, since `default-features`
was not specified for `workspace.dependencies.deltachat`, this could
become a hard error in the future
This is because when referring to a workspace dependency, it's not
possible to remove features, it's only possible to add features, so that
the `vendored` feature was always enabled with no possibility to disable
it.
This PR restores the wanted behavior of enabling vendoring by default
with the possibility to disable it with "default-features = false".
It fixes `nix build .#python-docs` by not passing
`--no-default-features` when building deltachat with nix.
If a message from an old contact's setup is received, the outdated Autocrypt header isn't applied,
so the contact verification preserves. But the chat protection breaks because the old message is
sorted to the bottom as it mustn't be sorted over the protection info message (which is `InNoticed`
moreover). Would be nice to preserve the chat protection too e.g. add a "protection broken" message,
then the old message and then a new "protection enabled" message, but let's record the current
behaviour first.
To avoid reordering, wait for "member removed" message
to be received before sending "member added".
The test failed at least once
because email server may reorder the messages internally
while delivering.
Otherwise instead of "old address"
ac2 may receive "member added",
resulting in this failure:
```
> assert msg_in_1.text == msg_out.text
E AssertionError: assert 'Member Me (c...hat.computer.' == 'old address'
E - old address
E + Member Me (ci-hfpxxe@***) added by ci-8e7mkr@***.
```
This ensures we do not get stuck trying DNS resolver results
when we have a known to work IP address in the cache
and DNS resolver returns garbage
either because it is a captive portal
or if it maliciously wants to get us stuck
trying a long list of unresponsive IP addresses.
This also limits the number of results we try to 10 overall.
If there are more results, we will retry later
with new resolution results.
It is impossible to set no display name anyway
in Delta Chat Android at least
because we don't want email addresses
in the UI.
This test does not work with long domains
that may get wrapped, so better remove it
instead of trying to prevent wrapping of domains.
Received messages shouldn't mingle with just sent ones and appear somewhere in the middle of the
chat, so we go after the newest non fresh message.
But if a received outgoing message is older than some `InSeen` message, better sort the received
message purely by timestamp (this is an heuristic in order not to break the Gmail-like case
simulated by `verified_chats::test_old_message_4()`). We could place the received message just
before that `InSeen` message, but anyway the user may not notice it.
At least this fixes outgoing messages sorting for shared accounts where messages from other devices
should be sorted the same way as incoming ones.
Otherwise backups exported from the current core and imported in versions < 1.144.0 have QR codes
not working. The breaking change which removed the column is
5a6efdff44.
Before file extensions were also limited to 32 chars, but extra chars in the beginning were just cut
off, e.g. "file.with_lots_of_characters_behind_point_and_double_ending.tar.gz" was considered to
have an extension "d_point_and_double_ending.tar.gz". Better to take only "tar.gz" then.
Also don't include whitespace-containing parts in extensions. File extensions generally don't
contain whitespaces.
There is already code below that emits
progress 0 or 1000 depending on whether
configuration succeeded or failed.
Before this change cancelling resulted
in progress 0 emitted,
immediately followed by progress 1000.
Before this change progress bar only started
when database is already transferred.
Database is usually the largest file
in the whole transfer, so the transfer appears
to be stuck for the sender.
With this change progress bar
starts for backup export
as soon as connection is received
and counts bytes transferred over the connection
using AsyncWrite wrapper.
Similarly for backup import,
AsyncRead wrapper counts the bytes
received and emits progress events.
Instead of treating NULL type error
as absence of the row,
handle NULL values with SQL.
Previously we sometimes
accidentally treated a single column
being NULL as the lack of the whole row.
The "Cannot establish guaranteed end-to-end encryption with ..." info
message can have lots of causes, and it happened twice to us now that it
took us some time to figure out which one it is.
So, include some more detail in the info message by simply adding the
non-translated error message in parantheses.
If we want to put in some more effort for nicer error messages, we
could:
- Introduce one new translated string "Cannot establish guaranteed
end-to-end encryption with …. Cause: %2$s" or similar (and remove the
old stock string)
- And/Or: Introduce new translated strings for all the possible errors
- And/Or: Maybe reword it in order to account better for the case that
the chat already is marked as g-e2ee, or use a different wording
(because if the chat is marked as g-e2ee then it might be nice to notify
the user that something may have gone wrong, but it's still working,
just that maybe the other side doesn't have us verified now)
Info messages are added
at the beginning of unpromoted group chats
("Others will only see this group after you sent a first message."),
may be created by WebXDC etc.
They are not sent outside
and have local Message-ID that
is not known to other recipients
so they should be skipped when constructing
In-Reply-To and References.
Without this fix IMAP loop may get stuck
trying to download non-existing message over and over
like this:
```
src/imap.rs:372: Logging into IMAP server with LOGIN.
src/imap.rs:388: Successfully logged into IMAP server
src/scheduler.rs:361: Failed to download message Msg#3467: Message Msg#3467 does not exist.
src/scheduler.rs:418: Failed fetch_idle: Failed to download messages: Message Msg#3467 does not exist
```
The whole download operation fails
due to attempt to set the state of non-existing message
to "failed". Now download of the message
will "succeed" if the message does not exist
and we don't try to set its state.
HTTPS requests are used to fetch
remote images in HTML emails,
to fetch autoconfig XML,
to POST requests for `DCACCOUNT:` QR codes
to make OAuth 2 API requests
and to connect to HTTPS proxies.
Rustls is more aggressive than OpenSSL
in deprecating cryptographic algorithms
so we cannot use it for IMAP and SMTP
to avoid breaking compatibility,
but for HTTPS requests listed
above this should not result in problems.
As HTTPS requests use only strict TLS checks,
there is no `strict_tls` argument
in `wrap_rustls` function.
Rustls is already used by iroh,
so this change does not introduce new dependencies.
This should fix ad-hoc groups splitting when messages are fetched out of order from different
folders or otherwise reordered, or some messages are missing so that the messages reference chain is
broken, or a member was removed from the thread and readded later, etc. Even if this way two
different threads are merged, it looks acceptable, having many threads with the same name/subject
and members isn't a common use case.
Instead of generating 72 random bits
and reducing them to 66 bits of Base64 characters,
generate 144 bits (18 bytes)
which is exactly 24 Base64 characters.
This should still be accepted by existing
Delta Chat clients which expect group ID
to be between 11 and 32 characters.
Message-ID creation is also simplified
to not have `Mr.` prefix
and dot in between two IDs.
Now it is a single ID followed by `@localhost`.
Some outdated documentation comments
are removed, e.g. group messages
don't start with `Gr.` already.
The greeting is now always read manually,
even for STARTTLS connections,
so the errors returned on failure to read form the stream
are the same regardless of the connection type.
Why:
- With IMAP APPEND we can upload messages directly to the DeltaChat folder (for non-chatmail
accounts).
- We can set the `\Seen` flag immediately so that if the user has other MUA, it doesn't alert about
a new message if it's just a sync message (there were several such reports on the support
forum). Though this also isn't useful for chatmail.
- We don't need SMTP envelope and overall remove some overhead on processing sync messages.
If a displayname equals to the address, adding it looks excessive.
Moreover, it's not useful for Delta Chat receiving the message because
`sanitize_name_and_addr()` removes such a displayname anyway. Also now
at least DC Android requires specifying profile name, so there should be
a fallback for users having meaningful addresses to keep the old
behaviour when Core generates `From` w/o the profile name, and this
question has already appeared on the forum.
This reverts commit 1caf672904.
Otherwise public key signature is regenerated each time the key is
loaded and test `key::tests::test_load_self_existing` which loads the
key twice fails when two loads happen on different seconds.
Closes#5976
Groups promotion to other devices and QR code tokens synchronisation are not synchronised processes,
so there are reasons why a QR code token may arrive earlier than the first group message:
- We are going to upload sync messages via IMAP while group messages are sent by SMTP.
- If sync messages go to the mvbox, they can be fetched earlier than group messages from Inbox.
`chat::create_send_msg_jobs()` already handles `Config::BccSelf` as needed. The only exception is
Autocrypt setup messages. This change unifies the logic for the self-chat and groups only containing
`SELF`.
This makes possible to schedule one more sending of the message, the existing jobs are not
cancelled. Otherwise it's complicated to implement bots that resend messages when a new member joins
the group.
This change removes OAuth2 for Gmail
as Delta Chat does not have a working
client ID anymore.
Tests are adjusted to test against Yandex
and MX queries for OAuth2 are always disabled
because they were only used to detect Google Workspace.
This change introduces new config options
`proxy_enabled` and `proxy_url`
that replace `socks5_*`.
Tested with deltachat-repl
by starting it with
`cargo run --locked -p deltachat-repl -- deltachat-db` and running
```
> set proxy_enabled 1
> set proxy_url ss://...
> setqr dcaccount:https://chatmail.example.org/new
> configure
```
And also:
- Make it `pub(crate)`.
- Use it in `should_request_mdns()` as using `config_exists()` there isn't correct because the
latter doesn't look at environment.
I.e. treat `DeleteServerAfter == None` as "delete at once". But when a backup is exported, set
`DeleteServerAfter` to 0 so that the server decides when to delete messages, in order not to break
the multi-device case. Even if a backup is not aimed for deploying more devices, `DeleteServerAfter`
must be set to 0, otherwise the backup is half-useful because after a restoration the user wouldn't
see new messages deleted by the device after the backup was done. But if the user explicitly set
`DeleteServerAfter`, don't change it when exporting a backup. Anyway even for non-chatmail case the
app should warn the user before a backup export if they have `DeleteServerAfter` enabled.
Also do the same after a backup import. While this isn't reliable as we can crash in between, this
is a problem only for old backups, new backups already have `DeleteServerAfter` set if necessary.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
There are providers in the provider database
that do not have servers specified.
For such providers default list should be tried
just like when configuring unknown providers.
Documentation comment says forward and backward verification is set,
but the code was not doing it.
`vc-contact-confirm` and `vg-member-added` messages
indicate that other device finished securejoin protocol
so we know Bob has our key marked as verified.
bcc_self has been enabled by default
since core version 1.95.0
by merging
PR <https://github.com/deltachat/deltachat-core-rust/pull/3612>.
However deltachat.h documentation
still incorrectly said that bcc_self is disabled by default.
This change replaces
usage of `reqwest` and `hyper-util`
with custom connection establishment code
so it is done in the same way
as for IMAP and SMTP connections.
This way we control HTTP, IMAP and SMTP
connection establishment
and schedule connection attempts
to resolved IP addresses
in the same way for all 3 protocols.
With current implementation
every time connection fails
we take the next delay from `delays` iterator.
In the worst case first 4 DNS results
immediately refuse connection
and we start fifth connection attempt
with 1 year timeout,
effectively continuing all remaining
connection attempts without concurrency.
With new implementation
new connection attempts are
added to `connection_attempt_set`
independently of connection failures
and after 10 seconds
we always end up with five
parallel connection attempts
as long as there are enough IP addresses.
Previously for each connection candidate (essentially host and port
pair) after resolving the host to a list of IPs Delta Chat iterated IP
addresses one by one. Now for IMAP and SMTP we try up to 5 IP addresses
in parallel. We start with one connection and add more connections
later. If some connection fails, e.g. we try to connect to IPv6 on IPv4
network and get "Network is unreachable" (ENETUNREACH) error, we replace
failed connection with another one immediately.
Co-authored-by: Hocuri <hocuri@gmx.de>
This ensures we don't add multiple Auto-Submitted headers
when bots send vg-request or vc-request messages.
The change fixes failing
receive_imf::tests::test_bot_accepts_another_group_after_qr_scan
test.
This fixes the bug that sometimes made QR scans fail.
The problem was:
When sorting headers into unprotected/hidden/protected, the From: header
was added twice for all messages: Once into unprotected_headers and once
into protected_headers. For messages that are `is_encrypted && verified
|| is_securejoin_message`, the display name is removed before pushing it
into unprotected_headers.
Later, duplicate headers are removed from unprotected_headers right
before prepending unprotected_headers to the message. But since the
unencrypted From: header got modified a bit when removing the display
name, it's not exactly the same anymore, so it's not removed from
unprotected_headers and consequently added again.
I.e. to all messages except "v{c,g}-request" as they sent out on a QR code scanning which is a
manual action and "vg-member-added" as formally this message is auto-submitted, but the member
addition is a result of an explicit user action. Otherwise it would be strange to have the
Auto-Submitted header in "member-added" messages of verified groups only.
Before the fix HTTP client
had no connection timeout,
so it only had a chance
to test one IPv6 and one IPv4
address if the first addresses timed out.
Now it can test at least 4 addresses
of each family and more if some addresses
refuse connection rather than time out.
Otherwise if DNS server returns incorrect results,
we may never try preloaded DNS results.
For example, we may get our first results
from a captive portal.
To test, add `127.0.0.1 example.org`
and try to create an account.
Without this change we only try 127.0.0.1 and fail.
With this change preloaded DNS results are tried as well.
this PR adds the type DC_QR_SOCKS5_PROXY to `dc_check_qr()` for
**supporting telegram proxy QR codes**. if returned, the UI should ask
the user if they want to us the proxy and call
`dc_set_config_from_qr();` afterwards (plus maybe `dc_configure()`).
idea is to improve our proxy story, follow ups may be:
- in UI, - move proxy out of "Account & Password", as a **separate
"Proxy Activity"** (it should stay in "Advanced" for now, however, below
"Server", which might be moved up)
- allow **opening the "Proxy Activity" from the welcome screens**
three-dot-menu (that would also solve a long standing issue that
entering the email address bypasses the proxy
- show proxy usage in the "Connectivity View" and/or add an **icon** to
the main chatlist screen (beside three-dot menu) in case some proxy is
in use; tapping this icon will open the "Proxy Activity"
- the the new "Proxy Activity", add a **share / show proxy QR code**
button. that would generate invite links in the form
`https://i.delta.chat/socks#...` - so that tapping then opens the app.
support for these links need to be added to core then.
- handle a list of proxies in core, offer selection in UI. the list
could be one for all profiles and could be filled eg. by normal invite
links or other channels
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Fix a typo in the config name
(by using `Config::` to avoid it)
and make sure we don't panic on unknown values.
Also test that we don't panic on unknown
`configured_imap_certificate_checks` values.
Apparently some providers fail TLS connection
with "no_application_protocol" alert
even when requesting "imap" protocol for IMAP connection
and "smtp" protocol for SMTP connection.
Fixes <https://github.com/deltachat/deltachat-core-rust/issues/5892>.
configured_imap_certificate_checks=0 means
accept invalid certificates unless provider database
says otherwise or SOCKS5 is enabled.
It should not be saved into the database anymore.
This bug was introduced in
<https://github.com/deltachat/deltachat-core-rust/pull/5854>
(commit 6b4532a08e)
and affects released core 1.142.4, 1.142.5 and 1.142.6.
Fix reverts faulty fix from
<https://github.com/deltachat/deltachat-core-rust/pull/5886>
(commit a268946f8d)
which changed the way configured_imap_certificate_checks=0
is interpreted and introduced problems
for existing setups with configured_imap_certificate_checks=0:
<https://github.com/deltachat/deltachat-core-rust/issues/5889>.
Existing test from previous fix is not reverted
and still applies.
Regression test is added to check that
configured_imap_certificate_checks
is not "0" for new accounts.
If user has not set any settings manually
and provider is not configured,
default to strict TLS checks.
Bug was introduced in
<https://github.com/deltachat/deltachat-core-rust/pull/5854>
(commit 6b4532a08e)
and affects released core 1.142.4 and 1.142.5.
The problem only affects accounts configured
using these core versions with provider
not in the provider database or when using advanced settings.
`imap::Session::store_seen_flags_on_imap()` handles messages from multiple folders, so not being
able to select one folder mustn't fail the whole function.
`cargo install` ignores lockfile by default.
Without lockfile current build fails
due to iroh-net 0.21.0 depending on `derive_more` 1.0.0-beta.6
but failing to compile with `derive_more` 1.0.0.-beta.7.
This particular error will be fixed by upgrading to iroh 0.22.0,
but using lockfile will avoid similar problems in the future.
- To avoid receiving undecryptable MDNs by bots and replying to them if the bot's key changes.
- MDNs from bots don't look useful in general, usually the user expects some reply from the bot, not
just that the message is read.
Recently there are many questions on the Delta Chat forum why some unexpected encrypted messages
appear in Inbox. Seems they are mainly sync messages, though that also obviously happens to
SecureJoin messages. Anyway, regardless of the `MvboxMove` setting, auto-generated outgoing messages
should be moved to the DeltaChat folder so as not to complicate co-using Delta Chat with other MUAs.
Otherwise cargo emits these warnings:
warning: .../deltachat-core-rust/deltachat-ffi/Cargo.toml: `default-features` is ignored for deltachat, since `default-features` was not specified for `workspace.dependencies.deltachat`, this could become a hard error in the future
warning: .../deltachat-core-rust/deltachat-rpc-server/Cargo.toml: `default-features` is ignored for deltachat, since `default-features` was not specified for `workspace.dependencies.deltachat`, this could become a hard error in the future
warning: .../deltachat-core-rust/deltachat-rpc-server/Cargo.toml: `default-features` is ignored for deltachat-jsonrpc, since `default-features` was not specified for `workspace.dependencies.deltachat-jsonrpc`, this could become a hard error in the future
Previously Delta Chat tried to use local part of email address as well.
This configuration is very uncommon,
but trying it doubled the time of configuration try
in the worst case, e.g. when the password is typed in incorrectly.
Single function smtp::connect::connect_stream
returns a stream of a single `dyn` type
that can be a TLS, STARTTLS or plaintext
connection established over SOCKS5 or directly.
This way we can't get an account with missing blobs if there's not enough disk space.
Also delete already unpacked files if all files weren't unpacked successfully. Still, there are some
minor problems remaining:
- If a db wasn't imported successfully, unpacked blobs aren't deleted because we don't know at which
step the import failed and whether the db will reference the blobs after restart.
- If `delete_and_reset_all_device_msgs()` fails, the whole `import_backup()` fails also, but after a
restart delete_and_reset_all_device_msgs() isn't retried. Probably errors from it should be
ignored at all.
Previously Delta Chat tried all DNS resolution results
in sequence until TCP connection is established successfully,
then tried to establish TLS on top of the TCP connection.
If establishing TLS fails, the whole
connection establishment procedure failed
without trying next DNS resolution results.
In particular, in a scenario
where DNS returns incorrect result
pointing e.g. to a server
that listens on the TCP port
but does not have correpsponding TLS certificate,
Delta Chat now will fall back to the cached result
and connect successfully.
while adapting strings for the recent change about read receipts,
https://github.com/deltachat/deltachat-core-rust/pull/5712 , it turns
out in discussions eg. at
https://github.com/deltachat/deltachat-android/issues/3179 that
untranslated english for the read receipts seem to be sufficient or even
better:
- do not reveal the sender's language
- unexpected languages are confusing - even if you chat in english, you
may get Chinese read receipts
- many clients do not show the text anyways, iirc, eg. Outlook display
the read receipts in context, and Delta Chat of course as well
- afaik, we're leaving comparable `multipart/report` untranslated as
well (sync, but also webxdc updates are practically english only)
- less code, fewer translations needed :)
There are many reasons why we may fail to find valid signatures in a message, e.g. we don't yet know
a public key attached in the same message, anyway, if From is forged, the message must be rejected.
Also always take the displayname from encrypted From, even if no valid signatures are found.
This is a hint for apps that a WebXDC icon should be shown in the summary, e.g. in the
chatlist. Otherwise it's not clear when it should be shown, e.g. it shouldn't be shown in a reaction
summary.
For hardcoded built-in DNS results
there is no cache entry in `dns_cache` table
so they cannot be prioritized if DNS resolution
never returned these results yet.
If there is no entry, a new one should be created.
SQL UPSERT does this.
Follow-up to 3cf78749df "Emit DC_EVENT_MSGS_CHANGED for
DC_CHAT_ID_ARCHIVED_LINK when the number of archived chats with unread messages increases (#3940)".
In general we don't want to make an extra db query to know if a noticied chat is
archived. Emitting events should be cheap, better to allow false-positive `MsgsChanged` events.
`Context::send_sync_msg()` mustn't be called from multiple tasks in parallel to avoid sending the
same sync items twice because sync items are removed from the db only after successful
sending. Let's guarantee this by calling `send_sync_msg()` only from the SMTP loop. Before
`send_sync_msg()` could be called in parallel from the SMTP loop and another task doing
e.g. `chat::sync()` which led to `test_multidevice_sync_chat` being flaky because of events
triggered by duplicated sync messages.
New protocol streams .tar into iroh-net
stream without traversing all the files first.
Reception over old backup protocol
is still supported to allow
transferring backups from old devices
to new ones, but not vice versa.
The "Chat-Group-Member-Removed" header is added to ad-hoc group messages as well, so we should check
for its presense before creating an ad-hoc group as we do for DC-style groups.
Before, update sending might be delayed due to rate limits and later merged into large
messages. This is undesirable for apps that want to send large files over WebXDC updates because the
message with aggregated update may be too large for actual sending and hit the provider limit or
require multiple attempts on a flaky SMTP connection.
So, don't aggregate updates if the size of an aggregated update will exceed the limit of 100
KiB. This is a soft limit, so it may be exceeded if a single update is larger and it limits only the
update JSON size, so the message with all envelopes still may be larger. Also the limit may be
exceeded when updates are sent together with the WebXDC instance when resending it as the instance
size isn't accounted to not complicate the code. At least this is not worse than the previous
behaviour when all updates were attached.
Bridge bots like matterdelta need to set a quoted text without referencing the quoted message, this
makes easier bridging messages from other platforms to Delta Chat or even bridging Delta Chat groups
in different accounts where you can not set a quoted message by the message id from another account.
Even though SMTP ALPN is not officially registered (unlike IMAP),
it is an obvious choice that will allow
to multiplex SMTP and other protocols on the same TLS port.
Newer node-gyp uses newer gyp
which requires newer python
that is not available in Debian 10
container that is
used for building node prebuilds
with old glibc.
Follow-up to 5fa7cff46. Let's still not send a sync message if the contact wasn't modified. This is
not very important, but just for consistency with the `chat::rename_ex()` behaviour.
This way it's clearer which key is which and also adding the key fingerprint to the file name avoids
overwriting another previously exported key. I think this is better than adding an incremental
number as we do for backups, there's no need to export a key several times to different files.
For purposes of building a message it's better to consider the self-chat as verified. Particularly,
this removes unencrypted name from the "From" header.
If currently there are no multi-device bots, let's disable sync messages for bots at all. Another
option is to auto-disable sync messages when `Config::Bot` is set, so sync messages can be reenabled
if needed. But let's leave this option for the future.
About the first TODO: I tried this out, but it didn't actually improve
things, for two reasons:
1. The trick with `#![cfg_attr(not(test),
warn(clippy::indexing_slicing))]` that enables the lint everywhere
except for tests doesn't work with workspace-wide lints. (Context: We
want to lint against indexing because it might panic, but in a test
panicking is fine, so we don't want to enable the lint in tests).
2. Most of our crates have different sets of lints right now, so it
would only be very few crates that use the workspace-wide list of lints.
About the second TODO:
It's not feasible right now to fully parse vCards, and for our
good-enough parser the current behavior is fine, I think. If we fail to
parse some realworld vCards because of this, we can still improve it.
Notification server uses APNS server
for heartbeat notifications,
so registering FCM tokens there
will result in failing to notify them
and unregistering them anyway.
It's possible that when rebasing a PR adding a migration a merge-conflict doesn't occur if another
migration was added in the target branch. Better to have at least runtime checks that the migration
version is correct. Looks like compile-time checks are not possible because Rust doesn't allow to
redefine constants, only vars.
Instead of constructing lists of protected,
unprotected and hidden headers,
construct a single list of headers
and then sort them into separate lists
based on the well-defined policy.
This also fixes the bug
where Subject was not present in the IMF header
for signed-only messages.
Closes#5713
Bot processes are run asynchronously, so we shouldn't send messages to a bot before it's fully
initialised and skipped existing messages for processing, i.e. before DC_EVENT_IMAP_INBOX_IDLE is
emitted.
we're always checking the configuration encrypted.
saying it is 'preferred' encrypted is misleading,
therfore, just remove it.
i do not think, it is worth saying that we do not query 'http',
this is clear from the source code.
moreover, fix two typos.
Looks like it has no sense to send any profile data (From/To names, self-status; self-avatar was
never sent even before) in MDNs, they aren't normal messages and aren't seen in a MUA. Better not to
reveal profile data to the network and even to contacts in MDNs and make them more lightweight.
Follow-up to b771311593. Since that commit names are not revealed in
verified chats, but during verification (i.e. SecureJoin) they are still sent unencrypted. Moreover,
all profile data mustn't be sent even encrypted before the contact verification, i.e. before
"v{c,g}-request-with-auth". That was done for the selfavatar in
304e902fce, now it's done for From/To names and the self-status as
well. Moreover, "v{c,g}-request" and "v{c,g}-auth-required" messages are deleted right after
processing, so other devices won't see the received profile data anyway.
Probably sync messages generated for a not yet configured account are useless because there are no
other devices yet. And even if this is not true, we don't want to depend on the order of setting
`Config::SyncMsgs` and other keys. Also w/o this the Python tests don't work if we start syncing
`Config::MvboxMove` because they don't expect that sync messages are sent while configuring
accounts.
NB: We don't restart IO from the synchronisation code, so `MvboxMove` isn't effective immediately if
`ConfiguredMvboxFolder` is unset, but only after a reconnect to IMAP.
Before, if the user deleted a message too quickly after sending, it was deleted only locally. The
fix is to remember for tombstones that the corresponding message should be deleted on the server
too.
This is a way to prevent redownloading locally deleted messages. Otherwise if a message is deleted
quickly after sending and `bcc_self` is configured, the BCC copy is downloaded and appears as a new
message as it happens for messages sent from another device.
Delta Chat -style groups have names w/o prefixes like "Re: " even if the user is added to an already
existing group, so let's remove prefixes from ad-hoc group names too. Usually it's not very
important that the group is a classic email thread existed before, this info just eats up screen
space. Also this way a group name is likely to preserve if the first message was missed.
SQLite search with `LIKE` is case-insensitive only for ASCII chars. To make it case-insensitive for
all messages, create a new column `msgs.txt_normalized` defaulting to `NULL` (so we do not bump up
the database size in a migration) and storing lowercased/normalized text there when the row is
created/updated. When doing a search, search over `IFNULL(txt_normalized, txt)`.
Before, if `Config::FetchExistingMsgs` is set, existing messages were received with the `InSeen`
state set, but for bots they must be `InFresh` and also `IncomingMsg` events should be emitted for
them so that they are processed by bots as it happens with new messages.
Since commit c0a17df344
(PR https://github.com/deltachat/deltachat-core-rust/pull/3402)
`send_smtp_messages` returns an error
as soon as it encounters the first message it failed to send.
Since this worked like this for about 2 years
without any problems, there is no need to revert the change,
but outdated comment should be removed.
If a display name should be protected (i.e. opportunistically encrypted), only put the corresponding
address to the unprotected headers. We protect the From display name only for verified chats,
otherwise this would be incompatible with Thunderbird and K-9 who don't use display names from the
encrypted part. Still, we always protect To display names as compatibility seems less critical here.
When receiving a messge, overwrite the From display name but not the whole From field as that would
allow From forgery. For the To field we don't really care. Anyway as soon as we receive a message
from the user, the display name will be corrected.
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Otherwise it's impossible to remove a member with missing key from a protected group. In the worst
case a removed member will be added back due to the group membership consistency algo.
The "I left the group" message can't be sent to a protected group if some member's key is missing,
in this case we should remain in the group. The problem should be fixed first, then the user may
retry to leave the group.
There was a comment in `fetch_existing_msgs()`: "Bots don't want those messages". If a bot doesn't
want this setting, why enable it? It's disabled by default anyway.
`IsChatmail` is set also by `inbox_fetch_idle()`, but it isn't called during `configure()`. Setting
`IsChatmail` from `inbox_fetch_idle()` is necessary to handle client/server upgrades, but
`IsChatmail` also should be available for the app after configuring an account, e.g. DC Android
needs it to know whether to ask the user to disable battery optimisations.
Before this fix LogSink was dropped immediately,
resulting in no logs printed for contexts created using
TextContext::new_alice(), but printed for contexts created
using TextContext::new().
`!to_ids().is_empty()` check is needed in cases of 1:1 chat creation
because otherwise `to_id` is undefined,
but in case of outgoing group message without recipients
observed on a second device creating a group should be allowed.
Previously OpenSSL was pinned due to problems
with Zig toolchain used back then
and then not unpinned even after switching from Zig to Nix
due to <https://github.com/alexcrichton/openssl-src-rs/issues/235>.
The problem should be fixed now
and we can try to unpin OpenSSL again.
When doing an AEAP transition, we mustn't just delete the old peerstate as this would break
encryption to it. This is critical for non-verified groups -- if we can't encrypt to the old
address, we can't securely remove it from the group (to add the new one instead).
-remove `DELTA_CHAT_SKIP_PATH` environment variable
- remove version check / search for dc rpc server in $PATH
- remove `options.skipSearchInPath`
- add `options.takeVersionFromPATH`
async-broadcast returns Overflowed error once
if channel overflow happened.
Public APIs such as get_next_event JSON-RPC method
are only expecting an error if the channel is closed,
so we should not propagate overflow error outside.
In particular, Delta Chat Desktop
stop receiving events completely if an error
is returned once.
If overflow happens, we should ignore it
and try again until we get an event or an error because
the channel is closed (in case of recv())
or empty (in case of try_recv()).
If multiple headers with the same name are present,
we must take the last one.
This is the one that is DKIM-signed if
this header is DKIM-signed at all.
Ideally servers should prevent adding
more From, To and Cc headers by oversigning
them, but unfortunately it is not common
and OpenDKIM in its default configuration
does not oversign any headers.
closes#5533
adds the functions that were still missing for migration to jsonrpc (the
ones that the cffi already had, so just should be quick to review ;)
It's actually `deltachat::contact::Contact::authname` by semantics. `display_name`/`name` shouldn't
be shared, it's the name given by the user locally.
Add a function parsing a vCard file at the given path.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: Asiel Díaz Benítez <asieldbenitez@gmail.com>
When there are no parent references,
Delta Chat inserts Message-ID into References.
Such references should be ignored
because otherwise fully downloaded message
may be assigned to the same chat as previously incorrectly assigned
partially downloaded message.
Fixes receive_imf::tests::test_create_group_with_big_msg
Chat-Group-ID always correctly identifies the chat
message was sent to, while In-Reply-To and References
may point to a message that has itself been incorrectly
assigned to a chat.
This protects Bob (the joiner) of sending unexpected-unencrypted messages during an otherwise nicely
running SecureJoin.
If things get stuck, however, we do not want to block communication -- the chat is just
opportunistic as usual, but that needs to be communicated:
1. If Bob's chat with Alice is `Unprotected` and a SecureJoin is started, then add info-message
"Establishing guaranteed end-to-end encryption, please wait..." and let `Chat::can_send()` return
`false`.
2. Once the info-message "Messages are guaranteed to be e2ee from now on" is added, let
`Chat::can_send()` return `true`.
3. If after SECUREJOIN_WAIT_TIMEOUT seconds `2.` did not happen, add another info-message "Could not
yet establish guaranteed end-to-end encryption but you may already send a message" and also let
`Chat::can_send()` return `true`.
Both `2.` and `3.` require the event `ChatModified` being sent out so that UI pick up the change wrt
`Chat::can_send()` (this is the same way how groups become updated wrt `can_send()` changes).
SECUREJOIN_WAIT_TIMEOUT should be 10-20 seconds so that we are reasonably sure that the app remains
active and receiving also on mobile devices. If the app is killed during this time then we may need
to do step 3 for any pending Bob-join chats (right now, Bob can only join one chat at a time).
`ChatId::lookup_by_contact()` returns `None` for blocked chats, so it should be only used if we need
to filter out blocked chats, e.g. when building a chatlist.
E.g. the multi-device synchronisation creates the "Saved Messages" chat as blocked, in this case the
chat icon wasn't updated before and the user avatar was displayed instead.
- [x] figgure out how to build the packages (that it installs native
optional package automatically)
- [X] Make the gluecode
- [x] expose both the lowerlevel api that desktop uses (~~send objects
and receive objects~~, getting path of rpc-server is enough)
- [X] and the higher level api needed for bots (jsonrpc client)
- [X] typescript types
- [x] automatically pick the right binary from npm or allow getting it
from env var, or give out an error (throw error)
- [x] find out how to dev locally (use local built core in dc desktop) -
there is the question of how to link the typescript client and the task
to add a search in the cargo target folder for a debug build or a
different way, find out some good flow that we can use and document for
dc desktop + locally built core development
- [x] build the packages in ci
- [x] fix that deltachat-rpc-server is not executable
postponed:
- [ ] publish from ci
- [ ] add key/token to deploy to npm
Closes#4694
## Related prs
- https://github.com/deltachat-bot/echo/pull/69
- https://github.com/deltachat/deltachat-desktop/pull/3567
---------
Co-authored-by: link2xt <link2xt@testrun.org>
this PR checks if the quotes are used in a reasonable way:
- quoted messages should be send to the same chat
- or to one-to-one chats
if the quote's chat ID is not the same than the sending chat _and_ the
sending chat is not a one-to-one chat, sending is aborted.
usually, the UIs does not allow quoting in other ways, so, this check is
only a "last defence line" to avoid leaking data in case the UI has
bugs, as recently in
https://github.com/deltachat/deltachat-android/issues/3032
@iequidoo @link2xt @adbenitez i am not 100% sure about this PR, maybe
i've overseen a reasonable usecase where other quotes make sense
---------
Co-authored-by: link2xt <link2xt@testrun.org>
Otherwise, e.g. if a message is a large GIF, but its viewtype is set to `Image` by the app, this GIF
will be recoded to JPEG to reduce its size. GIFs and other special viewtypes must be always detected
and sent as is.
Otherwise location-only messages
that should be sent every 60 seconds
are never sent because location loop
waits until the end of location streaming
and is only interrupted by location streaming
ending in other chats or being enabled in other chats.
This is similar to `test_modify_chat_disordered`,
but tests that recovery works in the simplest case
where next message is not modifying group membership.
... and fails if file already exists. The UI should open the file saving dialog, defaulting to
Downloads and original filename, when asked to save the file. After confirmation it should call
dc_msg_save_file().
This way we avoid an immediate retry if the network is not yet ready exhausting the ratelimiter's
quota of two connection attempts. Also notify the ratelimiter only after a successful connection so
that it only limits the server load, but not connection attempts.
Otherwise an error
"The encoder or decoder for Jpeg does not support the color type `Rgba8`"
is returned if image has an alpha channel.
This is caused by the recent change of JPEG encoder
in `image` crate: <https://github.com/image-rs/image/issues/2211>
This makes `EventTracker` receive events immediately
instead of being moved from event emitter to event tracker
by a task spawned from `TestContext::new_internal`.
This makes `EventTracker.clear_events` reliable
as it is guaranteed to remove all events emitted
by the time it is called rather than only events
that have been moved already.
as discussed in several chats, this PR starts making it possible to use
Webxdc as integrations to the main app. In other word: selected parts of
the main app can be integrated as Webxdc, eg. Maps [^1]
this PR contains two parts:
- draft an Webxdc Integration API
- use the Webxdc Integration API to create a Maps Integration
to be clear: a Webxdc is not part of this PR. the PR is about marking a
Webxdc being used as a Map - and core then feeds the Webxdc with
location data. from the view of the Webxdc, the normal
`sendUpdate()`/`setUpdateListener()` is used.
things are still marked as "experimental", idea is to get that in to
allow @adbenitez and @nicodh to move forward on the integrations into
android and desktop, as well as improving the maps.xdc itself.
good news is that we currently can change the protocol between Webxdc
and core at any point :)
# Webxdc Integration API
see `dc_init_webxdc_integration()` in `deltachat.h` for overview and
documentation.
rust code is mostly in `webxdc/integration.rs` that is called by other
places as needed. current [user of the API is
deltachat-ios](https://github.com/deltachat/deltachat-ios/pull/1912),
android/desktop will probably follow.
the jsonrpc part is missing and can come in another PR when things are
settled and desktop is really starting [^2] (so we won't need to do all
iterations twice :) makes also sense, when this is done by someone
actually trying that out on desktop
while the API is prepared to allow other types of integrations (photo
editor, compose tools ...) internally, we currently ignore the type. if
that gets more crazy, we probably also need a dedicated table for the
integrations and not just a single param.
# Maps Integration
rust code is mostly in `webxdc/maps_integration.rs` that is called by
`webxdc/integration.rs` as needed.
EDIT: the idea of having a split here, is that
`webxdc/maps_integration.rs` really can focus on the json part, on the
communication with the .xdc, including tests
this PR is basic implementation, enabling to move forward on
integrations on iOS, but also on desktop and android.
the current implementation allows already the following:
- global and per-chat maps
- add and display POIs
- show positions and tracks of the last 24 hours
the current maps.xdc uses leaflet, and is in some regards better than
the current android/desktop implementations (much faster, show age of
positions, fade out positions, always show names of POIs, clearer UI).
however, we are also not bound to leaflet, it can be anything
> [**screenshots of the current
state**](https://github.com/deltachat/deltachat-ios/pull/1912)
> 👆
to move forward faster and to keep this PR small, the following will go
to a subsequent PR:
- consider allowing webxdc to use a different timewindow for the
location
- delete POIs
- jsonrpc
[^1]: maps are a good example as anyways barely native (see android
app), did cause a lot of pain on many levels in the past (technically,
bureaucratically), and have a comparable simple api
[^2]: only going for jsonrpc would only make sense if large parts of
android/ios would use jsonrpc, we're not there
---------
Co-authored-by: link2xt <link2xt@testrun.org>
As trashed messages can't be loaded from the db now by `Message::load_from_db()` returning an error
for such messages, errors from `Message::load_from_db()` should be logged as warnings. If it's
really an error like a db failure, it should be logged internally.
As trashed messages can't be loaded from the db now, `get_message_by_id()` returns None in some
tests e.g. in `test_deleted_msgs_dont_reappear()`. A `PerAccount` hook shouldn't be called if so.
There's no need to load an updated message state from the db to implement `is_outgoing()` and also
this function is implicitly called in some tests where a message is already trashed and a call to
`dc_get_msg()` generates an unexpected error.
Previously test did not trigger
deletion of ephemeral messages
and worked because clear_events() did not
remove just emitted events from `send_text_msg`.
It keeps timing out with the default timeout of 2 s on macOS runners.
Also fix comment in the integration test which
said that timeout is 1 minute but sets it to 3 minutes.
Set this timeout to 5 minutes as well.
API now pretends that trashed messages don't exist.
This way callers don't have to check if loaded message
belongs to trash chat.
If message may be trashed by the time it is attempted to be loaded,
callers should use Message::load_from_db_optional.
Most changes are around receive_status_update() function
because previously it relied on loading trashed status update
messages immediately after adding them to the database.
This way nightly clippy warnings are not generated
when devshell is used.
Nighly Rust is also not cached, e.g. rust-analyzer has to be rebuilt
if version from fenix is used.
This fixes things for Gmail f.e. Before, `Imap::fetch_move_delete()` was called before looking for
Trash and returned an error because of that failing the whole `fetch_idle()` which prevented
configuring Trash in turn.
`a.clone_from(&b)` is equivalent to `a = b.clone()` in functionality,
but can be overridden to reuse the resources of a to avoid unnecessary
allocations.
When the env var CARGO_BUILD_TARGET is set, cargo will crossbuild for the given arch triplet. In this case, the targets will not be put into target/release/, but target/$CARGO_BUILD_TARGET/release/. Add this subdirectory, if neccessary.
If `delete_device_after` is configured, that period should be counted for webxdc instances from the
last status update, otherwise nothing prevents from deleting them. Use `msgs.timestamp_rcvd` to
store the last status update timestamp, it anyway isn't used for anything except displaying a
detailed message info. Also, as `ephemeral::select_expired_messages()` now also checks
`timestamp_rcvd`, we have an improvement that a message is guaranteed not to be deleted for the
`delete_device_after` period since its receipt. Before only the sort timestamp was checked which is
derived from the "sent" timestamp.
Let's add a 1-minute tolerance to `Params::MemberListTimestamp`.
This adds to the group membership consistency algo the following properties:
- If remote group membership changes were made by two members in parallel, both of them are applied,
no matter in which order the messages are received.
- If we remove a member locally, only explicit remote member additions/removals made in parallel are
allowed, but not the synchronisation of the member list from "To". Before, if somebody managed to
reply earlier than receiving our removal of a member, we added it back which doesn't look good.
This change avoids the race between
`provide_backup` changing the state from NoProvider to Pending
and a call to `get_backup_qr` or `get_backup_qr_svg`.
With this change `get_backup_qr` and `get_backup_qr_svg`
always block until QR code is available,
even if `provide_backup` was not called yet.
- Always emit `ContactsChanged` from `contact::update_last_seen()` if a contact was seen recently
just for simplicity and symmetry with `RecentlySeenLoop::run()` which also may emit several events
for single contact.
- Fix sleep time calculation in `RecentlySeenLoop::run()` -- `now` must be updated on every
iteration, before the initial value was used every time which led to progressively long sleeps.
Instead, look up the 1:1 chat in `receive_imf::add_parts()`. This is a more generic approach to fix
assigning outgoing reactions to 1:1 chats in the multi-device setup. Although currently both
approaches give the same result, this way we can even implement a "react privately"
functionality. Maybe it sounds useless, but it seems better to have less reaction-specific code.
we checked for tombstones already using `is_trash()`,
however, we've overseen that tombstones get deleted at some point :)
therefore, just do not treat loading failures of the weak msg_id as errors -
usually, they are not - and if, just the normal summary is shown.
in theory, we could check for existance explicitly before tryong load_from_db,
however, that would be additional code (and maybe another database call)
and not worth the effort.
anyways, this commit also adds an explicit test
for physical deletion after housekeeping.
use get_summary_text_without_prefix() to get raw summaries without prefixes,
this is needed for reaction summaries,
where we also do not show the name, so we do not want to show "Forwarded" as well.
shows the last reaction in chatlist's summaries if there is no
newer message.
the reason to show reactions in the summary, is to make them a _little_
more visible when one is not in the chat. esp. in not-so-chatty or in
one-to-ones chats this becomes handy: imaging a question and someone
"answers" with "thumbs up" ...
otoh, reactions are still tuned down on purpose: no notifications, chats
are opend as usual, the chatlist is not sorted by reactions and also the
date in the summary refer to the last message - i thought quite a bit
about that, this seems to be good compromise and will raise the fewest
questions. it is somehow clear to the users that reactions are not the
same as a real message. also, it is comparable easy to implement - no
UI changes required :)
all that is very close to what whatsapp is doing (figured that out by
quite some testing ... to cite @adbenitez: if in doubt, we can blame
whatsapp :)
technically, i first wanted to go for the "big solution" and add two
more columns, chat_id and timestamp, however, it seemed a bit bloated if
we really only need the last one. therefore, i just added the last
reaction information to the chat's param, which seems more performant
but also easier to code :)
`is_probably_private_reply()` checks
if a message should better go to the one to one chat
instead of the chat already identified by the `In-Reply-To`.
this functionality is needed to make "Reply Privately" work.
however, this functionality is never true for reactions.
if we would return `true` here, own reactions seen by a second device
would not get the correct chat assiged.
We update version in several .toml and .json files
on every release anyway, so updating pyproject.toml is easy.
setuptools_scm makes it more difficult to build
python packages for software distributions
because it requires full git checkout
with tags rather than just a worktree.
It is also possible to remove or move tags
after the release, so git revision no longer
pins python package version if setuptools_scm is used.
- the `jobs` table is no longer in use,
no need to track files on housekeeping,
no need to clear it from repl tool
- some `Params` were used for jobs table only,
they can be used freely for other purposes on other tables.
param 'protection settings timestamp' was never used in practise,
its code is removed as well, so we can free the Param as well.
it was used by iOS to know when a background fetch was complete;
meanwhile the superiour `dc_accounts_background_fetch()` is used for
that.
there is still the corresponding context function `dc_all_work_done()`,
this not used by any UI as well, however, it is in use by a python
tests.
not sure, what to do with it, at a first glance, the test still seems
useful.
Don't attach selfavatar in "v{c,g}-request" and "v{c,g}-auth-required" messages:
- These messages are deleted right after processing, so other devices won't see the avatar.
- It's also good for privacy because the contact isn't yet verified and these messages are auto-sent
unlike usual unencrypted messages.
the rendered time output seems different on different systems and timezomes,
eg. on my local machine, it is
`Sent: 2024.03.20 10:00:01 ` and not `Sent: 2024.03.20 09:00:01`,
maybe because of winter/summer time, idk.
as the gist of the test is to check the name,
however, i just removed the whole time check.
Do not include oldest reference, because chat members
which have been added later and have not seen the first message
do not have referenced message in the database.
Instead, include up to 3 recent Message-IDs.
If user specifies build directory, respect this
instead of building inside the target/ directory.
Also remove outdated comment about Rust 1.50.0 support.
Current MSRV is 1.70.0 and we use 2021 edition
which implies version 2 resolver.
This is a test reproducing the problem
in <https://github.com/deltachat/deltachat-core-rust/issues/5339>.
Fix would be to avoid reordering on the server side,
so the test checks that the unverified message
is replaced with a square bracket error
as expected if messages arrive in the wrong order.
Add white background instead of the default black one to avatars when recoding to JPEG. But not for
"usual" images to spare the CPU. The motivation is to handle correctly
"black-on-transparent-background" avatars which are quite common.
as turned out on recent researches wrt a slow database, setting
`save_mime_headers` will result in 25x larger databases.
this is definetely something we want to know at least in the debug info.
(the option cannot be enabled in current UIs,
however, esp. devs find options to set this.
apart from that, save_mime_headers is only used in some python tests)
ftr: the huge database was more than 10 times slower, leading to missing
notifications on ios (as things do not finish within the few seconds iOS
gave us).
moreover, the hugeness also avoids exporting; this is fixed by
https://github.com/deltachat/deltachat-core-rust/pull/5349
this PR fixes one of the issues we had with an (honestly accidentally)
huge database of >2gb.
this database could not be exported on iOS, as ram memory is limited
there, leading to the app just crashing.
it is unclear if that would be better on eg. Android, however,
temp_store=FILE is not working as well there, this is the smaller
drawback there.
before merging, this PR should be tested on time with export/import on
iOS (export/import uses VACUUM which uses /tmp) EDIT: did so, works on
iphone7 with ios15
`Param::MemberListTimestamp` was updated only from `receive_imf::apply_group_changes()` i.e. for
received messages. If we sent a message, that timestamp wasn't updated, so remote group membership
changes always overrode local ones. Especially that was a problem when a message is sent offline so
that it doesn't incorporate recent group membership changes.
When scheduler is destroyed, e.g. during a key import,
there is some time between destroying the interrupt channel
and the loop task.
To avoid busy looping, tasks should terminate if
receiving from the interrupt loop fails
instead of treating it as the interrupt.
An unencrypted message with already known Autocrypt key, but sent from another address, means that
it's rather a new contact sharing the same key than the existing one changed its address, otherwise
it would already have our key to encrypt.
If Subject is multiline-formatted, `mailparse` adds the leading whitespace to it. The solution is to
always remove the leading whitespace, because if Subject isn't multiline-formatted, it never
contains the leading whitespace anyway. But as for the trailing whitespace -- i checked -- it's
never removed, so let's keep this as is.
This fixes the following identity-misbinding attack:
It appears that Bob’s messages in the SecureJoin protocol do not properly “bind” to Alice’s public
key or fingerprint. Even though Bob’s messages carry Alice’s public key and address as a gossip in
the protected payload, Alice does not reject the message if the gossiped key is different from her
own key. As a result, Mallory could perform an identity-misbinding attack. If Mallory obtained
Alice’s QR invite code, she could change her own QR code to contain the same tokens as in Alice’s QR
code, and convince Bob to scan the modified QR code, possibly as an insider attacker. Mallory would
forward messages from Bob to Alice and craft appropriate responses for Bob on his own. In the end,
Bob would believe he is talking to Mallory, but Alice would believe she is talking to Bob.
Secure-Join-Group is only expected by old core in vg-request-with-auth.
There is no reason to leak group ID in unencrypted vg-request.
Besides that, Secure-Join-Group is deprecated
as Alice knows Group ID corresponding to the auth code,
so the header can be removed completely eventually.
If a message is sent from SELF, but signed with a foreign key, it mustn't be considered
Autocrypt-encrypted and shown with a padlock. Currently this is broken.
Compilation of vendored OpenSSL inside Nix is broken since
<https://github.com/alexcrichton/openssl-src-rs/pull/229> due to build
script changes.
There is anyway no need to compile vendored OpenSSL as nixpkgs already
contains OpenSSL package.
This fixes `nix build .#deltachat-rpc-server-x86_64-linux` and similar
commands which are used during releases.
The way it was implemented it threw out all remaining messages after finding the next incoming
message. Better use FFIEventTracker functions, they are used in all the tests anyway.
Connection establishment now happens only in one place in each IMAP loop.
Now all connection establishment happens in one place
and is limited by the ratelimit.
Backoff was removed from fake_idle
as it does not establish connections anymore.
If connection fails, fake_idle will return an error.
We then drop the connection and get back to the beginning of IMAP
loop.
Backoff may be still nice to have to delay retries
in case of constant connection failures
so we don't immediately hit ratelimit if the network is unusable
and returns immediate error on each connection attempt
(e.g. ICMP network unreachable error),
but adding backoff for connection failures is out of scope for this change.
ratelimit can be exhausted quickly if the network is not available,
i.e. if every connection attempt returns "network unreachable" error.
When the network becomes available, we want to retry connecting
as soon as maybe_network is called without waiting for ratelimiter.
Before group avatar was sent as an attachment. Let's do the same as with user avatar and send group
avatar as base64. Receiver code uses the same functions for user and chat avatars, so base64 avatars
are supported for most receivers already.
`deltachat-rpc-server` releases are built
with Nix and LLVM/clang toolchains now
that fully support atomics.
Zig toolchain that required disabling atomics
and resulted in problems with OpenSSL 3.2 releases
is not used anymore.
It was broken completely and before "fix: apply Autocrypt headers if timestamp is unchanged" that
didn't show up because the message from the second Bob's device never had "Date" greater than one
from the message sent before from the first device.
In this case connection failure
may be a connection timeout (currently 1 minute),
so it does not make sense to fake idle for another minute immediately after.
However, failure may be immediate if the port is closed
and the server refuses connection every time.
To prevent busy loop in this case
we apply ratelimit to connection attempts rather than login attempts.
This partially reverts ccec26ffa7
Even if 1:1 chat with alice is protected,
we should send vc-request unencrypted.
This happens if Alice changed the key
and QR-code Bob scans contains fingerprint
that is different from the verified fingerprint.
Sending vc-request encrypted to the old key
does not help because Alice is not able
to decrypt it in this case.
Use sync messages for that as it is done for e.g. Config::Displayname. Maybe we need to remove
self-status synchronisation via usual messages then, but let's think of it a bit.
Use sync messages for that as it is done for e.g. Config::Displayname. Maybe we need to remove
avatar synchronisation via usual messages then, but let's think of it a bit.
Encryption subkey is incorrectly referred to as public key
in variable names.
This is incorrect because generated encryption key
is secret too just as the signing primary key.
Generated OpenPGP secret key consists of primary signing key
and encryption subkey.
Then OpenPGP public key consisting of
the primary signing public key
and encryption public key is generated.
Keypair consists of the secret OpenPGP key and public OpenPGP key,
each of them has a primary key and subkey inside.
Also disable --progress.
It is not disabled by default for backward compatibility,
but solves the problem of lots of progress lines
in the downloadable raw output:
https://github.com/actions/checkout/pull/1067
The limit is better enforced by webxdc distributors,
e.g. xdc store bots or actually email providers
to allow for experimentation with large frameworks
or porting existing apps and testing them
before reducing their size.
Besides that, the comment on WEBXDC_SENDING_LIMIT was outdated,
it was not updated when the limit was increased to 640 kB.
Even if `vc-request-with-auth` is received with a delay, the protection message must have the sort
timestamp equal to the Sent timestamp of `vc-request-with-auth`, otherwise subsequent chat messages
would also have greater sort timestamps and while it doesn't affect the chat itself (because Sent
timestamps are shown to a user), it affects the chat position in the chatlist because chats there
are sorted by sort timestamps of the last messages, so the user sees chats sorted out of
order. That's what happened in #5088 where a user restores the backup made before setting up a
verified chat with their contact and fetches new messages, including `vc-request-with-auth` and also
messages from other chats, after that.
Add a new crate `deltachat_time` with a fake `struct SystemTimeTools` for mocking
`SystemTime::now()` for test purposes. One still needs to use `std::time::SystemTime` as a struct
representing a system time. I think such a minimalistic approach is ok -- even if somebody uses the
original `SystemTime::now()` instead of the mock by mistake, that could break only tests but not the
program itself. The worst thing that can happen is that tests using `SystemTime::shift()` and
checking messages timestamps f.e. wouldn't catch the corresponding bugs, but now we don't have such
tests at all which is much worse.
after merging:
6. Tag the release: `git tag -a v1.135.0`.
7. Push the release tag: `git push origin v1.135.0`.
8. Create a GitHub release: `gh release create v1.135.0 -n ''`.
---------
Co-authored-by: link2xt <link2xt@testrun.org>
PR #5099 removed some columns in the database that were actually in use.
usually, to not worsen UX unnecessarily
(releases take time - in between, "Add Second Device", "Backup" etc.
would fail), we try to avoid such schema changes (checking for
db-version would avoid import etc. but would still worse UX),
see discussion at #2294.
these are the errors, the user will be confronted with otherwise:
<img width=400
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/e3f0fd6e-a7a9-43f6-9023-0ae003985425>
it is not great to maintain the old columns, but well :)
as no official releases with newer cores are rolled out yet, i think, it
is fine to change the "107" migration
and not copy things a second time in a newer migration.
(this issue happens to me during testing, and is probably also the issue
reported by @lk108 for ubuntu-touch)
The system clock may be adjusted and even go back, so caching system time in code sections where
it's not supposed to change may even protect from races/bugs.
If a time value doesn't need to be sent to another host, saved to the db or otherwise used across
program restarts, a monotonically nondecreasing clock (`Instant`) should be used. But as `Instant`
may use `libc::clock_gettime(CLOCK_MONOTONIC)`, e.g. on Android, and does not advance while being in
deep sleep mode, get rid of `Instant` in favor of using `SystemTime`, but add `tools::Time` as an
alias for it with the appropriate comment so that it's clear why `Instant` isn't used in those
places and to protect from unwanted usages of `Instant` in the future. Also this can help to switch
to another clock impl if we find any.
Restart the IO scheduler if needed to make the new config value effective (for `MvboxMove,
OnlyFetchMvbox, SentboxWatch` currently). Also add `set_config_internal()` which doesn't affect
running the IO scheduler. The reason is that `Scheduler::start()` itself calls `set_config()`,
although not for the mentioned keys, but still, and also Rust complains about recursive async calls.
Currently when a user sets up another device by logging in, a new key is created. If a message is
sent from either device outside, it cannot be decrypted by the other device.
The message is replaced with square bracket error like this:
```
<string name="systemmsg_cannot_decrypt">This message cannot be decrypted.\n\n• It might already help to simply reply to this message and ask the sender to send the message again.\n\n• If you just re-installed Delta Chat then it is best if you re-setup Delta Chat now and choose "Add as second device" or import a backup.</string>
```
(taken from Android repo `res/values/strings.xml`)
If the message is outgoing, it does not help to "simply reply to this message". Instead, we should
add a translatable device message of a special type so UI can link to the FAQ entry about second
device. But let's limit such notifications to 1 per day. And as for the undecryptable message
itself, let it go to Trash if it can't be assigned to a chat by its references.
Before it was emitted only on the source device and `test_sync()` didn't catch the bug because of
the interference of the previous call to `set_config_bool()`. Now there's a separate test that does
its job well.
Before moving emails to the mvbox we need to remember its UIDVALIDITY, otherwise emails moved before
that wouldn't be fetched but considered "old" instead.
Also:
- Don't use `session.create()` to create mvbox as `select_with_uidvalidity()` already creates mvbox
on its own.
- Don't try to create compat folders like "INBOX.DeltaChat", but only look for them.
If `escaper::decode_html_buf_sloppy()` just truncates the text (which happens when it fails to
html-decode it at some position), then it's probably not HTML at all and should be left as
is. That's what happens with hyperlinks f.e. and there was even a test on this wrong behaviour which
is fixed now. So, now hyperlinks are not truncated in messages and should work as expected.
If a Delta Chat message has the Message-ID already existing in the db, but a greater "Date", it's a
resent message that can be deleted. Messages having the same "Date" mustn't be deleted because they
can be already seen messages moved back to INBOX. Also don't delete messages having lesser "Date" to
avoid deleting both messages in a multi-device setting.
Use `create_smeared_timestamp()` for this. This allows to dedup messages on the receiver -- if it
sees the same Message-ID, but a different timestamp, then it's a resent message that can be deleted.
"Auto-Submitted: auto-replied" messages mustn't be considered as sent by either bots or non-bots,
e.g. MDNs have this header value and it's the same for bots and non-bots.
In particular TLSRPT reports
contain files that may be interesting for admins.
Currently Delta Chat drops the attachment
so message appears as a text message without actual payload.
@adbenitez wants this feature on Deltalab to display a bot tag.
Other UIs might also want to adopt this feature :)
---------
Co-authored-by: link2xt <link2xt@testrun.org>
If we wait for inviter success,
vg-member-added message may be still in flight
and reach ac2 after device resetup.
Making ac2 wait for joining the group ensures that old
device receives vg-member-added message
and new device will not receive it and fail to decrypt.
Other instances of wait_for_securejoin_inviter_success()
in the same tests are also replaced for reliability.
Previously the message was removed from `download` table,
but message bubble was stuck in InProgress state.
Now download state is updated by the caller,
so it cannot be accidentally skipped.
This allows to send existing messages (incoming and outgoing) taken from encrypted chats, to
unencrypted ones. `Param::ForcePlaintext` is removed as well -- if a message can be sent encrypted
this time, nothing bad with this.
Currently `Chat.send_msg()` modifies the source message and returns another message object
equivalent to the source one. That's how it works in the core and thus in Python bindings too.
This is needed to protect from ESPs (such as gmx.at) doing their own Quoted-Printable encoding and
thus breaking messages and signatures. It's unlikely that the reader uses a MUA not supporting
Quoted-Printable encoding. And RFC 2646 "4.6" also recommends it for encrypted messages.
Trying non-strict TLS checks is not necessary
for most servers with proper TLS setup,
but doubles the time needed to fail configuration
when the server is not responding, e.g.
when all connection attempts time out.
There is also a risk of accidentally
configuring non-strict TLS checks in a rare case
that strict TLS check configuration spuriously failed,
e.g. on a bad network.
If the server has a known broken TLS setup,
it can still be added to the provider database
or configured with non-strict TLS check manually.
User can also configure another email provider,
such as chatmail servers, instead of using the server
with invalid TLS hostname.
This change does not affect exising setups.
deltachat.h uses `@defgroup` commands to create topics
for groups of constants. Prior to Doxygen 1.9.8
defining a group created a "module"
and all constants were visible from the modules.html page.
In Doxygen 1.9.8 "modules" were renamed into "topics"
as C++20 modules have taken their place,
so Delta Chat documentation does not have modules
in Doxygen sense anymore.
The change is to replace "modules.html" with "topics.html"
in the DoxygenLayout.xml.
See <https://www.doxygen.nl/manual/grouping.html> for
Doxygen documentation about groups and their relation to topics.
This PR stops MDNs from being forced to be sent unencrypted.
If no encryption is possible (by `should_encrypt`), the fix#5152 still
applies.
close#5168
Send `EventType::ConnectivityChanged` when using the context methods
`start_io` and `stop_io`.
close#5097
---------
Co-authored-by: Septias <scoreplayer2000@gmail.comclear>
Add an event for a case if a multi-device synced config value changed. Maybe the app needs to
refresh smth on such an event. For uniformity it is emitted on the source device too. The value is
omitted, otherwise it would be logged which might not be good for privacy.
Mark 1:1 chat as verified as soon as Alice is forward-verified
so Bob can already start sending Chat-Verified headers.
This way Alice and Bob can scan each other's QR codes
and even if all Secure-Join headers are dropped from the network,
still get forward verifications via QR-code scans
and backward verifications via Chat-Verified messages in 1:1 chat.
Before, while a message is in OutPending state after resending is requested, the user still sees the
red marker with error and it is confusing, so the user don't know the sending state of the message.
Before this fix actual contents of the message
reposted by Schleuder is considered a mailing list footer and removed,
not visible even in the "Show Full Message..." view.
With this change there will be two message bubbles,
one for header and one for the contents,
but it is still better than losing the contents completely.
Attempting to parse header part is out of scope for this change.
This is broken since 44227d7b86
mimeparser only recognizes read receipts
by the Content-Type being "multipart/report".
If multipart/report is hidden inside multipart/mixed
and the message is not encrypted,
it degrades encryption.
a27e84ad89 "fix: Delete received outgoing messages from SMTP queue"
can break sending messages sent as several SMTP messages because they have a lot of recipients:
`pub(crate) const DEFAULT_MAX_SMTP_RCPT_TO: usize = 50;`
We should not cancel sending if it is such a message and we received BCC-self because it does not
mean the other part was sent successfully. For this, split such messages into separate jobs in the
`smtp` table so that only a job containing BCC-self is canceled from `receive_imf_inner()`. Although
this doesn't solve the initial problem with timed-out SMTP requests for such messages completely,
this enables fine-grained SMTP retries so we don't need to resend all SMTP messages if only some of
them failed to be sent.
The bug was made in 44227d7b86. Sql::execute() with placeholders must
be used to escape strings, one never should escape them manually as strings themselves can contain
escape symbols. Thanks to @link2xt for noticing.
Some SMTP servers are running slow before-queue filters, most commonly Postfix with `rspamd` filter
which is implemented as a [before-queue Milter](https://www.postfix.org/MILTER_README.html). Some of
`rspamd` plugin filters are slow on large mails.
We previously had problems with timing out during waiting for SMTP response:
https://github.com/deltachat/deltachat-core-rust/issues/1383. This is largely fixed by
https://github.com/async-email/async-smtp/pull/29 and currently we have 60-second timeout just for
reading a response but apparently it is not sufficient -- maybe connection gets killed by NAT while
we are waiting for response or `rspamd` takes more than 60 seconds for large messages.
As a result a message is resent multiple times and eventually fails with "too many retries" while
multiple BCC-self messages are received.
We should remove the message from the SMTP queue as soon as we receive it via IMAP as it is clear
the message was sent even if we did not manage to get actual SMTP server response.
First of all, it's just downloaded and hasn't been seen yet by the user. Also this changes nothing
as `msgs.state` isn't changed when replacing a message anyway.
Put a copy of Message-ID into hidden headers and prefer it over the one in the IMF header section
that servers mess up with.
This also reverts "Set X-Microsoft-Original-Message-ID on outgoing emails for amazonaws (#3077)".
As ratelimit was introduced to avoid reconnecting immediately after disconnecting
in case of bugs in IMAP protocol handling,
connection attempts should only be counted when IMAP is actually used,
i.e. when the first command (LOGIN) is sent.
As per the comment in `receive_imf.rs`, `chat.protected` must be maintained regardless of the
`Config::VerifiedOneOnOneChats`. The only thing that mustn't be done if `VerifiedOneOnOneChats` is
unset (i.e. for non-supporting UIs) is marking chats as "protection broken" because this needs
showing the corresponding dialog to a user.
Drafts mustn't affect sorting of any other messages, they aren't even displayed in the chat
window. Also hidden messages mustn't affect sorting of usual messages. But let hidden messages sort
together with protection messages because hidden messages also can be or not be verified, so let's
preserve this information -- even it's not useful currently, it can be useful in the future
versions.
Before in some places it was correctly calculated by passing the "sent" timestamp to
`calc_sort_timestamp()`, but in other places just the system time was used. In some complex
scenarios like #5088 (restoration of a backup made before a contact verification) it led to wrong
sort timestamps of protection messages and also messages following by them.
But to reduce number of args passed to functions needing to calculate the sort timestamp, add
message timestamps to `struct MimeMessage` which is anyway passed everywhere.
We do not want all_work_done() to return true immediately
after calling start_io(), but only when connection goes idle.
"Connected" state is set immediately after connecting to the server,
but it does not mean there is nothing to do.
This change make all_work_done() return false
from the Connected state and introduces a new Idle
connectivity state that is only set before connection
actually goes idle. For idle state all_work_done() returns true.
From the user point of view both old Connected state
and new Idle state look the same.
This change depends on async-imap update that resets the timeout
every time an `* OK Still here` is received.
Reducing timeout allows to detect lost connections
not later than 6 minutes
because Delta Chat will attempt to finish IDLE with DONE
after 5 minutes without keepalives
and will either get TCP RST directly
or, worst case, wait another minute for TCP socket read timeout.
Ad-hoc groups don't have grpid-s that can be used to identify them across devices and thus wasn't
synced until now.
The same problem already exists for assigning messages to ad-hoc groups and this assignment is done
by `get_parent_message()` and `lookup_chat_by_reply()`. Let's reuse this logic for the
synchronisation, it works well enough and this way we have less surprises than if we try to
implement grpids for ad-hoc groups. I.e. add an `Msgids` variant to `chat::SyncId` analogous to the
"References" header in messages and put two following Message-IDs to a sync message:
- The latest message A having `DownloadState::Done` and the state to be one of `InFresh, InNoticed,
InSeen, OutDelivered, OutMdnRcvd`.
- The message that A references in `In-Reply-To`.
This way the logic is almost the same to what we have in `Chat::prepare_msg_raw()` (the difference
is that we don't use the oldest Message-ID) and it's easier to reuse the existing code.
NOTE: If a chat has only an OutPending message f.e., the synchronisation wouldn't work, but trying
to work in such a corner case has no significant value and isn't worth complicating the code.
- Use TestContextManager
- Actually run receive_imf rather than only mimeparser on "received" messages
- Check that received message parts actually have a padlock
parse_mime_recursive() skips empty text parts,
so there may be no parts as the result of parsing.
In this case an empty part is added.
However, because it is added with parts.push()
rather than add_single_part(),
it is added without a padlock even if the message is encrypted.
`do_add_single_part()` adds padlock (GuaranteeE2EE param)
and should be used to add parts instead.
This test reproduces a bug preventing joining the group with a QR code
if the group already has a contact with inconsistent key state,
which means both Autocrypt and verified key exist,
but don't match.
This can happen when an invite QR code created by this contact
is scanned as scanning an invite code creates unprotected group
with the inviter for info messages.
If securejoin process never finishes because the inviter is offline,
group remains in this unprotected state with added inviter.
Normally the group becomes verified when a "Member added" (vg-member-added)
message is received in the chat.
However, current code checks that all members
of the chat are verified
(have a green checkmark, use verified key in 1:1 chat)
before marking the group as verified and fails otherwise.
- Remove "Detected Autocrypt-mime message" logs printed for every incoming Autocrypt message.
- Print only a single line at the beginning of receive_imf with both the Message-ID and seen flag.
- Print Securejoin step only once, inside handle_securejoin_handshake or observe_securejoin_on_other_device.
- Do not log "Not creating ad-hoc group" every time ad-hoc group is not created, log when it is created instead.
- Log ID of the chat where Autocrypt-Gossip for all members is received.
- Do not print "Secure-join requested." for {vg,vc}-request, we already log the step.
- Remove ">>>>>>>>>>>>>>>>>>>>>>>>>" noise from securejoin logs.
Otherwise it looks like the message creating a protected group is not verified. For this, use
`sent_timestamp` of the received message as an upper limit of the sort timestamp (`msgs.timestamp`)
of the protection message. As the protection message is added to the chat earlier, this way its
timestamp is always less or eq than the received message's timestamp.
Test that on the second device of a protected group creator the first message is
`SystemMessage::ChatProtectionEnabled` and the second one is the message populating the group.
Allowing outgoing unencrypted messages in groups with 2 members
breaks the test
`python/tests/test_0_complex_or_slow.py::test_verified_group_vs_delete_server_after`
This centralizes all Securejoin/verification checks and updates in one
place right before add_parts() even before we assign the message to
the chat, so we can decouple chat logic from verification logic.
It's not necessary and in other places like add_contact_to_chat_ex() sync messages are also sent
only if there are no system messages sent like MemberAddedToGroup.
We already synchronise status/footer when we see a self-sent message with a Chat-Version
header. Would be nice to do the same for display name.
But let's do it the same way as for `Config::{MdnsEnabled,ShowEmails}`. Otherwise, if we sync the
display name using the "From" header, smth like `Param::StatusTimestamp` is needed then to reject
outdated display names. Also this timestamp needs to be updated when `Config::Displayname` is set
locally. Also this wouldn't work if system time isn't synchronised on devices. Also using multiple
approaches to sync different config values would lead to more code and bugs while having almost no
value -- using "From" only saves some bytes and allows to sync some things w/o the synchronisation
itself to be enabled. But the latter also can be a downside -- if it's usual synchonisation, you can
(potentially) disable it and share the same email account across people in some organisation
allowing them to have different display names. With using "From" for synchronisation such a
capability definitely requires a new config option.
Motivation: Syncing these options will improve UX in very most cases and should be done. Other
candidates are less clear or are advanced options, we can reconsider that at some point later.
Approach:
- Sync options one-by-one when the corresponding option is set (even if to the same value).
- Don't sync when an option is reset to a default as defaults may differ across client versions.
- Check on both sides that the option should be synced so that if there are different client
versions, the synchronisation of the option is either done or not done in both directions.
Moreover, receivers of a config value need to check if a key can be synced because some settings
(e.g. Avatar path) could otherwise lead to exfiltration of files from a receiver's device if we
assume an attacker to have control of a device in a multi-device setting or if multiple users are
sharing an account.
- Don't sync `SyncMsgs` itself.
If a message is encrypted, but unsigned:
- Don't set `MimeMessage::from_is_signed`.
- Remove "secure-join-fingerprint" and "chat-verified" headers from `MimeMessage`.
- Minor: Preserve "Subject" from the unencrypted top level if there's no "Subject" in the encrypted
part, this message is displayed w/o a padlock anyway.
Apparently it didn't lead to any vulnerabilities because there are checks for
`MimeMessage::signatures.is_empty()` in all necessary places, but still the code looked dangerous,
especially because `from_is_singed` var name didn't correspond to its actual value (it was rather
`from_is_encrypted_maybe_signed`).
Currently this leads to
DEBUG root:rpc.py:136 account_id=1 got an event {'kind': 'Warning', 'msg': 'src/scheduler.rs:711: send_smtp_messages failed: failed to send message: failed to update retries count: database is locked: Error code 5: The database file is locked'}
and
FAILED tests/test_webxdc.py::test_webxdc_insert_lots_of_updates - deltachat_rpc_client.rpc.JsonRpcError: {'code': -1, 'message': 'database is locked\n\nCaused by:\n Error code 5: The database file is locked'}
Without this change
when the test returns a `Result`, `cargo test` does not show
the line number.
To make the tests as easy to debug as the panicking tests,
enable `backtrace` feature on `anyhow` and add debug information
to add source line numbers to backtraces.
Now running `RUST_BACKTRACE=1 cargo test` produces backtraces
with the line numbers. For example:
Error: near ",": syntax error in SELECT COUNT(,,*) FROM msgs_status_updates; at offset 13
Caused by:
Error code 1: SQL error or missing database
Stack backtrace:
0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/result.rs:1962:27
1: deltachat::sql::Sql::query_row::{{closure}}::{{closure}}
at ./src/sql.rs:466:23
2: deltachat::sql::Sql::call::{{closure}}::{{closure}}
at ./src/sql.rs:379:55
3: tokio::runtime::context::runtime_mt::exit_runtime
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/context/runtime_mt.rs:35:5
4: tokio::runtime::scheduler::multi_thread::worker::block_in_place
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/scheduler/multi_thread/worker.rs:438:9
5: tokio::runtime::scheduler::block_in_place::block_in_place
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/scheduler/block_in_place.rs:20:5
6: tokio::task::blocking::block_in_place
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/task/blocking.rs:78:9
7: deltachat::sql::Sql::call::{{closure}}
at ./src/sql.rs:379:19
8: deltachat::sql::Sql::query_row::{{closure}}
at ./src/sql.rs:469:10
9: deltachat::sql::Sql::count::{{closure}}
at ./src/sql.rs:443:76
10: deltachat::webxdc::tests::change_logging_webxdc::{{closure}}
at ./src/webxdc.rs:2644:18
11: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/future/future.rs:125:9
12: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/park.rs:282:63
13: tokio::runtime::coop::with_budget
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/coop.rs:107:5
tokio::runtime::coop::budget
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/coop.rs:73:5
tokio::runtime::park::CachedParkThread::block_on
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/park.rs:282:31
14: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/context/blocking.rs:66:9
...
Line 10 of the backtrace contains the line number in the test (2644).
Limit the number of IMAP connections to 1 per minute regardless of the reason of reconnection, but
allow one immediate retry. This is more reliable than ratelimiting only in error conditions because
ratelimiting can't be skipped by mistake. Anyway connections shouldn't be frequent in normal
operation mode.
send_webxdc_status_update JSON-RPC call
and corresponding Rust call sometimes fail in CI with
---
database is locked
Caused by:
Error code 5: The database file is locked
---
Adding more context to send_webxdc_status_update() errors
to better localize the error origin.
Otherwise when connection is lost IMAP may get into infinite loop
trying to parse remaining bytes:
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
Returning an error bubbles it up to `fetch_idle()`
which will call `trigger_reconnect()` and drop the connection.
To preview the docs, run:
```
scripts/build-python-docs.sh
firefox dist/html/index.html
```
I have removed the Makefile because modern Sphinx Makefile is just a
wrapper for `sphinx-build -M`:
3596590317/sphinx/templates/quickstart/Makefile.new_t
and sphinx-quickstart even has an option `--no-makefile`.
`make.bat` makes even less sense.
In `scripts/build-python-docs.sh` I use `sphinx-build` directly
without `make` wrapper.
Email addresses should generally be compared case-insensitively,
but there may be errors in comparison code.
To reduce the chance of problems, encode addresses
in Autocrypt and Autocrypt-Gossip in lowercase
to avoid propagating uppercase characters over the network
to other accounts potentially running buggy code.
If configured address is `Bob@example.net`,
but the message arrives adding `bob@example.net`,
Bob's device should still recognize it as addition of self
and fully recreate the group.
If the sender of the message in protected group chat
is not a member of the chat, mark the sender name with `~`
as we do it in non-protected chats and set the error
instead of replacing the whole message with
"Unknown sender for this chat. See 'info' for more details."
To send a message to a protected group this way
the sender needs to know the group ID
and sign the message with the current verified key.
Usually this is just a late message
delivered shortly after the user has left
the group or was removed from it.
Replacing the message with a single error text part
as done before this change makes it impossible
to access anything other than text, such as attached images.
Other devices should get the same chat name as the currently used device, i.e. the name a user sees
after renaming the chat. This fix is minor because `improve_single_line_input()` logic isn't going
to change often, but still, and also it simplifies the code.
This was introduced in https://github.com/deltachat/deltachat-core-rust/pull/4685
for internal use when looking up a chat based on In-Reply-To,
but actually affects the UIs as they should not display Download button
when the message is downloaded but cannot be decrypted.
If verified key for a contact is changed via securejoin,
gossip the keys in every group with this contact next time
we send a message there to let others learn new verified key
and let the contact who has resetup their device learn keys of others
in groups.
The returned error is unexpected and no UI I tested with stoped IO really handled it besides maybe displaying a toast.
(desktop and iOS don't handle it, deltatouch shows a toast)
This should not be shown to the user, it is only shown to the user if the UI has a bug, so that bug should be clearly visible.
Merge the code paths for verified and autocrypt key.
If both are changed, only one will be added.
Existing code path adds a message to all chats with the contact
rather than to 1:1 chat. If we later decide that
only 1:1 chat or only verified chats should be notified,
we can add a separate `verified_fingerprint_changed` flag.
Otherwise we will try to create an ad-hoc group
and failing because there are only two contacts
and then unblock a 1:1 chat just to assign
the message to trash in the end.
Hidden 1:1 chat is created for the "chat protected" message.
It should not appear simply because we received
a read receipt from the contact in a group.
Looks like this doesn't fix anything currently, because a better message from
`apply_group_changes()` doesn't appear in a context with another better message, but why drop it if
it's possible to add it, moreover, messages about implicit member additions are never dropped while
looking less important.
The second SQL statement calculating chat size
was already fixed in f656cb29be,
but more important statement calculating member list intersection
was overlooked.
As a result, trash chat with members added there due to former bugs
could still appear in similar chats.
- Reduce cross-module dependencies.
- Stop bloating the `sync` module while implementing synchronisation of more entities.
- Now there's the only `ChatId` :)
An error while executing an item mustn't prevent next items from being executed. There was a comment
that only critical errors like db write failures must be reported upstack, but in fact it's hard to
achieve in the current design, there are no error codes or so, so it's bug-prone. E.g.
`ChatAction::Block` and `Unblock` already reported all errors upstack. So, let's make error handling
the same as everywhere and just ignore any errors in the item execution loop. In the worst case we
just do more unsuccessful db writes f.e.
It's sufficient if the local state is updated successfully, no need to fail the whole
operation. Anyway delivery of sync messages and applying them on other devices are beyond of our
control. If an error occurs when generating a sync messages, probably a log message is sufficient,
no need to even show it to a user. As for the tests, anyway there are ones on synchronisation which
perform necessary checks. Particularly, some sync messages can't be generated if an account is
unconfigured. Adding the corresponding checks to the device synchronisation code (and maybe even
more checks in the future) would complicate the code unnecessarily. Even errors caused by bugs in
this code aren't a reason to fail a local operation.
Sync chat contacts across devices for broadcast lists and groups. This needs the corresponding chat
to exist on other devices which is not the case for unpromoted groups, so it fails for them now but
it's only a warning and will work once creation of unpromoted groups is synchronised too.
List-ID header is added for broadcast lists.
UTF-8 in email headers is allowed only if
all recipient MTAs support SMTPUTF8 extension,
which is not always the case even if our submission service
reports SMTPUTF8 support.
Don't grow timeout if interrupted early and slept not enough. Also:
- Don't grow timeout too fast, but 1.5--2 times (randomly) per iteration.
- Don't interrupt if rate-limited.
- Reset timeout if rate-limited. Rate limit isn't an error, so we can start from 30 secs again if an
error happens then.
When a key is gossiped for the contact in a verified chat,
it is stored in the secondary verified key slot.
The messages are then encrypted to the secondary verified key
if they are also encrypted to the contact introducing this secondary key.
Chat-Group-Member-Added no longer updates the verified key.
Verified group recovery only relies on the secondary verified key.
When a message is received from a contact
signed with a secondary verified key,
secondary verified key replaces the primary verified key.
When verified key is changed for the contact
in response to receiving a message
signed with a secondary verified key,
"Setup changed" message is added
to the same chat where the message is received.
this pr keeps and refines documentation added in #4951, however, reverts
the api introduced by #4951
which turns out to be not useful for UI in practise:
UI anyway check for chat/no-chat beforehand,
so a simple condition in profiles as
`green_checkmark = chat_exist ? chat_is_protected() :
contact_is_verified()`
is more useful in practise and is waht UI need and did already in the
past. (https://github.com/deltachat/deltachat-android/pull/2836 shows a
detailed discussion)
(as a side effect, beside saving code,
this PR saves up to three database calls
(get contact from chat in UI to pass it to profile_is_verified(), get
chat from contact in core, load is_protected in core) - instead, core
can use already is_protected from already loaded chat object)
/me did check rust-tests, fingers crossed for python tests
/me should re-setup python tests on local machine at some point :)
This message makes that partial messages do not change the group state.
A simple fix and a comprehensive test is added. This is a follow up to
the former #4841 which took a different approach.
This is another approach to provide group membership consistency for all members. Considerations:
- Classical MUA users usually don't intend to remove users from an email thread, so if they removed
a recipient then it was probably by accident.
- DC users could miss new member additions and then better to handle this in the same way as for
classical MUA messages. Moreover, if we remove a member implicitly, they will never know that and
continue to think they're still here.
But it shouldn't be a big problem if somebody missed a member removal, because they will likely
recreate the member list from the next received message. The problem occurs only if that "somebody"
managed to reply earlier. Really, it's a problem for big groups with high message rate, but let it
be for now.
Ensure the client does not busy loop
skipping IDLE if UIDNEXT of the mailbox is higher than
the last seen UID plus 1, e.g. if the message
with UID=UIDNEXT-1 was deleted before we fetched it.
We do not fallback to UIDNEXT=1 anymore
if the STATUS command cannot determine UIDNEXT.
There are no known IMAP servers with broken STATUS so far.
This allows to guarantee that select_with_uidvalidity()
sets UIDNEXT for the mailbox and use it in fetch_new_messages()
to ensure that UIDNEXT always advances even
when there are no messages to fetch.
Before, it was shown by UI when the chat is empty, however, in case of
verified groups, this is never the case any longer as the 'e2ee now
guaranteed' is always added.
Green checkmark in the contact profile
should only be displayed in the title
if the same checkmark is displayed in the title of 1:1 chat.
If 1:1 chat does not exist,
the checkmark should not be displayed in the title
of the contact profile.
Also add docs to is_protected
property of FullChat and BasicChat JSON objects.
This prevents accidentally going IDLE
when the last new message has arrived
while the folder was closed.
For example, this happened in some tests:
1. INBOX is selected to fetch, move and delete messages.
2. One of the messages is deleted.
3. INBOX is closed to expunge the message.
4. A new message arrives.
5. INBOX is selected with (CONDSTORE) to sync flags.
6. Delta Chat goes into IDLE without downloading the new message.
To determine that a new message has arrived
we need to notice that UIDNEXT has advanced when selecting the folder.
However, some servers such as Winmail Pro Mail Server 5.1.0616
do not return UIDNEXT in response to SELECT command.
To avoid interdependencies with the code
SELECTing the folder and having to implement
STATUS fallback after each SELECT even when we
may not want to go IDLE due to interrupt or unsolicited EXISTS,
we simply call STATUS unconditionally before IDLE.
Sync messages are only sent on explicit user actions and only one per action, so it's safe to send
them right away not worrying about the rate limit on the server.
This may happen if autocrypt key is changed after verification.
UI should still display who introduced the contact,
especially if the contact is still a member of some verified group.
If the contact is at the same time not verified,
i.e. its autocrypt key does not match verified key,
UI may display a crossed out checkmark instead of green checkmark.
Result::Err is reserved for local errors,
such as database failures.
Not found peerstate in the database is a protocol failure,
so just return Ok(false) in mark_peer_as_verified().
This allows to handle more errors with `?`.
Otherwise SELF contact in the beginning of the vector
and in to_ids may be repeated twice and not deduplicated.
dedup() only deduplicates consecutive elements.
We may not have a verified key for other members
because we lost a gossip message.
Still, if the message is signed with a verified key
of the sender, there is no reason to replace it with an error.
Even though r2d2 connection pool is removed,
deleting accounts still fails in Windows CI.
This reverts commit e88f21c010.
`try_many_times` documentation is modified to explain
why the workaround is still needed.
close#4620
This PR introduces a new core API to parse mailto links into a uniform
data format. This could be used to unify the different implementations
on the current platforms.
To complete this PR we have to decide for which APIs we want to expose
this (now) internal API (c, python, json-rpc, etc.), and if we want such
an API at all as it doesn't have a corresponding UI-PR and is not
_really_ needed.
And execute sync messages only if `Config::SyncMsgs` is enabled. Earlier executing was always
enabled, the messages are force-encrypted anyway. But for users it's probably more clear whether a
device is synchronised or not.
Using BufWriter ensures that `STARTTLS` command is sent
as a single packet.
Also refactor the code to ensure we only convert to
Box<dyn SessionStream> in the end.
Android calls get_connectivity_html()
every time connectivity changes, which in turn interrupts
IMAP loop and triggers change from "not connected" to "connecting"
state.
To avoid such infinite loop of IMAP interrupts when
there is not connectivity, update quota only when IMAP
loop is interrupted otherwise. This anyway happens
when a message is received or maybe_network is called.
Also remove outdated comments about `Action::UpdateRecentQuota` job
which does not exist anymore.
Winmail Pro Mail Server 5.1.0616 does not return UIDNEXT
in response to SELECT, but returns it when explicitly requested
via STATUS command:
? SELECT INBOX
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\Draft \Answered \Flagged \Deleted \Seen)] Limited
* 2 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1697802109] Ok
? OK [READ-WRITE] Ok SELECT completed
? STATUS INBOX (UIDNEXT)
* STATUS "INBOX" (UIDNEXT 4)
? OK STATUS completed
Previously used FETCH method is reported to fail for some users,
the FETCH command sometimes returns no results.
Besides, there is no guarantee that the message with
the highest sequence number has the highest UID.
In the worst case if STATUS does not return UIDNEXT
in response to explicit request, we fall back to setting
UIDNEXT to 1 instead of returning an error.
feat: Make broadcast lists create their own chat - UIs need to ask for
the name when creating broadcast lists now (see
https://github.com/deltachat/deltachat-android/pull/2653)
That's quite a minimal approach: Add a List-ID header to outgoing
broadcast lists, so that the receiving Delta Chat shows them as a
separate chat, as talked about with @r10s and @hpk42.
Done:
- [x] Fix an existing bug that the chat name isn't updated when the
broadcast/mailing list name changes (I already started this locally)
To be done in other PRs:
- [ ] Right now the receiving side shows "Mailing list" in the subtitle
of such a chat, it would be nicer if it showed "Broadcast list" (or
alternatively, rename "Broadcast list" to "Mailing list", too)
- [ ] The UIs should probably ask for a name before creating the
broadcast list, since it will actually be sent over the wire. (Android
PR: https://github.com/deltachat/deltachat-android/pull/2653)
Fixes https://github.com/deltachat/deltachat-core-rust/issues/4597
BREAKING CHANGE: This means that UIs need to ask for the name when creating a broadcast list, similar to https://github.com/deltachat/deltachat-android/pull/2653.
This approach uses a param field to enable forcing the sticker
`viewtype`. The first commit has the memory-only flag implemented, but
this flag is not persistent through the database conversion needed for
draft/undraft. That's why `param` has to be used.
follow up to #4814fixes#4739
---------
Co-authored-by: Septias <scoreplayer2000@gmail.comclear>
If there's a temporary SMTP error, pretend there are no more MDNs to send in send_mdn(). It's
unlikely that other MDNs could be sent successfully in case of connectivity problems. This approach
is simpler and perhaps even better than adding a progressive backoff between MDN sending retries --
MDNs just will be resent after a reconnection, which makes some sense.
This is needed to test periodic re-gossiping in existing chats.
Also add a test for verified groups on that even if "member added" message is missed by a device of
newly added member, after re-gossiping Autocrypt keys to the group it successfully learns these keys
and marks other members as verified.
Set connectivity status to "connected" at the end of connect() call.
Otherwise for servers that do not support IMAP IDLE
connect() is called at the beginning of fake idle
and connectivity stays in "connecting" status most of the time.
Before they were trashed. Note that for unencrypted ones DC works as expected creating the requested
group immediately because Chat-Group-Id is duplicated in the Message-Id header and Subject is
fetched.
Also add a test on downloading a message later. Although it doesn't reproduce #4700 for some reason,
it fails w/o the fix because before a message state was changing to `InSeen` after a full download
which doesn't look correct. The result of a full message download should be such as if it was fully
downloaded initially.
This ensures the server does not get SIGINT
when the bot is running in a terminal and user presses ^C
We want to send SIGTERM ourselves after clean shutdown,
e.g. stopping I/O for all accounts.
Add protected-headers="v1" directive to Content-Type of an encrypted/signed MIME so that other MUAs
like Thunderbird display the true message Subject instead of "...".
For example, reader_loop() may die
if readline() tries to read too large line
and thows an exception.
We want to at least log the exception in this case.
This certificate does not exist on older Android phones.
It is already added manually for IMAP and SMTP,
this commit adds the same certificate for HTTP requests.
it may happen that percentages larger than 100% are reported by the provider,
eg. for some time a storage usage of 120% may be accepted.
while we should report the values "as is" to the user,
for the bar, percentages larger 100% will destroy the layout.
9bd7ab72 brings a possibility of group membership inconsistency to the original Hocuri's algo
described and implemented in e12e026b in sake of security so that nobody can add themselves to a
group by forging "InReplyTo" and other headers. This commit fixes the problem by removing group
members locally if we see a discrepancy with the "To" list in the received message as it is better
for privacy than adding absent members locally. But it shouldn't be a big problem if somebody missed
a member addition, because they will likely recreate the member list from the next received
message. The problem occurs only if that "somebody" managed to reply earlier. Really, it's a problem
for big groups with high message rate, but let it be for now.
Also:
- Query chat contacts from the db only once.
- Update chat contacts in the only transaction, otherwise we can just break the chat contact list
halfway.
- Allow classic MUA messages to remove group members if a parent message is missing. Currently it
doesn't matter because unrelated messages go to new ad-hoc groups, but let this logic be outside
of apply_group_changes(). Just in case if there will be a MUA preserving "Chat-Group-ID" header
f.e.
The code removed is an incomplete implementation of skipping
the Legacy Display Part specified in
https://www.ietf.org/archive/id/draft-autocrypt-lamps-protected-headers-02.html#section-5.2
The code does not fully implement the specification, e.g.
it does not check that there are exactly two parts.
Delta Chat and Thunderbird are not adding this part anyway,
and it is defined as "transitional" in the draft.
This also removes misplaced warning "Ignoring nested protected headers"
that is printed for every incoming Delta Chat message
since commit 5690c48863
which is part of the PR <https://github.com/deltachat/deltachat-core-rust/pull/982>.
A few people got the impression that if you send 6 messages
in a burst you'll only be able to send the next one in 60 seconds.
Hopefully this can resolve it.
This is an RFC 2045 requirement for base64-encoded MIME parts.
Previously referenced RFC 5322 requirement
is a general Internet Message Format requirement
and is more generous.
Previously it was required that a directory path is provided to the import API.
Now it is possible to point directly to the .asc file containing a secret key.
This allows UI to present a file selection dialog to the user
and let select the file directly.
Selecting a directory is still supported for backwards compatibility.
Such a message may be assigned to a wrong chat (e.g. undecipherable group msgs often get assigned to
the 1:1 chat with the sender). Add `DownloadState::Undecipherable` so that messages referencing
undecipherable ones don't go to that wrong chat too. Also do not reply to not fully downloaded
messages. Before `Message.error` was checked for that purpose, but a message can be error for many
reasons.
Previously only one connection, the one used to change the key,
was working after passphrase change.
With this fix the whole pool of connections
is recreated on passphrase change, so there is no need
to reopen the database manually.
These constants are current defaults in `pgp` crate,
this change would prevent accidental change due to rPGP upgrade
and make it easier to change in a single place.
It can be not good for membership consistency if we missed a message adding a member, but improves
security because nobody can add themselves to a group from now on.
1:1 chat may be blocked while the contact is not
if 1:1 chat was created as a result of scanning
a verified group join QR code with the contact
as the inviter. In this case 1:1 chat is blocked to hide it
while the contact is unblocked.
- If we don't know the parent (=In-Reply-To) message, then completely recreate the group member list
(i.e. use the member list of the incoming message) (because we assume that we missed some messages
& have a wrong group state).
- If the message has a "Chat-Group-Member-Removed: member@example.com" header, then remove this
member.
- If the message has a "Chat-Group-Member-Added: member@example.com" header, then add this member.
That means:
- Remove checks for the presense of `ContactId::SELF` in the group. Thus all recipients of a message
take the same decision about group membership changes, no matter if they are in the group
currently. This fixes a situation when a recipient thinks it's not a member because it missed a
message about its addition before.
NOTE: But always recreate membership list if SELF has been added. The older versions of DC don't
always set "In-Reply-To" to the latest message they sent, but to the latest delivered message (so
it's a race), so we need this heuristic currently.
- Recreate the group member list if we don't know the parent (=In-Reply-To) message, even if the
sender isn't in the group as per our view, because we missed some messages and our view may be
stale.
The new message for which `parent_query()` is done may assume that it will be received in a context
affected by those messages, e.g. they could add new members to a group and the new message will
contain them in "To:". Anyway recipients must be prepared to orphaned references.
This is already the way `get_chatlist_entries` works.
`get_similar_chatlist_entries` is renamed into
`get_similar_chat_ids` because return values are not entries anymore.
If the user makes a backup from a UI that supports the experimental verified 1:1 chats (e.g. nightly Android) and imports it into a UI that doesn't, then this config should be cleared.
We already talked about this a long time ago after @Simon-Laux noticed this problem, but I think we didn't actually solve it back then?
Best to review with whitespace changes disabled.
I.e. exclude from the list the following chats as well:
- Read-only mailing lists.
- Chats we're not a member of.
But as for ProtectionBroken chats, we return them, as that may happen to a verified chat at any
time. It may be confusing if a chat that is normally in the list disappears suddenly. The UI need to
deal with that case anyway.
(cherry picked from commit 83ef25e7de)
I.e. exclude from the list the following chats as well:
- Read-only mailing lists.
- Chats we're not a member of.
But as for ProtectionBroken chats, we return them, as that may happen to a verified chat at any
time. It may be confusing if a chat that is normally in the list disappears suddenly. The UI need to
deal with that case anyway.
If a message is unsigned or signed with an unknown key, `MimeMessage::was_encrypted()` returns
false. So, it mustn't be checked when deciding whether to look into
`MimeMessage::decoded_data`. Looking through git history one can see that it's just a wrong check
left in the code for historical reasons.
Other MUAs don't set add/remove headers, so the only way for them to re-add us to the group is to
add us to To/CC/wherever. Previously it worked only for other members that are still in the group so
that they properly handled our re-addition, but we didn't.
Check if a sticker has at least one fully transparent corner and otherwise change the Sticker type
to Image. This would fix both Android and iOS at the same time and prevent similar bug on future
platforms that may get this bug like Ubuntu Touch.
I.e. from delete_msgs(). Otherwise messages must not be deleted from there, e.g. if a message is
ephemeral, but a network outage lasts longer than the ephemeral message timer, the message still
must be sent upon a successful reconnection.
I.e. from delete_msgs(). Otherwise messages must not be deleted from there, e.g. if a message is
ephemeral, but a network outage lasts longer than the ephemeral message timer, the message still
must be sent upon a successful reconnection.
If the Inbox is fetched before the Sentbox (as done currently), messages from the Sentbox will
correctly mingle with the Inbox messages in the end. So, this commit changes message ordering only
if we already have processed outgoing messages, e.g. if we just sent them in the chat as described
in #4621. Otherwise new incoming messages are displayed somewhere in the middle of the chat which
doesn't look usable.
Directly unwrap in TestContext::get_chat()
Turns out that all usages of get_chat() directly unwrapped, because in a
test it doesn't make sense to handle the error of there being no chat.
Don't show a contact as verified if their key changed in the meantime
If a contact's key changed since the verification, then it's very
unlikely that they still have the old, verified key. So, don't show them
as verified anymore.
This also means that you can't add a contact like this to a verified
group, which is good.
The documentation actually already described this (new) behavior:
```rust
/// and if the key has not changed since this verification.
```
so, this adapts the code to the documentation.
It can be used e.g. as a default in the file saving dialog. Also display the original filename in
the message info. For these purposes add Param::Filename in addition to Param::File and use it as an
attachment filename in sent emails.
Opening the same account (context) from multiple processes is dangerous, can result in duplicate
downloads of the same message etc. Same for account manager, attempts to modify the same
accounts.toml even if done atomically with may result in corrupted files as atomic replacement
procedure does not expect that multiple processes may write to the same temporary file.
accounts.toml cannot be used as a lockfile because it is replaced during atomic update. Therefore, a
new file next to accounts.toml is needed to prevent starting second account manager in the same
directory.
But iOS needs to be able to open accounts from multiple processes at the same time. This is required
as the "share-to-DC extension" is a separate process by iOS design -- this process may or may not be
started while the main app is running. Accounts are not altered however by this extension, so let's
add to the `Accounts::new()` constructor an `rdwr` parameter which allows to read the accounts
config w/o locking the lockfile.
Webxdc update messages may contain
long lines that get hard-wrapped
and may corrupt JSON if the message
is not encrypted.
base64-encode the update part
to avoid hard wrapping.
This is not necessary for encrypted
messages, but does not introduce
size overhead as OpenPGP messages
are compressed.
Correctly handle messages with old timestamps for verified chats:
* They must not be sorted over a protection-changed info message
* If they change the protection, then they must not be sorted over existing other messages, because then the protection-changed info message would also be above these existing messages.
This PR fixes this:
1. Even seen messages can't be sorted into already-noticed messages anymore. **This also changes DC's behavior in the absence of verified 1:1 chats**. Before this PR, messages that are marked as seen when they are downloaded will always be sorted by their timestamp, even if it's very old.
2. protection-changed info messages are always sorted to the bottom.
**Edit:**
3. There is an exception to rule 1: Outgoing messages are still allowed to be sorted purely by their timestamp, and don't influence old messages. This is to the problem described at [*].
Together, these rules also make sure that the protection-changed info message is always right above the message causing the change.
[*] If we receive messages from two different folders, e.g. `Sent` and `Inbox`, then this will lead to wrong message ordering in many cases. I need to think about this more, or maybe someone else has an idea. One new idea that came to my mind is:
* Always sort noticed messages under the newest info message (this PR sorts them under the newest noticed message, master sorts them purely by their sent timestamp)
* Always sort unnoticed messages under the newest noticed message (that's the same behavior as in this PR and on master)
* Always sort protection-changed info messages to the bottom (as in this PR)
However, after a talk with @link2xt we instead decided to add rule 3. (see above) because it seemed a little bit easier.
The test checks that if webxdc update is too large to
download with the current `download_limit`,
it is applied afterwards when the user manually downloads the update message.
tokio-tar 0.3.0 prints message "create_dir_all ..." to stdout during import.
tokio-tar 0.3.1 has removed this debug output which broke stdio JSON-RPC protocol.
If the verification is broken, `can_send()` is false.
But if the user was typing a message right when a verification-breaking message came in, the UI still needs to be able to save it as a draft.
Steps to reproduce the bug:
- Set a draft
- Your chat partner breaks verification
- Go back to the chats list
- Go to the chat again
- Accept the breakage
- Expected: The draft is still there
- Bug behavior: The draft is gone
Some Dovecot servers are configured
to alias "INBOX.DeltaChat" and "DeltaChat" to the same folder.
In this case Delta Chat moves new emails from "INBOX"
to "DeltaChat", but then discovers the same mail in "INBOX.DeltaChat"
and tries to move it to "DeltaChat" again.
Each time a MOVE command is issued to move the message
from "INBOX.DeltaChat" to "DeltaChat", the message gets a new UID.
To prevent such IMAP move loop between aliased folders,
we do not move the message if we have already seen it on IMAP,
i.e. we have its Message-ID in the `imap` table.
Note that we do not use `rfc724_mid_exists`,
because it checks the `msgs` table and would consider
BCC-self messages seen even before we see them in the Inbox,
preventing their move to the DeltaChat folder.
Duplicate messages and messages without Message-IDs
are not moved anymore, but this is better
than having an infinite move loop.
1:1 chats are automatically created as protected if the contact is
verified, there is no need to explicitly do this.
Plus, by removing this call, the test also tests that automatically
creating 1:1 chats as protected works.
receive_imf() calls add_parts()
which INSERTs or UPDATEs the message using UPSERT [1].
It then uses last_insert_rowid() to get
the ID of the inserted message.
However, it is incorrect to use last_insert_rowid()
if an UPDATE was executed instead of INSERT.
The solution is to use `RETURNING id` clause
to make the UPSERT statement return message ID in any case [2].
The fix is tested in test_webxdc_update_for_not_downloaded_instance()
and with a debug_assert!.
[1] https://www.sqlite.org/lang_UPSERT.html
[2] https://sqlite.org/forum/forumpost/9ce3bc1c4a85c15f
When `cargo test` is executed,
all examples are built by default
to ensure that they can be compiled.
This is a documented and expected behaviour,
even though it was previously reported as a bug:
<https://github.com/rust-lang/cargo/issues/6675>
In particular, `examples/simple.rs` is built into
a 67M binary `target/debug/examples/simple`.
This is unnecessary to do so every time
you change a line in the `deltachat` crate
and want to rerun the tests.
Workaround is to run `cargo test --tests`,
but it is easy to forget and is not discoverable
unless you read the "Target Selection" section of `cargo help test`.
We have a maintained example at https://github.com/deltachat-bot/echo,
so there is no need for an example in the core repository.
This commit adds new stock strings
"I added member ...",
"I removed member ..." and
"I left the group" that are sent over the network
and are visible in classic MUAs like Thunderbird.
Member name in these messages uses authname
instead of the display name,
so the name set locally does not get leaked when
a member is added or removed.
Message.set_text() and Message.get_text() are modified accordingly
to accept String and return String.
Messages which previously contained None text
are now represented as messages with empty text.
Use Message.set_text("".to_string())
instead of Message.set_text(None).
simplify() is written to process incoming plaintext messages
and extract footers and quotes from them.
Incoming messages contain various quote styles
and simplify() implements heuristics to detects them.
If dehtml() output is processed by simplify(),
simplify() heuristics may erroneously detect
footers and quotes in produced plaintext.
dehtml() should directly detect quotes
instead of converting them to plaintext quotes
for parsing with simplify().
This is a follow-up to cbe1671104
I changed `apt-get` arguments for x86_64 glibc builds,
but forgot to change it for x86_64 musl, aarch64 glibc and aarch64 musl.
Because of this, `upload-wheels` task failed with a message:
```
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt install python3.11-venv
You may need to use sudo with that command. After installing the python3-venv
package, recreate your virtual environment.
```
Mergeable is disabled because it was requiring
that PR title follows conventional commit notation
even when PR consisted of multiple commits
and was not planned to be squash merged.
`ouroboros` is deprecated with a security advisory recommending
switch to `self_cell` crate:
https://rustsec.org/advisories/RUSTSEC-2023-0042
async-imap 0.9.0 depends on `self_cell` instead of `ouroboros`.
This commit solves the "error: externally-managed-environment"
which started appearing since Debian 12 release.
`debian` is used as an Docker image to run devpi.
Specifying msg IDs that cannot be loaded in the event payload
results in an error when the UI tries to load the message.
Instead, emit an event without IDs
to make the UI reload the whole messagelist.
New algorithm improves group consistency
in cases of missing messages,
restored old backups and replies from classic MUAs.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: link2xt <link2xt@testrun.org>
When running `cargo test` in the deltachat-jsonrpc folder,
a new file `openrpc/openrpc.json` will be created with an
[OpenRPC](https://spec.open-rpc.org/) definition.
It can be copy-pasted into the
[OpenRPC playground](https://playground.open-rpc.org/)
and used to generate clients in other languages.
With Python 3.7 asynchronous tests randomly fail
with "RuntimeError: Event loop is closed" during shutdown.
Backtrace of the error includes `SafeChildWatcher` calls.
Python 3.8 has replaced `SafeChildWatcher`
with a new `ThreadedChildWatcher` by default [1]
as a bugfix for
"asyncio.create_subprocess_exec() only works with main event loop" bug [2].
Python 3.7 scheduled end of life is 2023-06-27
according to <https://devguide.python.org/versions/>.
[1] https://github.com/python/cpython/pull/14344
[2] https://bugs.python.org/issue35621
Normally the message has UID 1,
but this is not true when account is reused.
In this case the message may be
"Marked messages 2 in folder DeltaChat as seen."
and the test times out waiting for
"Marked messages 1 in folder DeltaChat as seen."
According to RFC 3501, EXISTS must always be sent in response to SELECT.
But if the server does not send it for some reason,
async-imap uses the default value, so we will essentially fetch `1:*`
and downloading all messages may take a long time.
convert `JSONRPCReactions.reactions` to a `Vec<JSONRPCReaction>` with unique reactions and their count, sorted in descending order.
---------
Co-authored-by: link2xt <link2xt@testrun.org>
According to RFC 3501, EXISTS must always be sent in response to SELECT.
But if the server does not send it for some reason,
async-imap uses the default value, so we will essentially fetch `1:*`
and downloading all messages may take a long time.
* jsonrpc: `MessageSearchResult`, always include `chat_name`(not an option anymore), also add `author_id` and `chat_type`
* replace .ok_or_else with anyhow context
* fix: jsonrpc: typescript client: fix types of events in event emitter
* add ts ignore to remove ts error (when building in ts5)
doing the types right via casting is to complicated IMO and has no real benefit here.
Why? because desktop currently fetches the chatlist multiple times, even though it just needs the
chatlistitem for one chat.
Note: @r10s was worried before that exposing the method to get a single updated chatlistitem could
lead to race conditions where the chatlist item is newer than the chatlist order. But I don't think
this will change anything for desktop besides making it a little faster (because currently desktop
fetches the whole chatlist instead of just the entry it needs when an entry updates).
This way is more compatible to JSON-RPC libraries
that do not support receiving notifications from the server
and allows describing event types in the OpenRPC specification.
Event thread converting events to notifications in the FFI
is removed, so it is now possible to construct a dc_jsonrpc_instance_t
while still retrieving events via dc_event_emitter_t.
Otherwise sending a message without plaintext part
resets the signature. It is particularly dangerous
in multidevice case, because it's easy to accidentally
reset the signature on your other device with a non-text message.
Move DebugLogging into debug logging module.
Remove Context.send_log_event().
Use blocking_read() instead of try_read() to
get read lock on debug_logging() in synchronous code.
Do not try to log events while holding a lock
on DebugLogging.
I.e. > 500K for the balanced media quality and 130K for the worse one. This can remove animation and
transparency from PNG/WebP, but then a user always can send an image as a file.
Also don't reduce wide/high images if they aren't huge. Among other benefits, this way most of PNG
screenshots aren't touched.
Also remove Exif from all images, not from JPEGs only.
This patch adds new C APIs
dc_get_next_msgs() and dc_wait_next_msgs(),
and their JSON-RPC counterparts
get_next_msgs() and wait_next_msgs().
New configuration "last_msg_id"
tracks the last message ID processed by the bot.
get_next_msgs() returns message IDs above
the "last_msg_id".
wait_next_msgs() waits for new message notification
and calls get_next_msgs().
wait_next_msgs() can be used to build
a separate message processing loop
independent of the event loop.
Async Python API get_fresh_messages_in_arrival_order()
is deprecated in favor of get_next_messages().
Introduced Python APIs:
- Account.wait_next_incoming_message()
- Message.is_from_self()
- Message.is_from_device()
Introduced Rust APIs:
- Context.set_config_u32()
- Context.get_config_u32()
* Don't let blocking be bypassed using groups
Fix#4313
* Fix another bug: A blocked group was sometimes not unblocked when an unblocked contact sent a message into it.
* Small performance improvement by not unnecessarily loading the peerstate
* Remove wrong info message "{contact} verified" when scanning a QR code with just an email
I think that this was a bug in the original C code and then slipped
through two refactorings.
Moved custom ToSql trait including Send + Sync from lib.rs to sql.rs.
Replaced most params! and paramsv! macro usage with tuples.
Replaced paramsv! and params_iterv! with params_slice!,
because there is no need to construct a vector.
bjoern <r10s@b44t.com> wrote:
> maybe_add_time_based_warnings() requires some date guaranteed to be in the near past. based on
this known date we check if the system clock is wrong (if earlier than known date) and if the used
Delta Chat version may be outdated (1 year passed since known date). while this does not catch all
situations, it catches quite some errors with comparable few effort.
>
> figuring out the date guaranteed to be in the near past is a bit tricky. that time, we added
get_provider_update_timestamp() for exactly that purpose - it is checked manually by some dev and
updated from time to time, usually before a release.
>
> however, meanwhile, the provider-db gets updated less frequently - things might be settled a bit
more - and, get_provider_update_timestamp() was also changed to return the date of the last commit,
instead of last run. while that seem to be more on-purpose, we cannot even do an “empty” database
update to update the known date.
>
> as get_provider_update_timestamp() is not used for anything else, maybe we should completely
remove that function and replace it by get_last_release_timestamp that is then updated by
./scripts/set_core_version.py - the result of that is reviewed manually anyway, so that seems to be
a good place (i prefer manual review here and mistrust further automation as also dev or ci clocks
may be wrong :)
This adds a few log items for imex::transfer::get_backup and the
ongoing process to give some more insights.
IO is now also paused after the ongoing process is allocated in
get_backup to avoid needlessly pausing IO.
that way, UI can just close the transfer dialog,
so that, at the end, both devices end in the chatlist.
we can also use this for troubleshooting -
if the device message is not present,
transfer did not succeed completely.
(a separate device message may be nice in that case -
but that is another effort,
same for making the device message reappear after deletion
or after some time)
some providers say that they support QUOTA in the IMAP CAPABILITY,
but return an empty list without any quota information then.
in our "Connectivity", this looks a bit of an error.
i have not seen this error often - only for testrun.org -
if it is usual, we could also just say "not supported"
(as we do in case QUOTA is not returned).
a translations seems not to be needed for now,
it seems unusual, and all other errors are not translated as well.
With this, all the builds happen in parallel in 3-6 minutes,
and then python tests run for 8-9 minutes.
Before, it was 6-17 minutes for compilation,
then 19-31 minutes for python tests.
Compilation of the library and test binary was not parallelized,
and then python tests included async python tests
and deltachat-rpc-server binary compilation.
Now, deltachat-rpc-server compilation,
Rust testing and async python tests are separate jobs.
Similarly to how `imex_inner()` runs migrations
after successful call to `import_backup()`,
migrations should be run after receiving a backup
using `transfer_from_provider()`.
This way if flaky python tests fail,
it is possible to rerun them without having to rerun library compilation.
Python test jobs will redownload already built library artifact
and compile the bindings against it.
This further reduces the cognitive overload of having many ways to do
something. The same is very easily done using composition. Followup
from 82ace72527.
`install_python_bindings.py` was not used by CI
and scripts, except for `scripts/run-python-test.sh`
which only used it to invoke `cargo`.
Instead of using an additional script,
run cargo directly.
The documentation is updated to remove
references to `install_python_bindings.py`.
The section "Installing bindings from source"
was in fact incorrect as it suggested
running `tox --devenv` first and only then
compiling the library with `install_python_bindings.py`.
Now it explicitly says to run cargo first
and then install the package without using `tox`.
`tox` is still used for running the tests
and setting up development environment.
* fix(imex): transfer::get_backup must always free ongoing process
When the ongoing process is cancelled it is still the responsibility
of whoever took out the ongoing process to free it. This code was
only freeing the ongoing process when completed normally but not when
cancelled.
* add changelog
Recommended file size is used for recoding media.
For the upper size, we rely on the provider to tell us back
if the message cannot be delivered to some recipients.
This allows to send large files, such as APKs,
when using providers that don't have such strict limits.
This removes the message that needed to be supplied to LogExt::log_err
calls. This was from a time before we adopted anyhow and now we are
better off using anyhow::Context::context for the message: it is more
consistent, composes better and is less custom.
The benefit of the composition can be seen in the FFI calls which need
to both log the error as well as return it to the caller via
the set_last_error mechanism.
It also removes the LogExt::ok_or_log_msg funcion for the same reason,
the message is obsoleted by anyhow's context.
This moves us back to a released version;
- Ticket is now opaque, need to use accessor functions.
- Ticket now ensures it is valid itself, no need to inspect it's
inners. Deserialisation would fail if it was bad.
- The git version was accidentally used with default-features enabled
and thus pulled in a few too many dependencies. They are now gone.
This ensures that the BackupProvider will be stopped as soon as the
struct is dropped and the imex progress error event is emitted. This
makes it easier to use and also makes sure that the ffi call
dc_backup_provider_unref() does not lead to dangling resources.
cargo-zigbuild is replaced with a simple zig-cc wrapper.
cargo-zigbuild still tries to use i386-linux-musl triple,
while in zig 0.11 x86-linux-musl should be used [1].
This also gives us more control over compilation flags used
and prevents changes due to cargo-zigbuild upgrades,
as its version was not pinned.
[1] Merged PR "all: rename i386 to x86 "
<https://github.com/ziglang/zig/pull/13101>
This uses the new iroh API to connect to all provider addresses
concurrently. It simplifies the implementation as well as we no
longer need to try the addresses manually.
* deps: Update iroh, remove default-net patch
The released version of default-net is now sufficient and iroh makes
sure this dependency is recent enough.
* Update cargo-deny config
* Newer version of spin, previous has been yanked
When trying IP addresses from the ticket, have a very rough sort order
in which to try them. Basically assume most local wifi's are
somewhere on 192.168.0.0/16 so prefer those first.
.wait_for_seen() is unreliable, because sometimes Dovecot
sends only EXISTS to the IDLE connection, but not the FETCH.
Dovecot sends updates like FETCH only if some
connection has already observed the message in previous state
without the \Seen flag.
To avoid this race condition, wait until the core sets the flag,
then FETCH the message manually and check that the flag is set.
The documentation says this blocks. This should block because it also
means the error reporting is more accurate by calling set_last_error
just before returning.
Otherwise it is possible for the context that is used in the spawn to
be unreferenced. Really this should be caught by the borrow checker
that ensures we only spawn things with a 'static lifetime, but we're
handling raw pointers so it doesn't.
This replaces the mechanism by which the IoPauseGuard makes sure the
IO scheduler is resumed: it really is a drop guard now by sending a
single message on drop.
This makes it not have to hold on to anything like the context so
makes it a lot easier to use.
The trade-off is that a long-running task is spawned when the guard is
created, this task needs to receive the message from the drop guard in
order for the scheduler to resume.
This changes the JSON-RPC APIs to get a QR code from the backup
provider to block. It means once you have a (blocking) call to
provide_backup() you can call get_backup_qr() or get_backup_qr_svg()
and they will block until the QR code is available.
Calling get_backup_qr() or get_backup_qr_svg() when there is no backup
provider will immediately error.
This adds a result extension trait to explicitly set the last error,
which *should* be the default for the FFI. Currently not touching all
APIs since that's potentially disruptive and we're close to a release.
The logging story is messy, as described in the doc comment. We
should further clean this up and tidy up these APIs so it's more
obvious to people how to do the right thing.
This makes the BackupProvider automatically invoke pause-io while it
is needed.
It needed to make the guard independent from the Context lifetime to
make this work. Which is a bit sad.
To handle backups the UIs have to make sure they do stop the IO
scheduler and also don't accidentally restart it while working on it.
Since they have to call start_io from a bunch of locations this can be
a bit difficult to manage.
This introduces a mechanism for the core to pause IO for some time,
which is used by the imex function. It interacts well with other
calls to dc_start_io() and dc_stop_io() making sure that when resumed
the scheduler will be running or not as the latest calls to them.
This was a little more invasive then hoped due to the scheduler. The
additional abstraction of the scheduler on the context seems a nice
improvement though.
async-smtp does not implement read buffering anymore
and expects library user to implement it.
To implement read buffer, we wrap streams into BufStream
instead of BufWriter.
The fact that `PRAGMA incremental_vacuum` may return a zero-column
SQLITE_ROW result is documented in `sqlite3_data_count()` documentation:
<https://www.sqlite.org/c3ref/data_count.html>
Previously successful auto_vacuum worked,
but resulted in a "Failed to run incremental vacuum" log.
With existing approach of constructing
the SQL query dynamically I get errors like this:
ephemeral.rs:575: update failed: too many SQL variables: Error code 1: SQL error or missing database
In my case it is trying to delete 143658 messages.
This is the result of importing a Desktop backup
and enabling device auto-deletion on the phone.
Current SQLite limit is 32766 variables
as stated in <https://www.sqlite.org/limits.html>.
This way we can test some recently added config options that we don't want to expose in UI like
DeleteToTrash or SignUnencrypted. Note that persistent config options like DeleteToTrash should
remain anyway because they allow fine-grained (per-account) control. Having them matters for tests
also.
This fixes use-after-free in case dc_context_unref() is called
while the background process spawned by dc_configure() is still
running.
Corresponding regression test in Python can be run with
`pytest tests/test_1_online.py::test_configure_unref`.
dc_configure() spawns a background configuration process.
It should increase the number of context references
so even if we unref the context, it is not dropped
until the end of the configuration process.
Currently running the test
with `pytest tests/test_1_online.py::test_configure_unref`
results in segmentation fault.
`<meta name="color-scheme" content="light dark" />` is a hint to the browsers
that the page can be rendered in light as well as in dark mode
and that the browser can apply corresponding defaults.
as we do not add css colors on our own,
this is sufficient for letting generated html messasge being rendered
in dark mode.
cmp. https://drafts.csswg.org/css-color-adjust/#color-scheme-propcloses#4146
standard footers meanwhile go the "contact status",
so they are no longer a reason to trigger "full message view".
this was already discussed when the HTML view was introduced at #2125
however, forgotten to change when the "contact status" was added at #2218
this pr will result in a cleaner chat view
with less "Show Full Message..." buttons
and will also save some storage
(in fact, i came over that when reviewing #4129 )
the contrast was decreased at
https://github.com/deltachat/deltachat-core-rust/pull/4136 ,
there were also suggestions to fix it there,
but it was probably just forgotten :)
this pr increases the contrast
between code background and code font again to the level it was before
and also to what similar themes are doing.
We are currently using libsqlite3-sys 0.25.2,
corresponding to SQLcipher 4.5.2
and SQLite 3.39.2.
SQLite supports ALTER TABLE DROP COLUMN since version 3.35.0,
and it has received critical database corruption bugfixes in 3.35.5.
There have been no fixes to it between SQLite 3.36.0 and 3.41.0,
so it appears stable now.
Explicit `prepare_msg` corresponding to `dc_prepare_msg`
is intended only for the case where the message is prepared
before the file is ready. It is not indented for calling
right before send_msg and requires that file path in the blobdir.
With explicit `prepare_msg` removed
all the tests still pass but there is no requirement
that the file is put into blobdir beforehand.
This adds more configuration to cargo-deny so that the output is not a
giant list of warnings hiding new entries. Instead the config now
lists the things that would emit warnings. It also adds -Dwarnings to
the CI job so that new warnings will be cleaned up: they can be added
to the config easily to fix the warnings.
Desktop does use this as it allows reading QR codes as text from the
clipboard as well as copying the QR text to the clipboard instead of
showing the QR code.
We do not make all transactions
[IMMEDIATE](https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions)
for more parallelism -- at least read transactions can be made DEFERRED to run in parallel
w/o any drawbacks. But if we make write transactions DEFERRED also w/o any external locking,
then they are upgraded from read to write ones on the first write statement. This has some
drawbacks:
- If there are other write transactions, we block the thread and the db connection until
upgraded. Also if some reader comes then, it has to get next, less used connection with a
worse per-connection page cache.
- If a transaction is blocked for more than busy_timeout, it fails with SQLITE_BUSY.
- Configuring busy_timeout is not the best way to manage transaction timeouts, we would
prefer it to be integrated with Rust/tokio asyncs. Moreover, SQLite implements waiting
using sleeps.
- If upon a successful upgrade to a write transaction the db has been modified by another
one, the transaction has to be rolled back and retried. It is an extra work in terms of
CPU/battery.
- Maybe minor, but we lose some fairness in servicing write transactions, i.e. we service
them in the order of the first write statement, not in the order they come.
The only pro of making write transactions DEFERRED w/o the external locking is some
parallelism between them. Also we have an option to make write transactions IMMEDIATE, also
w/o the external locking. But then the most of cons above are still valid. Instead, if we
perform all write transactions under an async mutex, the only cons is losing some
parallelism for write transactions.
scripts/update-provider-database.sh checks out the provider database
and updates data.rs file,
making it easier to update.
CI uses this script to check that checked in data.rs
corresponds to the provider database repository.
the slowest test was `test_modify_chat_disordered`;
due to several sleep(), it lasts at least 11 seconds.
however, many of the sleep() are not needed and were added
just to get things done that time.
this pr removes 7 superfluous sleeps,
making the test finish in about 3 seconds,
so that this test is no longer the bottleneck when running
`cargo test` on fast machines.
(this is not only useful to safe some seconds -
but also to notice degradations as of
https://github.com/deltachat/deltachat-core-rust/issues/4051#issuecomment-1436995166 )
Although it does a little for security, it will help to protect from unwanted server-side
modifications and bugs. And now we have a time to test "multipart/signed" messages compatibility
with other MUAs.
This makes DC compatible with "multipart/signed" messages thus allowing switching to them someday
from the current "multipart/mixed" unencrypted message format.
try_decrypt() is a CPU-bound task.
When called from async function,
it should be wrapped in tokio::task::spawn_blocking().
Using tokio::task::spawn_blocking() is difficult here
because of &mail, &private_keyring and &public_keyring borrows,
so we should at least use tokio::task::block_in_place()
to avoid blocking the executor.
Previously we used tags like "1.109.0" and "py-1.109.0".
Since Python bindings are always released
at the same time as the core,
it is easier to use a single tag for them.
"v" prefix is added to make matching by "v*" easier
in CI and scripts.
the speedup of #4065 is larger than measured first
running `cargo test`, first we thought there is only a slight speedup
from 13.5 seconds to 12.5 seconds on a m1pro.
however, there is one test that does 11 seconds of sleep() (test_modify_chat_disordered)
as this test is not run very first, this slows down things overall.
skipping this test,
speedup is from 13.5 seconds to 9.5 seconds -
28% faster - and this is something we should mention in the changelog :)
(this pr does not remove or change the slow test.
i think, due to the number of cores, this is not needed until someone
has a machine where the other tests run in 2 seconds or so)
Latest versions 0.0.248 and 0.0.249 report false positive:
src/deltachat_rpc_client/client.py:105:9: RET503 [*] Missing explicit `return` at the end of function able to return
.call() interface is safer because it ensures
that blocking operations on SQL connection
are called within tokio::task::block_in_place().
Previously some code called blocking operations
in async context, e.g. add_parts() in receive_imf module.
The underlying implementation of .call()
can later be replaced with an implementation
that does not require block_in_place(),
e.g. a worker pool,
without changing the code using the .call() interface.
IIRC, this was written this way back when we didn't have config caching,
in order to save database accesses by getting the config outside the
for loop.
Gmail archives messages marked as `\Deleted` by default if those messages aren't in the Trash. But
if move them to the Trash instead, they will be auto-deleted in 30 days.
With the introduction of transactions in Contact::add_or_lookup(),
python tests sometimes fail to create contacts with the folowing error:
Cannot create contact: add_or_lookup: database is locked: Error code 5: The database file is locked
`PRAGMA busy_timeout=60000` does not affect
this case as the error is returned before 60 seconds pass.
DEFERRED transactions with write operations need
to be retried from scratch
if they are rolled back
due to a write operation on another connection.
Using IMMEDIATE transactions for writing
is an attempt to fix this problem
without a retry loop.
If we later need DEFERRED transactions,
e.g. for reading a snapshot without locking the database,
we may introduce another function for this.
When connection pool is organized as a stack,
it always returns most recently used connection.
Because each connection has its own page cache,
using the connection with fresh cache improves performance.
I commented out `oauth2::tests::test_oauth_from_mx`
because it requires network connection,
turned off the Wi-Fi and ran the tests.
Before the change, with a queue:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 56.424 s ± 4.515 s [User: 183.181 s, System: 128.156 s]
Range (min … max): 52.123 s … 68.193 s 10 runs
```
With a stack:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 29.887 s ± 1.377 s [User: 101.226 s, System: 45.573 s]
Range (min … max): 26.591 s … 31.010 s 10 runs
```
On version 1.107.1:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
Time (mean ± σ): 43.658 s ± 1.079 s [User: 202.582 s, System: 50.723 s]
Range (min … max): 41.531 s … 45.170 s 10 runs
```
This also removes BackupProvider::join in favour of implementing
Future directly. I wondered about implementing a FusedFutre to make
this a little safer but it would introduce a dependency on the futures
crate in deltachat-ffi which did not exist yet, so I didn't do that.
the axum update broke the websocket server, because yerpc still uses a the old 5.9 version.
So I needed to downgrade the dependency until https://github.com/deltachat/yerpc/pull/35 is merged.
I want to retry the kaiOS client experiment, for this I need a working websocket server binary.
We no longer need that in the transfer case, that would give very
weird results. This also means there is nothing imex-specific about
this anymore so move it to blobs.rs
In execute_migration transaction first update the version, and only
then execute the rest of the migration. This ensures that transaction
is immediately recognized as write transaction and there is no need
to promote it from read transaction to write transaction later, e.g.
in the case of "DROP TABLE IF EXISTS" that is a read only operation if
the table does not exist. Promoting a read transaction to write
transaction may result in an error if database is locked.
* jsonrpc: `get_messages` now returns a map with `MessageLoadResult`
instead of failing completely if one of the requested messages could not be loaded.
* add pr number to changelog
* format errors with causes instead of debug output
also for chatlistitemfetchresult
get_chat_msgs() function is split into new get_chat_msgs() without flags
and get_chat_msgs_ex() which accepts booleans instead of bitflags.
FFI call dc_get_chat_msgs() is still using bitflags for compatibility.
JSON-RPC calls get_message_ids() and get_message_list_items()
accept booleans instead of bitflags now.
This adds functionality to send and receive a backup over the network
using a QR code.
The sender or provider prepares the backup, sets up a server that
waits for clients. It provides a ticket in the form of a QR code
which contains connection and authentication information.
The receiver uses the QR code to connect to the provider and fetches
backup, restoring it locally.
actions-rs/clippy-check runs clippy with --message-format=json option
and converts the output into annotations.
This makes clippy output unreadable, it is JSON
and you cannot quickly find the line number to fix.
Annotations in the code review view look nice,
but on large PRs they are less usable because you need
to scroll the whole page to find all the annotations.
There was the following bug:
- Bob has two devices, the second is offline.
- Alice creates a verified group and sends a QR invitation to Bob.
- Bob joins the group.
- Bob's second devices goes online, but sees a contact request instead of the verified group.
- The "member added" message is not a system message but a plain text message.
- Sending a message fails as the key is missing -- message info says "proper enc-key for <Alice>
missing, cannot encrypt".
This fixes the case with multiple devices on the joining side: if we observe such a message from
another device, we mark an inviter as verified and an accepted contact thus causing a subsequent
"vg-member-added" message -- in case of a verified group -- to create it properly.
DNS cache is used as a fallback if TCP connection
to all IP addresses returned by DNS failed.
If strict TLS checks are disabled,
DNS cache results are stored, but not used.
GitHub Pull Request: <https://github.com/deltachat/deltachat-core-rust/pull/3970>
This bug was introduced by 3cf78749df - there are three visibility
states: Archived, Pinned and Normal - in the database, this "visibility" is named historically
"archived" ... the original code has an AND archived=1 therefore.
It seems .abort() does not work on the recently seen loop
in some cases, e.g. if it is busy looping in a separate thread.
In my case after account reconfiguration recently seen loop
kept running and issuing warnings about closed interrupt channel.
Exit from recently seen loop on errors to avoid using 100% CPU
in such cases.
Derive Debug, PartialEq and Eq for Peerstate,
so `verifier` is included in Debug output and compared.
Store verifier as empty string
instead of NULL in the database.
- Return Result from set_verified() so that it can't be missed.
- Pass Fingerprint to set_verified() by value to avoid cloning it inside. This optimises out an
extra clone() if we already have a value that can be moved at the caller side. However, this may
add an extra clone() if set_verified() fails, but let's not optimise the fail scenario.
- Alice has two devices, the second is offline.
- Alice creates a verified group and sends a QR invitation to Bob.
- Bob joins the group and sends a message there. Alice sees it.
- Alice's second devices goes online, but doesn't see Bob in the group.
- If bcc_self is set, gossip headers must be added despite of the number of group members.
- If another device observes Secure-Join, instead of looking for Secure-Join-Fingerprint in
"vg-member-added"/"vc-contact-confirm" messages it must use keys from Autocrypt-Gossip headers as
described in the Countermitm doc
(https://countermitm.readthedocs.io/en/latest/new.html#joining-a-verified-group-secure-join).
This setting is true by default and causes
Windows build to cancel when Linux fails
due to flaky test and vice versa.
Cancelled test then has to be restarted
from scratch even though it was not going
to fail.
This allows to distinguish exceptions,
such as database errors, from invalid user input.
For example, if the From: field of the message
does not look like an email address, the mail
should be ignored. But if there is a database
failure while writing a new contact for the address,
this error should be bubbled up.
* Print chats after a test failed again
E.g.
```
========== Chats of bob: ==========
Single#Chat#10: alice@example.org [alice@example.org]
--------------------------------------------------------------------------------
Msg#10: (Contact#Contact#10): hellooo [FRESH]
Msg#11: (Contact#Contact#10): hellooo without mailing list [FRESH]
--------------------------------------------------------------------------------
========== Chats of alice: ==========
Single#Chat#10: bob@example.net [bob@example.net]
--------------------------------------------------------------------------------
Msg#10: Me (Contact#Contact#Self): hellooo √
Msg#11: Me (Contact#Contact#Self): hellooo without mailing list √
--------------------------------------------------------------------------------
```
I found this very useful sometimes, so, let's try to re-introduce it (it
was removed in #3449)
* Add failing test for TestContext::drop()
* Do not panic in TestContext::drop() if runtime is dropped
Co-authored-by: link2xt <link2xt@testrun.org>
this is a followup to #3918
I went with option "C" from my comment:
https://github.com/deltachat/deltachat-core-rust/pull/3918#issuecomment-1371224339
- Archive link is (still) very different from a normal chat, so most of the options would be empty / not relevant
- avatar path is not needed as desktop renders it differently anyway,
it's not really a chat like saved messages or device messages where it made more sense
for the core to supply the icon, vs. using the svg directly.
- translating the string in the coreas stock-string is more annoying than doing it from the ui, especially when
this special pseudo chat has different rendering anyway so also no need to provide a name property
* let marknoticed_chat() work for DC_CHAT_ID_ARCHIVED_LINK
* fix test_util::get_last_msg() - the first position may be the archive-link if 'adding specials' is allowed
* add a test for the archived-link message counter
* update CHANGELOG
* move 'archived link' betweeen pinned and normal cahts or above normal chats
* add icon for 'archived chats' link
* let get_fresh_msg_cnt() work for DC_CHAT_ID_ARCHIVED_LINK
* move 'archived link' topmost
* use less noticeable archived-icon
* slightly smaller archived icon
* update CHANGELOG
There are at least two user reports that fetching existing messages
sometimes results in infinite loop of retrying it. Account is working
if set up from the backup, but never starts working if set up
from scratch.
This change improves error reporting, but also sets FetchedExistingMsgs
before actually trying to do it. This way if the operation fails,
connection is reestablished, but fetching existing messages is not
retried again over and over.
async-imap does not do its own buffering, but calls flush() after
sending each command. Using BufWriter reduces the number of write()
system calls used to send a single command.
Note that BufWriter is set up on top of TLS streams, because
we can't guarantee that TLS libraries flush the stream before
waiting for response.
This makes it possible to fuzz test the functions
without exposing the module interface in the deltachat core
interface.
Also ensure that format_flowed will not grow a dependency
on deltachat core types.
If we move the detached signatures validation code out of try_decrypt(), we don't need to convert an
already parsed signed message part to Vec and then parse it back. Also this simplifies the
try_decrypt() semantics and return type. It can't make a good coffee anyway.
Bumping MSRV from 1.61.0 to 1.63.0, because `arbitrary` crate requires
it and fuzzing crates depend on it, at least by default. We still use
1.64.0 as our default rust toolchain.
Services like Lacre [1] on Disroot and Inbound Encryption on Posteo [2]
offer to encrypt all incoming messages with the provided OpenPGP
public key. Resulting messages are encrypted, but not end-to-end encrypted
and not signed by the sender, therefore should not have a padlock displayed.
However, such encrypted and unsigned message is also not an indication
of an error on ongoing attack, so we shoud not report this as a problem
to the user.
[1] https://lacre.io/
[2] https://posteo.de/en/help/how-do-i-activate-inbound-encryption-with-my-public-pgp-key
This way we don't need a separate code path for signatures validation for unencrypted
messages. Also, now we degrade encryption only if there are no valid signatures, so the code for
upgrading encryption back isn't needed.
Note that if the message is encrypted, we don't check whether it's signed with an attached key
currently, otherwise a massive refactoring of the code is needed because for encrypted messages a
signature is checked and discarded first now.
* do not `SELECT *` on old tables to fill new ones
the old table may contain deprecrated columns for whatever reason;
as a result the query fails as the statement tries to insert eg.
16 columns into 12 colums
(concrete error for acpeerstate that have several deprecated columns)
* update CHANGELOG
IMAP capabilities and selected folder are IMAP session,
not IMAP client property.
Moving most operations into IMAP session structure
removes the need to constantly check whether IMAP session exists
and reduces number of invalid states, e.g. when a folder is selected but
there is no connection.
Capabilities are determined immediately after logging in,
so there is no need for `capabilities_determined` flag anymore.
Capabilities of the server are always known if there is a session.
`should_reconnect` flag and `disconnect()` function are removed: we
drop the session on error. Even though RFC 3501 says that a client
SHOULD NOT close the connection without a LOGOUT, it is more reliable
to always just drop the connection, especially after an error.
* Because both only make problems with mailing lists, it's easiest to just disable them. If we want, we can make them work properly with mailing lists one day and re-enable them, but this needs some further thoughts.
Part of #3701
* Use load_from_db() in more tests
* clippy
* Changelog
* Downgrade warning to info, improve message
* Use lifetimes instead of cloning
When a batch of messages is moved from Inbox to DeltaChat folder with a single MOVE command, their
UIDs may be reordered (e.g. Gmail is known for that) which leads to that messages are processed by
receive_imf in the wrong order. But the INTERNALDATE attribute is preserved during a MOVE according
to RFC3501. So, use it for sorting fetched messages.
Both reversed and original order do not make much sense
for the bot. Ideally bots should have their own key
to get the list of fresh messages in the order of IDs.
This esp. speeds up receive_imf a bit when we recreate the member list (recreate_member_list == true).
It's a preparation for https://github.com/deltachat/deltachat-core-rust/issues/3768, which will be a one-line-fix, but recreate the member list more often, so that we want to optimize this case a bit.
It also adds a benchmark for this case. It's not that easy to make the benchmark non-flaky, but by closing all other programs and locking the CPU to 1.5GHz it worked. It is consistently a few percent faster than ./without-optim:
```
Receive messages/Receive 100 Chat-Group-Member-{Added|Removed} messages
time: [52.257 ms 52.569 ms 52.941 ms]
change: [-3.5301% -2.6181% -1.6697%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
```
It's a w/a for "Space added before long group names after MIME serialization/deserialization"
issue. DC itself never creates group names with leading/trailing whitespace, so it can be safely
removed. On the sender side there's no trim() because group names anyway go through
improve_single_line_input(). And I believe we should send the exact name we have in our db. Also
there's no check for leading/trailing whitespace because there may be existing user databases with
group names having such whitespaces.
Since switch to async we don't have spurious "database is busy"
errors anymore. Since an error is irrecoverable in most cases,
we can skip the message. The cost of this is we may
accidentally skip a correct message if I/O fails, but
the advantage is that we are guaranteed to never confuse
irrecoverable error with recoverable one and get stuck in
infinite loop redownloading the same message over and over.
That's a bug which @Simon-Laux and probably also @hpk42 had, where one malformed incoming (Spam-) mail blocked the receiving of all emails coming after it.
The problem was that from_field_to_contact_id() returned ContactId::UNDEFINED, and then lookup_by_contact() returned Err.
* Treat multiple From addresses as if there was no From: addr
* changelog
* Don't send invalid emails through the whole receive_imf pipeline
Instead, directly create a trash entry for them.
* Don't create trash entries for randomly generated Message-Id's
* clippy
* fix typo
Co-authored-by: link2xt <link2xt@testrun.org>
- over jsonrpc built with napi.rs: \[[📂 source](https://github.com/deltachat/napi-jsonrpc) | [📦 npm](https://www.npmjs.com/package/@deltachat/napi-jsonrpc)\]
[^1]: Out of date / unmaintained, if you like those languages feel free to start maintaining them. If you have questions we'll help you, please ask in the issues.
We format the code using `rustfmt`. Run `cargo fmt` prior to committing the code.
Run `scripts/clippy.sh` to check the code for common mistakes with [Clippy].
[Clippy]: https://doc.rust-lang.org/clippy/
## SQL
Multi-line SQL statements should be formatted using string literals,
for example
```
sql.execute(
"CREATE TABLE messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
text TEXT DEFAULT '' NOT NULL -- message text
) STRICT",
)
.await?;
```
Do not use macros like [`concat!`](https://doc.rust-lang.org/std/macro.concat.html)
or [`indoc!](https://docs.rs/indoc).
Do not escape newlines like this:
```
sql.execute(
"CREATE TABLE messages ( \
id INTEGER PRIMARY KEY AUTOINCREMENT, \
text TEXT DEFAULT '' NOT NULL \
) STRICT",
)
.await?;
```
Escaping newlines
is prone to errors like this if space before backslash is missing:
```
"SELECT foo\
FROM bar"
```
Literal above results in `SELECT fooFROM bar` string.
This style also does not allow using `--` comments.
---
Declare new SQL tables with [`STRICT`](https://sqlite.org/stricttables.html) keyword
to make SQLite check column types.
Declare primary keys with [`AUTOINCREMENT`](https://www.sqlite.org/autoinc.html) keyword.
This avoids reuse of the row IDs and can avoid dangerous bugs
like forwarding wrong message because the message was deleted
and another message took its row ID.
Declare all new columns as `NOT NULL`
and set the `DEFAULT` value if it is optional so the column can be skipped in `INSERT` statements.
Dealing with `NULL` values both in SQL and in Rust is tricky and we try to avoid it.
If column is already declared without `NOT NULL`, use `IFNULL` function to provide default value when selecting it.
Use `HAVING COUNT(*) > 0` clause
to [prevent aggregate functions such as `MIN` and `MAX` from returning `NULL`](https://stackoverflow.com/questions/66527856/aggregate-functions-max-etc-return-null-instead-of-no-rows).
Don't delete unused columns too early, but maybe after several months/releases, unused columns are
still used by older versions, so deleting them breaks downgrading the core or importing a backup in
an older version. Also don't change the column type, consider adding a new column with another name
instead. Finally, never change column semantics, this is especially dangerous because the `STRICT`
style="fill:#8c8c8c;fill-opacity:1;stroke:none;stroke-width:680.523;stroke-dasharray:none;paint-order:markers fill stroke"
id="rect1"
width="9951.9541"
height="9767.4756"
x="-71.697792"
y="-1012.83"
ry="0.43547946"/>
<path
d="m 2948.0033,5553.6941 q -130.7292,0 -228.7761,-96.3953 -98.0468,-96.3953 -98.0468,-224.9223 V 2447.6234 q 0,-128.527 98.0468,-224.9223 98.0469,-96.3953 228.7761,-96.3953 h 3703.9934 q 130.7292,0 228.776,96.3953 98.0469,96.3953 98.0469,224.9223 v 2784.7531 q 0,128.527 -98.0469,224.9223 -98.0468,96.3953 -228.776,96.3953 z M 4800,3936.3952 2948.0033,2742.1646 V 5232.3765 H 6651.9967 V 2742.1646 Z m 0,-321.3176 1830.2085,-1167.4541 h -3654.97 z m -1851.9967,-872.913 v -294.5412 2784.7531 z"
@@ -4,46 +4,16 @@ This crate provides a [JSON-RPC 2.0](https://www.jsonrpc.org/specification) inte
The JSON-RPC API is exposed in two fashions:
* A executable that exposes the JSON-RPC API through a [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) server running on localhost.
* The JSON-RPC API can also be called through the [C FFI](../deltachat-ffi). The C FFI needs to be built with the `jsonrpc` feature. It will then expose the functions `dc_jsonrpc_init`, `dc_jsonrpc_request`, `dc_jsonrpc_next_response` and `dc_jsonrpc_unref`. See the docs in the [header file](../deltachat-ffi/deltachat.h) for details.
* A executable`deltachat-rpc-server` that exposes the JSON-RPC API through stdio.
* The JSON-RPC API can also be called through the [C FFI](../deltachat-ffi). It exposes the functions `dc_jsonrpc_init`, `dc_jsonrpc_request`, `dc_jsonrpc_next_response` and `dc_jsonrpc_unref`. See the docs in the [header file](../deltachat-ffi/deltachat.h) for details.
We also include a JavaScript and TypeScript client for the JSON-RPC API. The source for this is in the [`typescript`](typescript) folder. The client can easily be used with the WebSocket server to build DeltaChat apps for web browsers or Node.js. See the [examples](typescript/example) for details.
We also include a JavaScript and TypeScript client for the JSON-RPC API. The source for this is in the [`typescript`](typescript) folder.
## Usage
#### Running the WebSocket server
From within this folder, you can start the WebSocket server with the following command:
```sh
cargo run --features webserver
```
If you want to use the server in a production setup, first build it in release mode:
```sh
cargo build --features webserver --release
```
You will then find the `deltachat-jsonrpc-server` executable in your `target/release` folder.
The executable currently does not support any command-line arguments. By default, once started it will accept WebSocket connections on `ws://localhost:20808/ws`. It will store the persistent configuration and databases in a `./accounts` folder relative to the directory from where it is started.
The server can be configured with environment variables:
|variable|default|description|
|-|-|-|
|`DC_PORT`|`20808`|port to listen on|
|`DC_ACCOUNTS_PATH`|`./accounts`|path to storage directory|
If you are targetting other architectures (like KaiOS or Android), the webserver binary can be cross-compiled easily with [rust-cross](https://github.com/cross-rs/cross):
The package includes a JavaScript/TypeScript client which is partially auto-generated through the JSON-RPC library used by this crate ([yerpc](https://github.com/Frando/yerpc/)). Find the source in the [`typescript`](typescript) folder.
The package includes a JavaScript/TypeScript client which is partially auto-generated through the JSON-RPC library used by this crate ([yerpc](https://github.com/chatmail/yerpc)). Find the source in the [`typescript`](typescript) folder.
To use it locally, first install the dependencies and compile the TypeScript code to JavaScript:
```sh
@@ -52,15 +22,7 @@ npm install
npm run build
```
The JavaScript client is not yet published on NPM (but will likely be soon). Currently, it is recommended to vendor the bundled build. After running `npm run build` as documented above, there will be a file `dist/deltachat.bundle.js`. This is an ESM module containing all dependencies. Copy this file to your project and import the DeltaChat class.
```typescript
import{DeltaChat}from'./deltachat.bundle.js'
constdc=newDeltaChat('ws://localhost:20808/ws')
constaccounts=awaitdc.rpc.getAllAccounts()
console.log('accounts',accounts)
```
The JavaScript client is [published on NPM](https://www.npmjs.com/package/@deltachat/jsonrpc-client).
A script is included to build autogenerated documentation, which includes all RPC methods:
```sh
@@ -73,18 +35,6 @@ Then open the [`typescript/docs`](typescript/docs) folder in a web browser.
#### Running the example app
We include a small demo web application that talks to the WebSocket server. It can be used for testing. Feel invited to expand this.
```sh
cd typescript
npm run build
npm run example:build
npm run example:start
```
Then, open [`http://localhost:8080/example.html`](http://localhost:8080/example.html) in a web browser.
Run `npm run example:dev` to live-rebuild the example app when files changes.
### Testing
The crate includes both a basic Rust smoke test and more featureful integration tests that use the TypeScript client.
@@ -104,14 +54,12 @@ cd typescript
npm run test
```
This will build the `deltachat-jsonrpc-server` binary and then run a test suite against the WebSocket server.
This will build the `deltachat-jsonrpc-server` binary and then run a test suite.
The test suite includes some tests that need online connectivity and a way to create test email accounts. To run these tests, talk to DeltaChat developers to get a token for the `testrun.org` service, or use a local instance of [`mailadm`](https://github.com/deltachat/docker-mailadm).
Then, set the `DCC_NEW_TMP_EMAIL` environment variable to your mailadm token before running the tests.
The test suite includes some tests that need online connectivity and a way to create test email accounts. To run these tests, set the `CHATMAIL_DOMAIN` environment variable to your testing email server domain.
```
DCC_NEW_TMP_EMAIL=https://testrun.org/new_email?t=yourtoken npm run test
CHATMAIL_DOMAIN=ci-chatmail.testrun.org npm run test
- [ ] different test type to simulate two devices: to test autocrypt_initiate_key_transfer & autocrypt_continue_key_transfer
## MVP - Websocket server&client
For kaiOS and other experiments, like a deltachat "web" over network from an android phone.
- [ ] coverage for a majority of the API
- [ ] Blobs served
- [ ] Blob upload (for attachments, setting profile-picture, importing backup and so on)
- [ ] other way blobs can be addressed when using websocket vs. jsonrpc over dc-node
- [ ] Web push API? At least some kind of notification hook closure this lib can accept.
### Other Ideas for the Websocket server
- [ ] make sure there can only be one connection at a time to the ws
- why? , it could give problems if its commanded from multiple connections
- [ ] encrypted connection?
- [ ] authenticated connection?
- [ ] Look into unit-testing for the proc macros?
- [ ] proc macro taking over doc comments to generated typescript file
## Desktop Apis
Incomplete todo for desktop api porting, just some remainders for points that might need more work:
- [ ] manual start/stop io functions in the api for context and accounts, so "not syncing all accounts" can still be done in desktop -> webserver should then not do start io on all accounts by default
/// The library-user may write an informational string to the log.
///
/// This event should *not* be reported to the end-user using a popup or something like
/// that.
Info{msg: String},
/// Emitted when SMTP connection is established and login was successful.
SmtpConnected{msg: String},
/// Emitted when IMAP connection is established and login was successful.
ImapConnected{msg: String},
/// Emitted when a message was successfully sent to the SMTP server.
SmtpMessageSent{msg: String},
/// Emitted when an IMAP message has been marked as deleted
ImapMessageDeleted{msg: String},
/// Emitted when an IMAP message has been moved
ImapMessageMoved{msg: String},
/// Emitted before going into IDLE on the Inbox folder.
ImapInboxIdle,
/// Emitted when an new file in the $BLOBDIR was created
NewBlobFile{file: String},
/// Emitted when an file in the $BLOBDIR was deleted
DeletedBlobFile{file: String},
/// The library-user should write a warning string to the log.
///
/// This event should *not* be reported to the end-user using a popup or something like
/// that.
Warning{msg: String},
/// The library-user should report an error to the end-user.
///
/// As most things are asynchronous, things may go wrong at any time and the user
/// should not be disturbed by a dialog or so. Instead, use a bubble or so.
///
/// However, for ongoing processes (eg. configure())
/// or for functions that are expected to fail (eg. autocryptContinueKeyTransfer())
/// it might be better to delay showing these events until the function has really
/// failed (returned false). It should be sufficient to report only the *last* error
/// in a message box then.
Error{msg: String},
/// An action cannot be performed because the user is not in the group.
/// Reported eg. after a call to
/// setChatName(), setChatProfileImage(),
/// addContactToChat(), removeContactFromChat(),
/// and messages sending functions.
ErrorSelfNotInGroup{msg: String},
/// Messages or chats changed. One or more messages or chats changed for various
/// reasons in the database:
/// - Messages sent, received or removed
/// - Chats created, deleted or archived
/// - A draft has been set
#[serde(rename_all = "camelCase")]
MsgsChanged{
/// Set if only a single chat is affected by the changes, otherwise 0.
chat_id: u32,
/// Set if only a single message is affected by the changes, otherwise 0.
msg_id: u32,
},
/// Reactions for the message changed.
#[serde(rename_all = "camelCase")]
ReactionsChanged{
/// ID of the chat which the message belongs to.
chat_id: u32,
/// ID of the message for which reactions were changed.
msg_id: u32,
/// ID of the contact whose reaction set is changed.
contact_id: u32,
},
/// A reaction to one's own sent message received.
/// Typically, the UI will show a notification for that.
///
/// In addition to this event, ReactionsChanged is emitted.
#[serde(rename_all = "camelCase")]
IncomingReaction{
/// ID of the chat which the message belongs to.
chat_id: u32,
/// ID of the contact whose reaction set is changed.
contact_id: u32,
/// ID of the message for which reactions were changed.
msg_id: u32,
/// The reaction.
reaction: String,
},
/// Incoming webxdc info or summary update, should be notified.
#[serde(rename_all = "camelCase")]
IncomingWebxdcNotify{
/// ID of the chat.
chat_id: u32,
/// ID of the contact sending.
contact_id: u32,
/// ID of the added info message or webxdc instance in case of summary change.
msg_id: u32,
/// Text to notify.
text: String,
/// Link assigned to this notification, if any.
href: Option<String>,
},
/// There is a fresh message. Typically, the user will show a notification
/// when receiving this message.
///
/// There is no extra #DC_EVENT_MSGS_CHANGED event sent together with this event.
#[serde(rename_all = "camelCase")]
IncomingMsg{
/// ID of the chat where the message is assigned.
chat_id: u32,
/// ID of the message.
msg_id: u32,
},
/// Downloading a bunch of messages just finished. This is an
/// event to allow the UI to only show one notification per message bunch,
/// instead of cluttering the user with many notifications.
#[serde(rename_all = "camelCase")]
IncomingMsgBunch,
/// Messages were seen or noticed.
/// chat id is always set.
#[serde(rename_all = "camelCase")]
MsgsNoticed{chat_id: u32},
/// A single message is sent successfully. State changed from DC_STATE_OUT_PENDING to
/// DC_STATE_OUT_DELIVERED, see `Message.state`.
#[serde(rename_all = "camelCase")]
MsgDelivered{
/// ID of the chat which the message belongs to.
chat_id: u32,
/// ID of the message that was successfully sent.
msg_id: u32,
},
/// A single message could not be sent. State changed from DC_STATE_OUT_PENDING or DC_STATE_OUT_DELIVERED to
/// DC_STATE_OUT_FAILED, see `Message.state`.
#[serde(rename_all = "camelCase")]
MsgFailed{
/// ID of the chat which the message belongs to.
chat_id: u32,
/// ID of the message that could not be sent.
msg_id: u32,
},
/// A single message is read by the receiver. State changed from DC_STATE_OUT_DELIVERED to
/// DC_STATE_OUT_MDN_RCVD, see `Message.state`.
#[serde(rename_all = "camelCase")]
MsgRead{
/// ID of the chat which the message belongs to.
chat_id: u32,
/// ID of the message that was read.
msg_id: u32,
},
/// A single message was deleted.
///
/// This event means that the message will no longer appear in the messagelist.
/// UI should remove the message from the messagelist
/// in response to this event if the message is currently displayed.
///
/// The message may have been explicitly deleted by the user or expired.
/// Internally the message may have been removed from the database,
/// moved to the trash chat or hidden.
///
/// This event does not indicate the message
/// deletion from the server.
#[serde(rename_all = "camelCase")]
MsgDeleted{
/// ID of the chat where the message was prior to deletion.
/// Never 0.
chat_id: u32,
/// ID of the deleted message. Never 0.
msg_id: u32,
},
/// Chat changed. The name or the image of a chat group was changed or members were added or removed.
/// See setChatName(), setChatProfileImage(), addContactToChat()
/// and removeContactFromChat().
///
/// This event does not include ephemeral timer modification, which
/// is a separate event.
#[serde(rename_all = "camelCase")]
ChatModified{chat_id: u32},
/// Chat ephemeral timer changed.
#[serde(rename_all = "camelCase")]
ChatEphemeralTimerModified{
/// Chat ID.
chat_id: u32,
/// New ephemeral timer value.
timer: u32,
},
/// Chat deleted.
ChatDeleted{
/// Chat ID.
chat_id: u32,
},
/// Contact(s) created, renamed, blocked or deleted.
#[serde(rename_all = "camelCase")]
ContactsChanged{
/// If set, this is the contact_id of an added contact that should be selected.
contact_id: Option<u32>,
},
/// Location of one or more contact has changed.
#[serde(rename_all = "camelCase")]
LocationChanged{
/// contact_id of the contact for which the location has changed.
/// If the locations of several contacts have been changed,
/// this parameter is set to `None`.
contact_id: Option<u32>,
},
/// Inform about the configuration progress started by configure().
ConfigureProgress{
/// Progress.
///
/// 0=error, 1-999=progress in permille, 1000=success and done
progress: usize,
/// Progress comment or error, something to display to the user.
comment: Option<String>,
},
/// Inform about the import/export progress started by imex().
///
#[serde(rename_all = "camelCase")]
ImexProgress{
/// 0=error, 1-999=progress in permille, 1000=success and done
progress: usize,
},
/// A file has been exported. A file has been written by imex().
/// This event may be sent multiple times by a single call to imex().
///
/// A typical purpose for a handler of this event may be to make the file public to some system
/// services.
///
/// @param data2 0
#[serde(rename_all = "camelCase")]
ImexFileWritten{path: String},
/// Progress event sent when SecureJoin protocol has finished
/// from the view of the inviter (Alice, the person who shows the QR code).
///
/// These events are typically sent after a joiner has scanned the QR code
/// generated by getChatSecurejoinQrCodeSvg().
#[serde(rename_all = "camelCase")]
SecurejoinInviterProgress{
/// ID of the contact that wants to join.
contact_id: u32,
/// The type of the joined chat.
/// This can take the same values
/// as `BasicChat.chatType` ([`crate::api::types::chat::BasicChat::chat_type`]).
chat_type: u32,
/// ID of the chat in case of success.
chat_id: u32,
/// Progress, always 1000.
progress: usize,
},
/// Progress information of a secure-join handshake from the view of the joiner
/// (Bob, the person who scans the QR code).
/// The events are typically sent while secureJoin(), which
/// may take some time, is executed.
#[serde(rename_all = "camelCase")]
SecurejoinJoinerProgress{
/// ID of the inviting contact.
contact_id: u32,
/// Progress as:
/// 400=vg-/vc-request-with-auth sent, typically shown as "alice@addr verified, introducing myself."
/// (Bob has verified alice and waits until Alice does the same for him)
/// 1000=vg-member-added/vc-contact-confirm received
progress: usize,
},
/// The connectivity to the server changed.
/// This means that you should refresh the connectivity view
/// and possibly the connectivtiy HTML; see getConnectivity() and
/// getConnectivityHtml() for details.
ConnectivityChanged,
/// Deprecated by `ConfigSynced`.
SelfavatarChanged,
/// A multi-device synced config value changed. Maybe the app needs to refresh smth. For
/// uniformity this is emitted on the source device too. The value isn't here, otherwise it
/// would be logged which might not be good for privacy.
ConfigSynced{
/// Configuration key.
key: String,
},
#[serde(rename_all = "camelCase")]
WebxdcStatusUpdate{
/// Message ID.
msg_id: u32,
/// Status update ID.
status_update_serial: u32,
},
/// Data received over an ephemeral peer channel.
#[serde(rename_all = "camelCase")]
WebxdcRealtimeData{
/// Message ID.
msg_id: u32,
/// Realtime data.
data: Vec<u8>,
},
/// Advertisement received over an ephemeral peer channel.
/// This can be used by bots to initiate peer-to-peer communication from their side.
#[serde(rename_all = "camelCase")]
WebxdcRealtimeAdvertisementReceived{
/// Message ID of the webxdc instance.
msg_id: u32,
},
/// Inform that a message containing a webxdc instance has been deleted
#[serde(rename_all = "camelCase")]
WebxdcInstanceDeleted{
/// ID of the deleted message.
msg_id: u32,
},
/// Tells that the Background fetch was completed (or timed out).
/// This event acts as a marker, when you reach this event you can be sure
/// that all events emitted during the background fetch were processed.
///
/// This event is only emitted by the account manager
AccountsBackgroundFetchDone,
/// Inform that set of chats or the order of the chats in the chatlist has changed.
///
/// Sometimes this is emitted together with `UIChatlistItemChanged`.
ChatlistChanged,
/// Inform that a single chat list item changed and needs to be rerendered.
/// If `chat_id` is set to None, then all currently visible chats need to be rerendered, and all not-visible items need to be cleared from cache if the UI has a cache.
#[serde(rename_all = "camelCase")]
ChatlistItemChanged{
/// ID of the changed chat
chat_id: Option<u32>,
},
/// Inform that the list of accounts has changed (an account removed or added or (not yet implemented) the account order changes)
///
/// This event is only emitted by the account manager
AccountsChanged,
/// Inform that an account property that might be shown in the account list changed, namely:
/// - is_configured (see is_configured())
/// - displayname
/// - selfavatar
/// - private_tag
///
/// This event is emitted from the account whose property changed.
AccountsItemChanged,
/// Inform than some events have been skipped due to event channel overflow.
EventChannelOverflow{
/// Number of events skipped.
n: u64,
},
/// Incoming call.
IncomingCall{
/// ID of the info message referring to the call.
msg_id: u32,
/// ID of the chat which the message belongs to.
chat_id: u32,
/// User-defined info as passed to place_outgoing_call()
place_call_info: String,
/// True if incoming call is a video call.
has_video: bool,
},
/// Incoming call accepted.
/// This is esp. interesting to stop ringing on other devices.
IncomingCallAccepted{
/// ID of the info message referring to the call.
msg_id: u32,
/// ID of the chat which the message belongs to.
chat_id: u32,
},
/// Outgoing call accepted.
OutgoingCallAccepted{
/// ID of the info message referring to the call.
msg_id: u32,
/// ID of the chat which the message belongs to.
chat_id: u32,
/// User-defined info passed to dc_accept_incoming_call(
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
