This is to avoid revalidating HTTP cache too frequently (and have many parallel revalidation tasks)
if revalidation fails or the HTTP request takes some time. The stale period >= 1 hour, so 1 more
minute won't be a problem.
It was called from `receive_imf` when an outgoing message is received. But
`Imap::fetch_new_messages()` already calls `chat::mark_old_messages_as_noticed()` which does the job
better (per-message).
SQL statements fail if the number of variables
exceeds `SQLITE_LIMIT_VARIABLE_NUMBER`.
Remaining repeat_vars() calls are difficult to replace
and use arrays passed from the UI,
e.g. forwarded message IDs or marked as seen IDs.
UIs may want to display smth like "Transferring..." after "Establishing connection between
devices..." on nonzero progress. Before, progress on the receiver side was starting with 2 after
receiving enough data.
It only contained proxy
that some users ran in Termux
to look at IMAP traffic.
The same can be achieved with `socat`, e.g.:
socat -v TCP-LISTEN:9999,bind=127.0.0.1 OPENSSL:nine.testrun.org:993
This fixes the HTML display of messages containing forwarded messages. Before, forwarded messages
weren't rendered in HTML and if a forwarded message is long and therefore truncated in the chat, it
could only be seen in the "Message Info". In #4462 it was suggested to display "Show Full
Message..." for each truncated message part and save to `msgs.mime_headers` only the corresponding
part, but this is a quite huge change and refactoring and also it may be good that currently we save
the full message structure to `msgs.mime_headers`, so i'd suggest not to change this for now.
Otherwise the "Show Full Message..." button appears somewhere in the middle of the multipart
message, e.g. after a text in the first message bubble, but before a text in the second
bubble. Moreover, if the second/n-th bubble's text is shortened (ends with "[...]"), the user should
scroll up to click on "Show Full Message..." which doesn't look reasonable. Scrolling down looks
more acceptable (e.g. if the first bubble's text is shortened in a multipart message).
I'd even suggest to show somehow that message bubbles belong to the same multipart message, e.g. add
"[↵]" to the text of all bubbles except the last one, but let's discuss this first.
without the prefix,
it looks as if it is part of the Message-ID,
esp. if Message-ID is longer,
a break on different delimiters may look exactly the same.
see #6329 for some screenshots that initially confused me :)
A user reported to me that after they left a group, they were implicitly readded, but there's no any
readdition message, so currently it looks in the chat like leaving it has no effect, just new
messages continue to arrive. The readdition probably happened because some member didn't receive the
user's self-removal message, anyway, at least there must be a message that the user is readded, even
if it isn't known by whom.
A NDN may arrive days after the message is sent when it's already impossible to tell which message
wasn't delivered looking at the "Failed to send" info message, so it only clutters the chat and
makes the user think they tried to send some message recently which isn't true. Moreover, the info
message duplicates the info already displayed in the error message behind the exclamation mark and
info messages do not point to the message that is failed to be sent.
Moreover it works rarely because `mimeparser.rs` only parses recipients from `x-failed-recipients`,
so it likely only works for Gmail. Postfix does not add this `X-Failed-Recipients` header. Let's
remove this parsing too. Thanks to @link2xt for pointing this out.
This PR:
- Moves the note about the false positive to the end of the test output,
where it is more likely to be noticed
- Also notes in test_modify_chat_disordered() and
test_setup_contact_*(), in addition to the existing note in
test_was_seen_recently()
This fixes HTML display of truncated (long) sent messages ("Show full message" in UIs). Before,
incorrect HTML was stored (with missing line breaks etc.) for them. Now stored plaintext is
formatted to HTML upon calling `MsgId::get_html()` and this results in the same HTML as on a
receiver side.
Also cleaned up test_connectivity()
which tested that state does not flicker to WORKING
when there are no messages to be fetched.
The state is expected to flicker to WORKING
when checking for new messages,
so the tests were outdated since
change 3b0b2379b8
The benchmark function (e.g. `recv_all_emails()`) is executed multiple
times on the same context. During the second iteration, all the emails
were already in the database, so, receiving them again failed.
This PR fixes that by passing in a second `iteration` counter that is
different for every invocation of the benchmark function.
This better reflects that this state means
we just connected and there may me work to do.
This state is converted to DC_CONNECTIVITY_WORKING
instead of DC_CONNECTIVITY_CONNECTED state now.
Before this change when IMAP connected
to the server, it switched
from DC_CONNECTIVITY_NOT_CONNECTED
to DC_CONNECTIVITY_CONNECTING,
then to DC_CONNECTIVITY_CONNECTED (actually preparing)
then to DC_CONNECTIVITY_WORKING
and then to DC_CONNECTIVITY_CONNECTED again (actually idle).
On fast connections this resulted in flickering "Connected"
string in the status bar right before "Updating..."
and on slow connections this "Connected" state
before "Updating..." lasted for a while
leaving the user to wonder if there are no new messages
or if Delta Chat will still switch to "Updating..."
before going into "Connected" state again.
Already apply rust beta (1.84) clippy suggestions now, before they let
CI fail in 6 weeks.
The newly used functions are available since 1.70, our MSRV is 1.77, so
we can use them.
A sync message for accepting or blocking a 1:1 chat may arrive before the first message from the
contact, when it does not exist yet. This frequently happens in non-chatmail accounts that have
moving to the DeltaChat folder disabled because Delta Chat unconditionally uploads sync messages to
the DeltaChat folder. Let's create a hidden contact in this case and a 1:1 chat for it.
this partly reverts experimental #3516
that allowed any .xdc sent to "Saved Messages" to request internet.
this helped on pushing map integration forward.
meanwhile, however, we have that map integration (#5461 and #5678),
that implies `info.internet_access` being set.
experimental `manifest.request_internet_access` is no longer needed therefore.
future will tell, if we revive the option at some point or
go for more intrations ('sending' is discussed often :) -
but currently it is not needed.
with this PR, when an `.xdc` with `request_integration = map` in the
manifest is added to the "Saved Messages" chat, it is used _locally_ as
an replacement for the shipped maps.xdc (other devices will see the
`.xdc` but not use it)
this allows easy development and adapting the map to use services that
work better in some area.
there are lots of known discussions and ideas about adding more barriers
of safety. however, after internal discussions, we decided to move
forward and also to allow internet, if requested by an integration (as
discussed at
https://github.com/deltachat/deltachat-core-rust/pull/3516).
the gist is to ease development and to make users who want to adapt,
actionable _now_, without making things too hard and adding too high
barriers or stressing our own resources/power too much.
note, that things are still experimental and will be the next time -
without the corresponding switch being enabled, nothing will work at
all, so we can be quite relaxed here :)
for android/ios, things will work directly. for desktop, allow_internet
needs to be accepted unconditionally from core. for the future, we might
add a question before using an integration and/or add signing. or sth.
completely different - but for now, the thing is to get started.
nb: "integration" field in the webxdc-info is experimental as well and
should not be used in UIs at all currently, it may vanish again and is
there mainly for simplicity of the code; therefore, no need to document
that.
successor of https://github.com/deltachat/deltachat-core-rust/pull/5461
this is how it looks like currently - again, please note that all that
is an experiment!
<img width=320
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/f659c891-f46a-4e28-9d0a-b6783d69be8d>
<img width=320
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/54549b3c-a894-4568-9e27-d5f1caea2d22>
... when going out of experimental, there are loots of ideas, eg.
changing "Start" to "integrate"
following `RELEASE.md`, after merging, the following is needed:
6. Tag the release: `git tag --annotate v1.151.2`.
7. Push the release tag: `git push origin v1.151.2`.
8. Create a GitHub release: `gh release create v1.151.2 --notes ''`.
this PR adds the `href` from `update.href` to the IncomingWebxdcNotify
event (DC_EVENT_INCOMING_WEBXDC_NOTIFY in cffi)
purpose is to add a "Start" button to the notifications that allow
starting the app immediately with the given href
using long options make things less mystical, it is clearer what
happens.
(apart from that i also disallowed `git -a` on my machine in general, as
`git commit -a` is considered harmful. as my approach to disallow that
is a bit greedy and disallows `-a` just for any git commands, this is
the only place where i regularly struggle :)
Text parts are using quoted-printable encoding
which takes care of wrapping long lines,
so using format=flowed is unnecessary.
This improves compatibility with receivers
which do not support format=flowed.
Receiving format=flowed messages is still possible, receiver side of
Delta Chat is unchanged.
- **feat: add `AccountsChanged` and `AccountsItemChanged` events**
- **emit event and add tests**
closes#6106
TODO:
- [x] test receiving synced config from second device
- [x] bug: investigate how to delay the configuration event until it is
actually configured - because desktop gets the event but still shows
account as if it was unconfigured, maybe event is emitted before the
value is written to the database?
- [x] update node bindings constants
this PR removes most usages of the `descr` parameter.
- to avoid noise in different branches etc. (as annoying on similar, at
a first glance simple changes), i left the external API stable
- also, the effort to do a database migration seems to be over the top,
so the column is left and set to empty strings on future updates - maybe
we can recycle the column at some point ;)
closes#6245
add `update.href` property option to update objects send via
`Context::send_webxdc_status_update()`.
when set together with `update.info`,
UI can implement the info message as a link that is passed to the webxdc
via `window.location.href`.
for that purpose, UI will read the link back from
`Message::get_webxdc_href()`.
Practically,
this allows e.g. an calendar.xdc
to emits clickable update messages
opening the calendar at the correct date.
closes#6219
documentation at https://github.com/webxdc/website/pull/90
it may be handy for an xdc to have only one list of all adresses, or
there may just be bugs.
in any case, do not notify SELF, e.g. in a multi-device setup; we're
also not doing this for other messages.
this is also a preparation for having an option to notify ALL.
I found the old documentation rather hard to understand. The new doc
string:
- uses whole sentences, leaving less space for misinterpretation
- explicitly mentions that it can happen that there is no
webxdc-info-message
- is clearly structured using bullet points.
this PR adds support for the property `update.notify` to notify about
changes in `update.info` or `update.summary`. the property can be set to
an array of addresses [^1]
core emits then the event `IncomingWebxdcNotify`, resulting in all UIs
to display a system notification, maybe even via PUSH.
for using the existing `update.info` and `update.summary`: the message
is no secret and should be visible to all group members as usual, to not
break the UX of having same group messages on all devices of all users -
as known already from the normal messages.
also, that way, there is no question what happens if user have disabled
notifications as the change is presented in the chat as well
doc counterpart at https://github.com/webxdc/website/pull/90closes#6217
[^1]: addresses come in either via the payload as currently or as an
explicit sender in the future - this does not affect this PR. same for
translations, see discussions at #6217 and #6097
---------
Co-authored-by: adb <asieldbenitez@gmail.com>
Co-authored-by: l <link2xt@testrun.org>
this PR adds the address to be used by the UI for
`window.webxdc.selfAddr` to webxdc-info. UIs need to be changed
accordingly and must not use configured_addr any longer.
the address is created by sha256(private-key + rfc724_mid) , which
results in different addresses for each webxdc, without the option to
find out the real address of the user.
this also returns the same address for a multi-device-setup - sending
totally random self address around might be an alternative, however
would require connectivity (both devices may be offline on first start).
for existing app, after the change, there will be a new user, resulting
eg. in a new highscore, otherwise, things should be mostly fine. this
assumption is also important as we might change the thing another time
when it comes to multi-transport.
ftr, addresses look like
`0f187e3f420748b03e3da76543e9a84ecff822687ce7e94f250c04c7c50398bc` now
when this is merged, we need to adapt #6230 and file issues for all UI
to use `info.selfAddr`
closes#6216
`receive_imf() is only used in tests and the REPL, which enables the
"internals" feature. This PR marks it as such, so that it's clear not
only from the comment that this function is not used for anything else.
If a message partially downloaded before is already IMAP-seen upon a full download, it should be
updated to `InSeen`. OTOH if it's not IMAP-seen, but already `InNoticed` locally, its state should
be preserved. So we take the maximum of two states.
This fixes sending MDNs for big messages when they are downloaded and really seen. Otherwise MDNs
are not sent for big encrypted messages because they "don't want MDN" until downloaded.
Add a test on what happens currently when apps call `markseen_msgs()` for not downloaded encrypted
messages. Such messages are marked as seen, but MDNs aren't sent for them. Also currently when such
a message is downloaded, it remains `InSeen` despite the full content hasn't yet been seen by the
user.
As discussed in #5467 we want to use `i.delta.chat` in QR codes in favor
of `OPENPGP4FPR:` scheme. This PR does the replacement in
`get_securejoin_qr` which is used in `get_securejoin_qr_svg`.
close#5467
This is needed for iOS (https://github.com/deltachat/deltachat-ios/pull/2393), see comment in the code. An alternative would be
to add an API `invalidate_config_cache()` or to do nothing and just
assume that things will be fine.
There is a change in behavior for the case
when name is the same as the suffix
(`name_len` == `namespc_len`),
but normally `files_in_use` should not contain empty filenames.
The message may be deleted while chatlist item is loading.
In this case displaying "No messages" is better than failing.
Ideally loading of the chatlist item
should happen in 1 database transaction and
always return some message if chat is not empty,
but this requires large refactoring.
this PR changes `DC_CERTCK_ACCEPT_*` to the same values in cffi as rust
does. and regards the same values as deprecated afterwards
there is some confusion about what is deprecated and what not, see
https://github.com/deltachat/deltachat-android/issues/3408
iOS needs to be adapted as it was following the docs in the CFFI before,
same desktop. both need to be graceful on reading and strict on writing.
~~**this PR is considered harmful,** so we should not merge that in
during 1.48 release, there is no urgency, things are fine (wondering if
it isn't even worth the effort, however, having different values and
deprecations is a call for trouble in the future ...)~~
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
due to async processing,
it may happen getConnectivityHtml() is called from UI before startIO() is actually called.
eg. on iOS, we may delay startIo() if another process is still processing a PUSH notification -
when during this time, the connectivity view is opened,
it is weird if a big error "CONTACT THE DEVELOPERS!11!!!" is shown :)
also, there is not really a function is_connected(),
for $reasons, as this turned out to be flacky,
so it is not even easy to check the state before calling getConnectivityHtml()
it is not worth in doing too much special,
we are talking about rare situaton,
also, the connectivity view gets updated some moments later.
Before I was getting
```
error: attribute 'targetPlatforms' missing
at /nix/store/dyzl40h25l04565n90psbhzgnc5vp2xr-source/pkgs/build-support/rust/build-rust-package/default.nix:162:7:
161| meta.platforms or lib.platforms.all
162| rustc.targetPlatforms;
| ^
163| };
```
This was probably an upstream issues as discussed in here
https://discourse.nixos.org/t/error-attribute-targetplatforms-missing-after-updating-inputs/54494
After this update it is fixed.
80 characters are a bit limited in practise ...
On Mon, 3 Jan, 2022 at 8:34 PM "Anonymous The Mighty" <anonymous@example.com> wrote:
... already breaks the limit. it is good to allow up to 40 additional characters
for name + email address.
allowing any length, however, may catch too much,
as the line could also be a normal paragraph with important content,
so 120 characters seems reasonable.
the idea of adding more complexity here would probably lead only to, well more complexity -
things can anyways go wrong -
and, we have the "show full message..." button for exactly that purpose,
so that the user can access everything as original.
so, if things go wrong sometimes,
this is expected and fine.
There were many cases in which "member added/removed" messages were added to chats even if they
actually do nothing because a member is already added or removed. But primarily this fixes a
scenario when Alice has several devices and shares an invite link somewhere, and both their devices
handle the SecureJoin and issue `ChatGroupMemberAdded` messages so all other members see a
duplicated group member addition.
This change adds support for receiving
Autocrypt header in the protected part of encrypted message.
Autocrypt header is now also allowed in mailing lists.
Previously Autocrypt header was rejected when
List-Post header was present,
but the check for the address being equal to the From: address
is sufficient.
New experimental `protect_autocrypt` config is disabled
by default because Delta Chat with reception
support should be released first on all platforms.
Over the past years, it happend two times that a user came to me worried
about a false-positive "Cannot login as ***. Please check if the e-mail
address and the password are correct." message.
I'm not sure why this happened, but this PR makes the logic for
showing this notification stricter:
- Before: The notification is shown if connection fails two times in a
row, and the second error contains the word "authentication".
- Now: The notification is shown if the connection fails two times in a
row, and _both_ error messages contain the word "authentication".
The second commit just renames `login_failed_once` to
`authentication_failed_once` in order to reflect this change.
Delta Chat for Android does not support Android 4 anymore,
so there is no reason to keep using unsupported NDK.
r27 is the latest LTS version of Android NDK.
Tested:
- `nix build .#deltachat-rpc-server-arm64-v8a-android`
- `nix build .#deltachat-rpc-server-armv6l-linux`
`nix build .#deltachat-rpc-server-x86_64-android`
and
`nix build .#deltachat-rpc-server-x86-android`
still fail, but we do not build it in CI.
Explicit check for `-----BEGIN PGP MESSAGE-----` is unnecessary
and not sufficient to ensure that the message is valid.
We have already checked the MIME type,
so ASCII-armored OpenPGP message should be inside.
If it's not, decryption will fail anyway.
close#2338
Concat error messages when receiving new ndns.
This PR adds a newline followed by the new NDN error to the error text.
Maybe we should use something more prominent like
```
-----------------------------------------------------------------------
```
or more newlines, but I'm not sure. This maybe has to be tested on a
real device to see what works best.
3f9242a saves name from all QR codes to `name` (i.e. manually edited name), but for SecureJoin QR
codes the name should be saved to `authname` because such QR codes are generated by the
inviter. Other QR codes may be generated locally and not only by Delta Chat, so the name from them
mustn't go to `authname` and be revealed to the network or other contacts.
If the number of retries for message is exceeded,
do not fail when marking it as failed if the message does not exist.
Otherwise we may never delete the message from SMTP queue
because corresponding msg_id is not valid anymore.
The server should decode the URL and according to RFC 3986
query parameters may or may not be URL-encoded,
but at some servers don't decode the dot correctly.
`@` is decoded correctly by autoconfig.murena.io
This may indicate that there was a new \Seen flag
that we don't want to skip.
Also don't drain unsolicited responses while scanning folders. Now we
only drain unsolicited responses right before IDLE and always redo the
whole fetch cycle if there have been some. Some message in the scanned
folder may not be fetched that would be previously fetched otherwise,
but it will be picked up on the next folder scan.
Just a small refactoring. Instead of rebinding res all the time just use
`and` and `and_then`how they are inteded to be used. Improves code
readability imo.
This adds a function to `Message`:
```rust
pub fn new_text(text: String) -> Self {
Message {
viewtype: Viewtype::Text,
text,
..Default::default()
}
}
```
I keep expecting that a function like this must exist and being
surprised that it doesn't.
Open question is whether it should be `pub` or `pub(crate)` - I made it
`pub` for now because it may be useful for others and we currently we
aren't thinking about the Rust API that much, anyway, but I can make it
`pub(crate)`, too (then it can't be used in deltachat-jsonrpc and
deltachat-repl).
I replaced some usages of Message::new(Viewtype::Text), but not all yet,
I'm going to do this in a follow-up, which will remove another around 65
LOC.
Right now, when there is an SMTP connection error, the connectivity view
will always show "Error: SMTP connection failure: SMTP failed to
connect".
Instead, I just used the same method that is used in imap connect()
already.
0a63083df7 (fix: Shorten message text in locally sent messages too)
sets `msgs.mime_modified` for long outgoing messages, but forgets to save full message text.
I.e. add the "Messages are guaranteed to be end-to-end encrypted from now on." message and mark the
chat as protected again because no user action is required in this case. There are a couple of
problems though:
- If the program crashes earlier than the protection is restored, the chat remains
protection-broken. But this problem already exists because `ChatId::set_protection()` is never
retried.
- If multiple old unverified messages are received, protection messages added in between don't
annihilate, so they clutter the chat.
Let's always set `Config::NotifyAboutWrongPw` before saving configuration, better if a wrong
password notification is shown once more than not shown at all. It shouldn't be a big problem
because reconfiguration is a manual action and isn't done frequently.
Also for the same reason reset `Config::NotifyAboutWrongPw` only after a successful addition of the
appropriate device message.
Otherwise it always fails with SQLITE_READONLY:
```
WARNING src/sql.rs:769: Failed to run incremental vacuum: attempt to write a readonly database: Error code 8: Attempt to write a readonly database.
```
this PR adds a function that can be used to create any QR code, in a raw
form.
this can be used to create add-contact as well as add-second-device QR
codes (eg. `dc_create_qr_svg(dc_get_securejoin_qr())`) - as well as for
other QR codes as proxies.
the disadvantage of the rich-formatted QR codes as created by
`dc_get_securejoin_qr_svg()` and `dc_backup_provider_get_qr_svg()` were:
- they do not look good and cannot interact with UI layout wise (but
also tapping eg. an address is not easily possible)
- esp. text really looks bad. even with
[some](e5dc8fe3d8)
[hacks](https://github.com/deltachat/deltachat-android/pull/2215) it
[stays buggy](https://github.com/deltachat/deltachat-ios/issues/2200);
the bugs mainly come from different SVG implementation, all need their
own quirks
- accessibility is probably bad as well
we thought that time, SVG is a great thing for QR codes, but apart from
basic geometrics, it is not.
so, we avoid text, this also means to avoid putting an avatar in the
middle of the QR code (we can put some generic symbol there, eg.
different ones for add-contact and add-second-device).
while this looks like a degradation, also other messengers use more raw
QR codes. also, we removed many data from the QR code anyway, eg. the
email address is no longer there. that time, sharing QR images was more
a thing, meanwhile we have invite links, that are much better for that
purpose.
in theory, we could also leave the SVG path completely and go for PNG -
which we did not that time as PNG and text looks bad, as the system font
is not easily usable :) but going for PNG would add further challenges
as passing binary data around, and also UI-implemtation-wise, that would
be a larger step. so, let's stay with SVG in a form we know is
compatible.
the old QR code functions are deprecated.
this PR allows setting a "private tag" for a profile, see
https://github.com/deltachat/deltachat-android/pull/3373 for a possible
UI.
currently, the core does not do anything with the tag (so, it could also
be a ui.-config option), however, this may change in the future - it
might bet synced, and become also otherwise useful in core. also, having
this in core is better documentation-wise, as otherwise each UI easily
does its own things :)
This adds "stale-while-revalidate" in-memory cache for DNS. Instead of
calling `tokio::net::lookup_host` we use previous result of
`tokio::net::lookup_host` immediately and spawn revalidation task in the
background. This way all lookups after the first successful one return
immediately.
Most of the time results returned by resolvers are the same anyway, but
with this cache we avoid waiting 60 second timeout if DNS request is
lost. Common reason result may be different is round-robin DNS load
balancing and switching from IPv4 to IPv6 network. For round-robin DNS
we don't break load balancing but simply use a different result, and for
IPv6 we anyway likely have a result in persistent cache and can use IPv4
otherwise.
Especially frequent should be the case when you send a message over SMTP
and SMTP connection is stale (older than 60 s), so we open a new one.
With this change new connection will be set up faster as you don't need
to wait for DNS resolution, so message will be sent faster.
rust-analyzer was showing warnings here because it is always also
building in the Test configuration, and EventType has a
```rust
#[cfg(test)]
Test,
```
variant, which was not matched.
On main, when running `cargo build`, the following warning is emitted:
> warning:
/home/jonathan/deltachat-android/jni/deltachat-core-rust/deltachat-ffi/Cargo.toml:
`default-features` is ignored for deltachat, since `default-features`
was not specified for `workspace.dependencies.deltachat`, this could
become a hard error in the future
This is because when referring to a workspace dependency, it's not
possible to remove features, it's only possible to add features, so that
the `vendored` feature was always enabled with no possibility to disable
it.
This PR restores the wanted behavior of enabling vendoring by default
with the possibility to disable it with "default-features = false".
It fixes `nix build .#python-docs` by not passing
`--no-default-features` when building deltachat with nix.
If a message from an old contact's setup is received, the outdated Autocrypt header isn't applied,
so the contact verification preserves. But the chat protection breaks because the old message is
sorted to the bottom as it mustn't be sorted over the protection info message (which is `InNoticed`
moreover). Would be nice to preserve the chat protection too e.g. add a "protection broken" message,
then the old message and then a new "protection enabled" message, but let's record the current
behaviour first.
To avoid reordering, wait for "member removed" message
to be received before sending "member added".
The test failed at least once
because email server may reorder the messages internally
while delivering.
Otherwise instead of "old address"
ac2 may receive "member added",
resulting in this failure:
```
> assert msg_in_1.text == msg_out.text
E AssertionError: assert 'Member Me (c...hat.computer.' == 'old address'
E - old address
E + Member Me (ci-hfpxxe@***) added by ci-8e7mkr@***.
```
This ensures we do not get stuck trying DNS resolver results
when we have a known to work IP address in the cache
and DNS resolver returns garbage
either because it is a captive portal
or if it maliciously wants to get us stuck
trying a long list of unresponsive IP addresses.
This also limits the number of results we try to 10 overall.
If there are more results, we will retry later
with new resolution results.
It is impossible to set no display name anyway
in Delta Chat Android at least
because we don't want email addresses
in the UI.
This test does not work with long domains
that may get wrapped, so better remove it
instead of trying to prevent wrapping of domains.
Received messages shouldn't mingle with just sent ones and appear somewhere in the middle of the
chat, so we go after the newest non fresh message.
But if a received outgoing message is older than some `InSeen` message, better sort the received
message purely by timestamp (this is an heuristic in order not to break the Gmail-like case
simulated by `verified_chats::test_old_message_4()`). We could place the received message just
before that `InSeen` message, but anyway the user may not notice it.
At least this fixes outgoing messages sorting for shared accounts where messages from other devices
should be sorted the same way as incoming ones.
Otherwise backups exported from the current core and imported in versions < 1.144.0 have QR codes
not working. The breaking change which removed the column is
5a6efdff44.
Before file extensions were also limited to 32 chars, but extra chars in the beginning were just cut
off, e.g. "file.with_lots_of_characters_behind_point_and_double_ending.tar.gz" was considered to
have an extension "d_point_and_double_ending.tar.gz". Better to take only "tar.gz" then.
Also don't include whitespace-containing parts in extensions. File extensions generally don't
contain whitespaces.
There is already code below that emits
progress 0 or 1000 depending on whether
configuration succeeded or failed.
Before this change cancelling resulted
in progress 0 emitted,
immediately followed by progress 1000.
Before this change progress bar only started
when database is already transferred.
Database is usually the largest file
in the whole transfer, so the transfer appears
to be stuck for the sender.
With this change progress bar
starts for backup export
as soon as connection is received
and counts bytes transferred over the connection
using AsyncWrite wrapper.
Similarly for backup import,
AsyncRead wrapper counts the bytes
received and emits progress events.
Instead of treating NULL type error
as absence of the row,
handle NULL values with SQL.
Previously we sometimes
accidentally treated a single column
being NULL as the lack of the whole row.
The "Cannot establish guaranteed end-to-end encryption with ..." info
message can have lots of causes, and it happened twice to us now that it
took us some time to figure out which one it is.
So, include some more detail in the info message by simply adding the
non-translated error message in parantheses.
If we want to put in some more effort for nicer error messages, we
could:
- Introduce one new translated string "Cannot establish guaranteed
end-to-end encryption with …. Cause: %2$s" or similar (and remove the
old stock string)
- And/Or: Introduce new translated strings for all the possible errors
- And/Or: Maybe reword it in order to account better for the case that
the chat already is marked as g-e2ee, or use a different wording
(because if the chat is marked as g-e2ee then it might be nice to notify
the user that something may have gone wrong, but it's still working,
just that maybe the other side doesn't have us verified now)
Info messages are added
at the beginning of unpromoted group chats
("Others will only see this group after you sent a first message."),
may be created by WebXDC etc.
They are not sent outside
and have local Message-ID that
is not known to other recipients
so they should be skipped when constructing
In-Reply-To and References.
Without this fix IMAP loop may get stuck
trying to download non-existing message over and over
like this:
```
src/imap.rs:372: Logging into IMAP server with LOGIN.
src/imap.rs:388: Successfully logged into IMAP server
src/scheduler.rs:361: Failed to download message Msg#3467: Message Msg#3467 does not exist.
src/scheduler.rs:418: Failed fetch_idle: Failed to download messages: Message Msg#3467 does not exist
```
The whole download operation fails
due to attempt to set the state of non-existing message
to "failed". Now download of the message
will "succeed" if the message does not exist
and we don't try to set its state.
HTTPS requests are used to fetch
remote images in HTML emails,
to fetch autoconfig XML,
to POST requests for `DCACCOUNT:` QR codes
to make OAuth 2 API requests
and to connect to HTTPS proxies.
Rustls is more aggressive than OpenSSL
in deprecating cryptographic algorithms
so we cannot use it for IMAP and SMTP
to avoid breaking compatibility,
but for HTTPS requests listed
above this should not result in problems.
As HTTPS requests use only strict TLS checks,
there is no `strict_tls` argument
in `wrap_rustls` function.
Rustls is already used by iroh,
so this change does not introduce new dependencies.
This should fix ad-hoc groups splitting when messages are fetched out of order from different
folders or otherwise reordered, or some messages are missing so that the messages reference chain is
broken, or a member was removed from the thread and readded later, etc. Even if this way two
different threads are merged, it looks acceptable, having many threads with the same name/subject
and members isn't a common use case.
Instead of generating 72 random bits
and reducing them to 66 bits of Base64 characters,
generate 144 bits (18 bytes)
which is exactly 24 Base64 characters.
This should still be accepted by existing
Delta Chat clients which expect group ID
to be between 11 and 32 characters.
Message-ID creation is also simplified
to not have `Mr.` prefix
and dot in between two IDs.
Now it is a single ID followed by `@localhost`.
Some outdated documentation comments
are removed, e.g. group messages
don't start with `Gr.` already.
The greeting is now always read manually,
even for STARTTLS connections,
so the errors returned on failure to read form the stream
are the same regardless of the connection type.
Why:
- With IMAP APPEND we can upload messages directly to the DeltaChat folder (for non-chatmail
accounts).
- We can set the `\Seen` flag immediately so that if the user has other MUA, it doesn't alert about
a new message if it's just a sync message (there were several such reports on the support
forum). Though this also isn't useful for chatmail.
- We don't need SMTP envelope and overall remove some overhead on processing sync messages.
If a displayname equals to the address, adding it looks excessive.
Moreover, it's not useful for Delta Chat receiving the message because
`sanitize_name_and_addr()` removes such a displayname anyway. Also now
at least DC Android requires specifying profile name, so there should be
a fallback for users having meaningful addresses to keep the old
behaviour when Core generates `From` w/o the profile name, and this
question has already appeared on the forum.
This reverts commit 1caf672904.
Otherwise public key signature is regenerated each time the key is
loaded and test `key::tests::test_load_self_existing` which loads the
key twice fails when two loads happen on different seconds.
Closes#5976
Groups promotion to other devices and QR code tokens synchronisation are not synchronised processes,
so there are reasons why a QR code token may arrive earlier than the first group message:
- We are going to upload sync messages via IMAP while group messages are sent by SMTP.
- If sync messages go to the mvbox, they can be fetched earlier than group messages from Inbox.
`chat::create_send_msg_jobs()` already handles `Config::BccSelf` as needed. The only exception is
Autocrypt setup messages. This change unifies the logic for the self-chat and groups only containing
`SELF`.
This makes possible to schedule one more sending of the message, the existing jobs are not
cancelled. Otherwise it's complicated to implement bots that resend messages when a new member joins
the group.
This change removes OAuth2 for Gmail
as Delta Chat does not have a working
client ID anymore.
Tests are adjusted to test against Yandex
and MX queries for OAuth2 are always disabled
because they were only used to detect Google Workspace.
This change introduces new config options
`proxy_enabled` and `proxy_url`
that replace `socks5_*`.
Tested with deltachat-repl
by starting it with
`cargo run --locked -p deltachat-repl -- deltachat-db` and running
```
> set proxy_enabled 1
> set proxy_url ss://...
> setqr dcaccount:https://chatmail.example.org/new
> configure
```
And also:
- Make it `pub(crate)`.
- Use it in `should_request_mdns()` as using `config_exists()` there isn't correct because the
latter doesn't look at environment.
I.e. treat `DeleteServerAfter == None` as "delete at once". But when a backup is exported, set
`DeleteServerAfter` to 0 so that the server decides when to delete messages, in order not to break
the multi-device case. Even if a backup is not aimed for deploying more devices, `DeleteServerAfter`
must be set to 0, otherwise the backup is half-useful because after a restoration the user wouldn't
see new messages deleted by the device after the backup was done. But if the user explicitly set
`DeleteServerAfter`, don't change it when exporting a backup. Anyway even for non-chatmail case the
app should warn the user before a backup export if they have `DeleteServerAfter` enabled.
Also do the same after a backup import. While this isn't reliable as we can crash in between, this
is a problem only for old backups, new backups already have `DeleteServerAfter` set if necessary.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
There are providers in the provider database
that do not have servers specified.
For such providers default list should be tried
just like when configuring unknown providers.
Documentation comment says forward and backward verification is set,
but the code was not doing it.
`vc-contact-confirm` and `vg-member-added` messages
indicate that other device finished securejoin protocol
so we know Bob has our key marked as verified.
bcc_self has been enabled by default
since core version 1.95.0
by merging
PR <https://github.com/deltachat/deltachat-core-rust/pull/3612>.
However deltachat.h documentation
still incorrectly said that bcc_self is disabled by default.
This change replaces
usage of `reqwest` and `hyper-util`
with custom connection establishment code
so it is done in the same way
as for IMAP and SMTP connections.
This way we control HTTP, IMAP and SMTP
connection establishment
and schedule connection attempts
to resolved IP addresses
in the same way for all 3 protocols.
With current implementation
every time connection fails
we take the next delay from `delays` iterator.
In the worst case first 4 DNS results
immediately refuse connection
and we start fifth connection attempt
with 1 year timeout,
effectively continuing all remaining
connection attempts without concurrency.
With new implementation
new connection attempts are
added to `connection_attempt_set`
independently of connection failures
and after 10 seconds
we always end up with five
parallel connection attempts
as long as there are enough IP addresses.
Previously for each connection candidate (essentially host and port
pair) after resolving the host to a list of IPs Delta Chat iterated IP
addresses one by one. Now for IMAP and SMTP we try up to 5 IP addresses
in parallel. We start with one connection and add more connections
later. If some connection fails, e.g. we try to connect to IPv6 on IPv4
network and get "Network is unreachable" (ENETUNREACH) error, we replace
failed connection with another one immediately.
Co-authored-by: Hocuri <hocuri@gmx.de>
This ensures we don't add multiple Auto-Submitted headers
when bots send vg-request or vc-request messages.
The change fixes failing
receive_imf::tests::test_bot_accepts_another_group_after_qr_scan
test.
This fixes the bug that sometimes made QR scans fail.
The problem was:
When sorting headers into unprotected/hidden/protected, the From: header
was added twice for all messages: Once into unprotected_headers and once
into protected_headers. For messages that are `is_encrypted && verified
|| is_securejoin_message`, the display name is removed before pushing it
into unprotected_headers.
Later, duplicate headers are removed from unprotected_headers right
before prepending unprotected_headers to the message. But since the
unencrypted From: header got modified a bit when removing the display
name, it's not exactly the same anymore, so it's not removed from
unprotected_headers and consequently added again.
I.e. to all messages except "v{c,g}-request" as they sent out on a QR code scanning which is a
manual action and "vg-member-added" as formally this message is auto-submitted, but the member
addition is a result of an explicit user action. Otherwise it would be strange to have the
Auto-Submitted header in "member-added" messages of verified groups only.
Before the fix HTTP client
had no connection timeout,
so it only had a chance
to test one IPv6 and one IPv4
address if the first addresses timed out.
Now it can test at least 4 addresses
of each family and more if some addresses
refuse connection rather than time out.
Otherwise if DNS server returns incorrect results,
we may never try preloaded DNS results.
For example, we may get our first results
from a captive portal.
To test, add `127.0.0.1 example.org`
and try to create an account.
Without this change we only try 127.0.0.1 and fail.
With this change preloaded DNS results are tried as well.
this PR adds the type DC_QR_SOCKS5_PROXY to `dc_check_qr()` for
**supporting telegram proxy QR codes**. if returned, the UI should ask
the user if they want to us the proxy and call
`dc_set_config_from_qr();` afterwards (plus maybe `dc_configure()`).
idea is to improve our proxy story, follow ups may be:
- in UI, - move proxy out of "Account & Password", as a **separate
"Proxy Activity"** (it should stay in "Advanced" for now, however, below
"Server", which might be moved up)
- allow **opening the "Proxy Activity" from the welcome screens**
three-dot-menu (that would also solve a long standing issue that
entering the email address bypasses the proxy
- show proxy usage in the "Connectivity View" and/or add an **icon** to
the main chatlist screen (beside three-dot menu) in case some proxy is
in use; tapping this icon will open the "Proxy Activity"
- the the new "Proxy Activity", add a **share / show proxy QR code**
button. that would generate invite links in the form
`https://i.delta.chat/socks#...` - so that tapping then opens the app.
support for these links need to be added to core then.
- handle a list of proxies in core, offer selection in UI. the list
could be one for all profiles and could be filled eg. by normal invite
links or other channels
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Fix a typo in the config name
(by using `Config::` to avoid it)
and make sure we don't panic on unknown values.
Also test that we don't panic on unknown
`configured_imap_certificate_checks` values.
Apparently some providers fail TLS connection
with "no_application_protocol" alert
even when requesting "imap" protocol for IMAP connection
and "smtp" protocol for SMTP connection.
Fixes <https://github.com/deltachat/deltachat-core-rust/issues/5892>.
configured_imap_certificate_checks=0 means
accept invalid certificates unless provider database
says otherwise or SOCKS5 is enabled.
It should not be saved into the database anymore.
This bug was introduced in
<https://github.com/deltachat/deltachat-core-rust/pull/5854>
(commit 6b4532a08e)
and affects released core 1.142.4, 1.142.5 and 1.142.6.
Fix reverts faulty fix from
<https://github.com/deltachat/deltachat-core-rust/pull/5886>
(commit a268946f8d)
which changed the way configured_imap_certificate_checks=0
is interpreted and introduced problems
for existing setups with configured_imap_certificate_checks=0:
<https://github.com/deltachat/deltachat-core-rust/issues/5889>.
Existing test from previous fix is not reverted
and still applies.
Regression test is added to check that
configured_imap_certificate_checks
is not "0" for new accounts.
If user has not set any settings manually
and provider is not configured,
default to strict TLS checks.
Bug was introduced in
<https://github.com/deltachat/deltachat-core-rust/pull/5854>
(commit 6b4532a08e)
and affects released core 1.142.4 and 1.142.5.
The problem only affects accounts configured
using these core versions with provider
not in the provider database or when using advanced settings.
`imap::Session::store_seen_flags_on_imap()` handles messages from multiple folders, so not being
able to select one folder mustn't fail the whole function.
`cargo install` ignores lockfile by default.
Without lockfile current build fails
due to iroh-net 0.21.0 depending on `derive_more` 1.0.0-beta.6
but failing to compile with `derive_more` 1.0.0.-beta.7.
This particular error will be fixed by upgrading to iroh 0.22.0,
but using lockfile will avoid similar problems in the future.
- To avoid receiving undecryptable MDNs by bots and replying to them if the bot's key changes.
- MDNs from bots don't look useful in general, usually the user expects some reply from the bot, not
just that the message is read.
Recently there are many questions on the Delta Chat forum why some unexpected encrypted messages
appear in Inbox. Seems they are mainly sync messages, though that also obviously happens to
SecureJoin messages. Anyway, regardless of the `MvboxMove` setting, auto-generated outgoing messages
should be moved to the DeltaChat folder so as not to complicate co-using Delta Chat with other MUAs.
Otherwise cargo emits these warnings:
warning: .../deltachat-core-rust/deltachat-ffi/Cargo.toml: `default-features` is ignored for deltachat, since `default-features` was not specified for `workspace.dependencies.deltachat`, this could become a hard error in the future
warning: .../deltachat-core-rust/deltachat-rpc-server/Cargo.toml: `default-features` is ignored for deltachat, since `default-features` was not specified for `workspace.dependencies.deltachat`, this could become a hard error in the future
warning: .../deltachat-core-rust/deltachat-rpc-server/Cargo.toml: `default-features` is ignored for deltachat-jsonrpc, since `default-features` was not specified for `workspace.dependencies.deltachat-jsonrpc`, this could become a hard error in the future
Previously Delta Chat tried to use local part of email address as well.
This configuration is very uncommon,
but trying it doubled the time of configuration try
in the worst case, e.g. when the password is typed in incorrectly.
Single function smtp::connect::connect_stream
returns a stream of a single `dyn` type
that can be a TLS, STARTTLS or plaintext
connection established over SOCKS5 or directly.
This way we can't get an account with missing blobs if there's not enough disk space.
Also delete already unpacked files if all files weren't unpacked successfully. Still, there are some
minor problems remaining:
- If a db wasn't imported successfully, unpacked blobs aren't deleted because we don't know at which
step the import failed and whether the db will reference the blobs after restart.
- If `delete_and_reset_all_device_msgs()` fails, the whole `import_backup()` fails also, but after a
restart delete_and_reset_all_device_msgs() isn't retried. Probably errors from it should be
ignored at all.
Previously Delta Chat tried all DNS resolution results
in sequence until TCP connection is established successfully,
then tried to establish TLS on top of the TCP connection.
If establishing TLS fails, the whole
connection establishment procedure failed
without trying next DNS resolution results.
In particular, in a scenario
where DNS returns incorrect result
pointing e.g. to a server
that listens on the TCP port
but does not have correpsponding TLS certificate,
Delta Chat now will fall back to the cached result
and connect successfully.
while adapting strings for the recent change about read receipts,
https://github.com/deltachat/deltachat-core-rust/pull/5712 , it turns
out in discussions eg. at
https://github.com/deltachat/deltachat-android/issues/3179 that
untranslated english for the read receipts seem to be sufficient or even
better:
- do not reveal the sender's language
- unexpected languages are confusing - even if you chat in english, you
may get Chinese read receipts
- many clients do not show the text anyways, iirc, eg. Outlook display
the read receipts in context, and Delta Chat of course as well
- afaik, we're leaving comparable `multipart/report` untranslated as
well (sync, but also webxdc updates are practically english only)
- less code, fewer translations needed :)
There are many reasons why we may fail to find valid signatures in a message, e.g. we don't yet know
a public key attached in the same message, anyway, if From is forged, the message must be rejected.
Also always take the displayname from encrypted From, even if no valid signatures are found.
This is a hint for apps that a WebXDC icon should be shown in the summary, e.g. in the
chatlist. Otherwise it's not clear when it should be shown, e.g. it shouldn't be shown in a reaction
summary.
For hardcoded built-in DNS results
there is no cache entry in `dns_cache` table
so they cannot be prioritized if DNS resolution
never returned these results yet.
If there is no entry, a new one should be created.
SQL UPSERT does this.
Follow-up to 3cf78749df "Emit DC_EVENT_MSGS_CHANGED for
DC_CHAT_ID_ARCHIVED_LINK when the number of archived chats with unread messages increases (#3940)".
In general we don't want to make an extra db query to know if a noticied chat is
archived. Emitting events should be cheap, better to allow false-positive `MsgsChanged` events.
`Context::send_sync_msg()` mustn't be called from multiple tasks in parallel to avoid sending the
same sync items twice because sync items are removed from the db only after successful
sending. Let's guarantee this by calling `send_sync_msg()` only from the SMTP loop. Before
`send_sync_msg()` could be called in parallel from the SMTP loop and another task doing
e.g. `chat::sync()` which led to `test_multidevice_sync_chat` being flaky because of events
triggered by duplicated sync messages.
New protocol streams .tar into iroh-net
stream without traversing all the files first.
Reception over old backup protocol
is still supported to allow
transferring backups from old devices
to new ones, but not vice versa.
The "Chat-Group-Member-Removed" header is added to ad-hoc group messages as well, so we should check
for its presense before creating an ad-hoc group as we do for DC-style groups.
Before, update sending might be delayed due to rate limits and later merged into large
messages. This is undesirable for apps that want to send large files over WebXDC updates because the
message with aggregated update may be too large for actual sending and hit the provider limit or
require multiple attempts on a flaky SMTP connection.
So, don't aggregate updates if the size of an aggregated update will exceed the limit of 100
KiB. This is a soft limit, so it may be exceeded if a single update is larger and it limits only the
update JSON size, so the message with all envelopes still may be larger. Also the limit may be
exceeded when updates are sent together with the WebXDC instance when resending it as the instance
size isn't accounted to not complicate the code. At least this is not worse than the previous
behaviour when all updates were attached.
Bridge bots like matterdelta need to set a quoted text without referencing the quoted message, this
makes easier bridging messages from other platforms to Delta Chat or even bridging Delta Chat groups
in different accounts where you can not set a quoted message by the message id from another account.
Even though SMTP ALPN is not officially registered (unlike IMAP),
it is an obvious choice that will allow
to multiplex SMTP and other protocols on the same TLS port.
Newer node-gyp uses newer gyp
which requires newer python
that is not available in Debian 10
container that is
used for building node prebuilds
with old glibc.
Follow-up to 5fa7cff46. Let's still not send a sync message if the contact wasn't modified. This is
not very important, but just for consistency with the `chat::rename_ex()` behaviour.
This way it's clearer which key is which and also adding the key fingerprint to the file name avoids
overwriting another previously exported key. I think this is better than adding an incremental
number as we do for backups, there's no need to export a key several times to different files.
For purposes of building a message it's better to consider the self-chat as verified. Particularly,
this removes unencrypted name from the "From" header.
If currently there are no multi-device bots, let's disable sync messages for bots at all. Another
option is to auto-disable sync messages when `Config::Bot` is set, so sync messages can be reenabled
if needed. But let's leave this option for the future.
About the first TODO: I tried this out, but it didn't actually improve
things, for two reasons:
1. The trick with `#![cfg_attr(not(test),
warn(clippy::indexing_slicing))]` that enables the lint everywhere
except for tests doesn't work with workspace-wide lints. (Context: We
want to lint against indexing because it might panic, but in a test
panicking is fine, so we don't want to enable the lint in tests).
2. Most of our crates have different sets of lints right now, so it
would only be very few crates that use the workspace-wide list of lints.
About the second TODO:
It's not feasible right now to fully parse vCards, and for our
good-enough parser the current behavior is fine, I think. If we fail to
parse some realworld vCards because of this, we can still improve it.
Notification server uses APNS server
for heartbeat notifications,
so registering FCM tokens there
will result in failing to notify them
and unregistering them anyway.
It's possible that when rebasing a PR adding a migration a merge-conflict doesn't occur if another
migration was added in the target branch. Better to have at least runtime checks that the migration
version is correct. Looks like compile-time checks are not possible because Rust doesn't allow to
redefine constants, only vars.
Instead of constructing lists of protected,
unprotected and hidden headers,
construct a single list of headers
and then sort them into separate lists
based on the well-defined policy.
This also fixes the bug
where Subject was not present in the IMF header
for signed-only messages.
Closes#5713
Bot processes are run asynchronously, so we shouldn't send messages to a bot before it's fully
initialised and skipped existing messages for processing, i.e. before DC_EVENT_IMAP_INBOX_IDLE is
emitted.
we're always checking the configuration encrypted.
saying it is 'preferred' encrypted is misleading,
therfore, just remove it.
i do not think, it is worth saying that we do not query 'http',
this is clear from the source code.
moreover, fix two typos.
Looks like it has no sense to send any profile data (From/To names, self-status; self-avatar was
never sent even before) in MDNs, they aren't normal messages and aren't seen in a MUA. Better not to
reveal profile data to the network and even to contacts in MDNs and make them more lightweight.
Follow-up to b771311593. Since that commit names are not revealed in
verified chats, but during verification (i.e. SecureJoin) they are still sent unencrypted. Moreover,
all profile data mustn't be sent even encrypted before the contact verification, i.e. before
"v{c,g}-request-with-auth". That was done for the selfavatar in
304e902fce, now it's done for From/To names and the self-status as
well. Moreover, "v{c,g}-request" and "v{c,g}-auth-required" messages are deleted right after
processing, so other devices won't see the received profile data anyway.
Probably sync messages generated for a not yet configured account are useless because there are no
other devices yet. And even if this is not true, we don't want to depend on the order of setting
`Config::SyncMsgs` and other keys. Also w/o this the Python tests don't work if we start syncing
`Config::MvboxMove` because they don't expect that sync messages are sent while configuring
accounts.
NB: We don't restart IO from the synchronisation code, so `MvboxMove` isn't effective immediately if
`ConfiguredMvboxFolder` is unset, but only after a reconnect to IMAP.
Before, if the user deleted a message too quickly after sending, it was deleted only locally. The
fix is to remember for tombstones that the corresponding message should be deleted on the server
too.
This is a way to prevent redownloading locally deleted messages. Otherwise if a message is deleted
quickly after sending and `bcc_self` is configured, the BCC copy is downloaded and appears as a new
message as it happens for messages sent from another device.
Delta Chat -style groups have names w/o prefixes like "Re: " even if the user is added to an already
existing group, so let's remove prefixes from ad-hoc group names too. Usually it's not very
important that the group is a classic email thread existed before, this info just eats up screen
space. Also this way a group name is likely to preserve if the first message was missed.
SQLite search with `LIKE` is case-insensitive only for ASCII chars. To make it case-insensitive for
all messages, create a new column `msgs.txt_normalized` defaulting to `NULL` (so we do not bump up
the database size in a migration) and storing lowercased/normalized text there when the row is
created/updated. When doing a search, search over `IFNULL(txt_normalized, txt)`.
Before, if `Config::FetchExistingMsgs` is set, existing messages were received with the `InSeen`
state set, but for bots they must be `InFresh` and also `IncomingMsg` events should be emitted for
them so that they are processed by bots as it happens with new messages.
Since commit c0a17df344
(PR https://github.com/deltachat/deltachat-core-rust/pull/3402)
`send_smtp_messages` returns an error
as soon as it encounters the first message it failed to send.
Since this worked like this for about 2 years
without any problems, there is no need to revert the change,
but outdated comment should be removed.
If a display name should be protected (i.e. opportunistically encrypted), only put the corresponding
address to the unprotected headers. We protect the From display name only for verified chats,
otherwise this would be incompatible with Thunderbird and K-9 who don't use display names from the
encrypted part. Still, we always protect To display names as compatibility seems less critical here.
When receiving a messge, overwrite the From display name but not the whole From field as that would
allow From forgery. For the To field we don't really care. Anyway as soon as we receive a message
from the user, the display name will be corrected.
Co-authored-by: iequidoo <dgreshilov@gmail.com>
Otherwise it's impossible to remove a member with missing key from a protected group. In the worst
case a removed member will be added back due to the group membership consistency algo.
The "I left the group" message can't be sent to a protected group if some member's key is missing,
in this case we should remain in the group. The problem should be fixed first, then the user may
retry to leave the group.
There was a comment in `fetch_existing_msgs()`: "Bots don't want those messages". If a bot doesn't
want this setting, why enable it? It's disabled by default anyway.
`IsChatmail` is set also by `inbox_fetch_idle()`, but it isn't called during `configure()`. Setting
`IsChatmail` from `inbox_fetch_idle()` is necessary to handle client/server upgrades, but
`IsChatmail` also should be available for the app after configuring an account, e.g. DC Android
needs it to know whether to ask the user to disable battery optimisations.
Before this fix LogSink was dropped immediately,
resulting in no logs printed for contexts created using
TextContext::new_alice(), but printed for contexts created
using TextContext::new().
`!to_ids().is_empty()` check is needed in cases of 1:1 chat creation
because otherwise `to_id` is undefined,
but in case of outgoing group message without recipients
observed on a second device creating a group should be allowed.
Previously OpenSSL was pinned due to problems
with Zig toolchain used back then
and then not unpinned even after switching from Zig to Nix
due to <https://github.com/alexcrichton/openssl-src-rs/issues/235>.
The problem should be fixed now
and we can try to unpin OpenSSL again.
When doing an AEAP transition, we mustn't just delete the old peerstate as this would break
encryption to it. This is critical for non-verified groups -- if we can't encrypt to the old
address, we can't securely remove it from the group (to add the new one instead).
-remove `DELTA_CHAT_SKIP_PATH` environment variable
- remove version check / search for dc rpc server in $PATH
- remove `options.skipSearchInPath`
- add `options.takeVersionFromPATH`
async-broadcast returns Overflowed error once
if channel overflow happened.
Public APIs such as get_next_event JSON-RPC method
are only expecting an error if the channel is closed,
so we should not propagate overflow error outside.
In particular, Delta Chat Desktop
stop receiving events completely if an error
is returned once.
If overflow happens, we should ignore it
and try again until we get an event or an error because
the channel is closed (in case of recv())
or empty (in case of try_recv()).
If multiple headers with the same name are present,
we must take the last one.
This is the one that is DKIM-signed if
this header is DKIM-signed at all.
Ideally servers should prevent adding
more From, To and Cc headers by oversigning
them, but unfortunately it is not common
and OpenDKIM in its default configuration
does not oversign any headers.
closes#5533
adds the functions that were still missing for migration to jsonrpc (the
ones that the cffi already had, so just should be quick to review ;)
It's actually `deltachat::contact::Contact::authname` by semantics. `display_name`/`name` shouldn't
be shared, it's the name given by the user locally.
Add a function parsing a vCard file at the given path.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: Asiel Díaz Benítez <asieldbenitez@gmail.com>
When there are no parent references,
Delta Chat inserts Message-ID into References.
Such references should be ignored
because otherwise fully downloaded message
may be assigned to the same chat as previously incorrectly assigned
partially downloaded message.
Fixes receive_imf::tests::test_create_group_with_big_msg
Chat-Group-ID always correctly identifies the chat
message was sent to, while In-Reply-To and References
may point to a message that has itself been incorrectly
assigned to a chat.
This protects Bob (the joiner) of sending unexpected-unencrypted messages during an otherwise nicely
running SecureJoin.
If things get stuck, however, we do not want to block communication -- the chat is just
opportunistic as usual, but that needs to be communicated:
1. If Bob's chat with Alice is `Unprotected` and a SecureJoin is started, then add info-message
"Establishing guaranteed end-to-end encryption, please wait..." and let `Chat::can_send()` return
`false`.
2. Once the info-message "Messages are guaranteed to be e2ee from now on" is added, let
`Chat::can_send()` return `true`.
3. If after SECUREJOIN_WAIT_TIMEOUT seconds `2.` did not happen, add another info-message "Could not
yet establish guaranteed end-to-end encryption but you may already send a message" and also let
`Chat::can_send()` return `true`.
Both `2.` and `3.` require the event `ChatModified` being sent out so that UI pick up the change wrt
`Chat::can_send()` (this is the same way how groups become updated wrt `can_send()` changes).
SECUREJOIN_WAIT_TIMEOUT should be 10-20 seconds so that we are reasonably sure that the app remains
active and receiving also on mobile devices. If the app is killed during this time then we may need
to do step 3 for any pending Bob-join chats (right now, Bob can only join one chat at a time).
`ChatId::lookup_by_contact()` returns `None` for blocked chats, so it should be only used if we need
to filter out blocked chats, e.g. when building a chatlist.
E.g. the multi-device synchronisation creates the "Saved Messages" chat as blocked, in this case the
chat icon wasn't updated before and the user avatar was displayed instead.
- [x] figgure out how to build the packages (that it installs native
optional package automatically)
- [X] Make the gluecode
- [x] expose both the lowerlevel api that desktop uses (~~send objects
and receive objects~~, getting path of rpc-server is enough)
- [X] and the higher level api needed for bots (jsonrpc client)
- [X] typescript types
- [x] automatically pick the right binary from npm or allow getting it
from env var, or give out an error (throw error)
- [x] find out how to dev locally (use local built core in dc desktop) -
there is the question of how to link the typescript client and the task
to add a search in the cargo target folder for a debug build or a
different way, find out some good flow that we can use and document for
dc desktop + locally built core development
- [x] build the packages in ci
- [x] fix that deltachat-rpc-server is not executable
postponed:
- [ ] publish from ci
- [ ] add key/token to deploy to npm
Closes#4694
## Related prs
- https://github.com/deltachat-bot/echo/pull/69
- https://github.com/deltachat/deltachat-desktop/pull/3567
---------
Co-authored-by: link2xt <link2xt@testrun.org>
this PR checks if the quotes are used in a reasonable way:
- quoted messages should be send to the same chat
- or to one-to-one chats
if the quote's chat ID is not the same than the sending chat _and_ the
sending chat is not a one-to-one chat, sending is aborted.
usually, the UIs does not allow quoting in other ways, so, this check is
only a "last defence line" to avoid leaking data in case the UI has
bugs, as recently in
https://github.com/deltachat/deltachat-android/issues/3032
@iequidoo @link2xt @adbenitez i am not 100% sure about this PR, maybe
i've overseen a reasonable usecase where other quotes make sense
---------
Co-authored-by: link2xt <link2xt@testrun.org>
Otherwise, e.g. if a message is a large GIF, but its viewtype is set to `Image` by the app, this GIF
will be recoded to JPEG to reduce its size. GIFs and other special viewtypes must be always detected
and sent as is.
Otherwise location-only messages
that should be sent every 60 seconds
are never sent because location loop
waits until the end of location streaming
and is only interrupted by location streaming
ending in other chats or being enabled in other chats.
This is similar to `test_modify_chat_disordered`,
but tests that recovery works in the simplest case
where next message is not modifying group membership.
... and fails if file already exists. The UI should open the file saving dialog, defaulting to
Downloads and original filename, when asked to save the file. After confirmation it should call
dc_msg_save_file().
This way we avoid an immediate retry if the network is not yet ready exhausting the ratelimiter's
quota of two connection attempts. Also notify the ratelimiter only after a successful connection so
that it only limits the server load, but not connection attempts.
Otherwise an error
"The encoder or decoder for Jpeg does not support the color type `Rgba8`"
is returned if image has an alpha channel.
This is caused by the recent change of JPEG encoder
in `image` crate: <https://github.com/image-rs/image/issues/2211>
This makes `EventTracker` receive events immediately
instead of being moved from event emitter to event tracker
by a task spawned from `TestContext::new_internal`.
This makes `EventTracker.clear_events` reliable
as it is guaranteed to remove all events emitted
by the time it is called rather than only events
that have been moved already.
as discussed in several chats, this PR starts making it possible to use
Webxdc as integrations to the main app. In other word: selected parts of
the main app can be integrated as Webxdc, eg. Maps [^1]
this PR contains two parts:
- draft an Webxdc Integration API
- use the Webxdc Integration API to create a Maps Integration
to be clear: a Webxdc is not part of this PR. the PR is about marking a
Webxdc being used as a Map - and core then feeds the Webxdc with
location data. from the view of the Webxdc, the normal
`sendUpdate()`/`setUpdateListener()` is used.
things are still marked as "experimental", idea is to get that in to
allow @adbenitez and @nicodh to move forward on the integrations into
android and desktop, as well as improving the maps.xdc itself.
good news is that we currently can change the protocol between Webxdc
and core at any point :)
# Webxdc Integration API
see `dc_init_webxdc_integration()` in `deltachat.h` for overview and
documentation.
rust code is mostly in `webxdc/integration.rs` that is called by other
places as needed. current [user of the API is
deltachat-ios](https://github.com/deltachat/deltachat-ios/pull/1912),
android/desktop will probably follow.
the jsonrpc part is missing and can come in another PR when things are
settled and desktop is really starting [^2] (so we won't need to do all
iterations twice :) makes also sense, when this is done by someone
actually trying that out on desktop
while the API is prepared to allow other types of integrations (photo
editor, compose tools ...) internally, we currently ignore the type. if
that gets more crazy, we probably also need a dedicated table for the
integrations and not just a single param.
# Maps Integration
rust code is mostly in `webxdc/maps_integration.rs` that is called by
`webxdc/integration.rs` as needed.
EDIT: the idea of having a split here, is that
`webxdc/maps_integration.rs` really can focus on the json part, on the
communication with the .xdc, including tests
this PR is basic implementation, enabling to move forward on
integrations on iOS, but also on desktop and android.
the current implementation allows already the following:
- global and per-chat maps
- add and display POIs
- show positions and tracks of the last 24 hours
the current maps.xdc uses leaflet, and is in some regards better than
the current android/desktop implementations (much faster, show age of
positions, fade out positions, always show names of POIs, clearer UI).
however, we are also not bound to leaflet, it can be anything
> [**screenshots of the current
state**](https://github.com/deltachat/deltachat-ios/pull/1912)
> 👆
to move forward faster and to keep this PR small, the following will go
to a subsequent PR:
- consider allowing webxdc to use a different timewindow for the
location
- delete POIs
- jsonrpc
[^1]: maps are a good example as anyways barely native (see android
app), did cause a lot of pain on many levels in the past (technically,
bureaucratically), and have a comparable simple api
[^2]: only going for jsonrpc would only make sense if large parts of
android/ios would use jsonrpc, we're not there
---------
Co-authored-by: link2xt <link2xt@testrun.org>
As trashed messages can't be loaded from the db now by `Message::load_from_db()` returning an error
for such messages, errors from `Message::load_from_db()` should be logged as warnings. If it's
really an error like a db failure, it should be logged internally.
As trashed messages can't be loaded from the db now, `get_message_by_id()` returns None in some
tests e.g. in `test_deleted_msgs_dont_reappear()`. A `PerAccount` hook shouldn't be called if so.
There's no need to load an updated message state from the db to implement `is_outgoing()` and also
this function is implicitly called in some tests where a message is already trashed and a call to
`dc_get_msg()` generates an unexpected error.
Previously test did not trigger
deletion of ephemeral messages
and worked because clear_events() did not
remove just emitted events from `send_text_msg`.
It keeps timing out with the default timeout of 2 s on macOS runners.
Also fix comment in the integration test which
said that timeout is 1 minute but sets it to 3 minutes.
Set this timeout to 5 minutes as well.
API now pretends that trashed messages don't exist.
This way callers don't have to check if loaded message
belongs to trash chat.
If message may be trashed by the time it is attempted to be loaded,
callers should use Message::load_from_db_optional.
Most changes are around receive_status_update() function
because previously it relied on loading trashed status update
messages immediately after adding them to the database.
This way nightly clippy warnings are not generated
when devshell is used.
Nighly Rust is also not cached, e.g. rust-analyzer has to be rebuilt
if version from fenix is used.
This fixes things for Gmail f.e. Before, `Imap::fetch_move_delete()` was called before looking for
Trash and returned an error because of that failing the whole `fetch_idle()` which prevented
configuring Trash in turn.
`a.clone_from(&b)` is equivalent to `a = b.clone()` in functionality,
but can be overridden to reuse the resources of a to avoid unnecessary
allocations.
When the env var CARGO_BUILD_TARGET is set, cargo will crossbuild for the given arch triplet. In this case, the targets will not be put into target/release/, but target/$CARGO_BUILD_TARGET/release/. Add this subdirectory, if neccessary.
If `delete_device_after` is configured, that period should be counted for webxdc instances from the
last status update, otherwise nothing prevents from deleting them. Use `msgs.timestamp_rcvd` to
store the last status update timestamp, it anyway isn't used for anything except displaying a
detailed message info. Also, as `ephemeral::select_expired_messages()` now also checks
`timestamp_rcvd`, we have an improvement that a message is guaranteed not to be deleted for the
`delete_device_after` period since its receipt. Before only the sort timestamp was checked which is
derived from the "sent" timestamp.
Let's add a 1-minute tolerance to `Params::MemberListTimestamp`.
This adds to the group membership consistency algo the following properties:
- If remote group membership changes were made by two members in parallel, both of them are applied,
no matter in which order the messages are received.
- If we remove a member locally, only explicit remote member additions/removals made in parallel are
allowed, but not the synchronisation of the member list from "To". Before, if somebody managed to
reply earlier than receiving our removal of a member, we added it back which doesn't look good.
This change avoids the race between
`provide_backup` changing the state from NoProvider to Pending
and a call to `get_backup_qr` or `get_backup_qr_svg`.
With this change `get_backup_qr` and `get_backup_qr_svg`
always block until QR code is available,
even if `provide_backup` was not called yet.
- Always emit `ContactsChanged` from `contact::update_last_seen()` if a contact was seen recently
just for simplicity and symmetry with `RecentlySeenLoop::run()` which also may emit several events
for single contact.
- Fix sleep time calculation in `RecentlySeenLoop::run()` -- `now` must be updated on every
iteration, before the initial value was used every time which led to progressively long sleeps.
Instead, look up the 1:1 chat in `receive_imf::add_parts()`. This is a more generic approach to fix
assigning outgoing reactions to 1:1 chats in the multi-device setup. Although currently both
approaches give the same result, this way we can even implement a "react privately"
functionality. Maybe it sounds useless, but it seems better to have less reaction-specific code.
we checked for tombstones already using `is_trash()`,
however, we've overseen that tombstones get deleted at some point :)
therefore, just do not treat loading failures of the weak msg_id as errors -
usually, they are not - and if, just the normal summary is shown.
in theory, we could check for existance explicitly before tryong load_from_db,
however, that would be additional code (and maybe another database call)
and not worth the effort.
anyways, this commit also adds an explicit test
for physical deletion after housekeeping.
use get_summary_text_without_prefix() to get raw summaries without prefixes,
this is needed for reaction summaries,
where we also do not show the name, so we do not want to show "Forwarded" as well.
shows the last reaction in chatlist's summaries if there is no
newer message.
the reason to show reactions in the summary, is to make them a _little_
more visible when one is not in the chat. esp. in not-so-chatty or in
one-to-ones chats this becomes handy: imaging a question and someone
"answers" with "thumbs up" ...
otoh, reactions are still tuned down on purpose: no notifications, chats
are opend as usual, the chatlist is not sorted by reactions and also the
date in the summary refer to the last message - i thought quite a bit
about that, this seems to be good compromise and will raise the fewest
questions. it is somehow clear to the users that reactions are not the
same as a real message. also, it is comparable easy to implement - no
UI changes required :)
all that is very close to what whatsapp is doing (figured that out by
quite some testing ... to cite @adbenitez: if in doubt, we can blame
whatsapp :)
technically, i first wanted to go for the "big solution" and add two
more columns, chat_id and timestamp, however, it seemed a bit bloated if
we really only need the last one. therefore, i just added the last
reaction information to the chat's param, which seems more performant
but also easier to code :)
`is_probably_private_reply()` checks
if a message should better go to the one to one chat
instead of the chat already identified by the `In-Reply-To`.
this functionality is needed to make "Reply Privately" work.
however, this functionality is never true for reactions.
if we would return `true` here, own reactions seen by a second device
would not get the correct chat assiged.
We update version in several .toml and .json files
on every release anyway, so updating pyproject.toml is easy.
setuptools_scm makes it more difficult to build
python packages for software distributions
because it requires full git checkout
with tags rather than just a worktree.
It is also possible to remove or move tags
after the release, so git revision no longer
pins python package version if setuptools_scm is used.
- the `jobs` table is no longer in use,
no need to track files on housekeeping,
no need to clear it from repl tool
- some `Params` were used for jobs table only,
they can be used freely for other purposes on other tables.
param 'protection settings timestamp' was never used in practise,
its code is removed as well, so we can free the Param as well.
it was used by iOS to know when a background fetch was complete;
meanwhile the superiour `dc_accounts_background_fetch()` is used for
that.
there is still the corresponding context function `dc_all_work_done()`,
this not used by any UI as well, however, it is in use by a python
tests.
not sure, what to do with it, at a first glance, the test still seems
useful.
Don't attach selfavatar in "v{c,g}-request" and "v{c,g}-auth-required" messages:
- These messages are deleted right after processing, so other devices won't see the avatar.
- It's also good for privacy because the contact isn't yet verified and these messages are auto-sent
unlike usual unencrypted messages.
the rendered time output seems different on different systems and timezomes,
eg. on my local machine, it is
`Sent: 2024.03.20 10:00:01 ` and not `Sent: 2024.03.20 09:00:01`,
maybe because of winter/summer time, idk.
as the gist of the test is to check the name,
however, i just removed the whole time check.
Do not include oldest reference, because chat members
which have been added later and have not seen the first message
do not have referenced message in the database.
Instead, include up to 3 recent Message-IDs.
If user specifies build directory, respect this
instead of building inside the target/ directory.
Also remove outdated comment about Rust 1.50.0 support.
Current MSRV is 1.70.0 and we use 2021 edition
which implies version 2 resolver.
This is a test reproducing the problem
in <https://github.com/deltachat/deltachat-core-rust/issues/5339>.
Fix would be to avoid reordering on the server side,
so the test checks that the unverified message
is replaced with a square bracket error
as expected if messages arrive in the wrong order.
Add white background instead of the default black one to avatars when recoding to JPEG. But not for
"usual" images to spare the CPU. The motivation is to handle correctly
"black-on-transparent-background" avatars which are quite common.
as turned out on recent researches wrt a slow database, setting
`save_mime_headers` will result in 25x larger databases.
this is definetely something we want to know at least in the debug info.
(the option cannot be enabled in current UIs,
however, esp. devs find options to set this.
apart from that, save_mime_headers is only used in some python tests)
ftr: the huge database was more than 10 times slower, leading to missing
notifications on ios (as things do not finish within the few seconds iOS
gave us).
moreover, the hugeness also avoids exporting; this is fixed by
https://github.com/deltachat/deltachat-core-rust/pull/5349
this PR fixes one of the issues we had with an (honestly accidentally)
huge database of >2gb.
this database could not be exported on iOS, as ram memory is limited
there, leading to the app just crashing.
it is unclear if that would be better on eg. Android, however,
temp_store=FILE is not working as well there, this is the smaller
drawback there.
before merging, this PR should be tested on time with export/import on
iOS (export/import uses VACUUM which uses /tmp) EDIT: did so, works on
iphone7 with ios15
`Param::MemberListTimestamp` was updated only from `receive_imf::apply_group_changes()` i.e. for
received messages. If we sent a message, that timestamp wasn't updated, so remote group membership
changes always overrode local ones. Especially that was a problem when a message is sent offline so
that it doesn't incorporate recent group membership changes.
When scheduler is destroyed, e.g. during a key import,
there is some time between destroying the interrupt channel
and the loop task.
To avoid busy looping, tasks should terminate if
receiving from the interrupt loop fails
instead of treating it as the interrupt.
An unencrypted message with already known Autocrypt key, but sent from another address, means that
it's rather a new contact sharing the same key than the existing one changed its address, otherwise
it would already have our key to encrypt.
If Subject is multiline-formatted, `mailparse` adds the leading whitespace to it. The solution is to
always remove the leading whitespace, because if Subject isn't multiline-formatted, it never
contains the leading whitespace anyway. But as for the trailing whitespace -- i checked -- it's
never removed, so let's keep this as is.
This fixes the following identity-misbinding attack:
It appears that Bob’s messages in the SecureJoin protocol do not properly “bind” to Alice’s public
key or fingerprint. Even though Bob’s messages carry Alice’s public key and address as a gossip in
the protected payload, Alice does not reject the message if the gossiped key is different from her
own key. As a result, Mallory could perform an identity-misbinding attack. If Mallory obtained
Alice’s QR invite code, she could change her own QR code to contain the same tokens as in Alice’s QR
code, and convince Bob to scan the modified QR code, possibly as an insider attacker. Mallory would
forward messages from Bob to Alice and craft appropriate responses for Bob on his own. In the end,
Bob would believe he is talking to Mallory, but Alice would believe she is talking to Bob.
Secure-Join-Group is only expected by old core in vg-request-with-auth.
There is no reason to leak group ID in unencrypted vg-request.
Besides that, Secure-Join-Group is deprecated
as Alice knows Group ID corresponding to the auth code,
so the header can be removed completely eventually.
If a message is sent from SELF, but signed with a foreign key, it mustn't be considered
Autocrypt-encrypted and shown with a padlock. Currently this is broken.
Compilation of vendored OpenSSL inside Nix is broken since
<https://github.com/alexcrichton/openssl-src-rs/pull/229> due to build
script changes.
There is anyway no need to compile vendored OpenSSL as nixpkgs already
contains OpenSSL package.
This fixes `nix build .#deltachat-rpc-server-x86_64-linux` and similar
commands which are used during releases.
The way it was implemented it threw out all remaining messages after finding the next incoming
message. Better use FFIEventTracker functions, they are used in all the tests anyway.
Connection establishment now happens only in one place in each IMAP loop.
Now all connection establishment happens in one place
and is limited by the ratelimit.
Backoff was removed from fake_idle
as it does not establish connections anymore.
If connection fails, fake_idle will return an error.
We then drop the connection and get back to the beginning of IMAP
loop.
Backoff may be still nice to have to delay retries
in case of constant connection failures
so we don't immediately hit ratelimit if the network is unusable
and returns immediate error on each connection attempt
(e.g. ICMP network unreachable error),
but adding backoff for connection failures is out of scope for this change.
ratelimit can be exhausted quickly if the network is not available,
i.e. if every connection attempt returns "network unreachable" error.
When the network becomes available, we want to retry connecting
as soon as maybe_network is called without waiting for ratelimiter.
Before group avatar was sent as an attachment. Let's do the same as with user avatar and send group
avatar as base64. Receiver code uses the same functions for user and chat avatars, so base64 avatars
are supported for most receivers already.
`deltachat-rpc-server` releases are built
with Nix and LLVM/clang toolchains now
that fully support atomics.
Zig toolchain that required disabling atomics
and resulted in problems with OpenSSL 3.2 releases
is not used anymore.
It was broken completely and before "fix: apply Autocrypt headers if timestamp is unchanged" that
didn't show up because the message from the second Bob's device never had "Date" greater than one
from the message sent before from the first device.
In this case connection failure
may be a connection timeout (currently 1 minute),
so it does not make sense to fake idle for another minute immediately after.
However, failure may be immediate if the port is closed
and the server refuses connection every time.
To prevent busy loop in this case
we apply ratelimit to connection attempts rather than login attempts.
This partially reverts ccec26ffa7
Even if 1:1 chat with alice is protected,
we should send vc-request unencrypted.
This happens if Alice changed the key
and QR-code Bob scans contains fingerprint
that is different from the verified fingerprint.
Sending vc-request encrypted to the old key
does not help because Alice is not able
to decrypt it in this case.
Use sync messages for that as it is done for e.g. Config::Displayname. Maybe we need to remove
self-status synchronisation via usual messages then, but let's think of it a bit.
Use sync messages for that as it is done for e.g. Config::Displayname. Maybe we need to remove
avatar synchronisation via usual messages then, but let's think of it a bit.
Encryption subkey is incorrectly referred to as public key
in variable names.
This is incorrect because generated encryption key
is secret too just as the signing primary key.
Generated OpenPGP secret key consists of primary signing key
and encryption subkey.
Then OpenPGP public key consisting of
the primary signing public key
and encryption public key is generated.
Keypair consists of the secret OpenPGP key and public OpenPGP key,
each of them has a primary key and subkey inside.
Also disable --progress.
It is not disabled by default for backward compatibility,
but solves the problem of lots of progress lines
in the downloadable raw output:
https://github.com/actions/checkout/pull/1067
The limit is better enforced by webxdc distributors,
e.g. xdc store bots or actually email providers
to allow for experimentation with large frameworks
or porting existing apps and testing them
before reducing their size.
Besides that, the comment on WEBXDC_SENDING_LIMIT was outdated,
it was not updated when the limit was increased to 640 kB.
Even if `vc-request-with-auth` is received with a delay, the protection message must have the sort
timestamp equal to the Sent timestamp of `vc-request-with-auth`, otherwise subsequent chat messages
would also have greater sort timestamps and while it doesn't affect the chat itself (because Sent
timestamps are shown to a user), it affects the chat position in the chatlist because chats there
are sorted by sort timestamps of the last messages, so the user sees chats sorted out of
order. That's what happened in #5088 where a user restores the backup made before setting up a
verified chat with their contact and fetches new messages, including `vc-request-with-auth` and also
messages from other chats, after that.
Add a new crate `deltachat_time` with a fake `struct SystemTimeTools` for mocking
`SystemTime::now()` for test purposes. One still needs to use `std::time::SystemTime` as a struct
representing a system time. I think such a minimalistic approach is ok -- even if somebody uses the
original `SystemTime::now()` instead of the mock by mistake, that could break only tests but not the
program itself. The worst thing that can happen is that tests using `SystemTime::shift()` and
checking messages timestamps f.e. wouldn't catch the corresponding bugs, but now we don't have such
tests at all which is much worse.
after merging:
6. Tag the release: `git tag -a v1.135.0`.
7. Push the release tag: `git push origin v1.135.0`.
8. Create a GitHub release: `gh release create v1.135.0 -n ''`.
---------
Co-authored-by: link2xt <link2xt@testrun.org>
PR #5099 removed some columns in the database that were actually in use.
usually, to not worsen UX unnecessarily
(releases take time - in between, "Add Second Device", "Backup" etc.
would fail), we try to avoid such schema changes (checking for
db-version would avoid import etc. but would still worse UX),
see discussion at #2294.
these are the errors, the user will be confronted with otherwise:
<img width=400
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/e3f0fd6e-a7a9-43f6-9023-0ae003985425>
it is not great to maintain the old columns, but well :)
as no official releases with newer cores are rolled out yet, i think, it
is fine to change the "107" migration
and not copy things a second time in a newer migration.
(this issue happens to me during testing, and is probably also the issue
reported by @lk108 for ubuntu-touch)
The system clock may be adjusted and even go back, so caching system time in code sections where
it's not supposed to change may even protect from races/bugs.
If a time value doesn't need to be sent to another host, saved to the db or otherwise used across
program restarts, a monotonically nondecreasing clock (`Instant`) should be used. But as `Instant`
may use `libc::clock_gettime(CLOCK_MONOTONIC)`, e.g. on Android, and does not advance while being in
deep sleep mode, get rid of `Instant` in favor of using `SystemTime`, but add `tools::Time` as an
alias for it with the appropriate comment so that it's clear why `Instant` isn't used in those
places and to protect from unwanted usages of `Instant` in the future. Also this can help to switch
to another clock impl if we find any.
Restart the IO scheduler if needed to make the new config value effective (for `MvboxMove,
OnlyFetchMvbox, SentboxWatch` currently). Also add `set_config_internal()` which doesn't affect
running the IO scheduler. The reason is that `Scheduler::start()` itself calls `set_config()`,
although not for the mentioned keys, but still, and also Rust complains about recursive async calls.
Currently when a user sets up another device by logging in, a new key is created. If a message is
sent from either device outside, it cannot be decrypted by the other device.
The message is replaced with square bracket error like this:
```
<string name="systemmsg_cannot_decrypt">This message cannot be decrypted.\n\n• It might already help to simply reply to this message and ask the sender to send the message again.\n\n• If you just re-installed Delta Chat then it is best if you re-setup Delta Chat now and choose "Add as second device" or import a backup.</string>
```
(taken from Android repo `res/values/strings.xml`)
If the message is outgoing, it does not help to "simply reply to this message". Instead, we should
add a translatable device message of a special type so UI can link to the FAQ entry about second
device. But let's limit such notifications to 1 per day. And as for the undecryptable message
itself, let it go to Trash if it can't be assigned to a chat by its references.
Before it was emitted only on the source device and `test_sync()` didn't catch the bug because of
the interference of the previous call to `set_config_bool()`. Now there's a separate test that does
its job well.
Before moving emails to the mvbox we need to remember its UIDVALIDITY, otherwise emails moved before
that wouldn't be fetched but considered "old" instead.
Also:
- Don't use `session.create()` to create mvbox as `select_with_uidvalidity()` already creates mvbox
on its own.
- Don't try to create compat folders like "INBOX.DeltaChat", but only look for them.
If `escaper::decode_html_buf_sloppy()` just truncates the text (which happens when it fails to
html-decode it at some position), then it's probably not HTML at all and should be left as
is. That's what happens with hyperlinks f.e. and there was even a test on this wrong behaviour which
is fixed now. So, now hyperlinks are not truncated in messages and should work as expected.
If a Delta Chat message has the Message-ID already existing in the db, but a greater "Date", it's a
resent message that can be deleted. Messages having the same "Date" mustn't be deleted because they
can be already seen messages moved back to INBOX. Also don't delete messages having lesser "Date" to
avoid deleting both messages in a multi-device setting.
Use `create_smeared_timestamp()` for this. This allows to dedup messages on the receiver -- if it
sees the same Message-ID, but a different timestamp, then it's a resent message that can be deleted.
"Auto-Submitted: auto-replied" messages mustn't be considered as sent by either bots or non-bots,
e.g. MDNs have this header value and it's the same for bots and non-bots.
In particular TLSRPT reports
contain files that may be interesting for admins.
Currently Delta Chat drops the attachment
so message appears as a text message without actual payload.
@adbenitez wants this feature on Deltalab to display a bot tag.
Other UIs might also want to adopt this feature :)
---------
Co-authored-by: link2xt <link2xt@testrun.org>
If we wait for inviter success,
vg-member-added message may be still in flight
and reach ac2 after device resetup.
Making ac2 wait for joining the group ensures that old
device receives vg-member-added message
and new device will not receive it and fail to decrypt.
Other instances of wait_for_securejoin_inviter_success()
in the same tests are also replaced for reliability.
Previously the message was removed from `download` table,
but message bubble was stuck in InProgress state.
Now download state is updated by the caller,
so it cannot be accidentally skipped.
This allows to send existing messages (incoming and outgoing) taken from encrypted chats, to
unencrypted ones. `Param::ForcePlaintext` is removed as well -- if a message can be sent encrypted
this time, nothing bad with this.
Currently `Chat.send_msg()` modifies the source message and returns another message object
equivalent to the source one. That's how it works in the core and thus in Python bindings too.
This is needed to protect from ESPs (such as gmx.at) doing their own Quoted-Printable encoding and
thus breaking messages and signatures. It's unlikely that the reader uses a MUA not supporting
Quoted-Printable encoding. And RFC 2646 "4.6" also recommends it for encrypted messages.
Trying non-strict TLS checks is not necessary
for most servers with proper TLS setup,
but doubles the time needed to fail configuration
when the server is not responding, e.g.
when all connection attempts time out.
There is also a risk of accidentally
configuring non-strict TLS checks in a rare case
that strict TLS check configuration spuriously failed,
e.g. on a bad network.
If the server has a known broken TLS setup,
it can still be added to the provider database
or configured with non-strict TLS check manually.
User can also configure another email provider,
such as chatmail servers, instead of using the server
with invalid TLS hostname.
This change does not affect exising setups.
deltachat.h uses `@defgroup` commands to create topics
for groups of constants. Prior to Doxygen 1.9.8
defining a group created a "module"
and all constants were visible from the modules.html page.
In Doxygen 1.9.8 "modules" were renamed into "topics"
as C++20 modules have taken their place,
so Delta Chat documentation does not have modules
in Doxygen sense anymore.
The change is to replace "modules.html" with "topics.html"
in the DoxygenLayout.xml.
See <https://www.doxygen.nl/manual/grouping.html> for
Doxygen documentation about groups and their relation to topics.
This PR stops MDNs from being forced to be sent unencrypted.
If no encryption is possible (by `should_encrypt`), the fix#5152 still
applies.
close#5168
Send `EventType::ConnectivityChanged` when using the context methods
`start_io` and `stop_io`.
close#5097
---------
Co-authored-by: Septias <scoreplayer2000@gmail.comclear>
Add an event for a case if a multi-device synced config value changed. Maybe the app needs to
refresh smth on such an event. For uniformity it is emitted on the source device too. The value is
omitted, otherwise it would be logged which might not be good for privacy.
Mark 1:1 chat as verified as soon as Alice is forward-verified
so Bob can already start sending Chat-Verified headers.
This way Alice and Bob can scan each other's QR codes
and even if all Secure-Join headers are dropped from the network,
still get forward verifications via QR-code scans
and backward verifications via Chat-Verified messages in 1:1 chat.
Before, while a message is in OutPending state after resending is requested, the user still sees the
red marker with error and it is confusing, so the user don't know the sending state of the message.
Before this fix actual contents of the message
reposted by Schleuder is considered a mailing list footer and removed,
not visible even in the "Show Full Message..." view.
With this change there will be two message bubbles,
one for header and one for the contents,
but it is still better than losing the contents completely.
Attempting to parse header part is out of scope for this change.
This is broken since 44227d7b86
mimeparser only recognizes read receipts
by the Content-Type being "multipart/report".
If multipart/report is hidden inside multipart/mixed
and the message is not encrypted,
it degrades encryption.
a27e84ad89 "fix: Delete received outgoing messages from SMTP queue"
can break sending messages sent as several SMTP messages because they have a lot of recipients:
`pub(crate) const DEFAULT_MAX_SMTP_RCPT_TO: usize = 50;`
We should not cancel sending if it is such a message and we received BCC-self because it does not
mean the other part was sent successfully. For this, split such messages into separate jobs in the
`smtp` table so that only a job containing BCC-self is canceled from `receive_imf_inner()`. Although
this doesn't solve the initial problem with timed-out SMTP requests for such messages completely,
this enables fine-grained SMTP retries so we don't need to resend all SMTP messages if only some of
them failed to be sent.
The bug was made in 44227d7b86. Sql::execute() with placeholders must
be used to escape strings, one never should escape them manually as strings themselves can contain
escape symbols. Thanks to @link2xt for noticing.
Some SMTP servers are running slow before-queue filters, most commonly Postfix with `rspamd` filter
which is implemented as a [before-queue Milter](https://www.postfix.org/MILTER_README.html). Some of
`rspamd` plugin filters are slow on large mails.
We previously had problems with timing out during waiting for SMTP response:
https://github.com/deltachat/deltachat-core-rust/issues/1383. This is largely fixed by
https://github.com/async-email/async-smtp/pull/29 and currently we have 60-second timeout just for
reading a response but apparently it is not sufficient -- maybe connection gets killed by NAT while
we are waiting for response or `rspamd` takes more than 60 seconds for large messages.
As a result a message is resent multiple times and eventually fails with "too many retries" while
multiple BCC-self messages are received.
We should remove the message from the SMTP queue as soon as we receive it via IMAP as it is clear
the message was sent even if we did not manage to get actual SMTP server response.
First of all, it's just downloaded and hasn't been seen yet by the user. Also this changes nothing
as `msgs.state` isn't changed when replacing a message anyway.
Put a copy of Message-ID into hidden headers and prefer it over the one in the IMF header section
that servers mess up with.
This also reverts "Set X-Microsoft-Original-Message-ID on outgoing emails for amazonaws (#3077)".
As ratelimit was introduced to avoid reconnecting immediately after disconnecting
in case of bugs in IMAP protocol handling,
connection attempts should only be counted when IMAP is actually used,
i.e. when the first command (LOGIN) is sent.
As per the comment in `receive_imf.rs`, `chat.protected` must be maintained regardless of the
`Config::VerifiedOneOnOneChats`. The only thing that mustn't be done if `VerifiedOneOnOneChats` is
unset (i.e. for non-supporting UIs) is marking chats as "protection broken" because this needs
showing the corresponding dialog to a user.
Drafts mustn't affect sorting of any other messages, they aren't even displayed in the chat
window. Also hidden messages mustn't affect sorting of usual messages. But let hidden messages sort
together with protection messages because hidden messages also can be or not be verified, so let's
preserve this information -- even it's not useful currently, it can be useful in the future
versions.
Before in some places it was correctly calculated by passing the "sent" timestamp to
`calc_sort_timestamp()`, but in other places just the system time was used. In some complex
scenarios like #5088 (restoration of a backup made before a contact verification) it led to wrong
sort timestamps of protection messages and also messages following by them.
But to reduce number of args passed to functions needing to calculate the sort timestamp, add
message timestamps to `struct MimeMessage` which is anyway passed everywhere.
We do not want all_work_done() to return true immediately
after calling start_io(), but only when connection goes idle.
"Connected" state is set immediately after connecting to the server,
but it does not mean there is nothing to do.
This change make all_work_done() return false
from the Connected state and introduces a new Idle
connectivity state that is only set before connection
actually goes idle. For idle state all_work_done() returns true.
From the user point of view both old Connected state
and new Idle state look the same.
This change depends on async-imap update that resets the timeout
every time an `* OK Still here` is received.
Reducing timeout allows to detect lost connections
not later than 6 minutes
because Delta Chat will attempt to finish IDLE with DONE
after 5 minutes without keepalives
and will either get TCP RST directly
or, worst case, wait another minute for TCP socket read timeout.
Ad-hoc groups don't have grpid-s that can be used to identify them across devices and thus wasn't
synced until now.
The same problem already exists for assigning messages to ad-hoc groups and this assignment is done
by `get_parent_message()` and `lookup_chat_by_reply()`. Let's reuse this logic for the
synchronisation, it works well enough and this way we have less surprises than if we try to
implement grpids for ad-hoc groups. I.e. add an `Msgids` variant to `chat::SyncId` analogous to the
"References" header in messages and put two following Message-IDs to a sync message:
- The latest message A having `DownloadState::Done` and the state to be one of `InFresh, InNoticed,
InSeen, OutDelivered, OutMdnRcvd`.
- The message that A references in `In-Reply-To`.
This way the logic is almost the same to what we have in `Chat::prepare_msg_raw()` (the difference
is that we don't use the oldest Message-ID) and it's easier to reuse the existing code.
NOTE: If a chat has only an OutPending message f.e., the synchronisation wouldn't work, but trying
to work in such a corner case has no significant value and isn't worth complicating the code.
- Use TestContextManager
- Actually run receive_imf rather than only mimeparser on "received" messages
- Check that received message parts actually have a padlock
parse_mime_recursive() skips empty text parts,
so there may be no parts as the result of parsing.
In this case an empty part is added.
However, because it is added with parts.push()
rather than add_single_part(),
it is added without a padlock even if the message is encrypted.
`do_add_single_part()` adds padlock (GuaranteeE2EE param)
and should be used to add parts instead.
This test reproduces a bug preventing joining the group with a QR code
if the group already has a contact with inconsistent key state,
which means both Autocrypt and verified key exist,
but don't match.
This can happen when an invite QR code created by this contact
is scanned as scanning an invite code creates unprotected group
with the inviter for info messages.
If securejoin process never finishes because the inviter is offline,
group remains in this unprotected state with added inviter.
Normally the group becomes verified when a "Member added" (vg-member-added)
message is received in the chat.
However, current code checks that all members
of the chat are verified
(have a green checkmark, use verified key in 1:1 chat)
before marking the group as verified and fails otherwise.
- Remove "Detected Autocrypt-mime message" logs printed for every incoming Autocrypt message.
- Print only a single line at the beginning of receive_imf with both the Message-ID and seen flag.
- Print Securejoin step only once, inside handle_securejoin_handshake or observe_securejoin_on_other_device.
- Do not log "Not creating ad-hoc group" every time ad-hoc group is not created, log when it is created instead.
- Log ID of the chat where Autocrypt-Gossip for all members is received.
- Do not print "Secure-join requested." for {vg,vc}-request, we already log the step.
- Remove ">>>>>>>>>>>>>>>>>>>>>>>>>" noise from securejoin logs.
Otherwise it looks like the message creating a protected group is not verified. For this, use
`sent_timestamp` of the received message as an upper limit of the sort timestamp (`msgs.timestamp`)
of the protection message. As the protection message is added to the chat earlier, this way its
timestamp is always less or eq than the received message's timestamp.
Test that on the second device of a protected group creator the first message is
`SystemMessage::ChatProtectionEnabled` and the second one is the message populating the group.
Allowing outgoing unencrypted messages in groups with 2 members
breaks the test
`python/tests/test_0_complex_or_slow.py::test_verified_group_vs_delete_server_after`
This centralizes all Securejoin/verification checks and updates in one
place right before add_parts() even before we assign the message to
the chat, so we can decouple chat logic from verification logic.
It's not necessary and in other places like add_contact_to_chat_ex() sync messages are also sent
only if there are no system messages sent like MemberAddedToGroup.
We already synchronise status/footer when we see a self-sent message with a Chat-Version
header. Would be nice to do the same for display name.
But let's do it the same way as for `Config::{MdnsEnabled,ShowEmails}`. Otherwise, if we sync the
display name using the "From" header, smth like `Param::StatusTimestamp` is needed then to reject
outdated display names. Also this timestamp needs to be updated when `Config::Displayname` is set
locally. Also this wouldn't work if system time isn't synchronised on devices. Also using multiple
approaches to sync different config values would lead to more code and bugs while having almost no
value -- using "From" only saves some bytes and allows to sync some things w/o the synchronisation
itself to be enabled. But the latter also can be a downside -- if it's usual synchonisation, you can
(potentially) disable it and share the same email account across people in some organisation
allowing them to have different display names. With using "From" for synchronisation such a
capability definitely requires a new config option.
Motivation: Syncing these options will improve UX in very most cases and should be done. Other
candidates are less clear or are advanced options, we can reconsider that at some point later.
Approach:
- Sync options one-by-one when the corresponding option is set (even if to the same value).
- Don't sync when an option is reset to a default as defaults may differ across client versions.
- Check on both sides that the option should be synced so that if there are different client
versions, the synchronisation of the option is either done or not done in both directions.
Moreover, receivers of a config value need to check if a key can be synced because some settings
(e.g. Avatar path) could otherwise lead to exfiltration of files from a receiver's device if we
assume an attacker to have control of a device in a multi-device setting or if multiple users are
sharing an account.
- Don't sync `SyncMsgs` itself.
If a message is encrypted, but unsigned:
- Don't set `MimeMessage::from_is_signed`.
- Remove "secure-join-fingerprint" and "chat-verified" headers from `MimeMessage`.
- Minor: Preserve "Subject" from the unencrypted top level if there's no "Subject" in the encrypted
part, this message is displayed w/o a padlock anyway.
Apparently it didn't lead to any vulnerabilities because there are checks for
`MimeMessage::signatures.is_empty()` in all necessary places, but still the code looked dangerous,
especially because `from_is_singed` var name didn't correspond to its actual value (it was rather
`from_is_encrypted_maybe_signed`).
Currently this leads to
DEBUG root:rpc.py:136 account_id=1 got an event {'kind': 'Warning', 'msg': 'src/scheduler.rs:711: send_smtp_messages failed: failed to send message: failed to update retries count: database is locked: Error code 5: The database file is locked'}
and
FAILED tests/test_webxdc.py::test_webxdc_insert_lots_of_updates - deltachat_rpc_client.rpc.JsonRpcError: {'code': -1, 'message': 'database is locked\n\nCaused by:\n Error code 5: The database file is locked'}
Without this change
when the test returns a `Result`, `cargo test` does not show
the line number.
To make the tests as easy to debug as the panicking tests,
enable `backtrace` feature on `anyhow` and add debug information
to add source line numbers to backtraces.
Now running `RUST_BACKTRACE=1 cargo test` produces backtraces
with the line numbers. For example:
Error: near ",": syntax error in SELECT COUNT(,,*) FROM msgs_status_updates; at offset 13
Caused by:
Error code 1: SQL error or missing database
Stack backtrace:
0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/result.rs:1962:27
1: deltachat::sql::Sql::query_row::{{closure}}::{{closure}}
at ./src/sql.rs:466:23
2: deltachat::sql::Sql::call::{{closure}}::{{closure}}
at ./src/sql.rs:379:55
3: tokio::runtime::context::runtime_mt::exit_runtime
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/context/runtime_mt.rs:35:5
4: tokio::runtime::scheduler::multi_thread::worker::block_in_place
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/scheduler/multi_thread/worker.rs:438:9
5: tokio::runtime::scheduler::block_in_place::block_in_place
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/scheduler/block_in_place.rs:20:5
6: tokio::task::blocking::block_in_place
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/task/blocking.rs:78:9
7: deltachat::sql::Sql::call::{{closure}}
at ./src/sql.rs:379:19
8: deltachat::sql::Sql::query_row::{{closure}}
at ./src/sql.rs:469:10
9: deltachat::sql::Sql::count::{{closure}}
at ./src/sql.rs:443:76
10: deltachat::webxdc::tests::change_logging_webxdc::{{closure}}
at ./src/webxdc.rs:2644:18
11: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/future/future.rs:125:9
12: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/park.rs:282:63
13: tokio::runtime::coop::with_budget
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/coop.rs:107:5
tokio::runtime::coop::budget
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/coop.rs:73:5
tokio::runtime::park::CachedParkThread::block_on
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/park.rs:282:31
14: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/context/blocking.rs:66:9
...
Line 10 of the backtrace contains the line number in the test (2644).
Limit the number of IMAP connections to 1 per minute regardless of the reason of reconnection, but
allow one immediate retry. This is more reliable than ratelimiting only in error conditions because
ratelimiting can't be skipped by mistake. Anyway connections shouldn't be frequent in normal
operation mode.
send_webxdc_status_update JSON-RPC call
and corresponding Rust call sometimes fail in CI with
---
database is locked
Caused by:
Error code 5: The database file is locked
---
Adding more context to send_webxdc_status_update() errors
to better localize the error origin.
Otherwise when connection is lost IMAP may get into infinite loop
trying to parse remaining bytes:
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
11-21 18:00:48.442 14858 12946 W DeltaChat: src/imap.rs:1457: Failed to process IMAP FETCH result: io: bytes remaining in stream.
Returning an error bubbles it up to `fetch_idle()`
which will call `trigger_reconnect()` and drop the connection.
To preview the docs, run:
```
scripts/build-python-docs.sh
firefox dist/html/index.html
```
I have removed the Makefile because modern Sphinx Makefile is just a
wrapper for `sphinx-build -M`:
3596590317/sphinx/templates/quickstart/Makefile.new_t
and sphinx-quickstart even has an option `--no-makefile`.
`make.bat` makes even less sense.
In `scripts/build-python-docs.sh` I use `sphinx-build` directly
without `make` wrapper.
Email addresses should generally be compared case-insensitively,
but there may be errors in comparison code.
To reduce the chance of problems, encode addresses
in Autocrypt and Autocrypt-Gossip in lowercase
to avoid propagating uppercase characters over the network
to other accounts potentially running buggy code.
If configured address is `Bob@example.net`,
but the message arrives adding `bob@example.net`,
Bob's device should still recognize it as addition of self
and fully recreate the group.
If the sender of the message in protected group chat
is not a member of the chat, mark the sender name with `~`
as we do it in non-protected chats and set the error
instead of replacing the whole message with
"Unknown sender for this chat. See 'info' for more details."
To send a message to a protected group this way
the sender needs to know the group ID
and sign the message with the current verified key.
Usually this is just a late message
delivered shortly after the user has left
the group or was removed from it.
Replacing the message with a single error text part
as done before this change makes it impossible
to access anything other than text, such as attached images.
Other devices should get the same chat name as the currently used device, i.e. the name a user sees
after renaming the chat. This fix is minor because `improve_single_line_input()` logic isn't going
to change often, but still, and also it simplifies the code.
This was introduced in https://github.com/deltachat/deltachat-core-rust/pull/4685
for internal use when looking up a chat based on In-Reply-To,
but actually affects the UIs as they should not display Download button
when the message is downloaded but cannot be decrypted.
If verified key for a contact is changed via securejoin,
gossip the keys in every group with this contact next time
we send a message there to let others learn new verified key
and let the contact who has resetup their device learn keys of others
in groups.
The returned error is unexpected and no UI I tested with stoped IO really handled it besides maybe displaying a toast.
(desktop and iOS don't handle it, deltatouch shows a toast)
This should not be shown to the user, it is only shown to the user if the UI has a bug, so that bug should be clearly visible.
Merge the code paths for verified and autocrypt key.
If both are changed, only one will be added.
Existing code path adds a message to all chats with the contact
rather than to 1:1 chat. If we later decide that
only 1:1 chat or only verified chats should be notified,
we can add a separate `verified_fingerprint_changed` flag.
Otherwise we will try to create an ad-hoc group
and failing because there are only two contacts
and then unblock a 1:1 chat just to assign
the message to trash in the end.
Hidden 1:1 chat is created for the "chat protected" message.
It should not appear simply because we received
a read receipt from the contact in a group.
Looks like this doesn't fix anything currently, because a better message from
`apply_group_changes()` doesn't appear in a context with another better message, but why drop it if
it's possible to add it, moreover, messages about implicit member additions are never dropped while
looking less important.
The second SQL statement calculating chat size
was already fixed in f656cb29be,
but more important statement calculating member list intersection
was overlooked.
As a result, trash chat with members added there due to former bugs
could still appear in similar chats.
- Reduce cross-module dependencies.
- Stop bloating the `sync` module while implementing synchronisation of more entities.
- Now there's the only `ChatId` :)
An error while executing an item mustn't prevent next items from being executed. There was a comment
that only critical errors like db write failures must be reported upstack, but in fact it's hard to
achieve in the current design, there are no error codes or so, so it's bug-prone. E.g.
`ChatAction::Block` and `Unblock` already reported all errors upstack. So, let's make error handling
the same as everywhere and just ignore any errors in the item execution loop. In the worst case we
just do more unsuccessful db writes f.e.
It's sufficient if the local state is updated successfully, no need to fail the whole
operation. Anyway delivery of sync messages and applying them on other devices are beyond of our
control. If an error occurs when generating a sync messages, probably a log message is sufficient,
no need to even show it to a user. As for the tests, anyway there are ones on synchronisation which
perform necessary checks. Particularly, some sync messages can't be generated if an account is
unconfigured. Adding the corresponding checks to the device synchronisation code (and maybe even
more checks in the future) would complicate the code unnecessarily. Even errors caused by bugs in
this code aren't a reason to fail a local operation.
Sync chat contacts across devices for broadcast lists and groups. This needs the corresponding chat
to exist on other devices which is not the case for unpromoted groups, so it fails for them now but
it's only a warning and will work once creation of unpromoted groups is synchronised too.
List-ID header is added for broadcast lists.
UTF-8 in email headers is allowed only if
all recipient MTAs support SMTPUTF8 extension,
which is not always the case even if our submission service
reports SMTPUTF8 support.
Don't grow timeout if interrupted early and slept not enough. Also:
- Don't grow timeout too fast, but 1.5--2 times (randomly) per iteration.
- Don't interrupt if rate-limited.
- Reset timeout if rate-limited. Rate limit isn't an error, so we can start from 30 secs again if an
error happens then.
When a key is gossiped for the contact in a verified chat,
it is stored in the secondary verified key slot.
The messages are then encrypted to the secondary verified key
if they are also encrypted to the contact introducing this secondary key.
Chat-Group-Member-Added no longer updates the verified key.
Verified group recovery only relies on the secondary verified key.
When a message is received from a contact
signed with a secondary verified key,
secondary verified key replaces the primary verified key.
When verified key is changed for the contact
in response to receiving a message
signed with a secondary verified key,
"Setup changed" message is added
to the same chat where the message is received.
this pr keeps and refines documentation added in #4951, however, reverts
the api introduced by #4951
which turns out to be not useful for UI in practise:
UI anyway check for chat/no-chat beforehand,
so a simple condition in profiles as
`green_checkmark = chat_exist ? chat_is_protected() :
contact_is_verified()`
is more useful in practise and is waht UI need and did already in the
past. (https://github.com/deltachat/deltachat-android/pull/2836 shows a
detailed discussion)
(as a side effect, beside saving code,
this PR saves up to three database calls
(get contact from chat in UI to pass it to profile_is_verified(), get
chat from contact in core, load is_protected in core) - instead, core
can use already is_protected from already loaded chat object)
/me did check rust-tests, fingers crossed for python tests
/me should re-setup python tests on local machine at some point :)
This message makes that partial messages do not change the group state.
A simple fix and a comprehensive test is added. This is a follow up to
the former #4841 which took a different approach.
This is another approach to provide group membership consistency for all members. Considerations:
- Classical MUA users usually don't intend to remove users from an email thread, so if they removed
a recipient then it was probably by accident.
- DC users could miss new member additions and then better to handle this in the same way as for
classical MUA messages. Moreover, if we remove a member implicitly, they will never know that and
continue to think they're still here.
But it shouldn't be a big problem if somebody missed a member removal, because they will likely
recreate the member list from the next received message. The problem occurs only if that "somebody"
managed to reply earlier. Really, it's a problem for big groups with high message rate, but let it
be for now.
Ensure the client does not busy loop
skipping IDLE if UIDNEXT of the mailbox is higher than
the last seen UID plus 1, e.g. if the message
with UID=UIDNEXT-1 was deleted before we fetched it.
We do not fallback to UIDNEXT=1 anymore
if the STATUS command cannot determine UIDNEXT.
There are no known IMAP servers with broken STATUS so far.
This allows to guarantee that select_with_uidvalidity()
sets UIDNEXT for the mailbox and use it in fetch_new_messages()
to ensure that UIDNEXT always advances even
when there are no messages to fetch.
Before, it was shown by UI when the chat is empty, however, in case of
verified groups, this is never the case any longer as the 'e2ee now
guaranteed' is always added.
Green checkmark in the contact profile
should only be displayed in the title
if the same checkmark is displayed in the title of 1:1 chat.
If 1:1 chat does not exist,
the checkmark should not be displayed in the title
of the contact profile.
Also add docs to is_protected
property of FullChat and BasicChat JSON objects.
This prevents accidentally going IDLE
when the last new message has arrived
while the folder was closed.
For example, this happened in some tests:
1. INBOX is selected to fetch, move and delete messages.
2. One of the messages is deleted.
3. INBOX is closed to expunge the message.
4. A new message arrives.
5. INBOX is selected with (CONDSTORE) to sync flags.
6. Delta Chat goes into IDLE without downloading the new message.
To determine that a new message has arrived
we need to notice that UIDNEXT has advanced when selecting the folder.
However, some servers such as Winmail Pro Mail Server 5.1.0616
do not return UIDNEXT in response to SELECT command.
To avoid interdependencies with the code
SELECTing the folder and having to implement
STATUS fallback after each SELECT even when we
may not want to go IDLE due to interrupt or unsolicited EXISTS,
we simply call STATUS unconditionally before IDLE.
Sync messages are only sent on explicit user actions and only one per action, so it's safe to send
them right away not worrying about the rate limit on the server.
This may happen if autocrypt key is changed after verification.
UI should still display who introduced the contact,
especially if the contact is still a member of some verified group.
If the contact is at the same time not verified,
i.e. its autocrypt key does not match verified key,
UI may display a crossed out checkmark instead of green checkmark.
Result::Err is reserved for local errors,
such as database failures.
Not found peerstate in the database is a protocol failure,
so just return Ok(false) in mark_peer_as_verified().
This allows to handle more errors with `?`.
Otherwise SELF contact in the beginning of the vector
and in to_ids may be repeated twice and not deduplicated.
dedup() only deduplicates consecutive elements.
We may not have a verified key for other members
because we lost a gossip message.
Still, if the message is signed with a verified key
of the sender, there is no reason to replace it with an error.
Even though r2d2 connection pool is removed,
deleting accounts still fails in Windows CI.
This reverts commit e88f21c010.
`try_many_times` documentation is modified to explain
why the workaround is still needed.
close#4620
This PR introduces a new core API to parse mailto links into a uniform
data format. This could be used to unify the different implementations
on the current platforms.
To complete this PR we have to decide for which APIs we want to expose
this (now) internal API (c, python, json-rpc, etc.), and if we want such
an API at all as it doesn't have a corresponding UI-PR and is not
_really_ needed.
And execute sync messages only if `Config::SyncMsgs` is enabled. Earlier executing was always
enabled, the messages are force-encrypted anyway. But for users it's probably more clear whether a
device is synchronised or not.
Using BufWriter ensures that `STARTTLS` command is sent
as a single packet.
Also refactor the code to ensure we only convert to
Box<dyn SessionStream> in the end.
Android calls get_connectivity_html()
every time connectivity changes, which in turn interrupts
IMAP loop and triggers change from "not connected" to "connecting"
state.
To avoid such infinite loop of IMAP interrupts when
there is not connectivity, update quota only when IMAP
loop is interrupted otherwise. This anyway happens
when a message is received or maybe_network is called.
Also remove outdated comments about `Action::UpdateRecentQuota` job
which does not exist anymore.
Winmail Pro Mail Server 5.1.0616 does not return UIDNEXT
in response to SELECT, but returns it when explicitly requested
via STATUS command:
? SELECT INBOX
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\Draft \Answered \Flagged \Deleted \Seen)] Limited
* 2 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1697802109] Ok
? OK [READ-WRITE] Ok SELECT completed
? STATUS INBOX (UIDNEXT)
* STATUS "INBOX" (UIDNEXT 4)
? OK STATUS completed
Previously used FETCH method is reported to fail for some users,
the FETCH command sometimes returns no results.
Besides, there is no guarantee that the message with
the highest sequence number has the highest UID.
In the worst case if STATUS does not return UIDNEXT
in response to explicit request, we fall back to setting
UIDNEXT to 1 instead of returning an error.
feat: Make broadcast lists create their own chat - UIs need to ask for
the name when creating broadcast lists now (see
https://github.com/deltachat/deltachat-android/pull/2653)
That's quite a minimal approach: Add a List-ID header to outgoing
broadcast lists, so that the receiving Delta Chat shows them as a
separate chat, as talked about with @r10s and @hpk42.
Done:
- [x] Fix an existing bug that the chat name isn't updated when the
broadcast/mailing list name changes (I already started this locally)
To be done in other PRs:
- [ ] Right now the receiving side shows "Mailing list" in the subtitle
of such a chat, it would be nicer if it showed "Broadcast list" (or
alternatively, rename "Broadcast list" to "Mailing list", too)
- [ ] The UIs should probably ask for a name before creating the
broadcast list, since it will actually be sent over the wire. (Android
PR: https://github.com/deltachat/deltachat-android/pull/2653)
Fixes https://github.com/deltachat/deltachat-core-rust/issues/4597
BREAKING CHANGE: This means that UIs need to ask for the name when creating a broadcast list, similar to https://github.com/deltachat/deltachat-android/pull/2653.
This approach uses a param field to enable forcing the sticker
`viewtype`. The first commit has the memory-only flag implemented, but
this flag is not persistent through the database conversion needed for
draft/undraft. That's why `param` has to be used.
follow up to #4814fixes#4739
---------
Co-authored-by: Septias <scoreplayer2000@gmail.comclear>
If there's a temporary SMTP error, pretend there are no more MDNs to send in send_mdn(). It's
unlikely that other MDNs could be sent successfully in case of connectivity problems. This approach
is simpler and perhaps even better than adding a progressive backoff between MDN sending retries --
MDNs just will be resent after a reconnection, which makes some sense.
This is needed to test periodic re-gossiping in existing chats.
Also add a test for verified groups on that even if "member added" message is missed by a device of
newly added member, after re-gossiping Autocrypt keys to the group it successfully learns these keys
and marks other members as verified.
Set connectivity status to "connected" at the end of connect() call.
Otherwise for servers that do not support IMAP IDLE
connect() is called at the beginning of fake idle
and connectivity stays in "connecting" status most of the time.
Before they were trashed. Note that for unencrypted ones DC works as expected creating the requested
group immediately because Chat-Group-Id is duplicated in the Message-Id header and Subject is
fetched.
Also add a test on downloading a message later. Although it doesn't reproduce #4700 for some reason,
it fails w/o the fix because before a message state was changing to `InSeen` after a full download
which doesn't look correct. The result of a full message download should be such as if it was fully
downloaded initially.
This ensures the server does not get SIGINT
when the bot is running in a terminal and user presses ^C
We want to send SIGTERM ourselves after clean shutdown,
e.g. stopping I/O for all accounts.
Add protected-headers="v1" directive to Content-Type of an encrypted/signed MIME so that other MUAs
like Thunderbird display the true message Subject instead of "...".
For example, reader_loop() may die
if readline() tries to read too large line
and thows an exception.
We want to at least log the exception in this case.
This certificate does not exist on older Android phones.
It is already added manually for IMAP and SMTP,
this commit adds the same certificate for HTTP requests.
it may happen that percentages larger than 100% are reported by the provider,
eg. for some time a storage usage of 120% may be accepted.
while we should report the values "as is" to the user,
for the bar, percentages larger 100% will destroy the layout.
9bd7ab72 brings a possibility of group membership inconsistency to the original Hocuri's algo
described and implemented in e12e026b in sake of security so that nobody can add themselves to a
group by forging "InReplyTo" and other headers. This commit fixes the problem by removing group
members locally if we see a discrepancy with the "To" list in the received message as it is better
for privacy than adding absent members locally. But it shouldn't be a big problem if somebody missed
a member addition, because they will likely recreate the member list from the next received
message. The problem occurs only if that "somebody" managed to reply earlier. Really, it's a problem
for big groups with high message rate, but let it be for now.
Also:
- Query chat contacts from the db only once.
- Update chat contacts in the only transaction, otherwise we can just break the chat contact list
halfway.
- Allow classic MUA messages to remove group members if a parent message is missing. Currently it
doesn't matter because unrelated messages go to new ad-hoc groups, but let this logic be outside
of apply_group_changes(). Just in case if there will be a MUA preserving "Chat-Group-ID" header
f.e.
The code removed is an incomplete implementation of skipping
the Legacy Display Part specified in
https://www.ietf.org/archive/id/draft-autocrypt-lamps-protected-headers-02.html#section-5.2
The code does not fully implement the specification, e.g.
it does not check that there are exactly two parts.
Delta Chat and Thunderbird are not adding this part anyway,
and it is defined as "transitional" in the draft.
This also removes misplaced warning "Ignoring nested protected headers"
that is printed for every incoming Delta Chat message
since commit 5690c48863
which is part of the PR <https://github.com/deltachat/deltachat-core-rust/pull/982>.
A few people got the impression that if you send 6 messages
in a burst you'll only be able to send the next one in 60 seconds.
Hopefully this can resolve it.
This is an RFC 2045 requirement for base64-encoded MIME parts.
Previously referenced RFC 5322 requirement
is a general Internet Message Format requirement
and is more generous.
Previously it was required that a directory path is provided to the import API.
Now it is possible to point directly to the .asc file containing a secret key.
This allows UI to present a file selection dialog to the user
and let select the file directly.
Selecting a directory is still supported for backwards compatibility.
Such a message may be assigned to a wrong chat (e.g. undecipherable group msgs often get assigned to
the 1:1 chat with the sender). Add `DownloadState::Undecipherable` so that messages referencing
undecipherable ones don't go to that wrong chat too. Also do not reply to not fully downloaded
messages. Before `Message.error` was checked for that purpose, but a message can be error for many
reasons.
Previously only one connection, the one used to change the key,
was working after passphrase change.
With this fix the whole pool of connections
is recreated on passphrase change, so there is no need
to reopen the database manually.
These constants are current defaults in `pgp` crate,
this change would prevent accidental change due to rPGP upgrade
and make it easier to change in a single place.
It can be not good for membership consistency if we missed a message adding a member, but improves
security because nobody can add themselves to a group from now on.
1:1 chat may be blocked while the contact is not
if 1:1 chat was created as a result of scanning
a verified group join QR code with the contact
as the inviter. In this case 1:1 chat is blocked to hide it
while the contact is unblocked.
- If we don't know the parent (=In-Reply-To) message, then completely recreate the group member list
(i.e. use the member list of the incoming message) (because we assume that we missed some messages
& have a wrong group state).
- If the message has a "Chat-Group-Member-Removed: member@example.com" header, then remove this
member.
- If the message has a "Chat-Group-Member-Added: member@example.com" header, then add this member.
That means:
- Remove checks for the presense of `ContactId::SELF` in the group. Thus all recipients of a message
take the same decision about group membership changes, no matter if they are in the group
currently. This fixes a situation when a recipient thinks it's not a member because it missed a
message about its addition before.
NOTE: But always recreate membership list if SELF has been added. The older versions of DC don't
always set "In-Reply-To" to the latest message they sent, but to the latest delivered message (so
it's a race), so we need this heuristic currently.
- Recreate the group member list if we don't know the parent (=In-Reply-To) message, even if the
sender isn't in the group as per our view, because we missed some messages and our view may be
stale.
The new message for which `parent_query()` is done may assume that it will be received in a context
affected by those messages, e.g. they could add new members to a group and the new message will
contain them in "To:". Anyway recipients must be prepared to orphaned references.
This is already the way `get_chatlist_entries` works.
`get_similar_chatlist_entries` is renamed into
`get_similar_chat_ids` because return values are not entries anymore.
If the user makes a backup from a UI that supports the experimental verified 1:1 chats (e.g. nightly Android) and imports it into a UI that doesn't, then this config should be cleared.
We already talked about this a long time ago after @Simon-Laux noticed this problem, but I think we didn't actually solve it back then?
Best to review with whitespace changes disabled.
I.e. exclude from the list the following chats as well:
- Read-only mailing lists.
- Chats we're not a member of.
But as for ProtectionBroken chats, we return them, as that may happen to a verified chat at any
time. It may be confusing if a chat that is normally in the list disappears suddenly. The UI need to
deal with that case anyway.
(cherry picked from commit 83ef25e7de)
I.e. exclude from the list the following chats as well:
- Read-only mailing lists.
- Chats we're not a member of.
But as for ProtectionBroken chats, we return them, as that may happen to a verified chat at any
time. It may be confusing if a chat that is normally in the list disappears suddenly. The UI need to
deal with that case anyway.
If a message is unsigned or signed with an unknown key, `MimeMessage::was_encrypted()` returns
false. So, it mustn't be checked when deciding whether to look into
`MimeMessage::decoded_data`. Looking through git history one can see that it's just a wrong check
left in the code for historical reasons.
Other MUAs don't set add/remove headers, so the only way for them to re-add us to the group is to
add us to To/CC/wherever. Previously it worked only for other members that are still in the group so
that they properly handled our re-addition, but we didn't.
Check if a sticker has at least one fully transparent corner and otherwise change the Sticker type
to Image. This would fix both Android and iOS at the same time and prevent similar bug on future
platforms that may get this bug like Ubuntu Touch.
I.e. from delete_msgs(). Otherwise messages must not be deleted from there, e.g. if a message is
ephemeral, but a network outage lasts longer than the ephemeral message timer, the message still
must be sent upon a successful reconnection.
I.e. from delete_msgs(). Otherwise messages must not be deleted from there, e.g. if a message is
ephemeral, but a network outage lasts longer than the ephemeral message timer, the message still
must be sent upon a successful reconnection.
If the Inbox is fetched before the Sentbox (as done currently), messages from the Sentbox will
correctly mingle with the Inbox messages in the end. So, this commit changes message ordering only
if we already have processed outgoing messages, e.g. if we just sent them in the chat as described
in #4621. Otherwise new incoming messages are displayed somewhere in the middle of the chat which
doesn't look usable.
Directly unwrap in TestContext::get_chat()
Turns out that all usages of get_chat() directly unwrapped, because in a
test it doesn't make sense to handle the error of there being no chat.
Don't show a contact as verified if their key changed in the meantime
If a contact's key changed since the verification, then it's very
unlikely that they still have the old, verified key. So, don't show them
as verified anymore.
This also means that you can't add a contact like this to a verified
group, which is good.
The documentation actually already described this (new) behavior:
```rust
/// and if the key has not changed since this verification.
```
so, this adapts the code to the documentation.
It can be used e.g. as a default in the file saving dialog. Also display the original filename in
the message info. For these purposes add Param::Filename in addition to Param::File and use it as an
attachment filename in sent emails.
Opening the same account (context) from multiple processes is dangerous, can result in duplicate
downloads of the same message etc. Same for account manager, attempts to modify the same
accounts.toml even if done atomically with may result in corrupted files as atomic replacement
procedure does not expect that multiple processes may write to the same temporary file.
accounts.toml cannot be used as a lockfile because it is replaced during atomic update. Therefore, a
new file next to accounts.toml is needed to prevent starting second account manager in the same
directory.
But iOS needs to be able to open accounts from multiple processes at the same time. This is required
as the "share-to-DC extension" is a separate process by iOS design -- this process may or may not be
started while the main app is running. Accounts are not altered however by this extension, so let's
add to the `Accounts::new()` constructor an `rdwr` parameter which allows to read the accounts
config w/o locking the lockfile.
Webxdc update messages may contain
long lines that get hard-wrapped
and may corrupt JSON if the message
is not encrypted.
base64-encode the update part
to avoid hard wrapping.
This is not necessary for encrypted
messages, but does not introduce
size overhead as OpenPGP messages
are compressed.
Correctly handle messages with old timestamps for verified chats:
* They must not be sorted over a protection-changed info message
* If they change the protection, then they must not be sorted over existing other messages, because then the protection-changed info message would also be above these existing messages.
This PR fixes this:
1. Even seen messages can't be sorted into already-noticed messages anymore. **This also changes DC's behavior in the absence of verified 1:1 chats**. Before this PR, messages that are marked as seen when they are downloaded will always be sorted by their timestamp, even if it's very old.
2. protection-changed info messages are always sorted to the bottom.
**Edit:**
3. There is an exception to rule 1: Outgoing messages are still allowed to be sorted purely by their timestamp, and don't influence old messages. This is to the problem described at [*].
Together, these rules also make sure that the protection-changed info message is always right above the message causing the change.
[*] If we receive messages from two different folders, e.g. `Sent` and `Inbox`, then this will lead to wrong message ordering in many cases. I need to think about this more, or maybe someone else has an idea. One new idea that came to my mind is:
* Always sort noticed messages under the newest info message (this PR sorts them under the newest noticed message, master sorts them purely by their sent timestamp)
* Always sort unnoticed messages under the newest noticed message (that's the same behavior as in this PR and on master)
* Always sort protection-changed info messages to the bottom (as in this PR)
However, after a talk with @link2xt we instead decided to add rule 3. (see above) because it seemed a little bit easier.
The test checks that if webxdc update is too large to
download with the current `download_limit`,
it is applied afterwards when the user manually downloads the update message.
tokio-tar 0.3.0 prints message "create_dir_all ..." to stdout during import.
tokio-tar 0.3.1 has removed this debug output which broke stdio JSON-RPC protocol.
If the verification is broken, `can_send()` is false.
But if the user was typing a message right when a verification-breaking message came in, the UI still needs to be able to save it as a draft.
Steps to reproduce the bug:
- Set a draft
- Your chat partner breaks verification
- Go back to the chats list
- Go to the chat again
- Accept the breakage
- Expected: The draft is still there
- Bug behavior: The draft is gone
Some Dovecot servers are configured
to alias "INBOX.DeltaChat" and "DeltaChat" to the same folder.
In this case Delta Chat moves new emails from "INBOX"
to "DeltaChat", but then discovers the same mail in "INBOX.DeltaChat"
and tries to move it to "DeltaChat" again.
Each time a MOVE command is issued to move the message
from "INBOX.DeltaChat" to "DeltaChat", the message gets a new UID.
To prevent such IMAP move loop between aliased folders,
we do not move the message if we have already seen it on IMAP,
i.e. we have its Message-ID in the `imap` table.
Note that we do not use `rfc724_mid_exists`,
because it checks the `msgs` table and would consider
BCC-self messages seen even before we see them in the Inbox,
preventing their move to the DeltaChat folder.
Duplicate messages and messages without Message-IDs
are not moved anymore, but this is better
than having an infinite move loop.
1:1 chats are automatically created as protected if the contact is
verified, there is no need to explicitly do this.
Plus, by removing this call, the test also tests that automatically
creating 1:1 chats as protected works.
receive_imf() calls add_parts()
which INSERTs or UPDATEs the message using UPSERT [1].
It then uses last_insert_rowid() to get
the ID of the inserted message.
However, it is incorrect to use last_insert_rowid()
if an UPDATE was executed instead of INSERT.
The solution is to use `RETURNING id` clause
to make the UPSERT statement return message ID in any case [2].
The fix is tested in test_webxdc_update_for_not_downloaded_instance()
and with a debug_assert!.
[1] https://www.sqlite.org/lang_UPSERT.html
[2] https://sqlite.org/forum/forumpost/9ce3bc1c4a85c15f
When `cargo test` is executed,
all examples are built by default
to ensure that they can be compiled.
This is a documented and expected behaviour,
even though it was previously reported as a bug:
<https://github.com/rust-lang/cargo/issues/6675>
In particular, `examples/simple.rs` is built into
a 67M binary `target/debug/examples/simple`.
This is unnecessary to do so every time
you change a line in the `deltachat` crate
and want to rerun the tests.
Workaround is to run `cargo test --tests`,
but it is easy to forget and is not discoverable
unless you read the "Target Selection" section of `cargo help test`.
We have a maintained example at https://github.com/deltachat-bot/echo,
so there is no need for an example in the core repository.
This commit adds new stock strings
"I added member ...",
"I removed member ..." and
"I left the group" that are sent over the network
and are visible in classic MUAs like Thunderbird.
Member name in these messages uses authname
instead of the display name,
so the name set locally does not get leaked when
a member is added or removed.
Message.set_text() and Message.get_text() are modified accordingly
to accept String and return String.
Messages which previously contained None text
are now represented as messages with empty text.
Use Message.set_text("".to_string())
instead of Message.set_text(None).
simplify() is written to process incoming plaintext messages
and extract footers and quotes from them.
Incoming messages contain various quote styles
and simplify() implements heuristics to detects them.
If dehtml() output is processed by simplify(),
simplify() heuristics may erroneously detect
footers and quotes in produced plaintext.
dehtml() should directly detect quotes
instead of converting them to plaintext quotes
for parsing with simplify().
This is a follow-up to cbe1671104
I changed `apt-get` arguments for x86_64 glibc builds,
but forgot to change it for x86_64 musl, aarch64 glibc and aarch64 musl.
Because of this, `upload-wheels` task failed with a message:
```
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt install python3.11-venv
You may need to use sudo with that command. After installing the python3-venv
package, recreate your virtual environment.
```
Mergeable is disabled because it was requiring
that PR title follows conventional commit notation
even when PR consisted of multiple commits
and was not planned to be squash merged.
`ouroboros` is deprecated with a security advisory recommending
switch to `self_cell` crate:
https://rustsec.org/advisories/RUSTSEC-2023-0042
async-imap 0.9.0 depends on `self_cell` instead of `ouroboros`.
This commit solves the "error: externally-managed-environment"
which started appearing since Debian 12 release.
`debian` is used as an Docker image to run devpi.
Specifying msg IDs that cannot be loaded in the event payload
results in an error when the UI tries to load the message.
Instead, emit an event without IDs
to make the UI reload the whole messagelist.
New algorithm improves group consistency
in cases of missing messages,
restored old backups and replies from classic MUAs.
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: link2xt <link2xt@testrun.org>
When running `cargo test` in the deltachat-jsonrpc folder,
a new file `openrpc/openrpc.json` will be created with an
[OpenRPC](https://spec.open-rpc.org/) definition.
It can be copy-pasted into the
[OpenRPC playground](https://playground.open-rpc.org/)
and used to generate clients in other languages.
With Python 3.7 asynchronous tests randomly fail
with "RuntimeError: Event loop is closed" during shutdown.
Backtrace of the error includes `SafeChildWatcher` calls.
Python 3.8 has replaced `SafeChildWatcher`
with a new `ThreadedChildWatcher` by default [1]
as a bugfix for
"asyncio.create_subprocess_exec() only works with main event loop" bug [2].
Python 3.7 scheduled end of life is 2023-06-27
according to <https://devguide.python.org/versions/>.
[1] https://github.com/python/cpython/pull/14344
[2] https://bugs.python.org/issue35621
Normally the message has UID 1,
but this is not true when account is reused.
In this case the message may be
"Marked messages 2 in folder DeltaChat as seen."
and the test times out waiting for
"Marked messages 1 in folder DeltaChat as seen."
According to RFC 3501, EXISTS must always be sent in response to SELECT.
But if the server does not send it for some reason,
async-imap uses the default value, so we will essentially fetch `1:*`
and downloading all messages may take a long time.
convert `JSONRPCReactions.reactions` to a `Vec<JSONRPCReaction>` with unique reactions and their count, sorted in descending order.
---------
Co-authored-by: link2xt <link2xt@testrun.org>
According to RFC 3501, EXISTS must always be sent in response to SELECT.
But if the server does not send it for some reason,
async-imap uses the default value, so we will essentially fetch `1:*`
and downloading all messages may take a long time.
* jsonrpc: `MessageSearchResult`, always include `chat_name`(not an option anymore), also add `author_id` and `chat_type`
* replace .ok_or_else with anyhow context
* fix: jsonrpc: typescript client: fix types of events in event emitter
* add ts ignore to remove ts error (when building in ts5)
doing the types right via casting is to complicated IMO and has no real benefit here.
Why? because desktop currently fetches the chatlist multiple times, even though it just needs the
chatlistitem for one chat.
Note: @r10s was worried before that exposing the method to get a single updated chatlistitem could
lead to race conditions where the chatlist item is newer than the chatlist order. But I don't think
this will change anything for desktop besides making it a little faster (because currently desktop
fetches the whole chatlist instead of just the entry it needs when an entry updates).
This way is more compatible to JSON-RPC libraries
that do not support receiving notifications from the server
and allows describing event types in the OpenRPC specification.
Event thread converting events to notifications in the FFI
is removed, so it is now possible to construct a dc_jsonrpc_instance_t
while still retrieving events via dc_event_emitter_t.
Otherwise sending a message without plaintext part
resets the signature. It is particularly dangerous
in multidevice case, because it's easy to accidentally
reset the signature on your other device with a non-text message.
Move DebugLogging into debug logging module.
Remove Context.send_log_event().
Use blocking_read() instead of try_read() to
get read lock on debug_logging() in synchronous code.
Do not try to log events while holding a lock
on DebugLogging.
I.e. > 500K for the balanced media quality and 130K for the worse one. This can remove animation and
transparency from PNG/WebP, but then a user always can send an image as a file.
Also don't reduce wide/high images if they aren't huge. Among other benefits, this way most of PNG
screenshots aren't touched.
Also remove Exif from all images, not from JPEGs only.
This patch adds new C APIs
dc_get_next_msgs() and dc_wait_next_msgs(),
and their JSON-RPC counterparts
get_next_msgs() and wait_next_msgs().
New configuration "last_msg_id"
tracks the last message ID processed by the bot.
get_next_msgs() returns message IDs above
the "last_msg_id".
wait_next_msgs() waits for new message notification
and calls get_next_msgs().
wait_next_msgs() can be used to build
a separate message processing loop
independent of the event loop.
Async Python API get_fresh_messages_in_arrival_order()
is deprecated in favor of get_next_messages().
Introduced Python APIs:
- Account.wait_next_incoming_message()
- Message.is_from_self()
- Message.is_from_device()
Introduced Rust APIs:
- Context.set_config_u32()
- Context.get_config_u32()
* Don't let blocking be bypassed using groups
Fix#4313
* Fix another bug: A blocked group was sometimes not unblocked when an unblocked contact sent a message into it.
* Small performance improvement by not unnecessarily loading the peerstate
* Remove wrong info message "{contact} verified" when scanning a QR code with just an email
I think that this was a bug in the original C code and then slipped
through two refactorings.
Moved custom ToSql trait including Send + Sync from lib.rs to sql.rs.
Replaced most params! and paramsv! macro usage with tuples.
Replaced paramsv! and params_iterv! with params_slice!,
because there is no need to construct a vector.
bjoern <r10s@b44t.com> wrote:
> maybe_add_time_based_warnings() requires some date guaranteed to be in the near past. based on
this known date we check if the system clock is wrong (if earlier than known date) and if the used
Delta Chat version may be outdated (1 year passed since known date). while this does not catch all
situations, it catches quite some errors with comparable few effort.
>
> figuring out the date guaranteed to be in the near past is a bit tricky. that time, we added
get_provider_update_timestamp() for exactly that purpose - it is checked manually by some dev and
updated from time to time, usually before a release.
>
> however, meanwhile, the provider-db gets updated less frequently - things might be settled a bit
more - and, get_provider_update_timestamp() was also changed to return the date of the last commit,
instead of last run. while that seem to be more on-purpose, we cannot even do an “empty” database
update to update the known date.
>
> as get_provider_update_timestamp() is not used for anything else, maybe we should completely
remove that function and replace it by get_last_release_timestamp that is then updated by
./scripts/set_core_version.py - the result of that is reviewed manually anyway, so that seems to be
a good place (i prefer manual review here and mistrust further automation as also dev or ci clocks
may be wrong :)
This adds a few log items for imex::transfer::get_backup and the
ongoing process to give some more insights.
IO is now also paused after the ongoing process is allocated in
get_backup to avoid needlessly pausing IO.
that way, UI can just close the transfer dialog,
so that, at the end, both devices end in the chatlist.
we can also use this for troubleshooting -
if the device message is not present,
transfer did not succeed completely.
(a separate device message may be nice in that case -
but that is another effort,
same for making the device message reappear after deletion
or after some time)
some providers say that they support QUOTA in the IMAP CAPABILITY,
but return an empty list without any quota information then.
in our "Connectivity", this looks a bit of an error.
i have not seen this error often - only for testrun.org -
if it is usual, we could also just say "not supported"
(as we do in case QUOTA is not returned).
a translations seems not to be needed for now,
it seems unusual, and all other errors are not translated as well.
With this, all the builds happen in parallel in 3-6 minutes,
and then python tests run for 8-9 minutes.
Before, it was 6-17 minutes for compilation,
then 19-31 minutes for python tests.
Compilation of the library and test binary was not parallelized,
and then python tests included async python tests
and deltachat-rpc-server binary compilation.
Now, deltachat-rpc-server compilation,
Rust testing and async python tests are separate jobs.
Similarly to how `imex_inner()` runs migrations
after successful call to `import_backup()`,
migrations should be run after receiving a backup
using `transfer_from_provider()`.
This way if flaky python tests fail,
it is possible to rerun them without having to rerun library compilation.
Python test jobs will redownload already built library artifact
and compile the bindings against it.
This further reduces the cognitive overload of having many ways to do
something. The same is very easily done using composition. Followup
from 82ace72527.
`install_python_bindings.py` was not used by CI
and scripts, except for `scripts/run-python-test.sh`
which only used it to invoke `cargo`.
Instead of using an additional script,
run cargo directly.
The documentation is updated to remove
references to `install_python_bindings.py`.
The section "Installing bindings from source"
was in fact incorrect as it suggested
running `tox --devenv` first and only then
compiling the library with `install_python_bindings.py`.
Now it explicitly says to run cargo first
and then install the package without using `tox`.
`tox` is still used for running the tests
and setting up development environment.
* fix(imex): transfer::get_backup must always free ongoing process
When the ongoing process is cancelled it is still the responsibility
of whoever took out the ongoing process to free it. This code was
only freeing the ongoing process when completed normally but not when
cancelled.
* add changelog
Recommended file size is used for recoding media.
For the upper size, we rely on the provider to tell us back
if the message cannot be delivered to some recipients.
This allows to send large files, such as APKs,
when using providers that don't have such strict limits.
This removes the message that needed to be supplied to LogExt::log_err
calls. This was from a time before we adopted anyhow and now we are
better off using anyhow::Context::context for the message: it is more
consistent, composes better and is less custom.
The benefit of the composition can be seen in the FFI calls which need
to both log the error as well as return it to the caller via
the set_last_error mechanism.
It also removes the LogExt::ok_or_log_msg funcion for the same reason,
the message is obsoleted by anyhow's context.
This moves us back to a released version;
- Ticket is now opaque, need to use accessor functions.
- Ticket now ensures it is valid itself, no need to inspect it's
inners. Deserialisation would fail if it was bad.
- The git version was accidentally used with default-features enabled
and thus pulled in a few too many dependencies. They are now gone.
This ensures that the BackupProvider will be stopped as soon as the
struct is dropped and the imex progress error event is emitted. This
makes it easier to use and also makes sure that the ffi call
dc_backup_provider_unref() does not lead to dangling resources.
cargo-zigbuild is replaced with a simple zig-cc wrapper.
cargo-zigbuild still tries to use i386-linux-musl triple,
while in zig 0.11 x86-linux-musl should be used [1].
This also gives us more control over compilation flags used
and prevents changes due to cargo-zigbuild upgrades,
as its version was not pinned.
[1] Merged PR "all: rename i386 to x86 "
<https://github.com/ziglang/zig/pull/13101>
This uses the new iroh API to connect to all provider addresses
concurrently. It simplifies the implementation as well as we no
longer need to try the addresses manually.
* deps: Update iroh, remove default-net patch
The released version of default-net is now sufficient and iroh makes
sure this dependency is recent enough.
* Update cargo-deny config
* Newer version of spin, previous has been yanked
When trying IP addresses from the ticket, have a very rough sort order
in which to try them. Basically assume most local wifi's are
somewhere on 192.168.0.0/16 so prefer those first.
.wait_for_seen() is unreliable, because sometimes Dovecot
sends only EXISTS to the IDLE connection, but not the FETCH.
Dovecot sends updates like FETCH only if some
connection has already observed the message in previous state
without the \Seen flag.
To avoid this race condition, wait until the core sets the flag,
then FETCH the message manually and check that the flag is set.
The documentation says this blocks. This should block because it also
means the error reporting is more accurate by calling set_last_error
just before returning.
Otherwise it is possible for the context that is used in the spawn to
be unreferenced. Really this should be caught by the borrow checker
that ensures we only spawn things with a 'static lifetime, but we're
handling raw pointers so it doesn't.
This replaces the mechanism by which the IoPauseGuard makes sure the
IO scheduler is resumed: it really is a drop guard now by sending a
single message on drop.
This makes it not have to hold on to anything like the context so
makes it a lot easier to use.
The trade-off is that a long-running task is spawned when the guard is
created, this task needs to receive the message from the drop guard in
order for the scheduler to resume.
- over jsonrpc built with napi.rs: \[[📂 source](https://github.com/deltachat/napi-jsonrpc) | [📦 npm](https://www.npmjs.com/package/@deltachat/napi-jsonrpc)\]
[^1]: Out of date / unmaintained, if you like those languages feel free to start maintaining them. If you have questions we'll help you, please ask in the issues.
We format the code using `rustfmt`. Run `cargo fmt` prior to committing the code.
Run `scripts/clippy.sh` to check the code for common mistakes with [Clippy].
[Clippy]: https://doc.rust-lang.org/clippy/
## SQL
Multi-line SQL statements should be formatted using string literals,
for example
```
sql.execute(
"CREATE TABLE messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
text TEXT DEFAULT '' NOT NULL -- message text
) STRICT",
)
.await?;
```
Do not use macros like [`concat!`](https://doc.rust-lang.org/std/macro.concat.html)
or [`indoc!](https://docs.rs/indoc).
Do not escape newlines like this:
```
sql.execute(
"CREATE TABLE messages ( \
id INTEGER PRIMARY KEY AUTOINCREMENT, \
text TEXT DEFAULT '' NOT NULL \
) STRICT",
)
.await?;
```
Escaping newlines
is prone to errors like this if space before backslash is missing:
```
"SELECT foo\
FROM bar"
```
Literal above results in `SELECT fooFROM bar` string.
This style also does not allow using `--` comments.
---
Declare new SQL tables with [`STRICT`](https://sqlite.org/stricttables.html) keyword
to make SQLite check column types.
Declare primary keys with [`AUTOINCREMENT`](https://www.sqlite.org/autoinc.html) keyword.
This avoids reuse of the row IDs and can avoid dangerous bugs
like forwarding wrong message because the message was deleted
and another message took its row ID.
Declare all new columns as `NOT NULL`
and set the `DEFAULT` value if it is optional so the column can be skipped in `INSERT` statements.
Dealing with `NULL` values both in SQL and in Rust is tricky and we try to avoid it.
If column is already declared without `NOT NULL`, use `IFNULL` function to provide default value when selecting it.
Use `HAVING COUNT(*) > 0` clause
to [prevent aggregate functions such as `MIN` and `MAX` from returning `NULL`](https://stackoverflow.com/questions/66527856/aggregate-functions-max-etc-return-null-instead-of-no-rows).
Don't delete unused columns too early, but maybe after several months/releases, unused columns are
still used by older versions, so deleting them breaks downgrading the core or importing a backup in
an older version. Also don't change the column type, consider adding a new column with another name
instead. Finally, never change column semantics, this is especially dangerous because the `STRICT`
@@ -108,10 +108,10 @@ This will build the `deltachat-jsonrpc-server` binary and then run a test suite
The test suite includes some tests that need online connectivity and a way to create test email accounts. To run these tests, talk to DeltaChat developers to get a token for the `testrun.org` service, or use a local instance of [`mailadm`](https://github.com/deltachat/docker-mailadm).
Then, set the `DCC_NEW_TMP_EMAIL` environment variable to your mailadm token before running the tests.
Then, set the `CHATMAIL_DOMAIN` environment variable to your testing email server domain.
```
DCC_NEW_TMP_EMAIL=https://testrun.org/new_email?t=yourtoken npm run test
/// This means that you should refresh the connectivity view
@@ -268,41 +240,91 @@ pub enum JSONRPCEventType {
/// getConnectivityHtml() for details.
ConnectivityChanged,
/// Deprecated by `ConfigSynced`.
SelfavatarChanged,
/// A multi-device synced config value changed. Maybe the app needs to refresh smth. For
/// uniformity this is emitted on the source device too. The value isn't here, otherwise it
/// would be logged which might not be good for privacy.
ConfigSynced{
/// Configuration key.
key: String,
},
#[serde(rename_all = "camelCase")]
WebxdcStatusUpdate{
msg_id: u32,
status_update_serial: u32,
},
/// Data received over an ephemeral peer channel.
#[serde(rename_all = "camelCase")]
WebxdcRealtimeData{msg_id: u32,data: Vec<u8>},
/// Advertisement received over an ephemeral peer channel.
/// This can be used by bots to initiate peer-to-peer communication from their side.
#[serde(rename_all = "camelCase")]
WebxdcRealtimeAdvertisementReceived{msg_id: u32},
/// Inform that a message containing a webxdc instance has been deleted
#[serde(rename_all = "camelCase")]
WebxdcInstanceDeleted{
msg_id: u32,
},
WebxdcInstanceDeleted{msg_id: u32},
/// Tells that the Background fetch was completed (or timed out).
/// This event acts as a marker, when you reach this event you can be sure
/// that all events emitted during the background fetch were processed.
///
/// This event is only emitted by the account manager
AccountsBackgroundFetchDone,
/// Inform that set of chats or the order of the chats in the chatlist has changed.
///
/// Sometimes this is emitted together with `UIChatlistItemChanged`.
ChatlistChanged,
/// Inform that a single chat list item changed and needs to be rerendered.
/// If `chat_id` is set to None, then all currently visible chats need to be rerendered, and all not-visible items need to be cleared from cache if the UI has a cache.
#[serde(rename_all = "camelCase")]
ChatlistItemChanged{chat_id: Option<u32>},
/// Inform that the list of accounts has changed (an account removed or added or (not yet implemented) the account order changes)
///
/// This event is only emitted by the account manager
AccountsChanged,
/// Inform that an account property that might be shown in the account list changed, namely:
/// - is_configured (see is_configured())
/// - displayname
/// - selfavatar
/// - private_tag
///
/// This event is emitted from the account whose property changed.
AccountsItemChanged,
/// Inform than some events have been skipped due to event channel overflow.
"CAN NOT RUN COVERAGE correctly: Missing DCC_NEW_TMP_EMAIL environment variable!\n\n",
"CAN NOT RUN COVERAGE correctly: Missing CHATMAIL_DOMAIN environment variable!\n\n",
"You can set COVERAGE_OFFLINE=1 to circumvent this check and skip the online tests, but those coverage results will be wrong, because some functions can only be tested in the online test"
assertsnapshot.text=="Group left by {}.".format(bob.get_config("addr"))
logging.info("Alice withdraws QR code.")
qr=alice.check_qr(qr_code)
assertqr["kind"]=="withdrawVerifyGroup"
alice.set_config_from_qr(qr_code)
logging.info("Bob scans withdrawn QR code.")
bob_chat=bob.secure_join(qr_code)
logging.info("Bob scanned withdrawn QR code")
whileTrue:
event=alice.wait_for_event()
if(
event.kind==EventType.WARNING
and"Ignoring vg-request-with-auth message because of invalid auth code."inevent.msg
):
break
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
