IMAP capabilities and selected folder are IMAP session,
not IMAP client property.
Moving most operations into IMAP session structure
removes the need to constantly check whether IMAP session exists
and reduces number of invalid states, e.g. when a folder is selected but
there is no connection.
Capabilities are determined immediately after logging in,
so there is no need for `capabilities_determined` flag anymore.
Capabilities of the server are always known if there is a session.
`should_reconnect` flag and `disconnect()` function are removed: we
drop the session on error. Even though RFC 3501 says that a client
SHOULD NOT close the connection without a LOGOUT, it is more reliable
to always just drop the connection, especially after an error.
* Because both only make problems with mailing lists, it's easiest to just disable them. If we want, we can make them work properly with mailing lists one day and re-enable them, but this needs some further thoughts.
Part of #3701
* Use load_from_db() in more tests
* clippy
* Changelog
* Downgrade warning to info, improve message
* Use lifetimes instead of cloning
When a batch of messages is moved from Inbox to DeltaChat folder with a single MOVE command, their
UIDs may be reordered (e.g. Gmail is known for that) which leads to that messages are processed by
receive_imf in the wrong order. But the INTERNALDATE attribute is preserved during a MOVE according
to RFC3501. So, use it for sorting fetched messages.
Both reversed and original order do not make much sense
for the bot. Ideally bots should have their own key
to get the list of fresh messages in the order of IDs.
This esp. speeds up receive_imf a bit when we recreate the member list (recreate_member_list == true).
It's a preparation for https://github.com/deltachat/deltachat-core-rust/issues/3768, which will be a one-line-fix, but recreate the member list more often, so that we want to optimize this case a bit.
It also adds a benchmark for this case. It's not that easy to make the benchmark non-flaky, but by closing all other programs and locking the CPU to 1.5GHz it worked. It is consistently a few percent faster than ./without-optim:
```
Receive messages/Receive 100 Chat-Group-Member-{Added|Removed} messages
time: [52.257 ms 52.569 ms 52.941 ms]
change: [-3.5301% -2.6181% -1.6697%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
```
It's a w/a for "Space added before long group names after MIME serialization/deserialization"
issue. DC itself never creates group names with leading/trailing whitespace, so it can be safely
removed. On the sender side there's no trim() because group names anyway go through
improve_single_line_input(). And I believe we should send the exact name we have in our db. Also
there's no check for leading/trailing whitespace because there may be existing user databases with
group names having such whitespaces.
Since switch to async we don't have spurious "database is busy"
errors anymore. Since an error is irrecoverable in most cases,
we can skip the message. The cost of this is we may
accidentally skip a correct message if I/O fails, but
the advantage is that we are guaranteed to never confuse
irrecoverable error with recoverable one and get stuck in
infinite loop redownloading the same message over and over.
That's a bug which @Simon-Laux and probably also @hpk42 had, where one malformed incoming (Spam-) mail blocked the receiving of all emails coming after it.
The problem was that from_field_to_contact_id() returned ContactId::UNDEFINED, and then lookup_by_contact() returned Err.
* Treat multiple From addresses as if there was no From: addr
* changelog
* Don't send invalid emails through the whole receive_imf pipeline
Instead, directly create a trash entry for them.
* Don't create trash entries for randomly generated Message-Id's
* clippy
* fix typo
Co-authored-by: link2xt <link2xt@testrun.org>
If we fetch messages out of order, then f.e. reactions don't work because if we process a reaction not yet having the corresponding message processed, the reaction is thrown away.
* allow deleting referenced contacts in UI
we are quite often getting requests of users
who want to get rid of some contact in the "new chat" list.
there is already a "delete" option,
but it does not work for referenced contacts -
however, it is not obvious for users that a contact is in use,
esp. of some mailing list or larger chat, old contacts, whatever.
this pr revives an old idea [^1] of "soft deleting" referenced contacts -
this way, the user can remove the annoying entry
without the need to understand complicated things
and finally saying that deletion is impossible :)
once the contact is reused, it will reappear,
however, this is already explained in the confirmation dialog of the UIs.
technically, this pr was simpler as expected as we already have
a Origin::Hidden, that is just reused here.
[^1]: https://github.com/deltachat/deltachat-core/pull/542
* update rust doccomment
* update changelog
* avoid races on contact deletion
chats may be created between checking for "no chats" and contact deletion.
this is prevented by putting the statement into an EXCLUSIVE transaction.
* fix failing python test
Note that `IMAP IDLE protocol timed out` was previously
added to the wrong error: not the timeout error (first `?`)
but actual error happened during IDLE, such as a network error.
* Add DC_EVENT_INCOMING_MSG event
* Fix lots of compile errors
* Docs
* Changelog
* Fix python tests
Adding DC_EVENT_INCOMING_MSG_BUNCH made the python tests fail because they use `get_matching("DC_EVENT_INCOMING_MSG")`, which also matches DC_EVENT_INCOMING_MSG_BUNCH, so the tests got confused.
This fixes `get_matching()` to only match whole event names.
* Also fix test_ac_setup_message_twice()
The built regex was ^EVENT_NAME1|EVENT_NAME2$, which becomes parsed as
"^EVENT_NAME1" OR "EVENT_NAME2$". Introduce a group (parentheses) to fix
this.
* desktop will use DC_EVENT_INCOMING_MSG_BUNCH,
so I would not call it experimental anymore
* add generated node constants
* msg_ids in the event as Vec<u32>
number[] in js land
this is way more convinient than a json encoded string.
* Apply suggestions from code review
Co-authored-by: bjoern <r10s@b44t.com>
Co-authored-by: Simon Laux <mobile.info@simonlaux.de>
Co-authored-by: Simon Laux <Simon-Laux@users.noreply.github.com>
Co-authored-by: bjoern <r10s@b44t.com>
Seems like consume_events() didn't work properly, i.e. in some cases it
didn't see the latest events and failed to consume them. So, the
IncomingMsg event from receiving DC_MAILINGLIST stayed in the events
channel, which made this fail:
```rust
// Check that no notification is displayed for blocked mailing list message.
while let Ok(event) = t.evtracker.try_recv() {
assert!(!matches!(event.typ, EventType::IncomingMsg { .. }));
}
```
Fix it by explicitly waiting for the first IncomingMsg event.
The problem was that a message without Autocrypt key or with a wrong
signature resets peerstate regardless of what DKIM check says. I
inserted sleep(1.1) to make sure reset always happens and make the bug
reproducible, then fixed it by forbidding reset if DKIM check fails.
https://github.com/deltachat/deltachat-core-rust/pull/3731
The way to create a Context is now rather burdensome, users have to
create and import a bunch of things just to get a Context. So let's
introduce a builder.
Notice that the builder can only produce an open context, if the
context can not be opened it is dropped. This is on purpose, the
Context itself can become RAII again at some point by doing this.
Only the FFI needs to have the concept of an open and a closed
Context.
Optimised debug builds result in an extremely slow code-build-test
cycle. The reason we do this is because we have a few tests which end
up overflowing the stack. This increases the stack instead.
mio 0.8.5 does not use libc epoll_create1() function on Android anymore,
so it will be possible to revert e29b6f9974
in the next Android release with the new core.
Fix#3507
Note that this is not intended for a release at this point! We first have to test whether it runs stable enough. If we want to make a release while we are not confident enough in authres-checking, then we have to disable it.
BTW, most of the 3000 new lines are in `test_data/messages/dkimchecks...`, not the actual code
da3a4b94 adds the results to the Message info. It currently does this by adding them to `hop_info`. Maybe we should rename `hop_info` to `extra_info` or something; this has the disadvantage that we can't rename the sql column name though.
Follow-ups for this could be:
- In `update_authservid_candidates()`: Implement the rest of the algorithm @hpk42 and me thought about. What's missing is remembering how sure we are that these are the right authserv-ids. Esp., when receiving a message sent from another account at the same domain, we can be quite sure that the authserv-ids in there are the ones of our email server. This will make authres-checking work with buzon.uy, disroot.org, yandex.ru, mailo.com, and riseup.net.
- Think about how we present this to the user - e.g. currently the only change is that we don't accept key changes, which will mean that the small lock on the message is not shown.
- And it will mean that we can fully enable AEAP, after revisiting the security implications of this, and assuming everyone (esp. @link2xt who pointed out the problems in the first place) feels comfortable with it.
* let search_msgs() return unaccepted requests
unaccepted chat requests are shown in the chatlist,
it should be returned by search_msgs() an by the other search functions as well.
form the view of the user, the search acts like a filter,
so there is no reason to hide things additionally.
also, the user may remember a word in a chat request,
maybe even an archived one (there is no need to accept a request before archiving)
that one wants to search later on.
* test searching for unaccepted requests
* simplyfy expression; `c.blocked!=1` is also what is used in similar statements
This way no temporary rows are created and it is easier to maintain
because UPDATE statement is right below the INSERT statement,
unlike `merge_messages` function which is easy to forget about.
* jsonrpc: typescript client: export constants
under `C` enum,
similar to how its exported from `deltachat-node`
* add pr number to changelog
* fix tests
* fix changelog entry position
* set timeout for node ci tests to 10min
set timeout for node ci tests to 10min for the test step,
macOS takes 12min for the whole workflow with cached core build,
so 10min just for the test step should be plenty.
* don't forget to set the limit on windows, too
* jsonrpc in cffi also sends events now
* add pr id to changelog
* jsonrpc: new format for events and better typescript autocompletion (#3663)
* jsonrpc: new format for events and better typescript autocompletion
* adjust doc comments
Very small PR; Motivation: Easier navigation using Go-To-definition.
Because, using go-to-definition of rust-analyzer on parse() doesn't take you to the actual parse() implementation but its trait definiton. On the other hand, it's very easy to find EmailAddress::new().
remove function `messageListGetMessageIds()`,
it is replaced by `getMessageIds()` and `getMessageListEntries()`
the latter returns a new `MessageListItem` type,
which is the now prefered way of using the message list.
All contexts created by the same account manager
share stock string translations. Setting translation on
a single context automatically sets translations for all other
accounts, so it is enough to set translations on the active account.
* jsonrpc js client: ci upload and new name
* Update jsonrpc-client-npm-package.yml
* Update jsonrpc-client-npm-package.yml
* Update jsonrpc-client-npm-package.yml
* change details message
* make sure to generate dist directory
* add method for desktop to get notification relevant information for a message
* add pr number to changelog
* rename MessageNotificationData to MessageNotificationInfo
* enable `BccSelf` by default
enabling `BccSelf` improves user experience as
it is easier to set up another device
and ppl will also see "all" messages in other user agents directly.
for uncounted user problems, after diving into the issue,
the resulting device was "turn on BccSelf".
disabled `BccSelf` was probably the the number one single reason
of user problems.
main drawback of the change are potentially double notifications
when using a shared account and having another mail app on the same device.
however, we meanwhile do not recommend shared accounts at all,
the issue is also fixable by the other mail apps (as done by K-9)
and could be even regarded as a feature (you can decide which app to use for ansering).
but at the end the drawback is probably much smaller than the issues reported above.
* adapt tests to `BccSelf` enabled
* update CHANGELOG
Instead of emitting single MsgsChanged event
with zero chat and msg IDs, emit one event per message.
Also emit WebxdcInstanceDeleted event if expired message
contains a webxdc.
"IMAP folder and UID information" is no longer stored
in the `msgs` table since creation of the `imap` table.
As a result `msgs` table entries do not contain UID information in the
first place.
This commit updates documentation to reflect this change and also
points to `prune_tombstones()` procedure which actually deletes `msgs`
rows.
* truncate incoming messages by lines,
because many linebreaks seem to cause the chat open delay on deltachat-ios
* run cargo fmt
* remove DC_DESIRED_TEXT_LINES_THRESHOLD
and use Strings instead of Cow<str>
* remove usage of clippy::indexing_slicing in truncate_by_lines (#3596)
* adjust comments
* Fix truncate_by_lines tests
* Reword indexing/slicing error
* Remove unnecessary conditional
* Fix a typo in the comment
Co-authored-by: link2xt <link2xt@testrun.org>
update node constants, this was forgotten in #3592
not a big deal as these are generated on building node,
so this is just a simple commit of these files after running `npm run build`
* add more functions, see changelog for details
* add pr number to changelog
* clarify doc comment
* clarify usage of BasicChat
and adjust properties acordingly
r10s is right it should only contain what we need of the expensive calls
* fix doc typos
* run cargo fmt
* jsonrpc: add connectivity functions
* fix typo
* fix typo
* Add get_contact_encryption_info and get_connectivity_html
Fix get_connectivity_html and get_encrinfo futures not being Send. See https://github.com/rust-lang/rust/issues/101650 for more information.
Co-authored-by: jikstra <jikstra@disroot.org>
* Update CHANGELOG
* Update typescript files
* remove todo from changelog
Co-authored-by: jikstra <jikstra@disroot.org>
* add request_internet_access manifest option
* test request_internet_access
* update CHANGELOG
* force warning when internet access is enabled
if internet access is enabled,
show a warning instead of the normal summary
(the internet access is currently mainly to test out integrations
as maps for video chat; the summary is dispensable in the cases currently)
* adapt json-rpc's WebxdcMessageInfo
* get_chat_media() from any chat similar to search_msgs()
* do not return hidden media
this fixes the issue that drafts
and other hidden messages pop up in the gallery.
* add a test for get_chat_media()
* use None instead of ChatId::new(0)
* clarify scope of 'any' in get_chat_media()
* adapt json rpc to changed get_chat_media()
* jsonrpc: chat_get_media turn chat_id into option
and also still allow `0` for dev convenience
(though I'm not totally sure if thats the right decision)
* cargo fmt
Co-authored-by: Simon Laux <mobile.info@simonlaux.de>
Python bindings expect all functions defined in deltachat.h
to be available, even if there is no high-level interface.
scripts/run-python-test.sh doesn't work without this.
* integrate json-rpc repo
https://github.com/deltachat/deltachat-jsonrpc
* get target dir from cargo
* fix clippy
* use node 16 in ci
use `npm i` instead of `npm ci`
try fix ci script
and fix a doc comment
* fix get_provider_info docs
* refactor function name
* fix formatting
make test pass
fix clippy
* update .gitignore
* change now returns event names as id
directly, no conversion method or number ids anymore
also longer timeout for requesting test accounts from mailadm
* fix compile after rebase
* add json api to cffi and expose it in dc node
* add some files to npm ignore
that don't need to be in the npm package
* add jsonrpc crate to set_core_version
* add jsonrpc feature flag
* call a jsonrpc function in segfault example
* break loop on empty response
* fix closing segfault
thanks again to link2xt for figguring this out
* activate other tests again
* remove selectAccount from highlevel client
* put jsonrpc stuff in own module
* disable jsonrpc by default
* add @deltachat/jsonrpc-client
to make sure its dependencies are installed, too
whwn installing dc-node
* commit types.ts
that dc-node has everything it needs to provide @deltachat/jsonrpc-client
without an extra ts compile step
* improve naming
* Changes for tokio compat, upgrade to yerpc 0.3
This also changes the webserver binary to use axum in place of tide.
* Improvements to typescript package
* Improve docs.
* improve docs, fix example
* Fix CFFI for JSON-RPC changes
* use stable toolchain not 1.56.0
* fix ci
* try to fix ci
* remove emtpy file
allow unused code for new_from_arc
* expose anyhow errors
feature name was wrong
* use multi-threaded runtime in JSON-RPC webserver
* improve test setup and code style
* don't wait for IO on webserver start
* Bump yerpc to 0.3.1 with fix for axum server
* update todo document
remove specific api stuff for now,
we now have the an incremental aproach on moving
not the all at-once effort I though it would be
* remove debug logs
* changelog entry about the jsonrpc
* Fix method name casings and cleanups
* Improve JSON-RPC CI, no need to build things multiple times
* Naming consistency: Use DeltaChat not Deltachat
* Improve documentation
* fix docs
* adress dig's comments
- description in cargo.toml
- impl From<EventType> for EventTypeName
- rename `CommandApi::new_from_arc` -> `CommandApi::from_arc`
- pre-allocate if we know the entry count already
- remove unused enumerate
- remove unused serde attribute comment
- rename `FullChat::from_dc_chat_id` -> `FullChat::try_from_dc_chat_id`
* make it more idiomatic:
rename `ContactObject::from_dc_contact -> `ContactObject::try_from_dc_contact`
* apply link2xt's suggestions:
- unref jsonrpc_instance in same thread it was created in
- increase `max_queue_size` from 1 to 1000
* reintroduce segfault test script
* remove unneeded context
thanks to link2xt for pointing that out
* Update deltachat-ffi/deltachat.h
Co-authored-by: bjoern <r10s@b44t.com>
* Update deltachat-ffi/deltachat.h
Co-authored-by: bjoern <r10s@b44t.com>
* make sure to use dc_str_unref instead of free
on cstrings returned/owned by rust
* Increase online test timeouts for CI
* fix the typos
thanks to ralphtheninja for finding them
* restore same configure behaviour as desktop:
make configure restart io with the old configuration if it had one on error
* found another segfault:
this time in batch_set_config
* remove print from test
* make dcn_json_rpc_request return undefined instead of not returning
this might have been the cause for the second segfault
* add set_config_from_qr to jsonrpc
* add `add_device_message` to jsonrpc
* jsonrpc: add `get_fresh_msgs` and `get_fresh_msg_cnt`
* jsonrpc: add dm_chat_contact to ChatListItemFetchResult
* add webxdc methods to jsonrpc:
- `webxdc_send_status_update`
- `webxdc_get_status_updates`
- `message_get_webxdc_info`
* add `chat_get_media` to jsonrpc
also add viewtype wrapper enum and use it in `MessageObject`,
additionally to using it in `chat_get_media`
* use camelCase in all js object properties
* Add check_qr function to jsonrpc
* Fixed clippy errors and formatting
* Fixed formatting
* fix changelog ordering after rebase
* fix compile after merging in master branch
Co-authored-by: Simon Laux <mobile.info@simonlaux.de>
Co-authored-by: Simon Laux <Simon-Laux@users.noreply.github.com>
Co-authored-by: bjoern <r10s@b44t.com>
Co-authored-by: flipsimon <28535045+flipsimon@users.noreply.github.com>
* improve error handling for account setup from qrcode
closes#3192
* replace issue id with pr id in changelog
* Update src/qr.rs
Co-authored-by: bjoern <r10s@b44t.com>
* show response when it's invalid in success case, too
* fix changelog entry
Co-authored-by: bjoern <r10s@b44t.com>
We install `tox` via `pipx` in `Dockerfile`, there is no need to
configure path to `python3.7`.
Also remove use of `pushd`/`popd` and switch from `bash` to `/bin/sh`.
#3491 introduced a bug that your address is only replaced in the first group you write to, which was rather hard to fix. In order to be able to release something, we agreed to revert it and instead only replace the contacts in verified groups (and in broadcast lists, if the signing key is verified).
Highlights:
* Revert "Only do the AEAP transition in the chat where it happened"
This reverts commit 22f4cd7b79.
* Only do the transition for verified groups (and broadcast lists)
To be exact, only do the transition if the signing key fingerpring is
verified. And only do it in verified groups and broadcast lists
* Slightly adapt string to this change
* Changelog
musllinux images miss PyPy interpreters,
we want to skip building PyPy wheels for musl
instead of failing the build.
Also remove workaround from CI scripts.
Wheels are now published to PyPI, recommend it instead of devpi. We
build the wheels only for releases anyway.
Suggest using tox instead of listing all the pytest dependencies to
avoid keeping them up to date in the readme.
We no longer publish `docker/coredeps` images, they cannot be
pulled from a container registry anymore.
Troubleshooting section is outdated, because vsyscall emulation is
only needed for manylinux2010 images, not manylinux2014 or
musllinux_1_1 images.
Mention Podman as an alternative to Docker.
Link to https://py.delta.chat/ instead of only examples.
Remove note about wheels for Mac and Windows. Nobody requests them
anyway.
* save webxdc-updates for not yet downloaded messages, that are probably webxdc instances then
* test webxdc updates received while instance is not yet downloaded
* keep msg_id on downloading messages
keeping msg_id on downloading messages
has the advantage that webxdc updates and other references to the msg_id
can be processed as usual.
if a message expands to multiple msg_id,
the last one is kept,
however, this does not affect webxdc at all.
(alternatives may be to update `msgs_status_updates`
but that seems more complicated and even less elegant,
another alternative would be to use different keys (eg. `rfc274_mid`),
but that also seems not to be much easier and would waste space as well.
also both alternatives would need adaption for other foreign keys)
* update CHANGELOG
* do not emit WebxdcStatusUpdate event in case the message is not yet downloaded
* move DELETE/UPDATE to an transaction
* make merge_msg_id() a little less confusing
* use some webxdc-update-param from placeholder
(the placeholder may be updated,
the just downloaded messages is not)
* more precise function name
* test not directly downloading status updates
* test not directly downloading mdn
* do not `SELECT timestamp` if not used
ordering is by `id` since #2364, selecting `timestamp` is not needed.
(came over this when keeping `id` on downloading in #3487 -
had in mind there was sth. special with ids ...
however, the assumption of #2364 is even more true with #3487 -
before, new (and then maybe much larger) ids were inserted
and could result in wrong search result ordering)
* remove another unused `SELECT timestamp`
Remove unused docker-doxygen Dockerfile.
Switch scripts/run-doxygen.sh from bash to sh.
Use docker.io/alpine image instead of unsupported hrektts/doxygen
It happened multiple times now that I wanted to quickly execute a test, but because of a warning that had become an error, it didn't execute.
This turns warnings into warnings again; our CI will fail if there is a warning, anyway (because of RUSTFLAGS: -Dwarnings)
* ignore status/footer updates from mailinglist messages
mailinglist software often modified existing footers
or adds footers on their own.
therefore,therefore, these footers often do not reflect the status/footer set by the user.
* test status footers not updated from mailinglists
some mailinglists have their name in square brackets prepended to the subject.
we pick up that name and remove the square brackets,
as these do not look good as the chat name in delta chat.
if a mailing list has a sequence of square brackets, we use all of them
(this seems to be okayish, at least i do not know of any complains).
however, the removal of square brackets was not nice for sequences,
resulting in `ml topic] [sub topic`.
this pr removes the square brackets only for the first name
and leave the other ones as is.
* more flexible render_webxdc_status_update_object()
* delay webxdc updates when ratelimit is reached
* inject updates messages to the SMTP loop as needed
this avoids starting several tasks
and allows sending updates out after a restart of the app.
* use mutex to prevent race conditions in status updates
* check ratelimiter only before the sending loop; it won't change until messages are actually sent out
* fix typo
* prefer standard type declaration over turbofish syntax
* use UNIQUE and ON CONFLICT for query smtp_status_updates
* combine DELETE+SELECT to one atomic statement
* as all operations on smtp_status_updates are now atomic, a mutex is no longer needed
* test DELETE+RETURNING statement
* simplify calls to can_send()
* comment about ratelimit boolean in send_smtp_messages()
This makes quotes created by user display properly in other MUAs like
Thunderbird and not start with space in MUAs that don't support format=flowed.
To distinguish user-created quotes from top-quote inserted by Delta
Chat, a newline is inserted if there is no top-quote and the first
line starts with ">".
Nested quotes are not supported, e.g. line starting with "> >" will
start with ">" on the next line if wrapped.
* do not use ratelimiter for bots
i tried some other approaches as having optional ratelimiter
or handle `can_send()` for bots differently,
but all that results in _far_ more code and/or indirections -
esp. as the "bot" config can change and is also persisted -
and the ratelimiter is created
at a point where the database is not yet available ...
of course, all that could be refactored as well,
but this two-liner also does the job :)
* update CHANGELOG
This allows account manager to construct a single event channel and
inject it into all created contexts instead of aggregating events from
separate event emitters.
This makes sure that under normal circumstances the LoginParam struct
is always fully validated, ensure future use does not have to be
careful with this.
The brittle handling of `server_flags` is also abstraced away from
users of it and is now handled entirely internally, as the flags is
really only a boolean a lot of the flag parsing complexity is removed.
The OAuth2 flag is moved into the ServerLoginParam struct as it really
belongs in there.
New ratelimiter module counts number of sent messages and calculates
the time until more messages can be sent.
Rate limiter is currently applied only to MDNs. Other messages are
sent without rate limiting even if quota is exceeded, but MDNs are not
sent until ratelimiter allows sending again.
* force a reason when calling `set_msg_failed()`
the string is displayed to the user,
so even _some_ context as "NDN without further details"
is better than an empty string.
* make clippy happy
* add CHANGELOG entry
* clarify webxdc reference wrt info-messages
* add from_id parameter to add_info_msg_with_cmd()
* flag webxdc-info-messages as such
* set from_id to sender for webxdc-info-messages
* test additional webxdc info properties
* do not add series of similar info messages
instead, if on adding the last info message
is already from the same webxdc and sender,
just update the text
* test cleanup of webxdc info messages series
* update changelog
* make clippy happy
there is no real complexity in the args,
so allowing one more arg is probably fine.
if really wanted, we can refactor the function in another pr;
this pr is already complex enough :)
* use cleaner function names and comments
* clarify CHANGELOG
Not completely sure it's worth it since some other dependencies still
depend on it. Anyway, proc macros are said to be bad for compile times, I just typed out what the proc macro generates and it's only 8 more lines, and we're already doing it this way in e.g. action_by_contact() and collect_texts_recursive() (the latter needs the boxed future both for the trait and for recursion).
* do not wipe info for drafts
* drafts and received webxdc-instances, resent or not, do not trigger info-messages
* test that resending a webxdc does not not add legacy info-messages
* make clippy happy
* update CHANGELOG
bcc_self-updates are not received
due to the normal prevention of not even downloading these messages.
there is no extra defense of that on webxdc-level;
status-updates do not even have a "global-unique" id
(they have a local id and the message where they're wrappted into have the
"global-unique" Message-Id)
I added this poison_sender and poison_receiver stuff back when we had event listeners (which were called "sinks", too, but anyway), i.e. user-definable closures that were run in the events loop. This was to make sure that the test fails if the closure panics. But since we don't have them anymore and this code isn't supposed to panic anyway:
```rust
while let Some(event) = events.recv().await {
for sender in senders.read().await.iter() {
sender.try_send(event.clone()).ok();
}
}
```
* test contact name changes applied everywhere
this test is failing on current master,
a changed authname is set to `contact.authname` but not cached at `chat.name`;
resulting in `dc_chat_get_name()` returning a name
undiscoverable by `dc_get_chatlist(name)`.
* fix: update chat.name on contact.authname changes
* do read contact.display_name from database only of chat.name is empty
* update CHANGELOG
This ensures that no invalid states are possible,
like the one where cancel channel exists, but
no ongoing process is running, or the one where
the ongoing process is not allocated, but
should not be stopped.
mimeparser now handles try_decrypt() errors instead of simply logging
them. If try_decrypt() returns an error, a single message bubble
with an error is added to the chat.
The case when encrypted part is found in a non-standard MIME structure
is not treated as an encryption failure anymore. Instead, encrypted
MIME part is presented as a file to the user, so they can download the
part and decrypt it manually.
Because try_decrypt() errors are handled by mimeparser now,
try_decrypt() was fixed to avoid trying to load private_keyring if the
message is not encrypted. In tests the context receiving message
usually does not have self address configured, so loading private
keyring via Keyring::new_self() fails together with the try_decrypt().
Skipping of all [Gmail] folders was introduced to avoid scanning
virtual "[Gmail]/All Mail" folder. However, it also skips Sent and
Spam folders which reside inside [Gmail]. As a result
configured_sentbox_folder becomes unset after folder scan, making it
impossible to watch Sent folder, and Spam folder is never scanned.
This change makes Delta Chat identify virtual Gmail folders by their
flags, so virtual folders are skipped while Sent and Spam folders are
still scanned.
* don't start workflow on py-*tag
* move node.js tests to separate action
* node.js CI: move PR ID and tags to environment variables
* node.js CI: small bracket mistake
* delete prebuilds.tar.gz before packaging
* use node/README.md as npm README
* Rename aeap-mvp.rst to aeap-mvp.md
* Update aeap-mvp.md
* Apply suggestions from hpk's review
Co-authored-by: holger krekel <holger@merlinux.eu>
* Additions to holger's review
* Decide on TODOs
* Drop the "If we are going to assign a message to a chat, but the sender is not a member of this chat" condition again
Co-authored-by: holger krekel <holger@merlinux.eu>
The problem was in the handle_fingerprint_change() function which
attempted to add a warning about fingerprint change to all chats with
the contact.
This failed because of the SQL query failing to find the contact for
self in the `contacts` table. So the warning was not added, but at the
same time the whole decryption process failed.
The fix is to skip handle_fingerprint_change() for self addreses.
If there are no MOVE/DELETE operations pending, the folder
should not be SELECTed.
When `only_fetch_mvbox` is enabled, `fetch_new_messages` skips
most folders, but `move_delete_messages` always selects the
folder even if there are no operations pending. Selecting
all folders wastes time during folder scan and turns
recent messages into non-recent.
* make upload to previews fail if it's executed on a tag not a PR
* node-package.yml: use tag in uploaded file name
* rename file to node-§tag.tar.gz before upload
* get rid of bash error?
in fact, `get_chat_msgs()` with `marker1before` is not used on iOS,
however, iOS still needs the special chat-id.
in #3274 we did not check this possibility.
it may be changed, of course, however, that would requore some refactorings
in an anyway complicated area and is probably not worth the effort.
Hold the same write lock while checking if ongoing
process is already allocated and while allocating it.
Otherwise it is possible for two parallel processes
running alloc_ongoing() to decide that no ongoing
process is allocated and allocate two ongoing processes.
* create same contact-colors when addresses differ in upper-/lowercase
this leaves group-colors based on group names as is,
so, "MY GROUP" creates a different color than "my group",
as these names are better synced and also not an ID in this sense,
this is probably fine here.
(also when looking at the examples from
https://xmpp.org/extensions/xep-0392.html#testvectors-fullrange-no-cvd ,
case-sensistifity for group names seems to be fine)
* add a test for upper-/lowercase in group names
* update CHANGELOG
Google Workspace has an option "Append footer" which appends standard
footer defined by administrator to all outgoing messages. However,
there is no plain text part in encrypted messages sent by Delta Chat,
so Google Workspace turn the message into multipart/mixed MIME, where
the first part is an empty plaintext part with a footer and the second
part is the original encrypted message.
This commit makes Delta Chat attempt to repair such messages,
similarly to how it already repairs "Mixed Up" MIME structure in
`get_mixed_up_mime`.
don't ignore core sourcefiles,
prevented npm installation on architectures with no prebuild
don't run lint checks on windows
github actions don't like double quotes apparently
minimize node.js CI
update ubuntu runner to 22.04
README: update link to node bindings source
simplify link in readme
node: fix crash with invalid account id
(throw error when getContext failed)
fix typo in readme
remove node specific changelog
change prebuild machine back to ubuntu 18.04
move package.json to root level to include rust source in npm package
change path in m1 patch
github action to upload to download.delta.chat/node/ on tag
try build with ubuntu 20.04
Update node/README.md
try building it with newer ubuntu because it wants glibc 2.33
fix path for prebuildify script
throw error when instanciating a wrapper class on `null`
(Context, Message, Chat, ChatList and so on)
try fix selecting the right cache
to fix the strange glibc bug
also revert back ubuntu version to 18.04
also bump package.json version with release script
fix paths since we moved around package.json
github action: fix path
document npm release - it's so much easier now!
there are no PR checks to post to if this action is executed on a tag
github action: fix artifact names
fix paths? wtf do I know, it's 3AM and I'm drunk
fix syntax error
don't upload preview if action is run on tag
is the tag ID an empty string or null?
node-package github action is done so far
also include scripts in package
only publish docs on push to master branch
actually bump package.json version in set_core_version script
prettify package.json
fix test - we don't really need to assert that
remove unnecessary ls statement from github action
adjust scripts to new location of deltachat-core-rust
adjust dc-node readme to repo change
mention old repository
migrate github actions, try out if they work
fix path to node docs in SSH github action
passing mailadm token to node tests
hopefully fixing the download paths for the artifacts
fixing download paths, this time for real
post upload link to details
fix scp command
forgot to remove platform_status dict
fixing paths in the github action
add github action to delete node preview builds when PR is closed
move environment variable to yaml
remove git trash
github action to release to npm
use different action which also works with branches for testing
we don't want to publish to NPM through the CI
see what lint issues windows has
Scheduler has no Stopped state anymore. If Scheduler exists, it is
always started. Scheduler is stopped via Scheduler.stop(), which
consumes Scheduler and cannot fail.
Unlike jobs which are executed before sending normal messages, MDNs
from `smtp_mdns` table are sent after sending messages from `smtp`
table. This way normal messages have higher priority than MDNs.
There are no SMTP jobs anymore. All jobs are IMAP jobs, so
`jobs.thread` column is not used anymore.
.expect() may panic, which is probably not what we want here.
it seems better to bubble up the error (as we are doing in the other cases)
(i was checking some .expect usages after we had a similar issue at #3264)
* tagger.put_raw() has changes sematics and escapes strings on its own now
an explicit escaping leads to double escaping and to wrong display.
this should also improve lenght calculation,
as a quote and other specials counts as 1 character and not as 4-6.
* test encoding of generated qr-code-svg
* streamline function argument wording
use qrcode_description instead of raw_qrcode_description -
there is nothing "raw" in the argument,
it is a string as used throughout the app.
i was wondering mainly about the whole line
"Failed to start IO: scheduler is already stopped".
i think it has to be "already started" and not "already stopped".
Hold scheduler lock during the whole procedure of scheduler starting
and stopping. This ensures that two processes can't get two read locks
in parallel and start loops or send the stop signal twice.
Also remove shutdown channels: it is enough to wait
for the loop handle without receiving a shutdown signal
from the end of the loop.
Assign location.kml message parts to the trash chat,
but return non-trash chat_id so locations are assigned
to the correct chat.
Due to a bug introduced in
7968f55191
previously location.kml messages resulted in
empty message bubbles on the receiver.
* add dc_resend_msgs() to ffi
* add 'resend' to repl
* implement resend_msgs()
* allow only resending if allowed by chat-protection
this means, resending is denied if a chat is protected and we cannot encrypt
(normally, however, we should not arrive in that state)
* allow only resending of normal, non-info-messages
* allow only resending of own messages
* reset sending state to OutPending on resending
the resulting state is always OutDelivered first,
OutMdnRcvd again would be applied when a read receipt is received.
preserving old state is doable, however,
maybe this simple approach is also good enough, at least for now
(or maybe the simple approach is even just fine :)
another thing: when we upgrade to resending foreign messages,
we do not have a simple way to mark them as pending
as incoming message just do not have such a state -
but this is sth. for the future.
The limit on the number of jobs executed in a row was introduced to
prevent large queues of small jobs like MarkseenMsgOnImap, MoveMsg and
DeleteMsgOnImap from delaying message fetching. Since all these jobs
are now removed and IMAP operations they did are now batched, it is
impossible to have 20 or more queued IMAP jobs.
When removing an account, try 60 times with 1 second sleep in between
in case removal of database files fails. This happens on Windows
platform sometimes due to a known bug in r2d2 which may result in 30
seconds delay until all connections are closed [1].
[1] https://github.com/sfackler/r2d2/issues/99
MarkseenMsgOnImap job, that was responsible for marking messages as
seen on IMAP and sending MDNs, has been removed.
Messages waiting to be marked as seen are now stored in a
single-column imap_markseen table consisting of foreign keys pointing
to corresponding imap table records.
Messages are marked as seen in batches in the inbox loop. UIDs are
grouped by folders to reduce the number of requests, including folder
selection requests. UID grouping logic has been factored out of
move_delete_messages into UidGrouper iterator to avoid code duplication.
Messages are marked as seen right before fetching from the inbox
folder. This ensures that even if new messages arrive into inbox while
the connection has another folder selected to mark messages there, all
messages are fetched before going IDLE. Ideally marking messages as
seen should be done after fetching and moving, as it is a low-priority
task, but this requires skipping IDLE if UIDNEXT has advanced since
previous time inbox has been selected. This is outside of the scope of
this change.
MDNs are now queued independently of marking the messages as seen.
SendMdn job is created directly rather than after marking the message
as seen on IMAP. Previously sending MDNs was done in MarkseenMsgOnImap
avoid duplicate MDN sending by setting $MDNSent flag together with
\Seen flag and skipping MDN sending if the flag is already set. This
is not the case anymore as $MDNSent flag support has been removed in
9c077c98cd and duplicate MDN sending in
multi-device case is avoided by synchronizing Seen status since
833e5f46cc as long as the server
supports CONDSTORE extension.
* show 'Not connected' if storage information are not yet available
'One moment' is a bit misleading in case the device is offline
as it will take more than a moment until that will be updated :)
* update CHANGELOG
this bypasses the replication safety introduced by letting the caller track the last serial,
however, in case of bots that do not track much state and do not playback updates anyway,
it is still useful.
at least on iOS we would need to inject a script to force a behavior,
that is hard to merge with settings set by the Webxdc.
therefore, the easiest approach seems to be to leave that completely up to the Webxdc -
and, in practise, many Webxdc already set viewports, including scaling.
There is very little to be gained by having to approve those PRs, and
it is a lot of UI interaction to get them approved.
They still will need to be merged manually regardless, so they might
as well be approved by a bot.
* make Connectivity-View-HTML not scalable
in practise, Android and Desktop already disallow scaling
by some other methods,
however, for iOS this is needed as we otherwise
have to do far more complicated things as drafted at
https://github.com/deltachat/deltachat-ios/pull/1531/files
* update CHANGELOG
* only do monthly dependabot updates
* increase dependabot pr limit from 10 to 50 (due to only monthly interval, 10 seems to be too low)
Co-authored-by: B. Petersen <r10s@b44t.com>
* Do ephemeral deletion in background loop
1. in start_io start ephemeral async task, in stop_io cancel ephemeral async task
2. start ephemeral async task which loops like this:
- wait until next time a message deletion is needed or an interrupt occurs (see 3.)
- perform delete_expired_messages including sending MSGS_CHANGED events
3. on new messages (incoming or outgoing) with ephemeral timer:
- interrupt ephemeral async task
* Changelog
* Fix and improve test
* no return value needed
* address @link2xt review comments
* slight normalization: have only one place where we wait for interrupt_receiver
* simplify sql statement -- and don't exit the ephemeral_task if there is an sql problem but rather wait
* Remove now-unused `ephemeral_task` JoinHandle
The JoinHandle is now inside the Scheduler.
* fix clippy
* Revert accidental move of the line
* Add log
Co-authored-by: holger krekel <holger@merlinux.eu>
Co-authored-by: link2xt <link2xt@testrun.org>
* add min_api to manifest.toml, define WEBXDC_API_VERSION
* return an error instead of index.html if the webxdc requires a newer api
* add a test with an webxdc requiring a newer api
* update CHANGELOG
* Configure: Try "imap.*"/"smtp.*"/"mail.*" first
Fix half of #3158.
Try "imap.ex.org"/"smtp.ex.org" and "mail.ex.org" first because if a server exists
under this address, it's likely the correct one.
Try "ex.org" last because if it's wrong and the server is configured to
not answer at all, configuration may be stuck for several minutes.
* Changelog
* Add test
Today, we solved an issue with holger's phone that he couldn't export
his account anymore because during the `VACUUM` the Android system
killed the app because of OOM. The solution was to drop the table
`backup_blobs`, so let's automatically do this in migration
This table was used back in the olden days when we backuped by exporting
the dbfile and then putting all blobs into it. During import, the
`backup_blobs` table should have been dropped, seems like this didn't
work here.
* do not unarchive muted chats on sending/receving messages
* add a test for unarchive muted chats
* update CHANGELOG
* improve test_unarchive_if_muted
Previously first try used an old connection and. If using stale
connection, an immediate retry was performed using a new connection.
Now if connection is stale, it is closed immediately without trying to
use it.
This should reduce the delay in cases when old connection is unusable.
This change is aimed at decoupling parsing and
add_parts() stages to eventually separate parsing
from database changes and pipeline message parsing and
decryption.
This seems not only wasteful but genuinly has the risk someone makes
their device useless by accidentally adding a huge file.
This also re-structures the checks a little: The if-conditions are
flattened out and cheap checks are done before more expensive ones.
Replaced mutable out parameters with explicit return of structure.
Also moved all decisions about emitted events out of add_parts(). Chat
ID is removed from created_db_entries as it is the same for all parts.
* add simple backup export/import test
this test fails on current master
until the context is recrated.
* avoid config_cache races
adds needed SQL-statements to config_cache locking.
otherwise, another thread may alter the database
eg. between SELECT and the config_cache update -
resulting in the wrong value being written to config_cache.
* also update config_cache on initializing tables
VERSION_CFG is also set later, however,
not doing it here will result in bugs when we change DBVERSION at some point.
as this alters only VERSION_CFG and that is executed sequentially anyway,
race conditions between SQL and config_cache
seems not to be an issue in this case.
* clear config_cache after backup import
import replaces the whole database,
so config_cache needs to be invalidated as well.
we do that before import,
so in case a backup is imported only partly,
the cache does not add additional problems.
* update CHANGELOG
due to an bug from Apple copying files from/to iPhones
(cmp. https://support.delta.chat/t/import-backup-to-ios/1628/7 )
it may easily happen that one gets corrupted/partly backups.
such imports usually fail with some error,
however, for debugging it is nice to have the concrete file size in the log.
Currently if user moves the message into some other folder and then
moves the message back, the message is considered duplicate even
though previous copy was already deleted. This is a common problem
reported by users at least twice.
Keeping duplicates does no harm except for additional storage usage.
If the message is later deleted by the user, all the copies on the
server will be deleted. anyway.
This brings the Display of ContactId in line with those of ChatId etc,
which is a bit clearer is logs and such places.
It also updates an SQL query to rely on the ToSql impl of ContactId
rather than it's Display when building the query.
Holger had a case where he wrote with someone using a classing MUA.
He opened a two-member-group with this person (which also allowed him to
set the subject).
At some point the other person replied from a different email address.
What he expected: This reply should be sorted into the two-member-group.
What happened: This reply was sorted into the 1:1 chat.
---
I had added the line && chat_contacts.contains(&from_id) months ago when I wrote
this code because it seemed vaguely sensible but without any real
reason. So, let's remove it and see if it creates other problems -
my gut feeling is no.
This makes the contact ID its own newtype instead of being a plain
u32. The change purposefully does not yet try and reap any benefits
from this yet, instead aiming for a boring change that's easy to
review. Only exception is the ToSql/FromSql as not doing that yet
would also have created churn in the database code and it is easier to
go straight for the right solution here.
Message-IDs are now retrieved only during fetching and saved into imap
table. dc_receive_imf_inner does not attempt to extract the Message-ID
anymore.
For messages without Message-ID the ID is now generated in
imap::fetch_new_messages rather than dc_receive_imf_inner,
so the same ID is used in the imap table (maintained by the imap
module) and msgs table (maintained by dc_receive_imf module).
Message-ID generation based on the Date, From and To field hashing has
been replaced with a simple dc_create_id() to avoid retrieving Date,
From, and To fields in the imap module, as it's hard to test that it
stays compatible between Delta Chat versions in this module. This
breaks jump-to-quote for quoted messages without Message-ID, which is
not critical.
Also prefetch X-Microsoft-Original-Message-ID, so retrieval of
duplicate messages with X-Microsoft-Original-Message-ID can be skipped
like it is done for messages with Message-ID header.
* (r10s, adb, hpk) remove getAllUpdates() and add a typical replica-API that works with increasing serials. Streamline docs a bit.
* adapt ffi to new api
* documentation: updates serials may have gaps
* get_webxdc_status_updates() return updates larger than a given serial
* remove status_update_id from status-update-event; it is not needed (ui should update from the last known serial) and easily gets confused with last_serial
* unify wording to 'StatusUpdateSerial'
* remove legacy payload format, all known webxdc should be adapted
* add serial and max_serial to status updates
* avoid races when getting max_serial by avoiding two SQL calls
* update changelog
Co-authored-by: B. Petersen <r10s@b44t.com>
The state bob needs to maintain during a secure-join process when
exchanging messages used to be stored on the context. This means if
the process was killed this state was lost and the securejoin process
would fail. Moving this state into the database should help this.
This still only allows a single securejoin process at a time, this may
be relaxed in the future. For now any previous securejoin process
that was running is killed if a new one is started (this was already
the case).
This can remove some of the complexity around BobState handling: since
the state is in the database we can already make state interactions
transactional and correct. We no longer need the mutex around the
state handling. This means the BobStateHandle construct that was
handling the interactions between always having a valid state and
handling the mutex is no longer needed, resulting in some nice
simplifications.
Part of #2777.
* Add AcManager
See https://github.com/deltachat/deltachat-core-rust/pull/2901#issuecomment-998285039
This reduces boilerplate code again therefore, improving the
signal-noise-ratio and reducing the mental barrier to start
writing a unit test.
Slightly off-topic:
I didn't add any advanced functions like `manager.get("alice");` because
they're not needed yet; however, once we have the AcManager we can
think about fancy things like:
```rust
acm.send_text(&alice, "Hi Bob, this is Alice!", &bob);
```
which automatically lets bob receive the message.
However, this may be less useful than it seems at first, since most of
the tests I looked at wouldn't benefit from it, so at least I won't do
it until I have a test that would benefit from it.
* Remove unnecessary RefCell
* Rename AcManager to TestContextManager
* Don't store TestContext's in a vec for now as we don't need this; we can re-add it later
* Rename acm -> tcm
This moves most common headers like From, To, Subject etc. defined in
the Internet Message Format standard at the top of the message
in the same order as used in RFC 5322.
When `smtp_send` returns `Status::Finished`,
the message should be removed from the queue even in case of
failure, such as a permanent error.
In addition to this bugfix, move the retry count increase to
the beginning of `send_msg_to_smtp` to ensure no message is
retried infinitely even in case of similar bugs.
fix#3007
My approach is:
We don't download any messages from the spam folder anymore, and only download them if they were moved out. This means that is-it-spam logic only resides in spam_target_folder(). This has some implications, see the comments.
pytest-timeout already handles all deadlocks and is configurable with
--timeout option. With this change it is possible to disable timeout
with --timeout 0 to run tests on extremely slow connections.
Also change how NO response is treated. NO response means there is an
error moving/copying the messages. When there are no matching
messages, the response is "OK No matching messages, so nothing copied"
according to some RFC 9051 examples.
this avoids archived chats containing fresh messages:
before, it could happen that between the two SQL calls
a new fresh message arrives,
unarchives the chat that is immediately archived by the second SQL call -
resulting in an archive chat containing fresh messages.
as fresh messages counter are shown on app icon etc.
this is pretty weird for the user as they do not see what is "fresh".
the other way round,
there is no transaction in receive_imf(),
however, receive_imf() only unarchives chats,
so that is visible and no big issue for the user.
the issue is rare at all,
however, annoying if you get that as the badge counter may be stuck at "1"
nearly forever (until you open the archived chat in question).
- do not attempt to mark reserved meessages as seen when
messages with empty Message-ID are marked as seen on IMAP
- do not reconnect on Seen flag synchronization failures This avoid
reconnection loops in case of permanent errors in `sync_seen_flags`
* Make set_config() look a bit nicer
* Add OnlyFetchMvbox option
* Add test for the config
* Add option to only watch mvbox
This is supposed to support having a server-side rule which moves
emails to the mvbox already. The new option makes sure the mvbox is
wathched and also makes sure no messages are feched from folders other
than the mvbox and the spam folder if enabled. It does not interact
with the other settings.
* Fixup ignore conditions
* Cleanup some bits
* Watch the mvbox when `WatchMvboxOnly` is set
* Rename back to only_fetch_mvbox (flub said it's OK for him)
* typo
* clippy, more typos
Co-authored-by: Hocuri <hocuri@gmx.de>
Synchronizing seen flags doubles the time required to scan all
folders. Delta Chat only marks messages as Seen on Inbox or DeltaChat,
so there is no need to check for Seen flag on other folders.
This operation takes roughly 0.3 s on a moderate size database.
Calling it once before scanning all folders and scanning
the watched folder instead of each time after downloading
a message from a folder speeds up IMAP loop.
This operation takes roughly 0.3 s on a moderate size database.
Calling it once before scanning all folders and scanning
the watched folder instead of each time after downloading
a message from a folder speeds up IMAP loop.
this uses `get_webxdc_info().name` for chatlist etc.
the previuosly used static strings comes from a time
where we just did not had the correct name.
i was also thinking about adding `get_webxdc_info().summary`,
however, as this information is dynamic,
that may open several issues, eg. quoted text may change
so that the answer is out of context.
As of #2980, the migrations were not run after importing a backup. This lead to errors like "Failed to send message: no such table: smtp" if you imported a backup from a previous DC version.
as we had the changelog-chat only in the last days,
- add missing pr
- update some wrong references
- unify some layout and wordings,
streamline headlines to "API Changes", "Changes" and "Fixes",
finer subdivisions only raise noise, duplicates and discussions
and, if not used consequently, do not add much benefit.
as discussed in dev chat,
100 kb is too low if one wants to use any game framework.
also, many game that doesn't have poor graphics
or uses magic CSS-tricks to do fancy draws,
will have more than 100 kb probably just in tiles and sprites.
even for regions with low resources,
100 kb is low for a game -
esp. if we compare that with stickers that are send around as well
are not deduplicated and also have several 100 kb in size -
for much less effect.
we should still encourage ppl to create tiny apps,
however, a too low limit also restricts possibilities wrt adaption.
already before,
we did _not_ sent updates to contact requests or other read-only-chats.
however, we _did_ modify the local database,
so that getAllUpdates() returns an update that was not sent out to other peers.
this is fixed by checking can_send() soon.
that way, all peers have the same state
also for contact requests or other read-only-chats
and webxdc in contact requests can be opened as usual.
further ui improvements may be needed for contact requests
(maybe allow the webxdc to know about that state or
maybe show a warning in the ui somewhere),
however, this is not part of this pr.
* let sending invalid webxdc fail
invalid webxdc can still be send as Viewtype::File, however
(maybe one want to discuss errors or so ;)
* clarify the supported zip compression methods
* add a test for unencrypted replies to encrypted webxdc instances
* make update messages work if a key is missing
even in opportunistic chats,
replies to encrypted messages are forced to be encrypted,
if that is not possbile, message sending fails.
while this is okay to not leak previously send text messages,
the quotes as used by webxdc are more artificial,
currently only the static text "Webxdc".
* changelog ...
This was written in a way which attempted to avoid easily creating an
infinite loop. But really that's a python idiom and doesn't work very
well in Rust. Worse, as shown by #2972 it is really easy to still get
this wrong. Instead do this the rust way, this way the compiler can
also reason properly about the branches and what is unreachable
removing some bogus dead code.
* fix forwarding webxdc instances, add a test for that
* adapt webxdc test to fail on info-messages added when in draft mode
* do not add info-messages when in draft-mode
* half the number of instance-loads per webxdc update send/receive
While experimenting with encrypted storage, once configuring failed between 920 and 940. But as the "configured" config had already been written after progress 910, some part of the code thought we are configured, some didn't.
With this PR, my encrypted-storage Android PR now works, at least I
couldn't find any further bugs.
Without it, configuring fails with: `Failed to create blob
icon-saved-messages-1803424689.png in
/data/user/0/com.b44t.messenger.beta/files/accounts/0e402b37dcd14a9586aea46294c908f2/dc.db-blobs:
No such file or directory (os error 2)`.
Also see https://github.com/deltachat/deltachat-core-rust/pull/2972.
To create encrypted account with account manager, call
dc_accounts_add_closed_account(). Open this account with
dc_context_open() using the passphrase you want to use for encryption.
When application is loaded next time and account manager is created,
it will open all accounts that have no passphrase set. For encrypted
accounts dc_context_is_open() will return 0. To open them, call
dc_context_open() with the correct passphrase. After opening, call
dc_context_start_io() on this account or just dc_accounts_start_io()
to start all accounts that are not started yet.
Support for legacy SQLite-based backup format is removed in this
commit.
- Replace .ok_or_else() and .map_err() with anyhow::Context where possible.
- Use .context() to check Option for None when it's an error
- Resultify Chatlist.get_chat_id()
- Add useful .context() to some errors
- IMAP error handling cleanup
Seen status is only synchronized on servers supporting IMAP CONDSTORE
extension. At the end of fetch loop iteration, flags are fetched for
all messages modified since previous synchronization and highest
modification sequence is stored into `imap_sync` table.
It slows down dc_get_chat_msgs() from ~50ms to seconds on Android. It
is disabled by default in SQLCipher 4.5.0, but currently SQLCipher 4.4.3
is bundled so this PRAGMA has to be disabled manually.
This is an irrecoverable error, dc_receive_imf must not fail on it
as it prevents last seen UID from advancing, so the same message
is prefetched on each iteration of IMAP loop.
the summary can be modified by the apps using
`sendUpdate({summary: "foo", payload: ...})`
the summary is updated when there is no newer update
and chat will be informed by the change as usual by
`DC_EVENT_MSGS_CHANGED`.
additional parameters will be `summary` and `msg` (for an info-message)
right now, and maybe more in the future (`aggregated`, `fast`, `to` ...),
so adding just more parameters easily gets wild.
also, this makes adaptions easier as no ffi needs to be adapted
when we add more parameters.
finally, sending is in-sync with receiving, database and wire now,
one receives similar objects as one sends,
which also looks like the right thing :)
for now, `sendUpdate()` also allows to use the old, unwrapped payload
to not immediately break existing `.xdc`, however, that will be removed soon,
probably before the first release.
Previously default of `auto_vacuum=NONE` was used.
`PRAGMA auto_vacuum=INCERMENTAL` allows running `PRAGMA
incremental_vacuum`. Unlike `VACUUM`, `PRAGMA incremental_vacuum`
frees unused pages without rebuilding the whole database, like the
`VACUUM` does, so it does not require additional disk space to build a
vacuumed copy of the database. This may even free enough space on the
disk to run `VACUUM` afterwards.
New setting will only be enabled for new databases or if the `VACUUM`
command runs successfully. Currently `VACUUM` is only executed on
backup export, but may fail nevertheless if there is not enough space
on the disk.
Also try to run `PRAGMA incremental_vacuum` during housekeeping. It
may not be the best strategy, but likely does not make any difference
under normal usage when the database only grows and there are no free
pages. Free pages are created only if enough data is deleted to free
at least one database page of 4096 bytes, for example when automatic
deletion of messages is deleted for the first time. In the future if
more data is placed into the database, like avatars and other blobs,
it may be necessary to revise this strategy, for example to keep some
free pages instead of removing all of them each time by querying
`PRAGMA freelist_pages` and running `PRAGMA incremental_vacuum(N)`.
Also resultified `get_watched_folders`.
Python test `test_moved_markseen` is modified to test using incoming
message instead of BCC-self message, because BCC-self message is
detected immediately in the Inbox.
We had an unhelpful log in the Testing group:
- it repeatedly says "src/imap.rs:1010: dc_receive_imf error: add_parts error" but the actual error is not shown
- it says "deltachat-ffi/src/lib.rs:186: dc_get_config(): invalid key" but it doesn't say what key it's trying to set
I considered removing it from the context by default, but the
migration test really wants to have the tracker initialised from the
very first event and not after the context is initialised. It is
easier for now to leave it hardcoded instead of adding an API to
explicitly require enabling it via the builder.
This replaces the EventSink callbacks with simple channel senders.
This simplifies the TestContext a lot as that is much simpler to
handle. It then also removes the special-casing of the LogSink since
it now is another even sender, only injected at the very start.
There are too many ways to create a TestContext, this introduces a
TestContextBuilder to try and keep this shorter. It also cleans up
the existing constructors keeping only the commonly used ones.
Without this the test output is written somewhere random and ends up
in the wrong test report. This buffers all the log output and prints
it inside the test on dropping the LogSink, or on dropping the
TestContext if no explicit LogSink was created.
`imap` table maps Message-IDs to UIDs on the server. `dc_receive_imf`
no longer gets the UID of the message as an argument and does not
insert the folder and UID of the message into the `msgs`
table. `server_folder` and `server_uid` columns in `msgs` table are
deprecated.
MoveMsg and DeleteMsgOnImap jobs are removed. Now messages are moved
and deleted only in the `fetch_move_delete` procedure that consults
the `target` column of the `imap` table to determine where the message
should go.
Where the message should go is determined after prefetching by the
`imap::target_folder()` procedure. Messages are only downloaded once
they reach their target folder to avoid race conditions in multidevice
setting, such as:
1. One device trying to FETCH the message while the other tries to
MOVE it.
2. One device marking the message as \Seen in the Inbox while the
other has already copied unseen message to the Movebox and is going to
delete the \Seen message in the Inbox.
3. Device downloads the message from the Inbox while there are newer
messages in the Movebox placed there by the other device, thus
processing the messages out of order.
This runs all CI jobs with -Dwarnings, turning warnings into errors
for CI.
This is useful since we run rustfmt and clippy on CI on stable rust,
while the builds and tests run with a specific older rustc. The
latter is also used for local development usually since it is encoded
in the rust-toolchain file. This warnings in clippy jobs of stable
rust would often go unnoticed, however stable rust usually finds more
genuine issues than the older compiler we use.
- Check that member modifying the group is in the group themselves.
It is not allowed to readd yourself to the group or remove someone without being in the group themselves.
- Unify code for group member addition and removal.
Removing a member now recreates the group member list from the To: field.
Removed member from the Chat-Group-Member-Removed was in the To: field in previous Delta Chat versions,
so it is excluded explicity. New versions of Delta Chat put removed member in Bcc: instead.
- Apply avatar changes after updating the group member list.
This allows to check that the contact modifying the avatar is actually a group member.
* store msg-id in msg-object on set_draft(), add a test for that
* test deleting drafts
* keep draft-ids on updating drafts, set draft-state
* do not allow forwarding of drafts
in general, it should be possbile,
however, it is not needed.
drafts and forwarding have lots of cornercases
even when not used in combination :)
* keep draft-ids on preparing and sending
* add comments about keeping msg_id
* early exit when trying to forward drafts
* tweak tests
* get rid of old C to Rust conversion code
* allow soon checking of increation-state, add a test for that
They contain X-MSExch-Correlation-Key header, but no
Original-Message-ID, so they cannot be used to find the original
message, but we want to recognize them as MDN nevertheless to assign
them to the trash chat.
* we no longer set a draft on group creation
* remove probably outdated comment
the comment was added at
https://github.com/deltachat/deltachat-core/pull/457
where draft revamping was done half only
and was probably forgotten to being removed later.
at least, i cannot make any sense out of the comment.
When there is an outgoing message in a chat, mark all older messages in this chat as seen.
Android already has a similar behavior, however, this led to the issue https://github.com/deltachat/deltachat-android/issues/2163 and should be changed back.
--
From the issue description at https://github.com/deltachat/deltachat-android/issues/2163, I implemented these fixes:
> Core should take care that if the last message in a chat is not fresh|noticed, no messages in the chat can be fresh. [...] Do this [...] in a function that's called at the end of fetch_new_messages(). Then dc_receive_imf() wouldn't get slower by this and we could re-use the same function for migration.
So, I didn't do this inside `dc_receive_imf()` in order not to make it take even longer. This obviously has the downside of higher complexity.
And I think we should implement this:
> On Androd, show the unread badge when unread!=0 again (see deltachat/deltachat-android@618af02). Then the user can see that there is a chat with an unread message and click it to get rid of it.
because it shouldn't be the UI's job to decide whether an unread badge is shown, but the core's.
ASCII armored keys can be easily generated with `sq key generate` and
used to encrypt and decrypt test messages with `sq` and `gpg` without
converting them to binary using `base64 -d` first.
unknown keys may come from upgrades (previously used key no longer defined)
or from downgrades (when an upgrade before uses a new key).
the latter was probalby responsible for some avatar loss,
mainly by testers, that usually switch versions forth and back much more often.
* fix missing MX resolver eg. on android
switching completely to /etc/resolv.conf (see #2780)
does not work at least on some Androids
(see https://github.com/deltachat/deltachat-android/issues/2151 )
therefore, we use the old approach as a fallback.
* log a warning, when we again have problems with figuring out MX resolvers
* test adding members in a multi-device setup
* fix system messages for multi-device-setup
* enhance test to check multi-device messages for removing-members and group-renames
* repl: allow groupname arguments with more than one word (came over that when testing qr codes)
* calcualte text-size from the real number of lines
* shift text and watermark apart when text get longer
* make clippy happy
This function is used to lookup the chat by `References` and
`In-Reply-To` header, so it does not make sense to return trashed
message when there is another non-trashed message in one of these
headers with a real chat ID.
- emit "contacts changed" event when the contact is no longer "seen recently" #3703
- do not allow peerstate reset if DKIM check failed #3731
## 1.98.0
### API-Changes
- jsonrpc: typescript client: export constants under `C` enum, similar to how its exported from `deltachat-node`#3681
- added reactions support #3644
- jsonrpc: reactions: added reactions to `Message` type and the `sendReaction()` method #3686
### Changes
- simplify `UPSERT` queries #3676
### Fixes
## 1.97.0
### API-Changes
- jsonrpc: add function: #3641, #3645, #3653
-`getChatContacts()`
-`createGroupChat()`
-`createBroadcastList()`
-`setChatName()`
-`setChatProfileImage()`
-`downloadFullMessage()`
-`lookupContactIdByAddr()`
-`sendVideochatInvitation()`
-`searchMessages()`
-`messageIdsToSearchResults()`
-`setChatVisibility()`
-`getChatEphemeralTimer()`
-`setChatEphemeralTimer()`
-`getLocations()`
-`getAccountFileSize()`
-`estimateAutoDeletionCount()`
-`setStockStrings()`
-`exportSelfKeys()`
-`importSelfKeys()`
-`sendSticker()`
-`changeContactName()`
-`deleteContact()`
-`joinSecurejoin()`
-`stopIoForAllAccounts()`
-`startIoForAllAccounts()`
-`startIo()`
-`stopIo()`
-`exportBackup()`
-`importBackup()`
-`getMessageHtml()`#3671
-`miscGetStickerFolder` and `miscGetStickers`#3672
- breaking: jsonrpc: remove function `messageListGetMessageIds()`, it is replaced by `getMessageIds()` and `getMessageListItems()` the latter returns a new `MessageListItem` type, which is the now prefered way of using the message list.
- jsonrpc: add type: #3641, #3645
-`MessageSearchResult`
-`Location`
- jsonrpc: add `viewType` to quoted message(`MessageQuote` type) in `Message` object type #3651
### Changes
- Look at Authentication-Results. Don't accept Autocrypt key changes
if they come with negative authentiation results while this contact
sent emails with positive authentication results in the past. #3583
- jsonrpc in cffi also sends events now #3662
- jsonrpc: new format for events and better typescript autocompletion
- Join all "[migration] vXX" log messages into one
### Fixes
- share stock string translations across accounts created by the same account manager #3640
- suppress welcome device messages after account import #3642
- fix unix timestamp used for daymarker #3660
## 1.96.0
### Changes
- jsonrpc js client:
- Change package name from `deltachat-jsonrpc-client` to `@deltachat/jsonrpc-client`
- remove relative file dependency to it from `deltachat-node` (because it did not work anyway and broke the nix build of desktop)
- ci: add github ci action to upload it to our download server automaticaly on realease
## 1.95.0
### API-Changes
- jsonrpc: add `mailingListAddress` property to `FullChat`#3607
-over jsonrpc built with napi.rs: \[[📂 source](https://github.com/deltachat/napi-jsonrpc) | [📦 npm](https://www.npmjs.com/package/@deltachat/napi-jsonrpc)\]
[^1]: Out of date / unmaintained, if you like those languages feel free to start maintaining them. If you have questions we'll help you, please ask in the issues.
This crate provides a [JSON-RPC 2.0](https://www.jsonrpc.org/specification) interface to DeltaChat.
The JSON-RPC API is exposed in two fashions:
* A executable that exposes the JSON-RPC API through a [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) server running on localhost.
* The JSON-RPC API can also be called through the [C FFI](../deltachat-ffi). The C FFI needs to be built with the `jsonrpc` feature. It will then expose the functions `dc_jsonrpc_init`, `dc_jsonrpc_request`, `dc_jsonrpc_next_response` and `dc_jsonrpc_unref`. See the docs in the [header file](../deltachat-ffi/deltachat.h) for details.
We also include a JavaScript and TypeScript client for the JSON-RPC API. The source for this is in the [`typescript`](typescript) folder. The client can easily be used with the WebSocket server to build DeltaChat apps for web browsers or Node.js. See the [examples](typescript/example) for details.
## Usage
#### Running the WebSocket server
From within this folder, you can start the WebSocket server with the following command:
```sh
cargo run --features webserver
```
If you want to use the server in a production setup, first build it in release mode:
```sh
cargo build --features webserver --release
```
You will then find the `deltachat-jsonrpc-server` executable in your `target/release` folder.
The executable currently does not support any command-line arguments. By default, once started it will accept WebSocket connections on `ws://localhost:20808/ws`. It will store the persistent configuration and databases in a `./accounts` folder relative to the directory from where it is started.
The server can be configured with environment variables:
|variable|default|description|
|-|-|-|
|`DC_PORT`|`20808`|port to listen on|
|`DC_ACCOUNTS_PATH`|`./accounts`|path to storage directory|
If you are targetting other architectures (like KaiOS or Android), the webserver binary can be cross-compiled easily with [rust-cross](https://github.com/cross-rs/cross):
The package includes a JavaScript/TypeScript client which is partially auto-generated through the JSON-RPC library used by this crate ([yerpc](https://github.com/Frando/yerpc/)). Find the source in the [`typescript`](typescript) folder.
To use it locally, first install the dependencies and compile the TypeScript code to JavaScript:
```sh
cd typescript
npm install
npm run build
```
The JavaScript client is not yet published on NPM (but will likely be soon). Currently, it is recommended to vendor the bundled build. After running `npm run build` as documented above, there will be a file `dist/deltachat.bundle.js`. This is an ESM module containing all dependencies. Copy this file to your project and import the DeltaChat class.
```typescript
import{DeltaChat}from'./deltachat.bundle.js'
constdc=newDeltaChat('ws://localhost:20808/ws')
constaccounts=awaitdc.rpc.getAllAccounts()
console.log('accounts',accounts)
```
A script is included to build autogenerated documentation, which includes all RPC methods:
```sh
cd typescript
npm run docs
```
Then open the [`typescript/docs`](typescript/docs) folder in a web browser.
## Development
#### Running the example app
We include a small demo web application that talks to the WebSocket server. It can be used for testing. Feel invited to expand this.
```sh
cd typescript
npm run build
npm run example:build
npm run example:start
```
Then, open [`http://localhost:8080/example.html`](http://localhost:8080/example.html) in a web browser.
Run `npm run example:dev` to live-rebuild the example app when files changes.
### Testing
The crate includes both a basic Rust smoke test and more featureful integration tests that use the TypeScript client.
#### Rust tests
To run the Rust test, use this command:
```
cargo test
```
#### TypeScript tests
```
cd typescript
npm run test
```
This will build the `deltachat-jsonrpc-server` binary and then run a test suite against the WebSocket server.
The test suite includes some tests that need online connectivity and a way to create test email accounts. To run these tests, talk to DeltaChat developers to get a token for the `testrun.org` service, or use a local instance of [`mailadm`](https://github.com/deltachat/docker-mailadm).
Then, set the `DCC_NEW_TMP_EMAIL` environment variable to your mailadm token before running the tests.
```
DCC_NEW_TMP_EMAIL=https://testrun.org/new_email?t=yourtoken npm run test
```
#### Test Coverage
Running `npm run test` will report test coverage. For the coverage to be accurate the online tests need to be run.
> If you are offline and want to see the coverage results anyway (even though they are inaccurate), you can bypass the errors of the online tests by setting the `COVERAGE_OFFLINE=1` environment variable.
A summary of the coverage will be reported in the terminal after the test run. Open `coverage/index.html` in a web browser for a detailed report.
- [ ] different test type to simulate two devices: to test autocrypt_initiate_key_transfer & autocrypt_continue_key_transfer
## MVP - Websocket server&client
For kaiOS and other experiments, like a deltachat "web" over network from an android phone.
- [ ] coverage for a majority of the API
- [ ] Blobs served
- [ ] Blob upload (for attachments, setting profile-picture, importing backup and so on)
- [ ] other way blobs can be addressed when using websocket vs. jsonrpc over dc-node
- [ ] Web push API? At least some kind of notification hook closure this lib can accept.
### Other Ideas for the Websocket server
- [ ] make sure there can only be one connection at a time to the ws
- why? , it could give problems if its commanded from multiple connections
- [ ] encrypted connection?
- [ ] authenticated connection?
- [ ] Look into unit-testing for the proc macros?
- [ ] proc macro taking over doc comments to generated typescript file
## Desktop Apis
Incomplete todo for desktop api porting, just some remainders for points that might need more work:
- [ ] manual start/stop io functions in the api for context and accounts, so "not syncing all accounts" can still be done in desktop -> webserver should then not do start io on all accounts by default
"CAN NOT RUN COVERAGE correctly: Missing DCC_NEW_TMP_EMAIL environment variable!\n\n",
"You can set COVERAGE_OFFLINE=1 to circumvent this check and skip the online tests, but those coverage results will be wrong, because some functions can only be tested in the online test"
- The secondary self addresses (see below) are shown in the UI, but not editable.
- When the user changed the email address in the configure screen, show a dialog to the user, either directly explaining things or with a link to the FAQ (see "Other" below)
Changes in the core
-------------------
- [x] We have one primary self address and any number of secondary self addresses. `is_self_addr()` checks all of them.
- [x] If the user does a reconfigure and changes the email address, the previous address is added as a secondary self address.
- don't forget to deduplicate secondary self addresses in case the user switches back and forth between addresses).
- The key stays the same.
- [x] No changes for 1:1 chats, there simply is a new one. (This works since, contrary to group messages, messages sent to a 1:1 chat are not assigned to the group chat but always to the 1:1 chat with the sender. So it's not a problem that the new messages might be put into the old chat if they are a reply to a message there.)
- [ ] When sending a message: If any of the secondary self addrs is in the chat's member list, remove it locally (because we just transitioned away from it). We add a log message for this (alternatively, a system message in the chat would be more visible).
- [x] ([#3385](https://github.com/deltachat/deltachat-core-rust/pull/3385)) When receiving a message: If the key exists, but belongs to another address (we may want to benchmark this)
AND there is a `Chat-Version` header\
AND the message is signed correctly
AND the From address is (also) in the encrypted (and therefore signed) headers <sup>[[1]](#myfootnote1)</sup>\
AND the message timestamp is newer than the contact's `lastseen` (to prevent changing the address back when messages arrive out of order) (this condition is not that important since we will have eventual consistency even without it):
Replace the contact in _all_ groups, possibly deduplicate the members list, and add a system message to all of these chats.
- Note that we can't simply compare the keys byte-by-byte, since the UID may have changed, or the sender may have rotated the key and signed the new key with the old one.
<aname="myfootnote1">[1]</a>: Without this check, an attacker could replay a message from Alice to Bob. Then Bob's device would do an AEAP transition from Alice's to the attacker's address, allowing for easier phishing.
<details>
<summary>More details about this</summary>
Suppose Alice sends a message to Evil (or to a group with both Evil and Bob). Evil then forwards the message to Bob, changing the From and To headers (and if necessary Message-Id) and replacing `addr=alice@example.org;` in the autocrypt header with `addr=evil@example.org;`.
Then Bob's device sees that there is a message which is signed by Alice's key and comes from Evil's address and would do the AEAP transition, i.e. replace Alice with Evil in all groups and show a message "Alice changed their address from alice@example.org to evil@example.org". Disadvantages for Evil are that Bob's message will be shown on Alice's device, possibly creating confusion/suspicion, and that the usual "Setup changed for..." message will be shown the next time Evil sends a message (because Evil doesn't know Alice's private key).
Possible mitigations:
- if we make the AEAP device message sth. like "Automatically removed alice@example.org and added evil@example.org", then this will create more suspicion, making the phishing harder (we didn't talk about what what the wording should be at all yet).
- Add something similar to replay protection to our Autocrypt implementation. This could be done e.g. by adding a second `From` header to the protected headers. If it's present, the receiver then requires it to be the same as the outer `From`, and if it's not present, we don't do AEAP --> **That's what we implemented**
Note that usually a mail is signed by a key that has a UID matching the from address.
That's not mandatory for Autocrypt (and in fact, we just keep the old UID when changing the self address, so with AEAP the UID will actually be different than the from address sometimes)
> The content of the user id packet is only decorative
</details>
### Notes:
- We treat protected and non-protected chats the same
- We leave the aeap transition statement away since it seems not to be needed, makes things harder on the sending side, wastes some network traffic, and is worse for privacy (since more pepole know what old addresses you had).
- As soon as we encrypt read receipts, sending a read receipt will be enough to tell a lot of people that you transitioned
- AEAP will make the problem of inconsistent group state worse, both because it doesn't work if the message is unencrypted (even if the design allowed it, it would be problematic security-wise) and because some chat partners may have gotten the transition and some not. We should do something against this at some point in the future, like asking the user whether they want to add/remove the members to restore consistent group state.
#### Downsides of this design:
- Inconsistent group state: Suppose Alice does an AEAP transition and sends a 1:1 message to Bob, so Bob rewrites Alice's contact. Alice, Bob and Charlie are together in a group. Before Alice writes to this group, Bob and Charlie will have different membership lists, and Bob will send messages to Alice's new address, while Charlie will send them to her old address.
#### Upsides:
- With this approach, it's easy to switch to a model where the info about the transition is encoded in the PGP key. Since the key is gossiped, the information about the transition will spread virally.
- Faster transation: If you send a message to e.g. "Delta Chat Dev", all members of the "sub-group" "delta android" will know of your transition.
### Alternatives and old discussions/plans:
- Change the contact instead of rewriting the group member lists. This seems to call for more trouble since we will end up with multiple contacts having the same email address.
- If needed, we could add a header a) indicating that the sender did an address transition or b) listing all the secondary (old) addresses. For now, there is no big enough benefit to warrant introducing another header and its processing on the receiver side (including all the neccessary checks and handling of error cases). Instead, we only check for the `Chat-Version` header to prevent accidental transitions when an MUA user sends a message from another email address with the same key.
- The condition for a transition temporarily was:
> When receiving a message: If we are going to assign a message to a chat, but the sender is not a member of this chat\
> AND the signing key is the same as the direct (non-gossiped) key of one of the chat members\
> AND ...
However, this would mean that in 1:1 messages can't trigger a transition, since we don't assign private messages to the parent chat, but always to the 1:1 chat with the sender.
<details>
<summary>Some previous state of the discussion, which temporarily lived in an issue description</summary>
Summarizing the discussions from https://github.com/deltachat/deltachat-core-rust/pull/2896, mostly quoting @hpk42:
1. (DONE) At the time of configure we push the current primary to become a secondary.
2. When a message is sent out to a chat, and the message is encrypted, and we have secondary addresses, then we
a) add a protected "AEAP-Replacement" header that contains all secondary addresses
b) if any of the secondary addresses is in the chat's member list, we remove it and leave a system message that we did so
3. When an encrypted message with a replacement header is received, replace the e-mail address of all secondary contacts (if they exist) with the new primary and drop a sysmessage in all chats the secondary is member off. This might (in edge cases) result in chats that have two or more contacts with the same e-mail address. We might ignore this for a first release and just log a warning. Let's maybe not get hung up on this case before everything else works.
Notes:
- for now we will send out aeap replacement headers forever, there is no termination condition other than lack of secondary addresses. I think that's fine for now. Later on we might introduce options to remove secondary addresses but i wouldn't do this for a first release/PR.
- the design is resilient against changing e-mail providers from A to B to C and then back to A, with partially updated chats and diverging views from recipients/contacts on this transition. In the end, you will have a primary and some secondaries, and when you start sending out messages everybody will eventually synchronize when they receive the current state of primaries/secondaries.
- of course on incoming message for need to check for each stated secondary address in the replacement header that it uses the same signature as the signature we verified as valid with the incoming message **--> Also we have to somehow make sure that the signing key was not just gossiped from some random other person in some group.**
- there are no extra flags/columns in the database needed (i hope)
#### Downsides of the chosen approach:
- Inconsistent group state: Suppose Alice does an AEAP transition and sends a 1:1 message to Bob, so Bob rewrites Alice's contact. Alice, Bob and Charlie are together in a group. Before Alice writes to this group, Bob and Charlie will have different membership lists, and Bob will send messages to Alice's new address, while Charlie will send them to her old address.
- There will be multiple contacts with the same address in the database. We will have to do something against this at some point.
The most obvious alternative would be to create a new contact with the new address and replace the old contact in the groups.
#### Upsides:
- With this approach, it's easier to switch to a model where the info about the transition is encoded in the PGP key. Since the key is gossiped, the information about the transition will spread virally.
- (Also, less important: Slightly faster transation: If you send a message to e.g. "Delta Chat Dev", all members of the "sub-group" "delta android" will know of your transition.)
- It's easier to implement (if too many problems turn up, we can still switch to another approach and didn't wast that much development time.)
- The user is responsible that messages to the old address arrive at the new address, for example by configuring the old provider to forward all emails to the new one.
Notes during implementing
========================
- As far as I understand the code, unencrypted messages are unsigned. So, the transition only works if both sides have the other side's key.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
