Commit Graph
100 Commits
Author SHA1 Message Date
link2xt a1f04d2129 imex: use param2 for passphrase 2022-01-16 13:22:08 +00:00
link2xt fb95573000 cargo update 2022-01-16 09:56:20 +00:00
link2xt 3c38fa6b70 Add API for passphrase-protected accounts
To create encrypted account with account manager, call
dc_accounts_add_closed_account(). Open this account with
dc_context_open() using the passphrase you want to use for encryption.

When application is loaded next time and account manager is created,
it will open all accounts that have no passphrase set. For encrypted
accounts dc_context_is_open() will return 0. To open them, call
dc_context_open() with the correct passphrase. After opening, call
dc_context_start_io() on this account or just dc_accounts_start_io()
to start all accounts that are not started yet.

Support for legacy SQLite-based backup format is removed in this
commit.
2022-01-06 08:54:58 +00:00
link2xt bfa641cea8 Error handling refactoring
- Replace .ok_or_else() and .map_err() with anyhow::Context where possible.
- Use .context() to check Option for None when it's an error
- Resultify Chatlist.get_chat_id()
- Add useful .context() to some errors
- IMAP error handling cleanup
2022-01-07 14:22:37 +00:00
link2xt 29c58efeb3 Remove default signature advertsing Delta Chat 2022-01-07 13:56:58 +00:00
link2xt 27eb82c556 context: hide ongoing process internals from public API
Also hide derive_blobdir and derive_walfile.
2022-01-07 06:25:03 +00:00
link2xt 652d67a20f Revert flaky "sql: enable auto_vacuum on all connections"
It results in "database is locked" errors on CI.

This reverts commit ce0984f02f.
2022-01-07 00:18:21 +03:00
link2xt ce0984f02f sql: enable auto_vacuum on all connections 2022-01-06 09:03:50 +00:00
link2xt bba3a25371 Add CONDSTORE to standards.md 2022-01-03 23:12:29 +00:00
link2xt 095b358aca Test multidevice synchronization of Seen status 2022-01-03 23:12:29 +00:00
link2xt 833e5f46cc Synchronize seen status across devices
Seen status is only synchronized on servers supporting IMAP CONDSTORE
extension. At the end of fetch loop iteration, flags are fetched for
all messages modified since previous synchronization and highest
modification sequence is stored into `imap_sync` table.
2022-01-03 23:12:29 +00:00
link2xt 3e0ce0e07a test_no_old_msg_is_fresh: compare DC_EVENT_MSGS_NOTICED argument to chat id
data1 of DC_EVENT_MSGS_NOTICED contains chat id, not message id
2022-01-03 23:12:29 +00:00
link2xt 1f31dd12fc Replace BTreeMap with BTreeSet in markseen_msgs 2022-01-03 23:12:29 +00:00
link2xt f63efc29bf Resultify update_msg_state 2022-01-03 23:12:29 +00:00
link2xt 3e394f14e8 sql: disable cipher_memory_security SQLCipher PRAGMA
It slows down dc_get_chat_msgs() from ~50ms to seconds on Android. It
is disabled by default in SQLCipher 4.5.0, but currently SQLCipher 4.4.3
is bundled so this PRAGMA has to be disabled manually.
2022-01-03 23:03:59 +00:00
link2xt 304c259a57 Update resync_folder_uids to use imap table 2022-01-02 17:04:16 +00:00
link2xt 630754b52e dc_receive_imf: don't fail on invalid address in the To field
This is an irrecoverable error, dc_receive_imf must not fail on it
as it prevents last seen UID from advancing, so the same message
is prefetched on each iteration of IMAP loop.
2022-01-02 00:20:44 +00:00
link2xt afd8c0d879 Add smtp table
It replaces SendMsgToSmtp job.

Prepared outgoing SMTP payloads are stored in the database now rather
than files in blobdir.
2022-01-01 19:14:53 +00:00
link2xt f3a716fac6 sql: enable auto_vacuum=INCREMENTAL
Previously default of `auto_vacuum=NONE` was used.

`PRAGMA auto_vacuum=INCERMENTAL` allows running `PRAGMA
incremental_vacuum`. Unlike `VACUUM`, `PRAGMA incremental_vacuum`
frees unused pages without rebuilding the whole database, like the
`VACUUM` does, so it does not require additional disk space to build a
vacuumed copy of the database. This may even free enough space on the
disk to run `VACUUM` afterwards.

New setting will only be enabled for new databases or if the `VACUUM`
command runs successfully. Currently `VACUUM` is only executed on
backup export, but may fail nevertheless if there is not enough space
on the disk.

Also try to run `PRAGMA incremental_vacuum` during housekeeping. It
may not be the best strategy, but likely does not make any difference
under normal usage when the database only grows and there are no free
pages. Free pages are created only if enough data is deleted to free
at least one database page of 4096 bytes, for example when automatic
deletion of messages is deleted for the first time. In the future if
more data is placed into the database, like avatars and other blobs,
it may be necessary to revise this strategy, for example to keep some
free pages instead of removing all of them each time by querying
`PRAGMA freelist_pages` and running `PRAGMA incremental_vacuum(N)`.
2021-12-31 21:33:38 +00:00
link2xt 72659580de sql: build rusqlite with sqlcipher 2022-01-01 00:00:00 +00:00
link2xt 30cb0cbcfd Reduce number of AsRef generics
They result in compilation of duplicate code.
2021-12-31 13:57:45 +00:00
link2xt 246cae5d9e Remove inbox_watch option
Also resultified `get_watched_folders`.

Python test `test_moved_markseen` is modified to test using incoming
message instead of BCC-self message, because BCC-self message is
detected immediately in the Inbox.
2021-12-25 18:07:14 +00:00
link2xt 12313543ca Fix leaving the group 2021-12-30 12:51:10 +00:00
link2xt 87e3dead14 Remove unused InterruptInfo.msg_id 2021-12-30 02:15:30 +00:00
link2xt 12823c2213 Add imap table to keep track of message UIDs
`imap` table maps Message-IDs to UIDs on the server. `dc_receive_imf`
no longer gets the UID of the message as an argument and does not
insert the folder and UID of the message into the `msgs`
table. `server_folder` and `server_uid` columns in `msgs` table are
deprecated.

MoveMsg and DeleteMsgOnImap jobs are removed. Now messages are moved
and deleted only in the `fetch_move_delete` procedure that consults
the `target` column of the `imap` table to determine where the message
should go.

Where the message should go is determined after prefetching by the
`imap::target_folder()` procedure.  Messages are only downloaded once
they reach their target folder to avoid race conditions in multidevice
setting, such as:

1. One device trying to FETCH the message while the other tries to
MOVE it.

2. One device marking the message as \Seen in the Inbox while the
other has already copied unseen message to the Movebox and is going to
delete the \Seen message in the Inbox.

3. Device downloads the message from the Inbox while there are newer
messages in the Movebox placed there by the other device, thus
processing the messages out of order.
2021-12-19 00:00:00 +00:00
link2xt 9e7bdc579e Test recovery from lost member removal message
This was fixed in d98d1857a4 but not
tested.

Revert d98d1857a4 to check that the test
is working.
2021-12-19 15:05:48 +00:00
link2xt 61af0c9ac4 python: remove API deprecated in Python 3.10 2021-12-18 17:47:08 +00:00
link2xt 91f02ad553 Rename asymetric test into asymmetric 2021-12-18 17:46:29 +00:00
link2xt 8d3a1e84c3 Remove mvbox_watch option and use mvbox_move instead 2021-12-18 13:55:01 +00:00
link2xt d009835210 Remove unnecessary getattr 2021-12-18 12:48:40 +00:00
link2xt 83a664ca68 Fix mypy error 2021-12-18 12:47:11 +00:00
link2xt d98d1857a4 Improve group updates
- Check that member modifying the group is in the group themselves.

  It is not allowed to readd yourself to the group or remove someone without being in the group themselves.

- Unify code for group member addition and removal.

  Removing a member now recreates the group member list from the To: field.

  Removed member from the Chat-Group-Member-Removed was in the To: field in previous Delta Chat versions,
  so it is excluded explicity. New versions of Delta Chat put removed member in Bcc: instead.

- Apply avatar changes after updating the group member list.

  This allows to check that the contact modifying the avatar is actually a group member.
2021-12-18 11:58:32 +00:00
link2xt dae80cbe35 Recognize MS Exchange read receipts as read receipts
They contain X-MSExch-Correlation-Key header, but no
Original-Message-ID, so they cannot be used to find the original
message, but we want to recognize them as MDN nevertheless to assign
them to the trash chat.
2021-12-11 21:44:54 +00:00
link2xt f8b4ef26b3 Update rustyline
The version was locked to 9.0.0 previsouly because Rust 1.56 requirement.
New version 9.1.1 should support 1.56 again.
2021-12-11 18:10:18 +00:00
link2xt db991453b0 Merge branch 'cargo-update' 2021-12-11 14:48:17 +00:00
link2xt c11ce4c8d4 Merge detached signature parsing 2021-12-11 14:46:58 +00:00
link2xt 92e300cb9f cargo update 2021-12-11 12:06:53 +00:00
link2xt 4d4968f358 Make the code compile with new tagger 2021-12-11 11:36:18 +00:00
link2xt 6d125028f5 Validate detached signatures 2021-12-10 22:56:42 +00:00
link2xt 7ff3cf4af0 Accept keyring by reference in pk_decrypt 2021-12-10 22:56:42 +00:00
link2xt bb3353397d Use return values instead of out arguments for PGP signatures 2021-12-10 22:56:42 +00:00
link2xt 572260ec29 Test RFC 1847 encapsulation
https://datatracker.ietf.org/doc/html/rfc3156#section-6.1
2021-12-10 22:56:42 +00:00
link2xt 1d0f6aad95 Add dc_msg_force_plaintext() API for bots (#2847)
This will allow implementing a special command for download bot to request that it sends back the downloaded file unencrypted.
2021-12-09 11:51:21 +01:00
link2xt 01db8d0130 test_utils: fix Alice's address in alice_keypair()
The key stored in test-data has @example.org identity.
2021-12-05 09:14:58 +00:00
link2xt 8ad9db5572 mimefactory: refactor generation of the To: field
Group all the code responsible for the generation of the To: field
2021-12-04 22:04:47 +00:00
link2xt 607cd23014 Test that removed member is not included in the To: field 2021-12-04 22:04:47 +00:00
link2xt 220758d244 Place removed members into Bcc: instead of To:
This way if someone replies to the member deletion messsage from a
classic MUA, they will not reply to the removed member.
2021-12-04 18:51:25 +00:00
link2xt 7ab71bb468 Convert test keys from base64 to ASCII armor
ASCII armored keys can be easily generated with `sq key generate` and
used to encrypt and decrypt test messages with `sq` and `gpg` without
converting them to binary using `base64 -d` first.
2021-12-04 19:56:03 +03:00
link2xt a74377b620 Fix Rust 1.57 compiler and clippy warnings (#2863)
* Remove unused os_name

* Disable clippy::manual_split_once lint

It's suggestions in stable 1.57 Rust are incorrect: https://github.com/rust-lang/rust-clippy/issues/7889

* Allow unused fields in OAuth2 response

* Fix clippy warning about `select_pk_for_encryption`
2021-12-04 18:56:14 +03:00
link2xt 6ce97bd0cd Merge fixes for chat assignment when forwarding messages
GitHub PR #2843
2021-11-27 00:00:00 +00:00
link2xt c29149e74c Add group forwarding test 2021-11-27 00:00:00 +00:00
link2xt 487f7593ce Reset In-Reply-To when forwarding a message 2021-11-27 00:00:00 +00:00
link2xt 6b3b33d2a0 Test forwarded quoted messages 2021-11-27 00:00:00 +00:00
link2xt 2d70ccc2bf Do not return a quoted message for forwarded messages
For forwarded messages, parent message is not a quoted message.
2021-11-27 00:00:00 +00:00
link2xt e90fc9504a Test get_parent_message 2021-11-27 00:00:00 +00:00
link2xt 5108314c03 Do not return trashed messages from get_rfc724_mid_in_list
This function is used to lookup the chat by `References` and
`In-Reply-To` header, so it does not make sense to return trashed
message when there is another non-trashed message in one of these
headers with a real chat ID.
2021-11-27 00:00:00 +00:00
link2xt 016fb2ceb2 Remove indexmap dependency
`indexmap` is a large dependency (4K SLoC) containing `unsafe` code.

Contact IDs are now passed around as a Vec<u32> or &[u32].

QUOTA roots are now sorted by name instead of perserving original order.
2021-11-22 16:40:06 +01:00
link2xt 5c571520a0 contact: use last_seen column
It was there since the C core, labeled with "/* last_seen is for
future use */" but never actually used. The comment was lost during
the translation from C to Rust.
2021-11-21 21:14:17 +03:00
link2xt ddefd2cf09 python: add cutil.from_optional_dc_charpointer()
`cutil.from_dc_charpointer()` is guaranteed to return `str`, while
`cutil.from_optional_dc_charpointer()` may return `None` if C function
returns `NULL`.
2021-11-21 20:00:29 +03:00
link2xt e885857875 Remove pretty_assertions dependency
It was only used in two places. Rather than adding `use
pretty_assertions::*` everywhere, it's easier to remove, and it
removes two additional dependency crates.
2021-11-20 11:42:52 +01:00
link2xt a1d57a2645 Disable unnecessary proptest features 2021-11-20 11:42:52 +01:00
link2xt a28aecd4d1 Update sqlite dependencies 2021-11-20 11:42:52 +01:00
link2xt a3f1ff2827 Disable xattr feature on async-tar
This removes `xattr` dependency

We only care about backed up file contents, not attributes.
2021-11-20 11:42:52 +01:00
link2xt c4d1a639b0 Remove itertools dependency
Collecting into Vec of &str and joining may even be faster according
to benchmarks:
https://gist.github.com/green-s/fbd0d374b290781ac9b3f8ff03e3245d
2021-11-20 11:42:52 +01:00
link2xt 60d41022ea scripts: switch from python 3.6 to python 3.7
PEP 562 (__getattr__ in deltachat.const module) is implemented only since python 3.7
2021-11-14 00:00:00 +00:00
link2xt 0d62069b67 python: add mypy support and some type hints
`deltachat.const` module now defines `__getattr__` and `__dir__` as
suggested by https://www.python.org/dev/peps/pep-0562/
mypy detects that `__getattr__` is defined and does not show errors
for `DC_*` constants which cannot be detected statically.
mypy is added to `tox.ini`, so type check can be run with `tox -e mypy`.
2021-11-14 11:06:44 +03:00
link2xt 56cf2e6596 Replace error! on verification failure with warn!
A message is added into 1:1 chat anyway, and user does not know what `StockMessage::ContactNotVerified` means.
2021-11-14 02:02:23 +03:00
link2xt 13bc8b78d7 python: enable isolated build in tox.ini
This makes tox install build system configured in pyproject.toml
according to PEP 518 rather than assuming setuptools.
2021-11-13 03:45:58 +00:00
link2xt c7c68094d9 setup.py: restore compatibility with setup.py sdist command
Otherwise source package with version 0.0.0 is created.
2021-11-12 23:41:17 +00:00
link2xt 1379f8a055 Factor apply_group_changes out of create_or_lookup_group
`apply_group_changes` is executed regardless of whether the group is
created via `create_or_lookup_group` or found via
`lookup_chat_by_reply`. This change removes the need for
`lookup_chat_by_reply` to return `None` when group ID exists in the
database to let `create_or_lookup_group` run. As a side effect of this
Delta Chat replies to ad hoc groups are now correctly assigned to
chats when there are multiple groups with the same group ID, such as
ad hoc groups with the same member lists.
2021-11-07 18:34:44 +03:00
link2xt 5e26b5bfdc Merge pull request #2743 from deltachat/improve-gossip
Optimize Autocrypt gossip
2021-11-01 02:01:51 +03:00
link2xt 60fbb6df5a Add python test for gossip optimization 2021-10-31 22:39:05 +00:00
link2xt 3e60ee9d3e python: use datetime.fromtimestamp() instead of datetime.utcfromtimestamp()
utcfromtimestamp() is not recommended by the official documentation,
because many methods, including timestamp(), work incorrectly with
"naive" datetimes returned by utcfromtimestamp().
2021-10-31 22:39:05 +00:00
link2xt 54e79409e6 Optimize Autocrypt gossip
Update gossiped_timestamp when someone else sends autocrypt gossip in
the group, so we postpone sending gossip again ourselves.

- Warn about failures to parse Autocrypt-Gossip header
- Move gossip-related methods into ChatId impl
- Fix a "gossi_pp_ed" typo
2021-11-01 01:17:51 +03:00
link2xt 3a014477e7 Fix dc_truncate proptest (#2781)
It was failing for approx_chars = 0.

Also reduce approx_chars range so approx_chars = 0 is tested more frequently.
2021-10-31 11:44:15 +01:00
link2xt 90d8c8baf5 Only apply ephemeral timers to non-special chats 2021-10-31 02:19:27 +03:00
link2xt 1dee17f980 Resultify dc_receive_imf::save_locations() 2021-10-31 02:19:27 +03:00
link2xt 6aeb21d3af dc_receive_imf: do not ignore lookup_by_contact errors 2021-10-31 02:19:27 +03:00
link2xt 4747ae2f1c Make is_dc_message and allow_creation immutable 2021-10-31 02:19:27 +03:00
link2xt 7968f55191 Trash messages instead of hiding them 2021-10-31 02:19:27 +03:00
link2xt 33aa3556d2 Use system resolver configuration instead of Google DNS for MX queries 2021-10-31 01:56:27 +03:00
link2xt ebb89e20b4 fixup! Modernize python setup 2021-10-28 22:03:52 +00:00
link2xt 8ef6b6089f receive_imf: simplify message state calculation
- Make `state` variable immutable.
- Don't mark contact requests as fresh when fetching existing messages: this is not needed since there is no contact request chat anymore.
- Don't mark messages from blocked chats as noticed, this is not needed since messages go to blocked chat anyway instead of contact request chat.
2021-10-28 01:59:54 +03:00
link2xt 5b5b26122e Modernize python setup
Use pyproject.toml instead of deprecated setup.py arguments and unpin dependencies in tox.ini.
2021-10-28 00:34:33 +03:00
link2xt ee3b40a59a Remove double reference from lookup_chat_by_reply argument 2021-10-23 12:34:36 +00:00
link2xt e511b87955 Remove unnecessary mut qualifier 2021-10-23 12:21:24 +00:00
link2xt 6061d71492 cargo: update strum to 0.22 2021-10-12 21:12:46 +00:00
link2xt dbd8814d2c Refactor qr module 2021-10-10 15:11:01 +03:00
link2xtandFloris Bruynooghe a3562c5940 Remove get_summarytext_by_raw
Use `Summary.truncated_text()` instead.

Co-Authored-By: Floris Bruynooghe <flub@devork.be>
2021-10-09 22:13:42 +03:00
link2xt 6e3c2fc839 dc_receive_imf: simplify timestamp calculation
Reduce the number of mutable variables and out parameters in
`add_parts`.

Also don't call `dc_create_smeared_timestamp` if there is no `Date`
header. Timestamps are only supposed to be "created" when the message
is sent, not received, to make sure sent messages are sorted properly
in MUAs that only use the date for sorting. Delta Chat uses database
IDs for sorting in addition to timestamps, so it can sort messages
with equal timestamps properly.

Update `dc_smeared_time` documentation.

Turn `dc_smeared_time` and `dc_create_smeared_timestamp` comments into
documentation comments.
2021-10-09 13:44:59 +03:00
link2xt 30e616f74f Increase MSRV to 1.51.0 and cargo update 2021-10-09 12:08:07 +03:00
link2xt 6c59b0de85 Remove unused Imap.interrupt
It is always set to `None`.
2021-10-03 01:14:50 +00:00
link2xt acbf363fc8 Try to lock strict TLS if certificate checks are automatic 2021-09-26 19:03:04 +03:00
link2xt 2309c7ca13 Emit events from account manager
Errors and warnings are emitted with a special 0 account ID.
2021-09-25 18:25:52 +03:00
link2xt 89d8b26192 Downgrade zeroize_derive to 1.1.0
Version 1.2.0 is not supported by Rust 1.48.0
2021-09-25 12:21:27 +00:00
link2xt ee32a7b00a Remove minor versions from deltachat_derive and deltachat-ffi
Also run `cargo update`
2021-09-25 12:15:59 +00:00
link2xt 1dbbf6b3be Create configured folders if they are deleted 2021-09-25 02:51:32 +03:00
link2xt f8a4a88fb2 Accept &str instead of AsRef<str> in fetch_new_mesages() 2021-09-25 02:51:32 +03:00
link2xt 3096193d58 ephemeral: always apply timers from system messages
Also extend the tests to catch regressions.
2021-09-25 02:51:23 +03:00
link2xt 085a899de2 Fix ephemeral timer rollback protection
Implement get_previous_message() that only looks for the last
Message-ID in the References: header instead of using
get_parent_message() which falls back to using other Message-IDs from
References and In-Reply-To field.
2021-09-20 01:54:51 +03:00