Commit Graph

514 Commits

Author SHA1 Message Date
link2xt
45abaff275 Update provider database 2023-02-21 14:41:13 +00:00
bjoern
38a62d92ba mention speedup of #4065 in CHANGELOG (#4073)
the speedup of #4065 is larger than measured first

running `cargo test`, first we thought there is only a slight speedup
from 13.5 seconds to 12.5 seconds on a m1pro.

however, there is one test that does 11 seconds of sleep() (test_modify_chat_disordered)
as this test is not run very first, this slows down things overall.

skipping this test,
speedup is from 13.5 seconds to 9.5 seconds -
28% faster - and this is something we should mention in the changelog :)

(this pr does not remove or change the slow test.
i think, due to the number of cores, this is not needed until someone
has a machine where the other tests run in 2 seconds or so)
2023-02-21 13:36:36 +01:00
Hocuri
17acbca576 Correctly clear database cache after import (#4067) 2023-02-21 11:23:34 +00:00
link2xt
92c7cc40d4 sql: replace .get_conn() interface with .call()
.call() interface is safer because it ensures
that blocking operations on SQL connection
are called within tokio::task::block_in_place().

Previously some code called blocking operations
in async context, e.g. add_parts() in receive_imf module.

The underlying implementation of .call()
can later be replaced with an implementation
that does not require block_in_place(),
e.g. a worker pool,
without changing the code using the .call() interface.
2023-02-20 18:54:29 +00:00
link2xt
56d10f7c42 Use transaction in update_blocked_mailinglist_contacts 2023-02-20 17:17:59 +00:00
iequidoo
604c4fcb71 Delete messages to the Trash folder for Gmail by default (#3957)
Gmail archives messages marked as `\Deleted` by default if those messages aren't in the Trash. But
if move them to the Trash instead, they will be auto-deleted in 30 days.
2023-02-20 14:09:27 -03:00
link2xt
4790ad0478 Merge switching SQLite connection pool from FIFO to LIFO 2023-02-20 16:43:19 +00:00
link2xt
eaa2ef5a44 sql: start transactions as IMMEDIATE
With the introduction of transactions in Contact::add_or_lookup(),
python tests sometimes fail to create contacts with the folowing error:

Cannot create contact: add_or_lookup: database is locked: Error code 5: The database file is locked

`PRAGMA busy_timeout=60000` does not affect
this case as the error is returned before 60 seconds pass.

DEFERRED transactions with write operations need
to be retried from scratch
if they are rolled back
due to a write operation on another connection.

Using IMMEDIATE transactions for writing
is an attempt to fix this problem
without a retry loop.

If we later need DEFERRED transactions,
e.g. for reading a snapshot without locking the database,
we may introduce another function for this.
2023-02-20 15:05:35 +00:00
link2xt
2eeacb0f8a sql: organize connection pool as a stack rather than a queue
When connection pool is organized as a stack,
it always returns most recently used connection.
Because each connection has its own page cache,
using the connection with fresh cache improves performance.

I commented out `oauth2::tests::test_oauth_from_mx`
because it requires network connection,
turned off the Wi-Fi and ran the tests.

Before the change, with a queue:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
  Time (mean ± σ):     56.424 s ±  4.515 s    [User: 183.181 s, System: 128.156 s]
  Range (min … max):   52.123 s … 68.193 s    10 runs
```

With a stack:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
  Time (mean ± σ):     29.887 s ±  1.377 s    [User: 101.226 s, System: 45.573 s]
  Range (min … max):   26.591 s … 31.010 s    10 runs
```

On version 1.107.1:
```
$ hyperfine "cargo test"
Benchmark 1: cargo test
  Time (mean ± σ):     43.658 s ±  1.079 s    [User: 202.582 s, System: 50.723 s]
  Range (min … max):   41.531 s … 45.170 s    10 runs
```
2023-02-20 15:00:37 +00:00
link2xt
446214fd7b sql: use transaction in Contact::add_or_lookup() 2023-02-19 23:16:44 +00:00
link2xt
5bdd49b451 Add Unreleased section to changelog 2023-02-19 21:57:36 +00:00
link2xt
44953d6bcc Release 1.109.0 2023-02-19 21:40:33 +00:00
link2xt
10066b2bc7 sql: use semaphore to limit access to the connection pool
This ensures that if multiple connections are returned
to the pool at the same time, waiters get them in the order
they were placed in the queue.
2023-02-19 17:06:25 +00:00
Hocuri
641d102aba Add dc_msg_set_subject() C FFI (#4057) 2023-02-19 13:42:39 +00:00
link2xt
ed8e2c4818 Replace r2d2 with an own connection pool
New connection pool does not use threads
and does not remove idle connections
or create new connections at runtime.
2023-02-19 02:26:26 +00:00
link2xt
870527de1e Remove r2d2_sqlite dependency 2023-02-17 11:06:17 +00:00
link2xt
7d2cca8633 sql: enable auto_vacuum on all connections 2023-02-15 18:59:47 +00:00
link2xt
893794f4e7 accounts: retry filesystem operations in migrate_account() 2023-02-15 18:57:37 +00:00
link2xt
032da4fb1a python: add py.typed file
It marks the package as supporting typing according to PEP 561 <https://peps.python.org/pep-0561/>

Also remove zip-safe option from setuptools configuration for deltachat-rpc-client.
It is deprecated in setuptools and not used for wheel package format according to <https://setuptools.pypa.io/en/latest/deprecated/zip_safe.html>
2023-02-15 18:56:34 +00:00
Simon Laux
0de5125de8 jsonrpc: get_messages now returns a map with MessageLoadResult instead of failing completely if one of the requested messages could not be loaded. (#4038)
* jsonrpc: `get_messages` now returns a map with `MessageLoadResult`
instead of failing completely if one of the requested messages could not be loaded.

* add pr number to changelog

* format errors with causes instead of debug output

also for chatlistitemfetchresult
2023-02-15 18:46:34 +00:00
link2xt
243f28f8a5 python: use dataclasses to reduce boilerplate 2023-02-15 12:35:52 +00:00
link2xt
78577594d0 deltachat-rpc-server: spawn request handlers 2023-02-15 12:19:03 +00:00
link2xt
00c14dd9f6 Move changelog entry to unreleased section 2023-02-14 10:58:43 +00:00
link2xt
71d9716117 Remove MimeMessage::from_bytes()
It was not used anywhere except the tests.
2023-02-14 10:57:57 +00:00
link2xt
2e4f63a290 Add Unreleased changelog section 2023-02-14 01:54:22 +00:00
link2xt
9dee725895 Fix header levels for 1.108.0 changelog 2023-02-14 01:53:57 +00:00
link2xt
267263dab7 Release 1.108.0 2023-02-13 21:15:34 +00:00
iequidoo
19d7632be0 Show non-deltachat emails by default for new installations 2023-02-13 14:48:53 -03:00
link2xt
6c8368fa23 jsonrpc: add API to check if the message is sent by a bot
Co-Authored-By: Asiel Díaz Benítez <asieldbenitez@gmail.com>
2023-02-12 18:53:26 +00:00
link2xt
07c510c178 Remove bitflags from get_chat_msgs() interface
get_chat_msgs() function is split into new get_chat_msgs() without flags
and get_chat_msgs_ex() which accepts booleans instead of bitflags.

FFI call dc_get_chat_msgs() is still using bitflags for compatibility.

JSON-RPC calls get_message_ids() and get_message_list_items()
accept booleans instead of bitflags now.
2023-02-12 15:11:04 +00:00
link2xt
fa198c3b5e Use SOCKS5 configuration for HTTP requests 2023-02-10 23:20:11 +00:00
link2xt
151b34ea79 python: handle NULL value returned from dc_get_msg
Returning None in this case and checked with mypy that callers can handle it.
2023-02-10 23:17:42 +00:00
link2xt
c9b2ad4ffa Prefer TLS over STARTTLS during autoconfiguration 2023-02-10 11:28:11 +00:00
link2xt
cd293e6f49 Update async-smtp to 0.8 2023-02-03 11:36:58 +00:00
link2xt
59f5cbe6f1 Merge from stable branch 2023-02-01 15:36:26 +00:00
link2xt
52a4b0c2b8 Revert to async-smtp 0.5 to disable SMTP pipelining 2023-02-01 14:43:22 +00:00
link2xt
74abb82de2 Log server security (TLS/STARTTLS/plain) type 2023-02-01 00:01:47 +00:00
iequidoo
39601be452 observe_securejoin_on_other_device(): Add handling of "v*-request-with-auth" messages (#3836)
This fixes the case with multiple devices on the joining side: if we observe such a message from
another device, we mark an inviter as verified and an accepted contact thus causing a subsequent
"vg-member-added" message -- in case of a verified group -- to create it properly.
2023-01-25 20:28:49 -03:00
Sebastian Klähn
ba860a2b61 Debug logging v2 (#3958)
debug logging
2023-01-25 13:22:15 +00:00
link2xt
37f20c6889 Prepare 1.107.0 2023-01-23 16:20:07 +00:00
link2xt
c4c4c977a6 Changelog 2023-01-19 16:50:50 +00:00
link2xt
677136f4ab Use SMTP pipelining 2023-01-19 01:30:32 +00:00
link2xt
de268b8225 Terminate recently seen loop if cannot receive interrupts
It seems .abort() does not work on the recently seen loop
in some cases, e.g. if it is busy looping in a separate thread.

In my case after account reconfiguration recently seen loop
kept running and issuing warnings about closed interrupt channel.

Exit from recently seen loop on errors to avoid using 100% CPU
in such cases.
2023-01-18 10:48:54 +00:00
link2xt
42c709e7b1 Fix SOCKS5 usage for IMAP
Connect to SOCKS5 server rather than target server
and send TCP connect command.
2023-01-18 10:13:17 +00:00
link2xt
cf0349acc8 configure: log SOCKS5 configuration for IMAP like we do for SMTP 2023-01-18 09:32:29 +00:00
link2xt
e43b36b61f Peerstate.verifier fixes
Derive Debug, PartialEq and Eq for Peerstate,
so `verifier` is included in Debug output and compared.

Store verifier as empty string
instead of NULL in the database.
2023-01-17 14:22:22 +00:00
Simon Laux
ed24867309 fix verifier-by addr was empty string instead of None (#3961)
fix verifier-by addr was empty string intead of None
2023-01-17 13:06:57 +00:00
iequidoo
3cf78749df Emit DC_EVENT_MSGS_CHANGED for DC_CHAT_ID_ARCHIVED_LINK when the number of archived chats with
unread messages increases (#3940)
2023-01-16 16:05:47 -03:00
link2xt
1c44135b41 Remove deprecated attach_selfavatar config
According to the comment it was added in Dec 2019
with an intention to remove it "after some time".
2023-01-12 20:52:47 +00:00
iequidoo
d644988845 Securejoin: Fix adding and handling Autocrypt-Gossip headers (#3836)
- If bcc_self is set, gossip headers must be added despite of the number of group members.
- If another device observes Secure-Join, instead of looking for Secure-Join-Fingerprint in
  "vg-member-added"/"vc-contact-confirm" messages it must use keys from Autocrypt-Gossip headers as
  described in the Countermitm doc
  (https://countermitm.readthedocs.io/en/latest/new.html#joining-a-verified-group-secure-join).
2023-01-12 15:13:30 -03:00