This change is aimed at decoupling parsing and
add_parts() stages to eventually separate parsing
from database changes and pipeline message parsing and
decryption.
This seems not only wasteful but genuinly has the risk someone makes
their device useless by accidentally adding a huge file.
This also re-structures the checks a little: The if-conditions are
flattened out and cheap checks are done before more expensive ones.
Replaced mutable out parameters with explicit return of structure.
Also moved all decisions about emitted events out of add_parts(). Chat
ID is removed from created_db_entries as it is the same for all parts.
* add simple backup export/import test
this test fails on current master
until the context is recrated.
* avoid config_cache races
adds needed SQL-statements to config_cache locking.
otherwise, another thread may alter the database
eg. between SELECT and the config_cache update -
resulting in the wrong value being written to config_cache.
* also update config_cache on initializing tables
VERSION_CFG is also set later, however,
not doing it here will result in bugs when we change DBVERSION at some point.
as this alters only VERSION_CFG and that is executed sequentially anyway,
race conditions between SQL and config_cache
seems not to be an issue in this case.
* clear config_cache after backup import
import replaces the whole database,
so config_cache needs to be invalidated as well.
we do that before import,
so in case a backup is imported only partly,
the cache does not add additional problems.
* update CHANGELOG
due to an bug from Apple copying files from/to iPhones
(cmp. https://support.delta.chat/t/import-backup-to-ios/1628/7 )
it may easily happen that one gets corrupted/partly backups.
such imports usually fail with some error,
however, for debugging it is nice to have the concrete file size in the log.
Currently if user moves the message into some other folder and then
moves the message back, the message is considered duplicate even
though previous copy was already deleted. This is a common problem
reported by users at least twice.
Keeping duplicates does no harm except for additional storage usage.
If the message is later deleted by the user, all the copies on the
server will be deleted. anyway.
This brings the Display of ContactId in line with those of ChatId etc,
which is a bit clearer is logs and such places.
It also updates an SQL query to rely on the ToSql impl of ContactId
rather than it's Display when building the query.
Holger had a case where he wrote with someone using a classing MUA.
He opened a two-member-group with this person (which also allowed him to
set the subject).
At some point the other person replied from a different email address.
What he expected: This reply should be sorted into the two-member-group.
What happened: This reply was sorted into the 1:1 chat.
---
I had added the line && chat_contacts.contains(&from_id) months ago when I wrote
this code because it seemed vaguely sensible but without any real
reason. So, let's remove it and see if it creates other problems -
my gut feeling is no.
This makes the contact ID its own newtype instead of being a plain
u32. The change purposefully does not yet try and reap any benefits
from this yet, instead aiming for a boring change that's easy to
review. Only exception is the ToSql/FromSql as not doing that yet
would also have created churn in the database code and it is easier to
go straight for the right solution here.
Message-IDs are now retrieved only during fetching and saved into imap
table. dc_receive_imf_inner does not attempt to extract the Message-ID
anymore.
For messages without Message-ID the ID is now generated in
imap::fetch_new_messages rather than dc_receive_imf_inner,
so the same ID is used in the imap table (maintained by the imap
module) and msgs table (maintained by dc_receive_imf module).
Message-ID generation based on the Date, From and To field hashing has
been replaced with a simple dc_create_id() to avoid retrieving Date,
From, and To fields in the imap module, as it's hard to test that it
stays compatible between Delta Chat versions in this module. This
breaks jump-to-quote for quoted messages without Message-ID, which is
not critical.
Also prefetch X-Microsoft-Original-Message-ID, so retrieval of
duplicate messages with X-Microsoft-Original-Message-ID can be skipped
like it is done for messages with Message-ID header.
* (r10s, adb, hpk) remove getAllUpdates() and add a typical replica-API that works with increasing serials. Streamline docs a bit.
* adapt ffi to new api
* documentation: updates serials may have gaps
* get_webxdc_status_updates() return updates larger than a given serial
* remove status_update_id from status-update-event; it is not needed (ui should update from the last known serial) and easily gets confused with last_serial
* unify wording to 'StatusUpdateSerial'
* remove legacy payload format, all known webxdc should be adapted
* add serial and max_serial to status updates
* avoid races when getting max_serial by avoiding two SQL calls
* update changelog
Co-authored-by: B. Petersen <r10s@b44t.com>
The state bob needs to maintain during a secure-join process when
exchanging messages used to be stored on the context. This means if
the process was killed this state was lost and the securejoin process
would fail. Moving this state into the database should help this.
This still only allows a single securejoin process at a time, this may
be relaxed in the future. For now any previous securejoin process
that was running is killed if a new one is started (this was already
the case).
This can remove some of the complexity around BobState handling: since
the state is in the database we can already make state interactions
transactional and correct. We no longer need the mutex around the
state handling. This means the BobStateHandle construct that was
handling the interactions between always having a valid state and
handling the mutex is no longer needed, resulting in some nice
simplifications.
Part of #2777.
* Add AcManager
See https://github.com/deltachat/deltachat-core-rust/pull/2901#issuecomment-998285039
This reduces boilerplate code again therefore, improving the
signal-noise-ratio and reducing the mental barrier to start
writing a unit test.
Slightly off-topic:
I didn't add any advanced functions like `manager.get("alice");` because
they're not needed yet; however, once we have the AcManager we can
think about fancy things like:
```rust
acm.send_text(&alice, "Hi Bob, this is Alice!", &bob);
```
which automatically lets bob receive the message.
However, this may be less useful than it seems at first, since most of
the tests I looked at wouldn't benefit from it, so at least I won't do
it until I have a test that would benefit from it.
* Remove unnecessary RefCell
* Rename AcManager to TestContextManager
* Don't store TestContext's in a vec for now as we don't need this; we can re-add it later
* Rename acm -> tcm
This moves most common headers like From, To, Subject etc. defined in
the Internet Message Format standard at the top of the message
in the same order as used in RFC 5322.
When `smtp_send` returns `Status::Finished`,
the message should be removed from the queue even in case of
failure, such as a permanent error.
In addition to this bugfix, move the retry count increase to
the beginning of `send_msg_to_smtp` to ensure no message is
retried infinitely even in case of similar bugs.
fix#3007
My approach is:
We don't download any messages from the spam folder anymore, and only download them if they were moved out. This means that is-it-spam logic only resides in spam_target_folder(). This has some implications, see the comments.
pytest-timeout already handles all deadlocks and is configurable with
--timeout option. With this change it is possible to disable timeout
with --timeout 0 to run tests on extremely slow connections.
Also change how NO response is treated. NO response means there is an
error moving/copying the messages. When there are no matching
messages, the response is "OK No matching messages, so nothing copied"
according to some RFC 9051 examples.
this avoids archived chats containing fresh messages:
before, it could happen that between the two SQL calls
a new fresh message arrives,
unarchives the chat that is immediately archived by the second SQL call -
resulting in an archive chat containing fresh messages.
as fresh messages counter are shown on app icon etc.
this is pretty weird for the user as they do not see what is "fresh".
the other way round,
there is no transaction in receive_imf(),
however, receive_imf() only unarchives chats,
so that is visible and no big issue for the user.
the issue is rare at all,
however, annoying if you get that as the badge counter may be stuck at "1"
nearly forever (until you open the archived chat in question).
- do not attempt to mark reserved meessages as seen when
messages with empty Message-ID are marked as seen on IMAP
- do not reconnect on Seen flag synchronization failures This avoid
reconnection loops in case of permanent errors in `sync_seen_flags`
* Make set_config() look a bit nicer
* Add OnlyFetchMvbox option
* Add test for the config
* Add option to only watch mvbox
This is supposed to support having a server-side rule which moves
emails to the mvbox already. The new option makes sure the mvbox is
wathched and also makes sure no messages are feched from folders other
than the mvbox and the spam folder if enabled. It does not interact
with the other settings.
* Fixup ignore conditions
* Cleanup some bits
* Watch the mvbox when `WatchMvboxOnly` is set
* Rename back to only_fetch_mvbox (flub said it's OK for him)
* typo
* clippy, more typos
Co-authored-by: Hocuri <hocuri@gmx.de>
Synchronizing seen flags doubles the time required to scan all
folders. Delta Chat only marks messages as Seen on Inbox or DeltaChat,
so there is no need to check for Seen flag on other folders.
This operation takes roughly 0.3 s on a moderate size database.
Calling it once before scanning all folders and scanning
the watched folder instead of each time after downloading
a message from a folder speeds up IMAP loop.
This operation takes roughly 0.3 s on a moderate size database.
Calling it once before scanning all folders and scanning
the watched folder instead of each time after downloading
a message from a folder speeds up IMAP loop.
this uses `get_webxdc_info().name` for chatlist etc.
the previuosly used static strings comes from a time
where we just did not had the correct name.
i was also thinking about adding `get_webxdc_info().summary`,
however, as this information is dynamic,
that may open several issues, eg. quoted text may change
so that the answer is out of context.
As of #2980, the migrations were not run after importing a backup. This lead to errors like "Failed to send message: no such table: smtp" if you imported a backup from a previous DC version.
as we had the changelog-chat only in the last days,
- add missing pr
- update some wrong references
- unify some layout and wordings,
streamline headlines to "API Changes", "Changes" and "Fixes",
finer subdivisions only raise noise, duplicates and discussions
and, if not used consequently, do not add much benefit.
as discussed in dev chat,
100 kb is too low if one wants to use any game framework.
also, many game that doesn't have poor graphics
or uses magic CSS-tricks to do fancy draws,
will have more than 100 kb probably just in tiles and sprites.
even for regions with low resources,
100 kb is low for a game -
esp. if we compare that with stickers that are send around as well
are not deduplicated and also have several 100 kb in size -
for much less effect.
we should still encourage ppl to create tiny apps,
however, a too low limit also restricts possibilities wrt adaption.
already before,
we did _not_ sent updates to contact requests or other read-only-chats.
however, we _did_ modify the local database,
so that getAllUpdates() returns an update that was not sent out to other peers.
this is fixed by checking can_send() soon.
that way, all peers have the same state
also for contact requests or other read-only-chats
and webxdc in contact requests can be opened as usual.
further ui improvements may be needed for contact requests
(maybe allow the webxdc to know about that state or
maybe show a warning in the ui somewhere),
however, this is not part of this pr.
* let sending invalid webxdc fail
invalid webxdc can still be send as Viewtype::File, however
(maybe one want to discuss errors or so ;)
* clarify the supported zip compression methods
* add a test for unencrypted replies to encrypted webxdc instances
* make update messages work if a key is missing
even in opportunistic chats,
replies to encrypted messages are forced to be encrypted,
if that is not possbile, message sending fails.
while this is okay to not leak previously send text messages,
the quotes as used by webxdc are more artificial,
currently only the static text "Webxdc".
* changelog ...
This was written in a way which attempted to avoid easily creating an
infinite loop. But really that's a python idiom and doesn't work very
well in Rust. Worse, as shown by #2972 it is really easy to still get
this wrong. Instead do this the rust way, this way the compiler can
also reason properly about the branches and what is unreachable
removing some bogus dead code.
* fix forwarding webxdc instances, add a test for that
* adapt webxdc test to fail on info-messages added when in draft mode
* do not add info-messages when in draft-mode
* half the number of instance-loads per webxdc update send/receive
While experimenting with encrypted storage, once configuring failed between 920 and 940. But as the "configured" config had already been written after progress 910, some part of the code thought we are configured, some didn't.
With this PR, my encrypted-storage Android PR now works, at least I
couldn't find any further bugs.
Without it, configuring fails with: `Failed to create blob
icon-saved-messages-1803424689.png in
/data/user/0/com.b44t.messenger.beta/files/accounts/0e402b37dcd14a9586aea46294c908f2/dc.db-blobs:
No such file or directory (os error 2)`.
Also see https://github.com/deltachat/deltachat-core-rust/pull/2972.
To create encrypted account with account manager, call
dc_accounts_add_closed_account(). Open this account with
dc_context_open() using the passphrase you want to use for encryption.
When application is loaded next time and account manager is created,
it will open all accounts that have no passphrase set. For encrypted
accounts dc_context_is_open() will return 0. To open them, call
dc_context_open() with the correct passphrase. After opening, call
dc_context_start_io() on this account or just dc_accounts_start_io()
to start all accounts that are not started yet.
Support for legacy SQLite-based backup format is removed in this
commit.
- Replace .ok_or_else() and .map_err() with anyhow::Context where possible.
- Use .context() to check Option for None when it's an error
- Resultify Chatlist.get_chat_id()
- Add useful .context() to some errors
- IMAP error handling cleanup
Seen status is only synchronized on servers supporting IMAP CONDSTORE
extension. At the end of fetch loop iteration, flags are fetched for
all messages modified since previous synchronization and highest
modification sequence is stored into `imap_sync` table.
It slows down dc_get_chat_msgs() from ~50ms to seconds on Android. It
is disabled by default in SQLCipher 4.5.0, but currently SQLCipher 4.4.3
is bundled so this PRAGMA has to be disabled manually.
This is an irrecoverable error, dc_receive_imf must not fail on it
as it prevents last seen UID from advancing, so the same message
is prefetched on each iteration of IMAP loop.
the summary can be modified by the apps using
`sendUpdate({summary: "foo", payload: ...})`
the summary is updated when there is no newer update
and chat will be informed by the change as usual by
`DC_EVENT_MSGS_CHANGED`.
additional parameters will be `summary` and `msg` (for an info-message)
right now, and maybe more in the future (`aggregated`, `fast`, `to` ...),
so adding just more parameters easily gets wild.
also, this makes adaptions easier as no ffi needs to be adapted
when we add more parameters.
finally, sending is in-sync with receiving, database and wire now,
one receives similar objects as one sends,
which also looks like the right thing :)
for now, `sendUpdate()` also allows to use the old, unwrapped payload
to not immediately break existing `.xdc`, however, that will be removed soon,
probably before the first release.
Previously default of `auto_vacuum=NONE` was used.
`PRAGMA auto_vacuum=INCERMENTAL` allows running `PRAGMA
incremental_vacuum`. Unlike `VACUUM`, `PRAGMA incremental_vacuum`
frees unused pages without rebuilding the whole database, like the
`VACUUM` does, so it does not require additional disk space to build a
vacuumed copy of the database. This may even free enough space on the
disk to run `VACUUM` afterwards.
New setting will only be enabled for new databases or if the `VACUUM`
command runs successfully. Currently `VACUUM` is only executed on
backup export, but may fail nevertheless if there is not enough space
on the disk.
Also try to run `PRAGMA incremental_vacuum` during housekeeping. It
may not be the best strategy, but likely does not make any difference
under normal usage when the database only grows and there are no free
pages. Free pages are created only if enough data is deleted to free
at least one database page of 4096 bytes, for example when automatic
deletion of messages is deleted for the first time. In the future if
more data is placed into the database, like avatars and other blobs,
it may be necessary to revise this strategy, for example to keep some
free pages instead of removing all of them each time by querying
`PRAGMA freelist_pages` and running `PRAGMA incremental_vacuum(N)`.
Also resultified `get_watched_folders`.
Python test `test_moved_markseen` is modified to test using incoming
message instead of BCC-self message, because BCC-self message is
detected immediately in the Inbox.
We had an unhelpful log in the Testing group:
- it repeatedly says "src/imap.rs:1010: dc_receive_imf error: add_parts error" but the actual error is not shown
- it says "deltachat-ffi/src/lib.rs:186: dc_get_config(): invalid key" but it doesn't say what key it's trying to set
I considered removing it from the context by default, but the
migration test really wants to have the tracker initialised from the
very first event and not after the context is initialised. It is
easier for now to leave it hardcoded instead of adding an API to
explicitly require enabling it via the builder.
This replaces the EventSink callbacks with simple channel senders.
This simplifies the TestContext a lot as that is much simpler to
handle. It then also removes the special-casing of the LogSink since
it now is another even sender, only injected at the very start.
There are too many ways to create a TestContext, this introduces a
TestContextBuilder to try and keep this shorter. It also cleans up
the existing constructors keeping only the commonly used ones.
Without this the test output is written somewhere random and ends up
in the wrong test report. This buffers all the log output and prints
it inside the test on dropping the LogSink, or on dropping the
TestContext if no explicit LogSink was created.
`imap` table maps Message-IDs to UIDs on the server. `dc_receive_imf`
no longer gets the UID of the message as an argument and does not
insert the folder and UID of the message into the `msgs`
table. `server_folder` and `server_uid` columns in `msgs` table are
deprecated.
MoveMsg and DeleteMsgOnImap jobs are removed. Now messages are moved
and deleted only in the `fetch_move_delete` procedure that consults
the `target` column of the `imap` table to determine where the message
should go.
Where the message should go is determined after prefetching by the
`imap::target_folder()` procedure. Messages are only downloaded once
they reach their target folder to avoid race conditions in multidevice
setting, such as:
1. One device trying to FETCH the message while the other tries to
MOVE it.
2. One device marking the message as \Seen in the Inbox while the
other has already copied unseen message to the Movebox and is going to
delete the \Seen message in the Inbox.
3. Device downloads the message from the Inbox while there are newer
messages in the Movebox placed there by the other device, thus
processing the messages out of order.
This runs all CI jobs with -Dwarnings, turning warnings into errors
for CI.
This is useful since we run rustfmt and clippy on CI on stable rust,
while the builds and tests run with a specific older rustc. The
latter is also used for local development usually since it is encoded
in the rust-toolchain file. This warnings in clippy jobs of stable
rust would often go unnoticed, however stable rust usually finds more
genuine issues than the older compiler we use.
- Check that member modifying the group is in the group themselves.
It is not allowed to readd yourself to the group or remove someone without being in the group themselves.
- Unify code for group member addition and removal.
Removing a member now recreates the group member list from the To: field.
Removed member from the Chat-Group-Member-Removed was in the To: field in previous Delta Chat versions,
so it is excluded explicity. New versions of Delta Chat put removed member in Bcc: instead.
- Apply avatar changes after updating the group member list.
This allows to check that the contact modifying the avatar is actually a group member.
* store msg-id in msg-object on set_draft(), add a test for that
* test deleting drafts
* keep draft-ids on updating drafts, set draft-state
* do not allow forwarding of drafts
in general, it should be possbile,
however, it is not needed.
drafts and forwarding have lots of cornercases
even when not used in combination :)
* keep draft-ids on preparing and sending
* add comments about keeping msg_id
* early exit when trying to forward drafts
* tweak tests
* get rid of old C to Rust conversion code
* allow soon checking of increation-state, add a test for that
They contain X-MSExch-Correlation-Key header, but no
Original-Message-ID, so they cannot be used to find the original
message, but we want to recognize them as MDN nevertheless to assign
them to the trash chat.
* we no longer set a draft on group creation
* remove probably outdated comment
the comment was added at
https://github.com/deltachat/deltachat-core/pull/457
where draft revamping was done half only
and was probably forgotten to being removed later.
at least, i cannot make any sense out of the comment.
When there is an outgoing message in a chat, mark all older messages in this chat as seen.
Android already has a similar behavior, however, this led to the issue https://github.com/deltachat/deltachat-android/issues/2163 and should be changed back.
--
From the issue description at https://github.com/deltachat/deltachat-android/issues/2163, I implemented these fixes:
> Core should take care that if the last message in a chat is not fresh|noticed, no messages in the chat can be fresh. [...] Do this [...] in a function that's called at the end of fetch_new_messages(). Then dc_receive_imf() wouldn't get slower by this and we could re-use the same function for migration.
So, I didn't do this inside `dc_receive_imf()` in order not to make it take even longer. This obviously has the downside of higher complexity.
And I think we should implement this:
> On Androd, show the unread badge when unread!=0 again (see deltachat/deltachat-android@618af02). Then the user can see that there is a chat with an unread message and click it to get rid of it.
because it shouldn't be the UI's job to decide whether an unread badge is shown, but the core's.
ASCII armored keys can be easily generated with `sq key generate` and
used to encrypt and decrypt test messages with `sq` and `gpg` without
converting them to binary using `base64 -d` first.
unknown keys may come from upgrades (previously used key no longer defined)
or from downgrades (when an upgrade before uses a new key).
the latter was probalby responsible for some avatar loss,
mainly by testers, that usually switch versions forth and back much more often.
* fix missing MX resolver eg. on android
switching completely to /etc/resolv.conf (see #2780)
does not work at least on some Androids
(see https://github.com/deltachat/deltachat-android/issues/2151 )
therefore, we use the old approach as a fallback.
* log a warning, when we again have problems with figuring out MX resolvers
* test adding members in a multi-device setup
* fix system messages for multi-device-setup
* enhance test to check multi-device messages for removing-members and group-renames
* repl: allow groupname arguments with more than one word (came over that when testing qr codes)
* calcualte text-size from the real number of lines
* shift text and watermark apart when text get longer
* make clippy happy
This function is used to lookup the chat by `References` and
`In-Reply-To` header, so it does not make sense to return trashed
message when there is another non-trashed message in one of these
headers with a real chat ID.
* generate qr code svg prototype
* qr code for groups
fix formatting
* - letter avatar in qrcode
- escape xml in userinput (display/groupname)
- fix "Me" display name
- merge import declarations
* remove dot at the end of VerifyContactQRDescription
* if addr == displayname, show only one of them
Especially useful for yggmail accounts without usernames,
because the text would overflow otherwise.
* use real clipPath for rounded avatar
* - center avatar text better (dominant-baseline)
- add "sans-serif" to font fallback for text if arial is missing
* make corner always blue
* add [logo + "get.delta.chat"] footer to qrcode
* Update deltachat-ffi/deltachat.h
Co-authored-by: bjoern <r10s@b44t.com>
* Apply suggestions from code review
Co-authored-by: bjoern <r10s@b44t.com>
* new card design
- add stockstrings
- update changelog
* make qrcode pixels also #f2f2f2 instead of full white
* rename VERIFY_CONTACT_QR_DESC to SETUP_CONTACT
make footer text a tiny bit darker upon r10s's request
* avoid using which is a doxygen command
* point out that one will join a group (this is still shorted and was also suggested in recent chats)
* add option to generate qr-code-svg to repl tool
* use same font-family in text and footer
* thinner card border
* remove superfluous <tspan> from footer to make color tweaking easier
* move font-weight to style, ios renderer does not pick it up from attribute; remove default font attributes not used consequently
* make get.delta.chat more visible
* align properly using dominant-baseline=central and alignment-baseline=middle, this makes things nice on all systems but android (before, ios was wrong and all others not 100% aligned as font metrics are ignored) (android needs a subsequent improvement)
Co-authored-by: bjoern <r10s@b44t.com>
`indexmap` is a large dependency (4K SLoC) containing `unsafe` code.
Contact IDs are now passed around as a Vec<u32> or &[u32].
QUOTA roots are now sorted by name instead of perserving original order.
It was there since the C core, labeled with "/* last_seen is for
future use */" but never actually used. The comment was lost during
the translation from C to Rust.
`cutil.from_dc_charpointer()` is guaranteed to return `str`, while
`cutil.from_optional_dc_charpointer()` may return `None` if C function
returns `NULL`.
It was only used in two places. Rather than adding `use
pretty_assertions::*` everywhere, it's easier to remove, and it
removes two additional dependency crates.
`deltachat.const` module now defines `__getattr__` and `__dir__` as
suggested by https://www.python.org/dev/peps/pep-0562/
mypy detects that `__getattr__` is defined and does not show errors
for `DC_*` constants which cannot be detected statically.
mypy is added to `tox.ini`, so type check can be run with `tox -e mypy`.
Generally we could also just remove the device message as we have the
connectivity view, OTOH if you are not using DC a lot, it may be useful
to be notified without opening DC.
And apart from this one bug it's working fine.
`apply_group_changes` is executed regardless of whether the group is
created via `create_or_lookup_group` or found via
`lookup_chat_by_reply`. This change removes the need for
`lookup_chat_by_reply` to return `None` when group ID exists in the
database to let `create_or_lookup_group` run. As a side effect of this
Delta Chat replies to ad hoc groups are now correctly assigned to
chats when there are multiple groups with the same group ID, such as
ad hoc groups with the same member lists.
* test one-to-one chats on setup-contact/secure-join
only one chat is created after scanning a QR code:
- on setup-contact, one-to-ones are created on both sided
- on secure-join, the joined group chat is created;
one-to-ones are not created intitally,
but should become visible on receiving messages
* make sure, Alice creates the chat with Bob on setup-contact
not totally sure if that change in #2508 was on-purpose,
however, all yet released versions
did create the one-to-one chat also on the Inviter's (Alice) side,
so, let's stay with that,
i do not see many reasons to change that.
* unblock hidden (Blocked::Yes) one-to-one chats
one-to-one chats may be hidden by secure-join,
in case someone later writes a message to it
(not unlikely), the chat needs to be shown.
before, messages are just not shown,
the corresponding chat did not appear.
the 'Blocked' wording of a 'Chat' must not be mixed with the
'Blocking' of a contact. 'Chat-Blocking' is mostly a visibility thing,
that may change as messages come in.
this change should not affect _really_ blocked contacts -
they are filtered out already before
and their messages are usually not even downloaded.
also, before allow_creation is checked,
that may disallow chat creation for show_emails reasons.
all in all, it just does the same
as if the user has manualy deleted the chat before and it would be created.
* simplify test
utcfromtimestamp() is not recommended by the official documentation,
because many methods, including timestamp(), work incorrectly with
"naive" datetimes returned by utcfromtimestamp().
Update gossiped_timestamp when someone else sends autocrypt gossip in
the group, so we postpone sending gossip again ourselves.
- Warn about failures to parse Autocrypt-Gossip header
- Move gossip-related methods into ChatId impl
- Fix a "gossi_pp_ed" typo
- Make `state` variable immutable.
- Don't mark contact requests as fresh when fetching existing messages: this is not needed since there is no contact request chat anymore.
- Don't mark messages from blocked chats as noticed, this is not needed since messages go to blocked chat anyway instead of contact request chat.
* refactor: cleanup send_handshake_msg()
- rename to send_alice_handshake_msg() as used by Alice only
- remove dead code from Bob
(Bob's code is at BobState::send_handshake_message() since some time)
- take a contact_id and not a chat_id;
this makes things less confusing when
info-messages are put to the final group chat
* always directly return chat-id from dc_join_securejoin()
* take care not to create a group twice
* adapt documentation
* add info-msg on group invites; add inviter directly after creation
* document existing 'joinqr' command in repl tool
* do not create empty one-to-one chats for group-joins
* refactor: cleanup fingerprint_equals_sender()
- the function takes a contact_id directly now.
before it consumes the first contact of a one-to-one chat -
which may be easily confused with the group-chat in creation.
moreover, the conversion contact_id -> chat_id -> contact_id
is unneeded overhead.
* show info-messages in destination chat for alice
* fingerprint_equals_sender() returns Err on database failure
* tweak documentation
* clarify what an 'unfinished tasks' task is.
* add regression test for create_for_contact_with_blocked()
* rename Blocked::Manually to better fitting Blocked::Yes
* tweak test_secure_join() and make sure, Alice and Bob have only on chat after a group-join
* add basic multi-device-sync functions
* generate json
* add context.parse_sync_items()
* add context.execute_sync_items()
* piggyback sync-commands message, add body for human-readable part
* avoid double json renderings
* mimeparser parses incoming .json sync-files
* do not piggyback sync-files
* execute sync items
* return status of send_sync_msg()
* send sync messages as multipart/report
* add a per-item-timestamp and also allow adding other per-item-fields in the future
* if the self-chat does not exist, create it blocked/hidden
* create tokens closer to real qr-code needs
* respect bcc_self setting, add test for that
* sync qr code tokens after promoting groups
* send sync-messages only if an experimental switch is set
* trigger send_sync_msg() after sending messages and after creating/redraw/revive qr-code
* add DC_STR_* constants to deltachat.h
* adapt to refactored qr module as of #2729
* tweak test
* use SendSyncMsgs config name instead of SendExperimentalSyncMsgs - we can remove or rename the config nevertheless, but have the option to keep it without renaming
* tweak docs
* remove currently unused effective timestamp calculation
* clarify when send_sync_msg() is called
* make sure, sync-messages are encrypted and are sent by SELF
* tweak docs, fix typos
Before this PR, when a test failed, we often got:
```
thread panicked while processing panic. aborting.
error: test failed, to rerun pass '-p deltachat --lib'
Caused by:
process didn't exit successfully: `/home/user/deltachat-android/jni/deltachat-core-rust/target/debug/deps/deltachat-33648fc4aaad608c 'contact::tests::test_selfavatar_changed_event' --nocapture` (signal: 4, SIGILL: illegal instruction)
```
instead of the error message and stack trace.
This PR fixes this.
* Add DC_EVENT_SELFAVATAR_CHANGED
* Add test.
Unfortunately I can't easily also test that the avatar is not copied
from unencrypted messages:
In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.
* Unfortunately I can't easily also test that the avatar is not copied from unencrypted messages:
In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.
* add let's encrypt certificate missing on some older android devices
* create Certificate with Lazy::new()
* document certificate source
* use smaller *.der format instead of *.pem
Reduce the number of mutable variables and out parameters in
`add_parts`.
Also don't call `dc_create_smeared_timestamp` if there is no `Date`
header. Timestamps are only supposed to be "created" when the message
is sent, not received, to make sure sent messages are sorted properly
in MUAs that only use the date for sorting. Delta Chat uses database
IDs for sorting in addition to timestamps, so it can sort messages
with equal timestamps properly.
Update `dc_smeared_time` documentation.
Turn `dc_smeared_time` and `dc_create_smeared_timestamp` comments into
documentation comments.
The problem was:
When opening the connectivity view while there is no network,
get_connectivity_html() calls schedule_quota_update(), which schedules a
UpdateRecentQuota job. But in update_recent_quota(), connecting fails
and a ConnectivityChanged event is emitted (connectivity changes from
Error to Connecting and back). Therefore the UI calls
get_connectivity_html() again, and the loop is complete.
This made the UI completely unresponsible. To reproduce, just turn wi-fi
off and open the connectivity view.
The fix is:
schedule_quota_update() now only schedules a new job if there is no old
job. To prevent the possible (though probably unlikely) problem that an
old quota update job has a backoff of, like, a day and therefore quota
is not updated, I reduced the backoff time for quota jobs to 10s.
Fixes possibly https://github.com/deltachat/deltachat-android/issues/2043, but we should "re-try" this
* better readable enum
* add a failing test
* let new secure-joins abort existing ones
before, a stale secure-join or setup-contact
made the whole qr-scanning unusable until
the app is restarted,
resulting in "QR process failed" errors.
this commit fixes the issue by
aborting existing scans -
in cases, a user really wants two concurrect joins running,
this is not perfect, but that did not worked before as well.
* remove unused AlreadyRunning variant
* make clippy happy
* add Chattype::Broadcast and create_broadcast_list()
* do not disclose recipients for broadcasts
* allow sending/add-/remove-member for broadcast
* set broadcast subject same as for one-to-one chats
* broadcast-recipient-list does not include SELF
* use special icon for broadcast groups
* generate initial broadcast names
* make clippy happy
* send BCC message unencrypted to avoid unexpected disclosing; encryption is opportunistic anyway. if we have 'protected chats' at some point, we can think that over.
* reword 'To:'-group
* simplify can-send-check
* add broadcast tests
* tweak comments
* Update deltachat-ffi/deltachat.h
Co-authored-by: Hocuri <hocuri@gmx.de>
* change name of can_edit() to is_self_in_chat()
Co-authored-by: Hocuri <hocuri@gmx.de>
this is needed for targeting "non-blocking group QR joins"
as create_multiuser_record() would also be needed from other places.
this will make rebasing/rewriting and finally reviewing #2508 easier.
in the past, group-messages were marked as "read by recipient"
only when at least 50% of the group members have send a read receipt -
in practise, this does happen never or much too late esp. in larger groups.
setting the state OutMdnRcvd already on the first read-receipt
seems to be much more intuitive and useful as you at least know
one person has read the message.
this is also what other messengers as telegram are doing here.
moreover, this fixes a bug that did not add all read-receipts
to the "Info" screen - once "enough" read-receipts were received,
and the state was already set to OutMdnRcvd, subsequent read-receipts
were ignored.
probably because the "Info" screen did not show the read-receipts since forever -
and for the second tick, the addutional read-receipts are not needed.
* let quota-warning reappear after import
an import removes all device-messages,
including the quota warning.
resetting `Config::QuotaExceeding` makes sure,
the warning reappears soon after import -
otherwise the warning would reappear only after
storage is cleaned up and exceeds again.
* a second quota warning when 95% storage exceeded
* factor out warning-check and add a test for that
* add missing stock strings for connectivity and quota
* reword quota error
* use 'Incoming/Outgoing Messages' instead of 'Inbox/Outbox'
* just say 'Not supported by your provider.' if quota cannot be read.
- the context is already given by the headline
'Storage on domain.org'
- the string is short and does not disturb much
(it is a very common error)
- the string does not mention 'quota' as such,
which will be hard to translate ('Kontingentsinformationen') -
even if ppl as we know about the Quota extensions an such things.
there was also the idea to hide the whole section,
however, that is confusing in a multi-device-usage
when things are sometimes shown and sometimes not.
before, MsgsChanged(chat_id, new_msg_id) was emitted,
but that does not cover the deleted message.
in theory, we could emit both,
however, that would just be a waste of refresh in uis.
also before, events were used this way,
however, also the documentations are updated to
reflect reality better.
Implement get_previous_message() that only looks for the last
Message-ID in the References: header instead of using
get_parent_message() which falls back to using other Message-IDs from
References and In-Reply-To field.
* add a test for xing mailinglists
* strip long hash-prefixes from mailinglist name if we got the name from the List-Id as a last resort
* add a test for newsletter@ mailinglists
these mailinglists have the list-name in `From:`
and can be detected by addresses starting with `newsletter@`.
* if not list-name is set, use the `From:` name for addresses starting with `newsletter@`
this is similar to what we do with `notifications@`
* Update src/dc_receive_imf.rs
Co-authored-by: Hocuri <hocuri@gmx.de>
* add an example to the regex
Co-authored-by: Hocuri <hocuri@gmx.de>
* draft a download-api
* basic implementation
* allow partial downloads for protected chats
* use a separate column for download_state
* force a minimal timeout for delete_server_after in combination with partial messages
* add a warning if a possible download may expire by delete_server_after
* test load_imap_deletion_msgid()
* add a test for a partial download
* improve documentation and visibility
* let get_download_limit() return Result<Option>
* rusty getters
* apply MIN_DELETE_SERVER_AFTER to shown availability time
* move stub-creation to download.rs, use stock-strings, nicer logging
* make clippy happy (cargo clippy --tests)
* refine tests and comments
* fix typo
* remove superfluous closure in ffi
* respect partial_download for immediately scheduled DeleteMsgOnImap jobs
Integer overflows crash the application by default.
On a first sight this is only a potential crash that can't be
triggered, because timestamps are stored as i64 and ephemeral timer
duration is u32.
* add a test for .xt.local mailinglists
* get correct names of .xt.local mailinglists
these mailinglist probably come from the xt:Commerce system
and are pretty widely used.
i have not seen an .xt.local mailinglist with name set in List-Id,
however, if that happens, it will still be taken.
only if the name is unset,
we use the name from the From-header.
summary::Summary replaces Lot in the Rust API for methods returning
chatlist summaries. Lot is a legacy type for C API compatibility, so
Summary can be converted into Lot.
There is a real risk of an active attack when connecting to non-.onion
servers over Tor, as bad Tor exit nodes are cheap to set up.
It's probably not needed for .onion domains, but we don't make an
exception for now.
* check update timestamps for signatures, user-avatars, ephemeral-settings, last-subject
* check update timestamp for group-avatars
* check update timestamp for group-names
* check update timestamp for memberlist
* check update timestamp for protection-settings
* add a more advanced test
* add another more advanced test
* set last-subject-timestamp more carefully
* bubble up errros from set_*timestamp() and check for from_id==0 before
* simplify Params::set_i64()
* remove comment that is more confusing than helpful
* use update_timestamp() wording consistently
Also make Accounts uncloneable. It is still possible to derive Clone,
but does not make sense to do so, as .clone() creates two separate
account managers which use the same files but different unsynchronized
in-memory data structures.
Debian deprecated `which` in `debianutils` in favor of `command`.
`which` outputs this to stderr now:
/usr/bin/which: this version of `which' is deprecated; use `command -v' in scripts instead.
* resultify update_recent_quota()
* add a device-message if quota exceeds QUOTA_WARN_THRESHOLD_PERCENTAGE
* check if a quota warning should be added during housekeeping, this is at least once a day
* dc_get_config("quota_exceeding") is useful for bots
* make clippy happy
* reword QuotaExceedingMsgBody message
* avoid lots of warnings if quota jitters around the warning threshold
* allow clippy::assertions_on_constants
these constants depend on each other, it makes sense to check that they are not changed in an incompatible way
This adds following settings:
- Socks5Enabled
- Socks5Host
- Socks5Port
- Socks5User
- Socks5Password
Currently http requests and dns requests are not getting executed as they currently can't get tunneled through socks5 proxy. Therefore gmail with oauth2 wont work through tor.
- Use the same limit for info: full text can be read in HTML anyway.
- Remove DC_MAX_GET_{TEXT,INFO}_LEN constants from deltachat.h
- Fix a typo: s/DC_ELLIPSE/DC_ELLIPSIS/
- Do not truncate the text when loading from the database.
- Update the documentation: limit is in Rust chars, not bytes
the 'device chat about' was shown as a single message
as at that time, there was just not 'status'.
meanwhile, we have the status option,
and it feels much more natural to get the information there,
esp. as the subtitle on all UIs already read
'Messages in this chat are generated locally' -
and a tap on that will show the hint, without scrolling or so.
this also teaches the user where to find such information -
and the "welcome" chat is less spammy and really starts with the text
"Welcome to Delta Chat!"
* allow dotless domains and hostnames
* update provider database
ran ./src/provider/update.py ../provider-db/_providers/ > src/provider/data.rs
to pull in recent changes from https://github.com/deltachat/provider-db
* add imap::get_quota_roots()
* schedule quote-checking job on getting connectivity-html
* get quota and debug print it
* basic quota output
* update quota at most once per minute, emit event on changes
* use more meaningful names
* add some comments, move update_recent_quota() to quota.rs
* show root name only if there are several roots
* make clippy happy, some refactorings
* allow only one update-quota job per time
* add now supported QUOTA to standards.md
This bug sometimes results in sending read receipts to self in
multi-device setups.
It happens consistently in a setup where the first device is
configured to move messages to DeltaChat folder and the second device
is not. When both devices receive BCC-self message simultaneously, the
first device moves the message to DeltaChat folder, while the second
device tries to mark the message as seen in the Inbox. Regardless of
whether the second device marks the message as seen successfully or
fails because the message is already moved by the first device,
`Job.markseen_msg_on_imap()` sends the read receipt to the From
address.
Recognizing these emails as forwarded would probably be too complicated and require too much special-casing, but now the user can access the email text via "Show full message".
fix#2599
Co-authored-by: B. Petersen <r10s@b44t.com>
* In dc_maybe_network_lost() directly set the connectivity "Not connected"
r10s reported that without doing this, the connectivity would stay at
"Connected" for 16 more seconds after network is gone and
dc_maybe_network_lost() was called.
* Set the state for all connections
fix#2539
It's always a bit ambiguous which function should do set_err or set last_send_error, I used this rule here:
If some code returns Status::RetryLater, then it sets last_send_error, because Status::RetryLater means that the user won't see the error directly on the message (if we returned Status::Failed(Err(_)), then the message would be shown as failed to the user) Also, smtp_send always sets last_send_error because, well, sending just failed or succeeded.
Also, remove unused field pending_error.
Previously system messages were always added to the end of the chat,
even if the message triggering them was sent earlier. This is
especially important for messages about disappearing timer reset
triggered by classic email messages, as they should be placed right
after the message resetting the timer.
- Use latest stable for rustfmt and clippy.
- Update Rust in rust-toolchain and coredeps Docker container to 1.54.0
and test against it.
- Test against 1.48.0 to ensure libdeltachat can be compiled with Debian
bullseye "rustc" and "cargo".
Gmail labels are not folders and should be skipped. For example, emails
appear in the inbox and under "All Mail" as soon as it is received. The
code used to wrongly conclude that the email had already been moved and
left it in the inbox.
the WAL files come from sqlite WAL mode,
they have the same name as the database file with an extra `-wal` suffix.
according to https://sqlite.org/wal.html#the_wal_file ,
the WAL file is part of the persistent state of the database
and should be kept with the database if the database is copied or moved.
"Archived Chats" also contain old contact requests now
(an maybe new one if the user archives them),
so, there may be easily some thousands chats in "Archived Chats" -
and the count has no real meaning to the user,
esp. as we not even display the number of "Real Chats".
Might be that this "Link" will go away anyway sooner or later,
however, for now, it is just fine to remove the badge counter.
* add a test for archived requests
* fix archived requests
* move requests but the last one to "Archived Chats"
this way, the app looks familiar after the contact request upgrade.
the subselect was copied from the old get_last_deaddrop_fresh_msg()
(which was removed by the contact request upgrade #2514)
* just move all old requests to "Archived Chats"
ux-wise, the advantage of keeping the last one is questionable,
one may think, always the last one is shown in chatlist.
showing _all_ fresh request is not doable
as past cores did not really take care of that
and the db-state is not consistent in that regard.
that would make the already complicated code even more complicated,
so we decided to go the easy way.
* accounts: update EventEmitter on add_account
* accounts: do not lock waiting for EventEmitter in dc_accounts_add_account
Otherwise dc_accounts_add_account blocks until an event arrives on
some existing account.
Contact request chats are not merged into a single virtual "deaddrop"
chat anymore. Instead, they are shown in the chatlist the same way as
other chats, but sending of messages to them is not allowed and MDNs
are not sent automatically until the chat is "accepted" by the user.
New API:
- dc_chat_is_contact_request(): returns true if chat is a contact
request. In this case option to accept and block the chat via
dc_accept_chat() and dc_block_chat() should be shown in the UI.
- dc_accept_chat(): accept contact request and unblock the chat
- dc_block_chat(): decline contact request and block the chat
Removed API:
- dc_create_chat_by_msg_id(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_marknoticed_contact(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_decide_on_contact_request(): this call requires a message ID from
deaddrop chat as input. As deaddrop chat is removed, this call can't
be used anymore.
- dc_msg_get_real_chat_id(): use dc_msg_get_chat_id() instead, the
only difference between these calls was in handling of deaddrop chat
- removed DC_CHAT_ID_DEADDROP and DC_STR_DEADDROP constants
* escape strings added to html
* use more common emojis for connectivity report
all emojis are from 2010 and older now.
an alternative would have been to use css,
however, that may have other issues
and as the whole report is subject to change anyway,
i go for the easy solution.
* use 'modern' meta pattern, remove unused div and styles
* use css instead emojis; looks better that way
also, we have the same look on all systems.
* add connectivity command to repl tool
fix https://github.com/deltachat/deltachat-core-rust/issues/2463fix#2116
The email could be private (i.e. only sent to me) or non-private (i.e. also sent to sb else). Also, it could be a classical email or a chat message. Below, I'm describing for each of the four combinations whether they should be assigned to a chat by create_or_lookup_group() or lookup_chat_by_reply(). Because I needed to use these function names a lot, I shortened them to l:group() and l:by_reply() in this PR description.
(!) means todo, (! -> Done) means I fixed something.
## Private classical email:
l:group() and l:by_reply() must both take care not to put it into group
l:group() no (! -> Done), l:by_reply no (! -> Done)
except for classical MUA replies to two-member-groups, they should be put into the parent group
### wrt alias-support:
A private classical email is very probably not going to be an answer to email that went to an alias address:
Suppose Alice writes to support@example.com.
support@example.com forwards to bob@example.com and claire@example.com.
When Bob answers, he will _probably_ answer `To: alice@example.com, support@example.com` (=> it's a non-private classical email).
With this PR, if he does only answer `To: alice@example.com`, (=> it's a private classical email), Alice's DC will show the answer in the private chat with Bob. Which actually makes more sense than showing it in the support@example.com chat I think. Also, if it was shown in the support@example.com chat, then Alice would answer in the support@example.com chat, then Claire would get Alice's message but not Bob's and therefore miss some context.
That being said - **I could change this**. Pretty easily actually, I would just have to remove the call to `is_probably_private_reply()` from `lookup_chat_by_reply()`. Didn't think through all edge cases yet, but should work.
## Private chat message:
l:group() has to put private chat messages into the group: It won't mistakenly put a message there (because it can rely on Chat-Group-Id). Currently, `is_probably_private_reply()` returns true for private chat messages, but l:group still has to put them into the group chat. (it's not nice that the function called is_probably_private_reply returns true for all private chat messages, but I didn't find any nicer solution) l:by_reply() must not assign it to sth special
l:group yes, l:by_reply no
By the way, for chat messages, `try_getting_grpid()` doesn't look at InReplyTo or References, in order not to put private chat-message replies into the group chat.
For alias-support, the same goes as for the private classical emails.
## Non-private classical email:
Just put it into any group, and if there is none, create one.
_Off-topic:
We currently don't look at the recipients lists, which means that a message can easily be assigned to a group although it was not sent to all group members. One day we could somehow compare the recipients list with the members list, but that needs some more discussing (esp. what do we do if they don't match? Create a new group? Show a hint in the UI?) and is nothing for a bug-fixing PR like this one._
l_group yes, l_by_reply yes, even for outgoing messages (! -> Done, this also was the issue reported by @gerryfrancis in the testing group:
Outgoing messages were not put into a chat by In-Reply-To/References, which is correct for chat messages, but for non-private classical outgoing emails, a new ad-hoc group was created everytime.
So, I added this: https://github.com/deltachat/deltachat-core-rust/pull/2465/files#diff-e7606b521f6710ddc6e5236ba5d7eefc917b7ad744b9e71762fd42830c55485bR703-R711)
## Non-private chat message:
Can be put into chat by l:by_reply() because it must be a group message
l:group yes, l:by_reply yes (to make alias support work if the support person uses DC)
Nothing to test or fix here; we have to put chat group messages into the group, which is trivial. And we have to make sure that alias-support works, which already was well tested.
* changelog: update to include prevous changes
and use new format/structure
* changelog: capitalize API
* changelog: add a missing pr refernece
* Apply suggestions from code review
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: Hocuri <hocuri@gmx.de>
See https://support.delta.chat/t/discussion-how-to-show-error-states/1363/10 <!-- comment -->
It turns out that it's pretty easy to distinguish between lots of states (currently Error/NotConnected, Connecting…, Getting new messages… and Connected). What's not that easy is distinguishing between an actual error and no network, because if the server just doesn't respond, it could mean that we don't have network or that we are trying ipv6, but only ipv4 works.
**WRT debouncing:**
Sending of EVENT_CONNECTIVITY_CHANGED is not debounced, but emitted every time one of the 3 threads (Inbox, Mvbox and Sentbox) has a network error, starts fetching data, or is done fetching data.
This means that it is emitted:
- 9 times when dc_maybe_network() is called or we get network connection
- 12 times when we lose network connection
Some measurements: dc_get_connectivity() takes a little more than 1ms (in my measurements back in March), dc_get_connectivity_html() takes 10-20ms. This means that it's no immmediate problem to call them very often, might increase battery drain though. For the UI it may be a lot of work to update the title everytime; at least Android is smart enough to update the title only once.
Possible problems (we don't have to worry about them now I think):
- Due to the scan_folders feature, if the user has lots of folders, the state could be "Connecting..." for quite a long time, generally DC seemed a little unresponsive to me because it took so long for "Connecting..." to go away. Telegram has a state "Updating..." that sometimes comes after "Connecting...".
To be done in other PRs:
- Better handle the case that the password was changed on the server and authenticating fails, see https://github.com/deltachat/deltachat-core-rust/issues/1923 and https://github.com/deltachat/deltachat-core-rust/issues/1768
- maybe event debouncing (except for "Connected" connectivity events)
fix https://github.com/deltachat/deltachat-android/issues/1760
in most (all?) UIs, import/export works on an already created account,
so, dc_accounts_import_account() does not really help here -
but adds some noise and confusion
eg. as for the other dc_accounts_t functions,
the corrsponding dc_context_t functions must not be called.
if really a new account is required for import,
it seems to be easier to call add_account() before import.
Do not process names, avatars, location XMLs, message signature
etc. for duplicate messages.
Previously only `add_parts` was stopped early, but not
`dc_receive_imf`. Also, `dc_receive_imf` processed From: and To:
fields and applied names to contacts even before checking the
Message-ID.
Fake Message-ID generation procedure is changed to operate on raw
header values to avoid interacting with the database.
as by reviving qr codes,
there may be more than one token for a chat,
ensure, the most recent token and only one token is returned
by the sql-command for looking up tokens
(used for generating new codes)
Raw MIME messages may contain non-ASCII characters. Attempting to
store them as TEXT by using String::from_utf8_lossy results in
non-ASCII characters being replaced with Unicode U+FFFD "REPLACEMENT
CHARACTER" which is later incorrectly decoded when attempting to parse
`mime_headers` content into HTML.
Normally they should not end up in contact requests, but if they do,
we want to show them. Otherwise it is completely impossible to see
them until the chat is moved out of contact requests.
Previously chats created by outgoing classic emails went into contact
requests (deaddrop). Outgoing messages are not shown in contact
requests, so created chat was not shown anywhere. With this fix chat
is created both for outgoing classic emails and outgoing chat emails.
Since classical messages are not deleted when show_emails setting is
set to "0" and remain in the database, they should be shown
somewhere. Otherwise they may reappear later when the setting is
enabled again.
New `dc_msg_is_bot()` C API and corresponding `Message.is_bot()`
Python API can be used to check if incoming message is sent by a bot,
e.g. to avoid two echo bots replying indefinitely to each other.
"Bot" flag is not set for outgoing messages, but may be set for
BCC-self messages. For now documentation says that `dc_msg_is_bot()`
return value is unspecified for outgoing messages. It can be better
specified later if needed for specific applications, e.g. sharing an
account with a helper bot.
the default account created by dc_accounts_new()
is annoying when doing a migrate/import account;
in this case, you must not forget to remove the default account.
in between you have two account,
and accounts are swiched.
it seems to be better
to just call add_account() on a new account object as needed.
systemli.org seems not to include Received: header in messages sent to
self, including BCC-self and messages sent to Saved message chat. This
results in all these messages not being displayed on other devices in
multi-device setting. To work around this problem we restrict draft
detection based on Received: header to classical mails.
Delta Chat already implements proper draft detection based on the
\Draft message flag and \Drafts folder flag. However, Thunderbird does
not set \Draft flag when message is stored as "Template". To make
draft detection for Thunderbird more robust, in case Received: header
check is removed altogether later, we check for X-Mozilla-Draft-Info
header which Thunderbird sets both for "Drafts" and "Templates".
probably they come in by the latest sqlx<->rusqlite moves,
however, as they are followed by subsequent migrations,
that should not have been a big bug in the past
(maybe unless the app was killed at a bad moment)
Even if chat has no messages, contacts should not be deleted.
Otherwise chat will reference invalid contact ID which can't be loaded
from the database, resulting in broken contact list in group chats.
`Imap` structure is always created in a configured state now. There is
no default value for `ImapConfig` anymore.
Also resultify Scheduler::start() to fail on database errors, for
example if IMAP configuration cannot be read from the database during
`start_io()`. Previosuly errors during reading keys such as
`mvbox_watch` were simply ignored and folders were not watched until
the application is completely restarted, now start_io() will fail and
scheduler will only be started at the next start_io() call which
usually happens when app is brought to the foreground.
* Add failing test for #2369
* Completely ignore Drafts folder
fix#2369
* Also ignore messages that have the Draft flag set but are not in the Drafts folder
Building aarch64 wheels under qemu takes a long time.
rust 1.52.1 includes fix for cargo that may result in
compilation being stuck: https://github.com/rust-lang/cargo/pull/9201
There was a bugreport for previous versions of DC that BCC-self
messages are not marked as seen if server-side Sieve rule moves them.
This test is an unsuccessful attempt to reproduce the bug,
which was confirmed to be fixed.
This test checks that MDN is marked as seen on ac1. Because ac1 also
receives BCC-self, it is possible that event of marking this message
as seen is confused with marking MDN as seen, and the test passes even
if MDN is not marked as seen. Explicitly disabling BCC-self for ac1
ensures it receives only one message (MDN).
Also start direct_imap as early as possible, before sending the
message. It is possible now, because BCC-self is not sent.
When watch_inbox was off, scan_folders failed and a toast "IMAP operation attempted while it is torn down" was shown.
--
The problem was:
When inbox_watch is off, scan_folders() is called at 244260a978/src/scheduler.rs (L107) but connect_configured() is not called before.
* Add test
* Don't only setup handle, but connect_configured() in scan_folders()
This allows to run `tox `on systems with only a single Python 3
version installed, such as Python 3.9.
CI script scripts/run-python-test.sh specifies `-e py37` explicitly
anyway.
The problem was:
If a message has the \Deleted flag set, we ignore it. But we forgot to
update last_uid, so that uid_next was not updated at 47e639b777/src/imap.rs (L730) and the same message was
fetched over and over again.
Fix#2429
Using `impl AsRef<str>` as the argument instead of `&str` makes it
possible to call the function with `&str`, `String` and other types
that implement `AsRef` trait.
The cost of it is that compiled binary contains mulitple versions of
the same function, one for each variant of types. If function contains
multiple generic `impl AsRef` arguments, the number of versions possibly
compiled into binary grows exponentially with the number of arguments.
Simple way to avoid it is to call `.as_ref()` on the caller side to
convert the argument to `&str`. In most cases even adding a `&` and
relying on `Deref` coercion is sufficient.
This patch changes many functions that accepted `impl AsRef<str>` and
`impl AsRef<Path>` to accept `&str` and `&Path` instead.
In some places `.clone()` calls are removed. Calling `.clone()` on
`String` and passing `String` to a function accepting `impl
AsRef<str>` is completely unnecessary as `&str` reference could be
passed instead. There is no clippy warning against it yet, but
changing argument type to `&str` allowed to find these cases.
The result of debloating is not impressive, several hundred kilobytes
are saved, which is about 3% of the `.so` binary, but the code is
cleaner too.
This moves the module-level lookup and creation functions to the
types, which make the naming more consistent. Now the lookup_* get_*
and create_* functions all behave similarly.
Peraps even more important the API of the lookup now allows
distinguishing failure from not found. This in turn is important to
be able to remove reliance on a ChatId with a 0 or "unset" value. The
locations where this ChatId(0) is still used is in database queries
which should be solved in an independed commit.
This introduces the explicit ChatIdBlocked struct to more explicitly
create a chat with a blocked status. It also adds a common shortcut
to ChatId itself which is more natural to use in many cases.
* Remove sql::error submodule
Use anyhow errors instead.
* Remove explicit checks for open SQL connection
An error will be thrown anyway during attempt to execute query.
* Don't use `with_conn()` and remove it
* Remove unused `with_conn_async`
* Resultify markseen_msgs
- Currently, group images are compressed as well because it was easier to implement that way.
- Currently, in the unlikely case that the avatar is compressed down to 20x20 pixels but still bigger than 20KB, the user doesn't get any indication of this, the avatar simply isn't changed (at least on Android).
If we want to change this, the easiest way is probably to let `dc_set_config()` in the ffi call `error!()` if `Selfavatar` can't be set. The same might make sense for some or all other configs. BUUUUUT: At least Android doesn't show error!() toasts anymore, probably because they were used too often and too spammy.
- The factor by which we scale down if the file is too big is 1.5.
Instead, allow specifying free-form BUILD_ID from the command line.
scripts/remote_python_packaging.sh still uses CIRCLE_ variables to avoid
changing working CircleCI config.
This feature is similar to existing avatar synchronization.
Whenever encrypted BCC-to-self copy of chat message is received, status
setting is updated with the signature of the message.
Rust does not work on manylinux2010 due to old GNU C Library.
We have been using manylinux2014 on CI machine already, but this change
was never commited.
messages are marked as 'noticed' already when the chat is opened
as all UIs call dc_marknoticed_chat();
also marking messages as 'noticed' when dc_markseen_msgs() is called
is not needed therefore - but stands in the way of further improvements
for the deaddrop, eg. UI may let the user decide when the deaddrop
can be removed from chatlist ('All done' button or so)
* Fix#2335 (delete job flooding)
The problem was:
- You are offline, and an ephemeral message is due to delete.
- load_imap_deletion_job() is called and returns a deletion job for it.
- The job fails because, well, we are offline.
- The job is saved into the database.
- load_imap_deletion_job() is called again, and so on.
* Add test
ct.name was insufficient, as authname, overridden name and email address
fallback were not taken into account.
Dropping this condition increases performance by 25% according to the
benchmark.
Also add a test for search_msgs.
It reduces the time by ~20% according to `search_msgs` benchmark.
Sorting by IDs is sufficient for global search as IDs increase in the
order of message reception.
This PR originally contained a fix for sqlx which turned out not not to be necessary. But on the way, I made some general improvements:
- Under some circumstances, a "normal" test failure led to a timeout, without printing a decent error message. See e.g. https://app.circleci.com/pipelines/github/deltachat/deltachat-core-rust/8069/workflows/ba2a9949-b4ad-4ceb-a930-073bba05e2db/jobs/30965.
(The problem was: if there is an exception in dc_account_extra_configure(), when trying to handle the exception the line account.log("===================", e, "===================") was called, which can't work as log() only expects one parameter)
- When a test fails: Call `dump_account_info()` even if there is no direct_imap
- In test_import_export_online_all(), add another 100KB file to the backup. This adds resilience against future size changes of the sql db file: The test tests the smoothness of the progress bar. And if there are there are not enough about-equally-sized files, the progress bar can't be smooth.
the user-given contact name may be set to an empty string;
in this case the authname or the email is used for the contact
and also for the name of possibly existing chats.
this works well for `dc_chat_get_name()` as that just uses
`dc_contact_get_display_name()` for single-chats.
it did not work for `dc_get_chatlist(query)` as that
uses the database for performance reasons -
however, in the database, the empty string is written
instead of the display name is written for a chat.
this is fixed by this pr by also using
dc_contact_get_display_name() when updating the chats-table
(similar to `dc_create_chat_by_contact_id()`)
message type directly (ex. `const.DC_MSG_STICKER`) so if new message types are
added, they can be used direcly without needing the python API to be updated.
This is an PGP/MIME format produced by Microsoft Exchange and ProtonMail IMAP/SMTP Bridge,
described in detail in https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html
This patch adds seamless support for "Mixed Up" Encryption, repairing
mangled Autocrypt messages without notifying the user.
&str queries are not persistent by default. To make queries persistent,
they have to be constructed with sqlx::query.
Upstream sqlx does not contain the change that make all queries
persistent, but it is not needed anymore. but
fix #2073fix#2292 (I think)
- Messages can be assigned to any chat by the References and In-Reply-To, also to 1:1 chats; this has higher priority than the group id because with ad-hoc groups, it can happen that two devices have different group ids for the same conversation thread.
- If `From` is not in the chat (we call this "shadow sender"), `OverrideSenderDisplayname` is set. This communicates to the UI that:
- A `~`should be added in front of the sender's displayname.
- Also in 1:1 chats, the sender's displayname and avatar is shown, as if this was a group.
The "Unknown sender for this chat" messages are completely removed for unprotected groups.
For protected chats, everything stays as it was before.
POSTPONED:
- Maybe (if it turns out to be still necessary):
- The ad-hoc group id is computed by the the References, instead of the member list, as it is currently done
- How do we prevent that the message can't be assigned to the correct chat as the parent message was deleted?
Switches from rusqlite to sqlx to have a fully async based interface
to sqlite.
Co-authored-by: B. Petersen <r10s@b44t.com>
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: link2xt <link2xt@testrun.org>
fix#2188
Also, if we notice that the server started reusing old UIDs, _also_ do a `ResyncFolders`, because the server likely forgot to change uid_validity
d="m 8.6780027,35.573064 0.032831,-11.910176 c 0.00138,-0.476406 0.4881282,-0.794259 0.9235226,-0.604877 l 4.1144877,2.345752 -0.02386,8.656315 -4.1268029,2.122946 C 9.1617452,36.370003 8.6766889,36.049472 8.6780027,35.573064 Z m 5.0469633,-1.508222 0.02386,-8.656314 31.145424,-9.537653 c 0.841472,-0.219211 1.65915,0.41667 1.656755,1.283728 l -0.06929,25.139995 c -0.0024,0.867062 -0.825942,1.500799 -1.663803,1.274581 z m 3.8042,6.892234 C 16.681121,40.104348 16.315444,38.819414 16.69043,37.591308 l 2.252234,-7.347193 c 0.2644,-0.861571 0.845185,-1.567441 1.641953,-1.989251 0.796769,-0.421808 1.706956,-0.509819 2.568531,-0.245419 l 7.263888,2.225804 c 1.775518,0.543235 2.780299,2.432591 2.232297,4.208094 L 30.3971,41.790532 c -0.545627,1.777887 -2.432591,2.780297 -4.208095,2.232298 l -7.263891,-2.225804 c -0.545033,-0.165864 -1.01825,-0.460162 -1.395948,-0.83995 z m 12.377693,-7.976728 c -0.07601,-0.07642 -0.17114,-0.133864 -0.280621,-0.167516 l -7.263891,-2.225803 c -0.233244,-0.07209 -0.421626,0.0013 -0.512275,0.04861 -0.09064,0.0474 -0.25772,0.166033 -0.327435,0.396899 l -2.252234,7.347191 c -0.108166,0.354628 0.09088,0.731541 0.447888,0.842099 l 7.263891,2.225802 c 0.354626,0.108174 0.731539,-0.09088 0.842099,-0.447888 l 2.249845,-7.344814 c 0.07453,-0.245145 0.0014,-0.504991 -0.167267,-0.67458 z"/>
- The secondary self addresses (see below) are shown in the UI, but not editable.
- When the user changed the email address in the configure screen, show a dialog to the user, either directly explaining things or with a link to the FAQ (see "Other" below)
Changes in the core
-------------------
- We have one primary self address and any number of secondary self addresses. `is_self_addr()` checks all of them.
- If the user does a reconfigure and changes the email address, the previous address is added as a secondary self address.
- don't forget to deduplicate secondary self addresses in case the user switches back and forth between addresses).
- The key stays the same.
- No changes for 1:1 chats, there simply is a new one
- When we send a message to a group, and the primary address is not a member of a group, but a secondary address is:
Add Chat-Group-Member-Removed=<old address> and Chat-Group-Member-Added=<new address> headers to this message
- On the receiving side, make sure that we accept this (even in verified groups) if the message is signed and the key stayed the same
Other
-----
- The user is responsible that messages to the old address arrive at the new address, for example by configuring the old provider to forward all emails to the new one.
@@ -58,12 +58,13 @@ end-to-end tests that require accounts on real e-mail servers.
running "live" tests with temporary accounts
---------------------------------------------
If you want to run live functional tests you can set ``DCC_NEW_TMP_EMAIL``::
If you want to run live functional tests you can set ``DCC_NEW_TMP_EMAIL`` to a URL that creates e-mail accounts. Most developers use https://testrun.org URLS created and managed by [mailadm](https://mailadm.readthedocs.io/en/latest/).
Please feel free to contact us through a github issue or by e-mail and we'll send you a URL that you can then use for functional tests like this:
With this, pytest runs create ephemeral e-mail accounts on the http://testrun.org server.
These accounts exists for one 1hour and then are removed completely.
export DCC_NEW_TMP_EMAIL=<URL you got from us>
With this account-creation setting, pytest runs create ephemeral e-mail accounts on the http://testrun.org server. These accounts exists only for one hour and then are removed completely.
One hour is enough to invoke pytest and run all offline and online tests:
pytest
@@ -142,7 +143,7 @@ This docker image can be used to run tests and build Python wheels for all inter
$ docker run -e DCC_NEW_TMP_EMAIL \
--rm -it -v \$(pwd):/mnt -w /mnt \
deltachat/coredeps ci_scripts/run_all.sh
deltachat/coredeps scripts/run_all.sh
Optionally build your own docker image
@@ -151,9 +152,9 @@ Optionally build your own docker image
If you want to build your own custom docker image you can do this::
$ cd deltachat-core # cd to deltachat-core checkout directory
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
