`indexmap` is a large dependency (4K SLoC) containing `unsafe` code.
Contact IDs are now passed around as a Vec<u32> or &[u32].
QUOTA roots are now sorted by name instead of perserving original order.
It was there since the C core, labeled with "/* last_seen is for
future use */" but never actually used. The comment was lost during
the translation from C to Rust.
`cutil.from_dc_charpointer()` is guaranteed to return `str`, while
`cutil.from_optional_dc_charpointer()` may return `None` if C function
returns `NULL`.
It was only used in two places. Rather than adding `use
pretty_assertions::*` everywhere, it's easier to remove, and it
removes two additional dependency crates.
`deltachat.const` module now defines `__getattr__` and `__dir__` as
suggested by https://www.python.org/dev/peps/pep-0562/
mypy detects that `__getattr__` is defined and does not show errors
for `DC_*` constants which cannot be detected statically.
mypy is added to `tox.ini`, so type check can be run with `tox -e mypy`.
Generally we could also just remove the device message as we have the
connectivity view, OTOH if you are not using DC a lot, it may be useful
to be notified without opening DC.
And apart from this one bug it's working fine.
`apply_group_changes` is executed regardless of whether the group is
created via `create_or_lookup_group` or found via
`lookup_chat_by_reply`. This change removes the need for
`lookup_chat_by_reply` to return `None` when group ID exists in the
database to let `create_or_lookup_group` run. As a side effect of this
Delta Chat replies to ad hoc groups are now correctly assigned to
chats when there are multiple groups with the same group ID, such as
ad hoc groups with the same member lists.
* test one-to-one chats on setup-contact/secure-join
only one chat is created after scanning a QR code:
- on setup-contact, one-to-ones are created on both sided
- on secure-join, the joined group chat is created;
one-to-ones are not created intitally,
but should become visible on receiving messages
* make sure, Alice creates the chat with Bob on setup-contact
not totally sure if that change in #2508 was on-purpose,
however, all yet released versions
did create the one-to-one chat also on the Inviter's (Alice) side,
so, let's stay with that,
i do not see many reasons to change that.
* unblock hidden (Blocked::Yes) one-to-one chats
one-to-one chats may be hidden by secure-join,
in case someone later writes a message to it
(not unlikely), the chat needs to be shown.
before, messages are just not shown,
the corresponding chat did not appear.
the 'Blocked' wording of a 'Chat' must not be mixed with the
'Blocking' of a contact. 'Chat-Blocking' is mostly a visibility thing,
that may change as messages come in.
this change should not affect _really_ blocked contacts -
they are filtered out already before
and their messages are usually not even downloaded.
also, before allow_creation is checked,
that may disallow chat creation for show_emails reasons.
all in all, it just does the same
as if the user has manualy deleted the chat before and it would be created.
* simplify test
utcfromtimestamp() is not recommended by the official documentation,
because many methods, including timestamp(), work incorrectly with
"naive" datetimes returned by utcfromtimestamp().
Update gossiped_timestamp when someone else sends autocrypt gossip in
the group, so we postpone sending gossip again ourselves.
- Warn about failures to parse Autocrypt-Gossip header
- Move gossip-related methods into ChatId impl
- Fix a "gossi_pp_ed" typo
- Make `state` variable immutable.
- Don't mark contact requests as fresh when fetching existing messages: this is not needed since there is no contact request chat anymore.
- Don't mark messages from blocked chats as noticed, this is not needed since messages go to blocked chat anyway instead of contact request chat.
* refactor: cleanup send_handshake_msg()
- rename to send_alice_handshake_msg() as used by Alice only
- remove dead code from Bob
(Bob's code is at BobState::send_handshake_message() since some time)
- take a contact_id and not a chat_id;
this makes things less confusing when
info-messages are put to the final group chat
* always directly return chat-id from dc_join_securejoin()
* take care not to create a group twice
* adapt documentation
* add info-msg on group invites; add inviter directly after creation
* document existing 'joinqr' command in repl tool
* do not create empty one-to-one chats for group-joins
* refactor: cleanup fingerprint_equals_sender()
- the function takes a contact_id directly now.
before it consumes the first contact of a one-to-one chat -
which may be easily confused with the group-chat in creation.
moreover, the conversion contact_id -> chat_id -> contact_id
is unneeded overhead.
* show info-messages in destination chat for alice
* fingerprint_equals_sender() returns Err on database failure
* tweak documentation
* clarify what an 'unfinished tasks' task is.
* add regression test for create_for_contact_with_blocked()
* rename Blocked::Manually to better fitting Blocked::Yes
* tweak test_secure_join() and make sure, Alice and Bob have only on chat after a group-join
* add basic multi-device-sync functions
* generate json
* add context.parse_sync_items()
* add context.execute_sync_items()
* piggyback sync-commands message, add body for human-readable part
* avoid double json renderings
* mimeparser parses incoming .json sync-files
* do not piggyback sync-files
* execute sync items
* return status of send_sync_msg()
* send sync messages as multipart/report
* add a per-item-timestamp and also allow adding other per-item-fields in the future
* if the self-chat does not exist, create it blocked/hidden
* create tokens closer to real qr-code needs
* respect bcc_self setting, add test for that
* sync qr code tokens after promoting groups
* send sync-messages only if an experimental switch is set
* trigger send_sync_msg() after sending messages and after creating/redraw/revive qr-code
* add DC_STR_* constants to deltachat.h
* adapt to refactored qr module as of #2729
* tweak test
* use SendSyncMsgs config name instead of SendExperimentalSyncMsgs - we can remove or rename the config nevertheless, but have the option to keep it without renaming
* tweak docs
* remove currently unused effective timestamp calculation
* clarify when send_sync_msg() is called
* make sure, sync-messages are encrypted and are sent by SELF
* tweak docs, fix typos
Before this PR, when a test failed, we often got:
```
thread panicked while processing panic. aborting.
error: test failed, to rerun pass '-p deltachat --lib'
Caused by:
process didn't exit successfully: `/home/user/deltachat-android/jni/deltachat-core-rust/target/debug/deps/deltachat-33648fc4aaad608c 'contact::tests::test_selfavatar_changed_event' --nocapture` (signal: 4, SIGILL: illegal instruction)
```
instead of the error message and stack trace.
This PR fixes this.
* Add DC_EVENT_SELFAVATAR_CHANGED
* Add test.
Unfortunately I can't easily also test that the avatar is not copied
from unencrypted messages:
In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.
* Unfortunately I can't easily also test that the avatar is not copied from unencrypted messages:
In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.
* add let's encrypt certificate missing on some older android devices
* create Certificate with Lazy::new()
* document certificate source
* use smaller *.der format instead of *.pem
Reduce the number of mutable variables and out parameters in
`add_parts`.
Also don't call `dc_create_smeared_timestamp` if there is no `Date`
header. Timestamps are only supposed to be "created" when the message
is sent, not received, to make sure sent messages are sorted properly
in MUAs that only use the date for sorting. Delta Chat uses database
IDs for sorting in addition to timestamps, so it can sort messages
with equal timestamps properly.
Update `dc_smeared_time` documentation.
Turn `dc_smeared_time` and `dc_create_smeared_timestamp` comments into
documentation comments.
The problem was:
When opening the connectivity view while there is no network,
get_connectivity_html() calls schedule_quota_update(), which schedules a
UpdateRecentQuota job. But in update_recent_quota(), connecting fails
and a ConnectivityChanged event is emitted (connectivity changes from
Error to Connecting and back). Therefore the UI calls
get_connectivity_html() again, and the loop is complete.
This made the UI completely unresponsible. To reproduce, just turn wi-fi
off and open the connectivity view.
The fix is:
schedule_quota_update() now only schedules a new job if there is no old
job. To prevent the possible (though probably unlikely) problem that an
old quota update job has a backoff of, like, a day and therefore quota
is not updated, I reduced the backoff time for quota jobs to 10s.
Fixes possibly https://github.com/deltachat/deltachat-android/issues/2043, but we should "re-try" this
* better readable enum
* add a failing test
* let new secure-joins abort existing ones
before, a stale secure-join or setup-contact
made the whole qr-scanning unusable until
the app is restarted,
resulting in "QR process failed" errors.
this commit fixes the issue by
aborting existing scans -
in cases, a user really wants two concurrect joins running,
this is not perfect, but that did not worked before as well.
* remove unused AlreadyRunning variant
* make clippy happy
* add Chattype::Broadcast and create_broadcast_list()
* do not disclose recipients for broadcasts
* allow sending/add-/remove-member for broadcast
* set broadcast subject same as for one-to-one chats
* broadcast-recipient-list does not include SELF
* use special icon for broadcast groups
* generate initial broadcast names
* make clippy happy
* send BCC message unencrypted to avoid unexpected disclosing; encryption is opportunistic anyway. if we have 'protected chats' at some point, we can think that over.
* reword 'To:'-group
* simplify can-send-check
* add broadcast tests
* tweak comments
* Update deltachat-ffi/deltachat.h
Co-authored-by: Hocuri <hocuri@gmx.de>
* change name of can_edit() to is_self_in_chat()
Co-authored-by: Hocuri <hocuri@gmx.de>
this is needed for targeting "non-blocking group QR joins"
as create_multiuser_record() would also be needed from other places.
this will make rebasing/rewriting and finally reviewing #2508 easier.
in the past, group-messages were marked as "read by recipient"
only when at least 50% of the group members have send a read receipt -
in practise, this does happen never or much too late esp. in larger groups.
setting the state OutMdnRcvd already on the first read-receipt
seems to be much more intuitive and useful as you at least know
one person has read the message.
this is also what other messengers as telegram are doing here.
moreover, this fixes a bug that did not add all read-receipts
to the "Info" screen - once "enough" read-receipts were received,
and the state was already set to OutMdnRcvd, subsequent read-receipts
were ignored.
probably because the "Info" screen did not show the read-receipts since forever -
and for the second tick, the addutional read-receipts are not needed.
* let quota-warning reappear after import
an import removes all device-messages,
including the quota warning.
resetting `Config::QuotaExceeding` makes sure,
the warning reappears soon after import -
otherwise the warning would reappear only after
storage is cleaned up and exceeds again.
* a second quota warning when 95% storage exceeded
* factor out warning-check and add a test for that
* add missing stock strings for connectivity and quota
* reword quota error
* use 'Incoming/Outgoing Messages' instead of 'Inbox/Outbox'
* just say 'Not supported by your provider.' if quota cannot be read.
- the context is already given by the headline
'Storage on domain.org'
- the string is short and does not disturb much
(it is a very common error)
- the string does not mention 'quota' as such,
which will be hard to translate ('Kontingentsinformationen') -
even if ppl as we know about the Quota extensions an such things.
there was also the idea to hide the whole section,
however, that is confusing in a multi-device-usage
when things are sometimes shown and sometimes not.
before, MsgsChanged(chat_id, new_msg_id) was emitted,
but that does not cover the deleted message.
in theory, we could emit both,
however, that would just be a waste of refresh in uis.
also before, events were used this way,
however, also the documentations are updated to
reflect reality better.
Implement get_previous_message() that only looks for the last
Message-ID in the References: header instead of using
get_parent_message() which falls back to using other Message-IDs from
References and In-Reply-To field.
* add a test for xing mailinglists
* strip long hash-prefixes from mailinglist name if we got the name from the List-Id as a last resort
* add a test for newsletter@ mailinglists
these mailinglists have the list-name in `From:`
and can be detected by addresses starting with `newsletter@`.
* if not list-name is set, use the `From:` name for addresses starting with `newsletter@`
this is similar to what we do with `notifications@`
* Update src/dc_receive_imf.rs
Co-authored-by: Hocuri <hocuri@gmx.de>
* add an example to the regex
Co-authored-by: Hocuri <hocuri@gmx.de>
* draft a download-api
* basic implementation
* allow partial downloads for protected chats
* use a separate column for download_state
* force a minimal timeout for delete_server_after in combination with partial messages
* add a warning if a possible download may expire by delete_server_after
* test load_imap_deletion_msgid()
* add a test for a partial download
* improve documentation and visibility
* let get_download_limit() return Result<Option>
* rusty getters
* apply MIN_DELETE_SERVER_AFTER to shown availability time
* move stub-creation to download.rs, use stock-strings, nicer logging
* make clippy happy (cargo clippy --tests)
* refine tests and comments
* fix typo
* remove superfluous closure in ffi
* respect partial_download for immediately scheduled DeleteMsgOnImap jobs
Integer overflows crash the application by default.
On a first sight this is only a potential crash that can't be
triggered, because timestamps are stored as i64 and ephemeral timer
duration is u32.
* add a test for .xt.local mailinglists
* get correct names of .xt.local mailinglists
these mailinglist probably come from the xt:Commerce system
and are pretty widely used.
i have not seen an .xt.local mailinglist with name set in List-Id,
however, if that happens, it will still be taken.
only if the name is unset,
we use the name from the From-header.
summary::Summary replaces Lot in the Rust API for methods returning
chatlist summaries. Lot is a legacy type for C API compatibility, so
Summary can be converted into Lot.
There is a real risk of an active attack when connecting to non-.onion
servers over Tor, as bad Tor exit nodes are cheap to set up.
It's probably not needed for .onion domains, but we don't make an
exception for now.
* check update timestamps for signatures, user-avatars, ephemeral-settings, last-subject
* check update timestamp for group-avatars
* check update timestamp for group-names
* check update timestamp for memberlist
* check update timestamp for protection-settings
* add a more advanced test
* add another more advanced test
* set last-subject-timestamp more carefully
* bubble up errros from set_*timestamp() and check for from_id==0 before
* simplify Params::set_i64()
* remove comment that is more confusing than helpful
* use update_timestamp() wording consistently
Also make Accounts uncloneable. It is still possible to derive Clone,
but does not make sense to do so, as .clone() creates two separate
account managers which use the same files but different unsynchronized
in-memory data structures.
Debian deprecated `which` in `debianutils` in favor of `command`.
`which` outputs this to stderr now:
/usr/bin/which: this version of `which' is deprecated; use `command -v' in scripts instead.
* resultify update_recent_quota()
* add a device-message if quota exceeds QUOTA_WARN_THRESHOLD_PERCENTAGE
* check if a quota warning should be added during housekeeping, this is at least once a day
* dc_get_config("quota_exceeding") is useful for bots
* make clippy happy
* reword QuotaExceedingMsgBody message
* avoid lots of warnings if quota jitters around the warning threshold
* allow clippy::assertions_on_constants
these constants depend on each other, it makes sense to check that they are not changed in an incompatible way
This adds following settings:
- Socks5Enabled
- Socks5Host
- Socks5Port
- Socks5User
- Socks5Password
Currently http requests and dns requests are not getting executed as they currently can't get tunneled through socks5 proxy. Therefore gmail with oauth2 wont work through tor.
- Use the same limit for info: full text can be read in HTML anyway.
- Remove DC_MAX_GET_{TEXT,INFO}_LEN constants from deltachat.h
- Fix a typo: s/DC_ELLIPSE/DC_ELLIPSIS/
- Do not truncate the text when loading from the database.
- Update the documentation: limit is in Rust chars, not bytes
the 'device chat about' was shown as a single message
as at that time, there was just not 'status'.
meanwhile, we have the status option,
and it feels much more natural to get the information there,
esp. as the subtitle on all UIs already read
'Messages in this chat are generated locally' -
and a tap on that will show the hint, without scrolling or so.
this also teaches the user where to find such information -
and the "welcome" chat is less spammy and really starts with the text
"Welcome to Delta Chat!"
* allow dotless domains and hostnames
* update provider database
ran ./src/provider/update.py ../provider-db/_providers/ > src/provider/data.rs
to pull in recent changes from https://github.com/deltachat/provider-db
* add imap::get_quota_roots()
* schedule quote-checking job on getting connectivity-html
* get quota and debug print it
* basic quota output
* update quota at most once per minute, emit event on changes
* use more meaningful names
* add some comments, move update_recent_quota() to quota.rs
* show root name only if there are several roots
* make clippy happy, some refactorings
* allow only one update-quota job per time
* add now supported QUOTA to standards.md
This bug sometimes results in sending read receipts to self in
multi-device setups.
It happens consistently in a setup where the first device is
configured to move messages to DeltaChat folder and the second device
is not. When both devices receive BCC-self message simultaneously, the
first device moves the message to DeltaChat folder, while the second
device tries to mark the message as seen in the Inbox. Regardless of
whether the second device marks the message as seen successfully or
fails because the message is already moved by the first device,
`Job.markseen_msg_on_imap()` sends the read receipt to the From
address.
Recognizing these emails as forwarded would probably be too complicated and require too much special-casing, but now the user can access the email text via "Show full message".
fix#2599
Co-authored-by: B. Petersen <r10s@b44t.com>
* In dc_maybe_network_lost() directly set the connectivity "Not connected"
r10s reported that without doing this, the connectivity would stay at
"Connected" for 16 more seconds after network is gone and
dc_maybe_network_lost() was called.
* Set the state for all connections
fix#2539
It's always a bit ambiguous which function should do set_err or set last_send_error, I used this rule here:
If some code returns Status::RetryLater, then it sets last_send_error, because Status::RetryLater means that the user won't see the error directly on the message (if we returned Status::Failed(Err(_)), then the message would be shown as failed to the user) Also, smtp_send always sets last_send_error because, well, sending just failed or succeeded.
Also, remove unused field pending_error.
Previously system messages were always added to the end of the chat,
even if the message triggering them was sent earlier. This is
especially important for messages about disappearing timer reset
triggered by classic email messages, as they should be placed right
after the message resetting the timer.
- Use latest stable for rustfmt and clippy.
- Update Rust in rust-toolchain and coredeps Docker container to 1.54.0
and test against it.
- Test against 1.48.0 to ensure libdeltachat can be compiled with Debian
bullseye "rustc" and "cargo".
Gmail labels are not folders and should be skipped. For example, emails
appear in the inbox and under "All Mail" as soon as it is received. The
code used to wrongly conclude that the email had already been moved and
left it in the inbox.
the WAL files come from sqlite WAL mode,
they have the same name as the database file with an extra `-wal` suffix.
according to https://sqlite.org/wal.html#the_wal_file ,
the WAL file is part of the persistent state of the database
and should be kept with the database if the database is copied or moved.
"Archived Chats" also contain old contact requests now
(an maybe new one if the user archives them),
so, there may be easily some thousands chats in "Archived Chats" -
and the count has no real meaning to the user,
esp. as we not even display the number of "Real Chats".
Might be that this "Link" will go away anyway sooner or later,
however, for now, it is just fine to remove the badge counter.
* add a test for archived requests
* fix archived requests
* move requests but the last one to "Archived Chats"
this way, the app looks familiar after the contact request upgrade.
the subselect was copied from the old get_last_deaddrop_fresh_msg()
(which was removed by the contact request upgrade #2514)
* just move all old requests to "Archived Chats"
ux-wise, the advantage of keeping the last one is questionable,
one may think, always the last one is shown in chatlist.
showing _all_ fresh request is not doable
as past cores did not really take care of that
and the db-state is not consistent in that regard.
that would make the already complicated code even more complicated,
so we decided to go the easy way.
* accounts: update EventEmitter on add_account
* accounts: do not lock waiting for EventEmitter in dc_accounts_add_account
Otherwise dc_accounts_add_account blocks until an event arrives on
some existing account.
Contact request chats are not merged into a single virtual "deaddrop"
chat anymore. Instead, they are shown in the chatlist the same way as
other chats, but sending of messages to them is not allowed and MDNs
are not sent automatically until the chat is "accepted" by the user.
New API:
- dc_chat_is_contact_request(): returns true if chat is a contact
request. In this case option to accept and block the chat via
dc_accept_chat() and dc_block_chat() should be shown in the UI.
- dc_accept_chat(): accept contact request and unblock the chat
- dc_block_chat(): decline contact request and block the chat
Removed API:
- dc_create_chat_by_msg_id(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_marknoticed_contact(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_decide_on_contact_request(): this call requires a message ID from
deaddrop chat as input. As deaddrop chat is removed, this call can't
be used anymore.
- dc_msg_get_real_chat_id(): use dc_msg_get_chat_id() instead, the
only difference between these calls was in handling of deaddrop chat
- removed DC_CHAT_ID_DEADDROP and DC_STR_DEADDROP constants
* escape strings added to html
* use more common emojis for connectivity report
all emojis are from 2010 and older now.
an alternative would have been to use css,
however, that may have other issues
and as the whole report is subject to change anyway,
i go for the easy solution.
* use 'modern' meta pattern, remove unused div and styles
* use css instead emojis; looks better that way
also, we have the same look on all systems.
* add connectivity command to repl tool
fix https://github.com/deltachat/deltachat-core-rust/issues/2463fix#2116
The email could be private (i.e. only sent to me) or non-private (i.e. also sent to sb else). Also, it could be a classical email or a chat message. Below, I'm describing for each of the four combinations whether they should be assigned to a chat by create_or_lookup_group() or lookup_chat_by_reply(). Because I needed to use these function names a lot, I shortened them to l:group() and l:by_reply() in this PR description.
(!) means todo, (! -> Done) means I fixed something.
## Private classical email:
l:group() and l:by_reply() must both take care not to put it into group
l:group() no (! -> Done), l:by_reply no (! -> Done)
except for classical MUA replies to two-member-groups, they should be put into the parent group
### wrt alias-support:
A private classical email is very probably not going to be an answer to email that went to an alias address:
Suppose Alice writes to support@example.com.
support@example.com forwards to bob@example.com and claire@example.com.
When Bob answers, he will _probably_ answer `To: alice@example.com, support@example.com` (=> it's a non-private classical email).
With this PR, if he does only answer `To: alice@example.com`, (=> it's a private classical email), Alice's DC will show the answer in the private chat with Bob. Which actually makes more sense than showing it in the support@example.com chat I think. Also, if it was shown in the support@example.com chat, then Alice would answer in the support@example.com chat, then Claire would get Alice's message but not Bob's and therefore miss some context.
That being said - **I could change this**. Pretty easily actually, I would just have to remove the call to `is_probably_private_reply()` from `lookup_chat_by_reply()`. Didn't think through all edge cases yet, but should work.
## Private chat message:
l:group() has to put private chat messages into the group: It won't mistakenly put a message there (because it can rely on Chat-Group-Id). Currently, `is_probably_private_reply()` returns true for private chat messages, but l:group still has to put them into the group chat. (it's not nice that the function called is_probably_private_reply returns true for all private chat messages, but I didn't find any nicer solution) l:by_reply() must not assign it to sth special
l:group yes, l:by_reply no
By the way, for chat messages, `try_getting_grpid()` doesn't look at InReplyTo or References, in order not to put private chat-message replies into the group chat.
For alias-support, the same goes as for the private classical emails.
## Non-private classical email:
Just put it into any group, and if there is none, create one.
_Off-topic:
We currently don't look at the recipients lists, which means that a message can easily be assigned to a group although it was not sent to all group members. One day we could somehow compare the recipients list with the members list, but that needs some more discussing (esp. what do we do if they don't match? Create a new group? Show a hint in the UI?) and is nothing for a bug-fixing PR like this one._
l_group yes, l_by_reply yes, even for outgoing messages (! -> Done, this also was the issue reported by @gerryfrancis in the testing group:
Outgoing messages were not put into a chat by In-Reply-To/References, which is correct for chat messages, but for non-private classical outgoing emails, a new ad-hoc group was created everytime.
So, I added this: https://github.com/deltachat/deltachat-core-rust/pull/2465/files#diff-e7606b521f6710ddc6e5236ba5d7eefc917b7ad744b9e71762fd42830c55485bR703-R711)
## Non-private chat message:
Can be put into chat by l:by_reply() because it must be a group message
l:group yes, l:by_reply yes (to make alias support work if the support person uses DC)
Nothing to test or fix here; we have to put chat group messages into the group, which is trivial. And we have to make sure that alias-support works, which already was well tested.
* changelog: update to include prevous changes
and use new format/structure
* changelog: capitalize API
* changelog: add a missing pr refernece
* Apply suggestions from code review
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: Hocuri <hocuri@gmx.de>
See https://support.delta.chat/t/discussion-how-to-show-error-states/1363/10 <!-- comment -->
It turns out that it's pretty easy to distinguish between lots of states (currently Error/NotConnected, Connecting…, Getting new messages… and Connected). What's not that easy is distinguishing between an actual error and no network, because if the server just doesn't respond, it could mean that we don't have network or that we are trying ipv6, but only ipv4 works.
**WRT debouncing:**
Sending of EVENT_CONNECTIVITY_CHANGED is not debounced, but emitted every time one of the 3 threads (Inbox, Mvbox and Sentbox) has a network error, starts fetching data, or is done fetching data.
This means that it is emitted:
- 9 times when dc_maybe_network() is called or we get network connection
- 12 times when we lose network connection
Some measurements: dc_get_connectivity() takes a little more than 1ms (in my measurements back in March), dc_get_connectivity_html() takes 10-20ms. This means that it's no immmediate problem to call them very often, might increase battery drain though. For the UI it may be a lot of work to update the title everytime; at least Android is smart enough to update the title only once.
Possible problems (we don't have to worry about them now I think):
- Due to the scan_folders feature, if the user has lots of folders, the state could be "Connecting..." for quite a long time, generally DC seemed a little unresponsive to me because it took so long for "Connecting..." to go away. Telegram has a state "Updating..." that sometimes comes after "Connecting...".
To be done in other PRs:
- Better handle the case that the password was changed on the server and authenticating fails, see https://github.com/deltachat/deltachat-core-rust/issues/1923 and https://github.com/deltachat/deltachat-core-rust/issues/1768
- maybe event debouncing (except for "Connected" connectivity events)
fix https://github.com/deltachat/deltachat-android/issues/1760
in most (all?) UIs, import/export works on an already created account,
so, dc_accounts_import_account() does not really help here -
but adds some noise and confusion
eg. as for the other dc_accounts_t functions,
the corrsponding dc_context_t functions must not be called.
if really a new account is required for import,
it seems to be easier to call add_account() before import.
Do not process names, avatars, location XMLs, message signature
etc. for duplicate messages.
Previously only `add_parts` was stopped early, but not
`dc_receive_imf`. Also, `dc_receive_imf` processed From: and To:
fields and applied names to contacts even before checking the
Message-ID.
Fake Message-ID generation procedure is changed to operate on raw
header values to avoid interacting with the database.
as by reviving qr codes,
there may be more than one token for a chat,
ensure, the most recent token and only one token is returned
by the sql-command for looking up tokens
(used for generating new codes)
Raw MIME messages may contain non-ASCII characters. Attempting to
store them as TEXT by using String::from_utf8_lossy results in
non-ASCII characters being replaced with Unicode U+FFFD "REPLACEMENT
CHARACTER" which is later incorrectly decoded when attempting to parse
`mime_headers` content into HTML.
Normally they should not end up in contact requests, but if they do,
we want to show them. Otherwise it is completely impossible to see
them until the chat is moved out of contact requests.
Previously chats created by outgoing classic emails went into contact
requests (deaddrop). Outgoing messages are not shown in contact
requests, so created chat was not shown anywhere. With this fix chat
is created both for outgoing classic emails and outgoing chat emails.
Since classical messages are not deleted when show_emails setting is
set to "0" and remain in the database, they should be shown
somewhere. Otherwise they may reappear later when the setting is
enabled again.
New `dc_msg_is_bot()` C API and corresponding `Message.is_bot()`
Python API can be used to check if incoming message is sent by a bot,
e.g. to avoid two echo bots replying indefinitely to each other.
"Bot" flag is not set for outgoing messages, but may be set for
BCC-self messages. For now documentation says that `dc_msg_is_bot()`
return value is unspecified for outgoing messages. It can be better
specified later if needed for specific applications, e.g. sharing an
account with a helper bot.
the default account created by dc_accounts_new()
is annoying when doing a migrate/import account;
in this case, you must not forget to remove the default account.
in between you have two account,
and accounts are swiched.
it seems to be better
to just call add_account() on a new account object as needed.
systemli.org seems not to include Received: header in messages sent to
self, including BCC-self and messages sent to Saved message chat. This
results in all these messages not being displayed on other devices in
multi-device setting. To work around this problem we restrict draft
detection based on Received: header to classical mails.
Delta Chat already implements proper draft detection based on the
\Draft message flag and \Drafts folder flag. However, Thunderbird does
not set \Draft flag when message is stored as "Template". To make
draft detection for Thunderbird more robust, in case Received: header
check is removed altogether later, we check for X-Mozilla-Draft-Info
header which Thunderbird sets both for "Drafts" and "Templates".
probably they come in by the latest sqlx<->rusqlite moves,
however, as they are followed by subsequent migrations,
that should not have been a big bug in the past
(maybe unless the app was killed at a bad moment)
Even if chat has no messages, contacts should not be deleted.
Otherwise chat will reference invalid contact ID which can't be loaded
from the database, resulting in broken contact list in group chats.
`Imap` structure is always created in a configured state now. There is
no default value for `ImapConfig` anymore.
Also resultify Scheduler::start() to fail on database errors, for
example if IMAP configuration cannot be read from the database during
`start_io()`. Previosuly errors during reading keys such as
`mvbox_watch` were simply ignored and folders were not watched until
the application is completely restarted, now start_io() will fail and
scheduler will only be started at the next start_io() call which
usually happens when app is brought to the foreground.
* Add failing test for #2369
* Completely ignore Drafts folder
fix#2369
* Also ignore messages that have the Draft flag set but are not in the Drafts folder
Building aarch64 wheels under qemu takes a long time.
rust 1.52.1 includes fix for cargo that may result in
compilation being stuck: https://github.com/rust-lang/cargo/pull/9201
There was a bugreport for previous versions of DC that BCC-self
messages are not marked as seen if server-side Sieve rule moves them.
This test is an unsuccessful attempt to reproduce the bug,
which was confirmed to be fixed.
This test checks that MDN is marked as seen on ac1. Because ac1 also
receives BCC-self, it is possible that event of marking this message
as seen is confused with marking MDN as seen, and the test passes even
if MDN is not marked as seen. Explicitly disabling BCC-self for ac1
ensures it receives only one message (MDN).
Also start direct_imap as early as possible, before sending the
message. It is possible now, because BCC-self is not sent.
When watch_inbox was off, scan_folders failed and a toast "IMAP operation attempted while it is torn down" was shown.
--
The problem was:
When inbox_watch is off, scan_folders() is called at 244260a978/src/scheduler.rs (L107) but connect_configured() is not called before.
* Add test
* Don't only setup handle, but connect_configured() in scan_folders()
This allows to run `tox `on systems with only a single Python 3
version installed, such as Python 3.9.
CI script scripts/run-python-test.sh specifies `-e py37` explicitly
anyway.
The problem was:
If a message has the \Deleted flag set, we ignore it. But we forgot to
update last_uid, so that uid_next was not updated at 47e639b777/src/imap.rs (L730) and the same message was
fetched over and over again.
Fix#2429
Using `impl AsRef<str>` as the argument instead of `&str` makes it
possible to call the function with `&str`, `String` and other types
that implement `AsRef` trait.
The cost of it is that compiled binary contains mulitple versions of
the same function, one for each variant of types. If function contains
multiple generic `impl AsRef` arguments, the number of versions possibly
compiled into binary grows exponentially with the number of arguments.
Simple way to avoid it is to call `.as_ref()` on the caller side to
convert the argument to `&str`. In most cases even adding a `&` and
relying on `Deref` coercion is sufficient.
This patch changes many functions that accepted `impl AsRef<str>` and
`impl AsRef<Path>` to accept `&str` and `&Path` instead.
In some places `.clone()` calls are removed. Calling `.clone()` on
`String` and passing `String` to a function accepting `impl
AsRef<str>` is completely unnecessary as `&str` reference could be
passed instead. There is no clippy warning against it yet, but
changing argument type to `&str` allowed to find these cases.
The result of debloating is not impressive, several hundred kilobytes
are saved, which is about 3% of the `.so` binary, but the code is
cleaner too.
This moves the module-level lookup and creation functions to the
types, which make the naming more consistent. Now the lookup_* get_*
and create_* functions all behave similarly.
Peraps even more important the API of the lookup now allows
distinguishing failure from not found. This in turn is important to
be able to remove reliance on a ChatId with a 0 or "unset" value. The
locations where this ChatId(0) is still used is in database queries
which should be solved in an independed commit.
This introduces the explicit ChatIdBlocked struct to more explicitly
create a chat with a blocked status. It also adds a common shortcut
to ChatId itself which is more natural to use in many cases.
* Remove sql::error submodule
Use anyhow errors instead.
* Remove explicit checks for open SQL connection
An error will be thrown anyway during attempt to execute query.
* Don't use `with_conn()` and remove it
* Remove unused `with_conn_async`
* Resultify markseen_msgs
- Currently, group images are compressed as well because it was easier to implement that way.
- Currently, in the unlikely case that the avatar is compressed down to 20x20 pixels but still bigger than 20KB, the user doesn't get any indication of this, the avatar simply isn't changed (at least on Android).
If we want to change this, the easiest way is probably to let `dc_set_config()` in the ffi call `error!()` if `Selfavatar` can't be set. The same might make sense for some or all other configs. BUUUUUT: At least Android doesn't show error!() toasts anymore, probably because they were used too often and too spammy.
- The factor by which we scale down if the file is too big is 1.5.
Instead, allow specifying free-form BUILD_ID from the command line.
scripts/remote_python_packaging.sh still uses CIRCLE_ variables to avoid
changing working CircleCI config.
This feature is similar to existing avatar synchronization.
Whenever encrypted BCC-to-self copy of chat message is received, status
setting is updated with the signature of the message.
Rust does not work on manylinux2010 due to old GNU C Library.
We have been using manylinux2014 on CI machine already, but this change
was never commited.
messages are marked as 'noticed' already when the chat is opened
as all UIs call dc_marknoticed_chat();
also marking messages as 'noticed' when dc_markseen_msgs() is called
is not needed therefore - but stands in the way of further improvements
for the deaddrop, eg. UI may let the user decide when the deaddrop
can be removed from chatlist ('All done' button or so)
* Fix#2335 (delete job flooding)
The problem was:
- You are offline, and an ephemeral message is due to delete.
- load_imap_deletion_job() is called and returns a deletion job for it.
- The job fails because, well, we are offline.
- The job is saved into the database.
- load_imap_deletion_job() is called again, and so on.
* Add test
ct.name was insufficient, as authname, overridden name and email address
fallback were not taken into account.
Dropping this condition increases performance by 25% according to the
benchmark.
Also add a test for search_msgs.
It reduces the time by ~20% according to `search_msgs` benchmark.
Sorting by IDs is sufficient for global search as IDs increase in the
order of message reception.
This PR originally contained a fix for sqlx which turned out not not to be necessary. But on the way, I made some general improvements:
- Under some circumstances, a "normal" test failure led to a timeout, without printing a decent error message. See e.g. https://app.circleci.com/pipelines/github/deltachat/deltachat-core-rust/8069/workflows/ba2a9949-b4ad-4ceb-a930-073bba05e2db/jobs/30965.
(The problem was: if there is an exception in dc_account_extra_configure(), when trying to handle the exception the line account.log("===================", e, "===================") was called, which can't work as log() only expects one parameter)
- When a test fails: Call `dump_account_info()` even if there is no direct_imap
- In test_import_export_online_all(), add another 100KB file to the backup. This adds resilience against future size changes of the sql db file: The test tests the smoothness of the progress bar. And if there are there are not enough about-equally-sized files, the progress bar can't be smooth.
the user-given contact name may be set to an empty string;
in this case the authname or the email is used for the contact
and also for the name of possibly existing chats.
this works well for `dc_chat_get_name()` as that just uses
`dc_contact_get_display_name()` for single-chats.
it did not work for `dc_get_chatlist(query)` as that
uses the database for performance reasons -
however, in the database, the empty string is written
instead of the display name is written for a chat.
this is fixed by this pr by also using
dc_contact_get_display_name() when updating the chats-table
(similar to `dc_create_chat_by_contact_id()`)
message type directly (ex. `const.DC_MSG_STICKER`) so if new message types are
added, they can be used direcly without needing the python API to be updated.
This is an PGP/MIME format produced by Microsoft Exchange and ProtonMail IMAP/SMTP Bridge,
described in detail in https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html
This patch adds seamless support for "Mixed Up" Encryption, repairing
mangled Autocrypt messages without notifying the user.
&str queries are not persistent by default. To make queries persistent,
they have to be constructed with sqlx::query.
Upstream sqlx does not contain the change that make all queries
persistent, but it is not needed anymore. but
fix #2073fix#2292 (I think)
- Messages can be assigned to any chat by the References and In-Reply-To, also to 1:1 chats; this has higher priority than the group id because with ad-hoc groups, it can happen that two devices have different group ids for the same conversation thread.
- If `From` is not in the chat (we call this "shadow sender"), `OverrideSenderDisplayname` is set. This communicates to the UI that:
- A `~`should be added in front of the sender's displayname.
- Also in 1:1 chats, the sender's displayname and avatar is shown, as if this was a group.
The "Unknown sender for this chat" messages are completely removed for unprotected groups.
For protected chats, everything stays as it was before.
POSTPONED:
- Maybe (if it turns out to be still necessary):
- The ad-hoc group id is computed by the the References, instead of the member list, as it is currently done
- How do we prevent that the message can't be assigned to the correct chat as the parent message was deleted?
Switches from rusqlite to sqlx to have a fully async based interface
to sqlite.
Co-authored-by: B. Petersen <r10s@b44t.com>
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: link2xt <link2xt@testrun.org>
fix#2188
Also, if we notice that the server started reusing old UIDs, _also_ do a `ResyncFolders`, because the server likely forgot to change uid_validity
`DoxygenLayout.xml` is picked up by doxygen automatically (as `Doxyfile`)
and its template can be generated by `doxygen -l DoxygenLayout.xml`.
after that, the following tweaks were done:
- remove all xml-parts but header (other defaults are fine so far)
- rename "Modules" header to "Constants" (there are no modules :)
- remove useless subitems from "Classes", eg. by name only shows as "d"
- reorder tabs so that "Classes" becomes the first one
save subject for messages:
- new api `dc_msg_get_subject()`,
- when quoting, use the subject of the quoted message as the new subject, instead of the
last subject in the chat
usually, there is at most one text/html and one text/plain part.
multiple text/plain parts,
are known only by mailinglist software that wants to add a footer
but cannot alter the original content part -
maybe because it is html and things are hard to stitch
or maybe because the content part is cryptographically signed
and cannot be altered therefore.
the test has multiple text parts because
the content text is signed and
the mailinglist software wants to add some text
(and cannot alter signed part without breaking the signature)
fix#2254: if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message), the db was corrupted.
* Add result.log() for logging with less boilerplate code
* Bugfix: Resultify housekeeping() to make it abort if the db is closed instead of just deleting everything
* Require the UI to call dc_stop_io() before backup export
* Prepare a bit better for closed-db: Resultify get_uidvalidity and get_uid_next and let job::load_next() wait until the db is open
About the bug (before this PR):
if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message):
- I don't know if it downloads the message, but of course at some point the process of receiving the message will fail
- In my test, DC is just in the process of moving a message when the imex starts, but then can't delete the job or update the msg server_uid
- Then, when job::load_next() is called, no job can be loaded. That's why it calls `load_housekeeping_job()`. As `load_housekeeping_job()` can't load the time of the last housekeeping, it assumes we never ran housekeeping and returns a new Housekeeping job, which is immediately executed.
- housekeeping can't find any blobs referenced in the db and therefore deletes almost all blobs.
blocked mailinglists addresses are added to the contact table
before blocked contact list is created -
(this allows unblocking of blocked lists in previous testing releases,
however, more importantly, it keeps all blocking/unblocking code inside
contacts)
on unblocking such a contact,
the corresponding chat is unblocked as well.
* deprecate mostly unused dc_get_blocked_cnt() api
instead, the size returned by get_blocked_contacts() should be checked,
this is safer and allows easier adaption of blocking rules.
ui or python seems not to use dc_get_blocked_cnt(),
however, there is one test in node,
therefore, the function will continue working for now
(by just returning Contact::get_all_blocked().len() then)
* add decision api to repl tool
* add block/unblock api to repl tool
* unify usage of @deprecated doxygen command
outlooks SMTP-server change the Message-ID of messages
and put the original Message-ID to X-Microsoft-Original-Message-ID.
the changed Message-ID has some issues:
- outgoing messages with bcc_self enabled are shown twice
as the self-copy got the changed Message-ID while the database uses
the original one
- read receipts do not work as they refer to the changed message id
- in general, sender and recipient see different Message-IDs
the issues can be fixed by
(1) let all receivers use the original Message-ID
this is what this pr is doing
and this should fix all issues with delta-to-delta communication,
including groups, group-images etc.
there may be issues left in communication
with other MUAs as they are using another Message-ID.
(2) ftr: updating the Message-ID in the database of the sender to the new one
this requires bcc_self always enabled (which is not the case)
and may also result easily in race conditions
(Bob answers before Alice sees its self-sent message),
however, has the advantage of better compatibility with other MUA.
if needed, the compatibility with other MUA could be improved by remembering
both Messages-IDs, maybe we could treat the modified as References or so,
however, i think, this could be part of another PR if we know better about
real, in the wild issues.
The chat::lookup_by_contact_id call is already resultified, the
database can not contain this since the auto-increment counter is
bumped to 7 by the time the database tables are created.
* add test for mailchimp mailinglists
* pass MimeMessage to create_or_lookup_mailinglist() (as of the other create*() routines) to allow more flexible name processing; document the function
* get mailing list name for mailchimp from From:-header
* make clippy happy
* add comment to '.list-id.mcsv.net' suffix
Now I know why the tests failed before 48c58a7 (i.e. after c923670) but
not on master (i.e. before c923670):
because of a bug, scan_folders() set the configured_sentbox to None if
it was set before. If it was None before, it restored the correct value.
On master, there was another bug that led to two runs of
scan_folders() being started at the same time. Therefore, the first run
set configured_sentbox to None, the second one restored the correct
value.
c923670 fixed the latter bug, so that only one run of scan_folders() was
started. Therefore, configured_sentbox stayed None incorrectly and
test_fetch_existing() failed.
48c58a7 fixed the former bug.
This commit adds checks to test_fetch_existing(), so that it definitely
checks for the former bug.
This does fix a bug and it makes the tests pass, but I'm not sure why it
makes the tests pass; maybe there is a race condition that made the
tests fail and my commit just leads to another timing.
This will allow MimeFactory.render() to put protected headers
into main_part and wrap it into single-part multipart/mixed if
protected headers are too large to put into the outermost IMF.
This should make colors used by Delta Chat for emails similar to colors
used by XMPP clients implementing the same specification[1], such as
Conversations and Snikket.
Previously Delta Chat used only hardcoded 16 colors, so this change
should also increase the number of colors and make it easier to
distinguish different contacts.
[1] https://xmpp.org/extensions/xep-0392.html
This changes the internal stock strings API to be more strongly typed,
ensuring that the caller can not construct the stock string in the
wrong way.
The old approach left it to the callers to figure out how a stock
string should be created, now each stock string has their specific
arguments and callers can not make mistakes. In particular all the
subtleties and different ways of calling stock_system_msg() disappear.
This could not use a trait for stock strings, as this would not allow
having per-message typed arguments. So we needed a type per message
with a custom method, only by convention this method is .stock_str().
The type is a enum without variants to avoid allowing someone to
create the type.
Sadly the fallback string and substitutions are still far away from
each other, but it is now only one place which needs to know how to
construct the string instead of many.
1.51.0-alpha.0 is somewhere between 1.50 and 1.51,
as various beta-versions come or already came out with the new core,
this is helps in logs/debugging.
The following rules apply now:
1. "name" column is only updated manually and never over the network
2. "authname" column is only updated over the network and never manually
3. Displayname is a "name" if it is non-empty, otherwise it is "authname".
This fixes a known (pytest.xfail) problem of "name" being changed over
the network when user has set it to non-empty string manually.
This also fixes the problem when "name" and "authname"
became unsynchronized accidentally, when they were equal and then
Origin::IncomingUnknownTo update arrived, setting "name" but not
"authname". Rust regression test is added for this case.
renaming mailing lists and setting mailinglists images
(possible locally, but not synced)
was blocked because the function real_group_exists() checked
the chattype on a numerically.
it is not 100% clear to me,
why this function exist at all,
it just checks if a record with type=120 (group) exists under the given id -
this is the same as loading the chat and check chat.typ afterwards.
might be a premature optimisation relict from core-c
or made some error handling easier there.
This does not affect the release profile and it significantly speeds
up compilation. In the rare cases where panics are not sufficient to
debug something you can manually enable this again.
The get_chat_id_by_grpid() call is already resultified. The database
can only contain consistent data since on creation it inserts rows for
the special chats which do not have a grpid set so can never match.
Thus we can safely remove another use of the ChatId::is_unset call
which should be removed to clean up the type.
* Copypaste-merge my old work
* Start implementing mailinglists the new way
* Create pseudo contact
* Fine-tune docs
* Remove some unnecessary changes
* style
* Make a stock str
* Fix a crash. Yes, this line caused a panic when reconfiguring on Android
(without a reasonable error log). Also, I could not receive any messages
anymore.
* rfmt
* Add tests and make them pass
* Even more tests
* rfmt
* Enhance test and fix bug
* Don't update the sender name when prefetching because maybe it's a mailing list
* Use an enum instead of numbers for the decision
* Don't remove anyone from mailing lists
* Fix bug in the regex
* Adjust error msg
* Compile error after rebase
* Correctly emit event
* Add dc_msg_is_mailing_list so that you can find out whether messages in the deaddrop belong to mailing lists.
* Add received headers to unit tests
* Comments, small tweaks
* Use dc_msg_get_override_sender_name instead of dc_msg_get_sender_name
* Add dc_msg_get_sender_first_name() because sometimes the first name was not correctly shown in mailing lists
* small fixes, don't let the user modify mailing list groups
* Hide contacts for addresses like noreply@github.com and reply+AEJ...@reply.github.com
When testing mailing lists, I noticed that sometimes a mailing list
contact got a name (like, hocuri <noreply@github.com>). It turned out
that ages ago, I had accidentally written an email to - in this example
- hocuri <noreply@github.com> and it had been added to the contacts
list.
This hides email addresses from the contacts list that are obviously not
meant to be written at and prevents updating the names.
* Comment, clippy
* Replace u32 with ChatId
* Resolve lost of small issues from the reviews
* remove dc_msg_get_sender_first_name
* add dc_msg_get_real_chat_id()
this allows to check if a contact request belongs to a mailing list
and to show name/avatar/whatever of the mailinglist.
another approach was to duplicate some chat-apis for messages
(eg. dc_msg_is_mailing_list()) however that would require far more new apis.
the idea to change the behavior of dc_msg_get_chat_id() would be
more clean, however, that easily breaks existing implementations
and can led to hard to find bugs.
* remove now unused Message.is_mailing_list()
* if a name for a mailing list is missing, use List-ID
* fix comment
* fix error message
* document how dc_get_chat_contacts() works for mailing lists
* refine decide api (#2185)
* add DC_DECIDE* constants to deltachat.h, tweak documentation
* use StartChat/Block/NotNow instead of Yes/Never/NotNow
* decide_on_contact_request works on ctx/msg-id
functions working on message-objects
usually do not read or write directly on the database.
therefore, decide_on_contact_request()
should not work with message-objects as well,
it is even a bit misleading, as eg. chat-id of the object is not modified.
instead, the function works on context,
similar to dc_send_msg(), dc_create_chat() and so on.
for now, i moved it to context, could maybe be part od MsgId.
* Update src/chatlist.rs
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: Hocuri <hocuri@gmx.de>
* refine documentation
* re-add accidentally deleted Param::MailingList
* remove pseudo-contact in domain @mailing.list
1. the pseudo-contact was added to the mailing list contacts,
which is not needed.
might be that we want to add the recent contacts there in a subsequent pr
(we do not know all contacts of a mailing list)
2. the pseudo-contact was used to block
mailing lists; this is done by setting the chat to Blocked::Manually now
3- the pseudo-contact was used for unblocking;
as it is very neat not to require additional ui for mailing list unblocking,
might be that we introduce a similar pseudo-contact for this limited purpose
in a subsequent pr, however, the pseudo-contact needs to exist only
during unblocking then, maybe also the special domain is not needed,
we'll see :)
* Move dc_decide_on_contact_request() up to the dc_context section as it's a member of dc_context now
More specifically, to the "handle messages" section
* re-introduce Chattype::Mailinglist (#2195)
* re-introduce Chattype::Mailinglist
* exhaustive chattype-check in fetch_existing_msgs() and get_summary2()
* exhaustive chattype-check in ndn_maybe_add_info_msg()
* exhaustive chattype-check in message::fill()
* remove dc_chat_is_mailing_list() from ffi
Co-authored-by: B. Petersen <r10s@b44t.com>
some videochat-services do not support adding random rooms to the url,
users let people in manually, so they can always use the same url.
as hacks as hiding the genrated random room behind `#$ROOM` or `?=foo=$ROOM`
does not always work, this pr introduces the parameter `$NOROOM` that is
just replaced by the empty string.
see https://support.delta.chat/t/videochat-with-webex/1412
for further details.
Return error for special IDs. Previously "No encryption." was returned for DC_CONTACT_ID_SELF.
Show "No encryption." if peerstate is reset.
Never talk about transport encryption: it is misleading. Even if TLS is used for client IMAP and SMTP connections, MTA-MTA connection may be unencrypted if MX does not support STARTTLS.
Add a Rust test.
* add failing test for From:-header bug
the test fails on the last check for quote-encapsulated utf-8 strings.
* Update mailparse
Co-authored-by: link2xt <link2xt@testrun.org>
Param should only be used to parse dictionaries stored in SQL database, not external data.
Since Param parser has been extended to escape newlines with newlines, QR-codes parser started to treat && as escaped &, which is wrong.
This change also uses String keys like "n" for "name" instead of completely unrelated constants like Param::SetLongitude.
- doc fixes
- make BobStateHandle safer by moving the state out of the handle.
- handle more match cases explicit in BobState returns
- fewer mutable variables
This does not only organise things better, but most importantly the
BobStateHandle is now not in the same module as its users. This means
it can guarantee safety about how it is initialised and use
unreachable!() to simplify it's API.
This introduces a state machine which takes care of managing the
handshake transitions in the secure-join protocol. This separates
user interactions from the protocol state handling.
This means that while handling the protocol state there are a bunch of
failures no longer possible due to all state information being
guaranteed to be present. As part of this the QR-code state has been
extracted from the generic Lot structure to something suitable just
for the SecureJoin protocol.
A LogSink has been added to the testing tools allowing log messages to
be correctly displayed on test failures.
* draft dc_msg_set_html() api
* implement setting 'html to be send'
* test sending html-parts
* more flexible html-partbuilder
* write html-parts to database and also send them
* add 'sendhtml' command to repl tool
Encryption preference was already applied since commit 78d855c5ca,
but only for already existing peerstates.
As a result, new users ignored gossiped encryption preference in the
"member added" message and had to wait until someone gossiped encryption
preferences for inactive users the second time.
This should help if parent message can't be found because messages were
reordered or deleted.
This does not prevent group IDs from being removed from Message-IDs in
the future, in which case it will become dead code.
This removes the proxy via crate::error to depend on anyhow directly.
There is no benefit to this indirection and this makes it simpler to
see which error types are used.
* add failing tests for forwarding html-mails
* let MsgId.get_html() return an option
* write html-part to local database on forwarding
* add html-part to forwarded non-dc messages
* read HTML-parts from encrypted messages
* avoid clone()
* Received:-header is no longer needed since #2152
* Update src/html.rs
Co-authored-by: Floris Bruynooghe <flub@devork.be>
* Update src/html.rs
Co-authored-by: Floris Bruynooghe <flub@devork.be>
* Update src/mimeparser.rs
Co-authored-by: Floris Bruynooghe <flub@devork.be>
* prefer 'orig' over 'org' as abbreviation for 'original'
* improve comment on tests
* prefer 'try_into()' over 'as u32' to avoid panics on bad data
* simplify ffi
Co-authored-by: Floris Bruynooghe <flub@devork.be>
instead, dc_contact_get_display_name() should be used.
dc_contact_get_first_name() was created to save some space on the screen,
esp. on mobile devices,
however, this does not always work and has issues on its own
with some names ("Dr. Strangelove", ":) Name" and so on).
as with mailing lists, more apis with first_name() would be needed,
we decided to drop that instead of following that way.
it is also less an issue as some years ago as screens have become larger,
if really, needed, the ui can handle that more gracefully,
however, just using dc_contact_get_display_name() should be fine as well.
We don't really have docs in the other crates yet, and there's a
naming conflict between deltachat and deltachat_ffi that needs to
sorted to build both. But deltachat_ffi is still documented with
doxygen for now so there's no need to sort this now.
This also checks that some of the correct user interactions happen,
checking we get a joiner event and the verified chat messages.
It also extends the test utils with the ability to distinguish the
different context logs by having them named.
Lots of new clippy lints due to toolchain upgrade.
Made the Message::error field pub(crate) again, it was the odd one out
and it seemed a reasonable way to shut up clippy.
Notable improvements:
- [track_caller](https://github.com/rust-lang/rfcs/blob/master/text/2091-inline-semantic.md): We can get the location of the caller of a message, which makes logging easier as you can have dedicated functions (not only macros), e.g. write a function `log(self: Result, msg: string) -> Option`
- The output after a panic is shorter and hides lots of uninteresting
function calls
- Test output is better captured
After merging, we should probably wait a bit with using the track_caller
feature to make going back easier, should we get problems again.
It is misleading and should never be checked.
dc_stop_io() also stops pending ephemeral message deletion tasks, so it
should be called in any case before releasing context.
Start implementing #1994
TODO (in later PRs):
- Add a hint to the watch settings that all folders are fetched from time to time (to be done in the individual UIs)
- folder names are case-insensitive, so double-check that all comparisons are case-insensitive
- The `scan_folders.rs` file didn't get as large as I expected and it's probably not worth it having an extra file for it. But if there are no objections, I'll make another PR to rename it to `folders.rs` and also put into it `configure_folders()` from `imap/mod.rs` and `needs_move()` with all its tests from `message.rs`.
Done:
- Most mailboxes have a "Drafts" folder where constantly new emails appear but we don't actually want to show them, what do we do about this? The most reliable way to detect such messages that we found up to now is:
If there is no `Received` header AND it's not in the `ConfiguredSentbox`, then ignore the email.
- before or after INBOX idle trigger a new "scan all folders for messages". It does a "list folders" and then goes through all folders with select-statements, checking if "next-uid" was changed since checked last time. This might be batchable but in any case should not consume a lot of traffic. We might debounce this scan activity to happen at most every N minutes
- if next-uid changed for a folder, we "prefetch" and "fetch" DC-messages as as needed ("dc-messages" are not just those with "Chat-Version" headers, but can also be regular emails)
- if we discover DC-messages in folders that have the "/Spam" flag (maybe excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
- for now, we do not change any user visible option, but introduce this "scan all" automatically and on top of what exists. The DeltaChat folder-watching does not perform scan-all-folders (maybe with the exception to trigger scan-all also with DeltaChat if INBOX is not watched)
- Tests (except if you have ideas to improve them)
- all folders, their last uidvalidity, next-seen etc. are kept in a separate "imap-sync" sqlite table. Maybe this can be used to streamline some of the "Sent" folder and "DeltaChat" folder detection code we already have.
- We now also move self-sent messages from the Inbox to the Sent folder if `mvbox_move` is off, as this was very easy to do now. This way, we now behave more like a normal MUA if the user wants this.
FOR LATER PRs:
- maybe for the first 50 messages or so, we could reduce the IDLE-timeout (currently 23 minutes or so) to faster detect messages sent to non-inbox folders. However, on Android and iOS, we would likely trigger scan-all when the app moves to foreground, and so it might not be neccessary to reduce the current idle-timeout at least for them. We can leave this "faster discovery" question for the end, after we move to real-life testing.
- (Later on, after the above works, we can consider heuristics on which folders to perform IDLE on, and remove the Watch-folder options (inbox, deltachat, sent). We tried to find a safe scheme for already doing it but failed to fine one, too many unknowns, also some questions regarding multi-device (you might have different settings with each of it, one moves, the other doesn't etc.) so we postponed this in favor of the above incremental suggestion.)
* Start implementing #1994
* Add debug logs, it seems like the SQL migration can go into another pr
* Let fetch_new_messages return whether there are new emails
* Code style
* Don't prefetch if there are no new emails
* clippy
* Even more debug logs
* If the folder was not newly selected, return always try to fetch as
uid_next is probably outdated
* Fix new bug
* Recognize spam folder
* if we discover DC-messages in folders that have the "/Spam" flag (excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
* Clippy, prioritize folder_meaning over folder_name_meaning
* Add a first test, for the first day after installation only debounce to 2s
* Start adding two tests (both of them fail)
* Don't abort folder scan if one folder fails
* More consts
* Replace bool return value by enum
* Split test up into multiple tests
* Print logs during rust tests
* Rust tests pass now
* .
* One of the Python tests passes now - reconfigure folders during scanning
* Make the last test pass - Delete emails in all folders when starting the test, not only inbox and mvbox
The problem had been that emails were left in the folder "xyz"
* lint
* DB migration (untested)
* Store uid_next in SQL instead of lastseen in a config
* Revert "If Inbox-watch is disabled and enabled again, do not fetch emails from in between"
all folders are always watched, anyway
* clippy, rm debug logs, comments
* Codestyle, comments
* fixing things again
* Fix another test: don't fetch from uid_next-1 but uid_next; make some {} to {:#} so that we can use `.context(...)`
* move self-sent, non-setupmessage chat messages to the Sent folder if `mvbox_move` is off
* comment
* Comments, make sure things work even if there is no uid_next
* Style
* Comments
* The rust test tested wrongly
* comments, small codestyle change
* Ignore emails that are probably only drafts
Most mailboxes have a "Drafts" folder where constantly new emails appear
but we don't actually want to show them.
So: If there is no Received header AND it's not in the ConfiguredSentbox,
then ignore the email.
Also: Add test.
* Fix occasional test failure, it was introduced as DC now moves messages from Inbox to Sent
* Add `Received` header to the rust tests
* After this PR we will always watch all folders and delete messages there if server_delete is enabled. So, for people who have server_delete on, disable it and add a hint to the devicechat
* comment, small fix
* link2xt's first review
* Use ON CONFLICT(FOLDER) DO to update and if it doesn't exist, then insert
Reason from link2xt: We had a problem with multiple peerstates inserted due to key fingerprint parsing error previously. With logic in Rust a similar problem can occur: an UPDATE can fail for reasons other than a conflict. PRIMARY KEY should ensure uniqueness in this case, but anyway.
* Remove two TODO statements, remove fetch_new_messages: ignoring uid {}, uid_next was {} log
* Next TODO: Make uidvalidity and uid_next DEFAULT 0
* rm two TODOs, Seems like we are not going to `exclude folders that are watched anyway` in this PR
* small tweak: Handle instants more carefully
* Add scan_all_folders_debounce_secs config for tests, set debounce to 60s (before it was just 2s during the first day)
* Don't use bold letters for the device message
* React to changes in the folders better
Before, if there was a configured Sent folder, but then it got
removed and replaced with another folder with a name meaning "Sent" but
without Sent flag, it would be ignored.
So, instead of checking against ConfiguredSentboxFolder,
create two Option variables at the beginning of the loop and replace
them with Some if it is None. At the end of the loop, store the new
values into ConfiguredSendboxFolder and ConfiguredSpamFolder, even if it
is None.
Also, derive some useful traits.
* move job: Return a meaningful error if server_folder is None instead of panicing
* small error-handling fix
* Fix test_fetch_existing() python test
Before, we sometimes got a race condition where scan_folders() sees that
there is a Sentbox and saves this info after we set the
ConfiguredSentbox to None and before the message is sent.
So, just expect that the message is moved to the sentbox.
* migration is 72 now
* rm 2 TODOs, Don't infinitely retry when dc_receive_imf() returns Err
* clippy: Remove glob imports
* Delete created folders at the beginning of tests
(some created folders made problems in the next tests because)
* Improve resetting accounts between tests
* draft API to deal with uncut message texts
* add column mime_modified
* add mime_modified flag to MimeParser and save it in the database
* save mime_headers also when mime_modified is set
* cargo fmt
* set mime_modified on parsed html-texts and when there are multiple alternative-parts; add test for that
* prototype functions, add to repl and ffi
* use correct mime_modified flag
* basically parse Mime-Structure to HTML
* add basic tests for HTML-parsing
* convert text/plain to html for getting original
* respect charset for plain texts
* make test more specific
* fix handling non-utf-8 charsets for plain messages
* add test for plain_to_html()
* add failing test for plaintext linkify
* linkify urls in plain text
* fix regex
* plain text linkify: add failing test for encapsulated links as <https://domain.com>
* plain text linkify: make encapsulated links as <https://domain.com> work
* plain text linkify: require word boundary at beginning of link, add tests for that
* plain text linkify: linkify emails
* plain text: support format=flowed
* plain text: support quotes
* make clippy happy
* set mime-modified also when simplify() cuts non-html messages, add tests for that
* streamline mime recursion
* repl tool: write original html to file for further processing
* convert cid:- to data:-protocol
* add a test for cid: to data: conversion
* make clippy happy
* fix html-tests to work with windows-lineends
* clarify what the returned html-code may contain
* add some more detailed doc comments
* add mime_modified column only if not exist
this additional check is needed
as the column may added with another dbversion in
some shipped beta-versions.
* incorporate documentation suggestions from review
* rename get_original_mime_html() to more simple get_html()
* rename api is_mime_modified() to more simple has_html(); internally, mime_modified-flag stays as-is, however
* rename MimeS to MimeMultipartType
* do not set mime-modified flag for encrypted messages that need extra-handling for saved mime-structure
* fix typo
* move get_msg_html() to MsgId.get_html()
* incorporate more documentation suggestions from review
* remove unused return value from collect_texts_recursive()
* avoid mime_modified being mutable in write-parts-loop
* move 'use futures::future::FutureExt' atop of html.rs
* move attributes defining plain-text to a dedicated structure
* more PlainText to separate file
* escape cid when building regex
* let dc_get_msg_html() return NULL when calling with bad param
in addition to merged pr #2123,
dbversion=70 was also used by pr #2125.
pr #2125 is not yet merged, however beta-versions were shipped with
that pr, so, dbversion=70 is already in use on some systems and on these
systems ConfiguredProvider won't be set.
the other way round, that ConfiguredProvider is set twice on some systems
affects only dev-installations and should not harm.
once this pr is merged, #2125 needs a rebase and will use dbversion=72 for
its changes to add the new column also n the dev-installations :)
(we need to check that the column to add does not exist, however)
Previously MX records were queried only for OAuth 2 configuration and
did not affect the list of servers tried. User was required to manually
configure the servers for Google Workspace (former GSuite) domains.
Now MX records are queried during configuration. If provider is found in
offline database, its ID, corresponding to the filename, is saved as
`configured_provider`.
`configured_provider` is also set during database migration if email
address uses the domain from the provider database, but no MX querying
is done.
- Make sure delete_expired_messages and housekeeping runs once a day
- delete more info about messages when putting them to trash (esp. also
delete txt_raw, from_id and to_id as we don't need those anymore, so
they are data that was unnecessarily kept)
fix#1926, fix#2090
Also:
* Nicer test_utils: add send_text() and print_chat()
* Adapt ephemeral messages for testing (make them accurate to the second)
* Add test for ephemeral messages
* Make pop_sent_msg() really pop the last sent message
If mvbox_move was on and an mdn was received, it was not marked as read.
Also remove a confusing log that said that we are adding a markseen job, even if we weren't. As job::save() logs itself, there is no need to log this again.
adding an asterisk to the filename key
REQUIRES two apostrope-delimiters in the value as of charset'lang'name,
see rfc2231
as we do use the value this way, we MUST not add the asterisk.
things worked correctly in the past as
consuming MUAs are pretty graceful.
i assume the error comes from the c to rust conversion -
core-c did add the asterisk, however also did the apostrope-encoding.
core-rust leaves the asterisk but changes encoding.
disabled and re-enabled
That's another narrow-stitching patch for a scenario where many emails could
be deleted all at once and surprisingly: user disables inbox-watch, enables delete-from-server, after a moth enables inbox-watch again -> currently all emails that arrived in the meantime will be deleted (if emails are not fetched, they won't be deleted)
This way we can better distinguish tagged from untagged core releases, also in logs etc.
We might, from time to time, increase the alpha.N "N" number if we are entering testing etc.
providers may have a maximum for the SMTP RCPT TO: header,
therefore, an outgoing message is split into several chunks.
the chunk size defaults to 50, however, if known to be larger or smaller
by a dedicated provider, this setting may be changed.
this does not affect the MIME To: headers,
if a provider has a maximum here, this is change would not help.
ephemeral_task holds a reference to Context, preventing event emitter
from returning NULL and terminating event loop. Prior to this change,
there was no way to quickly terminate pending ephemeral_task.
the saved-messages can be deleted as any other chat;
if the feature is unneeded, this probably also makes some sense.
however, if saved-messages was deleted accidentally (yeah, i've seen that :),
it is not intuitive to see how the saved-messages feature can be restored.
this change just drops a note about how-to-restore to the device chat.
fix#2020
When forwarding a message, the original `in_reply_to` stays in place.
I did not find any recent commit that changed anything about this,
but IIRC there was a bug that prevented setting the `in_reply_to` at
all.
So, for chat messages, do not look at the InReplyTo header (and also not
at the References header)
by #2001, processing of protection-enabled and protection-disabled
messages is only done if verification-checks pass.
unfortunately, these verification checks
were only applied on already protected chats,
so that enabling via a protection-enabled message was not possible.
this is fixed by this pr.
moreover, when inner_set_protection() fails,
the error is shown in the chat.
this avoids confusion - as long the current core
does not communicate with a UI that enables/disables protection,
everything looks just the same as in the past :)
update_device_chats() takes about 2 seconds on a modern device (Android) because the
welcome image file has to be written to the disk as a blob. The problem
was that this was done after the progress bar had vanished and before
anything else happened so that I thought that something had gone wrong
multiple times.
The UIs have to remove update_device_chats(), too..
this would exclude non-deltas from protected-chats
(where they could participate in verified-groups in the past)
and makes migration a bit harder when using a fuzzy multi-device-setup.
however, the test still takes place
when a group is created or protection is enabled/disabled.
should_encrypt() shall return Ok(false)
on peerstate=EncryptPreference::Reset
only for opportunistic groups.
for verified/protected groups (e2ee_guaranteed set),
Ok(true) or Error() should be returned.
this bug was introduced by #1946 (Require quorum to enable encryption).
a draft may contain only a quote,
without any text set yet.
these drafts cannot be sent, however, appear in the summary -
currently with the summary-text "", which results to sth.
as "Draft: " - which looks like an error or at least a bit weird.
this pr sets the summary text to "Reply" - similar to "Image", "Video" etc. -
the UI just expects some text here, not an empty string.
the result are summaries as "Draft: Reply" on all UIs -
which, btw. is also roughly the same what Signal does in this case.
the constant comes from c-core
and was used to define the size of the string-array that time.
this is no longer needed in core
and also the UIs seems not to use it
(they will treat DC_STR* like an id,
there should be no need to know the max. DC_STR* value)
Read all of an e-mail accounts messages and extract all To/CC addresses
if the From was from our own account.
Then, fetch existing messages from the server and show them.
Also, I fixed two other things:
- just by chance my test failed because of an completely unrelated bug.
The bug: bcc_self messages were not marked as read if mvbox_move was set
to true.
- add some color to the test output (minor change)
Sticky encryption rule, requiring that all replies to encrypted messages
are encrypted, applies only to messages with a quote now.
Co-Authored-By: B. Petersen <r10s@b44t.com>
This code is inconsistent with EncryptHelper::should_encrypt, which uses
quorum rule.
UI does not display or use encryption state in-between preparing and
sending message anyway.
In addition, "sticky encryption" rule, which required all replies to
encrypted messages to be encrypted, is dropped. It is going to be
restored with the introduction of quoting.
both are not used in productive:
- chat-id "in-creation" was dropped completely already in core-c
as it does not allow assigning messages to chats in time.
- message flag "starred" was added at some point and never used.
ux-wise, "disappearing messages" promises to clean up a chat -
this does not fit well with "starring" messages.
"saved messages" chat seems to be superior in that regard.
but anyway, it is not used but comes at maintainance costs,
so it is better to delete its functionality for now.
(the db entries, however, are left for now,
that might be more tricky to remove)
Address is used to generate Message-Id.
Previously error toast was shown, but MsgId was returned from
chat::prepare_msg anyway. Now chat::prepare_msg returns an error.
error!() is removed, because FFI library shows errors via
.unwrap_or_log_default().
When run against host that is not in the provider database or has multiple
IMAP servers listed, this test fails because one error is returned for
each server.
* Mention in dc_event_get_data2_str method that it can also return 0
* Update deltachat-ffi/deltachat.h
Co-authored-by: bjoern <r10s@b44t.com>
Co-authored-by: bjoern <r10s@b44t.com>
Previously, standard Autocrypt rule was used to determine whether
encryption is enabled: if at least one recipient does not prefer
encryption, encryption is disabled.
This rule has been problematic in large groups, because the larger
is the group, the higher is the chance that one of the users does not
prefer encryption.
New rule requires a majority of users to prefer encryption. Note that it
does not affect 1:1 chats, because it is required that *strictly* more
than a half users in a chat prefer encryption.
partly, we wrote '@return None.' for void functions,
however, some linter complain about that (recently swift)
and they're right, it does not make much sense.
On Android, there is to /tmp directory, so SQLite may throw
SQLITE_IOERR_GETTEMPPATH when trying to find a place for temporary
files. Storing temporary files in memory is a workaround that always
works (until out of memory).
See https://github.com/mozilla/mentat/issues/505 for discussion of a
similar issue.
In the current design the dc_receive_imf() pipeline calls
handle_securejoin_handshake() before it creates the group. However
handle_securejoin_handshake() already signals to securejoin() that the
chat exists, which is not true.
The proper fix would be to re-desing how group-creation works,
potentially allowinng handle_securejoin_handshake() to already create
the group and no longer require any further processing by
dc_receive_imf().
'Delete emails from server' was an experimental ad-hoc function
that was added before 'Automatically delete old messages' was introduced
to allow at least some way to cleanup.
'Delete emails from server'-UI was already removed from android/ios in june
(see https://github.com/deltachat/deltachat-android/pull/1406)
and the function never exists on desktop at all.
This currently tests the setup-contact full flow and the shortcut
flow. More securejoin tests are needed but this puts some
infrastructure in place to start writing these.
This commit fixes condition which ignores reordered autocrypt messages.
If a plaintext message resetting peerstate has been received, autocrypt
header should only be applied if it has higher timestamp.
Previously, timestamp of the last received autocrypt header was used,
which may be lower than last_seen timestamp.
# Conflicts:
# src/peerstate.rs
Previously DC_EVENT_ERROR_NETWORK events were emitted for each failed
attempt during autoconfiguration, even if eventually configuration
succeeded. Such error reports are not useful and often confusing,
especially if they report failures to connect to domains that don't
exist, such as imap.example.org when mail.example.org should be used.
Now DC_EVENT_ERROR_NETWORK is only emitted when attempting to connect
with existing IMAP and SMTP configuration already stored in the
database.
Configuration failure is still indicated by DC_EVENT_CONFIGURE_PROGRESS
with data1 set to 0. Python tests from TestOnlineConfigurationFails
group are changed to only wait for this event.
This turns the Bob::expects field into an enum, removing the old
constants. It also makes the field private, nothi0ng outside the
securejoin module uses it.
Finally it documents stuff, including a seemingly-unrelated Param.
But that param is used by the securejoin module... It's nice to have
doc tooltips be helpful.
LoginParamNew structure, which contained possible IMAP and SMTP
configurations to try is replaced with uniform vectors of ServerParams
structures. These vectors are initialized from provider database, online
Mozilla or Outlook XML configuration or user entered parameters.
During configuration, vectors of ServerParams are expanded to replace
unknown values with all possible variants, which are tried one by one
until configuration succeeds or all variants for a particular protocol
(IMAP or SMTP) are exhausted.
ServerParams structure is moved into configure submodule, and all
dependencies on it outside of this submodule are removed.
in general, Viewtypes other than File should be used only
when the added file type is tested on all platforms -
including Android4 currently.
as this is easily overseen,
i've added a comment.
this partly reverts Viewtype changes done by
https://github.com/deltachat/deltachat-core-rust/pull/1818
Fix#1791 and show all group messages if the user already is in the group, even if the sender is blocked
Also fix a comment
Co-authored-by: link2xt <ilabdsf@gmail.com>
This fixes#1804 in two ways: First, it removes a <!doctype html> from
the start of the mail, if there is any.
Then, it parses the html itself it quick-xml fails, just stripping
everything between < and >.
Both of these would have fixed this specific issue.
Also, add tests for both fixes.
threading was broken in core43 as this flags unencrypted messages as errors
and errors are not replied-to.
the fix is not to mark missing signatures for unencrypted messages as errors.
.tgs files are Telegram stickers. Internally they are gzipped JSON files,
containing a single Lottie animation.
MIME type application/x-tgsticker is commonly used for telegram documents
containing such stickers.
An EphemeralTimerChanged message with the same timer as already set can
be received when there are large delays or lost messages. Even though
inner_set_ephemeral_timer should not be called in this case, because it
emits an event indicating timer change, system message will be added to
the chat, so it should be translated with set_better_msg in any case.
Normally NULL is used when there is no fingerprint, but default value
for fingerprint columns is an empty string.
In this case, loading should not fail with an "invalid length" error,
but treat the fingerprint as missing.
Strict check was introduced in commit ca95f25639
Since try_decrypt does not check if the message has valid signatures
anymore, it may return empty signatures set, which means the message is
not a valid autocrypt message.
this pr allows prefixing custom jitsi urls with `jitsi:`
so that clients have a chance to detect these custom instances
and may start the corresponding app.
introduce the qr-code-type `DCWEBRTC:[webrtc_instance]`.
dc_check_qr() returns this as the type DC_QR_WEBRTC
and these qr-codes can be passed to dc_set_config_from_qr() then;
this finally sets `webrtc_instance` using dc_set_config().
needed for how node/js handles the chatlist currently
(it convertes the ffi-object to an js-object and throws ffi-object away directly,
this makes it hard to access dc_chatlist_get_summary() later.
Since introduction of provider database, these certificate checks are
enabled for test servers anyway, but this setting prevents usage of
local servers with self-signed certificates.
It presents no security issue, because properly configured servers will
only serve passwords on HTTPS and distribute only HTTPS QR codes, but
makes testing easier when HTTPS is not easy to deploy.
If attacker can control the URL used, they can change the URL to another
HTTPS URL controlled by them and act as a proxy between the client and
original server anyway.
Adhoc groups for group messages that don't have Chat-Group-ID are already
created above in another create_or_lookup_adhoc_group.
At this point it could be that there is a valid Chat-Group-ID header,
but no group was created because removed_id was non-zero i.e. received
message removes some group member.
If group was not explicitly left, current code creates an adhoc group
instead of trashing late or reordered messages that remove group
members. Such groups have adhoc group IDs locally, but proper group
message-IDs. Attempts to reply to such groups or leave them creates
"split groups" for all other members of original group. The solution is
not to create adhoc groups in this case.
Permanent error 550 5.1.1 is no longer considered temporary.
Enhanced status code is checked now, so only 550 5.5.0 is an exception
for misconfigured Postfix servers.
Yandex error 554 5.7.1 was handled correctly, but only because it had
response code 554, while the comment talks about enhanced status code
5.7.1. The comments are corrected.
Failed messages are now marked as such with message::set_msg_failed.
Previously they were left in a pending state.
If info message cannot be added to the chat, the error is displayed with
error! instead of being logged with warn!.
when seeing our own profile image send from other devices,
we use them as Selfavatar on the current device as well.
as there is no special message sent on avatar changes,
this is a simple approach to sync the avatar across devices.
Now MarkseenMsgOnImap sends MDN even if it can't mark the message as
seen on the server.
To prevent multiple MDNs from being sent, MarkseenMsgOnImap is postponed
until the message is detected in a folder from which it is not going to
be moved.
Mimeparser.was_encrypted() checks if the message is an Autocrypt encrypted
message. It already means the message has a valid signature.
This commit documents a few functions to make it clear that signatures
stored in Mimeparser must be valid and must always come from encrypted
messages.
Also one unwrap() is eliminated in encrypted_and_signed(). It is possible
to further simplify encrypted_and_signed() by skipping the was_encrypted()
check, because the function only returns true if there is a matching
signature, but it is helpful for debugging to distinguish between
non-Autocrypt messages and messages whose fingerprint does not match.
Add a mutex to prevent a race condition when a "your pw is wrong" warning is sent, resulting in multiple messeges being sent.
Do not mute the device chat but "only" send MsgsChanged event when no
notification shall be shown.
More logging.
Parallel processing of messages results in bugs such as messages sent by
a new member being processed before the message that adds this member to
the chat, even when it has lower UID. In this case, messages are shown as
"[Unknown sender for this chat. See 'info' for more details.]", while
there is a message "Member ... added" right before, because writing the
information about new member to the database takes longer then reading
old index from the database.
With this change, API user does not need to look at the timestamp of
the next message to display the day marker, but can get the timestamp
directly from the marker. This way there is no database query and no
risk of error as a result of database being busy or message being deleted.
d="m 8.6780027,35.573064 0.032831,-11.910176 c 0.00138,-0.476406 0.4881282,-0.794259 0.9235226,-0.604877 l 4.1144877,2.345752 -0.02386,8.656315 -4.1268029,2.122946 C 9.1617452,36.370003 8.6766889,36.049472 8.6780027,35.573064 Z m 5.0469633,-1.508222 0.02386,-8.656314 31.145424,-9.537653 c 0.841472,-0.219211 1.65915,0.41667 1.656755,1.283728 l -0.06929,25.139995 c -0.0024,0.867062 -0.825942,1.500799 -1.663803,1.274581 z m 3.8042,6.892234 C 16.681121,40.104348 16.315444,38.819414 16.69043,37.591308 l 2.252234,-7.347193 c 0.2644,-0.861571 0.845185,-1.567441 1.641953,-1.989251 0.796769,-0.421808 1.706956,-0.509819 2.568531,-0.245419 l 7.263888,2.225804 c 1.775518,0.543235 2.780299,2.432591 2.232297,4.208094 L 30.3971,41.790532 c -0.545627,1.777887 -2.432591,2.780297 -4.208095,2.232298 l -7.263891,-2.225804 c -0.545033,-0.165864 -1.01825,-0.460162 -1.395948,-0.83995 z m 12.377693,-7.976728 c -0.07601,-0.07642 -0.17114,-0.133864 -0.280621,-0.167516 l -7.263891,-2.225803 c -0.233244,-0.07209 -0.421626,0.0013 -0.512275,0.04861 -0.09064,0.0474 -0.25772,0.166033 -0.327435,0.396899 l -2.252234,7.347191 c -0.108166,0.354628 0.09088,0.731541 0.447888,0.842099 l 7.263891,2.225802 c 0.354626,0.108174 0.731539,-0.09088 0.842099,-0.447888 l 2.249845,-7.344814 c 0.07453,-0.245145 0.0014,-0.504991 -0.167267,-0.67458 z"/>
Note that all these problems (can) also happen with verified groups,
then raising "false alarms" which could lure people to ignore such issues.
For reference, please see https://github.com/deltachat/deltachat-core-rust/blob/master/spec.md#add-and-remove-members how MemberAdded/Removed messages are shaped.
IOW, it's clear we need to do something about it to improve overall
reliability in group-settings.
- All Chat-Group-Member-Added/Removed messages are recorded in their
full raw (signed and encrypted) mime-format in the DB
- If an incoming member-add/member-delete messages has a member list
which is, apart from the added/removed member, not consistent
with our own view, broadcast a "Chat-Group-Member-Correction" message to
all members, attaching the original added/removed mime-message for all mismatching
contacts. If we have no relevant add/del information, don't send a
correction message out.
Solution: replay group modification messages on inconsistencies
- Upong receiving added/removed attachments we don't do the
check_consistency+correction message dance.
This avoids recursion problems and hard-to-reason-about chatter.
For brevity let's abbreviate "group membership modification" as **GMM**.
Notes:
Delta chat has explicit GMM messages, typically encrypted to the group members
as seen by the device that sends the GMM. The `Spec <https://github.com/deltachat/deltachat-core-rust/blob/master/spec.md#add-and-remove-members>`_ details the Mime headers and format.
- mechanism works for both encrypted and unencrypted add/del messages
If we detect membership inconsistencies we can resend relevant GMM messages
to the respective chat. The receiving devices can process those GMM messages
as if it would be an incoming message. If for example they have already seen
the Message-ID of the GMM message, they will ignore the message. It's
probably useful to record GMM message in their original MIME-format and
not invent a new recording format. Few notes on three aspects:
-we already have a "mime_headers" column in the DB for each incoming message.
We could extend it to also include the payload and store mime unconditionally
for member-added/removed messages.
-**group-membership-tracking**: All valid GMM messages are persisted in
their full raw (signed/encrypted?) MIME-format in the DB. Note that GMM messages
already are in the msgs table, and there is a mime_header column which we could
extend to contain the raw Mime GMM message.
-multiple member-added/removed messages can be attached in a single
correction message
-**consistency_checking**: If an incoming GMM has a member list which is
not consistent with our own view, broadcast a "Group-Member-Correction"
message to all members containing a multipart list of GMMs.
-it is minimal on the number of overall messages to reach group consistency
(best-case: no extra messages, the ABCD case above: max two extra messages)
-**correcting_memberships**: Upon receiving a Group-Member-Correction
message we pass the contained GMMs to the "incoming mail pipeline"
(without **consistency_checking** them, to avoid recursion issues)
- somewhat backward compatible: older clients will probably ignore
messages which are signed by someone who is not the outer From-address.
- the correction-protocol also helps with dropped messages. If a member
did not see a member-added/removed message, the next member add/removed
message in the group will likely heal group consistency for this member.
Alice/Carol and Bob/Doris getting on the same page
This will run all "offline" tests and skip all functional
end-to-end tests that require accounts on real e-mail servers.
.._livetests:
running "live" tests with temporary accounts
---------------------------------------------
If you want to run live functional tests you can set ``DCC_NEW_TMP_EMAIL`` to a URL that creates e-mail accounts. Most developers use https://testrun.org URLS created and managed by [mailadm](https://mailadm.readthedocs.io/en/latest/).
Please feel free to contact us through a github issue or by e-mail and we'll send you a URL that you can then use for functional tests like this:
export DCC_NEW_TMP_EMAIL=<URL you got from us>
With this account-creation setting, pytest runs create ephemeral e-mail accounts on the http://testrun.org server. These accounts exists only for one hour and then are removed completely.
One hour is enough to invoke pytest and run all offline and online tests:
pytest
# or if you have installed pytest-xdist for parallel test execution
pytest -n6
Each test run creates new accounts.
.._sourceinstall:
Installing bindings from source (Updated: July 2020)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
