* repl: allow groupname arguments with more than one word (came over that when testing qr codes)
* calcualte text-size from the real number of lines
* shift text and watermark apart when text get longer
* make clippy happy
This function is used to lookup the chat by `References` and
`In-Reply-To` header, so it does not make sense to return trashed
message when there is another non-trashed message in one of these
headers with a real chat ID.
* generate qr code svg prototype
* qr code for groups
fix formatting
* - letter avatar in qrcode
- escape xml in userinput (display/groupname)
- fix "Me" display name
- merge import declarations
* remove dot at the end of VerifyContactQRDescription
* if addr == displayname, show only one of them
Especially useful for yggmail accounts without usernames,
because the text would overflow otherwise.
* use real clipPath for rounded avatar
* - center avatar text better (dominant-baseline)
- add "sans-serif" to font fallback for text if arial is missing
* make corner always blue
* add [logo + "get.delta.chat"] footer to qrcode
* Update deltachat-ffi/deltachat.h
Co-authored-by: bjoern <r10s@b44t.com>
* Apply suggestions from code review
Co-authored-by: bjoern <r10s@b44t.com>
* new card design
- add stockstrings
- update changelog
* make qrcode pixels also #f2f2f2 instead of full white
* rename VERIFY_CONTACT_QR_DESC to SETUP_CONTACT
make footer text a tiny bit darker upon r10s's request
* avoid using which is a doxygen command
* point out that one will join a group (this is still shorted and was also suggested in recent chats)
* add option to generate qr-code-svg to repl tool
* use same font-family in text and footer
* thinner card border
* remove superfluous <tspan> from footer to make color tweaking easier
* move font-weight to style, ios renderer does not pick it up from attribute; remove default font attributes not used consequently
* make get.delta.chat more visible
* align properly using dominant-baseline=central and alignment-baseline=middle, this makes things nice on all systems but android (before, ios was wrong and all others not 100% aligned as font metrics are ignored) (android needs a subsequent improvement)
Co-authored-by: bjoern <r10s@b44t.com>
`indexmap` is a large dependency (4K SLoC) containing `unsafe` code.
Contact IDs are now passed around as a Vec<u32> or &[u32].
QUOTA roots are now sorted by name instead of perserving original order.
It was there since the C core, labeled with "/* last_seen is for
future use */" but never actually used. The comment was lost during
the translation from C to Rust.
`cutil.from_dc_charpointer()` is guaranteed to return `str`, while
`cutil.from_optional_dc_charpointer()` may return `None` if C function
returns `NULL`.
It was only used in two places. Rather than adding `use
pretty_assertions::*` everywhere, it's easier to remove, and it
removes two additional dependency crates.
`deltachat.const` module now defines `__getattr__` and `__dir__` as
suggested by https://www.python.org/dev/peps/pep-0562/
mypy detects that `__getattr__` is defined and does not show errors
for `DC_*` constants which cannot be detected statically.
mypy is added to `tox.ini`, so type check can be run with `tox -e mypy`.
Generally we could also just remove the device message as we have the
connectivity view, OTOH if you are not using DC a lot, it may be useful
to be notified without opening DC.
And apart from this one bug it's working fine.
`apply_group_changes` is executed regardless of whether the group is
created via `create_or_lookup_group` or found via
`lookup_chat_by_reply`. This change removes the need for
`lookup_chat_by_reply` to return `None` when group ID exists in the
database to let `create_or_lookup_group` run. As a side effect of this
Delta Chat replies to ad hoc groups are now correctly assigned to
chats when there are multiple groups with the same group ID, such as
ad hoc groups with the same member lists.
* test one-to-one chats on setup-contact/secure-join
only one chat is created after scanning a QR code:
- on setup-contact, one-to-ones are created on both sided
- on secure-join, the joined group chat is created;
one-to-ones are not created intitally,
but should become visible on receiving messages
* make sure, Alice creates the chat with Bob on setup-contact
not totally sure if that change in #2508 was on-purpose,
however, all yet released versions
did create the one-to-one chat also on the Inviter's (Alice) side,
so, let's stay with that,
i do not see many reasons to change that.
* unblock hidden (Blocked::Yes) one-to-one chats
one-to-one chats may be hidden by secure-join,
in case someone later writes a message to it
(not unlikely), the chat needs to be shown.
before, messages are just not shown,
the corresponding chat did not appear.
the 'Blocked' wording of a 'Chat' must not be mixed with the
'Blocking' of a contact. 'Chat-Blocking' is mostly a visibility thing,
that may change as messages come in.
this change should not affect _really_ blocked contacts -
they are filtered out already before
and their messages are usually not even downloaded.
also, before allow_creation is checked,
that may disallow chat creation for show_emails reasons.
all in all, it just does the same
as if the user has manualy deleted the chat before and it would be created.
* simplify test
utcfromtimestamp() is not recommended by the official documentation,
because many methods, including timestamp(), work incorrectly with
"naive" datetimes returned by utcfromtimestamp().
Update gossiped_timestamp when someone else sends autocrypt gossip in
the group, so we postpone sending gossip again ourselves.
- Warn about failures to parse Autocrypt-Gossip header
- Move gossip-related methods into ChatId impl
- Fix a "gossi_pp_ed" typo
- Make `state` variable immutable.
- Don't mark contact requests as fresh when fetching existing messages: this is not needed since there is no contact request chat anymore.
- Don't mark messages from blocked chats as noticed, this is not needed since messages go to blocked chat anyway instead of contact request chat.
* refactor: cleanup send_handshake_msg()
- rename to send_alice_handshake_msg() as used by Alice only
- remove dead code from Bob
(Bob's code is at BobState::send_handshake_message() since some time)
- take a contact_id and not a chat_id;
this makes things less confusing when
info-messages are put to the final group chat
* always directly return chat-id from dc_join_securejoin()
* take care not to create a group twice
* adapt documentation
* add info-msg on group invites; add inviter directly after creation
* document existing 'joinqr' command in repl tool
* do not create empty one-to-one chats for group-joins
* refactor: cleanup fingerprint_equals_sender()
- the function takes a contact_id directly now.
before it consumes the first contact of a one-to-one chat -
which may be easily confused with the group-chat in creation.
moreover, the conversion contact_id -> chat_id -> contact_id
is unneeded overhead.
* show info-messages in destination chat for alice
* fingerprint_equals_sender() returns Err on database failure
* tweak documentation
* clarify what an 'unfinished tasks' task is.
* add regression test for create_for_contact_with_blocked()
* rename Blocked::Manually to better fitting Blocked::Yes
* tweak test_secure_join() and make sure, Alice and Bob have only on chat after a group-join
* add basic multi-device-sync functions
* generate json
* add context.parse_sync_items()
* add context.execute_sync_items()
* piggyback sync-commands message, add body for human-readable part
* avoid double json renderings
* mimeparser parses incoming .json sync-files
* do not piggyback sync-files
* execute sync items
* return status of send_sync_msg()
* send sync messages as multipart/report
* add a per-item-timestamp and also allow adding other per-item-fields in the future
* if the self-chat does not exist, create it blocked/hidden
* create tokens closer to real qr-code needs
* respect bcc_self setting, add test for that
* sync qr code tokens after promoting groups
* send sync-messages only if an experimental switch is set
* trigger send_sync_msg() after sending messages and after creating/redraw/revive qr-code
* add DC_STR_* constants to deltachat.h
* adapt to refactored qr module as of #2729
* tweak test
* use SendSyncMsgs config name instead of SendExperimentalSyncMsgs - we can remove or rename the config nevertheless, but have the option to keep it without renaming
* tweak docs
* remove currently unused effective timestamp calculation
* clarify when send_sync_msg() is called
* make sure, sync-messages are encrypted and are sent by SELF
* tweak docs, fix typos
Before this PR, when a test failed, we often got:
```
thread panicked while processing panic. aborting.
error: test failed, to rerun pass '-p deltachat --lib'
Caused by:
process didn't exit successfully: `/home/user/deltachat-android/jni/deltachat-core-rust/target/debug/deps/deltachat-33648fc4aaad608c 'contact::tests::test_selfavatar_changed_event' --nocapture` (signal: 4, SIGILL: illegal instruction)
```
instead of the error message and stack trace.
This PR fixes this.
* Add DC_EVENT_SELFAVATAR_CHANGED
* Add test.
Unfortunately I can't easily also test that the avatar is not copied
from unencrypted messages:
In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.
* Unfortunately I can't easily also test that the avatar is not copied from unencrypted messages:
In the second encrypted message, the avatar would not be sent again
then, because we only send avatars once a day or so.
* add let's encrypt certificate missing on some older android devices
* create Certificate with Lazy::new()
* document certificate source
* use smaller *.der format instead of *.pem
Reduce the number of mutable variables and out parameters in
`add_parts`.
Also don't call `dc_create_smeared_timestamp` if there is no `Date`
header. Timestamps are only supposed to be "created" when the message
is sent, not received, to make sure sent messages are sorted properly
in MUAs that only use the date for sorting. Delta Chat uses database
IDs for sorting in addition to timestamps, so it can sort messages
with equal timestamps properly.
Update `dc_smeared_time` documentation.
Turn `dc_smeared_time` and `dc_create_smeared_timestamp` comments into
documentation comments.
The problem was:
When opening the connectivity view while there is no network,
get_connectivity_html() calls schedule_quota_update(), which schedules a
UpdateRecentQuota job. But in update_recent_quota(), connecting fails
and a ConnectivityChanged event is emitted (connectivity changes from
Error to Connecting and back). Therefore the UI calls
get_connectivity_html() again, and the loop is complete.
This made the UI completely unresponsible. To reproduce, just turn wi-fi
off and open the connectivity view.
The fix is:
schedule_quota_update() now only schedules a new job if there is no old
job. To prevent the possible (though probably unlikely) problem that an
old quota update job has a backoff of, like, a day and therefore quota
is not updated, I reduced the backoff time for quota jobs to 10s.
Fixes possibly https://github.com/deltachat/deltachat-android/issues/2043, but we should "re-try" this
* better readable enum
* add a failing test
* let new secure-joins abort existing ones
before, a stale secure-join or setup-contact
made the whole qr-scanning unusable until
the app is restarted,
resulting in "QR process failed" errors.
this commit fixes the issue by
aborting existing scans -
in cases, a user really wants two concurrect joins running,
this is not perfect, but that did not worked before as well.
* remove unused AlreadyRunning variant
* make clippy happy
* add Chattype::Broadcast and create_broadcast_list()
* do not disclose recipients for broadcasts
* allow sending/add-/remove-member for broadcast
* set broadcast subject same as for one-to-one chats
* broadcast-recipient-list does not include SELF
* use special icon for broadcast groups
* generate initial broadcast names
* make clippy happy
* send BCC message unencrypted to avoid unexpected disclosing; encryption is opportunistic anyway. if we have 'protected chats' at some point, we can think that over.
* reword 'To:'-group
* simplify can-send-check
* add broadcast tests
* tweak comments
* Update deltachat-ffi/deltachat.h
Co-authored-by: Hocuri <hocuri@gmx.de>
* change name of can_edit() to is_self_in_chat()
Co-authored-by: Hocuri <hocuri@gmx.de>
this is needed for targeting "non-blocking group QR joins"
as create_multiuser_record() would also be needed from other places.
this will make rebasing/rewriting and finally reviewing #2508 easier.
in the past, group-messages were marked as "read by recipient"
only when at least 50% of the group members have send a read receipt -
in practise, this does happen never or much too late esp. in larger groups.
setting the state OutMdnRcvd already on the first read-receipt
seems to be much more intuitive and useful as you at least know
one person has read the message.
this is also what other messengers as telegram are doing here.
moreover, this fixes a bug that did not add all read-receipts
to the "Info" screen - once "enough" read-receipts were received,
and the state was already set to OutMdnRcvd, subsequent read-receipts
were ignored.
probably because the "Info" screen did not show the read-receipts since forever -
and for the second tick, the addutional read-receipts are not needed.
* let quota-warning reappear after import
an import removes all device-messages,
including the quota warning.
resetting `Config::QuotaExceeding` makes sure,
the warning reappears soon after import -
otherwise the warning would reappear only after
storage is cleaned up and exceeds again.
* a second quota warning when 95% storage exceeded
* factor out warning-check and add a test for that
* add missing stock strings for connectivity and quota
* reword quota error
* use 'Incoming/Outgoing Messages' instead of 'Inbox/Outbox'
* just say 'Not supported by your provider.' if quota cannot be read.
- the context is already given by the headline
'Storage on domain.org'
- the string is short and does not disturb much
(it is a very common error)
- the string does not mention 'quota' as such,
which will be hard to translate ('Kontingentsinformationen') -
even if ppl as we know about the Quota extensions an such things.
there was also the idea to hide the whole section,
however, that is confusing in a multi-device-usage
when things are sometimes shown and sometimes not.
before, MsgsChanged(chat_id, new_msg_id) was emitted,
but that does not cover the deleted message.
in theory, we could emit both,
however, that would just be a waste of refresh in uis.
also before, events were used this way,
however, also the documentations are updated to
reflect reality better.
Implement get_previous_message() that only looks for the last
Message-ID in the References: header instead of using
get_parent_message() which falls back to using other Message-IDs from
References and In-Reply-To field.
* add a test for xing mailinglists
* strip long hash-prefixes from mailinglist name if we got the name from the List-Id as a last resort
* add a test for newsletter@ mailinglists
these mailinglists have the list-name in `From:`
and can be detected by addresses starting with `newsletter@`.
* if not list-name is set, use the `From:` name for addresses starting with `newsletter@`
this is similar to what we do with `notifications@`
* Update src/dc_receive_imf.rs
Co-authored-by: Hocuri <hocuri@gmx.de>
* add an example to the regex
Co-authored-by: Hocuri <hocuri@gmx.de>
* draft a download-api
* basic implementation
* allow partial downloads for protected chats
* use a separate column for download_state
* force a minimal timeout for delete_server_after in combination with partial messages
* add a warning if a possible download may expire by delete_server_after
* test load_imap_deletion_msgid()
* add a test for a partial download
* improve documentation and visibility
* let get_download_limit() return Result<Option>
* rusty getters
* apply MIN_DELETE_SERVER_AFTER to shown availability time
* move stub-creation to download.rs, use stock-strings, nicer logging
* make clippy happy (cargo clippy --tests)
* refine tests and comments
* fix typo
* remove superfluous closure in ffi
* respect partial_download for immediately scheduled DeleteMsgOnImap jobs
Integer overflows crash the application by default.
On a first sight this is only a potential crash that can't be
triggered, because timestamps are stored as i64 and ephemeral timer
duration is u32.
* add a test for .xt.local mailinglists
* get correct names of .xt.local mailinglists
these mailinglist probably come from the xt:Commerce system
and are pretty widely used.
i have not seen an .xt.local mailinglist with name set in List-Id,
however, if that happens, it will still be taken.
only if the name is unset,
we use the name from the From-header.
summary::Summary replaces Lot in the Rust API for methods returning
chatlist summaries. Lot is a legacy type for C API compatibility, so
Summary can be converted into Lot.
There is a real risk of an active attack when connecting to non-.onion
servers over Tor, as bad Tor exit nodes are cheap to set up.
It's probably not needed for .onion domains, but we don't make an
exception for now.
* check update timestamps for signatures, user-avatars, ephemeral-settings, last-subject
* check update timestamp for group-avatars
* check update timestamp for group-names
* check update timestamp for memberlist
* check update timestamp for protection-settings
* add a more advanced test
* add another more advanced test
* set last-subject-timestamp more carefully
* bubble up errros from set_*timestamp() and check for from_id==0 before
* simplify Params::set_i64()
* remove comment that is more confusing than helpful
* use update_timestamp() wording consistently
Also make Accounts uncloneable. It is still possible to derive Clone,
but does not make sense to do so, as .clone() creates two separate
account managers which use the same files but different unsynchronized
in-memory data structures.
Debian deprecated `which` in `debianutils` in favor of `command`.
`which` outputs this to stderr now:
/usr/bin/which: this version of `which' is deprecated; use `command -v' in scripts instead.
* resultify update_recent_quota()
* add a device-message if quota exceeds QUOTA_WARN_THRESHOLD_PERCENTAGE
* check if a quota warning should be added during housekeeping, this is at least once a day
* dc_get_config("quota_exceeding") is useful for bots
* make clippy happy
* reword QuotaExceedingMsgBody message
* avoid lots of warnings if quota jitters around the warning threshold
* allow clippy::assertions_on_constants
these constants depend on each other, it makes sense to check that they are not changed in an incompatible way
This adds following settings:
- Socks5Enabled
- Socks5Host
- Socks5Port
- Socks5User
- Socks5Password
Currently http requests and dns requests are not getting executed as they currently can't get tunneled through socks5 proxy. Therefore gmail with oauth2 wont work through tor.
- Use the same limit for info: full text can be read in HTML anyway.
- Remove DC_MAX_GET_{TEXT,INFO}_LEN constants from deltachat.h
- Fix a typo: s/DC_ELLIPSE/DC_ELLIPSIS/
- Do not truncate the text when loading from the database.
- Update the documentation: limit is in Rust chars, not bytes
the 'device chat about' was shown as a single message
as at that time, there was just not 'status'.
meanwhile, we have the status option,
and it feels much more natural to get the information there,
esp. as the subtitle on all UIs already read
'Messages in this chat are generated locally' -
and a tap on that will show the hint, without scrolling or so.
this also teaches the user where to find such information -
and the "welcome" chat is less spammy and really starts with the text
"Welcome to Delta Chat!"
* allow dotless domains and hostnames
* update provider database
ran ./src/provider/update.py ../provider-db/_providers/ > src/provider/data.rs
to pull in recent changes from https://github.com/deltachat/provider-db
* add imap::get_quota_roots()
* schedule quote-checking job on getting connectivity-html
* get quota and debug print it
* basic quota output
* update quota at most once per minute, emit event on changes
* use more meaningful names
* add some comments, move update_recent_quota() to quota.rs
* show root name only if there are several roots
* make clippy happy, some refactorings
* allow only one update-quota job per time
* add now supported QUOTA to standards.md
This bug sometimes results in sending read receipts to self in
multi-device setups.
It happens consistently in a setup where the first device is
configured to move messages to DeltaChat folder and the second device
is not. When both devices receive BCC-self message simultaneously, the
first device moves the message to DeltaChat folder, while the second
device tries to mark the message as seen in the Inbox. Regardless of
whether the second device marks the message as seen successfully or
fails because the message is already moved by the first device,
`Job.markseen_msg_on_imap()` sends the read receipt to the From
address.
Recognizing these emails as forwarded would probably be too complicated and require too much special-casing, but now the user can access the email text via "Show full message".
fix#2599
Co-authored-by: B. Petersen <r10s@b44t.com>
* In dc_maybe_network_lost() directly set the connectivity "Not connected"
r10s reported that without doing this, the connectivity would stay at
"Connected" for 16 more seconds after network is gone and
dc_maybe_network_lost() was called.
* Set the state for all connections
fix#2539
It's always a bit ambiguous which function should do set_err or set last_send_error, I used this rule here:
If some code returns Status::RetryLater, then it sets last_send_error, because Status::RetryLater means that the user won't see the error directly on the message (if we returned Status::Failed(Err(_)), then the message would be shown as failed to the user) Also, smtp_send always sets last_send_error because, well, sending just failed or succeeded.
Also, remove unused field pending_error.
Previously system messages were always added to the end of the chat,
even if the message triggering them was sent earlier. This is
especially important for messages about disappearing timer reset
triggered by classic email messages, as they should be placed right
after the message resetting the timer.
- Use latest stable for rustfmt and clippy.
- Update Rust in rust-toolchain and coredeps Docker container to 1.54.0
and test against it.
- Test against 1.48.0 to ensure libdeltachat can be compiled with Debian
bullseye "rustc" and "cargo".
Gmail labels are not folders and should be skipped. For example, emails
appear in the inbox and under "All Mail" as soon as it is received. The
code used to wrongly conclude that the email had already been moved and
left it in the inbox.
the WAL files come from sqlite WAL mode,
they have the same name as the database file with an extra `-wal` suffix.
according to https://sqlite.org/wal.html#the_wal_file ,
the WAL file is part of the persistent state of the database
and should be kept with the database if the database is copied or moved.
"Archived Chats" also contain old contact requests now
(an maybe new one if the user archives them),
so, there may be easily some thousands chats in "Archived Chats" -
and the count has no real meaning to the user,
esp. as we not even display the number of "Real Chats".
Might be that this "Link" will go away anyway sooner or later,
however, for now, it is just fine to remove the badge counter.
* add a test for archived requests
* fix archived requests
* move requests but the last one to "Archived Chats"
this way, the app looks familiar after the contact request upgrade.
the subselect was copied from the old get_last_deaddrop_fresh_msg()
(which was removed by the contact request upgrade #2514)
* just move all old requests to "Archived Chats"
ux-wise, the advantage of keeping the last one is questionable,
one may think, always the last one is shown in chatlist.
showing _all_ fresh request is not doable
as past cores did not really take care of that
and the db-state is not consistent in that regard.
that would make the already complicated code even more complicated,
so we decided to go the easy way.
* accounts: update EventEmitter on add_account
* accounts: do not lock waiting for EventEmitter in dc_accounts_add_account
Otherwise dc_accounts_add_account blocks until an event arrives on
some existing account.
Contact request chats are not merged into a single virtual "deaddrop"
chat anymore. Instead, they are shown in the chatlist the same way as
other chats, but sending of messages to them is not allowed and MDNs
are not sent automatically until the chat is "accepted" by the user.
New API:
- dc_chat_is_contact_request(): returns true if chat is a contact
request. In this case option to accept and block the chat via
dc_accept_chat() and dc_block_chat() should be shown in the UI.
- dc_accept_chat(): accept contact request and unblock the chat
- dc_block_chat(): decline contact request and block the chat
Removed API:
- dc_create_chat_by_msg_id(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_marknoticed_contact(): deprecated 2021-02-07 in favor of
dc_decide_on_contact_request()
- dc_decide_on_contact_request(): this call requires a message ID from
deaddrop chat as input. As deaddrop chat is removed, this call can't
be used anymore.
- dc_msg_get_real_chat_id(): use dc_msg_get_chat_id() instead, the
only difference between these calls was in handling of deaddrop chat
- removed DC_CHAT_ID_DEADDROP and DC_STR_DEADDROP constants
* escape strings added to html
* use more common emojis for connectivity report
all emojis are from 2010 and older now.
an alternative would have been to use css,
however, that may have other issues
and as the whole report is subject to change anyway,
i go for the easy solution.
* use 'modern' meta pattern, remove unused div and styles
* use css instead emojis; looks better that way
also, we have the same look on all systems.
* add connectivity command to repl tool
fix https://github.com/deltachat/deltachat-core-rust/issues/2463fix#2116
The email could be private (i.e. only sent to me) or non-private (i.e. also sent to sb else). Also, it could be a classical email or a chat message. Below, I'm describing for each of the four combinations whether they should be assigned to a chat by create_or_lookup_group() or lookup_chat_by_reply(). Because I needed to use these function names a lot, I shortened them to l:group() and l:by_reply() in this PR description.
(!) means todo, (! -> Done) means I fixed something.
## Private classical email:
l:group() and l:by_reply() must both take care not to put it into group
l:group() no (! -> Done), l:by_reply no (! -> Done)
except for classical MUA replies to two-member-groups, they should be put into the parent group
### wrt alias-support:
A private classical email is very probably not going to be an answer to email that went to an alias address:
Suppose Alice writes to support@example.com.
support@example.com forwards to bob@example.com and claire@example.com.
When Bob answers, he will _probably_ answer `To: alice@example.com, support@example.com` (=> it's a non-private classical email).
With this PR, if he does only answer `To: alice@example.com`, (=> it's a private classical email), Alice's DC will show the answer in the private chat with Bob. Which actually makes more sense than showing it in the support@example.com chat I think. Also, if it was shown in the support@example.com chat, then Alice would answer in the support@example.com chat, then Claire would get Alice's message but not Bob's and therefore miss some context.
That being said - **I could change this**. Pretty easily actually, I would just have to remove the call to `is_probably_private_reply()` from `lookup_chat_by_reply()`. Didn't think through all edge cases yet, but should work.
## Private chat message:
l:group() has to put private chat messages into the group: It won't mistakenly put a message there (because it can rely on Chat-Group-Id). Currently, `is_probably_private_reply()` returns true for private chat messages, but l:group still has to put them into the group chat. (it's not nice that the function called is_probably_private_reply returns true for all private chat messages, but I didn't find any nicer solution) l:by_reply() must not assign it to sth special
l:group yes, l:by_reply no
By the way, for chat messages, `try_getting_grpid()` doesn't look at InReplyTo or References, in order not to put private chat-message replies into the group chat.
For alias-support, the same goes as for the private classical emails.
## Non-private classical email:
Just put it into any group, and if there is none, create one.
_Off-topic:
We currently don't look at the recipients lists, which means that a message can easily be assigned to a group although it was not sent to all group members. One day we could somehow compare the recipients list with the members list, but that needs some more discussing (esp. what do we do if they don't match? Create a new group? Show a hint in the UI?) and is nothing for a bug-fixing PR like this one._
l_group yes, l_by_reply yes, even for outgoing messages (! -> Done, this also was the issue reported by @gerryfrancis in the testing group:
Outgoing messages were not put into a chat by In-Reply-To/References, which is correct for chat messages, but for non-private classical outgoing emails, a new ad-hoc group was created everytime.
So, I added this: https://github.com/deltachat/deltachat-core-rust/pull/2465/files#diff-e7606b521f6710ddc6e5236ba5d7eefc917b7ad744b9e71762fd42830c55485bR703-R711)
## Non-private chat message:
Can be put into chat by l:by_reply() because it must be a group message
l:group yes, l:by_reply yes (to make alias support work if the support person uses DC)
Nothing to test or fix here; we have to put chat group messages into the group, which is trivial. And we have to make sure that alias-support works, which already was well tested.
* changelog: update to include prevous changes
and use new format/structure
* changelog: capitalize API
* changelog: add a missing pr refernece
* Apply suggestions from code review
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: Hocuri <hocuri@gmx.de>
See https://support.delta.chat/t/discussion-how-to-show-error-states/1363/10 <!-- comment -->
It turns out that it's pretty easy to distinguish between lots of states (currently Error/NotConnected, Connecting…, Getting new messages… and Connected). What's not that easy is distinguishing between an actual error and no network, because if the server just doesn't respond, it could mean that we don't have network or that we are trying ipv6, but only ipv4 works.
**WRT debouncing:**
Sending of EVENT_CONNECTIVITY_CHANGED is not debounced, but emitted every time one of the 3 threads (Inbox, Mvbox and Sentbox) has a network error, starts fetching data, or is done fetching data.
This means that it is emitted:
- 9 times when dc_maybe_network() is called or we get network connection
- 12 times when we lose network connection
Some measurements: dc_get_connectivity() takes a little more than 1ms (in my measurements back in March), dc_get_connectivity_html() takes 10-20ms. This means that it's no immmediate problem to call them very often, might increase battery drain though. For the UI it may be a lot of work to update the title everytime; at least Android is smart enough to update the title only once.
Possible problems (we don't have to worry about them now I think):
- Due to the scan_folders feature, if the user has lots of folders, the state could be "Connecting..." for quite a long time, generally DC seemed a little unresponsive to me because it took so long for "Connecting..." to go away. Telegram has a state "Updating..." that sometimes comes after "Connecting...".
To be done in other PRs:
- Better handle the case that the password was changed on the server and authenticating fails, see https://github.com/deltachat/deltachat-core-rust/issues/1923 and https://github.com/deltachat/deltachat-core-rust/issues/1768
- maybe event debouncing (except for "Connected" connectivity events)
fix https://github.com/deltachat/deltachat-android/issues/1760
in most (all?) UIs, import/export works on an already created account,
so, dc_accounts_import_account() does not really help here -
but adds some noise and confusion
eg. as for the other dc_accounts_t functions,
the corrsponding dc_context_t functions must not be called.
if really a new account is required for import,
it seems to be easier to call add_account() before import.
Do not process names, avatars, location XMLs, message signature
etc. for duplicate messages.
Previously only `add_parts` was stopped early, but not
`dc_receive_imf`. Also, `dc_receive_imf` processed From: and To:
fields and applied names to contacts even before checking the
Message-ID.
Fake Message-ID generation procedure is changed to operate on raw
header values to avoid interacting with the database.
as by reviving qr codes,
there may be more than one token for a chat,
ensure, the most recent token and only one token is returned
by the sql-command for looking up tokens
(used for generating new codes)
Raw MIME messages may contain non-ASCII characters. Attempting to
store them as TEXT by using String::from_utf8_lossy results in
non-ASCII characters being replaced with Unicode U+FFFD "REPLACEMENT
CHARACTER" which is later incorrectly decoded when attempting to parse
`mime_headers` content into HTML.
Normally they should not end up in contact requests, but if they do,
we want to show them. Otherwise it is completely impossible to see
them until the chat is moved out of contact requests.
Previously chats created by outgoing classic emails went into contact
requests (deaddrop). Outgoing messages are not shown in contact
requests, so created chat was not shown anywhere. With this fix chat
is created both for outgoing classic emails and outgoing chat emails.
Since classical messages are not deleted when show_emails setting is
set to "0" and remain in the database, they should be shown
somewhere. Otherwise they may reappear later when the setting is
enabled again.
New `dc_msg_is_bot()` C API and corresponding `Message.is_bot()`
Python API can be used to check if incoming message is sent by a bot,
e.g. to avoid two echo bots replying indefinitely to each other.
"Bot" flag is not set for outgoing messages, but may be set for
BCC-self messages. For now documentation says that `dc_msg_is_bot()`
return value is unspecified for outgoing messages. It can be better
specified later if needed for specific applications, e.g. sharing an
account with a helper bot.
the default account created by dc_accounts_new()
is annoying when doing a migrate/import account;
in this case, you must not forget to remove the default account.
in between you have two account,
and accounts are swiched.
it seems to be better
to just call add_account() on a new account object as needed.
systemli.org seems not to include Received: header in messages sent to
self, including BCC-self and messages sent to Saved message chat. This
results in all these messages not being displayed on other devices in
multi-device setting. To work around this problem we restrict draft
detection based on Received: header to classical mails.
Delta Chat already implements proper draft detection based on the
\Draft message flag and \Drafts folder flag. However, Thunderbird does
not set \Draft flag when message is stored as "Template". To make
draft detection for Thunderbird more robust, in case Received: header
check is removed altogether later, we check for X-Mozilla-Draft-Info
header which Thunderbird sets both for "Drafts" and "Templates".
probably they come in by the latest sqlx<->rusqlite moves,
however, as they are followed by subsequent migrations,
that should not have been a big bug in the past
(maybe unless the app was killed at a bad moment)
Even if chat has no messages, contacts should not be deleted.
Otherwise chat will reference invalid contact ID which can't be loaded
from the database, resulting in broken contact list in group chats.
`Imap` structure is always created in a configured state now. There is
no default value for `ImapConfig` anymore.
Also resultify Scheduler::start() to fail on database errors, for
example if IMAP configuration cannot be read from the database during
`start_io()`. Previosuly errors during reading keys such as
`mvbox_watch` were simply ignored and folders were not watched until
the application is completely restarted, now start_io() will fail and
scheduler will only be started at the next start_io() call which
usually happens when app is brought to the foreground.
* Add failing test for #2369
* Completely ignore Drafts folder
fix#2369
* Also ignore messages that have the Draft flag set but are not in the Drafts folder
Building aarch64 wheels under qemu takes a long time.
rust 1.52.1 includes fix for cargo that may result in
compilation being stuck: https://github.com/rust-lang/cargo/pull/9201
There was a bugreport for previous versions of DC that BCC-self
messages are not marked as seen if server-side Sieve rule moves them.
This test is an unsuccessful attempt to reproduce the bug,
which was confirmed to be fixed.
This test checks that MDN is marked as seen on ac1. Because ac1 also
receives BCC-self, it is possible that event of marking this message
as seen is confused with marking MDN as seen, and the test passes even
if MDN is not marked as seen. Explicitly disabling BCC-self for ac1
ensures it receives only one message (MDN).
Also start direct_imap as early as possible, before sending the
message. It is possible now, because BCC-self is not sent.
When watch_inbox was off, scan_folders failed and a toast "IMAP operation attempted while it is torn down" was shown.
--
The problem was:
When inbox_watch is off, scan_folders() is called at 244260a978/src/scheduler.rs (L107) but connect_configured() is not called before.
* Add test
* Don't only setup handle, but connect_configured() in scan_folders()
This allows to run `tox `on systems with only a single Python 3
version installed, such as Python 3.9.
CI script scripts/run-python-test.sh specifies `-e py37` explicitly
anyway.
The problem was:
If a message has the \Deleted flag set, we ignore it. But we forgot to
update last_uid, so that uid_next was not updated at 47e639b777/src/imap.rs (L730) and the same message was
fetched over and over again.
Fix#2429
Using `impl AsRef<str>` as the argument instead of `&str` makes it
possible to call the function with `&str`, `String` and other types
that implement `AsRef` trait.
The cost of it is that compiled binary contains mulitple versions of
the same function, one for each variant of types. If function contains
multiple generic `impl AsRef` arguments, the number of versions possibly
compiled into binary grows exponentially with the number of arguments.
Simple way to avoid it is to call `.as_ref()` on the caller side to
convert the argument to `&str`. In most cases even adding a `&` and
relying on `Deref` coercion is sufficient.
This patch changes many functions that accepted `impl AsRef<str>` and
`impl AsRef<Path>` to accept `&str` and `&Path` instead.
In some places `.clone()` calls are removed. Calling `.clone()` on
`String` and passing `String` to a function accepting `impl
AsRef<str>` is completely unnecessary as `&str` reference could be
passed instead. There is no clippy warning against it yet, but
changing argument type to `&str` allowed to find these cases.
The result of debloating is not impressive, several hundred kilobytes
are saved, which is about 3% of the `.so` binary, but the code is
cleaner too.
This moves the module-level lookup and creation functions to the
types, which make the naming more consistent. Now the lookup_* get_*
and create_* functions all behave similarly.
Peraps even more important the API of the lookup now allows
distinguishing failure from not found. This in turn is important to
be able to remove reliance on a ChatId with a 0 or "unset" value. The
locations where this ChatId(0) is still used is in database queries
which should be solved in an independed commit.
This introduces the explicit ChatIdBlocked struct to more explicitly
create a chat with a blocked status. It also adds a common shortcut
to ChatId itself which is more natural to use in many cases.
* Remove sql::error submodule
Use anyhow errors instead.
* Remove explicit checks for open SQL connection
An error will be thrown anyway during attempt to execute query.
* Don't use `with_conn()` and remove it
* Remove unused `with_conn_async`
* Resultify markseen_msgs
- Currently, group images are compressed as well because it was easier to implement that way.
- Currently, in the unlikely case that the avatar is compressed down to 20x20 pixels but still bigger than 20KB, the user doesn't get any indication of this, the avatar simply isn't changed (at least on Android).
If we want to change this, the easiest way is probably to let `dc_set_config()` in the ffi call `error!()` if `Selfavatar` can't be set. The same might make sense for some or all other configs. BUUUUUT: At least Android doesn't show error!() toasts anymore, probably because they were used too often and too spammy.
- The factor by which we scale down if the file is too big is 1.5.
Instead, allow specifying free-form BUILD_ID from the command line.
scripts/remote_python_packaging.sh still uses CIRCLE_ variables to avoid
changing working CircleCI config.
This feature is similar to existing avatar synchronization.
Whenever encrypted BCC-to-self copy of chat message is received, status
setting is updated with the signature of the message.
Rust does not work on manylinux2010 due to old GNU C Library.
We have been using manylinux2014 on CI machine already, but this change
was never commited.
messages are marked as 'noticed' already when the chat is opened
as all UIs call dc_marknoticed_chat();
also marking messages as 'noticed' when dc_markseen_msgs() is called
is not needed therefore - but stands in the way of further improvements
for the deaddrop, eg. UI may let the user decide when the deaddrop
can be removed from chatlist ('All done' button or so)
* Fix#2335 (delete job flooding)
The problem was:
- You are offline, and an ephemeral message is due to delete.
- load_imap_deletion_job() is called and returns a deletion job for it.
- The job fails because, well, we are offline.
- The job is saved into the database.
- load_imap_deletion_job() is called again, and so on.
* Add test
ct.name was insufficient, as authname, overridden name and email address
fallback were not taken into account.
Dropping this condition increases performance by 25% according to the
benchmark.
Also add a test for search_msgs.
It reduces the time by ~20% according to `search_msgs` benchmark.
Sorting by IDs is sufficient for global search as IDs increase in the
order of message reception.
This PR originally contained a fix for sqlx which turned out not not to be necessary. But on the way, I made some general improvements:
- Under some circumstances, a "normal" test failure led to a timeout, without printing a decent error message. See e.g. https://app.circleci.com/pipelines/github/deltachat/deltachat-core-rust/8069/workflows/ba2a9949-b4ad-4ceb-a930-073bba05e2db/jobs/30965.
(The problem was: if there is an exception in dc_account_extra_configure(), when trying to handle the exception the line account.log("===================", e, "===================") was called, which can't work as log() only expects one parameter)
- When a test fails: Call `dump_account_info()` even if there is no direct_imap
- In test_import_export_online_all(), add another 100KB file to the backup. This adds resilience against future size changes of the sql db file: The test tests the smoothness of the progress bar. And if there are there are not enough about-equally-sized files, the progress bar can't be smooth.
the user-given contact name may be set to an empty string;
in this case the authname or the email is used for the contact
and also for the name of possibly existing chats.
this works well for `dc_chat_get_name()` as that just uses
`dc_contact_get_display_name()` for single-chats.
it did not work for `dc_get_chatlist(query)` as that
uses the database for performance reasons -
however, in the database, the empty string is written
instead of the display name is written for a chat.
this is fixed by this pr by also using
dc_contact_get_display_name() when updating the chats-table
(similar to `dc_create_chat_by_contact_id()`)
message type directly (ex. `const.DC_MSG_STICKER`) so if new message types are
added, they can be used direcly without needing the python API to be updated.
This is an PGP/MIME format produced by Microsoft Exchange and ProtonMail IMAP/SMTP Bridge,
described in detail in https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html
This patch adds seamless support for "Mixed Up" Encryption, repairing
mangled Autocrypt messages without notifying the user.
&str queries are not persistent by default. To make queries persistent,
they have to be constructed with sqlx::query.
Upstream sqlx does not contain the change that make all queries
persistent, but it is not needed anymore. but
fix #2073fix#2292 (I think)
- Messages can be assigned to any chat by the References and In-Reply-To, also to 1:1 chats; this has higher priority than the group id because with ad-hoc groups, it can happen that two devices have different group ids for the same conversation thread.
- If `From` is not in the chat (we call this "shadow sender"), `OverrideSenderDisplayname` is set. This communicates to the UI that:
- A `~`should be added in front of the sender's displayname.
- Also in 1:1 chats, the sender's displayname and avatar is shown, as if this was a group.
The "Unknown sender for this chat" messages are completely removed for unprotected groups.
For protected chats, everything stays as it was before.
POSTPONED:
- Maybe (if it turns out to be still necessary):
- The ad-hoc group id is computed by the the References, instead of the member list, as it is currently done
- How do we prevent that the message can't be assigned to the correct chat as the parent message was deleted?
Switches from rusqlite to sqlx to have a fully async based interface
to sqlite.
Co-authored-by: B. Petersen <r10s@b44t.com>
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: link2xt <link2xt@testrun.org>
fix#2188
Also, if we notice that the server started reusing old UIDs, _also_ do a `ResyncFolders`, because the server likely forgot to change uid_validity
`DoxygenLayout.xml` is picked up by doxygen automatically (as `Doxyfile`)
and its template can be generated by `doxygen -l DoxygenLayout.xml`.
after that, the following tweaks were done:
- remove all xml-parts but header (other defaults are fine so far)
- rename "Modules" header to "Constants" (there are no modules :)
- remove useless subitems from "Classes", eg. by name only shows as "d"
- reorder tabs so that "Classes" becomes the first one
save subject for messages:
- new api `dc_msg_get_subject()`,
- when quoting, use the subject of the quoted message as the new subject, instead of the
last subject in the chat
usually, there is at most one text/html and one text/plain part.
multiple text/plain parts,
are known only by mailinglist software that wants to add a footer
but cannot alter the original content part -
maybe because it is html and things are hard to stitch
or maybe because the content part is cryptographically signed
and cannot be altered therefore.
the test has multiple text parts because
the content text is signed and
the mailinglist software wants to add some text
(and cannot alter signed part without breaking the signature)
fix#2254: if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message), the db was corrupted.
* Add result.log() for logging with less boilerplate code
* Bugfix: Resultify housekeeping() to make it abort if the db is closed instead of just deleting everything
* Require the UI to call dc_stop_io() before backup export
* Prepare a bit better for closed-db: Resultify get_uidvalidity and get_uid_next and let job::load_next() wait until the db is open
About the bug (before this PR):
if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message):
- I don't know if it downloads the message, but of course at some point the process of receiving the message will fail
- In my test, DC is just in the process of moving a message when the imex starts, but then can't delete the job or update the msg server_uid
- Then, when job::load_next() is called, no job can be loaded. That's why it calls `load_housekeeping_job()`. As `load_housekeeping_job()` can't load the time of the last housekeeping, it assumes we never ran housekeeping and returns a new Housekeeping job, which is immediately executed.
- housekeeping can't find any blobs referenced in the db and therefore deletes almost all blobs.
blocked mailinglists addresses are added to the contact table
before blocked contact list is created -
(this allows unblocking of blocked lists in previous testing releases,
however, more importantly, it keeps all blocking/unblocking code inside
contacts)
on unblocking such a contact,
the corresponding chat is unblocked as well.
* deprecate mostly unused dc_get_blocked_cnt() api
instead, the size returned by get_blocked_contacts() should be checked,
this is safer and allows easier adaption of blocking rules.
ui or python seems not to use dc_get_blocked_cnt(),
however, there is one test in node,
therefore, the function will continue working for now
(by just returning Contact::get_all_blocked().len() then)
* add decision api to repl tool
* add block/unblock api to repl tool
* unify usage of @deprecated doxygen command
outlooks SMTP-server change the Message-ID of messages
and put the original Message-ID to X-Microsoft-Original-Message-ID.
the changed Message-ID has some issues:
- outgoing messages with bcc_self enabled are shown twice
as the self-copy got the changed Message-ID while the database uses
the original one
- read receipts do not work as they refer to the changed message id
- in general, sender and recipient see different Message-IDs
the issues can be fixed by
(1) let all receivers use the original Message-ID
this is what this pr is doing
and this should fix all issues with delta-to-delta communication,
including groups, group-images etc.
there may be issues left in communication
with other MUAs as they are using another Message-ID.
(2) ftr: updating the Message-ID in the database of the sender to the new one
this requires bcc_self always enabled (which is not the case)
and may also result easily in race conditions
(Bob answers before Alice sees its self-sent message),
however, has the advantage of better compatibility with other MUA.
if needed, the compatibility with other MUA could be improved by remembering
both Messages-IDs, maybe we could treat the modified as References or so,
however, i think, this could be part of another PR if we know better about
real, in the wild issues.
The chat::lookup_by_contact_id call is already resultified, the
database can not contain this since the auto-increment counter is
bumped to 7 by the time the database tables are created.
* add test for mailchimp mailinglists
* pass MimeMessage to create_or_lookup_mailinglist() (as of the other create*() routines) to allow more flexible name processing; document the function
* get mailing list name for mailchimp from From:-header
* make clippy happy
* add comment to '.list-id.mcsv.net' suffix
Now I know why the tests failed before 48c58a7 (i.e. after c923670) but
not on master (i.e. before c923670):
because of a bug, scan_folders() set the configured_sentbox to None if
it was set before. If it was None before, it restored the correct value.
On master, there was another bug that led to two runs of
scan_folders() being started at the same time. Therefore, the first run
set configured_sentbox to None, the second one restored the correct
value.
c923670 fixed the latter bug, so that only one run of scan_folders() was
started. Therefore, configured_sentbox stayed None incorrectly and
test_fetch_existing() failed.
48c58a7 fixed the former bug.
This commit adds checks to test_fetch_existing(), so that it definitely
checks for the former bug.
This does fix a bug and it makes the tests pass, but I'm not sure why it
makes the tests pass; maybe there is a race condition that made the
tests fail and my commit just leads to another timing.
This will allow MimeFactory.render() to put protected headers
into main_part and wrap it into single-part multipart/mixed if
protected headers are too large to put into the outermost IMF.
This should make colors used by Delta Chat for emails similar to colors
used by XMPP clients implementing the same specification[1], such as
Conversations and Snikket.
Previously Delta Chat used only hardcoded 16 colors, so this change
should also increase the number of colors and make it easier to
distinguish different contacts.
[1] https://xmpp.org/extensions/xep-0392.html
This changes the internal stock strings API to be more strongly typed,
ensuring that the caller can not construct the stock string in the
wrong way.
The old approach left it to the callers to figure out how a stock
string should be created, now each stock string has their specific
arguments and callers can not make mistakes. In particular all the
subtleties and different ways of calling stock_system_msg() disappear.
This could not use a trait for stock strings, as this would not allow
having per-message typed arguments. So we needed a type per message
with a custom method, only by convention this method is .stock_str().
The type is a enum without variants to avoid allowing someone to
create the type.
Sadly the fallback string and substitutions are still far away from
each other, but it is now only one place which needs to know how to
construct the string instead of many.
1.51.0-alpha.0 is somewhere between 1.50 and 1.51,
as various beta-versions come or already came out with the new core,
this is helps in logs/debugging.
The following rules apply now:
1. "name" column is only updated manually and never over the network
2. "authname" column is only updated over the network and never manually
3. Displayname is a "name" if it is non-empty, otherwise it is "authname".
This fixes a known (pytest.xfail) problem of "name" being changed over
the network when user has set it to non-empty string manually.
This also fixes the problem when "name" and "authname"
became unsynchronized accidentally, when they were equal and then
Origin::IncomingUnknownTo update arrived, setting "name" but not
"authname". Rust regression test is added for this case.
renaming mailing lists and setting mailinglists images
(possible locally, but not synced)
was blocked because the function real_group_exists() checked
the chattype on a numerically.
it is not 100% clear to me,
why this function exist at all,
it just checks if a record with type=120 (group) exists under the given id -
this is the same as loading the chat and check chat.typ afterwards.
might be a premature optimisation relict from core-c
or made some error handling easier there.
This does not affect the release profile and it significantly speeds
up compilation. In the rare cases where panics are not sufficient to
debug something you can manually enable this again.
The get_chat_id_by_grpid() call is already resultified. The database
can only contain consistent data since on creation it inserts rows for
the special chats which do not have a grpid set so can never match.
Thus we can safely remove another use of the ChatId::is_unset call
which should be removed to clean up the type.
* Copypaste-merge my old work
* Start implementing mailinglists the new way
* Create pseudo contact
* Fine-tune docs
* Remove some unnecessary changes
* style
* Make a stock str
* Fix a crash. Yes, this line caused a panic when reconfiguring on Android
(without a reasonable error log). Also, I could not receive any messages
anymore.
* rfmt
* Add tests and make them pass
* Even more tests
* rfmt
* Enhance test and fix bug
* Don't update the sender name when prefetching because maybe it's a mailing list
* Use an enum instead of numbers for the decision
* Don't remove anyone from mailing lists
* Fix bug in the regex
* Adjust error msg
* Compile error after rebase
* Correctly emit event
* Add dc_msg_is_mailing_list so that you can find out whether messages in the deaddrop belong to mailing lists.
* Add received headers to unit tests
* Comments, small tweaks
* Use dc_msg_get_override_sender_name instead of dc_msg_get_sender_name
* Add dc_msg_get_sender_first_name() because sometimes the first name was not correctly shown in mailing lists
* small fixes, don't let the user modify mailing list groups
* Hide contacts for addresses like noreply@github.com and reply+AEJ...@reply.github.com
When testing mailing lists, I noticed that sometimes a mailing list
contact got a name (like, hocuri <noreply@github.com>). It turned out
that ages ago, I had accidentally written an email to - in this example
- hocuri <noreply@github.com> and it had been added to the contacts
list.
This hides email addresses from the contacts list that are obviously not
meant to be written at and prevents updating the names.
* Comment, clippy
* Replace u32 with ChatId
* Resolve lost of small issues from the reviews
* remove dc_msg_get_sender_first_name
* add dc_msg_get_real_chat_id()
this allows to check if a contact request belongs to a mailing list
and to show name/avatar/whatever of the mailinglist.
another approach was to duplicate some chat-apis for messages
(eg. dc_msg_is_mailing_list()) however that would require far more new apis.
the idea to change the behavior of dc_msg_get_chat_id() would be
more clean, however, that easily breaks existing implementations
and can led to hard to find bugs.
* remove now unused Message.is_mailing_list()
* if a name for a mailing list is missing, use List-ID
* fix comment
* fix error message
* document how dc_get_chat_contacts() works for mailing lists
* refine decide api (#2185)
* add DC_DECIDE* constants to deltachat.h, tweak documentation
* use StartChat/Block/NotNow instead of Yes/Never/NotNow
* decide_on_contact_request works on ctx/msg-id
functions working on message-objects
usually do not read or write directly on the database.
therefore, decide_on_contact_request()
should not work with message-objects as well,
it is even a bit misleading, as eg. chat-id of the object is not modified.
instead, the function works on context,
similar to dc_send_msg(), dc_create_chat() and so on.
for now, i moved it to context, could maybe be part od MsgId.
* Update src/chatlist.rs
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: Hocuri <hocuri@gmx.de>
* refine documentation
* re-add accidentally deleted Param::MailingList
* remove pseudo-contact in domain @mailing.list
1. the pseudo-contact was added to the mailing list contacts,
which is not needed.
might be that we want to add the recent contacts there in a subsequent pr
(we do not know all contacts of a mailing list)
2. the pseudo-contact was used to block
mailing lists; this is done by setting the chat to Blocked::Manually now
3- the pseudo-contact was used for unblocking;
as it is very neat not to require additional ui for mailing list unblocking,
might be that we introduce a similar pseudo-contact for this limited purpose
in a subsequent pr, however, the pseudo-contact needs to exist only
during unblocking then, maybe also the special domain is not needed,
we'll see :)
* Move dc_decide_on_contact_request() up to the dc_context section as it's a member of dc_context now
More specifically, to the "handle messages" section
* re-introduce Chattype::Mailinglist (#2195)
* re-introduce Chattype::Mailinglist
* exhaustive chattype-check in fetch_existing_msgs() and get_summary2()
* exhaustive chattype-check in ndn_maybe_add_info_msg()
* exhaustive chattype-check in message::fill()
* remove dc_chat_is_mailing_list() from ffi
Co-authored-by: B. Petersen <r10s@b44t.com>
some videochat-services do not support adding random rooms to the url,
users let people in manually, so they can always use the same url.
as hacks as hiding the genrated random room behind `#$ROOM` or `?=foo=$ROOM`
does not always work, this pr introduces the parameter `$NOROOM` that is
just replaced by the empty string.
see https://support.delta.chat/t/videochat-with-webex/1412
for further details.
Return error for special IDs. Previously "No encryption." was returned for DC_CONTACT_ID_SELF.
Show "No encryption." if peerstate is reset.
Never talk about transport encryption: it is misleading. Even if TLS is used for client IMAP and SMTP connections, MTA-MTA connection may be unencrypted if MX does not support STARTTLS.
Add a Rust test.
* add failing test for From:-header bug
the test fails on the last check for quote-encapsulated utf-8 strings.
* Update mailparse
Co-authored-by: link2xt <link2xt@testrun.org>
Param should only be used to parse dictionaries stored in SQL database, not external data.
Since Param parser has been extended to escape newlines with newlines, QR-codes parser started to treat && as escaped &, which is wrong.
This change also uses String keys like "n" for "name" instead of completely unrelated constants like Param::SetLongitude.
- doc fixes
- make BobStateHandle safer by moving the state out of the handle.
- handle more match cases explicit in BobState returns
- fewer mutable variables
This does not only organise things better, but most importantly the
BobStateHandle is now not in the same module as its users. This means
it can guarantee safety about how it is initialised and use
unreachable!() to simplify it's API.
This introduces a state machine which takes care of managing the
handshake transitions in the secure-join protocol. This separates
user interactions from the protocol state handling.
This means that while handling the protocol state there are a bunch of
failures no longer possible due to all state information being
guaranteed to be present. As part of this the QR-code state has been
extracted from the generic Lot structure to something suitable just
for the SecureJoin protocol.
A LogSink has been added to the testing tools allowing log messages to
be correctly displayed on test failures.
* draft dc_msg_set_html() api
* implement setting 'html to be send'
* test sending html-parts
* more flexible html-partbuilder
* write html-parts to database and also send them
* add 'sendhtml' command to repl tool
Encryption preference was already applied since commit 78d855c5ca,
but only for already existing peerstates.
As a result, new users ignored gossiped encryption preference in the
"member added" message and had to wait until someone gossiped encryption
preferences for inactive users the second time.
This should help if parent message can't be found because messages were
reordered or deleted.
This does not prevent group IDs from being removed from Message-IDs in
the future, in which case it will become dead code.
This removes the proxy via crate::error to depend on anyhow directly.
There is no benefit to this indirection and this makes it simpler to
see which error types are used.
* add failing tests for forwarding html-mails
* let MsgId.get_html() return an option
* write html-part to local database on forwarding
* add html-part to forwarded non-dc messages
* read HTML-parts from encrypted messages
* avoid clone()
* Received:-header is no longer needed since #2152
* Update src/html.rs
Co-authored-by: Floris Bruynooghe <flub@devork.be>
* Update src/html.rs
Co-authored-by: Floris Bruynooghe <flub@devork.be>
* Update src/mimeparser.rs
Co-authored-by: Floris Bruynooghe <flub@devork.be>
* prefer 'orig' over 'org' as abbreviation for 'original'
* improve comment on tests
* prefer 'try_into()' over 'as u32' to avoid panics on bad data
* simplify ffi
Co-authored-by: Floris Bruynooghe <flub@devork.be>
instead, dc_contact_get_display_name() should be used.
dc_contact_get_first_name() was created to save some space on the screen,
esp. on mobile devices,
however, this does not always work and has issues on its own
with some names ("Dr. Strangelove", ":) Name" and so on).
as with mailing lists, more apis with first_name() would be needed,
we decided to drop that instead of following that way.
it is also less an issue as some years ago as screens have become larger,
if really, needed, the ui can handle that more gracefully,
however, just using dc_contact_get_display_name() should be fine as well.
We don't really have docs in the other crates yet, and there's a
naming conflict between deltachat and deltachat_ffi that needs to
sorted to build both. But deltachat_ffi is still documented with
doxygen for now so there's no need to sort this now.
This also checks that some of the correct user interactions happen,
checking we get a joiner event and the verified chat messages.
It also extends the test utils with the ability to distinguish the
different context logs by having them named.
Lots of new clippy lints due to toolchain upgrade.
Made the Message::error field pub(crate) again, it was the odd one out
and it seemed a reasonable way to shut up clippy.
Notable improvements:
- [track_caller](https://github.com/rust-lang/rfcs/blob/master/text/2091-inline-semantic.md): We can get the location of the caller of a message, which makes logging easier as you can have dedicated functions (not only macros), e.g. write a function `log(self: Result, msg: string) -> Option`
- The output after a panic is shorter and hides lots of uninteresting
function calls
- Test output is better captured
After merging, we should probably wait a bit with using the track_caller
feature to make going back easier, should we get problems again.
It is misleading and should never be checked.
dc_stop_io() also stops pending ephemeral message deletion tasks, so it
should be called in any case before releasing context.
Start implementing #1994
TODO (in later PRs):
- Add a hint to the watch settings that all folders are fetched from time to time (to be done in the individual UIs)
- folder names are case-insensitive, so double-check that all comparisons are case-insensitive
- The `scan_folders.rs` file didn't get as large as I expected and it's probably not worth it having an extra file for it. But if there are no objections, I'll make another PR to rename it to `folders.rs` and also put into it `configure_folders()` from `imap/mod.rs` and `needs_move()` with all its tests from `message.rs`.
Done:
- Most mailboxes have a "Drafts" folder where constantly new emails appear but we don't actually want to show them, what do we do about this? The most reliable way to detect such messages that we found up to now is:
If there is no `Received` header AND it's not in the `ConfiguredSentbox`, then ignore the email.
- before or after INBOX idle trigger a new "scan all folders for messages". It does a "list folders" and then goes through all folders with select-statements, checking if "next-uid" was changed since checked last time. This might be batchable but in any case should not consume a lot of traffic. We might debounce this scan activity to happen at most every N minutes
- if next-uid changed for a folder, we "prefetch" and "fetch" DC-messages as as needed ("dc-messages" are not just those with "Chat-Version" headers, but can also be regular emails)
- if we discover DC-messages in folders that have the "/Spam" flag (maybe excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
- for now, we do not change any user visible option, but introduce this "scan all" automatically and on top of what exists. The DeltaChat folder-watching does not perform scan-all-folders (maybe with the exception to trigger scan-all also with DeltaChat if INBOX is not watched)
- Tests (except if you have ideas to improve them)
- all folders, their last uidvalidity, next-seen etc. are kept in a separate "imap-sync" sqlite table. Maybe this can be used to streamline some of the "Sent" folder and "DeltaChat" folder detection code we already have.
- We now also move self-sent messages from the Inbox to the Sent folder if `mvbox_move` is off, as this was very easy to do now. This way, we now behave more like a normal MUA if the user wants this.
FOR LATER PRs:
- maybe for the first 50 messages or so, we could reduce the IDLE-timeout (currently 23 minutes or so) to faster detect messages sent to non-inbox folders. However, on Android and iOS, we would likely trigger scan-all when the app moves to foreground, and so it might not be neccessary to reduce the current idle-timeout at least for them. We can leave this "faster discovery" question for the end, after we move to real-life testing.
- (Later on, after the above works, we can consider heuristics on which folders to perform IDLE on, and remove the Watch-folder options (inbox, deltachat, sent). We tried to find a safe scheme for already doing it but failed to fine one, too many unknowns, also some questions regarding multi-device (you might have different settings with each of it, one moves, the other doesn't etc.) so we postponed this in favor of the above incremental suggestion.)
* Start implementing #1994
* Add debug logs, it seems like the SQL migration can go into another pr
* Let fetch_new_messages return whether there are new emails
* Code style
* Don't prefetch if there are no new emails
* clippy
* Even more debug logs
* If the folder was not newly selected, return always try to fetch as
uid_next is probably outdated
* Fix new bug
* Recognize spam folder
* if we discover DC-messages in folders that have the "/Spam" flag (excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
* Clippy, prioritize folder_meaning over folder_name_meaning
* Add a first test, for the first day after installation only debounce to 2s
* Start adding two tests (both of them fail)
* Don't abort folder scan if one folder fails
* More consts
* Replace bool return value by enum
* Split test up into multiple tests
* Print logs during rust tests
* Rust tests pass now
* .
* One of the Python tests passes now - reconfigure folders during scanning
* Make the last test pass - Delete emails in all folders when starting the test, not only inbox and mvbox
The problem had been that emails were left in the folder "xyz"
* lint
* DB migration (untested)
* Store uid_next in SQL instead of lastseen in a config
* Revert "If Inbox-watch is disabled and enabled again, do not fetch emails from in between"
all folders are always watched, anyway
* clippy, rm debug logs, comments
* Codestyle, comments
* fixing things again
* Fix another test: don't fetch from uid_next-1 but uid_next; make some {} to {:#} so that we can use `.context(...)`
* move self-sent, non-setupmessage chat messages to the Sent folder if `mvbox_move` is off
* comment
* Comments, make sure things work even if there is no uid_next
* Style
* Comments
* The rust test tested wrongly
* comments, small codestyle change
* Ignore emails that are probably only drafts
Most mailboxes have a "Drafts" folder where constantly new emails appear
but we don't actually want to show them.
So: If there is no Received header AND it's not in the ConfiguredSentbox,
then ignore the email.
Also: Add test.
* Fix occasional test failure, it was introduced as DC now moves messages from Inbox to Sent
* Add `Received` header to the rust tests
* After this PR we will always watch all folders and delete messages there if server_delete is enabled. So, for people who have server_delete on, disable it and add a hint to the devicechat
* comment, small fix
* link2xt's first review
* Use ON CONFLICT(FOLDER) DO to update and if it doesn't exist, then insert
Reason from link2xt: We had a problem with multiple peerstates inserted due to key fingerprint parsing error previously. With logic in Rust a similar problem can occur: an UPDATE can fail for reasons other than a conflict. PRIMARY KEY should ensure uniqueness in this case, but anyway.
* Remove two TODO statements, remove fetch_new_messages: ignoring uid {}, uid_next was {} log
* Next TODO: Make uidvalidity and uid_next DEFAULT 0
* rm two TODOs, Seems like we are not going to `exclude folders that are watched anyway` in this PR
* small tweak: Handle instants more carefully
* Add scan_all_folders_debounce_secs config for tests, set debounce to 60s (before it was just 2s during the first day)
* Don't use bold letters for the device message
* React to changes in the folders better
Before, if there was a configured Sent folder, but then it got
removed and replaced with another folder with a name meaning "Sent" but
without Sent flag, it would be ignored.
So, instead of checking against ConfiguredSentboxFolder,
create two Option variables at the beginning of the loop and replace
them with Some if it is None. At the end of the loop, store the new
values into ConfiguredSendboxFolder and ConfiguredSpamFolder, even if it
is None.
Also, derive some useful traits.
* move job: Return a meaningful error if server_folder is None instead of panicing
* small error-handling fix
* Fix test_fetch_existing() python test
Before, we sometimes got a race condition where scan_folders() sees that
there is a Sentbox and saves this info after we set the
ConfiguredSentbox to None and before the message is sent.
So, just expect that the message is moved to the sentbox.
* migration is 72 now
* rm 2 TODOs, Don't infinitely retry when dc_receive_imf() returns Err
* clippy: Remove glob imports
* Delete created folders at the beginning of tests
(some created folders made problems in the next tests because)
* Improve resetting accounts between tests
* draft API to deal with uncut message texts
* add column mime_modified
* add mime_modified flag to MimeParser and save it in the database
* save mime_headers also when mime_modified is set
* cargo fmt
* set mime_modified on parsed html-texts and when there are multiple alternative-parts; add test for that
* prototype functions, add to repl and ffi
* use correct mime_modified flag
* basically parse Mime-Structure to HTML
* add basic tests for HTML-parsing
* convert text/plain to html for getting original
* respect charset for plain texts
* make test more specific
* fix handling non-utf-8 charsets for plain messages
* add test for plain_to_html()
* add failing test for plaintext linkify
* linkify urls in plain text
* fix regex
* plain text linkify: add failing test for encapsulated links as <https://domain.com>
* plain text linkify: make encapsulated links as <https://domain.com> work
* plain text linkify: require word boundary at beginning of link, add tests for that
* plain text linkify: linkify emails
* plain text: support format=flowed
* plain text: support quotes
* make clippy happy
* set mime-modified also when simplify() cuts non-html messages, add tests for that
* streamline mime recursion
* repl tool: write original html to file for further processing
* convert cid:- to data:-protocol
* add a test for cid: to data: conversion
* make clippy happy
* fix html-tests to work with windows-lineends
* clarify what the returned html-code may contain
* add some more detailed doc comments
* add mime_modified column only if not exist
this additional check is needed
as the column may added with another dbversion in
some shipped beta-versions.
* incorporate documentation suggestions from review
* rename get_original_mime_html() to more simple get_html()
* rename api is_mime_modified() to more simple has_html(); internally, mime_modified-flag stays as-is, however
* rename MimeS to MimeMultipartType
* do not set mime-modified flag for encrypted messages that need extra-handling for saved mime-structure
* fix typo
* move get_msg_html() to MsgId.get_html()
* incorporate more documentation suggestions from review
* remove unused return value from collect_texts_recursive()
* avoid mime_modified being mutable in write-parts-loop
* move 'use futures::future::FutureExt' atop of html.rs
* move attributes defining plain-text to a dedicated structure
* more PlainText to separate file
* escape cid when building regex
* let dc_get_msg_html() return NULL when calling with bad param
in addition to merged pr #2123,
dbversion=70 was also used by pr #2125.
pr #2125 is not yet merged, however beta-versions were shipped with
that pr, so, dbversion=70 is already in use on some systems and on these
systems ConfiguredProvider won't be set.
the other way round, that ConfiguredProvider is set twice on some systems
affects only dev-installations and should not harm.
once this pr is merged, #2125 needs a rebase and will use dbversion=72 for
its changes to add the new column also n the dev-installations :)
(we need to check that the column to add does not exist, however)
Previously MX records were queried only for OAuth 2 configuration and
did not affect the list of servers tried. User was required to manually
configure the servers for Google Workspace (former GSuite) domains.
Now MX records are queried during configuration. If provider is found in
offline database, its ID, corresponding to the filename, is saved as
`configured_provider`.
`configured_provider` is also set during database migration if email
address uses the domain from the provider database, but no MX querying
is done.
- Make sure delete_expired_messages and housekeeping runs once a day
- delete more info about messages when putting them to trash (esp. also
delete txt_raw, from_id and to_id as we don't need those anymore, so
they are data that was unnecessarily kept)
fix#1926, fix#2090
Also:
* Nicer test_utils: add send_text() and print_chat()
* Adapt ephemeral messages for testing (make them accurate to the second)
* Add test for ephemeral messages
* Make pop_sent_msg() really pop the last sent message
If mvbox_move was on and an mdn was received, it was not marked as read.
Also remove a confusing log that said that we are adding a markseen job, even if we weren't. As job::save() logs itself, there is no need to log this again.
adding an asterisk to the filename key
REQUIRES two apostrope-delimiters in the value as of charset'lang'name,
see rfc2231
as we do use the value this way, we MUST not add the asterisk.
things worked correctly in the past as
consuming MUAs are pretty graceful.
i assume the error comes from the c to rust conversion -
core-c did add the asterisk, however also did the apostrope-encoding.
core-rust leaves the asterisk but changes encoding.
disabled and re-enabled
That's another narrow-stitching patch for a scenario where many emails could
be deleted all at once and surprisingly: user disables inbox-watch, enables delete-from-server, after a moth enables inbox-watch again -> currently all emails that arrived in the meantime will be deleted (if emails are not fetched, they won't be deleted)
This way we can better distinguish tagged from untagged core releases, also in logs etc.
We might, from time to time, increase the alpha.N "N" number if we are entering testing etc.
providers may have a maximum for the SMTP RCPT TO: header,
therefore, an outgoing message is split into several chunks.
the chunk size defaults to 50, however, if known to be larger or smaller
by a dedicated provider, this setting may be changed.
this does not affect the MIME To: headers,
if a provider has a maximum here, this is change would not help.
ephemeral_task holds a reference to Context, preventing event emitter
from returning NULL and terminating event loop. Prior to this change,
there was no way to quickly terminate pending ephemeral_task.
the saved-messages can be deleted as any other chat;
if the feature is unneeded, this probably also makes some sense.
however, if saved-messages was deleted accidentally (yeah, i've seen that :),
it is not intuitive to see how the saved-messages feature can be restored.
this change just drops a note about how-to-restore to the device chat.
fix#2020
When forwarding a message, the original `in_reply_to` stays in place.
I did not find any recent commit that changed anything about this,
but IIRC there was a bug that prevented setting the `in_reply_to` at
all.
So, for chat messages, do not look at the InReplyTo header (and also not
at the References header)
by #2001, processing of protection-enabled and protection-disabled
messages is only done if verification-checks pass.
unfortunately, these verification checks
were only applied on already protected chats,
so that enabling via a protection-enabled message was not possible.
this is fixed by this pr.
moreover, when inner_set_protection() fails,
the error is shown in the chat.
this avoids confusion - as long the current core
does not communicate with a UI that enables/disables protection,
everything looks just the same as in the past :)
update_device_chats() takes about 2 seconds on a modern device (Android) because the
welcome image file has to be written to the disk as a blob. The problem
was that this was done after the progress bar had vanished and before
anything else happened so that I thought that something had gone wrong
multiple times.
The UIs have to remove update_device_chats(), too..
this would exclude non-deltas from protected-chats
(where they could participate in verified-groups in the past)
and makes migration a bit harder when using a fuzzy multi-device-setup.
however, the test still takes place
when a group is created or protection is enabled/disabled.
should_encrypt() shall return Ok(false)
on peerstate=EncryptPreference::Reset
only for opportunistic groups.
for verified/protected groups (e2ee_guaranteed set),
Ok(true) or Error() should be returned.
this bug was introduced by #1946 (Require quorum to enable encryption).
a draft may contain only a quote,
without any text set yet.
these drafts cannot be sent, however, appear in the summary -
currently with the summary-text "", which results to sth.
as "Draft: " - which looks like an error or at least a bit weird.
this pr sets the summary text to "Reply" - similar to "Image", "Video" etc. -
the UI just expects some text here, not an empty string.
the result are summaries as "Draft: Reply" on all UIs -
which, btw. is also roughly the same what Signal does in this case.
the constant comes from c-core
and was used to define the size of the string-array that time.
this is no longer needed in core
and also the UIs seems not to use it
(they will treat DC_STR* like an id,
there should be no need to know the max. DC_STR* value)
Read all of an e-mail accounts messages and extract all To/CC addresses
if the From was from our own account.
Then, fetch existing messages from the server and show them.
Also, I fixed two other things:
- just by chance my test failed because of an completely unrelated bug.
The bug: bcc_self messages were not marked as read if mvbox_move was set
to true.
- add some color to the test output (minor change)
Sticky encryption rule, requiring that all replies to encrypted messages
are encrypted, applies only to messages with a quote now.
Co-Authored-By: B. Petersen <r10s@b44t.com>
This code is inconsistent with EncryptHelper::should_encrypt, which uses
quorum rule.
UI does not display or use encryption state in-between preparing and
sending message anyway.
In addition, "sticky encryption" rule, which required all replies to
encrypted messages to be encrypted, is dropped. It is going to be
restored with the introduction of quoting.
both are not used in productive:
- chat-id "in-creation" was dropped completely already in core-c
as it does not allow assigning messages to chats in time.
- message flag "starred" was added at some point and never used.
ux-wise, "disappearing messages" promises to clean up a chat -
this does not fit well with "starring" messages.
"saved messages" chat seems to be superior in that regard.
but anyway, it is not used but comes at maintainance costs,
so it is better to delete its functionality for now.
(the db entries, however, are left for now,
that might be more tricky to remove)
Address is used to generate Message-Id.
Previously error toast was shown, but MsgId was returned from
chat::prepare_msg anyway. Now chat::prepare_msg returns an error.
error!() is removed, because FFI library shows errors via
.unwrap_or_log_default().
When run against host that is not in the provider database or has multiple
IMAP servers listed, this test fails because one error is returned for
each server.
* Mention in dc_event_get_data2_str method that it can also return 0
* Update deltachat-ffi/deltachat.h
Co-authored-by: bjoern <r10s@b44t.com>
Co-authored-by: bjoern <r10s@b44t.com>
Previously, standard Autocrypt rule was used to determine whether
encryption is enabled: if at least one recipient does not prefer
encryption, encryption is disabled.
This rule has been problematic in large groups, because the larger
is the group, the higher is the chance that one of the users does not
prefer encryption.
New rule requires a majority of users to prefer encryption. Note that it
does not affect 1:1 chats, because it is required that *strictly* more
than a half users in a chat prefer encryption.
partly, we wrote '@return None.' for void functions,
however, some linter complain about that (recently swift)
and they're right, it does not make much sense.
On Android, there is to /tmp directory, so SQLite may throw
SQLITE_IOERR_GETTEMPPATH when trying to find a place for temporary
files. Storing temporary files in memory is a workaround that always
works (until out of memory).
See https://github.com/mozilla/mentat/issues/505 for discussion of a
similar issue.
In the current design the dc_receive_imf() pipeline calls
handle_securejoin_handshake() before it creates the group. However
handle_securejoin_handshake() already signals to securejoin() that the
chat exists, which is not true.
The proper fix would be to re-desing how group-creation works,
potentially allowinng handle_securejoin_handshake() to already create
the group and no longer require any further processing by
dc_receive_imf().
'Delete emails from server' was an experimental ad-hoc function
that was added before 'Automatically delete old messages' was introduced
to allow at least some way to cleanup.
'Delete emails from server'-UI was already removed from android/ios in june
(see https://github.com/deltachat/deltachat-android/pull/1406)
and the function never exists on desktop at all.
This currently tests the setup-contact full flow and the shortcut
flow. More securejoin tests are needed but this puts some
infrastructure in place to start writing these.
This commit fixes condition which ignores reordered autocrypt messages.
If a plaintext message resetting peerstate has been received, autocrypt
header should only be applied if it has higher timestamp.
Previously, timestamp of the last received autocrypt header was used,
which may be lower than last_seen timestamp.
# Conflicts:
# src/peerstate.rs
Previously DC_EVENT_ERROR_NETWORK events were emitted for each failed
attempt during autoconfiguration, even if eventually configuration
succeeded. Such error reports are not useful and often confusing,
especially if they report failures to connect to domains that don't
exist, such as imap.example.org when mail.example.org should be used.
Now DC_EVENT_ERROR_NETWORK is only emitted when attempting to connect
with existing IMAP and SMTP configuration already stored in the
database.
Configuration failure is still indicated by DC_EVENT_CONFIGURE_PROGRESS
with data1 set to 0. Python tests from TestOnlineConfigurationFails
group are changed to only wait for this event.
This turns the Bob::expects field into an enum, removing the old
constants. It also makes the field private, nothi0ng outside the
securejoin module uses it.
Finally it documents stuff, including a seemingly-unrelated Param.
But that param is used by the securejoin module... It's nice to have
doc tooltips be helpful.
LoginParamNew structure, which contained possible IMAP and SMTP
configurations to try is replaced with uniform vectors of ServerParams
structures. These vectors are initialized from provider database, online
Mozilla or Outlook XML configuration or user entered parameters.
During configuration, vectors of ServerParams are expanded to replace
unknown values with all possible variants, which are tried one by one
until configuration succeeds or all variants for a particular protocol
(IMAP or SMTP) are exhausted.
ServerParams structure is moved into configure submodule, and all
dependencies on it outside of this submodule are removed.
in general, Viewtypes other than File should be used only
when the added file type is tested on all platforms -
including Android4 currently.
as this is easily overseen,
i've added a comment.
this partly reverts Viewtype changes done by
https://github.com/deltachat/deltachat-core-rust/pull/1818
Fix#1791 and show all group messages if the user already is in the group, even if the sender is blocked
Also fix a comment
Co-authored-by: link2xt <ilabdsf@gmail.com>
This fixes#1804 in two ways: First, it removes a <!doctype html> from
the start of the mail, if there is any.
Then, it parses the html itself it quick-xml fails, just stripping
everything between < and >.
Both of these would have fixed this specific issue.
Also, add tests for both fixes.
threading was broken in core43 as this flags unencrypted messages as errors
and errors are not replied-to.
the fix is not to mark missing signatures for unencrypted messages as errors.
.tgs files are Telegram stickers. Internally they are gzipped JSON files,
containing a single Lottie animation.
MIME type application/x-tgsticker is commonly used for telegram documents
containing such stickers.
An EphemeralTimerChanged message with the same timer as already set can
be received when there are large delays or lost messages. Even though
inner_set_ephemeral_timer should not be called in this case, because it
emits an event indicating timer change, system message will be added to
the chat, so it should be translated with set_better_msg in any case.
Normally NULL is used when there is no fingerprint, but default value
for fingerprint columns is an empty string.
In this case, loading should not fail with an "invalid length" error,
but treat the fingerprint as missing.
Strict check was introduced in commit ca95f25639
Since try_decrypt does not check if the message has valid signatures
anymore, it may return empty signatures set, which means the message is
not a valid autocrypt message.
this pr allows prefixing custom jitsi urls with `jitsi:`
so that clients have a chance to detect these custom instances
and may start the corresponding app.
introduce the qr-code-type `DCWEBRTC:[webrtc_instance]`.
dc_check_qr() returns this as the type DC_QR_WEBRTC
and these qr-codes can be passed to dc_set_config_from_qr() then;
this finally sets `webrtc_instance` using dc_set_config().
needed for how node/js handles the chatlist currently
(it convertes the ffi-object to an js-object and throws ffi-object away directly,
this makes it hard to access dc_chatlist_get_summary() later.
Since introduction of provider database, these certificate checks are
enabled for test servers anyway, but this setting prevents usage of
local servers with self-signed certificates.
It presents no security issue, because properly configured servers will
only serve passwords on HTTPS and distribute only HTTPS QR codes, but
makes testing easier when HTTPS is not easy to deploy.
If attacker can control the URL used, they can change the URL to another
HTTPS URL controlled by them and act as a proxy between the client and
original server anyway.
Adhoc groups for group messages that don't have Chat-Group-ID are already
created above in another create_or_lookup_adhoc_group.
At this point it could be that there is a valid Chat-Group-ID header,
but no group was created because removed_id was non-zero i.e. received
message removes some group member.
If group was not explicitly left, current code creates an adhoc group
instead of trashing late or reordered messages that remove group
members. Such groups have adhoc group IDs locally, but proper group
message-IDs. Attempts to reply to such groups or leave them creates
"split groups" for all other members of original group. The solution is
not to create adhoc groups in this case.
Permanent error 550 5.1.1 is no longer considered temporary.
Enhanced status code is checked now, so only 550 5.5.0 is an exception
for misconfigured Postfix servers.
Yandex error 554 5.7.1 was handled correctly, but only because it had
response code 554, while the comment talks about enhanced status code
5.7.1. The comments are corrected.
Failed messages are now marked as such with message::set_msg_failed.
Previously they were left in a pending state.
If info message cannot be added to the chat, the error is displayed with
error! instead of being logged with warn!.
when seeing our own profile image send from other devices,
we use them as Selfavatar on the current device as well.
as there is no special message sent on avatar changes,
this is a simple approach to sync the avatar across devices.
Now MarkseenMsgOnImap sends MDN even if it can't mark the message as
seen on the server.
To prevent multiple MDNs from being sent, MarkseenMsgOnImap is postponed
until the message is detected in a folder from which it is not going to
be moved.
Mimeparser.was_encrypted() checks if the message is an Autocrypt encrypted
message. It already means the message has a valid signature.
This commit documents a few functions to make it clear that signatures
stored in Mimeparser must be valid and must always come from encrypted
messages.
Also one unwrap() is eliminated in encrypted_and_signed(). It is possible
to further simplify encrypted_and_signed() by skipping the was_encrypted()
check, because the function only returns true if there is a matching
signature, but it is helpful for debugging to distinguish between
non-Autocrypt messages and messages whose fingerprint does not match.
Add a mutex to prevent a race condition when a "your pw is wrong" warning is sent, resulting in multiple messeges being sent.
Do not mute the device chat but "only" send MsgsChanged event when no
notification shall be shown.
More logging.
Parallel processing of messages results in bugs such as messages sent by
a new member being processed before the message that adds this member to
the chat, even when it has lower UID. In this case, messages are shown as
"[Unknown sender for this chat. See 'info' for more details.]", while
there is a message "Member ... added" right before, because writing the
information about new member to the database takes longer then reading
old index from the database.
With this change, API user does not need to look at the timestamp of
the next message to display the day marker, but can get the timestamp
directly from the marker. This way there is no database query and no
risk of error as a result of database being busy or message being deleted.
instead of applying all config_defaults unconditionally
after the first configure, a config_defaults for a given key
is now applied when this key has never been set before.
this way, you can set some keys before calling configure()
and also, later versions can add new defaults for new keys
(that would require another call to configure() then, however)
for "fresh" messages, calc_sort_timestamp() makes sure,
the sorting timestamp is not less than the last-non-fresh message.
this commit fixes the "fresh" state we give to calc_sort_timestamp(),
it just uses the message-state we've already calculated.
the old assumption, that unseen messages are always fresh
is wrong as this "seen" flag just comes from an imap-flag
that is not set for self-sent messages.
therefore, in a multi-device-setup, things are totally messed up.
(the bug was probably in there since a long time,
however did not came to light until the async-move)
This tidies up our testing tools a little bit. We had several
functions which through various changes ended up doing the same and
some more which did very similar stuff, so I merged them to have
things simpler. Also moved towards methods on the TestContext struct
while cleaning this up anyway, seems like this structure is going to
stay around for a bit anyway.
The intersting change is in `test_utils.rs`, everything else is just
updating callers. A few tests used example.org which I moved to
example.com to be able to re-use more configuration of the test
context.
the last message shown in a chatlist
is the one with the largets timestamp that is not hidden.
in the past, we calcualted the last timestamp using a subquery
and uses that timestamp to finally get the message.
this may fail when there are two messages with the same max. timestamp.
with this fix, we return the id from the subquery and use that
(the subquery already filters by hidden etc.)
in practise, by time-smearing,
usually delta-chat avoids messages from the same device
having the same timestamp - however, this may not be true for multi-device
and/or read-receipts.
i have not seen this error all the years, however, it happens with
the async move several times - maybe because things are just sent faster
and things become more probabe.
Fix #1478
I changed my original plans a little because I had so many extra ideas and then sorted that I should rather look at actual NDNs and look what is necessary to parse them:
- Recognize NDNs by ~the sender address, which is in a regex the providers database. The problem with heuristics would be that someone could send fake-NDNs and mark messages as failed.~ the standard ("report/delivery-status") and heuristics ("subject contains 'fail' and sender contains 'daemon'"). If there is a valid Message-ID, then rely on that this is an NDN (of course, generally someone might try to find out a Message-ID and send a fake NDN).
- ~Look for `In-Reply-To`~ (only Gmail did this and Gmail uses rfc822 anyway.)
- ~Look for a mimepart `message/delivery-status`, which might contain a `X-Original-Message-ID`~ (only Gmail did this and Gmail uses rfc822, too, anyway.).
- Search through the body and look for a line `Message-ID: *` (remember to remove `<`, `>`), in the hope that that's the original header
- Look for a mime-part containing the string `rfc822`, which will contain the original header. Parse them with Mailparse and look for `Message-ID`.
CLOSE, which was used previously, only expunges messages and deselects
folder, and it should only be called if some folder is selected. For that,
Imap.close_folder() method is used.
This adds functionality to the cffi build script to also extract
defines so that we can ask the compiler to figure out what the correct
values are. To do this we need to be able to locate the header file
used in the first place, for which we add a small utility in the
header file itself guarded to only be compiled for this specific case.
Reply to group messages assigned to 1:1 chat on ac4 and see where they
end up on ac1 and ac2. Currently they contain group ID in Reply-To field
and go to group chat, so the test is failing.
The code in dc_receive_imf.rs looks a bit funny, an alternative would be a function:
fn upcate_chat_last_subject(context: &Context, chat_id: &ChatId, mime_parser: &mut MimeMessage) -> Result<()> {
let mut chat = Chat::load_from_db(context, *chat_id)?;
chat.param.set(Param::LastSubject, mime_parser.get_subject().ok_or_else(||Error::Message("No subject in email".to_string()))?);
chat.update_param(context)?;
Ok(())
}
get_rowid should not return 0, as we have inserted a row right above.
And using is_error() instead of comparing row_id to 0 is a strange way
to check this condition.
As all functions that actually returned 0 chat ID to indicate error have
been removed, the function is gone too.
Both methods do the same: compare chat_id to 0. However, in these cases
0 refers to the state when chat_id is not determined yet, because no
corresponding chat has been found.
All functions that returned 0 to indicate error have already been
resultified.
This uses the Fingerprint type more consistenly when handling
fingerprits rather then have various string representations passed
around and sometimes converted back and forth with slight differences
in strictness.
It fixes an important bug in the existing, but until now unused,
parsing behaviour of Fingerprint. It also adds a default length check
on the fingerprint as that was checked in some existing places.
Fially generating keys is no longer expensive, so let's not ignore
these tests.
the advanced options are not used anyway later,
but prevent imap/smtp connections from being altered.
nb: we want to stop altering when some advanced options
are entered, however, we want to do this probaby
not depending on autoconfig.
Fix#1474 "Sending message to contact with < or > in Recipient gets treated as "Sent" but is not received".
As I was at it, I also extracted the correct name and address from addresses like Mueller, Dave <dave@domain.com>.
This means all key conversions/serialisation/deserialisation can be
done with DcKey rather than Key. Also migrate all key conversion
tests to DcKey rather than Key.
This moves both the Keyring and the fingerprints to the DcKey trait,
unfortunately I was not able to disentangle these two changes. The
Keyring now ensures only the right kind of key is added to it.
The keyring now uses the DcKey::load_self method rather than
re-implement the SQL to load keys from the database. This vastly
simpliefies the use and fixes an error where a failed key load or
unconfigured would result in the message being treated as plain text
and benefits from the in-line key generation path.
For the fingerprint a new type representing it is introduced. The aim
is to replace more fingerpring uses with this type as now there are
various string representations being passed around and converted
between. The Display trait is used for the space-separated and
multiline format, which is perhaps not the most obvious but seems
right together with FromStr etc.
the document standards.md exist since a long time
to get an overview but also a nice reference to get some links.
however, its placement in the android-repo is a bit unlogical,
probably comes from a time where there was only one implementation :)
as ios and desktop use the same standards and as
all acutal implementations are in the core,
we move it to the core repo.
android can have a link to the new place.
journal_mode is persisted, it is sufficient to change it only for one handle.
with_init() is called for a bunch of handles directly on pool-creation,
so changing to WAL here easily results in busy-errors.
search for the sentbox in two passes:
- first check the folder attributes
- only if that fails, check for some known names
this way, the sent-attribute always has precedence
over the name; this was not the case before.
moreover, this fixes an possibly wrong early exist
when the attribute list is completely empty.
The condition remains from the time when expired messages were hidden
instead of being moved into trash chat. As a result, old hidden
messages, such as location messages, were not deleted.
This function should only return temporary errors, e.g. database errors,
as precheck_imf() and dc_receive_imf::add_parts() treat them as such,
retrying the fetch on failure. When permanent errors, like missing
Message-ID, are bubbled up, they cause infinite fetch loop.
* First try making get_recipients use MailHeader (nice and functional)
* Get it to compile by using not-so-functional style
* Add "empty-from" test, drop unnecessary check for error; continue using addrparse_header() instead of addrparse()
* Try to use functional style, unfortunately, I can't get the compiler to accept it
* Do it imperative-style: Do not overwrite To with Cc and vice versa
* Use addrparse_header() once more
* Still addrparse_header()
* Clippy
* Fix compile errors in tests
* Fix typo
* Fix tests again ;-)
* Code style
* Code style; try a HashMap<addr: String, display_name: String> as an address list but I am not convinced
* Code style; Use Vec<SingleInfo> as address list
* Clippy
* Add tests
* Add another test
* Remove stale comments
If error is ignored, the message will never be fetched again, even if
there was a database write error.
dc_receive_imf itself is modified to ignore unrecoverable errors, to
prevent endless refetching of incorrect messages.
these events take the message-id as parameter and might be used
to update an existing list (although to recommended)
if the event is issued for hidden messages,
this might led to "empty" messages flashing up -
the ui tries to get the message from the event,
after a moment, on the next update, the message disappears again
as hidden messages are of course not returned eg. by dc_get_chat_msgs().
the effect was probably always visible for secure-join-messages on ios,
however, become much more visible recently when read-receipts are added
as hidden messages as well (to make them auto-deletable).
the footer mark normally used in email-conversations is `-- `,
note the trailing space, see RFC 3676, §4.3
unfortunately, the final space is removed by some providers,
which lead to footers showing up on delta-to-delta-conversations
(on nondc-to-delta, this is not an issue as we cannot be sure anyway
and show a [...] therefore)
this change accepts lines with only `--` as a footer separator
if there is no other footer separator
and if the line before is empty and the line after is not.
as there is still some chance to remove text accidentally,
see tests, some protection against that is needed in another commit.
Let the thread load new messages. This may happen when user switches
the setting to delete messages on the server on and there are a lot of
messages to delete.
This prevents generation of a large number of jobs when IMAP deletion
setting is enabled for the first time. Writing jobs to the database
locks it for readers and may cause UI freezing, because chatlist and
messages can't be read until all jobs are written.
Note that on failure job will be written to the database, to make sure
it is postponed instead of being retried immediately.
Job::new() can be used to create jobs in-memory
Job.update() is replaced with Job.save() which can create new database
entries and consumes Job to avoid the need to update job ID after saving
it to the database.
This moves the loading of the keys from the database to the trait and
thus with types differing between public and secret keys. This
fetches the Config::ConfiguredAddr (configured_addr) directly from the
database in the SQL to simplify the API and consistency instead of
making this the responsiblity of all callers to get this right.
Since anyone invoking these methods also wants to be sure the keys
exist, move key generation here as well. This already simplifies some
code in contact.rs and will eventually replace all manual checks for
existing keys.
To make errors more manageable this gives EmailAddress it's own error
type and adds some conversions for it. Otherwise the general error
type leaks to far. The EmailAddress type also gets its ToSql trait impl
to be able to save it to the database directly.
If precheck_imf fails to check if message with the same rfc724_mid
already exists, the same message may be downloaded twice. Instead,
abort the whole operation and retry later.
For hidden messages, blobs are not deleted during housekeeping. To
actually free the space used by media files, messages should be moved to
trash instead of being hidden.
Some mobile email clients, such as apple mail, attach or inline images
after description, just like Delta Chat. It is better to display them
instead of ignoring.
Since introduction of server message deletion, message may not have
server UID even when it is not a draft or pending message. To handle
such messages correctly, we explicitly check message state.
Now there is only one function: hide_device_expired_messages().
If any messages are hidden event DC_EVENT_MSGS_CHANGED(0,0) is emitted
now, which is more correct than DC_EVENT_CHAT_MODIFIED and also triggers
chatlist reload.
In contrast to get_config_delete_server_after(), value 1 does not mean
"delete at once", because it does not make sense to delete messages
immediately after receivning them.
MarkseenMdnOnImap stored server folder and UID which are never updated
by update_server_uid. Now hidden entries are created for MDNs, so they
should be handled as ordinary messages.
Sometimes message deletion job marks message as unlinked without
actually deleting it. It is possible if the message was already moved
into another folder, possibly by second device, but not detected there
yet. It should be detected later in the other folder, and the
server_uid in the database should be set.
Since introduction of add_imap_deletion_jobs() any expired message
record will be marked as unlinked eventually, because another message
deletion job will be scheduled even if update_server_uid() resurrects
the message once.
This may result in messages not being deleted. Optimization and
traffic-saving is postponed for later, one idea is to optimize message
deletion to avoid checking if Message-ID on the server matches
Message-ID in the database.
If job returns Status::Finished, it will be deleted. Then
add_imap_deletion_jobs will recreate it immediately if the message is
expired. To actually backoff the job, we should postpone it instead of
removing.
This is to avoid creating thousands of jobs when user enables
"delete_server_after" setting for the first time.
If device is offline for more than 2 weeks, some messages may not be
deleted.
When "delete_server_after" setting is configured, postponed
DeleteMsgOnImap jobs are created for incoming messages.
This commit adds job::add_imap_deletion_jobs function which creates
DeleteMsgOnImap jobs right before performing IMAP jobs. This way even
messages that expired when the setting was disabled are going to be
deleted.
Job creation on message reception is unnecessary now, and even harmful
because it will create jobs with an expiration time which may later be
reduced. It is planned to be removed in following commits.
Now 0 means "never delete", 1 means "delete at once" and other values
indicate the number of seconds after which them message should be
deleted from the server.
Configuration value interpretation is moved into
Context.get_config_delete_server_after() function.
At least one entry is required for DeleteMsgOnImap job. Additionally,
adding a hidden entry makes it possible to avoid redownloading the
message if it gets a new UID on the server.
DeleteMsgOnImap deletes files from the server only when the last part
is deleted. Removing only the first part of the hidden or trashed
message does not result in message deletion.
Currently only trashed or hidden messages are deleted by
DeleteMsgOnImap, so it is safe to remove database records.
It is planned to delete messages on IMAP server after
user-configurable time to cleanup the server even for messages
displayed in chats. For such messages, we unlink them from the
Message-ID, but keep the database record to display them.
Mutable UID reference is removed. Instead, ImapActionResult is
returned immediately.
If message has changed and UID does not correspond to expected
message, ImapActionResult::Failed is returned. Temprorary IMAP errors
result in ImapActionResult::RetryLater.
On the job side, ImapActionResult::RetryLater does not result in
immediate job retry anymore. Instead, job is retried with a backoff.
In addition to moving the message into trash chat, this function
removes message text to make sure the message does not remain in the
database. Only the information necessary to delete message from the
server and avoid redownloading it should be kept, such as Message-Id
and IMAP UID.
CString::yolo was still used in the ffi, this was an unsafe
transitional thing. To remove it there were two choices: 1. make
errors in creating CStrings hard errors or 2. try and be as lenient as
possible. Given the to_string_lossy() convention adopted in the ffi
this choose the lenient option and simply skips over embedded null
bytes, leaving the rest of the strings intact.
Thus now CString::new_lossy(). It's only used for .strdup() however
so no longer a public trait.
This also cleans up the public visibility of things in the strings.rs
file:
- Rename StrExt/OptStrExt traits to what they actually do: provide
.strdup() -> Strdup/OptStrdup.
- dc_strdup() should be an implementation detail, replace all usages
with Strdup.strdup() method.
- Only allow visibility inside the crate for all things.
- Reduce visibility to only the module for things not used in lib.rs.
after fixing some issues wrt ecc keys, see #1319,
and waiting some time (three core releases, two ios/android/desktop releases),
it is now the time to switch again to ecc keys again,
after the first attempt was stopped in #1319
before, the list was empty
and trying to send to groups that only contain SELF lead to a crash.
in theory, this happens for both, bcc_self enabled or not, however,
if bcc_self was disabled (default setting),
things worked as the whole mimerendering was skipped on a higher level.
also, the saved-messages-chat was not affected as this was checked explicitly.
this pr changes the mimerendering so that From: is used as To:
if there is no recipient-list and the messasge will be sent to SELF only.
the currently released versions fail sometimes in encrypting to ecc keys,
see #1313, the issue is about to be fixed,
however, we should not generate ecc keys until the
fix is rolled out - otherwise new users will get encryption errors every some
messages if their counterpart is not yet using the most recent version.
we can start generating ecc keys a few weeks after the fix is rolled out.
up to now, Chat-Group-Member-Added and -Removed commands
result in a complete recreation of the memberlist
by collecting all addresses from the From: and To: headers.
this easily results in missed and accidentally removed members,
esp. when several people at the same time scan a qr code to join a group.
this commit changes the behavior of adding members by
not removing members on the Chat-Group-Member-Added command.
instead the existing memberlist is conjuncted
with the memberlist seen in the message.
only adding the member from the Chat-Group-Member-Added
seems not to be sufficient - imaging a group of Alice and Bob:
- Alice adds Claire
- Bob adds Dave _before_ seeing that Alice added Claire
- Dave would never get the information that Claire is in the group
wrt Chat-Group-Member-Removed: this command
does no longer recreate the memberlist but just remove _exactly_ the member
mentioned in the header. there are situations, where a just removed member
will be readded by out-of-order-messages, however, compared to missed
members, this is seems to be acceptable - also as this is more visible
and easier to fix (just remove the member again).
might be that, in practise, this is not a big issue. while adding members
is typically done in masses on bootstraping a group,
this is typically not true for removing members.
Instead, use last_insert_rowid() function to find the row.
There is no race condition in using last_insert_rowid(), because
last_insert_rowid() returns row id last inserted in this connection. As
we hold the connection during the whole transaction, it is impossible
that some other thread will execute INSERT statement in parallel.
This commit is part of the effort to get rid of sql::get_rowid hack and
use transactions more for related SQL statements.
Existing public methods that use these functions, like sql::execute, are
only suitable for executing a single statement.
Sometimes it is useful to execute multiple statements within one
connection, for example to begin a transaction, execute mutliple SELECT
and INSERT queries and commit or rollback the whole transaction.
We already have a .strdup() method on AsRef<str>, this adds this
method also to an option of this. In case the option is None a NULL
pointer is returned.
This is done by using a new trait, as the type system otherwise
considers such an implementation conflicting with the existing one.
Now configure module is no longer public. Users should call
Context.configure() and Context.is_configured() methods.
Configure module is completely hidden from documentation unless
--document-private-items option is specified.
RFC 2183 specifically allows filenames to be specified for MIME parts
with "inline" Content-Disposition.
Previously we treated such attachments as an error, causing the whole
message parsing to fail. Now it is only an error if Content-Disposition
cannot be parsed.
MIME parts with filename are now considered to be attachments
regardless of their Content-Disposition type. However, "attachment"
Content-Disposition is still processed differently: we generate a
filename for it even if it is not specified.
I had a look for two mintues at converting things to .try_inner() and
I don't think this is currently worth the effort or would increase
readability a lot. So let's leave this for now.
There is no need to have both a .log_warn() and .log_err(), even their
names are confusing by now. Let's just have the most liberal one and
reduce one more thing we need to know about.
Also, these traits don't need to be pub.
Using peek_online_config() results in a message being sent to some
other account, used by previous test. If tests are run in parallel,
for example with pytest-xdist, we could be sending a message to a still
online DC client.
Envelope has the same Message-ID, but using other fields from the
envelope, such as From field, is error-prone. They may contain values
different from the message body. As we don't parse the envelope later on,
it is better not to fetch it during prefetch too.
This capability is not used by Delta Chat. Moreover, XLIST is deprecated
in favor of https://tools.ietf.org/html/rfc6154, which should be
supported instead. Delta Chat already looks for \Sent attribute in
get_folder_meaning().
- Python should raise exceptions on error. Not return False.
- Negative durations are nonense.
- ChatID validity is already checked by the Rust API, let's not duplicate.
Configuring an online account generates a key, we would like this
code-path tested too. So add some functionality to the AccountManager
to not use the pre-generated keys.
Because this slows down interactively running the tests by hand add an
ignored marker which only runs if --ignored is used. This name was
chosen because this matches the naming used by rust/cargo #[ignored].
The difference however is that --ignored on cargo *only* runs ignored
tests while here it runs *all* tests.
To ensure the ignored/slow tests are run on CI we add it as an
argument to the tox configuration, which is used by the CI to run the
tests.
This changes the AccountMaker to use pre-generated keys when
available, speeding up test runs.
As a side-effect we no longer need to compile the integration tests in
release mode with debug symbols. Losing debug symbols (-g) means
cargo no longer wants to recompile everything all the time too.
Tested locally and seems to works.
The imex code was deleting the key if it was already used. Turns out
none of the callers would not want this behaviour, so let's just
handle this case generally.
I guess this is a little neater. Note that this does change all the
existing error logging to warning logging. I believe this is correct
as these error log messages are really programming errors rather than
errors which need to be show to the user.
Also chose to make entire with_inner unsafe, there is little to be
gained from hiding the unsafe as this is only used from unsafe
functions in the first place.
The user-visible change here is that it allows the FFI API to save
keys in the database for a context. This is primarily intended for
testing purposes as it allows you to get a key without having to
generate it.
Internally the most important change is to start using the
SignedPublicKey and SignedPrivateKey types from rpgp instead of
wrapping them into a single Key object. This allows APIs to be
specific about which they want instead of having to do runtime checks
like .is_public() or so. This means some of the functionality of the
Key impl now needs to be a trait.
A thid API change is to introduce the KeyPair struct, which binds
together the email address, public and private key for a keypair.
All these changes result in a bunch of cleanups, though more more
should be done to completely replace the Key type with the
SignedPublicKye/SignedPrivateKey + traits. But this change is large
enough already.
Testing-wise this adds two new keys which can be loaded from disk and
and avoids a few more key-generating tests. The encrypt/decrypt tests
are moved from the stress tests into the pgp tests and split up.
This should prevent accidental sending of unencrypted messages when
parent_is_encrypted returns an error. For example, if the database is
busy due to other thread activity, parent_is_encrypted should not return
false. Instead, message sending job should retry later.
error!() will show the error directly to the user,
this is not useful in this case,
and also the message with the function-name is not for the end-user.
instead, the ui shall (and already does)
show some explaining text if dc_has_import() returns false;
the error!() is disturbing here as this results in two hints shown to the user.
this changes the behavior when scanning a setup-contact qr-code.
instead of waiting, until the whole protocol is finished,
which may take something between 10 seconds and several minutes,
the dc_join_secure_join() returns instantly;
the uis typically show the created chat then.
the returned chat-id is the same than if we wait for the protocol to finish,
it is the opportunistic one-to-one chat-id, so no changes there.
all this works even when both devices are offline.
after dc_join_secure_join() returns, however,
the usual setup-contact continues. ux-wise, once the protocol finishes,
a system-info-messages is added to the chat (also unchanged),
this is directly visible, in the chat as well as in the chatlist.
while the prococol runs, the user can alredy send message to the chat,
or do other things in the app.
if the user scans a new qr-code while an existing protocol is not finished yet,
the old join will be aborted (however, of course, created chats are kept).
we could also allow multiple joins at the same time,
however, this would be much more effort that this little change
and i am not sure if it is worth the effort.
finally, if a verified-group shall be joined,
this is not possible instantly, this is not affected by this pr.
same for unverified-groups, however, this could maybe be improved,
but also here, not sure if it is worth the effort
(i think most scans are setup-contact scans)
oauth2 crashes in beta23
because we did not let Serde remove the quotes from a parsed JSON email-address.
for autoconfig, we try to get a well-known URL
containing the domain of the used address - with the quote that is
https://autoconfig.gmail.com"/mail/config-v1.1.xml?...
unfortunately, instead of just returning an error,
(the url does not exist anyway)
reqwest crashes on the attempt to get this URL.
the Serde-thing is not obvious to me:
while serde_json::Value.as_str() removes the quotes,
serde_json::Value.to_string() does not.
.truncate() is not safe because it panics if string length does not
lie on character boundary. We convert the string to characters and take
first n character instead.
before a message is actually deleted,
we verify that server_folder+server_uid matches the Message-ID,
so that we do not delete the wrong message by accident.
the reason for that is that, to my knowledge, server_folder+server_uid
are not necessarily stable but Message-ID is.
anyway, this check was broken and set the server_uid to 0 in all cases -
where it should be set to 0 _only_ on mismatches.
As a result of job, other jobs can be added or deleted. To avoid
processing deleted jobs and to process other jobs in the correct order,
we should reload them after performing each job.
However, we don't use "LIMIT 1" in SQL queries, because we want to be
able to skip over invalid jobs, and checking if job is invalid can only
be done in Rust.
This may or may not send a combined MDN out. We don't test for it,
but the test ensures that *if combined MDNs are sent in this case*,
then we receive them correctly.
This return value was very complicated to understand. Some failure
returns were returned as Err and some as Ok with no consistency, but
resulting in the same behaviour.
This refactor makes the handle_securejoin_handshake the sole place
responsible for maintaining the state of the secure join
process (context.bob) and also in charge of terminating the ongoing
process. This is none of receive_imf's business.
The remaining returns are now cleanly classified in application-errors
and protocol errors:
Applications errors result in an Err and mean there is a bug or
something else serious went wrong, like database access suddenly
failed or so. In this case receive_imf is still responsible for
clearing the state and resetting ongoing-process. It may be possible
this should still be moved back to securejoin.rs so that recieve_imf
doesn't need to know anything about this either.
Protocol errors are not failures for receive_imf, it just means the
received message didn't follow the protocol. Receive_imf in this case
is told to ignore the message: that is hide it but not delete it.
Other Ok returns also only say what needs to happen to the message:
- It's fully processed and needs no further processing, instead should
be removed
- It should still be processed as a normal received message.
This changes some behaviour: if the chat creation/lookup for the
contact fails this is treated as an application error. Previously
this was silently ignored and send_msg() would be called with a 0
chat_id without checking the response. This resulted in the protocol
quitely being blocked.
This all shhould now make it easier to resultify more of the functions
called by this function, instead of having to deal with very
complicated application logic hidden in the return code.
this commit sets the subject of groups to the name of the group.
if the message is a reply in the group, also the prefix `Re:` is added.
the Chat: prefix is removed for groups,
also the Fwd: prefix, as it this would be expected to be set to the
group name of the forwarded message, which is not really compatible with the
rule "group-name = subject" (forwarded messages are detected by the body,
not by the subject)
rusqlite already checks that SQLite is safe to use in multi-threaded mode when it opens the database and returns rusqlite::Error::SqliteSingleThreadedMode otherwise.
Also create a common parent_query() function to make sure we use the
same message as a parent when determining if it is encrypted and when
actually replying to it.
The struct really represents a parsed MIME message and is not used as
a parser itself. After the from_bytes() call (which should arguably
use the FromStr trait instead) the struct is fully populated.
This is a new feature in Rust 1.40, it means users outside the crate
will not be able to create these structs, allowing us to add fields
without breaking the public API.
Normally it happens in addr_cmp function, but update_gossip_peerstates
forms a HashSet of addresses, so they should be lowercased beforehand.
Also adjust the mail_with_cc.txt to test for non-lowercase addresses.
According to Rust documentation, usize is "The pointer-sized unsigned
integer type".
This change removes unnecessary libc dependency and makes top_evil_rs
script happier.
Since stock string callback has been deprecated, all event callbacks
return 0.
For compatibility, C declarations are not changed and FFI users are
expected to return 0 from their callbacks.
in the very beginning, we allowed using the cmdline partly from the ui
via a special function; this made info!() handy.
however, this is long gone and no longer needed.
It was used when timeout was set on the whole smtp.send() operation.
Now only the operations inside smtp.send() can timeout, and such timeout
errors result in SendError, so SendTimeout is unused.
-Moved wheels installation below source installation and added comment that they are outdated and not working atm
-Updated source compilation/installation to a more step by step process as current one was outdated and not working due to Rust now needing nightly build else failing to compile (subtle v2.2.2 do not work without nightly)
This is the first experiment towards using structs to define the API.
It adds it as a new method on the existing Chat struct.
The types in this struct could improve as well as many other things.
But this is a straight forward translation of the existing json! macro
into a struct.
15 minute timeout was used because it applied to the whole send()
operation. Now timeout applies to each I/O operation, such as read()
or write(), so it can be made much shorter. In particular, this timeout
applies to read() call right after connection to plain or STARTTLS server,
in which case it is not reasonable to wait 15 minutes to receive one
line of data.
The timeout is triggered only if no progress is made within 30 seconds. It
likely indicates a network failure anyway.
* Remove Simplify structure.
* Match for lines starting with 5 markers, not consisting of exactly 5 markers.
This is a regression from C to Rust conversion, see
2bb1c280d5/src/dc_simplify.c (L154)
* Add tests.
we change the name on the wire as the old Chat-Group-Image header
could not be used on random mails, it was the marker for a "Changed" message,
if we would keep this names, things will fail for exising installations
as messages are dropped and a "Group image changed" message is shown instead.
* add-action-rs-clippy-action
just to try this action out
* add --all argument
* rm --all argument again
* add cargo check and fmt
* disable circle ci counterparts(fetch, clippy, fmt)
- recreate the group list more carefully, fixes#985
- resultify a few functions in the dc_receive pipeline
- don't quote displaynames in email-addresses, use utf8, preliminrarily addresses #976
This effectively reverts
https://github.com/deltachat/deltachat-core-rust/pull/964 for chat.rs,
which in that PR was thought to fix something. So maybe something is
still broken? But after improving tests the previous code seems to be
correct.
- Update Python bindings to not always use dc_prepare_msg path when
sending messages with attachements. When using dc_prepare_msg the
blobs need to be created in the blobdir since they will not get
copied and many tests where not doing this.
- Add a test that ensures that calling dc_prepare_msg with a
file **not** in the blobdir fails.
- Add a test that ensures that calling dc_send_msg directly with a
file **not** in the blobdir copies the file to the blobdir. This
test cheats a little by knowing what the filename in the blobdir
will be which is implementation-dependent and thus a bit brittle.
But for now it proves correct behaviour so let's go with this.
- Improve the test_forward_increation test to ensure that the
in-creation file only has it's final state before calling
dc_send_msg. This checks the correct file data is sent out and not
the preparing data, this fails with the chat.rs changes in
#964 (reverted here to make this work again). Also fix the test to
actually create the in-creation file in the blobdir.
- Fix test_send_file_twice_unicode_filename_mangling to not use
in-creation. It was not creating it's files in the blobdir and that
is an error when using in-creation and it didn't seem it was trying
to test something about the in-creation logic (which is tested in
test_increation.py already).
- Fix Message._msgtate code which presumably was not used before?
- Rename `BlobObject::create_from_path` to
`BlobObject::new_from_path`. All the `BlobObject::create*` calls
now always create new files which is much more consistent. APIs
should do what is obious.
Rustls does not offer a documented way to accept valid certificates with
invalid hostnames. Implementation of certificate verification in Rustls
does not have a public API and reimplementing it is error-prone.
Previously, if any of a chat's peers set prefer_encrypt to false
(i.e. "e2ee_enabled=0" in the config, a misnomer btw) then a
previously encrypted chat would drop to cleartext easily.
eg. android relies on that and fallsback to wrong aspect ratios otherwise.
also, this makes job.rs a little more readable
as some things are moved to message.rs :)
This deletes a lot of code and complexity. Though comes at some cost:
- The type no longer fits in a register and will always be on the
stack.
- Constructing the errors is more verbose, no more auto Into casting.
this reverts the logical changes done in #811
but keeps the improvements done later eg. in #911.
the reason for the revert is that it is too hard to
find a started draft in a larget chatlist.
also the shown date would not be just descending.
Reset is an internal value that received messages should not be able to set.
Also return an error on any value other than "mutual" and "nopreference", errors are converted to NoPreference by the caller.
The ongoing process of dc_join_securejoin() was stopped before the
corresponding chat was created. This resulted in a race-condition
between the sqlite threads executing the creation and query
statements, thus sometimes the query would not find the group and
mysteriously fail.
Tripple-programming with hpk & r10s.
Be more defensive: if somehow we got corrupt jobs in the database skip
over them rather than fail to do anything.
This only modifies the query_map() call, the rest is only split off
into it's own function to make it testable. Smaller functions are
good anyway.
When a new Selfavatar is set the file needs to be copied into the
blobdir and as a proper blob it should be stored in the database as
$BLOBDIR/$filename.
This should fix#868.
First, try to use subkeys, because they are usually
short-term encryption keys. If none of the subkeys
are encryption keys, try to use the primary key.
rPGP is updated to the master branch because the
latest release does not have .is_encryption_key() yet.
Double extensions are sometimes used to identify files correctly,
e.g. .tar.gz or .html.zip. Breaking those extensions is not very
nice.
This fixes#865.
if two messages have the same time,
this results i all kinds of sorting failures,
esp. in non-delta-muas and on forwarding messages.
so, no two messages sent out from delta have the same timestamp.
normally, this is no problem, but when things are sent too fast,
eg. on forwarding, we lend us some time from the future.
however, all this did not work
because we forgot to write back the modified time,
this is fixed by this commit, as well as some cleanup.
To download and install the official compiler for the Rust programming language, and the Cargo package manager, run the command in your user environment:
```
curl https://sh.rustup.rs -sSf | sh
$ curl https://sh.rustup.rs -sSf | sh
```
## Using the CLI client
Compile and run Delta Chat Core using `cargo`:
Compile and run Delta Chat Core command line utility, using `cargo`:
```
cargo run --example repl -- /path/to/db
$ RUST_LOG=repl=info cargo run --example repl --features repl -- ~/deltachat-db
```
where ~/deltachat-db is the database file. Delta Chat will create it if it does not exist.
Configure your account (if not already configured):
d="m 8.6780027,35.573064 0.032831,-11.910176 c 0.00138,-0.476406 0.4881282,-0.794259 0.9235226,-0.604877 l 4.1144877,2.345752 -0.02386,8.656315 -4.1268029,2.122946 C 9.1617452,36.370003 8.6766889,36.049472 8.6780027,35.573064 Z m 5.0469633,-1.508222 0.02386,-8.656314 31.145424,-9.537653 c 0.841472,-0.219211 1.65915,0.41667 1.656755,1.283728 l -0.06929,25.139995 c -0.0024,0.867062 -0.825942,1.500799 -1.663803,1.274581 z m 3.8042,6.892234 C 16.681121,40.104348 16.315444,38.819414 16.69043,37.591308 l 2.252234,-7.347193 c 0.2644,-0.861571 0.845185,-1.567441 1.641953,-1.989251 0.796769,-0.421808 1.706956,-0.509819 2.568531,-0.245419 l 7.263888,2.225804 c 1.775518,0.543235 2.780299,2.432591 2.232297,4.208094 L 30.3971,41.790532 c -0.545627,1.777887 -2.432591,2.780297 -4.208095,2.232298 l -7.263891,-2.225804 c -0.545033,-0.165864 -1.01825,-0.460162 -1.395948,-0.83995 z m 12.377693,-7.976728 c -0.07601,-0.07642 -0.17114,-0.133864 -0.280621,-0.167516 l -7.263891,-2.225803 c -0.233244,-0.07209 -0.421626,0.0013 -0.512275,0.04861 -0.09064,0.0474 -0.25772,0.166033 -0.327435,0.396899 l -2.252234,7.347191 c -0.108166,0.354628 0.09088,0.731541 0.447888,0.842099 l 7.263891,2.225802 c 0.354626,0.108174 0.731539,-0.09088 0.842099,-0.447888 l 2.249845,-7.344814 c 0.07453,-0.245145 0.0014,-0.504991 -0.167267,-0.67458 z"/>
d="M 13.5,7.5 V 39 h 0.08654 L 22.533801,29.370239 31.482419,39 h 0.01758 V 7.5 Z m 9.004056,4.108698 1.879508,4.876388 5.039514,0.359779 -3.879358,3.363728 1.227764,5.095749 -4.276893,-2.796643 -4.280949,2.788618 1.237229,-5.093073 -3.873949,-3.371754 5.040866,-0.350417 z"
- The secondary self addresses (see below) are shown in the UI, but not editable.
- When the user changed the email address in the configure screen, show a dialog to the user, either directly explaining things or with a link to the FAQ (see "Other" below)
Changes in the core
-------------------
- We have one primary self address and any number of secondary self addresses. `is_self_addr()` checks all of them.
- If the user does a reconfigure and changes the email address, the previous address is added as a secondary self address.
- don't forget to deduplicate secondary self addresses in case the user switches back and forth between addresses).
- The key stays the same.
- No changes for 1:1 chats, there simply is a new one
- When we send a message to a group, and the primary address is not a member of a group, but a secondary address is:
Add Chat-Group-Member-Removed=<old address> and Chat-Group-Member-Added=<new address> headers to this message
- On the receiving side, make sure that we accept this (even in verified groups) if the message is signed and the key stayed the same
Other
-----
- The user is responsible that messages to the old address arrive at the new address, for example by configuring the old provider to forward all emails to the new one.
For brevity let's abbreviate "group membership modification" as **GMM**.
Delta chat has explicit GMM messages, typically encrypted to the group members
as seen by the device that sends the GMM. The `Spec <https://github.com/deltachat/deltachat-core-rust/blob/master/spec.md#add-and-remove-members>`_ details the Mime headers and format.
If we detect membership inconsistencies we can resend relevant GMM messages
to the respective chat. The receiving devices can process those GMM messages
as if it would be an incoming message. If for example they have already seen
the Message-ID of the GMM message, they will ignore the message. It's
probably useful to record GMM message in their original MIME-format and
not invent a new recording format. Few notes on three aspects:
-**group-membership-tracking**: All valid GMM messages are persisted in
their full raw (signed/encrypted?) MIME-format in the DB. Note that GMM messages
already are in the msgs table, and there is a mime_header column which we could
extend to contain the raw Mime GMM message.
-**consistency_checking**: If an incoming GMM has a member list which is
not consistent with our own view, broadcast a "Group-Member-Correction"
message to all members containing a multipart list of GMMs.
-**correcting_memberships**: Upon receiving a Group-Member-Correction
message we pass the contained GMMs to the "incoming mail pipeline"
(without **consistency_checking** them, to avoid recursion issues)
Alice/Carol and Bob/Doris getting on the same page
If you want to run live functional tests you can set ``DCC_NEW_TMP_EMAIL`` to a URL that creates e-mail accounts. Most developers use https://testrun.org URLS created and managed by [mailadm](https://mailadm.readthedocs.io/en/latest/).
Please feel free to contact us through a github issue or by e-mail and we'll send you a URL that you can then use for functional tests like this:
export DCC_NEW_TMP_EMAIL=<URL you got from us>
With this account-creation setting, pytest runs create ephemeral e-mail accounts on the http://testrun.org server. These accounts exists only for one hour and then are removed completely.
One hour is enough to invoke pytest and run all offline and online tests:
pytest
# or if you have installed pytest-xdist for parallel test execution
pytest -n6
Each test run creates new accounts.
running "live" tests (experimental)
-----------------------------------
.._sourceinstall:
If you want to run "liveconfig" functional tests you can set
``DCC_PY_LIVECONFIG`` to:
Installing bindings from source (Updated: July 2020)
"you must specify --email and --password once to configure this database/account"
)
ac.set_config("addr",args.email)
ac.set_config("mail_pw",args.password)
ac.set_config("mvbox_move","0")
ac.set_config("mvbox_watch","0")
ac.set_config("sentbox_watch","0")
ac.set_config("bot","1")
configtracker=ac.configure()
configtracker.wait_finish()
# start IO threads and configure if neccessary
ac.start_io()
print("{}: waiting for message".format(ac.get_config("addr")))
ac.wait_shutdown()
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
