* Fix#2335 (delete job flooding)
The problem was:
- You are offline, and an ephemeral message is due to delete.
- load_imap_deletion_job() is called and returns a deletion job for it.
- The job fails because, well, we are offline.
- The job is saved into the database.
- load_imap_deletion_job() is called again, and so on.
* Add test
ct.name was insufficient, as authname, overridden name and email address
fallback were not taken into account.
Dropping this condition increases performance by 25% according to the
benchmark.
Also add a test for search_msgs.
It reduces the time by ~20% according to `search_msgs` benchmark.
Sorting by IDs is sufficient for global search as IDs increase in the
order of message reception.
This PR originally contained a fix for sqlx which turned out not not to be necessary. But on the way, I made some general improvements:
- Under some circumstances, a "normal" test failure led to a timeout, without printing a decent error message. See e.g. https://app.circleci.com/pipelines/github/deltachat/deltachat-core-rust/8069/workflows/ba2a9949-b4ad-4ceb-a930-073bba05e2db/jobs/30965.
(The problem was: if there is an exception in dc_account_extra_configure(), when trying to handle the exception the line account.log("===================", e, "===================") was called, which can't work as log() only expects one parameter)
- When a test fails: Call `dump_account_info()` even if there is no direct_imap
- In test_import_export_online_all(), add another 100KB file to the backup. This adds resilience against future size changes of the sql db file: The test tests the smoothness of the progress bar. And if there are there are not enough about-equally-sized files, the progress bar can't be smooth.
the user-given contact name may be set to an empty string;
in this case the authname or the email is used for the contact
and also for the name of possibly existing chats.
this works well for `dc_chat_get_name()` as that just uses
`dc_contact_get_display_name()` for single-chats.
it did not work for `dc_get_chatlist(query)` as that
uses the database for performance reasons -
however, in the database, the empty string is written
instead of the display name is written for a chat.
this is fixed by this pr by also using
dc_contact_get_display_name() when updating the chats-table
(similar to `dc_create_chat_by_contact_id()`)
message type directly (ex. `const.DC_MSG_STICKER`) so if new message types are
added, they can be used direcly without needing the python API to be updated.
This is an PGP/MIME format produced by Microsoft Exchange and ProtonMail IMAP/SMTP Bridge,
described in detail in https://tools.ietf.org/id/draft-dkg-openpgp-pgpmime-message-mangling-00.html
This patch adds seamless support for "Mixed Up" Encryption, repairing
mangled Autocrypt messages without notifying the user.
&str queries are not persistent by default. To make queries persistent,
they have to be constructed with sqlx::query.
Upstream sqlx does not contain the change that make all queries
persistent, but it is not needed anymore. but
fix #2073fix#2292 (I think)
- Messages can be assigned to any chat by the References and In-Reply-To, also to 1:1 chats; this has higher priority than the group id because with ad-hoc groups, it can happen that two devices have different group ids for the same conversation thread.
- If `From` is not in the chat (we call this "shadow sender"), `OverrideSenderDisplayname` is set. This communicates to the UI that:
- A `~`should be added in front of the sender's displayname.
- Also in 1:1 chats, the sender's displayname and avatar is shown, as if this was a group.
The "Unknown sender for this chat" messages are completely removed for unprotected groups.
For protected chats, everything stays as it was before.
POSTPONED:
- Maybe (if it turns out to be still necessary):
- The ad-hoc group id is computed by the the References, instead of the member list, as it is currently done
- How do we prevent that the message can't be assigned to the correct chat as the parent message was deleted?
Switches from rusqlite to sqlx to have a fully async based interface
to sqlite.
Co-authored-by: B. Petersen <r10s@b44t.com>
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: link2xt <link2xt@testrun.org>
fix#2188
Also, if we notice that the server started reusing old UIDs, _also_ do a `ResyncFolders`, because the server likely forgot to change uid_validity
`DoxygenLayout.xml` is picked up by doxygen automatically (as `Doxyfile`)
and its template can be generated by `doxygen -l DoxygenLayout.xml`.
after that, the following tweaks were done:
- remove all xml-parts but header (other defaults are fine so far)
- rename "Modules" header to "Constants" (there are no modules :)
- remove useless subitems from "Classes", eg. by name only shows as "d"
- reorder tabs so that "Classes" becomes the first one
save subject for messages:
- new api `dc_msg_get_subject()`,
- when quoting, use the subject of the quoted message as the new subject, instead of the
last subject in the chat
usually, there is at most one text/html and one text/plain part.
multiple text/plain parts,
are known only by mailinglist software that wants to add a footer
but cannot alter the original content part -
maybe because it is html and things are hard to stitch
or maybe because the content part is cryptographically signed
and cannot be altered therefore.
the test has multiple text parts because
the content text is signed and
the mailinglist software wants to add some text
(and cannot alter signed part without breaking the signature)
fix#2254: if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message), the db was corrupted.
* Add result.log() for logging with less boilerplate code
* Bugfix: Resultify housekeeping() to make it abort if the db is closed instead of just deleting everything
* Require the UI to call dc_stop_io() before backup export
* Prepare a bit better for closed-db: Resultify get_uidvalidity and get_uid_next and let job::load_next() wait until the db is open
About the bug (before this PR):
if the DB was closed without calling stop_io() and then an interrupt arrives (e.g. incoming message):
- I don't know if it downloads the message, but of course at some point the process of receiving the message will fail
- In my test, DC is just in the process of moving a message when the imex starts, but then can't delete the job or update the msg server_uid
- Then, when job::load_next() is called, no job can be loaded. That's why it calls `load_housekeeping_job()`. As `load_housekeeping_job()` can't load the time of the last housekeeping, it assumes we never ran housekeeping and returns a new Housekeeping job, which is immediately executed.
- housekeeping can't find any blobs referenced in the db and therefore deletes almost all blobs.
blocked mailinglists addresses are added to the contact table
before blocked contact list is created -
(this allows unblocking of blocked lists in previous testing releases,
however, more importantly, it keeps all blocking/unblocking code inside
contacts)
on unblocking such a contact,
the corresponding chat is unblocked as well.
* deprecate mostly unused dc_get_blocked_cnt() api
instead, the size returned by get_blocked_contacts() should be checked,
this is safer and allows easier adaption of blocking rules.
ui or python seems not to use dc_get_blocked_cnt(),
however, there is one test in node,
therefore, the function will continue working for now
(by just returning Contact::get_all_blocked().len() then)
* add decision api to repl tool
* add block/unblock api to repl tool
* unify usage of @deprecated doxygen command
outlooks SMTP-server change the Message-ID of messages
and put the original Message-ID to X-Microsoft-Original-Message-ID.
the changed Message-ID has some issues:
- outgoing messages with bcc_self enabled are shown twice
as the self-copy got the changed Message-ID while the database uses
the original one
- read receipts do not work as they refer to the changed message id
- in general, sender and recipient see different Message-IDs
the issues can be fixed by
(1) let all receivers use the original Message-ID
this is what this pr is doing
and this should fix all issues with delta-to-delta communication,
including groups, group-images etc.
there may be issues left in communication
with other MUAs as they are using another Message-ID.
(2) ftr: updating the Message-ID in the database of the sender to the new one
this requires bcc_self always enabled (which is not the case)
and may also result easily in race conditions
(Bob answers before Alice sees its self-sent message),
however, has the advantage of better compatibility with other MUA.
if needed, the compatibility with other MUA could be improved by remembering
both Messages-IDs, maybe we could treat the modified as References or so,
however, i think, this could be part of another PR if we know better about
real, in the wild issues.
The chat::lookup_by_contact_id call is already resultified, the
database can not contain this since the auto-increment counter is
bumped to 7 by the time the database tables are created.
* add test for mailchimp mailinglists
* pass MimeMessage to create_or_lookup_mailinglist() (as of the other create*() routines) to allow more flexible name processing; document the function
* get mailing list name for mailchimp from From:-header
* make clippy happy
* add comment to '.list-id.mcsv.net' suffix
Now I know why the tests failed before 48c58a7 (i.e. after c923670) but
not on master (i.e. before c923670):
because of a bug, scan_folders() set the configured_sentbox to None if
it was set before. If it was None before, it restored the correct value.
On master, there was another bug that led to two runs of
scan_folders() being started at the same time. Therefore, the first run
set configured_sentbox to None, the second one restored the correct
value.
c923670 fixed the latter bug, so that only one run of scan_folders() was
started. Therefore, configured_sentbox stayed None incorrectly and
test_fetch_existing() failed.
48c58a7 fixed the former bug.
This commit adds checks to test_fetch_existing(), so that it definitely
checks for the former bug.
This does fix a bug and it makes the tests pass, but I'm not sure why it
makes the tests pass; maybe there is a race condition that made the
tests fail and my commit just leads to another timing.
This will allow MimeFactory.render() to put protected headers
into main_part and wrap it into single-part multipart/mixed if
protected headers are too large to put into the outermost IMF.
This should make colors used by Delta Chat for emails similar to colors
used by XMPP clients implementing the same specification[1], such as
Conversations and Snikket.
Previously Delta Chat used only hardcoded 16 colors, so this change
should also increase the number of colors and make it easier to
distinguish different contacts.
[1] https://xmpp.org/extensions/xep-0392.html
This changes the internal stock strings API to be more strongly typed,
ensuring that the caller can not construct the stock string in the
wrong way.
The old approach left it to the callers to figure out how a stock
string should be created, now each stock string has their specific
arguments and callers can not make mistakes. In particular all the
subtleties and different ways of calling stock_system_msg() disappear.
This could not use a trait for stock strings, as this would not allow
having per-message typed arguments. So we needed a type per message
with a custom method, only by convention this method is .stock_str().
The type is a enum without variants to avoid allowing someone to
create the type.
Sadly the fallback string and substitutions are still far away from
each other, but it is now only one place which needs to know how to
construct the string instead of many.
1.51.0-alpha.0 is somewhere between 1.50 and 1.51,
as various beta-versions come or already came out with the new core,
this is helps in logs/debugging.
The following rules apply now:
1. "name" column is only updated manually and never over the network
2. "authname" column is only updated over the network and never manually
3. Displayname is a "name" if it is non-empty, otherwise it is "authname".
This fixes a known (pytest.xfail) problem of "name" being changed over
the network when user has set it to non-empty string manually.
This also fixes the problem when "name" and "authname"
became unsynchronized accidentally, when they were equal and then
Origin::IncomingUnknownTo update arrived, setting "name" but not
"authname". Rust regression test is added for this case.
renaming mailing lists and setting mailinglists images
(possible locally, but not synced)
was blocked because the function real_group_exists() checked
the chattype on a numerically.
it is not 100% clear to me,
why this function exist at all,
it just checks if a record with type=120 (group) exists under the given id -
this is the same as loading the chat and check chat.typ afterwards.
might be a premature optimisation relict from core-c
or made some error handling easier there.
This does not affect the release profile and it significantly speeds
up compilation. In the rare cases where panics are not sufficient to
debug something you can manually enable this again.
The get_chat_id_by_grpid() call is already resultified. The database
can only contain consistent data since on creation it inserts rows for
the special chats which do not have a grpid set so can never match.
Thus we can safely remove another use of the ChatId::is_unset call
which should be removed to clean up the type.
* Copypaste-merge my old work
* Start implementing mailinglists the new way
* Create pseudo contact
* Fine-tune docs
* Remove some unnecessary changes
* style
* Make a stock str
* Fix a crash. Yes, this line caused a panic when reconfiguring on Android
(without a reasonable error log). Also, I could not receive any messages
anymore.
* rfmt
* Add tests and make them pass
* Even more tests
* rfmt
* Enhance test and fix bug
* Don't update the sender name when prefetching because maybe it's a mailing list
* Use an enum instead of numbers for the decision
* Don't remove anyone from mailing lists
* Fix bug in the regex
* Adjust error msg
* Compile error after rebase
* Correctly emit event
* Add dc_msg_is_mailing_list so that you can find out whether messages in the deaddrop belong to mailing lists.
* Add received headers to unit tests
* Comments, small tweaks
* Use dc_msg_get_override_sender_name instead of dc_msg_get_sender_name
* Add dc_msg_get_sender_first_name() because sometimes the first name was not correctly shown in mailing lists
* small fixes, don't let the user modify mailing list groups
* Hide contacts for addresses like noreply@github.com and reply+AEJ...@reply.github.com
When testing mailing lists, I noticed that sometimes a mailing list
contact got a name (like, hocuri <noreply@github.com>). It turned out
that ages ago, I had accidentally written an email to - in this example
- hocuri <noreply@github.com> and it had been added to the contacts
list.
This hides email addresses from the contacts list that are obviously not
meant to be written at and prevents updating the names.
* Comment, clippy
* Replace u32 with ChatId
* Resolve lost of small issues from the reviews
* remove dc_msg_get_sender_first_name
* add dc_msg_get_real_chat_id()
this allows to check if a contact request belongs to a mailing list
and to show name/avatar/whatever of the mailinglist.
another approach was to duplicate some chat-apis for messages
(eg. dc_msg_is_mailing_list()) however that would require far more new apis.
the idea to change the behavior of dc_msg_get_chat_id() would be
more clean, however, that easily breaks existing implementations
and can led to hard to find bugs.
* remove now unused Message.is_mailing_list()
* if a name for a mailing list is missing, use List-ID
* fix comment
* fix error message
* document how dc_get_chat_contacts() works for mailing lists
* refine decide api (#2185)
* add DC_DECIDE* constants to deltachat.h, tweak documentation
* use StartChat/Block/NotNow instead of Yes/Never/NotNow
* decide_on_contact_request works on ctx/msg-id
functions working on message-objects
usually do not read or write directly on the database.
therefore, decide_on_contact_request()
should not work with message-objects as well,
it is even a bit misleading, as eg. chat-id of the object is not modified.
instead, the function works on context,
similar to dc_send_msg(), dc_create_chat() and so on.
for now, i moved it to context, could maybe be part od MsgId.
* Update src/chatlist.rs
Co-authored-by: Hocuri <hocuri@gmx.de>
Co-authored-by: Hocuri <hocuri@gmx.de>
* refine documentation
* re-add accidentally deleted Param::MailingList
* remove pseudo-contact in domain @mailing.list
1. the pseudo-contact was added to the mailing list contacts,
which is not needed.
might be that we want to add the recent contacts there in a subsequent pr
(we do not know all contacts of a mailing list)
2. the pseudo-contact was used to block
mailing lists; this is done by setting the chat to Blocked::Manually now
3- the pseudo-contact was used for unblocking;
as it is very neat not to require additional ui for mailing list unblocking,
might be that we introduce a similar pseudo-contact for this limited purpose
in a subsequent pr, however, the pseudo-contact needs to exist only
during unblocking then, maybe also the special domain is not needed,
we'll see :)
* Move dc_decide_on_contact_request() up to the dc_context section as it's a member of dc_context now
More specifically, to the "handle messages" section
* re-introduce Chattype::Mailinglist (#2195)
* re-introduce Chattype::Mailinglist
* exhaustive chattype-check in fetch_existing_msgs() and get_summary2()
* exhaustive chattype-check in ndn_maybe_add_info_msg()
* exhaustive chattype-check in message::fill()
* remove dc_chat_is_mailing_list() from ffi
Co-authored-by: B. Petersen <r10s@b44t.com>
some videochat-services do not support adding random rooms to the url,
users let people in manually, so they can always use the same url.
as hacks as hiding the genrated random room behind `#$ROOM` or `?=foo=$ROOM`
does not always work, this pr introduces the parameter `$NOROOM` that is
just replaced by the empty string.
see https://support.delta.chat/t/videochat-with-webex/1412
for further details.
Return error for special IDs. Previously "No encryption." was returned for DC_CONTACT_ID_SELF.
Show "No encryption." if peerstate is reset.
Never talk about transport encryption: it is misleading. Even if TLS is used for client IMAP and SMTP connections, MTA-MTA connection may be unencrypted if MX does not support STARTTLS.
Add a Rust test.
* add failing test for From:-header bug
the test fails on the last check for quote-encapsulated utf-8 strings.
* Update mailparse
Co-authored-by: link2xt <link2xt@testrun.org>
Param should only be used to parse dictionaries stored in SQL database, not external data.
Since Param parser has been extended to escape newlines with newlines, QR-codes parser started to treat && as escaped &, which is wrong.
This change also uses String keys like "n" for "name" instead of completely unrelated constants like Param::SetLongitude.
- doc fixes
- make BobStateHandle safer by moving the state out of the handle.
- handle more match cases explicit in BobState returns
- fewer mutable variables
This does not only organise things better, but most importantly the
BobStateHandle is now not in the same module as its users. This means
it can guarantee safety about how it is initialised and use
unreachable!() to simplify it's API.
This introduces a state machine which takes care of managing the
handshake transitions in the secure-join protocol. This separates
user interactions from the protocol state handling.
This means that while handling the protocol state there are a bunch of
failures no longer possible due to all state information being
guaranteed to be present. As part of this the QR-code state has been
extracted from the generic Lot structure to something suitable just
for the SecureJoin protocol.
A LogSink has been added to the testing tools allowing log messages to
be correctly displayed on test failures.
* draft dc_msg_set_html() api
* implement setting 'html to be send'
* test sending html-parts
* more flexible html-partbuilder
* write html-parts to database and also send them
* add 'sendhtml' command to repl tool
Encryption preference was already applied since commit 78d855c5ca,
but only for already existing peerstates.
As a result, new users ignored gossiped encryption preference in the
"member added" message and had to wait until someone gossiped encryption
preferences for inactive users the second time.
This should help if parent message can't be found because messages were
reordered or deleted.
This does not prevent group IDs from being removed from Message-IDs in
the future, in which case it will become dead code.
This removes the proxy via crate::error to depend on anyhow directly.
There is no benefit to this indirection and this makes it simpler to
see which error types are used.
* add failing tests for forwarding html-mails
* let MsgId.get_html() return an option
* write html-part to local database on forwarding
* add html-part to forwarded non-dc messages
* read HTML-parts from encrypted messages
* avoid clone()
* Received:-header is no longer needed since #2152
* Update src/html.rs
Co-authored-by: Floris Bruynooghe <flub@devork.be>
* Update src/html.rs
Co-authored-by: Floris Bruynooghe <flub@devork.be>
* Update src/mimeparser.rs
Co-authored-by: Floris Bruynooghe <flub@devork.be>
* prefer 'orig' over 'org' as abbreviation for 'original'
* improve comment on tests
* prefer 'try_into()' over 'as u32' to avoid panics on bad data
* simplify ffi
Co-authored-by: Floris Bruynooghe <flub@devork.be>
instead, dc_contact_get_display_name() should be used.
dc_contact_get_first_name() was created to save some space on the screen,
esp. on mobile devices,
however, this does not always work and has issues on its own
with some names ("Dr. Strangelove", ":) Name" and so on).
as with mailing lists, more apis with first_name() would be needed,
we decided to drop that instead of following that way.
it is also less an issue as some years ago as screens have become larger,
if really, needed, the ui can handle that more gracefully,
however, just using dc_contact_get_display_name() should be fine as well.
We don't really have docs in the other crates yet, and there's a
naming conflict between deltachat and deltachat_ffi that needs to
sorted to build both. But deltachat_ffi is still documented with
doxygen for now so there's no need to sort this now.
This also checks that some of the correct user interactions happen,
checking we get a joiner event and the verified chat messages.
It also extends the test utils with the ability to distinguish the
different context logs by having them named.
Lots of new clippy lints due to toolchain upgrade.
Made the Message::error field pub(crate) again, it was the odd one out
and it seemed a reasonable way to shut up clippy.
Notable improvements:
- [track_caller](https://github.com/rust-lang/rfcs/blob/master/text/2091-inline-semantic.md): We can get the location of the caller of a message, which makes logging easier as you can have dedicated functions (not only macros), e.g. write a function `log(self: Result, msg: string) -> Option`
- The output after a panic is shorter and hides lots of uninteresting
function calls
- Test output is better captured
After merging, we should probably wait a bit with using the track_caller
feature to make going back easier, should we get problems again.
It is misleading and should never be checked.
dc_stop_io() also stops pending ephemeral message deletion tasks, so it
should be called in any case before releasing context.
Start implementing #1994
TODO (in later PRs):
- Add a hint to the watch settings that all folders are fetched from time to time (to be done in the individual UIs)
- folder names are case-insensitive, so double-check that all comparisons are case-insensitive
- The `scan_folders.rs` file didn't get as large as I expected and it's probably not worth it having an extra file for it. But if there are no objections, I'll make another PR to rename it to `folders.rs` and also put into it `configure_folders()` from `imap/mod.rs` and `needs_move()` with all its tests from `message.rs`.
Done:
- Most mailboxes have a "Drafts" folder where constantly new emails appear but we don't actually want to show them, what do we do about this? The most reliable way to detect such messages that we found up to now is:
If there is no `Received` header AND it's not in the `ConfiguredSentbox`, then ignore the email.
- before or after INBOX idle trigger a new "scan all folders for messages". It does a "list folders" and then goes through all folders with select-statements, checking if "next-uid" was changed since checked last time. This might be batchable but in any case should not consume a lot of traffic. We might debounce this scan activity to happen at most every N minutes
- if next-uid changed for a folder, we "prefetch" and "fetch" DC-messages as as needed ("dc-messages" are not just those with "Chat-Version" headers, but can also be regular emails)
- if we discover DC-messages in folders that have the "/Spam" flag (maybe excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
- for now, we do not change any user visible option, but introduce this "scan all" automatically and on top of what exists. The DeltaChat folder-watching does not perform scan-all-folders (maybe with the exception to trigger scan-all also with DeltaChat if INBOX is not watched)
- Tests (except if you have ideas to improve them)
- all folders, their last uidvalidity, next-seen etc. are kept in a separate "imap-sync" sqlite table. Maybe this can be used to streamline some of the "Sent" folder and "DeltaChat" folder detection code we already have.
- We now also move self-sent messages from the Inbox to the Sent folder if `mvbox_move` is off, as this was very easy to do now. This way, we now behave more like a normal MUA if the user wants this.
FOR LATER PRs:
- maybe for the first 50 messages or so, we could reduce the IDLE-timeout (currently 23 minutes or so) to faster detect messages sent to non-inbox folders. However, on Android and iOS, we would likely trigger scan-all when the app moves to foreground, and so it might not be neccessary to reduce the current idle-timeout at least for them. We can leave this "faster discovery" question for the end, after we move to real-life testing.
- (Later on, after the above works, we can consider heuristics on which folders to perform IDLE on, and remove the Watch-folder options (inbox, deltachat, sent). We tried to find a safe scheme for already doing it but failed to fine one, too many unknowns, also some questions regarding multi-device (you might have different settings with each of it, one moves, the other doesn't etc.) so we postponed this in favor of the above incremental suggestion.)
* Start implementing #1994
* Add debug logs, it seems like the SQL migration can go into another pr
* Let fetch_new_messages return whether there are new emails
* Code style
* Don't prefetch if there are no new emails
* clippy
* Even more debug logs
* If the folder was not newly selected, return always try to fetch as
uid_next is probably outdated
* Fix new bug
* Recognize spam folder
* if we discover DC-messages in folders that have the "/Spam" flag (excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam
* Clippy, prioritize folder_meaning over folder_name_meaning
* Add a first test, for the first day after installation only debounce to 2s
* Start adding two tests (both of them fail)
* Don't abort folder scan if one folder fails
* More consts
* Replace bool return value by enum
* Split test up into multiple tests
* Print logs during rust tests
* Rust tests pass now
* .
* One of the Python tests passes now - reconfigure folders during scanning
* Make the last test pass - Delete emails in all folders when starting the test, not only inbox and mvbox
The problem had been that emails were left in the folder "xyz"
* lint
* DB migration (untested)
* Store uid_next in SQL instead of lastseen in a config
* Revert "If Inbox-watch is disabled and enabled again, do not fetch emails from in between"
all folders are always watched, anyway
* clippy, rm debug logs, comments
* Codestyle, comments
* fixing things again
* Fix another test: don't fetch from uid_next-1 but uid_next; make some {} to {:#} so that we can use `.context(...)`
* move self-sent, non-setupmessage chat messages to the Sent folder if `mvbox_move` is off
* comment
* Comments, make sure things work even if there is no uid_next
* Style
* Comments
* The rust test tested wrongly
* comments, small codestyle change
* Ignore emails that are probably only drafts
Most mailboxes have a "Drafts" folder where constantly new emails appear
but we don't actually want to show them.
So: If there is no Received header AND it's not in the ConfiguredSentbox,
then ignore the email.
Also: Add test.
* Fix occasional test failure, it was introduced as DC now moves messages from Inbox to Sent
* Add `Received` header to the rust tests
* After this PR we will always watch all folders and delete messages there if server_delete is enabled. So, for people who have server_delete on, disable it and add a hint to the devicechat
* comment, small fix
* link2xt's first review
* Use ON CONFLICT(FOLDER) DO to update and if it doesn't exist, then insert
Reason from link2xt: We had a problem with multiple peerstates inserted due to key fingerprint parsing error previously. With logic in Rust a similar problem can occur: an UPDATE can fail for reasons other than a conflict. PRIMARY KEY should ensure uniqueness in this case, but anyway.
* Remove two TODO statements, remove fetch_new_messages: ignoring uid {}, uid_next was {} log
* Next TODO: Make uidvalidity and uid_next DEFAULT 0
* rm two TODOs, Seems like we are not going to `exclude folders that are watched anyway` in this PR
* small tweak: Handle instants more carefully
* Add scan_all_folders_debounce_secs config for tests, set debounce to 60s (before it was just 2s during the first day)
* Don't use bold letters for the device message
* React to changes in the folders better
Before, if there was a configured Sent folder, but then it got
removed and replaced with another folder with a name meaning "Sent" but
without Sent flag, it would be ignored.
So, instead of checking against ConfiguredSentboxFolder,
create two Option variables at the beginning of the loop and replace
them with Some if it is None. At the end of the loop, store the new
values into ConfiguredSendboxFolder and ConfiguredSpamFolder, even if it
is None.
Also, derive some useful traits.
* move job: Return a meaningful error if server_folder is None instead of panicing
* small error-handling fix
* Fix test_fetch_existing() python test
Before, we sometimes got a race condition where scan_folders() sees that
there is a Sentbox and saves this info after we set the
ConfiguredSentbox to None and before the message is sent.
So, just expect that the message is moved to the sentbox.
* migration is 72 now
* rm 2 TODOs, Don't infinitely retry when dc_receive_imf() returns Err
* clippy: Remove glob imports
* Delete created folders at the beginning of tests
(some created folders made problems in the next tests because)
* Improve resetting accounts between tests
* draft API to deal with uncut message texts
* add column mime_modified
* add mime_modified flag to MimeParser and save it in the database
* save mime_headers also when mime_modified is set
* cargo fmt
* set mime_modified on parsed html-texts and when there are multiple alternative-parts; add test for that
* prototype functions, add to repl and ffi
* use correct mime_modified flag
* basically parse Mime-Structure to HTML
* add basic tests for HTML-parsing
* convert text/plain to html for getting original
* respect charset for plain texts
* make test more specific
* fix handling non-utf-8 charsets for plain messages
* add test for plain_to_html()
* add failing test for plaintext linkify
* linkify urls in plain text
* fix regex
* plain text linkify: add failing test for encapsulated links as <https://domain.com>
* plain text linkify: make encapsulated links as <https://domain.com> work
* plain text linkify: require word boundary at beginning of link, add tests for that
* plain text linkify: linkify emails
* plain text: support format=flowed
* plain text: support quotes
* make clippy happy
* set mime-modified also when simplify() cuts non-html messages, add tests for that
* streamline mime recursion
* repl tool: write original html to file for further processing
* convert cid:- to data:-protocol
* add a test for cid: to data: conversion
* make clippy happy
* fix html-tests to work with windows-lineends
* clarify what the returned html-code may contain
* add some more detailed doc comments
* add mime_modified column only if not exist
this additional check is needed
as the column may added with another dbversion in
some shipped beta-versions.
* incorporate documentation suggestions from review
* rename get_original_mime_html() to more simple get_html()
* rename api is_mime_modified() to more simple has_html(); internally, mime_modified-flag stays as-is, however
* rename MimeS to MimeMultipartType
* do not set mime-modified flag for encrypted messages that need extra-handling for saved mime-structure
* fix typo
* move get_msg_html() to MsgId.get_html()
* incorporate more documentation suggestions from review
* remove unused return value from collect_texts_recursive()
* avoid mime_modified being mutable in write-parts-loop
* move 'use futures::future::FutureExt' atop of html.rs
* move attributes defining plain-text to a dedicated structure
* more PlainText to separate file
* escape cid when building regex
* let dc_get_msg_html() return NULL when calling with bad param
in addition to merged pr #2123,
dbversion=70 was also used by pr #2125.
pr #2125 is not yet merged, however beta-versions were shipped with
that pr, so, dbversion=70 is already in use on some systems and on these
systems ConfiguredProvider won't be set.
the other way round, that ConfiguredProvider is set twice on some systems
affects only dev-installations and should not harm.
once this pr is merged, #2125 needs a rebase and will use dbversion=72 for
its changes to add the new column also n the dev-installations :)
(we need to check that the column to add does not exist, however)
Previously MX records were queried only for OAuth 2 configuration and
did not affect the list of servers tried. User was required to manually
configure the servers for Google Workspace (former GSuite) domains.
Now MX records are queried during configuration. If provider is found in
offline database, its ID, corresponding to the filename, is saved as
`configured_provider`.
`configured_provider` is also set during database migration if email
address uses the domain from the provider database, but no MX querying
is done.
- Make sure delete_expired_messages and housekeeping runs once a day
- delete more info about messages when putting them to trash (esp. also
delete txt_raw, from_id and to_id as we don't need those anymore, so
they are data that was unnecessarily kept)
fix#1926, fix#2090
Also:
* Nicer test_utils: add send_text() and print_chat()
* Adapt ephemeral messages for testing (make them accurate to the second)
* Add test for ephemeral messages
* Make pop_sent_msg() really pop the last sent message
If mvbox_move was on and an mdn was received, it was not marked as read.
Also remove a confusing log that said that we are adding a markseen job, even if we weren't. As job::save() logs itself, there is no need to log this again.
adding an asterisk to the filename key
REQUIRES two apostrope-delimiters in the value as of charset'lang'name,
see rfc2231
as we do use the value this way, we MUST not add the asterisk.
things worked correctly in the past as
consuming MUAs are pretty graceful.
i assume the error comes from the c to rust conversion -
core-c did add the asterisk, however also did the apostrope-encoding.
core-rust leaves the asterisk but changes encoding.
disabled and re-enabled
That's another narrow-stitching patch for a scenario where many emails could
be deleted all at once and surprisingly: user disables inbox-watch, enables delete-from-server, after a moth enables inbox-watch again -> currently all emails that arrived in the meantime will be deleted (if emails are not fetched, they won't be deleted)
This way we can better distinguish tagged from untagged core releases, also in logs etc.
We might, from time to time, increase the alpha.N "N" number if we are entering testing etc.
providers may have a maximum for the SMTP RCPT TO: header,
therefore, an outgoing message is split into several chunks.
the chunk size defaults to 50, however, if known to be larger or smaller
by a dedicated provider, this setting may be changed.
this does not affect the MIME To: headers,
if a provider has a maximum here, this is change would not help.
ephemeral_task holds a reference to Context, preventing event emitter
from returning NULL and terminating event loop. Prior to this change,
there was no way to quickly terminate pending ephemeral_task.
the saved-messages can be deleted as any other chat;
if the feature is unneeded, this probably also makes some sense.
however, if saved-messages was deleted accidentally (yeah, i've seen that :),
it is not intuitive to see how the saved-messages feature can be restored.
this change just drops a note about how-to-restore to the device chat.
fix#2020
When forwarding a message, the original `in_reply_to` stays in place.
I did not find any recent commit that changed anything about this,
but IIRC there was a bug that prevented setting the `in_reply_to` at
all.
So, for chat messages, do not look at the InReplyTo header (and also not
at the References header)
by #2001, processing of protection-enabled and protection-disabled
messages is only done if verification-checks pass.
unfortunately, these verification checks
were only applied on already protected chats,
so that enabling via a protection-enabled message was not possible.
this is fixed by this pr.
moreover, when inner_set_protection() fails,
the error is shown in the chat.
this avoids confusion - as long the current core
does not communicate with a UI that enables/disables protection,
everything looks just the same as in the past :)
update_device_chats() takes about 2 seconds on a modern device (Android) because the
welcome image file has to be written to the disk as a blob. The problem
was that this was done after the progress bar had vanished and before
anything else happened so that I thought that something had gone wrong
multiple times.
The UIs have to remove update_device_chats(), too..
this would exclude non-deltas from protected-chats
(where they could participate in verified-groups in the past)
and makes migration a bit harder when using a fuzzy multi-device-setup.
however, the test still takes place
when a group is created or protection is enabled/disabled.
should_encrypt() shall return Ok(false)
on peerstate=EncryptPreference::Reset
only for opportunistic groups.
for verified/protected groups (e2ee_guaranteed set),
Ok(true) or Error() should be returned.
this bug was introduced by #1946 (Require quorum to enable encryption).
a draft may contain only a quote,
without any text set yet.
these drafts cannot be sent, however, appear in the summary -
currently with the summary-text "", which results to sth.
as "Draft: " - which looks like an error or at least a bit weird.
this pr sets the summary text to "Reply" - similar to "Image", "Video" etc. -
the UI just expects some text here, not an empty string.
the result are summaries as "Draft: Reply" on all UIs -
which, btw. is also roughly the same what Signal does in this case.
the constant comes from c-core
and was used to define the size of the string-array that time.
this is no longer needed in core
and also the UIs seems not to use it
(they will treat DC_STR* like an id,
there should be no need to know the max. DC_STR* value)
Read all of an e-mail accounts messages and extract all To/CC addresses
if the From was from our own account.
Then, fetch existing messages from the server and show them.
Also, I fixed two other things:
- just by chance my test failed because of an completely unrelated bug.
The bug: bcc_self messages were not marked as read if mvbox_move was set
to true.
- add some color to the test output (minor change)
Sticky encryption rule, requiring that all replies to encrypted messages
are encrypted, applies only to messages with a quote now.
Co-Authored-By: B. Petersen <r10s@b44t.com>
This code is inconsistent with EncryptHelper::should_encrypt, which uses
quorum rule.
UI does not display or use encryption state in-between preparing and
sending message anyway.
In addition, "sticky encryption" rule, which required all replies to
encrypted messages to be encrypted, is dropped. It is going to be
restored with the introduction of quoting.
both are not used in productive:
- chat-id "in-creation" was dropped completely already in core-c
as it does not allow assigning messages to chats in time.
- message flag "starred" was added at some point and never used.
ux-wise, "disappearing messages" promises to clean up a chat -
this does not fit well with "starring" messages.
"saved messages" chat seems to be superior in that regard.
but anyway, it is not used but comes at maintainance costs,
so it is better to delete its functionality for now.
(the db entries, however, are left for now,
that might be more tricky to remove)
Address is used to generate Message-Id.
Previously error toast was shown, but MsgId was returned from
chat::prepare_msg anyway. Now chat::prepare_msg returns an error.
error!() is removed, because FFI library shows errors via
.unwrap_or_log_default().
When run against host that is not in the provider database or has multiple
IMAP servers listed, this test fails because one error is returned for
each server.
* Mention in dc_event_get_data2_str method that it can also return 0
* Update deltachat-ffi/deltachat.h
Co-authored-by: bjoern <r10s@b44t.com>
Co-authored-by: bjoern <r10s@b44t.com>
Previously, standard Autocrypt rule was used to determine whether
encryption is enabled: if at least one recipient does not prefer
encryption, encryption is disabled.
This rule has been problematic in large groups, because the larger
is the group, the higher is the chance that one of the users does not
prefer encryption.
New rule requires a majority of users to prefer encryption. Note that it
does not affect 1:1 chats, because it is required that *strictly* more
than a half users in a chat prefer encryption.
partly, we wrote '@return None.' for void functions,
however, some linter complain about that (recently swift)
and they're right, it does not make much sense.
On Android, there is to /tmp directory, so SQLite may throw
SQLITE_IOERR_GETTEMPPATH when trying to find a place for temporary
files. Storing temporary files in memory is a workaround that always
works (until out of memory).
See https://github.com/mozilla/mentat/issues/505 for discussion of a
similar issue.
In the current design the dc_receive_imf() pipeline calls
handle_securejoin_handshake() before it creates the group. However
handle_securejoin_handshake() already signals to securejoin() that the
chat exists, which is not true.
The proper fix would be to re-desing how group-creation works,
potentially allowinng handle_securejoin_handshake() to already create
the group and no longer require any further processing by
dc_receive_imf().
'Delete emails from server' was an experimental ad-hoc function
that was added before 'Automatically delete old messages' was introduced
to allow at least some way to cleanup.
'Delete emails from server'-UI was already removed from android/ios in june
(see https://github.com/deltachat/deltachat-android/pull/1406)
and the function never exists on desktop at all.
This currently tests the setup-contact full flow and the shortcut
flow. More securejoin tests are needed but this puts some
infrastructure in place to start writing these.
This commit fixes condition which ignores reordered autocrypt messages.
If a plaintext message resetting peerstate has been received, autocrypt
header should only be applied if it has higher timestamp.
Previously, timestamp of the last received autocrypt header was used,
which may be lower than last_seen timestamp.
# Conflicts:
# src/peerstate.rs
Previously DC_EVENT_ERROR_NETWORK events were emitted for each failed
attempt during autoconfiguration, even if eventually configuration
succeeded. Such error reports are not useful and often confusing,
especially if they report failures to connect to domains that don't
exist, such as imap.example.org when mail.example.org should be used.
Now DC_EVENT_ERROR_NETWORK is only emitted when attempting to connect
with existing IMAP and SMTP configuration already stored in the
database.
Configuration failure is still indicated by DC_EVENT_CONFIGURE_PROGRESS
with data1 set to 0. Python tests from TestOnlineConfigurationFails
group are changed to only wait for this event.
This turns the Bob::expects field into an enum, removing the old
constants. It also makes the field private, nothi0ng outside the
securejoin module uses it.
Finally it documents stuff, including a seemingly-unrelated Param.
But that param is used by the securejoin module... It's nice to have
doc tooltips be helpful.
LoginParamNew structure, which contained possible IMAP and SMTP
configurations to try is replaced with uniform vectors of ServerParams
structures. These vectors are initialized from provider database, online
Mozilla or Outlook XML configuration or user entered parameters.
During configuration, vectors of ServerParams are expanded to replace
unknown values with all possible variants, which are tried one by one
until configuration succeeds or all variants for a particular protocol
(IMAP or SMTP) are exhausted.
ServerParams structure is moved into configure submodule, and all
dependencies on it outside of this submodule are removed.
in general, Viewtypes other than File should be used only
when the added file type is tested on all platforms -
including Android4 currently.
as this is easily overseen,
i've added a comment.
this partly reverts Viewtype changes done by
https://github.com/deltachat/deltachat-core-rust/pull/1818
Fix#1791 and show all group messages if the user already is in the group, even if the sender is blocked
Also fix a comment
Co-authored-by: link2xt <ilabdsf@gmail.com>
This fixes#1804 in two ways: First, it removes a <!doctype html> from
the start of the mail, if there is any.
Then, it parses the html itself it quick-xml fails, just stripping
everything between < and >.
Both of these would have fixed this specific issue.
Also, add tests for both fixes.
threading was broken in core43 as this flags unencrypted messages as errors
and errors are not replied-to.
the fix is not to mark missing signatures for unencrypted messages as errors.
.tgs files are Telegram stickers. Internally they are gzipped JSON files,
containing a single Lottie animation.
MIME type application/x-tgsticker is commonly used for telegram documents
containing such stickers.
An EphemeralTimerChanged message with the same timer as already set can
be received when there are large delays or lost messages. Even though
inner_set_ephemeral_timer should not be called in this case, because it
emits an event indicating timer change, system message will be added to
the chat, so it should be translated with set_better_msg in any case.
Normally NULL is used when there is no fingerprint, but default value
for fingerprint columns is an empty string.
In this case, loading should not fail with an "invalid length" error,
but treat the fingerprint as missing.
Strict check was introduced in commit ca95f25639
Since try_decrypt does not check if the message has valid signatures
anymore, it may return empty signatures set, which means the message is
not a valid autocrypt message.
this pr allows prefixing custom jitsi urls with `jitsi:`
so that clients have a chance to detect these custom instances
and may start the corresponding app.
introduce the qr-code-type `DCWEBRTC:[webrtc_instance]`.
dc_check_qr() returns this as the type DC_QR_WEBRTC
and these qr-codes can be passed to dc_set_config_from_qr() then;
this finally sets `webrtc_instance` using dc_set_config().
needed for how node/js handles the chatlist currently
(it convertes the ffi-object to an js-object and throws ffi-object away directly,
this makes it hard to access dc_chatlist_get_summary() later.
Since introduction of provider database, these certificate checks are
enabled for test servers anyway, but this setting prevents usage of
local servers with self-signed certificates.
It presents no security issue, because properly configured servers will
only serve passwords on HTTPS and distribute only HTTPS QR codes, but
makes testing easier when HTTPS is not easy to deploy.
If attacker can control the URL used, they can change the URL to another
HTTPS URL controlled by them and act as a proxy between the client and
original server anyway.
Adhoc groups for group messages that don't have Chat-Group-ID are already
created above in another create_or_lookup_adhoc_group.
At this point it could be that there is a valid Chat-Group-ID header,
but no group was created because removed_id was non-zero i.e. received
message removes some group member.
If group was not explicitly left, current code creates an adhoc group
instead of trashing late or reordered messages that remove group
members. Such groups have adhoc group IDs locally, but proper group
message-IDs. Attempts to reply to such groups or leave them creates
"split groups" for all other members of original group. The solution is
not to create adhoc groups in this case.
Permanent error 550 5.1.1 is no longer considered temporary.
Enhanced status code is checked now, so only 550 5.5.0 is an exception
for misconfigured Postfix servers.
Yandex error 554 5.7.1 was handled correctly, but only because it had
response code 554, while the comment talks about enhanced status code
5.7.1. The comments are corrected.
Failed messages are now marked as such with message::set_msg_failed.
Previously they were left in a pending state.
If info message cannot be added to the chat, the error is displayed with
error! instead of being logged with warn!.
when seeing our own profile image send from other devices,
we use them as Selfavatar on the current device as well.
as there is no special message sent on avatar changes,
this is a simple approach to sync the avatar across devices.
Now MarkseenMsgOnImap sends MDN even if it can't mark the message as
seen on the server.
To prevent multiple MDNs from being sent, MarkseenMsgOnImap is postponed
until the message is detected in a folder from which it is not going to
be moved.
Mimeparser.was_encrypted() checks if the message is an Autocrypt encrypted
message. It already means the message has a valid signature.
This commit documents a few functions to make it clear that signatures
stored in Mimeparser must be valid and must always come from encrypted
messages.
Also one unwrap() is eliminated in encrypted_and_signed(). It is possible
to further simplify encrypted_and_signed() by skipping the was_encrypted()
check, because the function only returns true if there is a matching
signature, but it is helpful for debugging to distinguish between
non-Autocrypt messages and messages whose fingerprint does not match.
Add a mutex to prevent a race condition when a "your pw is wrong" warning is sent, resulting in multiple messeges being sent.
Do not mute the device chat but "only" send MsgsChanged event when no
notification shall be shown.
More logging.
Parallel processing of messages results in bugs such as messages sent by
a new member being processed before the message that adds this member to
the chat, even when it has lower UID. In this case, messages are shown as
"[Unknown sender for this chat. See 'info' for more details.]", while
there is a message "Member ... added" right before, because writing the
information about new member to the database takes longer then reading
old index from the database.
With this change, API user does not need to look at the timestamp of
the next message to display the day marker, but can get the timestamp
directly from the marker. This way there is no database query and no
risk of error as a result of database being busy or message being deleted.
instead of applying all config_defaults unconditionally
after the first configure, a config_defaults for a given key
is now applied when this key has never been set before.
this way, you can set some keys before calling configure()
and also, later versions can add new defaults for new keys
(that would require another call to configure() then, however)
for "fresh" messages, calc_sort_timestamp() makes sure,
the sorting timestamp is not less than the last-non-fresh message.
this commit fixes the "fresh" state we give to calc_sort_timestamp(),
it just uses the message-state we've already calculated.
the old assumption, that unseen messages are always fresh
is wrong as this "seen" flag just comes from an imap-flag
that is not set for self-sent messages.
therefore, in a multi-device-setup, things are totally messed up.
(the bug was probably in there since a long time,
however did not came to light until the async-move)
This tidies up our testing tools a little bit. We had several
functions which through various changes ended up doing the same and
some more which did very similar stuff, so I merged them to have
things simpler. Also moved towards methods on the TestContext struct
while cleaning this up anyway, seems like this structure is going to
stay around for a bit anyway.
The intersting change is in `test_utils.rs`, everything else is just
updating callers. A few tests used example.org which I moved to
example.com to be able to re-use more configuration of the test
context.
the last message shown in a chatlist
is the one with the largets timestamp that is not hidden.
in the past, we calcualted the last timestamp using a subquery
and uses that timestamp to finally get the message.
this may fail when there are two messages with the same max. timestamp.
with this fix, we return the id from the subquery and use that
(the subquery already filters by hidden etc.)
in practise, by time-smearing,
usually delta-chat avoids messages from the same device
having the same timestamp - however, this may not be true for multi-device
and/or read-receipts.
i have not seen this error all the years, however, it happens with
the async move several times - maybe because things are just sent faster
and things become more probabe.
Fix #1478
I changed my original plans a little because I had so many extra ideas and then sorted that I should rather look at actual NDNs and look what is necessary to parse them:
- Recognize NDNs by ~the sender address, which is in a regex the providers database. The problem with heuristics would be that someone could send fake-NDNs and mark messages as failed.~ the standard ("report/delivery-status") and heuristics ("subject contains 'fail' and sender contains 'daemon'"). If there is a valid Message-ID, then rely on that this is an NDN (of course, generally someone might try to find out a Message-ID and send a fake NDN).
- ~Look for `In-Reply-To`~ (only Gmail did this and Gmail uses rfc822 anyway.)
- ~Look for a mimepart `message/delivery-status`, which might contain a `X-Original-Message-ID`~ (only Gmail did this and Gmail uses rfc822, too, anyway.).
- Search through the body and look for a line `Message-ID: *` (remember to remove `<`, `>`), in the hope that that's the original header
- Look for a mime-part containing the string `rfc822`, which will contain the original header. Parse them with Mailparse and look for `Message-ID`.
CLOSE, which was used previously, only expunges messages and deselects
folder, and it should only be called if some folder is selected. For that,
Imap.close_folder() method is used.
This adds functionality to the cffi build script to also extract
defines so that we can ask the compiler to figure out what the correct
values are. To do this we need to be able to locate the header file
used in the first place, for which we add a small utility in the
header file itself guarded to only be compiled for this specific case.
Reply to group messages assigned to 1:1 chat on ac4 and see where they
end up on ac1 and ac2. Currently they contain group ID in Reply-To field
and go to group chat, so the test is failing.
The code in dc_receive_imf.rs looks a bit funny, an alternative would be a function:
fn upcate_chat_last_subject(context: &Context, chat_id: &ChatId, mime_parser: &mut MimeMessage) -> Result<()> {
let mut chat = Chat::load_from_db(context, *chat_id)?;
chat.param.set(Param::LastSubject, mime_parser.get_subject().ok_or_else(||Error::Message("No subject in email".to_string()))?);
chat.update_param(context)?;
Ok(())
}
get_rowid should not return 0, as we have inserted a row right above.
And using is_error() instead of comparing row_id to 0 is a strange way
to check this condition.
As all functions that actually returned 0 chat ID to indicate error have
been removed, the function is gone too.
Both methods do the same: compare chat_id to 0. However, in these cases
0 refers to the state when chat_id is not determined yet, because no
corresponding chat has been found.
All functions that returned 0 to indicate error have already been
resultified.
This uses the Fingerprint type more consistenly when handling
fingerprits rather then have various string representations passed
around and sometimes converted back and forth with slight differences
in strictness.
It fixes an important bug in the existing, but until now unused,
parsing behaviour of Fingerprint. It also adds a default length check
on the fingerprint as that was checked in some existing places.
Fially generating keys is no longer expensive, so let's not ignore
these tests.
the advanced options are not used anyway later,
but prevent imap/smtp connections from being altered.
nb: we want to stop altering when some advanced options
are entered, however, we want to do this probaby
not depending on autoconfig.
Fix#1474 "Sending message to contact with < or > in Recipient gets treated as "Sent" but is not received".
As I was at it, I also extracted the correct name and address from addresses like Mueller, Dave <dave@domain.com>.
This means all key conversions/serialisation/deserialisation can be
done with DcKey rather than Key. Also migrate all key conversion
tests to DcKey rather than Key.
This moves both the Keyring and the fingerprints to the DcKey trait,
unfortunately I was not able to disentangle these two changes. The
Keyring now ensures only the right kind of key is added to it.
The keyring now uses the DcKey::load_self method rather than
re-implement the SQL to load keys from the database. This vastly
simpliefies the use and fixes an error where a failed key load or
unconfigured would result in the message being treated as plain text
and benefits from the in-line key generation path.
For the fingerprint a new type representing it is introduced. The aim
is to replace more fingerpring uses with this type as now there are
various string representations being passed around and converted
between. The Display trait is used for the space-separated and
multiline format, which is perhaps not the most obvious but seems
right together with FromStr etc.
the document standards.md exist since a long time
to get an overview but also a nice reference to get some links.
however, its placement in the android-repo is a bit unlogical,
probably comes from a time where there was only one implementation :)
as ios and desktop use the same standards and as
all acutal implementations are in the core,
we move it to the core repo.
android can have a link to the new place.
journal_mode is persisted, it is sufficient to change it only for one handle.
with_init() is called for a bunch of handles directly on pool-creation,
so changing to WAL here easily results in busy-errors.
search for the sentbox in two passes:
- first check the folder attributes
- only if that fails, check for some known names
this way, the sent-attribute always has precedence
over the name; this was not the case before.
moreover, this fixes an possibly wrong early exist
when the attribute list is completely empty.
The condition remains from the time when expired messages were hidden
instead of being moved into trash chat. As a result, old hidden
messages, such as location messages, were not deleted.
This function should only return temporary errors, e.g. database errors,
as precheck_imf() and dc_receive_imf::add_parts() treat them as such,
retrying the fetch on failure. When permanent errors, like missing
Message-ID, are bubbled up, they cause infinite fetch loop.
* First try making get_recipients use MailHeader (nice and functional)
* Get it to compile by using not-so-functional style
* Add "empty-from" test, drop unnecessary check for error; continue using addrparse_header() instead of addrparse()
* Try to use functional style, unfortunately, I can't get the compiler to accept it
* Do it imperative-style: Do not overwrite To with Cc and vice versa
* Use addrparse_header() once more
* Still addrparse_header()
* Clippy
* Fix compile errors in tests
* Fix typo
* Fix tests again ;-)
* Code style
* Code style; try a HashMap<addr: String, display_name: String> as an address list but I am not convinced
* Code style; Use Vec<SingleInfo> as address list
* Clippy
* Add tests
* Add another test
* Remove stale comments
If error is ignored, the message will never be fetched again, even if
there was a database write error.
dc_receive_imf itself is modified to ignore unrecoverable errors, to
prevent endless refetching of incorrect messages.
these events take the message-id as parameter and might be used
to update an existing list (although to recommended)
if the event is issued for hidden messages,
this might led to "empty" messages flashing up -
the ui tries to get the message from the event,
after a moment, on the next update, the message disappears again
as hidden messages are of course not returned eg. by dc_get_chat_msgs().
the effect was probably always visible for secure-join-messages on ios,
however, become much more visible recently when read-receipts are added
as hidden messages as well (to make them auto-deletable).
the footer mark normally used in email-conversations is `-- `,
note the trailing space, see RFC 3676, §4.3
unfortunately, the final space is removed by some providers,
which lead to footers showing up on delta-to-delta-conversations
(on nondc-to-delta, this is not an issue as we cannot be sure anyway
and show a [...] therefore)
this change accepts lines with only `--` as a footer separator
if there is no other footer separator
and if the line before is empty and the line after is not.
as there is still some chance to remove text accidentally,
see tests, some protection against that is needed in another commit.
Let the thread load new messages. This may happen when user switches
the setting to delete messages on the server on and there are a lot of
messages to delete.
This prevents generation of a large number of jobs when IMAP deletion
setting is enabled for the first time. Writing jobs to the database
locks it for readers and may cause UI freezing, because chatlist and
messages can't be read until all jobs are written.
Note that on failure job will be written to the database, to make sure
it is postponed instead of being retried immediately.
Job::new() can be used to create jobs in-memory
Job.update() is replaced with Job.save() which can create new database
entries and consumes Job to avoid the need to update job ID after saving
it to the database.
This moves the loading of the keys from the database to the trait and
thus with types differing between public and secret keys. This
fetches the Config::ConfiguredAddr (configured_addr) directly from the
database in the SQL to simplify the API and consistency instead of
making this the responsiblity of all callers to get this right.
Since anyone invoking these methods also wants to be sure the keys
exist, move key generation here as well. This already simplifies some
code in contact.rs and will eventually replace all manual checks for
existing keys.
To make errors more manageable this gives EmailAddress it's own error
type and adds some conversions for it. Otherwise the general error
type leaks to far. The EmailAddress type also gets its ToSql trait impl
to be able to save it to the database directly.
If precheck_imf fails to check if message with the same rfc724_mid
already exists, the same message may be downloaded twice. Instead,
abort the whole operation and retry later.
For hidden messages, blobs are not deleted during housekeeping. To
actually free the space used by media files, messages should be moved to
trash instead of being hidden.
Some mobile email clients, such as apple mail, attach or inline images
after description, just like Delta Chat. It is better to display them
instead of ignoring.
Since introduction of server message deletion, message may not have
server UID even when it is not a draft or pending message. To handle
such messages correctly, we explicitly check message state.
Now there is only one function: hide_device_expired_messages().
If any messages are hidden event DC_EVENT_MSGS_CHANGED(0,0) is emitted
now, which is more correct than DC_EVENT_CHAT_MODIFIED and also triggers
chatlist reload.
In contrast to get_config_delete_server_after(), value 1 does not mean
"delete at once", because it does not make sense to delete messages
immediately after receivning them.
MarkseenMdnOnImap stored server folder and UID which are never updated
by update_server_uid. Now hidden entries are created for MDNs, so they
should be handled as ordinary messages.
Sometimes message deletion job marks message as unlinked without
actually deleting it. It is possible if the message was already moved
into another folder, possibly by second device, but not detected there
yet. It should be detected later in the other folder, and the
server_uid in the database should be set.
Since introduction of add_imap_deletion_jobs() any expired message
record will be marked as unlinked eventually, because another message
deletion job will be scheduled even if update_server_uid() resurrects
the message once.
This may result in messages not being deleted. Optimization and
traffic-saving is postponed for later, one idea is to optimize message
deletion to avoid checking if Message-ID on the server matches
Message-ID in the database.
If job returns Status::Finished, it will be deleted. Then
add_imap_deletion_jobs will recreate it immediately if the message is
expired. To actually backoff the job, we should postpone it instead of
removing.
This is to avoid creating thousands of jobs when user enables
"delete_server_after" setting for the first time.
If device is offline for more than 2 weeks, some messages may not be
deleted.
When "delete_server_after" setting is configured, postponed
DeleteMsgOnImap jobs are created for incoming messages.
This commit adds job::add_imap_deletion_jobs function which creates
DeleteMsgOnImap jobs right before performing IMAP jobs. This way even
messages that expired when the setting was disabled are going to be
deleted.
Job creation on message reception is unnecessary now, and even harmful
because it will create jobs with an expiration time which may later be
reduced. It is planned to be removed in following commits.
Now 0 means "never delete", 1 means "delete at once" and other values
indicate the number of seconds after which them message should be
deleted from the server.
Configuration value interpretation is moved into
Context.get_config_delete_server_after() function.
At least one entry is required for DeleteMsgOnImap job. Additionally,
adding a hidden entry makes it possible to avoid redownloading the
message if it gets a new UID on the server.
DeleteMsgOnImap deletes files from the server only when the last part
is deleted. Removing only the first part of the hidden or trashed
message does not result in message deletion.
Currently only trashed or hidden messages are deleted by
DeleteMsgOnImap, so it is safe to remove database records.
It is planned to delete messages on IMAP server after
user-configurable time to cleanup the server even for messages
displayed in chats. For such messages, we unlink them from the
Message-ID, but keep the database record to display them.
Mutable UID reference is removed. Instead, ImapActionResult is
returned immediately.
If message has changed and UID does not correspond to expected
message, ImapActionResult::Failed is returned. Temprorary IMAP errors
result in ImapActionResult::RetryLater.
On the job side, ImapActionResult::RetryLater does not result in
immediate job retry anymore. Instead, job is retried with a backoff.
In addition to moving the message into trash chat, this function
removes message text to make sure the message does not remain in the
database. Only the information necessary to delete message from the
server and avoid redownloading it should be kept, such as Message-Id
and IMAP UID.
CString::yolo was still used in the ffi, this was an unsafe
transitional thing. To remove it there were two choices: 1. make
errors in creating CStrings hard errors or 2. try and be as lenient as
possible. Given the to_string_lossy() convention adopted in the ffi
this choose the lenient option and simply skips over embedded null
bytes, leaving the rest of the strings intact.
Thus now CString::new_lossy(). It's only used for .strdup() however
so no longer a public trait.
This also cleans up the public visibility of things in the strings.rs
file:
- Rename StrExt/OptStrExt traits to what they actually do: provide
.strdup() -> Strdup/OptStrdup.
- dc_strdup() should be an implementation detail, replace all usages
with Strdup.strdup() method.
- Only allow visibility inside the crate for all things.
- Reduce visibility to only the module for things not used in lib.rs.
after fixing some issues wrt ecc keys, see #1319,
and waiting some time (three core releases, two ios/android/desktop releases),
it is now the time to switch again to ecc keys again,
after the first attempt was stopped in #1319
before, the list was empty
and trying to send to groups that only contain SELF lead to a crash.
in theory, this happens for both, bcc_self enabled or not, however,
if bcc_self was disabled (default setting),
things worked as the whole mimerendering was skipped on a higher level.
also, the saved-messages-chat was not affected as this was checked explicitly.
this pr changes the mimerendering so that From: is used as To:
if there is no recipient-list and the messasge will be sent to SELF only.
the currently released versions fail sometimes in encrypting to ecc keys,
see #1313, the issue is about to be fixed,
however, we should not generate ecc keys until the
fix is rolled out - otherwise new users will get encryption errors every some
messages if their counterpart is not yet using the most recent version.
we can start generating ecc keys a few weeks after the fix is rolled out.
up to now, Chat-Group-Member-Added and -Removed commands
result in a complete recreation of the memberlist
by collecting all addresses from the From: and To: headers.
this easily results in missed and accidentally removed members,
esp. when several people at the same time scan a qr code to join a group.
this commit changes the behavior of adding members by
not removing members on the Chat-Group-Member-Added command.
instead the existing memberlist is conjuncted
with the memberlist seen in the message.
only adding the member from the Chat-Group-Member-Added
seems not to be sufficient - imaging a group of Alice and Bob:
- Alice adds Claire
- Bob adds Dave _before_ seeing that Alice added Claire
- Dave would never get the information that Claire is in the group
wrt Chat-Group-Member-Removed: this command
does no longer recreate the memberlist but just remove _exactly_ the member
mentioned in the header. there are situations, where a just removed member
will be readded by out-of-order-messages, however, compared to missed
members, this is seems to be acceptable - also as this is more visible
and easier to fix (just remove the member again).
might be that, in practise, this is not a big issue. while adding members
is typically done in masses on bootstraping a group,
this is typically not true for removing members.
Instead, use last_insert_rowid() function to find the row.
There is no race condition in using last_insert_rowid(), because
last_insert_rowid() returns row id last inserted in this connection. As
we hold the connection during the whole transaction, it is impossible
that some other thread will execute INSERT statement in parallel.
This commit is part of the effort to get rid of sql::get_rowid hack and
use transactions more for related SQL statements.
Existing public methods that use these functions, like sql::execute, are
only suitable for executing a single statement.
Sometimes it is useful to execute multiple statements within one
connection, for example to begin a transaction, execute mutliple SELECT
and INSERT queries and commit or rollback the whole transaction.
We already have a .strdup() method on AsRef<str>, this adds this
method also to an option of this. In case the option is None a NULL
pointer is returned.
This is done by using a new trait, as the type system otherwise
considers such an implementation conflicting with the existing one.
Now configure module is no longer public. Users should call
Context.configure() and Context.is_configured() methods.
Configure module is completely hidden from documentation unless
--document-private-items option is specified.
RFC 2183 specifically allows filenames to be specified for MIME parts
with "inline" Content-Disposition.
Previously we treated such attachments as an error, causing the whole
message parsing to fail. Now it is only an error if Content-Disposition
cannot be parsed.
MIME parts with filename are now considered to be attachments
regardless of their Content-Disposition type. However, "attachment"
Content-Disposition is still processed differently: we generate a
filename for it even if it is not specified.
I had a look for two mintues at converting things to .try_inner() and
I don't think this is currently worth the effort or would increase
readability a lot. So let's leave this for now.
There is no need to have both a .log_warn() and .log_err(), even their
names are confusing by now. Let's just have the most liberal one and
reduce one more thing we need to know about.
Also, these traits don't need to be pub.
Using peek_online_config() results in a message being sent to some
other account, used by previous test. If tests are run in parallel,
for example with pytest-xdist, we could be sending a message to a still
online DC client.
Envelope has the same Message-ID, but using other fields from the
envelope, such as From field, is error-prone. They may contain values
different from the message body. As we don't parse the envelope later on,
it is better not to fetch it during prefetch too.
This capability is not used by Delta Chat. Moreover, XLIST is deprecated
in favor of https://tools.ietf.org/html/rfc6154, which should be
supported instead. Delta Chat already looks for \Sent attribute in
get_folder_meaning().
- Python should raise exceptions on error. Not return False.
- Negative durations are nonense.
- ChatID validity is already checked by the Rust API, let's not duplicate.
Configuring an online account generates a key, we would like this
code-path tested too. So add some functionality to the AccountManager
to not use the pre-generated keys.
Because this slows down interactively running the tests by hand add an
ignored marker which only runs if --ignored is used. This name was
chosen because this matches the naming used by rust/cargo #[ignored].
The difference however is that --ignored on cargo *only* runs ignored
tests while here it runs *all* tests.
To ensure the ignored/slow tests are run on CI we add it as an
argument to the tox configuration, which is used by the CI to run the
tests.
This changes the AccountMaker to use pre-generated keys when
available, speeding up test runs.
As a side-effect we no longer need to compile the integration tests in
release mode with debug symbols. Losing debug symbols (-g) means
cargo no longer wants to recompile everything all the time too.
Tested locally and seems to works.
The imex code was deleting the key if it was already used. Turns out
none of the callers would not want this behaviour, so let's just
handle this case generally.
I guess this is a little neater. Note that this does change all the
existing error logging to warning logging. I believe this is correct
as these error log messages are really programming errors rather than
errors which need to be show to the user.
Also chose to make entire with_inner unsafe, there is little to be
gained from hiding the unsafe as this is only used from unsafe
functions in the first place.
The user-visible change here is that it allows the FFI API to save
keys in the database for a context. This is primarily intended for
testing purposes as it allows you to get a key without having to
generate it.
Internally the most important change is to start using the
SignedPublicKey and SignedPrivateKey types from rpgp instead of
wrapping them into a single Key object. This allows APIs to be
specific about which they want instead of having to do runtime checks
like .is_public() or so. This means some of the functionality of the
Key impl now needs to be a trait.
A thid API change is to introduce the KeyPair struct, which binds
together the email address, public and private key for a keypair.
All these changes result in a bunch of cleanups, though more more
should be done to completely replace the Key type with the
SignedPublicKye/SignedPrivateKey + traits. But this change is large
enough already.
Testing-wise this adds two new keys which can be loaded from disk and
and avoids a few more key-generating tests. The encrypt/decrypt tests
are moved from the stress tests into the pgp tests and split up.
This should prevent accidental sending of unencrypted messages when
parent_is_encrypted returns an error. For example, if the database is
busy due to other thread activity, parent_is_encrypted should not return
false. Instead, message sending job should retry later.
error!() will show the error directly to the user,
this is not useful in this case,
and also the message with the function-name is not for the end-user.
instead, the ui shall (and already does)
show some explaining text if dc_has_import() returns false;
the error!() is disturbing here as this results in two hints shown to the user.
this changes the behavior when scanning a setup-contact qr-code.
instead of waiting, until the whole protocol is finished,
which may take something between 10 seconds and several minutes,
the dc_join_secure_join() returns instantly;
the uis typically show the created chat then.
the returned chat-id is the same than if we wait for the protocol to finish,
it is the opportunistic one-to-one chat-id, so no changes there.
all this works even when both devices are offline.
after dc_join_secure_join() returns, however,
the usual setup-contact continues. ux-wise, once the protocol finishes,
a system-info-messages is added to the chat (also unchanged),
this is directly visible, in the chat as well as in the chatlist.
while the prococol runs, the user can alredy send message to the chat,
or do other things in the app.
if the user scans a new qr-code while an existing protocol is not finished yet,
the old join will be aborted (however, of course, created chats are kept).
we could also allow multiple joins at the same time,
however, this would be much more effort that this little change
and i am not sure if it is worth the effort.
finally, if a verified-group shall be joined,
this is not possible instantly, this is not affected by this pr.
same for unverified-groups, however, this could maybe be improved,
but also here, not sure if it is worth the effort
(i think most scans are setup-contact scans)
oauth2 crashes in beta23
because we did not let Serde remove the quotes from a parsed JSON email-address.
for autoconfig, we try to get a well-known URL
containing the domain of the used address - with the quote that is
https://autoconfig.gmail.com"/mail/config-v1.1.xml?...
unfortunately, instead of just returning an error,
(the url does not exist anyway)
reqwest crashes on the attempt to get this URL.
the Serde-thing is not obvious to me:
while serde_json::Value.as_str() removes the quotes,
serde_json::Value.to_string() does not.
.truncate() is not safe because it panics if string length does not
lie on character boundary. We convert the string to characters and take
first n character instead.
before a message is actually deleted,
we verify that server_folder+server_uid matches the Message-ID,
so that we do not delete the wrong message by accident.
the reason for that is that, to my knowledge, server_folder+server_uid
are not necessarily stable but Message-ID is.
anyway, this check was broken and set the server_uid to 0 in all cases -
where it should be set to 0 _only_ on mismatches.
As a result of job, other jobs can be added or deleted. To avoid
processing deleted jobs and to process other jobs in the correct order,
we should reload them after performing each job.
However, we don't use "LIMIT 1" in SQL queries, because we want to be
able to skip over invalid jobs, and checking if job is invalid can only
be done in Rust.
This may or may not send a combined MDN out. We don't test for it,
but the test ensures that *if combined MDNs are sent in this case*,
then we receive them correctly.
This return value was very complicated to understand. Some failure
returns were returned as Err and some as Ok with no consistency, but
resulting in the same behaviour.
This refactor makes the handle_securejoin_handshake the sole place
responsible for maintaining the state of the secure join
process (context.bob) and also in charge of terminating the ongoing
process. This is none of receive_imf's business.
The remaining returns are now cleanly classified in application-errors
and protocol errors:
Applications errors result in an Err and mean there is a bug or
something else serious went wrong, like database access suddenly
failed or so. In this case receive_imf is still responsible for
clearing the state and resetting ongoing-process. It may be possible
this should still be moved back to securejoin.rs so that recieve_imf
doesn't need to know anything about this either.
Protocol errors are not failures for receive_imf, it just means the
received message didn't follow the protocol. Receive_imf in this case
is told to ignore the message: that is hide it but not delete it.
Other Ok returns also only say what needs to happen to the message:
- It's fully processed and needs no further processing, instead should
be removed
- It should still be processed as a normal received message.
This changes some behaviour: if the chat creation/lookup for the
contact fails this is treated as an application error. Previously
this was silently ignored and send_msg() would be called with a 0
chat_id without checking the response. This resulted in the protocol
quitely being blocked.
This all shhould now make it easier to resultify more of the functions
called by this function, instead of having to deal with very
complicated application logic hidden in the return code.
this commit sets the subject of groups to the name of the group.
if the message is a reply in the group, also the prefix `Re:` is added.
the Chat: prefix is removed for groups,
also the Fwd: prefix, as it this would be expected to be set to the
group name of the forwarded message, which is not really compatible with the
rule "group-name = subject" (forwarded messages are detected by the body,
not by the subject)
rusqlite already checks that SQLite is safe to use in multi-threaded mode when it opens the database and returns rusqlite::Error::SqliteSingleThreadedMode otherwise.
Also create a common parent_query() function to make sure we use the
same message as a parent when determining if it is encrypted and when
actually replying to it.
The struct really represents a parsed MIME message and is not used as
a parser itself. After the from_bytes() call (which should arguably
use the FromStr trait instead) the struct is fully populated.
This is a new feature in Rust 1.40, it means users outside the crate
will not be able to create these structs, allowing us to add fields
without breaking the public API.
Normally it happens in addr_cmp function, but update_gossip_peerstates
forms a HashSet of addresses, so they should be lowercased beforehand.
Also adjust the mail_with_cc.txt to test for non-lowercase addresses.
According to Rust documentation, usize is "The pointer-sized unsigned
integer type".
This change removes unnecessary libc dependency and makes top_evil_rs
script happier.
Since stock string callback has been deprecated, all event callbacks
return 0.
For compatibility, C declarations are not changed and FFI users are
expected to return 0 from their callbacks.
in the very beginning, we allowed using the cmdline partly from the ui
via a special function; this made info!() handy.
however, this is long gone and no longer needed.
It was used when timeout was set on the whole smtp.send() operation.
Now only the operations inside smtp.send() can timeout, and such timeout
errors result in SendError, so SendTimeout is unused.
-Moved wheels installation below source installation and added comment that they are outdated and not working atm
-Updated source compilation/installation to a more step by step process as current one was outdated and not working due to Rust now needing nightly build else failing to compile (subtle v2.2.2 do not work without nightly)
This is the first experiment towards using structs to define the API.
It adds it as a new method on the existing Chat struct.
The types in this struct could improve as well as many other things.
But this is a straight forward translation of the existing json! macro
into a struct.
15 minute timeout was used because it applied to the whole send()
operation. Now timeout applies to each I/O operation, such as read()
or write(), so it can be made much shorter. In particular, this timeout
applies to read() call right after connection to plain or STARTTLS server,
in which case it is not reasonable to wait 15 minutes to receive one
line of data.
The timeout is triggered only if no progress is made within 30 seconds. It
likely indicates a network failure anyway.
* Remove Simplify structure.
* Match for lines starting with 5 markers, not consisting of exactly 5 markers.
This is a regression from C to Rust conversion, see
2bb1c280d5/src/dc_simplify.c (L154)
* Add tests.
we change the name on the wire as the old Chat-Group-Image header
could not be used on random mails, it was the marker for a "Changed" message,
if we would keep this names, things will fail for exising installations
as messages are dropped and a "Group image changed" message is shown instead.
* add-action-rs-clippy-action
just to try this action out
* add --all argument
* rm --all argument again
* add cargo check and fmt
* disable circle ci counterparts(fetch, clippy, fmt)
- recreate the group list more carefully, fixes#985
- resultify a few functions in the dc_receive pipeline
- don't quote displaynames in email-addresses, use utf8, preliminrarily addresses #976
This effectively reverts
https://github.com/deltachat/deltachat-core-rust/pull/964 for chat.rs,
which in that PR was thought to fix something. So maybe something is
still broken? But after improving tests the previous code seems to be
correct.
- Update Python bindings to not always use dc_prepare_msg path when
sending messages with attachements. When using dc_prepare_msg the
blobs need to be created in the blobdir since they will not get
copied and many tests where not doing this.
- Add a test that ensures that calling dc_prepare_msg with a
file **not** in the blobdir fails.
- Add a test that ensures that calling dc_send_msg directly with a
file **not** in the blobdir copies the file to the blobdir. This
test cheats a little by knowing what the filename in the blobdir
will be which is implementation-dependent and thus a bit brittle.
But for now it proves correct behaviour so let's go with this.
- Improve the test_forward_increation test to ensure that the
in-creation file only has it's final state before calling
dc_send_msg. This checks the correct file data is sent out and not
the preparing data, this fails with the chat.rs changes in
#964 (reverted here to make this work again). Also fix the test to
actually create the in-creation file in the blobdir.
- Fix test_send_file_twice_unicode_filename_mangling to not use
in-creation. It was not creating it's files in the blobdir and that
is an error when using in-creation and it didn't seem it was trying
to test something about the in-creation logic (which is tested in
test_increation.py already).
- Fix Message._msgtate code which presumably was not used before?
- Rename `BlobObject::create_from_path` to
`BlobObject::new_from_path`. All the `BlobObject::create*` calls
now always create new files which is much more consistent. APIs
should do what is obious.
Rustls does not offer a documented way to accept valid certificates with
invalid hostnames. Implementation of certificate verification in Rustls
does not have a public API and reimplementing it is error-prone.
Previously, if any of a chat's peers set prefer_encrypt to false
(i.e. "e2ee_enabled=0" in the config, a misnomer btw) then a
previously encrypted chat would drop to cleartext easily.
eg. android relies on that and fallsback to wrong aspect ratios otherwise.
also, this makes job.rs a little more readable
as some things are moved to message.rs :)
This deletes a lot of code and complexity. Though comes at some cost:
- The type no longer fits in a register and will always be on the
stack.
- Constructing the errors is more verbose, no more auto Into casting.
this reverts the logical changes done in #811
but keeps the improvements done later eg. in #911.
the reason for the revert is that it is too hard to
find a started draft in a larget chatlist.
also the shown date would not be just descending.
Reset is an internal value that received messages should not be able to set.
Also return an error on any value other than "mutual" and "nopreference", errors are converted to NoPreference by the caller.
The ongoing process of dc_join_securejoin() was stopped before the
corresponding chat was created. This resulted in a race-condition
between the sqlite threads executing the creation and query
statements, thus sometimes the query would not find the group and
mysteriously fail.
Tripple-programming with hpk & r10s.
Be more defensive: if somehow we got corrupt jobs in the database skip
over them rather than fail to do anything.
This only modifies the query_map() call, the rest is only split off
into it's own function to make it testable. Smaller functions are
good anyway.
When a new Selfavatar is set the file needs to be copied into the
blobdir and as a proper blob it should be stored in the database as
$BLOBDIR/$filename.
This should fix#868.
First, try to use subkeys, because they are usually
short-term encryption keys. If none of the subkeys
are encryption keys, try to use the primary key.
rPGP is updated to the master branch because the
latest release does not have .is_encryption_key() yet.
Double extensions are sometimes used to identify files correctly,
e.g. .tar.gz or .html.zip. Breaking those extensions is not very
nice.
This fixes#865.
if two messages have the same time,
this results i all kinds of sorting failures,
esp. in non-delta-muas and on forwarding messages.
so, no two messages sent out from delta have the same timestamp.
normally, this is no problem, but when things are sent too fast,
eg. on forwarding, we lend us some time from the future.
however, all this did not work
because we forgot to write back the modified time,
this is fixed by this commit, as well as some cleanup.
resorting the chatlist on changing drafts has some ux issues.
eg. when the chatlist is visible together with the input field,
if may come to flickering resorting during input
or to a resorting just when the user leave the chat
as this might trigger set_draft().
but also on mobiles, the resorting is visible and a bit unexpected.
also it is unclear what happens when a chat with a draft is entered
and left without modifications.
the solution proposed here is to ignore draft on sorting
while still showing them in the chatlist
if they're newer as the last message.
a possible disadvantage is
that the date for the chat with a draft does not follow the ordering
(the ordering is by the last message),
however, the date is not shown as a "primary sort" criterion or so,
so it might be that this is completely okay.
also, of course, it affects only draft :)
the point of this pr is to get an overview
how and where DC_CONTACT_ID_DEVICE is used,
to prepare introducing a device-"chat".
i did not change the sql statements for now
as this would require some more refactoring
and has the potential to introduce bugs.
Before we created an empty file and asked the OS to copy the file.
The OS is very good at this so this is a good idea generally. However
it seems that in some cases, possibly an Android Dowload folder, we
might be able to create a file but not overwrite it. Thus refactor
this a bit so we are copying the file ourselves.
There are no new tests here since the behaviour remains identical.
The good news is that the existing tests were good enough to catch
some bugs already.
the error led to unusable contact requests,
at least on android and ios (probably also desktop)
because msg_id=dc_chatlist_get_msg_id() always returns 0
and create_chat_by_msg_id(msg_id)
or dc_marknoticed_contact(<get sender from msg_id>)
failed therefore.
This more strongly types the ubiquitous message id type by no longer
making it an integer. It keeps the actual ID opaque. Only for the
generic job API the number keeps being used. Some locations also need
to create it from an integer and call MsgId::new().
Turns out that anyone that uses these either justs wants a file or
wants a blob. Consolidate those patterns into one place and simplify
all the callers.
This creates a specific type for blobs, with well defined conversions
at the borders. It also introduces a strong type for the Param::File
value since that param is often used used by the public API to set
filenames using absolute paths, but then core changes the param to a
blob before it gets to the database.
This eliminates a few more functions with very mallable C-like
arguments behaviour which combine a number of operations in one.
Because blob filenames are stored so often in arbitrary strings this
does add more code when receiving those, until the storage is fixed.
File name sanitisation is now deletated to the sanitize-filename crate
which should do a slightly better job at this.
Previously, "flags" argument of Sql.open was of type libc::c_int, but
only one bit was used: whether to open database read-only. This commit
makes it explicit by changing type to bool and renaming argument.
if the past we had lots of crashes because of unexpected unwrap failures,
mostly related to string.
this commit avoids them eg. for string-conversions that may panic
eg. when encountering a null-byte or by logical programming errors
where an object is assumed to be set but is not under unexpected circumstances.
- increase the number of simultan database-connections handled by the r2d2 pool.
currently we have already at least 5 threads threads,
but also the ui may call from any thread.
- the busy-timeout for all connections is set to 10 seconds.
this means, if a connection-A wants to write,
but connection B-is already writing,
connection-A waits multiple times a few ms and tries over.
this is repeated until the 10 seconds are accumulated.
contact-requests of non-blocked senders are shown in the chatlist,
so that the user gets aware of them without opening the contact-request-chat.
however, for that, the contact-request virtual-chat-id
must be added to the beginning of the list,
otherwise it won't get noticed by the user.
* add sticker type
this pr adds the message type 'sticker'.
stickers are handled as normal images
but tagged with the header `Chat-Content: sticker`
it's up to the ui to render these stickers appropriate.
* cargo fmt
there are 3 key blobs in the database, gossip_key, public_key and verified_key.
the verification_key should not change if, for any reasons,
the public_key or the gossip_key changes.
Since Rust strings operations are assumed to never fail, this commit
removes a lot of checking, whether appending to mmapstring fails. Now it
is Rust runtime burden.
core-c has truncated messages in the contact requests.
this is questionable in general, as
- all messages, including contact requests,
are already truncated, unquoted, simplified etc.
- the ui should be capable of showing the full text anyway
(when the contact request is accepted, the whole messase is shown)
- also, all current ui show the contact requests by name only in the
chatlist; the user often does not even come to the contact request view.
- if the ui wants to show the contact request is a special way,
it is probably better to leave this truncation up to the ui
There is very little API guarantees about this, but clients seem to
expect *something* to work on a closed context. So split this up into
static info an more dynamic context-related info, but let's not
provide any guarantees about what keys are available when.
Fixes#599
Function "quote_word" accepts display charset argument, but on only call
site it is string constant. Change function to always use "utf-8"
display charset, and remove useless argument.
Note: I strongly suggest reviewing this commit in side-by-side mode.
Note: This commit fails CI due incorrect formatting. It is done
deliberately to simplify review process.
Previously, "set_draft_raw" function was used with "msg = None" only for
side-effect of removing current draft message in chat.
After draft removal functionality was factored into separate
"maybe_delete_draft" function, it is used directly in only call site,
which used to invoke "set_draft_raw" with optional message.
This function is private, and this refactoring does not change FFI
interface.
Note: This commit fails CI due incorrect formatting. It is done
deliberately to simplify review process.
* src/dc_mimefactory.rs(new): add factory method to have verbose
initialization of all (more than 10) MimeFactory fields only in one place.
* src/dc_mimefactory.rc(dc_mimefactory_load_msg, dc_mimefactory_load_mdn):
simplify code (and reduce linecount) using Mimefactory::new
* src/dc_mimefactory.rs(MimeFactory): change type of `loaded` field
* src/dc_mimefactory.rs(Loaded): new enum, describing possible
values of `loaded` field of `MimeFactory` structure
* src/dc_mimefactory.rs(dc_mimefactory_loaded_t): remove unused type alias
* src/job.rs(add_smtp_job): adjust call site by removing multiple casts
* src/dc_mimefactory.rs(dc_mimefactory_render): ditto
Replace named constants with enum to improve type-safety and make
exhausiveness checks possible.
Note, that since this enum never pass FFI border, its numeric values
does not need to be specified explicitly and can be left on compiler's
discretion.
Previously, this function accepted both key id (integer) and is_default
(boolean). If `is_default` flag was set, value `id` parameter wasn't
used. This commit compresses two argument into single `id: Option<i64>`.
To download and install the official compiler for the Rust programming language, and the Cargo package manager, run the command in your user environment:
```
curl https://sh.rustup.rs -sSf | sh
$ curl https://sh.rustup.rs -sSf | sh
```
## Using the CLI client
Compile and run Delta Chat Core using `cargo`:
Compile and run Delta Chat Core command line utility, using `cargo`:
```
cargo run --example repl -- /path/to/db
$ RUST_LOG=repl=info cargo run --example repl --features repl -- ~/deltachat-db
```
where ~/deltachat-db is the database file. Delta Chat will create it if it does not exist.
Configure your account (if not already configured):
@@ -89,6 +90,16 @@ $ cargo test --all
$ cargo build -p deltachat_ffi --release
```
## Debugging environment variables
-`DCC_IMAP_DEBUG`: if set IMAP protocol commands and responses will be
printed
-`DCC_MIME_DEBUG`: if set outgoing and incoming message will be printed
-`RUST_LOG=repl=info,async_imap=trace,async_smtp=trace`: enable IMAP and
SMTP tracing in addition to info messages.
### Expensive tests
Some tests are expensive and marked with `#[ignore]`, to run these
@@ -101,9 +112,24 @@ $ cargo test -- --ignored
-`vendored`: When using Openssl for TLS, this bundles a vendored version.
-`nightly`: Enable nightly only performance and security related features.
-`ringbuf`: Enable the use of [`slice_deque`](https://github.com/gnzlbg/slice_deque) in pgp.
d="M 13.5,7.5 V 39 h 0.08654 L 22.533801,29.370239 31.482419,39 h 0.01758 V 7.5 Z m 9.004056,4.108698 1.879508,4.876388 5.039514,0.359779 -3.879358,3.363728 1.227764,5.095749 -4.276893,-2.796643 -4.280949,2.788618 1.237229,-5.093073 -3.873949,-3.371754 5.040866,-0.350417 z"
For brevity let's abbreviate "group membership modification" as **GMM**.
Delta chat has explicit GMM messages, typically encrypted to the group members
as seen by the device that sends the GMM. The `Spec <https://github.com/deltachat/deltachat-core-rust/blob/master/spec.md#add-and-remove-members>`_ details the Mime headers and format.
If we detect membership inconsistencies we can resend relevant GMM messages
to the respective chat. The receiving devices can process those GMM messages
as if it would be an incoming message. If for example they have already seen
the Message-ID of the GMM message, they will ignore the message. It's
probably useful to record GMM message in their original MIME-format and
not invent a new recording format. Few notes on three aspects:
-**group-membership-tracking**: All valid GMM messages are persisted in
their full raw (signed/encrypted?) MIME-format in the DB. Note that GMM messages
already are in the msgs table, and there is a mime_header column which we could
extend to contain the raw Mime GMM message.
-**consistency_checking**: If an incoming GMM has a member list which is
not consistent with our own view, broadcast a "Group-Member-Correction"
message to all members containing a multipart list of GMMs.
-**correcting_memberships**: Upon receiving a Group-Member-Correction
message we pass the contained GMMs to the "incoming mail pipeline"
(without **consistency_checking** them, to avoid recursion issues)
Alice/Carol and Bob/Doris getting on the same page
# Seeds for failure cases proptest has generated in the past. It is
# automatically read and these particular cases re-run before any
# novel cases are generated.
#
# It is recommended to check this file in to source control so that
# everyone who runs the test benefits from these saved cases.
cc 03cab93c6d1f3a8245f63cf84dacb307944294fe6333c1e38f078a6600659c7a # shrinks to data = "a\t0aA\ta\t0 \ta\t0 \ta a\t\ta A\tAA0a0a 0\t a\t aA \t a\t A0\t AAa\taA0\taAAaA\t0\taa0a\ta Aa Aaaa A0A\t a aA 0\t A\t0\t0\t\t\t\t\t\tA \t\t a\tA Aa aAA0A0AA0aaA A\t\t aa0\ta\t \tAa\taA\t00 AA A a\tA a aAAa \t 00\t0 \t\t a A 0\t\t\t aAA Aa \taAAAA0a A0A\t\t1\\E$\t$R\tc\t^=\t\"\tQ<Uk\t\t>A\t\t&\t}v&\tM^\'|\tW5?dn\t\t+\t\tP\te`\t\t>:Brlq--?\t$#Q\tK=zJ\tb\"9*.\"\t`\tF&T*\tBs,\tg\'*\\\t:\t?l$\t\t|A\"HR:Hk\t\\KkV\t\t{=&!^e%:|_*0wV\t[$`\t:\t$%f\t\t[!\"Y. \tP\t\th\'?\'/?%:++NfQo#@\"+?\t(\\??{\t\'\'$Dzj\t0.?{s4?&?Y=/yj]Z=\t4n\t?Ja\"\t{I\t$\t;I}_8V\t&\t?N\'\tI2/\t9.\tFT%9%`\'\tz\to7Y\t|AXP&@G12g\t\'w\t\t%??\t\"h$?F\"\"6%q\\\\{\tT\t\"]$87$}h\'\t<\t$\tc%U:mT2:<v\t#Rl!;U\t\t\"^D\tRZ(BZ{n\t%[$fgH`\t{B}:*\t\t%%*F`%W\t//B}PQ\t\tsu2\tLz<1*!p-X\tnKv&&0\thm4<n\\.\\/.w\'\t<)E1g3#ood\t`?\t\\({N?,8_?h\ty\t0%\t*$A\t\t*w-ViQUj\tTiH\t%\t%&0p\'\'\tA%r**Fo\'Z\\\tNI*ely4=I?}$.o`\t$\ts\'<\t\",:~=Nv/0O%=+<LS\t%P\'\t$r%d.\t{G{/L:\t&I&8-`cy*\"{\t/%fP9.P<\t\t\'/`\t\t`\t\t`!t:\t::\t\tW\'X^\t@.uL&a\tN\t\t\t.\t?0*\tvUK>UJ\\\tQbj%w]P=Js\t\"R\t&s^?</=%\t\'VI:\" kT`{b*<\t\tF&?Y\t\t:/\t!C\'e0`\t\t\tx-\t*\\N\\wwu.I4\tb/y\t\"P:P%\"\t\tQ][\\\t^x\t\t):\t\t&s\t$1-\t\t\tXt`0\t;\t/UeCP*\"G\t\t\':\tk@91Hf&#\t(Uzz&`p2;\t{]\t\"I_%=\\%,XD\"\'06QfCd<;$:q^\t8:\"*\"H\t\to\t&xK/\t\ty0/<}<j<|*`~::b\t=S.=}=Ki\t<Y.\'{\tf\t{Ub*H%(_4*?*\tn2\t/\'\t\t\t/,4]\tt\t<y\t\t\tWi\t\tT&\"\t\t\t\t\t=/1Wu?\t\'A\"W-P\t$?e\\\t`\t6`vD\t8`\t\tccD J\tY&jI//?_\t\\j\t_\tsiq$\t?9\tQ\t.^%&..?%Jm??A%+<\tN&*\t.g\tS$W\"\"\tMpq\t\t:&\\\thTz&<iz%/%T&\'F\t\\]&\t\t}\t\t\tXr=dtp`:+<,\t%60Y*<(&K*kJ\todO\t=<V?&\tMU/\"\t= Y!)<\tV\t9\t)\t&v8q{\t\t&pT\t3\ttB,qcq\'i$;gv%j_%M_{[\"&;\t\t\t.B;6Y\\%\t\"\tY/a\t\\`:?\t<\t?]\taNwD;\\\t%l*74%5T?QS :~yR/E*R\t\t=u\t\\\t\t.Q<;\\\t_S/{&F$^\tw_>\'h=|%\t\t:W!\\<U2\'$\tb&`\t=|=L9\t\t\t\\WZ:! }{\t ;\t;\t\t 0.*\t.%\"\'r%{<Mh_t[[}\t-\tJo\"b/AC*-=`=T\tz$2\tC\t\t/k{47\"\t\t,q%\tZ\tT3\t\tf>%\t\'?%@^tx\t7\"1Bk{b{n\t\"Pj3\tHc\t\tt\tY<\t#?\tSh\\yk/N\\\t8 7=R4*9Cw&m\t\\-\'f\t|\'#t(Etu.Hdu(E\t%&v:\'aqW~A5\t\t w.s{J%lX<\"\t\'*%m<&:/B<&\':U}$&`.{)\t\t6S\t:/$*kQ-Z\t^\'t${/tH*\'v\t3\t=\t\tDyp:B\t`I_R&4SO\t\t&-j=*.\t87&\'e*( \t\t\t\'<$\\DJ<$p?{}\'&\tv\t\\Xk<Y:Y`!$K{\tF&\tzd\t\t*i$\tj\'\t<)R*\t%?\t!.\t=\"@#~:=*\t\tXO=_T,1\"\'.%%\"`{\\:\t\"\tfkeOb/\'$I~\ta\t|&\t[\\KK\"1&Z\t<k\t\t)%\'-~\"2n\tj\tW?*<@w{g%d\ta\\\'\'I\t;:ySR%ke:4\tc\t$=\t&9P]x4\tJ=\t6C6%a\t`0\tF\tm-\tTr\t}\t\tQum\t&@\typ|w2&\t\t3`i&t\t\tT5\"\t.&b&e*/==1.\'*\\[U*\tqPL%?$-0/}~|q`\t\t}\t$\tq==o+T$\'!H\t\ti&um\"?\"%%\t/\'p\tg>?{0{J{\t\t/\t\t{zKZ&>=\t[\"1h<H%z/8,/]s\tv{7\t\t:j*H,M//\t\t\td\'.)\t"
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
