fix#3007
My approach is:
We don't download any messages from the spam folder anymore, and only download them if they were moved out. This means that is-it-spam logic only resides in spam_target_folder(). This has some implications, see the comments.
- Replace .ok_or_else() and .map_err() with anyhow::Context where possible.
- Use .context() to check Option for None when it's an error
- Resultify Chatlist.get_chat_id()
- Add useful .context() to some errors
- IMAP error handling cleanup
This is an irrecoverable error, dc_receive_imf must not fail on it
as it prevents last seen UID from advancing, so the same message
is prefetched on each iteration of IMAP loop.
I considered removing it from the context by default, but the
migration test really wants to have the tracker initialised from the
very first event and not after the context is initialised. It is
easier for now to leave it hardcoded instead of adding an API to
explicitly require enabling it via the builder.
`imap` table maps Message-IDs to UIDs on the server. `dc_receive_imf`
no longer gets the UID of the message as an argument and does not
insert the folder and UID of the message into the `msgs`
table. `server_folder` and `server_uid` columns in `msgs` table are
deprecated.
MoveMsg and DeleteMsgOnImap jobs are removed. Now messages are moved
and deleted only in the `fetch_move_delete` procedure that consults
the `target` column of the `imap` table to determine where the message
should go.
Where the message should go is determined after prefetching by the
`imap::target_folder()` procedure. Messages are only downloaded once
they reach their target folder to avoid race conditions in multidevice
setting, such as:
1. One device trying to FETCH the message while the other tries to
MOVE it.
2. One device marking the message as \Seen in the Inbox while the
other has already copied unseen message to the Movebox and is going to
delete the \Seen message in the Inbox.
3. Device downloads the message from the Inbox while there are newer
messages in the Movebox placed there by the other device, thus
processing the messages out of order.
- Check that member modifying the group is in the group themselves.
It is not allowed to readd yourself to the group or remove someone without being in the group themselves.
- Unify code for group member addition and removal.
Removing a member now recreates the group member list from the To: field.
Removed member from the Chat-Group-Member-Removed was in the To: field in previous Delta Chat versions,
so it is excluded explicity. New versions of Delta Chat put removed member in Bcc: instead.
- Apply avatar changes after updating the group member list.
This allows to check that the contact modifying the avatar is actually a group member.
When there is an outgoing message in a chat, mark all older messages in this chat as seen.
Android already has a similar behavior, however, this led to the issue https://github.com/deltachat/deltachat-android/issues/2163 and should be changed back.
--
From the issue description at https://github.com/deltachat/deltachat-android/issues/2163, I implemented these fixes:
> Core should take care that if the last message in a chat is not fresh|noticed, no messages in the chat can be fresh. [...] Do this [...] in a function that's called at the end of fetch_new_messages(). Then dc_receive_imf() wouldn't get slower by this and we could re-use the same function for migration.
So, I didn't do this inside `dc_receive_imf()` in order not to make it take even longer. This obviously has the downside of higher complexity.
And I think we should implement this:
> On Androd, show the unread badge when unread!=0 again (see deltachat/deltachat-android@618af02). Then the user can see that there is a chat with an unread message and click it to get rid of it.
because it shouldn't be the UI's job to decide whether an unread badge is shown, but the core's.
* test adding members in a multi-device setup
* fix system messages for multi-device-setup
* enhance test to check multi-device messages for removing-members and group-renames
This function is used to lookup the chat by `References` and
`In-Reply-To` header, so it does not make sense to return trashed
message when there is another non-trashed message in one of these
headers with a real chat ID.
`indexmap` is a large dependency (4K SLoC) containing `unsafe` code.
Contact IDs are now passed around as a Vec<u32> or &[u32].
QUOTA roots are now sorted by name instead of perserving original order.
It was there since the C core, labeled with "/* last_seen is for
future use */" but never actually used. The comment was lost during
the translation from C to Rust.
`apply_group_changes` is executed regardless of whether the group is
created via `create_or_lookup_group` or found via
`lookup_chat_by_reply`. This change removes the need for
`lookup_chat_by_reply` to return `None` when group ID exists in the
database to let `create_or_lookup_group` run. As a side effect of this
Delta Chat replies to ad hoc groups are now correctly assigned to
chats when there are multiple groups with the same group ID, such as
ad hoc groups with the same member lists.
* test one-to-one chats on setup-contact/secure-join
only one chat is created after scanning a QR code:
- on setup-contact, one-to-ones are created on both sided
- on secure-join, the joined group chat is created;
one-to-ones are not created intitally,
but should become visible on receiving messages
* make sure, Alice creates the chat with Bob on setup-contact
not totally sure if that change in #2508 was on-purpose,
however, all yet released versions
did create the one-to-one chat also on the Inviter's (Alice) side,
so, let's stay with that,
i do not see many reasons to change that.
* unblock hidden (Blocked::Yes) one-to-one chats
one-to-one chats may be hidden by secure-join,
in case someone later writes a message to it
(not unlikely), the chat needs to be shown.
before, messages are just not shown,
the corresponding chat did not appear.
the 'Blocked' wording of a 'Chat' must not be mixed with the
'Blocking' of a contact. 'Chat-Blocking' is mostly a visibility thing,
that may change as messages come in.
this change should not affect _really_ blocked contacts -
they are filtered out already before
and their messages are usually not even downloaded.
also, before allow_creation is checked,
that may disallow chat creation for show_emails reasons.
all in all, it just does the same
as if the user has manualy deleted the chat before and it would be created.
* simplify test
Update gossiped_timestamp when someone else sends autocrypt gossip in
the group, so we postpone sending gossip again ourselves.
- Warn about failures to parse Autocrypt-Gossip header
- Move gossip-related methods into ChatId impl
- Fix a "gossi_pp_ed" typo
- Make `state` variable immutable.
- Don't mark contact requests as fresh when fetching existing messages: this is not needed since there is no contact request chat anymore.
- Don't mark messages from blocked chats as noticed, this is not needed since messages go to blocked chat anyway instead of contact request chat.
* refactor: cleanup send_handshake_msg()
- rename to send_alice_handshake_msg() as used by Alice only
- remove dead code from Bob
(Bob's code is at BobState::send_handshake_message() since some time)
- take a contact_id and not a chat_id;
this makes things less confusing when
info-messages are put to the final group chat
* always directly return chat-id from dc_join_securejoin()
* take care not to create a group twice
* adapt documentation
* add info-msg on group invites; add inviter directly after creation
* document existing 'joinqr' command in repl tool
* do not create empty one-to-one chats for group-joins
* refactor: cleanup fingerprint_equals_sender()
- the function takes a contact_id directly now.
before it consumes the first contact of a one-to-one chat -
which may be easily confused with the group-chat in creation.
moreover, the conversion contact_id -> chat_id -> contact_id
is unneeded overhead.
* show info-messages in destination chat for alice
* fingerprint_equals_sender() returns Err on database failure
* tweak documentation
* clarify what an 'unfinished tasks' task is.
* add regression test for create_for_contact_with_blocked()
* rename Blocked::Manually to better fitting Blocked::Yes
* tweak test_secure_join() and make sure, Alice and Bob have only on chat after a group-join
* add basic multi-device-sync functions
* generate json
* add context.parse_sync_items()
* add context.execute_sync_items()
* piggyback sync-commands message, add body for human-readable part
* avoid double json renderings
* mimeparser parses incoming .json sync-files
* do not piggyback sync-files
* execute sync items
* return status of send_sync_msg()
* send sync messages as multipart/report
* add a per-item-timestamp and also allow adding other per-item-fields in the future
* if the self-chat does not exist, create it blocked/hidden
* create tokens closer to real qr-code needs
* respect bcc_self setting, add test for that
* sync qr code tokens after promoting groups
* send sync-messages only if an experimental switch is set
* trigger send_sync_msg() after sending messages and after creating/redraw/revive qr-code
* add DC_STR_* constants to deltachat.h
* adapt to refactored qr module as of #2729
* tweak test
* use SendSyncMsgs config name instead of SendExperimentalSyncMsgs - we can remove or rename the config nevertheless, but have the option to keep it without renaming
* tweak docs
* remove currently unused effective timestamp calculation
* clarify when send_sync_msg() is called
* make sure, sync-messages are encrypted and are sent by SELF
* tweak docs, fix typos
Reduce the number of mutable variables and out parameters in
`add_parts`.
Also don't call `dc_create_smeared_timestamp` if there is no `Date`
header. Timestamps are only supposed to be "created" when the message
is sent, not received, to make sure sent messages are sorted properly
in MUAs that only use the date for sorting. Delta Chat uses database
IDs for sorting in addition to timestamps, so it can sort messages
with equal timestamps properly.
Update `dc_smeared_time` documentation.
Turn `dc_smeared_time` and `dc_create_smeared_timestamp` comments into
documentation comments.
this is needed for targeting "non-blocking group QR joins"
as create_multiuser_record() would also be needed from other places.
this will make rebasing/rewriting and finally reviewing #2508 easier.
before, MsgsChanged(chat_id, new_msg_id) was emitted,
but that does not cover the deleted message.
in theory, we could emit both,
however, that would just be a waste of refresh in uis.
also before, events were used this way,
however, also the documentations are updated to
reflect reality better.
Implement get_previous_message() that only looks for the last
Message-ID in the References: header instead of using
get_parent_message() which falls back to using other Message-IDs from
References and In-Reply-To field.
