fix python lint errors
receive pre-mesages, start with changes to imap loop.
refactor: move download code from `scheduler.rs` to `download.rs`, also
move `get_msg_id_by_rfc724_mid` to `MsgId::get_by_rfc724_mid`
`MAX_FETCH_MSG_SIZE` is no longer unused
Parse if it is a pre-message or full-message
start with receiving logic
get rid of `MsgId::get_by_rfc724_mid` because it was a duplicate of
`message::rfc724_mid_exists`
docs: add hint to `MimeMessage::from_bytes` stating that it has
side-effects.
receiving full message
send and receive `attachment_size` and set viewtype to text in
pre_message
metadata as struct in pre-message in header. And fill params that we can
already fill from the metadata. Also add a new api to check what
viewtype the message will have once downloaded.
api: jsonrpc: add `full_message_view_type` to `Message` and
`MessageInfo`
make PreMsgMetadata.to_header_value not consume self/PreMsgMetadata
add api to merge params
on download full message: merge new params into old params and remove
full-message metadata params
move tests to `src/tests/pre_messages.rs`
dynamically allocate test attachment bytes
fix detection of pre-messages. (it looked for the ChatFullMessageId
header in the unencrypted headers before)
fix setting dl state to avaiable on pre-messages
fix: save pre message with rfc724_mid of full message als disable
replacement for full messages
add some receiving tests and update test todo for premessage metadata
test: process full message before pre-message
test receive normal message
some serialization tests for PreMsgMetadata
remove outdated todo comment
test that pre-message contains message text
PreMsgMetadata: test_build_from_file_msg and test_build_from_file_msg
test: test_receive_pre_message_image
Test receiving the full message after receiving an edit after receiving
the pre-message
test_reaction_on_pre_message
test_full_download_after_trashed
test_webxdc_update_for_not_downloaded_instance
simplify fake webxdc generation in
test_webxdc_update_for_not_downloaded_instance
test_markseen_pre_msg
test_pre_msg_can_start_chat and test_full_msg_can_start_chat
test_download_later_keeps_message_order
test_chatlist_event_on_full_msg_download
fix download not working
log splitting into pre-message
add pre-message info to text when loading from db. this can be disabled
with config key `hide_pre_message_metadata_text` if ui wants to display
it in a prettier way.
update `download_limit` documentation
more logging: log size of pre and post messages
rename full message to Post-Message
split up the pre-message tests into multiple files
dedup test code by extracting code to create test messages into util
methods
remove post_message_view_type from api, now it is only used internally
for tests
remove `hide_pre_message_metadata_text` config option, as there
currently is no way to get the full message viewtype anymore
Update src/download.rs
resolve comment
use `parse_message_id` instead of removing `<>`parenthesis it manually
fix available_post_msgs gets no entries
handle forwarding and add a test for it.
convert comment to log warning event on unexpected download failure
add doc comment to `simple_imap_loop`
more logging
handle saving pre-message to self messages and test.
part of #7367
progress / what's to do:
- [x] send pre-message
- [x] The message's state must be set to MessageState::OutDelivered only
after both messages are sent. If a read receipt is received, the message
can be OutMdnRcvd or OutPending; let's just do whatever is easiest for
now. Take care not to revert from OutMdnReceived to OutDelivered if we
first receive a read receipt and then deliver the full message.
- this is already the case:
- `OutDelivered` is set when a message is sent out and has no remaining
send jobs in the smtp table for this message id - so already works since
full message and pre message have same msgId in that table.
- `OutMdnRcvd` is a "virtual" state
(https://github.com/chatmail/core/issues/7367#issuecomment-3500891040),
so going back to `OutDelivered` can't happen anymore
- [x] delimit `ChatFullMessageId` with `<` and `>` like the other
message ids
- [x] add tests
- [x] test that pre message is sent for attachment larger than X
- test that correct headers are present on both messages
- also test that Autocrypt-gossip and selfavatar should never go into
full-messages
- [x] test that no pre message is sent for attachment smaller than X
- no "is full message" header should be present
- [x] test that pre message is not send for large webxdc update or large
text
- [x] fix test
`receive_imf::receive_imf_tests::test_dont_reverify_by_self_on_outgoing_msg`
---------
Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>
Co-authored-by: Hocuri <hocuri@gmx.de>
We do not try to delete resent messages
anymore. Previously resent messages
were distinguised by having duplicate Message-ID,
but future Date, but now we need to download
the message before we even see the Date.
We now move the message to the destination folder
but do not fetch it.
It may not be a good idea to delete
the duplicate in multi-device setups anyway,
because the device which has a message
may delete the duplicate of a message
the other device missed.
To avoid triggering IMAP busy move loop
described in the comments
we now only move the messages
from INBOX and Spam folders.
Ignoring `receive_imf_inner()` errors, i.e. silently skipping messages on failures, leads to bugs
never fixed. As for temporary I/O errors, ignoring them leads to lost messages, in this case it's
better to bubble up the error and get the IMAP loop stuck. However if there's some logic error, it's
better to show it to the user so that it's more likely reported, and continue receiving messages. To
distinguish these cases, on error, try adding the message as partially downloaded with the error set
to `msgs.error`, this way the user also can retry downloading the message to finally see it if the
problem is fixed.
this PR adds a info message "messages are end-to-end-encrypted" also for
chats created by eg. vcards. by the removal of lock icons, this is a
good place to hint for that in addition; this is also what eg. whatsapp
and others are doing
the wording itself is tweaked at
https://github.com/deltachat/deltachat-android/pull/3817 (and there is
also the rough idea to make the message a little more outstanding, by
some more dedicated colors)
~~did not test in practise, if this leads to double "e2ee info messages"
on secure join, tests look good, however.~~ EDIT: did lots of practise
tests meanwhile :)
most of the changes in this PR are about test ...
ftr, in another PR, after 2.0 reeases, there could probably quite some
code cleanup wrt set-protection, protection-disabled etc.
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
Connection sometimes fails while processing FETCH
responses. In this case `fetch_new_messages` exits early
and does not advance next expected UID even if
some messages were processed.
This results in prefetching the same messages
after reconnection and log messages
similar to
"Not moving the message ab05c85a-e191-4fd2-a951-9972bc7e167f@localhost that we have seen before.".
With this change we advance next expected UID
even if `fetch_new_messages` returns a network error.
fetch_existing option is not enabled in existing clients
and does not work with encrypted messages
without importing the key into a newely created account.
When receiving messages, blobs will be deduplicated with the new
function `create_and_deduplicate_from_bytes()`. For sending files, this
adds a new function `set_file_and_deduplicate()` instead of
deduplicating by default.
This is for
https://github.com/deltachat/deltachat-core-rust/issues/6265; read the
issue description there for more details.
TODO:
- [x] Set files as read-only
- [x] Don't do a write when the file is already identical
- [x] The first 32 chars or so of the 64-character hash are enough. I
calculated that if 10b people (i.e. all of humanity) use DC, and each of
them has 200k distinct blob files (I have 4k in my day-to-day account),
and we used 20 chars, then the expected value for the number of name
collisions would be ~0.0002 (and the probability that there is a least
one name collision is lower than that) [^1]. I added 12 more characters
to be on the super safe side, but this wouldn't be necessary and I could
also make it 20 instead of 32.
- Not 100% sure whether that's necessary at all - it would mainly be
necessary if we might hit a length limit on some file systems (the
blobdir is usually sth like
`accounts/2ff9fc096d2f46b6832b24a1ed99c0d6/dc.db-blobs` (53 chars), plus
64 chars for the filename would be 117).
- [x] "touch" the files to prevent them from being deleted
- [x] TODOs in the code
For later PRs:
- Replace `BlobObject::create(…)` with
`BlobObject::create_and_deduplicate(…)` in order to deduplicate
everytime core creates a file
- Modify JsonRPC to deduplicate blob files
- Possibly rename BlobObject.name to BlobObject.file in order to prevent
confusion (because `name` usually means "user-visible-name", not "name
of the file on disk").
[^1]: Calculated with both https://printfn.github.io/fend/ and
https://www.geogebra.org/calculator, both of which came to the same
result
([1](https://github.com/user-attachments/assets/bbb62550-3781-48b5-88b1-ba0e29c28c0d),
[2](https://github.com/user-attachments/assets/82171212-b797-4117-a39f-0e132eac7252))
---------
Co-authored-by: l <link2xt@testrun.org>
Add a `create` param to `select_with_uidvalidity()` instead of always trying to create the folder
and return `Ok(false)` from it if the folder doesn't exist and shouldn't be created, and handle this
in `store_seen_flags_on_imap()` by just removing "jobs" from the `imap_markseen` table. Also don't
create the folder in other code paths where it's not necessary.
this PR removes most usages of the `descr` parameter.
- to avoid noise in different branches etc. (as annoying on similar, at
a first glance simple changes), i left the external API stable
- also, the effort to do a database migration seems to be over the top,
so the column is left and set to empty strings on future updates - maybe
we can recycle the column at some point ;)
closes#6245
Without this fix IMAP loop may get stuck
trying to download non-existing message over and over
like this:
```
src/imap.rs:372: Logging into IMAP server with LOGIN.
src/imap.rs:388: Successfully logged into IMAP server
src/scheduler.rs:361: Failed to download message Msg#3467: Message Msg#3467 does not exist.
src/scheduler.rs:418: Failed fetch_idle: Failed to download messages: Message Msg#3467 does not exist
```
The whole download operation fails
due to attempt to set the state of non-existing message
to "failed". Now download of the message
will "succeed" if the message does not exist
and we don't try to set its state.
API now pretends that trashed messages don't exist.
This way callers don't have to check if loaded message
belongs to trash chat.
If message may be trashed by the time it is attempted to be loaded,
callers should use Message::load_from_db_optional.
Most changes are around receive_status_update() function
because previously it relied on loading trashed status update
messages immediately after adding them to the database.
Connection establishment now happens only in one place in each IMAP loop.
Now all connection establishment happens in one place
and is limited by the ratelimit.
Backoff was removed from fake_idle
as it does not establish connections anymore.
If connection fails, fake_idle will return an error.
We then drop the connection and get back to the beginning of IMAP
loop.
Backoff may be still nice to have to delay retries
in case of constant connection failures
so we don't immediately hit ratelimit if the network is unusable
and returns immediate error on each connection attempt
(e.g. ICMP network unreachable error),
but adding backoff for connection failures is out of scope for this change.
If a Delta Chat message has the Message-ID already existing in the db, but a greater "Date", it's a
resent message that can be deleted. Messages having the same "Date" mustn't be deleted because they
can be already seen messages moved back to INBOX. Also don't delete messages having lesser "Date" to
avoid deleting both messages in a multi-device setting.
Previously the message was removed from `download` table,
but message bubble was stuck in InProgress state.
Now download state is updated by the caller,
so it cannot be accidentally skipped.
Also add a test on downloading a message later. Although it doesn't reproduce #4700 for some reason,
it fails w/o the fix because before a message state was changing to `InSeen` after a full download
which doesn't look correct. The result of a full message download should be such as if it was fully
downloaded initially.
Such a message may be assigned to a wrong chat (e.g. undecipherable group msgs often get assigned to
the 1:1 chat with the sender). Add `DownloadState::Undecipherable` so that messages referencing
undecipherable ones don't go to that wrong chat too. Also do not reply to not fully downloaded
messages. Before `Message.error` was checked for that purpose, but a message can be error for many
reasons.
Message.set_text() and Message.get_text() are modified accordingly
to accept String and return String.
Messages which previously contained None text
are now represented as messages with empty text.
Use Message.set_text("".to_string())
instead of Message.set_text(None).
Moved custom ToSql trait including Send + Sync from lib.rs to sql.rs.
Replaced most params! and paramsv! macro usage with tuples.
Replaced paramsv! and params_iterv! with params_slice!,
because there is no need to construct a vector.
Gmail archives messages marked as `\Deleted` by default if those messages aren't in the Trash. But
if move them to the Trash instead, they will be auto-deleted in 30 days.
get_chat_msgs() function is split into new get_chat_msgs() without flags
and get_chat_msgs_ex() which accepts booleans instead of bitflags.
FFI call dc_get_chat_msgs() is still using bitflags for compatibility.
JSON-RPC calls get_message_ids() and get_message_list_items()
accept booleans instead of bitflags now.
