There was a bug that file extensions were removed when recoding an
avatar. The problem was that `recode_avatar` used `name` to check for
the extension, but some functions passed an empty string.
There even were two tests from before the decision to keep the
extensions that tested for the faulty behavior.
These were the last places in the `deltachat` crate where files were
stored without deduplication. The CFFI python bindings are the last
thing that's still missing.
Deduplicate:
- In the REPL
- In `store_from_base64()`, which writes avatars received in headers
- In a few tests
- The saved messages, broadcast, device, archive icons
- The autocrypt setup message
1-2 more PRs, and we can get rid of `BlobObject::create`,
`sanitise_name()`, and some others
When receiving messages, blobs will be deduplicated with the new
function `create_and_deduplicate_from_bytes()`. For sending files, this
adds a new function `set_file_and_deduplicate()` instead of
deduplicating by default.
This is for
https://github.com/deltachat/deltachat-core-rust/issues/6265; read the
issue description there for more details.
TODO:
- [x] Set files as read-only
- [x] Don't do a write when the file is already identical
- [x] The first 32 chars or so of the 64-character hash are enough. I
calculated that if 10b people (i.e. all of humanity) use DC, and each of
them has 200k distinct blob files (I have 4k in my day-to-day account),
and we used 20 chars, then the expected value for the number of name
collisions would be ~0.0002 (and the probability that there is a least
one name collision is lower than that) [^1]. I added 12 more characters
to be on the super safe side, but this wouldn't be necessary and I could
also make it 20 instead of 32.
- Not 100% sure whether that's necessary at all - it would mainly be
necessary if we might hit a length limit on some file systems (the
blobdir is usually sth like
`accounts/2ff9fc096d2f46b6832b24a1ed99c0d6/dc.db-blobs` (53 chars), plus
64 chars for the filename would be 117).
- [x] "touch" the files to prevent them from being deleted
- [x] TODOs in the code
For later PRs:
- Replace `BlobObject::create(…)` with
`BlobObject::create_and_deduplicate(…)` in order to deduplicate
everytime core creates a file
- Modify JsonRPC to deduplicate blob files
- Possibly rename BlobObject.name to BlobObject.file in order to prevent
confusion (because `name` usually means "user-visible-name", not "name
of the file on disk").
[^1]: Calculated with both https://printfn.github.io/fend/ and
https://www.geogebra.org/calculator, both of which came to the same
result
([1](https://github.com/user-attachments/assets/bbb62550-3781-48b5-88b1-ba0e29c28c0d),
[2](https://github.com/user-attachments/assets/82171212-b797-4117-a39f-0e132eac7252))
---------
Co-authored-by: l <link2xt@testrun.org>
- This [is said to lead improve compilation
speed](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html#Assorted-Tricks)
- When grepping for a function invocation, this makes it easy to see whether it's from a test or "real" code
- We're calling the files e.g. `chat_tests.rs` instead of `tests.rs` for the same reason why we moved `imap/mod.rs` to `imap.rs`: Otherwise, your editor always shows you that you're in the file `tests.rs` and you don't know which one.
This is only moving mimeparser and chat tests, because these were the
biggest files; we can move more files in subsequent PRs if we like it.
If we create an unpromoted group,
add a member there and then remove it
before we promote a group, there is no need to
add such member to the list of past members
and send the address of this member to the group
when it is promoted.
> _took quite some time until i found the time to finish this PR and to
find a time window that does not disturb other developments too much,
but here we are:_
this PR enables UI to improve "Saved messages" hugely, bringing it on
WhatsApp's "Starred Messages" or Telegram's "Saved Messages" level. with
this PR, UIs can add the following functionality with few effort ([~100
loc on iOS](https://github.com/deltachat/deltachat-ios/pull/2527)):
- add a "Save" button in the messages context menu, allowing to save a
message
- show directly in the chat if a message was saved, eg. by a little star
★
- in "Saved Messages", show the message in its context - so with author,
avatar, date and keep incoming/outgoing state
- in "Saved Messages", a button to go to the original message can be
added
- if the original message was deleted, one can still go to the original
chat
these features were often requested, in the forum, but also in many
one-to-one discussions, recently at the global gathering.
moreover, in contrast to the old method with "forward to saved", no
traffic is wasted - the messages are saved locally, and only a small
sync messages is sent around
this is how it looks out on iOS:
<img width="260" alt="Screenshot 2025-01-17 at 00 02 51"
src="https://github.com/user-attachments/assets/902741b6-167f-4af1-9f9a-bd0439dd20d0"
/> <img width="353" alt="Screenshot 2025-01-17 at 00 05
33"
src="https://github.com/user-attachments/assets/97eceb79-6e43-4a89-9792-2e077e7141cd"
/>
technically, still a copy is done from the original message (with
already now deduplicated blobs), so that things work nicely with
deletion modes; eg. you can save an important message and preserve it
from deletion that way.
jsonrpc can be done in a subsequent PR, i was implementing the UI on iOS
where that was not needed (and most API were part of message object that
is not in use in jsonrpc atm)
@hpk42 the forward issue we discussed earller that day is already solved
(first implementation did not had an explict save_msgs() but was using
forward_msgs(SELF) as saving - with the disadvantage that forwarding to
SELF is not working, eg. if one wants the old behaviour) acutally, that
made the PR a lot nicer, as now very few existing code is changed
<details>
<summary>previous considerations and abandoned things</summary>
while working on this PR, there was also the idea to just set a flag
“starred” in the message table and not copy anything. however, while
tempting, that would result in more complexity, questions and drawbacks
in UI:
- delete-message, delete-chat, auto-deletion, clear-chat would raise
questions - what do do with the “Starred”? having a copy in “Saved
Messages” does not raise this question
- newly saved messages appear naturally as “new” in “Saved Messages”,
simply setting a flag would show them somewhere in between - unless we
do additional effort
- “Saved Messages” already has its place in the UI - and allows to
_directly_ save things there as well - not easily doable with “starring”
- one would need to re-do many things that already exist in “Saved
Messages”, at least in core
- one idea to solve some of the issues could be to have “Starred” as
well as “Saved Messages” - however, that would irritate user, one would
remember exactly what was done with a message, and understand the fine
differences
whatsapp does this “starred”, btw, so when original is deleted, starred
is deleted as well. Telegram does things similar to us, Signal does
nothing. Whatsapp has a per-chat view for starred messages - if needed,
we could do sth. like that as well, however, let’s first see how far the
“all view” goes (in contrast to whatsapp, we have profiles for
separation as well)
for the wording: “saving” is what we’re doing here, this is much more on
point as “starring” - which is more the idea of a “bookmark”, and i
think, whatsapp uses this wording to not raise false expectations
(still, i know of ppl that were quite upset that a “starred” message was
deleted when eg. the chat was cleared to save some memory)
wrt webxdc app updates: options that come into mind were: _empty_ (as
now), _snapshot_ (copy all updates) or _shortcut_ (always open
original). i am not sure what the best solution is, the easiest was
_empty_, so i went for that, as it is (a) obvious, and what is already
done with forwarding and (b) the original is easy to open now (in
contrast to forwarding).
so, might totally be that we need or want to tweak things here, but i
would leave that outside the first iteration, things are not worsened in
that area
wrt reactions: as things are detached, similar to webxdc updates, we do
not not to show the original reactions - that way, one can use reactions
for private markers (telegram is selling that feature :)
to the icon: a disk or a bookmark might be other options, but the star
is nicer and also know from other apps - and anyways a but vague UX
wise. so i think, it is fine
finally, known issue is that if a message was saved that does not exist
on another device, it does not get there. i think, that issue is a weak
one, and can be ignored mostly, most times, user will save messages soon
after receiving, and if on some devices auto-deletion is done, it is
maybe not even expected to have suddenly another copy there
</details>
EDIT: once this is merged, detailed issues about what to do should be
filed for android/desktop (however, they do not have urgency to adapt,
things will continue working as is)
---------
Co-authored-by: Hocuri <hocuri@gmx.de>
This partially restores the fix from c9cf2b7f2e
that was removed during the addition of new group consistency at de63527d94
but only for "Member added" messages.
Multiple "Member added" messages happen
when the same QR code is processed multiple times
by multiple devices.
Using `repeat_vars()` to generate SQL statements led to some of them having more than
`SQLITE_MAX_VARIABLE_NUMBER` parameters and thus failing, so let's get rid of this pattern. But
let's not optimise for now and just repeat executing an SQL statement in a loop, all the places
where `repeat_vars()` is used seem not performance-critical and containing functions execute other
SQL statements in loops. If needed, performance can be improved by preparing a statement and
executing it in a loop. An exception is `lookup_chat_or_create_adhoc_group()` where `repeat_vars()`
can't be replaced with a query loop, there we need to replace the `SELECT` query with a read
transaction creating a temporary table which is used to perform the SELECT query then.
Before, `Chat::why_cant_send()` just returned `CantSendReason` after the first unsuccessful check
allowing to handle the result and finally send the message if the condition is acceptable in which
case the remaining checks are not done. This didn't result in any bugs, but to make the code more
robust let's add a functional parameter to filter failed checks without early return.
This implements new group consistency algorithm described in
<https://github.com/deltachat/deltachat-core-rust/issues/6401>
New `Chat-Group-Member-Timestamps` header is added
to send timestamps of member additions and removals.
Member is part of the chat if its addition timestamp
is greater or equal to the removal timestamp.
Change `BccSelf` default to 0 for chatmail configurations and enable it upon a backup export. As for
`DeleteServerAfter` who was set to 0 upon a backup export before, make its default dependent on
`BccSelf` for chatmail. We don't need `BccSelf` for chatmail by default because we assume
single-device use. Also `BccSelf` is needed for "classic" email accounts even if `DeleteServerAfter`
is set to "immediately" to detect that a message was sent if SMTP server is slow to respond and
connection is lost before receiving the status line which isn't a problem for chatmail servers.
Even if UIs don't call `Message::force_sticker()`, they don't want conversions of `Sticker` to
`Image` if it's obviously not an image, particularly, has non-image extension. Also UIs don't want
conversions of `Sticker` to anything other than `Image`, so let's keep the `Sticker` viewtype in
this case.
It was called from `receive_imf` when an outgoing message is received. But
`Imap::fetch_new_messages()` already calls `chat::mark_old_messages_as_noticed()` which does the job
better (per-message).
SQL statements fail if the number of variables
exceeds `SQLITE_LIMIT_VARIABLE_NUMBER`.
Remaining repeat_vars() calls are difficult to replace
and use arrays passed from the UI,
e.g. forwarded message IDs or marked as seen IDs.
This PR:
- Moves the note about the false positive to the end of the test output,
where it is more likely to be noticed
- Also notes in test_modify_chat_disordered() and
test_setup_contact_*(), in addition to the existing note in
test_was_seen_recently()
This fixes HTML display of truncated (long) sent messages ("Show full message" in UIs). Before,
incorrect HTML was stored (with missing line breaks etc.) for them. Now stored plaintext is
formatted to HTML upon calling `MsgId::get_html()` and this results in the same HTML as on a
receiver side.
A sync message for accepting or blocking a 1:1 chat may arrive before the first message from the
contact, when it does not exist yet. This frequently happens in non-chatmail accounts that have
moving to the DeltaChat folder disabled because Delta Chat unconditionally uploads sync messages to
the DeltaChat folder. Let's create a hidden contact in this case and a 1:1 chat for it.
with this PR, when an `.xdc` with `request_integration = map` in the
manifest is added to the "Saved Messages" chat, it is used _locally_ as
an replacement for the shipped maps.xdc (other devices will see the
`.xdc` but not use it)
this allows easy development and adapting the map to use services that
work better in some area.
there are lots of known discussions and ideas about adding more barriers
of safety. however, after internal discussions, we decided to move
forward and also to allow internet, if requested by an integration (as
discussed at
https://github.com/deltachat/deltachat-core-rust/pull/3516).
the gist is to ease development and to make users who want to adapt,
actionable _now_, without making things too hard and adding too high
barriers or stressing our own resources/power too much.
note, that things are still experimental and will be the next time -
without the corresponding switch being enabled, nothing will work at
all, so we can be quite relaxed here :)
for android/ios, things will work directly. for desktop, allow_internet
needs to be accepted unconditionally from core. for the future, we might
add a question before using an integration and/or add signing. or sth.
completely different - but for now, the thing is to get started.
nb: "integration" field in the webxdc-info is experimental as well and
should not be used in UIs at all currently, it may vanish again and is
there mainly for simplicity of the code; therefore, no need to document
that.
successor of https://github.com/deltachat/deltachat-core-rust/pull/5461
this is how it looks like currently - again, please note that all that
is an experiment!
<img width=320
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/f659c891-f46a-4e28-9d0a-b6783d69be8d>
<img width=320
src=https://github.com/deltachat/deltachat-core-rust/assets/9800740/54549b3c-a894-4568-9e27-d5f1caea2d22>
... when going out of experimental, there are loots of ideas, eg.
changing "Start" to "integrate"
There were many cases in which "member added/removed" messages were added to chats even if they
actually do nothing because a member is already added or removed. But primarily this fixes a
scenario when Alice has several devices and shares an invite link somewhere, and both their devices
handle the SecureJoin and issue `ChatGroupMemberAdded` messages so all other members see a
duplicated group member addition.
This adds a function to `Message`:
```rust
pub fn new_text(text: String) -> Self {
Message {
viewtype: Viewtype::Text,
text,
..Default::default()
}
}
```
I keep expecting that a function like this must exist and being
surprised that it doesn't.
Open question is whether it should be `pub` or `pub(crate)` - I made it
`pub` for now because it may be useful for others and we currently we
aren't thinking about the Rust API that much, anyway, but I can make it
`pub(crate)`, too (then it can't be used in deltachat-jsonrpc and
deltachat-repl).
I replaced some usages of Message::new(Viewtype::Text), but not all yet,
I'm going to do this in a follow-up, which will remove another around 65
LOC.
0a63083df7 (fix: Shorten message text in locally sent messages too)
sets `msgs.mime_modified` for long outgoing messages, but forgets to save full message text.
