There were many cases in which "member added/removed" messages were added to chats even if they
actually do nothing because a member is already added or removed. But primarily this fixes a
scenario when Alice has several devices and shares an invite link somewhere, and both their devices
handle the SecureJoin and issue `ChatGroupMemberAdded` messages so all other members see a
duplicated group member addition.
This adds a function to `Message`:
```rust
pub fn new_text(text: String) -> Self {
Message {
viewtype: Viewtype::Text,
text,
..Default::default()
}
}
```
I keep expecting that a function like this must exist and being
surprised that it doesn't.
Open question is whether it should be `pub` or `pub(crate)` - I made it
`pub` for now because it may be useful for others and we currently we
aren't thinking about the Rust API that much, anyway, but I can make it
`pub(crate)`, too (then it can't be used in deltachat-jsonrpc and
deltachat-repl).
I replaced some usages of Message::new(Viewtype::Text), but not all yet,
I'm going to do this in a follow-up, which will remove another around 65
LOC.
0a63083df7 (fix: Shorten message text in locally sent messages too)
sets `msgs.mime_modified` for long outgoing messages, but forgets to save full message text.
Received messages shouldn't mingle with just sent ones and appear somewhere in the middle of the
chat, so we go after the newest non fresh message.
But if a received outgoing message is older than some `InSeen` message, better sort the received
message purely by timestamp (this is an heuristic in order not to break the Gmail-like case
simulated by `verified_chats::test_old_message_4()`). We could place the received message just
before that `InSeen` message, but anyway the user may not notice it.
At least this fixes outgoing messages sorting for shared accounts where messages from other devices
should be sorted the same way as incoming ones.
Instead of treating NULL type error
as absence of the row,
handle NULL values with SQL.
Previously we sometimes
accidentally treated a single column
being NULL as the lack of the whole row.
Info messages are added
at the beginning of unpromoted group chats
("Others will only see this group after you sent a first message."),
may be created by WebXDC etc.
They are not sent outside
and have local Message-ID that
is not known to other recipients
so they should be skipped when constructing
In-Reply-To and References.
Instead of generating 72 random bits
and reducing them to 66 bits of Base64 characters,
generate 144 bits (18 bytes)
which is exactly 24 Base64 characters.
This should still be accepted by existing
Delta Chat clients which expect group ID
to be between 11 and 32 characters.
Message-ID creation is also simplified
to not have `Mr.` prefix
and dot in between two IDs.
Now it is a single ID followed by `@localhost`.
Some outdated documentation comments
are removed, e.g. group messages
don't start with `Gr.` already.
Why:
- With IMAP APPEND we can upload messages directly to the DeltaChat folder (for non-chatmail
accounts).
- We can set the `\Seen` flag immediately so that if the user has other MUA, it doesn't alert about
a new message if it's just a sync message (there were several such reports on the support
forum). Though this also isn't useful for chatmail.
- We don't need SMTP envelope and overall remove some overhead on processing sync messages.
Groups promotion to other devices and QR code tokens synchronisation are not synchronised processes,
so there are reasons why a QR code token may arrive earlier than the first group message:
- We are going to upload sync messages via IMAP while group messages are sent by SMTP.
- If sync messages go to the mvbox, they can be fetched earlier than group messages from Inbox.
This makes possible to schedule one more sending of the message, the existing jobs are not
cancelled. Otherwise it's complicated to implement bots that resend messages when a new member joins
the group.
Follow-up to 3cf78749df "Emit DC_EVENT_MSGS_CHANGED for
DC_CHAT_ID_ARCHIVED_LINK when the number of archived chats with unread messages increases (#3940)".
In general we don't want to make an extra db query to know if a noticied chat is
archived. Emitting events should be cheap, better to allow false-positive `MsgsChanged` events.
`Context::send_sync_msg()` mustn't be called from multiple tasks in parallel to avoid sending the
same sync items twice because sync items are removed from the db only after successful
sending. Let's guarantee this by calling `send_sync_msg()` only from the SMTP loop. Before
`send_sync_msg()` could be called in parallel from the SMTP loop and another task doing
e.g. `chat::sync()` which led to `test_multidevice_sync_chat` being flaky because of events
triggered by duplicated sync messages.
Before, update sending might be delayed due to rate limits and later merged into large
messages. This is undesirable for apps that want to send large files over WebXDC updates because the
message with aggregated update may be too large for actual sending and hit the provider limit or
require multiple attempts on a flaky SMTP connection.
So, don't aggregate updates if the size of an aggregated update will exceed the limit of 100
KiB. This is a soft limit, so it may be exceeded if a single update is larger and it limits only the
update JSON size, so the message with all envelopes still may be larger. Also the limit may be
exceeded when updates are sent together with the WebXDC instance when resending it as the instance
size isn't accounted to not complicate the code. At least this is not worse than the previous
behaviour when all updates were attached.
SQLite search with `LIKE` is case-insensitive only for ASCII chars. To make it case-insensitive for
all messages, create a new column `msgs.txt_normalized` defaulting to `NULL` (so we do not bump up
the database size in a migration) and storing lowercased/normalized text there when the row is
created/updated. When doing a search, search over `IFNULL(txt_normalized, txt)`.
Otherwise it's impossible to remove a member with missing key from a protected group. In the worst
case a removed member will be added back due to the group membership consistency algo.
This protects Bob (the joiner) of sending unexpected-unencrypted messages during an otherwise nicely
running SecureJoin.
If things get stuck, however, we do not want to block communication -- the chat is just
opportunistic as usual, but that needs to be communicated:
1. If Bob's chat with Alice is `Unprotected` and a SecureJoin is started, then add info-message
"Establishing guaranteed end-to-end encryption, please wait..." and let `Chat::can_send()` return
`false`.
2. Once the info-message "Messages are guaranteed to be e2ee from now on" is added, let
`Chat::can_send()` return `true`.
3. If after SECUREJOIN_WAIT_TIMEOUT seconds `2.` did not happen, add another info-message "Could not
yet establish guaranteed end-to-end encryption but you may already send a message" and also let
`Chat::can_send()` return `true`.
Both `2.` and `3.` require the event `ChatModified` being sent out so that UI pick up the change wrt
`Chat::can_send()` (this is the same way how groups become updated wrt `can_send()` changes).
SECUREJOIN_WAIT_TIMEOUT should be 10-20 seconds so that we are reasonably sure that the app remains
active and receiving also on mobile devices. If the app is killed during this time then we may need
to do step 3 for any pending Bob-join chats (right now, Bob can only join one chat at a time).
`ChatId::lookup_by_contact()` returns `None` for blocked chats, so it should be only used if we need
to filter out blocked chats, e.g. when building a chatlist.
E.g. the multi-device synchronisation creates the "Saved Messages" chat as blocked, in this case the
chat icon wasn't updated before and the user avatar was displayed instead.
this PR checks if the quotes are used in a reasonable way:
- quoted messages should be send to the same chat
- or to one-to-one chats
if the quote's chat ID is not the same than the sending chat _and_ the
sending chat is not a one-to-one chat, sending is aborted.
usually, the UIs does not allow quoting in other ways, so, this check is
only a "last defence line" to avoid leaking data in case the UI has
bugs, as recently in
https://github.com/deltachat/deltachat-android/issues/3032
@iequidoo @link2xt @adbenitez i am not 100% sure about this PR, maybe
i've overseen a reasonable usecase where other quotes make sense
---------
Co-authored-by: link2xt <link2xt@testrun.org>
Otherwise, e.g. if a message is a large GIF, but its viewtype is set to `Image` by the app, this GIF
will be recoded to JPEG to reduce its size. GIFs and other special viewtypes must be always detected
and sent as is.
This is similar to `test_modify_chat_disordered`,
but tests that recovery works in the simplest case
where next message is not modifying group membership.
as discussed in several chats, this PR starts making it possible to use
Webxdc as integrations to the main app. In other word: selected parts of
the main app can be integrated as Webxdc, eg. Maps [^1]
this PR contains two parts:
- draft an Webxdc Integration API
- use the Webxdc Integration API to create a Maps Integration
to be clear: a Webxdc is not part of this PR. the PR is about marking a
Webxdc being used as a Map - and core then feeds the Webxdc with
location data. from the view of the Webxdc, the normal
`sendUpdate()`/`setUpdateListener()` is used.
things are still marked as "experimental", idea is to get that in to
allow @adbenitez and @nicodh to move forward on the integrations into
android and desktop, as well as improving the maps.xdc itself.
good news is that we currently can change the protocol between Webxdc
and core at any point :)
# Webxdc Integration API
see `dc_init_webxdc_integration()` in `deltachat.h` for overview and
documentation.
rust code is mostly in `webxdc/integration.rs` that is called by other
places as needed. current [user of the API is
deltachat-ios](https://github.com/deltachat/deltachat-ios/pull/1912),
android/desktop will probably follow.
the jsonrpc part is missing and can come in another PR when things are
settled and desktop is really starting [^2] (so we won't need to do all
iterations twice :) makes also sense, when this is done by someone
actually trying that out on desktop
while the API is prepared to allow other types of integrations (photo
editor, compose tools ...) internally, we currently ignore the type. if
that gets more crazy, we probably also need a dedicated table for the
integrations and not just a single param.
# Maps Integration
rust code is mostly in `webxdc/maps_integration.rs` that is called by
`webxdc/integration.rs` as needed.
EDIT: the idea of having a split here, is that
`webxdc/maps_integration.rs` really can focus on the json part, on the
communication with the .xdc, including tests
this PR is basic implementation, enabling to move forward on
integrations on iOS, but also on desktop and android.
the current implementation allows already the following:
- global and per-chat maps
- add and display POIs
- show positions and tracks of the last 24 hours
the current maps.xdc uses leaflet, and is in some regards better than
the current android/desktop implementations (much faster, show age of
positions, fade out positions, always show names of POIs, clearer UI).
however, we are also not bound to leaflet, it can be anything
> [**screenshots of the current
state**](https://github.com/deltachat/deltachat-ios/pull/1912)
> 👆
to move forward faster and to keep this PR small, the following will go
to a subsequent PR:
- consider allowing webxdc to use a different timewindow for the
location
- delete POIs
- jsonrpc
[^1]: maps are a good example as anyways barely native (see android
app), did cause a lot of pain on many levels in the past (technically,
bureaucratically), and have a comparable simple api
[^2]: only going for jsonrpc would only make sense if large parts of
android/ios would use jsonrpc, we're not there
---------
Co-authored-by: link2xt <link2xt@testrun.org>
