feat: add API to save messages (#5606)

> _took quite some time until i found the time to finish this PR and to
find a time window that does not disturb other developments too much,
but here we are:_

this PR enables UI to improve "Saved messages" hugely, bringing it on
WhatsApp's "Starred Messages" or Telegram's "Saved Messages" level. with
this PR, UIs can add the following functionality with few effort ([~100
loc on iOS](https://github.com/deltachat/deltachat-ios/pull/2527)):

- add a "Save" button in the messages context menu, allowing to save a
message
- show directly in the chat if a message was saved, eg. by a little star
★
- in "Saved Messages", show the message in its context - so with author,
avatar, date and keep incoming/outgoing state
- in "Saved Messages", a button to go to the original message can be
added
- if the original message was deleted, one can still go to the original
chat

these features were often requested, in the forum, but also in many
one-to-one discussions, recently at the global gathering.

moreover, in contrast to the old method with "forward to saved", no
traffic is wasted - the messages are saved locally, and only a small
sync messages is sent around

this is how it looks out on iOS:

<img width="260" alt="Screenshot 2025-01-17 at 00 02 51"
src="https://github.com/user-attachments/assets/902741b6-167f-4af1-9f9a-bd0439dd20d0"
/> &nbsp; &nbsp; <img width="353" alt="Screenshot 2025-01-17 at 00 05
33"
src="https://github.com/user-attachments/assets/97eceb79-6e43-4a89-9792-2e077e7141cd"
/>

technically, still a copy is done from the original message (with
already now deduplicated blobs), so that things work nicely with
deletion modes; eg. you can save an important message and preserve it
from deletion that way.

jsonrpc can be done in a subsequent PR, i was implementing the UI on iOS
where that was not needed (and most API were part of message object that
is not in use in jsonrpc atm)

@hpk42 the forward issue we discussed earller that day is already solved
(first implementation did not had an explict save_msgs() but was using
forward_msgs(SELF) as saving - with the disadvantage that forwarding to
SELF is not working, eg. if one wants the old behaviour) acutally, that
made the PR a lot nicer, as now very few existing code is changed

<details>
<summary>previous considerations and abandoned things</summary>

while working on this PR, there was also the idea to just set a flag
“starred” in the message table and not copy anything. however, while
tempting, that would result in more complexity, questions and drawbacks
in UI:

- delete-message, delete-chat, auto-deletion, clear-chat would raise
questions - what do do with the “Starred”? having a copy in “Saved
Messages” does not raise this question
- newly saved messages appear naturally as “new” in “Saved Messages”,
simply setting a flag would show them somewhere in between - unless we
do additional effort
- “Saved Messages” already has its place in the UI - and allows to
_directly_ save things there as well - not easily doable with “starring”
- one would need to re-do many things that already exist in “Saved
Messages”, at least in core
- one idea to solve some of the issues could be to have “Starred” as
well as “Saved Messages” - however, that would irritate user, one would
remember exactly what was done with a message, and understand the fine
differences

whatsapp does this “starred”, btw, so when original is deleted, starred
is deleted as well. Telegram does things similar to us, Signal does
nothing. Whatsapp has a per-chat view for starred messages - if needed,
we could do sth. like that as well, however, let’s first see how far the
“all view” goes (in contrast to whatsapp, we have profiles for
separation as well)

for the wording: “saving” is what we’re doing here, this is much more on
point as “starring” - which is more the idea of a “bookmark”, and i
think, whatsapp uses this wording to not raise false expectations
(still, i know of ppl that were quite upset that a “starred” message was
deleted when eg. the chat was cleared to save some memory)

wrt webxdc app updates: options that come into mind were: _empty_ (as
now), _snapshot_ (copy all updates) or _shortcut_ (always open
original). i am not sure what the best solution is, the easiest was
_empty_, so i went for that, as it is (a) obvious, and what is already
done with forwarding and (b) the original is easy to open now (in
contrast to forwarding).
so, might totally be that we need or want to tweak things here, but i
would leave that outside the first iteration, things are not worsened in
that area

wrt reactions: as things are detached, similar to webxdc updates, we do
not not to show the original reactions - that way, one can use reactions
for private markers (telegram is selling that feature :)

to the icon: a disk or a bookmark might be other options, but the star
is nicer and also know from other apps - and anyways a but vague UX
wise. so i think, it is fine

finally, known issue is that if a message was saved that does not exist
on another device, it does not get there. i think, that issue is a weak
one, and can be ignored mostly, most times, user will save messages soon
after receiving, and if on some devices auto-deletion is done, it is
maybe not even expected to have suddenly another copy there

</details>

EDIT: once this is merged, detailed issues about what to do should be
filed for android/desktop (however, they do not have urgency to adapt,
things will continue working as is)

---------

Co-authored-by: Hocuri <hocuri@gmx.de>

src/chat.rs

@@ -4209,6 +4209,80 @@ pub async fn forward_msgs(context: &Context, msg_ids: &[MsgId], chat_id: ChatId)
     Ok(())
 }
 
+/// Save a copy of the message in "Saved Messages"
+/// and send a sync messages so that other devices save the message as well, unless deleted there.
+pub async fn save_msgs(context: &Context, msg_ids: &[MsgId]) -> Result<()> {
+    for src_msg_id in msg_ids {
+        let dest_rfc724_mid = create_outgoing_rfc724_mid();
+        let src_rfc724_mid = save_copy_in_self_talk(context, src_msg_id, &dest_rfc724_mid).await?;
+        context
+            .add_sync_item(SyncData::SaveMessage {
+                src: src_rfc724_mid,
+                dest: dest_rfc724_mid,
+            })
+            .await?;
+    }
+    context.send_sync_msg().await?;
+    Ok(())
+}
+
+/// Saves a copy of the given message in "Saved Messages" using the given RFC724 id.
+/// To allow UIs to have a "show in context" button,
+/// the copy contains a reference to the original message
+/// as well as to the original chat in case the original message gets deleted.
+/// Returns data needed to add a `SaveMessage` sync item.
+pub(crate) async fn save_copy_in_self_talk(
+    context: &Context,
+    src_msg_id: &MsgId,
+    dest_rfc724_mid: &String,
+) -> Result<String> {
+    let dest_chat_id = ChatId::create_for_contact(context, ContactId::SELF).await?;
+    let mut msg = Message::load_from_db(context, *src_msg_id).await?;
+    msg.param.remove(Param::Cmd);
+    msg.param.remove(Param::WebxdcDocument);
+    msg.param.remove(Param::WebxdcDocumentTimestamp);
+    msg.param.remove(Param::WebxdcSummary);
+    msg.param.remove(Param::WebxdcSummaryTimestamp);
+
+    if !msg.original_msg_id.is_unset() {
+        bail!("message already saved.");
+    }
+
+    let copy_fields = "from_id, to_id, timestamp_sent, timestamp_rcvd, type, txt, txt_raw, \
+                             mime_modified, mime_headers, mime_compressed, mime_in_reply_to, subject, msgrmsg";
+    let row_id = context
+        .sql
+        .insert(
+            &format!(
+                "INSERT INTO msgs ({copy_fields}, chat_id, rfc724_mid, state, timestamp, param, starred) \
+                            SELECT {copy_fields}, ?, ?, ?, ?, ?, ? \
+                            FROM msgs WHERE id=?;"
+            ),
+            (
+                dest_chat_id,
+                dest_rfc724_mid,
+                if msg.from_id == ContactId::SELF {
+                    MessageState::OutDelivered
+                } else {
+                    MessageState::InSeen
+                },
+                create_smeared_timestamp(context),
+                msg.param.to_string(),
+                src_msg_id,
+                src_msg_id,
+            ),
+        )
+        .await?;
+    let dest_msg_id = MsgId::new(row_id.try_into()?);
+
+    context.emit_msgs_changed(msg.chat_id, *src_msg_id);
+    context.emit_msgs_changed(dest_chat_id, dest_msg_id);
+    chatlist_events::emit_chatlist_changed(context);
+    chatlist_events::emit_chatlist_item_changed(context, dest_chat_id);
+
+    Ok(msg.rfc724_mid)
+}
+
 /// Resends given messages with the same Message-ID.
 ///
 /// This is primarily intended to make existing webxdcs available to new chat members.
@@ -4703,7 +4777,7 @@ mod tests {
     use crate::chatlist::get_archived_cnt;
     use crate::constants::{DC_GCL_ARCHIVED_ONLY, DC_GCL_NO_SPECIALS};
     use crate::headerdef::HeaderDef;
-    use crate::message::delete_msgs;
+    use crate::message::{delete_msgs, MessengerMessage};
     use crate::receive_imf::receive_imf;
     use crate::test_utils::{sync, TestContext, TestContextManager, TimeShiftFalsePositiveNote};
     use strum::IntoEnumIterator;
@@ -6836,6 +6910,136 @@ mod tests {
         Ok(())
     }
 
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_save_msgs() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+        let bob = TestContext::new_bob().await;
+        let alice_chat = alice.create_chat(&bob).await;
+
+        let sent = alice.send_text(alice_chat.get_id(), "hi, bob").await;
+        let sent_msg = Message::load_from_db(&alice, sent.sender_msg_id).await?;
+        assert!(sent_msg.get_saved_msg_id(&alice).await?.is_none());
+        assert!(sent_msg.get_original_msg_id(&alice).await?.is_none());
+
+        let self_chat = alice.get_self_chat().await;
+        save_msgs(&alice, &[sent.sender_msg_id]).await?;
+
+        let saved_msg = alice.get_last_msg_in(self_chat.id).await;
+        assert_ne!(saved_msg.get_id(), sent.sender_msg_id);
+        assert!(saved_msg.get_saved_msg_id(&alice).await?.is_none());
+        assert_eq!(
+            saved_msg.get_original_msg_id(&alice).await?.unwrap(),
+            sent.sender_msg_id
+        );
+        assert_eq!(saved_msg.get_text(), "hi, bob");
+        assert!(!saved_msg.is_forwarded()); // UI should not flag "saved messages" as "forwarded"
+        assert_eq!(saved_msg.is_dc_message, MessengerMessage::Yes);
+        assert_eq!(saved_msg.get_from_id(), ContactId::SELF);
+        assert_eq!(saved_msg.get_state(), MessageState::OutDelivered);
+        assert_ne!(saved_msg.rfc724_mid(), sent_msg.rfc724_mid());
+
+        let sent_msg = Message::load_from_db(&alice, sent.sender_msg_id).await?;
+        assert_eq!(
+            sent_msg.get_saved_msg_id(&alice).await?.unwrap(),
+            saved_msg.id
+        );
+        assert!(sent_msg.get_original_msg_id(&alice).await?.is_none());
+
+        let rcvd_msg = bob.recv_msg(&sent).await;
+        let self_chat = bob.get_self_chat().await;
+        save_msgs(&bob, &[rcvd_msg.id]).await?;
+        let saved_msg = bob.get_last_msg_in(self_chat.id).await;
+        assert_ne!(saved_msg.get_id(), rcvd_msg.id);
+        assert_eq!(
+            saved_msg.get_original_msg_id(&bob).await?.unwrap(),
+            rcvd_msg.id
+        );
+        assert_eq!(saved_msg.get_text(), "hi, bob");
+        assert!(!saved_msg.is_forwarded());
+        assert_eq!(saved_msg.is_dc_message, MessengerMessage::Yes);
+        assert_ne!(saved_msg.get_from_id(), ContactId::SELF);
+        assert_eq!(saved_msg.get_state(), MessageState::InSeen);
+        assert_ne!(saved_msg.rfc724_mid(), rcvd_msg.rfc724_mid());
+
+        // delete original message
+        delete_msgs(&bob, &[rcvd_msg.id]).await?;
+        let saved_msg = Message::load_from_db(&bob, saved_msg.id).await?;
+        assert!(saved_msg.get_original_msg_id(&bob).await?.is_none());
+
+        // delete original chat
+        rcvd_msg.chat_id.delete(&bob).await?;
+        let msg = Message::load_from_db(&bob, saved_msg.id).await?;
+        assert!(msg.get_original_msg_id(&bob).await?.is_none());
+
+        Ok(())
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_saved_msgs_not_added_to_shared_chats() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let alice = tcm.alice().await;
+        let bob = tcm.bob().await;
+
+        let msg = tcm.send_recv_accept(&alice, &bob, "hi, bob").await;
+
+        let self_chat = bob.get_self_chat().await;
+        save_msgs(&bob, &[msg.id]).await?;
+        let msg = bob.get_last_msg_in(self_chat.id).await;
+        let contact = Contact::get_by_id(&bob, msg.get_from_id()).await?;
+        assert_eq!(contact.get_addr(), "alice@example.org");
+
+        let shared_chats = Chatlist::try_load(&bob, 0, None, Some(contact.id)).await?;
+        assert_eq!(shared_chats.len(), 1);
+        assert_eq!(
+            shared_chats.get_chat_id(0).unwrap(),
+            bob.get_chat(&alice).await.id
+        );
+
+        Ok(())
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_forward_from_saved_to_saved() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+        let bob = TestContext::new_bob().await;
+        let sent = alice.send_text(alice.create_chat(&bob).await.id, "k").await;
+
+        bob.recv_msg(&sent).await;
+        let orig = bob.get_last_msg().await;
+        let self_chat = bob.get_self_chat().await;
+        save_msgs(&bob, &[orig.id]).await?;
+        let saved1 = bob.get_last_msg().await;
+        assert_eq!(
+            saved1.get_original_msg_id(&bob).await?.unwrap(),
+            sent.sender_msg_id
+        );
+        assert_ne!(saved1.from_id, ContactId::SELF);
+
+        forward_msgs(&bob, &[saved1.id], self_chat.id).await?;
+        let saved2 = bob.get_last_msg().await;
+        assert!(saved2.get_original_msg_id(&bob).await?.is_none(),);
+        assert_eq!(saved2.from_id, ContactId::SELF);
+
+        Ok(())
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_save_from_saved_to_saved_failing() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+        let bob = TestContext::new_bob().await;
+        let sent = alice.send_text(alice.create_chat(&bob).await.id, "k").await;
+
+        bob.recv_msg(&sent).await;
+        let orig = bob.get_last_msg().await;
+        save_msgs(&bob, &[orig.id]).await?;
+        let saved1 = bob.get_last_msg().await;
+
+        let result = save_msgs(&bob, &[saved1.id]).await;
+        assert!(result.is_err());
+
+        Ok(())
+    }
+
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_resend_own_message() -> Result<()> {
         // Alice creates group with Bob and sends an initial message

src/html.rs

@@ -291,7 +291,7 @@ pub fn new_html_mimepart(html: String) -> PartBuilder {
 mod tests {
     use super::*;
     use crate::chat;
-    use crate::chat::forward_msgs;
+    use crate::chat::{forward_msgs, save_msgs};
     use crate::config::Config;
     use crate::contact::ContactId;
     use crate::message::{MessengerMessage, Viewtype};
@@ -499,6 +499,38 @@ test some special html-characters as < > and & but also " and &#x
         assert!(html.contains("this is <b>html</b>"));
     }
 
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_html_save_msg() -> Result<()> {
+        // Alice receives a non-delta html-message
+        let alice = TestContext::new_alice().await;
+        let chat = alice
+            .create_chat_with_contact("", "sender@testrun.org")
+            .await;
+        let raw = include_bytes!("../test-data/message/text_alt_plain_html.eml");
+        receive_imf(&alice, raw, false).await?;
+        let msg = alice.get_last_msg_in(chat.get_id()).await;
+
+        // Alice saves the message
+        let self_chat = alice.get_self_chat().await;
+        save_msgs(&alice, &[msg.id]).await?;
+        let saved_msg = alice.get_last_msg_in(self_chat.get_id()).await;
+        assert_ne!(saved_msg.id, msg.id);
+        assert_eq!(
+            saved_msg.get_original_msg_id(&alice).await?.unwrap(),
+            msg.id
+        );
+        assert!(!saved_msg.is_forwarded()); // UI should not flag "saved messages" as "forwarded"
+        assert_ne!(saved_msg.get_from_id(), ContactId::SELF);
+        assert_eq!(saved_msg.get_from_id(), msg.get_from_id());
+        assert_eq!(saved_msg.is_dc_message, MessengerMessage::No);
+        assert!(saved_msg.get_text().contains("this is plain"));
+        assert!(saved_msg.has_html());
+        let html = saved_msg.get_id().get_html(&alice).await?.unwrap();
+        assert!(html.contains("this is <b>html</b>"));
+
+        Ok(())
+    }
+
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_html_forwarding_encrypted() {
         // Alice receives a non-delta html-message

src/message.rs

@@ -470,6 +470,7 @@ pub struct Message {
     /// `In-Reply-To` header value.
     pub(crate) in_reply_to: Option<String>,
     pub(crate) is_dc_message: MessengerMessage,
+    pub(crate) original_msg_id: MsgId,
     pub(crate) mime_modified: bool,
     pub(crate) chat_blocked: Blocked,
     pub(crate) location_id: u32,
@@ -536,6 +537,7 @@ impl Message {
                     "    m.download_state AS download_state,",
                     "    m.error AS error,",
                     "    m.msgrmsg AS msgrmsg,",
+                    "    m.starred AS original_msg_id,",
                     "    m.mime_modified AS mime_modified,",
                     "    m.txt AS txt,",
                     "    m.subject AS subject,",
@@ -592,6 +594,7 @@ impl Message {
                         error: Some(row.get::<_, String>("error")?)
                             .filter(|error| !error.is_empty()),
                         is_dc_message: row.get("msgrmsg")?,
+                        original_msg_id: row.get("original_msg_id")?,
                         mime_modified: row.get("mime_modified")?,
                         text,
                         subject: row.get("subject")?,
@@ -1256,6 +1259,35 @@ impl Message {
         Ok(None)
     }
 
+    /// Returns original message ID for message from "Saved Messages".
+    pub async fn get_original_msg_id(&self, context: &Context) -> Result<Option<MsgId>> {
+        if !self.original_msg_id.is_special() {
+            if let Some(msg) = Message::load_from_db_optional(context, self.original_msg_id).await?
+            {
+                return if msg.chat_id.is_trash() {
+                    Ok(None)
+                } else {
+                    Ok(Some(msg.id))
+                };
+            }
+        }
+        Ok(None)
+    }
+
+    /// Check if the message was saved and returns the corresponding message inside "Saved Messages".
+    /// UI can use this to show a symbol beside the message, indicating it was saved.
+    /// The message can be un-saved by deleting the returned message.
+    pub async fn get_saved_msg_id(&self, context: &Context) -> Result<Option<MsgId>> {
+        let res: Option<MsgId> = context
+            .sql
+            .query_get_value(
+                "SELECT id FROM msgs WHERE starred=? AND chat_id!=?",
+                (self.id, DC_CHAT_ID_TRASH),
+            )
+            .await?;
+        Ok(res)
+    }
+
     /// Force the message to be sent in plain text.
     pub fn force_plaintext(&mut self) {
         self.param.set_int(Param::ForcePlaintext, 1);
@@ -2168,7 +2200,8 @@ mod tests {
 
     use super::*;
     use crate::chat::{
-        self, add_contact_to_chat, marknoticed_chat, send_text_msg, ChatItem, ProtectionStatus,
+        self, add_contact_to_chat, forward_msgs, marknoticed_chat, save_msgs, send_text_msg,
+        ChatItem, ProtectionStatus,
     };
     use crate::chatlist::Chatlist;
     use crate::config::Config;
@@ -2477,6 +2510,41 @@ mod tests {
         );
     }
 
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_get_original_msg_id() -> Result<()> {
+        let alice = TestContext::new_alice().await;
+        let bob = TestContext::new_bob().await;
+
+        // normal sending of messages does not have an original ID
+        let one2one_chat = alice.create_chat(&bob).await;
+        let sent = alice.send_text(one2one_chat.id, "foo").await;
+        let orig_msg = Message::load_from_db(&alice, sent.sender_msg_id).await?;
+        assert!(orig_msg.get_original_msg_id(&alice).await?.is_none());
+        assert!(orig_msg.parent(&alice).await?.is_none());
+        assert!(orig_msg.quoted_message(&alice).await?.is_none());
+
+        // forwarding to "Saved Messages", the message gets the original ID attached
+        let self_chat = alice.get_self_chat().await;
+        save_msgs(&alice, &[sent.sender_msg_id]).await?;
+        let saved_msg = alice.get_last_msg_in(self_chat.get_id()).await;
+        assert_ne!(saved_msg.get_id(), orig_msg.get_id());
+        assert_eq!(
+            saved_msg.get_original_msg_id(&alice).await?.unwrap(),
+            orig_msg.get_id()
+        );
+        assert!(saved_msg.parent(&alice).await?.is_none());
+        assert!(saved_msg.quoted_message(&alice).await?.is_none());
+
+        // forwarding from "Saved Messages" back to another chat, detaches original ID
+        forward_msgs(&alice, &[saved_msg.get_id()], one2one_chat.get_id()).await?;
+        let forwarded_msg = alice.get_last_msg_in(one2one_chat.get_id()).await;
+        assert_ne!(forwarded_msg.get_id(), saved_msg.get_id());
+        assert_ne!(forwarded_msg.get_id(), orig_msg.get_id());
+        assert!(forwarded_msg.get_original_msg_id(&alice).await?.is_none());
+
+        Ok(())
+    }
+
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     async fn test_markseen_msgs() -> Result<()> {
         let alice = TestContext::new_alice().await;

src/sql/migrations.rs

@@ -116,8 +116,6 @@ CREATE INDEX msgs_mdns_index1 ON msgs_mdns (msg_id);"#,
             r#"
 ALTER TABLE chats ADD COLUMN archived INTEGER DEFAULT 0;
 CREATE INDEX chats_index2 ON chats (archived);
--- 'starred' column is not used currently
--- (dropping is not easily doable and stop adding it will make reusing it complicated)
 ALTER TABLE msgs ADD COLUMN starred INTEGER DEFAULT 0;
 CREATE INDEX msgs_index5 ON msgs (starred);"#,
             17,

src/sync.rs

@@ -16,7 +16,7 @@ use crate::param::Param;
 use crate::sync::SyncData::{AddQrToken, AlterChat, DeleteQrToken};
 use crate::token::Namespace;
 use crate::tools::time;
-use crate::{stock_str, token};
+use crate::{message, stock_str, token};
 
 /// Whether to send device sync messages. Aimed for usage in the internal API.
 #[derive(Debug, PartialEq)]
@@ -62,6 +62,10 @@ pub(crate) enum SyncData {
         key: Config,
         val: String,
     },
+    SaveMessage {
+        src: String,  // RFC724 id (i.e. "Message-Id" header)
+        dest: String, // RFC724 id (i.e. "Message-Id" header)
+    },
 }
 
 #[derive(Debug, Serialize, Deserialize)]
@@ -259,6 +263,7 @@ impl Context {
                     DeleteQrToken(token) => self.delete_qr_token(token).await,
                     AlterChat { id, action } => self.sync_alter_chat(id, action).await,
                     SyncData::Config { key, val } => self.sync_config(key, val).await,
+                    SyncData::SaveMessage { src, dest } => self.save_message(src, dest).await,
                 },
                 SyncDataOrUnknown::Unknown(data) => {
                     warn!(self, "Ignored unknown sync item: {data}.");
@@ -282,6 +287,13 @@ impl Context {
         token::delete(self, Namespace::Auth, &token.auth).await?;
         Ok(())
     }
+
+    async fn save_message(&self, src_rfc724_mid: &str, dest_rfc724_mid: &String) -> Result<()> {
+        if let Some((src_msg_id, _)) = message::rfc724_mid_exists(self, src_rfc724_mid).await? {
+            chat::save_copy_in_self_talk(self, &src_msg_id, dest_rfc724_mid).await?;
+        }
+        Ok(())
+    }
 }
 
 #[cfg(test)]