feat!: QR codes and symmetric encryption for broadcast channels (#7268)

Follow-up for https://github.com/chatmail/core/pull/7042, part of
https://github.com/chatmail/core/issues/6884.

This will make it possible to create invite-QR codes for broadcast
channels, and make them symmetrically end-to-end encrypted.

- [x] Go through all the changes in #7042, and check which ones I still
need, and revert all other changes
- [x] Use the classical Securejoin protocol, rather than the new 2-step
protocol
- [x] Make the Rust tests pass
- [x] Make the Python tests pass
- [x] Fix TODOs in the code
- [x] Test it, and fix any bugs I find
- [x] I found a bug when exporting all profiles at once fails sometimes,
though this bug is unrelated to channels:
https://github.com/chatmail/core/issues/7281
- [x] Do a self-review (i.e. read all changes, and check if I see some
things that should be changed)
- [x] Have this PR reviewed and merged
- [ ] Open an issue for "TODO: There is a known bug in the securejoin
protocol"
- [ ] Create an issue that outlines how we can improve the Securejoin
protocol in the future (I don't have the time to do this right now, but
want to do it sometime in winter)
- [ ] Write a guide for UIs how to adapt to the changes (see
https://github.com/deltachat/deltachat-android/pull/3886)

## Backwards compatibility

This is not very backwards compatible:
- Trying to join a symmetrically-encrypted broadcast channel with an old
device will fail
- If you joined a symmetrically-encrypted broadcast channel with one
device, and use an old core on the other device, then the other device
will show a mostly empty chat (except for two device messages)
- If you created a broadcast channel in the past, then you will get an
error message when trying to send into the channel:

> The up to now "experimental channels feature" is about to become an officially supported one. By that, privacy will be improved, it will become faster, and less traffic will be consumed.
> 
> As we do not guarantee feature-stability for such experiments, this means, that you will need to create the channel again. 
> 
> Here is what to do:
>  • Create a new channel
>  • Tap on the channel name
>  • Tap on "QR Invite Code"
>  • Have all recipients scan the QR code, or send them the link
> 
> If you have any questions, please send an email to delta@merlinux.eu or ask at https://support.delta.chat/.


## The symmetric encryption

Symmetric encryption uses a shared secret. Currently, we use AES128 for
encryption everywhere in Delta Chat, so, this is what I'm using for
broadcast channels (though it wouldn't be hard to switch to AES256).

The secret shared between all members of a broadcast channel has 258
bits of entropy (see `fn create_broadcast_shared_secret` in the code).

Since the shared secrets have more entropy than the AES session keys,
it's not necessary to have a hard-to-compute string2key algorithm, so,
I'm using the string2key algorithm `salted`. This is fast enough that
Delta Chat can just try out all known shared secrets. [^1] In order to
prevent DOS attacks, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted` [^2].

## The "Securejoin" protocol that adds members to the channel after they
scanned a QR code

This PR uses the classical securejoin protocol, the same that is also
used for group and 1:1 invitations.

The messages sent back and forth are called `vg-request`,
`vg-auth-required`, `vg-request-with-auth`, and `vg-member-added`. I
considered using the `vc-` prefix, because from a protocol-POV, the
distinction between `vc-` and `vg-` isn't important (as @link2xt pointed
out in an in-person discussion), but
1. it would be weird if groups used `vg-` while broadcasts and 1:1 chats
used `vc-`,
2. we don't have a `vc-member-added` message yet, so, this would mean
one more different kind of message
3. we anyways want to switch to a new securejoin protocol soon, which
will be a backwards incompatible change with a transition phase. When we
do this change, we can make everything `vc-`.



[^1]: In a symmetrically encrypted message, it's not visible which
secret was used to encrypt without trying out all secrets. If this does
turn out to be too slow in the future, then we can remember which secret
was used more recently, and and try the most recent secret first. If
this is still too slow, then we can assign a short, non-unique (~2
characters) id to every shared secret, and send it in cleartext. The
receiving Delta Chat will then only try out shared secrets with this id.
Of course, this would leak a little bit of metadata in cleartext, so, I
would like to avoid it.
[^2]: A DOS attacker could send a message with a lot of encrypted
session keys, all of which use a very hard-to-compute string2key
algorithm. Delta Chat would then try to decrypt all of the encrypted
session keys with all of the known shared secrets. In order to prevent
this, as I said, Delta Chat will not attempt to decrypt with a
string2key algorithm other than `salted`

BREAKING CHANGE: A new QR type AskJoinBroadcast; cloning a broadcast
channel is no longer possible; manually adding a member to a broadcast
channel is no longer possible (only by having them scan a QR code)

src/pgp.rs

@@ -3,7 +3,7 @@
 use std::collections::{BTreeMap, HashSet};
 use std::io::{BufRead, Cursor};
 
-use anyhow::{Context as _, Result};
+use anyhow::{Context as _, Result, bail};
 use chrono::SubsecRound;
 use deltachat_contact_tools::EmailAddress;
 use pgp::armor::BlockType;
@@ -12,12 +12,13 @@ use pgp::composed::{
     Message, MessageBuilder, SecretKeyParamsBuilder, SignedPublicKey, SignedPublicSubKey,
     SignedSecretKey, SubkeyParamsBuilder, TheRing,
 };
+use pgp::crypto::aead::{AeadAlgorithm, ChunkSize};
 use pgp::crypto::ecc_curve::ECCCurve;
 use pgp::crypto::hash::HashAlgorithm;
 use pgp::crypto::sym::SymmetricKeyAlgorithm;
 use pgp::packet::{SignatureConfig, SignatureType, Subpacket, SubpacketData};
 use pgp::types::{CompressionAlgorithm, KeyDetails, Password, PublicKeyTrait, StringToKey};
-use rand_old::thread_rng;
+use rand_old::{Rng as _, thread_rng};
 use tokio::runtime::Handle;
 
 use crate::key::{DcKey, Fingerprint};
@@ -25,7 +26,7 @@ use crate::key::{DcKey, Fingerprint};
 #[cfg(test)]
 pub(crate) const HEADER_AUTOCRYPT: &str = "autocrypt-prefer-encrypt";
 
-pub const HEADER_SETUPCODE: &str = "passphrase-begin";
+pub(crate) const HEADER_SETUPCODE: &str = "passphrase-begin";
 
 /// Preferred symmetric encryption algorithm.
 const SYMMETRIC_KEY_ALGORITHM: SymmetricKeyAlgorithm = SymmetricKeyAlgorithm::AES128;
@@ -236,13 +237,17 @@ pub fn pk_calc_signature(
     Ok(sig.to_armored_string(ArmorOptions::default())?)
 }
 
-/// Decrypts the message with keys from the private key keyring.
+/// Decrypts the message:
+/// - with keys from the private key keyring (passed in `private_keys_for_decryption`)
+///   if the message was asymmetrically encrypted,
+/// - with a shared secret/password (passed in `shared_secrets`),
+///   if the message was symmetrically encrypted.
 ///
-/// Receiver private keys are provided in
-/// `private_keys_for_decryption`.
-pub fn pk_decrypt(
+/// Returns the decrypted and decompressed message.
+pub fn decrypt(
     ctext: Vec<u8>,
     private_keys_for_decryption: &[SignedSecretKey],
+    mut shared_secrets: &[String],
 ) -> Result<pgp::composed::Message<'static>> {
     let cursor = Cursor::new(ctext);
     let (msg, _headers) = Message::from_armor(cursor)?;
@@ -251,18 +256,41 @@ pub fn pk_decrypt(
     let empty_pw = Password::empty();
 
     let decrypt_options = DecryptionOptions::new();
+    let symmetric_encryption_res = check_symmetric_encryption(&msg);
+    if symmetric_encryption_res.is_err() {
+        shared_secrets = &[];
+    }
+
+    // We always try out all passwords here,
+    // but benchmarking (see `benches/decrypting.rs`)
+    // showed that the performance impact is negligible.
+    // We can improve this in the future if necessary.
+    let message_password: Vec<Password> = shared_secrets
+        .iter()
+        .map(|p| Password::from(p.as_str()))
+        .collect();
+    let message_password: Vec<&Password> = message_password.iter().collect();
+
     let ring = TheRing {
         secret_keys: skeys,
         key_passwords: vec![&empty_pw],
-        message_password: vec![],
+        message_password,
         session_keys: vec![],
         decrypt_options,
     };
-    let (msg, ring_result) = msg.decrypt_the_ring(ring, true)?;
-    anyhow::ensure!(
-        !ring_result.secret_keys.is_empty(),
-        "decryption failed, no matching secret keys"
-    );
+
+    let res = msg.decrypt_the_ring(ring, true);
+
+    let (msg, _ring_result) = match res {
+        Ok(it) => it,
+        Err(err) => {
+            if let Err(reason) = symmetric_encryption_res {
+                bail!("{err:#} (Note: symmetric decryption was not tried: {reason})")
+            } else {
+                bail!("{err:#}");
+            }
+        }
+    };
 
     // remove one layer of compression
     let msg = msg.decompress()?;
@@ -270,6 +298,34 @@ pub fn pk_decrypt(
     Ok(msg)
 }
 
+/// Returns Ok(()) if we want to try symmetrically decrypting the message,
+/// and Err with a reason if symmetric decryption should not be tried.
+///
+/// A DOS attacker could send a message with a lot of encrypted session keys,
+/// all of which use a very hard-to-compute string2key algorithm.
+/// We would then try to decrypt all of the encrypted session keys
+/// with all of the known shared secrets.
+/// In order to prevent this, we do not try to symmetrically decrypt messages
+/// that use a string2key algorithm other than 'Salted'.
+fn check_symmetric_encryption(msg: &Message<'_>) -> std::result::Result<(), &'static str> {
+    let Message::Encrypted { esk, .. } = msg else {
+        return Err("not encrypted");
+    };
+
+    if esk.len() > 1 {
+        return Err("too many esks");
+    }
+
+    let [pgp::composed::Esk::SymKeyEncryptedSessionKey(esk)] = &esk[..] else {
+        return Err("not symmetrically encrypted");
+    };
+
+    match esk.s2k() {
+        Some(StringToKey::Salted { .. }) => Ok(()),
+        _ => Err("unsupported string2key algorithm"),
+    }
+}
+
 /// Returns fingerprints
 /// of all keys from the `public_keys_for_validation` keyring that
 /// have valid signatures there.
@@ -310,8 +366,8 @@ pub fn pk_validate(
     Ok(ret)
 }
 
-/// Symmetric encryption.
-pub async fn symm_encrypt(passphrase: &str, plain: Vec<u8>) -> Result<String> {
+/// Symmetric encryption for the autocrypt setup message (ASM).
+pub async fn symm_encrypt_autocrypt_setup(passphrase: &str, plain: Vec<u8>) -> Result<String> {
     let passphrase = Password::from(passphrase.to_string());
 
     tokio::task::spawn_blocking(move || {
@@ -328,6 +384,46 @@ pub async fn symm_encrypt(passphrase: &str, plain: Vec<u8>) -> Result<String> {
     .await?
 }
 
+/// Symmetrically encrypt the message.
+/// This is used for broadcast channels and for version 2 of the Securejoin protocol.
+/// `shared secret` is the secret that will be used for symmetric encryption.
+pub async fn symm_encrypt_message(
+    plain: Vec<u8>,
+    private_key_for_signing: SignedSecretKey,
+    shared_secret: &str,
+    compress: bool,
+) -> Result<String> {
+    let shared_secret = Password::from(shared_secret.to_string());
+
+    tokio::task::spawn_blocking(move || {
+        let msg = MessageBuilder::from_bytes("", plain);
+        let mut rng = thread_rng();
+        let mut salt = [0u8; 8];
+        rng.fill(&mut salt[..]);
+        let s2k = StringToKey::Salted {
+            hash_alg: HashAlgorithm::default(),
+            salt,
+        };
+        let mut msg = msg.seipd_v2(
+            &mut rng,
+            SymmetricKeyAlgorithm::AES128,
+            AeadAlgorithm::Ocb,
+            ChunkSize::C8KiB,
+        );
+        msg.encrypt_with_password(&mut rng, s2k, &shared_secret)?;
+
+        msg.sign(&*private_key_for_signing, Password::empty(), HASH_ALGORITHM);
+        if compress {
+            msg.compression(CompressionAlgorithm::ZLIB);
+        }
+
+        let encoded_msg = msg.to_armored_string(&mut rng, Default::default())?;
+
+        Ok(encoded_msg)
+    })
+    .await?
+}
+
 /// Symmetric decryption.
 pub async fn symm_decrypt<T: BufRead + std::fmt::Debug + 'static + Send>(
     passphrase: &str,
@@ -351,7 +447,10 @@ mod tests {
     use tokio::sync::OnceCell;
 
     use super::*;
-    use crate::test_utils::{alice_keypair, bob_keypair};
+    use crate::{
+        key::{load_self_public_key, load_self_secret_key},
+        test_utils::{TestContextManager, alice_keypair, bob_keypair},
+    };
     use pgp::composed::Esk;
     use pgp::packet::PublicKeyEncryptedSessionKey;
 
@@ -364,7 +463,7 @@ mod tests {
         HashSet<Fingerprint>,
         Vec<u8>,
     )> {
-        let mut msg = pk_decrypt(ctext.to_vec(), private_keys_for_decryption)?;
+        let mut msg = decrypt(ctext.to_vec(), private_keys_for_decryption, &[])?;
         let content = msg.as_data_vec()?;
         let ret_signature_fingerprints =
             valid_signature_fingerprints(&msg, public_keys_for_validation);
@@ -560,6 +659,105 @@ mod tests {
         assert_eq!(valid_signatures.len(), 0);
     }
 
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_encrypt_decrypt_broadcast() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let alice = &tcm.alice().await;
+        let bob = &tcm.bob().await;
+
+        let plain = Vec::from(b"this is the secret message");
+        let shared_secret = "shared secret";
+        let ctext = symm_encrypt_message(
+            plain.clone(),
+            load_self_secret_key(alice).await?,
+            shared_secret,
+            true,
+        )
+        .await?;
+
+        let bob_private_keyring = crate::key::load_self_secret_keyring(bob).await?;
+        let mut decrypted = decrypt(
+            ctext.into(),
+            &bob_private_keyring,
+            &[shared_secret.to_string()],
+        )?;
+
+        assert_eq!(decrypted.as_data_vec()?, plain);
+
+        Ok(())
+    }
+
+    /// Test that we don't try to decrypt a message
+    /// that is symmetrically encrypted
+    /// with an expensive string2key algorithm
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_dont_decrypt_expensive_message() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let bob = &tcm.bob().await;
+
+        let plain = Vec::from(b"this is the secret message");
+        let shared_secret = "shared secret";
+
+        // Create a symmetrically encrypted message
+        // with an IteratedAndSalted string2key algorithm:
+
+        let shared_secret_pw = Password::from(shared_secret.to_string());
+        let msg = MessageBuilder::from_bytes("", plain);
+        let mut rng = thread_rng();
+        let s2k = StringToKey::new_default(&mut rng); // Default is IteratedAndSalted
+
+        let mut msg = msg.seipd_v2(
+            &mut rng,
+            SymmetricKeyAlgorithm::AES128,
+            AeadAlgorithm::Ocb,
+            ChunkSize::C8KiB,
+        );
+        msg.encrypt_with_password(&mut rng, s2k, &shared_secret_pw)?;
+
+        let ctext = msg.to_armored_string(&mut rng, Default::default())?;
+
+        // Trying to decrypt it should fail with a helpful error message:
+
+        let bob_private_keyring = crate::key::load_self_secret_keyring(bob).await?;
+        let error = decrypt(
+            ctext.into(),
+            &bob_private_keyring,
+            &[shared_secret.to_string()],
+        )
+        .unwrap_err();
+
+        assert_eq!(
+            error.to_string(),
+            "missing key (Note: symmetric decryption was not tried: unsupported string2key algorithm)"
+        );
+
+        Ok(())
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn test_decryption_error_msg() -> Result<()> {
+        let mut tcm = TestContextManager::new();
+        let alice = &tcm.alice().await;
+        let bob = &tcm.bob().await;
+
+        let plain = Vec::from(b"this is the secret message");
+        let pk_for_encryption = load_self_public_key(alice).await?;
+
+        // Encrypt a message, but only to self, not to Bob:
+        let ctext = pk_encrypt(plain, vec![pk_for_encryption], None, true, true).await?;
+
+        // Trying to decrypt it should fail with an OK error message:
+        let bob_private_keyring = crate::key::load_self_secret_keyring(bob).await?;
+        let error = decrypt(ctext.into(), &bob_private_keyring, &[]).unwrap_err();
+
+        assert_eq!(
+            error.to_string(),
+            "missing key (Note: symmetric decryption was not tried: not symmetrically encrypted)"
+        );
+
+        Ok(())
+    }
+
     /// Tests that recipient key IDs and fingerprints
     /// are omitted or replaced with wildcards.
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]