Connectivity view (instead of spamming the user with error_network when sth fails) (#2319)

See https://support.delta.chat/t/discussion-how-to-show-error-states/1363/10 <!-- comment -->

It turns out that it's pretty easy to distinguish between lots of states (currently Error/NotConnected, Connecting…, Getting new messages… and Connected). What's not that easy is distinguishing between an actual error and no network, because if the server just doesn't respond, it could mean that we don't have network or that we are trying ipv6, but only ipv4 works.

**WRT debouncing:**

Sending of EVENT_CONNECTIVITY_CHANGED is not debounced, but emitted every time one of the 3 threads (Inbox, Mvbox and Sentbox) has a network error, starts fetching data, or is done fetching data.
This means that it is emitted:
- 9 times when dc_maybe_network() is called or we get network connection
- 12 times when we lose network connection

Some measurements: dc_get_connectivity() takes a little more than 1ms (in my measurements back in March), dc_get_connectivity_html() takes 10-20ms. This means that it's no immmediate problem to call them very often, might increase battery drain though. For the UI it may be a lot of work to update the title everytime; at least Android is smart enough to update the title only once.

Possible problems (we don't have to worry about them now I think):
- Due to the scan_folders feature, if the user has lots of folders, the state could be "Connecting..." for quite a long time, generally DC seemed a little unresponsive to me because it took so long for "Connecting..." to go away. Telegram has a state "Updating..." that sometimes comes after "Connecting...".

To be done in other PRs:
- Better handle the case that the password was changed on the server and authenticating fails, see https://github.com/deltachat/deltachat-core-rust/issues/1923 and https://github.com/deltachat/deltachat-core-rust/issues/1768
- maybe event debouncing  (except for "Connected" connectivity events)

fix https://github.com/deltachat/deltachat-android/issues/1760

src/accounts.rs

@@ -179,6 +179,25 @@ impl Accounts {
         self.accounts.read().await.keys().copied().collect()
     }
 
+    /// This is meant especially for iOS, because iOS needs to tell the system when its background work is done.
+    ///
+    /// Returns whether all accounts finished their background work.
+    /// DC_EVENT_CONNECTIVITY_CHANGED will be sent when this turns to true.
+    ///
+    /// iOS can:
+    /// - call dc_start_io() (in case IO was not running)
+    /// - call dc_maybe_network()
+    /// - while dc_accounts_all_work_done() returns false:
+    ///   -  Wait for DC_EVENT_CONNECTIVITY_CHANGED
+    pub async fn all_work_done(&self) -> bool {
+        for account in self.accounts.read().await.values() {
+            if !account.all_work_done().await {
+                return false;
+            }
+        }
+        true
+    }
+
     pub async fn start_io(&self) {
         let accounts = &*self.accounts.read().await;
         for account in accounts.values() {

src/events.rs

@@ -185,21 +185,6 @@ pub enum EventType {
     #[strum(props(id = "400"))]
     Error(String),
 
-    /// An action cannot be performed because there is no network available.
-    ///
-    /// The library will typically try over after a some time
-    /// and when dc_maybe_network() is called.
-    ///
-    /// Network errors should be reported to users in a non-disturbing way,
-    /// however, as network errors may come in a sequence,
-    /// it is not useful to raise each an every error to the user.
-    ///
-    /// Moreover, if the UI detects that the device is offline,
-    /// it is probably more useful to report this to the user
-    /// instead of the string from data2.
-    #[strum(props(id = "401"))]
-    ErrorNetwork(String),
-
     /// An action cannot be performed because the user is not in the group.
     /// Reported eg. after a call to
     /// dc_set_chat_name(), dc_set_chat_profile_image(),
@@ -330,4 +315,11 @@ pub enum EventType {
     ///     (Bob has verified alice and waits until Alice does the same for him)
     #[strum(props(id = "2061"))]
     SecurejoinJoinerProgress { contact_id: u32, progress: usize },
+
+    /// The connectivity to the server changed.
+    /// This means that you should refresh the connectivity view
+    /// and possibly the connectivtiy HTML; see dc_get_connectivity() and
+    /// dc_get_connectivity_html() for details.
+    #[strum(props(id = "2100"))]
+    ConnectivityChanged,
 }

src/imap.rs

@@ -8,13 +8,12 @@ use std::{cmp, cmp::max, collections::BTreeMap};
 use anyhow::{bail, format_err, Context as _, Result};
 use async_imap::{
     error::Result as ImapResult,
-    types::{Fetch, Flag, Mailbox, Name, NameAttribute},
+    types::{Fetch, Flag, Mailbox, Name, NameAttribute, UnsolicitedResponse},
 };
 use async_std::channel::Receiver;
 use async_std::prelude::*;
 use num_traits::FromPrimitive;
 
-use crate::config::Config;
 use crate::constants::{
     Chattype, ShowEmails, Viewtype, DC_FETCH_EXISTING_MSGS_COUNT, DC_FOLDERS_CONFIGURED_VERSION,
     DC_LP_AUTH_OAUTH2,
@@ -36,6 +35,7 @@ use crate::provider::Socket;
 use crate::scheduler::InterruptInfo;
 use crate::stock_str;
 use crate::{chat, constants::DC_CONTACT_ID_SELF};
+use crate::{config::Config, scheduler::connectivity::ConnectivityStore};
 
 mod client;
 mod idle;
@@ -96,6 +96,8 @@ pub struct Imap {
     /// True if CAPABILITY command was run successfully once and config.can_* contain correct
     /// values.
     capabilities_determined: bool,
+
+    pub(crate) connectivity: ConnectivityStore,
 }
 
 #[derive(Debug)]
@@ -194,6 +196,7 @@ impl Imap {
             interrupt: None,
             should_reconnect: false,
             login_failed_once: false,
+            connectivity: Default::default(),
             capabilities_determined: false,
         };
 
@@ -227,9 +230,6 @@ impl Imap {
     ///
     /// It is safe to call this function if already connected, actions are performed only as needed.
     ///
-    /// Does not emit network errors, can be used to try various parameters during
-    /// autoconfiguration.
-    ///
     /// Calling this function is not enough to perform IMAP operations. Use [`Imap::prepare`]
     /// instead if you are going to actually use connection rather than trying connection
     /// parameters.
@@ -245,6 +245,8 @@ impl Imap {
             return Ok(());
         }
 
+        self.connectivity.set_connecting(context).await;
+
         let oauth2 = self.config.oauth2;
 
         let connection_res: ImapResult<Client> = if self.config.lp.security == Socket::Starttls
@@ -343,7 +345,7 @@ impl Imap {
                     self.login_failed_once = true;
                 }
 
-                self.trigger_reconnect();
+                self.trigger_reconnect(context).await;
                 Err(format_err!("{}\n\n{}", message, err))
             }
         }
@@ -377,18 +379,15 @@ impl Imap {
     ///
     /// Ensure that IMAP client is connected, folders are created and IMAP capabilities are
     /// determined.
-    ///
-    /// This function emits network error if it fails.  It should not be used during configuration
-    /// to avoid showing failed attempt errors to the user.
     pub async fn prepare(&mut self, context: &Context) -> Result<()> {
-        let res = self.connect(context).await;
-        if let Err(ref err) = res {
-            emit_event!(context, EventType::ErrorNetwork(err.to_string()));
+        if let Err(err) = self.connect(context).await {
+            self.connectivity.set_err(context, &err).await;
+            return Err(err);
         }
 
         self.ensure_configured_folders(context, true).await?;
         self.determine_capabilities().await?;
-        res
+        Ok(())
     }
 
     async fn disconnect(&mut self, context: &Context) {
@@ -417,7 +416,8 @@ impl Imap {
         self.should_reconnect
     }
 
-    pub fn trigger_reconnect(&mut self) {
+    pub async fn trigger_reconnect(&mut self, context: &Context) {
+        self.connectivity.set_connecting(context).await;
         self.should_reconnect = true;
     }
 
@@ -678,6 +678,10 @@ impl Imap {
             }
         }
 
+        if !uids.is_empty() {
+            self.connectivity.set_working(context).await;
+        }
+
         let (largest_uid_processed, error_cnt) = self
             .fetch_many_msgs(context, folder, uids, fetch_existing_msgs)
             .await;
@@ -845,7 +849,7 @@ impl Imap {
 
         if self.session.is_none() {
             // we could not get a valid imap session, this should be retried
-            self.trigger_reconnect();
+            self.trigger_reconnect(context).await;
             warn!(context, "Could not get IMAP session");
             return (None, server_uids.len());
         }
@@ -1363,6 +1367,31 @@ impl Imap {
         info!(context, "FINISHED configuring IMAP-folders.");
         Ok(())
     }
+
+    /// Return whether the server sent an unsolicited EXISTS response.
+    /// Drains all responses from `session.unsolicited_responses` in the process.
+    /// If this returns `true`, this means that new emails arrived and you should
+    /// fetch again, even if you just fetched.
+    fn server_sent_unsolicited_exists(&self, context: &Context) -> bool {
+        let session = match &self.session {
+            Some(s) => s,
+            None => return false,
+        };
+        let mut unsolicited_exists = false;
+        while let Ok(response) = session.unsolicited_responses.try_recv() {
+            match response {
+                UnsolicitedResponse::Exists(_) => {
+                    info!(
+                        context,
+                        "Need to fetch again, got unsolicited EXISTS {:?}", response
+                    );
+                    unsolicited_exists = true;
+                }
+                _ => info!(context, "ignoring unsolicited response {:?}", response),
+            }
+        }
+        unsolicited_exists
+    }
 }
 
 /// Try to get the folder meaning by the name of the folder only used if the server does not support XLIST.
@@ -1595,7 +1624,7 @@ pub(crate) async fn prefetch_should_download(
     let is_reply_to_chat_message = parent.is_some();
     if let Some(parent) = &parent {
         let chat = chat::Chat::load_from_db(context, parent.get_chat_id()).await?;
-        if chat.typ == Chattype::Group {
+        if chat.typ == Chattype::Group && !chat.id.is_special() {
             // This might be a group command, like removing a group member.
             // We really need to fetch this to avoid inconsistent group state.
             return Ok(true);

src/imap/idle.rs

@@ -2,7 +2,6 @@ use super::Imap;
 
 use anyhow::{bail, format_err, Result};
 use async_imap::extensions::idle::IdleResponse;
-use async_imap::types::UnsolicitedResponse;
 use async_std::prelude::*;
 use std::time::{Duration, SystemTime};
 
@@ -32,24 +31,11 @@ impl Imap {
         let timeout = Duration::from_secs(23 * 60);
         let mut info = Default::default();
 
+        if self.server_sent_unsolicited_exists(context) {
+            return Ok(info);
+        }
+
         if let Some(session) = self.session.take() {
-            // if we have unsolicited responses we directly return
-            let mut unsolicited_exists = false;
-            while let Ok(response) = session.unsolicited_responses.try_recv() {
-                match response {
-                    UnsolicitedResponse::Exists(_) => {
-                        warn!(context, "skip idle, got unsolicited EXISTS {:?}", response);
-                        unsolicited_exists = true;
-                    }
-                    _ => info!(context, "ignoring unsolicited response {:?}", response),
-                }
-            }
-
-            if unsolicited_exists {
-                self.session = Some(session);
-                return Ok(info);
-            }
-
             if let Ok(info) = self.idle_interrupt.try_recv() {
                 info!(context, "skip idle, got interrupt {:?}", info);
                 self.session = Some(session);
@@ -181,7 +167,7 @@ impl Imap {
                         }
                         Err(err) => {
                             error!(context, "could not fetch from folder: {:#}", err);
-                            self.trigger_reconnect()
+                            self.trigger_reconnect(context).await;
                         }
                     }
                 }

src/imap/scan_folders.rs

@@ -61,9 +61,19 @@ impl Imap {
 
             // Don't scan folders that are watched anyway
             if !watched_folders.contains(&folder.name().to_string()) && !is_drafts {
-                self.fetch_new_messages(context, folder.name(), false)
-                    .await
-                    .ok_or_log_msg(context, "Can't fetch new msgs in scanned folder");
+                // Drain leftover unsolicited EXISTS messages
+                self.server_sent_unsolicited_exists(context);
+
+                loop {
+                    self.fetch_new_messages(context, folder.name(), false)
+                        .await
+                        .ok_or_log_msg(context, "Can't fetch new msgs in scanned folder");
+
+                    // If the server sent an unsocicited EXISTS during the fetch, we need to fetch again
+                    if !self.server_sent_unsolicited_exists(context) {
+                        break;
+                    }
+                }
             }
         }
 

src/imap/select_folder.rs

@@ -37,7 +37,7 @@ impl Imap {
                         info!(context, "close/expunge succeeded");
                     }
                     Err(err) => {
-                        self.trigger_reconnect();
+                        self.trigger_reconnect(context).await;
                         return Err(Error::CloseExpungeFailed(err));
                     }
                 }
@@ -70,7 +70,7 @@ impl Imap {
         if self.session.is_none() {
             self.config.selected_folder = None;
             self.config.selected_folder_needs_expunge = false;
-            self.trigger_reconnect();
+            self.trigger_reconnect(context).await;
             return Err(Error::NoSession);
         }
 
@@ -103,7 +103,7 @@ impl Imap {
                         Ok(NewlySelected::Yes)
                     }
                     Err(async_imap::error::Error::ConnectionLost) => {
-                        self.trigger_reconnect();
+                        self.trigger_reconnect(context).await;
                         self.config.selected_folder = None;
                         Err(Error::ConnectionLost)
                     }
@@ -112,7 +112,7 @@ impl Imap {
                     }
                     Err(err) => {
                         self.config.selected_folder = None;
-                        self.trigger_reconnect();
+                        self.trigger_reconnect(context).await;
                         Err(Error::Other(err.to_string()))
                     }
                 }

src/job.rs

@@ -249,10 +249,14 @@ impl Job {
             info!(context, "smtp-sending out mime message:");
             println!("{}", String::from_utf8_lossy(&message));
         }
+
+        smtp.connectivity.set_working(context).await;
+
         let status = match smtp.send(context, recipients, message, job_id).await {
             Err(crate::smtp::send::Error::SendError(err)) => {
                 // Remote error, retry later.
-                warn!(context, "SMTP failed to send: {:?}", err);
+                warn!(context, "SMTP failed to send: {:?}", &err);
+                smtp.connectivity.set_err(context, &err).await;
                 self.pending_error = Some(err.to_string());
 
                 let res = match err {

src/log.rs

@@ -42,17 +42,6 @@ macro_rules! error {
     }};
 }
 
-#[macro_export]
-macro_rules! error_network {
-    ($ctx:expr, $msg:expr) => {
-        error_network!($ctx, $msg,)
-    };
-    ($ctx:expr, $msg:expr, $($args:expr),* $(,)?) => {{
-        let formatted = format!($msg, $($args),*);
-        emit_event!($ctx, $crate::EventType::ErrorNetwork(formatted));
-    }};
-}
-
 #[macro_export]
 macro_rules! emit_event {
     ($ctx:expr, $event:expr) => {

src/scheduler.rs

@@ -13,6 +13,10 @@ use crate::job::{self, Thread};
 use crate::message::MsgId;
 use crate::smtp::Smtp;
 
+use self::connectivity::ConnectivityStore;
+
+pub(crate) mod connectivity;
+
 pub(crate) struct StopToken;
 
 /// Job and connection scheduler.
@@ -35,7 +39,9 @@ pub(crate) enum Scheduler {
 impl Context {
     /// Indicate that the network likely has come back.
     pub async fn maybe_network(&self) {
-        self.scheduler.read().await.maybe_network().await;
+        let lock = self.scheduler.read().await;
+        lock.maybe_network().await;
+        connectivity::idle_interrupted(lock).await;
     }
 
     pub(crate) async fn interrupt_inbox(&self, info: InterruptInfo) {
@@ -107,6 +113,9 @@ async fn inbox_loop(ctx: Context, started: Sender<()>, inbox_handlers: ImapConne
                     } else {
                         if let Err(err) = connection.scan_folders(&ctx).await {
                             warn!(ctx, "{}", err);
+                            connection.connectivity.set_err(&ctx, err).await;
+                        } else {
+                            connection.connectivity.set_not_configured(&ctx).await;
                         }
                         connection.fake_idle(&ctx, None).await
                     };
@@ -132,26 +141,24 @@ async fn fetch(ctx: &Context, connection: &mut Imap) {
     match ctx.get_config(Config::ConfiguredInboxFolder).await {
         Ok(Some(watch_folder)) => {
             if let Err(err) = connection.prepare(ctx).await {
-                error_network!(ctx, "{}", err);
+                warn!(ctx, "Could not connect: {}", err);
                 return;
             }
 
             // fetch
             if let Err(err) = connection.fetch(ctx, &watch_folder).await {
-                connection.trigger_reconnect();
+                connection.trigger_reconnect(ctx).await;
                 warn!(ctx, "{:#}", err);
             }
         }
         Ok(None) => {
-            warn!(ctx, "Can not fetch inbox folder, not set");
-            connection.fake_idle(ctx, None).await;
+            info!(ctx, "Can not fetch inbox folder, not set");
         }
         Err(err) => {
             warn!(
                 ctx,
                 "Can not fetch inbox folder, failed to get config: {:?}", err
             );
-            connection.fake_idle(ctx, None).await;
         }
     }
 }
@@ -167,8 +174,9 @@ async fn fetch_idle(ctx: &Context, connection: &mut Imap, folder: Config) -> Int
 
             // fetch
             if let Err(err) = connection.fetch(ctx, &watch_folder).await {
-                connection.trigger_reconnect();
+                connection.trigger_reconnect(ctx).await;
                 warn!(ctx, "{:#}", err);
+                return InterruptInfo::new(false, None);
             }
 
             if folder == Config::ConfiguredInboxFolder {
@@ -180,22 +188,25 @@ async fn fetch_idle(ctx: &Context, connection: &mut Imap, folder: Config) -> Int
                 }
             }
 
+            connection.connectivity.set_connected(ctx).await;
+
             // idle
             if connection.can_idle() {
-                connection
-                    .idle(ctx, Some(watch_folder))
-                    .await
-                    .unwrap_or_else(|err| {
-                        connection.trigger_reconnect();
+                match connection.idle(ctx, Some(watch_folder)).await {
+                    Ok(v) => v,
+                    Err(err) => {
+                        connection.trigger_reconnect(ctx).await;
                         warn!(ctx, "{}", err);
                         InterruptInfo::new(false, None)
-                    })
+                    }
+                }
             } else {
                 connection.fake_idle(ctx, Some(watch_folder)).await
             }
         }
         Ok(None) => {
-            warn!(ctx, "Can not watch {} folder, not set", folder);
+            connection.connectivity.set_not_configured(ctx).await;
+            info!(ctx, "Can not watch {} folder, not set", folder);
             connection.fake_idle(ctx, None).await
         }
         Err(err) => {
@@ -280,6 +291,7 @@ async fn smtp_loop(ctx: Context, started: Sender<()>, smtp_handlers: SmtpConnect
                 None => {
                     // Fake Idle
                     info!(ctx, "smtp fake idle - started");
+                    connection.connectivity.set_connected(&ctx).await;
                     interrupt_info = idle_interrupt_receiver.recv().await.unwrap_or_default();
                     info!(ctx, "smtp fake idle - interrupted")
                 }
@@ -338,6 +350,11 @@ impl Scheduler {
                 .send(())
                 .await
                 .expect("mvbox start send, missing receiver");
+            mvbox_handlers
+                .connection
+                .connectivity
+                .set_not_configured(&ctx)
+                .await
         }
 
         if ctx.get_config_bool(Config::SentboxWatch).await? {
@@ -356,6 +373,11 @@ impl Scheduler {
                 .send(())
                 .await
                 .expect("sentbox start send, missing receiver");
+            sentbox_handlers
+                .connection
+                .connectivity
+                .set_not_configured(&ctx)
+                .await
         }
 
         let smtp_handle = {
@@ -508,6 +530,8 @@ struct ConnectionState {
     stop_sender: Sender<()>,
     /// Channel to interrupt idle.
     idle_interrupt_sender: Sender<InterruptInfo>,
+    /// Mutex to pass connectivity info between IMAP/SMTP threads and the API
+    connectivity: ConnectivityStore,
 }
 
 impl ConnectionState {
@@ -550,6 +574,7 @@ impl SmtpConnectionState {
             shutdown_receiver,
             stop_sender,
             idle_interrupt_sender,
+            connectivity: handlers.connection.connectivity.clone(),
         };
 
         let conn = SmtpConnectionState { state };
@@ -597,6 +622,7 @@ impl ImapConnectionState {
             shutdown_receiver,
             stop_sender,
             idle_interrupt_sender,
+            connectivity: handlers.connection.connectivity.clone(),
         };
 
         let conn = ImapConnectionState { state };

src/scheduler/connectivity.rs

@@ -0,0 +1,349 @@
+use core::fmt;
+use std::{ops::Deref, sync::Arc};
+
+use async_std::sync::{Mutex, RwLockReadGuard};
+
+use crate::events::EventType;
+use crate::{config::Config, scheduler::Scheduler};
+use crate::{context::Context, log::LogExt};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, EnumProperty, PartialOrd, Ord)]
+pub enum Connectivity {
+    NotConnected = 1000,
+    Connecting = 2000,
+    /// Fetching or sending messages
+    Working = 3000,
+    Connected = 4000,
+}
+
+// The order of the connectivities is important: worse connectivities (i.e. those at
+// the top) take priority. This means that e.g. if any folder has an error - usually
+// because there is no internet connection - the connectivity for the whole
+// account will be `Notconnected`.
+#[derive(Debug, Clone, PartialEq, Eq, EnumProperty)]
+enum DetailedConnectivity {
+    Error(String),
+    Uninitialized,
+    Connecting,
+    Working,
+    InterruptingIdle,
+    Connected,
+
+    /// The folder was configured not to be watched or configured_*_folder is not set
+    NotConfigured,
+}
+
+impl Default for DetailedConnectivity {
+    fn default() -> Self {
+        DetailedConnectivity::Uninitialized
+    }
+}
+
+impl DetailedConnectivity {
+    fn to_basic(&self) -> Option<Connectivity> {
+        match self {
+            DetailedConnectivity::Error(_) => Some(Connectivity::NotConnected),
+            DetailedConnectivity::Uninitialized => Some(Connectivity::NotConnected),
+            DetailedConnectivity::Connecting => Some(Connectivity::Connecting),
+            DetailedConnectivity::Working => Some(Connectivity::Working),
+            DetailedConnectivity::InterruptingIdle => Some(Connectivity::Connected),
+            DetailedConnectivity::Connected => Some(Connectivity::Connected),
+
+            // Just don't return a connectivity, probably the folder is configured not to be
+            // watched or there is e.g. no "Sent" folder, so we are not interested in it
+            DetailedConnectivity::NotConfigured => None,
+        }
+    }
+
+    fn to_string_imap(&self, _context: &Context) -> String {
+        match self {
+            DetailedConnectivity::Error(e) => format!("🔴 Error: {}", e),
+            DetailedConnectivity::Uninitialized => "🔴 Not started".to_string(),
+            DetailedConnectivity::Connecting => "🟡 Connecting…".to_string(),
+            DetailedConnectivity::Working => "⬇️ Getting new messages…".to_string(),
+            DetailedConnectivity::InterruptingIdle | DetailedConnectivity::Connected => {
+                "🟢 Connected".to_string()
+            }
+            DetailedConnectivity::NotConfigured => "🔴 Not configured".to_string(),
+        }
+    }
+
+    fn to_string_smtp(&self, _context: &Context) -> String {
+        match self {
+            DetailedConnectivity::Error(e) => format!("🔴 Error: {}", e),
+            DetailedConnectivity::Uninitialized => {
+                "(You did not try to send a message recently)".to_string()
+            }
+            DetailedConnectivity::Connecting => "🟡 Connecting…".to_string(),
+            DetailedConnectivity::Working => "⬆️ Sending…".to_string(),
+
+            // We don't know any more than that the last message was sent successfully;
+            // since sending the last message, connectivity could have changed, which we don't notice
+            // until another message is sent
+            DetailedConnectivity::InterruptingIdle | DetailedConnectivity::Connected => {
+                "🟢 Your last message was sent successfully".to_string()
+            }
+            DetailedConnectivity::NotConfigured => "🔴 Not configured".to_string(),
+        }
+    }
+
+    fn all_work_done(&self) -> bool {
+        match self {
+            DetailedConnectivity::Error(_) => true,
+            DetailedConnectivity::Uninitialized => false,
+            DetailedConnectivity::Connecting => false,
+            DetailedConnectivity::Working => false,
+            DetailedConnectivity::InterruptingIdle => false,
+            DetailedConnectivity::Connected => true,
+            DetailedConnectivity::NotConfigured => true,
+        }
+    }
+}
+
+#[derive(Clone, Default)]
+pub(crate) struct ConnectivityStore(Arc<Mutex<DetailedConnectivity>>);
+
+impl ConnectivityStore {
+    async fn set(&self, context: &Context, v: DetailedConnectivity) {
+        {
+            *self.0.lock().await = v;
+        }
+        context.emit_event(EventType::ConnectivityChanged);
+    }
+
+    pub(crate) async fn set_err(&self, context: &Context, e: impl ToString) {
+        self.set(context, DetailedConnectivity::Error(e.to_string()))
+            .await;
+    }
+    pub(crate) async fn set_connecting(&self, context: &Context) {
+        self.set(context, DetailedConnectivity::Connecting).await;
+    }
+    pub(crate) async fn set_working(&self, context: &Context) {
+        self.set(context, DetailedConnectivity::Working).await;
+    }
+    pub(crate) async fn set_connected(&self, context: &Context) {
+        self.set(context, DetailedConnectivity::Connected).await;
+    }
+    pub(crate) async fn set_not_configured(&self, context: &Context) {
+        self.set(context, DetailedConnectivity::NotConfigured).await;
+    }
+
+    async fn get_detailed(&self) -> DetailedConnectivity {
+        self.0.lock().await.deref().clone()
+    }
+    async fn get_basic(&self) -> Option<Connectivity> {
+        self.0.lock().await.to_basic()
+    }
+    async fn get_all_work_done(&self) -> bool {
+        self.0.lock().await.all_work_done()
+    }
+}
+
+/// Set all folder states to InterruptingIdle in case they were `Connected` before.
+/// Called during `dc_maybe_network()` to make sure that `dc_accounts_all_work_done()`
+/// returns false immediately after `dc_maybe_network()`.
+pub(crate) async fn idle_interrupted(scheduler: RwLockReadGuard<'_, Scheduler>) {
+    let [inbox, mvbox, sentbox] = match &*scheduler {
+        Scheduler::Running {
+            inbox,
+            mvbox,
+            sentbox,
+            ..
+        } => [
+            inbox.state.connectivity.clone(),
+            mvbox.state.connectivity.clone(),
+            sentbox.state.connectivity.clone(),
+        ],
+        Scheduler::Stopped => return,
+    };
+    drop(scheduler);
+
+    let mut connectivity_lock = inbox.0.lock().await;
+    // For the inbox, we also have to set the connectivity to InterruptingIdle if it was
+    // NotConfigured before: If all folders are NotConfigured, dc_get_connectivity()
+    // returns Connected. But after dc_maybe_network(), dc_get_connectivity() must not
+    // return Connected until DC is completely done with fetching folders; this also
+    // includes scan_folders() which happens on the inbox thread.
+    if *connectivity_lock == DetailedConnectivity::Connected
+        || *connectivity_lock == DetailedConnectivity::NotConfigured
+    {
+        *connectivity_lock = DetailedConnectivity::InterruptingIdle;
+    }
+    drop(connectivity_lock);
+
+    for state in &[&mvbox, &sentbox] {
+        let mut connectivity_lock = state.0.lock().await;
+        if *connectivity_lock == DetailedConnectivity::Connected {
+            *connectivity_lock = DetailedConnectivity::InterruptingIdle;
+        }
+    }
+    // No need to send ConnectivityChanged, the user-facing connectivity doesn't change because
+    // of what we do here.
+}
+
+impl fmt::Debug for ConnectivityStore {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        if let Some(guard) = self.0.try_lock() {
+            write!(f, "ConnectivityStore {:?}", &*guard)
+        } else {
+            write!(f, "ConnectivityStore [LOCKED]")
+        }
+    }
+}
+
+impl Context {
+    /// Get the current connectivity, i.e. whether the device is connected to the IMAP server.
+    /// One of:
+    /// - DC_CONNECTIVITY_NOT_CONNECTED (1000-1999): Show e.g. the string "Not connected" or a red dot
+    /// - DC_CONNECTIVITY_CONNECTING (2000-2999): Show e.g. the string "Connecting…" or a yellow dot
+    /// - DC_CONNECTIVITY_WORKING (3000-3999): Show e.g. the string "Getting new messages" or a spinning wheel
+    /// - DC_CONNECTIVITY_CONNECTED (>=4000): Show e.g. the string "Connected" or a green dot
+    ///
+    /// We don't use exact values but ranges here so that we can split up
+    /// states into multiple states in the future.
+    ///
+    /// Meant as a rough overview that can be shown
+    /// e.g. in the title of the main screen.
+    ///
+    /// If the connectivity changes, a DC_EVENT_CONNECTIVITY_CHANGED will be emitted.
+    pub async fn get_connectivity(&self) -> Connectivity {
+        let lock = self.scheduler.read().await;
+        let stores: Vec<_> = match &*lock {
+            Scheduler::Running {
+                inbox,
+                mvbox,
+                sentbox,
+                ..
+            } => [&inbox.state, &mvbox.state, &sentbox.state]
+                .iter()
+                .map(|state| state.connectivity.clone())
+                .collect(),
+            Scheduler::Stopped => return Connectivity::NotConnected,
+        };
+        drop(lock);
+
+        let mut connectivities = Vec::new();
+        for s in stores {
+            if let Some(connectivity) = s.get_basic().await {
+                connectivities.push(connectivity);
+            }
+        }
+        connectivities
+            .into_iter()
+            .min()
+            .unwrap_or(Connectivity::Connected)
+    }
+
+    /// Get an overview of the current connectivity, and possibly more statistics.
+    /// Meant to give the user more insight about the current status than
+    /// the basic connectivity info returned by dc_get_connectivity(); show this
+    /// e.g., if the user taps on said basic connectivity info.
+    ///
+    /// If this page changes, a DC_EVENT_CONNECTIVITY_CHANGED will be emitted.
+    ///
+    /// This comes as an HTML from the core so that we can easily improve it
+    /// and the improvement instantly reaches all UIs.
+    pub async fn get_connectivity_html(&self) -> String {
+        let mut ret =
+            "<!DOCTYPE html>\n<html><head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" /></head><body>\n".to_string();
+
+        let lock = self.scheduler.read().await;
+        let (folders_states, smtp) = match &*lock {
+            Scheduler::Running {
+                inbox,
+                mvbox,
+                sentbox,
+                smtp,
+                ..
+            } => (
+                [
+                    (
+                        Config::ConfiguredInboxFolder,
+                        Config::InboxWatch,
+                        inbox.state.connectivity.clone(),
+                    ),
+                    (
+                        Config::ConfiguredMvboxFolder,
+                        Config::MvboxWatch,
+                        mvbox.state.connectivity.clone(),
+                    ),
+                    (
+                        Config::ConfiguredSentboxFolder,
+                        Config::SentboxWatch,
+                        sentbox.state.connectivity.clone(),
+                    ),
+                ],
+                smtp.state.connectivity.clone(),
+            ),
+            Scheduler::Stopped => {
+                ret += "Not started</body></html>\n";
+                return ret;
+            }
+        };
+        drop(lock);
+
+        ret += "<div><h3>Incoming messages:</h3><ul>";
+        for (folder, watch, state) in &folders_states {
+            let w = self.get_config(*watch).await.ok_or_log(self);
+
+            let mut folder_added = false;
+            if w.flatten() == Some("1".to_string()) {
+                let f = self.get_config(*folder).await.ok_or_log(self).flatten();
+
+                if let Some(foldername) = f {
+                    ret += "<li><b>&quot;";
+                    ret += &foldername;
+                    ret += "&quot;:</b> ";
+                    ret += &state.get_detailed().await.to_string_imap(self);
+                    ret += "</li>";
+
+                    folder_added = true;
+                }
+            }
+
+            if !folder_added && folder == &Config::ConfiguredInboxFolder {
+                let detailed = &state.get_detailed().await;
+                if let DetailedConnectivity::Error(_) = detailed {
+                    // On the inbox thread, we also do some other things like scan_folders and run jobs
+                    // so, maybe, the inbox is not watched, but something else went wrong
+                    ret += "<li>";
+                    ret += &detailed.to_string_imap(self);
+                    ret += "</li>";
+                }
+            }
+        }
+        ret += "</ul></div>";
+
+        ret += "<h3>Outgoing messages:</h3><ul style=\"list-style-type: none;\"><li>";
+        ret += &smtp.get_detailed().await.to_string_smtp(self);
+        ret += "</li></ul>";
+
+        ret += "</body></html>\n";
+        ret
+    }
+
+    pub async fn all_work_done(&self) -> bool {
+        let lock = self.scheduler.read().await;
+        let stores: Vec<_> = match &*lock {
+            Scheduler::Running {
+                inbox,
+                mvbox,
+                sentbox,
+                smtp,
+                ..
+            } => [&inbox.state, &mvbox.state, &sentbox.state, &smtp.state]
+                .iter()
+                .map(|state| state.connectivity.clone())
+                .collect(),
+            Scheduler::Stopped => return false,
+        };
+        drop(lock);
+
+        for s in &stores {
+            if !s.get_all_work_done().await {
+                return false;
+            }
+        }
+        true
+    }
+}

src/smtp.rs

@@ -8,12 +8,11 @@ use async_smtp::smtp::client::net::ClientTlsParameters;
 use async_smtp::{error, smtp, EmailAddress};
 
 use crate::constants::DC_LP_AUTH_OAUTH2;
-use crate::context::Context;
 use crate::events::EventType;
 use crate::login_param::{dc_build_tls, CertificateChecks, LoginParam, ServerLoginParam};
 use crate::oauth2::dc_get_oauth2_access_token;
 use crate::provider::Socket;
-use crate::stock_str;
+use crate::{context::Context, scheduler::connectivity::ConnectivityStore};
 
 /// SMTP write and read timeout in seconds.
 const SMTP_TIMEOUT: u64 = 30;
@@ -28,11 +27,11 @@ pub enum Error {
         #[source]
         error: error::Error,
     },
-    #[error("SMTP: failed to connect: {0}")]
+    #[error("SMTP failed to connect: {0}")]
     ConnectionFailure(#[source] smtp::error::Error),
-    #[error("SMTP: failed to setup connection {0:?}")]
+    #[error("SMTP failed to setup connection: {0}")]
     ConnectionSetupFailure(#[source] smtp::error::Error),
-    #[error("SMTP: oauth2 error {address}")]
+    #[error("SMTP oauth2 error {address}")]
     Oauth2Error { address: String },
     #[error("TLS error {0}")]
     Tls(#[from] async_native_tls::Error),
@@ -53,6 +52,8 @@ pub(crate) struct Smtp {
     /// (eg connect or send succeeded). On initialization and disconnect
     /// it is set to None.
     last_success: Option<SystemTime>,
+
+    pub(crate) connectivity: ConnectivityStore,
 }
 
 impl Smtp {
@@ -97,6 +98,7 @@ impl Smtp {
             return Ok(());
         }
 
+        self.connectivity.set_connecting(context).await;
         let lp = LoginParam::from_database(context, "configured_").await?;
         let res = self
             .connect(
@@ -107,16 +109,10 @@ impl Smtp {
                 lp.provider.map_or(false, |provider| provider.strict_tls),
             )
             .await;
-        if let Err(ref err) = res {
-            let message = stock_str::server_response(
-                context,
-                format!("SMTP {}:{}", lp.smtp.server, lp.smtp.port),
-                err.to_string(),
-            )
-            .await;
 
-            context.emit_event(EventType::ErrorNetwork(message));
-        };
+        if let Err(err) = &res {
+            self.connectivity.set_err(context, err).await;
+        }
         res
     }
 

src/stock_str.rs

@@ -130,9 +130,6 @@ pub enum StockMessage {
     ))]
     CannotLogin = 60,
 
-    #[strum(props(fallback = "Could not connect to %1$s: %2$s"))]
-    ServerResponse = 61,
-
     #[strum(props(fallback = "%1$s by %2$s."))]
     MsgActionByUser = 62,
 
@@ -583,18 +580,6 @@ pub(crate) async fn cannot_login(context: &Context, user: impl AsRef<str>) -> St
         .replace1(user)
 }
 
-/// Stock string: `Could not connect to %1$s: %2$s`.
-pub(crate) async fn server_response(
-    context: &Context,
-    server: impl AsRef<str>,
-    details: impl AsRef<str>,
-) -> String {
-    translated(context, StockMessage::ServerResponse)
-        .await
-        .replace1(server)
-        .replace2(details)
-}
-
 /// Stock string: `%1$s by %2$s.`.
 pub(crate) async fn msg_action_by_user(
     context: &Context,
@@ -1000,10 +985,7 @@ mod tests {
     #[async_std::test]
     async fn test_stock_string_repl_str2() {
         let t = TestContext::new().await;
-        assert_eq!(
-            server_response(&t, "foo", "bar").await,
-            "Could not connect to foo: bar"
-        );
+        assert_eq!(msg_action_by_user(&t, "foo", "bar").await, "foo by bar.");
     }
 
     #[async_std::test]

src/test_utils.rs

@@ -631,7 +631,6 @@ fn receive_event(event: &Event) {
         EventType::SmtpMessageSent(msg) => format!("[SMTP_MESSAGE_SENT] {}", msg),
         EventType::Warning(msg) => format!("WARN: {}", yellow.paint(msg)),
         EventType::Error(msg) => format!("ERROR: {}", red.paint(msg)),
-        EventType::ErrorNetwork(msg) => format!("{}", red.paint(format!("[NETWORK] msg={}", msg))),
         EventType::ErrorSelfNotInGroup(msg) => {
             format!("{}", red.paint(format!("[SELF_NOT_IN_GROUP] {}", msg)))
         }