Merge branch 'feat/dynamic_buffer_tls1.3' into 'master'

feat(mbedtls): add support for dynamic buffer for TLS1.3

Closes IDFGH-14708, IDF-12469, IDF-9178, and IDF-1725

See merge request espressif/esp-idf!38258

components/mbedtls/CMakeLists.txt

@@ -351,6 +351,7 @@ if(CONFIG_MBEDTLS_DYNAMIC_BUFFER)
     set(WRAP_FUNCTIONS
         mbedtls_ssl_write_client_hello
         mbedtls_ssl_handshake_client_step
+        mbedtls_ssl_tls13_handshake_client_step
         mbedtls_ssl_handshake_server_step
         mbedtls_ssl_read
         mbedtls_ssl_write

components/mbedtls/Kconfig

@@ -172,10 +172,10 @@ menu "mbedTLS"
         default 4 if MBEDTLS_DEBUG_LEVEL_VERBOSE
 
     menu "mbedTLS v3.x related"
-        # NOTE: MBEDTLS_DYNAMIC_BUFFER feature is not supported with TLS 1.3 yet. Ref: IDF-4762
         config MBEDTLS_SSL_PROTO_TLS1_3
             bool "Support TLS 1.3 protocol"
-            depends on MBEDTLS_TLS_ENABLED && MBEDTLS_SSL_KEEP_PEER_CERTIFICATE && !MBEDTLS_DYNAMIC_BUFFER
+            depends on MBEDTLS_TLS_ENABLED && MBEDTLS_SSL_KEEP_PEER_CERTIFICATE
+            select MBEDTLS_CLIENT_SSL_SESSION_TICKETS if MBEDTLS_DYNAMIC_BUFFER
             select MBEDTLS_HKDF_C
             default n
 

components/mbedtls/port/dynamic/dynamic_buffer_architecture.md

@@ -0,0 +1,221 @@
+# Dynamic Buffer Management in mbedTLS for ESP-IDF
+
+## Executive Summary
+
+ESP-IDF implements a dynamic buffer management system for mbedTLS to optimize memory usage during TLS/SSL connections. This architecture significantly reduces RAM requirements on memory-constrained ESP devices by intelligently allocating buffers only when needed and sizing them according to actual message requirements rather than worst-case scenarios.
+
+## The Problem We're Solving
+
+Standard TLS implementations allocate large static buffers that:
+- Reserve memory for the entire connection lifetime
+- Are sized for worst-case scenarios (large certificates, messages)
+- Remain allocated even when not in use
+
+On memory-constrained IoT devices like ESP32, this traditional approach is inefficient and limits the number of concurrent connections possible.
+
+## Our Solution: The Dynamic Buffer Approach
+
+Instead of static allocation, our system:
+1. **Allocates buffers only when needed**
+2. **Right-sizes buffers** based on actual message requirements
+3. **Releases memory** when it's not needed
+4. **Preserves critical state** in small cache buffers
+
+## How It Works: A Conceptual View
+
+### Buffer Lifecycle
+
+1. **Starting state**: Begin with minimal or no buffer allocation
+2. **Just before data transmission/reception**: Allocate right-sized buffer
+3. **During data processing**: Use the allocated buffer
+4. **After processing**: Replace large buffer with small cache buffer
+5. **Repeat** as needed during the connection
+
+### Key Concepts Illustrated
+
+#### Transmission (TX) Buffer Handling
+
+```
+[Small idle buffer] → [Right-sized TX buffer] → [Back to small buffer]
+```
+
+#### Reception (RX) Buffer Handling
+
+```
+[No buffer] → [Header buffer] → [Right-sized RX buffer] → [Small cache buffer]
+```
+
+### Implementation Strategy
+
+Our implementation follows these steps for each operation:
+
+1. **Before operation**: Check if we need to allocate or resize a buffer
+2. **During operation**: Use the standard mbedTLS functions with our buffers
+3. **After operation**: Shrink or release buffers that are no longer needed
+
+## Core Components
+
+### 1. Custom Buffer Structure
+
+We use a custom buffer structure that includes metadata:
+
+```c
+struct esp_mbedtls_ssl_buf {
+    esp_mbedtls_ssl_buf_states state;  // CACHED or NOT_CACHED
+    unsigned int len;                  // Buffer size
+    unsigned char buf[0];              // Flexible array for actual data
+};
+```
+
+This structure allows us to:
+
+- Track whether a buffer contains important state
+- Store the buffer's size for dynamic resizing
+- Use a flexible array member for efficient memory layout
+
+### 2. Buffer States
+
+- **CACHED**: Contains important cryptographic state that must be preserved
+- **NOT_CACHED**: Can be safely replaced or released
+
+The state tracking is critical for maintaining TLS protocol security while optimizing memory.
+
+### 3. Critical State Preservation
+
+When replacing large buffers with small cache buffers, we preserve:
+
+- **SSL counter values**: Used for replay protection
+- **Initialization vectors**: Required for encryption/decryption
+
+These small amounts of cryptographic state must be maintained between operations to keep the TLS connection secure.
+
+### 4. Memory Management Functions
+
+| Function | Purpose |
+|----------|---------|
+| `esp_mbedtls_add_tx_buffer()` | Allocates a transmission buffer sized for the outgoing message |
+| `esp_mbedtls_free_tx_buffer()` | Replaces TX buffer with small cache buffer after transmission |
+| `esp_mbedtls_add_rx_buffer()` | Reads record header and allocates right-sized reception buffer |
+| `esp_mbedtls_free_rx_buffer()` | Replaces RX buffer with small cache buffer after processing |
+
+### 5. Function Wrapping
+
+We intercept key mbedTLS functions using GCC's function wrapping:
+
+```c
+int __wrap_mbedtls_ssl_read(mbedtls_ssl_context *ssl, unsigned char *buf, size_t len) {
+    // 1. Allocate right-sized buffer
+    // 2. Call original function
+    // 3. Free buffer when done
+}
+```
+
+This allows seamless integration without modifying the mbedTLS source code.
+
+## The Handshake Process Design
+
+During TLS handshaking, memory needs change dramatically between steps. Our system tracks the handshake state and manages memory accordingly:
+
+### Client-Side Handshake Memory Management
+
+| Handshake Step | Memory Action |
+|----------------|---------------|
+| Client Hello | Allocate TX buffer |
+| Server Hello | Allocate RX buffer |
+| Server Certificate | Allocate RX buffer |
+| Certificate Verify | Allocate TX buffer |
+| Free certificate resources | Release CA certificates |
+| Client Key Exchange | Allocate TX buffer |
+| Change Cipher Spec | Small buffer |
+| Finished | Small buffers |
+
+
+
+## Implementation Design: SSL Read Operation
+
+The dynamic buffer allocation for SSL read operations follows this design:
+
+1. First, read just the TLS record header:
+
+```c
+// Read just the header to determine full message size
+ssl->in_hdr = msg_head;
+ssl->in_len = msg_head + 3;
+mbedtls_ssl_fetch_input(ssl, mbedtls_ssl_in_hdr_len(ssl));
+
+// Parse header to get message length
+esp_mbedtls_parse_record_header(ssl);
+in_msglen = ssl->in_msglen;
+```
+
+2. Once we know the exact message size, allocate a buffer that's precisely sized:
+
+```c
+// Allocate buffer of right size
+buffer_len = in_msglen + overhead; // Add necessary TLS overhead
+esp_buf = mbedtls_calloc(1, SSL_BUF_HEAD_OFFSET_SIZE + buffer_len);
+
+// Initialize and set up buffer
+esp_mbedtls_init_ssl_buf(esp_buf, buffer_len);
+init_rx_buffer(ssl, esp_buf->buf);
+```
+
+3. After processing, preserve critical state and free the large buffer:
+
+```c
+// Save critical state (counters and IVs)
+memcpy(buf, ssl->in_ctr, 8);
+memcpy(buf + 8, ssl->in_iv, 8);
+
+// Free large buffer
+esp_mbedtls_free_buf(ssl->in_buf);
+
+// Allocate small cache buffer
+esp_buf = mbedtls_calloc(1, SSL_BUF_HEAD_OFFSET_SIZE + 16);
+esp_mbedtls_init_ssl_buf(esp_buf, 16);
+
+// Restore critical state in small buffer
+memcpy(esp_buf->buf, buf, 16);
+```
+
+## Configuration Options
+
+The dynamic buffer system includes configurable options:
+
+- `CONFIG_MBEDTLS_DYNAMIC_FREE_CA_CERT`: Free CA certificates after verification
+- `CONFIG_MBEDTLS_DYNAMIC_FREE_CONFIG_DATA`: Free DHM parameters and key material when no longer needed
+
+These can be enabled in ESP-IDF's menuconfig system.
+
+## Integration Architecture
+
+The implementation uses function wrapping to seamlessly integrate with mbedTLS:
+
+```
+Application → mbedTLS API → Our Wrapper Functions → Original mbedTLS Functions
+```
+
+Key wrapped functions include:
+- `mbedtls_ssl_setup`: Initialize with minimal buffers
+- `mbedtls_ssl_read`/`mbedtls_ssl_write`: Dynamic buffer management during I/O 
+- `mbedtls_ssl_handshake_client_step`: Handshake-aware memory management
+- `mbedtls_ssl_free`: Clean up all allocated memory
+
+## Design Benefits
+
+The dynamic buffer management design provides several key benefits:
+
+1. **Memory Efficiency**: Significantly reduced peak memory usage
+2. **Scalability**: Adapts to different TLS message sizes dynamically
+3. **Transparency**: Application code doesn't need to be aware of the memory optimization
+4. **Compatibility**: Maintains full mbedTLS functionality
+
+## Summary
+
+The dynamic buffer management system in ESP-IDF's mbedTLS port follows a sophisticated architecture that:
+1. Allocates only what's needed, when it's needed
+2. Preserves critical state in small buffers
+3. Is aware of the TLS handshake flow
+4. Releases memory as soon as it's no longer required
+
+This architecture enables more efficient use of RAM on memory-constrained devices while maintaining the security guarantees of the TLS protocol.

components/mbedtls/port/dynamic/esp_mbedtls_dynamic_impl.c

@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: 2020-2024 Espressif Systems (Shanghai) CO LTD
+ * SPDX-FileCopyrightText: 2020-2025 Espressif Systems (Shanghai) CO LTD
  *
  * SPDX-License-Identifier: Apache-2.0
  */
@@ -352,6 +352,8 @@ int esp_mbedtls_add_rx_buffer(mbedtls_ssl_context *ssl)
             ESP_LOGD(TAG, "mbedtls_ssl_fetch_input reads data times out");
         } else if (ret == MBEDTLS_ERR_SSL_WANT_READ) {
             ESP_LOGD(TAG, "mbedtls_ssl_fetch_input wants to read more data");
+        } else if (ret == MBEDTLS_ERR_SSL_CONN_EOF) {
+            ESP_LOGD(TAG, "mbedtls_ssl_fetch_input connection EOF");
         } else {
             ESP_LOGE(TAG, "mbedtls_ssl_fetch_input error=%d", -ret);
         }

components/mbedtls/port/dynamic/esp_ssl_cli.c

@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: 2020-2022 Espressif Systems (Shanghai) CO LTD
+ * SPDX-FileCopyrightText: 2020-2025 Espressif Systems (Shanghai) CO LTD
  *
  * SPDX-License-Identifier: Apache-2.0
  */
@@ -9,9 +9,11 @@
 
 int __real_mbedtls_ssl_handshake_client_step(mbedtls_ssl_context *ssl);
 int __real_mbedtls_ssl_write_client_hello(mbedtls_ssl_context *ssl);
+int __real_mbedtls_ssl_tls13_handshake_client_step(mbedtls_ssl_context *ssl);
 
 int __wrap_mbedtls_ssl_handshake_client_step(mbedtls_ssl_context *ssl);
 int __wrap_mbedtls_ssl_write_client_hello(mbedtls_ssl_context *ssl);
+int __wrap_mbedtls_ssl_tls13_handshake_client_step(mbedtls_ssl_context *ssl);
 
 static const char *TAG = "SSL client";
 
@@ -52,6 +54,15 @@ static int manage_resource(mbedtls_ssl_context *ssl, bool add)
 
 
         case MBEDTLS_SSL_SERVER_HELLO:
+            if (add) {
+                CHECK_OK(esp_mbedtls_add_rx_buffer(ssl));
+            } else {
+                if (!ssl->MBEDTLS_PRIVATE(keep_current_message)) {
+                    CHECK_OK(esp_mbedtls_free_rx_buffer(ssl));
+                }
+            }
+            break;
+        case MBEDTLS_SSL_ENCRYPTED_EXTENSIONS:
             if (add) {
                 CHECK_OK(esp_mbedtls_add_rx_buffer(ssl));
             } else {
@@ -121,6 +132,13 @@ static int manage_resource(mbedtls_ssl_context *ssl, bool add)
                 CHECK_OK(esp_mbedtls_add_tx_buffer(ssl, buffer_len));
             }
             break;
+        case MBEDTLS_SSL_CLIENT_CERTIFICATE_VERIFY:
+            if (add) {
+                size_t buffer_len = MBEDTLS_SSL_OUT_BUFFER_LEN;
+
+                CHECK_OK(esp_mbedtls_add_tx_buffer(ssl, buffer_len));
+            }
+            break;
         case MBEDTLS_SSL_CLIENT_KEY_EXCHANGE:
             if (add) {
                 size_t buffer_len = MBEDTLS_SSL_OUT_BUFFER_LEN;
@@ -191,6 +209,39 @@ static int manage_resource(mbedtls_ssl_context *ssl, bool add)
             }
 #endif
             break;
+#if defined(MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE)
+        case MBEDTLS_SSL_CLIENT_CCS_BEFORE_2ND_CLIENT_HELLO:
+            if (add) {
+                CHECK_OK(esp_mbedtls_add_tx_buffer(ssl, MBEDTLS_SSL_OUT_BUFFER_LEN));
+            }
+            break;
+        case MBEDTLS_SSL_CLIENT_CCS_AFTER_SERVER_FINISHED:
+            if (add) {
+                CHECK_OK(esp_mbedtls_add_tx_buffer(ssl, MBEDTLS_SSL_OUT_BUFFER_LEN));
+            }
+            break;
+#if defined(MBEDTLS_SSL_EARLY_DATA)
+        case MBEDTLS_SSL_CLIENT_CCS_AFTER_CLIENT_HELLO:
+            if (add) {
+                CHECK_OK(esp_mbedtls_add_tx_buffer(ssl, MBEDTLS_SSL_OUT_BUFFER_LEN));
+            }
+            break;
+        case MBEDTLS_SSL_END_OF_EARLY_DATA:
+            size_t buffer_len = MBEDTLS_SSL_OUT_BUFFER_LEN;
+
+            CHECK_OK(esp_mbedtls_add_tx_buffer(ssl, buffer_len));
+            break;
+#endif /* MBEDTLS_SSL_EARLY_DATA */
+#endif /* MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE */
+#if defined(MBEDTLS_SSL_SESSION_TICKETS)
+        case MBEDTLS_SSL_TLS1_3_NEW_SESSION_TICKET:
+            if (add) {
+                CHECK_OK(esp_mbedtls_add_rx_buffer(ssl));
+            } else {
+                CHECK_OK(esp_mbedtls_free_rx_buffer(ssl));
+            }
+            break;
+#endif /* MBEDTLS_SSL_SESSION_TICKETS */
         default:
             break;
     }
@@ -209,6 +260,17 @@ int __wrap_mbedtls_ssl_handshake_client_step(mbedtls_ssl_context *ssl)
     return 0;
 }
 
+int __wrap_mbedtls_ssl_tls13_handshake_client_step(mbedtls_ssl_context *ssl)
+{
+    CHECK_OK(manage_resource(ssl, true));
+
+    CHECK_OK(__real_mbedtls_ssl_tls13_handshake_client_step(ssl));
+
+    CHECK_OK(manage_resource(ssl, false));
+
+    return 0;
+}
+
 int __wrap_mbedtls_ssl_write_client_hello(mbedtls_ssl_context *ssl)
 {
     CHECK_OK(manage_resource(ssl, true));

components/mbedtls/port/dynamic/esp_ssl_tls.c

@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: 2020-2024 Espressif Systems (Shanghai) CO LTD
+ * SPDX-FileCopyrightText: 2020-2025 Espressif Systems (Shanghai) CO LTD
  *
  * SPDX-License-Identifier: Apache-2.0
  */
@@ -347,6 +347,43 @@ int __wrap_mbedtls_ssl_read(mbedtls_ssl_context *ssl, unsigned char *buf, size_t
 
     ret = __real_mbedtls_ssl_read(ssl, buf, len);
 
+#if CONFIG_MBEDTLS_SSL_PROTO_TLS1_3
+        /*
+         * As per RFC 8446, section 4.6.1 the server may send a NewSessionTicket message at any time after the
+         * client Finished message.
+         * If a post-handshake message is received, connection state is changed to `MBEDTLS_SSL_TLS1_3_NEW_SESSION_TICKET`
+         * and when the message is parsed, the return value is `MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET`.
+         * When the session ticket is parsed, reduce the ssl->in_msglen by the length of the
+         * NewSessionTicket message.
+        */
+        if (mbedtls_ssl_get_version_number(ssl) == MBEDTLS_SSL_VERSION_TLS1_3) {
+            if (ret == MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET) {
+                ESP_LOGD(TAG, "got session ticket in TLS 1.3 connection, retry read");
+
+                /* At this stage, we have received a NewSessionTicket messages
+                 * We should decrement the ssl->in_msglen by the length of the
+                 * NewSessionTicket message.
+                 *
+                 * The NewSessionTicket message has been parsed internally by mbedTLS
+                 * and it is stored in the mbedTLS context. This msglen size update
+                 * is also handled by mbedTLS but in case of dynamic buffer,
+                 * we need to free the rx buffer if it is allocated
+                 * and prepare for the next read. So we have to update the msglen
+                 * by ourselves and free the rx buffer if no more data is available.
+                 */
+                if (ssl->MBEDTLS_PRIVATE(in_hslen) < ssl->MBEDTLS_PRIVATE(in_msglen)) {
+                    ssl->MBEDTLS_PRIVATE(in_msglen) -= ssl->MBEDTLS_PRIVATE(in_hslen);
+                    memmove(ssl->MBEDTLS_PRIVATE(in_msg), ssl->MBEDTLS_PRIVATE(in_msg) + ssl->MBEDTLS_PRIVATE(in_hslen),
+                        ssl->MBEDTLS_PRIVATE(in_msglen));
+                    MBEDTLS_PUT_UINT16_BE(ssl->MBEDTLS_PRIVATE(in_msglen), ssl->in_len, 0);
+                } else {
+                    ssl->MBEDTLS_PRIVATE(in_msglen) = 0;
+                }
+                ssl->MBEDTLS_PRIVATE(in_hslen) = 0;
+            }
+        }
+#endif // CONFIG_MBEDTLS_SSL_PROTO_TLS1_3
+
     if (rx_done(ssl)) {
         CHECK_OK(esp_mbedtls_free_rx_buffer(ssl));
     }