mirror of
https://github.com/espressif/esp-idf.git
synced 2026-10-02 11:10:54 +03:00
docs(nvs_flash): improved example of NVS fragmentation
This commit is contained in:
radek.tandler
committed by
Zhang Shuxian
Zhang Shuxian
parent
56a01ca822
commit
c6c0734a7b
@@ -20,39 +20,54 @@ Detailed functional description of NVS and API is provided in [documentation](ht
|
||||
|
||||
## Blob Storage-Overhead Measurement
|
||||
|
||||
In addition to the basic statistics demonstration, the example can measure how much usable storage a blob actually consumes, taking the NVS metadata and free-space fragmentation into account. This part is enabled by default and can be turned off via `idf.py menuconfig` → *Example Configuration* → *Run NVS blob storage-overhead measurement*.
|
||||
In addition to the basic statistics demonstration, the example can measure how many entries a single blob actually consumes compared to its ideal (non-fragmented) cost, taking the NVS metadata and free-space fragmentation into account. This part is disabled by default and can be turned on via `idf.py menuconfig` → *Example Configuration* → *Run NVS blob storage-overhead measurement*.
|
||||
|
||||
The measurement sweeps a matrix of **partition sizes** × **blob sizes** and, for each cell, fills the partition to capacity and reports the heap demand, NVS entry usage and the resulting storage overhead.
|
||||
For every measured cell the example erases and initializes a dedicated partition, pre-populates its pages with strings so that a target number of free entries remains on each page, writes a single blob and compares its real entry footprint against the ideal one. The results are printed as a single table.
|
||||
|
||||
### Variable partition size
|
||||
### Configurable sweep
|
||||
|
||||
The example uses a custom partition table (`partitions.csv`) that defines several NVS partitions of different sizes (`nvs_16k`, `nvs_32k`, `nvs_64k`). The measurement iterates over them using `nvs_flash_init_partition()` / `nvs_get_stats()`, so the influence of the partition size on the relative overhead becomes directly visible.
|
||||
The measurement iterates over two configurable arrays defined at the top of `nvs_statistics_example.c`:
|
||||
|
||||
### Variable blob size
|
||||
* `measurement_combos[]` — partition size / blob size combinations. The default set sweeps three partition sizes and, for the 16 kB partition, a few blob sizes that straddle the point where the blob no longer fits: `16k / 128 B`, `16k / 350 B`, `16k / 416 B`, `16k / 417 B`, `32k / 768 B`, `32k / 920 B` and `64k / 1612 B`. The partition names refer to the NVS partitions declared in the custom partition table (`partitions.csv`).
|
||||
* `page_free_ranges[]` — per-page free-entry targets, each a `{ first_page_free, rest_free }` pair given as an absolute number of free (available) entries that must remain on a page (out of 126 per page). The default set is `{14, 11}`, `{14, 7}`, `{14, 4}` and `{13, 4}`.
|
||||
|
||||
For each partition, blobs of increasing size (128, 256, 512, 1024, 2048 and 4096 bytes) are stored with unique keys until `nvs_set_blob()` returns `ESP_ERR_NVS_NOT_ENOUGH_SPACE`. The number of stored blobs is then compared against the theoretical (ideal) count derived from the documented per-blob entry cost:
|
||||
Every combination is measured at every free-entry target.
|
||||
|
||||
```
|
||||
entries_per_blob = 1 (BLOB_INDEX) + k (per-page BLOB_DATA chunk headers) + ceil(blob_size / 32)
|
||||
```
|
||||
|
||||
where `k` is the number of pages the blob data is split across.
|
||||
|
||||
### Worst-case fragmentation
|
||||
### Per-page pre-population
|
||||
|
||||
By design, each NVS page is a 4096-byte flash sector holding 126 usable 32-byte entries. A string occupies `1 + ceil((len + 1) / 32)` entries and must fit contiguously within a single page, while a blob may split its data into per-page chunks.
|
||||
|
||||
To demonstrate the worst case, the example optionally pre-populates a partition so that **every page is filled up to its last 2 entries** (enabled via *Add a worst-case (pre-fragmented) measurement pass*). This is achieved by writing large strings:
|
||||
Each free-entry target leaves `first_page_free` free entries on the first page and `rest_free` on every remaining page (the namespace entries on page 0 count towards page 0's occupancy). To leave the requested number of free entries on a page, the example writes a "keep" string that occupies all but the requested free entries, followed by a "filler" string that occupies those remaining entries. Once the whole partition has been filled this way, all filler strings are erased. Every page therefore exposes exactly the requested number of reclaimable free entries, scattering the free space into per-page gaps — the fragmentation pattern the subsequent blob write has to cope with.
|
||||
|
||||
* the first string is sized to leave 2 free entries on page 0 while accounting for the namespace entry,
|
||||
* every following string fills a fresh page to 124 entries, leaving exactly 2 free.
|
||||
### Reported columns
|
||||
|
||||
After this step the partition reports a large amount of `free_entries`, but the largest contiguous run of free entries on any page is only 2. The consequences are then measured by filling the partition with blobs:
|
||||
A single blob is then written into the pre-populated partition and its consumption is tabulated:
|
||||
|
||||
* each blob chunk can only use 1 chunk-header + 1 data entry per page, so roughly half of the consumed space becomes metadata overhead,
|
||||
* because the number of chunks per blob is bounded, large blobs may become unstoreable even though many `free_entries` remain.
|
||||
| Column | Meaning |
|
||||
| ------ | ------- |
|
||||
| `Blob Size [B]` | Blob payload size. |
|
||||
| `Partition Size [k]` | Partition size in kibibytes. |
|
||||
| `Free Entries per Page` → `First [-]` | Number of free (available) entries left on the first NVS page. |
|
||||
| `Free Entries per Page` → `Remaining [-]` | Number of free (available) entries left on every remaining NVS page. |
|
||||
| `Available Entries [-]` | `available_entries` from `nvs_get_stats()` right before the blob is written. |
|
||||
| `Expected Entries [-]` | Ideal (non-fragmented) blob cost: `1 (BLOB_INDEX) + chunks + ceil(blob_size / 32)`. |
|
||||
| `Actual Entries [-]` | Real entry consumption of the written blob (`used_entries` delta). |
|
||||
| `Overhead Entries [-]` | `Actual Entries − Expected Entries`. |
|
||||
| `Overhead [%]` | `100 × Overhead Entries / Expected Entries`. |
|
||||
|
||||
This makes the relationship between fragmentation, remaining free space and resulting overhead measurable, instead of presenting a single (best-case) number that could create false expectations.
|
||||
If the blob does not fit into the pre-populated partition, `nvs_set_blob()` returns `ESP_ERR_NVS_NOT_ENOUGH_SPACE` and the corresponding row reports `FAIL` for the actual consumption and the derived overhead columns.
|
||||
|
||||
### Why a blob may not fit even when `Available Entries` looks sufficient
|
||||
|
||||
`Available Entries` is a *partition-wide* total of free, non-reserved entries. It is tempting to compare it against `Expected Entries` and conclude that any blob whose ideal cost is smaller must fit — but a fragmented partition does not work that way, which is exactly what this measurement demonstrates. Several NVS design properties make the real requirement larger, and the usable free space smaller, than that single number suggests:
|
||||
|
||||
* **A blob is stored as one data chunk per page, not as one contiguous run.** When the free space is scattered (as the per-page pre-population deliberately arranges it), NVS splits the blob so that each page receives at most one `BLOB_DATA` chunk. The chunks are sized to whatever run of free entries a page can offer, and the split continues page by page until the whole payload is stored.
|
||||
* **Every chunk carries its own metadata entry.** Each `BLOB_DATA` chunk costs one header entry on top of its payload entries, and the blob as a whole costs one `BLOB_INDEX` entry. The more the free space is fragmented, the more chunks are needed and the more of these header entries are spent — this is precisely the difference between `Expected Entries` (ideal, minimum chunking) and `Actual Entries` (real, fragmented chunking), and it can push `Overhead [%]` well above 100 %.
|
||||
* **A page needs at least two free entries to hold any chunk** (one header entry plus at least one payload entry). Free space that survives only as isolated single-entry gaps still counts towards `Available Entries`, yet no blob chunk can ever be placed there, so that space is effectively unusable for the blob.
|
||||
* **The *first* chunk needs a minimum starting free run.** Unless the whole blob fits into the current page, NVS refuses to place the very first `BLOB_DATA` chunk on a page whose free var-data run is smaller than `CHUNK_MAX_SIZE / 10`. With the default `CHUNK_MAX_SIZE = ENTRY_SIZE × (ENTRY_COUNT − 1) = 32 × 125 = 4000` bytes, this threshold is `400` bytes — roughly 13 entries. Such a page is marked full and NVS looks for a page offering a larger free run; if none exists, the write fails with `ESP_ERR_NVS_NOT_ENOUGH_SPACE` *before any chunk is written*. A partition whose free space survives only as many small per-page gaps (for example the `rest_free = 4` target, i.e. ≈ 96 bytes of free var-data per page) therefore offers no page able to *start* a multi-page blob — even though those same gaps could still host the blob's *subsequent* chunks, and even though every one of their entries is counted in `Available Entries`.
|
||||
* **One page is always held in reserve** for the power-loss-safe space-reclaim algorithm, and space reclaim only consolidates free entries within a single candidate page per call. There is no operation that gathers scattered free entries from many pages into one large contiguous run for a single write.
|
||||
|
||||
Put together, these effects mean the blob's real footprint (`Actual Entries`) can be far larger than its ideal footprint (`Expected Entries`), while the portion of `Available Entries` that a single blob can actually reach is smaller than the raw figure. When the fragmented free space can no longer absorb the chunked-and-metadata-inflated blob, the write fails with `ESP_ERR_NVS_NOT_ENOUGH_SPACE` — even though `Available Entries` on its own looked large enough. In the table below this shows up as rows where `Available Entries` ≥ `Expected Entries` yet `Actual Entries` still reads `FAIL`.
|
||||
|
||||
## How to use example
|
||||
|
||||
@@ -92,44 +107,35 @@ I (565) nvs_statistics_example: Newly used entries match expectation.
|
||||
...
|
||||
```
|
||||
|
||||
The second part reports the blob storage overhead for each partition / blob size, in both a pristine and a pre-fragmented partition:
|
||||
The second part reports, for every partition size / blob size combination and every per-page free-entry target, the ideal vs. real blob entry consumption in a single table:
|
||||
|
||||
```
|
||||
...
|
||||
I (1265) nvs_statistics_example: Starting NVS blob storage-overhead measurement...
|
||||
|
||||
NVS BLOB TEST - 128 B (partition 'nvs_16k', pristine):
|
||||
======================
|
||||
heap before NVS init: 299220 B
|
||||
heap after NVS init: 255472 B (diff 43748 B)
|
||||
|
||||
available heap after fill: 254100 B (diff 1372 B)
|
||||
expected blobs count: 21
|
||||
stored blobs count: 20
|
||||
used_entries: 120 (3840 B)
|
||||
free_entries: 6 (192 B)
|
||||
total_entries: 126 (4032 B)
|
||||
STORAGE OVERHEAD: 36%
|
||||
|
||||
NVS BLOB TEST - 128 B (partition 'nvs_16k', fragmented):
|
||||
======================
|
||||
heap before NVS init: 299220 B
|
||||
heap after NVS init: 255472 B (diff 43748 B)
|
||||
fragmentation strings written: 3
|
||||
|
||||
available heap after fill: 254100 B (diff 1372 B)
|
||||
expected blobs count: 21
|
||||
stored blobs count: 2
|
||||
used_entries: 124 (3968 B)
|
||||
free_entries: 2 (64 B)
|
||||
total_entries: 126 (4032 B)
|
||||
STORAGE OVERHEAD: 98%
|
||||
+---------------+--------------------+---------------------------+-----------------------+----------------------+--------------------+----------------------+--------------+
|
||||
| Blob Size | Partition Size | Free Entries per Page | Available Entries | Expected Entries | Actual Entries | Overhead Entries | Overhead |
|
||||
| | +-----------+---------------+ | | | | |
|
||||
| [B] | [k] | First [-] | Remaining [-] | [-] | [-] | [-] | [-] | [%] |
|
||||
+---------------+--------------------+-----------+---------------+-----------------------+----------------------+--------------------+----------------------+--------------+
|
||||
| 128 | 16 | 14 | 11 | 39 | 6 | 6 | 0 | 0.0 |
|
||||
| 128 | 16 | 14 | 4 | 18 | 6 | 6 | 0 | 0.0 |
|
||||
| 350 | 16 | 14 | 7 | 28 | 13 | 15 | 2 | 15.4 |
|
||||
| 416 | 16 | 14 | 4 | 18 | 15 | 18 | 3 | 20.0 |
|
||||
| 417 | 16 | 14 | 4 | 18 | 16 | FAIL | - | - |
|
||||
| 768 | 32 | 14 | 7 | 56 | 26 | 30 | 4 | 15.4 |
|
||||
| 1612 | 64 | 14 | 4 | 60 | 53 | 71 | 18 | 34.0 |
|
||||
...
|
||||
+---------------+--------------------+-----------+---------------+-----------------------+----------------------+--------------------+----------------------+--------------+
|
||||
|
||||
I (9000) nvs_statistics_example: NVS blob storage-overhead measurement done.
|
||||
I (9010) nvs_statistics_example: Returning from app_main().
|
||||
...
|
||||
```
|
||||
|
||||
> The exact numbers depend on the target, partition size and blob size; the values above are illustrative. The key takeaway is the difference in `STORAGE OVERHEAD` and `stored blobs count` between the pristine and fragmented passes.
|
||||
> The exact numbers depend on the target, partition size, blob size and per-page free-entry target; the values above are illustrative. Note the following patterns:
|
||||
>
|
||||
> * `Overhead [%]` grows as fewer free entries are left per page (each additional chunk adds one metadata entry), so the same blob costs more on a heavily fragmented partition than on a lightly fragmented one.
|
||||
> * The neighbouring `416 B` and `417 B` rows straddle the fit boundary. With only 4 free entries (≈ 96 bytes) left on every page except the first, the first page — with 14 free entries (≈ 416 bytes of free var-data) — is the only one whose free run clears the ~400-byte minimum-first-chunk threshold, so it is the only page on which a blob can start. The `416 B` blob fits entirely into that first chunk (consuming all 18 available entries), whereas a single extra payload byte in the `417 B` blob no longer fits and there is no further usable space, so the write reports `FAIL` — even though its ideal `Expected Entries` (16) is below the reported `Available Entries` (18). This is the "seems like enough space, but the write still fails" case explained above.
|
||||
|
||||
To reset NVS data, erase the contents of flash memory using `idf.py erase-flash`, then upload the program again as described above.
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
idf_component_register(SRCS "nvs_statistics_example.c"
|
||||
PRIV_REQUIRES nvs_flash
|
||||
PRIV_REQUIRES nvs_flash esp_partition
|
||||
INCLUDE_DIRS ".")
|
||||
|
||||
@@ -2,20 +2,12 @@ menu "Example Configuration"
|
||||
|
||||
config EXAMPLE_RUN_OVERHEAD_MEASUREMENT
|
||||
bool "Run NVS blob storage-overhead measurement"
|
||||
default y
|
||||
default n
|
||||
help
|
||||
When enabled, the example sweeps several NVS partition sizes and blob
|
||||
sizes, fills each partition to capacity with blobs and reports the
|
||||
resulting heap demand, entry usage and storage overhead.
|
||||
|
||||
config EXAMPLE_INDUCE_FRAGMENTATION
|
||||
bool "Add a worst-case (pre-fragmented) measurement pass"
|
||||
depends on EXAMPLE_RUN_OVERHEAD_MEASUREMENT
|
||||
default y
|
||||
help
|
||||
When enabled, every measured partition is first pre-populated with
|
||||
large strings that leave only two free entries per NVS page. This
|
||||
scatters the free space into non-coalescable gaps and demonstrates
|
||||
the worst-case blob storage overhead caused by fragmentation.
|
||||
When enabled, the example sweeps several partition size / blob size
|
||||
combinations. For each combination it pre-populates every NVS page
|
||||
with strings up to a set of population levels, writes a single blob
|
||||
and reports its ideal vs. real entry consumption (and the resulting
|
||||
overhead) in a table.
|
||||
|
||||
endmenu
|
||||
|
||||
@@ -22,10 +22,10 @@
|
||||
#include "freertos/FreeRTOS.h"
|
||||
#include "freertos/task.h"
|
||||
#include "esp_check.h"
|
||||
#include "esp_system.h"
|
||||
#include "esp_log.h"
|
||||
#include "nvs_flash.h"
|
||||
#include "nvs.h"
|
||||
#include "esp_partition.h"
|
||||
|
||||
#define MOCK_DATA_NAMESPACE "_mock_data"
|
||||
#define MOCK_DATA_BACKUP_NAMESPACE "_mock_backup"
|
||||
@@ -38,16 +38,12 @@
|
||||
#define NVS_ENTRIES_PER_PAGE 126
|
||||
#define NVS_PAGE_CHUNK_MAX_SIZE (NVS_ENTRY_SIZE * (NVS_ENTRIES_PER_PAGE - 1)) // 4000 B
|
||||
|
||||
/* String lengths (strlen, excluding the NUL terminator) used to fragment a
|
||||
* partition. A string of length L occupies 1 header entry + ceil((L+1)/32) data
|
||||
* entries and must fit contiguously within a single page.
|
||||
* - FRAG_STR_LEN fills a fresh page to 124 entries, leaving exactly 2 free.
|
||||
* - FRAG_STR_FIRST_LEN is one entry shorter to account for the namespace entry
|
||||
* that is written on the first page, so that page also keeps 2 free entries. */
|
||||
#define FRAG_STR_LEN (123 * NVS_ENTRY_SIZE - 1) // 3935 -> 124 entries
|
||||
#define FRAG_STR_FIRST_LEN (122 * NVS_ENTRY_SIZE - 1) // 3903 -> 123 entries
|
||||
/* Namespace directory entries that live on the first NVS page before any
|
||||
* pre-population strings are written: one for the blob namespace and one for the
|
||||
* pre-population namespace. They are accounted for when filling page 0. */
|
||||
#define FIRST_PAGE_NS_ENTRIES 2
|
||||
|
||||
#define FRAG_NAMESPACE "_frag"
|
||||
#define PREP_NAMESPACE "_prep"
|
||||
#define BLOB_NAMESPACE "_blobs"
|
||||
|
||||
static const char *TAG = "nvs_statistics_example";
|
||||
@@ -220,193 +216,390 @@ static esp_err_t read_mock_data_from_namespace(const char* namespace_name)
|
||||
|
||||
#if CONFIG_EXAMPLE_RUN_OVERHEAD_MEASUREMENT
|
||||
|
||||
// Partitions of various sizes (declared in partitions.csv) swept by the measurement.
|
||||
static const char* measured_partitions[] = {
|
||||
"nvs_16k",
|
||||
"nvs_32k",
|
||||
"nvs_64k",
|
||||
// One measured combination of partition and blob size. The partition names must
|
||||
// match NVS partitions declared in partitions.csv.
|
||||
typedef struct {
|
||||
const char *partition_name; // NVS partition to run the measurement on
|
||||
size_t blob_size; // blob payload size in bytes
|
||||
} measurement_combo_t;
|
||||
|
||||
// Partition size / blob size combinations to measure. Adjust freely.
|
||||
static const measurement_combo_t measurement_combos[] = {
|
||||
{ "nvs_16k", 128 },
|
||||
{ "nvs_16k", 350 },
|
||||
{ "nvs_16k", 416 },
|
||||
{ "nvs_16k", 417 },
|
||||
{ "nvs_32k", 768 },
|
||||
{ "nvs_32k", 920 },
|
||||
{ "nvs_64k", 1612 },
|
||||
};
|
||||
|
||||
// Blob sizes (in bytes) measured for each partition.
|
||||
static const size_t blob_sizes[] = {128, 256, 512, 1024, 2048, 4096};
|
||||
// A per-page free-space target, expressed as an absolute number of free
|
||||
// (available) entries that must remain on a page after pre-population (out of
|
||||
// NVS_ENTRIES_PER_PAGE per page). The first page of the partition keeps
|
||||
// 'first_page_free' free entries and every remaining page keeps 'rest_free'.
|
||||
typedef struct {
|
||||
unsigned first_page_free; // free (available) entries left on the first NVS page
|
||||
unsigned rest_free; // free (available) entries left on every remaining NVS page
|
||||
} page_free_range_t;
|
||||
|
||||
// Theoretical entries consumed by a single blob in a non-fragmented partition:
|
||||
// 1 BLOB_INDEX entry + 'chunks' chunk-header entries + ceil(size/32) data entries.
|
||||
// Per-page free-entry targets to measure (free entries out of 126 per page).
|
||||
static const page_free_range_t page_free_ranges[] = {
|
||||
{ 14, 11 },
|
||||
{ 14, 7 },
|
||||
{ 14, 4 },
|
||||
{ 13, 4 },
|
||||
};
|
||||
|
||||
// Results-table layout (see run_overhead_measurement()). Columns 2 and 3 ("First"
|
||||
// and "Remaining") are grouped under a shared "Free Entries per Page" header.
|
||||
#define TABLE_COL_COUNT 9
|
||||
#define TABLE_GROUP_C0 2 // "First"
|
||||
#define TABLE_GROUP_C1 3 // "Remaining"
|
||||
static const int table_col_width[TABLE_COL_COUNT] = { 9, 14, 9, 13, 17, 16, 14, 16, 8 };
|
||||
|
||||
// Theoretical (ideal) number of entries consumed by a single blob stored in a
|
||||
// non-fragmented partition:
|
||||
// 1 BLOB_INDEX entry + 'chunks' chunk-header entries + ceil(size/32) data entries.
|
||||
static size_t entries_per_blob_ideal(size_t blob_size)
|
||||
{
|
||||
size_t chunks = (blob_size + NVS_PAGE_CHUNK_MAX_SIZE - 1) / NVS_PAGE_CHUNK_MAX_SIZE;
|
||||
if (chunks == 0) {
|
||||
chunks = 1;
|
||||
chunks = 1; // an empty blob still needs one (empty) data chunk
|
||||
}
|
||||
size_t data_entries = (blob_size + NVS_ENTRY_SIZE - 1) / NVS_ENTRY_SIZE;
|
||||
return 1 + chunks + data_entries;
|
||||
}
|
||||
|
||||
// Pre-populate a partition with large strings so that every page is filled up to
|
||||
// its last 2 entries. Returns the number of strings written.
|
||||
static size_t fragment_partition(const char* partition_name)
|
||||
// strlen (excluding the NUL terminator) of a string value occupying exactly
|
||||
// 'span' NVS entries. A string uses 1 header entry + ceil((strlen + 1) / 32) data
|
||||
// entries; choosing strlen = (span - 1) * 32 - 1 makes the data occupy exactly
|
||||
// (span - 1) entries with no rounding slack.
|
||||
static size_t string_len_for_span(size_t span)
|
||||
{
|
||||
return (span - 1) * NVS_ENTRY_SIZE - 1;
|
||||
}
|
||||
|
||||
// Create (and immediately close) a namespace so its single directory entry is
|
||||
// written to flash on the currently active page.
|
||||
static void create_namespace_entry(const char *partition_name, const char *namespace_name)
|
||||
{
|
||||
nvs_handle_t handle;
|
||||
esp_err_t err = nvs_open_from_partition(partition_name, FRAG_NAMESPACE, NVS_READWRITE, &handle);
|
||||
esp_err_t err = nvs_open_from_partition(partition_name, namespace_name, NVS_READWRITE, &handle);
|
||||
if (err != ESP_OK) {
|
||||
ESP_LOGE(TAG, "Error (%s) opening fragmentation handle on '%s'!", esp_err_to_name(err), partition_name);
|
||||
return 0;
|
||||
ESP_LOGE(TAG, "Error (%s) creating namespace '%s' on '%s'!", esp_err_to_name(err), namespace_name, partition_name);
|
||||
return;
|
||||
}
|
||||
nvs_close(handle);
|
||||
}
|
||||
|
||||
// Live-entry target for a page that must keep 'free' entries available, clamped so
|
||||
// that both the "keep" and the "filler" string stay writable.
|
||||
static size_t page_target_from_free(unsigned free)
|
||||
{
|
||||
long target = (long)NVS_ENTRIES_PER_PAGE - (long)free;
|
||||
if (target < (long)(FIRST_PAGE_NS_ENTRIES + 2)) {
|
||||
target = FIRST_PAGE_NS_ENTRIES + 2; // keep the page-0 'keep' string writable
|
||||
}
|
||||
if (target > (long)(NVS_ENTRIES_PER_PAGE - 2)) {
|
||||
target = NVS_ENTRIES_PER_PAGE - 2; // leave room for a >= 2-entry filler
|
||||
}
|
||||
return (size_t)target;
|
||||
}
|
||||
|
||||
// Pre-populate every usable NVS page with string values, leaving 'first_page_free'
|
||||
// free (available) entries on the first page and 'rest_free' on every remaining
|
||||
// page (the namespace entries on page 0 count towards page 0's occupancy). The
|
||||
// entries not meant to stay free are first filled with removable strings and then
|
||||
// erased, so each page ends up FULL from the page allocator's point of view while
|
||||
// still exposing the requested number of reclaimable free entries. This scatters
|
||||
// the free space into per-page gaps - the fragmentation pattern a single blob
|
||||
// write then has to cope with.
|
||||
static void prepopulate_partition(const char *partition_name, unsigned first_page_free, unsigned rest_free)
|
||||
{
|
||||
nvs_handle_t handle;
|
||||
esp_err_t err = nvs_open_from_partition(partition_name, PREP_NAMESPACE, NVS_READWRITE, &handle);
|
||||
if (err != ESP_OK) {
|
||||
ESP_LOGE(TAG, "Error (%s) opening pre-population handle on '%s'!", esp_err_to_name(err), partition_name);
|
||||
return;
|
||||
}
|
||||
|
||||
char* buffer = malloc(FRAG_STR_LEN + 1);
|
||||
// Live-entry targets derived from the requested free-entry counts.
|
||||
size_t first_target = page_target_from_free(first_page_free);
|
||||
size_t rest_target = page_target_from_free(rest_free);
|
||||
|
||||
// Size the reusable string buffer for the largest string span that can occur:
|
||||
// the largest "keep" string (highest live-entry target) or the largest "filler"
|
||||
// string (largest free gap), whichever is bigger.
|
||||
size_t max_target = (first_target > rest_target) ? first_target : rest_target;
|
||||
size_t min_target = (first_target < rest_target) ? first_target : rest_target;
|
||||
size_t max_span = max_target;
|
||||
size_t max_gap = NVS_ENTRIES_PER_PAGE - min_target;
|
||||
if (max_gap > max_span) {
|
||||
max_span = max_gap;
|
||||
}
|
||||
|
||||
char *buffer = malloc(string_len_for_span(max_span) + 1);
|
||||
if (buffer == NULL) {
|
||||
ESP_LOGE(TAG, "Failed to allocate fragmentation buffer!");
|
||||
ESP_LOGE(TAG, "Failed to allocate pre-population buffer!");
|
||||
nvs_close(handle);
|
||||
return 0;
|
||||
return;
|
||||
}
|
||||
memset(buffer, 'A', FRAG_STR_LEN);
|
||||
buffer[FRAG_STR_LEN] = '\0';
|
||||
memset(buffer, 'A', string_len_for_span(max_span));
|
||||
|
||||
size_t count = 0;
|
||||
char key[16];
|
||||
|
||||
// First string is one entry shorter to compensate for the namespace entry
|
||||
// written on page 0, so that page also retains exactly 2 free entries.
|
||||
buffer[FRAG_STR_FIRST_LEN] = '\0';
|
||||
snprintf(key, sizeof(key), "f%05u", (unsigned)count);
|
||||
err = nvs_set_str(handle, key, buffer);
|
||||
buffer[FRAG_STR_FIRST_LEN] = 'A';
|
||||
if (err == ESP_OK && nvs_commit(handle) == ESP_OK) {
|
||||
count++;
|
||||
}
|
||||
|
||||
// Remaining full-page strings until the partition cannot hold another one.
|
||||
unsigned page = 0;
|
||||
while (true) {
|
||||
snprintf(key, sizeof(key), "f%05u", (unsigned)count);
|
||||
char key[16];
|
||||
|
||||
// Live-entry target for this page: first_target on page 0, rest_target on all others.
|
||||
size_t target = (page == 0) ? first_target : rest_target;
|
||||
size_t free_gap = NVS_ENTRIES_PER_PAGE - target;
|
||||
|
||||
// On page 0 the two namespace entries already occupy part of the target share.
|
||||
size_t keep_span = (page == 0) ? (target - FIRST_PAGE_NS_ENTRIES) : target;
|
||||
size_t keep_len = string_len_for_span(keep_span);
|
||||
buffer[keep_len] = '\0';
|
||||
snprintf(key, sizeof(key), "k%05u", page);
|
||||
err = nvs_set_str(handle, key, buffer);
|
||||
buffer[keep_len] = 'A';
|
||||
if (err == ESP_ERR_NVS_NOT_ENOUGH_SPACE) {
|
||||
break;
|
||||
break; // partition full (one page kept in reserve)
|
||||
}
|
||||
if (err != ESP_OK) {
|
||||
ESP_LOGE(TAG, "Error (%s) writing fragmentation string!", esp_err_to_name(err));
|
||||
ESP_LOGE(TAG, "Error (%s) writing pre-population string!", esp_err_to_name(err));
|
||||
break;
|
||||
}
|
||||
if (nvs_commit(handle) != ESP_OK) {
|
||||
|
||||
// Fill the rest of this page with a removable string, marking the page FULL.
|
||||
size_t fill_len = string_len_for_span(free_gap);
|
||||
buffer[fill_len] = '\0';
|
||||
snprintf(key, sizeof(key), "x%05u", page);
|
||||
err = nvs_set_str(handle, key, buffer);
|
||||
buffer[fill_len] = 'A';
|
||||
if (err != ESP_OK) {
|
||||
break;
|
||||
}
|
||||
count++;
|
||||
nvs_commit(handle);
|
||||
page++;
|
||||
}
|
||||
|
||||
// Erase every filler, turning the completely full pages into pages that keep
|
||||
// 'target' live entries and expose 'free_gap' reclaimable entries each.
|
||||
for (unsigned i = 0; i < page; i++) {
|
||||
char key[16];
|
||||
snprintf(key, sizeof(key), "x%05u", i);
|
||||
nvs_erase_key(handle, key);
|
||||
}
|
||||
nvs_commit(handle);
|
||||
|
||||
free(buffer);
|
||||
nvs_close(handle);
|
||||
return count;
|
||||
}
|
||||
|
||||
// Fill a partition with same-sized blobs until it runs out of space.
|
||||
// Returns the number of blobs successfully stored.
|
||||
static size_t fill_with_blobs(const char* partition_name, size_t blob_size)
|
||||
// Print one horizontal separator line with a '+' at every column boundary.
|
||||
static void print_table_separator(void)
|
||||
{
|
||||
putchar('+');
|
||||
for (int c = 0; c < TABLE_COL_COUNT; c++) {
|
||||
for (int i = 0; i < table_col_width[c] + 2; i++) {
|
||||
putchar('-');
|
||||
}
|
||||
putchar('+');
|
||||
}
|
||||
putchar('\n');
|
||||
}
|
||||
|
||||
// Print the top border of the header: like print_table_separator(), but the two
|
||||
// grouped columns are merged into a single segment (no boundary between them).
|
||||
static void print_table_group_top(void)
|
||||
{
|
||||
for (int c = 0; c < TABLE_COL_COUNT; c++) {
|
||||
putchar((c == TABLE_GROUP_C1) ? '-' : '+');
|
||||
for (int i = 0; i < table_col_width[c] + 2; i++) {
|
||||
putchar('-');
|
||||
}
|
||||
}
|
||||
putchar('+');
|
||||
putchar('\n');
|
||||
}
|
||||
|
||||
// Print the divider between the group label and its sub-labels: the grouped
|
||||
// columns are split with a dashed rule, every other column stays blank.
|
||||
static void print_table_group_divider(void)
|
||||
{
|
||||
for (int c = 0; c < TABLE_COL_COUNT; c++) {
|
||||
bool group_edge = (c == TABLE_GROUP_C0 || c == TABLE_GROUP_C1 || c == TABLE_GROUP_C1 + 1);
|
||||
char fill = (c == TABLE_GROUP_C0 || c == TABLE_GROUP_C1) ? '-' : ' ';
|
||||
putchar(group_edge ? '+' : '|');
|
||||
for (int i = 0; i < table_col_width[c] + 2; i++) {
|
||||
putchar(fill);
|
||||
}
|
||||
}
|
||||
putchar('|');
|
||||
putchar('\n');
|
||||
}
|
||||
|
||||
// Print one table row from TABLE_COL_COUNT right-aligned cell strings.
|
||||
static void print_table_row(const char *cells[TABLE_COL_COUNT])
|
||||
{
|
||||
putchar('|');
|
||||
for (int c = 0; c < TABLE_COL_COUNT; c++) {
|
||||
printf(" %*s |", table_col_width[c], cells[c]);
|
||||
}
|
||||
putchar('\n');
|
||||
}
|
||||
|
||||
// Print a string centered within a 'width'-character field.
|
||||
static void print_centered(const char *s, int width)
|
||||
{
|
||||
int len = (int)strlen(s);
|
||||
if (len > width) {
|
||||
len = width;
|
||||
}
|
||||
int left = (width - len) / 2;
|
||||
int right = width - len - left;
|
||||
printf("%*s%.*s%*s", left, "", len, s, right, "");
|
||||
}
|
||||
|
||||
// Print the two-line header. The first line carries the column names (with "Free
|
||||
// Entries per Page" centered above the two grouped columns) and the second line
|
||||
// carries the unit of measurement of each column (and, for the grouped columns,
|
||||
// their "First" / "Remaining" sub-labels which keep their unit inline).
|
||||
static void print_table_header(void)
|
||||
{
|
||||
// Row 1: column names (units moved to row 2 below).
|
||||
static const char *names[TABLE_COL_COUNT] = {
|
||||
"Blob Size", "Partition Size", "Free Entries per Page", "",
|
||||
"Available Entries", "Expected Entries", "Actual Entries",
|
||||
"Overhead Entries", "Overhead"
|
||||
};
|
||||
// Row 2: units of measurement; the grouped columns keep their sub-labels here.
|
||||
static const char *units[TABLE_COL_COUNT] = {
|
||||
"[B]", "[k]", "First [-]", "Remaining [-]",
|
||||
"[-]", "[-]", "[-]", "[-]", "[%]"
|
||||
};
|
||||
|
||||
print_table_group_top();
|
||||
|
||||
// Row 1: column names, with the group name centered above the two grouped columns.
|
||||
putchar('|');
|
||||
for (int c = 0; c < TABLE_COL_COUNT; c++) {
|
||||
if (c == TABLE_GROUP_C0) {
|
||||
putchar(' ');
|
||||
print_centered(names[TABLE_GROUP_C0], table_col_width[TABLE_GROUP_C0] + table_col_width[TABLE_GROUP_C1] + 3);
|
||||
printf(" |");
|
||||
c = TABLE_GROUP_C1; // the second grouped column is covered by the merged name
|
||||
} else {
|
||||
printf(" %*s |", table_col_width[c], names[c]);
|
||||
}
|
||||
}
|
||||
putchar('\n');
|
||||
|
||||
print_table_group_divider();
|
||||
|
||||
// Row 2: units of measurement (grouped columns split into their sub-labels).
|
||||
print_table_row(units);
|
||||
|
||||
print_table_separator();
|
||||
}
|
||||
|
||||
// Measure the real vs. ideal blob entry consumption for one combination and one
|
||||
// per-page free-entry target, then print the corresponding results-table row.
|
||||
static void measure_blob_overhead(const measurement_combo_t *combo, page_free_range_t page_free)
|
||||
{
|
||||
const char *part = combo->partition_name;
|
||||
|
||||
ESP_ERROR_CHECK(nvs_flash_erase_partition(part));
|
||||
ESP_ERROR_CHECK(nvs_flash_init_partition(part));
|
||||
|
||||
unsigned size_kb = 0;
|
||||
const esp_partition_t *p = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_NVS, part);
|
||||
if (p != NULL) {
|
||||
size_kb = (unsigned)(p->size / 1024);
|
||||
}
|
||||
|
||||
// Create the blob namespace up-front so its entry is part of the population.
|
||||
create_namespace_entry(part, BLOB_NAMESPACE);
|
||||
|
||||
// Leave the requested number of free entries on the first page and remaining pages.
|
||||
prepopulate_partition(part, page_free.first_page_free, page_free.rest_free);
|
||||
|
||||
// Available entries reported right before the blob is written.
|
||||
nvs_stats_t stats_before;
|
||||
ESP_ERROR_CHECK(nvs_get_stats(part, &stats_before));
|
||||
|
||||
// Attempt to write a single blob and measure its real entry footprint.
|
||||
bool stored = false;
|
||||
size_t actual_entries = 0;
|
||||
nvs_handle_t handle;
|
||||
esp_err_t err = nvs_open_from_partition(partition_name, BLOB_NAMESPACE, NVS_READWRITE, &handle);
|
||||
if (err != ESP_OK) {
|
||||
ESP_LOGE(TAG, "Error (%s) opening blob handle on '%s'!", esp_err_to_name(err), partition_name);
|
||||
return 0;
|
||||
}
|
||||
|
||||
uint8_t* blob = malloc(blob_size);
|
||||
if (blob == NULL) {
|
||||
ESP_LOGE(TAG, "Failed to allocate %u B blob buffer!", (unsigned)blob_size);
|
||||
nvs_close(handle);
|
||||
return 0;
|
||||
}
|
||||
memset(blob, 0x5A, blob_size);
|
||||
|
||||
size_t count = 0;
|
||||
char key[16];
|
||||
while (true) {
|
||||
snprintf(key, sizeof(key), "b%05u", (unsigned)count);
|
||||
err = nvs_set_blob(handle, key, blob, blob_size);
|
||||
if (err == ESP_ERR_NVS_NOT_ENOUGH_SPACE) {
|
||||
break;
|
||||
ESP_ERROR_CHECK(nvs_open_from_partition(part, BLOB_NAMESPACE, NVS_READWRITE, &handle));
|
||||
uint8_t *blob = malloc(combo->blob_size);
|
||||
if (blob != NULL) {
|
||||
memset(blob, 0x5A, combo->blob_size);
|
||||
esp_err_t err = nvs_set_blob(handle, "the_blob", blob, combo->blob_size);
|
||||
if (err == ESP_OK) {
|
||||
err = nvs_commit(handle);
|
||||
}
|
||||
if (err != ESP_OK) {
|
||||
ESP_LOGE(TAG, "Error (%s) writing blob!", esp_err_to_name(err));
|
||||
break;
|
||||
free(blob);
|
||||
if (err == ESP_OK) {
|
||||
nvs_stats_t stats_after;
|
||||
ESP_ERROR_CHECK(nvs_get_stats(part, &stats_after));
|
||||
actual_entries = stats_after.used_entries - stats_before.used_entries;
|
||||
stored = true;
|
||||
}
|
||||
err = nvs_commit(handle);
|
||||
if (err == ESP_ERR_NVS_NOT_ENOUGH_SPACE) {
|
||||
break;
|
||||
}
|
||||
count++;
|
||||
// A failing write (e.g. ESP_ERR_NVS_NOT_ENOUGH_SPACE) is an expected outcome
|
||||
// for tightly populated partitions and is reported as "FAIL" in the table below.
|
||||
} else {
|
||||
ESP_LOGE(TAG, "Failed to allocate %u B blob buffer!", (unsigned)combo->blob_size);
|
||||
}
|
||||
|
||||
free(blob);
|
||||
nvs_close(handle);
|
||||
return count;
|
||||
}
|
||||
|
||||
// Run one measurement cell: erase + init a partition, optionally fragment it,
|
||||
// fill it with blobs of the given size and report heap/entry/overhead statistics.
|
||||
static void measure_blob_overhead(const char* partition_name, size_t blob_size, bool fragment)
|
||||
{
|
||||
ESP_ERROR_CHECK(nvs_flash_erase_partition(partition_name));
|
||||
size_t expected_entries = entries_per_blob_ideal(combo->blob_size);
|
||||
|
||||
uint32_t heap_before_init = esp_get_free_heap_size();
|
||||
ESP_ERROR_CHECK(nvs_flash_init_partition(partition_name));
|
||||
uint32_t heap_after_init = esp_get_free_heap_size();
|
||||
|
||||
nvs_stats_t stats;
|
||||
ESP_ERROR_CHECK(nvs_get_stats(partition_name, &stats));
|
||||
size_t total_entries = stats.total_entries;
|
||||
|
||||
size_t frag_strings = 0;
|
||||
if (fragment) {
|
||||
frag_strings = fragment_partition(partition_name);
|
||||
char c_blob[16], c_part[16], c_first[16], c_rest[16], c_avail[16], c_exp[16], c_act[16], c_ovh[16], c_ovhpct[16];
|
||||
snprintf(c_blob, sizeof(c_blob), "%u", (unsigned)combo->blob_size);
|
||||
snprintf(c_part, sizeof(c_part), "%u", size_kb);
|
||||
snprintf(c_first, sizeof(c_first), "%u", page_free.first_page_free);
|
||||
snprintf(c_rest, sizeof(c_rest), "%u", page_free.rest_free);
|
||||
snprintf(c_avail, sizeof(c_avail), "%u", (unsigned)stats_before.available_entries);
|
||||
snprintf(c_exp, sizeof(c_exp), "%u", (unsigned)expected_entries);
|
||||
if (stored) {
|
||||
long overhead_entries = (long)actual_entries - (long)expected_entries;
|
||||
double overhead_pct = (expected_entries != 0) ? (100.0 * (double)overhead_entries / (double)expected_entries) : 0.0;
|
||||
snprintf(c_act, sizeof(c_act), "%u", (unsigned)actual_entries);
|
||||
snprintf(c_ovh, sizeof(c_ovh), "%ld", overhead_entries);
|
||||
snprintf(c_ovhpct, sizeof(c_ovhpct), "%.1f", overhead_pct);
|
||||
} else {
|
||||
snprintf(c_act, sizeof(c_act), "%s", "FAIL");
|
||||
snprintf(c_ovh, sizeof(c_ovh), "%s", "-");
|
||||
snprintf(c_ovhpct, sizeof(c_ovhpct), "%s", "-");
|
||||
}
|
||||
|
||||
size_t stored = fill_with_blobs(partition_name, blob_size);
|
||||
uint32_t heap_after_fill = esp_get_free_heap_size();
|
||||
const char *cells[TABLE_COL_COUNT] = { c_blob, c_part, c_first, c_rest, c_avail, c_exp, c_act, c_ovh, c_ovhpct };
|
||||
print_table_row(cells);
|
||||
|
||||
ESP_ERROR_CHECK(nvs_get_stats(partition_name, &stats));
|
||||
|
||||
size_t per_blob = entries_per_blob_ideal(blob_size);
|
||||
size_t expected = (per_blob != 0) ? (total_entries / per_blob) : 0;
|
||||
size_t payload = stored * blob_size;
|
||||
size_t capacity = total_entries * NVS_ENTRY_SIZE;
|
||||
int overhead_pct = (capacity != 0) ? (int)(100 - (100ULL * payload) / capacity) : 0;
|
||||
|
||||
printf("\n");
|
||||
printf("NVS BLOB TEST - %u B (partition '%s', %s):\n",
|
||||
(unsigned)blob_size, partition_name, fragment ? "fragmented" : "pristine");
|
||||
printf("======================\n");
|
||||
printf("heap before NVS init: %" PRIu32 " B\n", heap_before_init);
|
||||
printf("heap after NVS init: %" PRIu32 " B (diff %" PRId32 " B)\n",
|
||||
heap_after_init, (int32_t)(heap_before_init - heap_after_init));
|
||||
if (fragment) {
|
||||
printf("fragmentation strings written: %u\n", (unsigned)frag_strings);
|
||||
}
|
||||
printf("\n");
|
||||
printf("available heap after fill: %" PRIu32 " B (diff %" PRId32 " B)\n",
|
||||
heap_after_fill, (int32_t)(heap_after_init - heap_after_fill));
|
||||
printf("expected blobs count: %u\n", (unsigned)expected);
|
||||
printf("stored blobs count: %u\n", (unsigned)stored);
|
||||
printf("used_entries: %u (%u B)\n", stats.used_entries, (unsigned)(stats.used_entries * NVS_ENTRY_SIZE));
|
||||
printf("free_entries: %u (%u B)\n", stats.free_entries, (unsigned)(stats.free_entries * NVS_ENTRY_SIZE));
|
||||
printf("total_entries: %u (%u B)\n", stats.total_entries, (unsigned)(stats.total_entries * NVS_ENTRY_SIZE));
|
||||
printf("STORAGE OVERHEAD: %d%%\n", overhead_pct);
|
||||
|
||||
ESP_ERROR_CHECK(nvs_flash_deinit_partition(partition_name));
|
||||
ESP_ERROR_CHECK(nvs_flash_deinit_partition(part));
|
||||
}
|
||||
|
||||
static void run_overhead_measurement(void)
|
||||
{
|
||||
const size_t partition_count = sizeof(measured_partitions) / sizeof(measured_partitions[0]);
|
||||
const size_t blob_size_count = sizeof(blob_sizes) / sizeof(blob_sizes[0]);
|
||||
const size_t combo_count = sizeof(measurement_combos) / sizeof(measurement_combos[0]);
|
||||
const size_t free_range_count = sizeof(page_free_ranges) / sizeof(page_free_ranges[0]);
|
||||
|
||||
ESP_LOGI(TAG, "Starting NVS blob storage-overhead measurement...");
|
||||
for (size_t p = 0; p < partition_count; p++) {
|
||||
for (size_t b = 0; b < blob_size_count; b++) {
|
||||
measure_blob_overhead(measured_partitions[p], blob_sizes[b], false);
|
||||
#if CONFIG_EXAMPLE_INDUCE_FRAGMENTATION
|
||||
measure_blob_overhead(measured_partitions[p], blob_sizes[b], true);
|
||||
#endif
|
||||
|
||||
printf("\n");
|
||||
print_table_header();
|
||||
|
||||
for (size_t i = 0; i < combo_count; i++) {
|
||||
for (size_t j = 0; j < free_range_count; j++) {
|
||||
measure_blob_overhead(&measurement_combos[i], page_free_ranges[j]);
|
||||
}
|
||||
}
|
||||
|
||||
print_table_separator();
|
||||
printf("\n");
|
||||
|
||||
ESP_LOGI(TAG, "NVS blob storage-overhead measurement done.");
|
||||
}
|
||||
|
||||
|
||||
@@ -14,9 +14,4 @@ def test_examples_nvs_statistics(dut: Dut) -> None:
|
||||
dut.expect('Getting post-commit NVS statistics...', timeout=5)
|
||||
dut.expect('Newly used entries match expectation.', timeout=5)
|
||||
|
||||
# Blob storage-overhead measurement (sweeps partition and blob sizes).
|
||||
dut.expect('Starting NVS blob storage-overhead measurement...', timeout=10)
|
||||
dut.expect('STORAGE OVERHEAD:', timeout=60)
|
||||
dut.expect('NVS blob storage-overhead measurement done.', timeout=120)
|
||||
|
||||
dut.expect('Returning from app_main().', timeout=5)
|
||||
|
||||
Reference in New Issue
Block a user