ESP-IDF compiles every component with -ffunction-sections/-fdata-sections and
links with -Wl,--gc-sections, so that code and data nothing references does not
reach the image. With CONFIG_COMPILER_LTO_LINKTIME the object files carry GIMPLE
instead of machine code and code generation is deferred to the link, where those
two options no longer apply. The LTO partition is therefore emitted as a single
.text/.rodata, and since --gc-sections works at section granularity it can only
keep or drop that section as a whole. Some function in it is always live, so
nothing is dropped.
The options are recorded in the object file (in the .gnu.lto_.opts section), but
they do not reach the code generator. Code generation at link time runs as a
separate compiler invocation (LTRANS) whose command line lto-wrapper
reconstructs from those recorded options, and append_compiler_options() in
gcc/lto-wrapper.cc forwards only CL_TARGET options plus a small hard-coded list:
-fPIC/-fpic/-fPIE/-fpie, -fcommon, -fgnu-tm, -fopenmp/-fopenacc, -fcf-protection=,
(-fasynchronous-)unwind-tables, -g, -O/-Os/-Og/-Ofast/-Oz and the diagnostics
formatting options. Everything else hits the default arm and is dropped.
-ffunction-sections/-fdata-sections are neither. They are plain Common options in
common.opt without the Optimization marker, so they are not part of the
per-function state that is streamed with each function (which is why -O2 and -Os
do survive per translation unit under LTO), and they are not target options
either. They reach LTRANS only if they are repeated on the link command line.
Pass them next to -flto=auto, in both build systems.
This can be verified on any LTO build by adding -save-temps to the link options
and inspecting the generated <output>.ltrans.mk, which contains the literal
LTRANS command line:
grep -o -- "-ffunction-sections\|-fdata-sections" build/*.ltrans.mk
Measured on esp32c3 with -Os and CONFIG_COMPILER_LTO_COMPILETIME, application
binary size in bytes:
no LTO LTO LTO+fix fix saves
hello_world 116112 119792 115808 -3984 (-3.33%)
wifi/getting_started 734720 792960 732976 -59984 (-7.56%)
Without this change, enabling LTO produces a larger image than not using LTO at
all; with it, LTO is size neutral to slightly positive.
Two effects contribute. The dead code inside the partition itself stays, which
scales with how much code was compiled with LTO. On top of that, every function
retained this way keeps whatever it references alive as well, transitively and
across object boundaries: constant data, other functions, and sections of objects
that were not compiled with LTO at all. That second effect is not bounded by the
size of the LTO partition and can dominate. In the wifi/getting_started case a
single retained function, esp_crt_bundle_attach(), is the only referrer of the
mbedTLS certificate bundle, so a 55 KB .rodata blob stayed in an application that
never uses TLS. A plain non-LTO build collects all of it, which is what this
change restores.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Link-time optimization (LTO) lets the compiler inline and optimize across
translation units. ESP-IDF relies heavily on linker-script placement rules
that match object files by name, which LTO does not preserve, so LTO cannot
be enabled for the whole framework. This change adds two opt-in options that
side-step that conflict:
- CONFIG_COMPILER_LTO_LINKTIME tells the linker to perform LTO on any object
files that carry LTO information (compiled with -flto). On its own this is
safe: users can add -flto to specific components (e.g. their own libraries)
to shrink them, without affecting components that use linker fragments.
- CONFIG_COMPILER_LTO_COMPILETIME automatically compiles most
components with -flto. A component is excluded when it has its own linker
fragments, when it opts out via the NO_LTO component property, or when its
object code is placed by *another* component's linker fragment (matched by
archive name). The last case is handled by tools/cmake/lto.cmake, which
scans linker fragments for explicit "archive: libNAME.a" placement and
excludes those components. Without it, functions that must run from IRAM
while the flash cache is disabled (e.g. the spi_flash / GDMA HAL routines,
placed in IRAM by spi_flash/esp_driver_dma fragments) would be moved to
flash by LTO and the device would panic with a cache error at run time.
Both options are disabled for the bootloader and ESP-TEE builds, which depend
on object-file-name based placement. LTO is also gated off for Clang
(needs LLD, IDF-8286) and host builds.
LTO works together with CONFIG_APP_REPRODUCIBLE_BUILD, but needs extra
flags: LTO defers code generation and most debug-info emission from compile
time to link time, where the reproducible-build path remapping (applied to
compile_options only) does not take effect. When both options are enabled,
three extra flags keep the .elf, .bin and .map byte-identical across build
directories (verified on esp32 / GCC 16.1):
- the -f*-prefix-map options are passed to the linker as well, so the LTO
code generator remaps DW_AT_comp_dir (otherwise the build dir leaks into
.debug_str, and cascades into esp_app_desc_t.app_elf_sha256 in the .bin);
- -save-temps makes lto-wrapper use stable LTRANS object names in the build
dir instead of random $TMPDIR paths that leak into the .map;
- -frandom-seed=1 makes LTO GIMPLE bytecode objects byte-identical (a
shared seed was verified not to collide, including for C++ file-local
static variables and anonymous namespaces promoted by LTO).
The gcc-ar / gcc-ranlib wrappers are selected in the GCC toolchain file so
that the LTO plugin is loaded when creating and indexing static archives;
plain ar/ranlib do not record LTO symbols in the archive index.
Note: LTO, like other inlining, can also increase binary size. Enable it
together with CONFIG_COMPILER_OPTIMIZATION_SIZE to get a code-size benefit.
Related: IDF-71, IDF-8286
Closes https://github.com/espressif/esp-idf/issues/18741
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add KASAN support for detecting heap memory safety bugs (buffer
overflows, underflows, use-after-free) at runtime using compiler
instrumentation and shadow memory. Gated behind
CONFIG_IDF_EXPERIMENTAL_FEATURES, with touch points kept to esp_system
and heap so other components stay untouched.
- Core runtime (esp_system/kasan.c, esp_kasan.h): nibble-based shadow
memory in DRAM, poison/unpoison, per-access validation, and __asan_*
stubs; hot-path stubs in IRAM so they stay valid with the flash cache
off. Shadow init runs before heap bring-up.
- Heap integration (heap/heap_kasan*.c): alloc/free hooks add redzones,
a quarantine FIFO, and shadow updates.
- Panic handling: disable checks once at the panic handler entry so
backtrace and stack dumps can read redzones without nested reports.
- Build system: -fsanitize=kernel-address for app code, with HAL, SoC,
esp_rom, SPI flash, esp_hw_support, bootloader_support, FreeRTOS, and
heap internals excluded from instrumentation.
- Test app (tools/test_apps/system/kasan_test): Unity tests for
overflow, underflow, use-after-free, and all sized __asan_* stubs,
with halt and no-halt configurations.
- Docs: document KASAN in the heap memory debugging guide (EN and CN).
Introduce ESPTOOLPY_FAST_REFLASHING configuration option. It instructs
ldgen to group entity mappings for libraries deemed mutable (prone to
change) separately from those considered immutable (unlikely to change).
Organizing mutable and immutable libraries separately in the linker
script allows the linker to form larger contiguous blocks of data for
immutable libraries in the application's output sections. These blocks
are likely to stay mostly unchanged between application recompilations,
enabling them to be skipped during reflashing.
Separating mutable and immutable libraries in the linker script to
minimize changes in the output sections is insufficient. Padding is
added after the input sections of mutable libraries in the default data
and text output sections. This creates a buffer for the mutable
libraries, allowing additional changes to be made without altering the
layout of the binary image.
Additionally two optimizations currently in use can still mix data from
these libraries, leading to significant changes even within the grouped
immutable libraries.
1. constant merging
Linker will try to merge input sections that have the MERGE and
STRING flags from different libraries (object files) to perform
optimizations like tail merging. For example, adding a string
literal in a mutable library will also change the addresses of
string literals from immutable libraries in such a merged section,
causing changes in the generated code when those literals are
referenced.
Disabled with COMPILER_DISABLE_MERGE_CONSTANTS(-fno-merge-constants)
2. literal pools on Xtensa
As optimization, the linker may merge literal pools from different
libraries (object files) to improve the generated code size. This
has the same effect as constant merging, and changes in mutable
libraries may cause changes in the generated code for immutable
libraries. To get larger unchanged continuous blocks in the text
output sections for immutable libraries, we need to ensure that the
Xtensa literal pools remain close to their references and are not
merged.
Disabled with CONFIG_COMPILER_ENABLE_TEXT_SECTION_LITERALS(-mtext-section-literals)
When ESPTOOLPY_FAST_REFLASHING is enabled, these two optimizations are
disabled to achieve larger unchaged continuous blocks for the grouped
immutable libraries, even though disabling these optimizations results
in slightly larger code.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
There are idf.py hints for helping the user to set component
dependencies properly instead of building sources out-of-component or
including headers from outside the component directory. These are
produced with
tools/idf_py_actions/hint_modules/component_requirements.py.
However, idf.py hints are printed only when the build fails. If the user
starts with a buildable solution then the suggestions to add component
dependencies are not printed.
This commit introduces cmake-level warnings for building source files
from outside the component and including header files without setting up
proper component dependencies.
Allow to disable implicit inlining of constexpr functions from libstdc++.
This is a known GCC issue https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93008
that may affect C++ application size depending on its structure.
GCC defines that malloc(0) returns a non-null pointer (glibc behavior).
This can break logic for some cases, because our implementation of
malloc(0) returns null pointer.
POSIX and the ISO C standard allow both behaviors.
GCC 15 introduces a regression in guaranteed zero-initialization of
padding bits. The suggested solution is to add the
-fzero-init-padding-bits=unions compile option.
To prevent similar issues in the future, this change adds the
-fzero-init-padding-bits=all build option. It ensures that padding bits
in unions and structs are properly zeroed, avoiding regressions.
See https://clang.llvm.org/docs/CommandGuide/clang.html:
>> Code Generation Options
>> -Oz Like -Os (and thus -O2), but reduces code size further.
Without -Oz enabled clang produced binaries that were too large.
This commit updates the ld linker flags to conditionally include the
-warn_commons flag when the linux target is built on MacOS. This is
because, not all versions of ld support the -warn_commons option.
Closes https://github.com/espressif/esp-idf/issues/13185
Probably GCC-13.x and on-wards uses "-fmerge-constants" to merge
the const section (string/floating-point) across compilation units.
This makes it difficult to properly analyze the size output of rodata
section across libraries, the merged section (big in size) is showed
across a single library.
The config option added here can help to disable this compiler behavior
and help to provide better size analysis. It can be used during
development phase only as it increases rodata section size.
The -Werror=all activates error for all warnings in -Wall, however, it
does not activate error for other default warnings, such as:
- int-conversion (pointer from integer w/o a cast)
- incompatible-pointer-types
- discarded-qualifiers
Which are IMO even more important that -Wall.
This commit fixes that by activating error for all warnings (i.e. from
-Wall and default ones) and removing those from -Wextra, as the culprit
commit seemed to address.
Fixes: 60f29236f6 "Build system: Raise warning level" (2016-11-16)
In order to avoid long analysis during esp-idf upgrade, provide a way to
restore the previous -Werror=all behavior that consider only warnings
from -Wall (and not default ones).
Also add a hint to use the Kconfig option on compilation error, but warn
that fixing the code is the preferred way.
Merges https://github.com/espressif/esp-idf/pull/11239
Suggested-By: Ivan Grokhotkov <ivan@espressif.com>
This updates the minimal supported version of CMake to 3.16, which in turn enables us to use more CMake features and have a cleaner build system.
This is the version that provides most new features and also the one we use in our latest docker image for CI.
