change(build): drop BUILD_COMPONENTS support and migrate test_apps to not use it for buildv2
Closes IDF-15859
See merge request espressif/esp-idf!51695
The compatibility shim populated BUILD_COMPONENTS from the library
interface's linked-components list so that consumers reading it kept
working. Those consumers now query the library interface directly, so
drop the shim population and reject reads of the property
unconditionally.
ESP-IDF compiles every component with -ffunction-sections/-fdata-sections and
links with -Wl,--gc-sections, so that code and data nothing references does not
reach the image. With CONFIG_COMPILER_LTO_LINKTIME the object files carry GIMPLE
instead of machine code and code generation is deferred to the link, where those
two options no longer apply. The LTO partition is therefore emitted as a single
.text/.rodata, and since --gc-sections works at section granularity it can only
keep or drop that section as a whole. Some function in it is always live, so
nothing is dropped.
The options are recorded in the object file (in the .gnu.lto_.opts section), but
they do not reach the code generator. Code generation at link time runs as a
separate compiler invocation (LTRANS) whose command line lto-wrapper
reconstructs from those recorded options, and append_compiler_options() in
gcc/lto-wrapper.cc forwards only CL_TARGET options plus a small hard-coded list:
-fPIC/-fpic/-fPIE/-fpie, -fcommon, -fgnu-tm, -fopenmp/-fopenacc, -fcf-protection=,
(-fasynchronous-)unwind-tables, -g, -O/-Os/-Og/-Ofast/-Oz and the diagnostics
formatting options. Everything else hits the default arm and is dropped.
-ffunction-sections/-fdata-sections are neither. They are plain Common options in
common.opt without the Optimization marker, so they are not part of the
per-function state that is streamed with each function (which is why -O2 and -Os
do survive per translation unit under LTO), and they are not target options
either. They reach LTRANS only if they are repeated on the link command line.
Pass them next to -flto=auto, in both build systems.
This can be verified on any LTO build by adding -save-temps to the link options
and inspecting the generated <output>.ltrans.mk, which contains the literal
LTRANS command line:
grep -o -- "-ffunction-sections\|-fdata-sections" build/*.ltrans.mk
Measured on esp32c3 with -Os and CONFIG_COMPILER_LTO_COMPILETIME, application
binary size in bytes:
no LTO LTO LTO+fix fix saves
hello_world 116112 119792 115808 -3984 (-3.33%)
wifi/getting_started 734720 792960 732976 -59984 (-7.56%)
Without this change, enabling LTO produces a larger image than not using LTO at
all; with it, LTO is size neutral to slightly positive.
Two effects contribute. The dead code inside the partition itself stays, which
scales with how much code was compiled with LTO. On top of that, every function
retained this way keeps whatever it references alive as well, transitively and
across object boundaries: constant data, other functions, and sections of objects
that were not compiled with LTO at all. That second effect is not bounded by the
size of the LTO partition and can dominate. In the wifi/getting_started case a
single retained function, esp_crt_bundle_attach(), is the only referrer of the
mbedTLS certificate bundle, so a 55 KB .rodata blob stayed in an application that
never uses TLS. A plain non-LTO build collects all of it, which is what this
change restores.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
On some Windows systems, antivirus, endpoint-security or DLP/encryption
software intercepts short-lived toolchain processes and strips their
stdout when the build captures it through a pipe, while the same command
prints its output normally when run by hand. The tool exits successfully
but returns nothing, and each build step that reads toolchain output then
failed with a different, cryptic error far from the real cause:
- CMake configuration aborted with "Unknown arguments specified" in
components/xtensa/project_include.cmake, or "check_expected_tool_version
invoked with incorrect arguments" in components/esp_common.
- ldgen turned the empty objdump output into an opaque pyparsing
"Expected 'In archive'" traceback.
- idf_tools.py silently reported the compiler/debugger version as
"unknown", sending users into a fruitless reinstall loop.
Detect the empty result at each consumer and fail (or warn) with an
actionable message that names the likely cause and the remedy:
- tools/cmake/compiler_query.cmake: new __compiler_query() helper runs a
compiler query and fails with a clear error on empty or failed output.
It is a standalone module included by both the cmakev1 and cmakev2
utilities, since the esp_common and xtensa project_include.cmake that
call it are shared by both build systems. The xtensa if() arguments are
now quoted so an empty result no longer collapses into a parse error.
- tools/ldgen/ldgen.py: _run_objdump() rejects empty objdump output, and
non-empty-but-unparsable section info is caught and re-raised as a clear
LdGenFailure instead of a raw pyparsing traceback.
- tools/idf_tools.py: empty version output now warns with the cause and
returns UNKNOWN_VERSION instead of silently reporting "unknown".
Closes https://github.com/espressif/esp-idf/issues/18727
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
idf_build_library() captured the LINK_OPTIONS build property before including
the components, so a link option a component appended while it was processed was
dropped from the executable link. Read the property as a generator expression so
these options are included.
Legacy ulp_embed_binary() call-site sources were attached directly to the CMake v2 child executable. However, the ULP runtime is linked with WHOLE_ARCHIVE through target_link_options(), which CMake emits before executable objects. This reversed the CMake v1 order, changed ULP code placement, and allowed runtime weak definitions to take precedence over legacy strong handlers.
Build the call-site sources as a dedicated static archive and place its whole-archive option before the runtime archive. Preserve the parent sdkconfig and include view, including FSM-specific include behavior, without generating a synthetic component tree.
Factor linker-specific whole-archive handling into a shared CMake v2 helper for GNU, Darwin, and ULP FSM linkers. Link the archive target normally as well so CMake tracks build and relink dependencies when a legacy ULP source changes.
Route ulp_embed_binary() through a built-in CMake v2 default ULP project instead of generating a compatibility project with a synthetic main component.
Attach legacy ULP sources directly to the ULP executable, pass the parent component include view through the existing ULP_S_SOURCES/COMPONENT_* channel, and remove the generated legacy project templates plus the component path ownership validation bypass they required.
Update the build-system coverage to assert that the builtin child project builds and rebuilds when parent-owned ULP sources change.
Preserve all linker scripts passed to target_linker_script() so LP-core child builds keep the full esp32p4 LP ROM script set instead of only the first script.
Link the ULP runtime component as WHOLE_ARCHIVE for ULP child builds, matching the legacy helper behavior where runtime sources were added directly to the executable. This keeps strong runtime handlers such as ulp_lp_core_panic_handler from being skipped in favor of weak defaults from already-extracted archive objects.
Validated with the esp32p4 lp_core_hp_uart panic and LP-ROM pytest cases, plus the focused local/CI build_all shape for the app.
Build ULP full subprojects through a dedicated entry file,
components/ulp/cmake/ulp_project.cmake, that wraps tools/cmakev2/idf.cmake
and layers a small ULP API on top, mirroring the cmakev2 layering:
ulp_project_init like idf_project_init (init, detect the ULP type,
reset the compile/link options inherited from the app)
ulp_build_executable like idf_build_executable, plus the embeddable
.bin/.h/.ld artifacts
ulp_project_default like idf_project_default (single-executable case)
A child project now includes this one file instead of idf.cmake and calls
these helpers directly, so idf_build_executable is used as-is for the
multi-binary case and the module-path indirection (include(IDFULPProject)
resolved via -DCMAKE_MODULE_PATH) is gone.
As a result:
- IDFULPProjectv2.cmake is removed; its setup moves into the wrapper.
- IDFULPProject.cmake becomes the CMake v1-only entry point.
- The ULP component no longer registers a POST_ELF callback; the binary
artifacts are produced by ulp_build_executable.
- The v2 full-subproject examples (lp_core, riscv, fsm, multi_binary,
combined) are updated to the new API.
- The ULP subproject API is documented in build-system-v2.rst.
Also fix a latent bug this exercises: idf_build_library emitted linker
scripts as "-T <name>" relying on a following "-L" search directory. GNU
ld only searches -L directories that precede -T, so the direct
esp32ulp-elf-ld link used for ULP FSM failed to open the script. Emit the
absolute path instead, matching what the CMake v1 ULP build already does.
Finally, replace the parent-argument bypass loop that used to live in
IDFULPProjectv2.cmake with --no-warn-unused-cli on the child configure,
and stop passing the unused IDF_PARENT_BUILD_DIR.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Register ULP memory linker scripts with target_linker_script so CMake v2 handles preprocessing and attachment through the component graph.
Keep the generated legacy outputs named .ld by stripping only the .in suffix, and pass full linker script paths to support direct ld invocation.
Resolve #include lines in linker script templates against the linked
component graph: idf_build_library appends every linked component's
INCLUDE_DIRS to the C preprocessor invocation for each .in linker
script, so a template can include any component header (e.g. a ULP
memory-layout template including soc/soc.h) without the owning
component having to know about it.
Also:
- Always pass -I<config_dir> so templates can include sdkconfig.h.
- Add a FLAGS option to target_linker_script for explicit preprocessor
flags (e.g. -D__ASSEMBLER__ or ld-snippet include dirs); when given,
FLAGS replaces the parent-dir==target include heuristic for that
script.
- Add a MEMORY option to target_linker_script that emits the marked
linker script as -T before all others, so section-placement scripts
in one component can reference MEMORY regions and REGION_ALIASes
declared by a memory-layout script in another component.
- Make C-preprocessor comment keeping (-C) part of the default flag set
rather than hardcoded, so FLAGS can drop it. linker_script_preprocessor
no longer forces -C; both the cmakev1 and cmakev2 preprocessors add it
to their default CFLAGS (output unchanged for existing scripts). A ULP
template that includes soc/soc.h omits -C so ld does not choke on the
header's // comments.
- Store per-script metadata (generated output, flags) in MD5-keyed
component properties instead of parallel lists.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Mirror the link-time optimization support into the cmakev2 build system so
that both build systems behave identically.
- project.cmake (__init_project_configuration): emit -flto=auto as a link
option when CONFIG_COMPILER_LTO_LINKTIME is set, except for bootloader and
ESP-TEE builds, otherwise keep -fno-lto.
- build.cmake (idf_build_library): when CONFIG_COMPILER_LTO_COMPILETIME is set,
compile each linked component with -flto=auto unless it has linker fragments,
is placed by another component's fragment (see tools/cmake/lto.cmake), has
opted out via NO_LTO, or is not a static library.
- project.cmake: when CONFIG_APP_REPRODUCIBLE_BUILD is also enabled, apply
the same three flags as the legacy build system to keep LTO output
reproducible: pass the prefix-map options to the linker (so link-time code
generation remaps DW_AT_comp_dir), add -save-temps (stable LTRANS object
names instead of random $TMPDIR paths in the .map), and pin -frandom-seed
(byte-identical LTO GIMPLE bytecode). See the commit message of
"feat(build): add options to enable link-time optimization (LTO)" for the
full analysis.
The gcc-ar / gcc-ranlib selection and the NO_LTO component property are shared
with the legacy build system through tools/cmake/toolchain.cmake and the common
component registration code, so no cmakev2-specific changes are needed there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sub-projects that grow their own configuration (e.g. the ULP cmakev2
flow, where the sub-project runs its own kconfgen and writes its own
sdkconfig) need a way to expose that configuration via `idf.py
menuconfig` without forcing every caller to know per-sub-project target
names.
Add a small registration + dispatcher mechanism in cmakev2:
* `idf_register_menuconfig(NAME <label> TARGET <cmake-target>)` records
a menuconfig target with the build. It is intentionally decoupled
from `idf_create_menuconfig` so that any custom target may be
registered, including the parent-side proxy targets that sub-projects
add via `externalproject_add` (e.g. ULP's `menuconfig-<app_name>`).
* `__finalize_menuconfig()` at project finalization creates the
user-facing `menuconfig` target:
- With a single registration, `menuconfig` is a thin alias for that
target. Behaviour is identical to before for all existing single
-menuconfig projects (bootloader/TEE share the main configuration,
so they do not register).
- With two or more registrations, `menuconfig` runs a small Python
dispatcher (`tools/kconfig_new/menuconfig_dispatcher.py`) that
lists the registered configurations and re-invokes the selected
target. The prompt surfaces the direct target name (e.g.
`idf.py menuconfig-ulp_app`) so users learn the per-sub-project
target after first use.
`__project_default()` is updated to register the main application as
`menuconfig-app` and then call `__finalize_menuconfig`. The component
side change is one line in `components/ulp/project_include.cmake`,
registering the existing `menuconfig-<app_name>` proxy target.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The GENERATE_SDKCONFIG build property was unconditionally set to 1 in
__init_kconfig(). Change it to use __get_default_value() so that a
value passed via -DGENERATE_SDKCONFIG=0 (e.g. from a parent build's
externalproject_add) is respected. This allows sub-projects like the
bootloader to skip writing back to the parent's sdkconfig file without
needing to explicitly set the property in their CMakeLists.txt.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Sub-projects like ULP are built as external CMake projects via
externalproject_add() and may use toolchains that are completely
different from the IDF target toolchain. For example, the ULP RISC-V
co-processor on an xtensa target (ESP32-S2/S3) uses
riscv32-esp-elf-gcc while the main project uses xtensa-esp-elf-gcc.
This causes problems in cmakev2:
1. __init_toolchain() in idf.cmake validates that CMAKE_TOOLCHAIN_FILE
matches IDF_TARGET and then overrides it. For sub-projects with
custom toolchains (passed via -DCMAKE_TOOLCHAIN_FILE from the
parent), this validation fails because the toolchain doesn't follow
the IDF naming convention (toolchain-<target>.cmake).
2. Several component project_include.cmake files use functions from
the IDF toolchain response file machinery (idf_toolchain_add_flags,
idf_toolchain_remove_flags, idf_toolchain_rerun_abi_detection) that
are not available in sub-projects with custom toolchains, or perform
validation that assumes the IDF target toolchain is in use.
Introduce IDF_CUSTOM_TOOLCHAIN, a CMake variable passed via -D from
the parent build to indicate that the toolchain was provided externally
and should not be resolved or validated by the IDF build system.
When IDF_CUSTOM_TOOLCHAIN is set:
- __init_toolchain() records the pre-set CMAKE_TOOLCHAIN_FILE and
skips IDF_TARGET-based resolution and validation.
- Component project_include.cmake files that manipulate IDF toolchain
flags (xtensa, esp_libc, esp_psram, soc) return early, as those
operations are not applicable to custom toolchains.
This is a generic mechanism usable by any sub-project that provides
its own toolchain (ULP, or any future sub-project with a non-IDF
toolchain).
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The cmakev2 path snapshots SDKCONFIG_DEFAULTS once during early init. Build
system v1 apps that mutate SDKCONFIG_DEFAULTS after `include(project.cmake)`
but before `project()` (e.g. `list(PREPEND SDKCONFIG_DEFAULTS ...)` to inject
tools/test_apps/configs/sdkconfig.debug_helpers) miss those mutations when
the shim drives the cmakev2 path, because the snapshot was taken before the
mutation.
Re-resolve SDKCONFIG_DEFAULTS at sdkconfig-generation time when
__V1_COMPAT_SHIM is active so the late-mutated value is honored. Native
cmakev2 apps still use the property-based contract via
idf_build_set_property(SDKCONFIG_DEFAULTS ... APPEND).
Promote the build system v2 functions and macros used by the
examples/build_system/cmakev2 examples to the generated API reference,
document the component-scope and version variables, the public build
properties, and the public component properties, and add cmakev2
build_property and component_property directives with dedicated Build
Properties and Component Properties sections to the esp-docs extension.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When an app sets __COMPONENT_REQUIRES_COMMON before project(), the
explicit value used to bypass the default branch that includes
${idf_target_arch}, dropping the arch component from the build.
IDF_TARGET_ARCH isn't known pre-project(), so apps shouldn't have to
hand-roll a target -> arch map. Append it after the explicit list during
common-component initialization; empty on linux means no append.
Read sdkconfig.cmake's CONFIG_* values and re-publish them on the
build-properties target so component CMakeLists.txt code that queries
Kconfig via idf_build_get_property(var CONFIG_FOO) returns the expected
value.
Add a COMMAND check on esptool_py_flash_target before invoking it in
__init_project_flash_targets. Apps that restrict the build set without
esptool_py would otherwise hit "Unknown CMake command".
Move managed-dependency injection and recursive inclusion to BEFORE the
component's add_subdirectory() call, so its CMakeLists.txt can resolve
managed-dep targets at register time. Inside idf_component_register,
union the manager-injected REQUIRES/PRIV_REQUIRES with what the
component author wrote instead of overwriting them.
Under __V1_COMPAT_SHIM, populate EXECUTABLE, EXECUTABLE_NAME, and
BUILD_COMPONENTS build properties in __project_default(), and propagate
project_elf to the caller scope. Relax the BUILD_COMPONENTS query gate
in build.cmake so component code that uses
idf_build_get_property(... BUILD_COMPONENTS) keeps working.
Co-authored-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Replicate Build system v1 behavior where the 'main' component implicitly
depends on the discovered components (or the app-restricted
__SHIM_COMPONENTS list) when no explicit REQUIRES/PRIV_REQUIRES is set.
Gated on __V1_COMPAT_SHIM so native Build system v2 projects retain
their explicit dependency contract.
Co-authored-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When IDF_BUILD_V2 evaluates to a CMake-truthy value, project.cmake
delegates to Build system v2 (cmakev2). The shim wraps project(),
forwards the app-declared COMPONENTS list via __SHIM_COMPONENTS, and
publishes __V1_COMPAT_SHIM so Build system v2 internals can gate Build
system v1 compatibility behavior. Existing app CMakeLists.txt files
build unchanged.
Co-authored-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
