change(build): drop BUILD_COMPONENTS support and migrate test_apps to not use it for buildv2
Closes IDF-15859
See merge request espressif/esp-idf!51695
The compatibility shim populated BUILD_COMPONENTS from the library
interface's linked-components list so that consumers reading it kept
working. Those consumers now query the library interface directly, so
drop the shim population and reject reads of the property
unconditionally.
ESP-IDF compiles every component with -ffunction-sections/-fdata-sections and
links with -Wl,--gc-sections, so that code and data nothing references does not
reach the image. With CONFIG_COMPILER_LTO_LINKTIME the object files carry GIMPLE
instead of machine code and code generation is deferred to the link, where those
two options no longer apply. The LTO partition is therefore emitted as a single
.text/.rodata, and since --gc-sections works at section granularity it can only
keep or drop that section as a whole. Some function in it is always live, so
nothing is dropped.
The options are recorded in the object file (in the .gnu.lto_.opts section), but
they do not reach the code generator. Code generation at link time runs as a
separate compiler invocation (LTRANS) whose command line lto-wrapper
reconstructs from those recorded options, and append_compiler_options() in
gcc/lto-wrapper.cc forwards only CL_TARGET options plus a small hard-coded list:
-fPIC/-fpic/-fPIE/-fpie, -fcommon, -fgnu-tm, -fopenmp/-fopenacc, -fcf-protection=,
(-fasynchronous-)unwind-tables, -g, -O/-Os/-Og/-Ofast/-Oz and the diagnostics
formatting options. Everything else hits the default arm and is dropped.
-ffunction-sections/-fdata-sections are neither. They are plain Common options in
common.opt without the Optimization marker, so they are not part of the
per-function state that is streamed with each function (which is why -O2 and -Os
do survive per translation unit under LTO), and they are not target options
either. They reach LTRANS only if they are repeated on the link command line.
Pass them next to -flto=auto, in both build systems.
This can be verified on any LTO build by adding -save-temps to the link options
and inspecting the generated <output>.ltrans.mk, which contains the literal
LTRANS command line:
grep -o -- "-ffunction-sections\|-fdata-sections" build/*.ltrans.mk
Measured on esp32c3 with -Os and CONFIG_COMPILER_LTO_COMPILETIME, application
binary size in bytes:
no LTO LTO LTO+fix fix saves
hello_world 116112 119792 115808 -3984 (-3.33%)
wifi/getting_started 734720 792960 732976 -59984 (-7.56%)
Without this change, enabling LTO produces a larger image than not using LTO at
all; with it, LTO is size neutral to slightly positive.
Two effects contribute. The dead code inside the partition itself stays, which
scales with how much code was compiled with LTO. On top of that, every function
retained this way keeps whatever it references alive as well, transitively and
across object boundaries: constant data, other functions, and sections of objects
that were not compiled with LTO at all. That second effect is not bounded by the
size of the LTO partition and can dominate. In the wifi/getting_started case a
single retained function, esp_crt_bundle_attach(), is the only referrer of the
mbedTLS certificate bundle, so a 55 KB .rodata blob stayed in an application that
never uses TLS. A plain non-LTO build collects all of it, which is what this
change restores.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Mirror the link-time optimization support into the cmakev2 build system so
that both build systems behave identically.
- project.cmake (__init_project_configuration): emit -flto=auto as a link
option when CONFIG_COMPILER_LTO_LINKTIME is set, except for bootloader and
ESP-TEE builds, otherwise keep -fno-lto.
- build.cmake (idf_build_library): when CONFIG_COMPILER_LTO_COMPILETIME is set,
compile each linked component with -flto=auto unless it has linker fragments,
is placed by another component's fragment (see tools/cmake/lto.cmake), has
opted out via NO_LTO, or is not a static library.
- project.cmake: when CONFIG_APP_REPRODUCIBLE_BUILD is also enabled, apply
the same three flags as the legacy build system to keep LTO output
reproducible: pass the prefix-map options to the linker (so link-time code
generation remaps DW_AT_comp_dir), add -save-temps (stable LTRANS object
names instead of random $TMPDIR paths in the .map), and pin -frandom-seed
(byte-identical LTO GIMPLE bytecode). See the commit message of
"feat(build): add options to enable link-time optimization (LTO)" for the
full analysis.
The gcc-ar / gcc-ranlib selection and the NO_LTO component property are shared
with the legacy build system through tools/cmake/toolchain.cmake and the common
component registration code, so no cmakev2-specific changes are needed there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sub-projects that grow their own configuration (e.g. the ULP cmakev2
flow, where the sub-project runs its own kconfgen and writes its own
sdkconfig) need a way to expose that configuration via `idf.py
menuconfig` without forcing every caller to know per-sub-project target
names.
Add a small registration + dispatcher mechanism in cmakev2:
* `idf_register_menuconfig(NAME <label> TARGET <cmake-target>)` records
a menuconfig target with the build. It is intentionally decoupled
from `idf_create_menuconfig` so that any custom target may be
registered, including the parent-side proxy targets that sub-projects
add via `externalproject_add` (e.g. ULP's `menuconfig-<app_name>`).
* `__finalize_menuconfig()` at project finalization creates the
user-facing `menuconfig` target:
- With a single registration, `menuconfig` is a thin alias for that
target. Behaviour is identical to before for all existing single
-menuconfig projects (bootloader/TEE share the main configuration,
so they do not register).
- With two or more registrations, `menuconfig` runs a small Python
dispatcher (`tools/kconfig_new/menuconfig_dispatcher.py`) that
lists the registered configurations and re-invokes the selected
target. The prompt surfaces the direct target name (e.g.
`idf.py menuconfig-ulp_app`) so users learn the per-sub-project
target after first use.
`__project_default()` is updated to register the main application as
`menuconfig-app` and then call `__finalize_menuconfig`. The component
side change is one line in `components/ulp/project_include.cmake`,
registering the existing `menuconfig-<app_name>` proxy target.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Promote the build system v2 functions and macros used by the
examples/build_system/cmakev2 examples to the generated API reference,
document the component-scope and version variables, the public build
properties, and the public component properties, and add cmakev2
build_property and component_property directives with dedicated Build
Properties and Component Properties sections to the esp-docs extension.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a COMMAND check on esptool_py_flash_target before invoking it in
__init_project_flash_targets. Apps that restrict the build set without
esptool_py would otherwise hit "Unknown CMake command".
Under __V1_COMPAT_SHIM, populate EXECUTABLE, EXECUTABLE_NAME, and
BUILD_COMPONENTS build properties in __project_default(), and propagate
project_elf to the caller scope. Relax the BUILD_COMPONENTS query gate
in build.cmake so component code that uses
idf_build_get_property(... BUILD_COMPONENTS) keeps working.
Co-authored-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When IDF_BUILD_V2 evaluates to a CMake-truthy value, project.cmake
delegates to Build system v2 (cmakev2). The shim wraps project(),
forwards the app-declared COMPONENTS list via __SHIM_COMPONENTS, and
publishes __V1_COMPAT_SHIM so Build system v2 internals can gate Build
system v1 compatibility behavior. Existing app CMakeLists.txt files
build unchanged.
Co-authored-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
kconfgen runs while the component manager iterates to convergence.
Those passes operate on partial component sets and emit "unknown
kconfig symbol" warnings for symbols defined in not-yet-downloaded
components — idf-build-apps treats those as build failures.
Suppress kconfgen output on the intermediate passes; only the final
pass against the converged set emits warnings.
The __init_project_configuration() function in cmakev2's project.cmake
unconditionally applied app-level compiler optimization flags based on
CONFIG_COMPILER_OPTIMIZATION_* Kconfig options. When the bootloader
subproject was built with cmakev2, these app-level flags leaked into the
bootloader compile command alongside the correct bootloader-specific
flags from CONFIG_BOOTLOADER_COMPILER_OPTIMIZATION_*.
For example, with the default configuration (app: DEBUG, bootloader:
SIZE), the bootloader received both "-Og -fno-shrink-wrap" (from app
config) and "-Os -freorder-blocks" (from bootloader config). While GCC
uses the last -O flag (-Os wins), the stray -fno-shrink-wrap persisted.
Introduce a SET_COMPILER_OPTIMIZATION build property that defaults to
YES when unset. Subprojects that manage their own optimization flags
(like the bootloader) can set this to NO before calling
idf_project_init() to prevent the default optimization flags from being
applied. This keeps project.cmake generic without requiring it to know
about specific subproject types.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_path variable was used in -fmacro-prefix-map and
-fdebug-prefix-map flags but never read from the IDF_PATH build
property, resulting in an empty substitution. This caused full
filesystem paths to leak into .rodata instead of being mapped to /IDF.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When the component manager is disabled via IDF_COMPONENT_MANAGER=0, the
cmakev2 build system still entered the full component manager flow
(__fetch_components_from_registry), which called
__download_managed_component(). That function returned early with
result=0 without creating the expected output file, violating the
post-condition in __download_component_level_managed_components() that
checks result==0 => file exists, causing a fatal error.
Instead of patching the low-level function to write an empty stub file,
this commit properly skips the entire component manager flow when the
manager is disabled:
- Move __init_component_manager() to idf.cmake global initialization
sequence alongside other __init_*() calls, so IDF_COMPONENT_MANAGER
build property is available early.
- Set __SDKCONFIG_ORIG to the real sdkconfig path in __init_kconfig()
as its default value. Previously it was only set inside
__create_sdkconfig_orig_copy() and had a defensive fallback in
__create_base_kconfgen_command(). The default ensures __SDKCONFIG_ORIG
is always valid regardless of whether the component manager runs.
- Guard __create_sdkconfig_orig_copy() with an IDF_COMPONENT_MANAGER
check. The sdkconfig backup exists solely to preserve unknown Kconfig
options from managed components during intermediate kconfgen rounds.
When the manager is disabled, no managed components exist, so the
backup is unnecessary.
- Guard __fetch_components_from_registry() call in project.cmake behind
IDF_COMPONENT_MANAGER == 1. When disabled, only the manifest warning
is issued. No download loop runs, no temp files are created, and no
"Component manager round N..." messages are printed.
- Remove the now-redundant IDF_COMPONENT_MANAGER guard from
__download_managed_component(), since it is only reachable when the
manager is enabled.
Closes https://github.com/espressif/esp-idf/issues/18372
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In cmake v1, __build_process_project_includes() exports all build
properties as CMake variables before including project_include.cmake
files. cmakev2 was missing this step, causing components like ULP that
reference build properties as CMake variables (e.g. ${SDKCONFIG_HEADER})
to receive empty values.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit introduces a new build property, __OPTIONAL_REQUIRES_MODE,
and uses it to either defer or link immediately, optional requirements
to components that request such linkage via the
idf_component_optional_requires() function in build system v2. The
DEFERRED mode is intended for single-binary projects where in the linking
of optional components happens after the library target is created the
dependency graph is available to the build system, thereby allowing it to
behave like the v1 version of the function.
Made-with: Cursor
When MAPFILE_TARGET is used in idf_build_executable, add the linker
--cref option so the cross-reference table is written to the map file
instead of stdout. Remove the global -Wl,--cref from default link
options in project.cmake so cref is only applied where a map file
is requested.
The sdkconfig file may contain configuration options defined in Kconfig
files of managed components. Since kconfgen runs before the component
manager fetches these components, the Kconfig definitions for managed
component options are not yet available. The kconfgen --output config
flag regenerates sdkconfig from kconfiglib's internal state, which only
knows about options with loaded Kconfig definitions. This causes unknown
options (i.e., those from managed components) to be silently dropped
from sdkconfig during intermediate regeneration rounds.
Note that kconfgen's --config flag (used for reading sdkconfig) only
performs deprecated option name replacement and does NOT drop unknown
options. The problem is exclusively in --output config, which writes a
fresh sdkconfig from the parsed Kconfig tree state.
Fix this by introducing a __SDKCONFIG_ORIG build property that provides
an indirection layer for the --config input path:
- Before the component manager runs: __SDKCONFIG_ORIG points to a copy
of the original sdkconfig (build/sdkconfig.orig), created by the new
__create_sdkconfig_orig_copy() function. This copy preserves all
original options, including those from managed components.
- During intermediate kconfgen runs: --config reads from the preserved
copy (so unknown options survive as input), while --output config
writes to the real sdkconfig (unknown options may be dropped there,
but this is harmless since kconfgen always reads from the copy).
- After the component manager completes: __SDKCONFIG_ORIG is reset to
point to the real sdkconfig and __BASE_KCONFGEN_CMD is rebuilt, so
that subsequent operations (menuconfig, save-defconfig, confserver)
read and write the actual sdkconfig file directly.
The flow is:
__create_sdkconfig_orig_copy()
-> __SDKCONFIG_ORIG = build/sdkconfig.orig
__generate_sdkconfig()
-> --config build/sdkconfig.orig --output config project/sdkconfig
__fetch_components_from_registry():
loop:
download_components()
__generate_sdkconfig()
-> --config build/sdkconfig.orig --output config project/sdkconfig
if success: break
endloop
-> __SDKCONFIG_ORIG = project/sdkconfig
-> rebuild __BASE_KCONFGEN_CMD
idf_create_menuconfig() / save-defconfig / confserver
-> uses --config project/sdkconfig (the real file)
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, idf_build_generate_metadata only accepts binary targets for
which it generates metadata (project_description.json). On Linux
targets, binary images are not generated, but we still need to generate
project_description.json. Extend the current function to accept both
executable and binary targets and ensure project_description.json is
generated when a Linux target is used.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the CONFIG_APP_BUILD_GENERATE_BINARIES option is used to
determine whether binary images will be generated in
idf_project_default. At present, this option is enabled even for the
Linux target, where the esptool_py component is not included, preventing
the generation of binary images. To address this issue, verify if the
target for esptool_py is present.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The IDF_TOOLCHAIN build property is currently incorrectly set to the
default `gcc` value for the linux target, whereas it should be empty.
This misconfiguration causes confusion for components like `soc`, which
adjust toolchain options based on the
IDF_TOOLCHAIN(CONFIG_IDF_TOOLCHAIN_GCC) build property's setting.
When sdkconfig is generated, the IDF_TOOLCHAIN build property is passed
as an environmental variable to kconfgen, and the CONFIG_IDF_TOOLCHAIN
configuration option is set based on this variable. Additionally, the
CONFIG_IDF_TOOLCHAIN_GCC and CONFIG_IDF_TOOLCHAIN_CLANG configuration
options are set accordingly. Subsequently, CONFIG_IDF_TOOLCHAIN_GCC is
used in several places, such as `components/soc/project_include.cmake`,
to configure the toolchain (compiler flags) by invoking functions from
`tools/cmake/toolchain_flags.cmake`, which is included only for
non-linux targets. As a result the configuration fails, because
functions from `tools/cmake/toolchain_flags.cmake` are not available on
linux target.
Since the IDF_TOOLCHAIN cmake cache variable is actually set in the
`tools/cmake/toolchain.cmake` file, the IDF_TOOLCHAIN build property
should be set after the toolchain is initialized in cmakev2's project
initialization. Note that each toolchain file, except for linux,
includes `toolchain.cmake`, which in turn includes
`toolchain_flags.cmake`. This means the IDF_TOOLCHAIN cmake cache
variable is set for every target except linux, because the toolchain
file for linux is empty. As a result CONFIG_IDF_TOOLCHAIN is empty and
CONFIG_IDF_TOOLCHAIN_GCC not set as for cmakev1.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In some instances, the COMPONENTS_DISCOVERED build property is used to
walk through the component list and obtain component properties, such as
during configuration preparation or the inclusion of project_include
files. Since we know the component interfaces from the
COMPONENT_INTERFACES build property, we can switch to the faster
`__idf_component_get_property_unchecked` method to obtain component properties
in these cases. This change reduces the reconfiguration time by 0.4
seconds on my computer.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit removes picolibc flags from project.cmake file since they
are added by the esp_libc component when project_include.cmake files are
evaluated.
This commit moves the project initialization by moving the
__init_project_configuration() after the sdkconfig is generated and
included. This is because __init_project_configuration() depends on
config options.
The idf_build_generate_depgraph function creates a component dependency
graph in dot (graphviz) format for a specified executable. It uses
existing helper functions from cmakev1, ensuring that the generated dot
files are produced in the same manner as in cmakev1. While adjustments
might be needed in the future if necessary, the current implementation
is intended to offer the same functionality as cmakev1. Similar to
cmakev1, the dot files are only generated only when the
__BUILD_COMPONENT_DEPGRAPH_ENABLED build property is set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_create_size_report function allows for the creation of size
report targets based on the generated link map file. The size report
targets are created using the TARGET option name: "<target>",
"<target>-files", and "<target>-components". These size report targets
are added to the idf_default_project with the TARGET set to "size",
resulting in the creation of "size", "size-files", and "size-components"
targets for the default project.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_project_init macro is evaluated within the global variable
scope, and currently, there are a few variables still set. Let's ensure
all variables are properly unset to avoid polluting the global variable
namespace.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The link flag for generating the map file is currently added globally to
the link_options. If multiple executables are generated, the link map
file is overwritten by the last created executable. Since cmakev2
supports building multiple executables, the link map file options need
to be set for each executable individually. To address this, add a new
MAPFILE_TARGET option to the idf_build_executable function. When set,
the link map file will be generated by the linker, and a target
specified in the MAPFILE_TARGET option will be created for the map file.
This also splits the idf_project_default macro. Only the
idf_project_init macro needs be called within the global scope, as it
includes the project_include.cmake files and the cmake version of the
configuration. The remaining functionality of the idf_project_default
macro is implemented in a __project_default helper function to avoid
polluting the global variable space.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the `idf_build_generate_metadata` function takes an
executable target as an argument, and the binary name is stored in the
`EXECUTABLE_BINARY` property. This approach is inconvenient because a
single executable might generate multiple binary images, making it
unreliable to store the binary image name in the executable property due
to the N:1 relationship. To address this, pass the binary image target
instead of the executable target to the `idf_build_generate_metadata`
function. This change is facilitated by linking the binary target,
executable target, and library interface targets with the following
properties:
binary:EXECUTABLE_TARGET -> executable:LIBRARY_INTERFACE -> library
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
With commit b719292b75 ("refactor(build-system): Simplify flash target creation"),
partition table and bootloader components are correctly adding their
dependency on the flash target, so there is no need for this to be done
in the build system. This removes the temporary workaround.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit adds support for uf2 targets for cmake2. The following
changes have been made:
- Adds a new tools/cmakev2/uf2.cmake.
- Adds the idf_create_uf2() function to create the uf2 targets. This
function now takes the executable as an argument thus allowing the uf2
target to be created per-executable.
- idf_project_default() is updated to create the uf2 targets.
This commit adds support for dfu targets for cmake2. The following
changes have been made:
- Adds a new tools/cmakev2/dfu.cmake file on similar lines as the
tools/cmake/dfu.cmake file.
- Adds the idf_create_dfu() function to create the dfu targets. This
function now takes the executable as an argument thus allowing the dfu
target to be created per-executable.
- idf_project_default() is updated to create the dfu target.
The COMPONENTS and EXCLUDE_COMPONENTS variables are unused in cmakev2. A
deprecation warning will be printed if they are set. This change also
includes a helper functions for printing deprecation warnings.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The cmakev1 dropped generation of prefix_map_gdbinit, reflect this
change also in cmakev2.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, we include numerous functions in the automatically generated
documentation for the build system API. Let's begin with only the
essential functions and gradually add more to the API based on requests
and actual needs.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
