ldgen skips regeneration when a fingerprint over the section names, the
linker fragment files, sdkconfig and Kconfig is unchanged. The linker
script template was not part of that fingerprint, even though everything
outside its mapping placeholders is copied into the generated script
verbatim.
The build system regenerates the preprocessed template whenever any of
its inputs changes, not only when a linker fragment changes. When such a
change reaches ldgen but leaves the section names alone, the fingerprint
still matches, generation is skipped, and the image is linked against the
previously generated script. The skip path updates the mtime of the
output, so nothing in the build output shows that the change was dropped.
Add the template to the fingerprint so that regenerating it always
regenerates the linker script.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The cxx component re-injects its own archive late on the link line so its
stack-unwind wrappers take precedence over libgcc's (when C++ exceptions
are disabled). It does this through an INTERFACE helper, libgcc_cxx, that
carries `$<TARGET_FILE:${cxx}>`. That helper was linked back into the cxx
target with PUBLIC, which places it in the cxx archive's own
LINK_LIBRARIES, so the cxx target ends up transitively referencing its
own output file.
When CMake computes the cxx target's sources it walks the target's own
link implementation, encounters `$<TARGET_FILE:${cxx}>`, and to resolve the
file it needs the target's link language, which in turn needs its
sources. This loop makes generation fail with:
The SOURCES of "..." use a generator expression that depends on the
SOURCES themselves.
CMake only hits this when it reaches the cxx target standalone before the
target has been pulled into a link by a consumer, and whether that
happens depends on the project layout and target evaluation order. That
is why it surfaces only in some projects, while the examples build fine.
Link libgcc_cxx as INTERFACE instead of PUBLIC. A static archive is not
linked, so its private LINK_LIBRARIES are inert; consumers still receive
libgcc_cxx through INTERFACE_LINK_LIBRARIES, so the intended link order is
preserved. Removing the self-reference makes generation independent of
evaluation order.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
On some Windows systems, antivirus, endpoint-security or DLP/encryption
software intercepts short-lived toolchain processes and strips their
stdout when the build captures it through a pipe, while the same command
prints its output normally when run by hand. The tool exits successfully
but returns nothing, and each build step that reads toolchain output then
failed with a different, cryptic error far from the real cause:
- CMake configuration aborted with "Unknown arguments specified" in
components/xtensa/project_include.cmake, or "check_expected_tool_version
invoked with incorrect arguments" in components/esp_common.
- ldgen turned the empty objdump output into an opaque pyparsing
"Expected 'In archive'" traceback.
- idf_tools.py silently reported the compiler/debugger version as
"unknown", sending users into a fruitless reinstall loop.
Detect the empty result at each consumer and fail (or warn) with an
actionable message that names the likely cause and the remedy:
- tools/cmake/compiler_query.cmake: new __compiler_query() helper runs a
compiler query and fails with a clear error on empty or failed output.
It is a standalone module included by both the cmakev1 and cmakev2
utilities, since the esp_common and xtensa project_include.cmake that
call it are shared by both build systems. The xtensa if() arguments are
now quoted so an empty result no longer collapses into a parse error.
- tools/ldgen/ldgen.py: _run_objdump() rejects empty objdump output, and
non-empty-but-unparsable section info is caught and re-raised as a clear
LdGenFailure instead of a raw pyparsing traceback.
- tools/idf_tools.py: empty version output now warns with the cause and
returns UNKNOWN_VERSION instead of silently reporting "unknown".
Closes https://github.com/espressif/esp-idf/issues/18727
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This file predates the adoption of ruff-format and was never reformatted,
because pre-commit only runs on the files touched by a change. GitHub PR
#18871 bumps the copyright year here, which pulls the file into the hook's
scope and makes check_pre_commit fail on a pre-existing formatting
violation unrelated to that change.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Build ULP full subprojects through a dedicated entry file,
components/ulp/cmake/ulp_project.cmake, that wraps tools/cmakev2/idf.cmake
and layers a small ULP API on top, mirroring the cmakev2 layering:
ulp_project_init like idf_project_init (init, detect the ULP type,
reset the compile/link options inherited from the app)
ulp_build_executable like idf_build_executable, plus the embeddable
.bin/.h/.ld artifacts
ulp_project_default like idf_project_default (single-executable case)
A child project now includes this one file instead of idf.cmake and calls
these helpers directly, so idf_build_executable is used as-is for the
multi-binary case and the module-path indirection (include(IDFULPProject)
resolved via -DCMAKE_MODULE_PATH) is gone.
As a result:
- IDFULPProjectv2.cmake is removed; its setup moves into the wrapper.
- IDFULPProject.cmake becomes the CMake v1-only entry point.
- The ULP component no longer registers a POST_ELF callback; the binary
artifacts are produced by ulp_build_executable.
- The v2 full-subproject examples (lp_core, riscv, fsm, multi_binary,
combined) are updated to the new API.
- The ULP subproject API is documented in build-system-v2.rst.
Also fix a latent bug this exercises: idf_build_library emitted linker
scripts as "-T <name>" relying on a following "-L" search directory. GNU
ld only searches -L directories that precede -T, so the direct
esp32ulp-elf-ld link used for ULP FSM failed to open the script. Emit the
absolute path instead, matching what the CMake v1 ULP build already does.
Finally, replace the parent-argument bypass loop that used to live in
IDFULPProjectv2.cmake with --no-warn-unused-cli on the child configure,
and stop passing the unused IDF_PARENT_BUILD_DIR.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Rename the ULP memory-layout linker scripts to .ld.in and let the
standard build system preprocess them, instead of the ULP build doing
its own C-preprocessor pass.
CMake v2 (full subproject): register each memory-layout template with
target_linker_script using the new FLAGS (the include dirs its #include
lines need) and MEMORY (emit before section-placement scripts) options.
soc/soc.h and the esp_common headers now resolve from the linked
component graph automatically, so the POST_ELF callback no longer
harvests include directories or preprocesses the template; it only
produces the .bin and the symbol header/linker exports from the ELF.
CMake v1 (legacy): the templates are now .ld.in, so strip the suffix
with NAME_WLE when deriving the generated script name. The v1-only
preprocessing helpers (__ulp_add_preprocessed_linker_script and
__ulp_create_arg_file) move from the shared IDFULPProjectCommon.cmake
to IDFULPProject.cmake next to their only caller, since CMake v2 no
longer references them.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Resolve #include lines in linker script templates against the linked
component graph: idf_build_library appends every linked component's
INCLUDE_DIRS to the C preprocessor invocation for each .in linker
script, so a template can include any component header (e.g. a ULP
memory-layout template including soc/soc.h) without the owning
component having to know about it.
Also:
- Always pass -I<config_dir> so templates can include sdkconfig.h.
- Add a FLAGS option to target_linker_script for explicit preprocessor
flags (e.g. -D__ASSEMBLER__ or ld-snippet include dirs); when given,
FLAGS replaces the parent-dir==target include heuristic for that
script.
- Add a MEMORY option to target_linker_script that emits the marked
linker script as -T before all others, so section-placement scripts
in one component can reference MEMORY regions and REGION_ALIASes
declared by a memory-layout script in another component.
- Make C-preprocessor comment keeping (-C) part of the default flag set
rather than hardcoded, so FLAGS can drop it. linker_script_preprocessor
no longer forces -C; both the cmakev1 and cmakev2 preprocessors add it
to their default CFLAGS (output unchanged for existing scripts). A ULP
template that includes soc/soc.h omits -C so ld does not choke on the
header's // comments.
- Store per-script metadata (generated output, flags) in MD5-keyed
component properties instead of parallel lists.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The ULP sub-project's editable sdkconfig and dependencies.lock previously
landed in the source tree (the defaults of __init_kconfig and the
component manager). A typical cleanup before changing target (rm -rf
build/ sdkconfig) did not reach them, so a stale CONFIG_IDF_TARGET would
survive into the next build and trip the cross-check in the ULP's
__init_idf_target.
In the cmakev2 path of __setup_ulp_project, anchor the ULP binary dir to
${BUILD_DIR}/subprojects/${app_name} and pass -DSDKCONFIG and
-DDEPENDENCIES_LOCK so both files live alongside the rest of the ULP
build artifacts. Wiping build/ (or idf.py fullclean, which set-target
pulls in) now reverts all sub-project state in one shot.
Also add a safety-net rename of any pre-existing sub-project sdkconfig
to .old when _IDF_PY_SET_TARGET_ACTION=1 is set in the parent's
configure environment. This mirrors what cmakev1 does for its own
sdkconfig and covers the off-path case where someone reconfigures via
cmake -B build -DIDF_TARGET=... directly, without running fullclean. In
the normal idf.py set-target flow the rename is a no-op because
fullclean has already removed the build tree.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Sub-projects that grow their own configuration (e.g. the ULP cmakev2
flow, where the sub-project runs its own kconfgen and writes its own
sdkconfig) need a way to expose that configuration via `idf.py
menuconfig` without forcing every caller to know per-sub-project target
names.
Add a small registration + dispatcher mechanism in cmakev2:
* `idf_register_menuconfig(NAME <label> TARGET <cmake-target>)` records
a menuconfig target with the build. It is intentionally decoupled
from `idf_create_menuconfig` so that any custom target may be
registered, including the parent-side proxy targets that sub-projects
add via `externalproject_add` (e.g. ULP's `menuconfig-<app_name>`).
* `__finalize_menuconfig()` at project finalization creates the
user-facing `menuconfig` target:
- With a single registration, `menuconfig` is a thin alias for that
target. Behaviour is identical to before for all existing single
-menuconfig projects (bootloader/TEE share the main configuration,
so they do not register).
- With two or more registrations, `menuconfig` runs a small Python
dispatcher (`tools/kconfig_new/menuconfig_dispatcher.py`) that
lists the registered configurations and re-invokes the selected
target. The prompt surfaces the direct target name (e.g.
`idf.py menuconfig-ulp_app`) so users learn the per-sub-project
target after first use.
`__project_default()` is updated to register the main application as
`menuconfig-app` and then call `__finalize_menuconfig`. The component
side change is one line in `components/ulp/project_include.cmake`,
registering the existing `menuconfig-<app_name>` proxy target.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add examples/system/ulp/lp_core/build_system_v2/ demonstrating the
cmakev2 ULP build with native IDF component support, Kconfig
integration, component manager usage, linker scripts (static and
ldgen-processed), and per-project menuconfig.
The example shows a custom ULP project (via ulp_add_project) structured
as a proper IDF-like project with a main component and
idf_build_executable(). The ULP sub-project runs its own kconfgen with
the parent's sdkconfig as SDKCONFIG_DEFAULTS, inheriting the parent's
IDF configuration while adding ULP-specific Kconfig options from its
own components.
Three components demonstrate different integration mechanisms:
- ulp_sum and ulp_math: pure cmakev2 components in the components/
subdirectory, using add_library(${COMPONENT_TARGET}) directly.
Each provides a Kconfig file with configurable options.
- ulp_clamp: an external component in the extra/ subdirectory,
declared as a local path dependency in ulp_sum/idf_component.yml.
The component manager resolves this dependency automatically.
ulp_clamp also provides a Kconfig file with a configurable
maximum clamp value (CONFIG_ULP_CLAMP_MAX_VALUE).
Linker script support is demonstrated in two ways:
- Static linker script: ulp_math registers ulp_math_sections.ld via
idf_component_set_property(LINKER_SCRIPTS), defining a custom
linker symbol (ulp_math_version).
- ldgen-processed template: ulp_math provides a linker fragment file
(ulp_math.lf) that describes a custom .ulp_math_fast section
placement. The main component registers ulp_sections.ld.in via
target_linker_script(PROCESS), which ldgen processes using the
fragment to generate the final linker script. The
ulp_math_multiply() function is placed in .ulp_math_fast using
__attribute__((section)). Both static and ldgen linker scripts
are handled by the standard idf_build_library() flow without any
ULP-specific linker script functions.
The ULP project has its own sdkconfig.defaults that overrides the
ulp_clamp default (100 instead of 255), demonstrating per-project
configuration layering. Users can append to SDKCONFIG_DEFAULTS
before include(idf.cmake) to add project-specific defaults.
A menuconfig-<app_name> target is registered in the parent build,
allowing ULP-specific configuration via:
idf.py --no-hints menuconfig-ulp_build_system_example
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The GENERATE_SDKCONFIG build property was unconditionally set to 1 in
__init_kconfig(). Change it to use __get_default_value() so that a
value passed via -DGENERATE_SDKCONFIG=0 (e.g. from a parent build's
externalproject_add) is respected. This allows sub-projects like the
bootloader to skip writing back to the parent's sdkconfig file without
needing to explicitly set the property in their CMakeLists.txt.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add cmakev2 support for the ULP sub-project, allowing it to be built
using the standard idf_project_init() flow when the main project uses
cmakev2. The existing cmakev1 path is preserved — the ULP
CMakeLists.txt checks IDF_BUILD_V2 and delegates to CMakeLists_v2.txt,
following the same pattern used by the bootloader sub-project.
The parent's project_include.cmake now has separate externalproject_add
blocks for cmakev1 and cmakev2, making the different requirements of
each build system explicit:
cmakev1 path:
Passes -DSDKCONFIG_HEADER and -DSDKCONFIG_CMAKE from the parent.
The ULP sub-project includes the parent's sdkconfig.cmake directly
(no kconfgen). This is the existing behavior.
cmakev2 path:
Passes -DSDKCONFIG_DEFAULTS pointing to the parent's sdkconfig.
The ULP sub-project runs its own kconfgen with its own component
set, generating separate sdkconfig.h and sdkconfig.cmake outputs.
The parent's sdkconfig serves as defaults, so the ULP inherits
the parent's IDF configuration while adding ULP-specific options
from its own components. A menuconfig-<app_name> proxy target is
registered in the parent build for ULP-specific configuration.
The proxy target depends on the ULP configure step to ensure the
cmake cache exists before menuconfig runs.
Passes -DIDF_CUSTOM_TOOLCHAIN=1 so that __init_toolchain() skips
IDF_TARGET-based toolchain resolution and component
project_include.cmake files skip IDF toolchain flag manipulation.
IDFULPProject.cmake is updated to skip the direct
include(${SDKCONFIG_CMAKE}) when running under cmakev2, as
idf_project_init() handles sdkconfig loading.
Verified with identical ULP artifacts (bin, ld, h) across all three
ULP types:
- LP Core (esp32c6, lp_core/lp_uart/lp_uart_print example)
- RISC-V (esp32s3, ulp_riscv/adc example)
- FSM (esp32, ulp_fsm/ulp example)
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Sub-projects like ULP are built as external CMake projects via
externalproject_add() and may use toolchains that are completely
different from the IDF target toolchain. For example, the ULP RISC-V
co-processor on an xtensa target (ESP32-S2/S3) uses
riscv32-esp-elf-gcc while the main project uses xtensa-esp-elf-gcc.
This causes problems in cmakev2:
1. __init_toolchain() in idf.cmake validates that CMAKE_TOOLCHAIN_FILE
matches IDF_TARGET and then overrides it. For sub-projects with
custom toolchains (passed via -DCMAKE_TOOLCHAIN_FILE from the
parent), this validation fails because the toolchain doesn't follow
the IDF naming convention (toolchain-<target>.cmake).
2. Several component project_include.cmake files use functions from
the IDF toolchain response file machinery (idf_toolchain_add_flags,
idf_toolchain_remove_flags, idf_toolchain_rerun_abi_detection) that
are not available in sub-projects with custom toolchains, or perform
validation that assumes the IDF target toolchain is in use.
Introduce IDF_CUSTOM_TOOLCHAIN, a CMake variable passed via -D from
the parent build to indicate that the toolchain was provided externally
and should not be resolved or validated by the IDF build system.
When IDF_CUSTOM_TOOLCHAIN is set:
- __init_toolchain() records the pre-set CMAKE_TOOLCHAIN_FILE and
skips IDF_TARGET-based resolution and validation.
- Component project_include.cmake files that manipulate IDF toolchain
flags (xtensa, esp_libc, esp_psram, soc) return early, as those
operations are not applicable to custom toolchains.
This is a generic mechanism usable by any sub-project that provides
its own toolchain (ULP, or any future sub-project with a non-IDF
toolchain).
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Document the create --format option (SPDX 2.2 tag/value and JSON, SPDX 3.0.1
JSON-LD, and CycloneDX 1.6) and the new --file option on idf.py sbom-create and
sbom-check, and generalize the previously SPDX-only wording. Note the NVDAPIKEY
variable for faster online scans, document that sbom-check exits with a non-zero
status when vulnerabilities are found, drop an unused link target, and update the
SBOM reference link. Fix the section heading levels to follow the esp-docs
conventions.
The build system v2 guide now references these examples with :example:
and :example_file: roles, so check_examples_documented.py no longer
expects them in the KNOWN_MISSING list. component_manager and the nested
import_prebuilt/prebuilt project are kept, since the guide does not link
them.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Split the single build-system-v2 page into a landing page that dispatches
to focused how-to guides (creating a project, creating components,
configuring dependencies, third-party libraries, ESP-IDF as a library,
multiple binaries) and reference pages (design and architecture, breaking
changes, and the generated API reference). Mirror the new structure on
zh_CN and update the API Guides table of contents.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Promote the build system v2 functions and macros used by the
examples/build_system/cmakev2 examples to the generated API reference,
document the component-scope and version variables, the public build
properties, and the public component properties, and add cmakev2
build_property and component_property directives with dedicated Build
Properties and Component Properties sections to the esp-docs extension.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When a component's manifest declared a managed dep `<ns>/<name>` and the
project had a local `components/<name>` shadowing it, configure failed
with `IDF: Failed to resolve component '<ns>__<name>'`.
cmakev1 does a project-wide injection so the manager's _choose_component
sees every component name in known_components and rewrites the
namespaced dep to its locally-shadowing short name. cmakev2 invokes the
manager per component with a single-component requirements file, so
known_components has only one entry and the rewrite cannot fire --
`<ns>__<name>` is written verbatim into MANAGED_REQUIRES and the
force-include at component.cmake:995 aborts.
Seed the per-component requirements file with one __COMPONENT_SOURCE
entry per discovered component so handle_project_requirements() builds
the same known_components list cmakev1 produces. The seeded entries are
inert on the cmake side -- the __component_set_property shim ignores
__COMPONENT_SOURCE -- they exist purely to feed the manager's rewrite.
Add a regression test in tools/test_build_system/buildv2/test_component.py.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add an OPTIONAL flag to idf_component_include so callers can ask
"include this component if it exists" without aborting the build
when the component is unknown.
Without OPTIONAL, the function behaves exactly as before -- it
calls __get_component_interface_or_die and the build aborts on a
miss. With OPTIONAL, the function performs a non-fatal interface
lookup and returns silently if the component is not known. When
combined with INTERFACE <variable>, the variable is set to the
empty string on miss and to the component's interface target on
hit, so callers can write:
idf_component_include(button OPTIONAL INTERFACE button_iface)
if(button_iface)
target_sources(${COMPONENT_TARGET} PRIVATE button_glue.c)
target_link_libraries(${COMPONENT_TARGET} PRIVATE ${button_iface})
endif()
This gives consumers a public, non-fatal way to wire up integrations
with components that may or may not be in the build (managed
dependencies pulled in only by some board configurations, optional
feature glue, etc.) without reaching into the private
__get_component_interface helper or doing dual-namespace
COMPONENTS_DISCOVERED checks against both <name> and <ns>__<name>
forms.
Backward compatible: existing call sites do not pass OPTIONAL and
continue to fail loudly on miss.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The bootloader subproject's full Kconfig discovery resolves CONFIG_SPIRAM=y
when the parent app has it enabled, even though esp_psram is not linked
into the bootloader (the CMake gate is
'if(NOT non_os_build) if(CONFIG_SPIRAM) idf_component_optional_requires(PRIVATE esp_psram)').
Shared sources in esp_hw_support that #include esp_psram private headers
or call esp_psram functions guarded only by '#if CONFIG_SPIRAM' then fail
to compile in the bootloader with
"fatal error: esp_private/esp_psram_extram.h: No such file or directory".
Mirror the CMake gate in source guards: every '#if CONFIG_SPIRAM' block in
a bootloader-compiled source that touches esp_psram becomes
'#if !BOOTLOADER_BUILD && CONFIG_SPIRAM'. The leaked CONFIG_SPIRAM value
in the bootloader's sdkconfig.h is then harmless because every dependent
block evaluates to false.
Sites updated:
- esp_memory_utils.c: include of esp_psram_extram.h and all
esp_psram_check_ptr_addr() call sites
- port/esp32{c5,c61,p4,s31}/cpu_region_protect.c: include of
esp_psram_extram.h (inner SPIRAM_FETCH/RODATA/PRE_CONFIGURE blocks are
already inside outer !BOOTLOADER_BUILD guards)
- port/esp32{p4,s31}/pmu_sleep.c: include of esp_psram_impl.h and
esp_psram_impl_{enter,exit}_halfsleep_mode() call sites
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The build system currently suffers from a bug where custom configuration
settings are lost when the component manager is invoked multiple times.
During an initial reconfigure or set-target command, the component
manager may return no managed components. Because the build system lacks
the Kconfig definitions for these components at this stage, it
automatically prunes any related configuration symbols—specifically
those defined in managed component Kconfig files—from the sdkconfig
file. By the time the component manager finishes downloading the
dependencies in a subsequent run, these original settings have already
been overwritten and discarded because they were previously unrecognized
by the build system.
To resolve this, the generation of the final sdkconfig should be
deferred until all component manager resolution cycles are complete. We
need to ensure that the build system does not treat missing Kconfig
definitions as invalid until the full dependency graph is loaded. A
potential fix involves using a temporary configuration file for
intermediate component manager passes to prevent the main sdkconfig from
being prematurely scrubbed of valid user settings that belong to
yet-to-be-resolved components.
Edit:
Added cleanup of sdkconfig.cm file and comments
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Edited by: Daniel Paul <daniel.paul@espressif.com>
Two tests added in commit c21d05e612 (MR !46354 "Added cmake target
listing in help") fail under pytest_buildv2_system /
pytest_buildv2_system_win:
* test_idf_py_help_after_configure_with_no_custom_targets_has_no_section
* test_idf_py_help_lists_cmake_custom_targets_after_configure
The tests assert that internal phony build targets do not appear
under "CMake Custom Targets" in idf.py --help. Filtering is driven
by tools/idf_py_actions/help_custom_targets_skip.py, whose exact-name
allowlist and shape policy were calibrated against cmakev1 target
names. cmakev1 either hard-codes the names ("app_check_size") or
uses a prefix already covered by HELP_PHONY_NAME_PREFIXES
("gen_<project>_binary"), so v1 stays out of the section.
cmakev2 parameterizes the same logical targets by the project name
in tools/cmakev2/build.cmake and tools/cmakev2/project.cmake and
produces, for project "build_test_app":
build_test_app_binary
build_test_app_binary_check_size
build_test_app_mapfile
build_test_app.map
ldgen_libraries.in_library_build_test_app
None of these match the v1-shaped allowlist or shape policy, so
both tests fail on every cmakev2 project rather than only on this
test app. Skip them under buildv2 with @pytest.mark.buildv2_skip
to unblock CI while a more robust filtering strategy is worked out.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Profiling shows that parsing the linker fragment (.lf) files with
pyparsing is the single largest cost in ldgen (~58% of total runtime).
parse_fragment_file() rebuilds its grammar from scratch on every call
because the conditional parser closure has to capture sdkconfig to
evaluate if/elif/else at parse time.
Add a cache that pickles the parsed FragmentFile object list to
<output>.lfcache. The cache is keyed on fragment file paths+mtimes
plus sdkconfig and kconfig mtimes — sdkconfig is included because
fragment if/elif/else blocks are evaluated against it at parse time.
This avoids the pyparsing cost on rebuilds where ldgen has to run
but the fragment files themselves are unchanged — for example when
a function is added or removed in a source file. Cache hits produce
the same sections.ld as a full parse, and any pickle load failure
falls through to a normal parse, so cache corruption or a Python
upgrade cannot produce a wrong build.
Print "Skipping linker fragment parsing, fragment files unchanged"
on the lf cache hit path so build logs show when the optimization
fired — this makes it easy to diagnose user reports of unexpected
ldgen behavior.
The LDGEN_NO_CACHE environment variable disables the lf cache. When
set, ldgen falls through to a normal parse without consulting or
writing the cache file.
Measured on wifi_station (205 libs, 67 .lf files), median of 10 runs:
before after
rebuild, section change, .lf unchanged 5.82s 1.34s
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
After running objdump on all libraries, extract section names from the
stored raw output using regex and compute a fingerprint (MD5 hash of
section names + mtimes of fragment files, sdkconfig, and kconfig).
If the fingerprint matches the cached value from the previous run,
skip the expensive pyparsing + generation step entirely and touch
the output file to update its timestamp. The fingerprint file is
stored next to the output (e.g. sections.ld.fingerprint) in the build
directory.
This avoids the pyparsing bottleneck for the common case of editing
function bodies without adding or removing symbols — section names
in the object files are unchanged, so the generated sections.ld
would be identical. When sections do change, the full generation
runs and the fingerprint is updated.
Print "Skipping linker script generation, section names unchanged"
on the fingerprint hit path so build logs show when the optimization
fired. This makes it easy to diagnose user reports of unexpected
ldgen behavior — the build log directly shows whether the cache was
used.
Add an LDGEN_NO_CACHE environment variable to disable the cache as a
workaround in case the optimization causes problems. When set, ldgen
prints "Linker script generation caches disabled by LDGEN_NO_CACHE"
and runs full generation as before, leaving any existing fingerprint
file untouched.
Measured on wifi_station (205 libs, 67 .lf files), median of 10 runs:
before after
rebuild, section names unchanged 5.82s 0.82s
Closes https://github.com/espressif/esp-idf/issues/18408
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When CONFIG_SECURE_BOOT_V2_ENABLED=y but
CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES is not set, produce the
binary directly as bootloader.bin instead of bootloader-unsigned.bin.
This matches the v1 behavior where the intermediate binary name is
conditional: bootloader-unsigned.bin only when build-time signing is
enabled (so the signed output can be named bootloader.bin), otherwise
the output is bootloader.bin directly.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The __init_project_configuration() function in cmakev2's project.cmake
unconditionally applied app-level compiler optimization flags based on
CONFIG_COMPILER_OPTIMIZATION_* Kconfig options. When the bootloader
subproject was built with cmakev2, these app-level flags leaked into the
bootloader compile command alongside the correct bootloader-specific
flags from CONFIG_BOOTLOADER_COMPILER_OPTIMIZATION_*.
For example, with the default configuration (app: DEBUG, bootloader:
SIZE), the bootloader received both "-Og -fno-shrink-wrap" (from app
config) and "-Os -freorder-blocks" (from bootloader config). While GCC
uses the last -O flag (-Os wins), the stray -fno-shrink-wrap persisted.
Introduce a SET_COMPILER_OPTIMIZATION build property that defaults to
YES when unset. Subprojects that manage their own optimization flags
(like the bootloader) can set this to NO before calling
idf_project_init() to prevent the default optimization flags from being
applied. This keeps project.cmake generic without requiring it to know
about specific subproject types.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_path variable was used in -fmacro-prefix-map and
-fdebug-prefix-map flags but never read from the IDF_PATH build
property, resulting in an empty substitution. This caused full
filesystem paths to leak into .rodata instead of being mapped to /IDF.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The bootloader does not use the heap allocator or TLSF, so the ROM
patch files esp_rom_tlsf.c and esp_rom_multi_heap.c are not needed
in the bootloader build.
In the cmakev1 build system this was never an issue because cmakev1
uses an early expansion phase where only Kconfig files for components
listed in BUILD_COMPONENTS are processed. Since the heap component is
not part of the bootloader's component list, its Kconfig options
(CONFIG_HEAP_TLSF_USE_ROM_IMPL, etc.) were never defined and the
conditional compilation of these source files was effectively skipped.
In the cmakev2 build system, Kconfig options from all discovered
components are visible regardless of whether the component is part
of the build. Because the bootloader reuses the main project's
sdkconfig (where CONFIG_HEAP_TLSF_USE_ROM_IMPL defaults to y on
targets with ROM TLSF support like esp32c2), the TLSF patch sources
were being compiled into the bootloader's esp_rom. This caused a
build failure because esp_rom_tlsf.c includes tlsf_block_functions.h
from the heap component, which is not a dependency of esp_rom and is
not part of the bootloader build.
Guard the TLSF and multi_heap ROM patch sources with NOT
BOOTLOADER_BUILD to prevent them from being compiled in the
bootloader context. This fix is compatible with both cmakev1 and
cmakev2.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Commit f62f45cf5c changed target_add_binary_data to resolve the
embedded file path relative to the component directory by using
idf_component_get_property to obtain the COMPONENT_DIR. This fixed the
path resolution when target_add_binary_data is called from
idf_component_include, which runs outside the component's directory
context.
However, target_add_binary_data can also be called directly from a
project's CMakeLists.txt with a non-component target, e.g.
target_add_binary_data(app.elf ...) as done in
examples/security/security_features_app. In this case,
idf_component_get_property fails because the target is not a component.
Fix this by moving the path resolution to idf_component_include, where
the embed file paths from EMBED_FILES and EMBED_TXTFILES component
properties are resolved to absolute paths relative to COMPONENT_DIR
before being passed to target_add_binary_data. This way,
target_add_binary_data receives already absolute paths from
idf_component_include and can use plain get_filename_component(ABSOLUTE)
for direct calls, which correctly resolves relative to
CMAKE_CURRENT_SOURCE_DIR.
Fixes: f62f45cf5c ("fix(cmakev2/utilities): add a dependency target for the embedded file")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add CMakeLists_v2.txt to the bootloader subproject, implementing the
bootloader build using the new cmakev2 IDF build framework.
The file covers the full bootloader build pipeline:
- Sets PROJECT_COMPONENTS_SOURCE to "idf_components" so that the
subproject's built-in components (main/, components/) are treated as
IDF components (priority 0) rather than project components (priority
3). This preserves the cmakev1 behaviour where user-supplied
components in bootloader_components/ can override the built-in ones.
- Registers optional user-supplied bootloader components from the
application project's bootloader_components/ directory, with support
for selectively excluding individual components via
IGNORE_EXTRA_COMPONENT.
- Bootstraps the cmakev2 framework (idf.cmake) and initialises the
project with BOOTLOADER_BUILD and NON_OS_BUILD properties, which are
also exposed as C preprocessor definitions.
- Sets GENERATE_SDKCONFIG to 0 to prevent the bootloader subproject
from regenerating the main project's sdkconfig, as the bootloader
has a different set of components and hence different Kconfig files.
- Sets the common implicit component dependencies shared by every
bootloader component (log, esp_rom, esp_common, esp_hw_support,
esp_libc, arch-specific component).
- Applies the compiler options specific for bootloader
- Selects the correct target-specific linker script, including a
separate script for ESP32-P4 silicon revisions < v3.
- Links the bootloader ELF via idf_build_executable and then converts it
to a flat binary via one of three paths depending on the secure boot
configuration:
* No secure boot: plain binary + size check + metadata.
* Secure Boot V1 one-time-flash: plain binary with post-build
instructions showing the esptool.py flash command.
* Secure Boot V1 reflashable: derives the symmetric eFuse key from
the ECDSA signing key, produces the reflash-digest image, and
prints burn/flash instructions.
* Secure Boot V2: produces an unsigned binary, optionally signs it
with the configured signing key (RSA-PSS 3072, ECDSA P-256, or
ECDSA P-384) via idf_sign_binary, and prints flash/multi-key
signing instructions.
- Adds comprehensive inline documentation explaining each section's
purpose, the rationale behind individual flags, and the relationships
between Kconfig symbols and generated artefacts.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a GENERATE_SDKCONFIG build property that controls whether kconfgen
writes the sdkconfig file (--output config) during the configuration
step. It defaults to 1 (enabled).
When building the bootloader as a subproject, the set of components
(and their Kconfig files) differs from the main project. Running
kconfgen with --output config in this context rewrites the main
project's sdkconfig. Even when the content is identical, the timestamp
update causes ninja to detect sdkconfig as newer than build.ninja
outputs (e.g. cmake_install.cmake), triggering an infinite CMake
re-run loop.
Setting GENERATE_SDKCONFIG to 0 in a subproject prevents this,
matching the behaviour of cmakev1's __OUTPUT_SDKCONFIG property.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add the PROJECT_COMPONENTS_SOURCE build property to control how project
components are categorised during component discovery. By default it is
set to "project_components" (priority 3 - highest), preserving the
existing behaviour.
Setting it to "idf_components" (priority 0) before calling
idf_project_init() makes the project's built-in components overridable
by user-supplied components through EXTRA_COMPONENT_DIRS (priority 2).
This is needed for sub-projects like the bootloader, whose built-in
components (e.g. main) are provided by ESP-IDF and should be overridable
by user-supplied components placed in bootloader_components/. The
cmakev1 build system allowed this override because it used a
last-one-wins strategy, but cmakev2 uses strict priority-based component
resolution where project_components always win.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add an optional ALL parameter to idf_build_binary and idf_sign_binary
functions. When specified, the created custom target is included in the
default build target. Without ALL, custom targets created by these
functions are excluded from the default build (add_custom_target
behavior), meaning they won't be built unless explicitly requested or
depended upon by another target in ALL.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a helper function for printing multi-line post-build messages on a
target. The message lines are joined with newlines, written to a file at
configure time, and printed at build time using a single cmake -E cat
command, replacing the need for repeated cmake -E echo invocations.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a public API function to check that the bootloader binary does not
overlap the partition table, mirroring the existing idf_check_binary_size
pattern for application binaries.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The KEYFILE argument value was stored in a misspelled variable
"keyfle" instead of "keyfile", causing custom keyfile paths to
be silently ignored.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
`CMAKE_CURRENT_LIST_DIR` is actually `components/bootloader`, so it
doesn’t need to be passed via `EXTRA_COMPONENT_DIRS`: the build already
recognizes it as an esp-idf component.
In **cmakev1**, this is silently ignored: if a component with the same
name already exists, its directory is updated and the previous directory
is stored in the `COMPONENT_OVERRIDEN_DIR` component property.
In **cmakev2**, this is correctly detected and reported.
CMake Warning at /home/fhrbata/work/esp-idf/tools/cmakev2/utilities.cmake:63 (message):
IDF: Component 'bootloader' directory '/home/fhrbata/work/esp-idf/components/bootloader'
with higher priority 'project_extra_components' will be used instead of component directory
'/home/fhrbata/work/esp-idf/components/bootloader' with lower priority 'idf_components'
Call Stack (most recent call first):
/home/fhrbata/work/esp-idf/tools/cmakev2/component.cmake:625 (idf_warn)
/home/fhrbata/work/esp-idf/tools/cmakev2/idf.cmake:411 (__init_component)
/home/fhrbata/work/esp-idf/tools/cmakev2/project.cmake:580 (__init_components)
CMakeLists_v2.txt:28 (idf_project_init)
CMakeLists.txt:19 (include)
Since it doesn’t make sense to explicitly add the bootloader as an extra
component, remove it.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The commonly required components are configured solely for backward
compatibility with cmakev1. Since the bootloader build explicitly sets
commonly required components, we need to support this feature in cmakev2
as well. Ideally, there should be no commonly required components, and
each component should specify its requirements explicitly.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
For the bootloader build, the `priv_include_dirs` variable is
uninitialized, but it is used in the common call to
`idf_component_register`.
Using uninitialized variables is not allowed in cmakev2 because a
component may be evaluated in the context of another component’s
variable scope.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the bootloader version of the `esp_partition` component sets
`PRIV_INCLUDE_DIRS` using the `private_include_dirs` variable. However,
this variable is not properly initialized, which causes issues in
cmakev2. In cmakev2, components are evaluated recursively, and a
component may be evaluated in the context of another component, so
components must initialize all variables before using them.
Moreover, there are effectively no `PRIV_INCLUDE_DIRS` to set when the
component is evaluated for the bootloader. Therefore, remove
`PRIV_INCLUDE_DIRS` entirely for bootloader and test builds.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The TestIdfRootDependency tests (test_basic_build,
test_build_only_when_required, test_cleanup_unused) were introduced in
MR 39704 to validate the idf_extra_components.yml root dependency
feature. This feature relies on component manager integration that is
not yet implemented in the cmakev2 build system, causing all three
tests to fail when run with --buildv2.
Skip these tests for cmakev2 using the buildv2_skip marker until root
components support is added (IDF-14259).
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In CMake v2's recursive component evaluation model, when a component
triggers the inclusion of another component via idf_component_include(),
the child component's directory scope inherits variables from the
caller's scope chain through the idf_component_include() function scope
and add_subdirectory().
The vfs component's Linux target block used list(APPEND ...) without
first initializing the srcs, inc, and priv_inc variables. This was
harmless when vfs was evaluated before esp_stdio, but after commit
5fac0b7386 ("feat(console): Move IO initialization outside of the
console component"), esp_stdio became a real component for the Linux
target and started calling idf_component_include(vfs). When esp_stdio
is evaluated first, its srcs variable (containing stdio_port.c and
linux/esp_stdio_linux.c) leaks into vfs's scope, causing the build
to fail with "Cannot find source file: components/vfs/stdio_port.c".
Fix by explicitly initializing all local variables at the top of the
Linux target block before appending to them.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Verify that setting IDF_COMPONENT_MANAGER=0 skips the component manager
flow entirely and produces a successful build. The test also asserts that
no "Component manager round" messages appear in the output, confirming
the manager loop is not entered.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When the component manager is disabled via IDF_COMPONENT_MANAGER=0, the
cmakev2 build system still entered the full component manager flow
(__fetch_components_from_registry), which called
__download_managed_component(). That function returned early with
result=0 without creating the expected output file, violating the
post-condition in __download_component_level_managed_components() that
checks result==0 => file exists, causing a fatal error.
Instead of patching the low-level function to write an empty stub file,
this commit properly skips the entire component manager flow when the
manager is disabled:
- Move __init_component_manager() to idf.cmake global initialization
sequence alongside other __init_*() calls, so IDF_COMPONENT_MANAGER
build property is available early.
- Set __SDKCONFIG_ORIG to the real sdkconfig path in __init_kconfig()
as its default value. Previously it was only set inside
__create_sdkconfig_orig_copy() and had a defensive fallback in
__create_base_kconfgen_command(). The default ensures __SDKCONFIG_ORIG
is always valid regardless of whether the component manager runs.
- Guard __create_sdkconfig_orig_copy() with an IDF_COMPONENT_MANAGER
check. The sdkconfig backup exists solely to preserve unknown Kconfig
options from managed components during intermediate kconfgen rounds.
When the manager is disabled, no managed components exist, so the
backup is unnecessary.
- Guard __fetch_components_from_registry() call in project.cmake behind
IDF_COMPONENT_MANAGER == 1. When disabled, only the manifest warning
is issued. No download loop runs, no temp files are created, and no
"Component manager round N..." messages are printed.
- Remove the now-redundant IDF_COMPONENT_MANAGER guard from
__download_managed_component(), since it is only reachable when the
manager is enabled.
Closes https://github.com/espressif/esp-idf/issues/18372
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The ULP sources include sdkconfig.h for compile-time configuration
values. The sdkconfig directory was previously available to the ULP
target indirectly through the COMPONENT_INCLUDES variable, which is
populated from the parent component's INTERFACE_INCLUDE_DIRECTORIES.
In cmake v1, idf_component_register() adds the config_dir as PUBLIC
include directory to every component (component.cmake:498), so it
ends up in INTERFACE_INCLUDE_DIRECTORIES and gets passed to the ULP
subproject via COMPONENT_INCLUDES.
In cmakev2, the config_dir is added as a build-level property and
applied to components as PRIVATE (component.cmake:1053), so it no
longer appears in INTERFACE_INCLUDE_DIRECTORIES. As a result, the
sdkconfig directory is missing from the ULP target's include paths.
Add the sdkconfig directory explicitly to the ULP target's include
directories. Note that SDKCONFIG_HEADER is already passed to the ULP
subproject and its directory is already extracted into sdkconfig_dir
for linker script preprocessing (IDFULPProject.cmake:40).
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In cmake v1, __build_process_project_includes() exports all build
properties as CMake variables before including project_include.cmake
files. cmakev2 was missing this step, causing components like ULP that
reference build properties as CMake variables (e.g. ${SDKCONFIG_HEADER})
to receive empty values.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The cmakev2 kconfig module sets sdkconfig output paths using internal
property names (__SDKCONFIG_HEADER, __SDKCONFIG_CMAKE, etc.), but
components like ULP read the public names (SDKCONFIG_HEADER,
SDKCONFIG_CMAKE). This results in empty values being passed to the ULP
sub-project, causing its CMake configure step to fail.
Add public aliases matching the cmake v1 property names for backward
compatibility.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When version.txt does not exist and git describe fails (e.g. in
release archives or environments without git), IDF_VER was set to
the raw git_describe output which resolves to "-128-NOTFOUND",
causing esp_get_idf_version() to return a garbled string.
Add a fallback that constructs the version string from the
IDF_VERSION_MAJOR, IDF_VERSION_MINOR and IDF_VERSION_PATCH
variables when git describe is not available.
Closes https://github.com/espressif/esp-idf/issues/18240
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Pytest's default import mode (prepend/rootdir) imports test modules as
top-level modules keyed by filename. When two directories contain test
files with the same basename (e.g. test_sdkconfig.py in both
test_build_system/ and test_build_system/buildv2/), pytest tries to
register both as the module name "test_sdkconfig". The second collection
fails with "import file mismatch" because the module object already
cached in sys.modules points to the first file.
Adding __init__.py to the buildv2/ directory makes it a proper Python
package. Pytest then imports its test modules under the package
namespace (buildv2.test_sdkconfig), which is distinct from the
top-level test_sdkconfig, resolving the collision.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The sdkconfig file may contain configuration options defined in Kconfig
files of managed components. Since kconfgen runs before the component
manager fetches these components, the Kconfig definitions for managed
component options are not yet available. The kconfgen --output config
flag regenerates sdkconfig from kconfiglib's internal state, which only
knows about options with loaded Kconfig definitions. This causes unknown
options (i.e., those from managed components) to be silently dropped
from sdkconfig during intermediate regeneration rounds.
Note that kconfgen's --config flag (used for reading sdkconfig) only
performs deprecated option name replacement and does NOT drop unknown
options. The problem is exclusively in --output config, which writes a
fresh sdkconfig from the parsed Kconfig tree state.
Fix this by introducing a __SDKCONFIG_ORIG build property that provides
an indirection layer for the --config input path:
- Before the component manager runs: __SDKCONFIG_ORIG points to a copy
of the original sdkconfig (build/sdkconfig.orig), created by the new
__create_sdkconfig_orig_copy() function. This copy preserves all
original options, including those from managed components.
- During intermediate kconfgen runs: --config reads from the preserved
copy (so unknown options survive as input), while --output config
writes to the real sdkconfig (unknown options may be dropped there,
but this is harmless since kconfgen always reads from the copy).
- After the component manager completes: __SDKCONFIG_ORIG is reset to
point to the real sdkconfig and __BASE_KCONFGEN_CMD is rebuilt, so
that subsequent operations (menuconfig, save-defconfig, confserver)
read and write the actual sdkconfig file directly.
The flow is:
__create_sdkconfig_orig_copy()
-> __SDKCONFIG_ORIG = build/sdkconfig.orig
__generate_sdkconfig()
-> --config build/sdkconfig.orig --output config project/sdkconfig
__fetch_components_from_registry():
loop:
download_components()
__generate_sdkconfig()
-> --config build/sdkconfig.orig --output config project/sdkconfig
if success: break
endloop
-> __SDKCONFIG_ORIG = project/sdkconfig
-> rebuild __BASE_KCONFGEN_CMD
idf_create_menuconfig() / save-defconfig / confserver
-> uses --config project/sdkconfig (the real file)
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
While the `esp-idf-sbom` `idf.py` extension is implemented within the
`esp-idf-sbom` python package, a few minor changes are also required on
the `idf.py` side.
1. Add `esp-idf-sbom` as a core requirement so it is automatically
installed by the installation scripts.
2. Add the `sbom-create` action to the list of actions for which the
flashing-related closing message is displayed.
3. Update the documentation in the api-guides for tools.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, embedded files can be added in two ways. First, by calling
the `target_add_binary_data` function directly within a component.
Second, by passing `EMBED_FILES` or `EMBED_TXTFILES` to
`idf_component_register`, or in cmakev2 by setting the `EMBED_FILES` or
`EMBED_TXTFILES` component properties.
The source file for an embedded file is generated using
`add_custom_command`. When the embedded file is added directly in the
component's CMakeLists.txt file by using the `target_add_binary_data`
function, the `add_custom_command` command is evaluated in the same
directory context where the component target is created. As a result,
the generated embedded file can be used automatically as a file
dependency in component's sources.
However, when an embedded file is added via `idf_component_register` or
by setting the component property, the call to `add_custom_command`
inside `target_add_binary_data` occurs after the component has been
evaluated, specifically after the `add_subdirectory` call, within
`idf_component_include`. This means it is not created in the same
directory context as the component's target. In this case, the embedded
file dependency is not added to the component's target, and the embedded
file is not generated. This behavior is described in the
`add_custom_command` documentation [1].
> A target created in the same directory (CMakeLists.txt file)
> that specifies any output of the custom command as a source
> file is given a rule to generate the file using the command at
> build time.
To fix this issue, an explicit custom target for the generated embedded
file is created and added as a dependency of the component's target,
ensuring that the file is generated correctly.
[1] - https://cmake.org/cmake/help/latest/command/add_custom_command.html
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In cmakev2, a component can be evaluated within the context of another
component, so it's important to properly initialize each variable used
by the component.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The minimal build property is simply a shorthand for `set(COMPONENTS
main)`. The issue is that there is currently no check to verify whether
the `main` component actually exists or is known to the build system.
If the `main` component is not present, print an error message along
with suggestions on how to fix this inconsistency.
Closes https://github.com/espressif/esp-idf/issues/18219
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_component_optional_requires compatibility function in cmakev2
currently attempts to mimic the behavior of cmakev1 by adding a
dependency only if a component is already included in the build. In
cmakev1, this is handled by checking the BUILD_COMPONENTS list, which is
created during an early evaluation phase. Because cmakev2 removed this
early phase to allow for configuration-based component dependencies, the
build system does not inherently know which components will be part of
the build beforehand. To compensate, the current compatibility function
relies on the TARGET_EXISTS generator expression to determine if a
component should be linked.
This approach poses a problem because generator expressions are not
evaluated until the generation phase, but cmakev2 processes linker
scripts during the configuration phase by recursively scanning targets
linked to a library. Because the scanner does not recognize and cannot
evaluate generator expressions, any component linked optionally through
generator expression is skipped. If that component carries a linker
script, the script is never added to the library interface, resulting in
build failures. Since cmakev2 aims to support multiple libraries, a
component might also exist globally, causing TARGET_EXISTS to evaluate
to true, yet still be missed during a specific library's recursive scan,
leading to the same omission of necessary linker scripts.
To resolve this, the implementation of idf_component_optional_requires
is changed to check if a component is known to the build system in
COMPONENTS_DISCOVERED build property. If so, it is explicitly included
and linked directly via its interface target. While this may pull more
components into a build than the previous generator expression method,
potentially increasing build times, it ensures that dependency trees are
fully visible to the library target scanner. This serves as a practical
middle ground that maintains compatibility with existing components and
ensures build stability. This approach allows for the gradual fixing of
idf_component_optional_requires usage in components for cmakev2, with
the eventual goal of removing its usage in cmakev2 entirely.
This change also fixes cases where idf_component_optional_requires is
used to conditionally add requirements based on configuration options.
In cmakev2, the presence of a configuration option does not guarantee[1]
that a component has been included via add_subdirectory, unlike the
behavior in cmakev1. With this change even constructions like
```cmake
if(CONFIG_VFS_SUPPORT_IO)
idf_component_optional_requires(PRIVATE vfs)
endif()
```
will now work in cmakev2, as the updated idf_component_optional_requires
explicitly includes the required component if it is available to the
build system.
Closes https://github.com/espressif/esp-idf/issues/18133
[1] https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/build-system-v2.html#id9
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a simple test to verify that the buildv2_test_app can be built for
the Linux target.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, cmakev2 is being tested only in backward-compatible mode by
using the existing cmakev1 tests with the cmakev2 test application. We
also need to add tests specific to cmakev2, and it is convenient to
reuse the existing build system testing framework. Let's add a `buildv2`
subdirectory to the existing `tools/test_build_system` directory and use
the `pytest_collection_modifyitems` hook to ignore tests in this
directory unless the `--buildv2` option is used.
Without the `--buildv2` option, only the existing cmakev1 tests are
executed and tests in `buildv2` directory are skipped. With the
`--buildv2` option, the existing cmakev1 tests run with the cmakev2
testing application for backward compatibility testing, and all cmakev2
tests within the `buildv2` subdirectory are also executed.
Note: we cannot use the `pytest_ignore_collect` hook, because the
`--buildv2` option is not known to the pytest, so the
`config.getoption('--buildv2', False)` returns always False. We would
likely need to add the `--buildv2` option in the conftest.py in the
esp-idf root directory.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
fix: lsadjf las jflasjfl aslfsald asl fsadlf sladsal jfsadfas
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This adds a clear header to the pytest output, indicating which build
system version is currently being tested.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In cmakev1, the initial order of components linked to the executable is
determined by the `__build_expand_requirements` function and stored as a
list in the `BUILD_COMPONENT_ALIASES` build property. Later, the
components from `BUILD_COMPONENT_ALIASES` are linked to the executable
using `target_link_libraries`. The issue is that
`BUILD_COMPONENT_ALIASES` contains components in a "reversed" order
compared to how they would typically appear on the link command line. In
other words, components (archives) deeper in the dependency chain are
placed first. As a result, cmake has to repeat the archives in the link
command more times than is actually necessary. This has the beneficial
side effect of placing libfreertos.a before libmain.a, allowing the
`app_main` symbol to be resolved.
In cmakev2, there is no predefined order because it does not perform
early evaluation. Instead, the entire link command is created based on
actual component dependencies using `target_link_libraries`.
Consequently, freertos must set the `app_main` symbols as undefined.
This is already done for non-Linux targets, but it must also be done for
the Linux target.
Another issue arises with macOS, which mangles C symbols by prepending
an underscore. We cannot use the same method as in
`components/linux/assert_func.c` for `__assert_func`, so for macOS,
`_app_main` is used. In the future, if more undefined symbols need to be
added for the Linux target, we should consider introducing a helper.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, idf_build_generate_metadata only accepts binary targets for
which it generates metadata (project_description.json). On Linux
targets, binary images are not generated, but we still need to generate
project_description.json. Extend the current function to accept both
executable and binary targets and ensure project_description.json is
generated when a Linux target is used.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Some components simply return when included in a build for the Linux
target. This is silently ignored in the `idf_component_include`, and the
component interface target is linked as requirement in
`idf_component_register`. This has the side
effect of such components being reported as included, with their
configuration displayed in the menuconfig for included components,
rather than in the submenu for excluded components. For example, the
`bt` component, if added as a dependency in `idf_component_register`,
will be displayed in menuconfig as an included component for Linux
target. The `idf_component_include` function sets the component's
`COMPONENT_REAL_TARGET` property to `NOTFOUND` in such situations. Use
this information when config.env is generated put such components into
excluded submenu. Note that we cannot avoid linking empty component interface
into library, because there might be recursive dependencies and at the
time when the component is included we might not now if it has a real
target or not.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the CONFIG_APP_BUILD_GENERATE_BINARIES option is used to
determine whether binary images will be generated in
idf_project_default. At present, this option is enabled even for the
Linux target, where the esptool_py component is not included, preventing
the generation of binary images. To address this issue, verify if the
target for esptool_py is present.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In cmakev2, the configuration for all components is included because the
complete configuration is generated very early, before any component is
actually evaluated. This poses a problem for the Linux target, which
does not have a controller Kconfig.in file, causing kconfgen to fail due
to the missing include. To address this, let's use the osource command,
which does not include the referenced Kconfig file if it does not exist.
Closes https://github.com/espressif/esp-idf/issues/18142
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The IDF_TOOLCHAIN build property is currently incorrectly set to the
default `gcc` value for the linux target, whereas it should be empty.
This misconfiguration causes confusion for components like `soc`, which
adjust toolchain options based on the
IDF_TOOLCHAIN(CONFIG_IDF_TOOLCHAIN_GCC) build property's setting.
When sdkconfig is generated, the IDF_TOOLCHAIN build property is passed
as an environmental variable to kconfgen, and the CONFIG_IDF_TOOLCHAIN
configuration option is set based on this variable. Additionally, the
CONFIG_IDF_TOOLCHAIN_GCC and CONFIG_IDF_TOOLCHAIN_CLANG configuration
options are set accordingly. Subsequently, CONFIG_IDF_TOOLCHAIN_GCC is
used in several places, such as `components/soc/project_include.cmake`,
to configure the toolchain (compiler flags) by invoking functions from
`tools/cmake/toolchain_flags.cmake`, which is included only for
non-linux targets. As a result the configuration fails, because
functions from `tools/cmake/toolchain_flags.cmake` are not available on
linux target.
Since the IDF_TOOLCHAIN cmake cache variable is actually set in the
`tools/cmake/toolchain.cmake` file, the IDF_TOOLCHAIN build property
should be set after the toolchain is initialized in cmakev2's project
initialization. Note that each toolchain file, except for linux,
includes `toolchain.cmake`, which in turn includes
`toolchain_flags.cmake`. This means the IDF_TOOLCHAIN cmake cache
variable is set for every target except linux, because the toolchain
file for linux is empty. As a result CONFIG_IDF_TOOLCHAIN is empty and
CONFIG_IDF_TOOLCHAIN_GCC not set as for cmakev1.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The `srcs` list variable is used without initialization when building
for linux target. Initialize it explicitly.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The `srcs` list variable is used without initialization when building
for linux target. Initialize it explicitly.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The dependency chain currently tracks component names that are included
recursively with `idf_component_include`. However, these component names
can be ambiguous because a component may be referenced by different
names, such as with a namespace. Additionally, `idf_component_include`
can accept anything that `__get_component_interface` accepts, meaning
even the component interface target can be used to include the component
in the build. To uniquely identify each component, use component
interface targets instead of names in the dependency chain.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In some instances, the COMPONENTS_DISCOVERED build property is used to
walk through the component list and obtain component properties, such as
during configuration preparation or the inclusion of project_include
files. Since we know the component interfaces from the
COMPONENT_INTERFACES build property, we can switch to the faster
`__idf_component_get_property_unchecked` method to obtain component properties
in these cases. This change reduces the reconfiguration time by 0.4
seconds on my computer.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The commit f951ae5b18 ("feat(cmakev2): Added component validation
checks for ...") introduced support for validating component sources and
include directories. It heavily relies on iterating through the
registered components to obtain their properties. For instance, it
examines all components for each component source file to ensure that
the source files do not originate from other components. However, it
turns out that `idf_component_get_property` is a bottleneck in this
process, causing reconfiguration to take 3-4 times longer than before
the introduction of this check. Since we know of all the component
interfaces, we can bypass the checks performed by
`idf_component_get_property` and instead use the raw version,
`__idf_component_get_property_unchecked`, which operates much faster.
Following are hello_world example reconfigure times before and after.
before:
-- Configuring done (11.1s)
-- Generating done (0.2s)
after:
-- Configuring done (3.9s)
-- Generating done (0.2s)
A new library property LIBRARY_COMPONENT_INTERFACES_LINKED is added that
keeps component interfaces linked to the library.
Fixes: f951ae5b18 ("feat(cmakev2): Added component validation checks for ..")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Retrieve the value of the specified component property quickly. The
existing `idf_component_get_property` function performs various checks
to identify the component interface target, which keeps the component
properties, and accepts a component name, target, target alias, or
interface. While `idf_component_get_property` uses a cache to identify
the component interface relatively quickly, it is still much slower
compared to the raw `get_property`. The
`__idf_component_get_property_unchecked` function provides a faster way how to
obtain component property if the component interface is already known.
It skips all the checks, so it must be used carefully.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
If `.gitlab/ci/rules\.yml` is changed, make sure the
`check-tools-files-patterns` pre-commit hooks is triggered to verify
that all `tools/*` files are covered by rules.yml.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The MINIMAL_BUILD build property and the COMPONENTS variable are both
used to determine the initial component list for the build.
Currently, if the COMPONENTS variable is set, the MINIMAL_BUILD logic
is ignored during component selection, but the MINIMAL_BUILD build
property remains set. This leads to an inconsistent state where
menuconfig displays information indicating MINIMAL_BUILD is active,
even though it was ignored in favor of the COMPONENTS variable.
Fix this by setting the MINIMAL_BUILD property to OFF if the
COMPONENTS variable is used.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The 8bbcdb116d ("ci(gitlab): Improve tools host test dependencies")
removed `kconfig_new` from rules.yml leading to the following pre-commit
failure.
```bash
Check tools dir files
patterns...........................................................Failed
- hook id: check-tools-files-patterns
- exit code: 1
This test is used for making sure of all the tools dir files are
recorded in .gitlab/ci/rules.yml to trigger the related tests, except
those files should be excluded.
Missing Files:
tools/kconfig_new/prepare_kconfig_files.py
tools/kconfig_new/config.env.in
tools/kconfig_new/confgen.py
tools/kconfig_new/confserver.py
tools/kconfig_new/config_buildv2.env.in
Please add these files or glob patterns to ".gitlab/ci/rules.yml" and
put related files under ".patterns-<test_group>" block to trigger
related tests.
Or add them to "tools/ci/exclude_check_tools_files.txt" to exclude them.
```
Fixes: 8bbcdb116d ("ci(gitlab): Improve tools host test dependencies")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In cmakev2, a component can be evaluated within the context of another
component, so it's important to properly initialize each variable used
by the component.
Fixes: 0c4cf75c35 ("feat(esp32s31): Introduce esp32s31 hello world")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_build_generate_depgraph function creates a component dependency
graph in dot (graphviz) format for a specified executable. It uses
existing helper functions from cmakev1, ensuring that the generated dot
files are produced in the same manner as in cmakev1. While adjustments
might be needed in the future if necessary, the current implementation
is intended to offer the same functionality as cmakev1. Similar to
cmakev1, the dot files are only generated only when the
__BUILD_COMPONENT_DEPGRAPH_ENABLED build property is set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
As part of inter-procedural optimizations (IPA), the compiler may
perform tasks such as constant propagation for functions. This involves
generating a specialized version of a given function with a new symbol
name that includes a suffix. For example, during constant propagation,
the compiler might create a specialized version named
`spiflash_start_core.constprop.0` for the `spiflash_start_core`
function. Additionally, the compiler may generate multiple clones of a
single function. Currently, when ldgen performs symbol placement, it
does not account for these compiler-generated functions, leading to
their incorrect or unexpected placement in memory (markers).
Consider a linker fragment with:
```
[mapping:spi_flash]
archive: libspi_flash.a
entries:
esp_flash_api: spiflash_start_core (noflash)
```
The `spiflash_start_core` function should be placed in IRAM. However,
the compiler might generate an optimized version of this function with a
`.constprop.0` suffix, resulting in a
`.text.spiflash_start_core.constprop.0` input section. Currently, ldgen
does not handle this situation, leading to misplaced symbols.
Since `.` is not allowed in C identifiers, it should be safe to consider
all input sections for a symbol with any `.` suffix as representing that
symbol. This means considering the symbol suffixes should not cause any
ambiguity.
This change automatically places all input sections, including those
with possible suffixes for a given symbol, into the specified memory. In
other words, specifying a function name like `spiflash_start_core` in a
linker fragment automatically includes input section names matching
`spiflash_start_core(\..*)?$`.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The build system keeps track of each component source. Currently
there are four types of sources:
1. "project_components" - project components
2. "project_extra_components" - components from EXTRA_COMPONENT_DIRS
3. "project_managed_components" - custom project dependencies managed by the IDF Component Manager
4. "idf_components" - ESP-IDF built-in components, typically under /components
This can be used to identify the component libraries that are likely to
change during application development and pass them to ldgen as mutable
libraries. Add all components with "project_components" as their source
as mutable.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a helper function `__should_generate_sdkconfig()` that checks
whether the sdkconfig file should be generated. The
`__generate_sdkconfig()` function can be called multiple times, such as
when the initial sdkconfig is generated at the start of the build
process and later after additional components are fetched by the
component manager. There might be no components fetched by the
component manager, for example, in the hello_world example, or the
downloaded components may not contain any configuration files. In such
cases, there is no need to regenerate the sdkconfig. This helper
function stores the list of configuration files in the
`__PREV_KCONFIGS`, `__PREV_KCONFIG_PROJBUILDS`, and
`__PREV_SDKCONFIG_RENAMES` build properties at its end, and at the
beginning, it compares them with the current lists of configuration
files.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_create_size_report function allows for the creation of size
report targets based on the generated link map file. The size report
targets are created using the TARGET option name: "<target>",
"<target>-files", and "<target>-components". These size report targets
are added to the idf_default_project with the TARGET set to "size",
resulting in the creation of "size", "size-files", and "size-components"
targets for the default project.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_project_init macro is evaluated within the global variable
scope, and currently, there are a few variables still set. Let's ensure
all variables are properly unset to avoid polluting the global variable
namespace.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Frantisek Hrbata
