Any idf.py invocation could hang indefinitely with no output while
spawning an unbounded chain of "idf.py --version" subprocesses,
eventually exhausting system memory.
During init_cli(), idf.py parses the project's dependencies.lock to
vet trusted component-provided idf_ext.py extensions. If the lock
contains a component whose manifest has an "if: idf_version" clause,
evaluating it calls idf-component-manager's _get_idf_version(). Outside
a CMake build the IDF_VERSION environment variable is not set, so that
function falls back to running "idf.py --version" as a subprocess,
which re-enters init_cli() and recurses without bound.
During a normal CMake build the component manager runs as a subprocess
that already has IDF_VERSION in its environment (see build/config.env),
so the fallback is never reached. The recursion happens only because
idf.py runs component-manager code in-process during its own CLI
startup, outside that context.
Seed IDF_VERSION into os.environ early in init_cli(), before any
dependencies.lock parsing, using the subprocess-free
idf_version_from_cmake() helper. This gives in-process component-manager
code the same IDF_VERSION a CMake build would provide.
When a component's manifest declared a managed dep `<ns>/<name>` and the
project had a local `components/<name>` shadowing it, configure failed
with `IDF: Failed to resolve component '<ns>__<name>'`.
cmakev1 does a project-wide injection so the manager's _choose_component
sees every component name in known_components and rewrites the
namespaced dep to its locally-shadowing short name. cmakev2 invokes the
manager per component with a single-component requirements file, so
known_components has only one entry and the rewrite cannot fire --
`<ns>__<name>` is written verbatim into MANAGED_REQUIRES and the
force-include at component.cmake:995 aborts.
Seed the per-component requirements file with one __COMPONENT_SOURCE
entry per discovered component so handle_project_requirements() builds
the same known_components list cmakev1 produces. The seeded entries are
inert on the cmake side -- the __component_set_property shim ignores
__COMPONENT_SOURCE -- they exist purely to feed the manager's rewrite.
Add a regression test in tools/test_build_system/buildv2/test_component.py.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Two tests added in commit c21d05e612 (MR !46354 "Added cmake target
listing in help") fail under pytest_buildv2_system /
pytest_buildv2_system_win:
* test_idf_py_help_after_configure_with_no_custom_targets_has_no_section
* test_idf_py_help_lists_cmake_custom_targets_after_configure
The tests assert that internal phony build targets do not appear
under "CMake Custom Targets" in idf.py --help. Filtering is driven
by tools/idf_py_actions/help_custom_targets_skip.py, whose exact-name
allowlist and shape policy were calibrated against cmakev1 target
names. cmakev1 either hard-codes the names ("app_check_size") or
uses a prefix already covered by HELP_PHONY_NAME_PREFIXES
("gen_<project>_binary"), so v1 stays out of the section.
cmakev2 parameterizes the same logical targets by the project name
in tools/cmakev2/build.cmake and tools/cmakev2/project.cmake and
produces, for project "build_test_app":
build_test_app_binary
build_test_app_binary_check_size
build_test_app_mapfile
build_test_app.map
ldgen_libraries.in_library_build_test_app
None of these match the v1-shaped allowlist or shape policy, so
both tests fail on every cmakev2 project rather than only on this
test app. Skip them under buildv2 with @pytest.mark.buildv2_skip
to unblock CI while a more robust filtering strategy is worked out.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a buildv2 test that reconfigures the test app with both GCC and
clang toolchains and asserts that the selected IDF_TOOLCHAIN value
is propagated to sdkconfig. The clang case is skipped when a clang
compiler is not available in PATH.
Profiling shows that parsing the linker fragment (.lf) files with
pyparsing is the single largest cost in ldgen (~58% of total runtime).
parse_fragment_file() rebuilds its grammar from scratch on every call
because the conditional parser closure has to capture sdkconfig to
evaluate if/elif/else at parse time.
Add a cache that pickles the parsed FragmentFile object list to
<output>.lfcache. The cache is keyed on fragment file paths+mtimes
plus sdkconfig and kconfig mtimes — sdkconfig is included because
fragment if/elif/else blocks are evaluated against it at parse time.
This avoids the pyparsing cost on rebuilds where ldgen has to run
but the fragment files themselves are unchanged — for example when
a function is added or removed in a source file. Cache hits produce
the same sections.ld as a full parse, and any pickle load failure
falls through to a normal parse, so cache corruption or a Python
upgrade cannot produce a wrong build.
Print "Skipping linker fragment parsing, fragment files unchanged"
on the lf cache hit path so build logs show when the optimization
fired — this makes it easy to diagnose user reports of unexpected
ldgen behavior.
The LDGEN_NO_CACHE environment variable disables the lf cache. When
set, ldgen falls through to a normal parse without consulting or
writing the cache file.
Measured on wifi_station (205 libs, 67 .lf files), median of 10 runs:
before after
rebuild, section change, .lf unchanged 5.82s 1.34s
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
After running objdump on all libraries, extract section names from the
stored raw output using regex and compute a fingerprint (MD5 hash of
section names + mtimes of fragment files, sdkconfig, and kconfig).
If the fingerprint matches the cached value from the previous run,
skip the expensive pyparsing + generation step entirely and touch
the output file to update its timestamp. The fingerprint file is
stored next to the output (e.g. sections.ld.fingerprint) in the build
directory.
This avoids the pyparsing bottleneck for the common case of editing
function bodies without adding or removing symbols — section names
in the object files are unchanged, so the generated sections.ld
would be identical. When sections do change, the full generation
runs and the fingerprint is updated.
Print "Skipping linker script generation, section names unchanged"
on the fingerprint hit path so build logs show when the optimization
fired. This makes it easy to diagnose user reports of unexpected
ldgen behavior — the build log directly shows whether the cache was
used.
Add an LDGEN_NO_CACHE environment variable to disable the cache as a
workaround in case the optimization causes problems. When set, ldgen
prints "Linker script generation caches disabled by LDGEN_NO_CACHE"
and runs full generation as before, leaving any existing fingerprint
file untouched.
Measured on wifi_station (205 libs, 67 .lf files), median of 10 runs:
before after
rebuild, section names unchanged 5.82s 0.82s
Closes https://github.com/espressif/esp-idf/issues/18408
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The TestIdfRootDependency tests (test_basic_build,
test_build_only_when_required, test_cleanup_unused) were introduced in
MR 39704 to validate the idf_extra_components.yml root dependency
feature. This feature relies on component manager integration that is
not yet implemented in the cmakev2 build system, causing all three
tests to fail when run with --buildv2.
Skip these tests for cmakev2 using the buildv2_skip marker until root
components support is added (IDF-14259).
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Verify that setting IDF_COMPONENT_MANAGER=0 skips the component manager
flow entirely and produces a successful build. The test also asserts that
no "Component manager round" messages appear in the output, confirming
the manager loop is not entered.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The check-all-apps-readmes pre-commit hook detected that
kconfig_test_app/README.md was missing the supported targets table.
Added the full table including esp32h21, esp32h4, and esp32s31.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Separate the misspelled KConfig.projbuild test case into its own test app
so it doesn't interfere with the general-purpose build_test_app. The test
now uses a pytest marker to select kconfig_test_app as the source.
Where actually building the app is not needed cmake reconfigure was introduced instead.
This should be performance upgrade especially for Windows runners, where build is quite slow
Pytest's default import mode (prepend/rootdir) imports test modules as
top-level modules keyed by filename. When two directories contain test
files with the same basename (e.g. test_sdkconfig.py in both
test_build_system/ and test_build_system/buildv2/), pytest tries to
register both as the module name "test_sdkconfig". The second collection
fails with "import file mismatch" because the module object already
cached in sys.modules points to the first file.
Adding __init__.py to the buildv2/ directory makes it a proper Python
package. Pytest then imports its test modules under the package
namespace (buildv2.test_sdkconfig), which is distinct from the
top-level test_sdkconfig, resolving the collision.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The sdkconfig file may contain configuration options defined in Kconfig
files of managed components. Since kconfgen runs before the component
manager fetches these components, the Kconfig definitions for managed
component options are not yet available. The kconfgen --output config
flag regenerates sdkconfig from kconfiglib's internal state, which only
knows about options with loaded Kconfig definitions. This causes unknown
options (i.e., those from managed components) to be silently dropped
from sdkconfig during intermediate regeneration rounds.
Note that kconfgen's --config flag (used for reading sdkconfig) only
performs deprecated option name replacement and does NOT drop unknown
options. The problem is exclusively in --output config, which writes a
fresh sdkconfig from the parsed Kconfig tree state.
Fix this by introducing a __SDKCONFIG_ORIG build property that provides
an indirection layer for the --config input path:
- Before the component manager runs: __SDKCONFIG_ORIG points to a copy
of the original sdkconfig (build/sdkconfig.orig), created by the new
__create_sdkconfig_orig_copy() function. This copy preserves all
original options, including those from managed components.
- During intermediate kconfgen runs: --config reads from the preserved
copy (so unknown options survive as input), while --output config
writes to the real sdkconfig (unknown options may be dropped there,
but this is harmless since kconfgen always reads from the copy).
- After the component manager completes: __SDKCONFIG_ORIG is reset to
point to the real sdkconfig and __BASE_KCONFGEN_CMD is rebuilt, so
that subsequent operations (menuconfig, save-defconfig, confserver)
read and write the actual sdkconfig file directly.
The flow is:
__create_sdkconfig_orig_copy()
-> __SDKCONFIG_ORIG = build/sdkconfig.orig
__generate_sdkconfig()
-> --config build/sdkconfig.orig --output config project/sdkconfig
__fetch_components_from_registry():
loop:
download_components()
__generate_sdkconfig()
-> --config build/sdkconfig.orig --output config project/sdkconfig
if success: break
endloop
-> __SDKCONFIG_ORIG = project/sdkconfig
-> rebuild __BASE_KCONFGEN_CMD
idf_create_menuconfig() / save-defconfig / confserver
-> uses --config project/sdkconfig (the real file)
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Frantisek Hrbata
Roland Dobai
Jakub Kocka
Marius Vikhammer
Fu Hanxi
Marek Fiala
Sergei Silnov
Sudeep Mohanty
Daniel Paul
Roland Dobai
Jan Beran