ldgen skips regeneration when a fingerprint over the section names, the
linker fragment files, sdkconfig and Kconfig is unchanged. The linker
script template was not part of that fingerprint, even though everything
outside its mapping placeholders is copied into the generated script
verbatim.
The build system regenerates the preprocessed template whenever any of
its inputs changes, not only when a linker fragment changes. When such a
change reaches ldgen but leaves the section names alone, the fingerprint
still matches, generation is skipped, and the image is linked against the
previously generated script. The skip path updates the mtime of the
output, so nothing in the build output shows that the change was dropped.
Add the template to the fingerprint so that regenerating it always
regenerates the linker script.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
On some Windows systems, antivirus, endpoint-security or DLP/encryption
software intercepts short-lived toolchain processes and strips their
stdout when the build captures it through a pipe, while the same command
prints its output normally when run by hand. The tool exits successfully
but returns nothing, and each build step that reads toolchain output then
failed with a different, cryptic error far from the real cause:
- CMake configuration aborted with "Unknown arguments specified" in
components/xtensa/project_include.cmake, or "check_expected_tool_version
invoked with incorrect arguments" in components/esp_common.
- ldgen turned the empty objdump output into an opaque pyparsing
"Expected 'In archive'" traceback.
- idf_tools.py silently reported the compiler/debugger version as
"unknown", sending users into a fruitless reinstall loop.
Detect the empty result at each consumer and fail (or warn) with an
actionable message that names the likely cause and the remedy:
- tools/cmake/compiler_query.cmake: new __compiler_query() helper runs a
compiler query and fails with a clear error on empty or failed output.
It is a standalone module included by both the cmakev1 and cmakev2
utilities, since the esp_common and xtensa project_include.cmake that
call it are shared by both build systems. The xtensa if() arguments are
now quoted so an empty result no longer collapses into a parse error.
- tools/ldgen/ldgen.py: _run_objdump() rejects empty objdump output, and
non-empty-but-unparsable section info is caught and re-raised as a clear
LdGenFailure instead of a raw pyparsing traceback.
- tools/idf_tools.py: empty version output now warns with the cause and
returns UNKNOWN_VERSION instead of silently reporting "unknown".
Closes https://github.com/espressif/esp-idf/issues/18727
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
- Harden prefix_map.cmake for paths with spaces on Windows
- ldgen: fragments-list-file, list normalization, CMake integration (tools/cmake + cmakev2)
- CI exclude list for check_tools; test_spaces bundle tweak
Made-with: Cursor
Profiling shows that parsing the linker fragment (.lf) files with
pyparsing is the single largest cost in ldgen (~58% of total runtime).
parse_fragment_file() rebuilds its grammar from scratch on every call
because the conditional parser closure has to capture sdkconfig to
evaluate if/elif/else at parse time.
Add a cache that pickles the parsed FragmentFile object list to
<output>.lfcache. The cache is keyed on fragment file paths+mtimes
plus sdkconfig and kconfig mtimes — sdkconfig is included because
fragment if/elif/else blocks are evaluated against it at parse time.
This avoids the pyparsing cost on rebuilds where ldgen has to run
but the fragment files themselves are unchanged — for example when
a function is added or removed in a source file. Cache hits produce
the same sections.ld as a full parse, and any pickle load failure
falls through to a normal parse, so cache corruption or a Python
upgrade cannot produce a wrong build.
Print "Skipping linker fragment parsing, fragment files unchanged"
on the lf cache hit path so build logs show when the optimization
fired — this makes it easy to diagnose user reports of unexpected
ldgen behavior.
The LDGEN_NO_CACHE environment variable disables the lf cache. When
set, ldgen falls through to a normal parse without consulting or
writing the cache file.
Measured on wifi_station (205 libs, 67 .lf files), median of 10 runs:
before after
rebuild, section change, .lf unchanged 5.82s 1.34s
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
After running objdump on all libraries, extract section names from the
stored raw output using regex and compute a fingerprint (MD5 hash of
section names + mtimes of fragment files, sdkconfig, and kconfig).
If the fingerprint matches the cached value from the previous run,
skip the expensive pyparsing + generation step entirely and touch
the output file to update its timestamp. The fingerprint file is
stored next to the output (e.g. sections.ld.fingerprint) in the build
directory.
This avoids the pyparsing bottleneck for the common case of editing
function bodies without adding or removing symbols — section names
in the object files are unchanged, so the generated sections.ld
would be identical. When sections do change, the full generation
runs and the fingerprint is updated.
Print "Skipping linker script generation, section names unchanged"
on the fingerprint hit path so build logs show when the optimization
fired. This makes it easy to diagnose user reports of unexpected
ldgen behavior — the build log directly shows whether the cache was
used.
Add an LDGEN_NO_CACHE environment variable to disable the cache as a
workaround in case the optimization causes problems. When set, ldgen
prints "Linker script generation caches disabled by LDGEN_NO_CACHE"
and runs full generation as before, leaving any existing fingerprint
file untouched.
Measured on wifi_station (205 libs, 67 .lf files), median of 10 runs:
before after
rebuild, section names unchanged 5.82s 0.82s
Closes https://github.com/espressif/esp-idf/issues/18408
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Changes generated by pre-commit related to ruff-format and an update to
the copyright date.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add the `--debug` option to print the entity tree to stdout. This can be
useful for viewing the entire entity tree state to identify potential
issues or incorrect placements.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Introduce a new `--mutable-libraries-file` option that accepts a file
containing the filenames of mutable libraries, each listed on a separate
line. Mutable libraries are component libraries expected to change
during development. In contrast, immutable component libraries are not
expected to change. In the generated linker script, the input sections
of mutable libraries are grouped together rather than being mixed with
those of immutable libraries. The goal is to create large continuous
areas in the ELF file's output sections that remain unchanged for
immutable libraries during application recompilation, allowing these
areas to be skipped during flashing.
The build system identifies the mutable libraries and passes them to
ldgen using the `--mutable-libraries-file` option. It maintains
information about component sources, one of which is
`project_components`. This source type identifies components that are
directly related to the project being developed and are very likely to
change.
Mappings for mutable libraries are explicitly created for all sections
in the default scheme. This happens before the entity
`(archive:object_file:symbol/input_section)` node tree with placements is
generated and is equivalent to having these mappings in the mapping
linker fragment. All placements for mappings, whether newly added or
already existing as defined in linker fragments, associated with mutable
libraries are flagged as `mutable` in the entity node tree. This flag
ensures that these placements are included in the final linker script.
Currently, ldgen only emits placements that are either significant or
forced. A placement is considered significant if, for example, it is not
already covered by a placement in parent node. For instance, `*(.iram1
.iram1.*)` placement already includes `*libapp_trace.a:(.iram1
.iram1.*)`, so the latter is not emitted by default. The `mutable` flag
ensures that placements for mutable libraries are emitted in the linker
script and placed at dedicated location.
The locations where placements for mutable libraries are specified in
the linker script are identified by a new `mutable` marker, for example,
`mutable[flash_text]`. The placements for immutable libraries remain in
the existing `mapping` marker, for example, `mapping[flash_text]`. The
`mutable` marker for each target is placed after the `mapping` marker.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This commit introduce SOC_MEM_NON_CONTIGUOUS_SRAM flag (that enebled for
esp32p4). If SOC_MEM_NON_CONTIGUOUS_SRAM is enabled:
- LDFLAGS+=--enable-non-contiguous-regions
- ldgen.py replaces "arrays[*]" from sections.ld.in with objects under
SURROUND keyword. (e.g. from linker.lf: data -> dram0_data SURROUND(foo))
- "mapping[*]" - refers to all other data
If SOC_MEM_NON_CONTIGUOUS_SRAM, sections.ld.in file should contain at
least one block of code like this (otherwise it does not make sense):
.dram0.bss (NOLOAD) :
{
arrays[dram0_bss]
mapping[dram0_bss]
} > sram_low
.dram1.bss (NOLOAD) :
{
/* do not place here arrays[dram0_bss] because it may be splited
* between segments */
mapping[dram0_bss]
} > sram_high
When using a Linux system configured with `zh_CN.UTF-8` as `$LANG`,
and running raw cmake command to build the project (rather than using
`idf.py build`), output of objdump will be Chinese
(like `在归档文件 libesp_pm.a 中`), resulting in parsing error
`pyparsing.ParseException: Expected "In archive" (at char 0), (line:1, col:1)`
at entity.py line 129.
This commit forces objdump to use raw locale setting (`C`), to ensure
it always make English output that's able to be parsed.
Closes https://github.com/espressif/esp-idf/pull/7903
Previously ldgen determines the output file name on its own. This commit
makes it so that user can dictate what the output file name will be
for the processed template, if the user needs it for something else.
Frantisek Hrbata
Vojtech Piroch
Jakub Kocka
Marek Fiala
Alexey Lapshin
simon.chupin
Fu Hanxi
Ivan Grokhotkov
Tian Yunhao
Renz Bagaporo
Roland Dobai
Angus Gratton
Renz Christian Bagaporo
Roland Dobai