esp-idf-sbom reports five ESP-IDF CVEs against this branch because NVD
pins them to 5.5.4 -- the version release/v5.5 still reports until 5.5.5
is released -- even though the fixes are already merged here:
- CVE-2026-45160 DHCP server OOB read (d51b107609)
- CVE-2026-45328 ESP-TEE hardening against REE manipulation (b471faf803)
- CVE-2026-45329 ESP-TEE service-call input validation (eebabaff2f, dcb758a30b)
- CVE-2026-45541 esp_http_server WebSocket NULL dereference (00a2f7fbbb)
- CVE-2026-45542 protocomm SRP6a heap overflow (71eb2dbe6a)
esp-idf-sbom merges this repository-local excluded_cves.yaml into its
exclusion list when scanning the tree, so these CVEs are reported as
excluded for this branch while the released v5.5.4 tag, which predates
this file, is still reported. Once version.cmake is bumped to 5.5.5 the
entries become no-ops (NVD does not list 5.5.5) and can be removed.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When version.txt does not exist and git describe fails (e.g. in
release archives or environments without git), IDF_VER was set to
the raw git_describe output which resolves to "-128-NOTFOUND",
causing esp_get_idf_version() to return a garbled string.
Add a fallback that constructs the version string from the
IDF_VERSION_MAJOR, IDF_VERSION_MINOR and IDF_VERSION_PATCH
variables when git describe is not available.
Closes https://github.com/espressif/esp-idf/issues/18240
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The minimal build property is simply a shorthand for `set(COMPONENTS
main)`. The issue is that there is currently no check to verify whether
the `main` component actually exists or is known to the build system.
If the `main` component is not present, print an error message along
with suggestions on how to fix this inconsistency.
Closes https://github.com/espressif/esp-idf/issues/18219
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
If `.gitlab/ci/rules\.yml` is changed, make sure the
`check-tools-files-patterns` pre-commit hooks is triggered to verify
that all `tools/*` files are covered by rules.yml.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The MINIMAL_BUILD build property and the COMPONENTS variable are both
used to determine the initial component list for the build.
Currently, if the COMPONENTS variable is set, the MINIMAL_BUILD logic
is ignored during component selection, but the MINIMAL_BUILD build
property remains set. This leads to an inconsistent state where
menuconfig displays information indicating MINIMAL_BUILD is active,
even though it was ignored in favor of the COMPONENTS variable.
Fix this by setting the MINIMAL_BUILD property to OFF if the
COMPONENTS variable is used.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Resolve ruff's UP031 errors related to the use of percent formatting for
strings and lines longer than 120 characters.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
As part of inter-procedural optimizations (IPA), the compiler may
perform tasks such as constant propagation for functions. This involves
generating a specialized version of a given function with a new symbol
name that includes a suffix. For example, during constant propagation,
the compiler might create a specialized version named
`spiflash_start_core.constprop.0` for the `spiflash_start_core`
function. Additionally, the compiler may generate multiple clones of a
single function. Currently, when ldgen performs symbol placement, it
does not account for these compiler-generated functions, leading to
their incorrect or unexpected placement in memory (markers).
Consider a linker fragment with:
```
[mapping:spi_flash]
archive: libspi_flash.a
entries:
esp_flash_api: spiflash_start_core (noflash)
```
The `spiflash_start_core` function should be placed in IRAM. However,
the compiler might generate an optimized version of this function with a
`.constprop.0` suffix, resulting in a
`.text.spiflash_start_core.constprop.0` input section. Currently, ldgen
does not handle this situation, leading to misplaced symbols.
Since `.` is not allowed in C identifiers, it should be safe to consider
all input sections for a symbol with any `.` suffix as representing that
symbol. This means considering the symbol suffixes should not cause any
ambiguity.
This change automatically places all input sections, including those
with possible suffixes for a given symbol, into the specified memory. In
other words, specifying a function name like `spiflash_start_core` in a
linker fragment automatically includes input section names matching
`spiflash_start_core(\..*)?$`.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The commit 51f3c02115 ("Add quotes around node ID in dependency
graph") introduced proper quoting for node IDs in the generated dot
file. Since the dot format does not allow dashes in node names without
quoting, and dashes are used in component names, the component names in
the generated dot file are now correctly quoted. Adjust the
g1_components test to accommodate this change by stripping the quotes.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The latest cJSON vulnerabilities appear to list "davegamble" as the
vendor instead of "cjson_project." Add a new CPE with "davegamble" as
the vendor.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The `setuptools` package starting with `v70.1.0`[1] contains built-in
`bdist_wheel` command. Before this version `setuptools` relied on the
`bdist_wheel` command implementation from the `wheel` package. Starting with
`setuptools` `v75.8.1` the `PEP 491`[3] restrictions on the distribution name
of a wheel package are enforced[4], replacting also `.` with `_`. Note that
`PEP 491` actually allows `.` in the distribution name, but for some reason the
latest packaging docs[10][11] does not, stating that `.` should be replaced
with `_`. This was discussion here[12].
Also the `wheel` package starting with `v0.45.0`[5] is using the `bdist_wheel`
command from `setuptools`. This means that any package which has `.` in its
distribution name, like `ruamel.yaml.clib`, can have different wheel name,
depending on which version of the `bdist_wheel` command was used.
The `bdist_wheel` command from setuptools prior `v75.8.1` or `wheel` prior
`v0.45.0` will keep the dots in distribution name preserved. For exaple the
`ruamel.yaml.clib` package will have distribution name
`ruamel.yaml.clib-0.2.12.dist-info. Newer versions will replace the dots with
`_` according to [10][11], creating distribution like
`ruamel_yaml_clib-0.2.12.dist-info`.
From packaging point of view `ruamel.yaml.clib-0.2.12.dist-info` and
`ruamel_yaml_clib-0.2.12.dist-info` are the same packages, but this is not
reflected in `importlib.metadata` prior python 3.10[9], which does not perform
name normalization prior the distribution search. This causes the `version`
from `importlib.metadata` to fail on python prior the 3.10 version if the
package with dots in distribution name was generated with normalized paths with
newer `setuptools`. Note that the distribution name normalization was
backported to some later 3.9 python version.
Let's demonstrate this behavior on a simple package with the
`my.minimal.package` name.
```
my_minimal_package/
├── pkg
│ └── __init__.py
└── setup.py
from setuptools import setup, find_packages
setup(
name='my.minimal.package',
version='0.1.0',
packages=find_packages(),
install_requires=[],
entry_points={},
)
```
With python 3.9.0 search for `my.minimal.package` fails because
of the missing name normalization.
```
docker run --rm -it --platform linux/x86_64 python:3.9.0 bash
python -m venv venv
. venv/bin/activate
pip install setuptools==v75.8.1
python setup.py bdist_wheel
pip install dist/my_minimal_package-0.1.0-py3-none-any.whl
python
Python 3.9.0 (default, Nov 18 2020, 13:28:38)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib.metadata import version as get_version
>>> get_version('my.minimal.package')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/importlib/metadata.py", line 551, in version
return distribution(distribution_name).version
File "/usr/local/lib/python3.9/importlib/metadata.py", line 524, in distribution
return Distribution.from_name(distribution_name)
File "/usr/local/lib/python3.9/importlib/metadata.py", line 187, in from_name
raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: my.minimal.package
>>> get_version('my_minimal_package')
'0.1.0'
```
With python 3.10.0 search for both `my.minimal.package` and
`my_minimal_package` succeeds.
```
docker run --rm -it --platform linux/x86_64 python:3.10.0 bash
python
Python 3.10.0 (default, Dec 3 2021, 00:21:30) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib.metadata import version as get_version
>>> get_version('my.minimal.package')
'0.1.0'
>>> get_version('my_minimal_package')
'0.1.0'
```
In our `tools/check_python_dependencies.py` we cannot relay on the default
distribution finder, used in the `version` function from `importlib.metadata`,
to do name normalization on older python versions. To cope with this,
implement a fallback version search. If `version` fails with
`PackageNotFoundError`, do the name normalization according to [10][11] and try
again.
Note: There is also a `wheel`[6][7] `v0.43.0` package embeded in `setuptools`
along with the new implementation[8]. This one seems to be used if the
external `wheel` package is not available but imported. TBH this is all kinda
messy and I may have overlooked something.
* [1] https://setuptools.pypa.io/en/stable/history.html#v70-1-0
* [2] https://setuptools.pypa.io/en/stable/history.html#v75-8-1
* [3] https://peps.python.org/pep-0491/#escaping-and-unicode
* [4] https://github.com/pypa/setuptools/pull/4766/files
* [5] https://wheel.readthedocs.io/en/stable/news.html
* [6] https://github.com/pypa/setuptools/blob/main/setuptools/_vendor/wheel/__init__.py
* [7] https://github.com/pypa/setuptools/issues/1386
* [8] https://github.com/pypa/setuptools/blob/main/setuptools/command/bdist_wheel.py
* [9] https://github.com/python/cpython/commit/c6ca368867bd68d44f333df840aa85d425a51410
* [10] https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization
* [11] https://packaging.python.org/en/latest/specifications/binary-distribution-format/
#escaping-and-unicode
* [12] https://github.com/pypa/setuptools/issues/3777
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In commit 70407df8c2 ("fix(build): don't call enable_language() before
proje.."), the C standard preferences were mistakenly changed to
`set(preferred_c_versions gnu99 gnu17 gnu11 gnu99)`. This was likely an
oversight from my testing. This corrects the C standard preferences to
the intended order.
Fixes: 70407df8c2 ("fix(build): don't call enable_language() before proje..")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
For the Linux target, we currently attempt to fallback to older C/CXX
lagnuage standards in the __build_set_lang_version() function. The
language standard support is checked using CMake's language-specific
functions, such as check_c_compiler_flag(). These functions require the
language to be enabled[1] in CMake beforehand, which is done by calling
project() or by enabling the languages later with enable_language(). At
present, we use enable_language() to enable C and CXX languages in
CMake, allowing us to set the standard early, before invoking project().
However, newer CMake versions (>3.29) issue a warning[2] if
enable_language() is called before project(), as noted in CMP0165[3].
It should generally be acceptable to call __build_set_lang_version()
after __project(), but doing so would alter the behavior of the
COMPILE_OPTIONS also for non-Linux targets. Currently, users can
add to COMPILE_OPTIONS even before calling project() in the project's
CMakeLists.txt and the options will be in the desired order. In other
words, appending to COMPILE_OPTIONS can occur either before or after
calling project() in the project's CMakeLists.txt, with the outcome
remaining consistent. This means the user's settings will appear later
and take priority. However, if __build_set_lang_version() is called
after __project(), the user's COMPILE_OPTIONS settings would be
overridden if set before calling project(). Our documentation[4] explicitly
states that COMPILE_OPTIONS and similar properties should be modified
using idf_build_set_property() after calling project() to prevent
default values from overwriting them.
Even with this guidance, some existing components that modify
COMPILE_OPTIONS before invoking project() might be impacted by this
change. Therefore, separate the language standard settings for non-Linux
and Linux targets. For non-Linux targets, these settings are applied in
__build_set_default_build_specifications(), maintaining the current
behavior. For the Linux target, the language standard is set with
__linux_build_set_lang_version() after calling __project(), ensuring the
languages are already enabled in CMake and no warning is issued. Since the Linux
target is still in preview, this approach should be acceptable,
especially with the existing documentation[4].
Closes https://github.com/espressif/esp-idf/issues/15488
[1] https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html#languages
[2] https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9396
[3] https://cmake.org/cmake/help/latest/policy/CMP0165.html#policy:CMP0165
[4] https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/api-guides/
build-system.html#overriding-default-build-specifications
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When minimal build is enabled, it only includes the main component and
its dependencies. This leads to test components specified through
TEST_COMPONENTS being ignored, meaning no tests are executed. The issue
arises because test components are also checked against the COMPONENTS
variable, and if they aren't listed there, they are disregarded. To fix
this, explicitly add TEST_COMPONENTS to COMPONENTS when the minimal
build is enabled.
Closes https://github.com/espressif/esp-idf/issues/15485
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The `idf_as_lib` example is used and tested in
`tools/test_build_system/test_cmake.py`. Include `idf_as_lib` in the
build_system rules to ensure the tests are executed whenever there is a
modification in the `idf_as_lib` example.
Expand the `test_build_custom_cmake_project` test to cover all supported
targets, including host build.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add the initial documentation for the idf.py diagnostic tool, detailing
how it can be used to gather information for troubleshooting issues
related to the ESP-IDF project.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Replace the original diag implementation in idf.py with the new one in a
separate esp-idf-diag python package. The interface is kept the same as
it was. The simple idf.py diag test is also preserved.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
At present, the diag tool uses its default purge file. However, users
may find it beneficial to specify and reuse their own purge file. A new
command line option, --purge, has been introduced to allow users to
provide their own purge file to diag. When this option is used, the
default purge file is ignored.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This should enable the collection of more detailed information about the
operating system and basic CPU details.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
At present, the steps are executed unconditionally. With this
modification, we can restrict each step individually to determine if it
should run on a particular system. For instance, we can execute
different commands on different systems. This is achieved by adding a
new key, "system," to the step dictionary, with possible values being
Linux, Windows, and Darwin.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, if a command run by the exec command returns an error
code, its stdout and stderr are not saved. It could be beneficial to
store at least the stderr if requested. Additionally, avoid creating
output files when there is no content and also store the stderr
of the failed command to diag.log.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When a symbol needs to be placed to a different target than the one
designated for the object file, the object file is expanded, which
includes the following steps:
1. Creating a new placement for the symbol's input section with the
specified target.
2. Excluding the object placement from the orignal target.
3. Creating a new intermediate placement for the object for the original
target, where its input sections are expanded, excluding the input
section for the symbol.
Let's illustrate the object expansion process with the following example:
[sections:rodata]
entries:
.rodata+
.sdata2+
.srodata+
[scheme:default]
entries:
text -> flash_text
rodata -> flash_rodata
[scheme:noflash]
entries:
text -> iram0_text
rodata -> dram0_data
[mapping:soc_pm]
archive: libsoc.a
entries:
gpio_periph: GPIO_HOLD_MASK (noflash)
gpio_periph section headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000000 000034 000000 00 AX 0 0 2
[ 2] .data PROGBITS 00000000 000034 000000 00 WA 0 0 1
[ 3] .bss NOBITS 00000000 000034 000000 00 WA 0 0 1
[ 4] .rodata.GPIO_HOLD_MASK PROGBITS 00000000 000034 000058 00 A 0 0 4
[ 5] .rodata.GPIO_PIN_MUX_REG PROGBITS 00000000 00008c 000058 00 A 0 0 4
[ 6] .debug_info PROGBITS 00000000 0000e4 0000d8 00 0 0 1
[ 7] .rela.debug_info RELA 00000000 0009d4 000108 0c I 16 6 4
[ 8] .debug_abbrev PROGBITS 00000000 0001bc 000070 00 0 0 1
[ 9] .debug_aranges PROGBITS 00000000 00022c 000018 00 0 0 1
[10] .rela.debug_aranges RELA 00000000 000adc 00000c 0c I 16 9 4
[11] .debug_line PROGBITS 00000000 000244 0001ab 00 0 0 1
[12] .debug_str PROGBITS 00000000 0003ef 00022d 01 MS 0 0 1
[13] .comment PROGBITS 00000000 00061c 000030 01 MS 0 0 1
[14] .note.GNU-stack PROGBITS 00000000 00064c 000000 00 0 0 1
[15] .riscv.attributes RISCV_ATTRIBUTES 00000000 00064c 000044 00 0 0 1
[16] .symtab SYMTAB 00000000 000690 000260 10 17 36 4
[17] .strtab STRTAB 00000000 0008f0 0000e1 00 0 0 1
[18] .shstrtab STRTAB 00000000 000ae8 0000d1 00 0 0 1
1. Creating a new placement
.dram0.data :
{
*libsoc.a:gpio_periph.*(.rodata.GPIO_HOLD_MASK .sdata2.GPIO_HOLD_MASK .srodata.GPIO_HOLD_MASK)
}
2. Excluding the object placement
.flash.rodata :
{
*(EXCLUDE_FILE(*libsoc.a:gpio_periph.*) .rodata.* ...)
}
3. Creating a new intermediate placement
.flash.rodata :
{
*libsoc.a:gpio_periph.*(.rodata.GPIO_PIN_MUX_REG)
}
Now, let's do the same, but also move GPIO_PIN_MUX_REG to noflash with an updated mapping.
[mapping:soc_pm]
archive: libsoc.a
entries:
gpio_periph: GPIO_HOLD_MASK (noflash)
gpio_periph: GPIO_PIN_MUX_REG (noflash)
1. Creating a new placement
.dram0.data :
{
*libsoc.a:gpio_periph.*(.rodata.GPIO_HOLD_MASK .sdata2.GPIO_HOLD_MASK .srodata.GPIO_HOLD_MASK)
*libsoc.a:gpio_periph.*(.rodata.GPIO_PIN_MUX_REG .sdata2.GPIO_PIN_MUX_REG
.srodata.GPIO_PIN_MUX_REG)
}
2. Excluding the object placement
.flash.rodata :
{
*(EXCLUDE_FILE(*libsoc.a:gpio_periph.*) .rodata.* ...)
}
3. Creating a new intermediate placement
.flash.rodata :
{
*libsoc.a:gpio_periph.*
}
The *libsoc.a:gpio_periph.* entity in step 3 no longer has input
sections, as there are no remaining .rodata input sections in the object
file. The linker behavior for this mapping is to include all object
input sections that have not yet been placed as described in
https://sourceware.org/binutils/docs/ld.html#Input-Section-Basics
"If you use a file name without a list of sections, then all sections in
the input file will be included in the output section. This is not
commonly done, but it may by useful on occasion."
The map file for such mapping now contains following input sections
.flash.rodata 0x3c0a0120 0x19b34
*libsoc.a:gpio_periph.*()
.debug_info 0x3c0b95bf 0xd8 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.debug_abbrev 0x3c0b9697 0x70 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.debug_aranges
0x3c0b9707 0x18 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.debug_line 0x3c0b971f 0x1ab esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.debug_str 0x3c0b98ca 0x21a esp-idf/soc/libsoc.a(gpio_periph.c.obj)
0x22d (size before relaxing)
.comment 0x3c0b9ae4 0x30 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.note.GNU-stack
0x3c0b9ae4 0x0 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.riscv.attributes
0x3c0b9ae4 0x44 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
This is incorrect, and such intermediate placement should not be
generated. This type of placement can be recognized because it is not
explicitly defined in the mapping and lacks input sections. We can
identify this in the significant function and prevent issuing commands
for such placement.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Set the COMPONENTS project variable to streamline the build process.
Although the minimal build feature could be used to specify plugin
components in the main component dependencies, it's more convenient to
set the plugin components at the project level. This approach also
allows for quick selection of which plugins to include in the
application.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The initial implementation of a diagnostic tool that collects valuable
information about esp-idf and failed build to assist in investigating
reported issues.
The gathered information includes environmental variables, details about
the python virtual environment, installed tools, platform information,
project_description.json, sdkconfig, build logs, map file, linker
scripts, and others.
usage:
1) create the default report
# allow diag to create the report directory name
$ idf.py diag
# explicitly specify the report directory
$ idf.py diag --output <report directory>
2) examine the contents of the generated <report directory> for
sensitive information and add additional content to the
<report directory>
3) create report archive zip file that can be shared or attached to
the reported issue
$ idf.py diag --zip <report directory>
The tool collects information as described in what are known as recipe
files. A recipe file is a YAML file, similar to an Ansible playbook or a
GitHub action, but much more simplified. Each recipe outlines how to
gather a set of related information. For instance, the manager.yml
recipe gathers data related to the component manager. Each recipe
includes metadata such as its description, tags, and steps. Tags are
used to determine which recipes to use; by default, all built-in recipes
located in tools/idf_py_actions/diag/recipes are used. Steps consist of
a list of commands to be executed. Currently, there are four commands:
file, exec, env, and glob. For more detailed information about recipes,
their format, and commands, please refer to
tools/idf_py_actions/diag/recipes/README.md.
Recipe example for component manager:
description: IDF Component Manager information
tags: [manager, base, project]
output: manager
steps:
- name: 'IDF Component Manager'
cmds:
- exec:
cmd: 'python -m idf_component_manager version'
output: manager.ver
- file:
path: '${PROJECT_DIR}/dependencies.lock'
- glob:
# Gather all idf_component.yml files from the project directory and
# save them in directories relative to the project directory within
# the idf_component directory.
pattern: 'idf_component.yml'
recursive: true
relative: true
path: '${PROJECT_DIR}'
output: 'idf_component/'
Create report for manager
1) all recipes with manager tag
$ idf.py diag --tag manager
2) use only the manager recipe explicitly; built-in recipes can be
referenced simply by their name, but all recipes can be referenced
by their path
$ idf.py diag --recipe manager
or
$ idf.py diag --recipe <full path>
To display available recipes, use
$ idf.py diag --list
and to verify recipes, use
$ idf.py diag --check
Both --list and --check honers the --tag and --recipe options.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currrently, all logs generated by RunTool are stored in files named
idf_py_(stdout|stderr)_output_$$, making it difficult to identify which
log corresponds to which command. To simplify this for idf-diag, include
the command arguments at the beginning of the log. This will allow
idf-diag to use regex to differentiate logs for build, monitor, flash,
and other commands and targets.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, we are only suggesting that the header file is likely not
part of the component's INCLUDE_DIRS. However, the header file may be
missing also because of the configuration settings. For instance, the
component might be disabled in sdkconfig, or the feature that supplies
the header might not be enabled. Enhance the hint message to address
this scenario as well.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The active.py script is currently clearing the IDF_PYTHON_ENV_PATH,
preventing it from being set to a custom location for the python virtual
environment directory. Although the install script checks to ensure that
an existing python virtual environment is not overwritten with one for a
different ESP-IDF version than it was originally created for, we should
still permit setting a custom path for the python virtual environment.
Closes https://github.com/espressif/esp-idf/issues/15006
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Enable @-argument completion only if '@' is not present in
COMP_WORDBREAKS. When '@' is included, the @-argument is not considered
part of the completion word, causing @-argument completion to function
unreliably in bash.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Merge Request !31081 introduced support for file autocompletion. This
adds a basic test for README files autocompletion in the root directory
of the esp-idf repository.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
esp-idf-sbom offers two ways to perform vulnerability scanning. The
primary method, which is the default, uses the NVD REST API. The
alternative method uses the esp-nvd-mirror repository. If there are
issues with accessing the NVD REST API, it can be useful to switch to
the esp-nvd-mirror easily. Allow to set the SBOM_CHECK_LOCAL_DB github
repository variable to switch to esp-nvd-mirror.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
With MINIMAL_BUILD enabled, menuconfig will only display configuration
options for components included in the build through main component
dependencies. This means that configuration options for excluded
components will not appear in menuconfig. This might be confusing, as
most examples currently do not use a trimmed build, and all components
detected by the build system are available in menuconfig, even if their
configuration has no effect because the components may not be linked or
used. Adding a note in the components menu of menuconfig could help
users realize they need to add a component to the main component
requirements to see its configuration options. Unfortunately, Kconfig
does not support multiline comments, so there are three separate
comments intended to adequately describe the potentially missing
component configurations.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Include a description of the MINIMAL_BUILD property, which serves as a
shortcut for set(COMPONENTS main). Additionally, add a note to the
relevant components indicating that they must be included in the build
to allow for their configuration.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The test_build_cmake_library_psram_workaround verifies that the
mfix-esp32-psram-cache-issue flag is applied to all compile commands in
compile_commands.json when CONFIG_SPIRAM_CACHE_WORKAROUND=y. Although
the import_lib does not need the esp_psram component, this test does. To
avoid adding the esp_psram dependency to the import_lib, disable
MINIMAL_BUILD for this test by explicitly setting the COMPONENTS
variable.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The check_public_headers.py script does not function well with
MINIMAL_BUILD enabled, as some headers from excluded components may be
missing. While it might be possible to address this issue with
exclusions, it is likely more effective and provides more coverage to
include all components, not just those required by the blink example.
Therefore, MINIMAL_BUILD has been disabled for this test, ensuring that
all registered components are included in the build. If defined, the
COMPONENTS variable takes precedence over the MINIMAL_BUILD property, so
setting "-DCOMPONENTS=" overrides the MINIMAL_BUILD property.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, several example dependencies rely on the fact that all
registered components are added to the build, along with components
specified in common requirements. This results in longer build times
because even unused components must be built. Switch all examples to use
idf_minimal_build to compile only the components actually required by
the example.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Introduce a MINIMAL_BUILD property to streamline the build process by
including only the main component and its direct and indirect
dependencies. This serves as a convenient shortcut for using
set(COMPONENTS main). The MINIMAL_BUILD build property is ignored if the
COMPONENTS variable is defined, as COMPONENTS takes precedence.
To enable a minimal build, add "idf_build_set_property(MINIMAL_BUILD ON)"
to the project's CMake configuration.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, several examples do not explicitly state their component
dependencies, relying instead on the default behavior that includes all
registered components and commonly required ones in the build.
Explicitly adding component dependencies can reduce build time when
set(COMPONENTS main) is used.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The object files are compiled simultaneously, which can result in
several error messages about missing headers. The current regex used to
find missing headers in the compiler's output employs a greedy search,
potentially capturing multiple error messages from different compilation
units. This can cause bug reports where the original component cannot be
identified. Strengthen the regex to ensure it only processes the first
reported error.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the new line ending conversion options for stdin and stdout
are specifically described for UART, which is inaccurate since the
console could be connected to USB serial/CDC or have secondary output.
Use a generic console term instead of UART.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The default newlib read/write syscalls automatically convert newline
endings from LF to CRLF on stdout. Therefore, the test should not
specifically check for LF in the expected output, as the standard
conversion for stdout is CRLF. While it is possible to change the
expected line ending in the test from LF to CRLF, it seems more
dependable to just avoid verifying line endings entirely.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The configuration of newline endings for stdout and stdin is supported
only by the VFS drivers. The write and read syscalls in newlib
automatically convert LF to CRLF for stdout and CR to LF for stdin,
ignoring the configured newline settings. Therefore, make the options
for newline endings visible only when VFS is enabled.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the newline conversions for the NEWLIB_STDOUT_LINE_ENDING and
NEWLIB_STDIN_LINE_ENDING options are only applied by VFS drivers.
Without VFS, newline conversion does not occur. Introduce the default
conversion of LF to CRLF on stdout and CF to LF on stdin for the default
read and write syscalls in newlib.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a basic description and usage overview of the esp-idf-sbom tool. A
more detailed description is already available on the GitHub pages,
which this documentation refers to.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The ImportError or ModuleNotFoundError might be raised without
specifying a module name. In this not so common situation, re-raise the
exception to print all the information that could assist in identifying
the problem.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
By default, esp-idf-size.ng displays all sections individually. This can
be confusing, especially if CONFIG_SOC_MEM_NON_CONTIGUOUS_SRAM is
enabled, resulting in sections like .dram0.data and .dram1.data being
abbreviated as two .data sections in the size report. To avoid confusion
for idf.py and cmake users, pass the --unify option to the underlying
esp_idf_size.ng by default.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the `SORT` flag mandates the inclusion of at least the
`sort_by_first` argument in the grammar, despite the documentation[1]
indicating that `SORT` can be utilized without any arguments, defaulting
to sorting input sections by name. Fix this by modifying the grammar
to allow a default `SORT` and update a test accordingly.
[1] https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/
linker-script-generation.html
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Explicitly set shell type in export.sh for bash and zsh
-Most of the issues reported with the updated export scripts are related
to shell detection. Since we already know the shell type for commonly
used ones like bash or zsh, we can pass the shell type directly to the
activate script. This should hopefully resolve the shell detection
problems for most users.
This update also modifies the shell type detection to rely solely on
psutil.Process().cmdline() rather than psutil.Process().exe(). This
change aims to resolve permission issues that may occur if, for example,
the Python binary has certain capabilities assigned.
Move shell completion to the init script
- Currently we are expanding the autocompletion in the activate script and
adding the expanded commands into the init script. To avoid
concerns about shell versions, move the entire expansion to the init
script.
Currently, we are using psutil.Process().name(), which includes the
command as entered on the command line. This becomes an issue if the
export script is initiated within another script.
run.sh
#!/usr/bin/bash
. ./export.sh
In this case the activate script will see `run.sh` instead of bash.
/proc/<pid>/com vs /proc/<pid>/exe
Use exe(), which contains the full executable path to fix this.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, when the venv is installed or updated, we attempt to
automatically update pip and setuptools within the venv. Unfortunately,
the setuptools package is installed or updated without adhering to the
constraints file, which restricts the setuptools version due to
https://github.com/pypa/setuptools/issues/4480. Resolve this issue by
applying the constraints file to the installation and update of both pip
and setuptools.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The documentation for the IDF Size tool has been relocated from the
performance section to the tools section and updated to reflect the
current refactored version of esp-idf-size.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
There is currently a bug in the __build_resolve_and_add_req function in
tools/cmake/build.cmake where the check for registered component
requirements is incorrectly applied to the component itself rather than
its dependencies. This issue likely originated from a typo, using
component_target instead of _component_target. To prevent further
confusion, _component_target has been renamed to _req_target.
Fixing this revealed multiple incorrect dependencies for the Linux
target, which have now been resolved by explicitly specifying the
dependencies for the Linux target.
Closes https://github.com/espressif/esp-idf/issues/13447
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This adds a possibility to specify --diff option to idf.py size,
size-components and size-files commands. This can be map file directly,
project directory or build directory.
Usage example:
idf.py size-components --diff ../hello_world2/build/hello_world.map
idf.py size-components --diff ../hello_world2/build
idf.py size-components --diff ../hello_world2
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This adds a simple test that tries to run idf_size.py help and check
if the process does not exit with error. This is just to make sure
that idf_size.py can be used with minimum required python version.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Previous 6caa4a17ac ("fix: display correct help in the idf_size.py wrapper")
introduced a regression, because it uses exit_on_error parameter for
argparse.ArgumentParser, which was added in python3.9, making
idf_size.py incompatible with idf.py minimal required python3.8.
The objective is to inspect the arguments of idf_size.py using a wrapper
argparse to determine whether the legacy or refactored version should be
initiated, while always displaying help for the underlying version. The
exit_on_error function was previously utilized to prevent argparse from
exiting and displaying help/usage. This replaces exit_on_error with a
workaround that makes the --format argument optional. Since this is the
sole instance where the wrapper argparse might fail, it achieves the
same outcome as using exit_on_error.
Fixes: 6caa4a17ac ("fix: display correct help in the idf_size.py wrapper")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently the wrapper tries to figure out which version of
the esp-idf-size should be started. The legacy version is
used if explicitly requested by the -l/--legacy option or
if json format is specified. This works fine, but if help
is requested, it is printed for the wrapper as shown bellow
$ idf_size.py -h
usage: idf_size.py [-h] [--format FORMAT] [-l]
options:
-h, --help show this help message and exit
--format FORMAT
-l, --legacy
This is not convenient and the full help from the underlying
version should be displayed.
Fix this by only peeking into the args to figure out if legacy or
refactored version should be started and always spawn the underlying
esp_idf_size python module. This is done by using exit_on_error=False and
add_help=False for the ArgumentParser. When help for refactored version
is requested a note as following is printed to notify users that the
legacy version can still be used.
$ idf_size.py -h
Note: legacy esp_idf_size version can be invoked by specifying the -l/--legacy
option or by setting the ESP_IDF_SIZE_LEGACY environment variable.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently when process is started through asyncio Runner and it is termited
e.g. with SIGINT(ctrl+c) a traceback is printed instead of gracefully
exit.
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7fe980970900>
Traceback (most recent call last):
File "/usr/lib64/python3.12/asyncio/base_subprocess.py", line 129, in __del__
self.close()
File "/usr/lib64/python3.12/asyncio/base_subprocess.py", line 107, in close
proto.pipe.close()
File "/usr/lib64/python3.12/asyncio/unix_events.py", line 568, in close
self._close(None)
File "/usr/lib64/python3.12/asyncio/unix_events.py", line 592, in _close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib64/python3.12/asyncio/base_events.py", line 793, in call_soon
self._check_closed()
File "/usr/lib64/python3.12/asyncio/base_events.py", line 540, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
This is caused because asyncio Runner context in asyncio.run is closing the event
loop and if exception is unhandled in coroutine(run_command) the transport is not
closed before the even loop is closed and we get RuntimeError: Event loop is closed
in the transport __del__ function because it's trying to use the closed
even loop.
Let's catch asyncio.CancelledError in case the process we are trying to
read from is terminated, print message, let the asyncio finish and exit
gracefully.
Closes https://github.com/espressif/esp-idf/issues/13418
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
It seems gcc is not producing debug information for sections which
are not properly marked as "ax", resulting in missing debug info
for _vector_table, _interrupt_handler and _panic_handler. This can be
verified e.g. with
readelf --debug=info ./build/esp-idf/riscv/CMakeFiles/__idf_riscv.dir/vectors.S.obj
readelf -SW ./build/esp-idf/riscv/CMakeFiles/__idf_riscv.dir/vectors.S.obj
for hello_world example on esp32c3 target. Mark the .exception_vectors.text and
.exception_vectors_table.text sections as writable and allocatable so the debug
info sections are generated.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently we silently ignore when the original component is not found
in a hope we can provide at least some meaningful hint. As it turned
out it's not true. Instead of providing misleading hint, just return
error. This adds several checks for situations, which should not happen,
but when they do it should be easier to identify the root cause of the
problem.
For example when hint module received malformed output with extra new
lines, e.g. caused by a bug in RunTool, it wrongly reported the original
component as source component.
This should also fix the tests on Windows.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently RunTool reads command's output with asyncio read, which
returns bytes. This is decoded into python's string and the output already
contains OS specific line endings, which on Windows is CRLF. Problem is
that the command output is saved by using python's text stream/file,
which replaces LF, native python's line ending, with OS specific line ending.
On Windows, and in this particular case, the CRLF from the command output is
translated into CRCRLF and saved in the commands output file. When this
file is read in again, e.g. for hint modules, the CRCRLF is replaced
with LFLF. Again the file is open as text file. Meaning a new emply line
is added.
Fix this by opening the output file with "newline=''", which prevents
this translation. We already have the OS specific line ending in the
command's output.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
With 09ba070b7e ("fix: allow to use older pre-commit hook version")
and b93cc581a5 ("fix: set default pre-commit hook stages to pre-commit..")
we set the default pre-commit hook stage to commit/pre-commit, which avoids
running the pre-commit plugins/hooks multiple times for different stages.
Unfortunately this also affects "pre-commit run" used in CI and
now it runs only plugins which are by default using the
commit/pre-commit stage. Meaning the sbom validation, which uses
post-commit, is not run at all. This adds an explicit pre-commit command
for validate-sbom-manifest plugin, which needs to be started in
post-commit stage.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
With b93cc581a5 ("fix: set default pre-commit hook stages to pre-commit..")
a dependency on pre-commit 3.3 was added. As it turned out, there are
people still using e.g. 2.16 version and this change is causing problems.
1. The minimum_pre_commit_version is checked after the config is parsed.
Meaning if there is a problem in the config file, like unknown
pre-commit for default_stages, pre-commit exits with an error not
suggesting that it should be updated.
2. "pre-commit" was added in 3.2 for consistency with git hook names and
we can use "commit" instead avoiding dependency on newer pre-commit.
For more info please see [1] and [2].
3. default_install_hook_types[4] were added in 2.18.0[3]. This was IMHO actually
a nice feature to have in our .pre-commit-config.yaml, because it
specifies which hooks should be installed when pre-commit install is
executed. Meaning it's not necessary to list the stages explicitly with the -t
options. Anyway we haven't used this feature till now, so it
hopefully should not be a problem if we remove it again.
With this change we can still avoid to start pre-commit plugins multiple
times for different stages, but also make it compatible with older
versions(2.16).
[1] https://github.com/pre-commit/pre-commit/issues/2732
[2] https://github.com/pre-commit/pre-commit/pull/2808
[3] https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md#2180---2022-04-02
[4] https://pre-commit.com/#top_level-default_install_hook_types
Fixes: b93cc581a5 ("fix: set default pre-commit hook stages to pre-commit..")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Following commit c3afbebf23 ("fix: bump esp-idf-sbom to v0.13.0 in pre-commit"),
the validation of submodule hash now relies solely on the information recorded
in the git-tree. Previously, the hash verification used submodule's
working tree hash if available. Since the new submodule hash is recorded
in git-tree only after the commit is created, we need to move the check
into post-commit, otherwise the hash validation checks the old value.
For example:
1. in .gitmodules
[submodule "components/json/cJSON"]
sbom-hash = cb8693b058ba302f4829ec6d03f609ac6f848546
2. update the cJSON
$ git -C components/json/cJSON checkout b45f48e600671feade0b6bd65d1c69de7899f2be
3. update cJSON hash in .gitmodules
[submodule "components/json/cJSON"]
sbom-hash = b45f48e600671feade0b6bd65d1c69de7899f2be
4. commit the changes
$ git commit -a -s
Step 4. will fail, because the validation is currently started in pre-commit stage,
where the hash for cJSON recorded in git-tree is still
cb8693b058ba302f4829ec6d03f609ac6f848546. The new hash b45f48e600671feade0b6bd65d1c69de7899f2be
will be stored in git-tree after the new commit is created.
Note that this means we cannot prevent the commit creation, but only
notify user about the hash inconsistency. If he/she still decides to
push it, it will fail in pre-commit checks in CI.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
If pre-commit hook does not specify stage[1], neither in
.pre-commit-config.yaml nor in .pre-commit-hooks.yaml, it's started for every
installed pre-commit hook. Limit the default stages to pre-commit only by default.
This also sets the default_install_hook_types and
minimum_pre_commit_version and sets conventional-precommit-linter to
v1.4.1.
[1] https://pre-commit.com/#confining-hooks-to-run-at-certain-stages
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
v0.13.0 contains fix for the submodule hash validation. Let's
bump its version in pre-commit, so the fix is used.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This adds a github action, which performs continuous vulnerability
scanning using the esp-idf-sbom-action github action. The test
is scheduled everyday at midnight and it's also possible to start
it as dispatched workflow. This scans all possible manifest files
in repository. The references for scanning are defined in github's
VULNERABILITY_SCAN_REFS variable and a json list. For example
['master', 'release/v5.2', 'release/v5.1', 'release/v5.0', 'release/v4.4']
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently sbom manifest is checked only in .gitmodules and
this check is done in pre-commit and also in CI. Meaning it's running
three times(pre-commit before push if user has it enabled, in CI
as there is the pre-commit run again and again with test in CI). Since
esp-idf-sbom contains a full manifest validation support and pre-commit
plugin for it, let's use it. This removes all the current sbom testing
and replaces it with a signle pre-commit plugin which validates all
manifests files(sbom.yml, idf_component.yml, .gitmodules and also
referenced manifests) in repository. Note that this checks all
manifests, not only ones which were modified. The check is reasonably
fast though, so it should not cause any problem. The reason for
validating all manifest files is that we want to make sure that the sbom
information in .gitmodules is updated too and that the hash
recorded in .gitmodules is up-to-date. Meaning submodule update
would not trigger this plugin, because no manifest was changed.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
MR espressif/esp-idf!27436 added sbom manifest file for fatfs,
but the originator is missing a prefix required by SPDX specification.
This results in malformed SPDX sbom files.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
With 8959555cee7e[1] ("setup_git_directory(): add an owner check for the top..")
git added an ownership check of the git directory and refuses to
run any git commands, even parsing the config file, if the git directory
is not owned by the current user. The "fatal: detected dubious ownership in repository"
is reported.
This fixes CVE-2022-24765[2], which allows to compromise user account. On a
multi-user system or e.g. on a shared file system, one user may create a "rogue"
git repository with e.g. core.fsmonitor set to an arbitrary command. Other user
may unwillingly execute this command by running e.g. git-diff or
git-status within the "rogue" git repository, which may be in one of the parent
directories. If e.g. PS1 is set to display information about a git
repository in CWD, as suggested in Git in Bash[3], the user do not need to run
any git command to trigger this, just entering some subdirectory under
this "rogue" git repository is enough, because the git command will be
started transparently through the script used in PS1. The core.fsmonitor
can be set to arbitrary command. It's purpose is to help git to identify changed files
and speed up the scanning for changed files.
rogue
├── .git # owned by user1
└── dir1 # owned by user2
├── dir2 # owned by user2
└── .git # owned by user2
user1 sets core.fsmonitor for git repository in rogue directory
$ git config --add core.fsmonitor "bash -c 'rm -rf \$HOME'"
user2 enters dir1 and runs e.g. git diff and triggers the core.fsmonitor command.
The ownership check may cause problems when running git commands in
ESP-IDF Docker container. For example user may run the container as
root, but the mounted project may be owned by a particular user.
In this case git will refuse to execute any git command within the
"/project" directory, because it's not owned by root. To overcome this,
git allows to set safe.directories, for which the ownership check is
skipped. The security check may be completely disabled by setting
safe.directories to "*". This solution was proposed in PR 12636[4], but
it would allow make it possible to exploit this vulnerability again.
This fix allows user to specify git's safe.directory in IDF_GIT_SAFE_DIR
environmental variable, which may be set during container startup.
The IDF_GIT_SAFE_DIR has same format as PATH and multiple directories can be
specified by using a ":" separator. To entirely disable this git security check
within the container, user may set IDF_GIT_SAFE_DIR='*'. This might be
heplfull in CI.
Closes https://github.com/espressif/esp-idf/pull/12636
[1] - https://github.com/git/git/commit/8959555cee7ec045958f9b6dd62e541affb7e7d9
[2] - https://nvd.nist.gov/vuln/detail/cve-2022-24765
[3] - https://git-scm.com/book/en/v2/Appendix-A%3A-Git-in-Other-Environments-Git-in-Bash
[4] - https://github.com/espressif/esp-idf/pull/12636
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This integrates esp_idf_size.ng, refactored esp-idf-size version, into
esp-idf and enables it by default. The esp_idf_size.ng may be enabled
by using the --ng option, but also via ESP_IDF_SIZE_NG env. variable,
which is used in this integration.
New -l/--legacy option is added, which enforces usage of the old version.
This option can be also set via "ESP_IDF_SIZE_LEGACY" env. variable.
This should allow to easily switch back to old version if there is any
problem.
The new version is used by default for all formats, except for the "json".
Examples:
$ idf.py size # uses refactored version
$ idf.py size --legacy # uses legacy version
$ idf.py size --l # uses legacy version
$ idf.py size --format json # uses legacy version
$ idf.py size --format json2 # uses refactored version
$ export ESP_IDF_SIZE_LEGACY="1" # use legacy version only from now on
ESP_IDF_SIZE_FORCE_TERMINAL, which forces terminal control codes(colors), is also set
when running from idf.py, so the colors are propagated even if stdout
for esp_idf_size.ng is not attached to tty.
The same changes are applied also to the idf_size.py wrapper.
There is an import check if esp_idf_size.ng is available. If not,
we switch into the legacy mode. This should cover situation when the
esp-idf has support for refactored version, but it's not installed.
This should also allow users to bind to a legacy version(<1.0.0) and the
idf.py size and idf_size.py should still work. This also allow us to
restring the version in constraints file if we need to switch back to
legacy version globally.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
pyyaml have problem with newer cython(3.0), this was fixed in pyyaml
6.0.1, which requires cython<3.0. The pyyaml dependency in astyle
was fixed in v1.0.5, but the pre-commit config is still useing v1.0.2.
As a result the pre-commit hooks installation of astyle can fail
on pyyaml. Fix this by bumping the astyle version for pre-commit.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently the component_requirements hint module does not work
as expected if the component list for a project is trimmed down.
With the new "all_component_info" dictionary info in project_description.json,
the module can produce hints even if cmake's COMPONENTS variable is
set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add new "all_component_info" dictionary into the
project_description.json file. It contains information about all
registered components presented in the __COMPONENT_TARGETS list.
Since components in this list are not fully evaluated, because only the first
stage of cmakefiles processing is done, it does not contain the same information
as the "build_component_info" dictionary. The "type", "file" and "sources" variables
are missing.
Most of the properties are already attached to the component target, so
this only adds INCLUDE_DIRS property to the target during the first cmakefiles
processing stage.
The "all_component_info" dict is generated in a separate function, even
though the original function for "build_component_info" could be
adjusted. This introduces a little bit of boilerplate, but keeps it
logically separated and probably easier if we want to extend it in the
future.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
There are orphaned .phyiram sections from components/esp_phy/lib/esp32/libphy.a when ESP_WIFI_SLP_IRAM_OPT
is not set on esp32. It can be seen in the wifi_station example examples/wifi/getting_started/station with
the default configuration. Even though they seem to be correctly placed in flash(linker got it right), I
believe this should be fixed. At least for the esp-idf-size, because these doesn't seem to be accounted.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
If there is a component(child) within a component(parent), like for test_apps, the parent
component may be wrongly identified as source component for the failed include. This may
lead to a false bug report if the parent component has component, which provides the missing
header, in requirements.
Fix this by looking for the longest matching source component directory.
Suggested-by: Ivan Grokhotkov <ivan@espressif.com>
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently idf.py reports just "Please use idf.py only in an ESP-IDF shell environment".
Sometimes it may be useful to know for which module the import failed.
Also the problem does not have to be related to shell environment only, but the
python venv can be corrupted. This adds a little bit more verbose error
message.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
1. The original test, before hint modules support was added, used
tempfile.NamedTemporaryFile in a way which is not supported on windows.
It was having the file open, which the hints tried to read it, leading
the EPERM exception. The docs[1] says this is not supported.
<quote>
Whether the name can be used to open the file a second time, while the
named temporary file is still open, varies across platforms (it can be
so used on Unix; it cannot on Windows)
</quote>
2. The hint module component_requirements test used the idf.py directly,
which is idf.py.exe on windows.Now it's starting idf.py through python.
We could probably used shell=True, but this approach is used in other
tests too.
Anyway the test are now passing on windows.
[1] https://docs.python.org/3/library/tempfile.html
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Following tests were added.
1. Test for missing header directory in component's INCLUDE_DIRS
2. Test for missing dependency in component's PRIV_REQUIRES
3. Test for missing dependency in component's REQUIRES
4. Test for dependency in PRIV_REQUIRES which should be in REQUIRES
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Introduces get_build_context() helper, which allows to get build context, e.g.
project description, at places where this info is not available. The
build context is set within ensure_build_directory.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The ticket below reports some very strange behaviour, where
even a simple command like the following
python3 -c "import subprocess; subprocess.run([''])"
actually spawns some process and pass instead of raising
PermissionError. Even though the problem is most probably somewhere
else, not in idf_tools, we may just return early if there is no
command available for get_version().
Closes https://github.com/espressif/esp-idf/issues/11880
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently ESP_PYTHON is used in the install.sh script, but it's
overwritten in detect_python.sh. This allows to explicitly specify
python binary, which should be used to create venv.
$ ESP_PYTHON=/usr/bin/python3.7 ./install.{sh,fish}
$ ESP_PYTHON=/usr/bin/python3.7 ./export.{sh,fish}
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Frantisek Hrbata
