In some instances, the COMPONENTS_DISCOVERED build property is used to
walk through the component list and obtain component properties, such as
during configuration preparation or the inclusion of project_include
files. Since we know the component interfaces from the
COMPONENT_INTERFACES build property, we can switch to the faster
`__idf_component_get_property_unchecked` method to obtain component properties
in these cases. This change reduces the reconfiguration time by 0.4
seconds on my computer.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The commit f951ae5b18 ("feat(cmakev2): Added component validation
checks for ...") introduced support for validating component sources and
include directories. It heavily relies on iterating through the
registered components to obtain their properties. For instance, it
examines all components for each component source file to ensure that
the source files do not originate from other components. However, it
turns out that `idf_component_get_property` is a bottleneck in this
process, causing reconfiguration to take 3-4 times longer than before
the introduction of this check. Since we know of all the component
interfaces, we can bypass the checks performed by
`idf_component_get_property` and instead use the raw version,
`__idf_component_get_property_unchecked`, which operates much faster.
Following are hello_world example reconfigure times before and after.
before:
-- Configuring done (11.1s)
-- Generating done (0.2s)
after:
-- Configuring done (3.9s)
-- Generating done (0.2s)
A new library property LIBRARY_COMPONENT_INTERFACES_LINKED is added that
keeps component interfaces linked to the library.
Fixes: f951ae5b18 ("feat(cmakev2): Added component validation checks for ..")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Retrieve the value of the specified component property quickly. The
existing `idf_component_get_property` function performs various checks
to identify the component interface target, which keeps the component
properties, and accepts a component name, target, target alias, or
interface. While `idf_component_get_property` uses a cache to identify
the component interface relatively quickly, it is still much slower
compared to the raw `get_property`. The
`__idf_component_get_property_unchecked` function provides a faster way how to
obtain component property if the component interface is already known.
It skips all the checks, so it must be used carefully.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
If `.gitlab/ci/rules\.yml` is changed, make sure the
`check-tools-files-patterns` pre-commit hooks is triggered to verify
that all `tools/*` files are covered by rules.yml.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The MINIMAL_BUILD build property and the COMPONENTS variable are both
used to determine the initial component list for the build.
Currently, if the COMPONENTS variable is set, the MINIMAL_BUILD logic
is ignored during component selection, but the MINIMAL_BUILD build
property remains set. This leads to an inconsistent state where
menuconfig displays information indicating MINIMAL_BUILD is active,
even though it was ignored in favor of the COMPONENTS variable.
Fix this by setting the MINIMAL_BUILD property to OFF if the
COMPONENTS variable is used.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The 8bbcdb116d ("ci(gitlab): Improve tools host test dependencies")
removed `kconfig_new` from rules.yml leading to the following pre-commit
failure.
```bash
Check tools dir files
patterns...........................................................Failed
- hook id: check-tools-files-patterns
- exit code: 1
This test is used for making sure of all the tools dir files are
recorded in .gitlab/ci/rules.yml to trigger the related tests, except
those files should be excluded.
Missing Files:
tools/kconfig_new/prepare_kconfig_files.py
tools/kconfig_new/config.env.in
tools/kconfig_new/confgen.py
tools/kconfig_new/confserver.py
tools/kconfig_new/config_buildv2.env.in
Please add these files or glob patterns to ".gitlab/ci/rules.yml" and
put related files under ".patterns-<test_group>" block to trigger
related tests.
Or add them to "tools/ci/exclude_check_tools_files.txt" to exclude them.
```
Fixes: 8bbcdb116d ("ci(gitlab): Improve tools host test dependencies")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In cmakev2, a component can be evaluated within the context of another
component, so it's important to properly initialize each variable used
by the component.
Fixes: 0c4cf75c35 ("feat(esp32s31): Introduce esp32s31 hello world")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_build_generate_depgraph function creates a component dependency
graph in dot (graphviz) format for a specified executable. It uses
existing helper functions from cmakev1, ensuring that the generated dot
files are produced in the same manner as in cmakev1. While adjustments
might be needed in the future if necessary, the current implementation
is intended to offer the same functionality as cmakev1. Similar to
cmakev1, the dot files are only generated only when the
__BUILD_COMPONENT_DEPGRAPH_ENABLED build property is set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
As part of inter-procedural optimizations (IPA), the compiler may
perform tasks such as constant propagation for functions. This involves
generating a specialized version of a given function with a new symbol
name that includes a suffix. For example, during constant propagation,
the compiler might create a specialized version named
`spiflash_start_core.constprop.0` for the `spiflash_start_core`
function. Additionally, the compiler may generate multiple clones of a
single function. Currently, when ldgen performs symbol placement, it
does not account for these compiler-generated functions, leading to
their incorrect or unexpected placement in memory (markers).
Consider a linker fragment with:
```
[mapping:spi_flash]
archive: libspi_flash.a
entries:
esp_flash_api: spiflash_start_core (noflash)
```
The `spiflash_start_core` function should be placed in IRAM. However,
the compiler might generate an optimized version of this function with a
`.constprop.0` suffix, resulting in a
`.text.spiflash_start_core.constprop.0` input section. Currently, ldgen
does not handle this situation, leading to misplaced symbols.
Since `.` is not allowed in C identifiers, it should be safe to consider
all input sections for a symbol with any `.` suffix as representing that
symbol. This means considering the symbol suffixes should not cause any
ambiguity.
This change automatically places all input sections, including those
with possible suffixes for a given symbol, into the specified memory. In
other words, specifying a function name like `spiflash_start_core` in a
linker fragment automatically includes input section names matching
`spiflash_start_core(\..*)?$`.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The build system keeps track of each component source. Currently
there are four types of sources:
1. "project_components" - project components
2. "project_extra_components" - components from EXTRA_COMPONENT_DIRS
3. "project_managed_components" - custom project dependencies managed by the IDF Component Manager
4. "idf_components" - ESP-IDF built-in components, typically under /components
This can be used to identify the component libraries that are likely to
change during application development and pass them to ldgen as mutable
libraries. Add all components with "project_components" as their source
as mutable.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a helper function `__should_generate_sdkconfig()` that checks
whether the sdkconfig file should be generated. The
`__generate_sdkconfig()` function can be called multiple times, such as
when the initial sdkconfig is generated at the start of the build
process and later after additional components are fetched by the
component manager. There might be no components fetched by the
component manager, for example, in the hello_world example, or the
downloaded components may not contain any configuration files. In such
cases, there is no need to regenerate the sdkconfig. This helper
function stores the list of configuration files in the
`__PREV_KCONFIGS`, `__PREV_KCONFIG_PROJBUILDS`, and
`__PREV_SDKCONFIG_RENAMES` build properties at its end, and at the
beginning, it compares them with the current lists of configuration
files.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_create_size_report function allows for the creation of size
report targets based on the generated link map file. The size report
targets are created using the TARGET option name: "<target>",
"<target>-files", and "<target>-components". These size report targets
are added to the idf_default_project with the TARGET set to "size",
resulting in the creation of "size", "size-files", and "size-components"
targets for the default project.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The idf_project_init macro is evaluated within the global variable
scope, and currently, there are a few variables still set. Let's ensure
all variables are properly unset to avoid polluting the global variable
namespace.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The MINIMAL_BUILD property is not relevant in cmakev2, as only
components explicitly linked through targets are included in the build
by design. Display a warning when this outdated build property is set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The link flag for generating the map file is currently added globally to
the link_options. If multiple executables are generated, the link map
file is overwritten by the last created executable. Since cmakev2
supports building multiple executables, the link map file options need
to be set for each executable individually. To address this, add a new
MAPFILE_TARGET option to the idf_build_executable function. When set,
the link map file will be generated by the linker, and a target
specified in the MAPFILE_TARGET option will be created for the map file.
This also splits the idf_project_default macro. Only the
idf_project_init macro needs be called within the global scope, as it
includes the project_include.cmake files and the cmake version of the
configuration. The remaining functionality of the idf_project_default
macro is implemented in a __project_default helper function to avoid
polluting the global variable space.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the `idf_build_generate_metadata` function takes an
executable target as an argument, and the binary name is stored in the
`EXECUTABLE_BINARY` property. This approach is inconvenient because a
single executable might generate multiple binary images, making it
unreliable to store the binary image name in the executable property due
to the N:1 relationship. To address this, pass the binary image target
instead of the executable target to the `idf_build_generate_metadata`
function. This change is facilitated by linking the binary target,
executable target, and library interface targets with the following
properties:
binary:EXECUTABLE_TARGET -> executable:LIBRARY_INTERFACE -> library
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
With commit b719292b75 ("refactor(build-system): Simplify flash target creation"),
partition table and bootloader components are correctly adding their
dependency on the flash target, so there is no need for this to be done
in the build system. This removes the temporary workaround.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Changes generated by pre-commit related to ruff-format and an update to
the copyright date.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Introduce ESPTOOLPY_FAST_REFLASHING configuration option. It instructs
ldgen to group entity mappings for libraries deemed mutable (prone to
change) separately from those considered immutable (unlikely to change).
Organizing mutable and immutable libraries separately in the linker
script allows the linker to form larger contiguous blocks of data for
immutable libraries in the application's output sections. These blocks
are likely to stay mostly unchanged between application recompilations,
enabling them to be skipped during reflashing.
Separating mutable and immutable libraries in the linker script to
minimize changes in the output sections is insufficient. Padding is
added after the input sections of mutable libraries in the default data
and text output sections. This creates a buffer for the mutable
libraries, allowing additional changes to be made without altering the
layout of the binary image.
Additionally two optimizations currently in use can still mix data from
these libraries, leading to significant changes even within the grouped
immutable libraries.
1. constant merging
Linker will try to merge input sections that have the MERGE and
STRING flags from different libraries (object files) to perform
optimizations like tail merging. For example, adding a string
literal in a mutable library will also change the addresses of
string literals from immutable libraries in such a merged section,
causing changes in the generated code when those literals are
referenced.
Disabled with COMPILER_DISABLE_MERGE_CONSTANTS(-fno-merge-constants)
2. literal pools on Xtensa
As optimization, the linker may merge literal pools from different
libraries (object files) to improve the generated code size. This
has the same effect as constant merging, and changes in mutable
libraries may cause changes in the generated code for immutable
libraries. To get larger unchanged continuous blocks in the text
output sections for immutable libraries, we need to ensure that the
Xtensa literal pools remain close to their references and are not
merged.
Disabled with CONFIG_COMPILER_ENABLE_TEXT_SECTION_LITERALS(-mtext-section-literals)
When ESPTOOLPY_FAST_REFLASHING is enabled, these two optimizations are
disabled to achieve larger unchaged continuous blocks for the grouped
immutable libraries, even though disabling these optimizations results
in slightly larger code.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The build system keeps track of each component source. Currently
there are four types of sources:
1. "project_components" - project components
2. "project_extra_components" - components from EXTRA_COMPONENT_DIRS
3. "project_managed_components" - custom project dependencies managed by the IDF Component Manager
4. "idf_components" - ESP-IDF built-in components, typically under /components
This can be used to identify the component libraries that are likely to
change during application development and pass them to ldgen as mutable
libraries. Add all components with "project_components" as their source
as mutable.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add the `--debug` option to print the entity tree to stdout. This can be
useful for viewing the entire entity tree state to identify potential
issues or incorrect placements.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Introduce a new `--mutable-libraries-file` option that accepts a file
containing the filenames of mutable libraries, each listed on a separate
line. Mutable libraries are component libraries expected to change
during development. In contrast, immutable component libraries are not
expected to change. In the generated linker script, the input sections
of mutable libraries are grouped together rather than being mixed with
those of immutable libraries. The goal is to create large continuous
areas in the ELF file's output sections that remain unchanged for
immutable libraries during application recompilation, allowing these
areas to be skipped during flashing.
The build system identifies the mutable libraries and passes them to
ldgen using the `--mutable-libraries-file` option. It maintains
information about component sources, one of which is
`project_components`. This source type identifies components that are
directly related to the project being developed and are very likely to
change.
Mappings for mutable libraries are explicitly created for all sections
in the default scheme. This happens before the entity
`(archive:object_file:symbol/input_section)` node tree with placements is
generated and is equivalent to having these mappings in the mapping
linker fragment. All placements for mappings, whether newly added or
already existing as defined in linker fragments, associated with mutable
libraries are flagged as `mutable` in the entity node tree. This flag
ensures that these placements are included in the final linker script.
Currently, ldgen only emits placements that are either significant or
forced. A placement is considered significant if, for example, it is not
already covered by a placement in parent node. For instance, `*(.iram1
.iram1.*)` placement already includes `*libapp_trace.a:(.iram1
.iram1.*)`, so the latter is not emitted by default. The `mutable` flag
ensures that placements for mutable libraries are emitted in the linker
script and placed at dedicated location.
The locations where placements for mutable libraries are specified in
the linker script are identified by a new `mutable` marker, for example,
`mutable[flash_text]`. The placements for immutable libraries remain in
the existing `mapping` marker, for example, `mapping[flash_text]`. The
`mutable` marker for each target is placed after the `mapping` marker.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Entity mappings for immutable libraries are placed in the existing
`mapping[target]` marker, while those for mutable libraries are placed
in the new `mutable[target]` marker, which comes after the `mapping`
marker. Additionally, include padding after the input sections of
mutable libraries in the default data and text output sections,
providing a headroom for the mutable libraries to grow. Padding is
currently not added, for example, in the `.iram0.data` output section,
which is not expected to change frequently. Padding for other mutable
input sections may be added in the future.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The cmakev2 build system may evaluate components recursively and all
components has to make sure all their variables are properly
initialized.
Fixes: a7115b6b36 ("refactor(jpeg): Split hal layer for jpeg")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The `tools/test_build_system/buildv2_test_app` is a cmakev2 build system
testing application, same as `build_test_app`, which should be removed
from the readme check performed by check_build_test_rules.py.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The COMPONENTS and EXCLUDE_COMPONENTS variables are unused in cmakev2. A
deprecation warning will be printed if they are set. This change also
includes a helper functions for printing deprecation warnings.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The cmakev1 dropped generation of prefix_map_gdbinit, reflect this
change also in cmakev2.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently both menu configs, for component's project configuration and
component's configuration, for components not included in the build have
the same name: "Configuration for components not included in the build".
This might be confusing. Let's use "Project configuration for components
not included in the build" for component Kconfig.projbuild files.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The cmakev1 added esp_stdio to common requires, add it in cmakev2 too
for backward compatibility.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the toolchain CMake files use the remove_duplicated_flags
function from utilities.cmake. The cmakev2 implementation also includes
this function for backward compatibility. Move the
remove_duplicated_flags function to a separate file,
deduplicate_flags.cmake, so it can be shared between cmakev1 and
cmakev2.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Enable check-kconfig-files. It was temporarily disabled because it was
not possible to add a new sourced file through an environment variable,
as kconfcheck had a fixed list of allowed environment variable names.
This issue should now be resolved.
Currently, the buildv2 tests are initiated using the same patterns as
the tests for the current build system. This means that any change in
the current build system will also trigger the buildv2 tests. Initially,
it might be wise not to block the CI for changes in the current build
system in case there is an issue with buildv2. Therefore, let's
explicitly start the buildv2 tests only when the buildv2 label is set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The esp_libc relies on the stdio implementation of the vfs component if
CONFIG_VFS_SUPPORT_IO is enabled. This is not an issue in cmakev1
because if the vfs component is not included in the project build, its
configuration is not available, and CONFIG_VFS_SUPPORT_IO is not set.
However, in cmakev2, the configuration for all components is available,
and the presence of some component configuration options does not
necessarily mean that the component is included in the project build.
When esp_libc is compiled without the vfs dependency but with
CONFIG_VFS_SUPPORT_IO enabled, the libc initialization will encounter a
panic due to a NULL pointer dereference (fp->_flags) in __swsetup_r, as
fopen in esp_libc_init_global_stdio will return NULL.
Although the current documentation exposes only limited functionality,
which is unlikely to change, add an explicit note that the build system
v2 is currently a technical preview intended for testing and early
evaluation.
Ensure consistency in referring to v2 throughout the documentation. In
some places, it is referred to as V2 or Version 2; unify this to v2.
Additionally, update the hello_world example for v2 to use the
idf::spi_flash alias instead of the interface variable to maintain
consistency with the rest of the documentation.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Extend the current documentation to recommend using the
idf_component_register function as the preferred method for creating a
new component for v2.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Using idf_component_register is the preferred method for creating new
components for cmakev2. This approach ensures compatibility with both
versions of the build system. The KCONFIG and KCONFIG_PROJBUILD options
have been removed from the API documentation, but are retained in code
in case a cmakev1 component uses these options to warn about
incompatibility. Also remove a note about `project_include.cmake`,
because cmakev2 includes all project_include files.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Enhance the `esp_target_info` component example by adding some of
the missing features that the component might require. This includes
adding a linker script, a linker fragment, linking the component archive
as a whole library, and providing an example of how the linker options
can be set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Components designed exclusively for cmakev2 are not compatible with
cmakev1 due to the lack of forward compatibility. It is important to
clarify this at the very beginning of the `Creating a New Component`
section. Additionally recommended a guidance on how to create a
component that will be compatible with both versions.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the archive file name is set only in the cmakev1
idf_component_register shim. The predictable component archive file name
is important, for example, for usage in linker fragments. Ensure that
the cmakev2 component also has the archive file name set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, when the cmakev2 component sources are set, the SRCS target
property is used. This is obviously wrong because the correct CMake
property is SOURCES.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Rename the LINKER_SCRIPTS_STATIC component property to LINKER_SCRIPTS.
This property stores linker scripts that are not processed by ldgen,
which essentially includes all of them. The only linker script processed
by ldgen is sections.ld, which is handled by the esp_system component.
This implies that there is likely no practical use case for other
components to utilize ldgen processed linker scripts. This change is
purely cosmetic to allow components to add linker scripts with:
idf_component_set_property(${COMPONENT_TARGET} LINKER_SCRIPTS linker_script.ld APPEND)
instead of
idf_component_set_property(${COMPONENT_TARGET} LINKER_SCRIPTS_STATIC linker_script.ld APPEND)
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add very basic documentation. It is currently written with the
expectation that the reader is already familiar with the existing build
system and focuses more on the changes that might be necessary to adapt
existing components to work with cmakev2. Additionally, it provides
basic information on how to create a new component and project.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, we include numerous functions in the automatically generated
documentation for the build system API. Let's begin with only the
essential functions and gradually add more to the API based on requests
and actual needs.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The current approach involves wrapping the library target within the
library interface target with the whole-archive flags, such as for the
GCC linker. However, this does not work as expected because the library
target is expanded, and the whole-archive flags are also applied to
other library targets that the wrapped library target depends on. IOW
the whole-archive flags surround multiple archives, not just the one
requrested. Generally, using linker flags like whole-archive in the
library INTERFACE(INTERFACE_LINK_LIBRARIES) does not seem to work as
CMake may perform deduplication and rearrange the flags [1].
The proof of concept used `CMAKE_LINK_LIBRARY_USING_<FEATURE>` with the
WHOLE_ARCHIVE feature, which was introduced in CMake 3.24 to specify how
the library target should be linked. There are two issues with this.
First, this feature is only available from CMake 3.24 onwards, while our
minimum CMake version is set to 3.22. More importantly, all occurrences
of a library on the link line are wrapped. For example, if a library
like `vfs` appears multiple times on the link line due to dependencies,
each occurrence will be wrapped with whole-archive, causing the linker
to complain about multiple symbol definitions. Therefore, even though
WHOLE_ARCHIVE is recommended for handling whole-archive linkage, it does
not seem suitable for our purposes. This was overlooked in the PoC
because only a simple testing component was used to check the
WHOLE_ARCHIVE behavior.
One way to address this issue is to adopt the same approach used in
cmakev1, which involves specifying the whole-archive flags when linking
component library targets to the final executable. In this case, CMake
retains the flags without any alteration, unlike when the flags are
specified in INTERFACE_LINK_LIBRARIES for the component interface
targets. While this approach is feasible, it would alter the current
logic, where we have a single library interface for the entire idf
library that can be linked to the executable. This change would also
complicate the direct use of the idf library, as it would no longer be
possible to simply link it to the executable. Instead, the executable
would need to correctly link the component libraries with the
whole-archive flags. We could encapsulate this process within the
idf_build_executable function, but projects that only use
idf_build_library would need to implement the same solution.
It appears possible to address this issue by using target_link_options
instead. This allows to specify exactly what should appear on the link
command line. One side effect of this approach is that the library
appears on the link command line multiple times: first when used with
target_link_options and second when used in target_link_libraries.
However, this does not seem to pose a problem. The flags specified with
target_link_options appear on the command line first, ensuring that the
whole-archive is prioritized, and the component archive libraries may be
repeated on the link line anyway due to dependencies. Essentially, the
final link command line is very similar to the one from cmakev1. We may
revisit this approach in the future, but for now, it seems to work as
expected.
[1] https://gitlab.kitware.com/cmake/cmake/-/issues/20078
[2] https://cmake.org/cmake/help/latest/variable/CMAKE_LINK_LIBRARY_USING_FEATURE.html
[3] https://discourse.cmake.org/t/automatically-wrapping-a-static-library-
in-whole-archive-no-whole-archive-when-used-during-linking/5883
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, when the cmakev1 properties for INCLUDE_DIRS and
PRIV_INCLUDE_DIRS are constructed in the
__set_component_cmakev1_properties function, the return values from the
get_target_property function for INCLUDE_DIRECTORIES and
INTERFACE_INCLUDE_DIRECTORIES properties are not checked. If a component
target does not set e.g. INCLUDE_DIRECTORIES property,
get_target_property will return a value such as `include_dirs-NOTFOUND`.
This value is subsequently passed to __get_relative_paths in the PATHS
argument, causing the file(RELATIVE_PATH) call in __get_relative_paths
to fail with an error.
```
CMake Error at /home/fhrbata/work/esp-idf/tools/cmakev2/utilities.cmake:235 (file):
file RELATIVE_PATH must be passed a full path to the file:
include_dirs-NOTFOUND
```
Fix this by explicitly set the PATHS to an empty list before passing it
to the __get_relative_paths function if the property is not set.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The commit adb2d5deee ("feat(cmake): Produce warnings when component dependen..")
introduced additional checks for source files and include directories
used by a component that are located outside the component's directory.
If these files and directories belong to another component, a warning is
issued. This feature has not yet been implemented in cmakev2, so related
tests are temporary disabled.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
In cmakev1, certain arguments provided to the idf_component_register
function are stored as component properties. These properties are used
to generate the project_description.json file, which other tools rely
on. Since the idf_component_register function is obsolete in cmakev2, we
need to recreate component properties such as INCLUDE_DIRS,
PRIV_INCLUDE_DIRS, REQUIRES, and PRIV_REQUIRES, which were previously
provided by the idf_component_register function in cmakev1. To achieve
this, let's examine the component's real target cmake properties and
reconstruct them to maintain compatibility with cmakev1.
The real target cmake properties may include generator expressions,
which are ignored. This should be acceptable, as they were not addressed
in cmakev1 either, and handling them is likely not feasible.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Return the difference between two lists, meaning the elements that are
present in the first list but not in the second.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This helper function removes entries containing generator expressions
from the given list. The list is modified in place.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This is a helper function that returns a list of paths relative to a
given base directory for a list of input paths.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When a linker script file with a .in extension is added using the
target_linker_script function, it is processed with the C preprocessor.
The linker scripts are preprocessed only once, even if they are used in
multiple libraries, because they are the same.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
During the initialization of a component in the __init_component
function, add the COMPONENT_BUILD_DIR property. This can be used to
store component-specific generated files, such as preprocessed linker
scripts.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The `includes` list variable is used without initialization. Initialize
it explicitly.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The component has been renamed, update the name in the common
requirements for the cmakev1 components.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The component can be referenced by multiple identifiers such as the
component name, which is derived from the component directory name,
component target, interface, or aliases. All component properties are
attached to the component interface target, which is also used for
declaring component dependencies. The cmakev2 build system has a
function called __get_component_interface, which is responsible for
identifying the component interface based on the given component
identifier. Since this function is called frequently, it needs to be
reasonably fast.
This introduces a new __idf_component_interface_cache INTERFACE target,
which serves as a mapping cache between component identifiers and the
component interface. The cache for each component is initialized in the
__init_component function, which introduces a component to the build
system. Currently, the component interface search is conducted by
examining the COMPONENTS_DISCOVERED and COMPONENT_INTERFACES lists
stored as build properties. Since the build system is aware of most
component identifiers, such as component name, target, and alias, during
component initialization, it can add mappings between component
identifiers and the component interface to a cache, which is built as
the components are initialized. This cache is used in the
__get_component_interface function instead of looking into the
COMPONENTS_DISCOVERED or COMPONENT_INTERFACES lists. This significantly
speeds up the component interface search and also makes the code much
simpler and more readable.
The component interface cache also completely replaces the existing
component name resolution, which was introduced because of the component
manager, and the cache used for resolved component names. This is
possible because all the necessary information is available during
component initialization when the component interface cache is
populated. The ambiguity of components is resolved based on component
source/priority.
Here is an example of the component interface mapping for the
espressif__led_strip component to the idf_espressif__led_strip interface
target. The component name, without the namespace, is referred to as the
short name. In this example, it is led_strip.
- espressif__led_strip -> idf_espressif__led_strip # name(directory name)
- idf_espressif__led_strip -> idf_espressif__led_strip # interface
- idf::espressif__led_strip -> idf_espressif__led_strip # alias
- _idf_espressif__led_strip -> idf_espressif__led_strip # real target
- led_strip -> idf_espressif__led_strip # short name
When another component with the same espressif__led_strip name is
initialized with a different priority, there is no need to change
anything in the cache, because the mapping stays the same.
The cache must be updated when two components share the same short name
but belong to different namespaces. This situation is likely uncommon.
For instance, consider espressif__led_strip and my__led_strip. If
my__led_strip has a higher priority, the cache is updated to reflect the
short name as follows:
- led_strip -> idf_my__led_strip
If both components have the same priority, the short name mapping for
led_strip is entirely removed. Conversely, if my__led_strip has a lower
priority, no short name mapping is added for it.
The short name is also added to the COMPONENT_SHORT_NAME property of the
component.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This is a revised version of the existing __create_confserver_target
function. It creates a specified confserver target for a given
executable. The kconfig_menus.json file, used by the IDEs, is generated
when the confserver starts. This differs from the previous behavior,
where kconfig_menus.json was created globally along with other sdkconfig
formats. The reason for this change is that kconfig_menus.json contains
the Kconfig menu hierarchy and it is not just a flat option-value
format. It needs to accurately reflect which configurations for which
components are included or excluded. The kconfig_menus.json is generated
at `build/kconfig_menus.json`, where IDEs expect it. This means the file
is overwritten every time the server starts by kconfig_menus.json
version for given executable, so only one confserver can run at a time.
This is likely acceptable, as I don't believe it's possible to safely
run multiple instances of confserver due to the potential race
conditions when the sdkconfig file are generated.
In the future, we may include the location of kconfig_menus.json in
project_description.json so it can be easily identified by IDEs for
each executable.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, when the initial sdkconfig is generated, the
kconfig_menus.json format is also created alongside other formats like
CMake, JSON, or C header. The kconfig_menus.json depends on the Kconfig
hierarchy and cannot be generated globally. It must be generated for
each executable to ensure that included and excluded components are
correctly positioned within the Kconfig menu hierarchy. Remove the
automatic generation of kconfig_menus.json.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The current base kconfgen command is stored in the __BASE_KCONFGEN_CMD
build property, and it includes the --env-file option pointing to the
`build/config.env` file. The `build/config.env` file does not separate
component configuration files into included and excluded components,
because it is created before any executable is added and we need it for
the initial sdkconfig generation. The global `build/config.env` can be
used for every sdkconfig format (cmake, json, txt), but not for the
kconfig_menus.json format, because kconfig_menus.json relies on the
Kconfig layout and hierarchy. Let's allow specifying the --env-file for
each kconfgen invocation and remove it from the base kconfgen command.
This allows you to specify an --env-file for each executable, each with
its own config.env file.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The __run_kconfgen function does not take any arguments. The output
formats and related files are stored in the __KCONFGEN_OUTPUTS_CMD build
property.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
This is basically a renamed version of __create_save_defconfig_target.
The creation of the save-defconfig target is done in
`idf_project_default()`.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
For a specified executable, create a menuconfig target using the name
provided in the TARGET option. The function generates a config.env file
specific to the executable, where Kconfig files for components linked to
the executable and Kconfig files for components not linked to it are
separated. This separation allows for a clear visual distinction between
the configuration of components that are linked and those that are not
linked to the executable within the menuconfig.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Generate the config.env file for the specified executable. The
configuration file will be stored in the build directory, within a
directory named after the executable target name. The kconfigs*.in
files, which are generated by prepare_kconfig_files.py, will also be
stored in this directory.
This function primarily prepares the arguments for the
__create_config_env_file function based on the components linked to the
executable, ensuring that component Kconfig files are stored in the
appropriate kconfigs*.in files, depending on whether the component is
linked to the executable.
The directory where the generated files are stored is added to the
executable CONFIG_ENV_DIR property.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The __create_config_env_file function generates a configuration file for
the prepare_kconfig_files.py script. Currently, it specifies the paths
for the kconfigs*.in files, which are generated by
prepare_kconfig_files.py, to be located in the build directory. The
kconfigs*.in files, along with the configuration file, must also be
generated for each executable, as each executable may have a
different set of linked components. Set the paths for the
kconfigs*.in files to be generated by the prepare_kconfig_files.py
script in the same directory as the configuration file.
Also allow explicit specification of which component configuration files
should be placed in different kconfigs*.in files. For sdkconfig
generation, it is not relevant whether a component's Kconfig file is
placed in the kconfigs.in or kconfigs_excluded.in file. This distinction
is only important for visually separating the configuration of included
and excluded components in the menuconfig. For the main sdkconfig, all
component Kconfig files are placed in the kconfigs.in file, and the
generated kconfigs*.in files are stored in the build directory. For
executables, the generated kconfigs*.in files will be stored in separate
directories for each executable.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The excluded Kconfig files cannot be collected globally, they must be
collected per executable. The distinction between included and excluded
Kconfig files is relevant only for menuconfig, which is specific to each
executable, as each executable may have a different set of linked
components. For the global sdkconfig file and its CMake and JSON
variants, whether the Kconfig files are excluded or not is irrelevant.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Commit 46b41dfec187 ("test(cmakev2): Enable CI tests for component manager ..")
introduced a temporary workaround for the cmakev2 testing
application by including the cmakev1 project.cmake file after the
return() statement. The issue is that some tests, such as
test_build_fail_on_build_time, append commands to the project's
CMakeLists.txt. These commands are placed after the newly added return()
statement, causing them to be ignored and resulting in test failures. To
resolve this, include the cmakev1 project.cmake within the cmake block
comment.
Fixes: 46b41dfec187 ("test(cmakev2): Enable CI tests for component manager ..")
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Three tests are disabled for cmakev2:
* test_build_dfu
* test_build_uf2
* test_build_loadable_elf
and one is modified:
* test_build_skdconfig_phy_init_data
the other 14 tests should pass.
The tests, test_build_dfu and test_build_uf2, are temporarily disabled
because cmakev2 currently does not include the dfu and uf2 targets.
These tests should be re-enabled once support is added to cmakev2.
The test_build_loadable_elf is disabled because the bootloader_support
component determines its requirements based on the sdkconfig values,
specifically the CONFIG_APP_BUILD_TYPE_RAM used in this test. When
CONFIG_APP_BUILD_TYPE_RAM is set, bootloader_support declares a
dependency on micro-ecc. This is wrong even in cmakev1 and should be
fixed.
The test_build_skdconfig_phy_init_data test requires the esp_phy
component but does not specify it in its requirements. It relies on the
cmakev1 behavior, where all components are included in the build by
default. This approach does not work for cmakev2. To resolve this, make
the dependency on esp_phy explicit, ensuring compatibility with both
versions.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the cmakev2 implementation of Kconfig name consistency has a
slightly different warning message compared to cmakev2, causing the test
`test_build.py::test_build_with_misspelled_kconfig` to fail. Make the
warning messages consistent with cmakev1.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The test_build_with_misspelled_kconfig test expects a KConfig.projbuild
file in the main component. However, the file is correctly named
Kconfig.projbuild, which causes the test to fail for cmakev2.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, cmakev2 evaluates each component using add_subdirectory with
the EXCLUDE_FROM_ALL option. The intention was to exclude all components
from being built by default unless they are explicitly linked to an
executable based on dependencies. This approach aims to avoid building
components that are included but not actually linked to the executable.
However, this has the side effect of preventing components from adding
their custom targets to the "all" target, which is used, for example, by
esp_phy. Generally, we should not restrict components from adding
targets to "all". Since components are only included if explicitly
requested by default, removing the EXCLUDE_FROM_ALL option should be
acceptable.
The downside is that if IDF_INCLUDE_ALL_COMPONENTS is set and the user
runs `idf.py build`, all components will be built, even if they are not
linked to the executable.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the check is based on the existence of sdkconfig, but there
may be situations where sdkconfig exists without containing
CONFIG_IDF_TARGET. Perform a target consistency check only if the target
is identified in the main sdkconfig file.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The function arguments in the documentation comments were using field
list, which caused text overflow in the generated documentation and
generally resulted in poor formatting. Let's use paragraphs for
the argument descriptions instead.
The documentation comments are written in reStructuredText, but
currently, they use inconsistent indentation. Standardize all the
documentation comments to use a four-character indentation.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add stump files for the build system v2 documentation with automatic API
generation from CMake files.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Load the esp_docs_cmakev2_extension to process documentation comments
from CMake files.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Add a Sphinx extension that introduces a new `cmakev2` domain with
multiple directives, allowing for the automatic extraction of
documentation comments from CMake files and their inclusion in the
Sphinx-generated documentation.
Directives:
- `cmakev2:include`: The included CMake file is processed for
documentation comments within the `#[[api` and `#]]` marks, which
should contain valid reStructuredText markup.
- `cmakev2:function`: Creates a CMake function node. All function nodes
are sorted by name and placed into the `_cmakev2_functions` section.
- `cmakev2:macro`: Creates a CMake macro node. All macro nodes are
sorted by name and placed into the `_cmakev2_macros` section.
- `cmakev2:variable`: Describes a CMake variable node. All variable
nodes are sorted by name and placed into the `_cmakev2_variables`
section.
Each node can be referenced with `` :cmakev2:ref:`<node name>` ``, where
the node name is the function, macro, or variable name as used in the
related directive.
Example:
CMake file:
```
#[[api
.. cmakev2:function:: idf_flash_binary
#]]
```
This function can be referenced with `` :cmakev2:ref:`idf_flash_binary` ``
and will be placed in the `.. _cmakev2_functions:` section.
The extension is currently located in esp-idf, but in the future, we
should consider moving it to esp-docs.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Generate the flasher_args.json file for the global flash target for
tools that require it.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Generate binary images and add flash targets for hello_world_example and
fatfs_example.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
All other functions, such as idf_build_binary, use OUTPUT_FILE. This is
more descriptive and provides more flexibility to add additional
file-related options in the future if necessary. Make
idf_build_generate_metadata consistent with the other function
interfaces by using OUTPUT_FILE.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Use idf_build_binary, idf_sign_binary, and idf_flash_binary to generate
and flash the default project application.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Given the binary target generated by the idf_build_binary or
idf_sign_binary function, add a new POST_BUILD dependency that displays
a message indicating the binary image must be manually signed before
flashing. This situation arises when CONFIG_SECURE_SIGNED_APPS is
enabled, but CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES is not set. As a
result, the binary image is not signed during the build process and
requires manual signing.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Given the binary target created by the idf_build_binary or
idf_sign_binary function, add a new dependency that verifies whether the
binary image fits within the smallest application partition.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Create a new flash target for a previously created binary target using
the idf_build_binary or idf_sign_binary function. Specify the new target
name with the TARGET option. The binary image path is stored in the
BINARY_PATH property of the binary target and is set when the binary
target is created using the idf_build_binary or idf_sign_binary
function. Use the NAME option as a prefix for the generated esptool
argument file, and the FLASH option can be used to include the binary
image flashing in the global flash target.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Create a signed binary image from a previously generated binary using
the `idf_build_binary` function. This image can be signed with an
optional key from a KEYFILE or, by default, with the key specified in
the sdkconfig. A new custom command is added to generate the signed
binary, which is stored at the location specified by the OUTPUT_FILE
option and a new custom target is created for it, with the name provided
in the TARGET option. The BINARY_PATH property is added to this new
TARGET, containing the path to the signed binary image.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The `idf_build_binary` function generates a binary image for a specified
executable target, which may or may not have been created using the
`idf_build_executable` function. It adds a custom command to generate
the binary image file at the location specified by the `OUTPUT_FILE`
option and creates a new target for it, with the name specified by the
`TARGET` option. The path of the generated binary file is also stored in
the `BINARY_PATH` property of the newly created binary target.
Additionally, this binary path is stored in the `EXECUTABLE_BINARY`
property of the given executable. This information is used by the
`idf_build_generate_metadata` function, which takes the executable
target as an argument and includes the binary name in the
`project_description.json`.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
The global flash targets are created using
__create_project_flash_targets, but these are merely placeholders that
allow components to declare their dependencies on the global flash or
encrypted-flash target in their project_include.cmake. The actual logic
for flash targets is located in the project_include.cmake of the
esptool_py component. Therefore, the targets must be initialized
after this file has been included. To address this, a new function,
__init_project_flash_targets, has been added for this purpose.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Create a phony target that intentionally fails the build, displaying the
error messages specified in the message arguments. An empty CMake file is
created and included, which is then deleted when the target is executed.
This forces CMake to rerun, as the file is added to the CMake rerun
dependencies.
This has been carried over from cmakev1 as it is used by several
components.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Currently, the lwip is linked to mbedtls if CONFIG_LWIP_ENABLE is set.
This approach works in cmakev1 because only the configuration for
BUILD_COMPONENTS is available during component evaluation. However, this
is not the case in cmakev2, where the configuration for all components
is available. This means that even if CONFIG_LWIP_ENABLE is set, it does
not guarantee that the lwip component is included in the build. For
cmakev2, modify the check to use a generator expression to ensure that
lwip is linked to mbedtls only if lwip is actually included in the
build.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Frantisek Hrbata