The `setuptools` package starting with `v70.1.0`[1] contains built-in
`bdist_wheel` command. Before this version `setuptools` relied on the
`bdist_wheel` command implementation from the `wheel` package. Starting with
`setuptools` `v75.8.1` the `PEP 491`[3] restrictions on the distribution name
of a wheel package are enforced[4], replacting also `.` with `_`. Note that
`PEP 491` actually allows `.` in the distribution name, but for some reason the
latest packaging docs[10][11] does not, stating that `.` should be replaced
with `_`. This was discussion here[12].
Also the `wheel` package starting with `v0.45.0`[5] is using the `bdist_wheel`
command from `setuptools`. This means that any package which has `.` in its
distribution name, like `ruamel.yaml.clib`, can have different wheel name,
depending on which version of the `bdist_wheel` command was used.
The `bdist_wheel` command from setuptools prior `v75.8.1` or `wheel` prior
`v0.45.0` will keep the dots in distribution name preserved. For exaple the
`ruamel.yaml.clib` package will have distribution name
`ruamel.yaml.clib-0.2.12.dist-info. Newer versions will replace the dots with
`_` according to [10][11], creating distribution like
`ruamel_yaml_clib-0.2.12.dist-info`.
From packaging point of view `ruamel.yaml.clib-0.2.12.dist-info` and
`ruamel_yaml_clib-0.2.12.dist-info` are the same packages, but this is not
reflected in `importlib.metadata` prior python 3.10[9], which does not perform
name normalization prior the distribution search. This causes the `version`
from `importlib.metadata` to fail on python prior the 3.10 version if the
package with dots in distribution name was generated with normalized paths with
newer `setuptools`. Note that the distribution name normalization was
backported to some later 3.9 python version.
Let's demonstrate this behavior on a simple package with the
`my.minimal.package` name.
```
my_minimal_package/
├── pkg
│ └── __init__.py
└── setup.py
from setuptools import setup, find_packages
setup(
name='my.minimal.package',
version='0.1.0',
packages=find_packages(),
install_requires=[],
entry_points={},
)
```
With python 3.9.0 search for `my.minimal.package` fails because
of the missing name normalization.
```
docker run --rm -it --platform linux/x86_64 python:3.9.0 bash
python -m venv venv
. venv/bin/activate
pip install setuptools==v75.8.1
python setup.py bdist_wheel
pip install dist/my_minimal_package-0.1.0-py3-none-any.whl
python
Python 3.9.0 (default, Nov 18 2020, 13:28:38)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib.metadata import version as get_version
>>> get_version('my.minimal.package')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.9/importlib/metadata.py", line 551, in version
return distribution(distribution_name).version
File "/usr/local/lib/python3.9/importlib/metadata.py", line 524, in distribution
return Distribution.from_name(distribution_name)
File "/usr/local/lib/python3.9/importlib/metadata.py", line 187, in from_name
raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: my.minimal.package
>>> get_version('my_minimal_package')
'0.1.0'
```
With python 3.10.0 search for both `my.minimal.package` and
`my_minimal_package` succeeds.
```
docker run --rm -it --platform linux/x86_64 python:3.10.0 bash
python
Python 3.10.0 (default, Dec 3 2021, 00:21:30) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib.metadata import version as get_version
>>> get_version('my.minimal.package')
'0.1.0'
>>> get_version('my_minimal_package')
'0.1.0'
```
In our `tools/check_python_dependencies.py` we cannot relay on the default
distribution finder, used in the `version` function from `importlib.metadata`,
to do name normalization on older python versions. To cope with this,
implement a fallback version search. If `version` fails with
`PackageNotFoundError`, do the name normalization according to [10][11] and try
again.
Note: There is also a `wheel`[6][7] `v0.43.0` package embeded in `setuptools`
along with the new implementation[8]. This one seems to be used if the
external `wheel` package is not available but imported. TBH this is all kinda
messy and I may have overlooked something.
* [1] https://setuptools.pypa.io/en/stable/history.html#v70-1-0
* [2] https://setuptools.pypa.io/en/stable/history.html#v75-8-1
* [3] https://peps.python.org/pep-0491/#escaping-and-unicode
* [4] https://github.com/pypa/setuptools/pull/4766/files
* [5] https://wheel.readthedocs.io/en/stable/news.html
* [6] https://github.com/pypa/setuptools/blob/main/setuptools/_vendor/wheel/__init__.py
* [7] https://github.com/pypa/setuptools/issues/1386
* [8] https://github.com/pypa/setuptools/blob/main/setuptools/command/bdist_wheel.py
* [9] https://github.com/python/cpython/commit/c6ca368867bd68d44f333df840aa85d425a51410
* [10] https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization
* [11] https://packaging.python.org/en/latest/specifications/binary-distribution-format/
#escaping-and-unicode
* [12] https://github.com/pypa/setuptools/issues/3777
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Using same IV in AES-GCM across multiple invocation of
encryption/decryption operations can pose a security risk. It can help
to reveal co-relation between different plaintexts.
This commit introduces a change to use part of IV as a monotonic
counter, which must be incremented after every AES-GCM invocation
on both the client and the device side.
Concept of patch version for a security scheme has been introduced here
which can help to differentiate a protocol behavior for the provisioning
entity. The security patch version will be available in the JSON
response for `proto-ver` endpoint request with the field
`sec_patch_ver`.
Please refer to documentation for more details on the changes required
on the provisioning entity side (e.g., PhoneApps).
The active.py script is currently clearing the IDF_PYTHON_ENV_PATH,
preventing it from being set to a custom location for the python virtual
environment directory. Although the install script checks to ensure that
an existing python virtual environment is not overwritten with one for a
different ESP-IDF version than it was originally created for, we should
still permit setting a custom path for the python virtual environment.
Closes https://github.com/espressif/esp-idf/issues/15006
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
When a symbol needs to be placed to a different target than the one
designated for the object file, the object file is expanded, which
includes the following steps:
1. Creating a new placement for the symbol's input section with the
specified target.
2. Excluding the object placement from the orignal target.
3. Creating a new intermediate placement for the object for the original
target, where its input sections are expanded, excluding the input
section for the symbol.
Let's illustrate the object expansion process with the following example:
[sections:rodata]
entries:
.rodata+
.sdata2+
.srodata+
[scheme:default]
entries:
text -> flash_text
rodata -> flash_rodata
[scheme:noflash]
entries:
text -> iram0_text
rodata -> dram0_data
[mapping:soc_pm]
archive: libsoc.a
entries:
gpio_periph: GPIO_HOLD_MASK (noflash)
gpio_periph section headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000000 000034 000000 00 AX 0 0 2
[ 2] .data PROGBITS 00000000 000034 000000 00 WA 0 0 1
[ 3] .bss NOBITS 00000000 000034 000000 00 WA 0 0 1
[ 4] .rodata.GPIO_HOLD_MASK PROGBITS 00000000 000034 000058 00 A 0 0 4
[ 5] .rodata.GPIO_PIN_MUX_REG PROGBITS 00000000 00008c 000058 00 A 0 0 4
[ 6] .debug_info PROGBITS 00000000 0000e4 0000d8 00 0 0 1
[ 7] .rela.debug_info RELA 00000000 0009d4 000108 0c I 16 6 4
[ 8] .debug_abbrev PROGBITS 00000000 0001bc 000070 00 0 0 1
[ 9] .debug_aranges PROGBITS 00000000 00022c 000018 00 0 0 1
[10] .rela.debug_aranges RELA 00000000 000adc 00000c 0c I 16 9 4
[11] .debug_line PROGBITS 00000000 000244 0001ab 00 0 0 1
[12] .debug_str PROGBITS 00000000 0003ef 00022d 01 MS 0 0 1
[13] .comment PROGBITS 00000000 00061c 000030 01 MS 0 0 1
[14] .note.GNU-stack PROGBITS 00000000 00064c 000000 00 0 0 1
[15] .riscv.attributes RISCV_ATTRIBUTES 00000000 00064c 000044 00 0 0 1
[16] .symtab SYMTAB 00000000 000690 000260 10 17 36 4
[17] .strtab STRTAB 00000000 0008f0 0000e1 00 0 0 1
[18] .shstrtab STRTAB 00000000 000ae8 0000d1 00 0 0 1
1. Creating a new placement
.dram0.data :
{
*libsoc.a:gpio_periph.*(.rodata.GPIO_HOLD_MASK .sdata2.GPIO_HOLD_MASK .srodata.GPIO_HOLD_MASK)
}
2. Excluding the object placement
.flash.rodata :
{
*(EXCLUDE_FILE(*libsoc.a:gpio_periph.*) .rodata.* ...)
}
3. Creating a new intermediate placement
.flash.rodata :
{
*libsoc.a:gpio_periph.*(.rodata.GPIO_PIN_MUX_REG)
}
Now, let's do the same, but also move GPIO_PIN_MUX_REG to noflash with an updated mapping.
[mapping:soc_pm]
archive: libsoc.a
entries:
gpio_periph: GPIO_HOLD_MASK (noflash)
gpio_periph: GPIO_PIN_MUX_REG (noflash)
1. Creating a new placement
.dram0.data :
{
*libsoc.a:gpio_periph.*(.rodata.GPIO_HOLD_MASK .sdata2.GPIO_HOLD_MASK .srodata.GPIO_HOLD_MASK)
*libsoc.a:gpio_periph.*(.rodata.GPIO_PIN_MUX_REG .sdata2.GPIO_PIN_MUX_REG
.srodata.GPIO_PIN_MUX_REG)
}
2. Excluding the object placement
.flash.rodata :
{
*(EXCLUDE_FILE(*libsoc.a:gpio_periph.*) .rodata.* ...)
}
3. Creating a new intermediate placement
.flash.rodata :
{
*libsoc.a:gpio_periph.*
}
The *libsoc.a:gpio_periph.* entity in step 3 no longer has input
sections, as there are no remaining .rodata input sections in the object
file. The linker behavior for this mapping is to include all object
input sections that have not yet been placed as described in
https://sourceware.org/binutils/docs/ld.html#Input-Section-Basics
"If you use a file name without a list of sections, then all sections in
the input file will be included in the output section. This is not
commonly done, but it may by useful on occasion."
The map file for such mapping now contains following input sections
.flash.rodata 0x3c0a0120 0x19b34
*libsoc.a:gpio_periph.*()
.debug_info 0x3c0b95bf 0xd8 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.debug_abbrev 0x3c0b9697 0x70 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.debug_aranges
0x3c0b9707 0x18 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.debug_line 0x3c0b971f 0x1ab esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.debug_str 0x3c0b98ca 0x21a esp-idf/soc/libsoc.a(gpio_periph.c.obj)
0x22d (size before relaxing)
.comment 0x3c0b9ae4 0x30 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.note.GNU-stack
0x3c0b9ae4 0x0 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
.riscv.attributes
0x3c0b9ae4 0x44 esp-idf/soc/libsoc.a(gpio_periph.c.obj)
This is incorrect, and such intermediate placement should not be
generated. This type of placement can be recognized because it is not
explicitly defined in the mapping and lacks input sections. We can
identify this in the significant function and prevent issuing commands
for such placement.
Signed-off-by: Frantisek Hrbata <frantisek.hrbata@espressif.com>
Roland Dobai
Xiao Xufeng
Frantisek Hrbata
Marek Fiala
Radim Karniš
Mahavir Jain
Jan Beran
Jiang Jiang Jian
Martin Vychodil
laokaiyao
Martin Vychodil
radek.tandler
akshat
Erhan Kurubas
Euripedes Rocha
Tomáš Rohlínek
Ondrej Kosta
morris
Fu Hanxi