feat(tools): add -j/--jobs option to idf.py

Add a global -j/--jobs option to idf.py that sets the number of parallel
build jobs passed to the underlying build tool. Previously parallelism was
only configurable via the IDF_PY_BUILD_JOBS environment variable, and only
for the Ninja generator.

The option applies to both Ninja and Make and defaults to IDF_PY_BUILD_JOBS
when not given, so the environment variable keeps working as before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Nebojša Cvetković
2026-07-22 16:02:52 +01:00
co-authored by Claude Opus 4.8
parent 055ba9d3f9
commit db6b310447
6 changed files with 31 additions and 15 deletions
+7 -1
View File
@@ -77,7 +77,13 @@ In the above list, the ``cmake`` command configures the project and generates bu
It's not necessary to run ``cmake`` more than once. After the first build, you only need to run ``ninja`` each time. ``ninja`` will automatically re-invoke ``cmake`` if the project needs reconfiguration. It's not necessary to run ``cmake`` more than once. After the first build, you only need to run ``ninja`` each time. ``ninja`` will automatically re-invoke ``cmake`` if the project needs reconfiguration.
When using ``idf.py`` with the Ninja generator, you can cap the number of parallel build jobs by setting the ``IDF_PY_BUILD_JOBS`` environment variable. For example: You can control the number of parallel build jobs passed to the underlying build tool (Ninja or Make) with the ``-j``/``--jobs`` option of ``idf.py``. For example:
.. code-block:: bash
idf.py -j 6 build
The same value can be set with the ``IDF_PY_BUILD_JOBS`` environment variable, which is used as the default when ``-j``/``--jobs`` is not given:
.. code-block:: bash .. code-block:: bash
+7 -1
View File
@@ -77,7 +77,13 @@ idf.py
没有必要多次运行 ``cmake``。第一次构建后,往后每次只需运行 ``ninja`` 即可。如果项目需要重新配置,``ninja`` 会自动重新调用 ``cmake``。 没有必要多次运行 ``cmake``。第一次构建后,往后每次只需运行 ``ninja`` 即可。如果项目需要重新配置,``ninja`` 会自动重新调用 ``cmake``。
使用 Ninja 生成器配合 ``idf.py`` 时,可以通过设置环境变量 ``IDF_PY_BUILD_JOBS`` 来限制并行构建任务数。例如: 使用 ``idf.py`` 时,可以通过 ``-j``/``--jobs`` 选项来控制传递给底层构建工具(Ninja 或 Make)的并行构建任务数。例如:
.. code-block:: bash
idf.py -j 6 build
也可以通过设置环境变量 ``IDF_PY_BUILD_JOBS`` 来指定该值,当未提供 ``-j``/``--jobs`` 时,该环境变量将作为默认值使用:
.. code-block:: bash .. code-block:: bash
+3 -1
View File
@@ -29,7 +29,9 @@ GENERATORS: dict[str, str | dict | list] = collections.OrderedDict(
if os.name != 'nt': if os.name != 'nt':
MAKE_CMD = 'gmake' if platform.system() == 'FreeBSD' else 'make' MAKE_CMD = 'gmake' if platform.system() == 'FreeBSD' else 'make'
GENERATORS['Unix Makefiles'] = { GENERATORS['Unix Makefiles'] = {
'command': [MAKE_CMD, '-j', str(multiprocessing.cpu_count() + 2)], # Make, unlike Ninja, does not parallelize by default; run_target() applies this as -j.
'default_jobs': multiprocessing.cpu_count() + 2,
'command': [MAKE_CMD],
'version': [MAKE_CMD, '--version'], 'version': [MAKE_CMD, '--version'],
'dry_run': [MAKE_CMD, '-n'], 'dry_run': [MAKE_CMD, '-n'],
'verbose_flag': 'VERBOSE=1', 'verbose_flag': 'VERBOSE=1',
+7
View File
@@ -537,6 +537,13 @@ def action_extensions(base_actions: dict, project_path: str) -> Any:
'help': 'CMake generator.', 'help': 'CMake generator.',
'type': click.Choice(GENERATORS.keys()), 'type': click.Choice(GENERATORS.keys()),
}, },
{
'names': ['-j', '--jobs'],
'help': 'Number of parallel build jobs passed to the build tool (Ninja or Make).',
'envvar': 'IDF_PY_BUILD_JOBS',
'type': click.IntRange(min=1),
'default': None,
},
{ {
'names': ['--dry-run'], 'names': ['--dry-run'],
'help': "Only process arguments, but don't execute actions.", 'help': "Only process arguments, but don't execute actions.",
+1 -1
View File
@@ -1,4 +1,4 @@
# SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD # SPDX-FileCopyrightText: 2022-2026 Espressif Systems (Shanghai) CO LTD
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
global_options = [{ global_options = [{
'names': ['-D', '--define-cache-entry'], 'names': ['-D', '--define-cache-entry'],
+6 -11
View File
@@ -667,18 +667,13 @@ def run_target(
generator_cmd = list(GENERATORS[args.generator]['command']) generator_cmd = list(GENERATORS[args.generator]['command'])
if args.generator == 'Ninja': # Parallel jobs from -j/--jobs (or IDF_PY_BUILD_JOBS), falling back to the generator's default.
parallel_level = os.environ.get('IDF_PY_BUILD_JOBS') jobs = getattr(args, 'jobs', None)
if parallel_level: if jobs is None:
try: jobs = GENERATORS[args.generator].get('default_jobs')
jobs = int(parallel_level)
except ValueError as e:
raise FatalError('Environment variable IDF_PY_BUILD_JOBS must be a positive integer') from e
if jobs <= 0: if jobs is not None:
raise FatalError('Environment variable IDF_PY_BUILD_JOBS must be a positive integer') generator_cmd += ['-j', str(jobs)]
generator_cmd += ['-j', str(jobs)]
if args.verbose: if args.verbose:
generator_cmd += [GENERATORS[args.generator]['verbose_flag']] generator_cmd += [GENERATORS[args.generator]['verbose_flag']]