Mirrors/esp-idf
1
0
Fork 0
mirror of https://github.com/espressif/esp-idf.git synced 2026-10-01 18:50:34 +03:00
Code Issues Packages Projects Releases Wiki Activity

ci: apply common-scripts CI refactor

Browse Source 
Squashes these original commits for release/v5.4 backport traceability:
- 9b289dc5ad6 ci: apply idf-ci 1.x
- 381df980d57 ci: remove pip-cache and other unused jobs
- 32375a0a15e ci: apply common-scripts CI refactor
This commit is contained in:
Fu Hanxi
2026-08-17 14:13:15 +02:00
parent 78acf95a61
commit b47f9850c7
156 changed files with 2376 additions and 7642 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitlab/ci/README.md
+8 -74
View File
@@ -23,12 +23,7 @@
- [Grammar](#grammar)
- [Special Rules](#special-rules)
- [Upload/Download Artifacts to Internal Minio Server](#uploaddownload-artifacts-to-internal-minio-server)
- [Users Without Access to Minio](#users-without-access-to-minio)
- [Users With Access to Minio](#users-with-access-to-minio)
- [Env Vars for Minio](#env-vars-for-minio)
- [Artifacts Types and File Patterns](#artifacts-types-and-file-patterns)
- [Upload](#upload)
- [Download](#download)
- [Revision Marker Naming Rules](#revision-marker-naming-rules)
## General Workflow
@@ -145,11 +140,11 @@ check if there's a suitable `.if-<if-anchor-you-need>` anchor
1. if there is, create a rule following [`rules` Template Naming Rules](#rules-template-naming-rules).For detail information, please refer to [GitLab Documentation `rules-if`](https://docs.gitlab.com/ee/ci/yaml/README.html#rulesif). Here's an example.
```yaml
.rules:patterns:python-files:
.rules:patterns:clang_tidy:
rules:
- <<: *if-protected
- <<: *if-dev-push
changes: *patterns-python-files
changes: *patterns-c-files
```
2. if there isn't
@@ -250,76 +245,15 @@ We're using the latest version of [idf-build-apps][idf-build-apps]. Please refer
In ESP-IDF CI, there's a few more special rules are additionally supported to disable the check app dependencies feature:
- Add MR labels `BUILD_AND_TEST_ALL_APPS`
- Pipeline variable `IDF_CI_SELECT_ALL_PYTEST_CASES=1`
- Run in protected branches
## Upload/Download Artifacts to Internal Minio Server
### Users Without Access to Minio
Please refer to the documentation [here](https://docs.espressif.com/projects/idf-ci/en/latest/guides/cli.html)
If you don't have access to the internal Minio server, you can still download the artifacts from the shared link in the job log.
## Revision Marker Naming Rules
The log will look like this:
When specifying chip revisions for testing markers, configurations or filenames, strictly follow the format `esp32<series>_rev<number>` (or similarly `<series>_rev<number>` when brief).
```shell
Pipeline ID : 587355
Job name : build_clang_test_apps_esp32
Job ID : 40272275
Created archive file: 40272275.zip, uploading as 587355/build_dir_without_map_and_elf_files/build_clang_test_apps_esp32/40272275.zip
Please download the archive file includes build_dir_without_map_and_elf_files from [INTERNAL_URL]
```
### Users With Access to Minio
#### Env Vars for Minio
Minio takes these env vars to connect to the server:
- `IDF_S3_SERVER`
- `IDF_S3_ACCESS_KEY`
- `IDF_S3_SECRET_KEY`
- `IDF_S3_BUCKET`
#### Artifacts Types and File Patterns
The artifacts types and corresponding file patterns are defined in tools/ci/artifacts_handler.py, inside `ArtifactType` and `TYPE_PATTERNS_DICT`.
#### Upload
```shell
python tools/ci/artifacts_handler.py upload
```
will upload the files that match the file patterns to minio object storage with name:
`<pipeline_id>/<artifact_type>/<job_name>/<job_id>.zip`
For example, job 39043328 will upload these four files:
- `575500/map_and_elf_files/build_pytest_examples_esp32/39043328.zip`
- `575500/build_dir_without_map_and_elf_files/build_pytest_examples_esp32/39043328.zip`
- `575500/logs/build_pytest_examples_esp32/39043328.zip`
- `575500/size_reports/build_pytest_examples_esp32/39043328.zip`
#### Download
You may run
```shell
python tools/ci/artifacts_handler.py download --pipeline_id <pipeline_id>
```
to download all files of the pipeline, or
```shell
python tools/ci/artifacts_handler.py download --pipeline_id <pipeline_id> --job_name <job_name_or_pattern>
```
to download all files with the specified job name or pattern, or
```shell
python tools/ci/artifacts_handler.py download --pipeline_id <pipeline_id> --job_name <job_name_or_pattern> --type <artifact_type> <artifact_type> ...
```
to download all files with the specified job name or pattern and artifact type(s).
You may check all detailed documentation with `python tools/ci/artifacts_handler.py download -h`
> e.g., `esp32_rev3`, `esp32c2_rev2`, `esp32c3_rev1`, `esp32p4_rev1`, `esp32c5_rev1`
.gitlab/ci/build.yml
+61 -309
View File
@@ -1,55 +1,13 @@
.build_template:
stage: build
extends:
- .after_script:build:ccache-show-stats:upload-failed-job-logs
- .after_script:build
image: $ESP_ENV_IMAGE
tags:
- build
# build only on shiny servers since shiny storage server is at the same location
- shiny
tags: [build, shiny]
variables:
# Enable ccache for all build jobs. See configure_ci_environment.sh for more ccache related settings.
IDF_CCACHE_ENABLE: "1"
dependencies: []
.build_cmake_clang_template:
extends:
- .build_template
- .before_script:build
- .after_script:build:ccache-show-stats
dependencies: # set dependencies to null to avoid missing artifacts issue
needs:
- job: fast_template_app
artifacts: false
- pipeline_variables
artifacts:
paths:
# The other artifacts patterns are defined under tools/ci/artifacts_handler.py
# Now we're uploading/downloading the binary files from our internal storage server
#
# keep the log file to help debug
- "**/build*/build_log.txt"
# keep the size info to help track the binary size
- size_info.txt
- "**/build*/size.json"
expire_in: 1 week
when: always
variables:
IDF_TOOLCHAIN: clang
TEST_BUILD_OPTS_EXTRA: ""
TEST_DIR: tools/test_apps/system/clang_build_test
PYTEST_IGNORE_COLLECT_IMPORT_ERROR: "1"
script:
# CI specific options start from "--parallel-count xxx". could ignore when running locally
- run_cmd python tools/ci/ci_build_apps.py $TEST_DIR -v
-t $IDF_TARGET
--copy-sdkconfig
--parallel-count ${CI_NODE_TOTAL:-1}
--parallel-index ${CI_NODE_INDEX:-1}
--modified-components ${MR_MODIFIED_COMPONENTS}
--modified-files ${MR_MODIFIED_FILES}
$TEST_BUILD_OPTS_EXTRA
- python tools/ci/artifacts_handler.py upload
######################
# build_template_app #
@@ -57,18 +15,11 @@
.build_template_app_template:
extends:
- .build_template
- .before_script:build
variables:
LOG_PATH: "${CI_PROJECT_DIR}/log_template_app"
BUILD_PATH: "${CI_PROJECT_DIR}/build_template_app"
BUILD_DIR: "${BUILD_PATH}/@t/@w"
BUILD_LOG_CMAKE: "${LOG_PATH}/cmake_@t_@w.txt"
BUILD_COMMAND_ARGS: ""
artifacts:
paths:
- log_template_app/*
- size_info.txt
- build_template_app/**/size.json
- build_template_app/**/size*.json
expire_in: 1 week
when: always
script:
@@ -82,224 +33,15 @@
# Only do the default cmake build for each target, remaining part are done in the build_template_app job
- tools/ci/build_template_app.sh ${BUILD_COMMAND_ARGS}
# build-related-pre-check-jobs ------------------------------------------------
# Build at least one project for each target at earliest stage to reduce build cost for obvious failing commits
fast_template_app:
extends:
- .build_template_app_template
- .rules:build:target_test
- .rules:build
stage: pre_check
tags: [fast_run, shiny]
variables:
BUILD_COMMAND_ARGS: "-p"
#------------------------------------------------------------------------------
#######################
# gnu_static_analyzer #
#######################
gcc_static_analyzer:
extends:
- .build_template_app_template
- .rules:build:target_test
stage: pre_check
tags: [build, shiny]
variables:
CI_CCACHE_DISABLE: 1
ANALYZING_APP: "examples/get-started/hello_world"
script:
- echo "CONFIG_COMPILER_STATIC_ANALYZER=y" >> ${ANALYZING_APP}/sdkconfig.defaults
- python -m idf_build_apps build -v -p ${ANALYZING_APP} -t all
########################################
# Clang Build Apps Without Tests Cases #
########################################
build_clang_test_apps_esp32:
extends:
- .build_cmake_clang_template
- .rules:build
variables:
IDF_TARGET: esp32
build_clang_test_apps_esp32s2:
extends:
- .build_cmake_clang_template
- .rules:build
variables:
IDF_TARGET: esp32s2
build_clang_test_apps_esp32s3:
extends:
- .build_cmake_clang_template
- .rules:build
variables:
IDF_TARGET: esp32s3
.build_clang_test_apps_riscv:
extends:
- .build_cmake_clang_template
variables:
# https://reviews.llvm.org/D90108.
# GNU 'as' lets .weak override .globl since binutils-gdb
# https://github.com/bminor/binutils-gdb/commit/5ca547dc2399a0a5d9f20626d4bf5547c3ccfddd (1996)
# while MC lets the last directive win (PR38921).
# For RISCV chips we use integrated assembler by default, so suppress this warning to pass CI pipeline.
TEST_BUILD_OPTS_EXTRA: "--ignore-warning-str 'changed binding to STB_WEAK'"
build_clang_test_apps_esp32c3:
extends:
- .build_clang_test_apps_riscv
- .rules:build
variables:
IDF_TARGET: esp32c3
build_clang_test_apps_esp32c2:
extends:
- .build_clang_test_apps_riscv
- .rules:build
variables:
IDF_TARGET: esp32c2
build_clang_test_apps_esp32c6:
extends:
- .build_clang_test_apps_riscv
- .rules:build
variables:
IDF_TARGET: esp32c6
build_clang_test_apps_esp32c5:
extends:
- .build_clang_test_apps_riscv
- .rules:build
variables:
IDF_TARGET: esp32c5
build_clang_test_apps_esp32h2:
extends:
- .build_clang_test_apps_riscv
- .rules:build
variables:
IDF_TARGET: esp32h2
build_clang_test_apps_esp32p4:
extends:
- .build_clang_test_apps_riscv
- .rules:build
variables:
IDF_TARGET: esp32p4
######################
# Build System Tests #
######################
.test_build_system_template:
stage: host_test
extends:
- .build_template
- .rules:build:check
dependencies: # set dependencies to null to avoid missing artifacts issue
needs:
- job: fast_template_app
artifacts: false
optional: true
artifacts:
reports:
junit: XUNIT_RESULT.xml
paths:
- XUNIT_RESULT.xml
- test_build_system
expire_in: 1 week
when: always
script:
- ${IDF_PATH}/tools/ci/test_configure_ci_environment.sh
- cd ${IDF_PATH}/tools/test_build_system
- python ${IDF_PATH}/tools/ci/get_known_failure_cases_file.py
- pytest
--cleanup-idf-copy
--parallel-count ${CI_NODE_TOTAL:-1}
--parallel-index ${CI_NODE_INDEX:-1}
--work-dir ${CI_PROJECT_DIR}/test_build_system
--junitxml ${CI_PROJECT_DIR}/XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
pytest_build_system:
extends: .test_build_system_template
parallel: 3
pytest_build_system_macos:
extends:
- .test_build_system_template
- .before_script:build:macos
- .after_script:build:macos:upload-failed-job-logs:ccache-show-stats
- .rules:build:macos
tags:
- macos_shell
parallel: 3
variables:
PYENV_VERSION: "3.8"
# CCACHE_DIR: "/cache/idf_ccache". On macOS, you cannot write to this folder due to insufficient permissions.
CCACHE_DIR: "" # ccache will use "$HOME/Library/Caches/ccache".
.test_build_system_minimal_cmake_template:
extends: .test_build_system_template
variables:
INSTALL_EXTRA_TOOLS: cmake@3.16.3
script:
- MINIMAL_SUPPORTED_CMAKE_VERSION=$(echo "${INSTALL_EXTRA_TOOLS}" | sed -n 's/.*cmake@\([0-9.]*\).*/\1/p')
- export PATH=$(echo "$PATH" | sed -E "s|/tools/cmake/[0-9.]+|/tools/cmake/${MINIMAL_SUPPORTED_CMAKE_VERSION}|")
- ACTUAL_CMAKE_VERSION=$(cmake --version | head -n1 | awk '{print $3}')
- |
if [ "${ACTUAL_CMAKE_VERSION}" != "${MINIMAL_SUPPORTED_CMAKE_VERSION}" ]; then
echo "ERROR: Wrong minimal CMake version! Detected: ${ACTUAL_CMAKE_VERSION}, but should be: ${MINIMAL_SUPPORTED_CMAKE_VERSION}"
exit 1
fi
- ${IDF_PATH}/tools/ci/test_configure_ci_environment.sh
- cd ${IDF_PATH}/tools/test_build_system
- python ${IDF_PATH}/tools/ci/get_known_failure_cases_file.py
- pytest
-k cmake
--cleanup-idf-copy
--work-dir ${CI_PROJECT_DIR}/test_build_system
--junitxml ${CI_PROJECT_DIR}/XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
pytest_build_system_minimal_cmake:
extends: .test_build_system_minimal_cmake_template
pytest_build_system_macos_minimal_cmake:
extends:
- .test_build_system_minimal_cmake_template
- .before_script:build:macos
- .after_script:build:macos:upload-failed-job-logs:ccache-show-stats
- .rules:build:macos
tags:
- macos_shell
variables:
PYENV_VERSION: "3.8"
# CCACHE_DIR: "/cache/idf_ccache". On macOS, you cannot write to this folder due to insufficient permissions.
CCACHE_DIR: "" # ccache will use "$HOME/Library/Caches/ccache".
INSTALL_EXTRA_TOOLS: ninja cmake@3.16.3
build_docker:
extends:
- .before_script:minimal
- .rules:build:docker
stage: host_test
needs:
- job: manual_gate
optional: true
image: espressif/docker-builder:1
tags: [shiny, dind]
variables:
DOCKER_TMP_IMAGE_NAME: "idf_tmp_image"
script:
# use different image tag for different concurrent job
- export DOCKER_TMP_IMAGE="${DOCKER_TMP_IMAGE_NAME}:${CI_CONCURRENT_ID:-latest}"
- export DOCKER_BUILD_ARGS="--build-arg IDF_CLONE_URL=${CI_REPOSITORY_URL} --build-arg IDF_CLONE_BRANCH_OR_TAG=${CI_COMMIT_REF_NAME} --build-arg IDF_CHECKOUT_REF=${CI_COMMIT_TAG:-$CI_COMMIT_SHA} --build-arg IDF_CLONE_SHALLOW=1 ${EXTRA_DOCKER_BUILD_ARGS:-} "
- time docker build --tag ${DOCKER_TMP_IMAGE} ${DOCKER_BUILD_ARGS} tools/docker/
# We can't mount $PWD/examples/get-started/blink into the container, see https://gitlab.com/gitlab-org/gitlab-ce/issues/41227.
# The workaround mentioned there works, but leaves around directories which need to be cleaned up manually.
# Therefore, build a copy of the example located inside the container.
- docker run --rm --workdir /opt/esp/idf/examples/get-started/blink ${DOCKER_TMP_IMAGE} idf.py build
# This job builds template app with permutations of targets and optimization levels
build_template_app:
@@ -307,57 +49,67 @@ build_template_app:
- .build_template_app_template
- .rules:build
stage: host_test
dependencies: # set dependencies to null to avoid missing artifacts issue
needs:
- job: fast_template_app
artifacts: false
########################################
# Clang Build Apps Without Tests Cases #
########################################
.build_cmake_clang_template:
extends:
- .build_template
- .rules:build
needs:
- job: fast_template_app
artifacts: false
- pipeline_variables
artifacts:
paths:
# The other artifacts patterns are defined under .idf_ci.toml
# Now we're uploading/downloading the binary files from our internal storage server
#
# keep the log file to help debug
- "**/build*/build_log.txt"
- "**/build*/size*.json"
expire_in: 1 week
when: always
variables:
IDF_TOOLCHAIN: clang
TEST_BUILD_OPTS_EXTRA: ""
script:
# CI specific options start from "--parallel-count xxx". could ignore when running locally
- run_cmd idf-build-apps build
-p tools/test_apps/system/clang_build_test
-t $IDF_TARGET
--parallel-count ${CI_NODE_TOTAL:-1}
--parallel-index ${CI_NODE_INDEX:-1}
--modified-components ${MR_MODIFIED_COMPONENTS}
--modified-files ${MR_MODIFIED_FILES}
$TEST_BUILD_OPTS_EXTRA
build_clang_test_apps_xtensa:
extends: .build_cmake_clang_template
parallel:
matrix:
- IDF_TARGET: [esp32, esp32s2, esp32s3]
build_clang_test_apps_riscv:
extends: .build_cmake_clang_template
variables:
# https://reviews.llvm.org/D90108.
# GNU 'as' lets .weak override .globl since binutils-gdb
# https://github.com/bminor/binutils-gdb/commit/5ca547dc2399a0a5d9f20626d4bf5547c3ccfddd (1996)
# while MC lets the last directive win (PR38921).
# For RISCV chips we use integrated assembler by default, so suppress this warning to pass CI pipeline.
TEST_BUILD_OPTS_EXTRA: "--ignore-warning-str 'changed binding to STB_WEAK'"
parallel:
matrix:
- IDF_TARGET: [esp32c3, esp32c2, esp32c6, esp32c5, esp32h2, esp32p4]
####################
# Dynamic Pipeline #
####################
generate_build_child_pipeline:
extends:
- .build_template
tags: [fast_run, shiny]
dependencies: # set dependencies to null to avoid missing artifacts issue
needs:
- pipeline_variables
- job: baseline_manifest_sha
optional: true
artifacts:
paths:
- build_child_pipeline.yml
- test_related_apps.txt
- non_test_related_apps.txt
expire_in: 1 week
when: always
variables:
PYTEST_IGNORE_COLLECT_IMPORT_ERROR: "1"
script:
# requires basic pytest dependencies
- run_cmd bash install.sh --enable-pytest
- run_cmd python tools/ci/dynamic_pipelines/scripts/generate_build_child_pipeline.py
build_child_pipeline:
stage: build
needs:
- job: fast_template_app
optional: true
artifacts: false
- pipeline_variables
- generate_build_child_pipeline
variables:
IS_MR_PIPELINE: $IS_MR_PIPELINE
MR_MODIFIED_COMPONENTS: $MR_MODIFIED_COMPONENTS
MR_MODIFIED_FILES: $MR_MODIFIED_FILES
PARENT_PIPELINE_ID: $CI_PIPELINE_ID
BUILD_AND_TEST_ALL_APPS: $BUILD_AND_TEST_ALL_APPS
REPORT_EXIT_CODE: $REPORT_EXIT_CODE
# https://gitlab.com/gitlab-org/gitlab/-/issues/214340
inherit:
variables: false
trigger:
include:
- artifact: build_child_pipeline.yml
job: generate_build_child_pipeline
strategy: depend
# !!!
# configurations moved to ci/actions/common/idf/
# !!!
.gitlab/ci/common.yml
+29 -318
View File
@@ -11,9 +11,18 @@ stages:
- target_test
- host_test
- test_deploy
- pre_deploy
- deploy
- post_deploy
- retry_failed_jobs
include:
- project: "ci/actions/common"
ref: "v2"
file: "templates/idf/common-scripts.yml"
inputs:
ci_python_constraint_file: "espidf.constraints.v5.4.txt"
macos_image: "registry.hub.docker.com/espressif/macos-idf:sonoma-v5.4"
windows_image: "windows_11_idf_v5_4"
variables:
# System environment
@@ -35,16 +44,12 @@ variables:
# - set GIT_STRATEGY: "clone" to shiny runners
# - set GIT_STRATEGY: "fetch" to brew runners
GIT_STRATEGY: clone
# we will download archive for each submodule instead of clone.
# we don't do "recursive" when fetch submodule as they're not used in CI now.
GIT_SUBMODULE_STRATEGY: none
# since we're using merged-result pipelines, the last commit should work for most cases
GIT_DEPTH: 1
# --no-recurse-submodules: we use cache for submodules
GIT_SUBMODULE_STRATEGY: none # here we use cache for submodules, so we don't need to fetch them every time
# since we're using merged-result pipelines, the last commit should work for most cases
# --prune --prune-tags: in case remote branch or tag is force pushed
GIT_FETCH_EXTRA_FLAGS: "--no-recurse-submodules --prune --prune-tags"
# we're using .cache folder for caches
GIT_CLEAN_FLAGS: -ffdx -e .cache/
LATEST_GIT_TAG: v5.4.4
SUBMODULE_FETCH_TOOL: "tools/ci/ci_fetch_submodule.py"
@@ -63,40 +68,36 @@ variables:
ESP_ENV_IMAGE: "${CI_REGISTRY}/ci/images/idf-v5.4:1"
ESP_IDF_DOC_ENV_IMAGE: "${CI_REGISTRY}/ci/images/idf-v5.4-docs:1"
TARGET_TEST_ENV_IMAGE: "${CI_REGISTRY}/ci/images/idf-v5.4-target-test:1"
SONARQUBE_SCANNER_IMAGE: "${CI_DOCKER_REGISTRY}/sonarqube-scanner:5"
PRE_COMMIT_IMAGE: "${CI_DOCKER_REGISTRY}/esp-idf-pre-commit:1"
# cache python dependencies
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
# Set this variable to the branch of idf-constraints repo in order to test a custom Python constraint file. The
# branch name must be without the remote part ("origin/"). Keep the variable empty in order to use the constraint
# file from https://dl.espressif.com/dl/esp-idf.
CI_PYTHON_CONSTRAINT_BRANCH: ""
# CI_PYTHON_CONSTRAINT_BRANCH could be defined as component input `ci_python_constraint_branch` in common-scripts above
# Update the filename for a specific ESP-IDF release. It is used only with CI_PYTHON_CONSTRAINT_BRANCH.
CI_PYTHON_CONSTRAINT_FILE: "espidf.constraints.v5.4.txt"
# CI_PYTHON_CONSTRAINT_FILE is defined as component input `ci_python_constraint_file` in common-scripts above
# Set this variable to repository name of a Python tool you wish to install and test in the context of ESP-IDF CI.
# Keep the variable empty when not used.
CI_PYTHON_TOOL_REPO: ""
# CI_PYTHON_TOOL_REPO could be defined as component input `ci_python_tool_repo` in common-scripts above
# Set this variable to the branch of a Python tool repo specified in CI_PYTHON_TOOL_REPO. The
# branch name must be without the remote part ("origin/"). Keep the variable empty when not used.
# This is used only if CI_PYTHON_TOOL_REPO is not empty.
CI_PYTHON_TOOL_BRANCH: ""
# CI_PYTHON_TOOL_BRANCH could be defined as component input `ci_python_tool_branch` in common-scripts above
# Set this variable to Clang toolchain distro URL to be used.
# NOTE: We have separate toolchains for Xtensa and RISCV, therefore jobs for one arch will fail.
# This is OK as far as we use CI_CLANG_DISTRO_URL for pre-release tests purposes only.
# Keep the variable empty when not used.
CI_CLANG_DISTRO_URL: ""
# CI_CLANG_DISTRO_URL could be defined as component input `ci_clang_distro_url` in common-scripts above
# Set this variable to specify the file name for the known failure cases.
KNOWN_FAILURE_CASES_FILE_NAME: "5.4.txt"
IDF_CI_BUILD: 1
PYTHONWARNINGS: "ignore"
# ccache settings
# some settings need to set in .gitlab-ci.yml as it takes effect while start-up the job
# https://ccache.dev/manual/latest.html#_configuring_ccache
@@ -106,295 +107,23 @@ variables:
CCACHE_MAXSIZE: "50G"
FF_USE_NEW_BASH_EVAL_STRATEGY: "true"
FORCE_COLOR: "1" # rich print with color
################################################
# `before_script` and `after_script` Templates #
################################################
.common_before_scripts: &common-before_scripts |
source tools/ci/utils.sh
is_based_on_commits $REQUIRED_ANCESTOR_COMMITS
if [[ -n "$IDF_DONT_USE_MIRRORS" ]]; then
export IDF_MIRROR_PREFIX_MAP=
fi
if echo "$CI_MERGE_REQUEST_LABELS" | egrep "(^|,)include_nightly_run(,|$)"; then
export INCLUDE_NIGHTLY_RUN="1"
export NIGHTLY_RUN="1"
fi
# configure cmake related flags
source tools/ci/configure_ci_environment.sh
if [[ "$CI_CCACHE_STATS" == 1 ]] && command -v ccache >/dev/null 2>&1 && [[ -n "$CCACHE_STATSLOG" ]]; then
mkdir -p "$(dirname "$CCACHE_STATSLOG")"
rm -f "$CCACHE_STATSLOG"
fi
# add extra python packages
export PYTHONPATH="$IDF_PATH/tools:$IDF_PATH/tools/esp_app_trace:$IDF_PATH/components/partition_table:$IDF_PATH/tools/ci/python_packages:$PYTHONPATH"
.setup_tools_and_idf_python_venv: &setup_tools_and_idf_python_venv |
# must use after setup_tools_except_target_test
# otherwise the export.sh won't work properly
# download constraint file for dev
if [[ -n "$CI_PYTHON_CONSTRAINT_BRANCH" ]]; then
wget -O /tmp/constraint.txt --header="Authorization:Bearer ${ESPCI_TOKEN}" ${GITLAB_HTTP_SERVER}/api/v4/projects/2581/repository/files/${CI_PYTHON_CONSTRAINT_FILE}/raw?ref=${CI_PYTHON_CONSTRAINT_BRANCH}
mkdir -p "${IDF_TOOLS_PATH:-$HOME/.espressif}"
mv /tmp/constraint.txt "${IDF_TOOLS_PATH:-$HOME/.espressif}/${CI_PYTHON_CONSTRAINT_FILE}"
fi
# Mirror
if [[ -n "$IDF_DONT_USE_MIRRORS" ]]; then
export IDF_MIRROR_PREFIX_MAP=
fi
if [[ "${CI_JOB_STAGE}" != "target_test" ]]; then
section_start "running_install_sh" "Running install.sh"
if [[ "${CI_JOB_STAGE}" == "build_doc" || "${CI_JOB_GROUP_NAME}" == *docs* ]]; then
run_cmd bash install.sh --enable-ci --enable-docs
elif [[ "${CI_JOB_STAGE}" == "build" ]]; then
run_cmd bash install.sh --enable-ci
else
if ! echo "${CI_JOB_NAME}" | egrep ".*pytest.*"; then
run_cmd bash install.sh --enable-ci
else
run_cmd bash install.sh --enable-ci --enable-pytest --enable-test-specific
fi
fi
section_end "running_install_sh"
else
section_start "install_python_env" "Install Python environment"
run_cmd python tools/idf_tools.py install-python-env --features ci,pytest,test-specific
section_end "install_python_env"
fi
if [[ ! -z "$INSTALL_EXTRA_TOOLS" ]]; then
section_start "installing_optional_tools" "Install optional tools ${INSTALL_EXTRA_TOOLS}"
$IDF_PATH/tools/idf_tools.py --non-interactive install $INSTALL_EXTRA_TOOLS
section_end "installing_optional_tools"
fi
# Install esp-clang if necessary (esp-clang is separately installed)
if [[ "$IDF_TOOLCHAIN" == "clang" && -z "$CI_CLANG_DISTRO_URL" ]]; then
$IDF_PATH/tools/idf_tools.py --non-interactive install esp-clang
fi
if [[ "${CI_JOB_STAGE}" == "target_test" ]]; then
section_start "IDF_SKIP_TOOLS_CHECK" "Skip required tools check"
export IDF_SKIP_TOOLS_CHECK=1
section_end "IDF_SKIP_TOOLS_CHECK"
fi
section_start "source_export" "Source export.sh"
source ./export.sh
section_end "source_export"
# we need the latest DEV release for esptool to be installed in every job without bumping the minimal requirement in
# the constraint file.
section_start "esptool_upgrade" "Upgrade esptool to the latest development version"
pip install --upgrade --pre esptool -c "${IDF_TOOLS_PATH:-$HOME/.espressif}/${CI_PYTHON_CONSTRAINT_FILE}"
section_end "esptool_upgrade"
# Custom clang toolchain
if [[ "$IDF_TOOLCHAIN" == "clang" && ! -z "$CI_CLANG_DISTRO_URL" ]]; then
echo "Using custom clang from ${CI_CLANG_DISTRO_URL}"
wget $CI_CLANG_DISTRO_URL
ARCH_NAME=$(basename $CI_CLANG_DISTRO_URL)
tar -x -f $ARCH_NAME
export PATH=$PWD/esp-clang/bin:$PATH
fi
# Custom OpenOCD
if [[ "$CI_JOB_STAGE" == "target_test" ]]; then
machine="$(uname -m)"
if [[ "$machine" == "armv7l" ]] ; then
OOCD_DISTRO_URL="$OOCD_DISTRO_URL_ARMHF"
elif [[ "$machine" == "aarch64" ]] ; then
OOCD_DISTRO_URL="$OOCD_DISTRO_URL_ARM64"
fi
if [[ ! -z "$OOCD_DISTRO_URL" ]]; then
echo "Using custom OpenOCD from ${OOCD_DISTRO_URL}"
wget $OOCD_DISTRO_URL
ARCH_NAME=$(basename $OOCD_DISTRO_URL)
tar -x -f $ARCH_NAME
export OPENOCD_SCRIPTS=$PWD/openocd-esp32/share/openocd/scripts
export PATH=$PWD/openocd-esp32/bin:$PATH
fi
fi
if [[ -n "$CI_PYTHON_TOOL_REPO" ]]; then
git clone --quiet --depth=1 -b ${CI_PYTHON_TOOL_BRANCH} https://gitlab-ci-token:${ESPCI_TOKEN}@${GITLAB_HTTPS_HOST}/espressif/${CI_PYTHON_TOOL_REPO}.git
pip install ./${CI_PYTHON_TOOL_REPO}
rm -rf ${CI_PYTHON_TOOL_REPO}
fi
info "setup tools and python venv done"
.show_ccache_statistics: &show_ccache_statistics |
# Show ccache statistics if enabled globally
section_start "ccache_show_stats" "Show ccache statistics"
if [[ "$CI_CCACHE_STATS" == 1 ]] && command -v ccache >/dev/null 2>&1; then
if ccache --help 2>/dev/null | grep -q -- '--show-log-stats'; then
if [[ -n "$CCACHE_STATSLOG" && -f "$CCACHE_STATSLOG" ]]; then
ccache --show-log-stats -vv
else
echo "INFO: No per-job ccache statistics were recorded"
fi
else
ccache --show-stats -vv
fi
fi || true
section_end "ccache_show_stats"
.upload_failed_job_log_artifacts: &upload_failed_job_log_artifacts |
if [ $CI_JOB_STATUS = "failed" ]; then
python tools/ci/artifacts_handler.py upload --type logs
fi
.before_script:minimal:
before_script:
- *common-before_scripts
- !reference [.common_before_scripts, minimal]
.before_script:build:macos:
before_script:
# macos is running shell executor, which means it would use
# the system installed /usr/local/bin/python3 by default.
# Ensure pyenv and PYENV_VERSION installed
- eval "$(pyenv init -)"
- *common-before_scripts
# On macOS, these tools need to be installed
- export IDF_TOOLS_PATH="${HOME}/.espressif_runner_${CI_RUNNER_ID}_${CI_CONCURRENT_ID}"
# remove idf-env.json, since it may contains enabled "features"
- rm -f $IDF_TOOLS_PATH/idf-env.json
# This adds tools (compilers) and the version-specific Python environment to PATH
- *setup_tools_and_idf_python_venv
- fetch_submodules
variables:
INSTALL_EXTRA_TOOLS: cmake ninja
.after_script:build:macos:upload-failed-job-logs:ccache-show-stats:
.after_script:build:
after_script:
# macos is running shell executor, which means it would use
# the system installed /usr/local/bin/python3 by default.
# Ensure pyenv and PYENV_VERSION installed
- eval "$(pyenv init -)"
- *upload_failed_job_log_artifacts
- *show_ccache_statistics
.before_script:build:
before_script:
- *common-before_scripts
- *setup_tools_and_idf_python_venv
- add_gitlab_ssh_keys
- fetch_submodules
- export EXTRA_CFLAGS=${PEDANTIC_CFLAGS}
- export EXTRA_CXXFLAGS=${PEDANTIC_CXXFLAGS}
.after_script:build:ccache-show-stats:
after_script:
- *show_ccache_statistics
.after_script:build:ccache-show-stats:upload-failed-job-logs:
after_script:
- *show_ccache_statistics
- *upload_failed_job_log_artifacts
##############################
# Git Strategy Job Templates #
##############################
.git_init: &git_init |
mkdir -p "${CI_PROJECT_DIR}"
cd "${CI_PROJECT_DIR}"
git init
.git_fetch_from_mirror_url_if_exists: &git_fetch_from_mirror_url_if_exists |
# check if set mirror
if [ -n "${LOCAL_GITLAB_HTTPS_HOST:-}" ] && [ -n "${ESPCI_TOKEN:-}" ]; then
MIRROR_REPO_URL="https://bot:${ESPCI_TOKEN}@${LOCAL_GITLAB_HTTPS_HOST}/${CI_PROJECT_PATH}"
elif [ -n "${LOCAL_GIT_MIRROR:-}" ]; then
MIRROR_REPO_URL="${LOCAL_GIT_MIRROR}/${CI_PROJECT_PATH}"
fi
# fetch from mirror first if set
if [ -n "${MIRROR_REPO_URL:-}" ]; then
if git remote -v | grep origin; then
git remote set-url origin "${MIRROR_REPO_URL}"
else
git remote add origin "${MIRROR_REPO_URL}"
fi
# mirror url may fail with authentication issue
git fetch origin --no-recurse-submodules || true
fi
# set remote url to CI_REPOSITORY_URL
if git remote -v | grep origin; then
git remote set-url origin "${CI_REPOSITORY_URL}"
else
git remote add origin "${CI_REPOSITORY_URL}"
fi
.git_checkout_ci_commit_sha: &git_checkout_ci_commit_sha |
git checkout $CI_COMMIT_SHA
git clean ${GIT_CLEAN_FLAGS}
# git diff requires two commits, with different CI env var
#
# By default, we use git strategy "clone" with depth 1 to speed up the clone process.
# But for jobs requires running `git diff`, we need to fetch more commits to get the correct diffs.
#
# Since there's no way to get the correct git_depth before the job starts,
# we can't set `GIT_DEPTH` in the job definition.
#
# Set git strategy to "none" and fetch manually instead.
.before_script:fetch:git_diff:
variables:
GIT_STRATEGY: none
before_script:
- *git_init
- *git_fetch_from_mirror_url_if_exists
- !reference [.common_after_scripts, ccache]
- run_cmd idf-ci gitlab upload-artifacts --type junit
- |
# merged results pipelines, by default
if [[ -n $CI_MERGE_REQUEST_SOURCE_BRANCH_SHA ]]; then
git fetch origin $CI_MERGE_REQUEST_DIFF_BASE_SHA --depth=1 ${GIT_FETCH_EXTRA_FLAGS}
git fetch origin $CI_MERGE_REQUEST_SOURCE_BRANCH_SHA --depth=1 ${GIT_FETCH_EXTRA_FLAGS}
export GIT_DIFF_OUTPUT=$(git diff --name-only $CI_MERGE_REQUEST_DIFF_BASE_SHA $CI_MERGE_REQUEST_SOURCE_BRANCH_SHA)
git fetch origin $CI_COMMIT_SHA --depth=1 ${GIT_FETCH_EXTRA_FLAGS}
# merge request pipelines, when the mr got conflicts
elif [[ -n $CI_MERGE_REQUEST_DIFF_BASE_SHA ]]; then
git fetch origin $CI_MERGE_REQUEST_DIFF_BASE_SHA --depth=1 ${GIT_FETCH_EXTRA_FLAGS}
git fetch origin $CI_COMMIT_SHA --depth=1 ${GIT_FETCH_EXTRA_FLAGS}
export GIT_DIFF_OUTPUT=$(git diff --name-only $CI_MERGE_REQUEST_DIFF_BASE_SHA $CI_COMMIT_SHA)
# other pipelines, like the protected branches pipelines
elif [[ "$CI_COMMIT_BEFORE_SHA" != "0000000000000000000000000000000000000000" ]]; then
git fetch origin $CI_COMMIT_BEFORE_SHA --depth=1 ${GIT_FETCH_EXTRA_FLAGS}
git fetch origin $CI_COMMIT_SHA --depth=1 ${GIT_FETCH_EXTRA_FLAGS}
export GIT_DIFF_OUTPUT=$(git diff --name-only $CI_COMMIT_BEFORE_SHA $CI_COMMIT_SHA)
else
# pipeline source could be web, scheduler, etc.
git fetch origin $CI_COMMIT_SHA --depth=2 ${GIT_FETCH_EXTRA_FLAGS}
export GIT_DIFF_OUTPUT=$(git diff --name-only $CI_COMMIT_SHA~1 $CI_COMMIT_SHA)
if [ $CI_JOB_STATUS = "failed" ]; then
run_cmd idf-ci gitlab upload-artifacts --type log
fi
- *git_checkout_ci_commit_sha
- *common-before_scripts
- *setup_tools_and_idf_python_venv
- add_gitlab_ssh_keys
# target test runners may locate in different places
# for runners set git mirror, we fetch from the mirror first, then fetch the HEAD commit
.before_script:fetch:target_test:
variables:
GIT_STRATEGY: none
before_script:
- *git_init
- *git_fetch_from_mirror_url_if_exists
- git fetch origin "${CI_COMMIT_SHA}" --depth=1 ${GIT_FETCH_EXTRA_FLAGS}
- *git_checkout_ci_commit_sha
- *common-before_scripts
- *setup_tools_and_idf_python_venv
- add_gitlab_ssh_keys
# no submodules
#############
# `default` #
@@ -403,12 +132,6 @@ default:
cache:
# pull only for most of the use cases since it's cache dir.
# Only set "push" policy for "upload_cache" stage jobs
- key: pip-cache-${LATEST_GIT_TAG}
fallback_keys:
- pip-cache
paths:
- .cache/pip
policy: pull
- key: submodule-cache-${LATEST_GIT_TAG}
fallback_keys:
- submodule-cache
@@ -416,20 +139,8 @@ default:
- .cache/submodule_archives
policy: pull
before_script:
- *common-before_scripts
- *setup_tools_and_idf_python_venv
- !reference [.common_before_scripts, with_tools]
- add_gitlab_ssh_keys
- fetch_submodules
# gitlab bug, setting them here doesn't work
# - expire_in: https://gitlab.com/gitlab-org/gitlab/-/issues/404563
# - when: https://gitlab.com/gitlab-org/gitlab/-/issues/440672
# artifacts:
# expire_in: 1 week
# when: always
retry:
max: 2
when:
# In case of a runner failure we could hop to another one, or a network error could go away.
- runner_system_failure
# Job execution timeout may be caused by a network issue.
- job_execution_timeout
- export EXTRA_CFLAGS=${PEDANTIC_CFLAGS}
- export EXTRA_CXXFLAGS=${PEDANTIC_CXXFLAGS}
.gitlab/ci/default-build-test-rules.yml
-20
View File
@@ -1,20 +0,0 @@
# this file support two keywords:
# - extra_default_build_targets:
# besides of the SUPPORTED_TARGETS in IDF,
# enable build for the specified targets by default as well.
# - bypass_check_test_targets:
# suppress the check_build_test_rules check-test-script warnings for the specified targets
#
# This file should ONLY be used during bringup. Should be reset to empty after the bringup process
extra_default_build_targets:
bypass_check_test_targets:
- esp32c5
- esp32c61
#
# These lines would
# - enable the README.md check for esp32c6. Don't forget to add the build jobs in .gitlab/ci/build.yml
# - disable the test script check with the manifest file.
#
.gitlab/ci/dependencies/dependencies.yml
-19
View File
@@ -1,19 +1,3 @@
.all_targets: &all_targets
- esp32
- esp32s2
- esp32s3
- esp32c3
- esp32c2
- esp32c6
- esp32c5
- esp32h2
- esp32p4
.target_test: &target_test
- example_test
- custom_test
- component_ut
##############
# Build Jobs #
##############
@@ -25,7 +9,6 @@
- build_system
- downloadable-tools
included_in:
- build:target_test
- build:check
# -------------------
@@ -57,8 +40,6 @@
"patterns:template-app":
patterns:
- build_template-app
included_in:
- build:target_test
"patterns:build-check":
patterns:
.gitlab/ci/deploy.yml
-48
View File
@@ -1,48 +0,0 @@
.deploy_job_template:
stage: deploy
image: $ESP_ENV_IMAGE
tags: [ deploy ]
# Check this before push_to_github
check_submodule_sync:
extends:
- .deploy_job_template
- .rules:test:submodule
stage: test_deploy
tags: [ brew, github_sync ]
retry: 2
variables:
# for brew runners, we always set GIT_STRATEGY to fetch
GIT_STRATEGY: fetch
SUBMODULES_TO_FETCH: "none"
PUBLIC_IDF_URL: "https://github.com/espressif/esp-idf.git"
dependencies: []
script:
- git submodule deinit --force .
- rm -rf .git/modules # remove all the cached metadata
# setting the default remote URL to the public one, to resolve relative location URLs
- git config remote.origin.url ${PUBLIC_IDF_URL}
# check if all submodules are correctly synced to public repository
- git submodule init
- git config --get-regexp '^submodule\..*\.url$' || true
- git submodule update --recursive
- echo "IDF was cloned from ${PUBLIC_IDF_URL} completely"
push_to_github:
extends:
- .deploy_job_template
- .before_script:minimal
- .rules:push_to_github
needs:
- check_submodule_sync
tags: [ brew, github_sync ]
variables:
# for brew runners, we always set GIT_STRATEGY to fetch
GIT_STRATEGY: fetch
# github also need full record of commits
GIT_DEPTH: 0
script:
- add_github_ssh_keys
- git remote remove github &>/dev/null || true
- git remote add github git@github.com:espressif/esp-idf.git
- tools/ci/push_to_github.sh
.gitlab/ci/host-test.yml
+244 -177
View File
@@ -2,18 +2,17 @@
extends: .rules:test:host_test
stage: host_test
image: $ESP_ENV_IMAGE
tags:
- host_test
tags: [build, shiny]
dependencies: # set dependencies to null to avoid missing artifacts issue
# run host_test jobs immediately, only after upload cache
needs:
- job: upload-pip-cache
optional: true
artifacts: false
- pipeline_variables
- job: upload-submodules-cache
optional: true
artifacts: false
- pipeline_variables
- job: fast_template_app
optional: true
artifacts: false
artifacts:
expire_in: 1 week
when: always
@@ -23,16 +22,28 @@ check_public_headers:
- .host_test_template
- .rules:build:check
script:
- IDF_TARGET=esp32 python tools/ci/check_public_headers.py --jobs 4 --prefix xtensa-esp32-elf-
- IDF_TARGET=esp32s2 python tools/ci/check_public_headers.py --jobs 4 --prefix xtensa-esp32s2-elf-
- IDF_TARGET=esp32s3 python tools/ci/check_public_headers.py --jobs 4 --prefix xtensa-esp32s3-elf-
- IDF_TARGET=esp32c3 python tools/ci/check_public_headers.py --jobs 4 --prefix riscv32-esp-elf-
- IDF_TARGET=esp32c2 python tools/ci/check_public_headers.py --jobs 4 --prefix riscv32-esp-elf-
- IDF_TARGET=esp32c6 python tools/ci/check_public_headers.py --jobs 4 --prefix riscv32-esp-elf-
- IDF_TARGET=esp32c5 python tools/ci/check_public_headers.py --jobs 4 --prefix riscv32-esp-elf-
- IDF_TARGET=esp32h2 python tools/ci/check_public_headers.py --jobs 4 --prefix riscv32-esp-elf-
- IDF_TARGET=esp32p4 python tools/ci/check_public_headers.py --jobs 4 --prefix riscv32-esp-elf-
- IDF_TARGET=esp32c61 python tools/ci/check_public_headers.py --jobs 4 --prefix riscv32-esp-elf-
- |
targets="esp32 \
esp32s2 \
esp32s3 \
esp32c3 \
esp32c2 \
esp32c6 \
esp32c5 \
esp32h2 \
esp32p4 \
esp32c61"
for target in ${targets}; do
case "${target}" in
esp32|esp32s2|esp32s3)
PREFIX="xtensa-${target}-elf-"
;;
*)
PREFIX="riscv32-esp-elf-"
;;
esac
IDF_TARGET=${target} python tools/ci/check_public_headers.py --jobs 4 --prefix ${PREFIX}
done
test_nvs_coverage:
extends:
@@ -51,14 +62,15 @@ test_partition_table_on_host:
extends: .host_test_template
script:
- cd components/partition_table/test_gen_esp32part_host
- ./gen_esp32part_tests.py
- pytest_for_ut ./gen_esp32part_tests.py
test_ldgen_on_host:
extends: .host_test_template
extends:
- .host_test_template
script:
- cd tools/ldgen/test
- export PYTHONPATH=$PYTHONPATH:..
- python -m unittest
- pytest_for_ut .
variables:
LC_ALL: C.UTF-8
@@ -66,15 +78,13 @@ test_spiffs_on_host:
extends: .host_test_template
script:
- cd components/spiffs/test_spiffsgen/
- ./test_spiffsgen.py
- pytest_for_ut ./test_spiffsgen.py
test_fatfsgen_on_host:
extends: .host_test_template
script:
- cd components/fatfs/test_fatfsgen/
- ./test_fatfsgen.py
- ./test_wl_fatfsgen.py
- ./test_fatfsparse.py
- pytest_for_ut ./test_fatfsgen.py ./test_wl_fatfsgen.py ./test_fatfsparse.py
test_multi_heap_on_host:
extends: .host_test_template
@@ -86,7 +96,7 @@ test_certificate_bundle_on_host:
extends: .host_test_template
script:
- cd components/mbedtls/esp_crt_bundle/test_gen_crt_bundle/
- ./test_gen_crt_bundle.py
- pytest_for_ut ./test_gen_crt_bundle.py
test_gdbstub_on_host:
extends: .host_test_template
@@ -115,15 +125,17 @@ test_cli_installer:
- python3 ${IDF_PATH}/tools/idf_tools.py download required qemu-riscv32 qemu-xtensa cmake cmake@3.16.3
- cd ${IDF_PATH}/tools/test_idf_tools
- python3 -m pip install jsonschema --break-system-packages
# Testing with system python3, so don't use any third-party packages
- python3 ./test_idf_tools.py -v
- python3 ./test_idf_tools_python_env.py
# It runs at the end because it modifies dependencies
- IDF_TEST_MAY_BREAK_DEPENDENCIES=1 python3 ./test_idf_tools.py -v TestSystemDependencies.test_commands_when_nodeps
.test_efuse_table_on_host_template:
test_efuse_table_on_host:
extends: .host_test_template
variables:
IDF_TARGET: "esp32"
parallel:
matrix:
- IDF_TARGET: [esp32, esp32s2, esp32c3, esp32s3, esp32c6, esp32h2]
artifacts:
when: on_failure
paths:
@@ -133,35 +145,7 @@ test_cli_installer:
- ./efuse_table_gen.py -t "${IDF_TARGET}" ${IDF_PATH}/components/efuse/${IDF_TARGET}/esp_efuse_table.csv
- git diff --exit-code -- ${IDF_TARGET}/esp_efuse_table.c || { echo 'Differences found for ${IDF_TARGET} target. Please run idf.py efuse-common-table and commit the changes.'; exit 1; }
- cd ${IDF_PATH}/components/efuse/test_efuse_host
- ./efuse_tests.py
test_efuse_table_on_host_esp32:
extends: .test_efuse_table_on_host_template
test_efuse_table_on_host_esp32s2:
extends: .test_efuse_table_on_host_template
variables:
IDF_TARGET: esp32s2
test_efuse_table_on_host_esp32s3:
extends: .test_efuse_table_on_host_template
variables:
IDF_TARGET: esp32s3
test_efuse_table_on_host_esp32c3:
extends: .test_efuse_table_on_host_template
variables:
IDF_TARGET: esp32c3
test_efuse_table_on_host_esp32h2:
extends: .test_efuse_table_on_host_template
variables:
IDF_TARGET: esp32h2
test_efuse_table_on_host_esp32c6:
extends: .test_efuse_table_on_host_template
variables:
IDF_TARGET: esp32c6
- pytest_for_ut ./efuse_tests.py
test_logtrace_proc:
extends: .host_test_template
@@ -199,16 +183,17 @@ test_tools:
INSTALL_EXTRA_TOOLS: "qemu-xtensa qemu-riscv32" # for test_idf_qemu.py
script:
- stat=0
- run_cmd idf-ci gitlab download-known-failure-cases-file ${KNOWN_FAILURE_CASES_FILE_NAME}
- cd ${IDF_PATH}/tools/ci/test_autocomplete
- pytest --noconftest test_autocomplete.py --junitxml=${IDF_PATH}/XUNIT_AUTOCOMP.xml || stat=1
- run_cmd pytest --noconftest test_autocomplete.py --junitxml=${IDF_PATH}/XUNIT_AUTOCOMP.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME} || stat=1
- cd ${IDF_PATH}/tools/test_idf_py
- pytest --noconftest test_idf_py.py --junitxml=${IDF_PATH}/XUNIT_IDF_PY.xml || stat=1
- pytest --noconftest test_hints.py --junitxml=${IDF_PATH}/XUNIT_HINTS.xml || stat=1
- pytest --noconftest test_idf_qemu.py --junitxml=${IDF_PATH}/XUNIT_IDF_PY_QEMU.xml || stat=1
- run_cmd pytest --noconftest test_idf_py.py --junitxml=${IDF_PATH}/XUNIT_IDF_PY.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME} || stat=1
- run_cmd pytest --noconftest test_hints.py --junitxml=${IDF_PATH}/XUNIT_HINTS.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME} || stat=1
- run_cmd pytest --noconftest test_idf_qemu.py --junitxml=${IDF_PATH}/XUNIT_IDF_PY_QEMU.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME} || stat=1
- cd ${IDF_PATH}/tools/test_mkdfu
- pytest --noconftest test_mkdfu.py --junitxml=${IDF_PATH}/XUNIT_MKDFU.xml || stat=1
- run_cmd pytest --noconftest test_mkdfu.py --junitxml=${IDF_PATH}/XUNIT_MKDFU.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME} || stat=1
- cd ${IDF_PATH}/tools/test_idf_size
- pytest --noconftest test_idf_size.py --junitxml=${IDF_PATH}/XUNIT_IDF_SIZE.xml || stat=1
- run_cmd pytest --noconftest test_idf_size.py --junitxml=${IDF_PATH}/XUNIT_IDF_SIZE.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME} || stat=1
- cd ${IDF_PATH}
- shellcheck -s sh tools/detect_python.sh || stat=1
- shellcheck -s bash tools/detect_python.sh || stat=1
@@ -223,12 +208,15 @@ test_split_path_by_spaces:
extends: .host_test_template
script:
- cd ${IDF_PATH}/tools
- python -m unittest split_paths_by_spaces.py
- pytest_for_ut ./split_paths_by_spaces.py
test_mqtt_on_host:
extends: .host_test_template
script:
- cd ${IDF_PATH}/components/mqtt/esp-mqtt/host_test
# no -Werror
- export EXTRA_CFLAGS=
- export EXTRA_CXXFLAGS=
- idf.py build
- LSAN_OPTIONS=verbosity=1:log_threads=1 build/host_mqtt_client_test.elf
@@ -269,122 +257,13 @@ test_gen_soc_caps_kconfig:
extends: .host_test_template
script:
- cd ${IDF_PATH}/tools/gen_soc_caps_kconfig/
- ./test/test_gen_soc_caps_kconfig.py
test_pytest_qemu:
extends:
- .host_test_template
- .before_script:build
artifacts:
paths:
- XUNIT_RESULT.xml
- pytest-embedded/
- "**/build*/*.bin"
reports:
junit: XUNIT_RESULT.xml
parallel:
matrix:
- IDF_TARGET: "esp32"
INSTALL_EXTRA_TOOLS: "qemu-xtensa"
- IDF_TARGET: "esp32c3"
INSTALL_EXTRA_TOOLS: "qemu-riscv32"
script:
- run_cmd python tools/ci/ci_build_apps.py . -v
--target $IDF_TARGET
--pytest-apps
-m qemu
--collect-app-info "list_job_${CI_JOB_NAME_SLUG}.txt"
--modified-components ${MR_MODIFIED_COMPONENTS}
--modified-files ${MR_MODIFIED_FILES}
- python tools/ci/get_known_failure_cases_file.py
- run_cmd pytest
--target $IDF_TARGET
--log-cli-level DEBUG
-m qemu
--embedded-services idf,qemu
--junitxml=XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
--app-info-filepattern \"list_job_*.txt\"
--qemu-extra-args \"-global driver=timer.$IDF_TARGET.timg,property=wdt_disable,value=true\"
test_pytest_linux:
extends:
- .host_test_template
- .before_script:build
artifacts:
paths:
- XUNIT_RESULT.xml
- pytest-embedded/
- "**/build*/build_log.txt"
reports:
junit: XUNIT_RESULT.xml
script:
- run_cmd python tools/ci/ci_build_apps.py components examples tools/test_apps -v
--target linux
--pytest-apps
-m host_test
--collect-app-info "list_job_${CI_JOB_NAME_SLUG}.txt"
--modified-components ${MR_MODIFIED_COMPONENTS}
--modified-files ${MR_MODIFIED_FILES}
- python tools/ci/get_known_failure_cases_file.py
- run_cmd pytest
--target linux
-m host_test
--embedded-services idf
--junitxml=XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
--app-info-filepattern \"list_job_*.txt\"
test_pytest_macos:
extends:
- .host_test_template
- .before_script:build:macos
tags:
- macos_shell
artifacts:
paths:
- XUNIT_RESULT.xml
- pytest-embedded/
- "**/build*/build_log.txt"
reports:
junit: XUNIT_RESULT.xml
variables:
PYTEST_IGNORE_COLLECT_IMPORT_ERROR: "1"
script:
- run_cmd python tools/ci/ci_build_apps.py components examples tools/test_apps -v
--target linux
--pytest-apps
-m \"host_test and macos_shell\"
--collect-app-info "list_job_${CI_JOB_NAME_SLUG}.txt"
--modified-components ${MR_MODIFIED_COMPONENTS}
--modified-files ${MR_MODIFIED_FILES}
- python tools/ci/get_known_failure_cases_file.py
- run_cmd pytest
--target linux
-m \"host_test and macos_shell\"
--junitxml=XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
--app-info-filepattern \"list_job_*.txt\"
test_idf_pytest_plugin:
extends:
- .host_test_template
- .rules:patterns:idf-pytest-plugin
variables:
SUBMODULES_TO_FETCH: "none"
artifacts:
reports:
junit: XUNIT_RESULT.xml
script:
- cd ${IDF_PATH}/tools/ci/dynamic_pipelines/tests/test_report_generator
- python -m unittest test_report_generator.py
- cd ${IDF_PATH}/tools/ci/idf_pytest
- pytest --junitxml=${CI_PROJECT_DIR}/XUNIT_RESULT.xml
- pytest_for_ut ./test/test_gen_soc_caps_kconfig.py
test_idf_build_apps_load_soc_caps:
extends: .host_test_template
script:
- python tools/ci/check_soc_headers_load_in_idf_build_apps.py
- cd tools/ci
- pytest_for_ut ./check_soc_headers_load_in_idf_build_apps.py
test_nvs_gen_check:
extends: .host_test_template
@@ -398,7 +277,7 @@ test_nvs_gen_check:
LC_ALL: C.UTF-8
script:
- cd ${IDF_PATH}/components/nvs_flash/nvs_partition_tool
- pytest --noconftest test_nvs_gen_check.py --junitxml=XUNIT_RESULT.xml
- pytest_for_ut test_nvs_gen_check.py
test_esp_rom:
extends: .host_test_template
@@ -409,7 +288,7 @@ test_esp_rom:
junit: XUNIT_RESULT.xml
script:
- cd ${IDF_PATH}/components/esp_rom/
- pytest --noconftest test_esp_rom.py --junitxml=XUNIT_RESULT.xml
- pytest_for_ut test_esp_rom.py
make_sure_soc_caps_compatible_in_idf_build_apps:
extends:
@@ -428,3 +307,191 @@ make_sure_soc_caps_compatible_in_idf_build_apps:
- git checkout $CI_COMMIT_SHA -- tools/ci/idf_build_apps_dump_soc_caps.py
- python tools/ci/idf_build_apps_dump_soc_caps.py base.json
- diff new.json base.json
build_docker:
extends:
- .before_script:minimal
- .rules:build:docker
stage: host_test
needs:
- job: manual_gate
optional: true
image: espressif/docker-builder:1
tags: [shiny, dind]
variables:
DOCKER_TMP_IMAGE_NAME: "idf_tmp_image"
script:
# use different image tag for different concurrent job
- export DOCKER_TMP_IMAGE="${DOCKER_TMP_IMAGE_NAME}:${CI_CONCURRENT_ID:-latest}"
- export DOCKER_BUILD_ARGS="--build-arg IDF_CLONE_URL=${CI_REPOSITORY_URL} --build-arg IDF_CLONE_BRANCH_OR_TAG=${CI_COMMIT_REF_NAME} --build-arg IDF_CHECKOUT_REF=${CI_COMMIT_TAG:-$CI_COMMIT_SHA} --build-arg IDF_CLONE_SHALLOW=1 ${EXTRA_DOCKER_BUILD_ARGS:-} "
- time docker build --tag ${DOCKER_TMP_IMAGE} ${DOCKER_BUILD_ARGS} tools/docker/
# We can't mount $PWD/examples/get-started/blink into the container, see https://gitlab.com/gitlab-org/gitlab-ce/issues/41227.
# The workaround mentioned there works, but leaves around directories which need to be cleaned up manually.
# Therefore, build a copy of the example located inside the container.
- docker run --rm --workdir /opt/esp/idf/examples/get-started/blink ${DOCKER_TMP_IMAGE} idf.py build
############################
# Host test with test apps #
############################
test_pytest_qemu:
extends:
- .host_test_template
artifacts:
paths:
- XUNIT_RESULT.xml
- pytest-embedded/
- "**/build*/*.bin"
reports:
junit: XUNIT_RESULT.xml
parallel:
matrix:
- IDF_TARGET: "esp32"
INSTALL_EXTRA_TOOLS: "qemu-xtensa"
# Skip Clang + Xtensa tests due to bootloader size issue
IDF_TOOLCHAIN: [gcc]
- IDF_TARGET: "esp32c3"
INSTALL_EXTRA_TOOLS: "qemu-riscv32"
IDF_TOOLCHAIN: [gcc, clang]
script:
- run_cmd idf-ci build run
--build-system cmake
--target $IDF_TARGET
-m qemu
--modified-files ${MR_MODIFIED_FILES}
- run_cmd idf-ci gitlab download-known-failure-cases-file ${KNOWN_FAILURE_CASES_FILE_NAME}
- run_cmd pytest
--target $IDF_TARGET
-m qemu
--junitxml=XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
--qemu-extra-args \"-global driver=timer.$IDF_TARGET.timg,property=wdt_disable,value=true\"
test_pytest_linux:
extends:
- .host_test_template
artifacts:
paths:
- XUNIT_RESULT.xml
- pytest-embedded/
- "**/build*/build_log.txt"
reports:
junit: XUNIT_RESULT.xml
script:
- run_cmd idf-ci build run
--build-system cmake
-p components -p examples -p tools/test_apps
--target linux
--only-test-related
--modified-files ${MR_MODIFIED_FILES}
- run_cmd idf-ci gitlab download-known-failure-cases-file ${KNOWN_FAILURE_CASES_FILE_NAME}
- run_cmd pytest
--target linux
-m \"not macos\"
--junitxml=XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
test_pytest_macos:
extends:
- .host_test_template
- .macos-settings
artifacts:
paths:
- XUNIT_RESULT.xml
- pytest-embedded/
- "**/build*/build_log.txt"
reports:
junit: XUNIT_RESULT.xml
script:
- run_cmd idf-ci build run
-p components -p examples -p tools/test_apps
--build-system cmake
--target linux
--only-test-related
-m macos
--modified-files ${MR_MODIFIED_FILES}
- run_cmd idf-ci gitlab download-known-failure-cases-file ${KNOWN_FAILURE_CASES_FILE_NAME}
- run_cmd pytest
--target linux
-m macos
--junitxml=XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
######################
# Build System Tests #
######################
.test_build_system_template:
stage: host_test
extends:
- .build_template
- .rules:build:check
dependencies: # set dependencies to null to avoid missing artifacts issue
needs:
- job: fast_template_app
artifacts: false
optional: true
artifacts:
reports:
junit: XUNIT_RESULT.xml
paths:
- XUNIT_RESULT.xml
- test_build_system
expire_in: 1 week
when: always
script:
- ${IDF_PATH}/tools/ci/test_configure_ci_environment.sh
- cd ${IDF_PATH}/tools/test_build_system
- run_cmd idf-ci gitlab download-known-failure-cases-file ${KNOWN_FAILURE_CASES_FILE_NAME}
- pytest
--cleanup-idf-copy
--parallel-count ${CI_NODE_TOTAL:-1}
--parallel-index ${CI_NODE_INDEX:-1}
--work-dir ${CI_PROJECT_DIR}/test_build_system
--junitxml ${CI_PROJECT_DIR}/XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
pytest_build_system:
extends: .test_build_system_template
parallel: 3
pytest_build_system_macos:
extends:
- .test_build_system_template
- .macos-settings
- .rules:build:macos
parallel: 3
.test_build_system_minimal_cmake_template:
extends: .test_build_system_template
variables:
INSTALL_EXTRA_TOOLS: cmake@3.16.3
script:
- MINIMAL_SUPPORTED_CMAKE_VERSION=$(echo "${INSTALL_EXTRA_TOOLS}" | sed -n 's/.*cmake@\([0-9.]*\).*/\1/p')
- export PATH=$(echo "$PATH" | sed -E "s|/tools/cmake/[0-9.]+|/tools/cmake/${MINIMAL_SUPPORTED_CMAKE_VERSION}|")
- ACTUAL_CMAKE_VERSION=$(cmake --version | head -n1 | awk '{print $3}')
- |
if [ "${ACTUAL_CMAKE_VERSION}" != "${MINIMAL_SUPPORTED_CMAKE_VERSION}" ]; then
echo "ERROR: Wrong minimal CMake version! Detected: ${ACTUAL_CMAKE_VERSION}, but should be: ${MINIMAL_SUPPORTED_CMAKE_VERSION}"
exit 1
fi
- ${IDF_PATH}/tools/ci/test_configure_ci_environment.sh
- cd ${IDF_PATH}/tools/test_build_system
- run_cmd idf-ci gitlab download-known-failure-cases-file ${KNOWN_FAILURE_CASES_FILE_NAME}
- pytest
-k cmake
--cleanup-idf-copy
--parallel-count ${CI_NODE_TOTAL:-1}
--parallel-index ${CI_NODE_INDEX:-1}
--work-dir ${CI_PROJECT_DIR}/test_build_system
--junitxml ${CI_PROJECT_DIR}/XUNIT_RESULT.xml
--ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
pytest_build_system_minimal_cmake:
extends: .test_build_system_minimal_cmake_template
pytest_build_system_macos_minimal_cmake:
extends:
- .test_build_system_minimal_cmake_template
- .macos-settings
- .rules:build:macos
variables:
INSTALL_EXTRA_TOOLS: ninja cmake@3.16.3
.gitlab/ci/post_deploy.yml
-13
View File
@@ -1,13 +0,0 @@
generate_failed_jobs_report:
stage: post_deploy
tags: [build, shiny]
image: $ESP_ENV_IMAGE
when: always
dependencies: [] # Do not download artifacts from the previous stages
artifacts:
expire_in: 1 week
when: always
paths:
- job_report.html
script:
- python tools/ci/dynamic_pipelines/scripts/generate_report.py --report-type job
.gitlab/ci/pre_check.yml
+8 -100
View File
@@ -1,16 +1,9 @@
.pre_check_template:
stage: pre_check
image: $ESP_ENV_IMAGE
tags:
- host_test
dependencies: # set dependencies to null to avoid missing artifacts issue
check_version:
# Don't run this for feature/bugfix branches, so that it is possible to modify
# esp_idf_version.h in a branch before tagging the next version.
extends:
- .pre_check_template
- .rules:protected
- .rules:protected:check
tags: [ brew, github_sync ]
variables:
# need a full clone to get the latest tag
@@ -116,108 +109,23 @@ check_version_tag:
check_test_scripts_build_test_rules:
extends:
- .pre_check_template
- .before_script:build
variables:
PYTEST_IGNORE_COLLECT_IMPORT_ERROR: "1"
script:
# requires basic pytest dependencies
- run_cmd bash install.sh --enable-pytest
- python tools/ci/check_build_test_rules.py check-test-scripts examples/ tools/test_apps components
check_configure_ci_environment_parsing:
check_gitlab_yaml:
extends:
- .pre_check_template
- .before_script:build
- .rules:build
script:
- cd tools/ci
- python -m unittest ci_build_apps.py
- python tools/ci/gitlab_yaml_linter.py
pipeline_variables:
extends:
- .pre_check_template
- .before_script:fetch:git_diff
tags: [fast_run, shiny]
script:
# MODIFIED_FILES is a list of files that changed, could be used everywhere
- MODIFIED_FILES=$(echo "$GIT_DIFF_OUTPUT" | xargs)
- echo "MODIFIED_FILES=$MODIFIED_FILES" >> pipeline.env
- echo "REPORT_EXIT_CODE=0" >> pipeline.env
# MR_MODIFIED_FILES and MR_MODIFIED_COMPONENTS are semicolon separated lists that is used in MR only
# for non MR pipeline, these are empty lists
- |
if [ $IS_MR_PIPELINE == "0" ]; then
echo "MR_MODIFIED_FILES=\"\"" >> pipeline.env
echo "MR_MODIFIED_COMPONENTS=\"\"" >> pipeline.env
else
MR_MODIFIED_FILES=$(echo "$GIT_DIFF_OUTPUT" | tr '\n' ';')
echo "MR_MODIFIED_FILES=\"$MR_MODIFIED_FILES\"" >> pipeline.env
MR_MODIFIED_COMPONENTS=$(run_cmd python tools/ci/ci_get_mr_info.py components --modified-files $MODIFIED_FILES | tr '\n' ';')
echo "MR_MODIFIED_COMPONENTS=\"$MR_MODIFIED_COMPONENTS\"" >> pipeline.env
fi
- |
if echo "$CI_MERGE_REQUEST_LABELS" | egrep "(^|,)BUILD_AND_TEST_ALL_APPS(,|$)"; then
echo "BUILD_AND_TEST_ALL_APPS=1" >> pipeline.env
fi
# run full pipeline if testing constraint branch
- |
if [ -n "$CI_PYTHON_CONSTRAINT_BRANCH" ]; then
echo "BUILD_AND_TEST_ALL_APPS=1" >> pipeline.env
fi
- echo "OOCD_DISTRO_URL_ARMHF=$OOCD_DISTRO_URL_ARMHF" >> pipeline.env
- echo "OOCD_DISTRO_URL_ARM64=$OOCD_DISTRO_URL_ARM64" >> pipeline.env
- python tools/ci/ci_process_description.py
- cat pipeline.env
- python tools/ci/artifacts_handler.py upload --type modified_files_and_components_report
artifacts:
reports:
dotenv: pipeline.env
paths:
- pipeline.env
expire_in: 1 week
when: always
baseline_manifest_sha:
retry_failed_jobs:
extends:
- .pre_check_template
- .rules:dev-push
tags: [fast_run, shiny]
script:
- |
# merged results pipelines, by default
# diff between target-branch-head and merged-result-head
if [ -n "$CI_MERGE_REQUEST_TARGET_BRANCH_SHA" ]; then
git fetch origin $CI_MERGE_REQUEST_TARGET_BRANCH_SHA --depth=1
git checkout FETCH_HEAD
idf-build-apps dump-manifest-sha \
--manifest-files $(find . -name ".build-test-rules.yml" | xargs) \
--output .manifest_sha
# merge request pipelines, when the mr got conflicts
# diff between diff-base-sha and merge-request-head
elif [ -n "$CI_MERGE_REQUEST_DIFF_BASE_SHA" ]; then
git fetch origin $CI_MERGE_REQUEST_DIFF_BASE_SHA --depth=1
git checkout FETCH_HEAD
idf-build-apps dump-manifest-sha \
--manifest-files $(find . -name ".build-test-rules.yml" | xargs) \
--output .manifest_sha
# other pipelines, like the protected branches pipelines
# not triggered in this job
fi
artifacts:
paths:
- .manifest_sha
expire_in: 1 week
when: always
redundant_pass_job:
stage: pre_check
tags: [shiny, fast_run]
image: $ESP_ENV_IMAGE
dependencies: null
before_script: []
cache: []
extends: []
allow_failure: true
script:
- echo "This job is redundant to ensure the 'retry_failed_jobs' job can exist and not be skipped"
when: always
- echo "Retrieving and retrying all failed jobs for the pipeline..."
- python tools/ci/python_packages/gitlab_api.py retry_failed_jobs $CI_MERGE_REQUEST_PROJECT_ID --pipeline_id $CI_PIPELINE_ID
when: manual
.gitlab/ci/pre_commit.yml
+13 -35
View File
@@ -2,33 +2,20 @@
extends:
- .before_script:minimal
stage: pre_check
image: $PRE_COMMIT_IMAGE
tags: [cache, shiny]
image: "${CI_REGISTRY}/ci/images/pre-commit-idf-v5.4:1"
tags: [pre-commit]
variables:
# Both shiny and brew runners can pick this job
GIT_STRATEGY: fetch
GIT_DEPTH: 1
SUBMODULES_TO_FETCH: "all"
needs:
- pipeline_variables
variables:
# cache pre_commit
PRE_COMMIT_HOME: "$CI_PROJECT_DIR/.cache/pre-commit"
script:
- fetch_submodules
- pre-commit run --files $MODIFIED_FILES
- pre-commit run --hook-stage post-commit validate-sbom-manifest
check_pre_commit_upload_cache:
extends:
- .check_pre_commit_template
rules:
- if: '($CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_BRANCH =~ /^release\/v/) && $CI_PIPELINE_SOURCE == "push"'
cache:
- key: pre_commit-cache-${LATEST_GIT_TAG}
paths:
- .cache/pre-commit
policy: pull-push
- key: submodule-cache-${LATEST_GIT_TAG}
paths:
- .cache/submodule_archives
policy: pull
check_pre_commit:
extends:
- .check_pre_commit_template
@@ -37,10 +24,6 @@ check_pre_commit:
when: never
- when: on_success
cache:
- key: pre_commit-cache-${LATEST_GIT_TAG}
paths:
- .cache/pre-commit
policy: pull
- key: submodule-cache-${LATEST_GIT_TAG}
paths:
- .cache/submodule_archives
@@ -56,25 +39,20 @@ check_powershell:
tags:
- dind
- amd64
- brew # faster "apk add"
needs:
- pipeline_variables
variables:
# cache pre_commit
PRE_COMMIT_HOME: "$CI_PROJECT_DIR/.cache/pre-commit"
# brew runners always use fetch
GIT_STRATEGY: fetch
GIT_DEPTH: 1
SUBMODULES_TO_FETCH: "none"
rules:
- changes:
- "*.ps1"
- ".gitlab/ci/pre_commit.yml"
script:
- apk add python3
- apk add py3-pip
- pip install pre-commit --break-system-packages
- pre-commit run --hook-stage manual check-powershell-scripts --files $MODIFIED_FILES
cache:
- key: pre_commit-cache-${LATEST_GIT_TAG}
paths:
- .cache/pre-commit
policy: pull
- key: submodule-cache-${LATEST_GIT_TAG}
paths:
- .cache/submodule_archives
policy: pull
.gitlab/ci/pre_deploy.yml
+24
View File
@@ -0,0 +1,24 @@
.pre_deploy_template:
stage: pre_deploy
image: $ESP_ENV_IMAGE
check_submodule_sync:
extends:
- .pre_deploy_template
- .rules:test:submodule
tags: [ brew, github_sync ]
retry: 2
variables:
GIT_STRATEGY: fetch # use brew local mirror first
SUBMODULES_TO_FETCH: "none"
PUBLIC_IDF_URL: "https://github.com/espressif/esp-idf.git"
script:
- git submodule deinit --force .
- rm -rf .git/modules # remove all the cached metadata
# setting the default remote URL to the public one, to resolve relative location URLs
- git config remote.origin.url ${PUBLIC_IDF_URL}
# check if all submodules are correctly synced to public repository
- git submodule init
- git config --get-regexp '^submodule\..*\.url$' || true
- git submodule update --recursive
- echo "IDF was cloned from ${PUBLIC_IDF_URL} completely"
.gitlab/ci/retry_failed_jobs.yml
-15
View File
@@ -1,15 +0,0 @@
retry_failed_jobs:
stage: retry_failed_jobs
tags: [shiny, fast_run]
allow_failure: true
image: $ESP_ENV_IMAGE
dependencies: null
before_script: []
cache: []
extends: []
script:
- echo "Retrieving and retrying all failed jobs for the pipeline..."
- python tools/ci/python_packages/gitlab_api.py retry_failed_jobs $CI_MERGE_REQUEST_PROJECT_ID --pipeline_id $CI_PIPELINE_ID
when: manual
needs:
- redundant_pass_job
.gitlab/ci/rules.yml
+25 -82
View File
@@ -9,19 +9,10 @@
- "**/*.{c,C,cpp}"
- "**/*.{h,H,hpp}"
- "Kconfig"
- "components/**/Kconfig"
- "components/**/CMakeList.txt"
- "components/**/CMakeLists.txt"
.patterns-python-cache: &patterns-python-cache
- "tools/requirements.json"
- "tools/requirements/requirements.*.txt"
.patterns-python-files: &patterns-python-files
- ".gitlab/ci/static-code-analysis.yml"
- "**/*.py"
.patterns-sonarqube-files: &patterns-sonarqube-files
- "tools/ci/sonar_exclude_list.txt"
.patterns-build_components: &patterns-build_components
# components files except "test*/" "host*/" folders
@@ -35,6 +26,7 @@
# Add folders excluded by "???[!t]" and "??[!s]?"
# pre-commit: tools/ci/check_rules_components_patterns.py
- "components/bt/host/**/*"
- "components/esp_psram/system_layer/*"
.patterns-downloadable-tools: &patterns-downloadable-tools
- "tools/idf_tools.py"
@@ -66,8 +58,10 @@
- "tools/ci/check_esp_memory_utils_headers.sh"
- "tools/ci/check_blobs.sh"
- "tools/ci/check_public_headers.py"
- "tools/ci/check_public_headers_exceptions.txt"
- "tools/ci/check_register_rw_half_word.cmake"
- "tools/ci/check_register_rw_half_word.py"
- "examples/build_system/**/*"
.patterns-host_test: &patterns-host_test
- ".gitlab/ci/host-test.yml"
@@ -97,6 +91,7 @@
- "tools/idf_tools.py"
- "tools/test_idf_tools/**/*"
- "tools/install_util.py"
- "tools/idf_extra_components.yml"
- "tools/export_utils/utils.py"
- "tools/export_utils/shell_types.py"
@@ -160,34 +155,22 @@
##############
# if anchors #
##############
.if-ref-master: &if-ref-master
if: '$CI_COMMIT_REF_NAME == "master"'
.if-master-push: &if-master-push
if: '$CI_COMMIT_REF_NAME == "master" && $CI_PIPELINE_SOURCE == "push"'
.if-tag-release: &if-tag-release
.if-release-tag: &if-release-tag
if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+(\.\d+)?($|-)/'
.if-protected: &if-protected
.if-protected-check: &if-protected-check
if: '($CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_BRANCH =~ /^release\/v/ || $CI_COMMIT_TAG =~ /^v\d+\.\d+(\.\d+)?($|-)/) || $CI_COMMIT_TAG =~ /^qa-test/'
.if-protected-deploy: &if-protected-deploy
if: '($CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_BRANCH =~ /^release\/v/ || $CI_COMMIT_TAG =~ /^v\d+\.\d+(\.\d+)?($|-)/) && ($CI_PIPELINE_SOURCE == "push" || $CI_PIPELINE_SOURCE == "api")'
.if-protected-no_label: &if-protected-no_label
if: '($CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_BRANCH =~ /^release\/v/ || $CI_COMMIT_TAG =~ /^v\d+\.\d+(\.\d+)?($|-)/) && $BOT_TRIGGER_WITH_LABEL == null'
.if-protected-ref-push: &if-protected-ref-push
.if-protected-branch-push: &if-protected-branch-push
# rules:changes always evaluates to true for new branch pipelines or when there is no Git push event
if: '($CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_BRANCH =~ /^release\/v/) && $CI_PIPELINE_SOURCE == "push"'
.if-qa-test-tag: &if-qa-test-tag
if: '$CI_COMMIT_TAG =~ /^qa-test/'
.if-dev-push: &if-dev-push
if: '$CI_COMMIT_REF_NAME != "master" && $CI_COMMIT_BRANCH !~ /^release\/v/ && $CI_COMMIT_TAG !~ /^v\d+\.\d+(\.\d+)?($|-)/ && $CI_COMMIT_TAG !~ /^qa-test/ && ($CI_PIPELINE_SOURCE == "push" || $CI_PIPELINE_SOURCE == "merge_request_event")'
.if-schedule: &if-schedule
if: '$CI_PIPELINE_SOURCE == "schedule"'
.if-schedule-nightly: &if-schedule-nightly
if: '$CI_PIPELINE_SOURCE == "schedule" && $INCLUDE_NIGHTLY_RUN == "1"'
@@ -207,39 +190,26 @@
# Rules #
#########
### Branches ###
.rules:protected:
.rules:protected:check:
rules:
- <<: *if-protected
- <<: *if-protected-check
.rules:push_to_github:
.rules:master:push:
rules:
- <<: *if-qa-test-tag
when: never
- <<: *if-protected-no_label
- <<: *if-master-push
.rules:tag:release:
rules:
- <<: *if-tag-release
- <<: *if-release-tag
.rules:dev-push:
rules:
- <<: *if-dev-push
# Do not upload caches on dev branches by default
.rules:upload-python-cache:
rules:
- <<: *if-tag-release
- <<: *if-schedule-nightly
- <<: *if-protected-ref-push
changes: *patterns-python-cache
- <<: *if-label-upload_cache
when: manual
.rules:upload-submodule-cache:
rules:
# Needn't upload submodule cache in schedule pipeline
- <<: *if-tag-release
- <<: *if-protected-ref-push
- <<: *if-release-tag
- <<: *if-protected-branch-push
changes: *patterns-submodule
- <<: *if-label-upload_cache
when: manual
@@ -247,16 +217,10 @@
### Patterns ###
.rules:patterns:clang_tidy:
rules:
- <<: *if-protected
- <<: *if-protected-check
- <<: *if-dev-push
changes: *patterns-c-files
.rules:patterns:python-files:
rules:
- <<: *if-protected
- <<: *if-dev-push
changes: *patterns-python-files
#.rules:patterns:static-code-analysis-preview:
# rules:
# - <<: *if-dev-push
@@ -266,12 +230,6 @@
# - <<: *if-dev-push
# changes: *patterns-sonarqube-files
.rules:patterns:idf-pytest-plugin:
rules:
- <<: *if-protected
- <<: *if-dev-push
changes: *patterns-idf-pytest-plugin
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# DO NOT place comments or maintain any code from this line
#
@@ -312,7 +270,7 @@
rules:
- <<: *if-revert-branch
when: never
- <<: *if-protected
- <<: *if-protected-check
- <<: *if-label-build
- <<: *if-dev-push
changes: *patterns-build_components
@@ -325,7 +283,7 @@
rules:
- <<: *if-revert-branch
when: never
- <<: *if-protected
- <<: *if-protected-check
- <<: *if-label-build
- <<: *if-dev-push
changes: *patterns-build_check
@@ -340,7 +298,7 @@
rules:
- <<: *if-revert-branch
when: never
- <<: *if-protected
- <<: *if-protected-check
- <<: *if-label-build
- <<: *if-label-docker
- <<: *if-dev-push
@@ -356,7 +314,7 @@
rules:
- <<: *if-revert-branch
when: never
- <<: *if-protected
- <<: *if-protected-check
- <<: *if-label-build
- <<: *if-label-macos
- <<: *if-label-macos_test
@@ -367,21 +325,6 @@
- <<: *if-dev-push
changes: *patterns-downloadable-tools
.rules:build:target_test:
rules:
- <<: *if-revert-branch
when: never
- <<: *if-protected
- <<: *if-label-build
- <<: *if-dev-push
changes: *patterns-build_components
- <<: *if-dev-push
changes: *patterns-build_system
- <<: *if-dev-push
changes: *patterns-build_template-app
- <<: *if-dev-push
changes: *patterns-downloadable-tools
.rules:labels:nvs_coverage:
rules:
- <<: *if-revert-branch
@@ -399,7 +342,7 @@
rules:
- <<: *if-revert-branch
when: never
- <<: *if-protected
- <<: *if-protected-check
- <<: *if-label-build-only
when: never
- <<: *if-label-host_test
@@ -410,7 +353,7 @@
rules:
- <<: *if-revert-branch
when: never
- <<: *if-protected
- <<: *if-protected-check
- <<: *if-label-build-only
when: never
- <<: *if-label-submodule
.gitlab/ci/static-code-analysis.yml
+11 -100
View File
@@ -16,106 +16,17 @@ clang_tidy_check:
--limit-file tools/ci/static-analysis-rules.yml
--xtensa-include-dir
check_pylint:
gcc_static_analyzer:
extends:
- .pre_check_template
- .rules:patterns:python-files
needs:
- pipeline_variables
artifacts:
reports:
codequality: pylint.json
paths:
- pylint.json
expire_in: 1 week
when: always
- .rules:patterns:clang_tidy
variables:
CI_CCACHE_DISABLE: 1
ANALYZING_APP: "examples/get-started/hello_world"
script:
- |
if [ -n "$CI_MERGE_REQUEST_IID" ]; then
export files=$(echo "$GIT_DIFF_OUTPUT" | grep ".py$" | xargs);
else
export files=$(git ls-files "*.py" | xargs);
fi
- if [ -z "$files" ]; then echo "No python files found"; exit 0; fi
- run_cmd pylint --exit-zero --load-plugins=pylint_gitlab --output-format=gitlab-codeclimate:pylint.json $files
#
## build stage
## Sonarqube related jobs put here for this reason:
## Here we have two jobs. code_quality_check and code_quality_report.
##
## code_quality_check will analyze the code changes between your MR and
## code repo stored in sonarqube server. The analysis result is only shown in
## the comments under this MR and won't be transferred to the server.
##
## code_quality_report will analyze and transfer both of the newly added code
## and the analysis result to the server.
##
## Put in the front to ensure that the newly merged code can be stored in
## sonarqube server ASAP, in order to avoid reporting unrelated code issues
#.sonar_scan_template:
# stage: build
# extends: .pre_check_template
# # full clone since this image does not support fetch --shallow-since-cutoff
# # shiny runners are used for full clone
# tags: [build, shiny]
# image: $SONARQUBE_SCANNER_IMAGE
# before_script:
# - source tools/ci/utils.sh
# - export PYTHONPATH="$CI_PROJECT_DIR/tools:$CI_PROJECT_DIR/tools/ci/python_packages:$PYTHONPATH"
# - fetch_submodules
# # Exclude the submodules, all paths ends with /**
# - submodules=$(get_all_submodules)
# # get all exclude paths specified in tools/ci/sonar_exclude_list.txt | ignore lines start with # | xargs | replace all <space> to <comma>
# - custom_excludes=$(cat $CI_PROJECT_DIR/tools/ci/sonar_exclude_list.txt | grep -v '^#' | xargs | sed -e 's/ /,/g')
# # Exclude the report dir as well
# - export EXCLUSIONS="$custom_excludes,$submodules"
# - export SONAR_SCANNER_OPTS="-Xmx2048m"
# variables:
# GIT_DEPTH: 0
# REPORT_PATTERN: clang_tidy_reports/**/*.txt
# artifacts:
# paths:
# - $REPORT_PATTERN
# expire_in: 1 week
# when: always
# dependencies: # Here is not a hard dependency relationship, could be skipped when only python files changed. so we do not use "needs" here.
# - clang_tidy_check
#
#code_quality_check:
# extends:
# - .sonar_scan_template
# - .rules:patterns:static-code-analysis-preview
# allow_failure: true # it's using exit code to indicate the code analysis result,
# # we don't want to block ci when critical issues founded
# script:
# - export CI_MERGE_REQUEST_COMMITS=$(python ${CI_PROJECT_DIR}/tools/ci/ci_get_mr_info.py commits --src-branch ${CI_COMMIT_REF_NAME} | tr '\n' ',')
# # test if this branch have merge request, if not, exit 0
# - test -n "$CI_MERGE_REQUEST_IID" || exit 0
# - test -n "$CI_MERGE_REQUEST_COMMITS" || exit 0
# - sonar-scanner
# -Dsonar.analysis.mode=preview
# -Dsonar.branch.name=$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
# -Dsonar.cxx.clangtidy.reportPath=$REPORT_PATTERN
# -Dsonar.exclusions=$EXCLUSIONS
# -Dsonar.gitlab.ci_merge_request_iid=$CI_MERGE_REQUEST_IID
# -Dsonar.gitlab.commit_sha=$CI_MERGE_REQUEST_COMMITS
# -Dsonar.gitlab.merge_request_discussion=true
# -Dsonar.gitlab.ref_name=$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
# -Dsonar.host.url=$SONAR_HOST_URL
# -Dsonar.login=$SONAR_LOGIN
#
#code_quality_report:
# extends:
# - .sonar_scan_template
# - .rules:protected
# allow_failure: true # it's using exit code to indicate the code analysis result,
# # we don't want to block ci when critical issues founded
# script:
# - sonar-scanner
# -Dsonar.branch.name=$CI_COMMIT_REF_NAME
# -Dsonar.cxx.clangtidy.reportPath=$REPORT_PATTERN
# -Dsonar.exclusions=$EXCLUSIONS
# -Dsonar.gitlab.commit_sha=$PIPELINE_COMMIT_SHA
# -Dsonar.gitlab.ref_name=$CI_COMMIT_REF_NAME
# -Dsonar.host.url=$SONAR_HOST_URL
# -Dsonar.login=$SONAR_LOGIN
- echo "CONFIG_COMPILER_STATIC_ANALYZER=y" >> ${ANALYZING_APP}/sdkconfig.defaults
- sed -i 's/.*MINIMAL_BUILD.*//g' ${ANALYZING_APP}/CMakeLists.txt
# no -Werror
- export EXTRA_CFLAGS=
- export EXTRA_CXXFLAGS=
- idf-build-apps build -p ${ANALYZING_APP} --no-check-warnings
.gitlab/ci/test-win.yml
+29 -74
View File
@@ -1,21 +1,16 @@
# Host tests
.host_test_win_template:
extends: .rules:test:host_test
extends:
- .windows-settings
- .rules:test:host_test
stage: host_test
image: $ESP_ENV_IMAGE
tags:
- windows-build
dependencies: # set dependencies to null to avoid missing artifacts issue
# run host_test jobs immediately, only after upload cache
needs:
- job: upload-pip-cache
optional: true
artifacts: false
- job: upload-submodules-cache
optional: true
artifacts: false
before_script: []
after_script: []
- job: manual_gate
optional: true
test_cli_installer_win:
rules:
@@ -30,14 +25,11 @@ test_cli_installer_win:
- tools/tools.new.json
- tools/test_idf_tools/test_python_env_logs.txt
expire_in: 1 week
variables:
IDF_PATH: "$CI_PROJECT_DIR"
script:
# Tools must be downloaded for testing
# cmake@version that is supported
- python ${IDF_PATH}\tools\idf_tools.py download required qemu-riscv32 qemu-xtensa cmake cmake@3.16.3
- cd ${IDF_PATH}\tools\test_idf_tools
- python -m pip install jsonschema
- cd tools\test_idf_tools
- python .\test_idf_tools.py
- python .\test_idf_tools_python_env.py
@@ -48,84 +40,48 @@ test_tools_win:
parallel: 4
artifacts:
paths:
- ${IDF_PATH}/*.out
- ${IDF_PATH}/XUNIT_*.xml
- "*.out"
- "XUNIT_*.xml"
reports:
junit: ${IDF_PATH}/XUNIT_*.xml
junit: "XUNIT_*.xml"
expire_in: 1 week
when: always
variables:
LC_ALL: C.UTF-8
PYTHONPATH: "$PYTHONPATH;$IDF_PATH\\tools;$IDF_PATH\\tools\\esp_app_trace;$IDF_PATH\\components\\partition_table;$IDF_PATH\\tools\\ci\\python_packages"
script:
- python -m pip install jsonschema
- .\install.ps1 --enable-ci --enable-pytest
- .\export.ps1
- python "${SUBMODULE_FETCH_TOOL}" -s "all"
- cd ${IDF_PATH}/tools/test_idf_py
- pytest --noconftest test_idf_py.py --junitxml=${IDF_PATH}/XUNIT_IDF_PY.xml
- pytest --noconftest test_hints.py --junitxml=${IDF_PATH}/XUNIT_HINTS.xml
- cd tools\test_idf_py
- idf-ci gitlab download-known-failure-cases-file ${KNOWN_FAILURE_CASES_FILE_NAME}
- pytest --parallel-count ${CI_NODE_TOTAL} --parallel-index ${CI_NODE_INDEX} --junitxml ${CI_PROJECT_DIR}\XUNIT_RESULT.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
# Build tests
.test_build_system_template_win:
stage: host_test
variables:
# Enable ccache for all build jobs. See configure_ci_environment.sh for more ccache related settings.
IDF_CCACHE_ENABLE: "1"
PYTHONPATH: "$PYTHONPATH;$IDF_PATH\\tools;$IDF_PATH\\tools\\esp_app_trace;$IDF_PATH\\components\\partition_table;$IDF_PATH\\tools\\ci\\python_packages"
before_script: []
after_script: []
extends: .host_test_win_template
timeout: 4 hours
artifacts:
paths:
- XUNIT_RESULT.xml
- test_build_system
expire_in: 2 days
when: always
reports:
junit: XUNIT_RESULT.xml
script:
- .\install.ps1 --enable-ci --enable-pytest
- . .\export.ps1
- python "${SUBMODULE_FETCH_TOOL}" -s "all"
- cd ${IDF_PATH}\tools\test_build_system
- python ${IDF_PATH}\tools\ci\get_known_failure_cases_file.py
- pytest --parallel-count ${CI_NODE_TOTAL} --parallel-index ${CI_NODE_INDEX} --junitxml=${CI_PROJECT_DIR}\XUNIT_RESULT.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
- cd tools\test_build_system
- idf-ci gitlab download-known-failure-cases-file ${KNOWN_FAILURE_CASES_FILE_NAME}
- pytest --parallel-count ${CI_NODE_TOTAL} --parallel-index ${CI_NODE_INDEX} --junitxml ${CI_PROJECT_DIR}\XUNIT_RESULT.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
pytest_build_system_win:
extends:
- .test_build_system_template_win
- .rules:labels:windows_pytest_build_system
parallel: 2
tags:
- windows-build
needs:
- job: manual_gate
optional: true
artifacts:
paths:
- XUNIT_RESULT.xml
- test_build_system
expire_in: 2 days
reports:
junit: XUNIT_RESULT.xml
when: always
parallel: 6
pytest_build_system_win_minimal_cmake:
extends:
- .test_build_system_template_win
- .rules:labels:windows_pytest_build_system
needs:
- job: manual_gate
optional: true
tags:
- windows-build
artifacts:
paths:
- XUNIT_RESULT.xml
- test_build_system
expire_in: 2 days
reports:
junit: XUNIT_RESULT.xml
when: always
variables:
MINIMAL_CMAKE_VERSION: "3.16.3"
script:
- .\install.ps1 --enable-ci --enable-pytest
- . .\export.ps1
- python ${IDF_PATH}\tools\idf_tools.py install cmake@${MINIMAL_CMAKE_VERSION}
- python tools\idf_tools.py install cmake@${MINIMAL_CMAKE_VERSION}
- $Env:PATH = "$Env:USERPROFILE\.espressif\tools\cmake\${MINIMAL_CMAKE_VERSION}\bin;$Env:PATH"
- |
$actualVersion = (& cmake --version).Split()[2]
@@ -133,7 +89,6 @@ pytest_build_system_win_minimal_cmake:
Write-Error "ERROR: Wrong CMake version! Detected: $actualVersion, but expected: $Env:MINIMAL_CMAKE_VERSION"
exit 1
}
- python "${SUBMODULE_FETCH_TOOL}" -s "all"
- cd ${IDF_PATH}\tools\test_build_system
- python ${IDF_PATH}\tools\ci\get_known_failure_cases_file.py
- pytest -k cmake --junitxml=${CI_PROJECT_DIR}\XUNIT_RESULT.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
- cd tools\test_build_system
- idf-ci gitlab download-known-failure-cases-file ${KNOWN_FAILURE_CASES_FILE_NAME}
- pytest -k cmake --junitxml ${CI_PROJECT_DIR}\XUNIT_RESULT.xml --ignore-result-files ${KNOWN_FAILURE_CASES_FILE_NAME}
.gitlab/ci/upload_cache.yml
-20
View File
@@ -6,26 +6,6 @@
stage: upload_cache
image: $ESP_ENV_IMAGE
upload-pip-cache:
extends:
- .upload_cache_template
- .before_script:minimal
- .rules:upload-python-cache
tags:
- $GEO
- cache
cache:
key: pip-cache-${LATEST_GIT_TAG}
paths:
- .cache/pip
policy: push
script:
- rm -rf .cache/pip # clear old packages
- bash install.sh --enable-ci --enable-pytest
parallel:
matrix:
- GEO: [ 'shiny', 'brew' ]
upload-submodules-cache:
extends:
- .upload_cache_template