Unverified Commit b304b4bd authored by Massimiliano Culpo's avatar Massimiliano Culpo Committed by GitHub
Browse files

Speed-up CI by reorganizing tests (#22247)

* unit tests: mark slow tests as "maybeslow"

This commit also removes the "network" marker and
marks every "network" test as "maybeslow". Tests
marked as db are maintained, but they're not slow
anymore.

* GA: require style tests to pass before running unit-tests

* GA: make MacOS unit tests fail fast

* GA: move all unit tests into the same workflow, run style tests as a prerequisite

All the unit tests have been moved into the same workflow so that a single
run of the dorny/paths-filter action can be used to ask for coverage based
on the files that have been changed in a PR. The basic idea is that for PRs
that introduce only changes to packages coverage is not necessary, this
resulting in a faster execution of the tests.

Also, for package only PRs slow unit tests are skipped.

Finally, MacOS and linux unit tests are now conditional on style tests passing
meaning that e.g. we won't waste a MacOS worker if we know that the PR has
flake8 issues.

* Addressed review comments

* Skipping slow tests on MacOS for package only recipes

* QA: make tests on changes correct before merging
parent d36de79b
name: macos tests
on:
push:
branches:
- develop
- releases/**
pull_request:
branches:
- develop
- releases/**
jobs:
build:
runs-on: macos-latest
strategy:
matrix:
python-version: [3.8]
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install Python packages
run: |
pip install --upgrade pip six setuptools
pip install --upgrade codecov coverage
pip install --upgrade flake8 pep8-naming mypy
- name: Setup Homebrew packages
run: |
brew install dash fish gcc gnupg2 kcov
- name: Run unit tests
run: |
git --version
. .github/workflows/setup_git.sh
. share/spack/setup-env.sh
coverage run $(which spack) unit-test
coverage combine
coverage xml
- uses: codecov/codecov-action@v1
with:
file: ./coverage.xml
flags: unittests,macos
name: style and docs
on:
push:
branches:
- develop
- releases/**
pull_request:
branches:
- develop
- releases/**
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Install Python Packages
run: |
pip install --upgrade pip
pip install --upgrade vermin
- name: Minimum Version (Spack's Core)
run: vermin --backport argparse --backport typing -t=2.6- -t=3.5- -v lib/spack/spack/ lib/spack/llnl/ bin/
- name: Minimum Version (Repositories)
run: vermin --backport argparse --backport typing -t=2.6- -t=3.5- -v var/spack/repos
style:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Install Python packages
run: |
pip install --upgrade pip six setuptools flake8 mypy>=0.800 black
- name: Setup git configuration
run: |
# Need this for the git tests to succeed.
git --version
. .github/workflows/setup_git.sh
- name: Run style tests
run: |
share/spack/qa/run-style-tests
documentation:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Install System packages
run: |
sudo apt-get -y update
sudo apt-get install -y coreutils ninja-build graphviz
- name: Install Python packages
run: |
pip install --upgrade pip six setuptools
pip install --upgrade -r lib/spack/docs/requirements.txt
- name: Build documentation
run: |
share/spack/qa/run-doc-tests
...@@ -10,13 +10,112 @@ on: ...@@ -10,13 +10,112 @@ on:
- develop - develop
- releases/** - releases/**
jobs: jobs:
# Validate that the code can be run on all the Python versions
# supported by Spack
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Install Python Packages
run: |
pip install --upgrade pip
pip install --upgrade vermin
- name: vermin (Spack's Core)
run: vermin --backport argparse --backport typing -t=2.6- -t=3.5- -v lib/spack/spack/ lib/spack/llnl/ bin/
- name: vermin (Repositories)
run: vermin --backport argparse --backport typing -t=2.6- -t=3.5- -v var/spack/repos
# Run style checks on the files that have been changed
style:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Install Python packages
run: |
pip install --upgrade pip six setuptools flake8 mypy>=0.800 black
- name: Setup git configuration
run: |
# Need this for the git tests to succeed.
git --version
. .github/workflows/setup_git.sh
- name: Run style tests
run: |
share/spack/qa/run-style-tests
# Build the documentation
documentation:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Install System packages
run: |
sudo apt-get -y update
sudo apt-get install -y coreutils ninja-build graphviz
- name: Install Python packages
run: |
pip install --upgrade pip six setuptools
pip install --upgrade -r lib/spack/docs/requirements.txt
- name: Build documentation
run: |
share/spack/qa/run-doc-tests
# Check which files have been updated by the PR
changes:
runs-on: ubuntu-latest
# Set job outputs to values from filter step
outputs:
core: ${{ steps.filter.outputs.core }}
packages: ${{ steps.filter.outputs.packages }}
with_coverage: ${{ steps.coverage.outputs.with_coverage }}
steps:
# For pull requests it's not necessary to checkout the code
- uses: dorny/paths-filter@v2
id: filter
with:
# See https://github.com/dorny/paths-filter/issues/56 for the syntax used below
filters: |
core:
- './!(var/**)/**'
packages:
- 'var/**'
# Some links for easier reference:
#
# "github" context: https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context
# job outputs: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idoutputs
# setting environment variables from earlier steps: https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#setting-an-environment-variable
#
- id: coverage
# Run the subsequent jobs with coverage if this is a PR and core has been modified
# or if this workflow is triggered by a push event (this means that once a PR is
# merged we'll perform a full run with CI on develop even though the PR was only
# modifying packages)
run: |
echo Core changes: ${{ steps.filter.outputs.core }}
echo Event name: ${{ github.event_name }}
if [ "${{ steps.filter.outputs.core }}" == "true" ] || [ "${{ github.event_name }}" == 'push' ]
then
echo "::set-output name=with_coverage::true"
else
echo "::set-output name=with_coverage::false"
fi
# Run unit tests with different configurations on linux
unittests: unittests:
needs: [ validate, style, documentation, changes ]
runs-on: ubuntu-latest runs-on: ubuntu-latest
strategy: strategy:
matrix: matrix:
python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9] python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9]
concretizer: ['original', 'clingo'] concretizer: ['original', 'clingo']
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v2
with: with:
...@@ -42,6 +141,7 @@ jobs: ...@@ -42,6 +141,7 @@ jobs:
git --version git --version
. .github/workflows/setup_git.sh . .github/workflows/setup_git.sh
- name: Install kcov for bash script coverage - name: Install kcov for bash script coverage
if: ${{ needs.changes.outputs.with_coverage == 'true' }}
env: env:
KCOV_VERSION: 34 KCOV_VERSION: 34
run: | run: |
...@@ -57,7 +157,8 @@ jobs: ...@@ -57,7 +157,8 @@ jobs:
. share/spack/setup-env.sh . share/spack/setup-env.sh
spack external find --not-buildable cmake bison spack external find --not-buildable cmake bison
spack -v solve zlib spack -v solve zlib
- name: Run unit tests - name: Run unit tests (full suite with coverage)
if: ${{ needs.changes.outputs.with_coverage == 'true' }}
env: env:
COVERAGE: true COVERAGE: true
SPACK_TEST_SOLVER: ${{ matrix.concretizer }} SPACK_TEST_SOLVER: ${{ matrix.concretizer }}
...@@ -65,10 +166,20 @@ jobs: ...@@ -65,10 +166,20 @@ jobs:
share/spack/qa/run-unit-tests share/spack/qa/run-unit-tests
coverage combine coverage combine
coverage xml coverage xml
- name: Run unit tests (reduced suite without coverage)
if: ${{ needs.changes.outputs.with_coverage == 'false' }}
env:
ONLY_PACKAGES: true
SPACK_TEST_SOLVER: ${{ matrix.concretizer }}
run: |
share/spack/qa/run-unit-tests
- uses: codecov/codecov-action@v1 - uses: codecov/codecov-action@v1
if: ${{ needs.changes.outputs.with_coverage == 'true' }}
with: with:
flags: unittests,linux,${{ matrix.concretizer }} flags: unittests,linux,${{ matrix.concretizer }}
# Test shell integration
shell: shell:
needs: [ validate, style, documentation, changes ]
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v2
...@@ -94,6 +205,7 @@ jobs: ...@@ -94,6 +205,7 @@ jobs:
git --version git --version
. .github/workflows/setup_git.sh . .github/workflows/setup_git.sh
- name: Install kcov for bash script coverage - name: Install kcov for bash script coverage
if: ${{ needs.changes.outputs.with_coverage == 'true' }}
env: env:
KCOV_VERSION: 38 KCOV_VERSION: 38
run: | run: |
...@@ -103,17 +215,23 @@ jobs: ...@@ -103,17 +215,23 @@ jobs:
mkdir -p ${KCOV_ROOT}/build mkdir -p ${KCOV_ROOT}/build
cd ${KCOV_ROOT}/build && cmake -Wno-dev ${KCOV_ROOT}/kcov-${KCOV_VERSION} && cd - cd ${KCOV_ROOT}/build && cmake -Wno-dev ${KCOV_ROOT}/kcov-${KCOV_VERSION} && cd -
make -C ${KCOV_ROOT}/build && sudo make -C ${KCOV_ROOT}/build install make -C ${KCOV_ROOT}/build && sudo make -C ${KCOV_ROOT}/build install
- name: Run shell tests - name: Run shell tests (without coverage)
if: ${{ needs.changes.outputs.with_coverage == 'false' }}
run: |
share/spack/qa/run-shell-tests
- name: Run shell tests (with coverage)
if: ${{ needs.changes.outputs.with_coverage == 'true' }}
env: env:
COVERAGE: true COVERAGE: true
run: | run: |
share/spack/qa/run-shell-tests share/spack/qa/run-shell-tests
- uses: codecov/codecov-action@v1 - uses: codecov/codecov-action@v1
if: ${{ needs.changes.outputs.with_coverage == 'true' }}
with: with:
flags: shelltests,linux flags: shelltests,linux
centos6:
# Test for Python2.6 run on Centos 6 # Test for Python2.6 run on Centos 6
centos6:
needs: [ validate, style, documentation ]
runs-on: ubuntu-latest runs-on: ubuntu-latest
container: spack/github-actions:centos6 container: spack/github-actions:centos6
steps: steps:
...@@ -127,7 +245,9 @@ jobs: ...@@ -127,7 +245,9 @@ jobs:
git checkout test-branch git checkout test-branch
share/spack/qa/run-unit-tests share/spack/qa/run-unit-tests
# Test RHEL8 UBI with platform Python
rhel8-platform-python: rhel8-platform-python:
needs: [ validate, style, documentation ]
runs-on: ubuntu-latest runs-on: ubuntu-latest
container: registry.access.redhat.com/ubi8/ubi container: registry.access.redhat.com/ubi8/ubi
steps: steps:
...@@ -151,8 +271,9 @@ jobs: ...@@ -151,8 +271,9 @@ jobs:
run: | run: |
source share/spack/setup-env.sh source share/spack/setup-env.sh
spack unit-test -k 'not svn and not hg' -x --verbose spack unit-test -k 'not svn and not hg' -x --verbose
clingo-cffi:
# Test for the clingo based solver (using clingo-cffi) # Test for the clingo based solver (using clingo-cffi)
clingo-cffi:
needs: [ validate, style, documentation, changes ]
runs-on: ubuntu-latest runs-on: ubuntu-latest
container: spack/github-actions:clingo-cffi container: spack/github-actions:clingo-cffi
steps: steps:
...@@ -170,5 +291,47 @@ jobs: ...@@ -170,5 +291,47 @@ jobs:
coverage combine coverage combine
coverage xml coverage xml
- uses: codecov/codecov-action@v1 - uses: codecov/codecov-action@v1
if: ${{ needs.changes.outputs.with_coverage == 'true' }}
with: with:
flags: unittests,linux,clingo flags: unittests,linux,clingo
# Run unit tests on MacOS
build:
needs: [ validate, style, documentation, changes ]
runs-on: macos-latest
strategy:
matrix:
python-version: [3.8]
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install Python packages
run: |
pip install --upgrade pip six setuptools
pip install --upgrade codecov coverage
pip install --upgrade flake8 pep8-naming mypy
- name: Setup Homebrew packages
run: |
brew install dash fish gcc gnupg2 kcov
- name: Run unit tests
run: |
git --version
. .github/workflows/setup_git.sh
. share/spack/setup-env.sh
if [ "${{ needs.changes.outputs.with_coverage }}" == "true" ]
then
coverage run $(which spack) unit-test -x
coverage combine
coverage xml
else
echo "ONLY PACKAGE RECIPES CHANGED [skipping coverage]"
$(which spack) unit-test -x -m "not maybeslow"
fi
- uses: codecov/codecov-action@v1
if: ${{ needs.changes.outputs.with_coverage == 'true' }}
with:
file: ./coverage.xml
flags: unittests,macos
# <img src="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo.svg" width="64" valign="middle" alt="Spack"/> Spack # <img src="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo.svg" width="64" valign="middle" alt="Spack"/> Spack
[![MacOS Tests](https://github.com/spack/spack/workflows/macos%20tests/badge.svg)](https://github.com/spack/spack/actions) [![Unit Tests](https://github.com/spack/spack/workflows/linux%20tests/badge.svg)](https://github.com/spack/spack/actions)
[![Linux Tests](https://github.com/spack/spack/workflows/linux%20tests/badge.svg)](https://github.com/spack/spack/actions)
[![Linux Builds](https://github.com/spack/spack/workflows/linux%20builds/badge.svg)](https://github.com/spack/spack/actions) [![Linux Builds](https://github.com/spack/spack/workflows/linux%20builds/badge.svg)](https://github.com/spack/spack/actions)
[![macOS Builds (nightly)](https://github.com/spack/spack/workflows/macOS%20builds%20nightly/badge.svg?branch=develop)](https://github.com/spack/spack/actions?query=workflow%3A%22macOS+builds+nightly%22) [![macOS Builds (nightly)](https://github.com/spack/spack/workflows/macOS%20builds%20nightly/badge.svg?branch=develop)](https://github.com/spack/spack/actions?query=workflow%3A%22macOS+builds+nightly%22)
[![codecov](https://codecov.io/gh/spack/spack/branch/develop/graph/badge.svg)](https://codecov.io/gh/spack/spack) [![codecov](https://codecov.io/gh/spack/spack/branch/develop/graph/badge.svg)](https://codecov.io/gh/spack/spack)
......
...@@ -114,6 +114,7 @@ def test_get_concrete_specs(config, mock_packages): ...@@ -114,6 +114,7 @@ def test_get_concrete_specs(config, mock_packages):
assert('archive-files' in spec_map) assert('archive-files' in spec_map)
@pytest.mark.maybeslow
def test_register_cdash_build(): def test_register_cdash_build():
build_name = 'Some pkg' build_name = 'Some pkg'
base_url = 'http://cdash.fake.org' base_url = 'http://cdash.fake.org'
......
...@@ -38,6 +38,9 @@ ...@@ -38,6 +38,9 @@
git = exe.which('git', required=True) git = exe.which('git', required=True)
pytestmark = pytest.mark.maybeslow
@pytest.fixture() @pytest.fixture()
def env_deactivate(): def env_deactivate():
yield yield
......
...@@ -25,8 +25,10 @@ ...@@ -25,8 +25,10 @@
# everything here uses the mock_env_path # everything here uses the mock_env_path
pytestmark = pytest.mark.usefixtures( pytestmark = [
'mutable_mock_env_path', 'config', 'mutable_mock_repo') pytest.mark.usefixtures('mutable_mock_env_path', 'config', 'mutable_mock_repo'),
pytest.mark.maybeslow
]
env = SpackCommand('env') env = SpackCommand('env')
install = SpackCommand('install') install = SpackCommand('install')
......
...@@ -71,7 +71,7 @@ def test_url_with_no_version_fails(): ...@@ -71,7 +71,7 @@ def test_url_with_no_version_fails():
url('parse', 'http://www.netlib.org/voronoi/triangle.zip') url('parse', 'http://www.netlib.org/voronoi/triangle.zip')
@pytest.mark.network @pytest.mark.maybeslow
@pytest.mark.skipif( @pytest.mark.skipif(
sys.version_info < (2, 7), sys.version_info < (2, 7),
reason="Python 2.6 tests are run in a container, where " reason="Python 2.6 tests are run in a container, where "
...@@ -106,7 +106,7 @@ def test_url_list(): ...@@ -106,7 +106,7 @@ def test_url_list():
assert 0 < correct_version_urls < total_urls assert 0 < correct_version_urls < total_urls
@pytest.mark.network @pytest.mark.maybeslow
@pytest.mark.skipif( @pytest.mark.skipif(
sys.version_info < (2, 7), sys.version_info < (2, 7),
reason="Python 2.6 tests are run in a container, where " reason="Python 2.6 tests are run in a container, where "
......
...@@ -14,7 +14,6 @@ def test_safe_only_versions(): ...@@ -14,7 +14,6 @@ def test_safe_only_versions():
"""Only test the safe versions of a package. """Only test the safe versions of a package.
(Using the deprecated command line argument) (Using the deprecated command line argument)
""" """
versions('--safe-only', 'zlib') versions('--safe-only', 'zlib')
...@@ -24,21 +23,21 @@ def test_safe_versions(): ...@@ -24,21 +23,21 @@ def test_safe_versions():
versions('--safe', 'zlib') versions('--safe', 'zlib')
@pytest.mark.network @pytest.mark.maybeslow
def test_remote_versions(): def test_remote_versions():
"""Test a package for which remote versions should be available.""" """Test a package for which remote versions should be available."""
versions('zlib') versions('zlib')
@pytest.mark.network @pytest.mark.maybeslow
def test_remote_versions_only(): def test_remote_versions_only():
"""Test a package for which remote versions should be available.""" """Test a package for which remote versions should be available."""
versions('--remote', 'zlib') versions('--remote', 'zlib')
@pytest.mark.network @pytest.mark.maybeslow
@pytest.mark.usefixtures('mock_packages') @pytest.mark.usefixtures('mock_packages')
def test_new_versions_only(): def test_new_versions_only():
"""Test a package for which new versions should be available.""" """Test a package for which new versions should be available."""
...@@ -46,28 +45,28 @@ def test_new_versions_only(): ...@@ -46,28 +45,28 @@ def test_new_versions_only():
versions('--new', 'brillig') versions('--new', 'brillig')
@pytest.mark.network @pytest.mark.maybeslow
def test_no_versions(): def test_no_versions():
"""Test a package for which no remote versions are available.""" """Test a package for which no remote versions are available."""
versions('converge') versions('converge')
@pytest.mark.network @pytest.mark.maybeslow
def test_no_unchecksummed_versions(): def test_no_unchecksummed_versions():
"""Test a package for which no unchecksummed versions are available.""" """Test a package for which no unchecksummed versions are available."""
versions('bzip2') versions('bzip2')
@pytest.mark.network @pytest.mark.maybeslow
def test_versions_no_url(): def test_versions_no_url():
"""Test a package with versions but without a ``url`` attribute.""" """Test a package with versions but without a ``url`` attribute."""
versions('graphviz') versions('graphviz')
@pytest.mark.network @pytest.mark.maybeslow
def test_no_versions_no_url(): def test_no_versions_no_url():
"""Test a package without versions or a ``url`` attribute.""" """Test a package without versions or a ``url`` attribute."""
......
...@@ -65,6 +65,7 @@ def test_parse_gpg_output_case_three(): ...@@ -65,6 +65,7 @@ def test_parse_gpg_output_case_three():
@pytest.mark.skipif(not spack.util.gpg.GpgConstants.user_run_dir, @pytest.mark.skipif(not spack.util.gpg.GpgConstants.user_run_dir,