Continuous Integration (CircleCI)
What is continuous integration?
Continuous integration (CI) is a software development strategy that increases the speed of development while ensuring the quality of the code that teams deploy. Developers continually commit code in small increments (at least daily, or even several times a day), which is then automatically built and tested before it is merged with the shared repository.
Source: CircleCI documentation
Configuration
Our CircleCI configuration resides in .circleci/config.yml. It contains all the jobs listed below. See Configuring CircleCI for a full reference.
Workflow develop
This workflow gets triggered everytime a commit is pushed to the develop branch or a PR is opened.
pip-install
This job executes pip install -e .[dev-pinned,pinned] and makes use of the CircleCI Dependency Cache.
It passes the virtual environment .venv to the subsequent jobs.
webpack
This job executes npm ci and makes use of the CircleCI Dependency Cache.
After that, it compiles all static files with webpack (npm run prod) and passes the output in
integreat_cms/static/dist to the subsequent jobs.
pylint
This job executes pylint_runner, which checks whether the Linting throws any errors or warnings.
black
This job executes black --check ., which checks whether the code matches the Black code style.
check-migrations
This job checks whether there are any changes to the models that need a database migration
setup-test-reporter
This job sets up the test coverage reporter for CodeClimate.
tests
This job runs the tests in 16 parallel containers. It sets up a temporary postgres database and runs the migrations
before testing. It runs pytest and passes the coverage in the test-results directory to the build artifacts.
upload-test-coverage
This job joins all separate coverage data files generated from the previous steps and uploads it to CodeClimate.
check-translations
This job uses the dev-tool ./tools/check_translations.sh to check whether the translation file is up to date and
does not contain any empty or fuzzy entries.
compile-translations
This job compiles the translation file and passes the resulting django.mo to the packaging job.
bump-dev-version
This job modifies the bumpver config to make sure the changes are only temporary and not committed, then it bumps
the version to the next alpha version which is not yet published on
TestPyPI.
This is only executed on develop (usually after PRs have been merged) or on branches that end with -publish-dev-package.
build-package
This job creates a python package and passes the resulting files in dist to the publish-package job.
See Packaging for more information.
publish-package
This job publishes the built package to TestPyPI via Twine.
This is only executed on develop (usually after PRs have been merged) or on branches that end with -publish-dev-package.
build-documentation
This job checks whether the documentation can be generated without any errors by running
./tools/make_docs.sh.
It passes the html documentation in docs to the deploy-documentation job.
deploy-documentation
This job authenticates as the user DigitalfabrikMember and commits all changes to the
documentation to the branch gh-pages
which is then deployed to https://digitalfabrik.github.io/integreat-cms/ by GitHub.
shellcheck/check
This job makes use of the ShellCheck CircleCI Orb and
executes the pre-defined job shellcheck/check. It is configured to check the directory tools
and to allow external sources because all dev tools source one common function script. Also see Shellcheck.
Workflow main
This workflow gets executed when a commit is pushed to the main branch. Typically, this is a release PR from develop.
pip-install
See pip-install.
bump-version
This job authenticates as the deliverino app and runs bumpver update to bump the version and commit the
changes to the main branch. Additionally, it merges the version bump commit into the develop branch.
Workflow deploy
This workflow gets executed when a commit is tagged.
pip-install
See pip-install.
webpack
See webpack.
compile-translations
See compile-translations.
build-package
See build-package.
publish-package
See publish-package. The only difference is that PyPI is used as repository instead of TestPyPI.
create-release
This job authenticates as Deliverino app and creates a GitHub release with .circleci/scripts/create_release.py.
notify-mattermost
This job sends a release notification to Mattermost into the integreat-releases channel. It needs the Mattermost
webhook which is injected via the mattermost context.
Debugging with SSH
If you encounter any build failures which you cannot reproduce on your local machine, you can SSH into the build server and examine the problem. See Debugging with SSH for more information.