* we come back to idea of having one CI workflow
* cancel and openapi are incorporated into that CI workflow
* cancel retrieves workflow id automatically (works for forks)
* static checks are now merged into one job
* less dependencies between jobs so that waiting is minimised
* better name for check if tests should be run
* separated out script for tests should be run check
* add git sync sidecars
* add a helm test
* add more tests
* allow users to provide git username and pass via a k8s secrets
* set default values for airflow worker repository & tag
* change ci timeout
* fix link
* add credentials_secret to airflow.cfg configmap
* set GIT_SYNC_ADD_USER on kubernetes worker pods, set uid
* add fsGroup to webserver and kubernete workers
* move gitSync to dags.gitSync
* rename valueFields
* turn off git sync and dag persistence by default
* provide option to specify known_hosts
* add git-sync details into the chart documentation
* Update .gitignore
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* make git sync max failures configurable
* Apply suggestions from code review
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* add back requirements.lock
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Sometimes tests were not triggered when they should be.
This change will cause the tests to be triggered when anything
changes in "airflow" or "charts" additionally to what we had
before.
The Kubernetes tests are now run using Helm chart
rather than the custom templates we used to have.
The Helm Chart uses locally build production image
so the tests are testing not only Airflow but also
Helm Chart and a Production image - all at the
same time. Later on we will add more tests
covering more functionalities of both Helm Chart
and Production Image. This is the first step to
get all of those bundle together and become
testable.
This change introduces also 'shell' sub-command
for Breeze's kind-cluster command and
EMBEDDED_DAGS build args for production image -
both of them useful to run the Kubernetes tests
more easily - without building two images
and with an easy-to-iterate-over-tests
shell command - which works without any
other development environment.
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Daniel Imberman <daniel@astronomer.io>
Local caching is now default strategy when building
the Production image.
You can still change it to pulled - similar to CI builds
by providing the right build flag and this is what
is used in CI by default. The flags in Breeze are now updated
to be more eplanatory and friendly (build-cache-*) and a flag
for "disabled" cache option is added as well.
Also the Dockerfile and Dockerfile.ci files are not needed
any more in the docker context. They used to be needed when
we built the Kubernetes image in the container, but since
we are now using production image directly - we do not need
them any nmore.
Combining setting the default strategy to local and removing
the Dockerfile from the context has the nice effect that you
can iterate much faster on the Production image without
triggering rebuilds of half of the docker image
as soon as the Dockerfile changes.
For a long time the way how entrypoint worked in ci scripts
was wrong. The way it worked was convoluted and short of black
magic. This did not allow to pass multiple test targets and
required separate execute command scripts in Breeze.
This is all now straightened out and both production and
CI image are always using the right entrypoint by default
and we can simply pass parameters to the image as usual without
escaping strings.
This also allowed to remove some breeze commands and
change names of several flags in Breeze to make them more
meaningful.
Both CI and PROD image have now embedded scripts for log
cleaning.
History of image releases is added for 1.10.10-*
alpha quality images.
* Upload kind logs to Github Artifacts
* get_ci_environment needed to correctly dump kind logs
Without this the "setup cluster" step dumps the logs as
`kind_logs_2020-06-11_default_default.tar.gz`
Since this is now used outside of just building images, I have moved the
function to _initialization.sh
* Github already compresses artifacts for download.
If we upload a .tar.gz, the download is a .zip file containing that
.tar.gz, which isn't ideal
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
There is some DNS problem causing it to be unable to resolve github.com
in the container.
To unblock tests we are disabling the git-sync mode tests for the
moment.
Tests requiring Kubernetes Cluster are now moved out of
the regular CI tests and moved to "kubernetes_tests" folder
so that they can be run entirely on host without having
the CI image built at all. They use production image
to run the tests on KinD cluster and we add tooling
to start/stop/deploy the application to the KinD cluster
automatically - for both CI testing and local development.
This is a pre-requisite to convert the tests to convert the
tests to use the official Helm Chart and Docker images or
Apache Airflow.
It closes#8782
Breeze had --push-images switch to also push images to repo
but it was often needed to build and push images separately.
We have now a possibility to push an already built image with
separate push-image command instead and also you can choose
to push to cache registry in GitHub rather than to DockerHub
with --registry-cache switch.
* Use production image for k8s tests
The CI image has become too large to load into KinD,
it also only really makes sense to use the production image for
integration tests
* nit
Co-authored-by: Daniel Imberman <daniel@astronomer.io>
* Push CI images to Docker packcage cache for v1-10 branches
This is done as a commit to master so that we can keep the two branches
in sync
Co-Authored-By: Ash Berlin-Taylor <ash_github@firemirror.com>
* Run Github Actions against v1-10-stable too
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
All PRs will used cached "latest good" version of the python
base images from our GitHub registry. The python versions in
the Github Registry will only get updated after a master
build (which pulls latest Python image from DockerHub) builds
and passes test correctly.
This is to avoid problems that we had recently with Python
patchlevel releases breaking our Docker builds.
By default github actions checks out only latest commit but in order to
see if there are any changes since the last readme generated
we need to see the whole history so we need to fetch it all.
We also skip generating the new README in case there is only one
commit in the history since the last release. The nature of readme
generation is that the commit with the README itself will never
be in the list of commits for the previous release so there is
always at least one commit more than the one listed in the readme.
Right now requirements will be only checked during the
CI build if the setup.py has changed and if yes, clear instructions
will be given. The diff will still be printed otherwise but
it will not cause the job to fail
The CRON job from previous runs did not have everything working
after the emergency migration to Github Actions.
This change brings back following improvements:
* rebuilding images from the scratch in CRON job
* automatically upgrading all requirements to test if they are new
* pushing production images to github packages as cache
* pushing nightly tag to github
* tests are not executed for doc-only changes
* images will be (once merged) downloaded from GitHub Registry so likely much faster
* we have a "scheduled" nightly build that will build everything from scratch and check if no requirements have been broken
* improved split of static checks between two static check jobs - to utilise parallelism better.
* reorganised some fast jobs (requirements, prod image) that do not depend on tests so that they can run earlier
* shorter names for jobs so that they are nicer to view in the actions view
* matrix definitions of the jobs so that we can manage them better
The periodic labeler had a bug that it was not using pagination.
In effect the labeler was checking only 30 oldest request and
if they all contained proper labels, it was not checking any more
PRs.
Details are available in
https://github.com/paulfantom/periodic-labeler/pull/4
But until it is merged, we switch to our own fixed version.
