When only task-sdk-integration tests are changed, the tests should
run. Currently they don't run because in this case PROD image has
is not build.
This PR fixes the selective checks.
The script had no good support for auto-complete and the like and
since everyone now use breeze for some tasks, we also should
have it as a regular breeze command.
Also the sorting order implemented by eslint jsonc natural
sorting was quite a bit different from what you'd expect:
* it sorts character-by-character case-insensitive
* it uses original sorting as tie-breker in case of characters
case-insensitve order
* handles numbers natuarlly (2<10)
* treats _ as end of word and sort it before everything else
* similarly with end of word - shorter words are before longer
if they have the same prefix
This PR updates the command and also fixes sorting order and
add unit tests to test vital parts of the translation script.
After adding -source package with release date to providers release,
we also need to update the source download documentation to link to
the right source tarball.
This PR adds the last missing bits and pieces to do that:
1) Updates information about format of RELEASE_DATE and supported
(potentially) YYYY_MM_DD_NN format
2) The release date parameter is used and verified at release
docuemntation preparation time and it's stored in
"providers/.last_release_date.txt"
3) The relase date of the current release (stored in the commit
which is used to release providers) is used to produce the
right links - both in "providers summary" and in individual
provider's package documntation
There were a few of things that made task-sdk-tests iteration
a bit unobvious and slower. With this PR, we should be able
to iterate over task-sdk-integration-tests WAY faster and get
more contributors involved in contributing to those.
* It was not clear that prerequisite of running the tests was building
PROD image for Pyton 3.10. This is now clear in the documentation.
* PROD images can be built in two different modes - from sources
with --installation-method equal to . or from packages with
the --installatio-method equal to "apache-airflow". This was
not clearly communicated during build and it is now printed at
output
* It was not clear that when you build PROD images from sources,
you should first compile ui assets, because otehrwise the
assets are not added as part of the image. With this PR the
`breeze prod-image build` command checks if the .vite manifest
is present in the right `dist` folders and will error out,
suggesting to run `breeze compile-ui-assets` before.
* If the PROD image has not been built before, breeze will propose
to build it and even do it automatically if the answer is not
provided within 20 seconds.
* when building PROD images from sources, it is faster to rebuild
the images with `uv` than with `pip`. the --use-uv parameter now
defaults to False when building from packages and to True when
building from sources.
* There was an error in .dockerignore where generated dist files
were not added to context when PROD image was built from sources.
This resulted in "permission denied' when such PROD images were used
to run tests.
* The test compose had fallback of Airflow 3.0.3 which would be
misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory
* We are now mounting sources of Airflow to inside the image by default
and skip it in CI. This mounting happens in local environment where PROD
image is built usually from sources, and it is disabled in CI by using
--skip-mounting-local-volumes flag. We also do not stop docker compose
by default when runnig it locally in order to make fast iteration the
default.
* We pass host operating system when starting the compose, and we only
change ownership on Linux - this is a long running operation on MacOS
because mounted filesystem is slow, but it's also not needed on MacOS
because the file system also maps ownershipt and files created by
Airflow are created with local user id.
* We pass local user id to containers to make sure that the files
created on linux are created by the local user (logs and the like).
* We are now detecting whether docker-compose is running and when we run
with locally mounted sources, we reuse those running containers. When
we don't mount local sources, we shut-down the compose before running
to make sure we do not have sources mounted - and we close the compose
by default when we do not mount local sources.
* When sources are mounted we are only enabling DEV_MODE inside the containers
so that components are hot-reloading (new feature added in #57741
last weeks. This way you do not have to restart anything when sources
are changed and you can re-run the tests when docker compose is running.
* The environment are passsed now via .env file so that you can easily
reproduce docke compose command locally
* The docker compose files are not copied any more, they are moved directly
to the top of 'task-sdk-integraiton-tests' and used from there.
* A `--down` flag is added to breeze testing task-sdk-integration-tests
that tears down running docker compose.
* Additional diagnostics added to show what's going on.
* Handling verbose option from breeze by adding more debugging information
* Updated documentation about the tests to be more comprehensive about
the options, updated file structure etc.
* Small QOL immprovement - if expected dags are not yet parsed by dag file
processor, when test starts, getting their status will return 404 "Not
Found". In such case our tests implemented a short retry scheme with tenacity
We have some semi-complex set of tools that take care about upgrading
of all our CI infrastructure periodically. While dependabot has a lot of
use, a lot of cases is not handled yet in it - such as upgrading the
charts, important dependencies in our scripts, dockerfiles and so on.
While we already have a set of prek hooks and github Actions that
do a lot there, some of that has to be done periodically outside of
dependabot - this CI wraps a number of those upgrade tools into a
single `ci upgrade` command that can be run locally, generate and
open PR - and can be used in both - main and any of the v*test
branches.
Future improvement areas:
* docker container upgrades for Helm chart and docker compose
* slack notification after such upgrade PR is opened
* more .....
We had pin-versions prek-hook implemented in a separate workflow
under the `dev` folder, but it has not been working since workspace
switch because prek workspace only works on sub-folders of where the
.pre-commit-config.yaml file is placed. It was in a separate file
because it needed python 3.11 to run, but it is possible to have
a specific python verison as separate language version in the hook
itself, so we can move it back to the main .pre-commit-config.yaml
This PR:
* moves the pin-version hook to main .pre-commit-config.yaml
* sets python 3.11 as version of python used in the hook independently
from default python version
* fix github actions and docs to use the hook from the main
.pre-commit-config.yaml
When we are publishing docs and building SBOMs the regular disk
space on / is not enough to hold all the necessary images. We need
to move it to /mnt to make it works - /mnt has way more space.
We now have same workflow running for both ARM and AMD and we need
a bit better diagnostics printed to distinguish those different
run types.
* Tha name of the workflow is just changed to "Tests"
* There is a job added that should immediately show the platform
in the left-sidebar of GitHub Actions
* The title containing platform is printed at the top of summary
- before the constraints summary.
* Make single workflow to run both AMD and ARM builds
* Add condition to exclude mysql tests for arm
* Fix mypy issues
* delete arm and amd workflows
* Fix artifact name
- Update uv from 0.9.4 to 0.9.5
- Update ruff from 0.14.1 to 0.14.2
- Update mypy to 1.18.2
- Update Python to 3.12.12
- Update various other dependencies
As discussed in https://lists.apache.org/thread/cwfmqbzxsm0gobtpo8kmfr99nfv29c2y
we are temporarily (or not) removing MySQL client support from Airflow
container images in order to stop our CI canary builds from failing.
If consensus or vote will be reached to remove it, we will leave it as
is, if we will find other ways how to keep mysql client support, we
will revert that change and restore MySQL client support.
* Implement initial integration test for airflowctl with 3.1
* password can be passed without interaction, update integration tests
* Add AIRFLOW_CLI_DEBUG_MODE for enhanced CLI debugging and update integration tests to skip keyring
* Warn user while running each command if debug mode enabled and explicitly state it shouldn't be used unless debugging or integration tests
* Move python-on-whales to devel-common, use shared docker-composer file, update documentation mistakes
* remove shared python-on-whales from airflow-ctl-tests/
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Decouple docker compose logic from test method to pytest_sessionstart in conftest
* Move python_on_whale import to file level
* Reorder dependencies in pyproject.toml for consistency
* Add workspace to main pyproject.toml, remove unused variable, move console to singleton __init__.py
* Add workspace to main pyproject.toml, remove unused variable, move console to singleton __init__.py
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
GitHub action uses name derived from the composit workflow not
from running workflow, so we must pass the name of tests down
as input parameter to be able to distinguish the two test types
by name.
There were a few issues with ARM workflows:
* not all jobs were run in ARM tests - we do not want to run mysql
of course, but other tests should be fine to run on ARM
* some conditions were not updated (we have to somehow duplicate
amd and arm job definition because we run out of composite
workflows - so sometimes conditions are not synced)
* most importantly - we uploaded prek cache in build-info job, but
that job only run on AMD, not on ARM so the ARM cache was really
an AMD one (and it caused unterminated strings in doctoc installation
It's not possible to upload same artifact twice in the same job and
since we use prek in several jobs we should make sure that the cache
is only uploaded once per job. This was the reason why it was initially
uploaded in build-info job (and save-cache was set to false elsewhere).
With this PR, we have save-cache in 3 places:
* basic checks
* static CI-image bound checks
* in octopin (Python 3.11)
Basic checks and static checks are mutually exclusive (controlled by
basic-checks-only flag) - so we can safely upload cache in both.
In all other places we only install prek with cache, but we do not
save the cache as artifact.
Apparently the prek cache mechanism has been somewhat broken for a
while - after we split prek to monorepo. The hash files used to
determine prek-cache was different for save and restore step
(the `**/` has been missing in the save cache step. Which means
that we always failed to restore cache and created it from the
scratch.
Also, it seems that the prek cache-when prepared refers to the uv
version that is pre-installed for it in case uv is not installed
in the system. And it refers to the uv version when creating the
virtual environments used by prek, and we first attempted to
install prek and create cache, and only after we installed uv, which
had a side-effect that in some cases the installed venvs referred
to a missing python binary.
Finally - there is a bug in prek https://github.com/j178/prek/issues/918
that pygrep cache contains reference to a non-existing python binary
that should be run when pygrep runs.
Also it's possible that some of the cache installed in workspace by the
github worker remained, and we did not preemptively clean the cache when
we attempted to restore it and failed.
This PR attempts to restore the cache usage in a more robust way:
* fixed cache key on save to save cache with proper name
* added uv version to cache key for prek
* always install uv in desired version before installing prek
* if we faile to cache-hit and restore the cache, we clean-up
the .cache/prek folder
* we do not look at skipped hooks when installing prek and restoring
or saving cache. There is very little saving on some hooks and
since we are preparing the cache in "build-info" now - it's better
to always use the same cache, no matter if some checks are skipped
* upgraded to prek 0.2.10 that fixed the issue with pygrep cache
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Update gotestsum to latest version, and enable its github-actions output mode
so we get grouping automatically
And most importantly, enable CodeQL scanning on Go
