* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Update gotestsum to latest version, and enable its github-actions output mode
so we get grouping automatically
And most importantly, enable CodeQL scanning on Go
* Introduce e2e testing with testcontainers
* Fix test command
* Fix test command
* Upload test report
* Add option to trigger with workflow_dispatch
* Add test to trigger example dags
* Upload logs
* Upload logs
* zip logs
* Fix example_bash_decorator file stat function
* Add breeze commands and docs
* Update breeze commands
* Make docker-image-tag to empty and determine in conftest for canary build
* Fix mnt writable
This PR upgrades latest versions of the important dependencies, also
it implements the possibility of using the token that is used
locally via `gh` GitHub CLI - this way if you did `gh auth login`
to use `gh` for your repository, there is no need to create and
pass GITHUB_TOKEN via UI of GitHub.
Also added ruff to the upgraded packages, and synchronized the variables
in upgrade checks to include all the packages we are checking for
We decided to clean-up runner and remove all the potential garbage
we get when we start runner - we also move docker to a separate
/mnt partitiion available on the runners - because for unit testing
especially we need as much space as possible. Also when we build
images, it is best if we use clean, pristine docker engine located
on the /mnt folder.
All of our other jobs are ok with small disk space and sometimes that cleanup
can take up to a minute (usually it is 10-20s)
This change (experimentally) removes all cleanup from all jobs
except the unit tests and image building to see if we can get stability
and speed at the same time.
Previously we were only using prek cache from canary builds, but
since we are starting to use several separate prek runs, it makes
sense to install prek hooks only once and store them in cache and
reuse even in the same build.
This PR removes "only-canary" prek cache preparation - now all
builds including all PRs from fork preapare prek cache once and
upload them as artifacts - then restoring prek cache for every
prek run should be much faster.
* Check if plural forms are correct for languages.
It turned out, that some of the languages we had, did not have
proper plural forms defined. There were few problems with it:
* some languages had "default" form defined where they should have
other forms
* there were no check if language was missing in the plural forms,
so new languages got the "default" form by - well - default.
* we had no check in CI to check if languages are defined in the
plural forms
This PR modifies the `check-translations-completeness` script and
uses it as a prek (manual) hook to make it easier to get run, also
to be able to run it in CI - it exits with non-zero exit only when
there is a clear issue with configuration or call parameters,
rather than when there are missing translations, it only prints
green/red message summarizing the output.
* Update dev/i18n/check_translations_completeness.py
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
---------
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
* feat(breeze): enable airflowctl docs to be released
* Update dev/breeze/src/airflow_breeze/utils/selective_checks.py
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Update docs/README.md
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* feat(docs): add function to retrieve airflowctl version and use it in DocsPublisher
* feat(docs): add function to retrieve airflowctl version and use it in DocsPublisher
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
There were a little inconsistencies how uv and prek were installed
in our CI
* install-prek action was not used in ci-amd.yaml
* UV_VERSION was not fixed in release-dockerhub-images
* uv was installed with pip rather than installation script from
astral in install-prek action
* Dynamically detect shared distributions in selective checks
* Run the tests as matrix
* Skip cleaning the worker and setting up docker and breeze
* Install the version of uv that we are bound with
* Run tests in parallel with xdist
After https://github.com/j178/prek/issues/35 has been completed and
released in `prek` 0.1.1 we can bring back the functionality of
flagging when our prek hooks need to be updated.
This PR brings minimum version of prek to 0.1.1 as well as
upgrades all other important versions up, upgrades breeze uv.lock
and brings back the steps in upgrade check to check for auto-upgradeable
hooks.
The go tests do not depend on CI image being built, also they should
not be run independently on the `run-tests` flag - because
run-tests is generally for our unit tests.
They should be run when `run-go-sdk-tests` is set and without
dependency on ci-images being built. There were also a few
other issues found during this cleanuup:
Several outputs were inconsistently named:
* run-tests was renamed to run-unit-tests
* need-* outputs were renamed to run-*
Some other fixes were applied:
* only-new-ui-files was removed as it is not needed any more as
we do not really need that check to be exposed to actions, it
is only used internally - to not run other tests when only ui
files changed
* run-system-tests was not properly used (it is currently an
alias to run-unit-tests anyway - but foreseen for future
separation$a
* need-javascript-scans and need-python-scans have been removed
as they are no longer used
We are not using black-related functionality in prek hooks any more - the
only remaining hook is 'blacken-docs` - but there is no need to synchronize
and upgrade black version any more for it.
Upgraded to latest black version for blacken-docs manually.
This upgrades prek to **just released** 0.0.29 and removes the
workaround implemented in #54613 to add build-essential to the
image that is used to build distributions - because prek 0.0.29
comes with all the necessary binary wheels/platforms that remove
the need to build in in our Linux ARM image (issue
https://github.com/j178/prek/issues/451 has been fixed).
The 0.0.28 prek has an important fix - avoids craching when you run
`git commit -a` https://github.com/j178/prek/issues/385 - we bump
the minimum_version to 0.0.28 as well as upgrading the CI and Breeze
version to 0.0.28 - also, we change the strategy of failing
upgrade check in CI - we do not fail the upgrade check when there
is a new upgrade of `prek` - similar like with `uv` we expect
new versions to be released quickly over the `teething` period.
