Prek 0.2.22 added the new feature that pretty much everyone adds
now after all shei-hulud attacks to get some cooldown period
of upgrades, to give chance for github scanners and "bleeding edge"
users to find out tha there are some malicious modifications.
This PR adds cooldown period to prek auto-upgrade in our CI
for `breeze ci upgrade` method.
We maintain release calendar with release page in Confluence in parallel.
This script (AI generated) verifies if what is in Release page is correctly
reflected in the calendar.
We heve a few release-manager acting people who can publish docs
and build images using GH actions workflows.
The list of this peaple is growing.
Adding Jens and Vinc and making sure that the list is synchronized
for both workflows we have.
* allocating pseudo-terminal inside the python script creating
the images instead of trying to do it by docker compose run
* better diagnostics in case of error (verbosity handling)
* properly allocating console with forcing pesudo-terminal creation
inside the container when --tty command is used with breeze shell
via `enable-tty.yaml`
* upgrading prek + uv to latest versions
* a bit of refactoring how the docker-compose files are referrred to
* Console in the script also uses pseudo-terminal
When only task-sdk-integration tests are changed, the tests should
run. Currently they don't run because in this case PROD image has
is not build.
This PR fixes the selective checks.
The script had no good support for auto-complete and the like and
since everyone now use breeze for some tasks, we also should
have it as a regular breeze command.
Also the sorting order implemented by eslint jsonc natural
sorting was quite a bit different from what you'd expect:
* it sorts character-by-character case-insensitive
* it uses original sorting as tie-breker in case of characters
case-insensitve order
* handles numbers natuarlly (2<10)
* treats _ as end of word and sort it before everything else
* similarly with end of word - shorter words are before longer
if they have the same prefix
This PR updates the command and also fixes sorting order and
add unit tests to test vital parts of the translation script.
After adding -source package with release date to providers release,
we also need to update the source download documentation to link to
the right source tarball.
This PR adds the last missing bits and pieces to do that:
1) Updates information about format of RELEASE_DATE and supported
(potentially) YYYY_MM_DD_NN format
2) The release date parameter is used and verified at release
docuemntation preparation time and it's stored in
"providers/.last_release_date.txt"
3) The relase date of the current release (stored in the commit
which is used to release providers) is used to produce the
right links - both in "providers summary" and in individual
provider's package documntation
The cooldown period delays updating dependencies until they are
N-days old. This allows the dependencies to be tried by others
(including security researchers) and get them flagged and
removed in case some security issues are found.
There were a few of things that made task-sdk-tests iteration
a bit unobvious and slower. With this PR, we should be able
to iterate over task-sdk-integration-tests WAY faster and get
more contributors involved in contributing to those.
* It was not clear that prerequisite of running the tests was building
PROD image for Pyton 3.10. This is now clear in the documentation.
* PROD images can be built in two different modes - from sources
with --installation-method equal to . or from packages with
the --installatio-method equal to "apache-airflow". This was
not clearly communicated during build and it is now printed at
output
* It was not clear that when you build PROD images from sources,
you should first compile ui assets, because otehrwise the
assets are not added as part of the image. With this PR the
`breeze prod-image build` command checks if the .vite manifest
is present in the right `dist` folders and will error out,
suggesting to run `breeze compile-ui-assets` before.
* If the PROD image has not been built before, breeze will propose
to build it and even do it automatically if the answer is not
provided within 20 seconds.
* when building PROD images from sources, it is faster to rebuild
the images with `uv` than with `pip`. the --use-uv parameter now
defaults to False when building from packages and to True when
building from sources.
* There was an error in .dockerignore where generated dist files
were not added to context when PROD image was built from sources.
This resulted in "permission denied' when such PROD images were used
to run tests.
* The test compose had fallback of Airflow 3.0.3 which would be
misleading if it happened. Now, AIRFLOW_IMAGE_NAME is mandatory
* We are now mounting sources of Airflow to inside the image by default
and skip it in CI. This mounting happens in local environment where PROD
image is built usually from sources, and it is disabled in CI by using
--skip-mounting-local-volumes flag. We also do not stop docker compose
by default when runnig it locally in order to make fast iteration the
default.
* We pass host operating system when starting the compose, and we only
change ownership on Linux - this is a long running operation on MacOS
because mounted filesystem is slow, but it's also not needed on MacOS
because the file system also maps ownershipt and files created by
Airflow are created with local user id.
* We pass local user id to containers to make sure that the files
created on linux are created by the local user (logs and the like).
* We are now detecting whether docker-compose is running and when we run
with locally mounted sources, we reuse those running containers. When
we don't mount local sources, we shut-down the compose before running
to make sure we do not have sources mounted - and we close the compose
by default when we do not mount local sources.
* When sources are mounted we are only enabling DEV_MODE inside the containers
so that components are hot-reloading (new feature added in #57741
last weeks. This way you do not have to restart anything when sources
are changed and you can re-run the tests when docker compose is running.
* The environment are passsed now via .env file so that you can easily
reproduce docke compose command locally
* The docker compose files are not copied any more, they are moved directly
to the top of 'task-sdk-integraiton-tests' and used from there.
* A `--down` flag is added to breeze testing task-sdk-integration-tests
that tears down running docker compose.
* Additional diagnostics added to show what's going on.
* Handling verbose option from breeze by adding more debugging information
* Updated documentation about the tests to be more comprehensive about
the options, updated file structure etc.
* Small QOL immprovement - if expected dags are not yet parsed by dag file
processor, when test starts, getting their status will return 404 "Not
Found". In such case our tests implemented a short retry scheme with tenacity
Rather than having multiple PRs - separately for each dependency
upgrade, we should group the dependency upgrades accross several
distributions into single PRs as much as possible.
We have some semi-complex set of tools that take care about upgrading
of all our CI infrastructure periodically. While dependabot has a lot of
use, a lot of cases is not handled yet in it - such as upgrading the
charts, important dependencies in our scripts, dockerfiles and so on.
While we already have a set of prek hooks and github Actions that
do a lot there, some of that has to be done periodically outside of
dependabot - this CI wraps a number of those upgrade tools into a
single `ci upgrade` command that can be run locally, generate and
open PR - and can be used in both - main and any of the v*test
branches.
Future improvement areas:
* docker container upgrades for Helm chart and docker compose
* slack notification after such upgrade PR is opened
* more .....
We had pin-versions prek-hook implemented in a separate workflow
under the `dev` folder, but it has not been working since workspace
switch because prek workspace only works on sub-folders of where the
.pre-commit-config.yaml file is placed. It was in a separate file
because it needed python 3.11 to run, but it is possible to have
a specific python verison as separate language version in the hook
itself, so we can move it back to the main .pre-commit-config.yaml
This PR:
* moves the pin-version hook to main .pre-commit-config.yaml
* sets python 3.11 as version of python used in the hook independently
from default python version
* fix github actions and docs to use the hook from the main
.pre-commit-config.yaml
