* allocating pseudo-terminal inside the python script creating
the images instead of trying to do it by docker compose run
* better diagnostics in case of error (verbosity handling)
* properly allocating console with forcing pesudo-terminal creation
inside the container when --tty command is used with breeze shell
via `enable-tty.yaml`
* upgrading prek + uv to latest versions
* a bit of refactoring how the docker-compose files are referrred to
* Console in the script also uses pseudo-terminal
The script had no good support for auto-complete and the like and
since everyone now use breeze for some tasks, we also should
have it as a regular breeze command.
Also the sorting order implemented by eslint jsonc natural
sorting was quite a bit different from what you'd expect:
* it sorts character-by-character case-insensitive
* it uses original sorting as tie-breker in case of characters
case-insensitve order
* handles numbers natuarlly (2<10)
* treats _ as end of word and sort it before everything else
* similarly with end of word - shorter words are before longer
if they have the same prefix
This PR updates the command and also fixes sorting order and
add unit tests to test vital parts of the translation script.
We have some semi-complex set of tools that take care about upgrading
of all our CI infrastructure periodically. While dependabot has a lot of
use, a lot of cases is not handled yet in it - such as upgrading the
charts, important dependencies in our scripts, dockerfiles and so on.
While we already have a set of prek hooks and github Actions that
do a lot there, some of that has to be done periodically outside of
dependabot - this CI wraps a number of those upgrade tools into a
single `ci upgrade` command that can be run locally, generate and
open PR - and can be used in both - main and any of the v*test
branches.
Future improvement areas:
* docker container upgrades for Helm chart and docker compose
* slack notification after such upgrade PR is opened
* more .....
- Update uv from 0.9.4 to 0.9.5
- Update ruff from 0.14.1 to 0.14.2
- Update mypy to 1.18.2
- Update Python to 3.12.12
- Update various other dependencies
There were a few issues with ARM workflows:
* not all jobs were run in ARM tests - we do not want to run mysql
of course, but other tests should be fine to run on ARM
* some conditions were not updated (we have to somehow duplicate
amd and arm job definition because we run out of composite
workflows - so sometimes conditions are not synced)
* most importantly - we uploaded prek cache in build-info job, but
that job only run on AMD, not on ARM so the ARM cache was really
an AMD one (and it caused unterminated strings in doctoc installation
It's not possible to upload same artifact twice in the same job and
since we use prek in several jobs we should make sure that the cache
is only uploaded once per job. This was the reason why it was initially
uploaded in build-info job (and save-cache was set to false elsewhere).
With this PR, we have save-cache in 3 places:
* basic checks
* static CI-image bound checks
* in octopin (Python 3.11)
Basic checks and static checks are mutually exclusive (controlled by
basic-checks-only flag) - so we can safely upload cache in both.
In all other places we only install prek with cache, but we do not
save the cache as artifact.
Apparently the prek cache mechanism has been somewhat broken for a
while - after we split prek to monorepo. The hash files used to
determine prek-cache was different for save and restore step
(the `**/` has been missing in the save cache step. Which means
that we always failed to restore cache and created it from the
scratch.
Also, it seems that the prek cache-when prepared refers to the uv
version that is pre-installed for it in case uv is not installed
in the system. And it refers to the uv version when creating the
virtual environments used by prek, and we first attempted to
install prek and create cache, and only after we installed uv, which
had a side-effect that in some cases the installed venvs referred
to a missing python binary.
Finally - there is a bug in prek https://github.com/j178/prek/issues/918
that pygrep cache contains reference to a non-existing python binary
that should be run when pygrep runs.
Also it's possible that some of the cache installed in workspace by the
github worker remained, and we did not preemptively clean the cache when
we attempted to restore it and failed.
This PR attempts to restore the cache usage in a more robust way:
* fixed cache key on save to save cache with proper name
* added uv version to cache key for prek
* always install uv in desired version before installing prek
* if we faile to cache-hit and restore the cache, we clean-up
the .cache/prek folder
* we do not look at skipped hooks when installing prek and restoring
or saving cache. There is very little saving on some hooks and
since we are preparing the cache in "build-info" now - it's better
to always use the same cache, no matter if some checks are skipped
* upgraded to prek 0.2.10 that fixed the issue with pygrep cache
This PR upgrades latest versions of the important dependencies, also
it implements the possibility of using the token that is used
locally via `gh` GitHub CLI - this way if you did `gh auth login`
to use `gh` for your repository, there is no need to create and
pass GITHUB_TOKEN via UI of GitHub.
Also added ruff to the upgraded packages, and synchronized the variables
in upgrade checks to include all the packages we are checking for
We decided to clean-up runner and remove all the potential garbage
we get when we start runner - we also move docker to a separate
/mnt partitiion available on the runners - because for unit testing
especially we need as much space as possible. Also when we build
images, it is best if we use clean, pristine docker engine located
on the /mnt folder.
All of our other jobs are ok with small disk space and sometimes that cleanup
can take up to a minute (usually it is 10-20s)
This change (experimentally) removes all cleanup from all jobs
except the unit tests and image building to see if we can get stability
and speed at the same time.
Previously we were only using prek cache from canary builds, but
since we are starting to use several separate prek runs, it makes
sense to install prek hooks only once and store them in cache and
reuse even in the same build.
This PR removes "only-canary" prek cache preparation - now all
builds including all PRs from fork preapare prek cache once and
upload them as artifacts - then restoring prek cache for every
prek run should be much faster.
* Check if plural forms are correct for languages.
It turned out, that some of the languages we had, did not have
proper plural forms defined. There were few problems with it:
* some languages had "default" form defined where they should have
other forms
* there were no check if language was missing in the plural forms,
so new languages got the "default" form by - well - default.
* we had no check in CI to check if languages are defined in the
plural forms
This PR modifies the `check-translations-completeness` script and
uses it as a prek (manual) hook to make it easier to get run, also
to be able to run it in CI - it exits with non-zero exit only when
there is a clear issue with configuration or call parameters,
rather than when there are missing translations, it only prints
green/red message summarizing the output.
* Update dev/i18n/check_translations_completeness.py
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
---------
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
* Dynamically detect shared distributions in selective checks
* Run the tests as matrix
* Skip cleaning the worker and setting up docker and breeze
* Install the version of uv that we are bound with
* Run tests in parallel with xdist
After https://github.com/j178/prek/issues/35 has been completed and
released in `prek` 0.1.1 we can bring back the functionality of
flagging when our prek hooks need to be updated.
This PR brings minimum version of prek to 0.1.1 as well as
upgrades all other important versions up, upgrades breeze uv.lock
and brings back the steps in upgrade check to check for auto-upgradeable
hooks.
The go tests do not depend on CI image being built, also they should
not be run independently on the `run-tests` flag - because
run-tests is generally for our unit tests.
They should be run when `run-go-sdk-tests` is set and without
dependency on ci-images being built. There were also a few
other issues found during this cleanuup:
Several outputs were inconsistently named:
* run-tests was renamed to run-unit-tests
* need-* outputs were renamed to run-*
Some other fixes were applied:
* only-new-ui-files was removed as it is not needed any more as
we do not really need that check to be exposed to actions, it
is only used internally - to not run other tests when only ui
files changed
* run-system-tests was not properly used (it is currently an
alias to run-unit-tests anyway - but foreseen for future
separation$a
* need-javascript-scans and need-python-scans have been removed
as they are no longer used
We are not using black-related functionality in prek hooks any more - the
only remaining hook is 'blacken-docs` - but there is no need to synchronize
and upgrade black version any more for it.
Upgraded to latest black version for blacken-docs manually.
The 0.0.28 prek has an important fix - avoids craching when you run
`git commit -a` https://github.com/j178/prek/issues/385 - we bump
the minimum_version to 0.0.28 as well as upgrading the CI and Breeze
version to 0.0.28 - also, we change the strategy of failing
upgrade check in CI - we do not fail the upgrade check when there
is a new upgrade of `prek` - similar like with `uv` we expect
new versions to be released quickly over the `teething` period.
