Eclipse's octopin has been archived in Feb and will not get any
more updates https://github.com/eclipse-csi/octopin.
Dependabot should be good enough to do the updates for us.
Devlist Discussion: https://lists.apache.org/thread/7n4pklzcc4lxtxsy9g69ssffg9qbdyvb
A static-site provider registry for discovering and browsing Airflow providers and their modules. Deployed at `airflow.apache.org/registry/` alongside the existing docs infrastructure (S3 + CloudFront).
Staging preview: https://airflow.staged.apache.org/registry/
## Acknowledgments
Many of you know the [Astronomer Registry](https://registry.astronomer.io), which has been the go-to for discovering providers for years. Big thanks to **Astronomer** and @josh-fell for building and maintaining it. This new registry is designed to be a community-owned successor on `airflow.apache.org`, with the eventual goal of redirecting `registry.astronomer.io` traffic here once it's stable. Thanks also to @ashb for suggesting and prototyping the Eleventy-based approach.
## What it does
The registry indexes all 99 official providers and 840 modules (operators, hooks, sensors, triggers, transfers, bundles, notifiers, secrets backends, log handlers, executors) from the existing
`providers/*/provider.yaml` files and source code in this repo. No external data sources beyond PyPI download stats.
**Pages:**
- **Homepage** — search bar (Cmd+K), stats counters, featured and new providers
- **Providers listing** — filterable by lifecycle stage (stable/incubation/deprecated), category, and sort order (downloads, name, recently updated)
- **Provider detail** — module counts by type, install command with extras/version selection, dependency info, connection builder, and a tabbed module browser with category sidebar and per-module search
- **Explore by Category** — providers grouped into Cloud, Databases, Data Warehouses, Messaging, AI/ML, Data Processing, etc.
- **Statistics** — module type distribution, lifecycle breakdown, top providers by downloads and module count
- **JSON API** — `/api/providers.json`, `/api/modules.json`, per-provider endpoints for modules, parameters, and connections
**Connection Builder** — pick a connection type (e.g. `aws`, `redshift`), fill in the form fields with placeholders and sensitivity markers, and export as URI, JSON, or environment variable format. Fields are
extracted from provider.yaml connection metadata.
* Add CI workflow to validate newsfragment PR numbers
Newsfragment files follow the naming convention `{pr_number}.{type}.rst`,
but nothing currently validates that the PR number in the filename matches
the actual PR number. This has led to cases where a contributor copies a
newsfragment from another PR or makes a typo, and the mismatch goes
unnoticed until a reviewer catches it manually.
The existing `scripts/ci/prek/newsfragments.py` validation script runs as
a local pre-commit hook where the PR number is not yet known, so it cannot
perform this check. Rather than extending that script with optional CLI
args and a separate CI invocation, this adds a standalone lightweight
workflow that uses `gh pr diff --name-only` to get the list of changed
files, greps for newsfragment `.rst` files, and checks that none have a
mismatched PR number — all in a single piped command, no checkout needed.
Notes for reviewers:
- `gh pr diff --name-only` includes deleted files. In practice, newsfragment
deletions only happen during towncrier releases on main, not in contributor
PRs, so this is not a concern for the `pull_request` trigger.
- `GH_TOKEN: ${{ github.token }}` follows the same pattern as
`milestone-tag-assistant.yml` and `backport-cli.yml` which also call `gh`
CLI directly.
- The `pull-requests: read` permission is required for `gh pr diff` to work
on fork PRs.
* fixup! Add CI workflow to validate newsfragment PR numbers
fixup! Add CI workflow to validate newsfragment PR numbers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Fix all build-system/requires including transitive dependencies
Add a feature to fix all build-system/requires to make build
reproducibility works. The build-system requires are managed
automatically by the `upgrade-important-versions` prek hook.
This should not only provide reproducibility, but also it should
prevent cases where transient dependency upgrade might break
building sdist (for example as it happened with virtualenv 21
breaking hatchling with https://github.com/pypa/hatch/issues/2193
* Fix mypy 'Cannot infer type of lambda' error in upgrade script
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The #62264 changed the build docs process to use current commit
for workflows/scripts to support building docs for Airflow 2,
however it broke building commit docs from history because fetch
depth was set to 1 (default) - but it should be 0 to fetch all
history.
In case we built docs for past version of airflow we checkout
both - current (main) version of code to use latest environment
and tag version of code to build the documentation. The workflow
mixed sub-workflows and run them from the "tag" version of workflow
rather than from current.
We are moving SBOM building to the publishing job, and we are also
pre-processing the generated docs to include the right stable
versions - including Airflow 2 compatibility.
The first workflow builds the docs and stores the stable versions
in stable.txt file, where the second workflow creates SBOMS
and publishes the folders in S3 using the stable.txt files stored.
* Remove `UV_HTTP_TIMEOUT`
See https://github.com/apache/airflow/discussions/61205.
Airflow is a large, popular project and a reference for other projects. I'd like to remove the historic use of `UV_HTTP_TIMEOUT`, that isn't required today anymore with the changes in uv. This will also benefit airflow, as broken network connections will fail much faster (either triggering a retry quicker, or reporting to the user in less than 4 x 300s = 20min)
I've kept the CLI option in the dev script to not break existing invocations, I can also remove it if you think it's safe
* Remove `--uv-http-timeout` from launchers too
* Add Milestone Tag Assistant
* Refactor the set-milestone utils to breeze ci_group
* Improves milestone assignment and adds tests for PR tagging
Refines the logic for auto-assigning GitHub milestones to merged PRs, prioritizing backport labels, version branches, and latest open milestones for main branch merges. Introduces user notification comments when no suitable milestone is found and prompts manual action. Adds comprehensive tests covering milestone detection, assignment, and notification flows to ensure robust behavior and edge case handling.
* Refactor milestone comment generation and improve tests for mention handling
* Add CLI testing fixtures and enhance milestone comment generation in tests
* Fix test scenarios
* Add check to skip setting milestone if already assigned in Breeze
* Remove merging into main logic
* Update Milestone Tag Assistant documentation to clarify triggering conditions and rules for milestone assignment
* Fix mypy error
* Enable dynamic generation of assets for Edge UI Plugin
* Review feedback
* Revert changes in template and other providers pyproject.toml
* Fixes in config
* Exclude __init__.py
* Ensure assets are generated on provider packaging
* Script beautification
* Ensure assets are compiled always
* Fix static checks by lazy loading providers details
* Fix missing dist in pytest
* Fix static check on openapi-spec generation for edge
* build assets via pnpm to prevent static check fails
* build assets via pnpm to prevent static check fails
* Fix hatch to ensure dist is compiled
* Call hatch in the venv of breeze tool
* Call asset generation script directly w/o prek
* Re-add prek dependency during provider build, add prek to GH pipeline
* Review feedback and fixes
* Use empty env for hatch build for providers
* Add prek for constrains generation
The release candidate commands were failing in CI because SVN operations
require credentials and network access to the ASF SVN repository.
This change detects when running in CI (via CI environment variable) and
simulates SVN operations instead of executing them:
- svn checkout: creates empty directory structure
- svn mkdir: uses regular mkdir
- svn add/commit/rm: prints simulation messages
This allows the release candidate workflow to be validated in CI without
requiring SVN access, while preserving full functionality in production.
The run-breeze-integration-tests job in .github/workflows/basic-tests.yml (line 122-125) downloads GPG keys from https://dist.apache.org/repos/dist/release/airflow/KEYS via wget. This network call frequently fails when the Apache SVN infrastructure is under high load, causing the CI step to fail (the || true only covers gpg --import, not wget itself).
The only consumer of these imported GPG keys is one integration test class: TestValidateSignaturesIntegration in dev/breeze/tests/test_airflow_release_validator_integration.py:199-217. This test already handles missing keys gracefully — when gpg --verify fails due to missing public keys, the test asserts that the failure details contain hints about downloading keys, and the test passes either way.
PRs with the pending-response label now go stale after 7 days instead
of 45 days. When the author responds or pushes, the pending-response
and stale labels are automatically removed.
We added structure to these to keep track of the large number of
breaking changes that were coming for Airflow 3. We no longer need any
of this structure or enforcement - we can go back to letting authors/reviewers
steer the content.
