2021-03-21 09:08:10 +01:00
|
|
|
.. Licensed to the Apache Software Foundation (ASF) under one
|
|
|
|
|
or more contributor license agreements. See the NOTICE file
|
|
|
|
|
distributed with this work for additional information
|
|
|
|
|
regarding copyright ownership. The ASF licenses this file
|
|
|
|
|
to you under the Apache License, Version 2.0 (the
|
|
|
|
|
"License"); you may not use this file except in compliance
|
|
|
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
|
|
|
|
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
|
|
|
|
|
|
.. Unless required by applicable law or agreed to in writing,
|
|
|
|
|
software distributed under the License is distributed on an
|
|
|
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
|
|
|
KIND, either express or implied. See the License for the
|
|
|
|
|
specific language governing permissions and limitations
|
|
|
|
|
under the License.
|
|
|
|
|
|
2022-01-23 14:15:45 +01:00
|
|
|
.. WARNING:
|
|
|
|
|
IF YOU ARE UPDATING THIS FILE, CONSIDER UPDATING README.MD TOO.
|
|
|
|
|
|
2021-03-21 09:08:10 +01:00
|
|
|
.. image:: /img/docker-logo.png
|
|
|
|
|
:width: 100
|
|
|
|
|
|
|
|
|
|
Docker Image for Apache Airflow
|
|
|
|
|
===============================
|
|
|
|
|
|
|
|
|
|
.. toctree::
|
|
|
|
|
:hidden:
|
|
|
|
|
|
|
|
|
|
Home <self>
|
|
|
|
|
build
|
|
|
|
|
entrypoint
|
2022-03-27 19:19:02 +02:00
|
|
|
changelog
|
2021-03-21 09:08:10 +01:00
|
|
|
recipes
|
|
|
|
|
|
|
|
|
|
.. toctree::
|
|
|
|
|
:hidden:
|
|
|
|
|
:caption: References
|
|
|
|
|
|
|
|
|
|
build-arg-ref
|
|
|
|
|
|
|
|
|
|
For the ease of deployment in production, the community releases a production-ready reference container
|
|
|
|
|
image.
|
|
|
|
|
|
|
|
|
|
|
2021-06-02 01:08:44 +02:00
|
|
|
The Apache Airflow community, releases Docker Images which are ``reference images`` for Apache Airflow.
|
|
|
|
|
Every time a new version of Airflow is released, the images are prepared in the
|
|
|
|
|
`apache/airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_
|
|
|
|
|
for all the supported Python versions.
|
|
|
|
|
|
2025-05-06 18:15:34 +02:00
|
|
|
The images we release are multi-platform AMD/ARM images.
|
|
|
|
|
|
Add Python 3.13 support for Airflow. (#46891)
* Breeze changes for Python 3.13
* Pyproject.toml provider changes for Python 3.13
* README.rst changes for providers for Python 3.13
* Add Python 3.13 support for Airflow.
Added Python 3.13 support across the codebase, including:
* CI/CD workflow updates to handle Python 3.13.
* Dockerfile and environment variable adjustments for compatibility.
* Documentation updates to list Python 3.13 as supported.
* Dependency and code changes for compatibility with new/changed
libraries (e.g., numpy 2.x, greenlet, pendulum).
* Test and plugin manager adjustments for Python 3.13-specific behaviors.
* Removal or skipping of tests and features not compatible with Python 3.13.
* Improved error handling and logging for unsupported plugins/providers
on Python 3.13.
* Updated all provider README.rst files to:
* List Python 3.13 as a supported version.
* Adjust dependency requirements for Python 3.13 (e.g., pandas,
pyarrow, ray, etc.).
* Add or update notes about excluded or conditionally supported
dependencies for Python 3.13.
* Document any known incompatibilities or workarounds for specific
providers.
* Updated all provider pyproject.toml files and automation to:
* Add Python 3.13 to the supported classifiers.
* Adjust requires-python and dependency constraints for Python 3.13.
* Exclude Python 3.13 for providers that are not yet compatible (e.g.,
those depending on Flask AppBuilder).
* Add conditional dependencies for new versions of libraries required
by Python 3.13.
* Updated Breeze and related scripts to:
* Build and test with Python 3.13.
* Update documentation and release scripts to include Python 3.13.
* Ensure all dev tooling and test environments work with Python 3.13.
* Removed remaining FAB related code from core and removed tests for
older FAB provider versions (<1.3.0)
* some of the tests in core still depend on FAB - but we make them
optional - both for parsing (we need to skip some imports) and
execution - because they test a legacy behaviour of embedding
Flask plugins that is still supported in core.
* k8s tests now run also with SimpleAuthManager and SimpleAuthManager
is used automatically when runnin tests in Python 3.13
* migration tests had to be excluded fo Python 3.13 because we cannot
migrate down on Python 3.13 without FAB tables created. The tests
will be brought back when FAB is supported or when we release 3.1
and start working on 3.2, we should be able to migrate down to 3.1
* Weaviate tests have been updated to handle case where weaviate-client
is > 2.10.0 - because it started to use httpx instead of requests,
and Python 3.13 actually unblocks using older version of weaviate
(it was previously blocked indirectly via Flask dependencies and
grpcio).
* Added doc change in our rules on what is the "default" Python version
for airflow images - handling the case where we cannot use "newest"
Python version as default where not all "regular" image providers are
supported in the latest version
* Connection is now flushed in `configure_git_connection_for_dag_bundle`
just before `clear_db_connections` - in order to handle Python 3.13
case where FAB app is not initialized by other fixtures - performing
implicit flush along the way
* Cleaning db connections was added to test_dag_run and test_dags test
in fast_api as the test implicitly rely on having connections
cleared (no test git connections added) - this will make the tests
side-effect free.
* Conditionally skipping some serialization tests involving yandex
and ray on Python 3.13 until they support Python 3.13
2025-07-17 16:53:49 +02:00
|
|
|
The default Python version for the reference image - one that you get when you do not specify version -
|
|
|
|
|
is the newest supported Python version at the time of the Airflow release that supports
|
|
|
|
|
all providers released in the regular reference image.
|
|
|
|
|
|
|
|
|
|
For example, even when we release Airflow 3.0 with Python 3.13 support, but some of the providers that
|
|
|
|
|
are installed by default in the regular reference image do not support Python 3.13,
|
|
|
|
|
the "default" Python version for the image should be Python 3.12.
|
|
|
|
|
|
|
|
|
|
The same applies to the "slim" images.
|
|
|
|
|
|
2022-01-23 14:15:45 +01:00
|
|
|
You can find the following images there (Assuming Airflow version :subst-code:`|airflow-version|`):
|
2021-06-02 01:08:44 +02:00
|
|
|
|
2024-10-08 12:00:11 +02:00
|
|
|
* :subst-code:`apache/airflow:latest` - the latest released Airflow image with default Python version (3.12 currently)
|
2021-08-19 10:02:58 +01:00
|
|
|
* :subst-code:`apache/airflow:latest-pythonX.Y` - the latest released Airflow image with specific Python version
|
2024-10-08 12:00:11 +02:00
|
|
|
* :subst-code:`apache/airflow:|airflow-version|` - the versioned Airflow image with default Python version (3.12 currently)
|
2021-08-19 10:02:58 +01:00
|
|
|
* :subst-code:`apache/airflow:|airflow-version|-pythonX.Y` - the versioned Airflow image with specific Python version
|
2021-06-02 01:08:44 +02:00
|
|
|
|
2022-05-10 17:24:26 +02:00
|
|
|
Those are "reference" regular images. They contain the most common set of extras, dependencies and providers that are
|
2021-12-01 10:10:56 +01:00
|
|
|
often used by the users and they are good to "try-things-out" when you want to just take Airflow for a spin,
|
2021-06-02 01:08:44 +02:00
|
|
|
|
2025-04-19 16:47:32 +05:30
|
|
|
You can also use "slim" images that contain only core Airflow and are about half the size of the "regular" images
|
2022-07-06 13:29:53 +02:00
|
|
|
but you need to add all the :doc:`apache-airflow:extra-packages-ref` and providers that you need separately
|
2022-05-10 17:24:26 +02:00
|
|
|
via :ref:`Building the image <build:build_image>`.
|
|
|
|
|
|
2024-10-08 12:00:11 +02:00
|
|
|
* :subst-code:`apache/airflow:slim-latest` - the latest released Airflow image with default Python version (3.12 currently)
|
2022-05-10 17:24:26 +02:00
|
|
|
* :subst-code:`apache/airflow:slim-latest-pythonX.Y` - the latest released Airflow image with specific Python version
|
2024-10-08 12:00:11 +02:00
|
|
|
* :subst-code:`apache/airflow:slim-|airflow-version|` - the versioned Airflow image with default Python version (3.12 currently)
|
2022-05-10 17:24:26 +02:00
|
|
|
* :subst-code:`apache/airflow:slim-|airflow-version|-pythonX.Y` - the versioned Airflow image with specific Python version
|
|
|
|
|
|
2021-06-02 01:08:44 +02:00
|
|
|
The Apache Airflow image provided as convenience package is optimized for size, and
|
2021-03-21 09:08:10 +01:00
|
|
|
it provides just a bare minimal set of the extras and dependencies installed and in most cases
|
2022-07-06 13:29:53 +02:00
|
|
|
you want to either extend or customize the image. You can see all possible extras in :doc:`apache-airflow:extra-packages-ref`.
|
2021-06-02 01:08:44 +02:00
|
|
|
The set of extras used in Airflow Production image are available in the
|
2024-02-23 19:19:06 +01:00
|
|
|
`Dockerfile <https://github.com/apache/airflow/blob/|airflow-version|/Dockerfile>`__.
|
2021-06-02 01:08:44 +02:00
|
|
|
|
|
|
|
|
However, Airflow has more than 60 community-managed providers (installable via extras) and some of the
|
|
|
|
|
default extras/providers installed are not used by everyone, sometimes others extras/providers
|
|
|
|
|
are needed, sometimes (very often actually) you need to add your own custom dependencies,
|
|
|
|
|
packages or even custom providers. You can learn how to do it in :ref:`Building the image <build:build_image>`.
|
2021-03-21 09:08:10 +01:00
|
|
|
|
|
|
|
|
The production images are build in DockerHub from released version and release candidates. There
|
|
|
|
|
are also images published from branches but they are used mainly for development and testing purpose.
|
2024-12-17 21:57:37 +01:00
|
|
|
See `Airflow Git Branching <https://github.com/apache/airflow/blob/main/contributing-docs/10_working_with_git.rst#airflow-git-branches>`_
|
2021-03-21 09:08:10 +01:00
|
|
|
for details.
|
2021-05-23 06:02:34 +02:00
|
|
|
|
2026-02-16 13:46:29 +01:00
|
|
|
.. _fixing-image-at-release-time:
|
|
|
|
|
|
2023-01-06 10:29:28 +01:00
|
|
|
Fixing images at release time
|
|
|
|
|
=============================
|
|
|
|
|
|
|
|
|
|
The released "versioned" reference images are mostly ``fixed`` when we release Airflow version and we only
|
|
|
|
|
update them in exceptional circumstances. For example when we find out that there are dependency errors
|
|
|
|
|
that might prevent important Airflow or embedded provider's functionalities working. In normal circumstances,
|
|
|
|
|
the images are not going to change after release, even if new version of Airflow dependencies are released -
|
|
|
|
|
not even when those versions contain critical security fixes. The process of Airflow releases is designed
|
|
|
|
|
around upgrading dependencies automatically where applicable but only when we release a new version of Airflow,
|
|
|
|
|
not for already released versions.
|
|
|
|
|
|
2024-02-23 19:19:06 +01:00
|
|
|
What should I do if my security scan shows critical and high vulnerabilities in the image?
|
|
|
|
|
==========================================================================================
|
|
|
|
|
|
|
|
|
|
We often hear questions that our users use various security scanners on the image and find out that
|
|
|
|
|
there are some critical and high vulnerabilities in the image - not coming from Airflow but for some other
|
|
|
|
|
components. In general, this is normal and expected that such vulnerabilities are found in the image after
|
|
|
|
|
it's been released and fixed - precisely because we are NOT updating the images after they are released as
|
|
|
|
|
explained above. Also sometimes even the latest releases contain vulnerabilities that are not yet fixed
|
|
|
|
|
in the base image we use or in the dependencies we use and cannot upgrade, because some of our providers
|
|
|
|
|
have limits and did not manage to upgrade yet and we have no control over that. So it is possible
|
|
|
|
|
that even the most recent release of our image there are some High and Critical vulnerabilities that
|
|
|
|
|
are not yet fixed.
|
|
|
|
|
|
|
|
|
|
**What can you do in such case?**
|
|
|
|
|
|
|
|
|
|
First of all - you should know what you should NOT do.
|
|
|
|
|
|
|
|
|
|
Do NOT send private email to the Airflow Security Team with scan results and asking what to do.
|
|
|
|
|
The Security team at Airflow takes care exclusively about undisclosed vulnerabilities in Airflow itself, not
|
|
|
|
|
in the dependencies or in the base image. The security email should only be used to report privately any
|
|
|
|
|
security issues that can be exploited via Airflow. This is nicely explained in our
|
|
|
|
|
`Security Policy <https://github.com/apache/airflow/security/policy>`__ where you can find more details
|
|
|
|
|
including the need to provide reproducible scenarios and submitting ONE issue per email. NEVER submit multiple
|
|
|
|
|
vulnerabilities in one email - those are rejected immediately, as they make the process of handling the issue
|
|
|
|
|
way harder for everyone, including the reporters.
|
|
|
|
|
|
2024-02-26 17:43:28 +01:00
|
|
|
Also DO NOT open a GitHub Issue with the scan results and asking what to do. The GitHub Issues are for
|
2024-02-23 19:19:06 +01:00
|
|
|
reporting bugs and feature requests to Airflow itself, not for asking for help with the security scans on
|
|
|
|
|
3rd party components.
|
|
|
|
|
|
|
|
|
|
So what are your options?
|
|
|
|
|
|
|
|
|
|
You have four options:
|
|
|
|
|
|
|
|
|
|
1. Build your own custom image following the examples we share there - using the latest base image and
|
|
|
|
|
possibly manually bumping dependencies you want to bump. There are quite a few examples
|
|
|
|
|
in :ref:`Building the image <build:build_image>` which you can follow. You can use "slim" image as a base
|
|
|
|
|
for your images and rather than basing your image on the "reference" image that has a number of extras
|
|
|
|
|
and providers installed, you can only install what you actually need and upgrade some dependencies that
|
|
|
|
|
otherwise would not be possible to upgrade - because some of the provider libraries have limits and
|
|
|
|
|
did not manage to upgrade yet and we have no control over that. This is the most flexible way to
|
|
|
|
|
build your image and you can build your process to combine it with quickly upgrading to latest Airflow
|
|
|
|
|
versions (see point 2. below).
|
|
|
|
|
|
|
|
|
|
2. Wait for a new version of Airflow and upgrade to it. Airflow images are updated to latest "non-conflicting"
|
|
|
|
|
dependencies and use latest "base" image at release time, so what you have in the reference images
|
|
|
|
|
at the moment we publish the image / release the version is what is "latest and greatest"
|
|
|
|
|
available at the moment with the base platform we use (Debian Bookworm is the reference image we use).
|
|
|
|
|
This is one of good strategies you can take - build a process to upgrade your Airflow version regularly
|
|
|
|
|
- quickly after it has been released by the community, this will help you to keep up with the latest
|
|
|
|
|
security fixes in the dependencies.
|
|
|
|
|
|
|
|
|
|
3. If the base platform we use (currently Debian Bookworm) does not contain the latest versions you want
|
|
|
|
|
and you want to use other base images, you can take a look at what system dependencies are installed
|
2025-04-19 16:47:32 +05:30
|
|
|
and scripts in the latest ``Dockerfile`` of Airflow and take inspiration from it and build your own image
|
2024-02-23 19:19:06 +01:00
|
|
|
or copy it and modify it to your needs. See the
|
|
|
|
|
`Dockerfile <https://github.com/apache/airflow/blob/|airflow-version|/Dockerfile>`__ for the latest version.
|
|
|
|
|
|
|
|
|
|
4. Research if the vulnerability affects you or not. Even if there is a dependency with high or critical
|
|
|
|
|
vulnerability, it does not mean that it can be exploited in Airflow (or specifically in the way you are
|
|
|
|
|
using Airflow). If you do have a reproducible scenario how a vulnerability can be exploited in Airflow, you should -
|
|
|
|
|
of course - privately report it to the security team. But if you do not have reproducible
|
2024-02-26 17:43:28 +01:00
|
|
|
scenario, please do some research and try to understand the impact of the vulnerability on Airflow. That
|
2024-02-23 19:19:06 +01:00
|
|
|
research might result in a public GitHub Discussion where you can discuss the impact of the
|
|
|
|
|
vulnerability if you research will indicate Airflow might not be impacted or private security email if
|
|
|
|
|
you find a reproducible scenario on how to exploit it.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**How do I discuss publicly about public CVEs in the image?**
|
|
|
|
|
|
|
|
|
|
The security scans report public vulnerabilities in 3rd-party components of Airflow. Since those are
|
|
|
|
|
already public vulnerabilities, this is something you can talk about but others also are talking about.
|
|
|
|
|
So you can do research on your own first. Try to find discussions about the issues, how others were handling
|
|
|
|
|
it and possibly even try to explore, whether the vulnerability can be exploited in Airflow or not.
|
|
|
|
|
This is a very valuable contribution to the community you can do in order to help others to
|
|
|
|
|
understand the impact of the vulnerability on Airflow. We highly appreciate our commercial users do it,
|
|
|
|
|
because Airflow is maintained by volunteers, so if you or your company can spend some time and skills of
|
|
|
|
|
security researchers to help the community to understand the impact of the vulnerability on Airflow, it
|
|
|
|
|
could be a fantastic contribution to the community and way to give back to the project that your company uses
|
|
|
|
|
for free.
|
|
|
|
|
|
2024-02-26 17:43:28 +01:00
|
|
|
You are free to discuss it publicly, open a `GitHub Discussion <https://github.com/apache/airflow/discussions>`_
|
2024-02-23 19:19:06 +01:00
|
|
|
mentioning your findings and research you've done so far. Ideally (as a way to contribute to Airflow) you
|
|
|
|
|
should explain the findings of your own security team in your company to help to research and understand
|
|
|
|
|
the impact of the vulnerability on Airflow (and your way of using it).
|
|
|
|
|
Again - strong suggestion is to open ONE discussion per vulnerability. You should NOT post scan results in
|
|
|
|
|
bulk - this is not helpful for a discussion, and you will not get meaningful answers if you will attempt to
|
|
|
|
|
discuss all the issues in one discussion thread.
|
|
|
|
|
|
|
|
|
|
Yes - we know it's the easy way to copy & paste your result and ask others what to do, but doing it is
|
|
|
|
|
going to likely result in silence because such actions in the community as seen as pretty selfish way of
|
|
|
|
|
getting your problems solved by tapping into time of other volunteers, without spending your time on making it
|
|
|
|
|
easier for them to help. If you really want to get help from the community, focus your discussion on
|
|
|
|
|
particular CVE, provide your findings - including analyzing your report in detail and finding which
|
|
|
|
|
binaries and base images exactly are causing the scanner to report the vulnerability. Remember that only
|
|
|
|
|
you have access to your scanner and you should bring as much helpful information so that others can
|
|
|
|
|
comment on it. Show that you have done your homework and that you bring valuable information to the community.
|
|
|
|
|
|
|
|
|
|
Opening a GitHub Discussion for this kind of issues is also a great way to communicate with the
|
|
|
|
|
maintainers and security team in an open and transparent way - without reverting to the private security
|
|
|
|
|
mailing list (which serves different purpose as explained above). If after such a discussion there will be
|
|
|
|
|
a way to remove such a vulnerability from the scanned image - great, you can even contribute a PR to the
|
|
|
|
|
Dockerfile to remove the vulnerability from the image. Maybe such a discussion will lead to a PR to allow
|
|
|
|
|
Airflow to upgrade to newer dependency that fixes the vulnerability or remove it altogether, or maybe
|
|
|
|
|
there is already a way to mitigate it or maybe there is already a PR that someone works on to fix it.
|
|
|
|
|
All this can (and should) be discussed publicly and transparently in a GitHub Discussion, not via private
|
|
|
|
|
security email, nor GitHub Issues which are exclusively about Airflow Issues not 3rd-party components
|
|
|
|
|
public security issues.
|
2023-01-06 10:29:28 +01:00
|
|
|
|
2022-12-22 14:02:24 +01:00
|
|
|
Support
|
2024-08-27 21:13:32 +08:00
|
|
|
========
|
2022-12-22 14:02:24 +01:00
|
|
|
|
|
|
|
|
The reference Docker Image supports the following platforms and database:
|
|
|
|
|
|
|
|
|
|
|
2023-07-06 17:28:29 +02:00
|
|
|
* Postgres Client
|
2025-05-06 18:15:34 +02:00
|
|
|
* MySQL Client (8+)
|
2023-07-06 17:28:29 +02:00
|
|
|
* MSSQL Client
|
|
|
|
|
|
2025-05-06 18:15:34 +02:00
|
|
|
Note that MySQL on ``ARM`` platform has experimental support through MariaDB client library.
|
2021-05-23 06:02:34 +02:00
|
|
|
|
|
|
|
|
Usage
|
|
|
|
|
=====
|
|
|
|
|
|
2025-03-20 07:21:39 -07:00
|
|
|
The :envvar:`AIRFLOW_HOME` is set by default to ``/opt/airflow/`` - this means that dags
|
2022-08-17 07:10:22 +08:00
|
|
|
are by default in the ``/opt/airflow/dags`` folder and logs are in the ``/opt/airflow/logs``
|
2021-05-23 06:02:34 +02:00
|
|
|
|
|
|
|
|
The working directory is ``/opt/airflow`` by default.
|
|
|
|
|
|
2022-04-11 23:00:25 +01:00
|
|
|
If no :envvar:`AIRFLOW__DATABASE__SQL_ALCHEMY_CONN` variable is set then SQLite database is created in
|
2021-05-23 06:02:34 +02:00
|
|
|
``${AIRFLOW_HOME}/airflow.db``.
|
|
|
|
|
|
|
|
|
|
For example commands that start Airflow see: :ref:`entrypoint:commands`.
|
|
|
|
|
|
|
|
|
|
Airflow requires many components to function as it is a distributed application. You may therefore also be interested
|
2022-08-24 12:12:47 -04:00
|
|
|
in launching Airflow in the Docker Compose environment, see: :doc:`apache-airflow:howto/docker-compose/index`.
|
2021-05-23 06:02:34 +02:00
|
|
|
|
|
|
|
|
You can use this image in :doc:`Helm Chart <helm-chart:index>` as well.
|
