SIGN IN SIGN UP
apache / airflow UNCLAIMED

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

44817 0 0 Python
#!/usr/bin/env python3
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
from __future__ import annotations
import sys
Move airflow sources to airflow-core package (#47798) This is continuation of the separation of the Airflow codebase into separate distributions. This one splits airflow into two of them: * apache-airflow - becomes an empty, meta no-code distribution that only has dependencies to apache-airflow-core and task-sdk distributions and it has preinstalled provider distributions added in standard "wheel" distribution. All "extras" lead either to "apache-airflow-core" extras or to providers - the dependencies and optional dependencies are calculated differently depending on "editable" or "standard" mode - in editable mode, just provider dependencies are installed for preinstalled providers in standard mode - those preinstalled providers are dependencies. * the apache-airflow-core distribution contains all airflow core sources (previously in apache-airflow) and it has no provider extras. Thanks to that apache-airflow distribution does not have any dynamically calculated dependencies. * the apache-airflow-core distribution hs "hatch_build_airflow_core.py" build hooks that add custom build target and implement custom cleanup in order to implement compiling assets as part of the build. * During the move, the following changes were applied for consistency: * packages when used in context of distribution packages have been renamed to "distributions" - including all documentations and commands in breeze to void confusion with import packages (see https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/) * all tests in `airflow-core` follow now the same convention where tests are in `unit`, `system` and `integration` package. no extra package has been as second level, because all the provider tests have "<PROVIDER>" there, so we just have to avoid naming airflow unit."<PROVIDER>" with the same name as provider. * all tooling in CI/DEV have been updated to follow the new structure. We should always build to packages now when we are building them using `breeze`.
2025-03-21 14:25:26 +01:00
from in_container_utils import AIRFLOW_DIST_PATH, click, console, run_command
Move airflow sources to airflow-core package (#47798) This is continuation of the separation of the Airflow codebase into separate distributions. This one splits airflow into two of them: * apache-airflow - becomes an empty, meta no-code distribution that only has dependencies to apache-airflow-core and task-sdk distributions and it has preinstalled provider distributions added in standard "wheel" distribution. All "extras" lead either to "apache-airflow-core" extras or to providers - the dependencies and optional dependencies are calculated differently depending on "editable" or "standard" mode - in editable mode, just provider dependencies are installed for preinstalled providers in standard mode - those preinstalled providers are dependencies. * the apache-airflow-core distribution contains all airflow core sources (previously in apache-airflow) and it has no provider extras. Thanks to that apache-airflow distribution does not have any dynamically calculated dependencies. * the apache-airflow-core distribution hs "hatch_build_airflow_core.py" build hooks that add custom build target and implement custom cleanup in order to implement compiling assets as part of the build. * During the move, the following changes were applied for consistency: * packages when used in context of distribution packages have been renamed to "distributions" - including all documentations and commands in breeze to void confusion with import packages (see https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/) * all tests in `airflow-core` follow now the same convention where tests are in `unit`, `system` and `integration` package. no extra package has been as second level, because all the provider tests have "<PROVIDER>" there, so we just have to avoid naming airflow unit."<PROVIDER>" with the same name as provider. * all tooling in CI/DEV have been updated to follow the new structure. We should always build to packages now when we are building them using `breeze`.
2025-03-21 14:25:26 +01:00
ALLOWED_DISTRIBUTION_FORMAT = ["wheel", "sdist", "both"]
def find_airflow_python_client(extension: str):
Move airflow sources to airflow-core package (#47798) This is continuation of the separation of the Airflow codebase into separate distributions. This one splits airflow into two of them: * apache-airflow - becomes an empty, meta no-code distribution that only has dependencies to apache-airflow-core and task-sdk distributions and it has preinstalled provider distributions added in standard "wheel" distribution. All "extras" lead either to "apache-airflow-core" extras or to providers - the dependencies and optional dependencies are calculated differently depending on "editable" or "standard" mode - in editable mode, just provider dependencies are installed for preinstalled providers in standard mode - those preinstalled providers are dependencies. * the apache-airflow-core distribution contains all airflow core sources (previously in apache-airflow) and it has no provider extras. Thanks to that apache-airflow distribution does not have any dynamically calculated dependencies. * the apache-airflow-core distribution hs "hatch_build_airflow_core.py" build hooks that add custom build target and implement custom cleanup in order to implement compiling assets as part of the build. * During the move, the following changes were applied for consistency: * packages when used in context of distribution packages have been renamed to "distributions" - including all documentations and commands in breeze to void confusion with import packages (see https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/) * all tests in `airflow-core` follow now the same convention where tests are in `unit`, `system` and `integration` package. no extra package has been as second level, because all the provider tests have "<PROVIDER>" there, so we just have to avoid naming airflow unit."<PROVIDER>" with the same name as provider. * all tooling in CI/DEV have been updated to follow the new structure. We should always build to packages now when we are building them using `breeze`.
2025-03-21 14:25:26 +01:00
packages = [f.as_posix() for f in AIRFLOW_DIST_PATH.glob(f"apache_airflow_client-[0-9]*.{extension}")]
if len(packages) > 1:
console.print(f"\n[red]Found multiple airflow client packages: {packages}\n")
sys.exit(1)
elif len(packages) == 0:
console.print("\n[red]No airflow client package found\n")
sys.exit(1)
if packages:
console.print(f"\n[bright_blue]Found airflow client package: {packages[0]}\n")
else:
console.print("\n[yellow]No airflow client package found.\n")
return packages[0]
@click.command()
@click.option(
Move airflow sources to airflow-core package (#47798) This is continuation of the separation of the Airflow codebase into separate distributions. This one splits airflow into two of them: * apache-airflow - becomes an empty, meta no-code distribution that only has dependencies to apache-airflow-core and task-sdk distributions and it has preinstalled provider distributions added in standard "wheel" distribution. All "extras" lead either to "apache-airflow-core" extras or to providers - the dependencies and optional dependencies are calculated differently depending on "editable" or "standard" mode - in editable mode, just provider dependencies are installed for preinstalled providers in standard mode - those preinstalled providers are dependencies. * the apache-airflow-core distribution contains all airflow core sources (previously in apache-airflow) and it has no provider extras. Thanks to that apache-airflow distribution does not have any dynamically calculated dependencies. * the apache-airflow-core distribution hs "hatch_build_airflow_core.py" build hooks that add custom build target and implement custom cleanup in order to implement compiling assets as part of the build. * During the move, the following changes were applied for consistency: * packages when used in context of distribution packages have been renamed to "distributions" - including all documentations and commands in breeze to void confusion with import packages (see https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/) * all tests in `airflow-core` follow now the same convention where tests are in `unit`, `system` and `integration` package. no extra package has been as second level, because all the provider tests have "<PROVIDER>" there, so we just have to avoid naming airflow unit."<PROVIDER>" with the same name as provider. * all tooling in CI/DEV have been updated to follow the new structure. We should always build to packages now when we are building them using `breeze`.
2025-03-21 14:25:26 +01:00
"--distribution-format",
default=ALLOWED_DISTRIBUTION_FORMAT[0],
envvar="DISTRIBUTION_FORMAT",
show_default=True,
Move airflow sources to airflow-core package (#47798) This is continuation of the separation of the Airflow codebase into separate distributions. This one splits airflow into two of them: * apache-airflow - becomes an empty, meta no-code distribution that only has dependencies to apache-airflow-core and task-sdk distributions and it has preinstalled provider distributions added in standard "wheel" distribution. All "extras" lead either to "apache-airflow-core" extras or to providers - the dependencies and optional dependencies are calculated differently depending on "editable" or "standard" mode - in editable mode, just provider dependencies are installed for preinstalled providers in standard mode - those preinstalled providers are dependencies. * the apache-airflow-core distribution contains all airflow core sources (previously in apache-airflow) and it has no provider extras. Thanks to that apache-airflow distribution does not have any dynamically calculated dependencies. * the apache-airflow-core distribution hs "hatch_build_airflow_core.py" build hooks that add custom build target and implement custom cleanup in order to implement compiling assets as part of the build. * During the move, the following changes were applied for consistency: * packages when used in context of distribution packages have been renamed to "distributions" - including all documentations and commands in breeze to void confusion with import packages (see https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/) * all tests in `airflow-core` follow now the same convention where tests are in `unit`, `system` and `integration` package. no extra package has been as second level, because all the provider tests have "<PROVIDER>" there, so we just have to avoid naming airflow unit."<PROVIDER>" with the same name as provider. * all tooling in CI/DEV have been updated to follow the new structure. We should always build to packages now when we are building them using `breeze`.
2025-03-21 14:25:26 +01:00
type=click.Choice(ALLOWED_DISTRIBUTION_FORMAT),
help="Package format to use",
)
@click.option(
Move airflow sources to airflow-core package (#47798) This is continuation of the separation of the Airflow codebase into separate distributions. This one splits airflow into two of them: * apache-airflow - becomes an empty, meta no-code distribution that only has dependencies to apache-airflow-core and task-sdk distributions and it has preinstalled provider distributions added in standard "wheel" distribution. All "extras" lead either to "apache-airflow-core" extras or to providers - the dependencies and optional dependencies are calculated differently depending on "editable" or "standard" mode - in editable mode, just provider dependencies are installed for preinstalled providers in standard mode - those preinstalled providers are dependencies. * the apache-airflow-core distribution contains all airflow core sources (previously in apache-airflow) and it has no provider extras. Thanks to that apache-airflow distribution does not have any dynamically calculated dependencies. * the apache-airflow-core distribution hs "hatch_build_airflow_core.py" build hooks that add custom build target and implement custom cleanup in order to implement compiling assets as part of the build. * During the move, the following changes were applied for consistency: * packages when used in context of distribution packages have been renamed to "distributions" - including all documentations and commands in breeze to void confusion with import packages (see https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/) * all tests in `airflow-core` follow now the same convention where tests are in `unit`, `system` and `integration` package. no extra package has been as second level, because all the provider tests have "<PROVIDER>" there, so we just have to avoid naming airflow unit."<PROVIDER>" with the same name as provider. * all tooling in CI/DEV have been updated to follow the new structure. We should always build to packages now when we are building them using `breeze`.
2025-03-21 14:25:26 +01:00
"--use-distributions-from-dist",
is_flag=True,
default=True,
show_default=True,
Move airflow sources to airflow-core package (#47798) This is continuation of the separation of the Airflow codebase into separate distributions. This one splits airflow into two of them: * apache-airflow - becomes an empty, meta no-code distribution that only has dependencies to apache-airflow-core and task-sdk distributions and it has preinstalled provider distributions added in standard "wheel" distribution. All "extras" lead either to "apache-airflow-core" extras or to providers - the dependencies and optional dependencies are calculated differently depending on "editable" or "standard" mode - in editable mode, just provider dependencies are installed for preinstalled providers in standard mode - those preinstalled providers are dependencies. * the apache-airflow-core distribution contains all airflow core sources (previously in apache-airflow) and it has no provider extras. Thanks to that apache-airflow distribution does not have any dynamically calculated dependencies. * the apache-airflow-core distribution hs "hatch_build_airflow_core.py" build hooks that add custom build target and implement custom cleanup in order to implement compiling assets as part of the build. * During the move, the following changes were applied for consistency: * packages when used in context of distribution packages have been renamed to "distributions" - including all documentations and commands in breeze to void confusion with import packages (see https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/) * all tests in `airflow-core` follow now the same convention where tests are in `unit`, `system` and `integration` package. no extra package has been as second level, because all the provider tests have "<PROVIDER>" there, so we just have to avoid naming airflow unit."<PROVIDER>" with the same name as provider. * all tooling in CI/DEV have been updated to follow the new structure. We should always build to packages now when we are building them using `breeze`.
2025-03-21 14:25:26 +01:00
envvar="USE_DISTRIBUTIONS_FROM_DIST",
help="Should install distributions from dist folder if set.",
)
@click.option(
"--github-actions",
is_flag=True,
default=False,
show_default=True,
envvar="GITHUB_ACTIONS",
help="Running in GitHub Actions",
)
Move airflow sources to airflow-core package (#47798) This is continuation of the separation of the Airflow codebase into separate distributions. This one splits airflow into two of them: * apache-airflow - becomes an empty, meta no-code distribution that only has dependencies to apache-airflow-core and task-sdk distributions and it has preinstalled provider distributions added in standard "wheel" distribution. All "extras" lead either to "apache-airflow-core" extras or to providers - the dependencies and optional dependencies are calculated differently depending on "editable" or "standard" mode - in editable mode, just provider dependencies are installed for preinstalled providers in standard mode - those preinstalled providers are dependencies. * the apache-airflow-core distribution contains all airflow core sources (previously in apache-airflow) and it has no provider extras. Thanks to that apache-airflow distribution does not have any dynamically calculated dependencies. * the apache-airflow-core distribution hs "hatch_build_airflow_core.py" build hooks that add custom build target and implement custom cleanup in order to implement compiling assets as part of the build. * During the move, the following changes were applied for consistency: * packages when used in context of distribution packages have been renamed to "distributions" - including all documentations and commands in breeze to void confusion with import packages (see https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/) * all tests in `airflow-core` follow now the same convention where tests are in `unit`, `system` and `integration` package. no extra package has been as second level, because all the provider tests have "<PROVIDER>" there, so we just have to avoid naming airflow unit."<PROVIDER>" with the same name as provider. * all tooling in CI/DEV have been updated to follow the new structure. We should always build to packages now when we are building them using `breeze`.
2025-03-21 14:25:26 +01:00
def install_airflow_python_client(
distribution_format: str, use_distributions_from_dist: bool, github_actions: bool
):
if use_distributions_from_dist and distribution_format not in ["wheel", "sdist"]:
console.print(
f"[red]DISTRIBUTION_FORMAT must be one of 'wheel' or 'sdist' and not {distribution_format}"
)
sys.exit(1)
Move airflow sources to airflow-core package (#47798) This is continuation of the separation of the Airflow codebase into separate distributions. This one splits airflow into two of them: * apache-airflow - becomes an empty, meta no-code distribution that only has dependencies to apache-airflow-core and task-sdk distributions and it has preinstalled provider distributions added in standard "wheel" distribution. All "extras" lead either to "apache-airflow-core" extras or to providers - the dependencies and optional dependencies are calculated differently depending on "editable" or "standard" mode - in editable mode, just provider dependencies are installed for preinstalled providers in standard mode - those preinstalled providers are dependencies. * the apache-airflow-core distribution contains all airflow core sources (previously in apache-airflow) and it has no provider extras. Thanks to that apache-airflow distribution does not have any dynamically calculated dependencies. * the apache-airflow-core distribution hs "hatch_build_airflow_core.py" build hooks that add custom build target and implement custom cleanup in order to implement compiling assets as part of the build. * During the move, the following changes were applied for consistency: * packages when used in context of distribution packages have been renamed to "distributions" - including all documentations and commands in breeze to void confusion with import packages (see https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/) * all tests in `airflow-core` follow now the same convention where tests are in `unit`, `system` and `integration` package. no extra package has been as second level, because all the provider tests have "<PROVIDER>" there, so we just have to avoid naming airflow unit."<PROVIDER>" with the same name as provider. * all tooling in CI/DEV have been updated to follow the new structure. We should always build to packages now when we are building them using `breeze`.
2025-03-21 14:25:26 +01:00
extension = "whl" if distribution_format == "wheel" else "tar.gz"
install_airflow_python_client_cmd = [
Migrated prod builds to use python built from source (#53770) Moved the prod build to also use the src built python as done by CI. This let's us iterate faster on our python versions without needing to wait for the community image of python. This involves a number of changes: * 3.0.5 -> 3.0.6 airflow version change * use released official packages rather than git repo to install Python * added wget (it was added in the original image to pull python packages so added for compatibility) added those flags to python build (same as in the * original build) * --with-ensurepip --build="$gnuArch" * --enable-loadable-sqlite-extension * --enable-option-checking=fatal * --enable-shared * --with-lto * added cleanup of apt after installing packages * added removal of .pyc/.test etc. files (saves 350 MB) * added relinking of symbolic links from /usr/python/bin to /usr/local/bin in the "main" image as well as in the build image. * we do not need AIRFLOW_SETUPTOOLS_VERSION any more - this was only added to upgrade setuptools, because the Python official image had a very old setuptools version. * checked all "customize" scripts and make them "work" * Updated the 'version upgrade" script to upgrade AIRFLOW_PYTHON_VERSION everywhere * Updated Changelog and documentation to update new ways of building images * removed installation with GitHub URL (it won't work easily after splitting to multiple packages - not easily at least) and it's not needed * all python, pip and similar links are created in /usr/python/bin * /usr/python/bin is always first in the PATH - before /usr/local/bin * added changelog entry explaining that Python's installation home has been moved to /usr/python/ from /usr/local * removal of installed editable distributions in breeze happens now first and THEN we install when --use-airflow-version is used. * LD_LIBRARY_PATH was not set so the shared python libraries could not be loaded when venv was created Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
2025-08-30 20:18:30 +05:30
"uv",
"pip",
"install",
find_airflow_python_client(extension),
]
console.print("\n[bright_blue]Installing airflow python client\n")
run_command(install_airflow_python_client_cmd, github_actions=github_actions, check=True)
if __name__ == "__main__":
install_airflow_python_client()