Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
|
|
|
# or more contributor license agreements. See the NOTICE file
|
|
|
|
|
# distributed with this work for additional information
|
|
|
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
|
|
|
# to you under the Apache License, Version 2.0 (the
|
|
|
|
|
# "License"); you may not use this file except in compliance
|
|
|
|
|
# with the License. You may obtain a copy of the License at
|
|
|
|
|
#
|
|
|
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
|
#
|
|
|
|
|
# Unless required by applicable law or agreed to in writing,
|
|
|
|
|
# software distributed under the License is distributed on an
|
|
|
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
|
|
|
# KIND, either express or implied. See the License for the
|
|
|
|
|
# specific language governing permissions and limitations
|
|
|
|
|
# under the License.
|
|
|
|
|
#
|
|
|
|
|
# PEP 723 compliant inline script metadata (not yet widely supported)
|
|
|
|
|
# /// script
|
2025-06-29 17:29:46 +02:00
|
|
|
# requires-python = ">=3.10"
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
# dependencies = [
|
|
|
|
|
# "apache-airflow-client",
|
2025-08-25 21:59:08 +02:00
|
|
|
# "rich>=13.6.0",
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
# ]
|
|
|
|
|
# ///
|
|
|
|
|
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
|
|
|
|
|
import sys
|
2024-11-27 13:14:52 +00:00
|
|
|
import time
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
import uuid
|
|
|
|
|
|
|
|
|
|
import airflow_client.client
|
2024-11-27 13:14:52 +00:00
|
|
|
import pytest
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
|
2025-04-11 18:41:34 +08:00
|
|
|
from tests_common.test_utils.api_client_helpers import generate_access_token
|
2025-03-06 21:23:00 +01:00
|
|
|
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
try:
|
|
|
|
|
# If you have rich installed, you will have nice colored output of the API responses
|
|
|
|
|
from rich import print
|
|
|
|
|
except ImportError:
|
|
|
|
|
print("Output will not be colored. Please install rich to get colored output: `pip install rich`")
|
|
|
|
|
pass
|
2025-05-17 05:17:45 +05:30
|
|
|
from airflow_client.client.api import config_api, dag_api, dag_run_api, task_api
|
|
|
|
|
from airflow_client.client.models.trigger_dag_run_post_body import TriggerDAGRunPostBody
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
|
|
|
|
|
# The client must use the authentication and authorization parameters
|
|
|
|
|
# in accordance with the API server security policy.
|
|
|
|
|
# Examples for each auth method are provided below, use the example that
|
|
|
|
|
# satisfies your auth use case.
|
|
|
|
|
#
|
2025-03-06 21:23:00 +01:00
|
|
|
# The example below use the default FabAuthManager, in case your airflow api server use a different
|
|
|
|
|
# auth manager for instance AwsAuthManagerUser or SimpleAuthManager make sure to generate the token with
|
|
|
|
|
# appropriate AuthManager.
|
|
|
|
|
# This is defined in the `[api]` section of your `airflow.cfg`:
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
#
|
2025-03-07 23:40:01 +00:00
|
|
|
# auth_manager = airflow.api_fastapi.auth.managers.simple.simple_auth_manager.SimpleAuthManager
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
#
|
|
|
|
|
# Make sure that your user/name are configured properly - using the user/password that has admin
|
|
|
|
|
# privileges in Airflow
|
|
|
|
|
|
2025-03-06 21:23:00 +01:00
|
|
|
# Used to initialize FAB and the auth manager, necessary for creating the token.
|
|
|
|
|
|
2025-03-20 09:15:19 -04:00
|
|
|
|
2025-04-11 18:41:34 +08:00
|
|
|
access_token = generate_access_token("admin", "admin", "localhost:8080")
|
2025-05-20 22:26:50 +05:30
|
|
|
configuration = airflow_client.client.Configuration(host="http://localhost:8080", access_token=access_token)
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
|
|
|
|
|
# Make sure in the [core] section, the `load_examples` config is set to True in your airflow.cfg
|
|
|
|
|
# or AIRFLOW__CORE__LOAD_EXAMPLES environment variable set to True
|
2025-04-30 21:45:09 +02:00
|
|
|
DAG_ID = "example_simplest_dag"
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
|
2024-11-27 13:14:52 +00:00
|
|
|
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
# Enter a context with an instance of the API client
|
2024-11-27 13:14:52 +00:00
|
|
|
@pytest.mark.execution_timeout(400)
|
|
|
|
|
def test_python_client():
|
2025-05-20 22:26:50 +05:30
|
|
|
with airflow_client.client.ApiClient(configuration) as api_client:
|
2024-11-27 13:14:52 +00:00
|
|
|
errors = False
|
|
|
|
|
|
|
|
|
|
print("[blue]Getting DAG list")
|
|
|
|
|
max_retries = 10
|
|
|
|
|
while max_retries > 0:
|
|
|
|
|
try:
|
|
|
|
|
dag_api_instance = dag_api.DAGApi(api_client)
|
|
|
|
|
api_response = dag_api_instance.get_dags()
|
|
|
|
|
except airflow_client.client.OpenApiException as e:
|
|
|
|
|
print(f"[red]Exception when calling DagAPI->get_dags: {e}\n")
|
|
|
|
|
errors = True
|
|
|
|
|
time.sleep(6)
|
|
|
|
|
max_retries -= 1
|
|
|
|
|
else:
|
|
|
|
|
print("[green]Getting DAG list successful")
|
|
|
|
|
break
|
|
|
|
|
|
|
|
|
|
print("[blue]Getting Tasks for a DAG")
|
|
|
|
|
try:
|
2025-05-17 05:17:45 +05:30
|
|
|
task_api_instance = task_api.TaskApi(api_client)
|
|
|
|
|
api_response = task_api_instance.get_tasks(DAG_ID)
|
2024-11-27 13:14:52 +00:00
|
|
|
print(api_response)
|
|
|
|
|
except airflow_client.client.exceptions.OpenApiException as e:
|
|
|
|
|
print(f"[red]Exception when calling DagAPI->get_tasks: {e}\n")
|
|
|
|
|
errors = True
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
else:
|
2024-11-27 13:14:52 +00:00
|
|
|
print("[green]Getting Tasks successful")
|
|
|
|
|
|
|
|
|
|
print("[blue]Triggering a DAG run")
|
2025-05-17 05:17:45 +05:30
|
|
|
dag_run_api_instance = dag_run_api.DagRunApi(api_client)
|
2024-11-27 13:14:52 +00:00
|
|
|
try:
|
|
|
|
|
# Create a DAGRun object (no dag_id should be specified because it is read-only property of DAGRun)
|
|
|
|
|
# dag_run id is generated randomly to allow multiple executions of the script
|
2025-05-17 05:17:45 +05:30
|
|
|
dag_run = TriggerDAGRunPostBody(
|
2024-11-27 13:14:52 +00:00
|
|
|
dag_run_id="some_test_run_" + uuid.uuid4().hex,
|
2025-03-06 21:23:00 +01:00
|
|
|
logical_date=None,
|
2024-11-27 13:14:52 +00:00
|
|
|
)
|
2025-05-17 05:17:45 +05:30
|
|
|
api_response = dag_run_api_instance.trigger_dag_run(DAG_ID, dag_run)
|
2024-11-27 13:14:52 +00:00
|
|
|
print(api_response)
|
|
|
|
|
except airflow_client.client.exceptions.OpenApiException as e:
|
Generate Python client in reproducible way (#36763)
Client source code and package generation was done using the code
generated and committed to `airflow-client-python` and while the
repository with such code is useful to have, it's just a convenience
repo, because all sources are (and should be) generated from the
API specification which is present in the Airflow repository.
This also made the reproducible builds and package generation not really
possible, because we never knew if the source generated in the
`airflow-client-python` repository has been generated and not tampered
with.
While implementing it, it turned out that there were some issues in
the past that nade our client generation somewhat broken..
* In 2.7.0 python client, we added the same code twice
(See https://github.com/apache/airflow-client-python/pull/93) on
top of "airflow_client.client" package, we also added copy of the
API client generated in "airflow_client.airflow_client" - that was
likely due to bad bash scripts and tools that were used to generate
it and errors during generation the clients.
* We used to generate the code for "client" package and then moved
the "client" package to "airflow_client.client" package, while
manually modifying imports with `sed` (!?). That was likely due to
limitations in some old version of the client generator. However the
client generator we use now is capable of generating code directly in
the "airflow_client.client" package.
* We also manually (via pre-commit) added Apache Licence to the
generated files. Whieh was completely unnecessary, because ASF rules
do not require licence headers to be added to code automatically
generated from a code that already has ASF licence.
* We also generated source tarball packages from such generated code,
which was completely unnecessary - because sdist packages are already
fulfilling all the reqirements of such source pacakges - the code
in the packages is enough to build the package from the sources and
it does not contain any binary code, moreover the code is generated
out of the API specificiation, which means that anyone can take
the code and genearate the pacakged software from just sources in
sdist. Similarly as in case of provider packages, we do not need
to produce separate -source.tar.gz files.
This PR fixes all of it.
First of all the source that lands in the source repository
`airflow-client-python` and sdist/wheel packages are generated directly
from the openapi specification.
They are generated using breeze release_management command from airflow
source tagged with specific tag in the Airflow repo (including the
source of reproducible build date that is updated together with airflow
release notes. This means that any PMC member can regenerate packages
(binary identical) straight from the Airflow repository - without
going through "airflow-client-python" repository.
No source tarball is generated - it is not needed, sdist is enough.
The `test_python_client.py` has been also moved over to Airflow repo
and updated with handling case when expose_config is not enabled and
it is used to automatically test the API client after it has been
generated.
2024-01-14 18:17:20 +01:00
|
|
|
print(f"[red]Exception when calling DAGRunAPI->post_dag_run: {e}\n")
|
2025-03-06 21:23:00 +01:00
|
|
|
errors = True
|
2024-11-27 13:14:52 +00:00
|
|
|
else:
|
|
|
|
|
print("[green]Posting DAG Run successful")
|
|
|
|
|
|
|
|
|
|
# Get current configuration. Note, this is disabled by default with most installation.
|
|
|
|
|
# You need to set `expose_config = True` in Airflow configuration in order to retrieve configuration.
|
|
|
|
|
conf_api_instance = config_api.ConfigApi(api_client)
|
|
|
|
|
try:
|
|
|
|
|
api_response = conf_api_instance.get_config()
|
|
|
|
|
print(api_response)
|
|
|
|
|
except airflow_client.client.OpenApiException as e:
|
2025-03-06 21:23:00 +01:00
|
|
|
if "Your Airflow administrator chose" in str(e):
|
2024-11-27 13:14:52 +00:00
|
|
|
print(
|
|
|
|
|
"[yellow]You need to set `expose_config = True` in Airflow configuration"
|
|
|
|
|
" in order to retrieve configuration."
|
|
|
|
|
)
|
|
|
|
|
print("[bright_blue]This is OK. Exposing config is disabled by default.")
|
|
|
|
|
else:
|
|
|
|
|
print(f"[red]Exception when calling DAGRunAPI->post_dag_run: {e}\n")
|
|
|
|
|
errors = True
|
|
|
|
|
else:
|
|
|
|
|
print("[green]Config retrieved successfully")
|
|
|
|
|
|
|
|
|
|
if errors:
|
|
|
|
|
print("\n[red]There were errors while running the script - see above for details")
|
|
|
|
|
sys.exit(1)
|
|
|
|
|
else:
|
|
|
|
|
print("\n[green]Everything went well")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
if __name__ == "__main__":
|
|
|
|
|
test_python_client()
|
