The SPDX licence should not have any extra characters after the
specification, moving the URL to a separate line is better in
context of automated parsers.
While we have some discussion on-going whether we should use some
shorter, machine-readable friendly versions of licence specification
in our source code headers here [1], the notion is that:
a) PMC can make judgment calls when to include different versions of
the licence
b) This expectation only applies to the code we actually release
in our official releases.
This change makes some judgment call on using much shorter, SPDX
driven licence headers in some specific files:
* markdown files that are intended to be consumed by agents
(AGENTS.md, SKILLS.md, CLAUDE.md and so on)
* all the markdown .github/* files that are clearly meta-data for
GitHub and which we exclude from released sources
We also make sure all those files are excluded from the official
source releases and distribution packages we prepare.
[1] https://lists.apache.org/thread/j1tn63r2lf13v3d1tnnqff8fkcl4nx53
This is continuation of the separation of the Airflow codebase into
separate distributions. This one splits airflow into two of them:
* apache-airflow - becomes an empty, meta no-code distribution that
only has dependencies to apache-airflow-core and task-sdk
distributions and it has preinstalled provider distributions
added in standard "wheel" distribution. All "extras" lead
either to "apache-airflow-core" extras or to providers - the
dependencies and optional dependencies are calculated differently
depending on "editable" or "standard" mode - in editable mode,
just provider dependencies are installed for preinstalled providers
in standard mode - those preinstalled providers are dependencies.
* the apache-airflow-core distribution contains all airflow core
sources (previously in apache-airflow) and it has no provider
extras. Thanks to that apache-airflow distribution does not
have any dynamically calculated dependencies.
* the apache-airflow-core distribution hs "hatch_build_airflow_core.py"
build hooks that add custom build target and implement custom
cleanup in order to implement compiling assets as part of the build.
* During the move, the following changes were applied for consistency:
* packages when used in context of distribution packages have been
renamed to "distributions" - including all documentations and
commands in breeze to void confusion with import packages
(see
https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/)
* all tests in `airflow-core` follow now the same convention
where tests are in `unit`, `system` and `integration` package.
no extra package has been as second level, because all the
provider tests have "<PROVIDER>" there, so we just have to avoid
naming airflow unit."<PROVIDER>" with the same name as provider.
* all tooling in CI/DEV have been updated to follow the new
structure. We should always build to packages now when we
are building them using `breeze`.
Following #36936 and the fact that GitHub stopped rendering big .rst
files, we also split CONTRIBUTING.rst into multiplet files. It will be
much easier to follow and it will render in GitHub.
Our security policy was quite a bit too long and to unstructured
to provide good guideline for the Security Reporters. It was basically
a wall of text where - even if you wanted - you could not easily find
the part that you could be really interested in - for example assessing
the severity of the issue or what happens after you reported the issue.
The change is also not about the structure but also about the content
that resulted from a number of discussions we had with securtity
reporters - answering straight away the questions they had - for
example how long they should wait for answer, how should they
monitor if the CVE has been announced etc. etc.
Having a structured form also allows to link to the particular
sections of the policy in case someone had not read and follow it -
having it structured and linkable, make it much more fair to a) expect
they will be read, b) point out (including pointing out negligence
by the reporter) to the particular section. That sounds fair and
reasonable if made it clear and directly answering questions the
reporters might have.
* Add back link to the ASF blog about severity to the policy
The security policy should be the place where researchers are
looking on how to assign severity to their reports. We had the
link to the ASF blog post decribing how we assess the severity
but it has been moved out in #32496 somewhat accidentally to the
information about the security team. It can stay there (as a
reference for the security team members/internal, but it would
be great to keep it in our Policy targeted for the researchers.
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
* Update security model of Airflow
This change updates the security model of Airflow to better explain
what are the capabilities of various kinds of users in Airflow
deployments and give both users and security researchers a way to
understand what security measures they can take and whether they
can qualify potential security issues in Airflow properly - taking
into account that various users of Airflow have various capabilities
and behaviours considered by some of the users as security
vulnerabilities, are standard capabilities of the users.
It also splits the security information of ours in two separate
pages:
* .github/SECURITY.md where we explain how to report the issues
to Apache Airflow security team by the researchers
* documentation security/index.html which is available via
Airflow Website where we explain what our security model is
and the different kinds of users we have.
Both serve slightly different purpose and both contain cross-reference
links to each other in order to be able to redirect people who read
about the security model to find out how they can report the issues
but also to guide security researchers who want to assess whether
their findings are real vulnerabilities, or rather normal behaviours
following the Airflow Security model.
Security has been also moved to be a top level topic, so that it
is easier to find and navigate to. Old links have been redirected
to the new locations.
Also chapters were added explaining Airflow vs. Providers security
releases, what is the relation between Airflow and Providers
security issues and how users should treat security announcements
in providers.
* Update .github/SECURITY.md
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
* Apply Niko's suggestions from code review
Co-authored-by: Niko Oliveira <onikolas@amazon.com>
* fixup! Apply Niko's suggestions from code review
---------
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Niko Oliveira <onikolas@amazon.com>
