SIGN IN SIGN UP
apache / airflow UNCLAIMED

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

44796 0 0 Python

Switch all airflow logging to structlog (#52651)

This change continues the work that was started in Airflow 3.0 and AIP-72 to
use Structlog in place of stdlib logging in Airflow.

The primary change here is to make LoggingMixin return a customized
structlogger; customized to maintain surface compatability with logging.Logger
(surface meaning things like handlers and filters aren't preserved) and to
then capture/redirect all logging via stdlib via Structlog processors.

This is the first step in allowing all Airflow components to be able to
produce JSON logs natively.

Things of note in this implementation:

- We have a customized structlog filtering logger that has a "per-logger-tree"
  level concept.

  This is implemented using a prefix trie[1] to lookup the logging level to
  efficiently be able to look up the level for child loggers when configured
  only at the parent level.

- We have a custom `PercentFormatRender` class that renders non-JSON logs that
  understands the stdlib style format strings, meaning users custom logging
  config will be respected again for the daemon components.

  (Note though: this won't help with Task logs, as those are always JSON and
  the UI does the rendering of those.)

- There is no longer a need for a different log format for colored and plain -- using
  color format specifiers (`%(blue)s`, `%(log_level)s` etc) when colors are
  disabled/not available will output nothing in their place.

- Introduce an mechanism for users to easily set the log level for loggers --
  for instance if you are debugging the scheduler, it would be nice to be able
  to set the `airflow.jobs.scheduler_job_runner` logger to DEBUG while keeping
  everything else at info.

[1]: https://en.wikipedia.org/wiki/Trie


The reason for not using caplog has gone away with the switch to structlog,
and we already override the builtin `caplog` fixture to use our structlog
version
A
Ash Berlin-Taylor committed
36fc820f27a25df2997cc2c85010d5caa02e8211
Parent: cde507a
Committed by GitHub <noreply@github.com> on 9/9/2025, 4:14:31 PM