Switch all airflow logging to structlog (#52651)
This change continues the work that was started in Airflow 3.0 and AIP-72 to use Structlog in place of stdlib logging in Airflow. The primary change here is to make LoggingMixin return a customized structlogger; customized to maintain surface compatability with logging.Logger (surface meaning things like handlers and filters aren't preserved) and to then capture/redirect all logging via stdlib via Structlog processors. This is the first step in allowing all Airflow components to be able to produce JSON logs natively. Things of note in this implementation: - We have a customized structlog filtering logger that has a "per-logger-tree" level concept. This is implemented using a prefix trie[1] to lookup the logging level to efficiently be able to look up the level for child loggers when configured only at the parent level. - We have a custom `PercentFormatRender` class that renders non-JSON logs that understands the stdlib style format strings, meaning users custom logging config will be respected again for the daemon components. (Note though: this won't help with Task logs, as those are always JSON and the UI does the rendering of those.) - There is no longer a need for a different log format for colored and plain -- using color format specifiers (`%(blue)s`, `%(log_level)s` etc) when colors are disabled/not available will output nothing in their place. - Introduce an mechanism for users to easily set the log level for loggers -- for instance if you are debugging the scheduler, it would be nice to be able to set the `airflow.jobs.scheduler_job_runner` logger to DEBUG while keeping everything else at info. [1]: https://en.wikipedia.org/wiki/Trie The reason for not using caplog has gone away with the switch to structlog, and we already override the builtin `caplog` fixture to use our structlog version
A
Ash Berlin-Taylor committed
36fc820f27a25df2997cc2c85010d5caa02e8211
Parent: cde507a
Committed by GitHub <noreply@github.com>
on 9/9/2025, 4:14:31 PM