Add gunicorn support for API server with rolling worker restarts (#60940)
Add optional gunicorn server type for the API server that provides: - Memory sharing via preload + fork copy-on-write - Rolling worker restarts through GunicornMonitor - Correct FIFO signal handling (SIGTTOU kills oldest worker) New configuration options in [api] section: - server_type: uvicorn (default) or gunicorn - worker_refresh_interval: seconds between refresh cycles (0=disabled) - worker_refresh_batch_size: workers to refresh per cycle - master_timeout: gunicorn master timeout - reload_on_plugin_change: reload on plugin file changes Requires apache-airflow-core[gunicorn] extra for gunicorn mode. * Run GunicornMonitor in main thread instead of daemon thread Matches Airflow 2's webserver pattern: monitor runs in main thread, so if it crashes, the whole process exits (fail-fast). No silent degradation where gunicorn keeps running without worker recycling. Also triggers monitor when reload_on_plugin_change is enabled, even if worker_refresh_interval is 0. * Refactor gunicorn support to use custom Arbiter instead of external monitor This refactor changes the gunicorn worker monitoring architecture from an external thread-based approach to using a custom Arbiter subclass, which is gunicorn's recommended extension pattern. Changes: - New gunicorn_app.py with AirflowArbiter and AirflowGunicornApp - AirflowArbiter integrates worker refresh into manage_workers() loop - Removed gunicorn_monitor.py (no longer needed) - Simplified api_server_command.py (no subprocess, direct gunicorn API) - Updated tests for new architecture Benefits: - Simpler architecture (no separate thread or subprocess) - Direct access to worker state via self.WORKERS - Uses gunicorn's internal spawn_worker/kill_worker methods - Follows gunicorn's documented extension pattern
K
Kaxil Naik committed
b906080fe82cd72c9024dd62cf1b64c15f403ed8
Parent: d12f444
Committed by GitHub <noreply@github.com>
on 2/4/2026, 9:45:09 AM