Astronomer, the leader in modern data orchestration and the driving force behind Apache Airflow, and Apache are announcing the release of Airflow 2.8. Building off a collection of smaller releases, Airflow 2.8 brings more than 20 new features, over 60 improvements, and over 50 bug fixes. The latest iteration of Airflow buckles down on four major themes: expanding the DAG authoring experience, improving logging, enhancing daily operations, and building a foundation for later releases.
Airflow 2.8 launches the usage of AIP 58, also known as the Airflow ObjectStore. Acting as an abstraction layer over common object stores—including LocalFilesystem, S3, GCS, and Azure Blob Storage—Airflow ObjectStore enables users to interact with files stored in any of these object stores with the same code. Meaning, users no longer have to depend on multiple specific operators, ultimately reducing maintenance workloads when, for example, transferring files.
Astronomer notes that Airflow ObjectStore is in an experimental stage and is likely subject to change.
Airflow 2.8 also improves upon another experimental feature—Airflow listeners—which implements an event subscription function for specific events occurring in an Airflow environment.
Now, users can utilize a listener hook for datasets, which allows users to subscribe to dataset creation and update events. This is especially useful for sending alerts when several datasets have been updated, or for triggering an external process based on dataset events, according to Kenten Danas, manager, developer relations at Astronomer.
Airflow 2.8 introduces improvements to logging capabilities, directly addressing the issue of Airflow tasks failing due to something external, and the subsequent debugging. With TaskContextLogger, component logs are forwarded to the task logs, enabling users to figure out the root of failure without leaving the Airflow UI.
Speaking of UI, Airflow 2.8 ushers in a variety of UI updates to improve the end user experience, including:
- Consolidation of actions into views where users spend their time
- Clear downstream task instances via List Task Instances view
- Improved usability for key features, such as the Trigger DAG run UI
These updates represent a small collection of the features and improvements brought with Airflow 2.8 To learn more, you can view the full release notes, or visit https://www.astronomer.io/