Apache Airflow is an open-source mission for scheduling and managing workflows, written in Python.
Kaxil Naik, director of Airflow engineering at Astronomer and one of many core committers of Airflow, informed SD Occasions: “It’s used to automate your each day jobs or each day duties, and duties could be so simple as working a Python script or it may be as difficult as bringing in all the info from 500 completely different knowledge warehouses and manipulating it.”
It was created at Airbnb in 2014 and is about to have a good time its 10 12 months anniversary later this 12 months. It joined the Apache Software program Basis in March 2016 on the Incubation stage and was made a top-level mission in 2019.
Airflow was initially designed for simply ETL use circumstances, however has through the years developed so as to add options that make it helpful for all elements associated to knowledge engineering.
“It has continued to be the chief on this house, as a result of we now have maintained stability between innovation and stability. Due to this virtually 10 years of Airflow in the identical house, we now have added so many options that permit Airflow to be very dependable and secure,” he stated.
The latest launch, 2.9, got here out earlier this week and added new options like the power to mix dataset and time-based schedules, customized names for Dynamic Process Mapping, and the power to group process logs.
The mission could be discovered on GitHub right here.
