Airflow
To Manage Data Pipelines
Presented by Mahendra Yadav
Mahendra Yadav
Data Engineer at Azri Solutions
- Twitter: https://www.twitter.com/@userimack/
- Github: http://github.com/userimack/
- Blog: https://userimack.wordpress.com/
Introduction to Airflow
Airflow is a platform to programmatically author, schedule and monitor data pipelines.
Data Pipeline
Existing challenges
- Hard to debug and maintain.
- Lack of stats
- Logging
- Retrying
Main Advantages
Stats
Logs
Handle task failures
Alert on failures
Monitoring of tasks over time
Pipelines written in Python - Code View 1
Pipelines written in Python - Code View 2
Basic Concepts
Workflow as DAG
- DAG (Directed Acyclic Graph) is a collection of all the tasks
- Reflects the relationships and dependencies between the tasks
- Addition of new DAG is very easy.
Tasks as Operators
- BashOperator
- PythonOperator
- PostgresOperator
- MySqlOperator
- ExternalTaskSensor
- EmailOperator
- SlackOperator
- And many more...
Scheduling the DAGs
DAGs can be scheduled to run at certain frequency by providing the parameter schedule interval.
- Cron expression
- Python datetime object
Executors
Executor is responsible for execution of the jobs.
- Sequential (Default)
- Local
- Celery
DAG Anatomy
Default Arguments
Instantiation
Tasks
Airflow in practice - case study
ETL Pipelines
Download Data
↓
Process
↓
Load
↓
Send Report
☺
Demo
Who uses Airflow
Resources
Thank You!
Mahendra Yadav
- Email: mahendra@azrisolutions.com
- Github: http://github.com/userimack/
- Blog: https://userimack.wordpress.com/
- Linkedin: https://www.linkedin.com/in/userimack/
- Twitter: https://www.twitter.com/@userimack/
- Slides: https://userimack.github.io/airflow_slides