Apache Airflow

wikipedia:Apache Airflow Workflow management system is a platform to programmatically author, schedule, and monitor workflows.

• https://airflow.apache.org/

• Cloud Composer
• Installation

Commands

• airflow --help
• airflow db
• airflow webserver
• airflow version

Logs

• Started process
• Fluentd recommended for logs
• StatsD recommended for metrics to Prometheus

Related

• Directed acyclic graph (DAG)
• Airflow Scheduler
• Amazon Managed Workflows for Apache Airflow (MWAA)
• Apache Airflow providers: https://airflow.apache.org/docs/apache-airflow-providers/

/etc/metricbeat/metricbeat.reference.yml
- module: airflow
  host: "localhost"
  port: "8126"
  metricsets: [ 'statsd' ]

Activities

• Read https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag/

See also

• Astronomer and Astronomer Cloud
• Cloud composer service on Google Cloud Platform
• airflow [ info | celery | config | db upgrad | config list | db init | scheduler | webserver | users | dags | kubernetes | pools | roles | tasks | variables | cheat-sheet | kereberos | plugins | rotate-fernet-key | sync-perm ] airflow --help
• Apache Airflow, Installation, configuration airflow.cfg, monitoring, Amazon Managed Workflows for Apache Airflow (MWAA), airflow, Airflow Scheduler, providers, Workflows, Airflow executors, Airflow versions, Datadog integration
• DAG, Airflow
• ETL, AWS Glue, Fivetran, Apache Airflow, Amazon Kinesis Data Firehose, SnapLogic, Hightouch