Apache Airflow

wikipedia:Apache Airflow Workflow management system is a platform to programmatically author, schedule, and monitor workflows.

• https://airflow.apache.org/

• Cloud Composer
• Installation:

pip3 install apache-airflow

There is now homebrew package available

helm repo add apache-airflow

Commands

• airflow db

Related

• Directed acyclic graph (DAG)
• Airflow Scheduler
• Amazon Managed Workflows for Apache Airflow (MWAA)
• Apache Airflow providers: https://airflow.apache.org/docs/apache-airflow-providers/

/etc/metricbeat/metricbeat.reference.yml
- module: airflow
  host: "localhost"
  port: "8126"
  metricsets: [ 'statsd' ]

See also

• Astronomer and Astronomer Cloud
• Cloud composer service on Google Cloud Platform
• Apache Airflow, Installation, configuration airflow.cfg, monitoring, Amazon Managed Workflows for Apache Airflow (MWAA), airflow, Airflow Scheduler, providers, Workflows, Airflow executors, Airflow versions, Datadog integration
• DAG, Airflow