Apache Airflow

wikipedia:Apache Airflow Workflow management system is a platform to programmatically author, schedule, and monitor workflows.

• https://airflow.apache.org/

• Cloud Composer
• Installation:
  • PIP:
    • pip3 install apache-airflow
      • pip install apache-airflow-providers-elasticsearch
      • pip3 install apache-airflow[celery]
  • Homebrew: not available
  • Configuration
  • Kubernetes:

helm repo add apache-airflow https://airflow.apache.org

helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace

Commands

• airflow --help
• airflow db

Logs

• Started process

Related

• Directed acyclic graph (DAG)
• Airflow Scheduler
• Amazon Managed Workflows for Apache Airflow (MWAA)
• Apache Airflow providers: https://airflow.apache.org/docs/apache-airflow-providers/

/etc/metricbeat/metricbeat.reference.yml
- module: airflow
  host: "localhost"
  port: "8126"
  metricsets: [ 'statsd' ]

Activities

• Read https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag/

See also

• Astronomer and Astronomer Cloud
• Cloud composer service on Google Cloud Platform
• airflow [ info | celery | config | db upgrad | config list | db init | scheduler | webserver | users | dags | kubernetes | pools | roles | tasks | variables | cheat-sheet | kereberos | plugins | rotate-fernet-key | sync-perm ] airflow --help
• Apache Airflow, Installation, configuration airflow.cfg, monitoring, Amazon Managed Workflows for Apache Airflow (MWAA), airflow, Airflow Scheduler, providers, Workflows, Airflow executors, Airflow versions, Datadog integration
• DAG, Airflow