Difference between revisions of "Apache Airflow"
Jump to navigation
Jump to search
| Line 35: | Line 35: | ||
== Activities == | == Activities == | ||
* Read https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag/ | * Read https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag/ | ||
| + | * [[Datadog Airflow integration]] | ||
== See also == | == See also == | ||
Revision as of 14:38, 1 December 2022
wikipedia:Apache Airflow Workflow management system is a platform to programmatically author, schedule, and monitor workflows.
Contents
Commands
Logs
- Started process
- Fluentd recommended for logs
- StatsD recommended for metrics to Prometheus
Related
- Directed acyclic graph (DAG)
- Drawing DAGs
- Airflow Scheduler
- Airflow Operators
- Job scheduler
- Apache NiFi
- Amazon Managed Workflows for Apache Airflow (MWAA) (May 2021))
- Apache Airflow providers: https://airflow.apache.org/docs/apache-airflow-providers/
/etc/metricbeat/metricbeat.reference.yml - module: airflow host: "localhost" port: "8126" metricsets: [ 'statsd' ]
Activities
- Read https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag/
- Datadog Airflow integration
See also
- Astronomer and Astronomer Cloud
- Cloud composer service on Google Cloud Platform
airflow [ info | celery | config | db upgrad | config list | db init | scheduler | webserver | users | dags | kubernetes | pools | roles | tasks | variables | cheat-sheet | kereberos | plugins | rotate-fernet-key | sync-perm ] airflow --help- Apache Airflow, Installation, configuration
airflow.cfg, monitoring, Amazon Managed Workflows for Apache Airflow (MWAA),airflow, Airflow Scheduler, providers, Workflows, Airflow executors, Airflow versions, Datadog integration - DAG, Airflow
- ETL, AWS Glue, Fivetran, Apache Airflow, Amazon Kinesis Data Firehose, SnapLogic, Hightouch
Advertising:
