Difference between revisions of "Apache Airflow"
Jump to navigation
Jump to search
| (52 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
| − | [[wikipedia:Apache Airflow]] [[Workflow management system]] | + | [[wikipedia:Apache Airflow]] [[Workflow management system]] is a platform to programmatically author, schedule, and monitor workflows. |
* https://airflow.apache.org/ | * https://airflow.apache.org/ | ||
* [[Cloud Composer]] | * [[Cloud Composer]] | ||
| + | * [[Apache Airflow installation|Installation]] | ||
| + | == Commands == | ||
| + | * <code>[[airflow --help]]</code> | ||
| + | * <code>[[airflow db]]</code> | ||
| + | * <code>[[airflow webserver]]</code> | ||
| + | * <code>[[airflow standalone]]</code> | ||
| + | * <code>[[airflow version]]</code> | ||
| + | |||
| + | == Logs == | ||
| + | * [[Started process]] | ||
| + | * [[Fluentd]] recommended for logs | ||
| + | * [[StatsD]] recommended for metrics to [[Prometheus]] | ||
== Related == | == Related == | ||
| − | * [[Directed acyclic graph]] | + | * [[Directed acyclic graph (DAG)]] |
| − | * [[Amazon Managed Workflows for Apache Airflow (MWAA)]] | + | * [[Drawing DAGs]] |
| + | * [[Airflow Scheduler]] | ||
| + | * [[Airflow Operators]] | ||
| + | * [[Job scheduler]] | ||
| + | * [[Apache NiFi]] | ||
| + | * [[Amazon Managed Workflows for Apache Airflow (MWAA)]] (May 2021)) | ||
| + | * [[Apache Airflow providers]]: https://airflow.apache.org/docs/apache-airflow-providers/ | ||
| + | |||
| + | [[/etc/metricbeat/metricbeat.reference.yml]] | ||
| + | - module: airflow | ||
| + | host: "localhost" | ||
| + | port: "8126" | ||
| + | metricsets: [ '[[statsd]]' ] | ||
| + | * [[Apache airflow metrics]] | ||
| + | * [[Running Airflow in Docker]] | ||
| + | |||
| + | == Activities == | ||
| + | * Read https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag/ | ||
| + | * [[Datadog Airflow integration]] | ||
== See also == | == See also == | ||
* [[Astronomer]] and [[Astronomer Cloud]] | * [[Astronomer]] and [[Astronomer Cloud]] | ||
* [[Cloud composer]] service on [[Google Cloud Platform]] | * [[Cloud composer]] service on [[Google Cloud Platform]] | ||
| + | * {{airflow cmd}} | ||
* {{Airflow}} | * {{Airflow}} | ||
| − | |||
| − | [[Category: | + | [[Category:Apache]] |
| + | [[Category:Airflow]] | ||
Latest revision as of 17:07, 18 January 2023
wikipedia:Apache Airflow Workflow management system is a platform to programmatically author, schedule, and monitor workflows.
Contents
Commands[edit]
Logs[edit]
- Started process
- Fluentd recommended for logs
- StatsD recommended for metrics to Prometheus
Related[edit]
- Directed acyclic graph (DAG)
- Drawing DAGs
- Airflow Scheduler
- Airflow Operators
- Job scheduler
- Apache NiFi
- Amazon Managed Workflows for Apache Airflow (MWAA) (May 2021))
- Apache Airflow providers: https://airflow.apache.org/docs/apache-airflow-providers/
/etc/metricbeat/metricbeat.reference.yml - module: airflow host: "localhost" port: "8126" metricsets: [ 'statsd' ]
Activities[edit]
- Read https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag/
- Datadog Airflow integration
See also[edit]
- Astronomer and Astronomer Cloud
- Cloud composer service on Google Cloud Platform
airflow [ info | celery | config | db upgrad | config list | db init | scheduler | webserver | users | dags | kubernetes | pools | roles | tasks | variables | cheat-sheet | kereberos | plugins | rotate-fernet-key | sync-perm ] airflow --help- Apache Airflow, Installation, configuration
airflow.cfg, monitoring, Amazon Managed Workflows for Apache Airflow (MWAA),airflow, Airflow Scheduler, providers, Workflows, Airflow executors, Airflow versions, Datadog integration
Advertising:
