Difference between revisions of "Datadog Airflow integration"
Jump to navigation
Jump to search
(9 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
− | |||
* [[Kubernetes]]: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#datadog | * [[Kubernetes]]: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#datadog | ||
* Windows/Linux https://docs.datadoghq.com/integrations/airflow/ | * Windows/Linux https://docs.datadoghq.com/integrations/airflow/ | ||
* Official documentation with <code>[[dogstatsd_mapper_profile:]]</code> configuration for the agent: https://app.datadoghq.eu/integrations/airflow | * Official documentation with <code>[[dogstatsd_mapper_profile:]]</code> configuration for the agent: https://app.datadoghq.eu/integrations/airflow | ||
+ | * https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html | ||
<code>[[statsd_datadog_enabled]]</code> | <code>[[statsd_datadog_enabled]]</code> | ||
Line 10: | Line 10: | ||
== Airflow monitoring == | == Airflow monitoring == | ||
{{airflow monitoring}} | {{airflow monitoring}} | ||
+ | |||
+ | |||
+ | == Logs == | ||
+ | |||
+ | [[triggerer]] | [2022-12-05 14:02:35,051] {base.py:317} INFO - [[Statsd]] buffering is disabled | ||
+ | triggerer | [2022-12-05 14:02:35,052] {base.py:335} INFO - Statsd periodic buffer flush is disabled | ||
+ | triggerer | [2022-12-05 14:02:35,053] {base.py:317} INFO - Statsd buffering is disabled | ||
+ | triggerer | [2022-12-05 14:02:35,053] {base.py:335} INFO - Statsd periodic buffer flush is disabled | ||
+ | |||
+ | [[scheduler]] | [2022-12-05 14:02:35,223] {base.py:317} INFO - Statsd buffering is disabled | ||
+ | scheduler | [2022-12-05 14:02:35,234] {base.py:335} INFO - Statsd periodic buffer flush is disabled | ||
+ | scheduler | [2022-12-05 14:02:35,235] {base.py:317} INFO - Statsd buffering is disabled | ||
+ | scheduler | [2022-12-05 14:02:35,235] {base.py:335} INFO - Statsd periodic buffer flush is disabled | ||
+ | |||
+ | [[webserver]] | [2022-12-05 14:02:36,135] {base.py:317} INFO - Statsd buffering is disabled | ||
+ | webserver | [2022-12-05 14:02:36,135] {base.py:335} INFO - Statsd periodic buffer flush is disabled | ||
+ | webserver | [2022-12-05 14:02:36,136] {base.py:317} INFO - Statsd buffering is disabled | ||
+ | webserver | [2022-12-05 14:02:36,136] {base.py:335} INFO - Statsd periodic buffer flush is disabled | ||
+ | |||
+ | |||
+ | |||
+ | * https://docs.datadoghq.com/integrations/airflow/?tab=containerized | ||
+ | INTEGRATION_NAME = {"airflow"} | ||
+ | INIT_CONFIG = {} | ||
+ | INSTANCE_CONFIG = {"url": http://%%host%%:8080"} | ||
+ | |||
+ | LOG_CONFIG = {"source": "airflow", "service": "<YOUR_APP_NAME>"} | ||
+ | |||
+ | == Errors == | ||
+ | * <code>[[ERROR airflow.stats Could not configure StatsClient: No module named 'datadog', using DummyStatsLogger instead]]</code> | ||
+ | * <code>[[Error submitting packet]]: [Errno 111] Connection refused, dropping the packet and closing the socket, "level": "WARNING</code> | ||
== Related == | == Related == | ||
* <code>[[airflow.cfg]]</code> | * <code>[[airflow.cfg]]</code> | ||
* <code>[[airflow config list]] | grep [[statsd]]</code> | * <code>[[airflow config list]] | grep [[statsd]]</code> | ||
− | * [[8125]] port | + | * <code>[[8125]] port</code> |
* [[Airflow]] <code>[[base_url]]</code> default value <code>http://localhost:8080</code> | * [[Airflow]] <code>[[base_url]]</code> default value <code>http://localhost:8080</code> | ||
* [[DogStatsD]]: <code>[[pip install]] 'apache-airflow[statsd]'</code> | * [[DogStatsD]]: <code>[[pip install]] 'apache-airflow[statsd]'</code> | ||
+ | * <code>[[DD_DOGSTATSD_MAPPER_PROFILES]]</code> | ||
== See also == | == See also == |
Latest revision as of 14:11, 20 June 2024
- Kubernetes: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#datadog
- Windows/Linux https://docs.datadoghq.com/integrations/airflow/
- Official documentation with
dogstatsd_mapper_profile:
configuration for the agent: https://app.datadoghq.eu/integrations/airflow - https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
statsd_datadog_enabled
statsd_datadog_tags
Airflow monitoring[edit]
[metrics] statsd_on = True statsd_host = localhost statsd_port = 8125 statsd_prefix = airflow #statsd_allow_list = scheduler,executor,dagrun statsd_datadog_enabled = False
Logs[edit]
triggerer | [2022-12-05 14:02:35,051] {base.py:317} INFO - Statsd buffering is disabled triggerer | [2022-12-05 14:02:35,052] {base.py:335} INFO - Statsd periodic buffer flush is disabled triggerer | [2022-12-05 14:02:35,053] {base.py:317} INFO - Statsd buffering is disabled triggerer | [2022-12-05 14:02:35,053] {base.py:335} INFO - Statsd periodic buffer flush is disabled
scheduler | [2022-12-05 14:02:35,223] {base.py:317} INFO - Statsd buffering is disabled scheduler | [2022-12-05 14:02:35,234] {base.py:335} INFO - Statsd periodic buffer flush is disabled scheduler | [2022-12-05 14:02:35,235] {base.py:317} INFO - Statsd buffering is disabled scheduler | [2022-12-05 14:02:35,235] {base.py:335} INFO - Statsd periodic buffer flush is disabled
webserver | [2022-12-05 14:02:36,135] {base.py:317} INFO - Statsd buffering is disabled webserver | [2022-12-05 14:02:36,135] {base.py:335} INFO - Statsd periodic buffer flush is disabled webserver | [2022-12-05 14:02:36,136] {base.py:317} INFO - Statsd buffering is disabled webserver | [2022-12-05 14:02:36,136] {base.py:335} INFO - Statsd periodic buffer flush is disabled
INTEGRATION_NAME = {"airflow"} INIT_CONFIG = {} INSTANCE_CONFIG = {"url": http://%%host%%:8080"}
LOG_CONFIG = {"source": "airflow", "service": "<YOUR_APP_NAME>"}
Errors[edit]
ERROR airflow.stats Could not configure StatsClient: No module named 'datadog', using DummyStatsLogger instead
Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket, "level": "WARNING
Related[edit]
airflow.cfg
airflow config list | grep statsd
8125 port
- Airflow
base_url
default valuehttp://localhost:8080
- DogStatsD:
pip install 'apache-airflow[statsd]'
DD_DOGSTATSD_MAPPER_PROFILES
See also[edit]
- Datadog, Datadog agent, Forwarder, Spans, DBM, standard attribute list, facets, log explorer, Terraform datadog, Airflow,
dog, agent, DogStatsD
, Airflow integration, Datadog pipelines,app.datadoghq.eu
, Datadog Cloud SIEM, Cloud Security Management, Datadog AWS Integration, Datadog Monitors,dd-trace-go
, DataDog Continuous Profiler, CoScreen, Kubernetes APM, flex logs, Datadog notifications
Advertising: