Datadog Airflow integration

TO_MERGE: Datadog Airflow

• Kubernetes: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#datadog
• Windows/Linux https://docs.datadoghq.com/integrations/airflow/
• Official documentation with dogstatsd_mapper_profile: configuration for the agent: https://app.datadoghq.eu/integrations/airflow
• https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html

statsd_datadog_enabled
statsd_datadog_tags

Airflow monitoring

[metrics]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
#statsd_allow_list = scheduler,executor,dagrun
statsd_datadog_enabled = False

Logs

triggerer | [2022-12-05 14:02:35,051] {base.py:317} INFO - Statsd buffering is disabled
triggerer | [2022-12-05 14:02:35,052] {base.py:335} INFO - Statsd periodic buffer flush is disabled
triggerer | [2022-12-05 14:02:35,053] {base.py:317} INFO - Statsd buffering is disabled
triggerer | [2022-12-05 14:02:35,053] {base.py:335} INFO - Statsd periodic buffer flush is disabled

scheduler | [2022-12-05 14:02:35,223] {base.py:317} INFO - Statsd buffering is disabled
scheduler | [2022-12-05 14:02:35,234] {base.py:335} INFO - Statsd periodic buffer flush is disabled
scheduler | [2022-12-05 14:02:35,235] {base.py:317} INFO - Statsd buffering is disabled
scheduler | [2022-12-05 14:02:35,235] {base.py:335} INFO - Statsd periodic buffer flush is disabled

webserver | [2022-12-05 14:02:36,135] {base.py:317} INFO - Statsd buffering is disabled
webserver | [2022-12-05 14:02:36,135] {base.py:335} INFO - Statsd periodic buffer flush is disabled
webserver | [2022-12-05 14:02:36,136] {base.py:317} INFO - Statsd buffering is disabled
webserver | [2022-12-05 14:02:36,136] {base.py:335} INFO - Statsd periodic buffer flush is disabled

• https://docs.datadoghq.com/integrations/airflow/?tab=containerized

INTEGRATION_NAME = {"airflow"}
INIT_CONFIG = {}
INSTANCE_CONFIG = {"url": http://%%host%%:8080"}

LOG_CONFIG = {"source": "airflow", "service": "<YOUR_APP_NAME>"}

Errors

• ERROR airflow.stats Could not configure StatsClient: No module named 'datadog', using DummyStatsLogger instead
• Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket, "level": "WARNING

Related

• airflow.cfg
• airflow config list | grep statsd
• 8125 port
• Airflow base_url default value http://localhost:8080
• DogStatsD: pip install 'apache-airflow[statsd]'
• DD_DOGSTATSD_MAPPER_PROFILES

See also

• Datadog, Datadog agent, Forwarder, Spans, DBM, standard attribute list, facets, log explorer, Terraform datadog, Airflow, dog, agent, DogStatsD, Airflow integration, Datadog pipelines, app.datadoghq.eu, Datadog Cloud SIEM, Cloud Security Management, Datadog AWS Integration, Datadog Monitors, dd-trace-go, DataDog Continuous Profiler, CoScreen, Kubernetes APM, flex logs, Datadog notifications