Difference between revisions of "Datadog Airflow integration"
Jump to navigation
Jump to search
(→Logs) |
|||
Line 29: | Line 29: | ||
webserver | [2022-12-05 14:02:36,136] {base.py:317} INFO - Statsd buffering is disabled | webserver | [2022-12-05 14:02:36,136] {base.py:317} INFO - Statsd buffering is disabled | ||
webserver | [2022-12-05 14:02:36,136] {base.py:335} INFO - Statsd periodic buffer flush is disabled | webserver | [2022-12-05 14:02:36,136] {base.py:335} INFO - Statsd periodic buffer flush is disabled | ||
+ | |||
+ | |||
+ | |||
+ | * https://docs.datadoghq.com/integrations/airflow/?tab=containerized | ||
+ | INTEGRATION_NAME = {"airflow"} | ||
+ | INIT_CONFIG = {} | ||
+ | INSTANCE_CONFIG = {"url": http://%%host%%:8080"} | ||
+ | |||
+ | LOG_CONFIG = {"source": "airflow", "service": "<YOUR_APP_NAME>"} | ||
== Errors == | == Errors == |
Revision as of 14:10, 20 June 2024
TO_MERGE: Datadog Airflow
- Kubernetes: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#datadog
- Windows/Linux https://docs.datadoghq.com/integrations/airflow/
- Official documentation with
dogstatsd_mapper_profile:
configuration for the agent: https://app.datadoghq.eu/integrations/airflow - https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
statsd_datadog_enabled
statsd_datadog_tags
Airflow monitoring
[metrics] statsd_on = True statsd_host = localhost statsd_port = 8125 statsd_prefix = airflow #statsd_allow_list = scheduler,executor,dagrun statsd_datadog_enabled = False
Logs
triggerer | [2022-12-05 14:02:35,051] {base.py:317} INFO - Statsd buffering is disabled triggerer | [2022-12-05 14:02:35,052] {base.py:335} INFO - Statsd periodic buffer flush is disabled triggerer | [2022-12-05 14:02:35,053] {base.py:317} INFO - Statsd buffering is disabled triggerer | [2022-12-05 14:02:35,053] {base.py:335} INFO - Statsd periodic buffer flush is disabled
scheduler | [2022-12-05 14:02:35,223] {base.py:317} INFO - Statsd buffering is disabled scheduler | [2022-12-05 14:02:35,234] {base.py:335} INFO - Statsd periodic buffer flush is disabled scheduler | [2022-12-05 14:02:35,235] {base.py:317} INFO - Statsd buffering is disabled scheduler | [2022-12-05 14:02:35,235] {base.py:335} INFO - Statsd periodic buffer flush is disabled
webserver | [2022-12-05 14:02:36,135] {base.py:317} INFO - Statsd buffering is disabled webserver | [2022-12-05 14:02:36,135] {base.py:335} INFO - Statsd periodic buffer flush is disabled webserver | [2022-12-05 14:02:36,136] {base.py:317} INFO - Statsd buffering is disabled webserver | [2022-12-05 14:02:36,136] {base.py:335} INFO - Statsd periodic buffer flush is disabled
INTEGRATION_NAME = {"airflow"} INIT_CONFIG = {} INSTANCE_CONFIG = {"url": http://%%host%%:8080"}
LOG_CONFIG = {"source": "airflow", "service": "<YOUR_APP_NAME>"}
Errors
ERROR airflow.stats Could not configure StatsClient: No module named 'datadog', using DummyStatsLogger instead
Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket, "level": "WARNING
Related
airflow.cfg
airflow config list | grep statsd
8125 port
- Airflow
base_url
default valuehttp://localhost:8080
- DogStatsD:
pip install 'apache-airflow[statsd]'
DD_DOGSTATSD_MAPPER_PROFILES
See also
- Datadog, Datadog agent, Forwarder, Spans, DBM, standard attribute list, facets, log explorer, Terraform datadog, Airflow,
dog, agent, DogStatsD
, Airflow integration, Datadog pipelines,app.datadoghq.eu
, Datadog Cloud SIEM, Cloud Security Management, Datadog AWS Integration, Datadog Monitors,dd-trace-go
, DataDog Continuous Profiler, CoScreen, Kubernetes APM, flex logs, Datadog notifications
Advertising: