Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit released in 2012. Prometheus design is focused to collect and process metrics, not as an event logging system for logs.. Grafana is usually used to produce dashboards.

• https://prometheus.io/

Prometheus Installation

• http://localhost:9090
• Node exporter: curl -sSL http://localhost:9100/metrics

Kubernetes

• helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
• helm install myprometheus prometheus-community/prometheus

Configuration files

• Linux: /etc/prometheus/prometheus.yml
• macOS: /usr/local/etc/prometheus.args
• Prometheus storage: /var/lib/prometheus/

Binaries

• prometheus
• promtool
• tsdb

Alternatives

• Thanos
• VictoriaMetrics
• Grafana Mimir (Mar 2022)

Configuration

• Configuration file: /etc/prometheus/prometheus.yml

• In the global settings, define the default interval for scraping metrics. Note that Prometheus will apply these settings to every exporter unless an individual exporter's own settings override the global.

• This scrape_interval value tells Prometheus to collect metrics from its exporters every 15 seconds, which is long enough for most exporters. Now, add Prometheus itself to the list of exporters to scrape from with the following scrape_configs directive:

• Prometheus uses the job_name to label exporters in queries and on graphs, so be sure to pick something descriptive here. And, as Prometheus exports important data about itself that you can use for monitoring performance and debugging, we've overridden the global scrape_interval directive from 15 seconds to 5 seconds for more frequent updates. Lastly, Prometheus uses the static_configs and targets directives to determine where exporters are running. Since this particular exporter is running on the same server as Prometheus itself, we can use localhost instead of an IP address along with the default port, 9090.

Your configuration file should now look like this:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Linux Basic Prometheus Operations

• Start Prometheus: :sudo systemctl start prometheus
• Reload systemd: :sudo systemctl daemon-reload prometheus
• Verify the service's status: :sudo systemctl status prometheus

• Delete all data:

Enable --web.enable-admin-api and execute:

http://prometheus.domain.com:9090/api/v2/admin/tsdb/delete_series

http://prometheus.domain.com:9090/api/v2/admin/tsdb/clean_tombstones

Activities

1. Install Prometheus in Linux, /Run Prometheus in Linux/ and connect to graphical interface http://localhost:9090/graph
2. Read Prometheus changelog: https://github.com/prometheus/prometheus/blob/master/CHANGELOG.md
3. Read Stackoverflow prometheus questions: https://stackoverflow.com/questions/tagged/prometheus?tab=Frequent
4. Export dockerd metrics to Prometheus: https://docs.docker.com/config/daemon/prometheus/, modify /etc/docker/daemon.json

Related terms

• PromQL
• Elasticsearch
• Exporters: Nginx (nginx-prometheus-exporter), Node exporter
• Cortex, Prometheus as a service
• Loki
• CKA: Understand how to monitor applications in Kubernetes
• .tpl
• alert:
• AWS CloudWatch Container Insights
• Amazon Managed Service for Prometheus
• Kubernetes Metrics Server
• Fluent Bit
• GitLab 12.5 Automatically close GitLab issues with recovery alerts from Prometheus
• Prometheus-node-exporter Debian configuration file
• /metrics
• Amazon Managed Prometheus
• Google Cloud Managed Prometheus
• level=error ... no space left on device
• Prometheus API
• Logz.io

See also

• Prometheus, Thanos, Grafana Mimir, VictoriaMetrics
• Prometheus: Prometheus installation, 9090, storage, Prometheus exporters: Node exporter 9100, Blackbox exporter, Prometheus alertmanager (changelog), Prometheus versions, Prometheus Node exporter releases, promtool, amtool, tsdb, /etc/default/prometheus, /etc/prometheus/prometheus.yml, journalctl -u prometheus, /etc/default/prometheus-node-exporter, PromQL, alert:, Prometheus Operator, Thanos, VictoriaMetrics, PushGateway, Prometheus API, values.yml, Amazon Managed Service for Prometheus
• Monitoring: Prometheus, Cacti, monit, munin, RRDtool, Zabbix, Netdata, Nagios, Check MK, Icinga, Pingdom, OpsGenie and Datadog, Opsgenie, PRTG, Checkmk
• TSDB: Prometheus, InfluxDB, RRDtool, Graphite, Telegraf, OpenTSDB, TimescaleDB, tsdbadmin

Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy.

Source: https://en.wikiversity.org/wiki/Prometheus_monitoring