Logstash

wikipedia:Logstash is a light-weight, open-source, server-side data processing pipeline that allows you to collect data from a variety of sources, transform it on the fly, and send it to your desired destination. It is most often used as a data pipeline for Elasticsearch, an open-source analytics and search engine. Logstash integrates with Elasticsearch and has over 200 pre-built open-source plugins that can help to index your data.

Plugins: https://www.elastic.co/guide/en/logstash/current/output-plugins.html

• Kafka, XMMP, Zabbix

• logstash --help

Installation

• https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

Linux:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install logstash

macOS installation:

• brew install logstash
• brew cask install homebrew/cask-versions/adoptopenjdk8

Usage:

• logstash (command)

Docker Logstash

Official Logstash docker image is around 800 MB size.

docker pull docker.elastic.co/logstash/logstash:7.8.0
docker run --rm -it -v ~/pipeline/:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:7.8.0

• Docker Logstash configurations:
  • Config: /usr/share/logstash/config/logstash.yml
  • Pipeline configurations: /usr/share/logstash/pipeline/

Activities

• Review homepage: https://www.elastic.co/logstash
• Review Logstash logs
• docker run --log-driver=syslog --log-opt syslog-address=tcp://<logstash-system-ip>:5000 hello-world

Related terms

• MetricBeat
• Lumberjack protocol
• Filebeat
• Vector
• Fluent Bit use Logstash format to ingest the logs

See also

• Logstash, logstash (command), logstash.yml, Logstash: docker run, Logstash changelog
• Logs, Log collector, log management, log explorer, Linux logging, docker logs, minikube logs, Vector, Logstash, Filebeat, promtail, logfmt, Elasticsearch, fluentd, Mezmo (LogDNA), Scalyr, Loggly, Loki, tlog, cockpit, NXLog, Winston, Amazon CloudWatch Logs Insights, Logz.io, Logflare, Coralogix