Apache Kafka

wikipedia:Apache Kafka (Jan 2011, Trends, Scala, Java) is a distributed event store and stream-processing platform.

• https://kafka.apache.org/
• Topics, Consumer, Producer, Pub/Sub)

• Installation:
  • macOS: brew install kafka && brew install zookeeper && brew install kcat
  • Docker: docker pull apache/kafka:3.7.0 && docker run -d --name kafka -p 9092:9092 apache/kafka:3.7.0 && docker exec -it kafka bash
  • Kubernetes: Helm install kafka
• Status: brew services info kafka && brew services info zookeeper
• Configuration: /etc/kafka/
  • server.properties
• Cloud: Amazon Managed Streaming for Apache Kafka (MSK) (Nov 2018)
• Components:
  • Kafka controller

Apache Kafka also works with external stream processing systems such as Apache Apex, Apache Beam, Apache Flink, Apache Spark, Apache Storm, and Apache NiFi.

• Optimized for writing

Binaries

kafka-acls
kafka-broker-api-versions
kafka-cluster
kafka-configs
kafka-console-consumer
kafka-console-producer
kafka-consumer-groups
kafka-consumer-perf-test
kafka-delegation-tokens
kafka-delete-records
kafka-dump-log
kafka-features
kafka-leader-election
kafka-log-dirs
kafka-metadata-shell
kafka-mirror-maker
kafka-preferred-replica-election
kafka-producer-perf-test
kafka-reassign-partitions
kafka-replica-verification
kafka-run-class
kafka-server-start
kafka-server-stop
kafka-storage
kafka-streams-application-reset
kafka-topics
kafka-verifiable-consumer
kafka-verifiable-producer

News

• Oct 2022, v3.3.0 https://www.confluent.io/blog/apache-kafka-3-3-0-new-features-and-updates/
  • KRaft mode available for new cluster production ready

Activities

• Create a Topic using kafka-topics --create and send (kafka-console-producer‎‎) and receive messages (kafka-console-consumer)

Related terms

• PubSub
• Apache Zookeeper distributed and hierarchical key-value store.
• AWS Kinesis
• Sentry
• Conduktor platform
• Zipkin
• Scala
• apt-get install kafkacat
• KRaft (Kafka Raft)
• Cloudera
• Linkedin Cruise-control
• Kafka Streams
• KIP-405: Kafka Tiered Storage
• Pub-sub
• Retention
• Ordering: messages are strictly ordered
• Confluent
• brew install --cask confluent-cli

See also

• kafka-topics [ --list ]
• Kafka, AWS MSK, aws kafka
• Kafka, kafka-cluster -h, kafka-configs, kafka-features, kafka-topics, kafka-consumer-groups, kafka-storage, kafka-server-start, kafka-server-stop, kafka-reassign-partitions, kafka-topics, kafka-console-producer, Amazon MSK, Kafka Connect, 9092, server.properties, KRaft, Kafka changelog, Kafka Streams, Confluent, Kafka Connectors, KAFKA_HEAP_OPTS, Kafka Broker, Kafka versions