Kafka-cli
Apache Kafka is a distributed event streaming platform that is widely used for building real-time data pipelines and streaming applications. In this guide, we’ll walk through the steps to install Kafka and Zookeeper on a Mac using Homebrew, and how to perform basic operations such as creating a topic, producing messages, and consuming messages using the command line interface.
Prerequisites
Before getting started, ensure that you have Homebrew installed on your Mac. Install HomeBrew : https://brew.sh/
Installing Kafka and Zookeeper
First, let’s install Zookeeper, which is a centralized service for managing configuration information, naming, providing distributed synchronization, and providing group services.
zkserver start
brew install zookeeper
Next, we’ll install Apache Kafka.
brew install kafka
brew services start kafka
This will start the Kafka server as a background service.
Creating a Topic
Topics are categories to which records are published. Let’s create a topic named “test” with a replication factor of 1 and a single partition.
kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
Producing Messages
To send messages to the “test” topic, we can use the Kafka console producer.
kafka-console-producer --bootstrap-server localhost:9092 --topic test
This will start a console where you can type messages to send to the topic.
Consuming Messages
To receive messages from the “test” topic, we can use the Kafka console consumer.
kafka-console-consumer --bootstrap-server localhost:9092 --topic test
This will start a console where you can see the messages being consumed in real-time.
Topics List
To fetch all the topics configured on machine, we can use kafka-topics.
kafka-topics --list --bootstrap-server localhost:9092
Send messages to topic
echo '{"adId": "123456789"}' | kcat -P -b localhost:9092 -t topic_name
Consume messages from topic
kcat -C -b localhost:9092 -t topic_name -o beginning
revisar config consumer_group
Look for the consumer_group name on application.yml and use it to consume messages from the topic.
kafka-consumer-groups --bootstrap-server localhost:9092 --group consumer_group_name --describe