Skip to content

Kcat

Kcat is client for kafka

https://github.com/edenhill/kcat

brew install kcat

configuration

setup as many profiles as you want: - config/kafkacat-pre.conf - config/kafkacat-pro.conf

bootstrap.servers=kafka-server.com:9094
security.protocol=sasl_ssl
sasl.mechanisms=SCRAM-SHA-256
sasl.username=my-username
sasl.password=my-encryptedpass-tylMsBLdkKZyjrQbH9GBCzkvtQ9vn99h

consuming events

In order to consume over a topic, the kafka admin has to give rights to the connected user on the topic

#!/bin/bash

# consuming messages on PROD
# kcat -F ~/.config/kafkacat-pro.conf -t pub.topic.xxx

kcat -b localhost:9092 -t pub.topic.xxx

# Read messages from stdin, produce to 'syslog' topic with snappy compression

tail -f /var/log/syslog | kcat -b mybroker -t syslog -z snappy

# Read the last 2000 messages from 'syslog' topic, then exit

kcat -C -b mybroker -t syslog -p 0 -o -2000 -e

# Decode Avro key (-s key=avro), value (-s value=avro) or both (-s avro) to JSON using schema from the Schema-Registry:

kcat -b mybroker -t ledger -s avro -r http://schema-registry-url:8080

# Output consumed messages according to format string:

kcat -b mybroker -t syslog -f 'Topic %t[%p], offset: %o, key: %k, payload: %S bytes: %s\n'

kcat -F ~/.config/kafkacat-pro.conf  -t "pub.topic.xxx" -Z -C -o s@1691020800000 -o e@1691107199000  -e -f '%s\n' | grep "492089954"

kcat -F ~/.config/kafkacat-pro.conf  -t "pub.topic.xxx" -Z -C -e -f '%s\n' | grep "492089954"

kcat -F ~/.config/kafkacat-pro.conf  -t "pub.topic.xxx" -C -f '%s\n' | grep "492089954"

kcat -F ~/.config/kafkacat-pro.conf  -t "pub.topic.xxx" -Z -C -o s@1690934400000 -o e@1691107200000 | jq -c 'select( .adMedia."adId" == "492089954" )'

kcat -F ~/.config/kafkacat-pro.conf  -t "pub.topic.xxx" -Z -C -o s@1692921600000 -o e@1693267200000 | grep "AdGenerated"


## https://www.epochconverter.com/ to filter by dates start end s@1690934400000 e@1691107200000

kcat -F ~/.config/kafkacat-pro.conf -t "pub.topic.xxx" -Z -C -e -f '%s\n' | grep "492089954"

kcat -F ~/.config/kafkacat-pro.conf -t "pub.topic.xxx" -Z -C -o s@1691020800000 -o e@1691107199000 -e -f '%s\n' >> mediaEvents. txt

kcat -F ~/.config/kafkacat-pro.conf -t "pub.topic.xxx" -Z -C -o s@1691020800000 -o e@1691107199000 | jq -c 'select(.adMedia."adId"="487481813"'

kcat -F ~/.config/kafkacat-pro.conf -t "pub.topic.xxx" -o -2000 -e -f 'Topic %t[%p], offset: %o, key: %k, payload: %S bytes: %s\n' | grep "http://schema.net/events/cross/-Message.json/1.7.json"

kcat -b mybroker -t syslog -f 'Topic %t[%p], offset: %o, key: %k, payload: %S bytes: %s\n'

Producing events

In order to produce over a topic, the kafka admin has to give rights to the connected user on the topic

Create the ad_generared.json file, and execute the next command line:

# producing messages on local
kcat -b localhost:9092 -t pub.topic.xxx -P -T ad_generared.json