In this article, we will be using Kafka via docker and try producing and consuming messages. We will learn some keywords and basic tools in Kafka.
Firstly, let’s create a new folder named kafka (or whatever name of your project), and go to that folder
Inside the kafka folder, we create a new docker-compose.yml file with the content below:
services:
kafka: # service name, can change to any name
image: ‘bitnami/kafka:3.6.2’ # docker image from https://hub.docker.com/r/bitnami/kafka
environment:
– KAFKA_CFG_NODE_ID=0
– KAFKA_CFG_PROCESS_ROLES=controller,broker
– KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
– KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
– KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
– KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
Run Kafka server
We need the container name to access into it. Check the NAMES column when running:
—
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f9258b058804 bitnami/kafka:3.6.2 “/opt/bitnami/script…” 24 hours ago Up 39 minutes 9092/tcp kafka_kafka_1
Access into docker container command line (In my case in kafka_kafka_1, please change to your service name if mismatch)
Now you are in the container and can use a set of tools below:
kafka-topics.sh: Manage topics
kafka-console-producer.sh: Manage producer
kafka-console-consumer.sh: Manage consumer
Kafka treat message as “event”. Events and grouped into a category called “topic”.
Assume we are creating a chat application. Let’s create a new topic called “chat-message” which is a stream of message records, as well as create a new topic called “user-presence” to manage the online status of users. We will use the server kafka:9092 mentioned in docker-compose configuration
kafka-topics.sh –create –topic user-presence –bootstrap-server kafka:9092
We can check how many topics we have by using:
—
__consumer_offsets
chat-message
user-presence
And view detail each topic:
—
Topic: chat-message TopicId: 0XIXOlZ-TAWqEBlBUP7WAA PartitionCount: 1 ReplicationFactor: 1 Configs:
Topic: chat-message Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Now we can use 2 separate terminals to set up the producer and consumer, and see how the consumer get message from the producer in real time.
In the first terminal, let’s use the tool kafka-console-consumer.sh to setup a consumer to waiting for the messages in the topic chat-message to consume:
Open the second terminal and use the tool kafka-console-producer.sh to setup the producer by running the command:
The kafka-console-producer.sh will waiting for our inputs. Let’s try pasting 3 events below and hit enter:
Event: {“message_id”: 2, “sender_id”: “user2”, “receiver_id”: “user1”, “content”: “I’m good, thanks! How about you?”, “timestamp”: “2024-04-10T09:01:00”}
Event3: some random words…
Note that the message format is plaintext, in the example above we use both plain text and some JSON-like format to show that the format is not strict. The consumer will need to have logic to decode the plaintext to some meaningful messages.
Now we should see events appear in the consumer terminal
That is the very basic setup of Kafka using docker. If you have any questions please ask in the comment section.
Bonus:
If you don’t want to access the container, in the host machine you can replace the bash with the kafka tools like kafka-topics.sh to run the Kafka command outside of container like this:
Original port: https://www.anvo.dev/articles/getting-start-with-kafka-using-docker
