30/08/2024 11:28 about:blank
Hands-on Lab: Kafka Python Client
Estimated time needed: 30 minutes
Objectives
After completing this lab, you will be able to:
Use kafka-python to interact with Kafka server in Python
Send and receive messages through the Kafka-python client
About Skills Network Cloud IDE
Skills Network Cloud IDE (based on Theia and Docker) provides an environment for hands on labs for
course and project-related labs. Theia is an open-source IDE (Integrated Development Environment) that can
be run on desktop or on the cloud. To complete this lab, we will be using the Cloud IDE based on Theia
running in a Docker container.
Important notice about this lab environment
Please be aware that sessions for this lab environment are not persistent. A new environment is created for
you every time you connect to this lab. Any data you may have saved in an earlier session will get lost. To
avoid losing your data, please plan to complete these labs in a single session.
Exercise 1: Download and extract Kafka
1. Open a new terminal by clicking the menu bar and selecting Terminal->New Terminal.
This will open a new terminal at the bottom of the screen.
Run the commands below on the newly opened terminal.
Note: You can copy the code by clicking the little copy button on the bottom right of the code
snippet below and then paste it, wherever you wish.
2. Download Kafka by running the command below:
1. 1
1. wget https://downloads.apache.org/kafka/3.8.0/kafka_2.13-3.8.0.tgz
Copied! Executed!
3. Extract Kafka from the zip file by running the command below.
1. 1
1. tar -xzf kafka_2.13-3.8.0.tgz
Copied! Executed!
about:blank 1/5
30/08/2024 11:28 about:blank
This creates a new directory kafka_2.13-3.8.0 in the current directory.
Exercise 2: Configure KRaft and start server
1. Change to the kafka_2.13-3.8.0 directory.
1. 1
1. cd kafka_2.13-3.8.0
Copied! Executed!
2. Generate a Cluster UUID that will uniquely identify the Kafka cluster.
1. 1
1. KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
Copied! Executed!
Note: This cluster ID will be used by the KRaft controller.
3. KRaft requires the log directories to be configured. Run the following command to configure the log
directories passing the cluster ID.
1. 1
1. bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties
Copied! Executed!
4. Now that KRaft is configured, you can start the Kafka server by running the following command.
1. 1
1. bin/kafka-server-start.sh config/kraft/server.properties
Copied!
Note: You can be sure it has started when you see an output contains messages that confirm the
Kafka Server started successfully.
Exercise 3: Create a topic in the admin.py file
Install kafka-python
1. Open a new terminal and navigate to the kafka_2.13-3.8.0 directory.
1. 1
1. cd kafka_2.13-3.8.0
Copied! Executed!
2. Install the kafka-python package by running the following command.
about:blank 2/5
30/08/2024 11:28 about:blank
1. 1
1. pip3 install kafka-python
Copied!
3. Create a file named admin.py by running the following command.
1. 1
1. touch admin.py
Copied! Executed!
4. Click the button below to open the file in edit mode and paste the following content in the file and save
it.
Open admin.py in IDE
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
1. from kafka.admin import KafkaAdminClient,NewTopic
2. admin_client = KafkaAdminClient(bootstrap_servers="localhost:9092", client_id='test')
3. topic_list = []
4. new_topic = NewTopic(name="bankbranch", num_partitions= 2, replication_factor=1)
5. topic_list.append(new_topic)
6. admin_client.create_topics(new_topics=topic_list)
Copied!
Note: We are creating a topic "bankbranch" through this code.
Exercise 4: Create the producer.py file
You need a producer to send messages to Kafka. You will find the code for the producer in the producer.py
file.
1. Create a file named producer.py by running the following command.
1. 1
1. touch producer.py
Copied! Executed!
2. Click the button below to open the file in edit mode and paste the following content in the file and save
it.
Open producer.py in IDE
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
about:blank 3/5
30/08/2024 11:28 about:blank
9. 9
1. from kafka import KafkaProducer
2. import json
3. producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))
4. producer.send("bankbranch", {'atmid':1, 'transid':100})
5. producer.send("bankbranch", {'atmid':2, 'transid':101})
6.
7. producer.flush()
8.
9. producer.close()
Copied!
In the above code, the producer is sending across two messages through this code. These messages will be
received by the consumer.
Exercise 5: Create the consumer.py file
You need a consumer to read messages from Kafka. The code for consumer will be written in the
consumer.py file.
1. Create a file named consumer.py by running the following command.
1. 1
1. touch consumer.py
Copied! Executed!
2. Click the button below to open the file in edit mode and paste the following content in the file and save
it.
Open consumer.py in IDE
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
1. from kafka import KafkaConsumer
2. consumer = KafkaConsumer('bankbranch',
3. group_id=None,
4. bootstrap_servers=['localhost:9092'],
5. auto_offset_reset = 'earliest')
6. print("Hello")
7. print(consumer)
8.
9. for msg in consumer:
10. print(msg.value.decode("utf-8"))
Copied!
Exercise 6: Execute the three Python files
1. Execute admin.py and producer.py using the following commands in terminal:
about:blank 4/5
30/08/2024 11:28 about:blank
1. 1
2. 2
1. python3 admin.py
2. python3 producer.py
Copied! Executed!
2. Open a new terminal and execute the following commands to run consumer.py:
1. 1
2. 2
1. cd kafka_2.13-3.8.0
2. python3 consumer.py
Copied! Executed!
3. Your consumer should print the messages sent by the producer as follows:
Practice Exercise
1. Create a new producer from bankbranch in a file named new_producer.py which will take user input
as long as the user wants and accept user input for the ATM number they want to transact with (1 or 2)
and stream the transaction.
2. Observe the consumer getting the events streamed by the producer in real time.
Click here for the solution.
Authors
Ramesh Sana Reddy
Lavanya T S
Shreya Khurana
© IBM Corporation. All rights reserved.
about:blank 5/5
