Understanding Topics, Partitions, and

Brokers

Ryan Plant
COURSE AUTHOR

@ryan_plant blog.ryanplant.com
 Basic Apache Kafka installation:
- Download the binary package
Demo - Extract the archive
- Explore the installation directory
contents
Prerequisites:
- Linux operating system
- Java 8 JDK installed
- Scala 2.11.x installed
Apache Kafka Topics

Central Kafka abstraction

Named feed or category of messages
- Producers produce to a topic
- Consumers consume from a topic

Logical entity
Physically represented as a log
 to: “my_topic”
to: “my_other_topic”

Broker Broker Broker Broker

“MY_TOPIC”
“MY_OTHER_TOPIC”

from: “my_topic” from: “my_other_topic”

 ordered sequence (by time)
append-only
older
immutable facts as events newer

0 1 2 3 4 5 6 7 8 9

“MY_TOPIC”
Event Sourcing
An architectural style or approach to maintaining an
application’s state by capturing all changes as a
sequence of time-ordered, immutable events.
older newer

0 1 2 3 4 5 6 7 8 9

“MY_TOPIC”
 Message Content

{timestamp}
Each message has a:
- Timestamp

{id} - Referenceable identifier

- Payload (binary)

[data content]
0 1 2 3 4 5 6 7 8 9

“MY_TOPIC”
The Offset

A placeholder:
- Last read message position
- Maintained by the Kafka Consumer
- Corresponds to the message identifier
0 1 2 3 n

“MY_TOPIC”

“from beginning”

offset: 0
offset: 0
offset: 3
0 1 2 3 4 5 6 7 8 9

“MY_TOPIC”

“from last offset”

3 8
Message Retention Policy

Apache Kafka retains all published

messages regardless of consumption
Retention period is configurable
- Default is 168 hours or seven days

Retention period is defined on a per-topic

basis
Physical storage resources can constrain
message retention
 Simple Kafka cluster setup
Creating an Apache Kafka topic
Producing some messages to the topic
Demo Consuming the messages from the topic
Look for:
- Built-in Producer and Consumer
clients
- The ordering of the messages within a
topic
Don’t get too caught up on:
- The command line parameters and
options
 Does Look This Look Familiar?

older newer

0 1 2 3 4 5 6 7 8 9

TOPIC

 Append-only
 Ordered sequence (by time)
 Immutable facts as events
Transaction or Commit Logs

Source of truth
Physically stored and maintained
Higher-order data structures derive from
the log
- Tables, indexes, views, etc.

Point of recovery
Basis for replication and distribution
 Apache Kafka is publish-
subscribe messaging
rethought as a distributed
commit log.
 Kafka Partitions

Each topic has one or more partitions

A partition is the basis for which Kafka can:
- Scale
- Become fault-tolerant
- Achieve higher levels of throughput

Partition Partition Partition Each partition is maintained on at least one

or more Brokers
(Partition == Log)
 Creating a Topic: Single Partition

~$ bin/kafka-topics.sh --create --topic my_topic \

> --zookeeper localhost:2181 \
> --partitions 1 \
> --replication-factor 1
 “my_topic”
Partition 0

0 1 2 3 4 5 6 7 8 n

/tmp/kafka-logs/{topic}-{partition}
“my_topic-0”
.index
Broker
.log

Each partition must fit entirely on one machine.

In general, the scalability of
Apache Kafka is determined
by the number of partitions
being managed by multiple
broker nodes.
 Creating a Topic: Multiple Partitions

~$ bin/kafka-topics.sh --create --topic my_topic \

> --zookeeper localhost:2181 \
> --partitions 3 \
> --replication-factor 1
 “my_topic”

Partition 0

0 1 2 3 4 5 6 7 8 9

Partition 1

0 1 2 3 4 5 6 7 8 9

Partition 2

0 1 2 3 4 5 6 7 8

older newer
 $.s
--create h
1 available brokers
--topic my_topic ZK
--partitions 3 2
--rep-factor 1 assign partitions to leaders

3 status checks

I have
partition 1 of
my_topic I have
id=0 id=1 partition 2 of id=2
my_topic

I have
partition 0 of
my_topic-1 my_topic my_topic-0 my_topic-2
metadata metadata metadata
 $.s
h
--broker-list {broker:2}
--topic my_topic ZK
metadata
…

id=0 id=1 id=2

my_topic-1 my_topic-0 my_topic-2

metadata metadata metadata
 $.s
h
--zookeeper {..}
--topic my_topic metadata ZK
…

3 1
2
id=0 id=1 id=2

my_topic-1 my_topic-0 my_topic-2

metadata metadata metadata
Partitioning Trade-offs

The more partitions the greater the

Zookeeper overhead
- With large partition numbers ensure
proper ZK capacity
Message ordering can become complex
- Single partition for global ordering
- Consumer-handling for ordering

The more partitions the longer the leader

fail-over time
Something Is Missing

What about fault-tolerance?

- Broker failure
- Network issue
- Disk failure
 Houston, we
have a
my_topic problem…
broker id: 2
partitions: 0, 2
ZK

I now have
id=0 id=1 partition 0 of id=2
my_topic

my_topic-0
my_topic-1 my_topic-0 my_topic-2

No redundancy between nodes…

 Don’t Forget the Replication Factor

~$ bin/kafka-topics.sh --create --topic my_topic \

> --zookeeper localhost:2181 \
> --partitions 3 \
> --replication-factor 1
Replication Factor

Reliable work distribution

- Redundancy of messages
- Cluster resiliency
- Fault-tolerance

Guarantees
- N-1 broker failure tolerance
- 2 or 3 minimum

Configured on a per-topic basis

 Multiple Replica Sets

~$ bin/kafka-topics.sh --create --topic my_topic \

> --zookeeper localhost:2181 \
> --partitions 3 \
> --replication-factor 3
 $.s
--create h
--topic my_topic ZK
--partitions 3
--rep-factor 3

id=0 id=1 id=2

my_topic-1 my_topic-0 my_topic-2

metadata metadata metadata

ISR: ISR: ISR:

id=7 3 id=5 id=8 3 id=3 id=4 3 id=6

my_topic-1 my_topic-1 my_topic-0 my_topic-0 my_topic-2 my_topic-2

(replica) (replica) (replica) (replica) (replica) (replica)
 $.s
--create h
--topic my_topic ZK
--partitions 3
--rep-factor 3

id=0 id=1 id=2

my_topic-1 my_topic-0 my_topic-2

metadata metadata metadata

ISR: ISR: ISR:

id=7 3 id=5 id=8 2 id=3 id=4 3 id=6

my_topic-1 my_topic-1 my_topic-0 my_topic-2 my_topic-2

(replica) (replica) (replica) (replica) (replica)
 Viewing Topic State

~$ bin/kafka-topics.sh --describe --topic my_topic \

> --zookeeper localhost:2181
 Multi-broker Kafka Setup
Demo Single Partition Topic
Replication Factor of 3
Look for:
- Using the --describe command
- Failure handling
- Continued operation
 Detailed explanation and view:
- Topics and Partitions
- Broker partition management and
behavior
Summary Aligned with distributed systems
principles
- Leader election of partitions
- Work distribution and failover

Kafka in action
- Demos
- Configuration

Foundation upon which to dive deeper

into Producers and Consumers