4 Architecting IoT Cloud
185
a publish-subscribe system in which messages persist and are categorized based on
the topic. Message creators are known as publishers and the consumers of messages
are known as subscribers. Consumers are able to subscribe to multiple topics and
consume/receive all messages for that particular topic (see Fig. 4.6). There are five
basic components required to move data in and out of Kafka as described below [8]:
• Topics – A topic is a category, channel, or a queue defined by the user under
which messages are published. Within Kafka, topics are multi-subscriber, meaning they can have zero to many consumers subscribed to receive the published
data.
• Partitions – Topics can be divided into partitions which allow users to scale and
parallelize a particular topic by breaking its data across many different brokers
(Kafka Server/Machine). In this approach, each partition can be allocated to an
individual machine, enabling many consumers to read a topic simultaneously.
Every message in Kafka is identifiable by a tuple made up of the message
topic, partition, and offset inside the partition. An offset is a specific immutable
order in which messages are arranged in a partition. In other words, an offset
identifies the location of one record in the partition. Consumers are able to
read messages beginning at a specific offset and can read from any chosen
offset point, enabling consumers to join a topic/partition at any point. To support
fault tolerance features, Kafka can replicate each partition across a configurable
number of servers (see Fig. 4.7). In this case, a partition can function either as
a leader or as a replica. All reads and writes for a partition must pass through
the leader, and the leader handles all read and write requests for a partition. In
addition, the leader is responsible for organizing replica updates with newer data.
In the case of leader failure, a replica is able to function as the leader.
• Producers – Producers are responsible for posting a message to topics. A
message is made up of a topic name referencing where the record will be sent, a
partition number (optional), key (optional), and value (optional). If the partition
is specified by the producer, the message will be routed to that specific partition.
If a partition is not chosen by producer, but a key is indicated, the partition is
picked by the broker based on the key’s hash. In the case that no partition or
Topic A
Topic B
Consumer 1
Consumer 2
Consumer 3
Producer 2
Producer 1
Producer 3
Fig. 4.6 The publish-subscribe mechanism in Kafka
185
a publish-subscribe system in which messages persist and are categorized based on
the topic. Message creators are known as publishers and the consumers of messages
are known as subscribers. Consumers are able to subscribe to multiple topics and
consume/receive all messages for that particular topic (see Fig. 4.6). There are five
basic components required to move data in and out of Kafka as described below [8]:
• Topics – A topic is a category, channel, or a queue defined by the user under
which messages are published. Within Kafka, topics are multi-subscriber, meaning they can have zero to many consumers subscribed to receive the published
data.
• Partitions – Topics can be divided into partitions which allow users to scale and
parallelize a particular topic by breaking its data across many different brokers
(Kafka Server/Machine). In this approach, each partition can be allocated to an
individual machine, enabling many consumers to read a topic simultaneously.
Every message in Kafka is identifiable by a tuple made up of the message
topic, partition, and offset inside the partition. An offset is a specific immutable
order in which messages are arranged in a partition. In other words, an offset
identifies the location of one record in the partition. Consumers are able to
read messages beginning at a specific offset and can read from any chosen
offset point, enabling consumers to join a topic/partition at any point. To support
fault tolerance features, Kafka can replicate each partition across a configurable
number of servers (see Fig. 4.7). In this case, a partition can function either as
a leader or as a replica. All reads and writes for a partition must pass through
the leader, and the leader handles all read and write requests for a partition. In
addition, the leader is responsible for organizing replica updates with newer data.
In the case of leader failure, a replica is able to function as the leader.
• Producers – Producers are responsible for posting a message to topics. A
message is made up of a topic name referencing where the record will be sent, a
partition number (optional), key (optional), and value (optional). If the partition
is specified by the producer, the message will be routed to that specific partition.
If a partition is not chosen by producer, but a key is indicated, the partition is
picked by the broker based on the key’s hash. In the case that no partition or
Topic A
Topic B
Consumer 1
Consumer 2
Consumer 3
Producer 2
Producer 1
Producer 3
Fig. 4.6 The publish-subscribe mechanism in Kafka