186
F. Firouzi and B. Farahani
5
4
3
2
1
0
2
1
0
4
3
2
1
0
Producer
ParƟƟon 1
ParƟƟon 2
ParƟƟon 3
The concept of part¶t¶ons of a topic
Partition 1
Partition 2
Partition 3
Partition 1
Partition 2
Partition 3
Partition 1
Partition 2
Partition 3
Broker 1
Broker 2
Broker 3
Leader and replica
Publish
Publish
Replica
write
Replica
write
Leader
Replica
Fig. 4.7 How topics and partitions work in Kafka
key is specified, a partition is chosen in a round-robin manner (i.e., partitions are
selected based on a circular order). Kafka broker can also create a timestamp for
each message when it wants to store the message in the partition. Producers can
attach keys to messages, guaranteeing that all messages with the same key are
sent to the same topic partition. Kafka ensures order in a partition but does not
guarantee order across partitions within topics. Therefore, not utilizing a key will
result in round-robin style distribution to all partitions, and it will not maintain
the same order.
• Consumers – Consumers subscribe to a particular topic(s) and process messages
that are posted in the corresponding topic(s).
• Brokers – A Kafka broker, Kafka node, or Kafka server all reference the same
concept and the terms can be used interchangeably. Kafka brokers take in
producer messages and store them on disk categorized by topics/partitions. The
broker also enables consumers to obtain messages based on offset, topic, or
partition.
4.4.1.3 Apache Nifi
Apache Nifi is designed to automate data movement among different systems.
Nifi simplifies data movement between a source and a destination through realtime control, and it supports different and distributed sources with various formats,
protocols, schemas, sizes, and speeds (i.e., clickstreams, social media feeds, geolocation devices, videos, log files, etc.). Apache Nifi can be configured to move
data similar to the manner in which UPS or FedEx moves, tracks, and delivers
parcels because this platform enables the user to track real-time data much like
a package delivery. Nifi is also very useful for sensitive data flows which need
F. Firouzi and B. Farahani
5
4
3
2
1
0
2
1
0
4
3
2
1
0
Producer
ParƟƟon 1
ParƟƟon 2
ParƟƟon 3
The concept of part¶t¶ons of a topic
Partition 1
Partition 2
Partition 3
Partition 1
Partition 2
Partition 3
Partition 1
Partition 2
Partition 3
Broker 1
Broker 2
Broker 3
Leader and replica
Publish
Publish
Replica
write
Replica
write
Leader
Replica
Fig. 4.7 How topics and partitions work in Kafka
key is specified, a partition is chosen in a round-robin manner (i.e., partitions are
selected based on a circular order). Kafka broker can also create a timestamp for
each message when it wants to store the message in the partition. Producers can
attach keys to messages, guaranteeing that all messages with the same key are
sent to the same topic partition. Kafka ensures order in a partition but does not
guarantee order across partitions within topics. Therefore, not utilizing a key will
result in round-robin style distribution to all partitions, and it will not maintain
the same order.
• Consumers – Consumers subscribe to a particular topic(s) and process messages
that are posted in the corresponding topic(s).
• Brokers – A Kafka broker, Kafka node, or Kafka server all reference the same
concept and the terms can be used interchangeably. Kafka brokers take in
producer messages and store them on disk categorized by topics/partitions. The
broker also enables consumers to obtain messages based on offset, topic, or
partition.
4.4.1.3 Apache Nifi
Apache Nifi is designed to automate data movement among different systems.
Nifi simplifies data movement between a source and a destination through realtime control, and it supports different and distributed sources with various formats,
protocols, schemas, sizes, and speeds (i.e., clickstreams, social media feeds, geolocation devices, videos, log files, etc.). Apache Nifi can be configured to move
data similar to the manner in which UPS or FedEx moves, tracks, and delivers
parcels because this platform enables the user to track real-time data much like
a package delivery. Nifi is also very useful for sensitive data flows which need