4 Architecting IoT Cloud
207
each shard contains all of the document properties, but contains fewer JSON objects
than a full index. Horizontal separations allow shards to act as independent nodes
that can be stored in any cluster. Sharding is vital because:
• You can horizontally split or scale content volume.
• You can distribute and parallelize operations across shards on multiple nodes,
increasing performance and throughput.
Elasticsearch enables the user to create one or more copies of index shards known
as replica shards or simply “replicas” (see Fig. 4.22). Replication is vital because:
• It provides increased availability in the case of shard or node failure; therefore,
a replica should never be allocated to the same node as the original shard that it
was created from.
• It supports scalability of search volume and throughput because searches can be
performed on all replicas simultaneously.
The default in Elasticsearch allows each index to allocate five primary shards
and one replica, meaning if you have two nodes in a cluster, the index will have
five primary shards with five subsequent replicas (one complete replica), totaling
ten shards for each index.
Utilizing Elasticsearch requires data to be stored in the JSON document format
that is then queried for retrieval. The query domain-specific language used by
Elasticsearch is Query DSL and requires that you query in JSON format. The user
also has the ability to nest queries to perform very complex searching in stored
documents. Figure 4.23 below demonstrates how an Elasticsearch query functions
[25].
Aggregation Elasticsearch is also able to support aggregations, enabling the user to
a group or extract statistics from data. You can think about aggregations by similarly
equating it to the SQL “group by” and aggregate functions. Within Elasticsearch,
users can execute searches that generate hits and simultaneously return aggregated
results, all in a single response.
Logstash Generally, Elasticsearch is used with Logstash, an open-source, serverside data processing conduit that ingests data from a legion of sources simultaneNode 1
Node 2
Node 3
1
2
3
Node 1
Node 2
Node 3
1
2
3
4
5
6
Node 1
Node 2
Node 3
1
2
3
4
5
6
3
1
4
5
6
2
Shard
Replica
Three shards with zero replicas
Six shards with zero replicas
Six shards with one replica
Index
Cluster
Fig. 4.22 A simple example of cluster, shards, and replica in Elasticsearch
207
each shard contains all of the document properties, but contains fewer JSON objects
than a full index. Horizontal separations allow shards to act as independent nodes
that can be stored in any cluster. Sharding is vital because:
• You can horizontally split or scale content volume.
• You can distribute and parallelize operations across shards on multiple nodes,
increasing performance and throughput.
Elasticsearch enables the user to create one or more copies of index shards known
as replica shards or simply “replicas” (see Fig. 4.22). Replication is vital because:
• It provides increased availability in the case of shard or node failure; therefore,
a replica should never be allocated to the same node as the original shard that it
was created from.
• It supports scalability of search volume and throughput because searches can be
performed on all replicas simultaneously.
The default in Elasticsearch allows each index to allocate five primary shards
and one replica, meaning if you have two nodes in a cluster, the index will have
five primary shards with five subsequent replicas (one complete replica), totaling
ten shards for each index.
Utilizing Elasticsearch requires data to be stored in the JSON document format
that is then queried for retrieval. The query domain-specific language used by
Elasticsearch is Query DSL and requires that you query in JSON format. The user
also has the ability to nest queries to perform very complex searching in stored
documents. Figure 4.23 below demonstrates how an Elasticsearch query functions
[25].
Aggregation Elasticsearch is also able to support aggregations, enabling the user to
a group or extract statistics from data. You can think about aggregations by similarly
equating it to the SQL “group by” and aggregate functions. Within Elasticsearch,
users can execute searches that generate hits and simultaneously return aggregated
results, all in a single response.
Logstash Generally, Elasticsearch is used with Logstash, an open-source, serverside data processing conduit that ingests data from a legion of sources simultaneNode 1
Node 2
Node 3
1
2
3
Node 1
Node 2
Node 3
1
2
3
4
5
6
Node 1
Node 2
Node 3
1
2
3
4
5
6
3
1
4
5
6
2
Shard
Replica
Three shards with zero replicas
Six shards with zero replicas
Six shards with one replica
Index
Cluster
Fig. 4.22 A simple example of cluster, shards, and replica in Elasticsearch