Service) have been widely considered as a huge success in lifting all the data storage
and process businesses into cloud (Amazon 2010, 2013; Marx 2013; Varia and
Mathew 2014). It can hold data ranging from gigabytes to petabytes in size. The
capability is dedicated to deal with the big data challenges which can be represented
by the four Vs: volume, velocity, variety, and veracity (Hashem et al. 2015). To
address these challenges and manage the explosively growing datasets, cloud seems
to be the only way out until now.
Cloud sourcing basically means the data are hosted on clouds, rather than
collected by clouds. After being acquired, the data are uploaded to clouds for people
to access it via the cloud toolkits. As mentioned in 4.3.1, data owners are the first
batch of cloud followers. Probably many people don’t realize that we are inevitably
heading to ZB (zettabyte, equal to 1000 exabytes; 1 exabyte is equal to 1000
terabytes) era. Maintaining a database containing a PB of both structured and
nonstructured content is never an easy task, actually more complex than people
can imagine. Very powerful storage and computer hardware are required and must
be maintained 24/7. Cloud providers brought together these requirements and
supplied a pool of solutions for various databases to choose. So far, many public
datasets have been archived by the major cloud providers. For instance, Amazon
launched an Earth data plan
13, 14 to archive important public datasets to benefit the
educators, researchers, and students (Palankar et al. 2008). The plan hosts Landsat
8 imagery, NEXRAD (Next-Generation Weather Radar, a network of 160 highresolution Doppler radar sites that detect precipitation and atmospheric movement
and disseminate data in approximately 5-min intervals from each site), SpaceNet
machine learning imagery, National Agriculture Imagery Program, digital elevation
model (DEM) Terrain Titles, GDELT dataset, NASA Earth Exchange (NEX)
datasets, GSOD (Global Surface Summary of the Day), Sentinel-2 imagery, and
DigitalGlobe open data. Google Earth Engine hosts the widely used datasets all over
the world and let people use them as free as usual (Gorelick 2013). Microsoft Azure
is a heavyweight player and hosts a huge volume of public datasets.
15 Most datasets
from the US government agencies, including NASA, DOT, the US Census, EPA,
etc., are hosted on Azure currently. The performances have been recognized by the
public. All the data are available online. HTTP URL is the simplest and direct option
to access them. The cloud providers build user-friendly websites for users to browse,
discover, and download data. API interface is also offered for client programs to
access and download the data via system-to-system exchanges. The users can
manipulate their data in the cloud although they don’t physically possess the data
(Chow et al. 2009).
Public clouds may offer low-cost instance VMs (Amazon as low as $0.0059 per
hour
16 ). They are still not fit for all use cases, especially when handling security
13 https://aws.amazon.com/public-datasets/
14 https://aws.amazon.com/cn/earth/
15 https://docs.microsoft.com/en-us/azure/sql-database/sql-database-public-data-sets
16 https://aws.amazon.com/cn/ec2/pricing/on-demand/
4 Agro-geoinformatics Data Sources and Sourcing
59
and process businesses into cloud (Amazon 2010, 2013; Marx 2013; Varia and
Mathew 2014). It can hold data ranging from gigabytes to petabytes in size. The
capability is dedicated to deal with the big data challenges which can be represented
by the four Vs: volume, velocity, variety, and veracity (Hashem et al. 2015). To
address these challenges and manage the explosively growing datasets, cloud seems
to be the only way out until now.
Cloud sourcing basically means the data are hosted on clouds, rather than
collected by clouds. After being acquired, the data are uploaded to clouds for people
to access it via the cloud toolkits. As mentioned in 4.3.1, data owners are the first
batch of cloud followers. Probably many people don’t realize that we are inevitably
heading to ZB (zettabyte, equal to 1000 exabytes; 1 exabyte is equal to 1000
terabytes) era. Maintaining a database containing a PB of both structured and
nonstructured content is never an easy task, actually more complex than people
can imagine. Very powerful storage and computer hardware are required and must
be maintained 24/7. Cloud providers brought together these requirements and
supplied a pool of solutions for various databases to choose. So far, many public
datasets have been archived by the major cloud providers. For instance, Amazon
launched an Earth data plan
13, 14 to archive important public datasets to benefit the
educators, researchers, and students (Palankar et al. 2008). The plan hosts Landsat
8 imagery, NEXRAD (Next-Generation Weather Radar, a network of 160 highresolution Doppler radar sites that detect precipitation and atmospheric movement
and disseminate data in approximately 5-min intervals from each site), SpaceNet
machine learning imagery, National Agriculture Imagery Program, digital elevation
model (DEM) Terrain Titles, GDELT dataset, NASA Earth Exchange (NEX)
datasets, GSOD (Global Surface Summary of the Day), Sentinel-2 imagery, and
DigitalGlobe open data. Google Earth Engine hosts the widely used datasets all over
the world and let people use them as free as usual (Gorelick 2013). Microsoft Azure
is a heavyweight player and hosts a huge volume of public datasets.
15 Most datasets
from the US government agencies, including NASA, DOT, the US Census, EPA,
etc., are hosted on Azure currently. The performances have been recognized by the
public. All the data are available online. HTTP URL is the simplest and direct option
to access them. The cloud providers build user-friendly websites for users to browse,
discover, and download data. API interface is also offered for client programs to
access and download the data via system-to-system exchanges. The users can
manipulate their data in the cloud although they don’t physically possess the data
(Chow et al. 2009).
Public clouds may offer low-cost instance VMs (Amazon as low as $0.0059 per
hour
16 ). They are still not fit for all use cases, especially when handling security
13 https://aws.amazon.com/public-datasets/
14 https://aws.amazon.com/cn/earth/
15 https://docs.microsoft.com/en-us/azure/sql-database/sql-database-public-data-sets
16 https://aws.amazon.com/cn/ec2/pricing/on-demand/
4 Agro-geoinformatics Data Sources and Sourcing
59