The cost of delay needs to be taken in account.
Currently most of the data providers’ systems have been equiped with APIs,
which allow the customers’ systems to make machine-to-machine connection for
direct data access. However, this docking process is very tough. The failure in any
step will lead to the discard of the source. It is a matter involving two teams from
different institutions. Before the standards of services were established, a person
from the source team has to work together with the consumer team. He/she is
responsible for providing the API information to the consumer team and guiding
the team to use provider’s data services. Such a strategy is old-fashioned, and in
order to achieve the final goal of machine-to-machine interoperability, meetings and
workshops have to be held frequently. The very disappointing thing is that even
direct human-to-human collabration cannot guarantee the machine-tomachine sourcing works. Maybe after 1 or 2 weeks of communication, the consumers find the source lacks of some fields or structures in the API of the provider's
system, which require the data providers to completely reorganize or update. If the
data provider refuses to do so, the collaboration is over.
All the data-consuming industries, not only agriculture, deeply recognized this
problem after many fundamental projects. It is aware that the difficulty has to be
overcome by both data providers and consumers. The core solution is one word:
standardization. Comprehensive standardization in every aspect of data communication interface, such as data format, service interface, message channel, protocol,
Web API, encoding, decoding, parameter, condition, etc., is the ultimate answer.
Standard-compliant data and services can be easily used by consumers without
communication among persons. Once standards are employed, the data sources
only need to develop standard-compliant services, and the consumers only need
standard-compliant clients. The data providers no longer have to send a person to
accommdate every customer. The clients no longer need asking a person to figure out
the content of the data and the use of the services since there are clear rules and
explanations in the specifications.
We have seen many progresses on standardization in data sourcing industries.
Typically, the satellite databases provide a variety of standard formats and access
methods including HTTP, FTP, OGC (Open Geospatial Consortium) W*S,
OPeNDAP, netCDF-CF (standard format for gridded and point monitoring data
and models) (Rew et al. 1997), HDF4, HDF5, GRIB (GRidded Information in
Binary) (SCHMUNCK 2002), GeoTiff, and KML (Keyhole Markup Language).
The WCS (Web Coverage Service) is a standard for web services distributing raster
data. The WMS (Web Map Service) is a service standard for real-time composition
of data into visible maps. Some NASA EOS datasets such as AIRS products are
accessible through standard OGC WCS and WMS protocols (Yang 2010; Yang and
Di 2002). The formats adopted in NASA are mainly GeoTiff and HDF (Burnett et al.
2007; Han et al. 2008; NASA 2014, 2016; Savtchenko et al. 2004; Zhao et al. 2015).
NOAA mainly uses netCDF and GRIB (Hankin et al. 2010; Williams 2015;
Williams et al. 2009). These data formats and services have been around for a
long time and very familiar to the relevant community. It is convenient to find
4 Agro-geoinformatics Data Sources and Sourcing
57
Currently most of the data providers’ systems have been equiped with APIs,
which allow the customers’ systems to make machine-to-machine connection for
direct data access. However, this docking process is very tough. The failure in any
step will lead to the discard of the source. It is a matter involving two teams from
different institutions. Before the standards of services were established, a person
from the source team has to work together with the consumer team. He/she is
responsible for providing the API information to the consumer team and guiding
the team to use provider’s data services. Such a strategy is old-fashioned, and in
order to achieve the final goal of machine-to-machine interoperability, meetings and
workshops have to be held frequently. The very disappointing thing is that even
direct human-to-human collabration cannot guarantee the machine-tomachine sourcing works. Maybe after 1 or 2 weeks of communication, the consumers find the source lacks of some fields or structures in the API of the provider's
system, which require the data providers to completely reorganize or update. If the
data provider refuses to do so, the collaboration is over.
All the data-consuming industries, not only agriculture, deeply recognized this
problem after many fundamental projects. It is aware that the difficulty has to be
overcome by both data providers and consumers. The core solution is one word:
standardization. Comprehensive standardization in every aspect of data communication interface, such as data format, service interface, message channel, protocol,
Web API, encoding, decoding, parameter, condition, etc., is the ultimate answer.
Standard-compliant data and services can be easily used by consumers without
communication among persons. Once standards are employed, the data sources
only need to develop standard-compliant services, and the consumers only need
standard-compliant clients. The data providers no longer have to send a person to
accommdate every customer. The clients no longer need asking a person to figure out
the content of the data and the use of the services since there are clear rules and
explanations in the specifications.
We have seen many progresses on standardization in data sourcing industries.
Typically, the satellite databases provide a variety of standard formats and access
methods including HTTP, FTP, OGC (Open Geospatial Consortium) W*S,
OPeNDAP, netCDF-CF (standard format for gridded and point monitoring data
and models) (Rew et al. 1997), HDF4, HDF5, GRIB (GRidded Information in
Binary) (SCHMUNCK 2002), GeoTiff, and KML (Keyhole Markup Language).
The WCS (Web Coverage Service) is a standard for web services distributing raster
data. The WMS (Web Map Service) is a service standard for real-time composition
of data into visible maps. Some NASA EOS datasets such as AIRS products are
accessible through standard OGC WCS and WMS protocols (Yang 2010; Yang and
Di 2002). The formats adopted in NASA are mainly GeoTiff and HDF (Burnett et al.
2007; Han et al. 2008; NASA 2014, 2016; Savtchenko et al. 2004; Zhao et al. 2015).
NOAA mainly uses netCDF and GRIB (Hankin et al. 2010; Williams 2015;
Williams et al. 2009). These data formats and services have been around for a
long time and very familiar to the relevant community. It is convenient to find
4 Agro-geoinformatics Data Sources and Sourcing
57