Grouped Calculations in Tables and Timetables
Grouped calculations can help you interpret large datasets such as time-series data. In such
calculations, you use a grouping variable to split a dataset into groups and apply a function to each
group. A grouping variable contains values, such as time periods or station locations, that you can use
to group other data values, such as temperature readings or atmospheric concentrations of a gas. In
MATLAB®, you can store such data in tables or timetables. With grouped calculations in a table you
can often calculate results in-place, in one table, instead of breaking data out into separate tables and
merging results later.
This example shows how to import nitrogen dioxide (NO2) data from the US Environmental
Protection Agency (EPA) into a table and do grouped calculations on this data. NO2 is one of the
Criteria Air Pollutants regulated under the US Clean Air Act. It is toxic by itself and is also a key
component of photochemical smog that results in ground-level ozone production. NO2 is produced
through high-temperature processes that can split nitrogen and oxygen gases and enable them to
recombine. Natural processes contribute NO2 to the atmosphere, but so do human activities such as
combustion in automobile engines and power plants, lightning, and biomass burning. The
concentration of NO2 in the atmosphere is also influenced by the photochemical cycling between NO
and NO2, atmospheric transport, and ultimately oxidation to nitric acid, causing acid rain. Different
processes contribute NO2 to the atmosphere on different timescales, leading to daily (diurnal),
weekly, and annual cycles in its atmospheric concentration. Time-series analysis of such data relies
heavily on grouped calculations to examine different periodic behavior or to average the data over
time to smooth out high-frequency variability and reveal long-term trends.
The example first shows how to do preliminary data cleaning, including conversion of the table to a
timetable. Then it shows simple ways to group the data by one grouping variable and calculate annual
mean NO2 concentrations. It also shows how to group the NO2 data by two grouping variables
together, time and location, enabling calculations that find locations exceeding EPA standards at
various times. You can also group the NO2 data by time period to look for daily or yearly cycles.
Finally it shows how to apply a function that requires inputs from multiple table variables to find the
times at which the maximum NO2 concentrations occurred at each site.
Import NO2 Data to Table
First, import NO2 data from the Air Quality System (AQS) database maintained by the EPA. This data
consists of hourly measurements of NO2 concentrations from outdoor monitors across the United
States, Puerto Rico, and the U.S. Virgin Islands. It is stored as a set of zipped spreadsheets, one for
each year starting with 1980.
Download hourly NO2 measurements for the years 1985–1989. You can download and unzip the
compressed spreadsheets by using the unzip function. The result is set of files in your current folder
with names such as hourly_42602_1985.csv. Here, 42602 is an EPA code for NO2. (Data from the
US Environmental Protection Agency. Air Quality System Data Mart available via https://
www.epa.gov/airdata. Accessed July 15, 2021.)
yrs = string(1985:1989);
urls = "https://aqs.epa.gov/aqsweb/airdata/hourly_42602_" + yrs + ".zip";
fnames = strings(numel(yrs),1);
for ii = 1:numel(yrs)
fnames(ii) = unzip(urls(ii));
end
fnames
fnames = 5×1 string
"hourly_42602_1985.csv"
9 Tables
9-86
Grouped calculations can help you interpret large datasets such as time-series data. In such
calculations, you use a grouping variable to split a dataset into groups and apply a function to each
group. A grouping variable contains values, such as time periods or station locations, that you can use
to group other data values, such as temperature readings or atmospheric concentrations of a gas. In
MATLAB®, you can store such data in tables or timetables. With grouped calculations in a table you
can often calculate results in-place, in one table, instead of breaking data out into separate tables and
merging results later.
This example shows how to import nitrogen dioxide (NO2) data from the US Environmental
Protection Agency (EPA) into a table and do grouped calculations on this data. NO2 is one of the
Criteria Air Pollutants regulated under the US Clean Air Act. It is toxic by itself and is also a key
component of photochemical smog that results in ground-level ozone production. NO2 is produced
through high-temperature processes that can split nitrogen and oxygen gases and enable them to
recombine. Natural processes contribute NO2 to the atmosphere, but so do human activities such as
combustion in automobile engines and power plants, lightning, and biomass burning. The
concentration of NO2 in the atmosphere is also influenced by the photochemical cycling between NO
and NO2, atmospheric transport, and ultimately oxidation to nitric acid, causing acid rain. Different
processes contribute NO2 to the atmosphere on different timescales, leading to daily (diurnal),
weekly, and annual cycles in its atmospheric concentration. Time-series analysis of such data relies
heavily on grouped calculations to examine different periodic behavior or to average the data over
time to smooth out high-frequency variability and reveal long-term trends.
The example first shows how to do preliminary data cleaning, including conversion of the table to a
timetable. Then it shows simple ways to group the data by one grouping variable and calculate annual
mean NO2 concentrations. It also shows how to group the NO2 data by two grouping variables
together, time and location, enabling calculations that find locations exceeding EPA standards at
various times. You can also group the NO2 data by time period to look for daily or yearly cycles.
Finally it shows how to apply a function that requires inputs from multiple table variables to find the
times at which the maximum NO2 concentrations occurred at each site.
Import NO2 Data to Table
First, import NO2 data from the Air Quality System (AQS) database maintained by the EPA. This data
consists of hourly measurements of NO2 concentrations from outdoor monitors across the United
States, Puerto Rico, and the U.S. Virgin Islands. It is stored as a set of zipped spreadsheets, one for
each year starting with 1980.
Download hourly NO2 measurements for the years 1985–1989. You can download and unzip the
compressed spreadsheets by using the unzip function. The result is set of files in your current folder
with names such as hourly_42602_1985.csv. Here, 42602 is an EPA code for NO2. (Data from the
US Environmental Protection Agency. Air Quality System Data Mart available via https://
www.epa.gov/airdata. Accessed July 15, 2021.)
yrs = string(1985:1989);
urls = "https://aqs.epa.gov/aqsweb/airdata/hourly_42602_" + yrs + ".zip";
fnames = strings(numel(yrs),1);
for ii = 1:numel(yrs)
fnames(ii) = unzip(urls(ii));
end
fnames
fnames = 5×1 string
"hourly_42602_1985.csv"
9 Tables
9-86