616
P.-Y. S. Hsueh et al.
delivery of interventions. This method applies a regression model to approximate
the expected reward with an additional threshold finding step, referred to as multistage threshold Q-learning (mTQL), in order to produce explainable policies. The
estimated policy is a tree-structured policy that maps from historical observations to
a next-step recommendation, with the split threshold as a sub-goal of intermediate
reward change that helps differentiate whether a subject can obtain an optimal outcome by adopting a different strategy. By applying mTQL to study the relationship
between stress and exercise behavior, the task of multi-stage stress management can
be cast as a sequential decision-making task of recommending exercise based on
micro-level feedback for stress reduction [68, 69].
In the stress-activity case study, the mTQL models stress reduction as the reward
to maximize and exercise behavior as the action to intervene. The action defined
as a binary variable to indicate whether the mean duration of daily moderate or
vigorous physical activity (MVPA) bout over the time period was greater than
30 minutes or not. The stress reduction is defined as the stress level difference of
current stage from a previous stage. The estimated policy is then used to understand
how exercise patterns affect users’ perceived stress levels and to perform coaching
more effectively.
An example of the estimated policy is shown in Fig. 13.2. For individuals who
have baseline stress level lower than a certain threshold, it would be important to
observe whether being active in the first week followed by a decrease of mean stress
level in the next week. If the mean stress level decreases less than a threshold, then
it is better not to suggest the active action any further to accommodate the individual
behavioral preferences and barriers. The analysis results show that dividing the
study period into multiple stages and incorporating more sub-goals for microlevel feedback potentially help individuals achieving better stress reduction. Also,
when an individual has started going astray from his usual path, a corresponding
recommendation should be proposed for adaptation.
Moreover, another challenge is inherent in the large volume of mobile and
IOT e-health data, which can often be collected continuously over time due to
sensing technologies. In order to extract key features from these time series, [70]
proposed a novel quantile coarsening analysis (QCA) and applied to analyze daily
physical activity data. The key idea of QCA is to represent a time series by a
small number of quantiles of time of activity. By construct of this representation,
QCA is effective at capturing the time regions with high level of activities, in
addition to the magnitude of activity. These are usual key features of behavioral
patterns. In the analysis of about 20,000 time series of daily activity counts, [70]
represented each time series of 1440 time points by 19 quantile features (1.3%
of original data), used the quantiles as inputs in unsupervised learning, and was
able to identify clusters that were distinctly associated with users’ characteristics
(e.g., employment status) and extraneous variables (e.g., weekday vs weekend).
From a computational viewpoint, since quantile transformation requires only simple
and scalable computations, QCA can be efficiently applied to large-scale data and
facilitates “on-the-device” computations in mobile phones. While developed in the
context of activity counts, QCA is a versatile analytical tool that can be applied to
P.-Y. S. Hsueh et al.
delivery of interventions. This method applies a regression model to approximate
the expected reward with an additional threshold finding step, referred to as multistage threshold Q-learning (mTQL), in order to produce explainable policies. The
estimated policy is a tree-structured policy that maps from historical observations to
a next-step recommendation, with the split threshold as a sub-goal of intermediate
reward change that helps differentiate whether a subject can obtain an optimal outcome by adopting a different strategy. By applying mTQL to study the relationship
between stress and exercise behavior, the task of multi-stage stress management can
be cast as a sequential decision-making task of recommending exercise based on
micro-level feedback for stress reduction [68, 69].
In the stress-activity case study, the mTQL models stress reduction as the reward
to maximize and exercise behavior as the action to intervene. The action defined
as a binary variable to indicate whether the mean duration of daily moderate or
vigorous physical activity (MVPA) bout over the time period was greater than
30 minutes or not. The stress reduction is defined as the stress level difference of
current stage from a previous stage. The estimated policy is then used to understand
how exercise patterns affect users’ perceived stress levels and to perform coaching
more effectively.
An example of the estimated policy is shown in Fig. 13.2. For individuals who
have baseline stress level lower than a certain threshold, it would be important to
observe whether being active in the first week followed by a decrease of mean stress
level in the next week. If the mean stress level decreases less than a threshold, then
it is better not to suggest the active action any further to accommodate the individual
behavioral preferences and barriers. The analysis results show that dividing the
study period into multiple stages and incorporating more sub-goals for microlevel feedback potentially help individuals achieving better stress reduction. Also,
when an individual has started going astray from his usual path, a corresponding
recommendation should be proposed for adaptation.
Moreover, another challenge is inherent in the large volume of mobile and
IOT e-health data, which can often be collected continuously over time due to
sensing technologies. In order to extract key features from these time series, [70]
proposed a novel quantile coarsening analysis (QCA) and applied to analyze daily
physical activity data. The key idea of QCA is to represent a time series by a
small number of quantiles of time of activity. By construct of this representation,
QCA is effective at capturing the time regions with high level of activities, in
addition to the magnitude of activity. These are usual key features of behavioral
patterns. In the analysis of about 20,000 time series of daily activity counts, [70]
represented each time series of 1440 time points by 19 quantile features (1.3%
of original data), used the quantiles as inputs in unsupervised learning, and was
able to identify clusters that were distinctly associated with users’ characteristics
(e.g., employment status) and extraneous variables (e.g., weekday vs weekend).
From a computational viewpoint, since quantile transformation requires only simple
and scalable computations, QCA can be efficiently applied to large-scale data and
facilitates “on-the-device” computations in mobile phones. While developed in the
context of activity counts, QCA is a versatile analytical tool that can be applied to