626
P.-Y. S. Hsueh et al.
In another follow-up evaluation study, the subjects also provided positive
feedback to the knowledge delivery strategies. In addition, another expert group
was also consulted to assess the validity of the knowledge delivery strategies. In
this follow-up evaluation study, the experts were asked to judge the orderings of
knowledge resources for given profiles. The test scenarios used in this part of
evaluation included not only the orders obtained by the MoCaB system but also
artificial orders that were designed to be (partly) incorrect. This evaluation process
was repeated a few weeks later with the same group of experts.
The evaluation results show that the experts were able to capture and correct
inconsistencies in the implicit part of the domain model. But on the one hand,
it is not possible to find all inconsistencies with this approach, and, on the other
hand, the experts’ judgments from the first and second repetition differed in
some cases, illustrating the elusiveness of their implicit knowledge. Moreover, this
evaluation only incorporates the opinions from a small number of experts, leaving
its generalizability to be questionable.
However, in this case study, the focus is on educating informal caregivers by
providing them personalized knowledge, and this group of users should be able
to judge and provide feedback on whether the knowledge profile created for them
fits their own expectation. By utilizing the additional information, the system
can address the abovementioned problems by adding an algorithm-driven learning
capacity. Since the internal structure of delivery strategy after feedback optimization
is not known, the algorithm must have the capability to adapt. For example, an
artificial neural network (ANN) is particularly suitable for its strength in capturing
intricate relationship in high dimensional data.
Even though the knowledge delivery algorithms mentioned above seem to work
quite well [101], it could be further improved by user feedback. To achieve this, both
the domain model and the tutoring model should be integrated into the artificial
neural network. There are multiple ways to implement this. The first option is to
train the ANN for importance scoring. Because the goal of the overall task is to
determine the ordering of available knowledge resources, the ANN could calculate
the scores for all knowledge resources in parallel, which then must be ordered
externally. Another option would be to train the ANN for both scoring and ordering.
To enable the second option, a more complex network structure is needed, which
would in turn result in a much longer training phase. The extra effort is non-trivial,
compared to simply applying a common sorting algorithm.
The schematic representation of a working ANN structure is shown in Fig. 13.5.
It first receives the input from the student model (via the CBI and NBA assessments) and returns a score for the importance of every knowledge resource, each
represented by an output neuron. In this setup, the neural network must extract the
implicit knowledge (e.g., experts’ domain knowledge and the delivery strategies
from the tutoring model) from observation data while training. The regression
models themselves are each performed by a strain inside the network. The strains
consist of three fully connected layers. Whereas one layer is for extracting the
expert weightings while training, the other two layers are for regression itself. In
the first training step, the artificially created profiles are used to teach the ANN
P.-Y. S. Hsueh et al.
In another follow-up evaluation study, the subjects also provided positive
feedback to the knowledge delivery strategies. In addition, another expert group
was also consulted to assess the validity of the knowledge delivery strategies. In
this follow-up evaluation study, the experts were asked to judge the orderings of
knowledge resources for given profiles. The test scenarios used in this part of
evaluation included not only the orders obtained by the MoCaB system but also
artificial orders that were designed to be (partly) incorrect. This evaluation process
was repeated a few weeks later with the same group of experts.
The evaluation results show that the experts were able to capture and correct
inconsistencies in the implicit part of the domain model. But on the one hand,
it is not possible to find all inconsistencies with this approach, and, on the other
hand, the experts’ judgments from the first and second repetition differed in
some cases, illustrating the elusiveness of their implicit knowledge. Moreover, this
evaluation only incorporates the opinions from a small number of experts, leaving
its generalizability to be questionable.
However, in this case study, the focus is on educating informal caregivers by
providing them personalized knowledge, and this group of users should be able
to judge and provide feedback on whether the knowledge profile created for them
fits their own expectation. By utilizing the additional information, the system
can address the abovementioned problems by adding an algorithm-driven learning
capacity. Since the internal structure of delivery strategy after feedback optimization
is not known, the algorithm must have the capability to adapt. For example, an
artificial neural network (ANN) is particularly suitable for its strength in capturing
intricate relationship in high dimensional data.
Even though the knowledge delivery algorithms mentioned above seem to work
quite well [101], it could be further improved by user feedback. To achieve this, both
the domain model and the tutoring model should be integrated into the artificial
neural network. There are multiple ways to implement this. The first option is to
train the ANN for importance scoring. Because the goal of the overall task is to
determine the ordering of available knowledge resources, the ANN could calculate
the scores for all knowledge resources in parallel, which then must be ordered
externally. Another option would be to train the ANN for both scoring and ordering.
To enable the second option, a more complex network structure is needed, which
would in turn result in a much longer training phase. The extra effort is non-trivial,
compared to simply applying a common sorting algorithm.
The schematic representation of a working ANN structure is shown in Fig. 13.5.
It first receives the input from the student model (via the CBI and NBA assessments) and returns a score for the importance of every knowledge resource, each
represented by an output neuron. In this setup, the neural network must extract the
implicit knowledge (e.g., experts’ domain knowledge and the delivery strategies
from the tutoring model) from observation data while training. The regression
models themselves are each performed by a strain inside the network. The strains
consist of three fully connected layers. Whereas one layer is for extracting the
expert weightings while training, the other two layers are for regression itself. In
the first training step, the artificially created profiles are used to teach the ANN