Autonomy: Risk Assessment 24.5 Risk of Loss 537
Part B | 24.5
where responses can be used to calibrate individual
assessments.
5. Analyzing and aggregating: This is a critical phase
of the assessment. Here the decision maker uses
a strategy for combining the expert judgments into
a single assessment that represents the group’s view.
This can be done mathematically or behaviorally.
A mathematical aggregation uses analytical functions for combining the expert judgments. The analysis of expert judgments must be conducted prior to
the aggregation to ensure that any misunderstandings and bias are removed from the assessments.
6. Complete analysis and write-up: The assessments
provided by the experts are documented, and disagreements between experts are recorded. A report
is submitted to the experts for review. Following this
review, a report is submitted to the decision maker
informing the results of the risk assessment exercise.
Mathematical Aggregation
The use of mathematical methods for consensus building has the benefit of facilitating the application of
the elicitation process because experts are not asked to
agree on the final judgment; in fact experts do not have
to be in the same room during the elicitation. Supporters
of mathematical methods have argued that behavioral
methods, in addition to being time-consuming, can
lead to systematic biases caused by group polarization [24.34].
The analytical function for combining individual
judgments can take into account the level of expertise
in the topic and the effectiveness of the calibration.
These factors are captured in a weight w i , where i D
1; : : : ; n or in an a priori distribution judgment. Methods for mathematically aggregating expert judgments
were developed based on a set of widely accepted
axioms [24.35]. Perhaps the most appealing method
for combining expert judgments is the linear opinion
pool [24.36]. Here the expert judgments are aggregated
by taking the weighted average of the assessments provided by the experts. The linear opinion pool complies
with the property of consistent marginalization [24.37].
A different mathematical aggregation method, the logarithmic opinion pool, uses a multiplicative averaging of
the expert assessments [24.36]. The logarithmic opinion pool does not comply with the marginalization
principle but unlike the linear opinion pool it complies with the principle of external Bayesianity. This
can be explained as follows: suppose that the decision
maker or facilitator has reached an aggregated p. / but
has subsequently learned new information relevant to
 . One way to update the judgment p. / in light of
the new information is by updating the individual expert judgments p i . / and then aggregate all judgments
using an analytical function. A second way is to update the original p. / without updating the individual
expert judgments, using Bayes theory. A mathematical
aggregation method that complies with the principle of
Bayesianity will produce the same result regardless of
the process used for creating the final judgment.
Other mathematical aggregation methods are more
complete from the axiomatic viewpoint. But this comes
at a price of being more difficult to implement.
Bayesian aggregation methods adopt a completely different prerogative to the linear and logarithmic pools.
Here, the decision maker begins by defining his own
prior distribution f . /. The expert judgments are then
incorporated, using Bayes’ rule, to form the decision maker’s posterior distribution f . j D/, where D D
ff 1 . /; : : : ; f n . /g is the set of experts’ elicited distributions. In this context, according to Bayes theorem,
the f . j D/ is proportional to f . / multiplied by the
likelihood term f .D j Â/. The problem with Bayesian
methods is that the decision maker must specify his own
prior belief about the risk f . / and the knowledge and
beliefs about the experts f .D j Â/ [24.35].
In 2008, in preparation for the Autosub3 deployment under the Pine Island glacier, a risk model for
Autosub3 was developed based on the assessments provided by eight independent experts. Experts were asked
to assess the probability of AUV loss given that a fault X
emerges during the deployment [24.10]. In addition to
providing an estimated probability of loss, experts provided a weight, from 15, capturing their confidence
in the assessment. A weight of 1 represented little confidence in the assessment, while a weight of 5 meant
that the expert was very confident. The experts considered 63 faults, in four environments: open water, coastal
water, under sea ice, and under ice shelf. A detailed description of each environment was provided prior to the
assessment. The faults were collected during six Autosub3 campaigns from mid 2005 to mid 2008.
Figure 24.4a shows the relative frequency of the
average of the weights used by the experts. The figure shows that on average, for 70% of the assessments,
experts assigned a weight of 3 or higher, for open water, coastal waters, and ice shelf environments. Experts
were least confident with the sea ice environment.
Figure 24.4b shows the un-weighted linear pool
probability of loss, for the five most critical failures in
the four different environments. This makes it clear that,
in these experts’ opinion, faults that would have a low
probability of leading to loss in open or coastal water
would have a high probability of loss under an ice shelf,
but not through simple scaling.
Expert judgment has also been applied to estimating the risk of loss of commercial AUVs, including
two Remus 100 vehicles in coastal areas and under
Part B | 24.5
where responses can be used to calibrate individual
assessments.
5. Analyzing and aggregating: This is a critical phase
of the assessment. Here the decision maker uses
a strategy for combining the expert judgments into
a single assessment that represents the group’s view.
This can be done mathematically or behaviorally.
A mathematical aggregation uses analytical functions for combining the expert judgments. The analysis of expert judgments must be conducted prior to
the aggregation to ensure that any misunderstandings and bias are removed from the assessments.
6. Complete analysis and write-up: The assessments
provided by the experts are documented, and disagreements between experts are recorded. A report
is submitted to the experts for review. Following this
review, a report is submitted to the decision maker
informing the results of the risk assessment exercise.
Mathematical Aggregation
The use of mathematical methods for consensus building has the benefit of facilitating the application of
the elicitation process because experts are not asked to
agree on the final judgment; in fact experts do not have
to be in the same room during the elicitation. Supporters
of mathematical methods have argued that behavioral
methods, in addition to being time-consuming, can
lead to systematic biases caused by group polarization [24.34].
The analytical function for combining individual
judgments can take into account the level of expertise
in the topic and the effectiveness of the calibration.
These factors are captured in a weight w i , where i D
1; : : : ; n or in an a priori distribution judgment. Methods for mathematically aggregating expert judgments
were developed based on a set of widely accepted
axioms [24.35]. Perhaps the most appealing method
for combining expert judgments is the linear opinion
pool [24.36]. Here the expert judgments are aggregated
by taking the weighted average of the assessments provided by the experts. The linear opinion pool complies
with the property of consistent marginalization [24.37].
A different mathematical aggregation method, the logarithmic opinion pool, uses a multiplicative averaging of
the expert assessments [24.36]. The logarithmic opinion pool does not comply with the marginalization
principle but unlike the linear opinion pool it complies with the principle of external Bayesianity. This
can be explained as follows: suppose that the decision
maker or facilitator has reached an aggregated p. / but
has subsequently learned new information relevant to
 . One way to update the judgment p. / in light of
the new information is by updating the individual expert judgments p i . / and then aggregate all judgments
using an analytical function. A second way is to update the original p. / without updating the individual
expert judgments, using Bayes theory. A mathematical
aggregation method that complies with the principle of
Bayesianity will produce the same result regardless of
the process used for creating the final judgment.
Other mathematical aggregation methods are more
complete from the axiomatic viewpoint. But this comes
at a price of being more difficult to implement.
Bayesian aggregation methods adopt a completely different prerogative to the linear and logarithmic pools.
Here, the decision maker begins by defining his own
prior distribution f . /. The expert judgments are then
incorporated, using Bayes’ rule, to form the decision maker’s posterior distribution f . j D/, where D D
ff 1 . /; : : : ; f n . /g is the set of experts’ elicited distributions. In this context, according to Bayes theorem,
the f . j D/ is proportional to f . / multiplied by the
likelihood term f .D j Â/. The problem with Bayesian
methods is that the decision maker must specify his own
prior belief about the risk f . / and the knowledge and
beliefs about the experts f .D j Â/ [24.35].
In 2008, in preparation for the Autosub3 deployment under the Pine Island glacier, a risk model for
Autosub3 was developed based on the assessments provided by eight independent experts. Experts were asked
to assess the probability of AUV loss given that a fault X
emerges during the deployment [24.10]. In addition to
providing an estimated probability of loss, experts provided a weight, from 15, capturing their confidence
in the assessment. A weight of 1 represented little confidence in the assessment, while a weight of 5 meant
that the expert was very confident. The experts considered 63 faults, in four environments: open water, coastal
water, under sea ice, and under ice shelf. A detailed description of each environment was provided prior to the
assessment. The faults were collected during six Autosub3 campaigns from mid 2005 to mid 2008.
Figure 24.4a shows the relative frequency of the
average of the weights used by the experts. The figure shows that on average, for 70% of the assessments,
experts assigned a weight of 3 or higher, for open water, coastal waters, and ice shelf environments. Experts
were least confident with the sea ice environment.
Figure 24.4b shows the un-weighted linear pool
probability of loss, for the five most critical failures in
the four different environments. This makes it clear that,
in these experts’ opinion, faults that would have a low
probability of leading to loss in open or coastal water
would have a high probability of loss under an ice shelf,
but not through simple scaling.
Expert judgment has also been applied to estimating the risk of loss of commercial AUVs, including
two Remus 100 vehicles in coastal areas and under