Part B | 24.5
536 Part B Autonomous Ocean Vehicles, Subsystems and Control
in open-water operations; however, if the same failure
takes place while the vehicle is under an ice shelf, sea
ice, or busy coastal area it may lead to vehicle loss.
The human element is also of great importance when
it comes to failure mitigation, as highlighted by Stokey
et al. [24.11], many technical failures in AUV operations can be mitigated by a more experienced team.
These factors are difficult if not impossible to capture deterministically, particularly if the number of
previous deployments in the target operational environment is small or if the environment and operational conditions are impossible to predict. Autonomous ocean
systems are not unique in this challenge, and many techniques have been developed and widely accepted, for
risk quantification in areas such as nuclear, process, oil
and gas, and aerospace safety. Common to these approaches is the formal use of expert judgment.
In this section, we present formal expert judgment
elicitation methods focusing on their pragmatic characteristics for estimating AOV risk. When studying
component or systems reliability, it is not sufficient to
have the MTBF, designers and operators are often interested in knowing the component or system reliability
as a function of time in operation or mission length.
This is imperative for the implementation of corrective
measures for risk reduction. Similarly, when operating
AOVs the decision to go, or not go, for a mission is
better informed by a risk profile that captures the probability of loss with mission time or length. The creation
of a probability of loss profile is possible by integrating
experts’ subjective judgments with statistical survival
techniques.
24.5.1 Expert Judgment Elicitation
Processes
Governments and companies are increasingly relying on
expert panels for providing risk assessments for highrisk or high-profile projects such as the safety assessment of a nuclear power station or the implementation of
a major science program [24.28]. There are many dangers in eliciting expert judgments in an informal way.
When providing assessments of events, people often follow one of a number of mental shortcuts, denoted as
heuristics [24.29, 30]. Research has shown that when
used incorrectly, these heuristics can lead to systematic
and predictable bias [24.31]. In their work, Tversky and
Kahnemann [24.29] have identified three predominant
heuristics: representativeness, availability, and adjustment and anchoring. Representativeness is described as
the tendency to judge the probability that A belongs
to B by how representative A is to B. For example, if
the description of Peter is highly representative of an
engineer then people tend to judge the probability of
Peter being an engineer as high regardless of the base
rate, that is, the proportion of engineers in society. The
base rate neglect is one type of bias that can be introduced by following the representativeness heuristics;
other types of biases are, for example, insensitivity to
sample size, insensitivity to predictability, and misconceptions of chance and regression [24.29]. Availability
heuristics is the tendency to judge a particular event by
the ease with which the instances and events can be
brought to mind. For example, one may estimate the risk
of heart attack among middle-aged men by remembering specific personal examples. Here bias can be introduced due to exposure to negative outcomes; they tend
to be easier to remember than positive ones. Anchoring and adjustment is a tendency to anchor probability
estimates at an initial estimate. Insufficient adjustment
results in biases of underestimation or overestimation.
Formal judgment elicitation has been proposed as
a solution to reduce biases and facilitate reproducibility
of the results [24.32]. A formal expert judgment elicitation is a structured process that consists of a number
of phases; a number have been proposed in the literature [24.33]. Below we describe the generic structure of
a formal judgment elicitation process tailored to AOVs:
1. Set out the issues: In this phase, the issue that is to
be addressed is described in general terms. This is
a scope definition stage; it defines the nature and
direction of the analysis, the choice of questions to
be asked and issues to be considered. For example,
the issue of interest is to estimate the probability of
vehicle loss.
2. Selecting the experts: An expert is someone with
specialist knowledge in the task in hand who is also
capable of decomposing a complex problem into
subsidiary problems. In vehicle risk assessment, an
expert should be someone with experience in the
type of fault scenarios and in AOV operations.
3. Clearly define the issues: The issue in hand is what
is the probability of failure F leading to loss (L) in
operational environment E? This is mathematically
represented as P.L j F; E/.
4. Training the experts and eliciting judgments: This
is the most important task in the elicitation process.
Different formal judgment elicitation processes follow different approaches. For some, training is quite
informal it consists of providing some examples
of typical assessments and an explanation of basic principles of probability theory. Other judgment
elicitation processes are stricter and encourage the
facilitator to elicit expert assessments for a number
of seed questions. These are questions for which the
facilitator knows the answer but the expert is not
familiar with the problem or question [24.32], and
536 Part B Autonomous Ocean Vehicles, Subsystems and Control
in open-water operations; however, if the same failure
takes place while the vehicle is under an ice shelf, sea
ice, or busy coastal area it may lead to vehicle loss.
The human element is also of great importance when
it comes to failure mitigation, as highlighted by Stokey
et al. [24.11], many technical failures in AUV operations can be mitigated by a more experienced team.
These factors are difficult if not impossible to capture deterministically, particularly if the number of
previous deployments in the target operational environment is small or if the environment and operational conditions are impossible to predict. Autonomous ocean
systems are not unique in this challenge, and many techniques have been developed and widely accepted, for
risk quantification in areas such as nuclear, process, oil
and gas, and aerospace safety. Common to these approaches is the formal use of expert judgment.
In this section, we present formal expert judgment
elicitation methods focusing on their pragmatic characteristics for estimating AOV risk. When studying
component or systems reliability, it is not sufficient to
have the MTBF, designers and operators are often interested in knowing the component or system reliability
as a function of time in operation or mission length.
This is imperative for the implementation of corrective
measures for risk reduction. Similarly, when operating
AOVs the decision to go, or not go, for a mission is
better informed by a risk profile that captures the probability of loss with mission time or length. The creation
of a probability of loss profile is possible by integrating
experts’ subjective judgments with statistical survival
techniques.
24.5.1 Expert Judgment Elicitation
Processes
Governments and companies are increasingly relying on
expert panels for providing risk assessments for highrisk or high-profile projects such as the safety assessment of a nuclear power station or the implementation of
a major science program [24.28]. There are many dangers in eliciting expert judgments in an informal way.
When providing assessments of events, people often follow one of a number of mental shortcuts, denoted as
heuristics [24.29, 30]. Research has shown that when
used incorrectly, these heuristics can lead to systematic
and predictable bias [24.31]. In their work, Tversky and
Kahnemann [24.29] have identified three predominant
heuristics: representativeness, availability, and adjustment and anchoring. Representativeness is described as
the tendency to judge the probability that A belongs
to B by how representative A is to B. For example, if
the description of Peter is highly representative of an
engineer then people tend to judge the probability of
Peter being an engineer as high regardless of the base
rate, that is, the proportion of engineers in society. The
base rate neglect is one type of bias that can be introduced by following the representativeness heuristics;
other types of biases are, for example, insensitivity to
sample size, insensitivity to predictability, and misconceptions of chance and regression [24.29]. Availability
heuristics is the tendency to judge a particular event by
the ease with which the instances and events can be
brought to mind. For example, one may estimate the risk
of heart attack among middle-aged men by remembering specific personal examples. Here bias can be introduced due to exposure to negative outcomes; they tend
to be easier to remember than positive ones. Anchoring and adjustment is a tendency to anchor probability
estimates at an initial estimate. Insufficient adjustment
results in biases of underestimation or overestimation.
Formal judgment elicitation has been proposed as
a solution to reduce biases and facilitate reproducibility
of the results [24.32]. A formal expert judgment elicitation is a structured process that consists of a number
of phases; a number have been proposed in the literature [24.33]. Below we describe the generic structure of
a formal judgment elicitation process tailored to AOVs:
1. Set out the issues: In this phase, the issue that is to
be addressed is described in general terms. This is
a scope definition stage; it defines the nature and
direction of the analysis, the choice of questions to
be asked and issues to be considered. For example,
the issue of interest is to estimate the probability of
vehicle loss.
2. Selecting the experts: An expert is someone with
specialist knowledge in the task in hand who is also
capable of decomposing a complex problem into
subsidiary problems. In vehicle risk assessment, an
expert should be someone with experience in the
type of fault scenarios and in AOV operations.
3. Clearly define the issues: The issue in hand is what
is the probability of failure F leading to loss (L) in
operational environment E? This is mathematically
represented as P.L j F; E/.
4. Training the experts and eliciting judgments: This
is the most important task in the elicitation process.
Different formal judgment elicitation processes follow different approaches. For some, training is quite
informal it consists of providing some examples
of typical assessments and an explanation of basic principles of probability theory. Other judgment
elicitation processes are stricter and encourage the
facilitator to elicit expert assessments for a number
of seed questions. These are questions for which the
facilitator knows the answer but the expert is not
familiar with the problem or question [24.32], and