205
Improving Safety
humans, when designing a complex system. This is also an important question for
certification of the autonomous transportation system.
Sensemaking and the principle of meaningful human control should be used to verify that the proper functions are allocated to the human or the automation. According
to Santoni de Sio and van der Hoven ( 2019), two design requirements should be satisfied for an autonomous system to remain under meaningful human control:
1. A “ tracing” condition, according to which the system should be designed in
such a way as to grant the possibility to always trace back the outcome of its
operations to at least one human along the chain of design and operation.
2. A “ tracking” condition, according to which the system should be able to
respond to both the relevant moral reasons of the humans designing and
deploying the system and the relevant facts in the environment in which the
system operates.
From a safety perspective, this can be placed in the bowtie model, where the design
principle of tracking are barriers preventing a technical fault, threat or unexpected
situation to lead to a dangerous situation, as a human alway has established the possibility to intervene and take over control. On the other side of the bowtie, once a
hazard has emerged, the outcome can be reduced by designing after a tracing condition making it possible to trace back the operation to a human who is in the position
to understand the capabilities of the system and the possible effects in the world of its
use and, hence, knows how to limit the consequences of an undesired event.
CONCLUSION
We have given a summary of ongoing projects and safety issues. The main issues
across the domains are technical reliability and maturity, the need for automation
transparency ( including awareness for the decision made by automation), the need
for defining what conditions the system can operate under and assigning responsibilities to human operators and the automation. Experiences from known accidents
involving a high level of automation, as in the cases of Boeing 737 MAX, Uber and
Tesla, have shown overreliance on automation and poor understanding of capabilities
and limitations. We need to collect and systemise data on accidents and incidents of
autonomous transportation systems and design with human factor practice to support
sensemaking and meaningful human control.
Design principles from meaningful human control should be used to verify if
the interaction between automation and the human is safe. This can be used as an
input to operational envelopes and to assist in the design of a good HAI supporting
sensemaking.
ACKNOWLEDGEMENT
This chapter has been funded by the Norwegian Research C ouncil – project 267860
SAREPTA.
Improving Safety
humans, when designing a complex system. This is also an important question for
certification of the autonomous transportation system.
Sensemaking and the principle of meaningful human control should be used to verify that the proper functions are allocated to the human or the automation. According
to Santoni de Sio and van der Hoven ( 2019), two design requirements should be satisfied for an autonomous system to remain under meaningful human control:
1. A “ tracing” condition, according to which the system should be designed in
such a way as to grant the possibility to always trace back the outcome of its
operations to at least one human along the chain of design and operation.
2. A “ tracking” condition, according to which the system should be able to
respond to both the relevant moral reasons of the humans designing and
deploying the system and the relevant facts in the environment in which the
system operates.
From a safety perspective, this can be placed in the bowtie model, where the design
principle of tracking are barriers preventing a technical fault, threat or unexpected
situation to lead to a dangerous situation, as a human alway has established the possibility to intervene and take over control. On the other side of the bowtie, once a
hazard has emerged, the outcome can be reduced by designing after a tracing condition making it possible to trace back the operation to a human who is in the position
to understand the capabilities of the system and the possible effects in the world of its
use and, hence, knows how to limit the consequences of an undesired event.
CONCLUSION
We have given a summary of ongoing projects and safety issues. The main issues
across the domains are technical reliability and maturity, the need for automation
transparency ( including awareness for the decision made by automation), the need
for defining what conditions the system can operate under and assigning responsibilities to human operators and the automation. Experiences from known accidents
involving a high level of automation, as in the cases of Boeing 737 MAX, Uber and
Tesla, have shown overreliance on automation and poor understanding of capabilities
and limitations. We need to collect and systemise data on accidents and incidents of
autonomous transportation systems and design with human factor practice to support
sensemaking and meaningful human control.
Design principles from meaningful human control should be used to verify if
the interaction between automation and the human is safe. This can be used as an
input to operational envelopes and to assist in the design of a good HAI supporting
sensemaking.
ACKNOWLEDGEMENT
This chapter has been funded by the Norwegian Research C ouncil – project 267860
SAREPTA.