193
Improving Safety
domains: road, sea, rail and air. Through these case studies, we aim to explore safety,
security, sensemaking and the human control of autonomous transport systems. We
have adopted the term “ meaningful human control” from discussion and debates
from another area ( lethal autonomous weapon systems; Cummings, 2019). The term
addresses the concerns of a “ responsibility gap” for harms caused by these systems,
i.e. humans, not computers and their algorithms should ultimately remain in control
of, and thus morally responsible for, relevant decisions about military operations.
The same concern must be the result of autonomous systems in transportation, i.e.
humans ( supported by computers and algorithms) should ultimately remain in control and responsible for relevant decisions. The responsibility may be on the designer
and producer of the autonomous systems, as Volvo and Mercedes Benz have stated
for their autonomous cars ( Chinen, 2019, p. 109).
BACKGROUND: SAFETY OF AUTONOMOUS SYSTEMS
Safety is commonly defined as freedom from unacceptable risk ( Hollnagel et al.,
2008). For autonomous transportation to become a success, It must prove to be at least
as safe and reliable as today’s transport systems. By some, it is claimed that increased
safety will be achieved by reducing the likelihood of human error when introducing
more autonomy ( Ramos et al., 2018). However, autonomy may create new types of
accidents that before were averted by the human in control, as demonstrated by the
Tesla fatal accident with Joshua Brown, NTSB ( 2017). Besides, the introduction of
new technology will create new accident types, as explained by Porathe et al. ( 2018),
Teoh and Kidd ( 2017), and Endsley ( 2019). The main safety challenges for autonomous systems are unexpected incidents not foreseen by automation, cybersecurity
TABLE 12.1
Levels of Automation – Simplified Description from SAE J3016 ( 2018).
Examples of automated
LoA
Humans in control Automation in control
features
0: No driving
All operations
No automated task. Warns;
Blind-spot monitoring
automation
protect
and lane-departure
warning
1: Driver assistance
All operations
Single automated systems:
Adaptive cruise control
assists
(ACC)
2: Limited assist; auto
Drives in-the-loop
Guides
Automated lane centring
throttle
combined with ACC
3: Assist, tactical;
On-the loop human Manage movement within
“ Traffic jam chauffeur”
supervised
monitors all time
defined limits
4: Automated assist
Out-of-loop asked
Operates, but may give back Self-driving mode with
strategic
by system
control
geofencing
5: Autonomous
Completely
Operates with graceful
None are yet available to
out-of-loop
degradation
the general public
Improving Safety
domains: road, sea, rail and air. Through these case studies, we aim to explore safety,
security, sensemaking and the human control of autonomous transport systems. We
have adopted the term “ meaningful human control” from discussion and debates
from another area ( lethal autonomous weapon systems; Cummings, 2019). The term
addresses the concerns of a “ responsibility gap” for harms caused by these systems,
i.e. humans, not computers and their algorithms should ultimately remain in control
of, and thus morally responsible for, relevant decisions about military operations.
The same concern must be the result of autonomous systems in transportation, i.e.
humans ( supported by computers and algorithms) should ultimately remain in control and responsible for relevant decisions. The responsibility may be on the designer
and producer of the autonomous systems, as Volvo and Mercedes Benz have stated
for their autonomous cars ( Chinen, 2019, p. 109).
BACKGROUND: SAFETY OF AUTONOMOUS SYSTEMS
Safety is commonly defined as freedom from unacceptable risk ( Hollnagel et al.,
2008). For autonomous transportation to become a success, It must prove to be at least
as safe and reliable as today’s transport systems. By some, it is claimed that increased
safety will be achieved by reducing the likelihood of human error when introducing
more autonomy ( Ramos et al., 2018). However, autonomy may create new types of
accidents that before were averted by the human in control, as demonstrated by the
Tesla fatal accident with Joshua Brown, NTSB ( 2017). Besides, the introduction of
new technology will create new accident types, as explained by Porathe et al. ( 2018),
Teoh and Kidd ( 2017), and Endsley ( 2019). The main safety challenges for autonomous systems are unexpected incidents not foreseen by automation, cybersecurity
TABLE 12.1
Levels of Automation – Simplified Description from SAE J3016 ( 2018).
Examples of automated
LoA
Humans in control Automation in control
features
0: No driving
All operations
No automated task. Warns;
Blind-spot monitoring
automation
protect
and lane-departure
warning
1: Driver assistance
All operations
Single automated systems:
Adaptive cruise control
assists
(ACC)
2: Limited assist; auto
Drives in-the-loop
Guides
Automated lane centring
throttle
combined with ACC
3: Assist, tactical;
On-the loop human Manage movement within
“ Traffic jam chauffeur”
supervised
monitors all time
defined limits
4: Automated assist
Out-of-loop asked
Operates, but may give back Self-driving mode with
strategic
by system
control
geofencing
5: Autonomous
Completely
Operates with graceful
None are yet available to
out-of-loop
degradation
the general public