7 Intelligent and Connected Cyber-Physical Systems: A Perspective. . .
379
Such requirements provide a clear measure of performance for the machine
learning environment, but also imply a number of assumptions on the system
context. These assumptions might include that the braking distance and speed are
sufficient to react when detecting persons, for example, 100 m ahead on the planned
trajectory of the vehicle and that other system measures (e.g., alternative sensing
channels) are available to decrease the overall false-negative and false-positive rate
to a sufficiently safe level, etc. The following list contains typical assumptions that
are relevant for the assurance case:
• Assumptions on the operational profile of the system’s environment. For example, the types and occurrence distribution of objects in the environment.
• Assumptions on attributes of inputs to the machine learning function. For
example, the camera resolution is sufficient to be able to detect persons from
a distance of 100 m with the required accuracy.
• Assumptions on the performance potential of machine learning. For example, the
chosen CNN approach has the intrinsic potential, given the right parameterization
and set of learning data to fulfil the allocated performance requirements.
Arguing safety when applying machine learning functions therefore not only
requires a demonstration that the performance requirements are met, especially
when evaluated on a specific data set that might be not representative for all driving
situations. Moreover, it is necessary to argue that requirements are appropriate given
the role of the function in the overall automated driving system and environmental
context and that all relevant assumptions have been explicitly defined and validated.
7.3.1.3 Causes of Functional Insufficiencies in Machine Learning
The inherent uncertainty associated with machine learning techniques coupled
with the open context environment lead to different causes of performance issues
compared to traditional, algorithmic, and control-theoretic approaches to vehicle
control. In order to argue the claim that functional insufficiencies within the machine
learning function (here: camera-based object detection, supervised training) are
minimized, it is important to understand the causes of such insufficiencies. As
interest in machine learning safety has grown, a number of authors [13, 14, 15]
have investigated different causes of performance limitations in machine learning
functions. Some examples applicable to the use case described here are described
below:
• Scalable oversight and distributional shift. One of the key differences in machine
learning techniques compared to algorithmic approaches is the lack of a detailed
specification of the target function. Instead, the functional specification can be
seen to be encoded within the set of training data. Therefore, if the training data
does not reflect the target operating context, then there is a strong likelihood that
the learned function will exhibit insufficiencies. Critical or ambiguous situations,
within which the system must react in a predictably safe manner, may occur
379
Such requirements provide a clear measure of performance for the machine
learning environment, but also imply a number of assumptions on the system
context. These assumptions might include that the braking distance and speed are
sufficient to react when detecting persons, for example, 100 m ahead on the planned
trajectory of the vehicle and that other system measures (e.g., alternative sensing
channels) are available to decrease the overall false-negative and false-positive rate
to a sufficiently safe level, etc. The following list contains typical assumptions that
are relevant for the assurance case:
• Assumptions on the operational profile of the system’s environment. For example, the types and occurrence distribution of objects in the environment.
• Assumptions on attributes of inputs to the machine learning function. For
example, the camera resolution is sufficient to be able to detect persons from
a distance of 100 m with the required accuracy.
• Assumptions on the performance potential of machine learning. For example, the
chosen CNN approach has the intrinsic potential, given the right parameterization
and set of learning data to fulfil the allocated performance requirements.
Arguing safety when applying machine learning functions therefore not only
requires a demonstration that the performance requirements are met, especially
when evaluated on a specific data set that might be not representative for all driving
situations. Moreover, it is necessary to argue that requirements are appropriate given
the role of the function in the overall automated driving system and environmental
context and that all relevant assumptions have been explicitly defined and validated.
7.3.1.3 Causes of Functional Insufficiencies in Machine Learning
The inherent uncertainty associated with machine learning techniques coupled
with the open context environment lead to different causes of performance issues
compared to traditional, algorithmic, and control-theoretic approaches to vehicle
control. In order to argue the claim that functional insufficiencies within the machine
learning function (here: camera-based object detection, supervised training) are
minimized, it is important to understand the causes of such insufficiencies. As
interest in machine learning safety has grown, a number of authors [13, 14, 15]
have investigated different causes of performance limitations in machine learning
functions. Some examples applicable to the use case described here are described
below:
• Scalable oversight and distributional shift. One of the key differences in machine
learning techniques compared to algorithmic approaches is the lack of a detailed
specification of the target function. Instead, the functional specification can be
seen to be encoded within the set of training data. Therefore, if the training data
does not reflect the target operating context, then there is a strong likelihood that
the learned function will exhibit insufficiencies. Critical or ambiguous situations,
within which the system must react in a predictably safe manner, may occur