226
T. Hanser et al.
Fig. 11.9 When supporting evidence converge towards the same conclusion, the model can build a
more decisive prediction, whereas if the supporting information is self-contradicting, the prediction
will be equivocal and therefore less decisive. Note that, in both cases, the reliability of the prediction
is the same (same number of data points, same distance and same dispersion), only the decidability
is different
In the case of the Derek Nexus expert system, the assertiveness in the prediction
is expressed by the reasoning engine, using one of the following likelihood levels:
impossible, improbable, doubted, equivocal, plausible, probable or certain [28].
If a class (or a value) is significantly more likely than others, it suggests that
the model is more decisive about the outcome; the user can have more trust in the
prediction and in turn, take a more confident decision based on this prediction. On
the other hand, if all the possible outcomes have similar likelihoods then the model
is not decisive, the prediction is inconclusive and the user is less confident when
making a decision. The decidability level can be observed by the gap between class
likelihoods or, in the case of regression, by the standard deviation of the predicted
value’s likelihood distribution (Fig. 11.10). This principle can be seen from the
distance to the decision boundary perspective; if the query compound lies far from
the decision boundary of the model, then the prediction will be decisive, whereas, if
the query is close to the decision boundary, the prediction is equivocal and therefore
non-decisive.
The level of decidability can be expressed in the form of a real value ranging
from 0 (all outcomes have equal likelihood) and 1 (the model is certain of a specific outcome). Like for the reliability, depending on the use case, the required level
of decidability may vary. For instance, it is not desirable to take a risk assessment
decision based on an inconclusive prediction and a high decidability threshold may
therefore be set. On the other hand, in the context of virtual screening, it is possible
to prioritize compounds using the relative values of decidability for given desired
properties and a low level of decidability may be sufficient. The decidability level
should be calibrated to actually reflect the accuracy expectation (correlation between
decidability and observed accuracy); conformal predictors provide a mathematical
framework to achieve this calibration and can be applied to any prediction methodologies [29, 30].
T. Hanser et al.
Fig. 11.9 When supporting evidence converge towards the same conclusion, the model can build a
more decisive prediction, whereas if the supporting information is self-contradicting, the prediction
will be equivocal and therefore less decisive. Note that, in both cases, the reliability of the prediction
is the same (same number of data points, same distance and same dispersion), only the decidability
is different
In the case of the Derek Nexus expert system, the assertiveness in the prediction
is expressed by the reasoning engine, using one of the following likelihood levels:
impossible, improbable, doubted, equivocal, plausible, probable or certain [28].
If a class (or a value) is significantly more likely than others, it suggests that
the model is more decisive about the outcome; the user can have more trust in the
prediction and in turn, take a more confident decision based on this prediction. On
the other hand, if all the possible outcomes have similar likelihoods then the model
is not decisive, the prediction is inconclusive and the user is less confident when
making a decision. The decidability level can be observed by the gap between class
likelihoods or, in the case of regression, by the standard deviation of the predicted
value’s likelihood distribution (Fig. 11.10). This principle can be seen from the
distance to the decision boundary perspective; if the query compound lies far from
the decision boundary of the model, then the prediction will be decisive, whereas, if
the query is close to the decision boundary, the prediction is equivocal and therefore
non-decisive.
The level of decidability can be expressed in the form of a real value ranging
from 0 (all outcomes have equal likelihood) and 1 (the model is certain of a specific outcome). Like for the reliability, depending on the use case, the required level
of decidability may vary. For instance, it is not desirable to take a risk assessment
decision based on an inconclusive prediction and a high decidability threshold may
therefore be set. On the other hand, in the context of virtual screening, it is possible
to prioritize compounds using the relative values of decidability for given desired
properties and a low level of decidability may be sufficient. The decidability level
should be calibrated to actually reflect the accuracy expectation (correlation between
decidability and observed accuracy); conformal predictors provide a mathematical
framework to achieve this calibration and can be applied to any prediction methodologies [29, 30].