11 Applicability Domain: Towards a More Formal …
227
Fig. 11.10 Result of a prediction can be expressed in the form of a distribution of likelihood for
each possible class in the case of classification models, or value in the case of regression estimators.
If the likelihood for a given class or value stands out, then the prediction is more decisive and induces
high confidence (a, c); if all outcomes have similar likelihoods, then the prediction is inconclusive
and induces low confidence (b, d)
11.3 Framework
We saw that the confidence in a prediction is the result of the combination of three
different aspects that can be handled separately in a cascade of assessments. The first
step in this sequence is to ensure that the model is suitable for the intended task and
can therefore be applied (applicability). If the model is applicable, the prediction
will be valid, and a first level of confidence is met. In the second step, the quantity,
quality and relevance of the supporting evidence is assessed in order to estimate how
reliable the prediction is (Reliability). If the prediction is reliable enough for a given
use case, it provides an additional level of confidence. Finally, if the prediction is
both, valid and reliable, then the last level of confidence depends on how decisive
the model is in terms of its results (Decidability). If the model expresses a decisive
outcome, then the confidence in the valid, reliable and conclusive prediction is high
(Fig. 11.11). Note that the chronology of these steps is important. If a model cannot
be applied, it is not legitimate to do a prediction and is therefore meaningless to
estimate the reliability; similarly, if the prediction is valid but not reliable, its result
cannot be trusted regardless of how decisive the outcome is, since this assertiveness
is itself not reliable.
11.4 Tardis
The confidence in a model’s outcome as defined by the decision domain is capturing
the objective and intrinsic components of a prediction. When used in the context
of decision-making and especially in the context of risk assessment, users leverage their expertise to validate the QSAR predictions; the latter should therefore be
interpretable and provide information regarding the evidence used by the model to
build a conclusion. For instance, a kNN model may expose the k-nearest neighbours
used to construct the prediction. Understanding the model’s rational and accessing
227
Fig. 11.10 Result of a prediction can be expressed in the form of a distribution of likelihood for
each possible class in the case of classification models, or value in the case of regression estimators.
If the likelihood for a given class or value stands out, then the prediction is more decisive and induces
high confidence (a, c); if all outcomes have similar likelihoods, then the prediction is inconclusive
and induces low confidence (b, d)
11.3 Framework
We saw that the confidence in a prediction is the result of the combination of three
different aspects that can be handled separately in a cascade of assessments. The first
step in this sequence is to ensure that the model is suitable for the intended task and
can therefore be applied (applicability). If the model is applicable, the prediction
will be valid, and a first level of confidence is met. In the second step, the quantity,
quality and relevance of the supporting evidence is assessed in order to estimate how
reliable the prediction is (Reliability). If the prediction is reliable enough for a given
use case, it provides an additional level of confidence. Finally, if the prediction is
both, valid and reliable, then the last level of confidence depends on how decisive
the model is in terms of its results (Decidability). If the model expresses a decisive
outcome, then the confidence in the valid, reliable and conclusive prediction is high
(Fig. 11.11). Note that the chronology of these steps is important. If a model cannot
be applied, it is not legitimate to do a prediction and is therefore meaningless to
estimate the reliability; similarly, if the prediction is valid but not reliable, its result
cannot be trusted regardless of how decisive the outcome is, since this assertiveness
is itself not reliable.
11.4 Tardis
The confidence in a model’s outcome as defined by the decision domain is capturing
the objective and intrinsic components of a prediction. When used in the context
of decision-making and especially in the context of risk assessment, users leverage their expertise to validate the QSAR predictions; the latter should therefore be
interpretable and provide information regarding the evidence used by the model to
build a conclusion. For instance, a kNN model may expose the k-nearest neighbours
used to construct the prediction. Understanding the model’s rational and accessing