216
T. Hanser et al.
kNN
K-Nearest Neighbours
QSAR Quantitative Structure Activity Relationship
OECD Organization of Economic Co-operation and Development
PCA
Principal Component Analysis
11.1 Introduction: Confidence in an Individual Prediction
Quantitative/Qualitative Structure Activity Relationship (QSAR) models can be used
in different contexts ranging from early virtual screening to late safety assessment
in the process of drug development. The ability to gauge the level of confidence
in predictions provided by such models becomes more and more important as the
drug candidates get closer to human exposure. In the early context of screening large
chemical libraries in the quest for active structures, it is acceptable for a model to make
a certain number of mistakes provided that the number of erroneous predictions is not
too big. In such a case, the global accuracy of a model gives us a sufficient estimate of
the confidence in the predictions and we expect a model with a good accuracy to lead
to a useful selection across the whole compound library. The model’s accuracy can
be measured a priori using internal and external validation methods and provides an
intrinsic sense of confidence in the predictions from a statistical standpoint; however,
it does not tell us how much we can trust individual predictions. On the other hand,
when it comes to human safety assessment, we are measuring the risk of adverse
events induced by a specific molecule. In this case, the global accuracy of the model is
not helpful, and we need a way to estimate the accuracy of an individual prediction.
Whereas in the context of virtual screening, we use the accuracy of the model,
in the context of human safety assessment, we need the accuracy of a prediction.
Model accuracy and prediction accuracy are two very different concepts. To better
illustrate the distinction, let us assume that we have built a good model providing
an 83% accuracy on a challenging toxicity prediction task. If we use this model in
the context of screening thousands of compounds, we can be relatively confident;
indeed, it feels like going to the casino playing with strongly biased dice; we know
that in the long run, we will achieve substantial gain (Fig. 11.1a). However, in the
context of risk assessment, we focus on one individual prediction with potentially
life-threatening consequences. In this case, the same 83% is a one in six chances of
a lethal outcome (Fig. 11.1b). The two different contexts lead to two distinct ways
to consider confidence in predictions.
When focusing on a specific compound, the question becomes “Can we trust this
specific individual prediction?” which combines elements of assessing the legitimacy
of the model, the reliability of the prediction for a specific compound and the level
of uncertainty that can be tolerated in making a decision. All these aspects make
perceiving the confidence in a prediction difficult to formalise. An attempt to formally
describe the legitimate scope of a model has been introduced as the AD. The role of
the AD is to define the boundaries within which a model can be used and provides
sufficiently accurate predictions. A well-defined AD has become a key feature for
T. Hanser et al.
kNN
K-Nearest Neighbours
QSAR Quantitative Structure Activity Relationship
OECD Organization of Economic Co-operation and Development
PCA
Principal Component Analysis
11.1 Introduction: Confidence in an Individual Prediction
Quantitative/Qualitative Structure Activity Relationship (QSAR) models can be used
in different contexts ranging from early virtual screening to late safety assessment
in the process of drug development. The ability to gauge the level of confidence
in predictions provided by such models becomes more and more important as the
drug candidates get closer to human exposure. In the early context of screening large
chemical libraries in the quest for active structures, it is acceptable for a model to make
a certain number of mistakes provided that the number of erroneous predictions is not
too big. In such a case, the global accuracy of a model gives us a sufficient estimate of
the confidence in the predictions and we expect a model with a good accuracy to lead
to a useful selection across the whole compound library. The model’s accuracy can
be measured a priori using internal and external validation methods and provides an
intrinsic sense of confidence in the predictions from a statistical standpoint; however,
it does not tell us how much we can trust individual predictions. On the other hand,
when it comes to human safety assessment, we are measuring the risk of adverse
events induced by a specific molecule. In this case, the global accuracy of the model is
not helpful, and we need a way to estimate the accuracy of an individual prediction.
Whereas in the context of virtual screening, we use the accuracy of the model,
in the context of human safety assessment, we need the accuracy of a prediction.
Model accuracy and prediction accuracy are two very different concepts. To better
illustrate the distinction, let us assume that we have built a good model providing
an 83% accuracy on a challenging toxicity prediction task. If we use this model in
the context of screening thousands of compounds, we can be relatively confident;
indeed, it feels like going to the casino playing with strongly biased dice; we know
that in the long run, we will achieve substantial gain (Fig. 11.1a). However, in the
context of risk assessment, we focus on one individual prediction with potentially
life-threatening consequences. In this case, the same 83% is a one in six chances of
a lethal outcome (Fig. 11.1b). The two different contexts lead to two distinct ways
to consider confidence in predictions.
When focusing on a specific compound, the question becomes “Can we trust this
specific individual prediction?” which combines elements of assessing the legitimacy
of the model, the reliability of the prediction for a specific compound and the level
of uncertainty that can be tolerated in making a decision. All these aspects make
perceiving the confidence in a prediction difficult to formalise. An attempt to formally
describe the legitimate scope of a model has been introduced as the AD. The role of
the AD is to define the boundaries within which a model can be used and provides
sufficiently accurate predictions. A well-defined AD has become a key feature for