108
T. Basu et al.
3.6 Conclusion
We have presented an overview over commonly used methods for uncertainty
quantification in the context of l 1 -penalized linear or logistic regression, comprising
refit, bootstrap, and Bayesian approaches.
We have illustrated these methods in the context of two datasets, both of which
have some relevance for aerospace engineering: one dataset relating to the current
Gaia space mission and another dataset involving the analysis of sonar signals.
For both modeling scenarios, we found good agreement of the parameter uncertainties obtained through the different methods. Standard errors of the bootstrap
and refit methods agreed particularly closely, noting however the limitation of the
latter to quantify uncertainty of inclusion as such. The Bayesian standard errors
were of the same magnitude as their frequentist counterparts; however they tended
to be larger and also did show some differences for specific parameters. For the
Sonar dataset, the refit indicated sparser models than Bayes or bootstrap, which
may appear unexpected at first glance but can be explained by the cut-off threshold
of 50% which happened to be just above the relative frequencies of occurrence for
many of the variables.
While the discussed uncertainty quantification methods are well-established and
investigated for the linear model, this is less the case for the logistic model. This is
not only reflected in the abundance of relevant literature, but also in the availability
of statistical software. Since we had not been able to locate an implementation of
the Bayesian logistic LASSO which could handle a model with 60 variables, we had
to reduce this dataset from the start to 48 variables. We did so for all methods, to
ensure comparability.
References
1. C.A.L. Bailer-Jones, The ILIUM forward modelling algorithm for multivariate parameter
estimation and its application to derive stellar parameters from Gaia spectrophotometry. Mon.
Not. R. Astron. Soc. 403(1), 96–116 (2010)
2. S. Boyd, L. Vandenberghe, Convex Optimization (Cambridge University Press, Cambridge,
2004)
3. L. Breiman, Better subset regression using the nonnegative garrote. Technometrics 37(4),
373–384 (1995)
4. K. Das, M. Sobel, Dirichlet Lasso: a Bayesian approach to variable selection. Stat. Modelling
15(3), 215–232 (2015)
5. N.R. Draper, H. Smith, Fitting a Straight Line by Least Squares: Applied Regression Analysis
(Wiley, New York, 1998), pp. 15–46
6. B. Efron, T. Hastie, I. Johnstone, R. Tibshirani, Least angle regression. Ann. Stat. 32(2),
407–499 (2004)
7. J. Einbeck, L. Evers, C. Bailer-Jones, Representing complex data using localized principal
components with application to astronomical data, in Principal Manifolds for Data Visualization and Dimension Reduction, ed. by A.N. Gorban, B. Kégl, D.C. Wunsch, A.Y. Zinovyev
(Springer, Berlin, 2008), pp. 178–201
T. Basu et al.
3.6 Conclusion
We have presented an overview over commonly used methods for uncertainty
quantification in the context of l 1 -penalized linear or logistic regression, comprising
refit, bootstrap, and Bayesian approaches.
We have illustrated these methods in the context of two datasets, both of which
have some relevance for aerospace engineering: one dataset relating to the current
Gaia space mission and another dataset involving the analysis of sonar signals.
For both modeling scenarios, we found good agreement of the parameter uncertainties obtained through the different methods. Standard errors of the bootstrap
and refit methods agreed particularly closely, noting however the limitation of the
latter to quantify uncertainty of inclusion as such. The Bayesian standard errors
were of the same magnitude as their frequentist counterparts; however they tended
to be larger and also did show some differences for specific parameters. For the
Sonar dataset, the refit indicated sparser models than Bayes or bootstrap, which
may appear unexpected at first glance but can be explained by the cut-off threshold
of 50% which happened to be just above the relative frequencies of occurrence for
many of the variables.
While the discussed uncertainty quantification methods are well-established and
investigated for the linear model, this is less the case for the logistic model. This is
not only reflected in the abundance of relevant literature, but also in the availability
of statistical software. Since we had not been able to locate an implementation of
the Bayesian logistic LASSO which could handle a model with 60 variables, we had
to reduce this dataset from the start to 48 variables. We did so for all methods, to
ensure comparability.
References
1. C.A.L. Bailer-Jones, The ILIUM forward modelling algorithm for multivariate parameter
estimation and its application to derive stellar parameters from Gaia spectrophotometry. Mon.
Not. R. Astron. Soc. 403(1), 96–116 (2010)
2. S. Boyd, L. Vandenberghe, Convex Optimization (Cambridge University Press, Cambridge,
2004)
3. L. Breiman, Better subset regression using the nonnegative garrote. Technometrics 37(4),
373–384 (1995)
4. K. Das, M. Sobel, Dirichlet Lasso: a Bayesian approach to variable selection. Stat. Modelling
15(3), 215–232 (2015)
5. N.R. Draper, H. Smith, Fitting a Straight Line by Least Squares: Applied Regression Analysis
(Wiley, New York, 1998), pp. 15–46
6. B. Efron, T. Hastie, I. Johnstone, R. Tibshirani, Least angle regression. Ann. Stat. 32(2),
407–499 (2004)
7. J. Einbeck, L. Evers, C. Bailer-Jones, Representing complex data using localized principal
components with application to astronomical data, in Principal Manifolds for Data Visualization and Dimension Reduction, ed. by A.N. Gorban, B. Kégl, D.C. Wunsch, A.Y. Zinovyev
(Springer, Berlin, 2008), pp. 178–201