294
F. Firouzi et al.
Fig. 5.48 Schematic presentation of the sequential steps in the boosting algorithm
to grow the tree, first we select a random subset of features. And then, we select
the best attribute for that selected subset. The randomness of selecting a subset
of features produces a wider diversity that contributes to better results. Full-grown
decision trees are also at risk of overfitting. The randomness in selecting a subset of
features in a random forest method prevents overfitting of the model in most of the
cases.
There are two important hyperparameters in the random forest method: number
of estimators and maximum number of features. The former corresponds to the
number of trees the algorithm builds, and the latter is the maximum number
of features the algorithm is allowed to evaluate in an individual tree. The main
bottleneck of random forest method is its performance in real-time applications.
A large number of trees make the algorithm inefficient in this scenario. As a general
rule, RF is fast in training but makes predictions quite slow.
5.4.7.3 Boosting
Boosting is another method for producing ensembles, in which the ensembles are
created sequentially. Recall that in bagging methods are parallel because we create
and learn n base models in parallel. In contrast, the idea of boosting method is
to use feedback from one base model to produce the next base model. In other
words, we consider the training instances of the previously generated base models
which are misclassified (see Fig. 5.48). In comparison to bagging, boosting shows
a better performance for specific applications, but it increases the risk of overfitting
the model.
Adaptive Boosting (AdaBoost)
AdaBoost is one of the well-known implementations of boosting. The core idea of
AdaBoost is to sequentially modify a weak model to make it a better classifier.
In fact, AdaBoost creates a better model by combining several weak classifiers
linearly, and the final classification is performed using this combination, which can
be expressed as
F. Firouzi et al.
Fig. 5.48 Schematic presentation of the sequential steps in the boosting algorithm
to grow the tree, first we select a random subset of features. And then, we select
the best attribute for that selected subset. The randomness of selecting a subset
of features produces a wider diversity that contributes to better results. Full-grown
decision trees are also at risk of overfitting. The randomness in selecting a subset of
features in a random forest method prevents overfitting of the model in most of the
cases.
There are two important hyperparameters in the random forest method: number
of estimators and maximum number of features. The former corresponds to the
number of trees the algorithm builds, and the latter is the maximum number
of features the algorithm is allowed to evaluate in an individual tree. The main
bottleneck of random forest method is its performance in real-time applications.
A large number of trees make the algorithm inefficient in this scenario. As a general
rule, RF is fast in training but makes predictions quite slow.
5.4.7.3 Boosting
Boosting is another method for producing ensembles, in which the ensembles are
created sequentially. Recall that in bagging methods are parallel because we create
and learn n base models in parallel. In contrast, the idea of boosting method is
to use feedback from one base model to produce the next base model. In other
words, we consider the training instances of the previously generated base models
which are misclassified (see Fig. 5.48). In comparison to bagging, boosting shows
a better performance for specific applications, but it increases the risk of overfitting
the model.
Adaptive Boosting (AdaBoost)
AdaBoost is one of the well-known implementations of boosting. The core idea of
AdaBoost is to sequentially modify a weak model to make it a better classifier.
In fact, AdaBoost creates a better model by combining several weak classifiers
linearly, and the final classification is performed using this combination, which can
be expressed as