7 A Review of Feature Reduction Methods …
127
Table 7.1
A summary of feature selection techniques
Methods
Description
Strengths
Weaknesses
Examples
Filter
• Rank features using a criterion
calculated based on the data
properties
• Fast, computationally
inexpensive, and as such, can
be applied to higher
dimensions of data
• Multivariate methods take the
relationship between features
into consideration
• Univariate methods ignore
feature dependencies
• Insensitive to the learner’s
heuristics
• Deciding on the best threshold
when selecting from ranked
features is not deterministic
• Information gain
• Chi-square test
• Fisher score
• Correlation coefficient
• Variance threshold
Wrapper
• Use search strategies to
generate feature subsets which
are then evaluated by a learner
• Dependencies between
features in a subset are
considered
• Interaction with the learner
results in better performance
than filter
• Features are learner specific
• Interaction with the learner
increases the likelihood of
overfitting
• Computationally expensive
• Sequential feature selection or
elimination (e.g. RFE)
• Genetic algorithm
• Simulated annealing
Embedded
• Are learning algorithms that
can weigh the contribution of
each feature to its performance
• Interacts with the learner but is
less prone to overfitting
• Computationally less
expensive than wrapper and
has better performance than
filter
• Dependencies between
features are inherently
considered
• Features selected are learning
algorithm specific
• LASSO
• Ridge Regression
• Elastic Net
• Decision Trees
Hybrid
• Combines other methods to
achieve the accuracy of
wrappers and the efficiency of
filters
• Better performance than filters
and less computationally
demanding than wrappers
• The setbacks of the filter and
wrapper methods are not
eliminated, they are reduced.
The features remain specific to
the learning algorithm
• Filter followed by embedded
methods
• Hybrid genetic algorithms
Ensemble
• Aggregates the output of
different feature selection
methods or subsets
• Ensures stable and robust
feature selection
• Depending on the constituent
methods, it could be
computationally expensive
and difficult to understand
• Could be made up of multiple
feature selection methods
127
Table 7.1
A summary of feature selection techniques
Methods
Description
Strengths
Weaknesses
Examples
Filter
• Rank features using a criterion
calculated based on the data
properties
• Fast, computationally
inexpensive, and as such, can
be applied to higher
dimensions of data
• Multivariate methods take the
relationship between features
into consideration
• Univariate methods ignore
feature dependencies
• Insensitive to the learner’s
heuristics
• Deciding on the best threshold
when selecting from ranked
features is not deterministic
• Information gain
• Chi-square test
• Fisher score
• Correlation coefficient
• Variance threshold
Wrapper
• Use search strategies to
generate feature subsets which
are then evaluated by a learner
• Dependencies between
features in a subset are
considered
• Interaction with the learner
results in better performance
than filter
• Features are learner specific
• Interaction with the learner
increases the likelihood of
overfitting
• Computationally expensive
• Sequential feature selection or
elimination (e.g. RFE)
• Genetic algorithm
• Simulated annealing
Embedded
• Are learning algorithms that
can weigh the contribution of
each feature to its performance
• Interacts with the learner but is
less prone to overfitting
• Computationally less
expensive than wrapper and
has better performance than
filter
• Dependencies between
features are inherently
considered
• Features selected are learning
algorithm specific
• LASSO
• Ridge Regression
• Elastic Net
• Decision Trees
Hybrid
• Combines other methods to
achieve the accuracy of
wrappers and the efficiency of
filters
• Better performance than filters
and less computationally
demanding than wrappers
• The setbacks of the filter and
wrapper methods are not
eliminated, they are reduced.
The features remain specific to
the learning algorithm
• Filter followed by embedded
methods
• Hybrid genetic algorithms
Ensemble
• Aggregates the output of
different feature selection
methods or subsets
• Ensures stable and robust
feature selection
• Depending on the constituent
methods, it could be
computationally expensive
and difficult to understand
• Could be made up of multiple
feature selection methods