10 Sequential Parameter Optimization for Mixed-Discrete Problems
335
10.3 Challenges in Real-World Optimization
10.3.1 Problem Features
Over the years, a large number of optimization methods have been proposed, and
new algorithms are developed every day to improve their general performance.
However, it has been stated by Wolpert and Macready [57] that any algorithm’s
improved performance over one class of problems is offset by a performance loss
over another class. Hence, the identification of problem features becomes a crucial
stage in the development and selection of optimization algorithms. Among all
potential problem features, the ones that mostly affect the performances in mixeddiscrete optimization can be listed as follows: high dimensionality, uncertainties,
computationally expensive evaluations, complex landscapes and black-box problems [55].
Here, “black-box” implies that no knowledge about the function is available and
any knowledge can only be derived by evaluating the function itself. It is often
impossible to predict the response of the function because the physical phenomena
are not fully understood or the modeling strategy leads to bias and undesired,
unknown sensitivities. The integration of optimization methodologies with computational analysis and simulations is of some importance in this context. This
lack of knowledge is especially problematic when function evaluations are expensive: Black-box optimization processes inherently require numerous evaluations of
objective functions [28]. Therefore, although numerous non-gradient optimization
methods are available for cheap black-box functions, more sophisticated methods
are necessary to deal with limited evaluation budgets.
Determining whether the best solution currently known is a local or a global
optimum is often difficult. This issue typically arises for multi-modal problems
(if the function has multiple maxima and minima). Moreover, difficult fitness
landscapes may exhibit deceptiveness [10]. Deceptive objective functions can trap
the optimizer by a large basin of attraction, which leads the search process away
from the global optimum in favour of a local one.
If an area with a better average fitness compared to other regions has been
found, the optimization algorithm will consider it as promising and will focus on
the exploration of this region. This assumes that such areas are likely to contain the
true optimum. Hence, developing an algorithm that is able to interpret the function
response correctly is a demanding task.
In many cases, this problem can be solved by choosing the correct optimization
strategy and performing a preliminary algorithm tuning. For example, a populationbased optimizer’s ability to distinguish the global optimum from a local optimum
often relies on the chosen population size. Moreover, maintaining diversity in the
population helps to avoid premature convergence [55]. It is also clear that the
optimality of the algorithms’ parameters changes during the optimization process
in case of multi-modal problems: In the beginning, algorithms should be more
explorative. This leads to the fastest identification of all the promising areas and
335
10.3 Challenges in Real-World Optimization
10.3.1 Problem Features
Over the years, a large number of optimization methods have been proposed, and
new algorithms are developed every day to improve their general performance.
However, it has been stated by Wolpert and Macready [57] that any algorithm’s
improved performance over one class of problems is offset by a performance loss
over another class. Hence, the identification of problem features becomes a crucial
stage in the development and selection of optimization algorithms. Among all
potential problem features, the ones that mostly affect the performances in mixeddiscrete optimization can be listed as follows: high dimensionality, uncertainties,
computationally expensive evaluations, complex landscapes and black-box problems [55].
Here, “black-box” implies that no knowledge about the function is available and
any knowledge can only be derived by evaluating the function itself. It is often
impossible to predict the response of the function because the physical phenomena
are not fully understood or the modeling strategy leads to bias and undesired,
unknown sensitivities. The integration of optimization methodologies with computational analysis and simulations is of some importance in this context. This
lack of knowledge is especially problematic when function evaluations are expensive: Black-box optimization processes inherently require numerous evaluations of
objective functions [28]. Therefore, although numerous non-gradient optimization
methods are available for cheap black-box functions, more sophisticated methods
are necessary to deal with limited evaluation budgets.
Determining whether the best solution currently known is a local or a global
optimum is often difficult. This issue typically arises for multi-modal problems
(if the function has multiple maxima and minima). Moreover, difficult fitness
landscapes may exhibit deceptiveness [10]. Deceptive objective functions can trap
the optimizer by a large basin of attraction, which leads the search process away
from the global optimum in favour of a local one.
If an area with a better average fitness compared to other regions has been
found, the optimization algorithm will consider it as promising and will focus on
the exploration of this region. This assumes that such areas are likely to contain the
true optimum. Hence, developing an algorithm that is able to interpret the function
response correctly is a demanding task.
In many cases, this problem can be solved by choosing the correct optimization
strategy and performing a preliminary algorithm tuning. For example, a populationbased optimizer’s ability to distinguish the global optimum from a local optimum
often relies on the chosen population size. Moreover, maintaining diversity in the
population helps to avoid premature convergence [55]. It is also clear that the
optimality of the algorithms’ parameters changes during the optimization process
in case of multi-modal problems: In the beginning, algorithms should be more
explorative. This leads to the fastest identification of all the promising areas and