314
the registered historical data of the problem, then our goal is to find an approximation ( )
ˆ
F x
(
to the function Y = F(X) that minimizes a measured value of some loss function
1 denoted as
L y F
,
F ( )
x x
(
) so that
2
, as shown in Eq. 3:
( )
( )
i
ˆ
D F
(
F x
( )
i
arg m n
i ⎡
⎤
( )
(
)
( F (
,
(
⎣
⎦
( )
(
)
y F x
(
( ,
(
⎤ ⎤
))
F x
(
,
(
(3)
In gradient boosting methods, it is assumed that a real value “y” exists, for which we look
for an approximation ( )
ˆ
F x
( in the form of a weighted sum of functions h i (x) that belongs to
some class H, called weak classifiers (or learners), as shown in Eq. 4:
( )
( )
1
consta
ˆ
nt
M
i i
i
h x
(
i i
F x
( )
λ i
λ i
=
+
)
h x
(
i i
i
i
∑ ∑
(4)
where h i (x) is a randomized decision tree, for which it’s mth iteration fits the residual values
(differences) between the estimator function
( )
ˆ
,
m
F x
(
m
and Y. Let J m be the number of leaves
in the tree; this separates the defined input space by the training set in J m disjoint regions
R
R
m
J
R m
m
J
1
R R
, and predicts a constant value in each one of them. Using the indicator notation
3 , the output of h m (x) for a given input x can be written as the sum shown in Eq. 5:
h
b
m
h h
j
J
jm
b R
m
J J
jm
R
( )
x =
( )
x
=
∑
1
(5)
1. A loss function or cost function is a function that maps an event or values of one or more variables
onto a real number intuitively representing some “cost” associated with the event. An optimization
problem seeks to minimize a loss function. In machine learning, typically a loss function is used for
parameter estimation, and the event in question is some function of the difference between estimated
and true values for an instance of data.
2. In mathematics, the arguments of the minimum (abbreviated as arg min) are the points of the domain
of some function at which the function values are minimized. This way, given an arbitrary set X, a
totally ordered set Y, and a function f:X→Y, its arguments of the minimum over some subset S of X
are defined by
arg min
x S X
∈ ⊆
S
[
]
f ( )
x
{
}
x
y S f
f
( )
y
( )
x
= { x
f ( y
x S
y S
x
S
3. In mathematics, an indicator function or a characteristic function is a function defined on a set X that
indicates membership of an element in a subset A of X, having the value 1 for all elements of A and the
value 0 for all elements of X not in A. It is usually denoted by the symbol 1 A , being defined by
1 A
if x A
if x A
( )
x
⎧
⎨
⎧ ⎧
⎩
⎨ ⎨
=
∉
1
0
;
;
Figure 4. A simple scheme to explain the gradient boosting technique.
~
ERRORS I
e e e e e TRAIN
e ~'r, e e TRAIN
e ie:._e e TRAIN
e •• e e MODEL ,..,.,,
e r .. :.· MODEL • "'\, TEST • • • I e MODEL _ ... __
. :. : ·. ~ . ...
(_ ' .. ___ ~ -- ~_·_:_._ ~ _- - : : : : : : : ! :
1" : :
' .. :- ~ --~ ·_·_._·_·_·_·_._ ~ __ -_1 ... ·_ "' .. _
§
PREDICTION
the registered historical data of the problem, then our goal is to find an approximation ( )
ˆ
F x
(
to the function Y = F(X) that minimizes a measured value of some loss function
1 denoted as
L y F
,
F ( )
x x
(
) so that
2
, as shown in Eq. 3:
( )
( )
i
ˆ
D F
(
F x
( )
i
arg m n
i ⎡
⎤
( )
(
)
( F (
,
(
⎣
⎦
( )
(
)
y F x
(
( ,
(
⎤ ⎤
))
F x
(
,
(
(3)
In gradient boosting methods, it is assumed that a real value “y” exists, for which we look
for an approximation ( )
ˆ
F x
( in the form of a weighted sum of functions h i (x) that belongs to
some class H, called weak classifiers (or learners), as shown in Eq. 4:
( )
( )
1
consta
ˆ
nt
M
i i
i
h x
(
i i
F x
( )
λ i
λ i
=
+
)
h x
(
i i
i
i
∑ ∑
(4)
where h i (x) is a randomized decision tree, for which it’s mth iteration fits the residual values
(differences) between the estimator function
( )
ˆ
,
m
F x
(
m
and Y. Let J m be the number of leaves
in the tree; this separates the defined input space by the training set in J m disjoint regions
R
R
m
J
R m
m
J
1
R R
, and predicts a constant value in each one of them. Using the indicator notation
3 , the output of h m (x) for a given input x can be written as the sum shown in Eq. 5:
h
b
m
h h
j
J
jm
b R
m
J J
jm
R
( )
x =
( )
x
=
∑
1
(5)
1. A loss function or cost function is a function that maps an event or values of one or more variables
onto a real number intuitively representing some “cost” associated with the event. An optimization
problem seeks to minimize a loss function. In machine learning, typically a loss function is used for
parameter estimation, and the event in question is some function of the difference between estimated
and true values for an instance of data.
2. In mathematics, the arguments of the minimum (abbreviated as arg min) are the points of the domain
of some function at which the function values are minimized. This way, given an arbitrary set X, a
totally ordered set Y, and a function f:X→Y, its arguments of the minimum over some subset S of X
are defined by
arg min
x S X
∈ ⊆
S
[
]
f ( )
x
{
}
x
y S f
f
( )
y
( )
x
= { x
f ( y
x S
y S
x
S
3. In mathematics, an indicator function or a characteristic function is a function defined on a set X that
indicates membership of an element in a subset A of X, having the value 1 for all elements of A and the
value 0 for all elements of X not in A. It is usually denoted by the symbol 1 A , being defined by
1 A
if x A
if x A
( )
x
⎧
⎨
⎧ ⎧
⎩
⎨ ⎨
=
∉
1
0
;
;
Figure 4. A simple scheme to explain the gradient boosting technique.
~
ERRORS I
e e e e e TRAIN
e ~'r, e e TRAIN
e ie:._e e TRAIN
e •• e e MODEL ,..,.,,
e r .. :.· MODEL • "'\, TEST • • • I e MODEL _ ... __
. :. : ·. ~ . ...
(_ ' .. ___ ~ -- ~_·_:_._ ~ _- - : : : : : : : ! :
1" : :
' .. :- ~ --~ ·_·_._·_·_·_·_._ ~ __ -_1 ... ·_ "' .. _
§
PREDICTION