402
J. Ren et al.
where F i (y i−1 , y i , x) = (f 1 (y i−1 , y i , x, i), f 2 (y i−1 , y i , x, i), ..., f K (y i−1 , y i , x, i))
T is
local characteristic vector.
Later, use viterbi algorithm. First of all, the denormalized probability of each
mark j = 1, 2..., m to position 1 is:
δ 1 (j) = w · F 1 (y 0 = start, y 1 = j, x), j = 1, 2, ..., m.
(18)
In general, the recursive formula is used to find the maximum value of the
denormalization probability of each mark l = 1, 2, ..., m to position i:
δ i (j) = max
i≤j≤m
δ i−1 (j) + w · F i (y i−1 = j, y i = l, x), l = 1, 2, ..., m,
(19)
Φ i (l) = arg max
i≤j≤m
δ i−1 (j) + w · F i (y i−1 = j, y i = l, x), l = 1, 2, ..., m. (20)
When l = n, the maximum value of the denormalized probability is:
max
y
(w · F (y, x)) = max
1≤j≤m
δ n (j),
(21)
and the most optimal terminal:
y
∗
n = arg max
1≤j≤m
δ n (j)
(22)
Return from the end of this optimal path:
y
∗
i = Φ i+1 (y
∗
i+1 ), i = n − 1, n − 2, ..., 1,
(23)
Finally, the most optimal path is y
∗ = (y
∗
1 , y
∗
2 , ..., y
∗
n )
T .
2.3 Our Model Structure
This part totally explains what we do and how to combine improved BiLSTM
and CRF together.
Figure 1 is the whole structure of our network.
Firstly, because all the words in corpus are in the form of one-hot encoder and
the dimension is very high, we use an embedding layer to find the words’ lowdimension representation to reduce the learning time. Also, the low-dimension
dense representation has better representation ability than the high-dimension
sparse representation.
Secondly, we put the embedded words sequence into our improved BiLSTM
to learn the forward and backward deep semantic meaning. Then, use a merge
layer to concatenate the external state of the two-layer-forward LSTM and the
external state of the two-layer-backward LSTM.
Thirdly, the combined external state is regarded as the input of CRF layer.
CRF layer learns its parameters.
Finally, the output sequence of CRF layer is the name entities.
J. Ren et al.
where F i (y i−1 , y i , x) = (f 1 (y i−1 , y i , x, i), f 2 (y i−1 , y i , x, i), ..., f K (y i−1 , y i , x, i))
T is
local characteristic vector.
Later, use viterbi algorithm. First of all, the denormalized probability of each
mark j = 1, 2..., m to position 1 is:
δ 1 (j) = w · F 1 (y 0 = start, y 1 = j, x), j = 1, 2, ..., m.
(18)
In general, the recursive formula is used to find the maximum value of the
denormalization probability of each mark l = 1, 2, ..., m to position i:
δ i (j) = max
i≤j≤m
δ i−1 (j) + w · F i (y i−1 = j, y i = l, x), l = 1, 2, ..., m,
(19)
Φ i (l) = arg max
i≤j≤m
δ i−1 (j) + w · F i (y i−1 = j, y i = l, x), l = 1, 2, ..., m. (20)
When l = n, the maximum value of the denormalized probability is:
max
y
(w · F (y, x)) = max
1≤j≤m
δ n (j),
(21)
and the most optimal terminal:
y
∗
n = arg max
1≤j≤m
δ n (j)
(22)
Return from the end of this optimal path:
y
∗
i = Φ i+1 (y
∗
i+1 ), i = n − 1, n − 2, ..., 1,
(23)
Finally, the most optimal path is y
∗ = (y
∗
1 , y
∗
2 , ..., y
∗
n )
T .
2.3 Our Model Structure
This part totally explains what we do and how to combine improved BiLSTM
and CRF together.
Figure 1 is the whole structure of our network.
Firstly, because all the words in corpus are in the form of one-hot encoder and
the dimension is very high, we use an embedding layer to find the words’ lowdimension representation to reduce the learning time. Also, the low-dimension
dense representation has better representation ability than the high-dimension
sparse representation.
Secondly, we put the embedded words sequence into our improved BiLSTM
to learn the forward and backward deep semantic meaning. Then, use a merge
layer to concatenate the external state of the two-layer-forward LSTM and the
external state of the two-layer-backward LSTM.
Thirdly, the combined external state is regarded as the input of CRF layer.
CRF layer learns its parameters.
Finally, the output sequence of CRF layer is the name entities.