400
J. Ren et al.
h t = o t tanh(c t ),
(4)
where f t , i t , o t are forgotten gate, input gate, output gate, respectively. They are
from 0 to 1. means the product of the value of the gate and each element
in the vector. ˆ
c t is the candidate states obtained by nonlinear functions. The
details are shown as follows:
ˆ
c t = tanh(W c · [x t , h t−1 ] + b c ),
(5)
i t = σ(W i · [x t , h t−1 ] + b i ),
(6)
f t = σ(W f · [x t , h t−1 ] + b f ),
(7)
o t = σ(W o · [x t , h t−1 ] + b o ),
(8)
where [x t , h t−1 ] means to concatenate x t and h t−1 , together. σ is sigmoid function:
σ(x) =
1
1 + e −x .
(9)
Because of these three gates and the internal state, LSTM can successfully
remember the past information and delete the useless information.
Although LSTM is the most popular network in dealing with time sequence
problems, it also has a small shortcoming. It just learn the forward information but does not learn the backward information, which has limitations in the
language model.
The Meaning of BiLSTM: As we just mentioned above, sometimes, we need
a network to study the forward information and backward information together,
especially in text problem such as text classification, text translation, and NER.
BiLSTM is simple to understand. It has two LSTMs. The first one’s input is
from x 1 to x T and its external state is from h
1
1 to h
1
T . The second one’s input is
from x T to x 1 and its external state is from h
2
1 to h
2
T . Later, the concatenation
of these two LSTMs’ external state is h t = [h
1
t , h
2
t ].
Our Changed BiLSTM: In this paper, in order to get the deeper information of sentences and the forward and backward text connection, we change the
BiLSTM structure.
We divided the model into two parts. The first part is a forward LSTM
connected to another forward LSTM. The first forward LSTM’s input is the
input data, and the second forward LSTM’s input is the returned sequence from
the first forward LSTM. The second part is a backward LSTM connected to
another backward LSTM. Also, the first backward LSTM’s input is the input
data, and the second backward LSTM’s input is the returned sequence from the
first backward LSTM.
By using this structure, we can firstly get deeper meanings of forward and
backward text sequences. Later, we combine the two states together and use the
combinative result as the CRF’s input.
J. Ren et al.
h t = o t tanh(c t ),
(4)
where f t , i t , o t are forgotten gate, input gate, output gate, respectively. They are
from 0 to 1. means the product of the value of the gate and each element
in the vector. ˆ
c t is the candidate states obtained by nonlinear functions. The
details are shown as follows:
ˆ
c t = tanh(W c · [x t , h t−1 ] + b c ),
(5)
i t = σ(W i · [x t , h t−1 ] + b i ),
(6)
f t = σ(W f · [x t , h t−1 ] + b f ),
(7)
o t = σ(W o · [x t , h t−1 ] + b o ),
(8)
where [x t , h t−1 ] means to concatenate x t and h t−1 , together. σ is sigmoid function:
σ(x) =
1
1 + e −x .
(9)
Because of these three gates and the internal state, LSTM can successfully
remember the past information and delete the useless information.
Although LSTM is the most popular network in dealing with time sequence
problems, it also has a small shortcoming. It just learn the forward information but does not learn the backward information, which has limitations in the
language model.
The Meaning of BiLSTM: As we just mentioned above, sometimes, we need
a network to study the forward information and backward information together,
especially in text problem such as text classification, text translation, and NER.
BiLSTM is simple to understand. It has two LSTMs. The first one’s input is
from x 1 to x T and its external state is from h
1
1 to h
1
T . The second one’s input is
from x T to x 1 and its external state is from h
2
1 to h
2
T . Later, the concatenation
of these two LSTMs’ external state is h t = [h
1
t , h
2
t ].
Our Changed BiLSTM: In this paper, in order to get the deeper information of sentences and the forward and backward text connection, we change the
BiLSTM structure.
We divided the model into two parts. The first part is a forward LSTM
connected to another forward LSTM. The first forward LSTM’s input is the
input data, and the second forward LSTM’s input is the returned sequence from
the first forward LSTM. The second part is a backward LSTM connected to
another backward LSTM. Also, the first backward LSTM’s input is the input
data, and the second backward LSTM’s input is the returned sequence from the
first backward LSTM.
By using this structure, we can firstly get deeper meanings of forward and
backward text sequences. Later, we combine the two states together and use the
combinative result as the CRF’s input.