322
D. Zhu et al.
First, input audio data is segmented to frames at intervals of 10 ms: x =
(x 1 , x 2 , . . . , x t );
Second, webrtcvad algorithm and activity results: v = (v 1 , v 2 , . . . , v t );
Third, smooth windows with the constant length (n = 30) and get average
results: v
∗
t =
t
i=t−n vi
n
;
Fourth, abstract speech segments based on v
∗ value and product sequences.
3.2 End-to-End Speech Recognition Model
There are two widely used speech recognition models: “acoustic model and language model” and end-to-end recognition model, and our paper use the DeepSpeech2 model.
DeepSpeech2 model building: DeepSpeech2 model is based on PyTorch
architecture, we segment the single-channel audio data at 16,000 Hz frequency
into frames with 10 ms interval and compute mel-frequency cepstral coefficient
(MFCC) features. Convolution kernel size at CNN layer 1 is 41 ∗ 11, at CNN
layer 2 is 21 ∗ 11, as for the bidirectional recurrent layers, we apply four layers
GRU with 512 width. Output layer is a fully connected layer.
As for model label, we design four plans to choose. They are:
1. Plan A: English alphabet + number + non-specific Chinese character, each
label represents an English alphabet or a number or a Chinese character,
the total labels number is 3583.
2. Plan B: English alphabet + number + specific Chinese character, each label
represents an English alphabet or a number or a specific Chinese character
included in the command set, the total labels number is 197.
3. Plan C: English alphabet + number + Chinese phonetic alphabet, each label
represents an English alphabet or a number or a Chinese phonetic alphabet,
the total labels number is 1449.
4. Plan D: English alphabet + number + Chinese syllable I/F (initials or finals),
each label represents an English alphabet or a number or an i/F, the total
labels number is 220.
Plan C and Plan D use Viterbi algorithm to map pinyin or syllable I/F to
character. In the paper, we apply five-order language model to evaluate state
transition probability.
We choose plan D and apply initial and tonal final combination two-variable
label method which improves its correct recognition rate from 91 to 97%. The
main phases are as follows:
A. Training, encode Chinese characters to initial and tonal final combination
labels, single final character is decoded to one final label. For example, “ ”
is encoded to “¯ a n” tonal final label, “ ” is encoded to both “y” initial label
and “ˇ u” tonal final label.
B. Interface, decode label sequences to Chinese phonetic alphabet sequences, use
trie tree segment Chinese phonetic alphabet sequences to pinyin sequences,
then use kenlm language model and sparse viterbi decoder [4] to map pinyin
sequences to Chinese characters sequences.
D. Zhu et al.
First, input audio data is segmented to frames at intervals of 10 ms: x =
(x 1 , x 2 , . . . , x t );
Second, webrtcvad algorithm and activity results: v = (v 1 , v 2 , . . . , v t );
Third, smooth windows with the constant length (n = 30) and get average
results: v
∗
t =
t
i=t−n vi
n
;
Fourth, abstract speech segments based on v
∗ value and product sequences.
3.2 End-to-End Speech Recognition Model
There are two widely used speech recognition models: “acoustic model and language model” and end-to-end recognition model, and our paper use the DeepSpeech2 model.
DeepSpeech2 model building: DeepSpeech2 model is based on PyTorch
architecture, we segment the single-channel audio data at 16,000 Hz frequency
into frames with 10 ms interval and compute mel-frequency cepstral coefficient
(MFCC) features. Convolution kernel size at CNN layer 1 is 41 ∗ 11, at CNN
layer 2 is 21 ∗ 11, as for the bidirectional recurrent layers, we apply four layers
GRU with 512 width. Output layer is a fully connected layer.
As for model label, we design four plans to choose. They are:
1. Plan A: English alphabet + number + non-specific Chinese character, each
label represents an English alphabet or a number or a Chinese character,
the total labels number is 3583.
2. Plan B: English alphabet + number + specific Chinese character, each label
represents an English alphabet or a number or a specific Chinese character
included in the command set, the total labels number is 197.
3. Plan C: English alphabet + number + Chinese phonetic alphabet, each label
represents an English alphabet or a number or a Chinese phonetic alphabet,
the total labels number is 1449.
4. Plan D: English alphabet + number + Chinese syllable I/F (initials or finals),
each label represents an English alphabet or a number or an i/F, the total
labels number is 220.
Plan C and Plan D use Viterbi algorithm to map pinyin or syllable I/F to
character. In the paper, we apply five-order language model to evaluate state
transition probability.
We choose plan D and apply initial and tonal final combination two-variable
label method which improves its correct recognition rate from 91 to 97%. The
main phases are as follows:
A. Training, encode Chinese characters to initial and tonal final combination
labels, single final character is decoded to one final label. For example, “ ”
is encoded to “¯ a n” tonal final label, “ ” is encoded to both “y” initial label
and “ˇ u” tonal final label.
B. Interface, decode label sequences to Chinese phonetic alphabet sequences, use
trie tree segment Chinese phonetic alphabet sequences to pinyin sequences,
then use kenlm language model and sparse viterbi decoder [4] to map pinyin
sequences to Chinese characters sequences.