RNN Language Model with Word Clustering and Class-based Output Layer

Authors

Yongzhe Shi, Tsinghua University
Wei-Qiang Zhang, Tsinghua UniversityFollow
Jia Liu, Tsinghua UniversityFollow
Michael T. Johnson, Marquette UniversityFollow

Document Type

Article

Language

eng

Publication Date

7-22-2013

Publisher

Springer

Source Publication

EURASIP Journal on Audio, Speech, and Music Processing

Source ISSN

1687-4722

Original Item ID

doi: 10.1186/1687-4722-2013-22

Abstract

The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.

Comments

Published version. EURASIP Journal on Audio, Speech, and Music Processing, Vol. 2013, No. 22 (July 22, 2013). DOI. Published under Creative Commons Attribution License 2.0.

Recommended Citation

Shi, Yongzhe; Zhang, Wei-Qiang; Liu, Jia; and Johnson, Michael T., "RNN Language Model with Word Clustering and Class-based Output Layer" (2013). Electrical and Computer Engineering Faculty Research and Publications. 44.
https://epublications.marquette.edu/electric_fac/44