Statistical Language Models based on Neural Networks - Faculty of ...

More documents

Recommendations

Info

correlated) to the ability to predict words in a given context, then we can formally measure quality of our artificial models of natural languages. This AI test has been proposed for example in [44] and more discussion is given in [42]. While it is likely that attempts to build artificial language models that can understand text in the same way as humans do just by reading huge quantities of text data is unreal- istically hard (as humans would probably fail in such task themselves), language models estimated from huge amounts of data are very interesting due to their practical usage in wide variety of commercially successful applications. Among the most widely known ones are the statistical machine translation (for example popular Google Translate) and the automatic speech recognition. The goal of this thesis is to describe new techniques that have been developed to overcome the simple n-gram models that still remain basically state-of-the-art today. To prove usefulness of the new approaches, empirical results on several standard data sets will be extensively described. Finally, approaches and techniques that can possibly lead to automatic language learning by computers will be discussed, together with a simple plan how this could be achieved. 1.2 Structure of the Thesis Chapter 2 introduces the statistical language modeling and mathematically defines the problem. Simple and advanced language modeling techniques are discussed. Also, the most important data sets that are further used in the thesis are introduced. Chapter 3 introduces neural network language models and the recurrent architecture, as well as the extensions of the basic model. The training algorithm is described in detail. Chapter 4 provides extensive empirical comparison of results obtained with various advanced language modeling techniques on the Penn Treebank setup, and results after combination of these techniques. The Chapter 5 focuses on the results after application of the RNN language model to standard speech recognition setup, the Wall Street Journal task. Results and comparison are provided on two different setups; one is from the Johns Hopkins University and allows comparison with competitive techniques such as discriminatively trained LMs and structured LMs, and the other setup was obtained with an open-source ASR toolkit, Kaldi. 6
Chapter 6 presents further extensions of the basic recurrent neural network language model that allow efficient training on large data sets. Experiments are performed on data sets with up to 400 million training tokens with very large neural networks. Results are reported on state of the art setup for Broadcast News speech recognition (the NIST RT04 task) with a recognizer and baseline models provided by IBM. Chapter 7 presents further empirical results on various other tasks, such as machine translation, data compression and others. The purpose of this chapter is to prove that the developed techniques are very general and easily applicable to other domains where n-gram models are currently used. Chapter 8 discusses computational limitations of models that are commonly used for the statistical language modeling, and provides some insight into how further progress can be achieved. Finally, Chapter 9 summarizes the achieved results and concludes the work. 1.3 Claims of the Thesis The most important original contributions of this thesis are: • Development of statistical language model <strong>based</strong> on simple recurrent neural network • Extensions of the basic recurrent neural network language model: – Simple classes <strong>based</strong> on unigram frequency of words – Joint training of neural network and maximum entropy model – Adaptation of neural net language models by sorting the training data – Adaptation of neural net language models by training the model during pro- cessing of the test data • Freely available open source toolkit for training RNN-<strong>based</strong> language models that can be used to reproduce the described experiments • Empirical comparison with other advanced language modeling techniques, with new state of the art results achieved with RNN <strong>based</strong> LMs on the following tasks: – <strong>Language</strong> modeling of Penn Treebank Corpus – Wall Street Journal speech recognition 7
Page 1 and 2: VYSOKÉ UČENÍ TECHNICKÉ V BRNĚ
Page 3 and 4: Abstrakt Statistické jazykové mod
Page 5 and 6: Contents 1 Introduction 4 1.1 Motiv
Page 7 and 8: 6.2.3 Reduction of Vocabulary Size
Page 9: Maybe the most popular vision of fu
Page 13 and 14: Chapter 2 Overview of Stati
Page 15 and 16: 2.1 Evaluation 2.1.1 Perplexity Eva
Page 17 and 18: ALP can be used to obtain prior pro
Page 19 and 20: • Good theoretical motivation •
Page 21 and 22: abilities of n-grams are stored in
Page 23 and 24: main (static) n-gram model. As the
Page 25 and 26: There are many popular examples sho
Page 27 and 28: y Chen et al., who proposed a so-ca
Page 29 and 30: confusion among researchers, and ma
Page 31 and 32: language model took almost a week u
Page 33 and 34: w(t) s(t-1) s(t) U V W y(t) Figure
Page 35 and 36: ate is halved at start of every new
Page 37 and 38: or using matrix-vector notation as
Page 39 and 40: information for more than 5 time st
Page 41 and 42: A simple solution to the exploding
Page 43 and 44: output layer changes to computation
Page 45 and 46: While RNN models can overcome this
Page 47 and 48: complex or random architectures (su
Page 49 and 50: While for any of the previous point
Page 51 and 52: where λ is the interpolation weigh
Page 53 and 54: model with default SRILM cutoffs pr
Page 55 and 56: experiments, we have used the one i
Page 57 and 58: Perplexity (Penn corpus) 145 140 13
Page 59 and 60: with syntactical NNLMs would be pre
Page 61 and 62:
Table 4.3: Combination of individua
Page 63 and 64:
Table 4.6: Results on Penn Treebank
Page 65 and 66:
4.6 Conclusion of the Model Combina
Page 67 and 68:
were: 400 classes, hidden layer siz
Page 69 and 70:
Entropy per word on the WSJ test da
Page 71 and 72:
Table 5.3: Results on the WSJ setup
Page 73 and 74:
Table 5.5: Results for models <stro
Page 75 and 76:
trained together with a maximum ent
Page 77 and 78:
wt-3 wt-2 wt-1 D D D P(wt|context)
Page 79 and 80:
n-gram probabilities. However, it w
Page 81 and 82:
Table 6.1: Training corpora for NIS
Page 83 and 84:
Perplexity 360 340 320 300 280 260
Page 85 and 86:
Entropy per word 9 8.5 8 7.5 7 6.5
Page 87 and 88:
1 a a a 1 2 3 P(w(t)|*) ONE TWO THR
Page 89 and 90:
Table 6.4: Perplexity on the evalua
Page 91 and 92:
Entropy reduction per word over KN4
Page 93 and 94:
Table 6.6: Perplexity with the new
Page 95 and 96:
Entropy reduction over KN5 -0.04 -0
Page 97 and 98:
as a baseline, and 12.3% after resc
Page 99 and 100:
Table 7.1: BLEU on IWSLT 2005 Machi
Page 101 and 102:
Table 7.3: Size of compressed text
Page 103 and 104:
Table 7.4: Accuracy of different la
Page 105 and 106:
Table 7.6: Entropy on PTB with n-gr
Page 107 and 108:
8.1 Machine Learning One possible d
Page 109 and 110:
that almost every non-trivial compu
Page 111 and 112:
supervision such as one digit at a
Page 113 and 114:
Chapter 9 Conclusion and Future Wor
Page 115 and 116:
from the expensive part of the mode
Page 117 and 118:
Bibliography [1] A. Alexandrescu, K
Page 119 and 120:
[23] D. Filimonov, M. Harper. A joi
Page 121 and 122:
[50] T. Mikolov, S. Kombrink, L. Bu
Page 123 and 124:
[77] W. Wang, M. Harper. The SuperA
Page 125 and 126:
Test Phase After the model is train
Page 127 and 128:
• compute sentence-level scores g
Page 129 and 130:
Appendix B: Data generated from mod
Page 131 and 132:
Appendix C: Example of decoded utte
Page 133:
AND SAYS THIS SWING IS WITHIN A REA
show all

Statistical Language Models based on Neural Networks - Faculty of ...

Create successful ePaper yourself

Delete template?

Save as template?