Statistical Language Models based on Neural Networks - Faculty of ...

More documents

Recommendations

Info

3.4.1 Vocabulary Truncation . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.4.2 Factorization of the Output Layer . . . . . . . . . . . . . . . . . . . 37 3.4.3 Approximation of Complex <strong>Language</strong> Model by Backoff N-gram model 40 3.4.4 Dynamic Evaluation of the Model . . . . . . . . . . . . . . . . . . . 40 3.4.5 Combination of Neural Network <strong>Models</strong> . . . . . . . . . . . . . . . . 42 4 Evaluation and Combination of <strong>Language</strong> Modeling Techniques 44 4.1 Comparison of Different Types of <strong>Language</strong> <strong>Models</strong> . . . . . . . . . . . . . . 45 4.2 Penn Treebank Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.3 Performance of Individual <strong>Models</strong> . . . . . . . . . . . . . . . . . . . . . . . . 47 4.3.1 Backoff N-gram <strong>Models</strong> and Cache <strong>Models</strong> . . . . . . . . . . . . . . 48 4.3.2 General Purpose Compression Program . . . . . . . . . . . . . . . . 49 4.3.3 Advanced <strong>Language</strong> Modeling Techniques . . . . . . . . . . . . . . . 50 4.3.4 Neural network <strong>based</strong> models . . . . . . . . . . . . . . . . . . . . . . 51 4.3.5 Combinations of NNLMs . . . . . . . . . . . . . . . . . . . . . . . . 53 4.4 Comparison of Different Neural Network Architectures . . . . . . . . . . . . 54 4.5 Combination of all models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.5.1 Adaptive Linear Combination . . . . . . . . . . . . . . . . . . . . . . 60 4.6 Conclusion of the Model Combination Experiments . . . . . . . . . . . . . . 61 5 Wall Street Journal Experiments 62 5.1 WSJ-JHU Setup Description . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.1.1 Results on the JHU Setup . . . . . . . . . . . . . . . . . . . . . . . . 63 5.1.2 Performance with Increasing Size of the Training Data . . . . . . . . 63 5.1.3 Conclusion of WSJ Experiments (JHU setup) . . . . . . . . . . . . . 65 5.2 Kaldi WSJ Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.2.1 Approximation of RNNME using n-gram models . . . . . . . . . . . 68 6 Strategies for Training Large Scale Neural Network <strong>Language</strong> <strong>Models</strong> 70 6.1 Model Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 6.2 Computational Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.2.1 Reduction of Training Epochs . . . . . . . . . . . . . . . . . . . . . . 74 6.2.2 Reduction of Number of Training Tokens . . . . . . . . . . . . . . . 74 2
6.2.3 Reduction of Vocabulary Size . . . . . . . . . . . . . . . . . . . . . . 74 6.2.4 Reduction of Size of the Hidden Layer . . . . . . . . . . . . . . . . . 75 6.2.5 Parallelization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6.3 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 6.4 Automatic Data Selection and Sorting . . . . . . . . . . . . . . . . . . . . . 76 6.5 Experiments with large RNN models . . . . . . . . . . . . . . . . . . . . . . 78 6.6 Hash-<strong>based</strong> Implementation of Class-<strong>based</strong> Maximum Entropy Model . . . 81 6.6.1 Training of Hash-Based Maximum Entropy Model . . . . . . . . . . 82 6.6.2 Results with Early Implementation of RNNME . . . . . . . . . . . . 85 6.6.3 Further Results with RNNME . . . . . . . . . . . . . . . . . . . . . 86 6.6.4 <strong>Language</strong> Learning by RNN . . . . . . . . . . . . . . . . . . . . . . . 90 6.7 Conclusion of the NIST RT04 Experiments . . . . . . . . . . . . . . . . . . 92 7 Additional Experiments 94 7.1 Machine Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 7.2 Data Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 7.3 Microsoft Sentence Completion Challenge . . . . . . . . . . . . . . . . . . . 98 7.4 Speech Recognition of Morphologically Rich <strong>Language</strong>s . . . . . . . . . . . 100 8 Towards Intelligent <strong>Models</strong> of Natural <strong>Language</strong>s 102 8.1 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 8.2 Genetic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 8.3 Incremental Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 8.4 Proposal for Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . 107 9 Conclusion and Future Work 109 9.1 Future of <strong>Language</strong> Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 111 3
Page 1 and 2: VYSOKÉ UČENÍ TECHNICKÉ V BRNĚ
Page 3 and 4: Abstrakt Statistické jazykové mod
Page 5: Contents 1 Introduction 4 1.1 Motiv
Page 9 and 10: Maybe the most popular vision of fu
Page 11 and 12: Chapter 6 presents further extensio
Page 13 and 14: Chapter 2 Overview of Stati
Page 15 and 16: 2.1 Evaluation 2.1.1 Perplexity Eva
Page 17 and 18: ALP can be used to obtain prior pro
Page 19 and 20: • Good theoretical motivation •
Page 21 and 22: abilities of n-grams are stored in
Page 23 and 24: main (static) n-gram model. As the
Page 25 and 26: There are many popular examples sho
Page 27 and 28: y Chen et al., who proposed a so-ca
Page 29 and 30: confusion among researchers, and ma
Page 31 and 32: language model took almost a week u
Page 33 and 34: w(t) s(t-1) s(t) U V W y(t) Figure
Page 35 and 36: ate is halved at start of every new
Page 37 and 38: or using matrix-vector notation as
Page 39 and 40: information for more than 5 time st
Page 41 and 42: A simple solution to the exploding
Page 43 and 44: output layer changes to computation
Page 45 and 46: While RNN models can overcome this
Page 47 and 48: complex or random architectures (su
Page 49 and 50: While for any of the previous point
Page 51 and 52: where λ is the interpolation weigh
Page 53 and 54: model with default SRILM cutoffs pr
Page 55 and 56: experiments, we have used the one i
Page 57 and 58:
Perplexity (Penn corpus) 145 140 13
Page 59 and 60:
with syntactical NNLMs would be pre
Page 61 and 62:
Table 4.3: Combination of individua
Page 63 and 64:
Table 4.6: Results on Penn Treebank
Page 65 and 66:
4.6 Conclusion of the Model Combina
Page 67 and 68:
were: 400 classes, hidden layer siz
Page 69 and 70:
Entropy per word on the WSJ test da
Page 71 and 72:
Table 5.3: Results on the WSJ setup
Page 73 and 74:
Table 5.5: Results for models <stro
Page 75 and 76:
trained together with a maximum ent
Page 77 and 78:
wt-3 wt-2 wt-1 D D D P(wt|context)
Page 79 and 80:
n-gram probabilities. However, it w
Page 81 and 82:
Table 6.1: Training corpora for NIS
Page 83 and 84:
Perplexity 360 340 320 300 280 260
Page 85 and 86:
Entropy per word 9 8.5 8 7.5 7 6.5
Page 87 and 88:
1 a a a 1 2 3 P(w(t)|*) ONE TWO THR
Page 89 and 90:
Table 6.4: Perplexity on the evalua
Page 91 and 92:
Entropy reduction per word over KN4
Page 93 and 94:
Table 6.6: Perplexity with the new
Page 95 and 96:
Entropy reduction over KN5 -0.04 -0
Page 97 and 98:
as a baseline, and 12.3% after resc
Page 99 and 100:
Table 7.1: BLEU on IWSLT 2005 Machi
Page 101 and 102:
Table 7.3: Size of compressed text
Page 103 and 104:
Table 7.4: Accuracy of different la
Page 105 and 106:
Table 7.6: Entropy on PTB with n-gr
Page 107 and 108:
8.1 Machine Learning One possible d
Page 109 and 110:
that almost every non-trivial compu
Page 111 and 112:
supervision such as one digit at a
Page 113 and 114:
Chapter 9 Conclusion and Future Wor
Page 115 and 116:
from the expensive part of the mode
Page 117 and 118:
Bibliography [1] A. Alexandrescu, K
Page 119 and 120:
[23] D. Filimonov, M. Harper. A joi
Page 121 and 122:
[50] T. Mikolov, S. Kombrink, L. Bu
Page 123 and 124:
[77] W. Wang, M. Harper. The SuperA
Page 125 and 126:
Test Phase After the model is train
Page 127 and 128:
• compute sentence-level scores g
Page 129 and 130:
Appendix B: Data generated from mod
Page 131 and 132:
Appendix C: Example of decoded utte
Page 133:
AND SAYS THIS SWING IS WITHIN A REA
show all

Statistical Language Models based on Neural Networks - Faculty of ...

Create successful ePaper yourself

Delete template?

Save as template?