Statistical Language Models based on Neural Networks - Faculty of ...

More documents

Recommendations

Info

[36] R. Kneser, H. Ney. Improved backing-off for m-gram language modeling. In Proceed- ings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 1995. [37] S. Kombrink, M. Hannemann, L. Burget. Out-of-vocabulary word detection and be- yond, In: ECML PKDD Proceedings and Journal Content, 2010. [38] S. Kombrink, T. Mikolov, M. Karafiát, L. Burget. Recurrent Neural Network <strong>based</strong> <strong>Language</strong> Modeling in Meeting Recognition, In: Proceedings of Interspeech, 2011. [39] R. Lau, R. Rosenfeld, S. Roukos. Trigger-<strong>based</strong> language models: A maximum entropy approach. In Proceedings of ICASSP, 1993. [40] H.-S. Le, I. Oparin, A. Allauzen, J.-L. Gauvain, F. Yvon. Structured Output Layer Neural Network <strong>Language</strong> Model. In Proc. of ICASSP, 2011. [41] S. Hochreiter, J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735-1780, 1997. [42] S. Legg. Machine Super Intelligence. PhD thesis, University of Lugano, 2008. [43] M. Looks and B. Goertzel. Program representation for general intelligence. In Proc. of AGI 2009. [44] M. Mahoney. Text Compression as a Test for Artificial Intelligence. In AAAI/IAAI, 486-502, 1999. [45] M. Mahoney. Fast Text Compression with Neural Networks. In Proc. FLAIRS, 2000. [46] M. Mahoney et al. PAQ8o10t. Available at http://cs.fit.edu/~mmahoney/ compression/text.html [47] J. Martens, I. Sutskever. Learning Recurrent Neural Networks with Hessian-Free Op- timization, In: Proceedings of ICML, 2011. [48] T. Mikolov, J. Kopecký, L. Burget, O. Glembek and J. Černocký: Neural network <strong>based</strong> language models for higly inflective languages, In: Proc. ICASSP 2009. [49] T. Mikolov, M. Karafiát, L. Burget, J. Černocký, S. Khudanpur: Recurrent neural network <strong>based</strong> language model, In: Proceedings of Interspeech, 2010. 116
[50] T. Mikolov, S. Kombrink, L. Burget, J. Černocký, S. Khudanpur: Extensions of recurrent neural network language model, In: Proceedings of ICASSP 2011. [51] T. Mikolov, A. Deoras, S. Kombrink, L. Burget, J. Černocký. Empirical Evaluation and Combination of Advanced <strong>Language</strong> Modeling Techniques, In: Proceedings of Interspeech, 2011. [52] T. Mikolov, A. Deoras, D. Povey, L. Burget, J. Černocký. Strategies for Training Large Scale Neural Network <strong>Language</strong> <strong>Models</strong>, In: Proc. ASRU, 2011. [53] M. Minsky, S. Papert. Perceptrons: An Introduction to Computational Geometry, MIT Press, 1969. [54] A. Mnih, G. Hinton. Three new graphical models for statistical language modelling. Proceedings of the 24th international conference on Machine learning, 2007. [55] A. Mnih, G. Hinton. A Scalable Hierarchical Distributed <strong>Language</strong> Model. Advances in Neural Information Processing Systems 21, MIT Press, 2009. [56] S. Momtazi, F. Faubel, D. Klakow. Within and Across Sentence Boundary <strong>Language</strong> Model. In: Proc. Interspeech 2010. [57] F. Morin, Y. Bengio: Hierarchical Probabilistic Neural Network <strong>Language</strong> Model. AISTATS, 2005. [58] P. Norwig. Theorizing from Data: Avoiding the Capital Mistake. Video lecture available at http://www.youtube.com/watch?v=nU8DcBF-qo4. [59] I. Oparin, O. Glembek, L. Burget, J. Černocký: Morphological random forests for language modeling of inflectional languages, In Proc. IEEE Workshop on Spoken <strong>Language</strong> Technology, 2008. [60] D. Povey, A. Ghoshal et al. The Kaldi Speech Recognition Toolkit, In: Proceedings of ASRU, 2011. [61] A. Rastrow, M. Dreyer, A. Sethy, S. Khudanpur, B. Ramabhadran, M. Dredze. Hill climbing on speech lattices: a new rescoring framework. In Proceedings of ICASSP, 2011. [62] W. Reichl, W. Chou. Robust Decision Tree State Tying for Continuous Speech Recog- nition, IEEE Trans. Speech and Audio Processing, 2000. 117
Page 1 and 2:
VYSOKÉ UČENÍ TECHNICKÉ V BRNĚ
Page 3 and 4:
Abstrakt Statistické jazykové mod
Page 5 and 6:
Contents 1 Introduction 4 1.1 Motiv
Page 7 and 8:
6.2.3 Reduction of Vocabulary Size
Page 9 and 10:
Maybe the most popular vision of fu
Page 11 and 12:
Chapter 6 presents further extensio
Page 13 and 14:
Chapter 2 Overview of Stati
Page 15 and 16:
2.1 Evaluation 2.1.1 Perplexity Eva
Page 17 and 18:
ALP can be used to obtain prior pro
Page 19 and 20:
• Good theoretical motivation •
Page 21 and 22:
abilities of n-grams are stored in
Page 23 and 24:
main (static) n-gram model. As the
Page 25 and 26:
There are many popular examples sho
Page 27 and 28:
y Chen et al., who proposed a so-ca
Page 29 and 30:
confusion among researchers, and ma
Page 31 and 32:
language model took almost a week u
Page 33 and 34:
w(t) s(t-1) s(t) U V W y(t) Figure
Page 35 and 36:
ate is halved at start of every new
Page 37 and 38:
or using matrix-vector notation as
Page 39 and 40:
information for more than 5 time st
Page 41 and 42:
A simple solution to the exploding
Page 43 and 44:
output layer changes to computation
Page 45 and 46:
While RNN models can overcome this
Page 47 and 48:
complex or random architectures (su
Page 49 and 50:
While for any of the previous point
Page 51 and 52:
where λ is the interpolation weigh
Page 53 and 54:
model with default SRILM cutoffs pr
Page 55 and 56:
experiments, we have used the one i
Page 57 and 58:
Perplexity (Penn corpus) 145 140 13
Page 59 and 60:
with syntactical NNLMs would be pre
Page 61 and 62:
Table 4.3: Combination of individua
Page 63 and 64:
Table 4.6: Results on Penn Treebank
Page 65 and 66:
4.6 Conclusion of the Model Combina
Page 67 and 68:
were: 400 classes, hidden layer siz
Page 69 and 70: Entropy per word on the WSJ test da
Page 71 and 72: Table 5.3: Results on the WSJ setup
Page 73 and 74: Table 5.5: Results for models <stro
Page 75 and 76: trained together with a maximum ent
Page 77 and 78: wt-3 wt-2 wt-1 D D D P(wt|context)
Page 79 and 80: n-gram probabilities. However, it w
Page 81 and 82: Table 6.1: Training corpora for NIS
Page 83 and 84: Perplexity 360 340 320 300 280 260
Page 85 and 86: Entropy per word 9 8.5 8 7.5 7 6.5
Page 87 and 88: 1 a a a 1 2 3 P(w(t)|*) ONE TWO THR
Page 89 and 90: Table 6.4: Perplexity on the evalua
Page 91 and 92: Entropy reduction per word over KN4
Page 93 and 94: Table 6.6: Perplexity with the new
Page 95 and 96: Entropy reduction over KN5 -0.04 -0
Page 97 and 98: as a baseline, and 12.3% after resc
Page 99 and 100: Table 7.1: BLEU on IWSLT 2005 Machi
Page 101 and 102: Table 7.3: Size of compressed text
Page 103 and 104: Table 7.4: Accuracy of different la
Page 105 and 106: Table 7.6: Entropy on PTB with n-gr
Page 107 and 108: 8.1 Machine Learning One possible d
Page 109 and 110: that almost every non-trivial compu
Page 111 and 112: supervision such as one digit at a
Page 113 and 114: Chapter 9 Conclusion and Future Wor
Page 115 and 116: from the expensive part of the mode
Page 117 and 118: Bibliography [1] A. Alexandrescu, K
Page 119: [23] D. Filimonov, M. Harper. A joi
Page 123 and 124: [77] W. Wang, M. Harper. The SuperA
Page 125 and 126: Test Phase After the model is train
Page 127 and 128: • compute sentence-level scores g
Page 129 and 130: Appendix B: Data generated from mod
Page 131 and 132: Appendix C: Example of decoded utte
Page 133: AND SAYS THIS SWING IS WITHIN A REA
show all

Statistical Language Models based on Neural Networks - Faculty of ...

Create successful ePaper yourself

Delete template?

Save as template?