Statistical Language Models based on Neural Networks - Faculty of ...

More documents

Recommendations

Info

[63] D. Rohde, D. Plaut. <strong>Language</strong> acquisition in the absence of explicit negative evidence: How important is starting small?, Cognition, 72, 67-109, 1999. [64] R. Rosenfeld. Adaptive <strong>Statistical</strong> <strong>Language</strong> Modeling: A Maximum Entropy Ap- proach,. Ph.D. thesis, Carnegie Mellon University, 1994. [65] D. E. Rumelhart, G. E. Hinton, R. J. Williams. Learning internal representations by back-propagating errors. Nature, 323:533.536, 1986. [66] C. E. Shannon. Prediction and entropy of printed English. Bell Systems Technical Journal, 1951. [67] J. Schmidhuber, S. Heil. Sequential Neural Text Compression, IEEE Trans. on Neural Networks 7(1): 142-146, 1996. [68] H. Schwenk, J. Gauvain. Training Neural Network <strong>Language</strong> <strong>Models</strong> On Very Large Corpora. In Proceedings of Joint Conference HLT/EMNLP, 2005. [69] H. Schwenk. Continuous space language models. Computer Speech and <strong>Language</strong>, vol. 21, 2007. [70] R. Solomonoff. Machine Learning - Past and Future. The Dartmouth Artificial Intel- ligence Conference, Dartmouth, 2006. [71] H. Soltau, G. Saon, B. Kingsbury. The IBM Attila Speech Recognition Toolkit, In: Proceedings of IEEE Workshop on Spoken <strong>Language</strong> Technology, 2010. [72] A. Stolcke. SRILM – An Extensible <strong>Language</strong> Modeling Toolkit. Proc. Intl. Conf. on Spoken <strong>Language</strong> Processing, vol. 2, pp. 901-904, 2002. [73] A. Stolcke. Entropy-<strong>based</strong> pruning of backoff language models. In Proceedings DARPA Broadcast News Transcription and Understanding Workshop, Lansdowne, VA, 1998. [74] I. Sutskever, J. Martens, G. Hinton. Generating Text with Recurrent Neural Net- works. In: Proceedings of ICML, 2011. [75] I. Szöke. Hybrid word-subword spoken term detection. PhD thesis, Brno, CZ, FIT BUT, 2010. [76] A. M. Turing. Computing machinery and intelligence. Mind, LIX:433-460, 1950. 118
[77] W. Wang, M. Harper. The SuperARV language model: Investigating the effectiveness of tightly integrating multiple knowledge sources. In Proceedings of Conference of Empirical Methods in Natural <strong>Language</strong> Processing, 2002. [78] Peng Xu. Random forests and the data sparseness problem in language modeling, Ph.D. thesis, Johns Hopkins University, 2005. [79] Puyang Xu, D. Karakos, S. Khudanpur. Self-Supervised Discriminative Training of <strong>Statistical</strong> <strong>Language</strong> <strong>Models</strong>. In Proc. ASRU, 2009. [80] Puyang Xu, A. Gunawardana, S. Khudanpur. Efficient Subsampling for Training Complex <strong>Language</strong> <strong>Models</strong>, In: Proceedings of EMNLP, 2011. [81] W. Xu, A. Rudnicky. Can Artificial Neural Networks Learn <strong>Language</strong> <strong>Models</strong>? In- ternational Conference on <strong>Statistical</strong> <strong>Language</strong> Processing, 2000. [82] G. Zweig, P. Nguyen at al. Speech Recognition with Segmental Conditional Random Fields: A Summary of the JHU CLSP Summer Workshop. In Proceedings of ICASSP, 2011. [83] G. Zweig, C.J.C. Burges. The Microsoft Research Sentence Completion Challenge, Microsoft Research Technical Report MSR-TR-2011-129, 2011. 119
Page 1 and 2:
VYSOKÉ UČENÍ TECHNICKÉ V BRNĚ
Page 3 and 4:
Abstrakt Statistické jazykové mod
Page 5 and 6:
Contents 1 Introduction 4 1.1 Motiv
Page 7 and 8:
6.2.3 Reduction of Vocabulary Size
Page 9 and 10:
Maybe the most popular vision of fu
Page 11 and 12:
Chapter 6 presents further extensio
Page 13 and 14:
Chapter 2 Overview of Stati
Page 15 and 16:
2.1 Evaluation 2.1.1 Perplexity Eva
Page 17 and 18:
ALP can be used to obtain prior pro
Page 19 and 20:
• Good theoretical motivation •
Page 21 and 22:
abilities of n-grams are stored in
Page 23 and 24:
main (static) n-gram model. As the
Page 25 and 26:
There are many popular examples sho
Page 27 and 28:
y Chen et al., who proposed a so-ca
Page 29 and 30:
confusion among researchers, and ma
Page 31 and 32:
language model took almost a week u
Page 33 and 34:
w(t) s(t-1) s(t) U V W y(t) Figure
Page 35 and 36:
ate is halved at start of every new
Page 37 and 38:
or using matrix-vector notation as
Page 39 and 40:
information for more than 5 time st
Page 41 and 42:
A simple solution to the exploding
Page 43 and 44:
output layer changes to computation
Page 45 and 46:
While RNN models can overcome this
Page 47 and 48:
complex or random architectures (su
Page 49 and 50:
While for any of the previous point
Page 51 and 52:
where λ is the interpolation weigh
Page 53 and 54:
model with default SRILM cutoffs pr
Page 55 and 56:
experiments, we have used the one i
Page 57 and 58:
Perplexity (Penn corpus) 145 140 13
Page 59 and 60:
with syntactical NNLMs would be pre
Page 61 and 62:
Table 4.3: Combination of individua
Page 63 and 64:
Table 4.6: Results on Penn Treebank
Page 65 and 66:
4.6 Conclusion of the Model Combina
Page 67 and 68:
were: 400 classes, hidden layer siz
Page 69 and 70:
Entropy per word on the WSJ test da
Page 71 and 72: Table 5.3: Results on the WSJ setup
Page 73 and 74: Table 5.5: Results for models <stro
Page 75 and 76: trained together with a maximum ent
Page 77 and 78: wt-3 wt-2 wt-1 D D D P(wt|context)
Page 79 and 80: n-gram probabilities. However, it w
Page 81 and 82: Table 6.1: Training corpora for NIS
Page 83 and 84: Perplexity 360 340 320 300 280 260
Page 85 and 86: Entropy per word 9 8.5 8 7.5 7 6.5
Page 87 and 88: 1 a a a 1 2 3 P(w(t)|*) ONE TWO THR
Page 89 and 90: Table 6.4: Perplexity on the evalua
Page 91 and 92: Entropy reduction per word over KN4
Page 93 and 94: Table 6.6: Perplexity with the new
Page 95 and 96: Entropy reduction over KN5 -0.04 -0
Page 97 and 98: as a baseline, and 12.3% after resc
Page 99 and 100: Table 7.1: BLEU on IWSLT 2005 Machi
Page 101 and 102: Table 7.3: Size of compressed text
Page 103 and 104: Table 7.4: Accuracy of different la
Page 105 and 106: Table 7.6: Entropy on PTB with n-gr
Page 107 and 108: 8.1 Machine Learning One possible d
Page 109 and 110: that almost every non-trivial compu
Page 111 and 112: supervision such as one digit at a
Page 113 and 114: Chapter 9 Conclusion and Future Wor
Page 115 and 116: from the expensive part of the mode
Page 117 and 118: Bibliography [1] A. Alexandrescu, K
Page 119 and 120: [23] D. Filimonov, M. Harper. A joi
Page 121: [50] T. Mikolov, S. Kombrink, L. Bu
Page 125 and 126: Test Phase After the model is train
Page 127 and 128: • compute sentence-level scores g
Page 129 and 130: Appendix B: Data generated from mod
Page 131 and 132: Appendix C: Example of decoded utte
Page 133: AND SAYS THIS SWING IS WITHIN A REA
show all

Statistical Language Models based on Neural Networks - Faculty of ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?