Statistical Language Models based on Neural Networks - Faculty of ...

More documents

Recommendations

Info

Chapter 1 Introduction 1.1 Motivation From the first day of existence of the computers, people were dreaming about artificial intelligence - machines that would produce complex behaviour to reach goals specified by human users. Possibility of existence of such machines has been controversial, and many philosophical questions were raised - whether the intelligence is not unique only to humans, or only to animals etc. Very influential work of Alan Turing did show that any computable problem can be computed by Universal Turing Machine - thus, assuming that the human mind can be described by some algorithm, Turing Machine is powerful enough to represent it. Computers today are Turing-complete, ie. can represent any computable algorithm. Thus, the main problem is how to find configuration of the machine so that it would produce desired behaviour that humans consider intelligent. Assuming that the problem is too difficult to be solved immediately, we can think of several ways that would lead us towards intelligent machines - we can start with a simple machine that can recognize basic shapes and images such as written digits, then scale it towards more complex types of images such as human faces and so on, finally reaching machine that can recognize objects in the real world as well as humans can. Other possible way can be to simulate parts of the human brain on the level of individual brain cells, neurons. Computers today are capable of realistically simulating the real world, as can be seen in modern computer games - thus, it seems logical that with accurate simulation of neurons and more computational power, it should be possible to simulate the whole human brain one day. 4
Maybe the most popular vision of future AI as seen in science fiction movies are robots and computers communicating with humans using natural language. Turing himself proposed a test of intelligence <strong>based</strong> on ability of the machine to communicate with humans using natural language [76]. This choice has several advantages - amount of data that has to be processed can be very small compared to machine that recognizes images or sounds. Next, machine that will understand just the basic patterns in the language can be developed first, and scaled up subsequently. The basic level of understanding can be at level of a child, or a person that learns a new language - even such low level of understanding is sufficient to be tested, so that it would be possible to measure progress in ability of the machine to understand the language. Assuming that we would want to build such machine that can communicate in natural language, the question is how to do it. Reasonable way would be to mimic learning processes of humans. A language is learned by observing the real world, recognizing its regularities, and mapping acoustic and visual signals to higher level representations in the brain and back - the acoustic and visual signals are predicted using the higher level representations. Motivation for learning the language is to improve success of humans in the real world. The whole learning problem might be too difficult to be solved at once - there are many open questions regarding importance of individual factors, such as how much data has to be processed during training of the machine, how important is it to learn the language jointly with observing real world situations, how important is the innate knowledge, what is the best formal representation of the language, etc. It might be too ambitious to attempt to solve all these problems together, and to expect too much from models or techniques that even do not allow existence of the solution (an example might be the well-known limitations of finite state machines to represent efficiently longer term patterns). Important work that has to be mentioned here is the Information theory of Claude Shannon. In his famous paper Entropy of printed English [66], Shannon tries to estimate entropy of the English text using simple experiments involving humans and frequency <strong>based</strong> models of the language (n-grams <strong>based</strong> on history of several preceding characters). The conclusion was that humans are by far better in prediction of natural text than n- grams, especially as the length of the context is increased - this so-called ”Shannon game” can be effectively used to develop more precise test of intelligence than the one defined by Turing. If we assume that the ability to understand the language is equal (or at least highly 5
Page 1 and 2: VYSOKÉ UČENÍ TECHNICKÉ V BRNĚ
Page 3 and 4: Abstrakt Statistické jazykové mod
Page 5 and 6: Contents 1 Introduction 4 1.1 Motiv
Page 7: 6.2.3 Reduction of Vocabulary Size
Page 11 and 12: Chapter 6 presents further extensio
Page 13 and 14: Chapter 2 Overview of Stati
Page 15 and 16: 2.1 Evaluation 2.1.1 Perplexity Eva
Page 17 and 18: ALP can be used to obtain prior pro
Page 19 and 20: • Good theoretical motivation •
Page 21 and 22: abilities of n-grams are stored in
Page 23 and 24: main (static) n-gram model. As the
Page 25 and 26: There are many popular examples sho
Page 27 and 28: y Chen et al., who proposed a so-ca
Page 29 and 30: confusion among researchers, and ma
Page 31 and 32: language model took almost a week u
Page 33 and 34: w(t) s(t-1) s(t) U V W y(t) Figure
Page 35 and 36: ate is halved at start of every new
Page 37 and 38: or using matrix-vector notation as
Page 39 and 40: information for more than 5 time st
Page 41 and 42: A simple solution to the exploding
Page 43 and 44: output layer changes to computation
Page 45 and 46: While RNN models can overcome this
Page 47 and 48: complex or random architectures (su
Page 49 and 50: While for any of the previous point
Page 51 and 52: where λ is the interpolation weigh
Page 53 and 54: model with default SRILM cutoffs pr
Page 55 and 56: experiments, we have used the one i
Page 57 and 58: Perplexity (Penn corpus) 145 140 13
Page 59 and 60:
with syntactical NNLMs would be pre
Page 61 and 62:
Table 4.3: Combination of individua
Page 63 and 64:
Table 4.6: Results on Penn Treebank
Page 65 and 66:
4.6 Conclusion of the Model Combina
Page 67 and 68:
were: 400 classes, hidden layer siz
Page 69 and 70:
Entropy per word on the WSJ test da
Page 71 and 72:
Table 5.3: Results on the WSJ setup
Page 73 and 74:
Table 5.5: Results for models <stro
Page 75 and 76:
trained together with a maximum ent
Page 77 and 78:
wt-3 wt-2 wt-1 D D D P(wt|context)
Page 79 and 80:
n-gram probabilities. However, it w
Page 81 and 82:
Table 6.1: Training corpora for NIS
Page 83 and 84:
Perplexity 360 340 320 300 280 260
Page 85 and 86:
Entropy per word 9 8.5 8 7.5 7 6.5
Page 87 and 88:
1 a a a 1 2 3 P(w(t)|*) ONE TWO THR
Page 89 and 90:
Table 6.4: Perplexity on the evalua
Page 91 and 92:
Entropy reduction per word over KN4
Page 93 and 94:
Table 6.6: Perplexity with the new
Page 95 and 96:
Entropy reduction over KN5 -0.04 -0
Page 97 and 98:
as a baseline, and 12.3% after resc
Page 99 and 100:
Table 7.1: BLEU on IWSLT 2005 Machi
Page 101 and 102:
Table 7.3: Size of compressed text
Page 103 and 104:
Table 7.4: Accuracy of different la
Page 105 and 106:
Table 7.6: Entropy on PTB with n-gr
Page 107 and 108:
8.1 Machine Learning One possible d
Page 109 and 110:
that almost every non-trivial compu
Page 111 and 112:
supervision such as one digit at a
Page 113 and 114:
Chapter 9 Conclusion and Future Wor
Page 115 and 116:
from the expensive part of the mode
Page 117 and 118:
Bibliography [1] A. Alexandrescu, K
Page 119 and 120:
[23] D. Filimonov, M. Harper. A joi
Page 121 and 122:
[50] T. Mikolov, S. Kombrink, L. Bu
Page 123 and 124:
[77] W. Wang, M. Harper. The SuperA
Page 125 and 126:
Test Phase After the model is train
Page 127 and 128:
• compute sentence-level scores g
Page 129 and 130:
Appendix B: Data generated from mod
Page 131 and 132:
Appendix C: Example of decoded utte
Page 133:
AND SAYS THIS SWING IS WITHIN A REA
show all

Statistical Language Models based on Neural Networks - Faculty of ...

Create successful ePaper yourself

Delete template?

Save as template?