Statistical Language Models based on Neural Networks - Faculty of ...

of the task, as often the easiest way how to obtain good results is to choose crude but fast
techniques, and train models on as much data as available. This strategy is however not
getting any closer to solving the problems, rather avoiding them for as long as possible.
For many tasks today, the amount of the available training data is so huge that further
progress by adding another data is not very likely.
Another reason why advanced techniques are not used in practice is importance of the
achieved results: it is commonly known that most of the published papers report only
negligible improvements over basic baselines. Even the best techniques rarely affect the
word error rate of speech recognition systems by more than 10% relatively - and that is
hardly observable difference from the user perspective. However, even small difference can
be huge in the long term - competitions are often won by a slight margin. Also, even if
the improvements are small and hardly observable, it is likely that in the longer term, the
majority of users will tend to prefer the best system.
While I see integration of neural net language models into production systems as the
next step for the language modeling research, there is still much to do in the basic research.
Based on the history of the language modeling research that has been often rather chaotic,
it might be fruitful to first define a roadmap. While detailed proposal for future research
is out of scope of this work, the main points are:
• The involved models should be computationally much less restricted than the tradi-
tional ones; it should be clear that a compact solution to simple problems can exist
in the model space
• The progress should be measured on increasingly more complex tasks (for example,
finding the most likely word in an incomplete sentence, as in [83])
• The tasks and the training data should be coherent and publicly available
While such research would not be competitive with the common techniques in the
short term, it is certain that a progress beyond models such as finite state machines is
needed. It has been popular to claim that we need orders of magnitude more powerful
computers, and also much more training data to make progress towards AI - I find this
doubtful. In my opinion, what needs to be addressed is the capability of the machine
learning techniques to efficiently discover new patterns.
112