Deep-Learning-with-PyTorch

96 CHAPTER 4 Real-world data representation using tensors
4.5.3 One-hot encoding whole words
We have one-hot encoded our sentence into a representation that a neural network
could digest. Word-level encoding can be done the same way by establishing a vocabulary
and one-hot encoding sentences—sequences of words—along the rows of our
tensor. Since a vocabulary has many words, this will produce very wide encoded vectors,
which may not be practical. We will see in the next section that there is a more
efficient way to represent text at the word level, using embeddings. For now, let’s stick
with one-hot encodings and see what happens.
We’ll define clean_words, which takes text and returns it in lowercase and
stripped of punctuation. When we call it on our “Impossible, Mr. Bennet” line, we get
the following:
# In[6]:
def clean_words(input_str):
punctuation = '.,;:"!?”“_-'
word_list = input_str.lower().replace('\n',' ').split()
word_list = [word.strip(punctuation) for word in word_list]
return word_list
words_in_line = clean_words(line)
line, words_in_line
# Out[6]:
('“Impossible, Mr. Bennet, impossible, when I am not acquainted with him',
['impossible',
'mr',
'bennet',
'impossible',
'when',
'i',
'am',
'not',
'acquainted',
'with',
'him'])
Next, let’s build a mapping of words to indexes in our encoding:
# In[7]:
word_list = sorted(set(clean_words(text)))
word2index_dict = {word: i for (i, word) in enumerate(word_list)}
len(word2index_dict), word2index_dict['impossible']
# Out[7]:
(7261, 3394)
Note that word2index_dict is now a dictionary with words as keys and an integer as a
value. We will use it to efficiently find the index of a word as we one-hot encode it.
Let’s now focus on our sentence: we break it up into words and one-hot encode it—