Deep-Learning-with-PyTorch

Distinguishing birds from airplanes
177
0
EACH ELEMENT
BETWEeN
0 AND 1
x1, x2
=
x 1
1
+
x 1
x 1 x 2 x 2 x 1 x 2
+
x 1 x 2 x 1 x 2 x 1 x
+
2
SUM OF ELEMENTS
EQUALS 1
x 2 x 1 x 2
+
x 1
x 1
+
x 2
x 2 x 3
= =
1
x 1
x 1
x1, x2, x3
=
+
x 2 x 3 x 1 x 2 x 3 x 1 x 2 x 3
+ + + + +
x1,... , x
=
x 1
x
x 1 x
x 1
+ + + +
x
Figure 7.8
Handwritten softmax
As expected, it satisfies the constraints on probability:
# In[9]:
softmax(x).sum()
# Out[9]:
tensor(1.)
Softmax is a monotone function, in that lower values in the input will correspond to
lower values in the output. However, it’s not scale invariant, in that the ratio between
values is not preserved. In fact, the ratio between the first and second elements of the
input is 0.5, while the ratio between the same elements in the output is 0.3678. This is
not a real issue, since the learning process will drive the parameters of the model in a
way that values have appropriate ratios.
The nn module makes softmax available as a module. Since, as usual, input tensors
may have an additional batch 0th dimension, or have dimensions along which they
encode probabilities and others in which they don’t, nn.Softmax requires us to specify
the dimension along which the softmax function is applied:
# In[10]:
softmax = nn.Softmax(dim=1)
x = torch.tensor([[1.0, 2.0, 3.0],
[1.0, 2.0, 3.0]])