Deep-Learning-with-PyTorch

146 CHAPTER 6 Using a neural network to fit the data
CAPPING THE OUTPUT RANGE
We want to firmly constrain the output of our linear operation to a specific range so
that the consumer of this output doesn’t have to handle numerical inputs of puppies
at 12/10, bears at –10, and garbage trucks at –1,000.
One possibility is to just cap the output values: anything below 0 is set to 0, and anything
above 10 is set to 10. That’s a simple activation function called torch.nn.Hardtanh
(https://pytorch.org/docs/stable/nn.html#hardtanh, but note that the default range
is –1 to +1).
COMPRESSING THE OUTPUT RANGE
Another family of functions that work well is torch.nn.Sigmoid, which includes 1 /
(1 + e ** -x), torch.tanh, and others that we’ll see in a moment. These functions
have a curve that asymptotically approaches 0 or –1 as x goes to negative infinity,
approaches 1 as x increases, and have a mostly constant slope at x == 0. Conceptually,
functions shaped this way work well because there’s an area in the middle of our linear
function’s output that our neuron (which, again, is just a linear function followed
by an activation) will be sensitive to, while everything else gets lumped next to the
boundary values. As we can see in figure 6.4, our garbage truck gets a score of –0.97,
while bears and foxes and wolves end up somewhere in the –0.3 to 0.3 range.
3
2
1
NOT
DOGS
GRIZzLY
BEAR
GOoD
DOGgO
0
-1
VERY
MUCH
DOGS
-2
GARBAGE
TRUCK
-3
-3 -2 -1 0 1 2 3
UNDER-
SATURATED
SENSITIVE
OVER-
SATURATED
Figure 6.4 Dogs, bears, and garbage trucks being mapped to how dog-like they are
via the tanh activation function