Modeling and Multivariate Methods - SAS

284 Creating Neural Networks Chapter 10
Overview of Neural Networks
Table 10.4 Activation Functions (Continued)
Linear
Gaussian
The identity function. The linear combination of the X variables is not
transformed.
The Linear activation function is most often used in conjunction with one
of the non-linear activation functions. In this case, the Linear activation
function is placed in the second layer, and the non-linear activation
functions are placed in the first layer. This is useful if you want to first
reduce the dimensionality of the X variables, and then have a nonlinear
model for the Y variables.
For a continuous Y variable, if only Linear activation functions are used, the
model for the Y variable reduces to a linear combination of the X variables.
For a nominal or ordinal Y variable, the model reduces to a logistic
regression.
The Gaussian function. Use this option for radial basis function behavior, or
when the response surface is Gaussian (normal) in shape. The Gaussian
function is:
e–
x2
where x is a linear combination of the X variables.
Boosting
Boosting is the process of building a large additive neural network model by fitting a sequence of smaller
models. Each of the smaller models is fit on the scaled residuals of the previous model. The models are
combined to form the larger final model. The process uses validation to assess how many component
models to fit, not exceeding the specified number of models.
Boosting is often faster than fitting a single large model. However, the base model should be a 1 to 2 node
single-layer model, or else the benefit of faster fitting can be lost if a large number of models is specified.
Use the Boosting panel in the Model Launch to specify the number of component models and the learning
rate. Use the Hidden Layer Structure panel in the Model Launch to specify the structure of the base model.
The learning rate must be 0 < r ≤ 1. Learning rates close to 1 result in faster convergence on a final model,
but also have a higher tendency to overfit data. Use learning rates close to 1 when a small Number of Models
is specified.
As an example of how boosting works, suppose you specify a base model consisting of one layer and two
nodes, with the number of models equal to eight. The first step is to fit a one-layer, two-node model. The
predicted values from that model are scaled by the learning rate, then subtracted from the actual values to
form a scaled residual. The next step is to fit a different one-layer, two-node model on the scaled residuals of
the previous model. This process continues until eight models are fit, or until the addition of another model
fails to improve the validation statistic. The component models are combined to form the final, large model.
In this example, if six models are fit before stopping, the final model consists of one layer and 2 x 6 = 12
nodes.