Modeling and Multivariate Methods - SAS

634 Comparing Model Performance Chapter 25
Model Comparison Platform Options
Mean Abs Dev The average of the absolute values of the differences between the response and the
predicted response. For categorical responses, the differences are between 1 and p (the fitted probability
for the response level that actually occurred).
Misclassification Rate The rate for which the response category with the highest fitted probability is not
the observed category.
N
The number of observations.
Related Information
“Training and Validation Measures of Fit” on page 286 in the “Creating Neural Networks” chapter provides
more information about measures of fit for categorical responses.
Model Comparison Platform Options
Some options in the Model Comparison red triangle menu depend on your data.
Continuous and Categorical Responses
Model Averaging Makes a new column of the arithmetic mean of the predicted values (for continuous
responses) or the predicted.probabilities (for categorical responses).
Continuous Responses
Plot Actual by Predicted Shows a scatterplot of the actual versus the predicted values. The plots for the
different models are overlaid.
Plot Residual by Row
are overlaid.
Shows a plot of the residuals by row number. The plots for the different models
Profiler Shows a profiler for each response based on prediction formula columns in your data. The
profilers have a row for each model being compared.
Categorical Responses
ROC Curve Shows ROC curves for each level of the response variable. The curves for the different models
are overlaid.
AUC Comparison Provides a comparison of the area under the ROC curve (AUC) from each model. The
area under the curve is the indicator of the goodness of fit, with 1 being a perfect fit.
The report includes the following information:
– standard errors and confidence intervals for each AUC
– standard errors, confidence intervals, and hypothesis tests for the difference between each pair of
AUCs
– an overall hypothesis test for testing whether all AUCs are equal