Modeling and Multivariate Methods - SAS

214 Performing Logistic Regression on Nominal and Ordinal Responses Chapter 7
Validation
Confusion Matrix
Profiler
A confusion matrix is a two-way classification of the actual response levels and the predicted response levels.
For a good model, predicted response levels should be the same as the actual response levels. The confusion
matrix gives a way of assessing how the predicted responses align with the actual responses.
Brings up the prediction profiler, showing the fitted values for a specified response probability as the values
of the factors in the model are changed. This feature is available for both nominal and ordinal responses. For
detailed information about profiling features, refer to the “Visualizing, Optimizing, and Simulating
Response Surfaces” chapter on page 553.
Validation
Validation is the process of using part of a data set to estimate model parameters, and using the other part to
assess the predictive ability of the model.
• The training set is the part that estimates model parameters.
• The validation set is the part that assesses or validates the predictive ability of the model.
• The test set is a final, independent assessment of the model’s predictive ability. The test set is available
only when using a validation column.
The training, validation, and test sets are created by subsetting the original data into parts. This is done
through the use of a validation column in the Fit Model launch window.
The validation column’s values determine how the data is split, and what method is used for validation:
• If the column has two distinct values, then training and validation sets are created.
• If the column has three distinct values, then training, validation, and test sets are created.
• If the column has more than three distinct values, or only one, then no validation is performed.
When validation is used, model fit statistics are given for the training, validation, and test sets.
Example of a Nominal Logistic Model
A market research study was undertaken to evaluate preference for a brand of detergent (Ries and Smith
1963). The results are in the Detergent.jmp sample data table. The model is defined by the following:
• the response variable, brand with values m and x
• an effect called softness (water softness) with values soft, medium, and hard
• an effect called previous use with values yes and no
• an effect called temperature with values high and low
• a count variable, count, which gives the frequency counts for each combination of effect categories.