Modeling and Multivariate Methods - SAS

Chapter 13 Recursively Partitioning Data 335
Validation
Validation
If you grow a tree with enough splits, partitioning can overfit data. When this happens, the model predicts
the fitted data very well, but predicts future observations poorly. Validation is the process of using part of a
data set to estimate model parameters, and using the other part to assess the predictive ability of the model.
• The training set is the part that estimates model parameters.
• The validation set is the part that assesses or validates the predictive ability of the model.
• The test set is a final, independent assessment of the model’s predictive ability. The test set is available
only when using a validation column (see Table 13.1).
The training, validation, and test sets are created by subsetting the original data into parts. Table 13.2
describes several methods for subsetting a data set.
Table 13.2 Validation Methods
Excluded Rows
Holdback
KFold
Uses row states to subset the data. Rows that are unexcluded are used as
the training set, and excluded rows are used as the validation set.
For more information about using row states and how to exclude rows,
see Using JMP.
Randomly divides the original data into the training and validation data
sets. The Validation Portion (see Table 13.1) on the platform launch
window is used to specify the proportion of the original data to use as
the validation data set (holdback).
Divides the original data into K subsets. In turn, each of the K sets is
used to validate the model fit on the rest of the data, fitting a total of K
models. The model giving the best validation statistic is chosen as the
final model.
KFold validation can be used only with the Decision Tree method. To
use KFold, select K Fold Crossvalidation from the platform
red-triangle menu, see “Platform Options” on page 325.
This method is best for small data sets, because is makes efficient use of
limited amounts of data.