Peptide-Based Drug Design

152 Hilpert et al.
than control. Given the large amount of data available, a rigorous methodology
was feasible. We performed a 10-fold stratified, cross-validation where each of
the two sets of peptides were randomly divided into 10 groups, requiring the
number of active peptides in each section to be approximately the same in each
group. By consecutively leaving one group of 10% out for use in validation
and combining the other 9 groups into a single training group, 10 groups were
constructed for training the models. ANNs were constructed using simulation
software (86), by training on each of the 10 training groups and using the leftout
validation group to signal the end of training and preventing overfitting.
(Overfitting occurs when there are too few data for the complexity of the model:
the model tends then to “memorize” the data and while it describes the training
data very well, it performs poorly when used to predict data outside the training
set.)
The performance of any model built on data is best measured by accessing the
predictions of the model in situations that did not exist in the training data. Given
the large amounts of peptide activity data we had available, we chose to assess
the model predictions on the set of data that was not involved in any way in the
construction of the models: we used the Set A models to predict the activity of
Set B peptides, and Set B models to predict activity of Set A peptides. Rather
than attempt to predict IC50, each ANN produced a single number between 0
and 1 to indicate the likelihood that the peptide was more active than Bac2A.
An example of the performance of one selected ANN model built on 90% of
Set A peptides for predicting Set B peptide activity is shown in Fig. 3. The
positive predictive value (PPV, the fraction of peptides that are active out of
all peptides predicted active) is quite low for ANN output threshold values up
to 0.9; but the PPV increases dramatically to 1.0 for threshold values greater
than 0.9. For threshold values less than 0.9, most peptides predicted to be active
are in fact inactive (PPV < 0.5). However, for high threshold values ( 0.5), but few
peptides are predicted to be active (only 2% of peptides at threshold of 0.999).
For such high threshold values, many active peptides are incorrectly classified
as inactive. However, this may not be a problem for an in silico screening
of large numbers of peptides where high confidence is needed that peptides
predicted to be active are worth the expense of experimental investigation. Such
screening programs may not be concerned that many active peptides are incorrectly
classified as inactive, only that those predicted active are most likely
active.
This is illustrated by examining the activities of test peptides ranked according
to the ANN outputs (values near 1 indicating likelihood to be more active
peptides, and values close to 0 indicating likelihood to not be more active
peptides). The results are summarized in Table 4. Here, the proportion of