The Annoyance Filter.pdf - Fourmilab

§27 ANNOYANCE-FILTER DICTIONARY 31
27. Plot a histogram of the distribution of words in the dictionary by probability. Words with negative
probability are ignored, so there is no need to purge before plotting.
〈 Class implementations 11 〉 +≡
#ifdef HAVE_PLOT_UTILITIES
#define PLOT_DEBUG
void dictionary ::plotProbabilityHistogram (string fileName , unsigned int nBins ) const
{
if (verbose ) {
cerr ≪ "Plotting␣probability␣histogram␣to␣" ≪ fileName ≪ ".png" ≪ endl ;
}
ofstream gp((fileName + ".gp").c str ( )), dat ((fileName + ".dat").c str ( ));
〈 Build histogram of word probabilities 28 〉;
〈 Write GNUPLOT data table for probability histogram 29 〉;
/∗ Create GNUPLOT instructions to plot data ∗/
gp ≪ "set␣term␣pbm␣small␣color" ≪ endl ;
gp ≪ "set␣ylabel␣\"Number␣of␣Words\"" ≪ endl ;
gp ≪ "set␣xlabel␣\"Probability\"" ≪ endl ;
gp ≪ "plot␣\"" ≪ fileName ≪ ".dat\"␣using␣1:2␣title␣\"\"␣with␣boxes" ≪ endl ;
string command ("gnuplot␣");
command += fileName + ".gp␣|␣pnmtopng␣>" + fileName + ".png";
#ifdef PLOT_DEBUG
cout ≪ command ≪ endl ;
#else
command += "␣2>/dev/null";
#endif
gp.close ( );
dat .close ( );
system (command .c str ( ));
#ifndef PLOT_DEBUG /∗ Delete the temporary files used to create the plot ∗/
remove ((fileName + ".gp").c str ( ));
remove ((fileName + ".dat").c str ( ));
#endif
}
#endif /∗ HAVE_PLOT_UTILITIES ∗/
28. Walk through the dictionary and bin the probabilities of words into nBins equally sized bins and
compute a histogram of the numbers in each bin.
〈 Build histogram of word probabilities 28 〉 ≡
vector〈unsigned int〉 hist (nBins );
for (const iterator mp = begin ( ); mp ≠ end ( ); mp ++) {
if (mp ⃗ second .getJunkProbability ( ) ≥ 0) {
unsigned int bin = static cast〈unsigned int〉(mp ⃗ second .getJunkProbability ( ) ∗ nBins );
}
}
hist [bin ]++;
This code is used in section 27.