2DkcTXceO

56 Unorthodox journey to statistics(b) Most states, as well as some large cities, take part in state surveys, givingeducational information for those units separately. Suppose one wants tocompare every unit with every other. With, say, 45 units, the number ofcomparisons in the family is 45×44/2, or 990. On the other hand, people ina single unit (e.g., state) are usually interested mainly in how it compareswith other units; these comparisons result in a family size of 44. Resultsare likely to differ. How can one reconcile the different decisions, whenresults must be transmitted to a public that thinks that there should beasingledecisionforeachcomparison?Extensive background material and results for NAEP are available atnces.ed.gov/nationsreportcard. In addition, there is a book describing the developmentof the survey (Jones and Olkin, 2004). For information specificallyon handling of multiplicity issues, see nces.ed.gov/nationsreportcard/tdw/analysis/2000\_2001/infer\_multiplecompare\_fdr.asp.In summary, the work on multiplicity has multiplied with the advent of bigdata, although the ill-structured situations described above have been aroundfor a long time with little formal attention, and more guidance on handling offamily size issues with examples would be a contribution that could result inwider use of multiple comparison methods.ReferencesBenjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate:Apracticalandpowerfulapproachtomultipletesting.Journal of theRoyal Statistical Society, Series B, 57:289–300.Braun, H., Ed. (1994). The Collected Works of John W. Tukey. Vol. 8: MultipleComparisons, 1948–1983. Chapman & Hall, London.de Kruif, P. (1926). Microbe Hunters. Harcourt, Brace, Jovanovich, NewYork.Finner, H. (1999). Stepwise multiple test procedures and control of directionalerrors. The Annals of Statistics, 27:274–289.Guo, W., Sarkar, S.K., and Peddada, S.D. (2010). Controlling false discoveriesin multidimensional directional decisions, with applications to geneexpression data on ordered categories. Biometrics, 66:485–492.Ioannidis, J.P.A. (2005). Why most published research findings are false.PLoS Medicine, 2(8):e124.Jones, L.V. and Olkin, I., Eds. (2004). The Nation’s Report Card: Evolutionand Perspectives. Phi Delta Kappa Educational Foundation, Bloomington,IN.