Why use multiple-choice questions on accounting - Bryant

<strong>Why</strong> Use Multiple Choice Questions 41scores is zero). The analyses in Table 7 suggest that only the data from classes 1, 5 and 8 show nosignificant differences. In all other classes, the t-test statistic was significant at an alpha level of .05.Parametric tests require several assumptions regarding the normality of the distribution forthe test statistic of interest, and are also sensitive to outliers whose magnitudes can overwhelm theconforming behavior of the remaining data. Accordingly, we also performed a series of Wilcoxonsigned rank tests to overcome this potential problem in our ten sets of class data. Like a matchedpairtest, a Wilcoxon test also computes the difference in each student’s CR and MC scores, but thenassigns a rank order to each absolute difference. The test statistic is the mean of these ranks (afterreassigning the positive or negative sign to each rank from the original difference). Differences ofzero are removed from further consideration, explaining why the number of observations, n, differsfor the same class in some rows of Table 7.Under the null hypothesis that the MC and CR scores come from the same population, theaverage of these terms should cluster around zero. For each class tested, the results for the Wilcoxonsigned rank test were consistent with the results of the paired t-test. Again, only classes 1, 5 and 8were statistically insignificant at an alpha level of .05.Our finding of statistical significance for the majority of our data somewhat contrasts withsome of our regression results in terms of students performing consistently between the two testformats. The null hypothesis for both statistical tests reported in Table 7 is that the scores from theMC and CR portions of our student exams came from the same population. The statistical outcomesfor seven out of ten tests performed here were that they did not–i.e., that student performance on theseparate assessment measures was meaningfully different. In fact, in our sample classes, studentsconsistently performed better on the CR portions of their exams, with mean differences ranging from3.9 percentage points to 19.7 points. We speculate that a student’s ability to earn partial credit onthe CR portions of these tests may explain some of this.DISCUSSION AND CAVEATSOur statistical analysis suggests that a positive relationship exists between the MC and CRscores. This is to say that, while there may be differences in the magnitude of scores, the scores atleast vary in a similar direction. The good news in this finding is the implication that using MC teststo measure understanding of a given subject area at least points in the right direction–i.e., thoseindividuals who do well on MC tests are also likely to do well on CR tests, and vice versa. This isheartening news to certification test developers who seek exactly this relationship.The results of both our matched-pair tests and our non-parametric tests are somewhat lessencouraging, however–indicating that the MC and CR portions of each exam were not equivalenttests. The MC portions of all these student tests, even after adjusting for “gender” and other potentialdifferences in our sample classes, explained about 30 percent of the variation in the scores of the CRportions of those same tests. Certification test developers are likely to be disappointed that thisstatistic was not higher. It indicates that the independent variables of our model–particularly studentperformance on the MC portions of the sample examinations–explained a little less than one thirdof the variability of the dependent (CR) variable. To us, it also suggests that the MC and CR portionsof these tests are testing different types of understanding, or perhaps testing the same understandingat different depths.The reasons for using MC tests instead of CR tests are well documented, and the authorsacknowledge the convenience of employing them on certification examinations. We would also becomfortable with such formats if we can show that there is a clear and consistent positive