I__. - International Military Testing Association

Item Content Validity: Its Relationship
With Item Discrimination and Difficulty
Teresa M. Rushano
USAF Occupational Measurement Squadron
At the USAF Occupational Measurement Squadron (USAFOMS), subject-matter experts
(SMEs) rate the questions on promotion tests for content validity.
They also use standard statistical criteria to determine whether test questions
should be reused on subsequent test revisions. The purpose of this
research was to explore the relationship between SME content validity ratings
(CVRs) and item statistics.
The Specialty Knowledge Tests (SKTs) used for enlisted promotions in the Air
Force are written at USAFOMS by senior NCOs acting as SMEs under the guidance .
of USAFOMS psychologists. Within each specialty, one SKT is prepared for
promotion to staff sergeant (E-51, and one for promotion to technical and
master sergeant (E-6 and E-7).
The USAFOMS test development process includes a procedure based on the methodology
of Lawshe (1975) for quantifying content validity on the basis of
essentiality to job performance. As part of the process of revising an existing
SKT, each SME independently assigns each test question a rating using
the following scale:
Is the skill (or knowledge) measured by this test question:
(21, Essential
Useful but not essential (11, or
N o t n e c e s s a r y (01,
for successful performance on the job?
The SMEs as a team then use these ratings as a point of departure in discussing
whether individual items should be retained on subsequent test revisions.
Perry, Williams, and Stanley (1990) found that CVRs influence SME determina-
tion of an item's test-worthiness and its subsequent selection for continued
use or deactivation. However, the ratings are not the only factors which may
impact the SME decision whether to reuse an item on an SKT. After completing
the CVRs, SMEs review item statistics.
For each SKT question, item statistics are provided which indicate how well
,-an item is doing on the test. USAFOMS has an established set of statistical
Tcriteria for test items which must be met. Test questions that do not meet
these criteria must be revised in order to be incorporated on the revised
3 version of the test. The two statistical elements examined in this research
~.'are the difficulty index and discrimination index. The difficulty (DIFF) of
# a test item, sometimes known as its ease index, is defined as the total percentage
of examinees on a test who selected each choice. The DIFF value for
the correct answer is examined to see if the item as a whole is too easy or
too hard. For example, an item answered correctly by 97% of the examinees is
considered too easy for the purposes of the SKT and would not be reused on
subsequent test revisions.
The s,econd statistical element used in this research is the discrimination
index (DISC). This statistic is calculated for each item choice by subtract-
ing the percentage of low-scoring examinees (i.e., those scoring in the lower
50% of all examinees) who select a choice, from the percentage of high-scoring
examinees making that choice. If a test question is working properly,