I__. - International Military Testing Association

Implementation of Content Validity Ratings
in Air Force Promotion Test Construction
Carlene M. Perry
United States Air Force Academy
John E. Williams
Paul P. Stanley II
USAF Occupational Measurement Squadron
The USAFOMS has implemented a procedure in which subject-matter experts
(SMEs) rate the content validity of individual test questions on Air Force
promotion tests. This paper describes that procedure and assesses its impact
upon test content and the perceptions of those involved in test,developm.ent.
Specialty Knowledge Tests (SKTs) are loo-item multiple-choice tests which
measure airmen's knowledge in their assigned Air Force specialties. Promotion
to the ranks of staff sergeant (E-5) through master sergeant (E-7) is
determined solely by relative ranking under the Weighted Airman Promotion
System (WAPS). Each airman receives a single WAPS score, the sum of six com-
ponent measures, with the SKT accounting for UP to 22% of the total. Since
the other components generally yield little variance among individuals, the
SKT is often the deciding factor in determining who gets promoted.
SKTs are written by teams of senior NCOs with the assistance of USAFOMS psychologists.
They are constructed using the content validity strategy of validation.
The following components ensure a direct and close relationship
between test items and important facets of the specialty being tested: 1)
the specialty training standard, an Air Force document which identifies the
primary duties and responsibilities in a specialty, 2) CODAP-based occupational
analysis data collected by USAFOMS, and 3) the expertise of the SMEs
themselves.
Content validity is thoroughly documented for the more than 400 SKTs in the
USAF inventory. However, the documentation is predominantly qualitative
rather than quantitative in nature, as is the norm with tests based on this
strategy. Test developers at USAFOMS felt that a quantitative means of assessing
content validity would be useful to improve their feedback to test
writers and to make it possible to study test resulfs longitudinally to help
identify problem tests.
Lawshe (19751 developed just such an approach, the first to focus on content
validity in a quantitative, rather than a qualitative manner. His method
called for a panel of subject-matter experts (SMEs) to independently rate
test items using the following scale:
Is the skill (or knowledge) measured by this test question:
Essential (21,
Useful but not essential Cl>, or
- Not necessary (01,
for successful performance on the job?
He then used a test of statistical significance with the content validation
panel’s ratings as a basis for eliminating items from a test item pool.
Lawshe’s procedure has been applied by a number of investigators in a variety
of situations. Distefano, Pryer, and Craig (1980) used his content valida-
235
.