I__. - International Military Testing Association

ADDRESSING THE ISSUES OF "QUANTITATIVE OVERKILL" IN JOB ANALYSIS
Julie Rheinstein
Brian S. O'Leary
Donald E. McCauley, Jr.
U.S. Office of Personnel Management
Schmidt, Hunter and Pearlman (1981) have indicate3 that molecular job .
analyses are unnecessary in selection research involving traditional aptitude
tests. Fine-grained, detailed job analyses tend to create ttle appeararce of
large differences in jobs, whereas, in fact, the differences are of no practical
significance in selection' Our recent job analysis research has focussed 011
looking at how job analysis projects can be less detailed and less cumbersome
while still allowing one to obtain the necessary information for test
development.
O'Leary, Rheinstein and McCauley (1989, 1990) discussed several "holistic"
job-analytic approaches used in fcrming job families. Their research suggests
that the traditional fins-yrained,job-analytic approach may not 'always bt:
necessary, especially when one is in a fast reaction situation.
In the first phase of a project for tt;e development of an examination for
Federal proiussional and administrative career occupations, job families were
formed tising a procedure developed by Rosse, Borman, Campbell and Osburn (1985)
(see O'Leary, Rheinstein, and McCauley, 1990,for a detailed explanation of the
formation of job families). Once the families had been established it wiis
necessary to determine the importance of various abilities for job performance,
and which abilities to measure by a written test.
The "inferential leap,- (i.e., the inferring of htiiitan abilities important
for job performance) is traditionally performed by a panel of "subject matter'
experts." However, there is little gtiidsnce in the literature concerning the
composition of this panel of experts. As Ldndy (1988) has so ably iridicated,
incumbents are the ones n,ost famiiiar with the job itself but are ofteli
unfamiliar- with the conceptual or operational characteristics of tile abilities.
On the other- hand, job analysts (often psychologists) are familiar with tr&
characteristics of the abilities but are often not very familiar- with the job
itself.
The recent work of Butler and Harvey (1988) and Harvey (1939) showing that
different kinds of tixparts (e.g., incumbents versus supervisor-s) provide
different views of a job, arid often conflicting information, would seeni tc
suggest that one might yet different results in job-ability linkage studies
depending upon the composition of the panel of experts. We were able to address
this isstie by comparing the job-ability linkage ratings made by personnel
resewCtl psychologists to the scirne ratings made by job incumbents.
When one conducts a traditional job analysis, the question becomes how much
informAtion should be collected. Often raters are asked to rate tasks on several
scales such as importance, time spent, difficulty, or physical demands.
Weismuller, Staley and West's (1989) research indicates that ratings one scale
al-e contaminated by ratings on other scales. Anecdoctal findings from job
analysts indicates that obtaining ratings on importance, time spent, etc. i:;
unnecessary in most cases because the ratings are highly correlated across
scales.
This paper will look at several aspects of job analysis and how the
traditional, fine-grained methods may result in "quantitative overkill." We
Will present data on sever-a? techniques for- determir?iny tha importmce of
x+-...
52
---!!y