Technical Report - International Military Testing Association
Technical Report - International Military Testing Association
Technical Report - International Military Testing Association
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
UNCLASSIFIED/UNLIMITED<br />
<strong>Technical</strong><br />
<strong>Report</strong><br />
distributed by<br />
Defense <strong>Technical</strong> Information Center ’<br />
DEFENSE LOGISTICS AGENCY<br />
Cameron Station � Alexandria, Virginia 22314<br />
UNCLASSIFIED/UNLIMI<br />
0<br />
( q
,<br />
a<br />
8X<br />
I<br />
, . ‘0.: I. ! ‘, . !.‘.’<br />
c L f ,, il ! !I 5 ti 0 u 5 t<br />
. :a , , , ‘, I, !. ;‘I! ..*<br />
,>. ,- ,-, - i ,.,. , .,! :;;r,:<br />
Fort himin Harrissfl<br />
Indiasapolis, hdima<br />
---. -s-e -_---._ - -.... -. -m-s-- U-Ryl’ ,:., j i .,. -<br />
-__ - --<br />
,
. r<br />
� �<br />
. .<br />
rnl.ITAkf ies?ING ASSOCIATION OFFICERS<br />
PRZSIDENT<br />
Colonel Jemar C. Doneghey<br />
Comaending Of iicer<br />
US Amy Enlicted Evalurtioo Center<br />
PRESIDENT-ELEtX<br />
Colonel John V. Patterron, Jr.<br />
Cowending Off fcer<br />
6570th Parronnel Rerearch bboratory, US Air Force<br />
STEERING colonnm<br />
Dr. Doneld L. Warno. Chairman<br />
US Army Enlirted Evaluetion Center<br />
Colonel June@ C. Donaghey, US Army<br />
Colonel John V. Patterron, Jr., US Air Force<br />
Captain Richard ?f. Hayes, US Nwy<br />
Ceptein J. P. Martin, US Coast Guard<br />
Lieutenant Colonel C. J. Chatroon, US Amy<br />
Lfeutenent Colonel A. S. Kneuf, US Air Force<br />
Ua)or Frank L. HcLaoethan, US Air Force<br />
Litutenent Caxaendcr Prances S. Turner, US Coast Csard<br />
Major Doneld L. Diamond, US Xarine Corp.<br />
Lieutenant 8. R. William, US Coast Guard<br />
Hre.‘Mabel 0. Brunner. US Air Force<br />
Mr. C. J. Hmeluro. US Navy<br />
SIXTH ANNUAL CONFEREHCE CCFBIITTEE<br />
Mr. Frank $3. Price, Cha.irman<br />
Captein Henry H. Bahner. Vice Chairman<br />
Hlljor Herril R. Owen<br />
Hr. William W. Wance<br />
Second Lieutenant Martin S. Brown<br />
SIXTH ANNUAL CONFERENCE PRCKRAM CCWlITTEE<br />
Hr. Claude P. Rrfdger, Chairman<br />
Hr. Dale R. Baker<br />
Hr. John S. Brcnd<br />
Hr. Arthur E. Hoffman<br />
** .<br />
The opfnions exprerred in the paperr prercnted in these proceeding8<br />
are those of the authorr end ase not to be construed as being official or<br />
in my vry representative of any of the US Armed Servicer.<br />
.<br />
F I
_ - --- _<br />
.._<br />
----.---- --- -- _____ I ._--.<br />
.<br />
-. -<br />
.<br />
. _ . - -. .- - ----- -<br />
_ . _ _ -. . .<br />
- ..___ - ._.- . .: . . .-___<br />
. ---.<br />
. . ’<br />
� �<br />
. * -<br />
.<br />
. . .<br />
.<br />
.<br />
---._.<br />
. . ,<br />
.. --.-... -__ _____.<br />
2;<br />
._<br />
--<br />
.<br />
.<br />
.<br />
:<br />
.<br />
.<br />
,<br />
c<br />
.<br />
.<br />
.<br />
. ,- ;<br />
.<br />
.<br />
.<br />
:<br />
-, “.,,
:a--<br />
�<br />
.<br />
. __ ._ _ .._ _ .___. - - - --_- __.. - -. ._ _ _._--- __.__ _ - ._-_. . -<br />
. .<br />
. .<br />
.<br />
.- __-yI----- ---.-- . . __I_ -..- --.. --- -___ --.-.---...-----
. .<br />
. .<br />
. . . .-.<br />
.<br />
.<br />
C8owal Syrposiuu It Tart Conotruction Proceduree,<br />
Williaa u. Wbnca, c!-4nnem . a . . . . . . . . . . . . * . . . . .<br />
*<br />
Tut Colutructicm 88 8 Subrprtea Within � �␛����� Approach<br />
to Training Technology, Iredora J. bt- . , . . . , . . . . ,<br />
Sumaq of Pragmatic Creativity in B+tiaatfon Conotructioa,<br />
John hedifotd . . . . . . . . . .<br />
Bvaluatloa of Notor’& il;, ‘&l&*E!: &&id; . . . . . . . . . .<br />
Performance Teot Construction, tred B. Horn .-. . . . . . . . . .<br />
08narbl Symporiur 1st Xntorprotrtionr and U8u of Bvoluaticm<br />
Ruulto, )Prr. Genwieve Schulter, Chairman<br />
One Ioterpretatfcm of the Major Goale of Specialty Knowledge<br />
Tutlng La the United Statu Air Force,<br />
. Stephen �� ����� � � � � � � � � � � � � � � � � � � � � � �<br />
� U8eI of )oS Bveluation Teat ReIult8, J, B. Rohrsfter . . . . . .<br />
caner01 Symporium XIX: Job Anolysir for Teat Devolopwnt<br />
Purpour , Prank I.?. Rico Jr. Chainam . . . . . . . . . . . . . .<br />
Neu Pmrpectivar in Job Anrlp8i.8, Joreph R. Harsh . . . . . . . .<br />
)soS Ev8lurtion Teat Outline Sadnat, Curt18 D. KcBrldo . . . . .<br />
Group Dfrcurrion I: Ita, Writing Procedurer for Incremiag<br />
Validity, I. J, Newnun, Chafman . . . . . . . . . . . . . . . .<br />
Group DircuorFua 11: Non-Bnpirlcrl Velidbticm of Tart Item,<br />
J. L. Pinucans, Chaima~ . . . . . . . . . . . . . . . . . . . .<br />
Group Diacurrfon III: Te8t and Itr Revf~im Techniquer,<br />
* J. g. Partington, Chainsan . . . . . . . . . . . . . . � . . . ,<br />
Peprrr An8were t o Camon Criticimu o f Tolta,<br />
, t P~kR.RIceJr........................<br />
1<br />
Camnlttm <strong>Report</strong> I: Sturfng Comdttee . . . . . . . . . ,, � . . .<br />
Canmittm Rqmrt ZIr fca Writer8 Aptitude Tut Devolopaent<br />
onmlttu � � � � � � � � � � � � � � � � � � � � � � ��� � � �<br />
Roster of Confereer . e . . . . . . . . . . . . . . . . . u . . . . 184<br />
._~__ - . ^ ._ .-. .<br />
.<br />
.<br />
_ --‘T’--- -<br />
I<br />
e<br />
----<br />
.<br />
.<br />
.<br />
,rcc<br />
. .<br />
.’ -<br />
.<br />
.<br />
:<br />
.<br />
.<br />
121<br />
122<br />
126<br />
127<br />
132<br />
136<br />
141<br />
148<br />
149<br />
156<br />
160<br />
161<br />
171<br />
173<br />
181<br />
182<br />
P<br />
4 Y
.<br />
.<br />
.<br />
_ _- -_ _.. ..--._... -.. .-...__- --_- __ ,_ __ -<br />
.<br />
“1<br />
. I’<br />
. .<br />
__,._<br />
- F&r.. L...A.-.-.----..<br />
w-h_- _.__ _I_____._. I _.____..-me.- --_<br />
_______ _--------- --..- . .-. 5<br />
Foreword<br />
P<br />
Unharaldod at its inception Ln October 1959, lhs Xilituy Tsrting<br />
A88ocLatioo h88 dwelopul into a povorful influence in military tutlug.<br />
The purp03s of tha haadei.atiou ir to bring together annually rrpresontatiVM<br />
froo. the V8rioU8 ~m8d 88rvic08 to difKu88 and exchaqo idaas<br />
. WUCOYUtKlg job PrOfiCtOUCy WahatiOU Of enlirtad p8r8CWl81. nU8, one<br />
of the more important 8nd productive -pact8 of ‘:he annual couforanco<br />
i8 8 WOting of mind8. Confermcer h&v8 bean hold rithin 8 liberal<br />
. .A fr-k of camftter mestingr. While the liberty of ccmuitter dircu88ionr<br />
tur provided opportunity for crsrtive thinking and productive<br />
dircurrion. the rtructura ~sca888ry for exchanging &8 much informtion<br />
88 porrlblo rithin A thort ti.m wa8 lacking. h progrrn for the Sixth<br />
Annual Coafumca ~88 developed vithin uhich auimm opportunity would<br />
oxirt for the prerant~tkm of rorurch rwultr and operating lnfornutioo<br />
through tha prerentation of preprred paper8 and for the exchmgm of<br />
id-8 and crutive thinking through group dircusrion8. fh8 eonfsrsnca<br />
wa8 divided into two theoretical rad tw geueral 8ympori8. P8plrr 88<br />
prorented in the 8yPpo6ia 8nd report8 of group di8curriou8 are roprod&Iced<br />
fn thooe procauiiagr. It ir mticipat&d that there proc8ediagr<br />
of tha Sixth Annual Conference and the procoedingr of future couforencos<br />
~111 becau8 a 8ource of infornution on poychologicrl teoting in the<br />
ni1itaX-y 88tablirhment.<br />
. r > *, ‘. ,k ‘*> y,,. . .‘J 1-. i<br />
‘- The Sixth hnnu8l~bnfsre?e8j;]a8 hdld et the US Amy Enli8tad<br />
Evolurtion Center, Fort B8njtmin F~T~~OTI, Indiana. Tha c-ding<br />
officer and staff of the gnliotrd Eva?uatFoa C@nt8r -.irh to uxpreor<br />
8pprcciation to a:1 COnfOr for thoir rupport and ccanplrts cooperation.<br />
Tha 8ucce88 of thr Sixth Annual Conference wa8 dua to thr attitude and<br />
offortr of the participu\rr.<br />
iii<br />
.<br />
4<br />
,
__. . . -- .;- * . -_<br />
.<br />
_... - - - _<br />
+<br />
. .-._<br />
.<br />
. . w<br />
1) I. .-c -. - _ . . - _. ..--z--.-m - _..--._. -_- -_- -. _ .._ _..--- --.-..-w .Mrm. _<br />
I<br />
I<br />
1<br />
KCLXTARY TKSTXRC AssocIATxm<br />
Sixth Annual ~oafsrencr<br />
Colonrl Jm.rs C. DonaghcF<br />
Prwiding<br />
1
.-. c 1. - - . . __.<br />
.<br />
_ _ _ _ . -- - --__.- -_ - . _- --.- -.. .A---- - .--.. _.-___ - _--.---.-.<br />
.- - - -<br />
, .<br />
.<br />
.<br />
_ _. _ _ ._ I<br />
!<br />
3<br />
i<br />
~-RUB* ----_ _--<br />
.<br />
.<br />
~--~~----.A-‘-- .--.--.---- - - - - -<br />
I’<br />
a<br />
a - . - - - - . - - - .-<br />
!<br />
I !<br />
I<br />
. .-<br />
.<br />
\<br />
Rtnrrhr to<br />
Wilttrry Tortkg As8oclatloa Conference<br />
Ilepui~~O~~~l<br />
QMlce of Per-1 Operation6<br />
Deptubent of the Amy<br />
Tllmk m cool JJQw!%Y,<br />
I mnt to beginby exprecalng pry appreciation ror thle agporttmity<br />
Ican'tstsadberstoda;yandtdUtoyous6atastprryclhologletor<br />
expertlnpsychologlcal meaeuremEtibec6ll6eI6mIlot. IamzLnandIhera<br />
bMn for scoSx’6l par6 Mlrigned to pofdticm Where 1 hwe U66d the<br />
lxfonaetlon that you people product. In w prevlou~ ssel@mt, I have<br />
lhmedto eppmclatethe value of MIundtcrrt ma-t in h%lpfrq<br />
aDyon6 86eigmd a task and trybg t0 Eake the bO6t pO66ibh we Or<br />
per6omelzvoursesto ccmplata the task.<br />
9%~ cvalustloo of job proriLicncy orrem 6 challenge that 16 not<br />
easily overcam. bthfE&eyof eXpert6 and cadvanCedtCChtlO~~, it 16<br />
t%lf+ffcult to expect an accurak evahatlon of ld.ltidual potential<br />
accazplishcd with peper ard pencil. I 6.m rirmly cormfnced, htmmr,<br />
l&atycu,havetoalnrgedegree eccaupliehedthi6end. Just a6manag6m6ntbytheru.bdtbeth~6ibhssbeenre~cedby~<br />
with a<br />
633d8 ld.0, 80 too ha6 OYtdu6tion by fntUftiOXi been nphCad by 6ClOXltifiC<br />
eralMt10n. (3exztm, you have pa66ed the point or no zxturn ror<br />
ob;Jtctlm evalutlmof job mastcry. We in the Anuy are rlmly conW<br />
th6tOUl-Fnflstcd~~t~OXlcSyat~16 &FetO St&y. IbO~fB~that OUl'<br />
EnU6tedgv~tloaSy6temletheamo6t eclentlflcalJyedv6mxitoolwith<br />
vhichv6haoatoQtebeenablatoprwidethec~ r and hle per-<br />
6omel6tafY.<br />
Ursfoz.-tumtely,Im6t ln6ux-t thstws have notbeun able to instill<br />
in our cnnrmRndcr6 a degree of confidence in the sy&em which ym vcnild<br />
lfka to achlow. 6iovthatcoloabl~~yardhi8paoplsha~devaloped<br />
instmmentathaten~thmliable andvaluble lnmewurlngthe indivf.dud<br />
job proclrlsncy, my miesicm frcm Eeadquartere, Dcpartpnn~ or th6<br />
Amy16 tohelpthe6epeople developtb6 undemtmdlng andcoxLfldence of<br />
cur lbry c-ers. If I mkn no other point todfiy, I reel- to<br />
1<br />
.
,<br />
.<br />
M<br />
‘, .<br />
i<br />
i<br />
,-<br />
‘.<br />
.<br />
toll you th6t th. grrotert short-coalng in poychologlcal test wuraomt<br />
f&y i8 th6 16ck of undsratmding of proficlrncy 6ValMtfOft by the V6w<br />
p6OplU1 thrt muot U06 this tool. without 6 knouledga of th6 teehiqueo<br />
fnvolvsd, memagms6nt i6 fr6qu6ntly 810~ to 6ppreCiat6 th6 Cool th6t GUI<br />
b6 provldod by thlo 86rvfc6. Th6 tnliot6d Evelu6tion C6ntor h6r6 ho8<br />
taught 016 much of wh6t I knou about poychologicrl toot m6aourement with<br />
+6p6r and pencil. f don’t pr6tend to know th6 d6t6ilo--that io not w<br />
job-but I do beliov6 w6 h6V6 oaa6thing bore that m can n6v6r 6g6i!¶ do<br />
without 6nd thr largest tuk reaoining Lo to ukr those uho arot u66 thio<br />
infOrPrartiOn avare Of it0 Vo1U6.<br />
Enlioted rvoluatlon In th6 Am7 go60 b6ck long bofOr6 th6 curr6nt<br />
oyot6a. I can racall th6 Army’8 Carwr Cuid6nc6 PIan of 1940 vYIlch W6o<br />
6n 8ttempt to ~06 testing 0~0~66 to muoure prOewtfon pot6ntial md<br />
d6t6nain6 thr ultlm6te promotion, prior to World War II. Going back 6v6n<br />
prior to World War II, th6 lndlviduol br6nch68 of th6 Amy devrlop6d t6oto<br />
to 6ld thmo in the promotion of noncanioofon6d offlcrro. The � ␛���� w<br />
h8v6 in the Amy today go68 fro b6yond thoo6 iaftf61 6nduvoro. W6 ~06<br />
our 6nlioted 6r6lrution to doterain rinimum qu6lificrtfon in th6 rilitmy<br />
Op6Cidit7, to d6t6min6 th6 6w6rd Of prOfiCi6nCy popbnto for OUp6riOr<br />
quallfic&tion, 6nd u6 ~06 anlloted 6volu6tioa 48 8n fndfcatlm of potsnti61<br />
for th6 promotion of our enlioted pwoonn6l. I 6o1 our6 moot of you 6re<br />
ware of th606 rpplic6tiono for our � yotao of mliot6d 6V6lUotiOn. w6<br />
6100 u6e enlloted wolu6tioe for gr6da det6rmiMtfon for psroom6l r6vrrtlng<br />
io UI enlfotrd ot6tue. It lo not gmamlly kncml that W6 eloo<br />
Uo6 the 6UliOt6d WoluetiOU oCOr68 for 6OOi~t o618CtiOn purpOo60 at<br />
department81 16~61.<br />
In ths Enliot6d Poroonnol Dlrectoret6, ths 6ooignr.?nt of our<br />
mOt6r � ergunto, firot � �������� and 06rg68nto mojor io wAd6 ~1317<br />
after 6 couolder6tion of th6 6VolUotiOIl 8cor6. WO h6vo eotablirhed<br />
c6rt6ln minimum 6volu6tlon 8cor68 for 688igmrunt to hlgh priorIt+ and<br />
critical pooltlono. W6 hovr 166rned thr value of 6nliot6d evrlu6tion<br />
� ���� ����� ���������� ����� �������� �������� fOt ������� ������ for<br />
obvlouo ruoono, w vant uut b6rt ~60~16.<br />
Thr ~06 of our svolu6tion q8tm in th6 Amy h66 grown by leopo 6nd<br />
boundo. W6 hove r6centlp lnltiot6d 6 progrmu for 6voluating 8 roldi6r<br />
Ln okillo oth6r th6n hi8 prinmry. W6 er6 tqiug t0 68t8blioh 6 valid<br />
oklll 16~61 in wr6 th6n on6 op6ci6lty. 6n6bling uo to know fully tho<br />
pot6ntial of our emlioted p6roonnrl. W6 lik6 to f661 that thla<br />
axpuloion of our ev6luotion c6pebiliti60 will 6nabl6 us to Lee thr<br />
Who16 ooldl6r, r6ther thon juot the primary � killo of thr ooldi6r. Thuo,<br />
6v&luotim of ~rcottdarg okillr will 6n6bl6 uo to wrov6 the uoo of our<br />
omnpov6r r6oourcao. Th6 old FrOblu i8 that c6rt6in ok1118 ato n6odod<br />
in grut6r proportioa owrouo than fn th6 Unit6d St6too. If ve only<br />
2<br />
.<br />
.<br />
-A___ ._..--<br />
.<br />
__...--.---<br />
.<br />
*<br />
---<br />
.<br />
I<br />
.<br />
.<br />
.<br />
I . .<br />
. . . 2 . .<br />
’ .<br />
*.<br />
- ‘.<br />
,. ., i<br />
.I b<br />
1.
(i<br />
.<br />
* -. -._ _<br />
. .<br />
.<br />
.<br />
have one @kill identified for the roidier and tbst skill 18 one uhich<br />
lm need priorrily overrear, that roldiar doesn’t get hi8 fair ohare of<br />
tima in the Waited St8ter. Utmn’ two or more rkillr are idatiffed,<br />
more equitable dirtribution of ovormea duty ir posrible.<br />
hnothor uso of the Eelirted tvaluatiq System ie the waluation of’<br />
the Army Rererve uld Natiou81 Guard enlistrd personnel. Rvaluation of<br />
there people vi11 ‘aid ua in determining mbilitation pxentlal a8 wall<br />
&8 providing an edditiocml Incentive for masuberr of the Rcmrvr mad<br />
Mational rwrd to maintain tholr milltarp rklllo. We are extremely<br />
proud that our Enll8ted Evaluation Sy8fem has been axtandrd to our<br />
enlisted Reoerve carpoaaetr. We consider this � otep in the furtheronce<br />
of tho ona Army concept of teamwork betworn tba Army compomntr.<br />
tvaluating rrcondary aed additional rkillr and evaluating perrmoel<br />
of thr Rsaem compoearta hu b88.e made porribla by tha u8a of tho Hcil<br />
315 Sy8tm, vhich w na uaa to handle tha ta8t data here la tha Enlisted<br />
tvalwtioo Cent*r. rtre uao of � utaatlc data procraring bar enrblad UI<br />
to look fonnrd to future application8 for our enlisted ev8luation. Flora<br />
at thir iertallatfon next month, rrq-creotatfvar of the Continental Army<br />
C-d, my offica, and talioted Evaluation Center ~111 meat to dircurr<br />
the potoetlel fur providieg a training evaluation report to our ccnmmderr<br />
fras thr ccmpaay love1 to the field Amy. If the proreguirlte rkillr to<br />
8 glvm Job can bo Individually evaluated categorically, thy CM aleo<br />
bo evaluated end ourmarired bp the unit or by the’ cpaciallty.. %lo vould<br />
enable tho unit comsander to deternina area8 vhlch require mupharf8 in<br />
hir training progrm, em ~811 cd helpins u8 doviro ochool program which<br />
vi11 insure tho � daquata troinfng of pcrrroenel in their military rpecialty.<br />
We na dlvide.our evaluation of enlisted porronnrl into flvs to ten araa8<br />
appropriate to thr given specialty. It ir there areu vhtch we Lateed<br />
to analpre for -raining defic.iencfrr. It 18 anvirionad that all our<br />
canandorr vi11 be able to use the informetioe which ~8 provida. I wka<br />
this point hero becaure I want you to underotaed that we feel that the<br />
product of our enlirted evaluation rfll be ured by our opcretlom people<br />
and our training people. tnlirted Evaluation can and will rupport the<br />
vary hurt of our Army mir~ion of rudinerr to ������ � � uie lo limited<br />
or general ground combat.<br />
I vant to conclude on thir note. The contribution8 of objectivr<br />
rvalwtioa of indlviduel Job proficiency to the mi88ion of the maed<br />
rrrvicer of the United State8 are inmururable. I an convfecod, aed in<br />
fact I kwu, that more and wre of our ronior officer8 aro rocogeisieg<br />
&fly the catrlbution 0.’ proficiency evaluation to the task of tumagrmeat.<br />
A8 rout cyatrn becuner swra rophfrtlcated mnd you ruch for<br />
perfection in ths raliabiltty cmd validity of your teOC8, you Wu8t<br />
novor forgot &at vlthout the confidence of mnag8mut, your product<br />
doer little good. The ao8t valid evaluation � ecomplirhor nothing vhbe<br />
.‘<br />
. :<br />
I<br />
:“i f<br />
, .<br />
i<br />
1<br />
.<br />
I<br />
. , I i<br />
-. ---.<br />
7
it falls on the deaf eara of mt. Tberhalldngebef~youis<br />
lmm t?.iaD mm?4 illpmhg your f%al.uation, it. ia 6ou the uee of<br />
objactitre ovahlation for the many prpo6oe which at can 6ome. MO<br />
know that encb of you ViU contribute, In tb3 cow&o of this conferax,<br />
things that will help u8 llzgxxm our Imisted Eva3,uation SyrJtela. l&3<br />
hop0 that in your tima hero with the Fhluation Centor, we can offer<br />
you ~csaethl.ng to tab back to your jobs.<br />
Ithankyou.<br />
.<br />
Y
.--<br />
A-<br />
.<br />
__ .-._ -.-.. - -‘--- .- -.- :<br />
. _ ..-- - . ^-^ _ _ . . _ __.. - .._ - _. __. _ ~.<br />
.<br />
Remarks to<br />
Ifilitaq <strong>Testing</strong> and Aesociation Conference<br />
X&JOB DC&AID L. DIAZUMD<br />
Officer in Charge, Teat Sectioe<br />
US l&rim Corpe Inetitute<br />
Good afternoon Iadlee and Gentlesten. I me lookiag fonrard to<br />
having the opportunity of mingling with you gsntlaxn who are euparte<br />
in tha teeting fiald am I have very limited axpariancs and kawladge<br />
of thie aubfect. I vi11 briefly axplain the Marine Corps Teeting<br />
Syeteu by firet etatiag the laieeion of the Harine Corps Inrtitute.<br />
Our mieOioa lo to prmtids correepondonco coureau in baoic zailitary<br />
eubjacte to aelisted liarinee and :o prepare and procore sxaminatlone<br />
es directed by the Cawuedant of thr Marine Corpe. We have a third<br />
miesion whfch oftan makae It difficult to accaaplieh the first twoto<br />
provide ceranonial troop0 to the marine barracks for participation<br />
in weakly parades during the � wser month0 and other carcmumial c-ituente<br />
throughout the year such � e military fueerale, etate arrivale of<br />
foreign dignitariee, and other White Xouaa functione. To gat back to<br />
‘tha teeting bueineee,<br />
the teete we prepare and adaiaieter ure not pm-<br />
foruance evaluation teeto but are ueed to maaeure the szminee’e<br />
knwlodge of general military eubjtcte. WC prepare and � dminietor four<br />
groups of teete.<br />
The first g&p IO the Officerr’e Adminietrative Subjacte Exaeiinetioe<br />
which is aduiniotered twice a year to lieutenants and captaine. The purpose<br />
of thie sxarsinatfon ie to motivate company grade officore to fazeiliariee’<br />
thameelvee vlth variouo adminietrative type subjecte with vhich<br />
they will becoma involved throughout thelr’careere. The subject? are not<br />
tactical uor technical in naturo, but cover ouch � raae ae law and Lagal<br />
Mattere, Supply +&nagaeant, Plnanclal Haeagamant, Pereonnal Administration,<br />
Caneral Adminietratiou, and Organitation, Camand and C-d Relationehipo.<br />
The officer lo � dministared a teet ae a lieutenant and agafn’ ae a captain.<br />
He muet pare tha test or retaka it the subeaquaet yaar until ha doae pare<br />
it. To data, at laaet, the reeulte of thie teet do not reflect upon<br />
his fitnese for prcaeotiou � e the promotion board doer not eea the teet I<br />
raeulte. However, if the officer IO a repeated failure, hie comaeding<br />
$<br />
officer met rarka a<br />
c oeeeemt to thie effect in hir fitneee report. At<br />
this point, the toot reeulte will reflect upon hie fitneee for promotion<br />
as the praeotion board views all fitness reporte.<br />
The toot itema are prepared by a board of officere ealected from<br />
Haadquartere, Marine Corps. They are put into teet ����� � duinietered,<br />
� corad, md reportad upon by Xarlne Corpe Inetltute.<br />
The � acond group ie tha Central Xilitary Subject0 Teet which ie<br />
edeieietered three tine8 a year to corporalo, � ergaante, and � taff<br />
sergeants, both rrgulare and raearvee. This test ie ueed to deteneine<br />
thair eligibility for praaotion and muet be paeeed before they � ra<br />
coneiderad by the Promotion Board.<br />
.<br />
.<br />
5<br />
.<br />
,<br />
i<br />
.J<br />
.!<br />
.i
*<br />
1.2<br />
P :<br />
I ,<br />
. ’<br />
. c<br />
,<br />
. .-.<br />
. .<br />
-. .<br />
. . L<br />
-.--- ------<br />
_ _ ..-.~ _- ._ * _ ._- _.._. __- ____.__-.__ .- ____..__ - _-_-- --e-v-._<br />
In the Marine Carp8 we expect all !4arinsr, rogardloclr of their job,<br />
to be potential cozabat mariner. Thereforo, thio toet covora thoco oubject8<br />
vhich aro conridored to be ind18pen8able for wrines who may ba<br />
raquirad to perform duty ,in a ccxnbot aree. There rubjecta include Squad<br />
and Platoon Tactic8, Cuerilla Warfare, Scouting and Patrolling, NBC<br />
DOfenle, and Pir8t Aid, to name a faw.<br />
Technicnl proficiency within occupational field8 are deteminod by<br />
meena of fitnrlr report8 and proficiancy mark8 rulnaitted by the mmrino’o<br />
commanding officer on a periodic barris --but aot 1ePa than once every 6<br />
montho.<br />
Tho third group of tert8 ir the General <strong>Military</strong> Subject8 Proficiency<br />
EvalUation Tact prepared once a year and prcvided to unit C-dOr8<br />
in the field. Tboy a&inirter, score the tndt with tamplateo provld8d by<br />
MCI, and evaluate the rerultr with rt8tictical forma and inrtructionr provided<br />
by XX. Th8 pUrpO8e of thir tcot 18 to enable unit camandorr to<br />
avaluato their own Unit and 1ndividurl training progmnr urd epot the<br />
WUkIIe88e8 in variour rubject 8rOa8. The 8Ubj8Ct8 covered by thio tzot<br />
include thO8e previourly mentioned in the preceding tort plus: Clore<br />
Order Drill, Hilitary Courteey and Diocipline, Wap Reading, Comnunications,<br />
Milltarp Training, Danertic Dirturbancer, and Technique of In8tnxction.<br />
A differ8nt te8t lo prep8red for each of the three rank groupr-one<br />
for private-lance corporal, ona for corpxalr and rergeante, and one for<br />
otaff NCO8. The t88t range8 fran 1‘75 itcrrmr to 250 IteM whsru8 our<br />
othor tertr era limited to 100 itsmr.<br />
The fourth group 18 the Inopector Goneral TO8t ured by the XC te8m<br />
during their annual in8pectfon tour8 of 811 tlrrine COtp8 unit8. Th88a<br />
tcrts are ured to ev&luata the U’Ait training progrm � l8o. There te8t8<br />
� ro prepared to te8t the oame rank group8 a8 the previoU8 testr. Th8 Ic<br />
team � dmini8ter8 the te8t8, rnril~ the rerultr to MCI for Ilcocing, evaluating,<br />
end roporting rho ra8ultr back to the unit 8nd the IG teas.<br />
t!CI 18 currently prqaring rpecificationo for a computer syrtem to<br />
br inrtalled, hopefully, tow8rd the end of fircal yur 1366. Thio 8yrtan<br />
will be a boon to our curtent ta8k of evaluating corrarpondence cour8e8<br />
and the variour te8t8 which ve adminirter and Icoro. At thir point I<br />
wuld like to uprerr en appul to tho88 of you who h8ve computer inotrllationr<br />
rupporting either correrpcmdence Cour888 or your torting progrm-pluse<br />
cont8ct me at your earliert convenience end oxplain your ryrtemr<br />
end operatlonr. I wuld like to carry back your idea8 and rmthodr to<br />
incorporate into our propooed 8yrtem rpecificationr.<br />
In cloring, I an looking fonmrd to gaining a lot of valuable<br />
informtim to h8lp WI end the U8rino COrp8 in our future tortfng<br />
prograar .<br />
Thmk you:<br />
.<br />
6<br />
_.. _ _ . . . _-_<br />
. _1 -- --..<br />
.<br />
e<br />
.<br />
. . .<br />
. .<br />
.<br />
.<br />
J<br />
.
:<br />
i<br />
f<br />
I<br />
.<br />
.<br />
_.. _ ._ ” .-. -. ._ _ ---_ _-.-.. _- __._. _ c.-----_-.__- ___. - ..--.-.- -. .<br />
.<br />
.<br />
. .<br />
‘V-u--. -- - ___ ._. - -....- - ---- d- .__--.I_.._--- -- .-.... - .<br />
. .-. __-_._ -----.--<br />
.’<br />
.pai%rko to<br />
Ithasbeen auggeeteplthatazyrexuarks shouldconcerntheEXEUF6D<br />
uxlg in armtd-6ercTices examinations. SincamanyoftheMhlmmbershere<br />
todapnaybeas~~thateetingBamcasIam,lt~~proper<br />
that~lookboth~thepa~tandppneentbefo~ews~toZook<br />
to the futuzw. l!heresretwoquestlonsuhichbotholdaMnevmmbers<br />
mustta!u3tipssto considerbeforeb attempt toevsluatethe futursr<br />
1. Xaveuebulltourhcmeuftetrtlng6ub6tanti~ Isour<br />
rauudati0n alAd?<br />
2. Canw safe look bta the f’uture? Are w mare of our<br />
----?I<br />
~t~tblr? Are ve aoun in our prarent performance?<br />
!L%e snmerstoboth qumtloas becntobe positive. The pest rep.rts<br />
of our organization indicate that we have been united in our actitity.<br />
t&3 have three c~ ereas for self-pfaiec:<br />
1. We ten bmrt af our conwte attention to the vast mount<br />
or detail favolved in lndldbg summful testing witere.<br />
2, our c~acernwiththeproblcaof creatingbettertests conthluel<br />
to be urupmittlng.<br />
7<br />
‘_<br />
.<br />
.<br />
.<br />
.-
-<br />
,<br />
/ .<br />
,<br />
_ _ . _ . . . * .__.. -<br />
.<br />
m<br />
\<br />
. .<br />
.<br />
. -<br />
-<br />
__..__. _-- .-- ,-- - -.<br />
acccaupllshmint of our original alms has beccowof r&nor iapxtance to<br />
the und.reeoEd of expfmsicm that haa oecttred. In tha early fbqe, wa<br />
couldproject anicbalthntwsona ofhorizontalclai-. #s could<br />
veil lmaghm and work for impronmaats inquautycup?-oductionard<br />
tridnlng, an Increase in the rnrnber of tecte writton (euui at the tiESb,<br />
incraaecsiapor~onaeltocerryouttheloed)arrtaninmaeein~<br />
preatlge aftha indivlhal ixnters. Host of.tblshasbet~d.one--althm&<br />
it wsn% until the 1363~coderonce that WJ 8brted to mrk cm ~1~j3<br />
dellbarata~a~etomakethevalussoiourrrorklnww~.<br />
TIna has proved that the acccesplishmmt of our ndealon ha8 bean a<br />
lrtcp not 6n end. khavcincre55edladepth asrrrallaal&erally. Ve,<br />
theMIIA,havedcocloyaddbvicos,Prans,aJsdtecfmicZuasthatholdcballeDglae<br />
possibllltlee.<br />
lb havereachtdthc pointvhereve canexploretbeuut.er~cesof<br />
tQstl.ng: ua sro phyaicauy pYTqm?d. UO5X9OIlSOUdgD.d9~aUr<br />
takeoff. E?utarowpyd.10lnglc8llymsdy? Areyoureacl# Dow<br />
relax In cm tmethly nmning operations, or do vu met the challenge<br />
of what can ve ~DP pfy aoureeb regm-dlng the M.I!A'e poeoible &FIIVA fn<br />
exploxd$fi~eewtest~corrmoe are the reprtsoftbe pmwl0us cunrem-8.<br />
Lf1maytakethlsllberty,itwmulc? seemrraa them xvports<br />
that In ears aretm theHl!Ah.aabeentoo con-tith -ata Curicons<br />
to recofi~ Ga that the space beymd thea Is real~&%&i%<br />
UnllmLted." At the lWmlExminlngCe&erwe b.ave reachedthe poLt<br />
Mm-13 the order&q of emunlmtionsbytbe Fleet, apcessthathithmto<br />
took 742 man yea-8 per exmlnatloapcriod, cmnowbe acccmpllahed<br />
automatically. ~1ongorbwoeachunft~~tospcndthatiEsIl~tinS<br />
ztect?searyexeuldnatlons. !rha IEW autaaation nov coi7rplQtQs tho onttre<br />
mund o.? the examlnatlon process, In rddltiun, we have reac.hQd tha<br />
point whem itidividual =tura~ 5n3 6tia8diccilly r0m-c-l-<br />
PAKt system 80 thatallpersomelrecorda iwe ~~~@.~telyUg-to-clnti<br />
d.lwst 68 Bean 58 the individual 8COm 16 knUW!L %!hib affords m<br />
0fYicicnt distribut2oaofenllfstedperm.melthr~the~.<br />
In pas-t M?A zncotiags, acc~to the recoxYla* w ham been concerned<br />
tith the "nuts and bolt&" OdLy In 1963, vlzaa wo att-apted to<br />
glintbe recogxlitlonorDm,didwl dupartrrcmpurQlyut1lltarlan<br />
8834Xtt3. Wo have gdaed great twhnics3. itduty. RJW M mu8t CODcentrate<br />
on valulcr uf tmtlng, We rru3t roach out ml oxplor8 new<br />
.-. _-._ .______ . - - ---._-..--_.---_P_--__-<br />
.<br />
. . . .<br />
. . .<br />
8<br />
I<br />
.<br />
__ . _-. .-. . -. . -<br />
.<br />
. .<br />
f<br />
1I<br />
1<br />
f<br />
.<br />
Y
-.<br />
/<br />
,’<br />
i<br />
_. . . _. __ _. __. .._ _ ___ -. __- .-- -_ - ._ __-_ . _-.-___ -___ .__-..<br />
,<br />
. .<br />
. . . . . . --..-_ _ _._<br />
.<br />
. _ .__- .._. --__--. -- - -_---. . - . i ._..... -<br />
_. _--- _--. ------ .-<br />
.<br />
\r<br />
*<br />
Perhaps, BI a group we have teMod to remln Btatlc. Ia this<br />
becaueevofeelthataurinterastinaduc8tloDdhprsgressls t3ub60rrri0at<br />
tommber&lpinourtterrlaet Allafourllttaeaxwdevotedto~asas<br />
sspact of ta. )ie 6houUtharcfarercdiza that education ie one<br />
0fth8~6tp0wtr0~t3int.bk3~23f30f~.<br />
Slnc8 thetop~llftlce in the ileldaftwtlngam gathsmdbere,<br />
l!Betln&oxlacarmaDa gzpnmd ai f.xrterest, ham!t WeI tha ctblc8.l right to<br />
rtmoln etatic--tohats ouroretlrg6mmdy~nicelyorganized commutlan?<br />
1 umpasltlvem allbava a etrongtdth lnthu lnUvldualpotmrtlLl<br />
of the HEA. WebelLeve Rtroaglythatlt canbetcsmths (zEnAmmm<br />
IXTESTIRIwer&mm. Wsbulient&tveereprqxmxlardab&to<br />
arp- unknovnrields. Wsbc13evathst0u.rarp10r8tlsnati~bs mmad.<br />
I& us xmf, at thlr conPcxwlce, use aar CaUctlvd craatira lnBgiMt1on<br />
to aseart omwelwa<br />
,<br />
, . .<br />
.<br />
I<br />
. .<br />
I<br />
i<br />
4 ; i
* .<br />
_ . . _.-* --. .___ . _ _-. .- - .-.-<br />
.<br />
.c -<br />
.<br />
R8mA.B to<br />
KIlIta.ry Tertln~ Asaocintlon Confetanci<br />
LiQUtMQit CO1OCi81 8. 8. &UDf<br />
f o r<br />
COLQSSL JoBlD V. PXRZRSoB1, JR.<br />
ccQal8xldIn~ Off Icer<br />
657Otb Peroomrl ~srurcb laboortory<br />
us Air Ports<br />
Tuo Itcu of probable I~torart to the eoafrrear of tbr Wlltory<br />
Tutlng AssocI8tIw concwnIlq the Air torte SpecIalt7 KnwlQdge T*rtI!lg<br />
progmm are:<br />
(1) The clorar relatIon8bip developing betmm Air Force<br />
trainfn~ ahd Specialty Knowledge TutFag, ukd<br />
(i) ’ Rramt developmnt In tmhnlquoo of job Qnal78Ir.<br />
Although both of there, tolpiC8 wIl1 be the 8ubjQetr of tiditidual<br />
pruantatiour during tka cesfareaco, ‘L would 1IkQ to-brtefly Iatroduca<br />
then.<br />
Apptcolhatoly a ya8r aa0 tha Air Force 8dopted what 18 CilllQd (c<br />
&Sal-chmxml approach to on-the-job treinirq. tfadar tbir concept en<br />
airn~n ir prwided with 8 rorlas of Cue&r lMvQlcqsen?mt C~U~~QS ubich<br />
(l;r@ i.ntsndsd to cwer thQ broad fUd8WXltal8, theory, :md principles<br />
nhlcb ho Is raqoirQd to kaw to progrors or bo upgrQd@d ia hi8 QpaeIalty.<br />
‘LbO8a c0urs.u are dQvQ1op.d by Air Troininf Comaamd t8ChZliCd ldtQS8<br />
and prfntod and dI8trIbuted by thQ ltptenslon Cwrres Inrtltutr o f thr<br />
Air thlvorrity. Tb~y are Intenbad to be rQlf-study type cmr8e8 ubIcb<br />
the 8iSmM COLBplQtU 8 8 OZL ordinary COKSe8pond8W~ COUtQB.<br />
ibe other channel of tho &Ql-churorl CO6CQpt I8 BCtrQ COnCQtnCd<br />
with the 8wifiC tS8iniJl~ required to 8ccaPgli8h tb0 QpMIfIc doti88<br />
Of b18 iW8diAt8 job. Tbls � rpect Ir wr8 concwrd rith tkr parformaxe<br />
on OpQclfIc UQxrw 8rMs of thQ rpecialty or thr uoiqar equipwnt<br />
of hi8 pQrticulQr ualt. In this PhoBQ he � pp1b8 thQ fundaa8ut8~r and<br />
throry 14axmd through tha Curer Mtrlmt Courrer to hi8 QpacIfIc<br />
unit rlrrim. ‘2hIC pb~8Q I8 8up~~is8d Qnd Qccoxplfrhed priMrIly by<br />
the InditFduQl casu.ndo, rfn(es or oven dam to equadroa hvrlr, .<br />
m8 under Our pt8SQnt cwc’rpt end method of devrbpa8nt me rQ1nt.d<br />
directly to thr C8rQQr D8valopwnt CCRirQOQ. They 8~rtQ priMrIly M<br />
and-of-cctrra �⌧���������� � dmInIrtar@d under coatrollrd conditiona. A<br />
rQ8QQrch progrm Is prQmatly uQdQrwQp to QvQluQte thQ SKTo drtrlop~d<br />
to pQrQ3lrl thQ Cuoer Devolopent Cesrra8.<br />
10<br />
il<br />
.
.<br />
__ .<br />
____ . . .- ---__.._ -<br />
__.___ ___ __----.-<br />
.<br />
.<br />
-.- --- -<br />
l%r Air Force l8 ozl the vuga of opar.tia& ufoptfa$ a raw<br />
tecbique of andysing jobe drfch we are hopeful wit1 pravida more<br />
def?lftive fsfozzaatlon on wbicb to baaa both tr&fniag md mote. UadQr<br />
this @yetam task invcmtorler are prepared gad informatiou gathued frca<br />
fob incwhbmtr coucuuing mount of ttsw, hportanc*, etc., relative to<br />
each opocifie tuk. Through the use of high-apeed ceqmtero aad grumplrrg<br />
trchiqaaa, highly useful Lafornatiaa can be obtainad to aorirt in<br />
more occuratrly daaeribing vuiou8 jobr, to plan training curricula,<br />
aud to urite better tear. -<br />
If sithu of them tuo rrccmt devalopentr-tha clorer rolatiourhlp<br />
of trainiug md testing and tba edro~cor ia job malyria--achaally<br />
lln up to mar upectatiom, �� � �� rO8t optimfctic a8 to the future<br />
of the *trman terting progrrn in tha Air Force.
,<br />
I<br />
.<br />
,*<br />
.<br />
. -.<br />
- - . . ___<br />
.<br />
. . -<br />
.- .-_<br />
.<br />
�<br />
Raaarkn to<br />
Wlitary Teotihg Aooociatfon Conferanco<br />
CAPTAIN J. P. MARTXN<br />
CoexamdFng Officer<br />
US Coaot Oua:d Training Center<br />
AD you know, leot year’8 conference we6 held at the Coast Guard<br />
Training Station at Groton, Connecticut. It was hooted by the Coaot<br />
Guard Inotttute, undrr the c-d of Captain Kurcheohf, who woo chairman<br />
of that conference.<br />
Since then, a number of changea have taken place in our orgnnirutioa.<br />
Captain Kurchukl hu retired; tho Inltituto lo no longer an independent<br />
c-d but ha8 been ccabined with the training rtatfoo to form what io<br />
nou knoun a8 the Coaot Guard Training Center; uumorouo changea in parronnel<br />
havo occured; Llrutanent Camander Hallock, for example, hoe bera<br />
rruoigned � o Ccman ding Officer of a cutter out of FJeu Bedford. In<br />
fact, of thoom uho attended lart year’8 confareaca, Liwtanaut Williir<br />
the only return-.<br />
For the raot of uo, we are attending an Wu Meeting for the firot<br />
timI. But I have had good reportr on laot year’8 mting. By all<br />
account8 it WI a plearant and informotfve confsrente. So I have reawn<br />
to look forward to ‘get.ting bettor acquafnted with you. I (m our0 we<br />
will find many � ��� of c-n interect. We may find that we have<br />
� imilar problems which thir confw-ante mfght hprlp uo oolve.<br />
So on behalf of myoelf, the moaboro of my otaff who ara hare and<br />
officer8 reprorenting other Coaot Guard Unite, we are glad to be here,<br />
to joia in the dlocuooion , and to have a chance to develop ruu inoighto<br />
in tha field of milltaxy teoting.<br />
A year ago 1 wao oboard the Coaot Guard Cutter Northviad, conducting<br />
oceanographic � urveyo off the coaot of Horthero Siberia, and I can � ooure<br />
you that my oula concern8 at the tbaa wore far rroved from the field of<br />
tuting. If � meone had mentioned a “correlatfaml cluoter,” I night<br />
have thought it had o-thing to do with the Aurora Boreallo. And if �<br />
boatewaln’a mate flrrt claoo had came to ma and aoked, “Cap’s, how ccsme<br />
I didn’t do no better than I done on the Chief’r Exm?” it would never<br />
occurred to ma to inform hla that ha ��� � uffering fra a “regreooion<br />
� quatfon” and had wound up un the wrong old. of “an unrrlectad o~c~ooo<br />
ratio.”<br />
Throughout � rorvico career, of course, wa are apooed to a wide<br />
variety of teotlng; techalquao- but uwally in the role of victim. So<br />
it is a nova1 and intriguing experience to otmd “on the inride” at<br />
lart-to mope within � charmed circle whore one hoar0 mention of “*ufltlvariate<br />
mothodo” aad “attitudinal framer of raferenca.”<br />
.<br />
. .<br />
,<br />
12<br />
. ._____ - _-_.- -. -.-. ---.-----. ..----. .--____ _ ..__<br />
.<br />
.<br />
. .<br />
.<br />
.<br />
1 .<br />
_-u--w- .___<br />
.<br />
.<br />
V<br />
.<br />
.<br />
I
.,<br />
� �<br />
. 3<br />
.<br />
i’ i<br />
i I<br />
,<br />
‘<br />
.’<br />
But ot my rate, at tba Cacut Cuerd Ttaiuing Canter I have 6 gaod<br />
crev of mprrtr who havr bean putting out sang thouaanda of erxuoinationa<br />
over the yeara--and I aa sure they have has; accoepliehing their purporo<br />
@it8 Wdl.<br />
What 18 our pwpors ia conducting czaminatfonr In the Coaet Guardt<br />
!<br />
1<br />
/<br />
j<br />
/-<br />
8<br />
.<br />
Well, before m CM an-r that, it lo necearay to explain that wa hew<br />
ravoral dtffarsnt kinds of auminatim progrrrais. And thin calls for a<br />
furthar word on tha reorgrniutioa � t the Training Center.<br />
What hu hawned la tbio: Wo Coast Guard Training Station md<br />
the Coast Guard Institute--which ware two Bepareto ccmun da-ham bean<br />
eooblned-a8 I 8ald srrller-to form tha Coast Curd Training Center.<br />
I<br />
1<br />
I<br />
i<br />
,<br />
I lm urigmd a8 C-ding Of fleer of the Training Cmter. A s 8ucb.<br />
I report directly, to the Ccmtanbt of tba Cout Gumd in Warhingtoa.<br />
‘1 \<br />
.<br />
8<br />
.<br />
Tha Coast Guard Training Center porforma tbrea priaary mirrionr.<br />
Thr Inatituta baa fu?-nirhed tvo of tha primaq Th8 n.taaion diVi8ioa8:<br />
Io8titute Division for Correrpoadcoeo Couroea and the Buairmtioo Divi8ion<br />
for Servica-Wide Examimtion Progrw.<br />
The third prisury misaim divlalcm of the Training Cantar is the<br />
Romfdmt Training Divirlon. This division contiauod to perfore the<br />
dutier of what -;~a formerly called tha Training Station.<br />
Ihe Bcrident Trafniag Dfvfrion camprim & mmber o f technic4<br />
8chool8 which afford trn1nFng to a COntk’tUOU8 flow of atudsnta Who cOQO<br />
in from 811 over thr Coast Guard. At ray one time the atudmt body will<br />
aumbrr about 700 ma. ~hep learn hou to care for aida to nrvlgatiou<br />
equipnat, rcmging frm unlighted river bouyr to rm’s@ lighthourw.<br />
Thay praparo to operota Loran Stationr with 13O%foot tranolnitting towera<br />
rhlch � ra atrung fraa the far reach.8 of the South Pacific to icebound<br />
prmontorier north of the Arctf: Circle. They lcara to baka bread, to<br />
road the dft8 and doha of %x83 Coda, and they acquire & great variety<br />
Of other 8kill8.<br />
The Inrtitute Dlviaion vaa 80 nmwd in order to retain the term<br />
“In8tituta” which haa had an honorable hiatory datfng back to 1929.<br />
Over thr yurr it h8a brctano knovn to hundreda of thouasada of Coart<br />
Cuudrmen asrking advracment with tha aid of corrrapondmco cuuram.<br />
for<br />
man<br />
Th. In8titutO DiVi8lOII pr8plre8, priotl, and 8dlBfni8tcr8 cOur8.8<br />
nrrrly all cO88t Guard 8nlir-d r8ting8: frm aviation � lectronicrto<br />
yaunn.<br />
*<br />
.<br />
,<br />
.<br />
.<br />
‘.<br />
* *,<br />
\ :<br />
/* :<br />
:<br />
t<br />
,<br />
I<br />
I<br />
i -<br />
I<br />
.<br />
I<br />
,
--<br />
.<br />
. .<br />
.-I)-.-. _ _ _ .__- __ _..<br />
_- --<br />
At tha praoent t&ua � hundred and four coutaea ara ia the fiald.<br />
Ap addltional twenty couroeo are being developed. Naarly all ccw~oeo<br />
am undergoing conotant rwioicn. Currently, the enrollment overagea<br />
about 13,000 � tudento.ac any ene time. The average lesson oubo~looion<br />
rata IO .vory cl006 to ona per month par cnrollea.<br />
The preoent Director of the fnotitute Diviolon io C-nd8r Dohlbp.<br />
tilLa Ioyoalf, c- der Dahlby hao recently caspleted a tour of oaa-duty<br />
md he tello me that they didn’t talk much about “correlatlonol clustar8”<br />
on hio ship either. Having some fraa B coosand of I bouy &wider. tha<br />
Cutter Salvia, ho oeosao to think thot “Pcotingrr’o Notion of Cognitive<br />
Dlooonance” hao � uoethlng to do with fog � lgnalo. So I would � ay that<br />
it is juot � o well thot wa both are here.<br />
Both the Reofdant Training Divioion end the Inoeftuta Divlolan uoe<br />
wny quroriono ao laooon aide. Ttmy conduct claooroom and end-of-courao<br />
�⌧����������� � Thora exemo ladlcato tha degree of ouccaoo in completing<br />
a limited couroo of � tudy. They are traditional teaching devices.<br />
But the divioion of the Training Cantor which I would think lo moot<br />
ccmcbmod with tho aganda of thio conference lo the Examination Dlvioion.<br />
Thio divioion io under tha Dfrection’of Couxaandar Turner, yet anOth6r<br />
rljCent arrival. Mloo Turnor’o Diviofon lo primarily engrgsd in the oort<br />
of tooting that wo are here to talk about.<br />
the Examination Divirion conduct8 � rrvica-vide (that IP, Coaot<br />
Guard-wide) oxominationo of milttory poraonnol. Their usage io go>->rnod<br />
to a lorgo Bxtant by th8 manpwor noodo of the Regular Cooot O&ard to<br />
prrfotm ito poaco iaa mloriono and by the anticipated noodo of a grutly<br />
apondod Coaot Guard in tho wont of cobiliratlon. Peacetime miooioar<br />
rmain fairly c’onotant, 80, accept for updating mdtorial to take account<br />
of technological changoo, tha regular -inatIon program tendo to r-in<br />
fairly � toblo.<br />
Bocouoe war plan8 8ra rwioed, uponded, or occaoioaally cut back<br />
to accord rlth internation � graemento. aetloaal policiao, and changes in<br />
military concopto, our tralnlng and examination progrew-for tha<br />
roewver-utuot romoin flaxiblo.<br />
Bocaure of thio apparent dioparity in roquiromento botwoon Regular8<br />
and Raowvoo-and also bocauos of osparata � dm1nLotrative hiotoriro--us<br />
hava, in tho Coaot Guard, developed two diotinct axominatlun programo.<br />
I � hould like to uke a brief camparioon of the tvo.<br />
14<br />
. . . . - .<br />
. .<br />
,.
F<br />
‘.<br />
r2r<br />
. *<br />
. .<br />
�<br />
, ,<br />
! ,.<br />
.’<br />
\ .<br />
.<br />
-_’<br />
.<br />
.<br />
-- -.- -_._<br />
.<br />
,<br />
. -- - _ _..”<br />
..-..-_ ^. _-_..--- _ ” . _ _ . .__ . .._.<br />
Requitemnte<br />
�������� � ra rsquireC to taka rervice-wide axmtinatioue for odveacsm&nt<br />
to the thrao chief ptLy officer gredU --E-7, 8, And g--And for the<br />
firet wmrruat grade,. W-l.<br />
Ra8erven. an the other hand, are required to tAke ~xamLnacioa8 for<br />
advancewtnt to all petty officer grede8. In other word., they ere required<br />
for 3d class, 2GlAec, let clAee md chief--tbAt it pay grader E-4 through<br />
E-7. The Rertrvee do not have E-B end E-9 grade%. Easslinacisnr for<br />
Reserve mdvsncmtr are separate And dietinct fraa the exAminAtiona<br />
� daiaietered to the Regulari. A Raeerve applicAnt for WArrAnt grade,<br />
U-l, hovever, tdcer the sama ex8minAtlon a8 doer � Regular opplicAnt.<br />
hi8 difference in re?uira\lente for exmmlnetione would eppeer to<br />
t&8 into Account the fact that the Rseerve’e path of AdvAncemnt loAdo<br />
throua study couroee, veekly inetructions, and service-vida axunineefon.<br />
By contrut, the RsgulAr’r prospect8 for prmotion have e lerger elemcut<br />
of practical experience, daily performme of duty, And pcraozul evAlu-<br />
, ation by hle caimmding officer.<br />
Both Regular end Reserve are required to cmplete M appropriate<br />
corrrepcmdrnce coureo before bQing conridered for sdvmcentne. The<br />
d;ffarence enter8 tiei tha Resew8 aAplring to ps~ gr8der E-4, E-5, 8nd<br />
E-6 met not only rucceeefully complete an end-of-course oxem, but must<br />
aleo canpete in a service-wide mu&nation. life end-of-coures exam fe<br />
cdainirtered by the Inetitute DLvieion; him rrrvice-vida -inAtion is<br />
� dainietared by the Examination divirion.<br />
Prequancy of Examirution6<br />
For the Regular: Half of the CPO rpccisltfar Are exesized in even<br />
ycure. Thr other helf � re mined in odd yurr. WArrant officer exam8<br />
are &ministered during odd yeer8 only.<br />
lot the Reserve: drrainatione in all enlirted rpecioltier Are given<br />
twice each yur. Inrofar 88 posrfble, there -inAeiOn6 are bared on<br />
Hovy Exmliartion8. Rxminatfone for Advancement to warrant (U-1) Adnfn-<br />
Laterad once each year, we the 8eme a8 for rtgulArs.<br />
Prepsration of Examination6<br />
ExAminAtions for the regulAr program are caapltttly revised each<br />
time they us givm. hey conrirt of new itans furnished by itca-writer<br />
epeciallste brought to the Training Center on teeporAry AdditionAl duty<br />
prior to each group of txeminAtion8. Thair offering8 Are caabfaad vfth<br />
itar eelecteC from M item burk. Cur bank of quertione nw contAia8<br />
approximately 32,OQO ieaar of carefully guarded krowladge.<br />
,<br />
.<br />
,<br />
.<br />
I’
\<br />
i<br />
i(<br />
.f<br />
I<br />
.<br />
As for the Reserve examlnatim program: Xn 1953 Bupers authorised<br />
the Coast Guard to use Navy examinations in our Reserve Progress. These<br />
were administered frua Coast Guard Headquarters until 1962. At that time<br />
administration was transferred to the Coast Guard Institute. The Navy<br />
exama are adapted to Coast Guard use insofar as possibke. Sots8 Navy<br />
exams are not adaptable to Coaat Guard use--notably those for storakeeper<br />
and ye- since our qualifications for these ratings differ from those<br />
for Navy personnel. In addition, we have certain mergency service<br />
ratings such as Coastal yorcarun and Dangerous Carganan which not only<br />
have peculiar names but are peculiar to Coast Guard mobilization requirements.<br />
For these we must prepare our own exam1natlons.<br />
Finally, Scoring and Evaluation Procedures Differ:<br />
Exams taken by regular applicants are scored manually at the Coast<br />
Guard Training Center. Item analysis is also done manually. These exams<br />
have not been adapted for scoring snd analysis by data processing equipment.<br />
Exams aclministered to the Reserves, on the other hand, have been<br />
adapted to the Navy data processing equiprent at the Naval Emaining<br />
Center at Great Lakes. Ue obtain scores, item analysis, and exam analysis<br />
by means of twice yearly visf ts to Great Lakes. There Is, of courue,<br />
greater volume of Reserve exadning than of Regul8r.<br />
This then is the outline and ssope of our esminatlon progrms au<br />
conducted by the Exsmination Division.<br />
On basic purpose in giving exmainations IS to obtain an objectiva<br />
measurement of military and professional tmowledge to use as one of the<br />
factors In assisting in establishing an order of eligibility for promotion.<br />
By canbining the results of comprehensive examinations with other evaluation<br />
factors, we hope to find the persons who are best qualified for<br />
promotion.<br />
Service-vide examinations also give us a clue as to the overull<br />
effectlvenass of all our service training programs. As we standardize<br />
our methods and procedures ve hope to be able to ccunpare our training<br />
achievements with those of other services.<br />
I think ve can do wre to standardite our requlremzntu for both<br />
Regulars and Reoervas. Only by crsatlng a parity betveen Reserve<br />
rrquirements and Regular requirements will ve be able to judge the<br />
relative veakncsses and otrangtho in our tvo kinda of training.<br />
The diffarances aistfng between Regular and Reserve requirements<br />
,for examinations are currently under scrutiny by tha Coast Guard. plans<br />
.<br />
+,<br />
c<br />
‘ .<br />
*.<br />
.<br />
.<br />
.-.-._ - _<br />
.<br />
I
-<br />
;<br />
,<br />
. -_<br />
. .<br />
. .<br />
.’<br />
,<br />
i:<br />
‘.<br />
:<br />
‘. :<br />
. .<br />
. .<br />
.<br />
.<br />
.<br />
.<br />
ate afoot to bring the two examination programs into closer altgnnent.<br />
It is hoped that through the reorganization vhich is still taking shape<br />
at the Training Center, we will be able to streamline ths execution of<br />
our prarsnt callnitnents. And then we vi11 be able to assume an expanded<br />
� �������<br />
The big questions remain--and probably always vill.<br />
Bow vell do we measure what we are trying to mcaaure? What do ve<br />
wind up meaauting after all?<br />
Are present techniques actually reletting for us the kind of men<br />
we vaTin positions of trust and leadership?<br />
Can ve ever hope to accomplish thin goal with “objective” testing7<br />
If not, to vhat ext@nt are v6 “6tNCtUring” our group and our<br />
rociety with an Inadequate tool?<br />
Perhaps if ve could find the kind of person ve are looking for ve<br />
wouldn’t have to bo EO concerned with “pro-pay,” “fringe beneffts,”<br />
and “morale.”<br />
So once again, we of the Coaat Guard are glad to be here.<br />
We shall look forvard to beneficial changes in our “attitudinal<br />
framer of reference”.<br />
We hope that our “regression equation” canes down.<br />
(And I cartaialy hope that<br />
Thank YOU.<br />
.<br />
17<br />
ie a god thing to hope for!)<br />
i
.<br />
i<br />
. . . . .<br />
’ .<br />
.<br />
! --_ _ .<br />
I 1 . --- - . .._. _.___ __ ._ _- ____ _..._--_ - ______.. _ -.___ _-__.._ - _. -.. ..-__---.----.- _<br />
.’ ,<br />
.:.<br />
._<br />
. .<br />
.<br />
,a’<br />
i<br />
i<br />
lkmirk8 to<br />
<strong>Military</strong> <strong>Testing</strong> A88oclation Conference<br />
C- JAMES C. DONAGDEY<br />
Can6undlng Off tcer<br />
US Army Enlinted Evaluation Center<br />
It io with pleasure and a 6en6e of pride, 88 the C-d:Lng Off icar b<br />
of the US Army EnlFated Evaluation Center, that I have thir opportunity<br />
to present to you the asresrrment of the Army’8 syrteaa of maanrring anlirted<br />
proficiency. As you are the experts, I 0111 leave the technical otpOCt8<br />
of job proficiency testing and evaluation in your able hands and X will<br />
u8e the opportunity of thi6 brief prarentatioa to:<br />
Di8cu88 the progrsns of our Enlirted Evaluation Syotae, with p6rti-<br />
CUlar UaphaPir on the nev prOgrsln8 that have tripled the 8COpe of operation8<br />
of the Enlirted Evaluation Center since the lart <strong>Military</strong> feoting<br />
A88ociation Conference;<br />
Pre6ant rigntficant changer -da in operating procedures hara at’<br />
the Center and a new method of documenting and reporting tndtividual<br />
proficiency; and<br />
To point out the team effort that ir required for the davelopmant<br />
of avalu6tfon material8 to rupport the objective8 of per8oWel wemant<br />
arli the importance of 6ttaining there ObjeCtiVe8.<br />
The queltty coldier ha6 been sought by our Army 8ince its earliert<br />
inception; although mo6t American pioneer8 Vera proficient fn the u6e of<br />
the musket, it VP8 not 866~ to find proficient soldie- even ia there<br />
&y8 when a limited knowledge of technlcal material was required. The<br />
quart for quality m6npwar has continued down through the yearo. Wa all<br />
certainly would agree that the way to proficiency h&r to be the quality<br />
men -- the aright t man t hfor e the rright i gjobh t t i m e .<br />
A6 Commander of the Enlirted Evaluation Center, my raiseion i8 to<br />
develop and monitor technique8 for evaluating the occupational proficiency<br />
of enlisted perronnel in the Army -- to maiutain a field<br />
administrative syetan for operation of the program -- and to render<br />
ccntralited 8COring and reporting of the rerults of the terto.<br />
Fran obrcurity to the vorld’6 second lergent testing center in 8iX<br />
rhort year8 16 the outstanding achievement of the US Army Enlisted<br />
Evaluation Center. Since March 1958, the Center ha8 developed from a<br />
aina+aaa operation, located in a “borrowed” ClaS6 ro~ll, to more than<br />
200 military and civilian parronnel (not Including the field te6t control<br />
officer8 throughout the vorld).<br />
_ _..-_ . - . . -<br />
F I’ .<br />
.<br />
1 ,<br />
: * .<br />
1<br />
�<br />
18<br />
. .<br />
c<br />
:<br />
:<br />
-._-__-_- -.<br />
.<br />
.<br />
.<br />
���� � �<br />
�� ��<br />
.
‘.<br />
<br />
.<br />
The initial objective of the Enlisted Evaluation Center in 1358 vas<br />
to evalua: s enlisted personnil ln their primary MOS for award- of proficiency<br />
; s;.<br />
The succtsoful integration of this program into the Army’s systaz of<br />
personnel managcmcnt prompted adoption of additional programs that could<br />
be administered and controlled through the enlisted evaluation systezz.<br />
e Cur system now supporte--in addition to the proficiency pay program-the<br />
following functions of enlisted managcmcnt:<br />
.<br />
h<br />
Primary MIS Qua~ificntion.<br />
Qualification for Promotion. Although this is used on a pcnnissivc baais<br />
within the Army, the system dots furnioh the coumundcr an indication of<br />
the soldier who is best qualified to fill a position of responsibility at<br />
a higher gradc.<br />
Secondary and Additional MS Qualification. This function provides commanders<br />
and Dtpartmtnt of the Army the information required to tfftct<br />
broader utilization of the soldier by conridcring areas other than primary<br />
job in which he is qualified. It allows use of tht vholt man instead of<br />
part of him.<br />
Reserve Cocnpontnt Evaluation. This ncv program is aimed towards the tvaluation<br />
of our citizen soldier, the Army Reserve and the Army National Guard,<br />
on the same standard as the Activa Army and vi11 assure the use of the<br />
ssme means of measuring results toyards tha aamt goal--a better soldier<br />
throughout the Army structure. Thie is one more important otcp towards<br />
the attainment of a one Army concept.<br />
The Enlisted Evaluation System has answered the Army’8 need for an<br />
objective system of individual evaluation of enlisted personnel co support<br />
there progrsms.<br />
Prom a standing start of about 17,000 evaluations in 64 Army jobs in<br />
1959, the Ctnttr vi11 evaluate almost one mfll1on personnel during fiocal<br />
year 1965. Thfs will require publication of approzimataly 300 ttst aids<br />
and the development of over 1,000 evaluation tests.<br />
Our tests provide the commanding officer with the information with<br />
which to asssss the capabilities of the Individual roldicr assigntd to<br />
his unit.<br />
For example, a tank commander knows that the tanks aszigned to his<br />
unit have a ctrtain maximum speed, destruction capability, and crufsing<br />
rcngt . Likcwiet, an artillery battery commander knows that the guns<br />
with which he is equipped have prescribed muzzle velocities, ranges, end<br />
rate8 of fire. Not only must he know these specifications of his material<br />
but he must know the particulars of tht jobs employed in his unit to<br />
19<br />
,<br />
,<br />
.’
I’<br />
I’<br />
.<br />
-.<br />
_--. .- _<br />
. _ . ..__. ?..-. ..___<br />
.<br />
8ccowplieh unit lgiaaiona and the capabilitie8 of his m to perform<br />
these joba.<br />
The eoliated avaluhtion ayetesn provides the cocmendoro with 8 more<br />
factual basis for umting personnel mamgement decisions.<br />
The impact of the ~rmy’a tmlirted svalu,rtion eyotdlrn on management<br />
can be exprsoaed in three aroaa: Pirat, it afford8 enlletocl peruonnal<br />
motivat1oa for greater job m88tery, maintenance of currency in job proficiency,<br />
and la a morale factor oiace peroonnel d8ciaiona c:an bs baaed<br />
on objectively appraised writ. Second, it furniehoa tho unit comundar<br />
with 8n objoctivo muaure for 8ppraioing individual cunpetonco, for<br />
daterP1ning thoao poraonnel beat qualified for prcmotion, � �� for uoo<br />
la araigment 8nd utlliution of his peraonnol 8od highlighta tequitomento<br />
for individual 8nd unit training. Third, Depettment of the Army<br />
ia provided vith effective me8auroa fot implamonting personnel m8nagowont<br />
programs and 8pplying uniform � tand8rda in peraonnel rmnogammnt.<br />
Direct’aupport of theao paraqmel wnagownt functions la provided<br />
by the Ealiatod Evaluation Center.<br />
Since tho last conference, the Centet has achiovod 8 aign1ficent<br />
� cv8ncaneot in the method of documanting and reporting individual proficiency.<br />
Inotollation of tho Center’s cozuputet � yateta provided the<br />
capability for instituting a total redesigned evaluation report. Pl888Q<br />
tefet to the copy of the boos Evaluation Data <strong>Report</strong> which ha8 baon<br />
included in your brochure. Prir.: to canputar epplication, individual<br />
proficfency was reported as a single numeric rvaS.uatian 8core ma indicated<br />
in the upper right hand coroer of your sample report. Nw indivfdual<br />
reports of HO-S evalu8tion not only ahw tho axamineo’a attained evalu-<br />
8tion score, but 8180 his atrongtha and wcakneoaea in the functional<br />
8raa& of his occupational apocislty as shown in the lower portion of<br />
your amp10 report. Your aauple ahwa tho seven subject 8re88 on tho<br />
Evalu8tion Data <strong>Report</strong> for an infez:: y senior � erge8ot. XOS 115.9. A<br />
det8iled derctiption of those areas is given in the Evaluation Teat Aid<br />
for thia HOS. Sgt Baacomb lo Very Law, Lov, or Typical in a11 are88 of<br />
his 190s. lie and his unit would benefit from his study in 811 theao<br />
� ���� �<br />
Copier of this report are forvmded to tho examinee’a unit of<br />
8aa1gnmcnt and the individual concerned. Special added di8tt1bution<br />
la rude to DA, where results are used to determine various personnel<br />
� ctionr �<br />
With this Evaluation Data <strong>Report</strong>, corman dare can identify nacearary<br />
training requirements, on-the-job, and thoao � ubjoct-xaatter are88 in<br />
which form81 school training la aaaent1al.<br />
-_- .-.... ___-__-.<br />
- . --I-.---.<br />
__ _. -. .- -_ __- -<br />
.<br />
20<br />
. . -.<br />
.<br />
.<br />
,<br />
I<br />
--. c ---- -- ---_l-___<br />
.<br />
I<br />
,<br />
.<br />
.<br />
.<br />
1__1___<br />
.<br />
.
\I’<br />
. .<br />
i ’<br />
’ _<br />
. .<br />
I<br />
1<br />
Z-N -A*<br />
.<br />
*<br />
�<br />
.<br />
__. -- - ._ _-.. - ._ . . -- -. . -_-- -_._ --. .--_ --~_. ..^<br />
. .<br />
. .<br />
.<br />
.’<br />
�<br />
-...-. .._---.. -._- --.--. ----.- .-.-- __ -- .- _ ..__ _ - --..<br />
.<br />
-.<br />
At this point, f might add that la past MTh Conferences dlscusalona<br />
were hold concerning the cxpericnccs of the othl:r servlcce with proffllag<br />
evaluation scores. As a result of this exchange of lnfornation, the<br />
Ealfrted Evaluation Center va8 aided in ltr efforts to produce A Profile<br />
Data <strong>Report</strong> which accaepllohe8 those purpocea unique to,, the Army.<br />
SWry reports orb dlrtrlbuttd to -jot ccr=eandcrr in the field.<br />
Those report8 reflect tha’WS, tbe rpread of evaluation scores, And the<br />
total nunbcr achieving each @core by grade and unit. With this lnforrrmtion,<br />
cameender8 can ApprAt80 unit efftctlveness md emphu~lre training<br />
requlr-nts.<br />
The objective of Army Perronnel Kanageaeant IO rimply to obtain the<br />
uulm~m cf flclent uee of raenpwar. To oupport the att8lnment of thle<br />
objective through thr aunagesent programs I hava cited the evaluation<br />
process requires the coordinated effort of the Center, vhich is the<br />
oparatlog Agency of the ry8tea A group of 10 Auparvlrory and 31 question<br />
writing ~genclas who ftrrnlsh lndivld~al question8 And problms which<br />
maka up the various tests, and 81 monitoring c-ds who ~upcrvlsa and<br />
8UppOrt the actual operation of the tcitlng program at the many far flung<br />
locations throughout the world jlere soldiers are located.<br />
The drvelc+ent of evaluation material8 fs a team effort. Toot<br />
speclAlists here At the Center are in direct c~nlcatlon with thelr<br />
carnterparts at the question writing � �������� Thir te4un affort Insures<br />
that profesrltmal rtmdards of test develomt ara ir,;orporeted In the<br />
8yStam. Through the use of Appltcstlar of measurement prioclplos by<br />
professioual pcrconnel the Army’8 evaluation lnstr..srsntc are in liar<br />
with those rtandatds employed in professional prychologlca:l maa8ur@aeent.<br />
Thlr later-action batween the Center And the ochoolr Is neccrsary to<br />
8U8tAln the highest profesrlonar. standards in our devclopnmt of telt<br />
Aid8 cad tests.<br />
Distinct advantages ACCNO frau this tt)za effort. For ermple:<br />
It insureo inclusion of accurate and current: subject-mrtter in the taste<br />
by the question vrltlng agcnclcq it provides for � ppllcatlon of momsuremant<br />
prlnclplc8 by the Center’s proferrlonal parronnel, and thib in turn<br />
Insures that the test masuras the techaicel knowledge required of tha<br />
roldfer to rdoquatoly perform on hi8 job.<br />
The military mvlronment to&y is still one of the moat caaplu<br />
social organizations lu the country. The problgn of finding the right<br />
man for the right Job 18 A continuously expanding task.<br />
.<br />
21<br />
�<br />
. :<br />
.<br />
~-- --<br />
i i<br />
!<br />
i<br />
I<br />
t!
.<br />
.<br />
-- .<br />
� �<br />
.* ,_ _._._.-_ _ : ,.. : _<br />
. .”<br />
.-<br />
“G, i .<br />
.<br />
.<br />
.<br />
L<br />
Yhir ck8rification and aroigment technique involve8 all of the<br />
farm of mmagesumt fmiliar to em& of you, that 18 tha identification<br />
of aptitudar, laarued � killa and knowledge, qualification for odvancemeat<br />
and relectlcan of peraonnal for opeclal � �����������<br />
Our preuent � y8te6 ha8 conridereble merit since it is producing<br />
the right man for the right job.<br />
I believe the 8electicm nystcm, to find the right man for the<br />
right job, should be dafigltlve In purpo80, yet r-la adainl8tratlvsly<br />
r-10, and above all, arrlot the cmdar in the oelection proceru.<br />
You ladier and gentlawn realira the awunt of work and proferriorral<br />
application required to dovelop the tcating lnrtrumaatr to achieve thir<br />
objective.<br />
In my opinion, we have an excellent reearurercant devica to identify<br />
current lamvledga on what the men “ten do.” Rowever, soort ‘mult be<br />
accanpllshed m the ‘VI11 do” � ppllcatlon of the individual onca ha ha6<br />
been placed on the actual job. We muat not lore the confldance of<br />
camander8, vtroor we tell that the i?dlvidual PO job qualified, aad then<br />
have the mn fall mirorably when he is arrignrd to the job.<br />
I have examined the topico on the program for dtccurricm which I<br />
find nort thought provoking. I look forward to the opportunity to rtudy<br />
your dircurrlon group report8 end benefit from the trmrendous collective,<br />
technical end p~ofesriozul knowledge represented by your presence. The<br />
Eallrted Eveluation Center vi11 continue to rupport the Asrociatlon’~<br />
effort8 toward future lmprov-tu.<br />
Let me say once again that the US Army, and partlcu1arl.y the<br />
Enlirted Eveluatioa Center, � ppreclate8 the opportunity to hart thio<br />
Sixth Annual &CIA Ccmferenco at Fort Beujmia Harrison.<br />
Ihi concluder the formal oerrion tod.ey. I vi11 be followed by<br />
Mr. Price, our Conference Chairmm, who ha8 8ome announcement8 to make.<br />
22<br />
---- -- ---
. ,<br />
. .<br />
Perspective<br />
DR. HAROLD A. ED!XXION<br />
President, Performance Research, Inc.<br />
Washington, D. C.<br />
First of all, let me express my thanks to the speakers this afternoon.<br />
They have done a beautiful job of saying,in a different way, the<br />
same things that I want to say again tonight.<br />
I am listed here as the keynote speaket, and I have ‘been watching<br />
keynoters of the political parties this 8Umner to see just what they<br />
do, 80 I vi 11 know what to do tonight. First of all, th’ey seem to look<br />
at the glorious and constructive record of the past--what we have done<br />
that is notable, forward looking, and looks good on the ,record. Tha<br />
second step is to paint with dark colors the errors of the opposition<br />
and to offer many castigations for these errors. And thse third phase<br />
ie to point out the great and bounteous future which we offer. I<br />
propose to follow this as an outline this evening. It gfves us a good ,<br />
framework within which to take a look at our own problems. I have<br />
chosen the term “Perspective” aa a title. I thought thins was sufficient<br />
to offer maybe some direction, maybe some closures.<br />
To keep this keynote speech and its implications clear, we need to<br />
be sure that we are using some of the same points of view and some of<br />
the same definitions of terms. Let me define the term “teat” aa ueed in<br />
our discussion this evening. We are a group primarily ci=rned with<br />
the construction of achievement tests--testa which ahov job knovledge,<br />
understanding and skills. A teet, as we are using the term here, is a<br />
sample of behavior, dravn under such condition8 that cne may judge some<br />
set of skills abilities, aptitudes, attitudes, or achievements on the<br />
basic of this sample of behavior. <strong>Testing</strong> then becomes a problem of<br />
constructing tests which make it possible to draw an appropriate sample<br />
of behavior. We are concerned with the adequacy of this sampling, and<br />
with its reliability. Are the sampled behaviors percfnent to our purposes?<br />
Then there are questions of uniformity of sampling, from time to time,<br />
place to place, and group to group.<br />
his concept of test ae a sample of behavior ia a very useful concept.<br />
It reduces the %.agic” aspects of tests. Nothing seems mysteriotre about<br />
the idea of sample. Whether you have had a course in tee:ting ata university<br />
or whether you have sat at the feet of one of your technicians and had long<br />
discourses on the nature of tests, you know that when you draw a sample,<br />
there are errors of sampling. You know that you have to be careful what<br />
you are sampling. You know that you have to draw a large enough sample of<br />
behavior, so that it vi.11 represent what you are tryfng LO sample. I am<br />
including within this concept of a test all of our procedures for drawing<br />
samples of behavior. I include here the sort of things we ordinarily<br />
23<br />
\ b.,<br />
.<br />
.
. .<br />
.<br />
. _ . - - ._ ._ -.’ _ _ - _ _._. _..._ _---_. .__ .-. _ -__-.<br />
include as teste. I also include the performance of work sample type of<br />
teat. I include the subjective teat; and we too often look down our<br />
nodes at such meaaures at times, primarily because they have greater<br />
sampling errors under certain conditions than do other senples of behavior.<br />
I include an interview as a sample of beh,avior. I include ’<br />
anecdotal reporto as samples of behavior. <strong>Report</strong>s by superiors on the<br />
performance of subordinates is a report of their sampling of the<br />
subordinatea’e perforaancc. Even projective tests are rsmples of behavior,<br />
but it takes someone well-trained to tell you vhat he is ssnpling, how<br />
this sample is comparable to any standards, and what to do vith it. When<br />
we look at testing an sampling, this point of view also subordinates the<br />
techniques of testing to the real role oftarta,thnt of measurement.<br />
Techniques may take their proper place as voym and means of achieving the<br />
purposes of our tests.<br />
. --- /<br />
c<br />
. i<br />
And nov to look back briefly on our achievements. Our glorious past,<br />
from the point of viev of this group and this meeting, is one to which I<br />
can and do point with pride. I know of no group that has done aa competent<br />
a job in achievement testing. This statement is supported by the behavioral<br />
facts oi the membership of this group. The membership of this group has<br />
advanced the otatc of the art on all fronts -- job analysts, task analysls,<br />
job content, frequency, and so on -- the boscs for teat ccnstruction. They<br />
have improved our techniques of item construction, relating content to’re.-<br />
.<br />
quirecuent , and adjusting difficulty to appropriate levels. They have advanced<br />
the techniques of item analysis; they have also promoted the ncc.:rsity<br />
for item analysis. They have developed techniques for producing alternate<br />
,<br />
forms of teata. Por theae and for their other achievements, I salute the<br />
members of this group, and I take pride in being here and feeling that I<br />
I am really one of you.<br />
’ C’<br />
Then we come to the oppositfon, but the oppoeilton La hard to find, to<br />
identify. Who is our opposition? Are they the ones who refuse to use our<br />
products and try to make tests do thlngs that they were never intended to<br />
do? Arc they the ones that insist on using less adequate mean% of performance<br />
measurement than are readily available to them? Perhape ve are<br />
ourselves our own greatest opporiticn. That is a dangerous statement to<br />
make after pointing with such pride to the achievements of this group. But<br />
among us, I suspect that each of us, including your speaker, has been<br />
guilty of being his own greatest opposftion. When we see testing in terms<br />
of only one set of techniques, we are miaeing the boat. At least we are<br />
putting a good hole in its bottom so it can sink with us. When we fail to<br />
base our testing on adequate analysis of per.Cormance requirements, ve are<br />
damsging our program. When we fail to recognfte and state the specific<br />
purposes of the test in operational terms, we are running the risk of cutting<br />
our clvn professional throats.<br />
We want to know what it is we are trying to mecrure. You do one kind<br />
of job when you are trying to measure the status of knouledg~e. You do<br />
another job when you are trying to bring in skill and rcaronlng, and you<br />
_-<br />
.<br />
.<br />
24<br />
___L_._-.- .--.-_ -- _- _- _ _ - ._<br />
. .<br />
.<br />
. .<br />
.<br />
‘,<br />
.<br />
.*<br />
.<br />
i<br />
.
. .<br />
.<br />
. .<br />
.<br />
need another sort of test item when you are trying to measure memory for<br />
obscure facts. There ar,e times wher each of these purposes is pertinent,<br />
but do not USC them blindly. Use them knowingly. It. brings to mind one<br />
definition of a gentlemen --, one who never unwittingly offends another.<br />
We need to know how the test vi11 be used; we need to know what kinds of<br />
persona will take the test: ve need .to know where and how the test takers<br />
learn the material covered by the test; ve need to know these as part<br />
of the mission of the test we are trying to build, and until we know that<br />
ve run a serious risk of being our own best enemy.<br />
Another current form of opposition, and perhaps not as real as those<br />
just pointed out is represented by those who seem to enjoy swinging at<br />
straw men. They single out some of our weaker, less adequate test items=those<br />
that may be viewed with alarm, with scorn, derision, double-meanlng-and<br />
hold up to ridicule ail testing because they found a flaw in our work.<br />
Some openly question the use of tests at all. I have n.>lt found out what<br />
they propose to substitute for testing, but I suppose there must be something<br />
from the “good old days” which they find adequate. Now when you<br />
look at the kinds of things published by our critics - a castfgation of<br />
tests built on some misuse of test items from such tests as the Bemreuter<br />
or the Bell Inventory, an attack on testing in Fortune Magazine some ten<br />
years ago, and, more recently, a very interesting book b:y Dr. Benesch<br />
Hoffmann who seems to believe that multiple-choice questions are the<br />
bane of our social order -- should we ignore their strident and exaggerated<br />
criticisms? Within each such criticism is some useful information. This<br />
is why I cannot label them as our most dangerous opposition. I personally<br />
value Dr. Hoffmann as my most competent test item critic, He accuses me<br />
of the worst kind of ekulduggery and of a great desire to take unfair<br />
advantage in wrfting test questions. So when WC read these articles and<br />
hear these sp,eches that take us to task for what we may consider a minor<br />
sin, or some error that we have outgrown --- take a second look. We may<br />
learn something, perhaps ,comething useful.<br />
One cannot fight such critics effectively because o.f the many facts<br />
and assumptions which must be righted. But, from them we can see places<br />
where we might improve: how we might build more acceptable programs, how<br />
we might better market our products, and SO on.<br />
The third phase of this talk tonight is our look ahead to see if there<br />
is anything in the future other thsn the manufacture of more tests just<br />
like the latest models we have been nr.king. Recently I visited an automotive<br />
museum. One thing struck me forcibly; there were cars prior to 1912 that<br />
looked like overgrown buggies , pcvered with washing machine motors. But<br />
all of those ca:s built since 1312 were not so old-fashioned. About all<br />
we have done with automobiles since 1912 is improve them,, We have put on<br />
bigger motors and better tires; FJt we have made little real change in them-a<br />
here in a period of 52 years no ,.&al basic change; improvements, yes;<br />
change, yes; but no real breakthr *ugh-<br />
25.
,’<br />
” ._<br />
, i<br />
.<br />
.-<br />
.<br />
.<br />
L . _ _...__ - .- . -. __ _____--,<br />
I think those of us ia the testing business are in that same boat -- _<br />
rod to ray. The development of the format for objective tests, which we<br />
all use and think of quite highly was a product’of A. team like this. We<br />
hAve improved them, shined them up, And dressed them up in different forma;<br />
but We reslly haven’t mcldc A basic breakthrough in ~11 that time.<br />
The AmY hIphA teat used in World War I was A milestone in test developmerit.<br />
That test wan in objective form. It WAN certainly a useful teet,and<br />
it hsd in it many of the kinds of content we currently use. In fact, for<br />
AOCIIC purposer, it is still usable. In fact, it is set11 on the rmrket. But<br />
we build testr for many other purposes. We have used improvements of the<br />
techniquea, but we have yet to realize A real breakthrough. We write items<br />
with one, two, three, four, five,and six AltematiVcU; one of them is right:<br />
the re8t of them art wrong. They did not have at that time the refinements<br />
we have developed in item analysis, using biserial coefficients, or whatever<br />
other statistics you May use at your technical altar. We did not have factor<br />
AMly8i 8, AE A matter of hiStOriCA fact, psychometrics did not reelly<br />
get going until the days of Cm1 SpearmAn, followed later by Thurstone.<br />
The idea of item homogeneity is still A more modem concept. But I don’t<br />
consider there as breakthroughs, They Are improvements in juet the same<br />
way that bAllOn Circa were AU improvement over the old high-pressure hard<br />
tire8, or in the OMIC way some of our modern carburetors are improvement<br />
over the simple carburetors of the pest.<br />
. Despite all this, however, I Am optimistic about the real possibility<br />
thdt we are on the eve of some major breAkthroughr in testing. What they<br />
are, I do not know. I am equipped with neither that kind of vision nor<br />
wisdom. By A breAkthrough, I mean a change or new idea that alters the<br />
whole perspective of test c.onrtruction as well as test usage. Are there<br />
AituAtions and times when another technique would do better; for exttmple,<br />
better As the first step in determining what factors in A job are really<br />
crusial, or really pertinent? Instead of trying to cover the entire job,<br />
maybe there are uome element8 in the job that ere really crucial. There<br />
. were some early flags of that kind back in the 30’s in the development of<br />
job questions by the U.S. Employment Service. It was Asked that these<br />
test quettions discriminate batwern the master or journeyman, and helper.<br />
� Their particular purpose in the employment office uas to identify the men<br />
who would like to upgrade themselves, but who really did not have the kncr:-<br />
/<br />
�<br />
Adge of the mater craftsmen they had worked around. But many had worked<br />
enough so that they had learned something of the job. RAybe there are<br />
Aome clues in using the concept “How do you know?” ‘What did he do?” kind<br />
of questioning. Only once in my career have I velidatled a set of selection<br />
tests All of which measured up to the criterion of performance. Rather<br />
than using ratings of competencgwe tried a kind of job analysis attempting<br />
to set what aupervisorr saw when they rated A man’s performance es “good.”<br />
We asked “Who ie your best man? Whet does he do or fail to do by which you<br />
know that he is your top worker?” We would continue t4e conversation about<br />
one man for 30 to 60 minutes finding out whet he did, boo he worked, how<br />
�<br />
26<br />
. . . -
.___ ---.-.-a ---._- ~.-~“..CI.~.--..w.‘r -.--.-.-^_. -. ._ .<br />
.’<br />
!<br />
)<br />
he handled his contacts, how he cooperated vith the home office, everything<br />
we could about him. Then WC would ask, “gow about one of your<br />
poor ones? Who is.the next one you would drop if you had to cut iour<br />
work force? Haw do you know he is that bad?” We found enormous overlap<br />
in observable performance of the better as compared with the poorer men,<br />
I<br />
a<br />
’<br />
and how he worked, what he did and vhet he failed to do. but there wds<br />
a small area of difference. It did not tell us too much about job duties,<br />
but it certainly told us hw the company valued job-parform=ca.. And<br />
this vas the information that gave us the clues 8S to vhat tests to<br />
select. Maybe thin point of view contains 8n idea we, should investigate.<br />
Instead of looking at all the duties, functions, and rcsponsibflitiea of<br />
the men, try to leern in what wrys we can spot a good one when we see one.<br />
Let us not forget that there is a new set of tools at our disposal;<br />
they are known as computers. Computers and their accessoF equipment<br />
certainly ought to open nev d-ors to us. They have already opened many<br />
doors. The biggest and most important is thet they permit us to do jobs<br />
that were too big to do before. They handle much more data. Before<br />
computers, how big a job of item analysis did you tackle? One hundred<br />
high cases; one hundred low? That would be close to the limit of what<br />
you would try to do. With comp::ters ve don’t think anything of using<br />
revere1 thousand cases in our item analysis. We are tackling problem?<br />
that are bigger erd bigger. Some of you may remember the day, it is before<br />
my time I mfl;ht add, when six correlation coefficients would enable<br />
one to e8rn a I% ;i. degree. As time vent on more correlations and more<br />
complex correlat;.)n functions vere required. We are collecting and using<br />
bigger and bigger quantities’of data in support of our efforts. These<br />
are improvements, but not breekthrougha, because we have been using the<br />
s8me old techniques, but with increased effectiveness, Computers do uuggest<br />
that ve can do some things in ways never done before. For example: One<br />
group of people for vhcxu I heve constructed tests .-or some yeers are so<br />
bright I 8m not able to write items tough enough for them. These are the<br />
science talent perticipants for the Westinghouee Scholare;.ips; these boys<br />
and girls are really bright. Nevertheless, I’m still looking for talent<br />
even in that group. I want a test vhich discriminates particularly among<br />
those in the upper half of the contestants. I want a test which will be<br />
reflected by a frequency distribution that has a positive skew. Some test<br />
questions which come within the proper difficulty level may depend upon<br />
remote and unimportant bits of information. Is it, for example, really<br />
important to knov precisely hw many light years Arcturus is from the<br />
Earth? Or even to know the answer within a thousand light years? This<br />
fs the kfnd of question that is answered correctly by few people, but etfll<br />
did not seem to be just what the situation demanded. Then ve tried multiple<br />
choice questions; but instead of having just one right and 4 wrong answers,<br />
we thought we would have one, two, three, four, or even five right ansvers.<br />
A score of 1 is earned if all the alternatives are marked correctly, and<br />
if there are ;qy errors the earned score is zero, This produced test<br />
questions of appropriate difficulty and gave the right kind of frequency<br />
.<br />
.<br />
27 i<br />
-._ ._<br />
,<br />
.<br />
, 3 t,<br />
^><br />
.<br />
. ,<br />
I<br />
i<br />
1<br />
I’<br />
I<br />
/
, .<br />
distribution. Basically, I have used the simple concept of conpoutd<br />
,<br />
probability,<br />
.<br />
Psychologists and test makers seem to have assumed that all that<br />
one needs to measure, or perhaps can measure, are behavior samples drawn<br />
from individuals. have you ever discussed the validity of a test<br />
question with your field people and had the answer, ‘Well that depends.”<br />
That depends on circumstances of the job, of the man’s boss, of the<br />
working climate available, of the standards to which he is held.<br />
*<br />
Another area that we should explore is this matter of fncludtng in<br />
our factors of performance some which are outside the skin of the individual,<br />
but nonthelaar lnflu5aco the quality of hia pstformance and of hlr knowledge.<br />
These pieces of information should appear on the right-hand side<br />
of the equation; not on the criterion side, We have not explored this.<br />
Since the daya of Carl Pearson we have bowed down in front of linear<br />
measurement. tven out factor analysis is just a complex linear system. We<br />
have not broken with that tradition; we have not developed the tools to do<br />
it. Did you ever try to compute a correlation matrix using nOnlimdr<br />
regression lines? The first thing one does is rectify them to get back to<br />
linearity. You are well aware that there is ao much in human performance’<br />
a.rd human value that is nonlinear. Even starting from mince pies, there<br />
15 such a thing as too little and there is such a thing aa too much. In<br />
between them there f5 some kind of optimal amount. With most of our personality<br />
measures we find the same sort of thing, Perhaps we call them bi-polar,<br />
but we ought to look at these a, porentially nonlinear concepts.<br />
Then we talk about patterns of performance. Wfthfn a set of questions,<br />
suppose there are three for which the answer “yes” are "out." We might call<br />
these “lethals,” Conditional answers also might be explored. If you mark<br />
answer number 3 on question 1, then whatever you answer to questions 13 and<br />
26 is wrong because your prerequisite knowledge was inadequate. There are<br />
a number of these pattern concepts that might be useful and the method tested.<br />
Out computers open the door to such problems because this kind of exploration<br />
call5 for considerable amounts of data. And I might add, your organizations<br />
have sufficient data. You have the opportunity of exploring rationales for<br />
weighting and patterning of answers.<br />
Some of the possible questions are, How good is it for a man in a<br />
given HOS to be a member of the 1 percent who give the correct answer to<br />
quertion 471 Is thic worth more than 1 point relative to the other<br />
questfonr? One might explore the weighting of item5 inversely proportional<br />
to frequency of right anBver5, so that those who have the mote unique<br />
knowledge become more vieiSle.<br />
28<br />
. .<br />
.<br />
* .<br />
-<br />
.<br />
*
. .<br />
-<br />
. -. .<br />
. ‘.<br />
-<br />
-m.__ ’<br />
We need careful definition of what we w4nt aach texst to do. we<br />
IWad ~108tar 6tbt6mant6 of what we want it to ~1~1oura and the condition8<br />
under aiich it owe do ito job. For a given aiammxwnt task w4 may nsed<br />
4 toot which yield8 89% perfect @core4 , or one that nsbodp parscoo ualeoo<br />
ha nsakeo a perfect ocara,aud there ara time0 when thi4 is 4 reoooneble<br />
toquirement. What kiod of diocriaination doe6 your test need to aukt?<br />
Doe4 e4ch queotion naud to opcrata at tha one parcent level? Are you<br />
aokfng it tcs diocrFrninate in ouaa othar fashion? 50 you want the tMt<br />
to be uoed a8 a motivation:1 device -- to moko your teat takoro uu;iouo<br />
to get out 4nd do more, better, 4nd Cantor? Do you want 4 toot that is<br />
basically a teaching dcvica? That wa4 the initial motivr~tion of S. I..<br />
Preooey’o teaching machine. In uoing hio mochineo,ovory time o otudant<br />
puohod 4n answer button to uIewer the tart 4nd got the right 4n4wor, the<br />
-chine went on to tha noxt quartion. If he got 4 wrong rnswar he kept<br />
on prrooing anover buttono until he got tbo right enover,, How vc erll<br />
thio reinforcement; imediaco feed bock.<br />
Thare aro ocane othar quaotiono to which pa Q not h&ve uuwcr~. ilou<br />
volid ohould 4 toot be? We have ha4rd about maximum validity, about lack<br />
of validity; but molly wh4t io the optim41 v4lidity YU rlhould look for?<br />
Anothsr quartion io *How rapidily and under whet cmditicmn do tsot ocoreo<br />
loose thair validity?” I have conoidorad the po4slbility of submitting a<br />
rootirch propooal for the development of tcot record inb: -0 ink4 for cuking<br />
teot recordo -- 80 that the teat � ���� for each teat wou1.d be faded out Juot<br />
at the tlme tha tart 4cora had lort it4 validity. At me time in 4 cwanooling<br />
center, wo act up a hard and foot yle, “Any percouality, intorest, or<br />
oimil4r tert has lort it4 validity 4ftcr 3 months.” After that,o5t4in 4<br />
freoh ocore - 4 04mpla of whrt that individual lo like now. And it might<br />
be noted that people oeeking counoelfag are parh:?o leoot stable in ouch<br />
charoctariotico. For b4ric rptituda teat8 ouch a# tha Ohio Stats Poychological<br />
Tort, we aokod for 4 frarh tart or rumple after one year. We nacd<br />
more knowledge 4bout thfr queetion not only in toner of tho teat, but 4140<br />
the intervening aventr. In measuring how wall 4n electronic6 technician<br />
parform on tho Job you msy get 4 meaouramnt at the time he completer<br />
bio training. Suppore he had �� � it and wait 60 dayo for �� � rafgnmont;<br />
hw volid lo hfo ta4t bcore after 60 d4yr with no practfce in the electronico<br />
ohopt Hw valid would hi8 t44t ocoro bat<br />
In being here tonight, I have triad to point to 4om3 of the future<br />
and to � tir up your thinking. I hove tried to illurtrata the l’taot’@ 40 4<br />
sample of behavior. I w4nt to cheer for axtending 4nd trying out new concapto<br />
of 4n8wer pattarnr in tart fonnulatfon. I want to iace ue devalop the<br />
ure of computer6 to do things in test rcoring and teat analyrfa th4t wo<br />
hove only tolked or drarsled 4bout and to do � ome thing4 that wa h4ve not<br />
avan rtartad to inugine. Keep your w4ys and maano, your rechniques, in<br />
proper peropectiva; and keep in front of you a clsrr idea of the purport4<br />
of the test itself by defining it4 uoeo, 48 well as ftr limft4tion4, 4nd;<br />
through these, extending Its ueefulneoo a8 a me4euring device or w4y of<br />
drawing more 4dequ4te, 4nd more effective samples of behavior.<br />
I .<br />
I<br />
29<br />
_ . _ __ .._^_ - .._-_ . -. -<br />
-<br />
._. ,. .<br />
.<br />
:<br />
I<br />
.<br />
.<br />
*.- ._ ,(<br />
\~<br />
._,. _ _ .____. ._-. .- ___--__<br />
*<br />
--.-<br />
,I(<br />
- --<br />
,<br />
-_-
-.<br />
-. _-..<br />
.- .<br />
- ! e<br />
I<br />
_-.. .<br />
1<br />
.<br />
.-<br />
.<br />
2<br />
-.e.<br />
T-&<br />
‘. .<br />
. . .<br />
,<br />
.<br />
. . -<br />
Approschcr to Improved Haaaurcment<br />
CIAUDE F. BRIDCBS, Chairman<br />
US Army Enlisted Evaluation Center<br />
Xn response to the consensus of the recomendatione of the participating<br />
agencies, the conference theme is %xraaeSng Measuring Efficiency<br />
of Evaluation Inatmentr.” This series of theoretical symposia is designed<br />
to explore ways of obtaining marked increase in the percentage of<br />
overlap between job proficiency factors and the evaluations of the current<br />
job proficiency of enlisted personnel. Special attention to innovttions<br />
and possible brenkthroughs in wtyt of increasing the covarlance between<br />
achievemant and measuring instruments Is long overdue. Concerted, lnttnsive<br />
efforts will be ntcearary if we are to make rignificant inroads<br />
into uhrt Captain Hayes aptly referred to yesterday aa “the outer apace<br />
of terting.tV Mere t’poliehing’8 activities using usual test validation<br />
and item analyoie procedures obviously art not enough.<br />
Since World War I, such polfshing activities have indeed improved<br />
tests but, as Dr. Edgerton so clearly Pointad out in his keynote address,<br />
have not made genuine changes. Perhaps the met continuously extensive<br />
research in the intervening 45 or so years has been pointed towards improving<br />
the effectiveness with which academic aptitude tests predict<br />
success in school. However,afttr all these efforts, what it the ututl<br />
correlaiion between ouch teatr and school marks? Dr. Chester Harris in<br />
the Encyclopedia of ?ducational Research states that the correlation<br />
between “intelligzce” tests and scholastic achleveoent typically fallr<br />
within the range .40 and .50. The validation studies canplated by the<br />
Evaluation and Analysis Branch of the US Army Enlisted Svaluation Center<br />
indicate that met of our recent Enlisted HOS Evaluation Tests ‘have<br />
validities in this same range. Cur attempts to adapt the most crucial<br />
aspects of the usual @‘custom-made” test developmeat procedures to a<br />
high-apted closaly-timed test production-line and to apply previous and<br />
current research findings in the measurement area to our test dlevelopxrnt<br />
8ctivities during the last six years have enabled us to improve or<br />
polish t our tests o a level approximately comparable with the typical<br />
academic aptitude tests,<br />
This looks fine. But let us consider -&at the correlation of .50<br />
actually mean2 in terms of the percentage of variance in achievement<br />
typically being predicted after 45 yetrn. Squaring .50, the upper end<br />
of the range, we get only n 25X overlap between the two varitbleo.<br />
Figure 1 presents a graphic picture of this sltuatioa. Seventy-five<br />
percent of the variance NOT touchad yet!<br />
30
w - ------- --- -.-. ___..- - I _ .__-.. .- _._.. .<br />
.<br />
Combiaetioar of two typical evaluation instruments gf.ve slightly<br />
better rasultr. The recent validity studies indicated that thin year<br />
RRC can expect correletionr in the neighborhood of those s:hown in Figure<br />
2. The Rnlirted HCS Evaluation Tear combined with ratings on the<br />
Vxxmnander’r Rvaluation <strong>Report</strong>” for mOst military occupational specialtier<br />
should yield multiple correlation (R) with peer ratings of about .57.<br />
(Peer rating8 may not be the most valid pOsBible crittrfon of all erpects<br />
of over-all job proficiency. Hovever, there is considerable eatiefectory<br />
evidence that, when obtained frcaa raters who know they are for experiment81<br />
1lp9 & peer zngs can provide practical and useful appxeele of a<br />
� ignificent portion of complete job proficiency,) We are working especially<br />
to improve the teBts for the “problem” XCS and expect even higher veluee<br />
next year. HoYtvet, the improvements still will leave not tapped about<br />
two-thirds of the factors involved in differentiating between different<br />
levslrr of job proficiency. Continued pollehing will help decrease this<br />
unmeasured variance eomevhat, but the ueuel polishing techniquer or even<br />
the addition of other carxnonly ured types of measuring inrtrumentz,<br />
conwnonly raise a multiple correlation only a few hundredths of a point<br />
at the mat.<br />
Indeed we still do have a very long vay to go. Today YB will attempt<br />
to survey some of the possible routes to be followed. Rowever, the best<br />
modes of transportation over them will remain to be determined. The<br />
general territorial area.8 which the route8 muat cover are job factors and<br />
personnel factoro--i.e., analyees of the characteristics of the job end<br />
analyses of persons performing it well vereue thore performing Ierr veil.<br />
Probably the most ixnediately fruitful of the two general areas of<br />
investigetlon involve8 the development of ways to Improve the determination<br />
of the factors in the job that dircriminata between indiv!lduels with varying<br />
levels of job proficiency, Recaure of this, and of the intorest camnon to<br />
El1 the agencfea, a special tterion on “Job Analysis for Teat Development<br />
Purpoata” hia been mt up. The rterion on l’Nonstetistica:l Criteria for<br />
Evaluating Iteaur” likewise Is pertinent to the area of job factorr.<br />
The other general area of invertigetion involves new and better ways<br />
of measuring personal factor-r of ptrronntl end of relating them<br />
appropriately to the job factora. The major typo8 of personal characteristicr<br />
for which appropriate mtesuret are nstdsd for verioua jobo might be<br />
rtructurtd AB follows:<br />
1. Job Knowledge (texonwltr of job content),<br />
2. tlentel Skills (taxonomitr of mental functione--mental manipulatione-required<br />
by the job).<br />
3. Xotor Skills (phyrical manipulation).<br />
31 .<br />
--.
-.<br />
‘. .<br />
_.”<br />
1.<br />
2”<br />
--<br />
.,;<br />
.<br />
. -:<br />
.-<br />
.z<br />
/<br />
. -.<br />
.A .<br />
.<br />
._<br />
/<br />
,<br />
,t<br />
I<br />
!<br />
I<br />
. 1.<br />
. *<br />
.;’<br />
/ . .<br />
-e__.:_I_L---.. . ._ --. -__L.- -- .YU r--- -.,-., __.. ._. _._. - -.<br />
4. Interpersonal rtlationrhipc skills (social -niPulationo) .<br />
5. Job motivation (application and effort).<br />
6. Test taking motivation,<br />
.<br />
-<br />
� � � � �<br />
.<br />
7. Interaction of personal characteristics with situatianal<br />
characteristics (Effectiveness of leadership provided;, type of<br />
management; characteristics of subordinates, peers, supervfaorr,<br />
administrators, locale, spatial conditions, etc. Why does the<br />
oamt man perform very well in some places and not in others?)<br />
8. Other noncognitive personal characteristics (attitudes, interests,<br />
level of aspiration, initiative, drive, energy, perservtrance,<br />
etc.).<br />
In the relatively ahort tims available during this conference, it<br />
will be tiposeible to effectively consider all of the major kiudr of<br />
approaches to improved mtaauremtnt of the varioua types of ,ptrsonal factorc.<br />
In order to auggtat others for later consideration, atveral potentially<br />
fruitful approaches omitted in these aaninars are mentioned. We will not<br />
discuss the atatiatical treatment of individual response data, such as<br />
beta weighted correction for errora, analysts of response p.atttrns, and<br />
scaling. Another omittad but potentially useful area would be the � nalpsta<br />
of officinl personnel record8 (use of biographical data, aptitude testa,<br />
education, age, years in aarvict, training records, honors, edvencement<br />
rate, etc.).<br />
We vi.11 be able to explore only about one idea for the other possible<br />
approaches. For example, ve can consider the utility of only the<br />
readability aspect of the ccnmnunication problem in teats, omitting semantics,<br />
use of illustrations, granxaatical conrtruction of the stem, etc. Many oE<br />
the omitted ideas involve primarily test polishing activities, but some<br />
of them ahould be remarkably effective in decreasing the unmeasured variance<br />
for a few military occupational apecialtiea. Cronbach, in his Essential6 of<br />
Psychological <strong>Testing</strong>, 1960, page 331, cite8 the result8 of an unpublished<br />
study by the Training Aida Section, Ninth Naval District, Headquarters,<br />
Great Lakes, Illinola, 1945 entitled “A Comparative Study of Verbalized<br />
and Projected Pictorial Tests in Gunnery.” He rmriaod the result8 61<br />
follows:<br />
“Training of Navy gunnrrr had been validly evaluated by<br />
acorta made in operating the guaa. As an economical aubatitute,<br />
verbal and pictorial tests were developed, Identical information<br />
vaa teated in two forma, the same quaation being asked in<br />
vorda alone or by mean8 of picture8 supplemented by words.<br />
Questions dealt with part8 of tha gun, duties of the crew,<br />
appearance of tracera when the gona was properly aimed, etc.<br />
--<br />
32<br />
. ‘.<br />
. .<br />
a<br />
�<br />
;
.<br />
. . , - .<br />
.<br />
-<br />
.<br />
2<br />
. . .<br />
.--...-. . _---- .-- --__ L.--i- . .._._” . ..- _ . .._I. __ .I__^ ---._..-___ ._.-.--.--^ . . _. - --.. .- -,<br />
.-<br />
-. __...._e._.--.I_ ,i<br />
The pictorial test had a correlatfon of .90 vith instructore’ marks<br />
based on gun operation wherea the valfdfty of the verbal test was<br />
only .62. The verbal test war in large measure a reading tes‘t; it.<br />
correlated .59 with a Navy reading test, while the picture test<br />
correlated only .26 with reading.”<br />
After these introductory remarks were prepared, the October 1964<br />
i88Ue of the American Psychologist arrived with an outstanding presentation<br />
of concept8 closely paralleling some of those we have been considering.<br />
In fact,eurpriaingly enough he used a quadrant idea to represent variance<br />
in ,comon, soqedat like this hart (F’fgure 2) that had been prepared to -<br />
-depict clearly the magnitude of O’ar challange. There is no better way to<br />
close theae introductory problem defining remarks thsn to quote the opening<br />
sentence in Dr. Melvin R. Harkr’ article entitled “How tab Build Better<br />
Theories, Test8 and Therapiecr.*’ Dr. Hark8 aptly stated:<br />
‘@The theai of this paper ia that psychological researcher8<br />
too frequently define their problem8 in ways which intrinsically<br />
. . preclude solution; that they spend their time in Bflding lilies<br />
of dubious quality rather than in determining why or how’<br />
.- promieing new blooms were blighted; further, that this<br />
.deplorable state is characteristic of theories, therapies,<br />
a n d tests.”<br />
. .-.-.....-... .-._-__,- __.I____...I____ ---- -<br />
.<br />
*<br />
.<br />
,<br />
. ,:<br />
_^._ ._-__ _ .____ _I___ -__. _ . I.._. - ._..... _I. --<br />
.<br />
.<br />
.<br />
,/<br />
$’<br />
1 iI<br />
tI<br />
i<br />
I<br />
3. :<br />
I i<br />
-
i<br />
ACHIEVEMENT i<br />
t<br />
UNMEASURED h<br />
TEST<br />
5 Y =.50<br />
r2 % Y =.25<br />
Figure 1. Percerttage of Achievement Measured by Test Hoving<br />
o Validity Coefficient of.50<br />
I<br />
67.4 9% .<br />
UNMEASURED<br />
L<br />
I<br />
&. - - - - - - w - m - l<br />
RATING SCALES - -<br />
rx2 Y<br />
P x2 y<br />
(x2)<br />
= .45<br />
= .2025<br />
P2 r = .I632<br />
- - - - - - - - -<br />
1<br />
TEST<br />
f<br />
(Xl)<br />
I<br />
I<br />
rx, Y<br />
r2x, Y<br />
PI r = .1632 I<br />
rvP<br />
= .45<br />
= .2025 ;<br />
= .20<br />
r2 Xl x2 = .04<br />
Ry*x,x2= .57<br />
R2 = .3264<br />
Figure 2. Percentage of Job Proficiency Meosurcd by<br />
1<br />
Combining Jab Mastery Test and Ratings.<br />
i<br />
I
c.<br />
/<br />
I ’<br />
.<br />
a<br />
.<br />
c<br />
.<br />
’ ,<br />
Summary Comparison of Purposes and Programs d * i<br />
:<br />
of the Ullitary Services<br />
zlr. Chairman, Rellw Conferees,<br />
E. c. JoHHSo1’I<br />
US ?rmy Enlisted Evaluation Center )<br />
As we begin to explore possible approaches to improved measurement,<br />
it will be well to review briefly, as a point of departure, the programs<br />
that arc nw in operation within the military services. You each have a<br />
detailed s-ry prepared by each service on their evaluation programs.<br />
Yn this paper I will present a Summary Comparison of the I?urposes and<br />
Programs of the Hilltary Servicer. I have prepared a char: that depicts<br />
thi 8 canparison. his chart is listed as Appendix 1 at the end of this<br />
paper. All the military services use a paper-and-pencil multiple choice<br />
type test as the basic inrtrument of their evaluation progrsm. This<br />
te8t is supplemented in some cases by a performance test. These tests<br />
mre very similar, vith the exception of the Marine Corps, in that they<br />
are designed to mcamre the job knwledga required for satisfactory performance<br />
in the militury specialty that IS being evaluated. Each military<br />
mpecialty has a separate test for each subdivision of the specialty.<br />
These subdivisions are either pay grades or skill levels. The number<br />
of questiono in these tests varies from 65 to 150. he NC0 tests include<br />
coverage of general military and f.lpervisory abilities as vell as the job<br />
specialty.<br />
.ARmPRoGRAH<br />
The Army uses two inotruments in its evaluation progrsm. An M0.S<br />
evaluation test for each skill level and a rating form, Couxnanders<br />
Evaluation <strong>Report</strong> (CER). The CER Is a rating form ueed to evaluate a<br />
noldier’s performance characteristics in a specific HOS at an ertablished<br />
level of skfll. One rating is accanplished by the soldier’r immediate<br />
superfor and another by the immediate superior of the rater, The CER’s<br />
include scales for rating the individual’s cooperativeness, reliability,<br />
job performance, and other factors.<br />
After the tests are administered Army wide, the MOS evaluation test<br />
unswer cards and the canpleted CER’s are forwarded to the USAEEC. The<br />
two Instruments are scored at the Center by canputer and aI canposite<br />
evaluation score is obtained. An HOS Evaluation Data <strong>Report</strong> is prepared<br />
for each individual teuted. This report contains the individual’s<br />
Evaluation Score and a profile showing his standing in each subjectmatter<br />
area of the test. The unit coaxmander receives a copy of the<br />
individual’s EDR which he reviews and then forwards to the individual<br />
soldier. Thus the individual and his unit commander is made aware of<br />
the subject-matter areas’ in which the soldier stands “high” aa well as<br />
those areas in which he needs to improve.<br />
.<br />
.<br />
.<br />
. .<br />
._ .-- --.. - _--.__ _ ___ _ _-_ -___ -.-__,._ XI ..__w_._. ---I_-- --.. _.<br />
_ _--__ -_ -. ---. - --<br />
.<br />
i<br />
:<br />
i
.<br />
, -.<br />
.._..<br />
.-<br />
---- _..-.__ 1-1 .-.__ - ___... _<br />
Thir FSX Evaluation Score and the Profile Data <strong>Report</strong> i-8 then ured<br />
a8 a baris to:<br />
Source of Itmu<br />
1. Avard Pro Pay<br />
2 . nos Verification<br />
(Primary and Secondary HOS)<br />
3. Ronmtiou Qualification Score<br />
(U8ed by camanders on m optional bario)<br />
4. Pay Grade end MOS Determination<br />
(Officer reverting to EK)<br />
5. Identify Training Reeds<br />
(Both for the individual end the unit)<br />
The Army depend8 upon Army service school8 and other inrtallatioas<br />
to kite tbair te8t itaac. Host of the tlmo the item writer8 are lnrtructorr<br />
in the MO.5 qualifying coume. They are civilianr, enlirted men, or<br />
officer8 who are arrfgned thir duty in addition to their regular inetructoz<br />
re8ponribilitie8.<br />
The Nuvy u8a8 a 8sparrte examination for each petty officer pay<br />
grade of all Navy rating8 with a few exctptionr. That exuainationr are<br />
� dmlnlettrtd by t.xrmin?.ng boards vhtttvar naval parronntl are ttationtd.<br />
The completed tn8vtr card8 ure scored by the Naval Exuainlng Center.<br />
Pars/fail cattgorltr art 8et up for each exmnination bared upon a multitude<br />
of factor8 much as netdr of the rtrvict at large, budgetary problase, and<br />
above all, the dtgrtt of qualification and performance of each candidate.<br />
Each candidate ha8 to par8 the exmainatlon before the other factor8<br />
� re conridered. Final � dvancanant, � sauming the candidate has fulfilled<br />
all prtrrqui8itt rtquirarentr and has attalned a par*ing #core on the<br />
examinatfon, 18 made on the baair of relative rtanding on a final cwpoaitt<br />
acore. Thir canposits score includes there five factors with cueximut~<br />
valutr U8 follour:<br />
.-.._<br />
.<br />
. .<br />
.<br />
.<br />
_. . _ ., _- .._ _ _ - .<br />
.<br />
.<br />
36
--<br />
/-’<br />
1. Bxamfnatfoti Grade<br />
2. Performance Factor<br />
3. Length of Strvicc<br />
4, Time in Rate<br />
5. Naber of Awards<br />
Maxiaum Composite Score<br />
If the candidate ccnzpcter In an occupation where there am more<br />
vacancies than qualified ptrsonncl to fill them,then the exanination<br />
acore qualifies him for advancement.<br />
This is the procedure used for advancement to pay grader E-4<br />
through E-7 to fill Navy wide vacanqies. Selection of personnel for<br />
pay grade8 E-8 and E-9 to accaaplfrhed by a selection board convened<br />
in the Navy Department, vhich rtviewr exuninatfon result@ and fndividual<br />
record*.<br />
Source of Items<br />
The Navy’s item writers are aseigned to the Naval1 Examining Center<br />
for a regular tour of duty.<br />
TEE AIR FORCE PRCKZUX<br />
The Air Forct uses three rpccialty knowledge tests for each Air<br />
Force Specialty. They are 3 (#emi-skilled), 5 (skilled), and 7 (advanced)<br />
levelr. There epecialty knovledge terto are adminirtared to Air Force<br />
ptrronnel world-wide. After the 5 and 7 level tests have been given, the<br />
teat ansver cards are sent to the Personnel Research lAboratory where<br />
they are rcored. The paes/fail rates for these test8 are determfntd<br />
in accordance wfth criteria ertablinhed by Htadquarterr, US Air Force.<br />
Scores on rpecialty knowltdge ttata are reported In pcrctntile form and<br />
do not indicate the actual nunber of questions answered correctly. This<br />
score ie used to identify those airmen who posrerr sufficient knowledge<br />
about their jobs to be considered for up-grading to a hlghtr level job.<br />
Thie ecore ir then used vith &her factors to areers the overall compettnce<br />
of airmen for the avard of an Air Force Specialty.<br />
Source of Items<br />
The Air Force pl;:er ftr subject matter apecialfets on TD’f to the<br />
Personnel Research Isboratory. These subject matter specialists are<br />
senior NCO’s in tht Air Force Specialty. for which tests are being developed.<br />
They develop all the test items for the three tests in their specialty<br />
while they are on this TDY.<br />
37
’<br />
;;<br />
. ‘-_<br />
. w<br />
__ .,-_. . __-__._~.---~~x~~~~~c~ ---- - --_.._ . ._ - __ --.- --.. ---<br />
Tbb c’c -t @ad’s progrrs barically porollelr that ‘of t’he Navy<br />
80 I will not go into their program in detail.<br />
\<br />
Tffe KARXNE CORPS PROCRAH<br />
The Marine Cor~8 utiliner revere1 test lnrtnments in their evelu8tion<br />
program.<br />
The tart that ir 8-t caqmr8bla in u8e to the evaluation te8t8<br />
urbd by the other rbrvicb8 18 their Caneral t4ilitary Subject Tart (c;MsT)<br />
Thiv telt 18 ured to determine the eligibility of enllrted men for promotton.<br />
It 18 adminirttred to eligible corporal8 through rtaff rerguntr.<br />
(E-4 through E-6). It differ8 frau the other tt8ts dl8&88ed in that it<br />
doe8 not cover the militcry rpeciolty or occupation of the individual<br />
but h88 a broad rcopt ccncanpa881ng thO8e rubjectr considered inderpenreble<br />
for Mariner in a canbat area. Tha8e rubjectr include tactical subject8 at<br />
8qu&d and platoon level, equipment and uniforml, field 8anitAtiOn, fir8t<br />
aid, etc. Thin tt8t 18 given three time8 trch yur, the Mower card8 art<br />
8ent to the Marine Corpr Inrtitute where they are gmded, and the rerulto<br />
are foxwarded to Promotion Branch, He8.tquarter8, Marine Corpr end the unit<br />
cmnranderr .<br />
A Urine rm8t pa88 tha CZGT before he i8 conridered for prarotim.<br />
The other factor conridered ir hi8 promotion 18 hi8 proficiency in ha8<br />
technical 8QOCiUltv. Thi8 IO determined by UImU8 of fitness report8 rubmitted<br />
by hi8 immmdiate ruperior over a period of year8.<br />
A8 8 matter of gtntrwl interbrt thb HArime cOrQ8 h88 threa other<br />
tbltl.<br />
1. The Geucrql <strong>Military</strong> Subject Proficiency Evaluation Tert j@lSPET)-<br />
Thi8 tart cover8 genernl military rubjtctr in more detail than the<br />
Q4ST and ir urbd by the unit commander to identify the training need8<br />
tnd to judge thb bffeCtiVtie88 of hi8 training progrun. It i8 provided<br />
to thb wilt c- dsr by the Marine Corpr Inotitute. It ir provided<br />
8ohly for the locel caruaander. No report m tb8t rb8Ult8 arb made to<br />
highbr hudqtmrtbro.<br />
2. Inrpector General Te8t (IGT)-<br />
Thir tart 18 an bxtract of the GKSPET with limited rrrnpling. It<br />
i8 deeigned for ~80 by the XC on hi8 Mnual in8pbCtiOII. here tart8 are<br />
‘gradad by the Marine Corps Inrtitutb tnd rtrultr arb nuilbd to thb XC<br />
and unit in8QtCtbd. The r"QOrt rbflbCt8 Unit QWfOlDWCb and CCmtQarb8<br />
the unit with other typo unit8 end thb Marine Corpr norm.<br />
38<br />
-- _-__. . -- _.._._- --- - - --- _.._...- -_- _--. - .-<br />
.<br />
. .<br />
.<br />
,<br />
-.<br />
c<br />
.<br />
.<br />
.<br />
.<br />
:<br />
- _.. . _ . . . r . ( .. -'.. . - -. * _ .._m __ ,__ __._ _ -.--- ---- _.- - . -I .--<br />
.<br />
.<br />
.-<br />
.
: .<br />
A<br />
.<br />
.<br />
.<br />
3. Officarr’ Administrmtivs Subjects ExamlnatFon (QASI?~<br />
The purpose is tolwtivaCe avery company grade officer to faxitfarirrtlm<br />
with a varfrty of edxafnirtrativo typo eubfectt. Every officw<br />
muat t&a thm aroadnctfon, re a Lieutenant and again ae (II Captain.<br />
Soccurful cafspleticm of ME is not adatory for promotion but arch<br />
officrr baa to continue to take the cxmdnatla tech peer until ha<br />
ruccesrfi~lly parrrr it. The test itsar are eelected by CL board of fiald<br />
grade officero and acre&led into a te8t at WI. 3-m temt8 are grsded<br />
by m and 8 r8port 18 8ubmitted to\XArine Carp He8dqUarter8 which in<br />
turn publirhrr the rerults and direct8 an rpproprints entry in th8<br />
OffiC8r'8 jACk8t.<br />
lor thir 8~mtp08i.um, tvo pertinent conclu8ioa8 cw b8 mad8 -- 1.<br />
Althargh th8re are rlgalficant difference8 in th888 progr8m8, th8r8<br />
oboiour~y 16 con8td8rable CanauMlfty fa the U8es of th8 tcrto. 2.<br />
Purth8nmre, th8 baelc goal of th8 programs of all tbs rervfcco c8n b8<br />
srpt8408d, in a gtnrr81 v8y, ~8 being ured t0 provld8 mU8ure8 Of thr<br />
extent to uhich wincer have nmtsrad the rignific8nt aEpact8 of<br />
thrir milltorp 8peCi<y.<br />
! : 39
I<br />
.a ,<br />
I<br />
’ !<br />
; * .<br />
.<br />
. . .<br />
.<br />
,<br />
Am!t-<br />
NAYY<br />
AIR FOt Cl!<br />
COAST GJAxD<br />
zi<br />
TEST<br />
1<br />
Mm!&<br />
‘IEST<br />
.<br />
. * .<br />
AWARD PRO-PAY<br />
WOS YERIP ICATIOW<br />
FFuxYrIm QuALxFICATmN !SJBJECT-tL4TTKR SPZCIALISTS<br />
EVALWTICH SCORE SC~RR AT SERVW? SCHoOts AND<br />
PAY GRADE 6 tfOS<br />
Dl!mmma’L?rn<br />
IDEHTXFY TRAIMING NEEDS<br />
CYI’fiER AREt INSTALUTICNS<br />
PEW-E FACTOR<br />
SUBJECT-IUTI’RR SPECIALISTS<br />
x.mcTx OF SERYICL PRm(xp Assn?zED To FUVAL<br />
TIXKINRATE<br />
NUMKR OF AWARDSIIt<br />
EXAtUMI!G CFXTER<br />
TRST<br />
UI’RER FAcfCRS<br />
TZST<br />
PERF-CE PACKtR<br />
mm OF SKRYfcE<br />
-liiEZYXEARD,J<br />
SUBJECT-HATTKR SPECIALISTS<br />
AWARD OF M sPEcIALxlp TDY TO 6570th WRSmL<br />
RKSEARca IABCRATORY<br />
PRmoN SzmcT-#hTrRR SPECIALX~S<br />
PIMXWTIOR @MST)<br />
TEST IDEmIEY TRAINING<br />
NEEDS (CXSPE’T) SuBrn-PATrKR sPEcr.hLxsTs<br />
CYl’ER FACTORS<br />
nvsPEcTxoN (ET)<br />
tiOTIVATIfXl (OASR)<br />
. . . c<br />
.\ -_-_--<br />
, .<br />
. *<br />
I<br />
. .<br />
I
.<br />
.<br />
.<br />
Conctpta for Exploration in<br />
Prof iciepcy tieaaureraent<br />
EfICHAEL A. ZACCARIA<br />
Lackland Air Force Base, Texas<br />
XARVIN KAS’<br />
Randolph Air Force Base, Texan<br />
Roth the keynote speaker and the cbairam of the tboretical aympoaim<br />
on approaches to improve measuremat have asked that ve develop C~ccpte<br />
for exploration in proficiency mBaaurcment. Our theoretical sp~~osi~m<br />
chairman baa indicated that vc can gain very little by merely polishing<br />
up our present meaauremnt techniques. Thus, any approach to a real<br />
improvement in proficiency measureuxmt muat be a radical departure, at<br />
leaat in aow vaya to the conventional concepts presently applied in<br />
this area. The traditional appmach has atereotyped our thlnklng to such<br />
I an extent that ve are primarily concerned vith marely ru Lning and velidating<br />
proficiency measurement devices. It might be vell for ua to bc:!r.<br />
this discussion by reappraising aoaz of the concept8 that have ‘
..- .<br />
-*- .,-<br />
‘. _<br />
.<br />
When the matn objective is to use a umaoure to select or screen<br />
individual8 for a .program, there ia no question concerning the value<br />
of a norm-referenced syatcm. On tha other hand, the question arises<br />
ar to whether we should group achievement measures in order to con!4 up<br />
with one score.<br />
()ur program chairwa has stated quite tuccinctly that in order to<br />
mke real progress in thir realm of proficiency evaluation, id, .&et<br />
begin with the job. We are definitely in agreement vfr.h this comaznt<br />
end wish to present some concept8 of how this r,:$c be done. Sa<br />
individuals have approached thte task by :L+ use of 6upervi60zy ratings<br />
as an ultimate criterion on the ow ‘jtrd and the development of tests of<br />
proficiency on the other. ‘31 approach has been one of intercorrelating<br />
the teut batteries ar.d crl:eria thuo deriving optianm prediction weights<br />
for each test.<br />
Thrs courm of action ray vell remind us of the story of a team of<br />
prychologirtr baLrg called upon to develop prediction devicas for a large<br />
company. After considerable work on these predictore, thio group wa8 &la<br />
to predict quite highly with a large battery of temts the criterion which<br />
war based on rupervirory ratings. The paychologistr, being dfasatirfied<br />
vith marely being able to forecast there ratings with naariy perfect<br />
accuracy, decided to do mme further research and to factor unalyze the<br />
criterion and predictor variables: Lo and behold, the analytlia being<br />
complete, they found that they were Pearuring job saniority.<br />
Validity of perfownce amaouree riced not ba related to another<br />
criterion. These could be ultimate criteria in and of themselves. Let<br />
UP ruppose that we wish to ueaaure the performnce of secratariee. The<br />
first thing we should do ir etart vith the .job. We should enumerate the<br />
taoks that she doar in terma of product-lfke performr.ice. A secretary,<br />
for example, may do the following: type letters or finished copy fros<br />
draf te , take dictation, file correa~ndance, #elect mterialr from filer,<br />
receive vfstoro, and answer inqutrfer over the telephone. From this<br />
brief description of the recretarial dutleb, we can quite reudily develop<br />
uome of the following criterion perfonmnce measure#: We can wasure her<br />
speed and accuracy of typing, her rpeed and accuracy of taking shorthand,<br />
and her ability to transcribe her shorthand notes, her abill y to sppropriately<br />
file corrasporxience and to find mterials filed. Her ability<br />
to anmrcr qumfer on ths telephorm may be somevhat mre difficult. On<br />
the basin of each score that rhe maken, we can clarrffy her in terma of<br />
ratisfactoty. unratiefactory, end outstanding.<br />
The criterion for satirfsctory on typing, for example, mfght be an<br />
average of 40 wordr per minute on a cartrin specified copy vith no more<br />
than two error43 per page for a two-page amnuscript. Outstanding on thir<br />
charactari8tic mfght imrolve 60 words per minute with no mre than one<br />
error on a two page manuscript. Sfmllar stemjards could be worked out<br />
for each of the other criteria.<br />
. . . - _ _<br />
.<br />
.<br />
_ _.<br />
.<br />
_ .-<br />
42 .<br />
I<br />
-.<br />
c<br />
i<br />
.<br />
. . .<br />
.<br />
.<br />
:<br />
* .- ‘, .’<br />
.+<br />
. - . - ,.,- i... .-_.. -...<br />
__ _ _. _. - - - .- . ._ -_- . _-^_-dA- _______...._ - ^-.- .--. - -.<br />
. _<br />
,<br />
.<br />
.
Here; UC are trying to make acveral pointa. k’s ten define and<br />
develop criteria in term of certain relevant on-the-fob bchsviora,<br />
Each of the&e different types of behaviors coo be rrreaaured separately,<br />
and they would be made more meaningful if they-were aeaeured in terms<br />
of relevant requfrements. It Is quite clear that 8oa.a secretarial jobs<br />
may require different: degrees ‘of skills. Individual skills could more<br />
readily be matched to required job akills. There is no need to coc&ine<br />
all of these acorea into one score and then further putting this score<br />
into a rtandard score or percentile conversion. This obscures the data.<br />
Purthermrc, there is no need to correlate each of these scores<br />
or obtain a weight for these scores by optimlly correlating these<br />
againat auptwisory ratings. After all, it could lfkely be that WE<br />
throw out the baby with the bath because the auperviaory rating may<br />
not be baaed on any of the charecterI8tIca that are relevant to actual<br />
job � ucceaa . We have all at least heard of cases in which aecretarlaa<br />
were rated outstanding by the boas even though he has never deen them<br />
do meny of the taaka typically required of a secretary.<br />
Teat Reliabil1tp and .Accuracx<br />
For the purpose of this presentation, we‘have discerned three types<br />
of teat reliability. The first type deals with the cqufvalence of two<br />
or more form of a teat. The second concerm itself c:Lth the stability<br />
of the seam teat given at two djfferent timor. Youmgenelty or sameness<br />
throughout the teat ir the third kfnd. There 1: w question as to the<br />
importance of each of these types of reliabflfty .Y connection with<br />
aptitude and peraonaiity testing. A question does .clae, however, as to<br />
the appropriatmeaa of atabllity and hamogtneity coe...cienta In achievemant<br />
testing. There ,La no doubt a8 to the importance of equivalence of<br />
form for a particular achievement teat. The present author8 raise a<br />
question on how one determines the equivalence of hro fonaa in a training<br />
situation.<br />
In the ideal training situation any individual or group of ‘ndivduala<br />
would theoretically score zero on any test at the beginning of training<br />
and LOO percent at the end of training. While it la true that few, if<br />
any, ideal situations of this sort ever alat, it is important to take<br />
the ideal situation for conaidcration. In an ideal rituation such as<br />
aamtioned � bwe, If we used end-of-cmrae achievement teat as a measure<br />
end used the traditional coefficient of correlation as the tool to<br />
determine the degree of equivalence of two forma of a teat, we would<br />
find that the correlation coefficient is not very high. E%en in a amwhat<br />
less id-al situation vhere ve have acorta bunchlzg up at the low<br />
end during pretesting and bunching up at the high end at the end of the<br />
course, we would not obtain a very high reliability coefficient by computing<br />
it based on only poatttat acores.<br />
43<br />
I
. .<br />
-<br />
/:<br />
i<br />
.<br />
. --_<br />
- _ _. _<br />
.<br />
-. -. .-- . . ..-----...-;-.---.-<br />
_-...II -.--.- -_.. _.<br />
our proposal is, thus, that if we muet hnve a correlation coefficient<br />
:o indicate tha equivolansa of .two forw, that data be accumulated<br />
at the cad of vatloucr portiona of P muroe of ins*.ructioa. ‘fw0 forlao o f<br />
a test could be adAniatcrad :J 80~x3 indi~iduid a8 a pretct6t, t0 othar<br />
Individuals ,durfng the courcc of itwtruction,and to othere at the oad of<br />
. the COUTC~. In this mannat, a correlation coefficient would shw the<br />
true relaL:on.ship between two forms of a test rind would thca r,ot be low<br />
due to restriction of a tent and would thus not be low due to restriction<br />
of range when students arc vcll taught.<br />
Another posrfble manure of rslisbility of a teat would be in terms<br />
of accuracy vith which it vaasures that whfch it is suppoead to umiure.<br />
This ir closc’ly akfn to relevancy azA validity of zmmuremmt and vi11<br />
be discussed in the n)rt rcctioa.<br />
Tart Validity and ValKdation<br />
There are a aumber of crroneoua ccnceptr concerning achievmmznt taatfng<br />
which rhould be clarified. when we speak of t~hievment or proficiency<br />
tedting, we are, in emenee, dircusrfng a criterion. If thfs bo the ease.<br />
vhy do vc have to wonder whether our teat predicts job nucccss? Wfliltl i t<br />
is true that there-are more than just job comgomnta ta the sumesaful<br />
psrfmnce Of 8 job, if WC can 8t loAnt ACcurat6ly nremrure the VariOUII<br />
cmpments of a job, we will have gone a loa vay toward thu actual<br />
devclopnt of a crfterior.<br />
There are those who feel that tsmy echlcvczmnt or pcrfonmncc teat8<br />
must correlate highly with an ultinrte “criterion of job 8uccem 08<br />
dctsradned by supe~~sory ratings.” We have already -de our point<br />
concerning this aspect. If we have szaeured a seczetary’a proficiencies<br />
in trF& of speed aud quafftF of typing, .shorthand, and fLling, we have<br />
already gona a long way towards umaruring her total job performance,<br />
although we nay hma naglectcd to rrrtaaure her initiative and mtivation.<br />
There tvo latter F?;jlzits, we vi11 agree, can better be measured by supervirory<br />
ratinga. We see no need, on the other hand, to validate the other<br />
measures by correlating them wCth � upenrisory evaluation of perforrcancc.<br />
, Ue do aot thfnk that the traditional practice of obtaining item<br />
total test correlations on achievement tests in very worthwhile, Sam?<br />
test developers seem to forget that this practice leads to homogenlafng<br />
the test, A practice which is generally acceptable in aptitude of<br />
parsoumlity test development. In the case of achievement testing, hwever,<br />
this practice has very limited usefulness. He feel that item<br />
validation could better be achieved by admiairtrat
‘a<br />
: .<br />
-..<br />
. .<br />
1 .<br />
. .<br />
. .<br />
_. . . .<br />
‘.<br />
.-<br />
. I<br />
---__ _ _,___ - --_- -- . _-__ i_ -___-___ .__ __._ _. . _ ‘_ -. -_ - - _-.. _-..- .-<br />
.*<br />
;<br />
Pf we aauot valldate echievcmcnt and performance teoto ngajnrt<br />
on-the-job criteria, certain conditions muat be preoent. Part of the<br />
on-the-job criteria murt. include oimulated or real taoko or job ample<br />
.s<br />
teoto. Each of there job criteria ohould bt measured and ‘deoignatcd<br />
reparately . The relevant rchool or trairiing criteria should � l8o be<br />
measured in a 8imilar manner and relationrhipa should be eotabliohad.<br />
If correlation8 are desired, they can be obtained by correlating the<br />
variour relevmit 8cor’co of not only those individual8 who 8UCCeS8fUlLy<br />
complete a cour8e of inotruction, but a180 of those who do not OuCCtEQfully<br />
complete, or vho go on the job directly without any training.<br />
:<br />
We feel that there ruggcstionr will open new avenue8 for axploratioa<br />
in proficiancy an’d � chieveznmt evaluation. i<br />
.<br />
_<br />
Referenccr i .<br />
Prederick8en. N. Proficiency te5t8 for training evaluation. In Glaoer,<br />
R. (ed.), Psychological research & training and educatCon.<br />
Pittaburgh: Univerrity of Pitteburgh Prerr, 1961.<br />
Glaoer, R. Instructional technology and the mcaourement of learnfng !<br />
ou tCOr888. Amer. PlychOlOgi8t. 1963, 2, 519-521. i<br />
Glarer, R. 6 Rlauo, D. J. Proficiency meaaurfmmt; � 88e88ing human<br />
performcncs. I n Gape, R. R. end Gtherr, Psychological principlea<br />
& ryotcm development. New York: Iiolt, Rinehart, & Winrton, 1962.<br />
- -<br />
Nedelrky, L. Abrolute grading otandardr for objective tertr. e.<br />
Poychol. Measot., 1954. lb, 3-19.<br />
Thorndike, R. L. 6 Hagen, E. Haaourement and evaluation & p8ychology<br />
fi education. New York: Wiley, 19557<br />
Tyler, R. W. Achievement tc-!ing and curriculum conotruction. In<br />
Williamoon, R. G. (ed.), Trcndo - in - otudent pcrronnele.<br />
Univerrity of Hfnnsoota Prero. 1949.<br />
Zaccertq, H. A. h Oloen, J. Reapprairal of Achievement Raaourer<br />
USAF i.:-tr. J., 1953. 1 73-75.<br />
,’<br />
4 5 i :<br />
‘?<br />
!<br />
!<br />
1<br />
� �<br />
i<br />
t<br />
x<br />
i<br />
i j<br />
j<br />
I<br />
.I<br />
:<br />
I<br />
I P---<br />
�<br />
i<br />
___-.-..<br />
.<br />
.<br />
, . .<br />
_ - .- i.... -__ _._ “_^_~ . .__ -._ _. _.^.____. “_. _. - . _.-- . ..- .___a __.__ _.-.- . .._..--.-_-. -.- “.-.^..I-- ---- -.A.- _ _._--_. _-_.._ :..,.-, -._- --.-. -.---. ._,-_._ - . _______*__ _----- . _ - . - -_--.-_ - -<br />
. t<br />
i<br />
-III_L-- i
--<br />
.r<br />
. .<br />
“\.<br />
.._-.<br />
-.<br />
.- -__.<br />
,<br />
.<br />
-.<br />
,<br />
.<br />
.<br />
.<br />
.<br />
. --_ ._<br />
-----. - _ _ ._-._ -.._ ._ _ ^ .- . -- ._. _<br />
Item ~nalyrie Information for Test Revision<br />
VERN w. URRY<br />
US Arwy Enlisted Evaluation Center<br />
The item aMly8is coding procedures used at the US Army Enlisted<br />
Evaluation Center represent an application of Cullikren’e item &leCtion<br />
techniques to MOS Evaluation Test redevelopment. These techniques were<br />
advanced by Culliksen (1950) in his “Theory of Mental Tests.” They call<br />
for the use in item anrrlyeis of the point-biaerial correlation coefficient.<br />
Ws are currently using thir statistic for that purpose.<br />
XtCcP analysis codes are provided to indicate appropriate actions to<br />
be taken by item writers in regard to individual HOS evaluation test items.<br />
While a full explanation of the 8 Codes used vi11 not be given, the present<br />
paper will discuer the statistlcal bases for theme action code8 vhich have<br />
to do vith the selection of acceptable item or the revision of items<br />
needing minor changer. The advantage8 fn improved measurement3 to be<br />
gained by the implementation of these particular action code8 will al80<br />
be dircusscd.<br />
Basically, item analysis has am its purpose the development of a<br />
test vith a given set of statistical chararteristica either through<br />
item eelectioa or item revisfoa. For a lOO-item HOS Evaluation Test,<br />
the deeired set of statistical characterietice vould include a mean of<br />
62.5, a standard devtatioa of 12.5, and a reliability of .80 a6 a mFaimum.<br />
The desired mean equally divides the range from a chance score of 25<br />
to a mxinarm or perfect score of 100, and the derlred 1taadard deviation<br />
io determined by dividing 6 into the above range, since 3 atanda:d<br />
deviations on c ither side of the mean cmtaine, for all practical purposes,<br />
a normal dictribution. The chance scora ir a function of the fact that<br />
each l4m Evaluation Tec;t Item contains 4 plausible alterridtives; hence,<br />
if OM were guessing 11’4 or 25 of the item would, on the average, be<br />
msvered correctly.<br />
The it- analysis action codas to be useJ by iteta writers take<br />
into account the dcrlred set of statistical characteristica for NO!3<br />
Evaluation Tests and the item statistics required to cbtafn HOS Evaluation<br />
Tart8 vith these atatirtical characttrirtics. Item rtatisticr relate to<br />
the statistical characterirticr of a teat in the following annner<br />
(Culliksen, 1951):<br />
.<br />
.<br />
46 ’<br />
--. . __. . ..-- - - -_-_--<br />
.<br />
I<br />
e .<br />
.<br />
.<br />
.<br />
.<br />
.<br />
.<br />
:<br />
.<br />
_.. ,.<br />
‘-. .t .<br />
^I. .^_<br />
.<br />
.._.._ ^-<br />
. ..-<br />
.-.-<br />
- -- -- -. ---<br />
.<br />
#<br />
.<br />
.<br />
!
f<br />
-.<br />
‘Ibotertfaoul,X:-Cp I<br />
.-<br />
.<br />
._.<br />
.<br />
(Formula 1)<br />
Where: pi, or item p-values, la the<br />
proportion of individuals<br />
anawaring the item cor-<br />
, ractly<br />
The ‘teat � tan&rd deviation, s, - Critai (Foraul* 2)<br />
The teat reliability, rm =<br />
Uhere: rit8i la the point-biearial<br />
item-teat correlation<br />
multipled by the itar<br />
etandord dsviatior~<br />
‘1 ‘JPi (l-pi)<br />
(Formula 3)<br />
Where: I: lo the numbsr of iteqa in<br />
tho toot,<br />
� � 1 the itaa variance<br />
si<br />
‘I - Pi(l-Pi)*<br />
tits1 la the point-biaorial<br />
itaa-tect correlation<br />
nultipliod by the itcap<br />
standard devir tfon<br />
In this ragcrd, ony itt-a action coding proceduroa that moues from<br />
an ito analyaia should taka into account the end produce, or HOS tvaluation<br />
Teat, which vould result, given the selection or revlaion of itwa<br />
to k included in a rodevelopod teot.<br />
In keeping vith a deaired toot mean of 62.5, all itrn p-valuoa � hould<br />
������� � rouod .325. For a doaired teat atonderd deviation, I:- p-values<br />
uhould also rango around a mdian value, I.o., a8 nmr .50 &I la practicable<br />
la view of the dealred mean, rlnce al - pi( -pl) and lo at a maximum vhm<br />
pi - .50.<br />
Ap-?<br />
The mlacted or acceptable items � o coded in the present procedure<br />
with codes OK or M (mia1nslly acceptable). Itana to be rovlmed, Code<br />
RV:, reach the HA level for item-teat correlations but either dirtractorteat<br />
cotrelations or distractor proportions indicate they need minor<br />
modification to be in consonance with tho � tatiatical characterlotica d<br />
doaired in tt0S Eveluetion Teata.<br />
, i I<br />
. .<br />
i<br />
I
. .<br />
.<br />
_. .- --\<br />
..-._-_- . .._<br />
;--. . :<br />
.: ‘.<br />
. .<br />
‘,<br />
(- . .<br />
.<br />
.<br />
. .<br />
I<br />
.<br />
‘._ *<br />
-. 2’. -.‘.<br />
. s<br />
_ _. -.- : .--..- .: -.._..- -- ..--;-A---- -___ ._<br />
.<br />
1<br />
I<br />
Distractor-test corralstions indicate revision is ctcessarp if they<br />
exceed the correlation set at the HA level. Distractor proportions are<br />
indicated for rtvirion if they fall outside a range fro66 a proportion<br />
of .OS to a proportion of .25. A minimum proportion of .OS wnd elected<br />
to provide that scm6 plausiboility exist in all distractore in order thut<br />
a theoretical chaacS6 scort.of 25 is teuablt. A msxinarn proportion of<br />
.25 wa6 btltcted becauat (1) all rc6pons6s, distractox and correct<br />
*<br />
altemativts, would have s proportion of .25, if examinaao had no icnm- .<br />
ledge regarding the item and (2) when the.p-values of correct alter- ’<br />
native6 approach .25, chance rc6ponses tend to lovar item-tsot correlation6<br />
(Gullford, 1956, p. 437). .<br />
. *<br />
. .<br />
Applied in conjunction, the above limits provide: (1) that :tem<br />
p-values lrvy range ‘bstween .25 and .85 which ie rearonsble in view of<br />
the desired man; (2) that the resultant test mssu is near the desired<br />
valus; (3) that wximm item standard deviation8 art obtained rince<br />
item p-values am msintainsd at andiaa values; and (4) that item p-valuer<br />
trt within ranpr wherein item-tart correlations art higher since extrexe<br />
or nonmedian p-valu66 for items tend to reduce item-test correlations.<br />
To iIlu6trata how Ponda 2 works in regard to the item coding<br />
proctdurtt, we can take th? txmple of ittw coded CX and HA:<br />
Given 100 item6 with item-ttrt corrtlacions of .30 (the tX level<br />
for larger NCS 8+6ples) and item standard deviation6 of4-9<br />
.48<br />
the test standard dfmfation would be:<br />
Ix - 100(.30)(.48) - 14.4<br />
With 100 item having itmn ta6t correlationa of .20 (the minimslly<br />
for larger HGS ramplea) aud i-em standard deviation6 of<br />
.48, the test standard deviation would be:<br />
sx - 100 (.20)(.48) - 9 . 6<br />
’ ; With item 6tleCttd or revised to txcttd there item-test correlation6 and<br />
to � pprosch these item standard dsvistions it in apparent that a daairtd<br />
test standard deviation is tmdt more posriblt.<br />
!<br />
‘lb illustrate how ?oxw~la 3 worka in relation to tbt item coding<br />
procrdurtr,wt can use 100 items with the average variance of (.625)<br />
(.357) - .23 and a test standard dwiation of 12.5. The miniatum tt6t<br />
reliabflity in thir cast would be:<br />
rxx - loo<br />
99<br />
l-.=&-.86<br />
In this msnner adequate reliability tends to be iasursd by the item coding<br />
procedure6 .<br />
48<br />
.<br />
.<br />
.<br />
i
-..<br />
:.. . ,<br />
/<br />
_ :.<br />
. .A.<br />
.<br />
.<br />
-<br />
---. *<br />
I<br />
.<br />
;<br />
. -<br />
i<br />
, -.<br />
.<br />
.<br />
Since &zm statistics ralata to the statistical charactcrtstfcs<br />
of tests as ,indiceted fin tb above formulas, it is further possible<br />
for test spclaliats to exercise more control wet thase aspects. This<br />
furth3t control fs possibla becPus8 each stetistfcal charectaristic of<br />
a redeveloped test can I32 estimated from Ftem ztatfsttcs which ~111 %e<br />
provided on each 140s Rvaluation Item Card. The sumatioa of these .Ltcm<br />
statistics and entry Znto the proper equations for the eatkstion of<br />
aech of the test statistics is thereby fccillated. Items to be included<br />
in a redeveloped tent could then bo varied in order to more closaly<br />
approach tha desired set of statfatical charactcrlstics.<br />
la sunstmry, action coding procedures tsnd to channel efforts in<br />
item selection and item revisfcn touardr HOS EvalustFm Tests of dosired<br />
characteristics; and access to pertinent item statistics on the part of<br />
test specfallsta vFl1 prwlde for a more accurate approzcinution of the<br />
desired set of statistical charactertstics when redeveloped tests are<br />
subsequently used tn evaluation tasting.<br />
Reference<br />
Cullikaen, H., ‘fieor]! ef F?ental T& Xew ‘York: Wiley, 1958.<br />
.I -<br />
49<br />
‘.<br />
!<<br />
,!<br />
�� �<br />
.:
. -. ‘,‘,.. . . ‘-’ _- _.<br />
1. *’ * _. L-.--- -’ ._--- c ._.__ ..’ -:.‘:.--2 .. - --.---_-.2.e__--P ,-*<br />
.--m--L. --. __.. -_ .- ..._I_.___ _______<br />
I<br />
I;.;. : .i<br />
, :<br />
I’<br />
: .-<br />
,<br />
. . ,<br />
L<br />
/;<br />
s’<br />
;.<br />
, . L<br />
,.-<br />
,<br />
Item Validity Informetion ee a<br />
Beeie for Teet Rtvisim.<br />
VERN W. URRY and EDWXN C . S’RIIKEY<br />
US Amy Enlisted Evaluation Center<br />
. -<br />
Xort of ue would pro’bmbly agree thet ~concepte such 88 tee; reli- .<br />
ability and content validity are important chuecteriotice of teote<br />
which ut ueed to evaluate ability or knowledge of -human befngm. All<br />
too often, hovever, there concepte are over-eanphaaized at the axpenee<br />
of a tart characterietic which ir coneiderably more fmportmt.<br />
A teet mey be perftc,tly reliable, yet mi&w~be measuring entirely<br />
nonvalid varianct, i,,e., it could bt measuring the wrong thing but<br />
doing it in e beautifully perftct vey. Content validity, very important<br />
for obvioue reaaose, ie occaei~nelly viewed by layum a8 the only kind<br />
of toet velidity that ie rtquirtd in Order to 18bel � ttet ne being 8<br />
good ant. Now the abrurdity of this point of view become8 mert apparent<br />
when one tnalyxtr the types of dtciefone that era going to be rnadt ebout<br />
tnlitttd men on the beet8 of their tart tcortt. Hart of the dtcieione<br />
are bared upon.an important aeeumption. The aeeumptioo made is that<br />
those individual8 vho acore high on the teat are performing better on<br />
the job (end thue should be rewarded), and those rcoring losrtr on tht<br />
teat art accordingly poortur perfonnere on the job (and hence ehould not<br />
bt rewarded). Whether th:Lo ir a true asrumptLon should not be left to<br />
cr:etal-bell garing or “ermchair” peychology, but � hould be subjected to<br />
verification. Thir vtrificetion mey be referred to as eetablishing the<br />
dtgrte of azpirical validity of a test, 1.6.. hw do ttst score8 actually<br />
related to en external mtesurt of on the .job pcrfonnenca? Ut et the<br />
Depertment of the Army ham rectrtly ‘incorporated e plen of v8lldetion<br />
for both the eveluetion taete ana the C-der’e Eviluetion <strong>Report</strong>.<br />
Detail8 of theta validation procedurea will be diecursed in mother paper.<br />
The purpose of thie paper io to describe � sptcific facet of the etudy<br />
which we cell tht “preliminary validity report.”<br />
The prtlimfnary velidity report ir Intended primsrily for the u8e<br />
of tt8t rpecialisto in the Test kvelopmunt Branch of the US Arm9<br />
tnlieted Evrluation Center. The purpore of the rtport is to provide<br />
guidelines in ttrt rtvieion bared on pertinent � tatfetic~l date which ie<br />
not aV8ilAble from our reguler ���� � rulyeie etetirticr. Thfr prtrtnt<br />
validation policy reprerente an improvement ovtr prevloue velidation<br />
offortr in that validity data covering the entire ttrt outline and<br />
indivfdurl test iteuu can be provided prior to teat rtvieion to allow<br />
fat their uee in the decirioa mrklng process of tort dtvelopocnt.<br />
,<br />
. .<br />
, - _-. _.<br />
.<br />
, ” *<br />
50<br />
---,- - . .<br />
._--_-_.
Teat reliability? -- Certainly 6OmC reliability 4-6 BlCCC66~~ in our<br />
test6 in order to obtain empirical --alidlty. Content validity -- again<br />
we era certainly concerned ,that itema in our tests should adequately cover<br />
the relevant arpect# of job behavior. Eut UC also feel thet there are<br />
only the minimum dcrsirable~ct~aracteri6tic6 thnt a test should have. Our<br />
earlier validation rjtudics asked the question: Doe6 this tC6t pO66e66<br />
anpirical validity? In the present validation procedure, thara 16 no<br />
lOtl@W M empharia !BOlely upon the qu56tiOIl -- he6 this tC6t pO66e68<br />
empirical validity7 -- but more directly urd more appropriatoly, the<br />
qUC6tiOKI is Why is the test valid? and Eov can it6 validity be improved?<br />
. .<br />
As noted in. th‘ra &k&ue paper, 'this one al;o reprer~tr ‘thi<br />
&pplication of itan selection technique6 advocated by CullikrCn in hir<br />
“Theory of Mental TlBSt6." Again, these techniques raqufrs the use of<br />
the point-birerial correlation coefficient for it--tC6t and ittsncriterion<br />
correlat Ions. The specific question v6 ark is, How do each<br />
of there itanr fnflrreoce the validity of the total evaluation test?<br />
The data provided for ita in the preliminary validity report of<br />
our validation sample inclu.de: (a) item p-values, (b) standard deviations,<br />
(c) itar vaclancco, (d) point-biserial itan-total test correlations,<br />
(e) point-birerial itan-criterion correlations, (f) item relitbility<br />
indexes, and (8) itcun validity indexas.<br />
The indexer of rcliability'ami'vrlidfty are obtained by multiplying<br />
the standard devfat:lm of each item by it6 rarpactiva point-birerial<br />
itenwtC8t correlation or point b18crial item-criterion corralatfon.<br />
They reprccrent the contributfon that the ftan maker to the reliability<br />
end the validity of the total toot, respectively.<br />
It is the last concept, the index of vrlidity that nw pO66e66e6<br />
special rignificanccs to our tc6t 8pcciall8t6. By inspection of the<br />
indexer of validity for each it-. they can detcnaine which itaru are at<br />
hart raeking a pori~cive contributim to the validity of the total tt8t,<br />
and inrure their inclu6fou in rubre~.uent te8t rcvirionr. If item<br />
validity indexer are available for each item that 18 includad in a<br />
test, it will be poerible to cetimate the total tart validity before<br />
it 16 � dmini8tered. The paper On itan Mdy8i8 hd6 irlicatsd hw telt<br />
-. .<br />
characteristics ruch a6 the mean, rtandard deviation, and reliability<br />
can be more closely ccntrolled. USC of the item validity index enable8<br />
US to exercire additional control over the mo8t salient feature of a<br />
tart, it8 ability to dircriminatc bctvrcn those lndividual6 vho 8re<br />
.<br />
good and poor performer8 on the job.<br />
51<br />
.
.<br />
.<br />
.<br />
.<br />
. -+_<br />
.<br />
. .<br />
*, .<br />
_ _ _ _ , ^<br />
.<br />
.<br />
.__ _ _.,_ _C..I---I.- -..-: .-.-- - .-._ ^ ._<br />
To fllurtratr how the indexor of validity for aach itaa lcflucoca<br />
the validity of the tote1 tart, consider tho follcwin~~ formlo,<br />
Ubrt4:<br />
ber thb validity of the<br />
‘xv tart,<br />
Zri,ot, the itma validity lndasi;,<br />
io tha point-bioerlal itemcriterion<br />
currolatioo<br />
multiplied by tho itrar<br />
rtmdrrd deviation,<br />
Bitei, ths itaP roltebility<br />
index, io the pointbioerial<br />
ItCE-tQot<br />
correlation luultipliod<br />
by ths itom otmdard<br />
doviotfon.<br />
Prom one cf our validity � ������ #the volfdfry eoefflcieot for tha<br />
tot81 test was .52. It wao found that the Cricoi two equal to 5.541,<br />
and Eritoi wab equal to 10.5’77. Entering the above equotlon YO find:<br />
‘w -<br />
5.551<br />
10.577<br />
Solving for the above, the rtV oqualo .525 which corraopondo to the .52<br />
which woo the volldity of th’o totol evaluation teat cmputod ftm the<br />
Powron product-nt formula. uoe of the indu of validity ptanotoo<br />
the � oundneoo of the “item-b,ank” concept,vhstetn numtouD prwfourly<br />
tried Itmoo pro readily wcsooible to the toot conotructor.<br />
Another urrful placr of information that vi11 bo provided thr teat<br />
� pmialloto lo a grophlc plot of the itaa-validity indexer and the itwe<br />
reliability indoxeo for each of the icmaa. One CM, thrroforo, tall at<br />
a glance vhich itano are contributing moot to teat validfty and taut<br />
reliability. Thio lo eooentlally tho oome kind of graphic dzrtration<br />
ahown by cu~likmn $n hio chapter on itm � olyoio.<br />
Row there ore the major changer in our volidotion � tudioo which vo<br />
bavr preomtly incorporated. Since Incorporating thio nmthcd, we have<br />
cuoplated or aeorly completed studier on � pproximtely oevon different<br />
mllitmy occupational opoci*litioo. Portunotoly, nearly all of rho teoto<br />
evaluated no far poorsrr � oam drgrar of aapirical validity.<br />
-- ___.-_--- _--~_ . .., __ .----.-...----- --. -___ __- . --<br />
.<br />
.<br />
52<br />
._ __ -.<br />
.<br />
;<br />
.<br />
c<br />
.<br />
.<br />
.<br />
.<br />
/<br />
:’<br />
\..<br />
’ :<br />
.<br />
�
.<br />
.<br />
One of the p~oblmnr that we probably all encounter in OUP taste me<br />
hiuh intrPcorrel4tionr baltween rubfact-matter 4~448. mture t o r t v~lidation<br />
affortr at USAF&Z will eneapparr further changes aimad of helping<br />
LO 4llOvfate thfr 8ittUtiOtl. !?88Qntirlly, -14 Are now plannin~~ a factor<br />
aulytic approach to telt. devrlopumnt. l-ho uhury4fin8P KwltlfoctoPl4l<br />
itam-snolytlc method has been ool6cted for thir purpoor. It 16 a method<br />
for sotfnrting f4ctor lordiago vithout the &rtai&bla tark of eauputing<br />
4 10+tx-125 v4ri4bls fnt,oPCOPP4l4tfOU WLtriX. A6 coon a8 ccxeputer prograplo<br />
4ro devolopod 4nd caarputor tio+o bocarnrr 4v4il4bl0, ths Evalu4tiou<br />
Section will rprt6mtic4l.ly bogin utr4cting group8 of itan that rhatld<br />
br rcorod tog&hat. We hlopa to obtain 8ovo~4l orthogonal rubto8t8.<br />
Tbfr approach h48 two m4for adv8ntagor for the Army. Pir8t, it<br />
8hould rerult i n an improwed diagnortic profila bec4ure mco~o8 uould<br />
tend to be independent of uch other end thu8 vould be awra meaningful<br />
and more e48ily interpreted. A profile tend8 to be of llrafted 000 if<br />
the vufour profile � ��� being 8COTOd are highly intercorrelated.<br />
Moxinnm utility o f profile .8cor48 demndr r414tiVOly unique varf4blo8,<br />
which m4~nr th4t e4ch variable OP to8t ue4 me4ru~4o 4 factor uaivac4lly.<br />
We are hoping that the Wherry-Wtnor approach till help u8 achiovo thi.8<br />
end. A second advantage fr thrt we till lncr6686 our knoulodgc about<br />
valid factora. If our Vo.riOU8 rubtest 4rQ uaivoc4~, the; masruram4at<br />
of tha moat valid foctorr cm be BlltphaOfhad which w6 hope will lead to<br />
wan gr6ator incPue68 in the validity of the tot41 ovalurticra te8t.<br />
In 8u11ana~y, the ~80 of tha prerontly incorpor4tod proliminrry<br />
validity x-sport and, ln puticul4P thr index of validity rhould help<br />
to inprove the urafulnrrr of our sviluatfon tolta and incPea80 the<br />
4ccurocy of porronnol docfrionr based upon thorn. Out long-rurge raoc&rch<br />
probleo till ba that of dotsnalning the porribility of obtaining orthogonal<br />
rUbtOrt vithln our 4VAlMtiOn te8t8.<br />
53 i<br />
1<br />
i<br />
f’ .<br />
1 1<br />
I<br />
---.--- .I.<br />
i<br />
i<br />
. /.‘..: i,<br />
(2 2‘<br />
.<br />
I
I<br />
.<br />
Guilford, J. P. Reliability and validity of ma&mre~. Facto? At;slysfs.<br />
~uu York: ~~Cr~-H~111, 1954, 470-538.<br />
Chilford, J. P. Validity of Mururewntr. Pundmmeai ctatirticr In<br />
paycholoay end eductrtion. Rev York; McGraw-Hill, 1956, 461-486.<br />
Gullikaen, A. Item eolectfon to cusfmixe teat vel:dity. Proceediaer<br />
of the 1948 invitet!loml confemnce 011 terting problme, Prinastoa,<br />
I. J.: Bducatlouai Tooting Sc~vIca, 1948, U-17.<br />
Wherry, R. J., md Wider9 B. J. A rcthod for frctotiog large nmbara af<br />
’ ittam. PspchaaatrilSf, 1953. 18, 161-139.<br />
-<br />
.<br />
54<br />
.<br />
.<br />
*<br />
.<br />
-._. - .- - - -__-<br />
.<br />
.<br />
. ,<br />
” i . i<br />
.<br />
.<br />
.<br />
.<br />
.
.<br />
.<br />
.<br />
The Prsdictioa of fob Proficiency<br />
ALEXANDER A,iLCNCO<br />
Staff Research Department of the Chief of<br />
Naval Air Training, Naval Air Station<br />
Menphia , . Tefmaoeee<br />
TM development of thia papar proceeded Qlong the line of b-craic to<br />
Qpplied rQaearch in the prcdfctlon of job proficiency. TtxImdlkQ, in<br />
Veraonml Selection” (1949). diatinguirhad between proficiency on the<br />
job per aa, and proficiency in job training. Only the latter type of<br />
proficiency ia conafdercd hero. Tvo araQa of roaQarch Qro cxamfocd:<br />
i.e., (1) b~alc re6eQrch on %otivQtion“ in e training emriroment; (2)<br />
applied raacarch on the predfctlon of school achlovemant. The firat QreQ<br />
involvca cozrelatiorml tcchniquaa which arc conaidrred to he Qn iatprwcd<br />
� ppro8ch to the araaurcmsnt of oonrptltude varfance. Included i n nonaptltudc<br />
variance, of murae, la a certain degree of Ymtfvatioa” variance<br />
vhfch haa been Qluaiva gentrally to amasuremant in a pure form. Howaver,<br />
aa we ahall ace, oum intereating thing8 can be dons in “wtfvation”<br />
maaauremant by the we of part Qnd partial correlation proccduraa. The<br />
� pplicQtlon of fnfonnation obtained from partial corralatlon procedure8<br />
to cndivfduafa by the usa of ratio derived scores wLl1 also be discuascd.<br />
l’tm aocond area involves the uae of multiple correlation and p866/fotl<br />
frequency procedutaa to ottain the beat prediction of technical training<br />
perfonaance. In thie regard, variour aclectlon md training grndea were<br />
866e8aad for their potentfal to 3 edict final trainfng performance.<br />
Operationally, thQ6e tvo approaches to the prediction of fob proficiency<br />
may be conridered to overlap, however, as vQ rhall 6Qe, that0 arc certain<br />
diatinctionr betwaca them vhich justify their indtpandtnt conaLdtration<br />
to the m6aaurmccnt of job p,rofLcfency. Several studier conducted by<br />
the Stuff Rercarch Dapartmmt Qt the Nav81 Station, Mnphir, on thcro two<br />
areaa will form the aubjcct matter of thfr paper.<br />
1 would llka to outlima the content6 of tha paper at this time in<br />
order to give you the focal pofnta of the mtcrlal I will be prorenting.<br />
I<br />
fie Hcaauramant and prediction of Hotivation<br />
A. Residual Cain<br />
B. Ratfo Xndex<br />
C. Motivation NQmsuramcnt<br />
D. Application of the Achievaaont Index (An applied<br />
rtasarch study)<br />
If The HtaaurQmQnt Qnd prediction of School Achisvannnt<br />
A. C,orrtlatioMl prediction<br />
B. Paar/Pail Odds to Prediction<br />
5s<br />
.<br />
,<br />
.<br />
. !<br />
i .<br />
I<br />
_<br />
.<br />
.<br />
1:<br />
. I
-IThe - Mearurc?ront and Prediction of Motivation<br />
By way of introduction to the rtudiar on the marure&zmt of moth-<br />
. ’ vatfon tvo 8tati8tical concepc~ nerd to be defined co fit our context:<br />
residual gain and actual to predicted ratio.<br />
.<br />
A. Reridual Cain. “Residual gain@’ ~88 dercribtd b y DUBOIS .in hir<br />
text on ‘Hultiv8riate ~relatjonal Analysir” (1957) a~ tht rcrldual<br />
vhich rtmtin, vharc the variance, vhich an lnLtia1 8core ha8 in CQmnOn<br />
vi th the flntl score, lr partialed out of tha find 8cort. Actually,<br />
Tborndibt, Brtgmrn, Tilton, and modyard in a text on “Adult Laarning”<br />
(1928) are credited for the firrt research employing reridual gain.<br />
Hirtorically, “re8ldual gaie~” va8 developed and used in a context of<br />
gain in learning. The clarric que8tion in laarning rertrrch has been:<br />
What are the correlate8 of learning?” This a88ume8 of cour88, that<br />
learning LJ not to be confurtd vith intelligence, � chievtmmt and other<br />
factors for which ttrtr have been devfrtd. mutver, several traditional<br />
probltma have been arrociatad vith finding the correlate8 of learning:<br />
(a) Aov can g8in be txpretmd to avoid metric difference8 betvren initial<br />
and final mearur4mtntr? (b) How can vt hold cormtent the known differtnct8<br />
among Individual8 in learning (or gain in achievtmsnt) in the fiM1<br />
msnrurement? a,& (c) Art the usual lw corrtlationr among gain eteaJUrbJ<br />
due to the multiplex nature oE gain or perhapr due to inadequate maJureJ<br />
of gain? “Reridual gain ,” derived by part correlatton techniquer, appearo<br />
to provide Jn Jdequate solution to each of there problem. hi8 may not<br />
sppesr relevant to the mstter in qwrtion here--nmely, tha mesourmmt<br />
of motivation in a training tnvironment. Houwar, the evoiution of<br />
“reefdual gain” AJ an index of learning ability va8 df*.tctsd eventually<br />
toward the w88ureumnt of cwtivation by a minor change in the baric<br />
dtrign utilized in the correlation81 rtudltr on learning. Thorofore,<br />
in order to trace the role of 8’remidual gain” in rsrearch on uativation,<br />
it fr necerrary to digrarr a little further on the nature of<br />
tBresidurl gain” itself.<br />
The baric paradigm utillittd in rtuditr on ~‘rteidual gain” fnvolv~~<br />
(1) pre-tertlng of achievement of rkill in a particular area. (2) Subrequent<br />
to thir, a given block of related inrtruction 10 introduced<br />
axparlrmnt8lAy, or ir provided by a technical rchool (~8 a logical rtquence<br />
to the pre-test itself, to the circuiwtancer dictate. (3) Lastly, a<br />
fl.nnl grade ir obtained for thir given block of instruction. The problam<br />
prerented here is to � ����� the degree of learning that took place<br />
behmen the pre-test and post-tart. lhir can ba done reveral different<br />
way8 (f.e., crude gain, X g,rin, etc.); however, “re8ldual gain” appurr<br />
to be a more oatlafactory rtatirtic of gain Ln 1earnLng.<br />
.<br />
. .<br />
. .<br />
.<br />
56<br />
.<br />
.<br />
.
.<br />
The basic fomula uhic:!~ generates the part correlation between<br />
“reoidwl gain” and a third variable is as follow:<br />
r(2.1)3 -<br />
‘23 - r12r13<br />
Where : 1 - Pratcot<br />
2 - Poottest<br />
3 - 0xridr V6rZable<br />
The boric fotlsuia vhich generate6 the fntercmrelction 5etwcrn cry t v o<br />
rrtr of residual mores, i.e.. 22.1 end 64.3 16 � 6 follour:<br />
‘24 - ‘12=14<br />
r(2.1)(4.3).- -<br />
- ‘23=34 + ‘12’13’34<br />
- r2<br />
l 34<br />
Where : 1,3 are inftial scored on two<br />
different test6<br />
2,d are final scores on tvo<br />
different te6 t6.<br />
Thf6 bring6 W CO the point, vhfch I indicated earliet, regard-w the<br />
tolr of “re6idual .-.6in” in ?ixotLvrtlon research. The! formulas ju6t dlscwred<br />
prWid6 a technique to divide the measured v6ti6nca fn rkil]. or<br />
echiwetmmt into two pert6: a part which is completely predicted by<br />
msrruted skill or schievement obtained at BOWi CCrffet point and & part<br />
which ie unpredicted by the e6tliet w66ure. Due to the mtute of the<br />
v6ti6bleo 6nd the baeic p6radfgm outlined above, the logic of the ritwtion<br />
leads ub to posit that the unpredicted vati6nce of “rClidu61” in tepre-<br />
6entative of “g6fn” in lermlng. Houevet, if instead of uring an achlevcmerit<br />
or skill mSa6uru a6 out pte-test we u6ed en eptftude utersute, then<br />
the logic of the situ6tlon would 146d u6 to porit that the rcrlducl fs<br />
more rynonymous with %mtivatfon” thsn “gain” a6 described 6bwe. ThU6,<br />
nothing has ch6nged except the n6ture of the pre-te6t. The vrlue of thir<br />
residu61 e6 a predictor and criterion both in the measurement and prediction<br />
of wtivation rill be illU6trated below.<br />
1. Retfo Index. Iha second concept of importance in our treatment<br />
of motlv6tlon tsseatch 16 the ratio of actual to predicted achievement.<br />
This hss been reforrcd to popularly 66: wer6chfcvementf underachievement ;<br />
.achievement index; dF6crcpancy 6core; PAQ (preparatory achlevemnt quotient)<br />
at cl.<br />
57<br />
.
.<br />
-._<br />
. .<br />
. .<br />
- .<br />
.<br />
. *<br />
Duboie (1957) hor rlhovn thet, ‘*e properly conetructod retie (ouch<br />
u that described ebove) te eaecntlelly � opeciel cane of a reelduel,<br />
i.e., the velue in the rmrnoretor vorfeble lees thet pert wkfch fe cortolated<br />
vich the denanioator variable. Accordingly, a rotio � hould<br />
corralaza .OO with the dcnaminattir verieble.” Thin hoe practieel<br />
� ����������� fa that ue ten perfom our rcteaerch in tha form of port<br />
and pariiel corraletion technlquae, end utiilee the results, if lndlutad<br />
by the dete, la the form of a ratio Index. Traditionally, thfo<br />
index has been derived by dividing � atudent’e, obearked performance<br />
(i .a., any training grede) by hie predicted parformeaco. One could<br />
divide by � etudent’e rw eptftude grede, but it ie coaoiderad to be<br />
8 further reflnmeeat to luee e predicted score derived on the beeie of<br />
OQ@O � ptitudr grede. In � lthar ceee. the reeultent index 18 untorrelated<br />
tith � ptitude (i..., .OO). Thur. rlmller co the pert correletlcm, the<br />
retfo epproech dlvidec m trelning grede into tm pertr; thet verfrnce<br />
rrhich fe predicted by rpritude and that verience which is unpredkted<br />
b y aptitude: (i.e.,the achlavemeut index). 7714 ochiavemaant index<br />
will be 1.00 or < ) 1.00 dcpandfng on the megnltudee of the cmmrator<br />
end denmfnator � ������<br />
It me7 be benoflcfrl to clarify two uuttere at thie point in order<br />
ta define thlr index properly. Tha achisvaneat index 1s oiraller to the<br />
“reelduel goin” AS we Fndfcetsd. The former meeure ie indicative of<br />
"mot ivat ion ,I’ end the letter, of “gain” In leeruing. One rae7 Infer<br />
thet both maneuree Ore “pure” indices of motivotiou end gain in learning.<br />
his ehould not be � eeumed *inc.> they era both roeiduole repreeenting<br />
verieace unexplelned by mserured � ptitude on the one hand end athlsvement<br />
on the other. Dubole In “Correletlonel Analyrie i n Training Reeeerch”<br />
(APA Peper, 1956) and Hayo in “PAQ Kenual” (e CNAl7XHlXA Reeeerch <strong>Report</strong>,<br />
1957). rtepactlvaly, indfcete that each residue1 lo not � “pure” meeeura<br />
of Rein or motlvetloa.<br />
The � econd problem often encountered concerue the predlctLve<br />
cepeclty of the � chlevcmmt index. This cm ba viewed in tvo ueye. On<br />
the one heod, � trefnlag ;grede , � e I indiceted, emi be divided into tw<br />
pert8 which are uncorraletad vith � rch other. Therefore, in � correletlonel<br />
� enee, eech pert connrrlbutee uniquely to the prodlctlon of an<br />
outside criterion. Houevcsr, rfnce these two pert8 era breed oa tha aeae<br />
trrlning grade, tt � hould not be � rmmed thet the cwbiood prrdlction<br />
by there perte will be grtrrter then prediction by thet trelnLag grede<br />
iteelf. This vae diecurecd in � reeanS f3UTECHTR.A rorcerch report by<br />
-go titled “An ApprLerell of Retie Scores.” Date on thlr mtter ir<br />
conteiaed In Tcible 1. TV0 rchoolr era reprsrrnted. It ten be rem thee<br />
the � chievcsnt Index (elno celled PAQ) dose not add to the Multiple ‘1”<br />
AS coutpered to the rfmple “r” vtmn uring t h e training grade clone. Thue ,<br />
without PAQ the OLmple r’(i wre .761 end .767; vlth PAQ, the Multiple<br />
B’S vere .762 end .763, rcepcctfvely. However, there ere other prectieel<br />
_ ,.. -- ---- . . .-__ ^_. ._..- __ - -- .._-<br />
I<br />
.<br />
.<br />
.<br />
58<br />
.<br />
i<br />
‘.<br />
/- ;<br />
I *
.<br />
- -<br />
.<br />
.<br />
.<br />
. *<br />
.<br />
i<br />
’ f<br />
“. . z<br />
~ -<br />
. .<br />
. I-<br />
. . .<br />
.<br />
t ”<br />
. .<br />
.<br />
. .<br />
-, 1. _.- j .-._ _.. -.A-- -<br />
,<br />
.<br />
reamn8 for dividing a tr~fning grrdr into tvo partr. Tt~ir iavolver the<br />
a$e of the achievmmnt index for dlagnortic pwpooer end will be tremted<br />
in beetion D to follow. -<br />
.<br />
Teble 1<br />
A Comparfson of Simple and Hultipla Correlstion G~tnp<br />
the Ratio InGax for Am(A) end ATR(A) School8<br />
-Ma(A) ml(A) (414) .767<br />
PAQ-AFU(A) ATR(A) (414) .763 '<br />
,c. Motivrttioa Xoamrownt. Earing defined out working coneopts,<br />
of “reoldunl grin” and uchlrvmaent retie, I would !ika now to present<br />
eme rsrearch findlngr dnvolvi~g the me of both rpproechar in aotfvation<br />
reararch. The rerurch VB# conducted by the CXATRCRIIU Strf f Reaurch<br />
Depertmnt located at Hmphie.<br />
1. PAQ Wmocrl. Iclyo, C. D., ClUTgCRTRA Rereuch <strong>Report</strong> 1957.<br />
a. Ratare of AchiovrPont Index:<br />
me firrt quartioo roleter to the ruture of thr achlevemont<br />
index. A# wo fndlcrnted, the logic of the paredigm vhereln eptitude<br />
ir partialed out of a trelning grade luda UQ to hypothuine that the<br />
vuiance rmainfng ir related to motivotioa OT effort-put-forth. Of<br />
comae, �� � lro lo rocogaited that othor factora contribute to thlr<br />
unpredicted nrlmce: 1.0.~ error, parromlity dffferencer, lnterert<br />
end � ttltudee, and pteviour � chlovmeat la � porticulor orea aad ability<br />
not auusured, � tc . l%r tuk, tharefore, ir to detormino uhat axteat<br />
motivation ir contefned in the reridual vuianco.<br />
The method edoptod to roroloa this quertltm was tho u8e of paat<br />
r*tlnge. Tbo clerric UBO of peer rrtitgr vu employed uhereln clrrres<br />
of 15 to 25 uen wro erktrd to cuminate and rank thrso rtudaatr In their<br />
59<br />
.
. . - -._<br />
.<br />
.<br />
.<br />
. . a<br />
- -- ..-. - _. _.. .-_._ .1 ._-.- ---- --.--- - -- __._.. ..__ __.. -___. ___<br />
roctioa who vero trylag the hardert to meter the courno and, vice verea<br />
for three mn vlro vero trying rho lwrt hard. Thhs objective VW to uma<br />
aa early � chlweaeat iadut to predict effort en � �������� by peer rqtbgr<br />
later in training. (Ttm rooulrr are tontaiaod In Table 2u.) The carrelationa<br />
of them two mumura~ range fraP .17 to .40 for four tcchalcrl<br />
trdning ochoolr and are trll rlgnificant.<br />
trblr 2r<br />
Correlation of Achiovemmt Index vith<br />
Paor Rctiagr on Effort<br />
Avfat loll School w r<br />
1. h(littO biOCh&AiC 166 .40<br />
2. Structural kkchmic 161 .U<br />
3. glectrmico Technician 209 .17<br />
4. Training Dovicemn 266 .39<br />
.-<br />
Since peer r-.tlngr th.melvee ue not imum to haio effactr for<br />
rwaon~ � uch � # intelligence or clorr achfavment, further InvoatigrtLm<br />
wu made to remove the effect of rrtLngo on InteAlfgmao frar the<br />
correlotion of rating on rtffort wd the � chievumnt index. Ttlir was<br />
done on A ample of lb6 rtudmstr in the Engine ktachanic Schwl at ntnphie<br />
(cf. Toble 2b). ;he rowkant pertlal corroletion VU indicated to bo<br />
*lb. When compared to the original correlation of .40 for thoro #am<br />
rariablsr vo cun 8~ that halo did lnfluenco ratinge oa offort - but tho<br />
portlol corrolotion vu #till � ignlficont of tho .OS 1~01. Uhilo mro<br />
roroorch nudr to be conducted In thle � �� tt can bo raid that ea~ of<br />
the vsrianco In the achievement index ha8 been idmtiflod 01 bring<br />
related to offort.<br />
.<br />
60<br />
.-..___ ._.- _. __ ._ -.. _<br />
. *.<br />
.<br />
e<br />
.<br />
. .<br />
.<br />
.<br />
,
--<br />
1<br />
. *<br />
__-_..___ __.-_.- I .<br />
� �<br />
. .<br />
�<br />
._ _, _.-_l ____- -..c--..- -_.--- . ..-- .___ll-l.<br />
Table 2b<br />
Intrrcorreletkms kaong Pear Rating8 on Effort end<br />
lntelligmce end Achievmmt Index<br />
1. Peer Rrtlngg--ori Xntelligenee<br />
2. Peer htingr 00 I!ffort<br />
3. intelligence (#my CCT)<br />
4. Aehievaent Index<br />
b. Stebilicy of the Achievasaat Index:<br />
1. 2. 3. 4.<br />
- .66 .46 .46<br />
e .29 A0<br />
m .lO<br />
The � ecxmd quution invclvea the relfrbility of the<br />
ehievement index. fafcrrmti~ on thle quartloo. ~08 drawn from enother<br />
� tudy Involving 196 8tUdMt8 for the Aviation Structural Mechanic School<br />
at Kemphis. Correlations wre obtofnod brtww tha achievePant index<br />
derived urly in training end at the end of treiaiq in two 8chooli.<br />
h e COlTOl~tiCm8 cbtelmd wro .69 and .35. fii8 indlcatu a noderate<br />
atability in the achfevtnant index. At leeat, it indicator thet the<br />
re8iduel fr not compared entirely of random error.<br />
2. Uotlvation Uea8uranmt. tteyo e n d Umnning. l!ducetiaul and<br />
P8ychological Hsrrurazmx. Volume 21, #usher 1, 1961.<br />
lhir rtudy invertl~.&tul enothor bark arpect of the � chieveeeat<br />
index, nrrnoly the prediction of � rubrequent � cklevaent index by e<br />
orriety of motivation meaoure# Including en early achlevment index.<br />
Actually, the echievanmt index in thlr inrtanco took cho form of a<br />
part corroletion fnrtud of e ratio of obrerved to prullcted achievemmt.<br />
A8 we indiceted before, there WO 8tetfetiC8 � re 0888ntldly<br />
rimiler.<br />
The mocLvrtfon meaeure8 uaployed wore:<br />
a. Peor retlng cm offott<br />
b. Self r*ttngr on effort<br />
C. Chock llrt o f rtudeat behavior<br />
d . Pietorte Nearurea<br />
-._ ----- ----I’<br />
.<br />
,<br />
61<br />
.<br />
, .
.<br />
.<br />
e. Grader in an urlfer couree wizh aptitude rauoved.<br />
(Thie is the reeidual which ie basically einilar<br />
to the retfo achievwsent iodbx and wa6 tentatively<br />
ueed a6 e mexeure of motiwtion.)<br />
The analpele proceeded in three etepe. Piret, fntercorrslotione<br />
meong the ten variablee were obtained (cf. Table 3). Secondly, the<br />
three aptitude meaeures vare partfxled out of each of the reaefning<br />
variable6 which wore thn intercorrelated u partial8 (cf. Table 4).<br />
Leetly, multiple corroletione vere canputed batween the raeldualfred<br />
motivation and echool grade predictor6 on the one hand rad the reeidualixed<br />
eriterlon variablea on the other hand (cf. Table 5). A8 you<br />
��� � ee, In tha lattar table, all the predictor raeiduale combine to<br />
predict both criterion roelduale very well. The maximnn multiple lo<br />
a70 for criterion #I, aud A4 for criterion i2, however, the prisnxry<br />
variablee involved in the multiple partial correlation were the peer<br />
rating residual and the fundemeotale echool grade reeiduele; i.e.,<br />
4.123 and 8.123, which yfeldcd~emltiple partial correlation8 cf .67<br />
and .41, reepectlvely.<br />
Table 3<br />
Intotcorrelatio=a Ffeme and Standard<br />
Deviation6 of Verfater - (H-196)<br />
Ueaeure 1 2 3 4 5 6 7 8 9 10 Me&ii S.D.<br />
::<br />
43:<br />
5.<br />
6.<br />
2<br />
9.<br />
10.<br />
cm 29 29 11 -09 04 15 36 33 31<br />
AR1 04 00 -10 03 08 38 20 -01<br />
HECH 16 08 10 05 22 43 38<br />
Peer Rat inge 28 19 00 30 59 34<br />
Self Rating6 -03 -10 09 19 04<br />
check Llet of 02 11 22 22<br />
Student Behav.<br />
Plctorlel Motlv 09 01 03<br />
mch Fund 57 40<br />
Sheet Hetel Unit 61<br />
Welding Unit<br />
tiote: Decixul point6 aeitted<br />
55.61<br />
55.54<br />
56.26<br />
29.24<br />
30.20<br />
97.71<br />
7.18<br />
81.70<br />
79.37<br />
82.34<br />
--- .._- _-.- -~<br />
. . .<br />
.<br />
. . .<br />
.<br />
.<br />
62<br />
*<br />
6.61<br />
5.82<br />
6.76<br />
9.60<br />
9.49<br />
11.47<br />
2.27<br />
6.44<br />
4.89<br />
5.24<br />
. .<br />
.<br />
.<br />
-y i
. . .*<br />
.<br />
- ..-.. ___ -<br />
--- -___ -.__- FI_---wmw--L. - --. -..h-.-.__ .<br />
.<br />
l4eArura<br />
Tab18 4<br />
fntArcorraltrtfonr of XotivAtLon and Criterion<br />
kftmmrei, Aptitude PertiAled Cut.<br />
.<br />
x4. 123 ‘5.123 ‘6.123 ‘7.123 ‘8.123 x9.123 ‘10.123<br />
X4.123 Parr BAtingr, -<br />
29 18 -02 29 60 30<br />
X<br />
5.123 Self RAtbIg -03 -09 16 18 05 !<br />
‘6.123 Checklirt 01 09 20 20<br />
‘7.123 Pictorial HUB. 03 -04 -02<br />
‘8.123 Uech Fund 49 35 '.<br />
%.123 Sheet !tOtAl Unit 43 i<br />
‘10.123 Welding Unit ”<br />
Note: DeeFePel points amfttsd<br />
Table 5<br />
Hultiple - PArtfAl CorrelAtioa Coefficlentr<br />
Betveen HOtiVAtion and CCitSrfA VAriAbleA<br />
MtiVAtioO #eArUreA Ctitsrla<br />
Sheet MetAl Unit Weldiog Unit<br />
x4.123, xS.123, x6.123, IK7.123, ‘8.123 .70<br />
‘9.123 ‘10.123<br />
. x4.123, x5.123, x6.123, ‘K7.123 .62 .34 i i<br />
.<br />
‘4.123, ‘8.i23<br />
63<br />
.<br />
. -. / .7<br />
./ .i :<br />
,>:<br />
c<br />
LX,<br />
;<br />
i .<br />
i<br />
i<br />
i<br />
I
,<br />
- .- - . . ---- .-..<br />
--.<br />
-<br />
D. Application of tha Achievamant Indax. I would like to turn now<br />
to 8 study on an application of the � chievanont ladax in a training situation.<br />
A8 we Indicated, the � chievmnant index can most easily ba darlvad for<br />
individuala by men8 �� � ratio of. actual divided by predicted achievomeat.<br />
1. A Note on Under sod Ovarachiavcmunt. F’rochlich, A. P. and<br />
ttayo, C. D. Par8unr;el end Gu:ldanca Journal, Merch 1963.<br />
This ~jzxnel rrticle rcp:rorantr a susxmary of the raraarch on the’<br />
� chiovammt Index and ohoutd be � ppropos al80 to the termination of my<br />
own remarks on thin subject. Froahllch indicates two applicctionr of<br />
the � chiavamant index (a) Bareuch md (b) Prediction. We have dwelt<br />
at langth on the potrntlal thlr Index haa for purpoaar of baric rsrearch<br />
on learning gala and motiuatImon depandtng on the nature of tha variable(s)<br />
partialad out; i.a., whathar ,achiavmant or aptitude 18 ra8lduallrad. As<br />
regards It8 predictive aspacts, we elro heva indicated that the achlevaindex<br />
doe8 pradlct training criteria but doar not add to the training<br />
grade. Ae notad, it repromate the unpredicted variance of a training<br />
grade which, whan corabinad with the predicted variance of that grade<br />
sat8 the multiple I%” a8 quel to the rtmple “r” (by urlng the trslning<br />
grade alone) In predicting en external criterion. Proehlich cleerly<br />
point8 that fact out, but emphasized that the marit of this dichotomy<br />
liar in It8 iroletlon of variance vhich wa8 not predicted by the ability<br />
maa8urtr utilized. Thu8, .when derived aarly in training, tho � chlevantmt<br />
indax 18 conridarad to hava marit In counseling “low-achiaver8” (i.e.,<br />
those who scored 1.00). Iha study indicated that, “if tha unpredicted<br />
pert of academic achiewment measurer tnotlvationel varience, it might be<br />
bvpotherieed thet if underachla~var8 of various ability levelc could be<br />
motivated to put out more effort,<br />
heve more interest, or t8ke on more<br />
positive attitude8 toward their studfe8, Sane might im;*‘ovc their final<br />
gradas .‘I<br />
In resume, it can be reid thet the achiavemant Index hao 8 graat<br />
potential for basic research in tha difficult area mearurlng non-aptitude<br />
variance ruch a8 motivetion. The pertiel end part corrclatlonr offer a<br />
good technique to isolate andi identify the corralatom of this rariduel.<br />
SOW use of this Indax in tha form of a ratio ha8 been made in uavol air<br />
technical tralnfng for counsalinng purpor*r. It is conrldarad that further<br />
research Into the netura of the � chlavrsnent index will prove to<br />
extend It8 role in an applied sannre.<br />
II Prediction of School Achievamant<br />
The recond approach to job proficiency maarurar i8 the pradiction<br />
of school achlevamant. This anployed the cl~s8lcrl regrarrion<br />
end pens/fail frequency proctcdurar to derive prediction8 and sppronimate<br />
odd8 to succeed various ochools in the Naval Air <strong>Technical</strong> Training<br />
Comand. The vrriebler utll:itad arc the � veilebla relaction end treinlng<br />
grede8. The technical School8 involved were the machaolcal and avionic8<br />
ratings. Briefly, I would like to prarrst an axample of the correlational<br />
teblt8 end al80 the parr/fail frequency teblas and their role in tha<br />
quality control of traiatar.<br />
.<br />
, .<br />
64<br />
__ ._ _ _ . _ .._ .---- - ___. -__ _--<br />
.<br />
. .<br />
‘.<br />
‘<br />
___-- -.--I--.---.<br />
e .<br />
.<br />
*.<br />
a<br />
i<br />
a<br />
.<br />
/
_ . . . _ .- ._ ..-.-_ _---.---- - -_____ --- ._ _- -.. ..-<br />
. .<br />
.<br />
.<br />
---- -.-.. __-.-.- _.___.__. . ..- . ..--_.-__-- .._ _ -. _. ._ . ___._ ..-L-.-L ._ .<br />
A. Correlational Prediction:<br />
T8bler 6 8nd 7 cent&in the correlatlonr of the trrining grade8<br />
utillred at two different point8 in the mme school (IW-A). These<br />
correlatlonr yielded the regrerrion equation8 nccersorg to dorive the<br />
predicted 8core8 contained in figurer one and tuo. Prediction tableo<br />
ruch � r.there, rather th8.n regreorion equ8tioa8, greatly arolot the<br />
training officers to gra.rp the mewing and uoo of predicted scorer.<br />
It8 application, of cour88, con8f8tr in: 1. Vetezmiakg the a:udent’r<br />
grades on the vari8bler acro88 the top and left aide of ffgurcr one<br />
and tvo; end 2. Locating the predicted school final average within<br />
the prediction tabler.<br />
.,*<br />
Table 6<br />
fntercorrclatlons of Variables Utilized<br />
, at the End of Aviation Pamiliariratloa School<br />
m(A)<br />
(N=1342)<br />
2 ,’ 3 H u<br />
BTB 1. .374 .392 177.63 12.32<br />
APAn 2. .478 76.72 7.03<br />
AFu(A) 3. 73.80 9.73<br />
Table 7<br />
Intercorrelation of Variable8 Utilized<br />
at the End of Phase 1 of APU(A) School<br />
APu(A;<br />
(N-1322)<br />
2 3 4 H u<br />
BTB 1. .365 .367 .381 177.79 12.29<br />
APAn 2. .389 .461 76.86 6.94’<br />
Phaee 1 3. ,851 76.27 9.01<br />
Am(A) 4. 74.09 9.32<br />
.<br />
.<br />
65<br />
.<br />
,<br />
,I<br />
- .^.._ -..<br />
, ;<br />
I
.<br />
.<br />
. .<br />
. .<br />
.<br />
.<br />
--. _ ._. ._ . ^_ . - --_._--~---.“~-- --..- .- - - _._ - _. .-_. ___ _.-_<br />
200<br />
195<br />
190<br />
185<br />
180<br />
175<br />
* 170<br />
c-4<br />
' 165<br />
m<br />
2 160<br />
ok<br />
2 155<br />
2 ;;:<br />
150<br />
$ 145<br />
140<br />
135<br />
130<br />
125<br />
120<br />
R - .530<br />
~1.23-8.23<br />
Figure 1<br />
AFti Prediction Table<br />
(?P1342)<br />
AFAU Ffnal Average (X2)<br />
-. --<br />
53 56 59 62 65 68 71 74 77 80 83 86 89 92 95 98<br />
66 67 69 70 '72 74 75 77 78 80 8? 83 85 86 88 89<br />
65 66 68 69 71 73 74 76 77 79 81 82 84 85 87 88<br />
~....<br />
64 i65 67 68 ';O 72 73 75 76 78 80 81 83 84 86 88<br />
.--a4<br />
63 64 j66 67 69 71 72 74 75 77 79 80 82 83 85 87<br />
62 63 i65 66 (58 70 71 73 74 76 78 79 81 82 84 86<br />
m-o-(<br />
61 62 64 I65 (57 69 76 72 73 75 77 78 80 81 b3 85<br />
"'Y<br />
60 61 63 64 ;66 68 69 71 72 74 76 77 79 80 82 84<br />
8<br />
59 60 62 63 i65 67 68 70 71 73 75 76 78 79 81 83<br />
----<br />
58 59 61 63 64 I66 67 69 70 72 74 75 77 78 80 82<br />
t<br />
57 58 60 62 63 i65 66 68 69 71 73 74 76 77 79 81<br />
-.-.*<br />
56 57 59 61 62 64 i65 67 69 70 72 73 75 76 78 80<br />
-0-q<br />
55 56 58 60 61 63 64:66 68 69 71 72 74 76 77 79<br />
:<br />
54 55 57 59 60 62 63:65 67 68 70 71 73 75 76 78<br />
!-,<br />
53 54 56 58 159 61 62 64:66 67 69 70 72 74 75 77<br />
:<br />
52 53 55 57 ,58 60 61 63 i-6: 66 68 69 71 73 74 76<br />
51 52 54 56 .57 59 60 62 64165 67 68 70 72 73 75<br />
-0.<br />
50 51 53 55 .56 58 59 61 63 64: 66 67 69 71 72 74<br />
--- - .<br />
.<br />
. .<br />
Regrerrion Equation: ~,,.196Xl+.53u[2-1.77<br />
Outtfng Score: 65 (i.e., oddr of 2 to 1 to pare)<br />
Xl-Basic Teat Battery Composite<br />
X2-A.PAIf Final Average<br />
E3=AFU(A) Pine1 Average (predicted)<br />
.<br />
,<br />
66<br />
*.<br />
c<br />
.<br />
.<br />
.<br />
.
__ ._ __ ._ .^ . .._--- -. ____. ..____ ~-__----- ---I -_-- .__.-_ - .___-_ _-.-. - . -.<br />
. *<br />
. .<br />
-----<br />
::<br />
a’<br />
_ ; ____ __-_.___ _ _s .,--1 .. ..-Clr-<br />
_--- --..-- ._.-.__C .____<br />
_ --‘-..--<br />
98 58 61 63; 66 69 71 74 76 79 81 84 86 89 91 94 96<br />
95 58 60 63; 65 68 70 73 75 78 00 83 85 88 90 93 96<br />
92 157 60 62; 65 67 70 72 75 77 80 82 85 87 90 92 95<br />
89<br />
86<br />
83<br />
80<br />
%<br />
a 77<br />
i 74<br />
2 71<br />
lj 68<br />
i 65<br />
62<br />
59<br />
56<br />
53<br />
50<br />
Figure 2<br />
MU(A) Prediction Table<br />
(R-1322)<br />
Fbue 1 Fin81 Average (Xl) i<br />
53 56 59 62: 65 68 71 74 77 80 83 86 89 92 95 98<br />
57 59 62i 64 67 69 72 74 77 79 82 84 87 89 92 94<br />
(1-m*<br />
56 58 61 63;66 68 71 74 76 79 81 84 86 89 91 94<br />
:<br />
55 58 60 63;65 68 70 73 75 78 80 83 85 88 90 93<br />
:<br />
55 57 60 62; 6S 67 69 72 75 77 80 82 85 87 90 92<br />
:<br />
54 57 59 62j 64 67 69 72 74 77 79 82 84 87 89 92<br />
----I<br />
53 56 58 62 63 ;66 68 71 74 76 79 81 84 86 89 91<br />
:<br />
53 55 58 60 63 ;65 68 70 73 75, 78 80 83 85 88 90<br />
:<br />
52 55 57 60 62 :65 67 70 72 75 77 80 82 85 87 90<br />
:<br />
52 54 57 59 62 :64 67 69 72 74 77 79 82 84 87 89<br />
c ---I<br />
51 53 56 58 61 63;66 68 71 74 76 79 81 84 86 89<br />
:<br />
50 53 55 58 60 63:65 68 70 73 75 78 80 83 85 88<br />
:<br />
50 52 55 57 60 62165 67 70 72 75 77 80 82 85 87<br />
49 52 54 57 59 62i 64 67 69 72 74 77 79 82 84 87<br />
---a,<br />
48 51 53 56 58 61 63 i 66 68 71 74 76 79 81 84 86<br />
R - .863 Rcgrsrrioa Equation: ~3'.8388x1+.210x2-6.55<br />
01.234.82 Cutting Score: 64 (i.e.,oddr of 2 to 1 to paw)<br />
Xl-Fbue 1 Final Average<br />
~-pAFAH Final Average<br />
X3=AFCl(A) Final Average (prsdlcted)<br />
.<br />
.<br />
67<br />
!
--- .._ __. __^ _ __-.-_ -.- -- _<br />
.<br />
- -- . . . .- :.- -. -. . _... .- _ _ __ I - ._. -.<br />
.<br />
�<br />
��<br />
� � �<br />
.<br />
.<br />
. . .<br />
m<br />
- .---- . .I . ..r _ -. _.-_-1---- _ -- ._ _<br />
B. Parr/Pail Oddo of Predfctloa:<br />
�<br />
no second mrthod ured to deterarlee a 8tudent’s chaxice8 of<br />
succeeding in training is bared on the observed Parr/fail frequanctar<br />
for vulou8 intervals of grader moat correlated with trainlog parformancr.<br />
Tables 0 and 9 ot tha handout contain the parr/fail frsquancy<br />
data for 2 variable8 taken at differant point8 fn tr&fntng fn<br />
the Avionic8 Fuadammtalr School et Hsmphlr. Data In tablo 8 io based<br />
on an “N” of 1340 rind a correlation of ,481 and data In table 9 on<br />
8n “W* of 1291 and 8 correlation of 25. The pars/fail frequsncleo<br />
are trmrlatd into approxiaate odde to succeed that particular 8chool.<br />
Thr ame data MI translated into two graph8 to facilitate the tratnlng<br />
officer8 interpretation and application of thaia pars/fail frsquancy<br />
tables. There graph are contained In figure8 3 and 4 and are relfarplanatory,<br />
tith the acmplor given vithin ach table.<br />
The predicted stoma, based on selection and training grader hmvs<br />
proved to be a valuable index to ultimte training porfomunce. The<br />
application of there predictions is the
�<br />
Teble 8<br />
Thr Probubilitier of Grrduation MU(A) School<br />
after a Student Carcpleteo APAX School<br />
(H-1340)<br />
AFM x A&) WJ(A) Apprax’. Oddr x -<br />
Pfa.Aver. of Total Graduater Attritqr for 8g8Fnrt Graduating<br />
9 4<br />
90-93<br />
86-89<br />
82-85<br />
78-81<br />
34-77<br />
70-73<br />
66-69<br />
62-65<br />
58-61<br />
54-57<br />
-53<br />
.4<br />
2.4<br />
7.6<br />
16.3<br />
20.0<br />
21.0<br />
17.4<br />
9.7<br />
3.4<br />
1.2<br />
.7<br />
. l<br />
352<br />
97<br />
213<br />
254<br />
244<br />
178<br />
95<br />
31<br />
:<br />
5<br />
5<br />
14<br />
5375<br />
35<br />
1’:<br />
5<br />
1<br />
5 0<br />
0<br />
2i 1<br />
43 2<br />
18 1<br />
13 2<br />
1<br />
z 2<br />
2<br />
i<br />
z 5<br />
-7<br />
0 1<br />
100<br />
100<br />
95<br />
98<br />
95<br />
87<br />
76<br />
#73<br />
69<br />
ii<br />
00 -<br />
llS8 182- 6 1 86 - -<br />
Table 9<br />
l’ho Probebilit%e$ of GrAduatfng AKl(A) School<br />
����� � Ltudent Comphter Phare 1<br />
. (N-1291)<br />
PhAre 1 % m(A)<br />
Pfa.Aver of Total Credoate<br />
94.<br />
90-93<br />
86-89<br />
82-85<br />
78-81<br />
74-77<br />
70973<br />
66-69<br />
62-65<br />
58-61<br />
W-57<br />
2:<br />
9.4<br />
13.9<br />
19.7<br />
15.9<br />
14.7<br />
9.0<br />
6.5<br />
3.6<br />
.9<br />
22<br />
61<br />
121<br />
179<br />
252<br />
202<br />
177<br />
91<br />
41<br />
11<br />
1158<br />
�<br />
�<br />
,<br />
69<br />
Attriter<br />
�<br />
- -<br />
Approx.Odda x<br />
for Qafn8t Cradu8ting<br />
22 0 100<br />
61 0 100<br />
121 0 108<br />
179 1 99<br />
126 99<br />
67 : 99<br />
14<br />
93<br />
18 :<br />
78<br />
bV@Il 49<br />
1<br />
24<br />
- 1 1: - - -8- 9 1 90<br />
I.<br />
i<br />
i<br />
; I<br />
i<br />
.<br />
i<br />
I(<br />
.<br />
; i-<br />
t :<br />
i<br />
� �<br />
�<br />
i
90<br />
80<br />
70<br />
60<br />
10<br />
.<br />
A,ctual da<br />
I<br />
I Exawpl~: 64<br />
2. Interrect probability CUNQ<br />
1<br />
I ,3. Interpret odd8 for •UC~BID<br />
I Grade 64-7 to 3 to pur<br />
I<br />
I<br />
00 7V 80 90 il<br />
AFAX ?irul Avor*go<br />
Ptgura 3<br />
8umaary utilf?y chart of probability date<br />
coatrlad in T&h 5<br />
70<br />
, _ _-_ - .._ .-.-_ .<br />
.<br />
Odds<br />
GVSn<br />
2 to 3<br />
3 to 7<br />
1 to 4<br />
1 to 9<br />
.<br />
.<br />
.
60<br />
40<br />
30<br />
20<br />
10<br />
1. Identify PtlAEr 1 grrda<br />
Exam$ls: 66<br />
2. faterroct probability curw L<br />
3. Interpret oddr for eucce88<br />
orode 66-3 to 2 to par<br />
I I<br />
70 80 90 loo<br />
Fhsra 1 Final Avrrrgr<br />
Piyre 4<br />
Gummy utility chart of probability data<br />
contained in Table 6<br />
. .<br />
. .<br />
.<br />
.<br />
71<br />
.<br />
.I<br />
Oddsl<br />
9 to 11<br />
4 to 11<br />
7 to 3<br />
3 to 2<br />
EV&l<br />
2 to 3<br />
3 to 7<br />
1 to L<br />
1 to 9<br />
.
.<br />
,<br />
.<br />
.<br />
rteferencer<br />
Dubois, P. R., Multiv8rlate correlational analylfe. Ww York Harper &<br />
Brothers, fmt .<br />
buboir, P. B. h t4aml~, u. H., Methods of res-h ln tcch~ical training.<br />
QIR Contr. # ~cear 816(02), NATT, ~av. Ed. ~e.ch. Rapt. 83. April 1961.<br />
lroehlich. H. P. and ?fayo, C. D., A note cm undar md overe&iewmat.<br />
perronnal and Cuicirnca JournalI Mrch 1963.<br />
L0DfJ0, A. A., etioo of rtudant performance Ln five wchurlcrl<br />
8ch001.. v rm<br />
Iennsrrce, December 1963.<br />
lougo, A. A., An appraisal of ratio meores. Bc,earcb <strong>Report</strong>. Staff,<br />
CNATECXl=iU, NhS wznphi8, feaa~aea, Much 1964.<br />
Lwgo, A. A., The pradactioa of rtudrnt parforranca in oaven avionica<br />
re: oolr . Berearch Rsport. Staff, m .w-i00<br />
Tameawe, August 1964.<br />
Uayo, C. D., PAQ Hawal. Renaarch <strong>Report</strong>. Staff, C2UfeCEmUr tU!I<br />
Nemphia, TenncBsee, 1957.<br />
)slyo, G. D. and Xauning, U. Ii., Uotivatlon meaBut-t, Educational,<br />
end P8ychologfcal Xeaaur-nt, XXI, 7343, 1961.<br />
‘Lhorndlka, E. L.; Brsgmaa, J.; tilts, J. U. and Woodyard, I!., Adult<br />
learuing. New York: UacMlLan Co., 1928.<br />
__--_. .--- .<br />
72<br />
.-<br />
#<br />
_.__e. c_ .--A____<br />
-- .---<br />
. .<br />
. . . .<br />
.,<br />
,<br />
, . .<br />
. *. .<br />
.<br />
.<br />
I<br />
.<br />
.<br />
.<br />
i<br />
;<br />
1.<br />
.
.<br />
.<br />
.<br />
.<br />
Parronality Tooting md Job Proficiency<br />
.?OHH D. RJtAFT<br />
US Amy Enlirted Bvaluatim Center \,<br />
Chairman Bridgeo, yellow’-‘Confetee.8,<br />
It ir a privilege to lead thir dirwrrion on perrmality teeting<br />
and Job perfomanca,<br />
A8 a point of departure in exploring waya of iacruring the prediction<br />
of roldiere’ performance, I vi11 give a goneral di8cuorion of<br />
the rarurch vhich Rayumd B. Cattell and hi8 u8ociatr8 at the Univorrlty<br />
of Illiaoir have conducted for mumming motlvetional rtrangth. Aleo, I<br />
will dircu88 8-e of the u8ea which wo might put Cattell’a re8march. and<br />
relrtod motivatimal ramarch, to in evaluating tho total percent of<br />
variance in the Job perbormanc~ of the individual rerviceman.<br />
Currently, the Army u8es A fob “aa8tary” teat to measure what the<br />
� oldler “can do.” It *JUU a rating rcrle to measure vhrt the roldier<br />
“vi11 do.” Unfortunately, even though years of hard lrr&or have barn<br />
attended by indurtriel prychologirtr in crying to refir.,+ the rating recli<br />
and to minimire it8 error8. it rtill ir a veak evalu8tfng iustrumnt.<br />
Individual8 differ in their ability to accur8tely parcaiw the cheractsrirticr<br />
of otherr. Sane of the factor8 relevant to thla differential<br />
perception are in the age, 8az, end prrronality of the parcaiver, la<br />
there charreteri8tic8 of the percoivad, and in the content erer8 in which<br />
the prrdic t ion8 are madu.<br />
Thr rating 8~~10’8 %ill do” � reee � re csotivetionol in content. They<br />
rhw the “drive” or “atntiment” f8ctorr o f t h e roldler’r performance. Hr,<br />
Shirkey, who rpoke before you A fw chute8 ago, recently did A factor<br />
analyrir 03 the Army’8 new CER (rating rcale). He found five factor@<br />
which can tentatively be celled: Militery Bearing I and II, Xilitaxy<br />
Laaderrhip Potential, Initiative, end T88k Uotivation. Hr. Shirkey wa8<br />
8urprired to find that the rupervirory portion of the trot correleted<br />
rather poorly vith the Leaderrhip Potential Pector of the CER. Although<br />
it io porrible that there rcrultr ‘Are partielly A condition of the prrtitular<br />
HOS analytad, it la evident th8t the CER la mesrurlng the mot;-vational<br />
arue concerned with vhat the coldior “will do.”<br />
Since mo8t ratiaga are rpurlour and do vary with thr rater, conrider<br />
how much better it would br if predictive mea8ure8 of tha motivational<br />
rtrength in there ereee could be nwde in order to give more<br />
objectivity and exactnerr. An cz8mple of the porrrble work that can<br />
be done along there line8 18 th8t of the rsrs8rch currently being<br />
conducted by the US Army Perronnel Rersarch Office on thr valtler a$aoelated<br />
with military cereer motiv8tion. Tlw rarearcher there heve<br />
73<br />
,-<br />
: ..o
-. _<br />
. . -<br />
:<br />
;<br />
-.----.-.--.e. .- -<br />
Identified the strengeh of six dimensions of motivational values or<br />
factors which are useful in predicting whether an officer or en epliated<br />
mm will plan on staying in the military service as a career. These<br />
factors which are tested by objective type teats ure support, conformity,<br />
recognition, independence, bcnevalence, snd leadership. ~1~0, they are<br />
currently conducting extensive research in developing devices which ~111<br />
predict cthich officers and enlisted uen will be the nest proficient<br />
dcring their &my careers.<br />
Cattell and his co-workers hsve spent the last fifteen yesra in<br />
trying to develop objective measures of motivational strength. They<br />
studied the amjor evaluative instruments produced by others, their devices,<br />
and the known general psychological principles in the areas of moeivatlon,<br />
learning theory, etc. (& example of the kind of principle referred to<br />
here is that of information. In general, i.e., after we discount the<br />
Influence of intelligence and general breadth of interest, a person knows<br />
more about what he is interested in, more about those courses of action<br />
to which he is comieeed, than he does about what he is not interested<br />
in.) Cattell developed m-me seventy-five devices of his own and poktulatcd<br />
8omf2 sixty new principles. Prom this moss of nnterial, he found<br />
through factor ana?ytic atudies,*seven basic raotivational ccmponents.<br />
ihree of these seem to correepond in content to the functfonr ascribed by<br />
Prcud to the Id, Ego, and Superego. The others have been named Physlological<br />
Interest, Repressed Cmplexes, Impulsivity or Urgency, and<br />
Persistence. These qualities can be ascribed in varying proportion to<br />
any drive. Second order facto- analysis of these motivational components<br />
resulted in two second order factors: (1) Integrated, or that which is<br />
essentially a conscious and experienced expression, and (2) Unintegrated,<br />
or that which is essentially unconscious and mainly wishfug and tension.<br />
The two most important working concepts of Cattell in his theoretical<br />
treatment of motivation have been termed “erg” and “seneiment,”<br />
respectively, for the constitutions1 � ud acquired patterns found<br />
opmationally as factors in dynamic measures by experiments. The term<br />
“erg” is used instead of drive because ehe latter term drags in all<br />
manner of clinical and other assumptions about “instincts” and so on,<br />
whcrers the ergic paeterns are experimentally demonstrable. How?Wr, In<br />
popular terms an erg is a drive or source of reactive energy directed<br />
toward a pareicular goal, nuch as fear, mating, assertiveness, aad so on.<br />
By contrast, a rentiment is an acquired aggregrate of attitudes built up<br />
by learning and social experience, but also, like an erg, a source of<br />
motivational energy and interest. Both ergs and senti;?rants, though<br />
essentially ccmnmn In form, are developed to different degreee in different<br />
people. Cattell and his co-workers found factor analytic evidence for some<br />
twenty motivational dimemions in a broad rampling of variables.<br />
-- _ . . _-._ -.-..___ -_- ___ _ _<br />
.<br />
74<br />
. . -.<br />
.<br />
I<br />
L<br />
.<br />
__.<br />
_.- -.-<br />
.<br />
.<br />
s
-. _- _. ._. ,. _ -- _ ______-________ ---..--- ..-_--_--___. .--..- ----.. -... -- .<br />
. *<br />
. .<br />
& .’<br />
.<br />
_ _ -----.- --A- __.__.___” ____ _. ,__- _-_..... ..-. -.._<br />
S~aa of the major crgic patterns or dynamic factors for which Cattell<br />
found evidence through his research are: Escape, with associated emotion<br />
of fear; Hating, with associated emotion of sex; Seif-assertion, with<br />
associated emotion of pride; Gregariousness, with associated emotion of<br />
loneliness; Appeal, with assocfated emotion of despair; Exploration,<br />
with associated emotfon of curiosity; Rest seeking, with associated<br />
emotion of sleepiness or fatfgue; etc. Some of the major sentiment<br />
patterns or dyuamlc factors for which he found evidence are: Self;<br />
Superego; Career (levela of aspiration); Sports and games; Mechanical<br />
Interest; etc.<br />
Cattell and his assocfates then developed the experimental HotLvational<br />
Analysis Test to measure the tension operative in the ergs and<br />
sentiments that prior to nov was assessed only by rough means. Ten<br />
dynamic factors were chosen which were felt, as a result of careful correlational<br />
and factor analytic research, would be of greatest value to<br />
the test users as being reprerentative, and comprehensive in coverage,<br />
of adult motivations.<br />
In this test, Cattell mearured the strength of each of these ten<br />
dynamic factors by using forced-choice type of attitude-interest questions.<br />
The particular attitude-interests he used were carefully chosen because<br />
they were found to be substantially -related to and therefore best suited<br />
to represent the factors concerned. For example, he found the folloving<br />
attitudes and their motlvatfonal paradigms for the dynamic factor of<br />
self-sentiment: ,<br />
-.<br />
1. Good reputation - - “1 want to maintain a good reputation<br />
and a camon respect in my commmity.”<br />
2. Norm1 sex - - - - “1 want a norms1 socially acceptable<br />
relation to a person of the opposfte<br />
sex.”<br />
3. .Look after family - “I want to look after my family so that<br />
it reaches approved social standards .‘I<br />
4. Proficient career - “I want to be proficient in my career *”<br />
5. Control impulses - “I want to keep my impulses under sufficient<br />
and proper control.”<br />
6. Self respect - - - “I want never to damage my sense of<br />
self respect.”<br />
The motivatioru!l strength of these attitude-interest paradigms for<br />
each dynamic factor vere measured by using four different forced-choice<br />
type devices or subtests. These subtests, which are a little unusual<br />
in nature, are called: (1) Uses; (2) Estimates; (3) Paired Words; and<br />
(4) Information. They are illustrated in the folloving chart.<br />
75
�<br />
�����<br />
:<br />
���<br />
�☺�<br />
.-<br />
.*<br />
.<br />
- . -_ _ . __ ma - . -. -. . _. _ __ _<br />
_.I ,<br />
.i<br />
2.<br />
-:<br />
_. .’<br />
. -.<br />
.<br />
�<br />
�<br />
��<br />
�� �<br />
� �<br />
�<br />
� �<br />
--__~. _-. -. --. ._<br />
�<br />
�<br />
Subtests (Rxamples~<br />
1. “Ends-for-means” (Projection) or “Uses’.’<br />
A free afternoon could beet be spent:<br />
a. enjoying the out-of-doors<br />
b. earning overtime<br />
2. “Autism’@ or ~*Esthates”<br />
How many years does it take to make a secretary efficient?<br />
a. 2 b. 4 c. 6 . d. 8<br />
3. “Ready <strong>Association</strong>” or “Paired words”<br />
Stamps<br />
Collect<br />
Evidence<br />
4. “Mean-end Knowledge” or “Info~tion”<br />
Who was the first president of the United States?<br />
a. Lincoln B. Jefferson C. Washington D. Roosevelt<br />
The scores from the four su3tests are combined into two groups for<br />
each factor to give separate strength measures for: (1) the Integrated<br />
(or conscious) (Information plus Paired Words) component, and, (2) the<br />
Unintegrated (or unconscious)(Estfmates plus Uaea) component. Thus, the<br />
test user can either use two scores for each of the ten dynamic dimensions<br />
in the N. A. T., or Integrated and Unintegrated scores can be added to<br />
give a single total interest score for each of the ten dynamic dfmenslone.<br />
The final score on each of the ten dynamic structure factorcr i8<br />
actually a sum both Over four devices, or methods of measurement as just<br />
described, and aleo Over distinct attitudes which are known to constitute<br />
good representatives of the dynamic trait. There are a few other scales<br />
developed with this test. Perhaps the most interesting one ist the<br />
Conflict Scale. This scale shows the amount of conflict between the<br />
Integrated and Unintegrated scales.<br />
Cattell reported the scores in stens (standard scores) for each of<br />
these scales. The statistical procedure used in arriving at these scores<br />
is ipsative in nature. One can learn the total strength of each of these<br />
drives in the individual being tested and be able to make diagnostic predictions<br />
from them.<br />
if -. .-._. ..-.. _~__. _-- ---~--.. -._-_ -<br />
76<br />
. . .<br />
4
.<br />
.<br />
. .<br />
,’ . .<br />
NW we rhould recognize that what Cattell is measuring here a8 a<br />
“drive rtrength” is actually mO8t accurately rtferred to ae an ergic<br />
ten8iOn 1eVel. That i8 to 8ay, it i8 not 8omc donstituticnnl strength<br />
of that drive, but the end result, in actual motivational 8trcngth of<br />
four or more influencer. There are: (1) con8titutional endowment, (2)<br />
early experience8 (imprinting, reprerzioc), (3) current degree of environmental<br />
8timulation, and (4) current reduction of tcnzion by satisfactions<br />
and degree8 of recurrent need gratification existing in the prerent<br />
environment. Theae vary with time.<br />
The Motivational Analysis Test is etill an experimental tert.<br />
Hwevtr, a8 more rerearch i8 conducted on it, it8 theoretical predictive<br />
ability and ��� � cutal predictive ability a8 derived fran preliminary<br />
reraarch rerulta will increase.<br />
In predicting what an adult will do, it 18 evidently quite a8<br />
important to knw the motivetion available frop these acquired rentimentr<br />
a8 from hi8 basic ergs. For urimple, the evaluation of a man’8 potential<br />
In a career will need to include mea8ure8 of the rtrength of hi8 intertrt<br />
in a career a8 8uch, 8180 of hi8 degree of concern about hi8 general<br />
reputation and self-concept, and ezp&ially of the level of basic dependability,<br />
implied by his level of superego development.<br />
The 8tati8ticalpredictionzwhich rezearch has 80 far obtained<br />
against life criteria have borne out these poychological interpretations<br />
of 8core8. Rerearch ha8 shown that dynamic structure trait rtrengthe<br />
add to the prediction of scholaetic achievement over and above what ir<br />
predictable from ability and general perronality traits. John Butcher,<br />
one of Cattell’s a88ociate8, found that other variable8 being controlled.<br />
those students who are high in ruperego ��� � ��� high � chiever8, there<br />
who are low on radi8m are high � chiever8, those that are high on<br />
arrertivene8s are high � chiever8: but he found no relationrhip with<br />
8elf rentimcnt. Aleo, he found that the career sentiment mU8t be modifiedI<br />
for the particular setting.<br />
Mr. Claude Bridge8 talked with Dr. Arthur Swency a few week8 ago.<br />
Dr. Swenay, who ie an � 88ociate of Cattell, 8aid that the preliminary<br />
reoearch rhwr that general motivational factorz could contribute a<br />
great deal a8 predictor8 of achievement in 8ptCifiC occupational rpecialities.<br />
He raid that it would probably be neceroary for u8 to develop<br />
rpecific inrtrumentr for group8 of occupational rpecialitier. (fii8 got8<br />
in with another te8t that Cattell has developed called the Vocational<br />
Interrt Heawre, which ir built on exactly the came principle8 a8 the<br />
Motivation81 Analysis Test only it iz geared toward predicting in 8pecifi.c<br />
occupational groupinga. The Vocational lnttrt8t Hearure ir being validated<br />
againzt the Kuder Preference Record.)<br />
.<br />
77<br />
-. -<br />
- :r ---<br />
i .
’ ._ ,_<br />
1 _ +<br />
_ _ . . .<br />
.<br />
a<br />
--.<br />
s<br />
.<br />
. .-. . - . _<br />
. . .<br />
/ . . .u<br />
/ . .- - .- . . ~. :.. __..----.----._- 11.11_--___ _.<br />
*<br />
---- - __------.<br />
Cattell says that when the results of the Motivational hnale<br />
Test are applied to clinical and industrial work,ritarion re%tiona<br />
oreequally significant and comprehensible in terns of the concept<br />
validities of the measures as they are of school achievement. Since<br />
Cettell is currently publishing the predictive details of most of his<br />
industrial rcseareh, it will be some months before these results ara<br />
1 available. .<br />
I feel that Cattell’s work is vary important to us as he has<br />
broken through several barriers. First, by carefully reviewing the<br />
previous motivational materials and those of his own, he wae able to<br />
corns up with motivational factors which operate more or lesr in all<br />
adulto. Some of these factors could be very valuable to us in OUT<br />
efforts to evaluate the total, on-the-job effectfveuers of each soldier.<br />
For example, by utilizing test results on the relative tension levels<br />
of ~cnae of these factors, we could after careful etudy of their rigniffcance,<br />
utilize the test results to identify the cause of a soldier not<br />
working up to his pest effectiveness.<br />
Second, Cattell developed test procedurea whereby the variable<br />
being measured is 80 disguised that the subject usually has no desire<br />
to distort hir response (e.g., what is obstensibly an ability tert) OT<br />
else would be incapable of doing so in any ays:ematic way because he<br />
ie unaware of a “desirable response.” Cattell has gotten away from<br />
the drawbacks of deliberate faking, of personal illusion, snd of ruperficielity<br />
of Easurements which have vitiated against mast “projective,”<br />
“preference ,‘I and “opinionsire” methods employed In interest batteries.<br />
Since Cattell hes shown that a person’s rrotivational makeup ia baaed<br />
on both Integrated (conscious) and Unintegrated (unconscious) components,<br />
we can see that by simply asking a person how he foels about something<br />
this cannot possibly be sufficient to tell his rrPtivationa1 ten.sion in<br />
the requested area.<br />
Third, Cattell utilized new scoring procedures, ipsativa in nature,<br />
whereby he could learn the total strength of each of these drives in the<br />
individual and be able to make predictions from them.<br />
Although Cattell’s Motivational analysis Test ia rtfll a research<br />
instrument, it is a very major step forward in mJtivationa1 strength<br />
measurement. This test covers a person’s Interests, drives, and the<br />
rtrengths of his sentiment and value systems. It increaser the reliability<br />
of conclusions regarding perromlity dynamics and helps to<br />
locate areas of fixation and conflict.<br />
The test concentrates on ten prychologically meaningful unitary<br />
olotivation systems, established by comprehensive and objective factor<br />
analytic research. Also, it uses nawly validated objective test devices<br />
for measuring these interest strengths.<br />
78<br />
. . -. _... - _.... -.----.-____ .-.- -- -_--_--<br />
. .<br />
.<br />
.<br />
.<br />
.<br />
.<br />
.<br />
.<br />
�
_ . - . .._ - -_-..-.-- _-_- ._,__X_____.___r_C_ _ --_ .._.-.. - .<br />
- _ __ .-_. - _ -~. .__ ..__ .-_ -. - _____. - _- _ _. .- _ _ _<br />
.<br />
. .<br />
.<br />
_.- ---....-I- --...- -~..-.‘---.------.-. C....‘~e-.C.---F --*- -..s.*---1m-*r~- --...-,.. -..ea.,I .,,_.__. _.‘_,” ,_ I ,, ..<br />
By centering on the main mtfvvation nyetenm, ns demonstrated to<br />
be consistently present in the typlcal indFvFdua1 of our culture, rather<br />
than on “special purpose,“ arbitary conglomerations, .eubJectivelyconceivad<br />
interest divisions, or iamginary “drives,” the test design<br />
recognizes the whole adult.<br />
It may be possible that we can use the research findings behind<br />
these tests. We can develop our own motivational fastor tests from<br />
the great fund of current, veil-founded, and documented research.<br />
Since we have large populations (serviceman) avaiiable to us, we can<br />
validate the tests rather easily. ~160, with our quite adequate computer<br />
machinery to work with, we can easily do the required statistical work.<br />
The tests which we could develop can be wed in three areas: (1) to<br />
rupplement or replace parts of our rating wales, (2) to set up a criterion<br />
for validating and improving ‘*supervisory and cognitive” portions of our<br />
present teata, and (3) to set UP a criterion for validating our rating<br />
rcales against.<br />
.<br />
Conclusions<br />
When we look back at Chain.ln’s Bridges’ chart here, and see the<br />
large percent of variance that is not covered by general *‘achfevewnt”<br />
type job-knowledge tests and rating scales; and when we see in the<br />
one case of Mr. Shirkey’s Factor Analysis of, the Army’s new rating scale,<br />
that our supervisory section of the Army’s Job mastery test dt-.e not load<br />
highly vith *he factor of Leadership Potential; and when we remember that<br />
the rating scale is influenced by many factora of the rater, ratee, and<br />
the content setting, perhaps this ic a major area which we should explore.<br />
Chairman Bridges and I would like to nou open the session to a general<br />
discussion of personality testing and Job performance.<br />
.<br />
79<br />
.<br />
” *<br />
.<br />
!<br />
I<br />
I<br />
I
,’<br />
�� ���� � �<br />
����<br />
s<br />
. --.<br />
� � � ��� ���� �<br />
. *<br />
-. .<br />
. . I<br />
.- .A-..-. ..--7 -------.-_<br />
- : - . -. ._^_...<br />
� �� � ��� � �� �� � � �� �<br />
Concluding Cameente<br />
CLAUDE F’. BRIDGES<br />
US Army Enlietad Eveluetion Canter<br />
___ _--._-_ .-.- . -_ *.---. . -- -.-. _. -_---.--.<br />
-. ._--<br />
After tha etimuletfng preperetory presentations of Cenarel Z8i8,<br />
Ceptain Reyee, end Dr. Edgarton, the question � rieee ‘What have we<br />
� ccanpliehad in this eympoeiu~?” Cur axpectetion wee not thet WI would<br />
imeadietaly make eny breakthroughs into the outer specs of tasting. We<br />
did hope to go beyond mora axchenga of procedures end obtain acme etimu-<br />
1Ation end nascent ideee having potential for meking inroede into tha<br />
currently 1nadequetaly maaeureble percentega of job proficiency. I<br />
believe wa heve � ccanpliehad this purpoea. Three d1ecueeione hevc provided<br />
us with idaee, possible new � pproechce, or naw epplicetione of<br />
aetabliehad techniques thet are worthy of further axploretion.<br />
Dr. Zaccerie mey strike some treditionel1et8 as a bit of 8 maverick<br />
ia some of his ueegae of terminology, but the points mada in tha paper<br />
by him end Dr. Kerp marit serious coneidaretion. True psychological<br />
8nelyece of jobs, epccificnlly for � aeeurament and treining purpoeeel<br />
and the cfftctiva detenninetion of functions1 job requirements and<br />
� tenderde should prova to be one of our most fruitful � terting ?ointe.<br />
As Cronbech (1960) etetee, “The most important requiranent fot valid<br />
� eec-Jemant le...8 cleer undcretending of the psychologies1 rcc,uiremente<br />
of the criterion teek.” In this connection, severe1 recent publications<br />
by Dr. Robert N. Cegnc end Dr. Robert B. Hfller art pregnant with poeeibllitiee.<br />
The pepare by Hr. Urry end Hr. Shirkay indiceta not only that con-<br />
� idcrabla improvament of even good taste ten still result from further<br />
polishing, but � leo point out some new applications of etetFeticel<br />
� nelyeie end control techniques to npprecieble incraeea in test velidity.<br />
..- * AS expected, Lieutenant Longo’e peper provides eavarel significant<br />
contributions. The Nevel Air <strong>Technical</strong> Treining � teff, under Dr. Meyo,<br />
,<br />
for many yeere has bean conducting axteneivc reecerch on tha prediction<br />
of job proficiency end on � ctuel versus predicted � chievament. This<br />
pepar werrente some edditionel ccuxaente end I would like to meka sane<br />
further suggestions on the use of an indax beeed upon the relationship<br />
of (1) ectuel � chievament, es mteeured by a proficiency test, and (2)<br />
axpectcd � chitvament, as predicted by en � ppropri&te � ptituda test.<br />
Lt Longo pointed out, that such 8 “diecrepency score” or, as I prefar<br />
to cell it, “effcctfveneee indax,” reflects the composite affect of<br />
several verieblee including both error verienca end � ll verience not<br />
in cOrnmOn between the two varieblee. Included in these influencing<br />
vrrieblce era attitudes, work hebite, locelly unique job eituetion<br />
varieblee, interection effects, end motivetion-in other words, the<br />
affectiveneee with which the men � ppliee his eptltudee, ekil.8, and<br />
knowladgce to achievement in 8 specific � ituetion.<br />
80<br />
.<br />
. .
.<br />
-. . . . -- .--- ..-_ .--- . .--. - -- ..- -- ._ _.. - _.. -._<br />
.<br />
. .<br />
Two usea are proposed. First, careful study and anAlyses of the<br />
differences in characteristics of “over-achievers” and “under-achievera,”<br />
using what Dr. Helvin R. Marks calls “the off-quadrant upproach,” ahould<br />
help identify variables Chat need to be covered better by the tests.<br />
Second, is an adaptation of the usual procedure that may add significantly<br />
to the multiple Correlations.<br />
In his doctoral dissertation, Reverend Brother James C. Bates, F.S.C.<br />
Consultant to the Provencal, Christian Brothers of Ireland, Canadian<br />
Provence, recently analyzed sole of the concomitants of the “learning<br />
effectiveneso” of 381 male students in a liberal arts college. While<br />
checking out some of my hypotheses, he produced evidence that an<br />
appropriately developed effectiveness index combined with an achievement<br />
test and an aptitude test does yield a significantly greater multiple<br />
correlation with academic marks than using only the two variables from<br />
which it was derived. Naturally, the addition to a multiple regresoion<br />
equation of a variable which ir a linear function of variables already<br />
included cAnnot increase the multiple correlation coefficient. Even<br />
using the ratio of the two should Add little. The correlation between<br />
the two types of indexes USUALLY approaches unity. ,<br />
However, in one part of Brother Bates’s study, the effectiveness<br />
index formula was developed from a total entering Freshman group (381)<br />
and then applied to the 274 students who graduated. When combined with<br />
the other predictor variables, these effectiveness scores increased ..he<br />
multiple correlation with first year college marks .OS points, which was<br />
significant at the .OOl level. Furthermore, the inclusion of this total,freshman-group-bnsed<br />
effectiveness index in the regression equation to<br />
predict average of allcollegeEarks for four years yielded the remarkable<br />
increase of .13 in the R. The canposfte of averc.:e high school marks,<br />
Essential High School Content Battery (the achievement test used in<br />
developing the effectiveness index) and the American Council Psychologic’s<br />
Examination (the aptitude test used),yielded an R of .56, which WAS<br />
raised to .69 by the addition of this type of effectiveness index.<br />
Incidentally, due to the hfgh reliability of the tests used to develop thlcm,<br />
these effectiveness indexes had A .93 reliability coefficient.<br />
This same procedure possibly could be adapted to the military situation.<br />
It should prove useful and relatively easy to develop in connection with<br />
successive levels of training. For the Army, an effectiveness index for<br />
a given F@S based upon the best combination of aptitude measures vereu8 t.hs<br />
Enlisted M0.S Evaluation Test scores and derived from a total Ho.9 skill<br />
level population might increase significal:tly the prediction of the future<br />
proficiency and promotability for soldiers in one pay grade or other<br />
identifiable auhgroup. Other possibilities for meaningful rubgrapa<br />
might be devised and checked out. Verification of the applicability to<br />
OCCupationAl dpecialties of the findings in Brother Bates’s dissertation<br />
offers the possibility of yielding A simple inexpensive way of significantly<br />
increasing validity. A true moderator VAriAble seems to be involved here.<br />
81<br />
;
. . .-.<br />
._ .<br />
.<br />
-. -.<br />
Anyone investigating this area further will find several references<br />
helpful. A publication edited by Drs. Dubois and Hackett includer reports<br />
by Dr. tlayo and Dr. Froelfsh referred to by Lt Longo. Dr. Harris’s book<br />
contains an excellent presentation of soma of the complex statistfcal<br />
problems and theory involved in this area of research. Dr. Thorndike’a<br />
little monograph examines closely the concepts and methodological problems<br />
and makes specific suggestions for sound research studies in thia<br />
rather tricky area.<br />
The lines of approach reported in Mr. Kraft's paper offer some seminal<br />
points of departure. III fact,the greatest hope for a major increase in<br />
percentage of job proficiency measured probably lies in obtaining more<br />
reliable and precise measures of the aspects of personality related to<br />
job proficiency--better ways of measuring the factors comprising the<br />
“will do” of enlisted personnel as identified by psychological job<br />
analyses and research. There is a good indication of the soundness of<br />
this conclusion, On 3 August 1964 in a letter on “Revision of Army<br />
Aptitude Area System”, Headquarters, US Continental Army Command,<br />
stated as follows:<br />
. . . Numerous school-conducted studies have shown that age,<br />
previous education; previous civilian and Army experience \<br />
of students frequently correlate hi&her with course performance<br />
than do ACB test scores. It is recommended that<br />
appropriate statistical precaution be taken by USABRO to<br />
Insure that factors other than test results do not contaminate<br />
the validation.<br />
“Evidence presented at an April 1964 USCCMRC Basic<br />
Electronics Conference, plus observations of numerous<br />
key personnel throughout the USCONARC school system, indicate<br />
clearly that subjective factors, such as attitude,<br />
desire to study, perseverance and other non-intellectual<br />
attributes of USCONARC school students have as much or more<br />
influence on courses performance of these students than<br />
“aptitude” as measured by classification test battery such<br />
as currently employed or contemplated by USAPRO. To the<br />
extent that inputs to USCOMRC school and training center<br />
courses are governed by tools thst take into account only<br />
those cheracterfstics readily measurable and ignore more<br />
significant - though admittedly subjective - factors,<br />
there will be less than optimum regulation of manpower<br />
flow through the Army training system. USAPRO is urged<br />
to consider the development of measures of attitude,<br />
motivation and desire to learn as an integral part of its<br />
program to revise the Army Aptitude Area System.”<br />
.<br />
02<br />
c<br />
.<br />
.
.<br />
.<br />
.<br />
In their reply the US Army Personnel Research Office stated:<br />
..* the background data reported will include not only<br />
ACB scores but also such factors as age, cducatkon,<br />
cfvilfan experience, etc. These variables wtLK be included<br />
in the analysis with the experimental test data.<br />
“The non-cognitive factors referred to *.. have indeed<br />
been demonstrated to be important in training performance,<br />
and in jcb performance as well. In this regard<br />
it Is worth noting tbct the erperinen:al battery<br />
contains several non-aptitude type measures, designed<br />
to evaluate those enduring interests, attitudes, rind<br />
other personal characteristics which help determine<br />
what a man - vi11 - - do - rathtr than cnly what he can - do. -<br />
One goal in the revised ACB is to provide a<br />
Classification Inventory which vi11 measure personal<br />
characteristics predictive of performance in occupatfonai<br />
areas such as electronics, mechanical maintenance,<br />
clerical-admlnfstrative, and other areas, just as<br />
the present CI measures personal factors which predicted<br />
combat performance in the Korean War. 2<br />
-addition - to these enduring --s--Y characteristics. - - -a__- however *<br />
there are factorof my.--_ a~otivstion which .e-are prfmnrlQ<br />
-situatfocel. - The classification br,ttery cannot predfct<br />
these, but this r Isearch program must take Lnto<br />
account the effects of such factors on the flndinge.<br />
It will be very helpful to the USAPRO scfentieta if,<br />
in conjunction with thfe program, the training installationr<br />
can contribute inrlghtr end data on such<br />
factors.”<br />
In this theoretical sympooio?n M have been able to little more thrrr<br />
touch on possible approaches to more effectively measuring some of the<br />
eight types of characteristic8 listed in the tntroduction. Thin afternoon<br />
a fev more possible approaches wfll be presented, Some should<br />
prove productive, and all should be stimulating. For example, in some<br />
occuprtfonal specialties even gross methods of controllfng item readability<br />
should yield prcfitable improvements especiallqr in tests for occupational<br />
specialties in which the level of reading ability of incumbents f~ a<br />
significant contributor to invalid variance of Hcorca. These improvements<br />
should be even greater for such tests i.f three 8teps can be accomplished.<br />
First, If a more precise item readability index can be developed instead<br />
of using adaptations of the gross sampling methods that are adequate for<br />
masfies of regular prose. Second, if a practical method can be developed.<br />
for correcting this index for special technical terminology characteristic<br />
and counon inthe occupational specialty. Third, if some means of determining<br />
the distribution of reading ability in each specfalty ia<br />
feasible.<br />
83<br />
---- .-. ..- .--_ -_8_-.-.-- ..- - ._ -.-. -..- ._ .__.(._<br />
.<br />
*<br />
k
:/<br />
. - . -_ _* .-- - -.<br />
-- . .<br />
. . ’<br />
. \<br />
./:.<br />
‘,<br />
.\<br />
-. .<br />
. . -<br />
.<br />
.<br />
--.---_ -- .-._. ---- I- ._- . _ _ - __--------.._<br />
To mention one more forthcoming approach, the “Performance Check Test”<br />
concept may offer a practical aad logically 8&nd way of effectively<br />
measuring motor skills in tho8e occ’upational apeclaltiea for vhich such<br />
measure8 are important and for which the motor skills meet the required<br />
criteria.<br />
Haking what Dr. Edgerton identified a8 genuine change8 in testing<br />
obviouely will require much effort, ingenuity, and a fortuitous concourtit<br />
of circumstances. .<br />
T &us confident that at leaet 8ome of the profe88ional 8taff fras<br />
each of the military services and in other research agencier will<br />
� ggreeeivaly pursue any inrightr that may be stimulated by these 8ympo8ia.<br />
We have much at 8takC and many of the 8aae problem8 in coamon a8 indicated<br />
both by Hr. Johnson’8 mmxnary analysi8 and cauparf8on of the programs<br />
of each agency and by your answer8 to the program planning qutstionnairer.<br />
These tend to emphasize the coaxson lnterert in exploring wnyr of obtaining<br />
marked increase in the percentage of overlap between job factorc and<br />
the evaluation8 of enlieted ptrronatl. You relectcd a8 tha theme for<br />
this conference, “Xncrea8ing the tleaauring Efficiency of Evaluation<br />
Instrumento.” Your rerponse8 to the program planning questionnaire8 indicate<br />
that, regardlee of the current emphasis in the evaluation program<br />
of the five services, an inwcdiatt goal of each service 18 the objective<br />
eValUatlon of as many significant factor8 iu Currtnt job Mattry a8 ie<br />
practical. For all uae8 of the respective programr., apprairing the<br />
Current ltvtl of total job proficiency is importsnt. Por promotion8, the<br />
ft.ture level of job proficiency is the important intermediate criterion.<br />
Of course, for all of the servicer the ultimte crittr1c.i is vhather or<br />
not we win any military action and, for the individual enlisted man, is<br />
hov well he function8 in hi8 a8signtd job during such action.<br />
� �<br />
.<br />
� �<br />
I .<br />
- .- _ _ _ - .-..- - ---_.- . _-_-- ---<br />
.<br />
.<br />
.<br />
I .<br />
. .<br />
, .<br />
.<br />
�<br />
.<br />
.<br />
.<br />
.<br />
.<br />
.<br />
b<br />
.
.<br />
- _ _<br />
-<br />
.<br />
- - . .* _<br />
.<br />
.<br />
.<br />
-M1_1-_---.--“--- --I<br />
><br />
Reference8<br />
.<br />
-. .-<br />
.<br />
_... ._ _.- . . ..--.. - - -<br />
B&e., J. C. An inalyrir --I--.<br />
of some of tha concomitant8 af learning<br />
afftctfve~rs. PhD dferertation, St. John’8 Univerrity, New<br />
York, 1965.<br />
Cronbach, L . J. Elrentlalr of prpcholoRicr1 testing (2nd ad.).<br />
Nev York: Harper, 1960, 589.<br />
Dubofr, P. H. & Hackett, E. V. (edr.) Tb8 &8Ure¶IMIt and evaluation of<br />
over-and under-•chlsveznant. St. Gi8: Waohingto~nlverrlty, I.%2.<br />
time, R. H. The � cquirltfon of knowledge. P8whol Rev, 1962, 69, 41,,<br />
355-36s.<br />
Sagne, R. H., Major, J. R., Carrttnr, Helen L., & Paradise, N. E.<br />
Factor8 in acquiring knwltdge of a mathcmotfcal tark. P8ychol<br />
Monogr, 1962, 76, No. 7 (whole No. 526).<br />
Gagne, R, H. & Paradire, N. E., Abilititr and lernfng ret8 in knowledge<br />
8cquirition. Plychol Mono& 1061, 75, No. 14 (Whole No. 518).<br />
. - . _ _<br />
Harrla, C. W. (cd.). Problems & mearurlng Mchrnge. a d i s o n : T h 8 Unlvrrrfty<br />
of Wirconrin Prt88, 1963.<br />
Xillsr, R. B. Handbook 0” training & Jraining equipment dt8iRn.<br />
Wright-Patterron Air Force Base, Ohlo: Wright Air Development Cmnttr.<br />
Technicel <strong>Report</strong>, 53-136, 1953.<br />
HilIar, K. 8. A method for man-machine taok ana?..ysir. wright-Pattsrroo<br />
_I-<br />
Air Force ~8~i~Wrlght Air Development Center, <strong>Technical</strong><br />
<strong>Report</strong>, S3-137, 1953.<br />
Miller, R. B. Soma Pittrburglh:<br />
workin& COnCtpt8 of ryrtem8 analymir.<br />
American Ixtute for Rt8earch. 1954.<br />
Hilhr, R. B. Tark and part-task trelntrr. Wrfght-Patterron Aft Force<br />
- -<br />
Bole, Ohio: Wright Air Development Center. Ttchnlcal <strong>Report</strong> 60-460,<br />
ASTIA No. AD 245652, 1960.<br />
Miller, R. B. 6 Van Cott, H. P. The dtttmfnation of knowledge content<br />
-me<br />
for complex mm-machine jabr.-Pittrburgh: Ams&an Inrtttutt for<br />
Search, 1955.<br />
. Thorndike, R. L. The concept8 of over-and under-achievement. New York:<br />
Bureau of Pub-ationr, Tezherr College. Columbia Unlverrlty, 1963.<br />
85<br />
, !<br />
k;<br />
.<br />
. 1<br />
!<br />
/<br />
,
-<br />
._<br />
.<br />
. .<br />
. .<br />
\<br />
.’<br />
.-<br />
.--<br />
.:.<br />
‘.<br />
-<br />
-. _-<br />
.<br />
.-<br />
_<br />
_..<br />
-.<br />
t-.. .:.<br />
-<br />
_.<br />
. ;<br />
-. -7<br />
- .zsI,:<br />
- :<br />
---.i<br />
: -.+,<br />
.-<br />
.<br />
. -w.<br />
. *<br />
-. .<br />
. . m .<br />
.<br />
.<br />
---- -.- ..--.. ._ _. __ -__. _.-. -__-------.-- ._i-_.. - .- ---<br />
THEOREnGAhfYlsl~OSlUfali-LEROYJOHI1SlOn,CHAI~AR<br />
USWAVALEXAIBI#IIGGE#TEB<br />
Relationship Between Raadability and Validity<br />
of 140s Evaluation Tests<br />
.TCXW S. BRAND<br />
US Army Enlisted Evaluation Center<br />
Cronbach states‘in his book Essentiala of Psychological <strong>Testing</strong>:<br />
“Tests having the same ‘content’ may measure different abilities because<br />
of variables associated with itam form. Reading ability, for example,<br />
affects scores on almost all achievement tests. A valid measure of knowledge<br />
is not obtained if a pearson who knws a fact misses an item about<br />
it because of verbal difficulties.” Cronbach goes on to ahow the superior<br />
validity of picture items over verbal items.<br />
A number of methods have been proposed for measuring the readability<br />
or reading difficulty level of written matter. The principal ones investigated<br />
by this offfce are those of Gunning, Flesch, and the Farr-Jenkine-<br />
Paterson modifi:ation of the Flesch method. These three methods, which<br />
are essentially variants of the same basic principles, were initially<br />
applied to several samples of written matter. It was concluded that the<br />
results obtained from the three methods were essentially the same. The<br />
Gunning method, however, has the advantage of being simpler, and was<br />
therefore adopted as a method for the measurement of readability.<br />
This psychologist is only aware of two papere dealing with the<br />
readability of tests. Neither of these present methods for determining<br />
the readability of individual test itema.<br />
Long words and long sentences have baen shown to be the principle<br />
determiners of readability. The three methods mentioned above all use<br />
average sentence length as one determiner of reading difficulty. In<br />
determining average sentence length, independent clauses are counted<br />
as sentencea.<br />
Flesch counts all syllables to determine the average word length.<br />
Farr-Jenkins-Patterson suggest that the same information may be obtained<br />
more easily by counting one-syllable words. Gunning counts words vith<br />
three or more syllables to determine percentage of “hard” words.<br />
--_-_. -.. .._ _ - - -<br />
.<br />
--s-- --<br />
. . . . :<br />
86<br />
.<br />
.<br />
_--___-. ..-._. --<br />
’
-.<br />
. :a.<br />
.\<br />
-.<br />
-I-<br />
.’<br />
-.., 7<br />
-a!<br />
i<br />
-.-<br />
.- /-<br />
-, *-<br />
.‘.<br />
-..<br />
,’<br />
-<br />
..- ._<br />
.-<br />
. .<br />
/’<br />
-: ,-<br />
-y _<br />
--<br />
$11<br />
- .-<br />
-<br />
- -<br />
--___<br />
.-_<br />
.<br />
.<br />
In counting three syllable wOrd6, Gunning makea the following<br />
exccptlona:<br />
verb forms that are made three syllable8 by the addition<br />
of -CI pr med. For example: reducte, invested<br />
proper nouns or word8 that are capitalized<br />
combinations of simple words, like bookkeeper<br />
t&nning’S Fog Count, as he Call8 It, i8 found by adding average<br />
sentence length and percentage of herd words, end multiplying the resulting<br />
am by .4. The Fog Count is a grade equivalent. Twelve equals high school<br />
graduate, 16 equals college graduate, etc. I have celled the acore<br />
resulting fran the application of Gunning’8 method a “Readability Index”‘,<br />
or RI.<br />
The fOrUtUla8 develdped by Gunning end Flesch vere validated against<br />
the McCall-Crabbs retding tests which were etandardized against ext,enslve<br />
student population8. This critcriOn is admittedly not a good one for<br />
adult reading materiels, but it vaa the only one available for that<br />
purpose 0<br />
Application of Gunn’ng’s Fog Index to MOS Evaluation Teat Ire.2<br />
Plesch end Gunning recommend taking loo-word samples of long articles<br />
to determine reading difficulty. Counting only one alternative to avoid<br />
excts8ive repetition, the average test iten has about 20 words. This 01<br />
admittedly a small sample. However, much Useful information may be gained<br />
if a readability index can be determined for the individual item. An<br />
echlevement-type item will contain one or more complete sentences to<br />
determine average sentence length, end the number of 3-eylleble word8<br />
in en item can be counted to determine percentage of “hard” words.<br />
In counting 3-syllable worde, decieione mU8t be made about counting<br />
number8 and eerles of lettera, or numbers and letters, which occur feirlly<br />
frequently in MOS evaluation tests. In this respect, it is believed<br />
that Gunning’8 ryrtem is more adapteblc to determining the readability<br />
. of HCS evaluation test items then that of Flesch.<br />
Determination of Validity (ric)<br />
A criterion of job performance consisting of 3 peer ratings per<br />
subject on an overall performance scale hae been obtained by this Center<br />
for samples of EM in selected NOS. These ratings vere obtained in field<br />
trip8 to selected military installations. Item validity (ric) was<br />
obtained by the point bi8eri81, or Pearson correlation between item and<br />
criterion.<br />
. ,<br />
.<br />
87<br />
.<br />
.
“.<br />
.I<br />
-..-<br />
--;-<br />
_, ‘5<br />
.’ ,<br />
I<br />
. --_--<br />
--- - -*. --<br />
. . -<br />
^<br />
.<br />
.<br />
a<br />
.<br />
- -<br />
._ _. .,-. _ - _ _..- .___ . _.... . .._<br />
Analysis<br />
A bivariate distribution ~58 prepared betwoen readability level,<br />
shown on the horizontal axis in intervalt of 3, and ric, shown on the<br />
vertical axis, in intervals of .lO.<br />
Correlational analysis to detect nonlinear r&%tfanshipr of one<br />
sample produced nonsignificant results.<br />
The ric was cmputed for each column of the scatterplot. E5ch<br />
column represent5 different levels of reEdability ic intervals of 2,<br />
e.g., 2-3, 4-S, etc., through 22-23, 24-25, etc.<br />
The items of the teat were divided in several way8 on the basis of<br />
readability level. The-tic was computed for items above a certain readability<br />
level, and the ric was computed for iteXM below this level. If<br />
the former is lower than the latter the sigcificance of the difference<br />
was tested. A one-tailed te8t was used. Degrees of freedom for each<br />
ric vas taken aa the number of i:ezns included in computing the ric times<br />
N-3 of the validation sample. In other word5, each item contributes<br />
N-3 degree5 of freedom. .<br />
The critixal readability level WBR tnken oa that diviafon point which<br />
yields the largest s score or critical ratio betaeen the ricls obtained<br />
for the two parts of the test.<br />
The yic’s so obtained were also tested for significant deviation<br />
from zero.<br />
To dete,rcsults have been obtained in four MOS. Results for two<br />
of these are shown in the following tables. These two MOS include<br />
some 40,000 EM.<br />
.<br />
88<br />
_ ._ ..----.-.- -... .-. --__- --<br />
.<br />
.<br />
.
- .-_<br />
.!<br />
.<br />
.<br />
.<br />
__. _. . ._<br />
1<br />
_____.___ .-.-_.. . - . ..--- --.---. -. -- --- -.-<br />
. .<br />
--_ .-,,__ -____ -2 _._. _,_ . I ,-. . -.<br />
-._- --_a _-._____ _________, __-_- -..____._ .--a. .-._ . --.- -<br />
�<br />
.<br />
.<br />
MOS 94l.i, cook6 .<br />
Inrpection of the colum Gic’6 shown in Table 1 suggert that item6<br />
above a readability level of 9th grade fail to predict the criterion.<br />
The ric for 53 lteme with RI greater than 9th grade lo. .006, which ie<br />
not significantly different from zero. The ?i, for the 44 items with<br />
RI of 9 or lesr is .OSl. Thi6 Tic ir 6igdfiC6ntt,~ different from<br />
�����<br />
The difference between the two ‘Tic@6 is significant at the .Ol<br />
level.<br />
It is therefore concluded that the 53 item6 with RI greater than<br />
9 contribute little to the volidity of this MOS Evaluation Teet.<br />
The obtained validity of the test was .lO, Whatever validity the<br />
test ha8 appear6 to cane primarily from itUiI6 with lower level6 of<br />
readability.<br />
MOS 111.6, Light Weapon6 Infant-n<br />
Result6 for the LOO technical item6 are rhown at the left of Teblo<br />
If, and reault6 for the 25 supervisory item6 are ahown on the right.<br />
In the technical test,‘18 ittD6 with RI of 20 or higher have an<br />
a&rage validity of 0.014. The validity of the te6t, therefore,-which<br />
is .25, derive6 principally from item6 with a RI oelow 20. The ric<br />
for it-6 with RI below 20 is .065, which is significantly different<br />
from zero. The difference between the two riC’6 is significant at<br />
the .05 level.<br />
The supervieory tert can be divided st 14 itIm6 with RI up to 13,<br />
and 11 items with RI of 14 or higher. The Fi,‘s are respectively .086<br />
and .035. The former ie oignificantly different frcm zero and the<br />
latter i6 not. The difference between them ie t!ot rtati6tically significant.<br />
If the trend6 reflected in the rerults obtained to date are born<br />
out by additional MS 6emple6, it is believed that a strong relationship<br />
will have been demonstrated between the readability of teet items and<br />
item-criterion correlations. Step6 can then be taken to write test<br />
item6 at appropriate levels of readability. The result should be a<br />
significant increase in the power of items to predict job proficiency<br />
criteria.<br />
89<br />
.<br />
,*<br />
.<br />
.<br />
.<br />
/<br />
,.<br />
.<br />
I I
.=-<br />
. -’<br />
_<br />
,“ .<br />
.<br />
-<br />
. . .<br />
_- - . -.<br />
,<br />
-- .<br />
. . e<br />
. _ . ._~ -------.-..-<br />
- .A: .__ ._ ._<br />
_<br />
.<br />
L<br />
TABLE I<br />
MOS 94tl COOK N= 129 (3peer ratings) Apt. Area GT<br />
. .<br />
.<br />
Va I.= . IO<br />
24<br />
21<br />
Rt<br />
It3<br />
6<br />
RIla $ f z<br />
+tc .04 .06 .05 .05<br />
QC .05l<br />
tf 3.92<br />
p< .OOOl<br />
Omltted “M- 1337”<br />
beg* 8<br />
02 DO .02 DO -.02 -.06 .Ol<br />
.006 2 =2.50<br />
p< .OI<br />
.<br />
.<br />
-
s-t g<br />
CD L-9 "0<br />
aI<br />
- - -I -, 6-8 g<br />
4 v<br />
N a<br />
C!li! if 1: I<br />
1 ! p-4. 1 ! q 1 p-y<br />
! -1 -1 -1 ‘?<br />
I 61-W<br />
6 --<br />
Ci:!! :I.<br />
I I TI I<br />
12-02 0, =<br />
=j I--- = 12-02 is<br />
.<br />
16<br />
,<br />
.<br />
___ I.. ‘s<br />
.-...C -q.-~...v..-- --.-___ - __ _ . . ^ .- . ..__._.-_-- ---.---.------ _ _ _.^__<br />
_ & ._I. .,<br />
. . .._ _-. .-..<br />
.<br />
.<br />
_ .-- -- ___-.. . _.._ - .-- .--.<br />
. .
Performance Check Tests<br />
CLAUDE P. BRIDGES<br />
US Army Enlisted Evaluation Center<br />
Background<br />
For some occupational specialties, one way of making significant<br />
inroads into the variance in job proficiency currently unmeasured would<br />
be to measure more directly and effectively proficiencies involving motor<br />
sKills.<br />
In some occupational specialties, physical manipulation skills (nctor<br />
skills or the product5 thereof) are crucial and important determinators<br />
of significant portions of job proficieacy levels. The development of<br />
adaptations of written test techniques uhic!r will adequately measure such<br />
manual skills, ,at least indirectly, msy be possible ultimately in some of<br />
these jobs. The correlation between a man’s knowledge and his use of the<br />
knowledge on the job often tends to be quite high. However, there are<br />
several occupational specialties in which direct measures of such physical<br />
manipulation skills still will be necessary for good coverage of the job.<br />
In spite of the difficulties inherent in the world-wide use of performance<br />
tests, it has been possible, in the US Army Enlisted Hilitary<br />
Oc-:upational Specialties (HOS) in which such skills are most basic to<br />
differences in job profit-ency, to develop performance tests which can<br />
be administered world-wide under standard conditions and the results<br />
evaluated in an acceptably standard manner, However, such tests are quite<br />
expensive to develop, administer, and evaluate--often prohibitively so.<br />
As a result, it is quite important that every effort be made to explore<br />
other possibilities for directly evaluating motor skills and their<br />
physical products and to include such new measurement techniques among<br />
the evaluation instruments.<br />
This paper presents one such technique and the procedure for developing<br />
the required instrument. 0rl;inally I called it a Performance Check List.<br />
However , thie term is already used commonly for a significantly different<br />
type of instrument. In order to emphasize one of its two most important<br />
characteristic5 I now refer to it a5 a “Performance Check Test,” or in<br />
short a “PCT . ” On the basis of extensive experience in this measurement<br />
area, analysis of the pertinent reseqrch literature and logical considerations,<br />
the technique proposed herein can be expected to function effectively<br />
in the occupational specialties for vhich it is most appropriate, i.e. in<br />
the jobs which meet all the specified criteria.<br />
92<br />
�<br />
.<br />
.<br />
I
i’<br />
I<br />
.<br />
.<br />
When developed with ing;t,2uity, we may be surprised at some of the<br />
areas in which PCT’s would be appropriate. For example, Dr. Owen Waveless,<br />
an international specialist in linguistics, assures me that the development<br />
of an Instrument along these lines for interpreters should be ponsible.<br />
He said that such an instrument could be used effectively by an observer<br />
NOT skilled In a given language to evaluate an axaminee’a ~aneuaw’imJficiency.<br />
Even though it would NOT be a canpletely adequate aubstltute<br />
for more precise measurlas yielded by a good performance test, it would<br />
be much better than a traditional rating scale alone.<br />
. WHAT IS A PERFORIWXE CHECK TEST?<br />
- -<br />
.<br />
Y<br />
The proposed instrument is a true test developed In much the same wa,y<br />
aa a good written test, but the items are answered about--not by--the examinee<br />
and each item describes one specific mnnipulation, skill, or<br />
product that is marked by the observer, or supervisor, as being observed,<br />
or not -brerved in the examinee’s performance of the specified tasks of<br />
his job. It differs from the usual performance check list in that each<br />
item has been selected on the basis of experimental analysis (using<br />
standard item analysis techniques) frcxn a much larger list of specific<br />
proficiencies that first, can be readily and directly observed or<br />
inferred, second, can be so defined that any qualified observer would<br />
consistently state that the soldier observed either did or did not odequcll;ely<br />
perform the defined manipulation,and third, which distinguish between<br />
characteristics of various levels of proficiency. In some instances the ’<br />
manipulation described by the item might be a specific task. H,-wever,<br />
most items wou!d be limited to distinct individual skills or manipulation5<br />
involved in completing such tasks. For example, one item in a performanoe<br />
check test for a bandsman might be,“Consistently produces high C on his<br />
inetrument when called upon to do so.” A bandmaster should be able to<br />
answer this question about every one of his bandsmen, even without any<br />
special observation of them.<br />
Different items should describe consistently observable levels of<br />
skills possessed by examineee with different level5 of job proficiency,<br />
Some items should describe skills whfch are possessed only by the most<br />
proficient soldier in the MOS; Borne should cover skill8 which are<br />
porseoeed by the highly proficient but not by a man with average prof<br />
iciency; come should cover skills possessed by the average man but not<br />
poclstseed by those low in proficiency; a few should cover skill8 that a<br />
man low in proficiency can do but that a novice with some familiarity with<br />
the ffeld can not do at the level described by the item. Naturally,<br />
neither items covering things which practically everyone in the job can<br />
do, nor items which no one can do should be included for measurement purposes,<br />
.<br />
93<br />
---<br />
-<br />
,<br />
, . !<br />
1 .<br />
I<br />
i<br />
6<br />
,
,<br />
s<br />
.<br />
. ---_.<br />
. . .<br />
.<br />
.<br />
- -.--. .-. ._- .*__ ____-.._ -.-..-- -- --.. _.-<br />
“-. _-- ._-__.- .- ---<br />
Ideally, the items should NOT require the observer to evaluate the<br />
level at which the individual performs the skill covered by the item.<br />
In ‘other words, it should not be necessary to make qualitative judgment<br />
or inferences in deciding uhether or not a given exadntc performs the<br />
skill in the manner defined by the item.<br />
In btmary, the most important distinguishing characteristics of a<br />
Performance Check Test are that (1) the performances can readily be observed,<br />
(2) they can be consistently judged as being present to the degree described,<br />
or absent, and (3) they are selected by appropriate item analysis and<br />
validation techniques.<br />
The Performance Check Test might be considered as consisting of a<br />
special type of rating scales. Honever, the PCT is not a usual type of<br />
rating scale. It is a li.st of things which an observer can easily determine<br />
that a man can or cannot do. These should be weighted on the<br />
basis of experimental findings which will indicate each Item’s Importance<br />
as a component of the exsminee’s effectiveness cr job proficiency. The<br />
quantitative expression of the level of skill in the manual manipulative<br />
aspects of the job vould be obtained mensly by adding up the vei&tc<br />
assigned to the items checked for the individual evaluated on the<br />
Performance Check Test. A total score reliably and validly reflecting<br />
the exsminee’s skill is obtained. It covers discriminating motor aspects<br />
, of the job. When appropriately weighted and combined with the written<br />
test and other job,proficiency evaluations, a significant increase and<br />
in some jobs a considerable increase in validfty should result. Thus<br />
additional inroads into currently unmeasured variance in job proficiency<br />
should result.<br />
SlJHKARY OF DEVELOPMENTAL PROCEDURE<br />
1. ldenttfy the jobs for which current evaluation instruments are<br />
least valid in which motor skills or their product are important<br />
discriminators of d!.fferences in levels of job amstery, and in which<br />
the tasks involved most adequately meet the criteria, (See Annex 1).<br />
2. Analyze all of the pertinent information available or readily<br />
attainable and prepare a preliminary list of items to serve as an<br />
illustration for the subject-matter experts engaged in developing a long<br />
experimental list of potential PCT items. The “critical incident” approach<br />
developed by Dr. John C. Flanagan would be quite helpful. The “essay<br />
approach” proposed by Hr. Roberts, an USARES Supervisory Research Psychologi at,<br />
and his staff should be another helpful source of items. (See Annex 2 for<br />
their materials.)<br />
3. Convey these examples together with guidelines and appropriate<br />
accessory materials to the appropriate experts on the occupational<br />
specialty involved in the organizations at which such experimental items<br />
94<br />
.<br />
._
.<br />
.<br />
are to be developed. Discus6 with the subject-matter experts the posl;ible<br />
variation8 of the procedure 80, that they have clearly in mind what is<br />
deefred and how to provide it. (See Annex 3 for a list of criteria for<br />
m iteIII8.)<br />
4. Develop an experimental form of the PCT, coixsidering the "item<br />
criteria” lieted in Annex 3 and including all of the Items which are judged<br />
to be promising.<br />
5. Administer this experfmental form to a group of men working in<br />
the occupational specialty, a8 close as fe feasible to the date on which<br />
the specialty is being evaluated by the regular instruments. In the<br />
rample used, all levels of proficiency should be reprerented. At least<br />
two, and preferably three or more-observers of each men should apply the<br />
experimental PCT. At the 8ame time the be8t posrible externel criterion1<br />
data should be obtained,<br />
6. Analyze the item8 and the possible ways of scoring each against<br />
the total score on the experimental form of the PCT, against the written<br />
test, and against the external criterion. Obtain correlation8 also with<br />
all of the regular evaluation instruments. The consistency with which<br />
each ttw can be used will be analyzed. Techniques similar to thorc, used1<br />
fa refining biographical information blank8 and to regular item analyser<br />
should be applied. The desirability of assigning other than unitary<br />
weights, to the individu81 item8 In the check teat, should be determined<br />
empirically. The most valid and relipble scoring points for each item<br />
should lfkewise be determined empirically.<br />
7. Coneiderfng all the available data and the pertinent general<br />
measurement criteria, develop the revised form of the Performance Check<br />
Teat. If extensive modification of a eignlficant proportion of the experimental<br />
items has not been neceeeary, a useful estimate of the reliability<br />
of the total score on the revised Performance Check Test @CT)<br />
can be obtained at this time. The data also can be analyzed in relation<br />
to minimum acceptable proficiency 8nd promotion. Optimal Weight6 of all<br />
the evalucltion instruments naturally should be determined for the experimental<br />
tryout population snd appropriate personnel action8 determined<br />
for critical “cut score8.”<br />
Por jobs in which it ir pO8Sible to obtain an adequate number of PCT<br />
items, each of which can be consistently observed in a performance as<br />
meeting or not meeting the deecribed limits--e.g. for which the observer<br />
can reliably say,“This man can, (0. cannot) do this thing as specified,”<br />
and when this crucial statistical .-nalysis la made, the effectivenees<br />
of this Performance Check Test technique should be assured, The percentage<br />
95<br />
.
. _<br />
.<br />
‘. -7. . . .<br />
__ ‘. � ��<br />
���<br />
� �<br />
:<br />
�<br />
� �<br />
�<br />
�<br />
�<br />
�<br />
.<br />
__ . -.-_- _... . . . . ..- :<br />
_..-. -^-. -_ -. -... -<br />
of the veriancc in job proficiency mearured by the battery of evaluation<br />
inrtrumanf~ rhould be appreciably increased in any job in which motor<br />
� kille or their direct product are rignlficant detemlnerm of differencea<br />
in job/master, and thum in job proficiency.<br />
Reference<br />
Adkinr, Dorothy C., Primoff, E. S., McAdoo, H. L., Bridges, C. P., &<br />
Form, B. Conrtructioa and analysir of achievement teata.<br />
Warhlngton, D. C.: ;IS Go~nment Prfncng Officr, 1947, 211-265.<br />
96<br />
_. _ _-_ _.. -_ .- ---.- _-__. .._ --- --.- --<br />
. .<br />
*<br />
.<br />
: .<br />
. .<br />
.<br />
.<br />
,<br />
. .<br />
:<br />
* ‘% . -<br />
.<br />
.<br />
.<br />
.
---<br />
.<br />
.<br />
.<br />
.<br />
Aritm 1<br />
CRTTRRU FOR IDXNTIFYINC APPROPRIATE OCCUPATIONAL SPECIALTIES<br />
The general cha=actcristics of occupational apacialties in vhich<br />
performance check test6 might be expected to make a significant contrlbutfon<br />
tovard Increased validity are mmarizcd belov:<br />
a. Performance Check Tests should be considered for occupational<br />
specialties in which special motor skills or the direct product of such<br />
skills are critically involved. These rkille must NOT be comton to most<br />
men. Protzlnent among such occupational specialtie8, probably should be<br />
those having as their principal duty the operation, and perhaps in some<br />
instancea the maintenance of equipment.<br />
b. The motor okills ig the job must be consistently end readily<br />
obrervable to a. clearly definable level by an adequately informed evaluator.<br />
c. The rotor skill8 in the job must be crucial determinators of<br />
differences in level8 of proficiency, I.e., the physical skills must play<br />
an important role in discriminating betveen relative level8 of proficiency.<br />
Frequently MOST of the duties of such HOS vi11 entail special physical<br />
okills employed In the manipulation, use, and/or adjustment of toolfi,<br />
equipment, mechanism end/or meter:als. Typical are the equi-ent<br />
operator’e, .drivers, mechanic8 . machinfs t8, bandamen, fmne maintenance<br />
technicians, and 8ome automatLc data processing equipment technicians.
- --<br />
..a--<br />
\<br />
----+.<br />
-<br />
- i_<br />
7<br />
. . .<br />
- -7<br />
.)<br />
.<br />
-‘L. ..I<br />
yi-<br />
-+<br />
._ ..;<br />
. .._.<br />
.-<br />
‘..<br />
. . . +<br />
. ... ._<br />
- a.<br />
- s.<br />
,<br />
-7 ‘_..<br />
A<br />
.-<br />
,.<br />
--<br />
_. .-<br />
.-<br />
.-<<br />
.?-F-<br />
--CL<br />
_ .:-’<br />
- :<br />
.- .,<br />
_.<br />
.- _-.<br />
- _<br />
.<br />
“.<br />
:m.*<br />
,- <<br />
‘.<br />
$<br />
*<br />
.<br />
ANNEX2<br />
EXPIANATION<br />
Preeently, bandsman performance tests are being administered by<br />
having bandsmen tape record certain standard musical passages. The tape<br />
recordings are, then evaluated by specially selected and trained audition<br />
boards. This method of performance testing is time consuming, expencive,<br />
and pose8 many problem8 from an administration and scoring viewpoint.<br />
Any difference8 in scoring from one audition board to another decreases<br />
the reliability of the acore proportionally.<br />
The US Army Enlirted Evaluation Center ia conducting a study<br />
concerned with the development of a bandsman performance evaluation scale(s)<br />
which can be completed while the supervisor observes the bandsman performing<br />
hi8 duties fran day to day and/or by having the individual<br />
bandsman play special music passages or perform special duties.<br />
To construct bandsman performance evaluation scales, it is necessary<br />
to obtain information from as many sources as possible concerning the<br />
fmportant specific performance behaviors that are demonstrated by both<br />
good and poor bandsmen in the performance of their required duties. XndiVidual8<br />
holding a particular MOS are among the beot equipped to know<br />
what are the important types of behavior. One way of collecting this<br />
information from the individua’l bandsman ie to have him write a short<br />
essay about the individual he consider8 to be the best performer in his<br />
ttOS and to urite a short eosay about the individual he consider8 to be<br />
the poorest performer in his MOS.<br />
You are requested to write essays on the attached sheets in accordance<br />
with the brief directions given at the top of the respective sheets. One<br />
sheet la to describe the best performer you know; the other sheet is to<br />
describe the poorest performer you know.<br />
98<br />
. .<br />
_-. . _ . .<br />
__ __- _.-_ . -.__-- -.<br />
c<br />
,<br />
.<br />
-<br />
.<br />
.,‘!<br />
.<br />
.<br />
.
Page 1<br />
GR4ia --<br />
HOS --<br />
Length of<br />
Time XOS Held<br />
Total Years Hueic<br />
Training<br />
--<br />
Total Year8 Experience<br />
8a a Uuafcian --<br />
On thi8 page write a rhort essay describing the on-the-job performance<br />
behavior ,of the bent bandsman you know in your MOS. Emphasize the thing8<br />
he can do bttterxn mo8t in the HOS or can do that others crnnot. It is<br />
important that you write your array from your own viewpoint. D o n o t<br />
consult othero. to determine what you should write. The ‘combining7<br />
individual viewpoint8 will be much more v8luablt than having group<br />
opinion8 prtrtnttd a8 individual viewpoints. Write on the back of the<br />
page if nece88ary or use an cdditional oheet of paper.<br />
99
. .<br />
- _ ..‘.,. _.I_ -- ..- . ___-- - - - - .---TV-.---. -<br />
_ _______.-.... ?: _____.-____ __ ,-,. - ..,-.L4.. .--a I.----<br />
.<br />
: :<br />
. ---,<br />
. . . . - -- -- -<br />
..: ._<br />
.< ‘*<br />
. w<br />
.<br />
. . _i~<br />
.<br />
P&go 2<br />
‘.<br />
MOS<br />
Length of<br />
Time KOS Held<br />
On thir page write a rhort array describing the on-the-job performance<br />
behavior of the poorest Emphasixe bandsman you know in your MOS.<br />
thu thfngr he cannot do at all or as well aa the typical bandsman in the<br />
was. It is important that you write your era&y frw your own viewpoint.<br />
Do not conrult others to determine what you should write. The combining<br />
of Fndividual vlevpofntcl will be rrmch more valuable than having group<br />
opinionr pre’sentcd as individual vievpoints. Write on the back of the<br />
page if necerrary or use an addittonal sheet of paper.<br />
.<br />
.<br />
.<br />
.
.<br />
.<br />
ANNEX 3<br />
CRITERIA FCR PERFORMANCE CHECK TEST ITEHS<br />
1. Items should requfre as little judgment as to quality as possible.<br />
Ideally, each item would be evaluated as being “possessed” or “not possessed”<br />
by examinee.<br />
2. Item6 should emphasize licrftical’V motor skills that discriminate<br />
between various levels of performance.<br />
3. Readily observable differences should exist in the motor tasks,<br />
(or portion thereof, described by the item.<br />
4. The performance can be evaluated objectively in that it is<br />
directly measurable (i.e. can be evaluated as to percent accuracy of<br />
performance , count can be made of specific number of times something was<br />
done, is.performed at the defined level, etc., rather than having to<br />
evaluate it in a highly subjective manner where the standard6 will vary<br />
greatly from one observer to another.)<br />
5. Items selected for objective evaluation should be a representative<br />
trampling or cross section of the overall performance in order that the<br />
evaluation will not be one-sided or unfair to 6ome.<br />
6. Items of performance selected should be those that can be<br />
Ipresented in a uniform manner by different observers at different locations.<br />
7. Items should be those which when being used will have a high<br />
degree of agreement among observers.<br />
8. Items requiring a special performance of standard task may be used<br />
when required, but otherwise should be avoided to simplify completion of<br />
the PCT.<br />
9. Item6 which can be evaluated better and more efficiently by paper<br />
and pencil testing should not be included in this list. No item should<br />
have a lower correlation withtotal PCT score than with written test score.<br />
10. Items selected should be considered in terms of use for minimum<br />
qualification, proficiency, and pranotion score determination. This implies<br />
that items may need to be restricted to activities commonly per-<br />
IEormed by most or all.<br />
11. PCT items must conform with usual measurement principles and<br />
item and test criteria.<br />
.<br />
.<br />
,<br />
.<br />
:<br />
i
*. .<br />
_-<br />
*..<br />
--<br />
_-<br />
. . .<br />
:<br />
1.<br />
.-<br />
\ .<br />
L-<br />
:<br />
_ :a.-.<br />
:<br />
-_<br />
- .<br />
-_<br />
\’<br />
._<br />
-0.<br />
-.<br />
,.<br />
---.<br />
>.<br />
*\.<br />
-.<br />
T.<br />
,-- :<br />
&ntroduction<br />
Reliability of Checking Computer<br />
Produced StatfstfcS<br />
CASMER ‘S. WINIEWICZ<br />
US Naval Examining Center<br />
With the advent of computer processing, it became porrible to reduce<br />
large amounts of raw data into more easily handled and more meaningful<br />
forms. Previously, where estimates of the statistical characteristics<br />
of examination6 , such as Iawshe’s D or Flanagan’s r based on eithtr upper<br />
and lover 25 or 27% of a sample wtrt used, it became possible to awitch<br />
to the original point-biserial correlation concept bartd cm the total<br />
samplt. Whtre computer configurations are extended to include a tape<br />
system, it becomes possible to expand the samplt to include the entire<br />
group of examineeo taking a particular examination.<br />
Computer processing, when applied to item analysir, results in<br />
dtcrtated processing time and earlier availability of results. Itan<br />
correlations and difficulty indices based on the total sample result6<br />
in more precise measurement, while the expansion of the sample size to<br />
include the entire population results in even greater stability of<br />
measurement.<br />
Howtver, computer procesring has not been an L;unixtd blessing. With<br />
the onset of any new system, new problems are generated. One of these<br />
problems concerns the introduction of discrepancies into item analyoir<br />
results.<br />
The carue of these discrepancies can be divided into three general<br />
. areas � The first arta could be labeled Program Error. Although new<br />
computer programs are &toted on actual samples of the data they will<br />
process, there is a limit to the exhaustiveness of these ttst8. In<br />
areas of measurement where samples vary in size and characteristics,<br />
unforseen program difficulties may arise.<br />
The stcond arta might be called Machine Error. All machines, even<br />
those components with few or no moving parts, are subject to failure<br />
when used over a period of time. In the read and punch components of<br />
the computtr configuration, burned out circuits may occur, causing a<br />
failure to read or punch out particular card columns. Such discrepancies<br />
are generally consistent in afftcting a particular part of the itan<br />
a.nalyais processed before the failure is discovered. Ir the computtr,<br />
components cooling fans my fall causing over-heating which reeults in<br />
the variable .loea of data. This type of error is the most difficult<br />
to spot by visual inspection because it is variable.<br />
_._ . - --<br />
.<br />
. .<br />
.<br />
3<br />
102<br />
*_<br />
.<br />
-<br />
-t<br />
.
. .<br />
‘-<br />
-.-<br />
.<br />
.I :<br />
.<br />
-..<br />
(. .-<br />
,.’<br />
.‘;. . . .<br />
:<br />
.._<br />
. .<br />
../<br />
./’<br />
:<br />
.<br />
-.<br />
:’<br />
. .<br />
. ‘,\<br />
.’ ‘\<br />
.<br />
,<br />
.<br />
.<br />
.<br />
._<br />
������ ������ �������� ���������� ⌧ ���� ������ � �� �� � ��<br />
��<br />
- ~- . ---. .*._ -..-___ _,- .__^.. ____.<br />
.<br />
.<br />
���<br />
� �<br />
�� � ���� � �������� � � �������<br />
.<br />
.I..-.-..--.---. .._... _ __; _.., ._ ._<br />
c<br />
������� ��<br />
The third general area fall8 into the Human Error category. ~x~plea of<br />
thir type of error will vary from the use of the wrong cnsver key (eaey<br />
to spot vicually) to the inclusion in the ramplt of answer cards from<br />
another examination, or lack of coordination portafning to different<br />
elements of the program.<br />
In short, diacrcpancier do occur in machine-produced item analysis.<br />
and it is generally agreed that human inspection of item analysis rt6UltiB<br />
ir ncca88ary.<br />
Hethodr<br />
At this point it might be useful to conaider scme methods upon which<br />
vieual fnrpection of machine itear � naly8i8 re8ulte may be baaed.<br />
The firrrt method considered might be called the u8e of an outside<br />
criteria. One such outside criterion vould concern the Item difficulty<br />
index or item p-value. The p-value criterion 8tAte8 that a p-value may<br />
not be less than 0.00, nor greater than +l.OO. That In, not leaa than<br />
zero percent of the rample may an8wer an item correctly, nor more than<br />
100 percent. Another outside criterion would 8tate that the item-teat<br />
correlation or r-value (vhich 18 generalized a8 the discrimination index)<br />
may not be lese than -1.00 nor more than +l.OO without further invertigat<br />
ion. Another method of check:ng itan analyrir might be called the<br />
intra-item conristency method. If the item r and p values are bared on<br />
the total sample, and if the correct and incorrect item alternative8<br />
r and y ertimatar are baaed on a high and low group (upper and lower<br />
27% for fnrtrncc) then the correct alternative is being mearur.-d indcpendently<br />
on two overlapping groups to yield similar although not identical<br />
rerults. Empirical studier based on a large number of item8 will yeild<br />
probability table8 for variourr degree8 of diacrepancice betvecn there<br />
tvo ret8 of r and p valuer.<br />
A more involved method of checking item analysis might be termed<br />
the Item history method. This method is rertricted to pre-tested or<br />
control items, and COn8it3t8 of observing change8 in the p and r value8<br />
of the right and wrong alternative8 of an item from one examination<br />
period to another. Although the p-value of an item is canputed independently<br />
of other items, it8 value may change due to change8 in the examiaee<br />
population or varying degrees of item coatpromise or obsolescence. The<br />
r-value vould also be expected to have a history showing a degree of<br />
fluctuation which would be further affected by its dependence on the<br />
action of other itanu in the examination.<br />
Another method of checking item analysis data ir based on trends in<br />
the item results. When the p and r value8 of the examination items arc<br />
viewed am a whole, trendo may become apparent. For instance, if one or<br />
more clusters of negatfve r-values are obrerved wfthout cmpcnsatfng<br />
group8 of high r-values and the rav score standard deviation io adequate<br />
on the ballsi of experience or a rtatistical model, these trend8 may<br />
indicate error8 in the item analysis resultr.<br />
. .<br />
.<br />
1 0 3<br />
I<br />
� �<br />
� �<br />
f
c<br />
.”<br />
-. f<br />
. . . I...<br />
..’<br />
A varietioa of this method would consist of averaging the r-velucr<br />
� ppropriately end compering this average with the raw score standard<br />
deviation.<br />
These methods have two important chorattarirtice in couxon. Fir&,<br />
they lack prtciaioa. The Outoide Critaria, Intra-Itcln Conoiatency, Itan<br />
Hirtory, and Trend Method of checking iton anaiyeie irrvolve limits or<br />
ranges vithin which machine produced it- enalynie data CM vary for rea-<br />
OORO other than error. Although probability estimate6 may be determined<br />
for individual itgls variations within a “range” oc the basis of probrbilfty<br />
tables, the problem would remain as to tha range of probability estimatea<br />
which are apt to lndicata error in the iten enslyair data.<br />
The second camnon charocterietic of imporinnct concerns the expense<br />
involved in these methods. Checking the data, item by item, can be time<br />
consuming end require@ the time and effort of professional personnel.<br />
Extlnple of Interprogram Consistency<br />
A method of checking item analysis which may be called en interprogram<br />
consistency method has the advantages of both speed and prtclafon<br />
for computer application.<br />
This method a8aumta that there are tvo computer programa producing<br />
results on a given set of data. One program mill produce a raw score<br />
mean md standard deviation, and the other the item onalyeia data.<br />
The comparison of the result6 of the two programs 16 baro,d on the<br />
\ relationship of the sum of the item p-vaiues and the raw score mean, and<br />
the � ura of the ites reliabil!..y indicts end the raw &core etandord deviation.<br />
Two further arrumptionr concerning theaa rtatistical characterfrtica<br />
are that r;v ecorea ere computed on the barir of the number of correct<br />
� n8vera and ths;t the item discrimination index io a point-biatriai corrtlatloc<br />
(Gulliksen, 1958).<br />
me computetion of the sum of p and the sum of the reliability<br />
indlcer involves a third cmputer progrem which may be called the itemtent<br />
covariance progrem. The computation0 involvs summing the item pvaluer<br />
(Cp) to product a mean , end the une of the formula<br />
K<br />
L/T-G=<br />
1<br />
to product a rtandard devfation both derived from the itan e~IelyOi8<br />
(Gul’Llkrtn, 1958). The formula dm defines the index (&llikren,<br />
i958).<br />
-<br />
.<br />
. .<br />
104 -<br />
._ .- . . ---. ._~<br />
c<br />
, .<br />
.<br />
.<br />
.
. .<br />
.- . -<br />
.,<br />
.-<br />
2<br />
:.<br />
-W-.l.W.‘.. --c*--L- I_,__,__,_ __, __L I .- ..-.,_.-. “.. ..-. _ ,*. - .--. -<br />
.<br />
A comparison of the two Set8 or’ statbtics will reveal differences<br />
due to rounding error. The amount of this rounding error 18 basically<br />
controlled by the number of decimal placea compb:atfonr are curried out<br />
to before the rounding process is introduced. The raw acore mean and<br />
standard deviation will contain rounding errors 8~ Viii each itcm’p-.<br />
value and point-biserial correlation. The production and surming of<br />
reliability Fndictr vi11 result in varying mounts of rounding error<br />
in the derived mean and ottndard deviation. Consequently, the difference<br />
batvttn tha derived mean and standard deviation and the mean and standard<br />
* deviation based on the raw 8COitl vi11 vary froai ttlt to tert.<br />
One method that may be used to interpret the tlgniflconce of these<br />
difference8 conaisto of developing an empirical probability table of<br />
differences bared on data of known reliability. In its simplest form,<br />
thlr table would be produced by deriving means and standard deviations<br />
from a rample of one to two hundred eets of item analysis data, then<br />
comparing them vith mt@nS and standard deviations developed fran their<br />
respective raw score di&tributiona. The mean and standard deviation<br />
of the distribution of abroluta differences would form a rough probabi1it.y<br />
table which may be used In evaluating re8ult8 from future u&e of the<br />
interprogram method.<br />
If canputations art carried cut to the fifth decimal places before<br />
rounding, the difference between there #et6 of mtdns and etandard &viationo<br />
due to rounding error vi11 be minimal. Consequently, any difficulty<br />
that may occur in any of the three programs vi11 yicid rtsulte eubstantially<br />
outride those indicated in the rxnplrcal probability table.<br />
Under these conditions, the Jnttrprogram mtt’md can ba a prtcirt<br />
method of checking machine produced item analysis.<br />
The actual vorking time required, which includtr the running of<br />
thb* item-test covariance progrm to arrive at the final rtoultr, 18<br />
approximately one hour par thirty-two ratem, or thirty-tvo 150 quertion<br />
teats.<br />
The primary value of this method is to provide a precise and<br />
relatively quick check of sttm of itan analycllr data that will � tparatt<br />
out the nicety-five percent that is completely accurate.<br />
In thoaa instancea where the differences between the derived and<br />
raw #core � tsns and rtandard deviations art too great to be accounted<br />
for by rounding error, the difficulty may be in the raw score. the<br />
item analyria or derived rteult8, or finally an error In checking the<br />
derived results againrt the raw score results.<br />
Reference<br />
Cullikrtn, Ii., Theory of mental ttatu.<br />
- - -<br />
.<br />
105<br />
New York: Wiley, 1958.<br />
I<br />
..<br />
,
,b-.y<br />
.._<br />
HOS Evaluation Test Validation moiedurca<br />
RAYMIND 0. WALDKOETTER<br />
: US Army Eniirted Evaluation Center<br />
‘.\_<br />
‘-.._,<br />
\<br />
I’<br />
, -<br />
.<br />
,<br />
� � ����� ��<br />
The following points will be covered with the intent to give e<br />
condensed familiarization of this Center’o HOS Evaluation Past Procedures.<br />
The technical aspects of how it is done may be uxxe readily<br />
assimilated by checking vith the evaluation section and going to<br />
selected referencee.<br />
1. The emphasis on test validation has been reinforced by the use<br />
of the special rating of job perfomance as a criterion.<br />
2. The criterion is an appropriate rating sample of the job performance<br />
a8 experienced by peers under the guidance of the rating device.<br />
3. Test validation is concerned vith determining: first, just how<br />
the total evaluation test correlates vith the criterion; second, what<br />
makes up the valid portion and segments, and their individual and total<br />
correlations; and third, just how the outline for test developent can<br />
be used as a guide with the recommended nlnaber of item to increase<br />
validity.<br />
4. The validating procedure cmputations are coupleted with the<br />
multiple correlation between EZT, CER, and the criterion, wfth an additional<br />
validity coefficient given by the correlation between the weighted<br />
scoring formula and criterion.<br />
5. Validation activities will accelerate to give a hoped for<br />
iutprersive continuity in the qualitative and quantitative test control<br />
procedures.<br />
The achievement of MOS test validation has always been a basic task<br />
for USAEEC, but it has received a new impetus this paet March (1964) when<br />
the decision was reached to tise a newly drafted special rating of job<br />
perrformc:e as a criterion measure. Since, due to physical limitations,<br />
it is not immediately possible to validate all HOS, the HOS consequently<br />
oelected for validation were identified so that a maximun sampling of<br />
the personnel evaluated would be obtained during the prescribed test<br />
periods.<br />
A short treatment of the criterion is in order h,:ra. and possfbly<br />
a good word for peer ratings. An appropriate sampling of eM designated<br />
in specified HOS are rated by at least 3 co-workers and 1 eupervleor.<br />
Readability coefficients are estimated for ratings of each sample using<br />
a one-way analysis of variance (Wirier, 1964). The rateee must have been<br />
known by the raters at least one month, observed several times a week,<br />
�<br />
.<br />
��<br />
106<br />
_____. .._ _ . --__--I_-. ..-<br />
.<br />
, .<br />
�<br />
�����<br />
� �<br />
��<br />
.<br />
��
.<br />
.<br />
and performed .the duties 01 the Primary NOS. The ‘mean co-vorker rating<br />
given by 3 En for each of the men in the particular validation sample<br />
serves as the criterion. Since a sumnary of buddy ratings in military<br />
research by Hollander (1954) emphasized the relevant values of peer<br />
ratings, this information vae instrumental in the USAEEC research<br />
decision to apply the job performance criterion.<br />
An element of anxiety may surround the peer rating mthod in aom<br />
quarterr. It has been found wantfng because of ineffestfve application<br />
in most instances. For example, young recruits could not generally be<br />
expected to rate leadership in the military setting because they do not<br />
have adequate knowledge of the role and no degree of experience in making<br />
such a judgment. The fault lies not fn the racing process but tn the<br />
Ineffective preparation for fts use. The comparison most apt may be<br />
that of trying to ahoot a bull’s eye vfth a defective weapon. A stand wss<br />
made in our program to develop a rating form which would minimize rating<br />
format influences and insufficient knowledges of co-vorkers about each<br />
other. The rem1 t being, that peer ratings of a relatively structured<br />
style, are postulated with a sense of confidence toward obtatnfng a<br />
realistic estimate of job performance on an eleven-point scale.<br />
The administration of rating wa8 prefaced by special instructions<br />
to induce the raters’ acceptance of the task in a mre Informed and<br />
responsible may. A research psychologist conducted the rating session,<br />
while the teat control officer (‘X0) at esch installation we requested<br />
to schedule all of the available racers who were qualified to rate @I<br />
in the specified HOS. Croups of about 20 to 40 men were assigned to meet<br />
in suitable place8 for the rb;ing sessions, which were usually completed<br />
in about 20 minutes.<br />
Three phases of analysis compose the substance of the test valfdation<br />
procedure: (1) analysis of the total evaluation test; (2) analysis<br />
of the valid portion of the evaluation test; and (3) providtng rccomnencled<br />
numbers of item by evaluation test outline. These phases art organized<br />
to assist in recuring the desired evaluation test speclflcations through<br />
te8 t rev16 ion.<br />
Initially, in the first phase the relationships between item<br />
statistics and test statistics are thoroughly delineated. Results of<br />
aoalyeia in tabular form show the total test, technical test and Broad<br />
Subject-Hatter Areas (B!SMA’a) by number of items with the rtapective<br />
means, standard deviations, RR-20 reliability coefficients, validity<br />
coefficienta, beta weights, the multipltR,and coirected R after shrinkage.<br />
men follow in another table the correlation coefficients betveen each of<br />
the BStU’a, the BSHA’s and the criterion, and total evaluation test and<br />
criteriorr. In u third suxnary table the rerults reported for items<br />
include item p-vaiues for the total HOS population, p-values computed<br />
107<br />
:. .
.\<br />
.<br />
.<br />
.<br />
”<br />
for the validutioo e8mple, item standard deviations, item varibccee,<br />
item-test correlations, item-criterion correlations, indcxea of<br />
reliability, and indexes of validity. Mr. Wry and Hr. Shirkey h&ve<br />
presented in their reports, which treat the iten relationships, 8<br />
wider di6cour6c vith an8lyeis and implications in this ares using the<br />
needed examples for illustration.<br />
The second pha6e, concerning maximization of test validity by item<br />
alalection, ir appmached by tlimiruzttng item8 from the origin;ll evaiuation<br />
te6t which contributed nothing to validity. Correlation coefficient8<br />
appear in tabular form showing the relationehips between BSMA’s and each<br />
BSHA and the criterion vith the coefficient computed to indicate the<br />
validity of the revired evaluation teet.<br />
The third phase of the sequence Fn the telt validating procedure6<br />
occurs fn a m6ximlution of test validity wfth the optiml allocation6<br />
of items acccrding to the test outline. Techniques of multiple correlation<br />
uhfch determine optinval item ellocationa have been formulated<br />
by HOr8t (1949) and Taylor (195C). Theee technique8 reflect function6<br />
In relation to the correlations between B.SM’o, their reliabflitfcs, and<br />
validities. The correla t ions, reliabilities, and validitiec are systematically<br />
ctungcd. vhen the number of item per BSX4 are altered. By<br />
increseing the less reliable Bs;yA’s. validity is further enlarged, provided<br />
the valid vrriance is not measured alao by other BWA’r. After a clueter<br />
aMly6i6 (Br*Jchter. 1954) of KiNA’s the Horst technique is applied.<br />
(This procedure in applicable currently, but oay give way in deference to<br />
the Wherry - WIncr mathod for factoring large numbcr~ of items.) In<br />
tabular form a rwmary is given of the cluster analyeie showing each<br />
ciuster, optlpal nrrmber of item6 per cluster, and the BWA’s per cluster<br />
with the valid number of items In each BRIM, the proportlon of BStIh<br />
ltem6 in each cluster, and the optiraal number of iteuis per BSHA.<br />
From a ruemary reviev of the validating procedure8 used vfth the<br />
te6t analysis, a brief description of the overall validity approach fa<br />
desirable at this point. Since urch, some six MOS terts which eVAlUStU<br />
mboot 17% of EM under the Army EES, have received validity analysis and<br />
evaluation based upon the criterion of job performsme racingr.<br />
The ssmples used rnnge from 30 to 129 and were checked to assure<br />
reprerentative groupr. The evaluation test validity coefficients<br />
ranged from .lO to .52. The CER had validity coefficients rangihg from<br />
.20 t o .51. The multiple correlation coefficient6 betveen evaluation<br />
test, CER, and the criterion ranged from .36 to .S5sdemostrating a<br />
rlightly better prediction of validity from the BT and CER combined.<br />
The corrected multiple correlation coefficients after ehrinkqe range<br />
from .34 to .51.<br />
. . . .<br />
108<br />
- .I<br />
.<br />
.
’<br />
.<br />
I<br />
,<br />
._ ..--“.,.-.“..~...;..-. . . . . _ _. .<br />
.<br />
bt ft be remarked here that although some validity coefficier.ts<br />
.have not reached entirely sanctioned level8 of validity wet are<br />
at.taining significant levels of validity. With the identification of<br />
weak points in validity we can utilize greeter control thereby beginning<br />
the necessary corrective test developtint procedures. For example,<br />
the validity coefficient for the HOS 941.1 Waluatfon Test (Cook) wa6<br />
.l.O, vhich should be-susceptible to improventent by simply eelecting<br />
‘the proven valid items and adjusting the reading level of the test to<br />
that more usually experienced by cook6. Mom a mmparisoa of validity<br />
coefficients I would like to hazard a conjecture that the validity<br />
coefficients tend to be higher for the more technical jobs of HOS<br />
t!mn for the unskilled or perhaps motor-skilled jobs.<br />
Another combined validity coefficient fa given by the correlatfon<br />
between the rav composite score or scoring fomula and the criterfon.<br />
These coefficients for the six MOS ranged from .22 to .49.<br />
To realize that validation procedures fn &ma of man-hours, data<br />
puocessing, and interpretation presupposes a high degree of technical<br />
organization, requires very little analytical skill. But to utilize<br />
these procedures and insert the mdfficatfons needed to improve validatfon<br />
procedure6 demands that an organ?zatfon reach a noticeable stage<br />
of maturity. The maluJtlon and Ar~lysis Branch of USAEEC, nov embarked<br />
upon thts stage of maturity, r:il continue to improve and increase ft6<br />
vrlidation efforts fn 6a63p?.it!43 all area6 of the HOS 6trUCtUre.<br />
109<br />
, C<br />
.!<br />
i
References<br />
Pruchter B. Cluster enelysir. Introduction to fnctcr knalyaic.<br />
New York: D. Van Nostrand C-an>, Inc.,TF54, 12-17.<br />
Hollander, E. P. Buddy ratings: <strong>Military</strong> reosarch and induatrial<br />
implications. Personnel Paychol, 1954, 7, 385-393.<br />
Horst, P. Determination of optimal teat length to reurizaita the<br />
multiple correlation. Pepchometri&, l>G9, 14, 79-88.<br />
Taylor, C. W. Maximizing predictive efffc.ency for a fixed total<br />
testing tima. Psychometrfka. 1950, 2, 391-406.<br />
Wherry, R. J., and Wirier, B. J. A methqd for factoring large numbers<br />
of iteuxu. Paychometrika, 1953, 18, 361-179.<br />
Wirier, B. J. Single-factor experiment8 having repeated measures on the<br />
saxa8 elements. Statistical principles in experimental deeign.<br />
New York: McGraw-Hill, 1962, 105-139.<br />
I<br />
. s<br />
.<br />
110
.<br />
.<br />
.<br />
-<br />
. .<br />
. . -.. . ^. ‘. L ,... . . . . . j . .<br />
Background<br />
Approaches to Improved Measurement:<br />
Research in Progress<br />
. .<br />
CASLHER S. WINIF~ICZ<br />
US .Nsval Examining Center<br />
The research projects abstracted herein are primarily projects<br />
th8t the US Naval Examining Center has concentrated their efforts on<br />
sfnce the last <strong>Military</strong> <strong>Testing</strong> <strong>Association</strong> meetfng, held in 1963 at<br />
Groton, Connecticut. ThcSe.prOjeCtS should not be confused with the<br />
regular semiannual evaluations that are conducted on all examinations<br />
and their respective populatlon8. To eumarize a few of these autometic<br />
evaluations; each examination is anslyzed fn terms of it8 adherence to<br />
the criterion standard of construction; professional and military sections<br />
are analyzed independently and also theft interactions a8 major components<br />
0” the total instrument; intercorrelations between all minor aubsections<br />
on the professional pgrt, as well as vith the total composite;<br />
item atUllySi8; raw and standard score conversions along with their<br />
respective statfetics and graphic representation of each population a8<br />
well a8 the parameters; analysis of the final multiple components; check<br />
for compromise and collwion; and finally all the various activity and<br />
bureau reports that are required to adequsltely summarize thfe information.<br />
A susxaary listing of the following research projects fndir-pte those<br />
unique areas that h8ve been investigated to provide-additional technical<br />
Information TV improve and support the Naval Advancement System.<br />
Pro-Pay Survey<br />
A survey was conducted on the prevalent attitude toward a variable<br />
reenl!.stment bonus (guaranteed minimum bonus with a variable sum based<br />
on rating criticality) vice a proficiency pay program. Ten thousand<br />
canfdates were sampled from a cross-section of 231 different Naval Activitfee.<br />
The major extrapolation from the data tend to support the premise<br />
that,in general,the group would prefer a variable bonus system in lieu of the<br />
present pro-pay program. This I.8 based very briefly on the fact that<br />
although only 25% of the total group are currently receiving pro-pay in<br />
one form or another, they only favor pro-pay In a 60 to 30 ratio. However,<br />
the larger portion of the sample, the remaining 75% that represents about<br />
6000 people in this sutiey are in favor of the bonus over pro-pay in a 2<br />
to 1 majority.<br />
Automatic Sxamination Requisitioning<br />
The examination answer card8 have Jeen revised to Incorporate the<br />
collection of two variable factors of performance evaluation and awards.<br />
Basic battery scores ‘Ire constant, and length of service and time in rate<br />
*<br />
111<br />
I<br />
..- . - -_. ..-<br />
; -! :<br />
i i
.<br />
,<br />
,<br />
. -<br />
c<br />
increase by a constant for each examination seriee. A propoaed tape<br />
configuration s-1ded to the present computer would allow for the elimination<br />
of certain administrative responsibil.ities prior to the administration<br />
of each examination (elimination of !UV,PZRS 524 which contains<br />
biographical information on each candidate). A raster tape file cotttinuoualy<br />
updat,ed by NEC in conjunction with the collection of certain<br />
infonnntion directly from the answer card would eventually eliminate<br />
the necessity for ordering examinaticns and they could be sent out<br />
automatically by name vhen each candidate becarocs eligible, as indCcatRd<br />
by the file maintained by NEG.<br />
Actuarial Longevity Prediction Study<br />
Since the literature only covers conventional statistical approaches<br />
to prediction, such as regression analysis or multiple regression, and in<br />
some cases the application of the Poisaon distribution or the negative<br />
bi-nomial, raw data was acquired to experimentally determine the fcaafbility<br />
of utilizing the actuarial approach in predicting longevity. F31vfdence<br />
to data indicates that this approach will eventually be ctilized in<br />
certain areaa of prediction as a routine technique, To fully appreciate<br />
the success attained by this approach, addifional comparison studies<br />
between various predictive techniques will be conducted aa to the eventual<br />
superiority of one in terms of minimizing error, fewer assumptions, time<br />
clement, and the final extrapolations from the data that are poaeible.<br />
Class Scheduling Project<br />
A atudent, teacher, and class scheduling project was cmglcted by<br />
hand baaed on raw data (N=2000) for a single school, The application<br />
of resulta will eventually be duplicated by a computer program and the<br />
two aete of results will be analyzed for comparibility. The project<br />
will then be extended for application to the CNARESTRA problem (N-35,000)<br />
which will Involve a series of schools located geographically in various<br />
parts of the country and eventually encompass the mobility of atudenta<br />
from one area to another. Complete coordination will be poacible between<br />
the transportation, teacher, student, school, classroom, time sequence,<br />
and course information factors, all will be in one co;aplete computer<br />
program.<br />
Differential Weights of Final Multiple<br />
The final multiple acore is composed of five individual factor<br />
scores which are a-d on the basis of a simple weigLting formula.<br />
Each factor has a designated ideal contribution to the overall variance,<br />
but the actual or real factor contribution IZWIY and does obviously vary<br />
from this standard. A study was implemented to determine the empirical<br />
interaction of the five factors making up rhe total composite of the<br />
multiple variance of passing candidates in selected critfcal ratea frclm<br />
previous examination series. The results indicated that the factor<br />
112<br />
.
.<br />
.<br />
.<br />
contributions to the fir&l mulctple variance differ from the apparant<br />
atandard by rate and factor. @n the basis of pay grade medfans, differences<br />
from the standard were etatistfcally significaut for all factor8<br />
at pay gradeo E-4 and E-5 and not significant at pay grade E-6.<br />
Pre-<strong>Testing</strong> for Controlled ~minations<br />
The Navy’8 regular advancement examination is compoeed of 150<br />
4 option, multiple choice questiors. In the construction of theae<br />
instruments only 50% of the questions have item statistics and are<br />
utilized to control the standard of the examination agalnat the deoired<br />
criterion, The remainfw percentage of question8 are new and generally<br />
cover the entire spectrum of difficulty. Closer adherrnce.to overall<br />
a tandards can be acquired by simple pre-testing of the new questions<br />
on selected populations. This in eesence will provide for a completely<br />
controlled examination from the standpoint of item difficulty and discrimination.<br />
Reliability Computer Check Study<br />
i<br />
7.<br />
A detailed approach to one aspect of reliability checking for<br />
computer operation8 ie given in another part of this surmary. The<br />
paper wa8 presentad independently a8 prt of the Theoretical Seminar.<br />
Four Cycle Bcaminin~ Periods<br />
The regular advancement examfnattona are pr?marily discriminating<br />
and administered Navy-Wide each February and August. The regular input<br />
at pay grade E-4 cannot be maintained in 22 critical rates, vithout<br />
disturbing the balance of the total system. The possibility of examining<br />
these critical perrronnel duriw Hay and November with a qualifying type<br />
oli examination ha8 been investigated and proposed a8 a poseible solution.<br />
Simplified Tri-Serial Correlation<br />
A byproduct of a latger project yielded a simplified triserial<br />
correlation, result8 of which were presented at the national 1964 APA<br />
conference. J88pCn’8 original formula for the triserial r ie rather<br />
unwieldy and therefore has been passed by for 8Ome easier lees appropriate<br />
correlstion technique. Jaspen’s formula equals:<br />
‘tri p<br />
ZaYa + (Zb - za) Y b - ZbYc<br />
Za* + (Zb - Za)* +<br />
=r 8 b<br />
C<br />
113<br />
.<br />
.-<br />
.
,. . . ;<br />
./:.<br />
. .<br />
--.. ----.<br />
,.<br />
,.‘.<br />
5/’<br />
,./’ .<br />
\‘<br />
.\.,‘.<br />
2.<br />
-<br />
. _. ., _ .<br />
The formula derived which gives identical resuAts with Jaspen’e<br />
formula equals:<br />
I‘trf u<br />
CYa + (Na - Nu) (Yt) - L”y-2<br />
Ntuy (2, + 21)<br />
Weighting E Two Subteet composite<br />
Another byproduct and also presented at the APA, was the tteighting<br />
of individual subtests in I composite, where they become more critical QBL<br />
the total number of tests in the composite dscreaaes. The importance of<br />
controlling the weighting of individual eubtesta reaches a naxfmxu When<br />
there are only two subteeto in the composite. A method of controlling the<br />
contribution of each subtest in a two teat composite ia given by the<br />
following formula:<br />
Wa + rab<br />
Za<br />
’ + ‘ebWa p<br />
f-<br />
1 - za<br />
Open End Item Distractor Development<br />
Pay grade E-8 and g-9 have essentially an examination tbAt is part<br />
aptitude and part achievement. Some of the subteste contain leadership’<br />
and situational problem questions’which are extremely dfffic*tlt to<br />
develop. A technique of using an open end question to collect responses<br />
was utilized. ‘The various responses are then tabulated and arerbad into<br />
aimllar groupi. ge by a frequency count. The three most attractive incorrect<br />
rsplies are then merged with the correct anewer to form an ites along<br />
with the appropriate stem.<br />
Longitudinal QualFty Control<br />
A quality control study is routinely conducted after each advancement<br />
cycle in order to determine adherence to etandarda. It has the<br />
advantage of presenting current Fnformstion and aces as a warning system<br />
relative to overall examination quality. A lolyitudinal seudy based on<br />
previous quality control studies is now in progress and its effect will<br />
be cumulative in nature and will form the baeia to predict the desired<br />
atatfstfcal characteristics of future examinations,<br />
Effect of Automatic Advancement<br />
Various programs at the minimum petty officer level (pay grade E-4)<br />
were introduced to act as a stimulus for reenlistment in the &IVY. As<br />
a rettwlt of euch programs as STAR, SCORE, and CLASS A school, atitoPratic<br />
. .<br />
.<br />
.<br />
.
._<br />
_ --r<br />
-.<br />
:’ ,--_<br />
.I’-<br />
..__<br />
..- _<br />
_.<br />
.I. :<br />
‘\ ‘,.<br />
,<br />
.<br />
.<br />
--<br />
.<br />
-. .I<br />
. . .~ . .<br />
promotion to pay grade E-4 is possible without the necessity of taking<br />
an examination. Since the inception of these programs which are a<br />
recent innwat ion, a certain normal quantity of the examination population<br />
is missing. This mean8 inessence taking the quality off the top and<br />
having the remainder take examfnations. Tine standard at E-4 will obviously<br />
change and also when this scgt?ent of the population ia added back into the<br />
competitive examination popclation at the next higher level at E-5, certain<br />
ramifications will take place. This problem is currently being fnvestigated.<br />
Forced Sampling Technique<br />
The forced sampling technfque is a form of stratified sampling.<br />
It controls a sample by reducing biases caused by chance factors in<br />
random sampling. The procedure is to rank the total number of candidates<br />
from lw raw score to high raw score. The next procedure is to<br />
pull the sample in such a way that all raw scores are represented as<br />
equally as possible in the sample. The formula used to choose the<br />
individual samples was: 4Nt/N, + Nt/N, + Nt/Ns...Nt equals the total<br />
population, and Ns equals the sample size vhich is desired. The forced<br />
sampling procedure has been found to be extremely accurate in representing<br />
the population median mean, and standard deviation even with<br />
sample sites of less than 50.<br />
Population Analysis Instead of Random Sampling<br />
The sample sizes of the occupational rates are based on the limits<br />
of a confidence interval of a population. The confidence in the limits<br />
of a mean for a given parameter is the fiduciary probability. The<br />
fiduciary probability is better than .95 that the true mean 1i:e in the<br />
interval n + 2.00 S. E.,, and .OS that it falls outside the 1im:ts. A<br />
standard error of one raw score was the average standard error of our<br />
samples. The effects on item analysis ard reliability of the examination<br />
is noticeable, although by increasing the number of candidates used in<br />
the studies tne vatLance also increases slfghtly. However, the use of<br />
the total population in the studies provides more stability in all item<br />
statistics.<br />
Validity<br />
Concentrated effort has been placed on obtaining indirect measures<br />
of validity through analysis of class A school graduates versus non<br />
school graduates, frequency breakdowns of various elements of the population<br />
and conducting teats of significance on the differences, Conventional<br />
validity through supervisory ratings and peer ratings has<br />
yielded validity coefficients that average around .35.<br />
115<br />
.<br />
--- ‘-<br />
.
Approaches to Improved Measurement:<br />
Research in Progress<br />
RCMLD K, GOODNIGHT (<br />
US Army Enlfated Evaluation Center<br />
Throughout the confer&e we have heard and discussed numerous topic6<br />
epanning the realm of test immrovement and development. In every cane<br />
each method used, the reaulto obtained, and the utilization of these<br />
re6ults are all bared on reresrch. Reeearch is the guideline to BUECC~U<br />
in almoet any endeavor. Tnncrefore, I am plcaeed to present to you several<br />
of the more important reeearch projects which have been completed, as wel.1<br />
68 nome currently in progrerit, at the US Army Enlirted Evaluation Center.<br />
The first project, completed oeveral years ago, ir A Comparison of<br />
Seven Methods of Computing Total Test Reliability fromma Single Terr<br />
Adminietration. Ttdt reliability (that is--a determination of teat conriatency)<br />
is evaluated by Lnvtstigating tht relationship of the praiictor<br />
to itrtlf, or in other words, it is-the relationship of the ranking of<br />
scores on one administration with the ranking on a aubsequant adminlstratfon.<br />
The rsliabilfty of a test can generally be estimated adequately<br />
fran only one administration of a test.<br />
For tach MOS Evaluation Teat a reliability coefficient is computed<br />
via the Kuder-ttfchardoori Formula 20. However, it ~68 imperative tc know<br />
if thie wa6 the most accuratt indication of the consirtancy with which<br />
the tert was measuring job proficiency. Thtrefore, this project was<br />
designed to determine which of the odd-even, K-R 20 on the total group,<br />
K-R 20 on 54% of the group, Hoyt’s Ana1ysi.e of VerCanct. Horat Maximum.<br />
Horet Corrected, and Cleman’r Maximum reliabili*.v mearurement methods wan<br />
super for. The result8 showed the K-R 20 relfability sethod on the total<br />
group io the meet appropriate for the MOS Evaluation Te6ta, thus supporting<br />
It8 uragt by thL Enlisted Evaluation Center (EEC).<br />
Thi8 study was completed in February 1963, and prcaently another project,<br />
A Canpariron of Fivt Hethods of Computing Total Ttrt Reliability frus<br />
a Single Test Administration, ir under way. This 18 a replication of the<br />
earlier study with some minor variation6 tc further verify EECI currant<br />
procedure8 in tt6t reliability estimation. Also, another reason for con=<br />
ducting this study lies in the proposed technical rtcommendationr which weft<br />
presented at the 1964 American Psychological <strong>Association</strong> meeting&; their<br />
rscamntndationr indicate that the K-R 21 ia more appropriate to use then the<br />
K-R 20 under situations such as those at EEC. Therefore, in this rtudy five<br />
methods of reliability eatimation-- K-R 20, K-R 21, odd-even, Hoyt’8 Analysis<br />
of Variance, and Uorst’o Maximum--are beir.g compared to determine which is<br />
the better method of mearuring te6t cons,etency in view of APA’s rtccmmendations<br />
and to rtudy the effect8 of aample sire and MOS rkill level on the<br />
varioue reliability coefficlentr. Results are not available thus far.<br />
116<br />
-_ ._.....- .--- .---. --.<br />
c<br />
_I__...
,’<br />
‘.<br />
,-- -<br />
: ’<br />
‘.b<br />
:’<br />
_. _<br />
:<br />
:<br />
I ,<br />
_.<br />
. .<br />
f<br />
-.. \<br />
‘\<br />
-.<br />
‘.<br />
Another area of reoearch is the longitudinal studies on the cmmander’s<br />
Evaluation <strong>Report</strong> (CER). The CER is the officfal rating form<br />
which constitutes one of the cmpone’nte of the Enlfated t’valuation<br />
System. The primary purpose of the CER is to provide an sseessraent of<br />
the soldier’s job performance and potentinl-for advancement by his imdiate<br />
and secondary supervisors. Although ratings, per se, ‘are not perfect they<br />
are the best available evaluation method when no objective masure can be<br />
obtained; therefore, the enlisted Evaluation Center is quite interested<br />
in the perfomance and improvement of the CER.<br />
no studies presently beinS conducted to facilitate the continual<br />
improvement of the CER are A Factor Analysis of CER Scales, <strong>Technical</strong><br />
Subtest. and Suoervieorv Subtest. and<br />
I A CccJlparison of a Graphic Raa<br />
Method, A Normative Paired-Compartson Rating Method, and An IpeatFve<br />
Paired-Comparison Ratfng Method via the Multitraiz-Multimethod Hatrix.<br />
The factor analytic study was to determine the degree of c-n-factor<br />
variance of each of the twelve CER Pcalee and the technical and supervisory<br />
subteata ofanMOS Evaluat:on Test. prom the results of this<br />
research will come information necessary for proper revision of the CER<br />
to eliminate rhe measurement overlap or c-n-factor variance, thus<br />
improving the evaluative ability of the zating. Seven orthogonal factors<br />
emerged from the analysis, however, tbey a:1 have not been named. One of<br />
more important results thus far, however, was the high loadfng of the<br />
technical subtest on the initiative Factor.<br />
The second rating study, presently in the planning stages, ie A<br />
Comparison of a Graphic Rating Method, A Normative Paired-Cozparieon-<br />
Rating Method, and an Ipaative Paired-Comparison Rating Method via the<br />
Multitrsit-Hultim+thod rWtrix. This research endeavor was dec Lgned to<br />
statistically letemine whether the graphic rating method as used in CER<br />
ratings is adequately serving its purpose by providing valid and accurate<br />
information to the Enlisted Evaluation Center, or, if either the normative<br />
paired-canparison 4th certainty judgments method or the ipsative pairedcomparison<br />
with certainty judgments method would be superior and provide<br />
more accurate and valid results. Also, since the normative and ipaative<br />
data are aupplemntary to each other, it is possible to statistically<br />
combine the data from these two methods, thus yielding a fourth rating<br />
method for the analysis. ft is felt that this information will be<br />
valuable in assessing the adequacy of the CER ratfng method in comparison<br />
to the other rating methods. No data have been collected yet,<br />
One very important previuus study was The Effect of Rater-Ratee<br />
Acquaintance Period on CER Ratings. This research project was designed<br />
to determine ii a specified minimum period of acquaintance between the<br />
rater and ratee was neceesary for satisfactory and reliable CER ratings.<br />
The results obtained on various NOS skill levels indicated that a minimum<br />
period of two months (60 days) acquaintance was esaentfal for proper<br />
evaluation. This time period is now mandatory in the Department of the<br />
Amy for all CER ratings, although some leniency in this requirement is<br />
alloued.<br />
117<br />
-------<br />
.<br />
‘I<br />
;;
.\- .<br />
‘.<br />
,-.<br />
/<br />
I<br />
\ ,<br />
_. /’<br />
,<br />
/<br />
/’<br />
. .<br />
,_ -y...---<br />
. . --<br />
I<br />
.<br />
.<br />
.<br />
FoeoiL\‘ly the most important area of research at the Ealfated t-kaluation<br />
Center are, the validation ctudiee, of the Cczzindtr’i: ZvalusrLon<br />
,Repzt and the KG Evaluation Testa which ere conducted routinely.<br />
Formal val.idiry rcpmtlng is quite comprehanui:te and tncltrdes the<br />
validatioil of both the CER aitd the EvaloAt.fon Teat as *:ell as the interrelationship&<br />
between these instrrantc. Dee t o thrj lergthy processin,%<br />
t&e required for the forwl validity report, a prclimin~ry reporting<br />
procedure io employed to provide maxiliium data for te3t revicion.<br />
A preltmtnary validity report is to prwtda guidcltaea in test<br />
revision based on pertinent statistical deta. TF;fo vaiiGi.ty data<br />
cover&z’ the entire evalwtion test outlin 1) and individual test item<br />
are provided prior to test revision to allw for their UB~ in the<br />
decision-making process of test development. Emphasis is now placed<br />
on the WIfi and HOW of test vslidity.<br />
Statistical bases are provided to allow control in :c?~~t revhim<br />
over the evaluaticq test characteristics of wan, ste.ndard deviati.on,<br />
reliability, and v.llidity. The test characteristic of valfdity is<br />
given specfal atta.;tLon, and is considered at both the item and test<br />
outline levels. The statistical rationale and data necessary for thinpurpose<br />
are provided to enable their use In test develcpment procedures.<br />
Tha users of the information SGI data provided in the validation<br />
report should: (1) have a thorough working knout,cdye of the interrelttionshfpa<br />
between item and te6t stat!.st:cs to enable zzre control<br />
in test revision Over evaluation test means, atar.darJ devls.tkona,<br />
rcliabflfties, and validitlrs; (2) insofar a8 is ~racr.fcable, include<br />
items of substantial vaiic:ity in revised tests; (3) study itema of etibstanti.al<br />
valfdfty to deten?fne the particuler types of items which tend<br />
to be most valid far a given Rvaluation test; (4) ff ouClina revision<br />
is deeut?d appropriate, make such revisions in view of known intarrelatio.%hips<br />
between Eroad Subject-patter Areas; and (5) reconcile item<br />
requests vith both optin item allocations by Broad Subject-t!nttor<br />
Areas and considerations of n practical nature. In this way, mxtnwn<br />
control of the test results can be attefned.<br />
A Corcparative Study of a Short Form and a Lorq Porn of Performance<br />
- -<br />
Dictation Test for Legal Clerk or COure <strong>Report</strong>er - - te 8 research project<br />
presently being conducted at the Enlisted Evaluation Cuntar.<br />
his project was designed to detennfne whether a short form of the<br />
Dfctation Perfornnnce Teat may be relLable enough to use in place of the<br />
longer form, thus savfng administration and scoring time 88 vell as<br />
providing a more maruzgeable measurement of the selected examineen. fie<br />
short form is composed of preeelected sections of the dictation test<br />
crmaprising 4GX of the total test. In a preliminary aralyeis, a Peareon<br />
r correlation cofficiene of .95 was obtained bemeen the long form and<br />
the exparitintal short revfsion. It would appear further examination<br />
of this relationship wfll prove fruitful, and the recommendation for<br />
the shorter test can be anticipated.<br />
118<br />
-- .- , -. . ..____. _____-... - __ .- _..-- ___---- ___^____ l_--
.1 , .I . . ., .;., ., _-<br />
.<br />
-.< “. ,:. ,L ,._.,._. _<br />
Another group of continual research studies lies in the Inv’eetf-<br />
Bation of Possible Test Ccmpromiee. One study i.n this area vns the<br />
davelopwnt of investigation procedures.<br />
An MOS !Zvslustion Test would be comprolaiaed if one or more copltes<br />
, of the test booklet came into.tho possession of an unauthorized enlisted<br />
_ nun, and the Information vats used by him before or during the admints-<br />
/ tration of the test, An HOS Evaluation Test is subject to possible<br />
compromise if it is lost or is unaccounted for prior to the ccmpletion<br />
of an “02 evaluation p%riod. . .<br />
: . .<br />
. .<br />
., _,‘.<br />
.<br />
.<br />
In this study three methods for investigating possible compromise<br />
in a large military program in which tests are administered once a year<br />
were developed and experimentally tested. Compromise may be checked by<br />
(1) comparison of test scores of individuals or possible compromise groups<br />
with population parameters; (2) standardization and analysis of test<br />
scores wer two test periods; and (3) regression analysis of test scores<br />
over two test periods. prom the resulting statistical analyses the<br />
limitatfors and advantages of each method vere shovn, as well a8 a<br />
rationale for the interpretation of results and the formulation of<br />
subsequent administrative decisions and recolmlendations, Presently,<br />
any one of these metho,ds may be used depending on the circumstances.<br />
Two more recently carsrenced studies are A Comparison of Six Me,hoda<br />
of Item-Test Correlations and k Comparison E v a of%ur l u a t i o n Proceduref!<br />
for Pieasuring ParforuGG Efficiency. The first fesearch project was<br />
based on Cuilford’s (1950) study in which he compared the biaerfal r, the<br />
point-bieer?al r, the ordinary tetrachoric r, the Flanagan tetrachoric r,<br />
and two applications of the Phi Coefficient methc-;s of itein-tese correlation.<br />
~%a point-biseripl r proved to be the superior method. Therefore,<br />
thfe project was designed to replicate and extend Gutlford’e study<br />
since the point-bisarial correlation is sued in item analysis at EbC.<br />
This project will either substuntiate Guilford’s findings and give<br />
further proof of the value of the point-bisarial method, or it will<br />
reviee his results and indicate possibly another more accurate index of<br />
Item-teat correlation. ho data have been collected as yet.<br />
The second study noted above can be classified as a criterion study.<br />
Sufficient procedures are used by EEC for increasing and maintaining the<br />
reliability and content validity of the HOS Evaluation Tests. However,<br />
a number of problems are encountered in the establishment of concurrent<br />
or predfctive val:ldfty since an adequate criterion is necessary. The EEC<br />
has developed a rating scale which is presently being used for getting<br />
criterion data via co-worker or peer ratings. However, more satfsfactory<br />
criterion data msy be accessible by using other measurement procedures.<br />
The purpose of this project is then to compare four methods of<br />
measuring perfor=mancaefficiency to learn whether a superior rating<br />
119<br />
,
.L<br />
.: -_ _.<br />
.\<br />
.‘? .’<br />
’ I -. ,<br />
I .:<br />
. . . i+’<br />
,*!<br />
i _:<br />
procedure doeo, in fact, exist. This information wlll be valuable in<br />
facilitating in the quest for the most satisfnctcry criterion aaasureuent<br />
procedure to be used in valldatloa studies.<br />
The four rating procedures to be used a.6 criterion masure of<br />
performsnce efficiency are self ratings, peer or co-worker ratings,<br />
supervisory ratings (first and second level), and group-speciallet<br />
ratings. Theoe avaluatto~ will al1 be obtatned on fder.tlcal rating<br />
forms comprised of an appraisal of the subordimte’a “(Xterell perform<br />
ance” and a check list of his overall job-proficiency quallflcatfon_s.<br />
-_<br />
These data when collected, vi11 be analyzed in vhole as vell as by<br />
subparts to derive the most meaningful information from them.<br />
These have been just some of the more pertinent research projects<br />
conducted at EEC. Uany other research projects have been done in the<br />
p-t, and many more are now in the developnrent ataga. It is believed<br />
that only through dellgent und carefully designed research programs<br />
such as these will improvement in our evaluation syntem be realized.<br />
120<br />
,
.<br />
.<br />
Test Construction Procedures<br />
wILLI\c’ .,.. ‘: tP*?rE, ..a CIIA,.NN ,<br />
, US &my Enl;fir?d Evaluation Center<br />
The te5t construction procedure is an especially crucial one in<br />
,.obtainino, a valid instrument, especially for test6 such as o*.ir~ t’nat<br />
usuaily cannot’ be given experimentally prior to operational use.<br />
For evaluation activities to be most effective, they should consjs:<br />
of the best possible techniques, used In accordance with what we know to<br />
be the besr and most effective psychological principles.<br />
One feature that distin&uishes reputable *hotk in test development<br />
from that of the mas.; of self-styled “test constructots” or “test experts”<br />
a>d outright quacks is that the reputable worker in the field is continuously<br />
concerned with testing, verifying, and improving: the adeauoc)<br />
of his procedures, He, knows that he does not know all t!le answers, and<br />
he is cvcr on the alert to find out more and to improve his procedures.<br />
There is no easy road to scientific test construction.’ The’roat’ is loni;<br />
and tortuous and beset with many pitfalls.<br />
Xn our types of testins programs, there usually is no test available<br />
that corresponds satisfactorily tp a function h+iich seems important to<br />
test. We as test psycholoI:-.sts and subject-matter experts, and coordinators,<br />
are then truly put upon our own mettie to originate improved patterns of<br />
test performance and to develo? a crude test idea into a practical and<br />
reliable testing instrument. This constitutes the most exacting and, at<br />
the same tfme, the most interesting and rewardfng phase of tesK development<br />
work. It requires truly creative efforts.<br />
The topics to be presented by the four symposium members are some&at<br />
diverse in nature and should stimulate our thinking in the direction of<br />
improved test construction procedures. At this time, I would like to fntroducc<br />
the fou; symposium members:<br />
Mr. Isadore J. Newman, 6570th Personnel Research Laboratory,<br />
US Air Force<br />
Mr. John Crediford, US Naval Examining Center<br />
Mr. Charles E. Cassidy, US Army Enlisted Evaluation Center<br />
Mr. Fred B. Honn, US Army Enlisted Evnluation Ccntcr<br />
Since the four ?apers to be presented by these gentlemen arc somewhat<br />
diverse and to keep our thinking “warm” relative to the particular topic,<br />
we will have a short question-and-answer period following etch presentation.<br />
121 I
Educators nnd trr;incrs erc cor,tinuelly kin:: chnllcn$ad to project<br />
knov1cdge.s that en&le ir12ividunls fo not only txijas: co ti;cir cnu<br />
viroment b.lt 9llstcr i t . ‘Jnfortunately ,tb.cre ar.2 notable dtf fcrencee<br />
among some educators and trttinera in their rspncfty to perceive that a<br />
challenge exists. xnfonration is so rapidly accumulating t’nnt it la<br />
imperative that we seek new methode of dissemination, Since 60 much of<br />
man’s diverse behavior is a result of lcsrnfr?g, we must find CI system<br />
by which vcrlffcd facts and relationships may be projected to hisi in e<br />
systematized effort,<br />
Educators end treiners must seiect the cethod of bnatruttfon vh
.<br />
.<br />
*<br />
.-a . ..” .-. ,. ., “.. _... .,... “. _..._ “__e._.- *...+... . . .- ..^ _.a, .~f-*-i-Y. “. ,C^.. 1 I .h.<br />
sarfaa of taaka which their mission requires. The conxzands develop their<br />
owu measuring instrumenta to evaluate their o*m on-the-job training<br />
progra36, Our unlt has the r;Iselon of dcvelcping the meeauring instrument<br />
for USAF; b%ich, under the new conccptr will .cNJaoure the knowledge reeulting<br />
from studying the subject matter of the ca:ccr development courses.<br />
Where&a in the put we verc chnrgcd with the reeponeibility of r,;aauring<br />
cn aiman’s knowledge of his entire GP+a-ialty, we vi11 bc limited to<br />
measuring only that knovlcdgc tiich is boee.‘. on the mntcrial concerned<br />
with principlea and fundamentals found in the career development ccursc.<br />
The Air Force Training Cocnond people who monitored this ncv concept for<br />
the Air Force, have ststcd “Hodificatione of present methods will be used<br />
for testing knowledgea learned and tikills developed. Specialty Knouledge<br />
Testa (SkTe) will becorce in effect ‘end of course’ tests for airmen<br />
completing a career develo;tment course. The test will cover only the<br />
content of the self-study natcriale. The writers of course materials vi11<br />
assume a nev role ~+ich will require more corefully planned approachca to<br />
the dovelopncnt of effective coureee, Specialty Knowledge Teats will<br />
adhere to what the student h-s been presented in hie coursa. If he<br />
hoe learned the material well, he should be surceesful in passing the<br />
SAT.” With this background I CD projecting the theei that a nyatsms<br />
approach might be utilized in the hole training technology into which<br />
our test construction process might ‘be integrated as one of the crub.<br />
systems,<br />
T%e Air TraloLng Coonnend has described the development of a 6yStcmR<br />
npproach as one that ‘I.... views the mnny in~;ividuals and groups de*dcloplng<br />
a particular veapons syetez a8 individual coD+W?nt6, like cogs in 8<br />
machine working togethar to achieve a ccmnon goh~.’ This approach requires:<br />
(1) the definition in precise terms of each person’s job; (2) a task<br />
analysis; (3) a specificat:ion of performance requirements and tolcrfince<br />
limits; and (4) a statercnt of the necessary interactions and cmnications<br />
to be carried out be:rleen groups --each requirencnt established to meet<br />
the predetermined system Eoal.” (Oflesh, 1964)<br />
In this frameuork,the Air Training Command (a subsystem itself, in<br />
overall training and evaluation) require8 a systems approach to include<br />
a task analyst8 as the second step in the sequence. To be useful for the<br />
test construction process these tasks must be stated in measurable bahavioral<br />
terms. A criterion test rmst then be developed co that a<br />
starting point in the training may be dcrermincd. The training materials<br />
are then started at thia point of departure and are carried forvard to<br />
the desired skill level. Our unit then takes over as one subsystem to<br />
construct a measuring instrument which purports to measure the knowledge<br />
which the examf.noe has acqufred at a specific skill level. This subsystem<br />
ranks the students according to how much knowledge relative to other<br />
students he has acquired, the resultant being a percentile score.<br />
123<br />
,<br />
.
Th? systems concept to be mortt effective, demands a alngIC mancger 1<br />
to align the subsysths so that each nerrhoe vlth the other aa lt evolves<br />
into tha einglo eyaten. Affter KQ US&’ has ccrtlficd that the contents<br />
of a opeclfic Specialty Dcocrlptlon correlate significantly with the<br />
relevant tasks involved In the fob, Air Training Commd break@ the job<br />
down lnto tasks and level6 of proficiency vhlch rare ueed as u Job Trafning<br />
Standard, If these tvo documents, villch wa coneldor 8uboystems, are<br />
atgnlflcont, then the subsystezs charged vlth vrltlng the training materials<br />
la off to n good start in making its contrlbutlon to the total reyatm.<br />
Assuming that each eubsystes has been properly constructed, and la properly<br />
coordinated with the other subsyeteae, vork ln the SKI tooting subsystem<br />
ohould proceed amoothly since ve would then be charged only with conatructlng<br />
an instrument that memore: how ~011 a:n airman haa mastered the<br />
metcrlals found in a career development courao. With the syetcm working<br />
optfmall)i wltheach subsystem making Lts proper contribution there need<br />
be no queetlon or concern over &at the SKT ia meaouring. Under euch<br />
circwatnnccs it will be meaaurlng the objectlvce and crfterfa eetabllahcd<br />
by the subsystem charged ofth prcparlng the career development course.<br />
At this point it must be remembered, however. that the SKT ia only sampling<br />
rcpresantativc areas of a teak as datcxmlncd by the teat conatructlon<br />
rubaystain.<br />
ihccaria (1563) hna eaid in dlacuoelng the problem of maeursman~.<br />
“Too often students are meoeurcd to fraction8 of a percentage pctnt agalnet<br />
other students vlthout ever being raeesurad agelnst mlnirxnn job requirements.<br />
There are two t :ln reusona for this phenomenon. Flrat, tralntng oblectlvco<br />
ara seldom atated in definitive enough terms, and aacond, a relative<br />
rather than an abeolute measurement oyetem i8 employed.”<br />
Thlo leads u8 to a diacuseicn of the usea for which an evaluation<br />
ir made. It la important to the whole system that this use be aperiflcally<br />
announced so that it will be one of the objectlvee for each subsystem to<br />
keep in mlnd title making their contrlbutfon, If Dr. Zaccaria’s crlterlonbaeed<br />
evalua:lon isuacd,it will result in a certain group being found<br />
proficient without knowing hov proficient. If the percentile rank lo<br />
used, ltwlll tell us the relative standing of each lndlvlduel in the<br />
group. It 1s up to the eyrtem manager to decide vhlch evaluation<br />
correlates best vlth the use for bhlch the measuring instrument is dosigned.<br />
A system8 approach vi11 work oniy when each subsystem la working<br />
towurd the rame goal under the direction of a slngla manager.<br />
124<br />
--... . ._-_. ^. --- - - .----.-- - -- . _ _ __- -._-- _..,.-.<br />
.<br />
. .
Ceoly, W. D., 6 Cratn, J. L. Dual concept of on-the-job train?ng. -USAF- Instructora’ Journal, 1, (Hr. 1). July 1963, 13.<br />
Draorel, P. L. Evalwtlon procedures for gantral education objectives,<br />
@xcatfonal record, April 1950, 97-122.<br />
Rflgard, t?, R. Theorfas s learning. New York: Applaton-Century-Crofta,<br />
1956, ix.<br />
Judy, C. J. Achieveneat tenting in the Air Force. USAF Inrtructora’<br />
JOWla',, I, (Nr. l), July 1963, 17. ’ -<br />
Ltndgulat, E. P. Educational meaeur&ent, WaahFngton, d, C.: Mericm<br />
.----- -<br />
Couoell on Education, 1951.<br />
Mayor, Sylvia R. Raeemch on QuStRnrtad training at Eloctrooic6 Syotea<br />
Divirion. -111- Trends in pro-rcmmed -<br />
AosocFation, IYbb, 149.<br />
- inrtruction, National Educrtton<br />
Ofbooh, C. Air Tratnfng Comand’r System8 Dtvalopental Approach to<br />
Inatructioml Matertolo.<br />
Vitolr, 8. H. 6 Newman, L. J. A Cmpariron of Two Inotructlonrl Mthodo.<br />
Paper presented at Our Lady of tho Lake Collage, San Antonio,<br />
Tezaa, 1964.<br />
Zaccaria, #. A, Reappraisal of achievesent medeure#. USAF Inotructoro’<br />
Journal, r (Hr. 1). July 1963, 73.<br />
125<br />
,
Sucimary of<br />
Pragmatic Creativity in Excanlnntion CdnbtructLon<br />
A Paper Delivered by<br />
John Credifotd<br />
us lava1 E?mafaing cancer<br />
Hr. Ctedfford spoke on the pragmatic utilftatfon of purely divergent<br />
thinking of nonprofeseionnl item writers. Because of a change in the<br />
time schedule, his prepared material was suzxnarlzed ~6 follows: After<br />
dcacribing the f-CA aa one of the greatest potential force8 in today’s<br />
teatinS, a ploa was made for the deliberate 1ntroduct:on of opportunltiea<br />
for free-flowing thinking into our tottinS ettuotfon. Several teeto were<br />
.‘aacrlbed that wete the results of placing nonmilitary item vritera in<br />
tne position of creating their ovn teets in a purely permissive environment,<br />
Learning by doing, uohuqered by tho rcntrfctlone of<br />
claeeical precedence, they produced sevaral noteworthy teeta, including<br />
a music test on tape administered tk* :r.zmar school chilfiren, a nilitery<br />
eupervfeory test, a queatlonnalrc used in oeiactlng brig guardo, and a<br />
tast for the selection of nonbissed. s*Jp+rvteory personno ir. indust,y.<br />
In line with the top:c of the dfacuanfon, the grectcet bendfft from the<br />
pepar was in the very fluent discussion Mich followed.<br />
126
:.<br />
‘ci<br />
� �<br />
�<br />
�<br />
�<br />
i<br />
I<br />
1<br />
I<br />
. j<br />
.<br />
.<br />
.<br />
.<br />
EvaluatFoll o f zotor s:cs11s<br />
A hOmc;y Little anecdote help6 to clarify my objectives in this pc-per.<br />
My father was born in Lrelantl, ad I recall a story he told me n~ny years<br />
ago about n fellow “greenhorn” whose working hours required him to retu:n<br />
home in the darkness of late evening. his cour6e led h1.m through a p a r -<br />
ticularly dark and deserted district in uhlch anyone with a lively Irj.021<br />
imaginntion could envision the direst of misfortune6 taking plscc. Every<br />
night as he passed through thie area, he constantly repeated aloud,<br />
“Praise the Lord and the Civil ain’t tl bnd man either.” In like vein I<br />
would like to pay due rcspcce to -the written test for its past contributions<br />
to the field of achievement testing and if.8 prospects of even greater<br />
utility in the future, but at the 6ame time, to recognize that the t.eRtfng<br />
of mOtor skills does offer Q great potential to be explored in our<br />
constant effort6 t0 create incree?ingly more effective measurement<br />
instruments.<br />
The rvaluatfon of motor skills will be treated in R q*~itc brosd<br />
COntCXt. The covernge includes all activities pertelnfrq co or involving<br />
muscular movement, not to exclude those requiring previous, concurrent,<br />
and subsequent cogniti*ve procesecrr.<br />
In effect, performance testing in<br />
ita mo6t conprchcncive appllcntion ~111 be considered. All test eituatfono<br />
in b-hich the examine& is rcqulred to do something Other than take a<br />
paper-and-pencil test: will be included. The use of driving tears as a<br />
prcrequisftr forobtaining a driving license in many titetea ir B good in-<br />
dication of the w!de public acceptance of performance test6. Although one<br />
might question the validity of these test6 BS typically administered,<br />
there con be no doubt that n driving test ie a pnrticularly good exqle<br />
oE a test Uf motor 6kille.<br />
The purpose cf subject-netter or achievement tcstfng normslly fr to<br />
provide an evaluation of the level of fob nascery attained by the examtnee<br />
in a given Job, and frequently tc rank o group cf individUala in regard<br />
to their relattve succe68. Depending upon the nature of the activity to<br />
be meesurcd, and frequently certain extraneoufi restrictiona, one test or<br />
R battery of two or more typca of tents may be ueed. In the selection of<br />
measuring instruments we place cnphaeis upon the p:esence of three b8sf.c<br />
qunlitiee: (1) vnlidfty, cr measuring what w want to measure, (2) reliability,or<br />
consistency of tTICiiBIJrf?ment, and (3) objectivley, or the erc-<br />
elusion of personal feeling not based upon accurate observation. We have<br />
at OUr disposal a variety of measuring inetrumenta*<br />
Llritten tent - The multiple-choice ~ypa of written test hoe become I<br />
almost: univerxy accepted ns the basic meonure. It poesecses the i<br />
127
, *<br />
-. aa*<br />
Y<br />
. -..<br />
. . ’<br />
advantages of very complete coverege of the appropriate subSect matter<br />
on the verbnl level in a comparatively short time, simplicity and euee<br />
of administration, relatively IOU COBt, SimplC scoring proccdusee, und<br />
finally, it fs suttable for prqsenting problems that m?esure many cypvpco<br />
of abilities.<br />
Performance rating - This consista of an evaluation of the individual’s<br />
performance on the job by his supervisor; in the Army it is called the<br />
CEZR or Comandcr’s Evaluation hcport. Althcugh ratings of this nature<br />
are susceptible to subJective elements, the design of the reting form<br />
attempts to direct the rater’s attention to obJectfve observation of<br />
behavior,<br />
Pcrformnnce test - fifs mq be in the nature of a work umple such<br />
a8 a typing test, a tapping test to measure an aptitude such as finger<br />
dexterity, or a sitoational test in which actual vork problems nre aimsla&d.<br />
Certain ae?ection and promotion boards can be properly considered<br />
performance tests as can the selection interview in an employment office.<br />
This is true when the interviever or board is attempting to evaluate<br />
b?havlor required in the performance of Job dutiee.<br />
Evaluation of experience and trnu - In 8ome goverzticnt Jurisdictions<br />
a scnre is aiven to each applicant for selection or promotion<br />
bneed upon quality and quantity o.f jbb pertinent experience, iraining,<br />
and special recognition such as awards received. Thie wore become8 part<br />
of the examines’s final rating.<br />
It is obvioue that all Job; require some basic motor activity. Certain<br />
movements of the feet, hands, eyes, and other parte of the worker’s<br />
anntsmy are required to reach the L3.srk aituntlon, to position himself for<br />
the performance of his duties, and to control and manipulate the physical<br />
tools of his trade. The absolute requirement of some degree of motor skill<br />
is present from the most sedentary of occupations to those thrnt require<br />
almost constant motor activity. For our present co?zern, that of testing<br />
military personnel in their occupational apecfaltiee, the broad spectrum<br />
of Jobs is divided into three classes. This purely arbitrary taxonomy<br />
based upon relative importance of motor activities in discriminating<br />
between levels of Job mastery provides a starting point in the determination<br />
of the need for A test of motor skills. Class 1. A large group of service<br />
personnel are required to perform duties that require only the basic<br />
motor skills, the posoeseion of vhfch can be assumed from their<br />
cacceptunce into the service. Although M)tor coordination contributes<br />
something to the efficiency with which they perform their dutles, Its<br />
importance fs ovtrshedowed by the fmportance of cognitive functions which<br />
are central to the executfon of their normal work requirements. Administrative<br />
and general clerical occupations are exsnnplee of this group. It<br />
appears thst a pcrforoance test would contribute little, if anything, to<br />
the eveluetion of these personnel. Class 2, A large number of eervice<br />
. .<br />
.<br />
I.28<br />
,<br />
c<br />
! . *,!<br />
,)<br />
.
.<br />
personnel are assigned to jobe the duties of k%lch require rotor nkil.ls<br />
of 8 mre specific nnture thsn those in the previous category. ThCQC<br />
skilile till have usually been acquired in one of tha service achoola, or<br />
in some casea, will hove been poosesscd by the individual upon entry into<br />
the scrvicc. The motor aspect of these skills might contribute oubstantially<br />
to the quality of the performence of their job duties, but<br />
cognitive abilities are of central importance. The ability to smoothly<br />
nanipulate tools, position physical objects, and perform manual movements<br />
in the adjustment or operation of equipment is required, but the evaluation<br />
and diagnosis of the situation requiring these physical skills le a better<br />
determiner of competence. In this erea, the quart ion is vhether tbe<br />
central skills can be musurad adequately oy a written test, or should<br />
some form of performance test such as the performance check list be<br />
made part of :ho total evsluation. This decision must be made by the<br />
test psychologist vorking cooperatively with subject-matter expcrtn in<br />
the epproprlate field. Before arriving at 8 final decision, pertinent<br />
validation data nnd frequently experimental c’cita must be considered.<br />
Various types of repairmen, mechanics, and equipment operaeors are exsmples<br />
of personnel whose jobs are of this nature. -Class -3.<br />
A COFparntlvely<br />
small number of jobs exist in !&ich motor skills appear to be<br />
crucial discriminators between levels of job mastery. These positions<br />
are typified by duties that involve const.ant, rcpctitivc activities that<br />
lend themselves readily to qt;an::itlve and/or quolitatlve measurement.<br />
Horc important than the fact that these job activities are readily<br />
measurable, is the knowledge that studies of pc. formance on the job<br />
compared with scores on the typical wrftten tests have often indicated<br />
that there is little correspondence between the knowledge of vhat to do<br />
and the ability to do the wrk quickly snd accurately. For joba in this<br />
category a completely adequste measurement of job mastery must include a<br />
performance test. Typist and sterographcr are examples of jobs that are<br />
Included in this class.<br />
The decision as to bbether or not the evaluation of a particular job<br />
should include a performace tcet rent6 entirely upon the nature of the<br />
elements of the job and thelr susceptibl~lty to meaaureneet by a written<br />
test. This declslon can be made by the test psychologist only when<br />
complete job analysis lnfornatioa fa available. ‘I’he analysie of the job<br />
provide8 a liar of duties &ich are required for adequate job performance<br />
and must serve as a starting point in the determination of the types of<br />
tests appropriate to constitute a complete evaluation, The actlvftiee<br />
must be defined, analyaed, and elements necessary for job success must be<br />
Isolated. This process involves the dlvialon of the job into its basic<br />
elemental components, a process which some Cestaltisee will find objectionable;<br />
however, if a particular element of a job fs not adequately<br />
,<br />
129<br />
.<br />
.
-<br />
‘. .<br />
*. .<br />
;-<br />
. d<br />
--<br />
. . .<br />
.<br />
. I<br />
_.-. .’ - . . .<br />
.<br />
.-- __ _ . ._<br />
measured by a wrltton test, we are challenged to find A way to evaluate<br />
this factor. We can concern curaelvca with emergenta, whan WB have<br />
conquered the conetituenta. Dasic criteria for job 8ucce~a euch a5<br />
quantity of output, quality of output, accuracy, spoilage, and eafcty<br />
factor8 muot be determined. Decfslom mat be roade a8 to whether the<br />
performance will be neaeured in terns of pro&Act or process. If it ie<br />
to be product, should WC establish standards rrncnoblc to objective<br />
ueatiureaent such us size and weight, or will subjective mcasureti such<br />
aa moothnees, color quality, and aymetry serve as better doterninera<br />
of competence? If an emphasis i5 placed upon proceae, should we be<br />
primarily concerned with use of tools, proper work methods, or work<br />
aequencef The selection of universnlly superior work procedurea f~ not<br />
@Any. Should a mechnnic vho con consietently analyra correctly a mAlfunction<br />
8imply by listening to the en@& be penalized because ho<br />
doesn’t use the generally accepted tool5 and work nethods? Thie ie only<br />
one cxaarple of the type of problerna to be solved in the construction of<br />
a p,ood performance teat: but it ie a good indicntor of the obstacle5<br />
that a teet conetructor must hurdle.<br />
Aptitude - ~+is is A test of a rotor skill that attcmyta to predict<br />
auccea5 in a particular activity or potential for benefiting from<br />
training. Aptitude tests are not Job achievement tesia and are mentioned<br />
here only because they are one type of performance test and in 8ome<br />
situation5 can be of considerable value. Ihe distinction between attitude<br />
end achievement teeta Is not alway cryetal clear. The U&Z! of the ;Cet<br />
rather than ite n.turc io the important factor, The UBQ of a aubjcctmatter<br />
teat to predict success in a position of a hi.gher level ie closely<br />
related to the u5e of the typical aptitude test.<br />
Achievement Teets<br />
Work aample - l-hi5 test provide5 the examinee with a typical. performance<br />
oituation appropriate to the Job for which he la being evaluated<br />
including a teak or group of tacks characteristic of that required for<br />
actual Job performance. A work srmrple ie not actually a piece of a Job.<br />
Some part5 of any Job would yield little atatiatically u5eful variation<br />
in performance; other part8 might not be adaptable to a teotlng eituatfon.<br />
A good work aampla muat differentiate betvesn good and poor workcre, rind<br />
provide 5core8 reflecting degree8 of proficiency. Thie is poseiblo only<br />
if a fatr 85mpl@ of crucial determiner8 of job 5ucceaa 15 included, requiring<br />
the cxaminee to demonbtratc hia acquired ukilla uofng the tools,<br />
maCerfaL5, tnd method5 characteristic of hfa Job. Teats of this type<br />
have been developed and utilized in the EEC for typieto, stenographer8,<br />
bandsmen, court reporter6, and radio code receivers.<br />
Situational performance tests - Thoro test8 do not attempt to pctttsure<br />
a eimple activity, but one !&rich is rather complex and lea5 well defined<br />
and isolated than the work sample. A group oral teat in which such<br />
130
. . _-. .._. -...a_-e,-,. .-Y.-e--.. .-V.-d_ -..w-.... -..-- ._-. - .I. . . --.<br />
I<br />
. .<br />
131<br />
/<br />
.___,__ .__- -. . .._ -... . -.-.+ - .._, .I .<br />
personality factors as dominance, leadership, judgment, and emotional<br />
6tnbflity are evaluated is an example of this type of te8t. Probira<br />
eolving ability ha6 nlso been mesaured in thfa way by prerznting the<br />
examinee with a unique simulated s:tuation of the nature he might encounter<br />
on the job and rating hie hendling of the situation. Tne<br />
process test, which ia primarily concerned with proper vork procedure8<br />
and sequencea, and the perfonaanct? check lilt are also typec of the<br />
situational pcrformnnce te6t. Hr. Claude Bridge6 of EEC 5.3 presenting a<br />
paper at 1330 houro thie afternoon in the Vest Auditorium on the aubjcct<br />
of perfbmance check lists. Because cf the complcxitiee of administration<br />
and acoritg, high cost, and the availability of a variety of written<br />
tests covering the appropriate eubject matter, the eituational performance<br />
test has not been widely utilized in achievement tearing,<br />
Performance tests for evaluating murk proficfency should be uded<br />
only when a group multiple choice test cannot provide en adequate measure.<br />
In nany instances the mastery of the job con be inferred from the fact<br />
thnt the individual possesees eufficient kncwledge to perform hlc job<br />
duties. Certain jobs, which by their nature, are centrally concerned wit-h<br />
a re;)ctitive, manual activity require a performance teet to supplement<br />
the written test in providing a complete evaluation. A typist, for<br />
exzrple, might have. a good knowledge of the various parts of the typewrfter<br />
and their function, but experience haa shown that this knowledge is not<br />
highly correlated with typing ability a8 measured by a typing performance<br />
test or performance on the job. Jobs of this nature represent only a<br />
relatively small percentage of jobs in the military services. The most<br />
fruitful area for further research appears to be that of the situational<br />
performance test, or more specifically, the performance check list. As<br />
a starting point, more comprehenefve fob analysis is required, job<br />
element8 predicting GucceGG must be defined, the procedural verauo the end<br />
product problem must be rccolved, and better method8 of scoring muet be<br />
developed. The old bugaboos of increnPed expense, greater expenditure<br />
of time, and difficulty in providing adequate and equitable tett sites<br />
are still with us. The competitive group oral and situational problem<br />
solving test6 offer encouragement in evalusting certain hard to meatiute<br />
pcrsonnltty traits; however, their principal value will probably be in<br />
ccrt8in specific unusual teat situations rather than in the area of job<br />
proficiency testing.<br />
__ . - . . . .- --- --- /-<br />
,<br />
.
.<br />
Perfonnsrnra Teat Construction<br />
US Army Enlisted Evaluation Center<br />
Having sntcbliahad the need for a performance teat, we must firat<br />
gfve due conrideration to the objectives of all perfo-ncn teat-e. &r<br />
prime objective is to measure the levels of proffcisncy Fn those critIca<br />
skilla which are not measured adequately by estsblfshad written teaca.<br />
Our Becondary objective ia to deattnifne If the oxamincae achieves (demonotrateo)<br />
skilla which are important, or critical. to mfnfmal parformanca<br />
on the tarkr of tha job. Thnt Fa to nay, we must identify motor (ccilnurl)<br />
rkfIlo that are to be sampled by our parformrinca ta@t in chess categories:<br />
firrt, thora skilla which are critical Fn dlatlngufrhiap different lavalo<br />
oL job marcery; aacond, those vhtch or-6 eeaantial for accaptwblc performante<br />
of the motor trek8 of the job, and finally, those skills which cannot<br />
tanaon@bly be measured by written teats.<br />
Our first step fo to identify which Job taut8 rdquiro thaern rkills<br />
previously mantionad. Prom such A very cmprobcnefva study of the job<br />
tarkb we ~111 be able to pinpoint the actual test trsk. Think of the teet<br />
task a8 our roed F&p which tall8 ua how to get to where we P~Q going.<br />
To a~~ure that our analyrfa is complete end accurate, we ur? the<br />
tack analysir method. Thie cormonly entails the crccompllrhmtnt of a form<br />
utillxlng fivn columpa. Tha hsgdtngo for those colunne are: (1) ACTIVfZP<br />
STEPS (what the worker actually doca); (2) PilOCEPURPI (how the taskr muat<br />
be done); ;3) CAFE AKD USE OF ‘MATERIEL (the toolo and tqufpaent with which<br />
the tmekr are done); (4) SAFETY AND SPECLAL PR!XAUTIU?-JS: and (5) COXUlSIoNs<br />
(reeultr of havfng done these tarks, or work samples). Performance tc8ts 5re<br />
ured for several purpoosn. Providing a criterion measure Fr one goal for<br />
performance testing in tha anied 8CrviCcO. The UIIQ of the task analysis<br />
method ir the beat assurance we have for achieving this goal.<br />
Once thir a~lysir bar been completad, v4 are in a position that<br />
� nablea US to ~QO the whole picture. Hence, we can more readily identify<br />
critical pointr such aa, not measurable, not a ratable point, or to<br />
label A point a8 vary critical. Actually, the taok anslysLa (S~Q prefer<br />
“vork uamplr”) bacomca :he OOUTCQ of items, ratable points, for 0p.r<br />
chxperimsntal inatrumant. -Further, thir analyrir qrortly aide o u r aelection<br />
of the correct type of test inotrument. Will ft be a final product,<br />
a procQII, or even a ccmbinatfon of product and.p ocerr test?<br />
132<br />
I<br />
- ,. ._......____ - .__- ----<br />
.
The octond 6reA of opercatfon’ for performance tetrtero le the conetruction<br />
of the experimental instrument. Au we @tart the proceoo of<br />
&electing rateble points bared upon our AcYI~sIE, there cara et leeet<br />
three criterle which should guide our selecticn. The firot criterion is<br />
reprerantativenecr--the activitice, being measured are rcalietic (joblike),<br />
typicA of the taakr and okillr performed by A quAlifi.ad apectatiat<br />
on the actuA1 job. Performance tenters Are frequently tempted to aubatitute<br />
unraaliattc talka, for a variety of reasona, and often cacrifire<br />
all cr almat all of the criterion, repreaentativenasa. (3318 10 not A<br />
fccetioua refarence to honeat recognition teats or aimlated conditions<br />
tcrte.) Our second criterion le reliability--conaiotent rerulte yielded<br />
for a rerpecteble range of acoree --enough so that a standard can be<br />
developed for world-vide wage. Third, va tast conatructorr muut be<br />
mindful of the criterion, prcctlcality-- feaoible to u&e in tome of the<br />
time, equipment, parosnnel. und ex?enrc to adminirter and to @core.<br />
Raving relected the “itaml” for our eicperimentel instrument in<br />
accordance with the three criteria ve muet next develop the rating<br />
scale needed to evaluate the tasks to be maarurcd. Banically, tie hsve<br />
two typeo of acelee frcxs which to chooat+: the forced cholco;go/no go,<br />
did/did not or the degrees of skill, vhich can take any of oaveral forma,<br />
Auch � 0, numerical (1, 2, 3, 4, 5): deccrlpttvo word scale--for exempia,<br />
poor, good, &VOrAg%, etc. A poerible third acelo, phyrlcml characterirtics<br />
o f the final prodr.:t, ie aomkthea uoAd. Remaber, i t ir eaey<br />
to go overboard with numericel ocal~)s, AI K. L. Ecan (1953) pointe out.<br />
Only thooe rater8 vho are exceptfonally well qualified ohould attempt<br />
uoing A rcale of more than five pointr. The nature of the items of our<br />
inotrment will largely dictate which ocale ir more appropriate. A<br />
word of caution at thfr point Fe in ordar. We tmut ba careful to piAn<br />
porrfble weighting procedurer vlth an eye to chcckfng them egninrt our<br />
crltarle. Ao A rule of thumb, complicated, involved weighting rhould<br />
be avoided. Hany promising teats have been invalidated by cumberroma<br />
weighting.<br />
Our next mc\Jor area of activity f0 the preporstion of inrtructionA<br />
for the axamlnerr And the examtneer. Basic contsnte of the exaninee<br />
fnrtructionr should inc?.ude A deocriptlan of the tank to be done, A lirt<br />
of the pointe of the tetik that will be scored, all the tool8 and/or<br />
equipment to be used, � ���� lkaft (ff applicable) and any other necclaoary<br />
lnatructionr.<br />
Huch more detailed inmtruc:ionr are necessary for our examinerr.<br />
Not only murt we cover the darcription of the taako, ths materiel list<br />
And time limlta bur alro tip8 regarding careful obaervatfon of tha<br />
examfnea’r ptrfomnce--eopeciAlly prwetior?t for 8 btep vhich fr, or<br />
could be difficult to obeerve. Further, theae inotructionr must be<br />
explicit regarding the recording of reruLts with the rating scale. we<br />
P<br />
_ _._ -_.- ._-._ - _-. ---<br />
,
.<br />
.<br />
kxmws hat’ to do if the axamin~a makes<br />
, OR undo en alternate method, OR etarte<br />
ult in fnjury or dmz.aga to equlpmsnt.<br />
hi8 quota from Adkinr, at al (1947):<br />
capcafed by axplicit inotructfonr to the<br />
ratinge and what to look for.”<br />
f. The next activity in ths expsrbneata’l<br />
ctore uoe the etandord teat-reteot tachnique<br />
E fnrtrument. If applicable, rateeting<br />
a helpful. A critical point to rcmmbor<br />
arga -age of examinean--from Lhooe who<br />
s to the most capable onea. (Ideally,<br />
es would be divided into two groupr-y.)<br />
We muat be aura to check and rachack<br />
ru of our instrument. Rcaearch haa shown<br />
ter8 can pay big dividends in .the area of<br />
ility.<br />
our tryout include hslp in doter-mining<br />
the tert; holp in datewining the mfnirxn<br />
sskfng for the Amy. wa would be helped in<br />
:-rrder’s Evaluation and the ET (written<br />
tryout rorsults most ruroly will lead us<br />
the inrtrumont and, poaoibly, even eme<br />
tryout takear us a @lentic step toward<br />
10 into gripe with rcalf*tic analyefe and<br />
Thir necerrltates coneiderable coordlnatfon<br />
:ara end field pcreonnel, not to mention the<br />
I. The purpore of this thorough coordination<br />
reffne our efxperFmsnta1 Fnotrument until<br />
, reliable tast inatrumant. We will have<br />
!dmLnlrtrotions of the official instrument<br />
knd elfmfnats problane not pravfourly antlciidard<br />
forme, 0coritlJ keyr, manual8, etc.<br />
.rtfcal analyrLs of teat renulta In order<br />
Fnstnment for use on a world-vida baafo.<br />
.<br />
H. L., Brfdgeo, C. P., &<br />
of - --I echievment me tc@te. Waeh-<br />
; Offlca, 1947.<br />
i peraonnsl tests, Wew York:<br />
uant. iiarhingtoa, D. C.:
O&#&RAL%YWP011U0111l-SRRS.GE~SEVIF.VESCHULTER,CEIIAIRB;1AW<br />
us HAVAL EXAbllltZ:Q CEIPTER<br />
One Interpretation of the Mjor<br />
Coals of Specialty Knowledge TcatirLq in the<br />
United Stateo Afr Force*<br />
ST8P¶iEN w. FcmS<br />
6570th Peroonnrl Renearch Laboratory, US Air Porca<br />
Kaybe itDo an occupational disease afflicting home of ua who rbork on<br />
the aeocmbly line grindit% out the tests. But it tiometimee happens that<br />
we get so wrapped up in the daily routine of the job and BO preoccupied<br />
with the short-run goals that we’re apt to lone sight of the major goals of<br />
the task. We bscoam no involved in the workaday methods that we find it hard<br />
to urecramblo the enda from the means. So, from time to ticnz, it is worthwhile<br />
to climb aloft and take a fresh look at our referenca poit?ts and renew<br />
our perepective. It ie occalrionally necessary to forget the mode of travel<br />
and concentratr on the destinntion.<br />
IN SUPPORZ OT THI! A?CRMN CXASSlCFIC%TXON SYSTEK<br />
The expreesed goal of the Air Borce specialty knowledge testing program<br />
la to evaluate the technical knowledge possesued by nirran a8 required for<br />
qualification under the Air Porte enlisted personnel classification eyatem.<br />
Towards thie p,oal, the Specialty Knowledge Test (XT) is provided aa an Air<br />
Porce-widd etandard of measure by which to determine job knowledge, apart<br />
from job performance as 8~1~. The SKT is applied not only laterally by<br />
career specialty but also vertically by skill level--namely the apprentice<br />
or eeniskilled, the journeyman or ekilled, and the advanced levels of qualification.<br />
Ae a criterion for skill upgradinS, the SKT ie intended to supplement-but<br />
not supplant--other criteria, such a8 demonstrated proficiency on the job,<br />
job experfence and history, supervisor’s recorrenendation, and coaxzander’a<br />
approval.<br />
upgrading.<br />
Thus the SKT is by no mean8 intended to be the eole criterion for<br />
Toward8 managerial control. In effect, the SKT aervea a8 a managerial<br />
control device whereby Headquarter8 USAP ie enabled (1) to ensure that the<br />
airman manpower resources meet the oetabliehed minimum requirement6 in terms<br />
*It is emphasized that this is one individual’e interpretation. Thio u83er<br />
doer not neccssartly reflect the official policy of the Air Force. No; hoes<br />
it necessarily reflect the position, whether official or unofficial, of any<br />
major air commend. Appreciation Is eepacfally due Lt Co1 Albert S. Rnauf,<br />
USAP, for the stimulating dialogue that evoked many of the observations noted<br />
herein.<br />
.<br />
,<br />
136<br />
L<br />
_ . -- ._<br />
i<br />
_... * ,<br />
f
.<br />
.<br />
.<br />
of technical knowledge, and (2) to accwlish a rza~ure of sttndsrdtrat?on cf<br />
airman knowledge on 6n Air Force-wide boeis.<br />
Oualitv control<br />
_L - - • It is axioostic thut knowledge ie power. Eiobherc is<br />
the truth of thin axiom better founded than in a tmdsrn military organlrstion<br />
I in which the succc~o of the nfseion depends on the qualitative superiority,<br />
r&that than quantitative strength, of ire F;E.npower, Knowledge, then, may<br />
be regarded a- a form of resource. Like t’lZteria1 reBturce8, technical knowledge<br />
, io subject to deteriorntlon, Leepletlon, and obsoleacenrc. To be n-aintaiced<br />
i.n a conetant state of readiness, knowledge rnuot be continuouely reneved,<br />
restored, and cultivated. Unlike material reBource6, however, knowledge is<br />
not readily mnable to rneneuremcnt for inventory purpoaee. Nevcrtheleas,<br />
througi. the SKT program, the Air Force seeko to maintain a slose check on<br />
the job knowledge reaourcea that reside in the enlisteJ nanpowcr population.<br />
Gtandarditation: one AP language.<br />
-I Through etandarditatfon of knowledge,<br />
the Asorce seeks to neutr$tllzt the hazards of specialization and division<br />
of labor. One hczard is the propensity of each organization todrvelop its<br />
own concepts, its own doctrine, ita own private language of epeclalized<br />
teminologv and nomenclature. One could fancy the crnergence of an Air Force<br />
tower of babe1 as the thec,retical ootcorae of this tendency, carriei. to it6<br />
ridiculous extreme. However, it should not be unrealistic to credit the<br />
SKT with making a notable contrfbution to the CLPUEQ of standardization of<br />
knowledge. Acting aa a vital stiolulant to the currency of a coamn technical<br />
language, the SKT helps keep the Air Force family of comorande on the cm<br />
wavelength for purpoeee of concnunication.<br />
Standardization: the “c~leat” air.-c!an. Another hazard is the tendency<br />
for each mjor air con-x to foster that job infornntion that la directly<br />
relevant to it8 own mission, to the virtual exclusion of broader areaa of<br />
knowledge having wider applicability to the general Air Force miseion. This<br />
is understandable inasmuch as each command ia under relantles8 preaaure to<br />
meet the demands of ite irrnediate mission. On top of their many burdene,<br />
the commands bear the donkey’s share of tralnlng burden. If an individual<br />
does a creditable job towards the fulfillment of the conmvlnd mission, the<br />
commend will naturally tend to want to overlook any knowledge gaps that might<br />
limit the individual’s potential value to the Air Force at large, As long<br />
as he ia a good SAC man, who careo if this man could ever be of any earthly<br />
or airborne use to MATS, or vice versa? Well, Headquarters USAF cares, And<br />
. so should MATS care from the very standpoint of ite own long-range intereete.<br />
And 80 indeed should SAC. For the atrman’s breadth af knowledge stamps him<br />
at once with both his professional and hir, Air Force identity. It is the<br />
. mark of his versatility and employability within his specialty. l’hiti quality<br />
in the airman largely relieves the Air Force of the need for retraining him<br />
extensively with each change of assignment--whatever the command, whatever<br />
the fob, whatever the specific nature of the equiprent involved.<br />
*<br />
137<br />
.
.<br />
Systematic upgrading. At the same time, -the Air Force achieves other<br />
gains through the use at the SKT PS a managerial control device. By channeling<br />
the upgrading proce56, tile SKT regulates the flow of technically knovledgenblc<br />
personnel up a host of career laddere. Rclpero, apprentices, journeymen,<br />
tcchnici6ns, 5upervisor8, superintcr&nts--all arc kept advancing at on orderly<br />
pace, each group maintaining the preecribed distance of feparetion from the<br />
othere, A kind of braklng mechanisu, the SKT kcepo ampetition from disintegratin!<br />
into a chaotic ecramble and providcn a hedge againot the runaway Inflation of<br />
ratlnge. A sort of traffic control system, the SKT set8 rz&xm speed limit8<br />
in the form of qualifyfng percentile scorea.<br />
Wotwithetandfng limitations. To be But@, the teat doe8 not have the effect<br />
of sunnnarily and irrevocably eliminating in drove8 failures from any furthercompetition,<br />
On the contrary, through reteating and board action, all but a<br />
negligible proportion of the failure8 eventually get by. Fhat ie important,<br />
howover, 18 that the test doe8 serve to keep the rate of progression within<br />
manageable limit8 for purpose8 of effectrve personnel salectlon. And to be<br />
sure, one can envisage a c-are flexible application of speed llmite, depending<br />
upon what the traffic will bear. A rather permis81ve puss/fail ratio might<br />
be justified for a critically undermanned specialty, or a very restrictive<br />
ratio for an overcrowded one. However, :.xh manipulation to accomdate variable<br />
supply-demand relationships azxng rpecialtics would hevc to be carefully conoidered<br />
in thr. light of poaeible conflict with the goals of 5tandardfzat:on and quality<br />
control,<br />
All told, the resulatory effect of the SKT function cannot be denied. It8<br />
value a8 a absnagerial control device ie appreciable, elrpecially when a88eEsed<br />
against the alternative of no control.<br />
k’or the good cf al.l. The gains achieved by the SKT program are by no<br />
means confined to the hig!lest level8 of management. The beneficieries of the<br />
program am found at all levels right down to the individual airman. Couxnandere<br />
and 8upeTvi5or5, faced with mekfng selections from amxtg relatively homgeneous<br />
group8 of personnel, find A trusty catalyet in the SKT, helping to ease the onus<br />
of decision. Heanwhile, the individual airman is provided with an objective system<br />
of career progreoslon, giving him the opportunity to compete in a eervfcewfde<br />
arena. Under etandard conditions, the testing situation Afford8 him the chance to<br />
demonstrate anew his capacity, in term8 of specialized know-hgFI, for advancement<br />
in level of responeibility and authority.<br />
IN SUPPORT OF TRAINING .<br />
Thus far,the discussion has dwellod mainly on those goals of the SKT program<br />
that relate to the use of the test a5 a managerial device in support of the airman<br />
claeslfication system. There are certain other goals of the SKT that produce an<br />
impact on the training function.<br />
138<br />
.
#<br />
___.<br />
-<br />
: ..\ \<br />
c<br />
. .<br />
I<br />
. i<br />
I<br />
1 i<br />
.<br />
.<br />
.<br />
.<br />
.<br />
Screening byppasa cpecfalloc~. The use of the SKT for the identification<br />
of bypass specialists is ohe of these goals. A recruit, &IO has sme ~acialty<br />
backgourd derived from civilian ocoupational experfencc or schooling or fros<br />
prior military service, azy t&e the SKI to ascertain his qunlificntion in the<br />
spec‘lal ty. Cm passing, he is svarded the smtskflfed rating,. Every year the<br />
bypass-specialist program yields significant savircge in term of cfrc-umented<br />
trainfug costs.<br />
Hotivatinq study. There is another goal of the SKT progrran that brings Lt<br />
into an affinitive relationship with the training program. This goal is i-licit<br />
in the publication of study reference lists for the guidance of alrum in preparation<br />
for specialty testing. The effect is to motivate study on the part of the<br />
airman-to kindle his urge to acquire the technical knowledge that is considered<br />
crucial to his successful pcrfo,rmance on the job and to his long-range career<br />
development.<br />
Lrpact on Iraining. As a consequence, preparation for the test become big<br />
businese. &my people get into the act at all levclo. Training program go<br />
into high gear. Tne impact upon training is felt throughout the Air Force, but<br />
nowhere is it felt more acutely than at the Air Training timand, In turn, the<br />
latter undertakes to produce training standard,0 and job informtion for b.ir Porte<br />
publication. r;ot only are these made available to tha traLnee for his use in<br />
preparation for the test, but they ate also adopted by the XT progrfun for input<br />
to the test-construction process.<br />
A r.c*~ epproach to CUT. A noteworthy outcome of the interaction between trainin;>.<br />
and evaluation has been the recent adoption by the Air Force of the dualchannel<br />
concept of on-the-job training. This approach to CUT. provides for the<br />
synchronous development of the airmen’s cereer, on the one hand, and his job proficiency<br />
on the other. The end product of one of these training chaonels is the<br />
Career Development Couree (CCC), which is a self-study course geared to the<br />
aiman’s specialty for his use in preparing for the next higher skill level.<br />
Since it is a self-contained peckage of career specialty information on fundamentals<br />
and basic principles, the CDC is a welcome source reference for use in<br />
SKT construction. Thus has the SKT fulfilled itself, in part, through ite<br />
salutary *act on training.<br />
The isaue of fndependant evaluation, So salutary, in fact, has bean the<br />
two-way interaction between training and evaluation that, for practical purposes,<br />
a sort of symbiotic relationship has emerged betveen the two. The intimacy of<br />
this relationship has been the subject of considerable interpretation and, possibly,<br />
overinterpretation.<br />
-I_ The ca6e against. One strongly voiced interpretation holda that evaluation<br />
should be an integral part of training. Accordfng to this view, the SKT program<br />
would logically be assigned to the Air Training Cormmnd. It ie maintained that<br />
the closest possible coordination 1s needed to effect greater efficiency and<br />
economy of test production. Such a wedding would eupposedly enhance the mutually<br />
supporting relatfonehip between the two functions. Thus a higher degree of mutual<br />
responsiveness would become possible In a more intimate association.<br />
139<br />
;<br />
i<br />
,i
2<br />
. .<br />
. -<br />
“.,.>__ ., . _ _ _ _ . . . . . _ _ -<br />
4.h.. ._ _ Tha caBe for. Another view, the preveillng ona, is that eveluation should<br />
ram&in independent of treining. Evaluation, it is contended, is properly a<br />
coordinaea function vfe-a-vie training and should not be relegated to a s~ordinata<br />
role, To do so%uidx BornewhAt like Qlscing the bar examiners under the<br />
low-school faculty, the teacher-certification agency under the normal. school,<br />
or the Auditor under the bookksepar. tiir would ignore tha nosd for checka and<br />
balances between training and evaluation A% indcpend-antiy coexisting functions,<br />
As an evaluative instrument, then, the SKT is supposed not - -to<br />
bs a form of<br />
training de-rfca. To regard the SK? 8~1 euch would supposedly be a groea nioconception<br />
of it6 true function, even though the SKT dcee have the extrinsic<br />
effect of operAting As a ntudy LrcpeIler.<br />
I a’<br />
/<br />
* \.<br />
.’<br />
,<br />
--<br />
c<br />
According to its Qroponents, evAluatfon is better able to respond flexibly<br />
to rapid technological developments with the Introduction of new subject matter,<br />
And to cope with the sudden obsolescence of old subject matter, eince it is A<br />
far simpler mtter to revise a test than to revise a training courne. To submerge<br />
evaluation in tralnfng, it is pointed Out, would impoea upon the former<br />
the inherent drsubacke of the latter. It in considered cruciA1, therefore,<br />
that evaluation remain directly recrpooet.re to the clsesificntion stclndards<br />
rather than become elavishly dependent upon the errining standards. As for<br />
the ecorrcmies anticipated with the pro?oaed consolidation, it le believed ehicl<br />
would have to be calaulatsd in term of overell cffactivenecs rather than<br />
purely Dionetary units. The sacrifice of IimSted pecuniary aavi~~ge is be-ieved<br />
to be a relatively cheap price to pay a~ part of the coot of independent<br />
nvAtuAtion.<br />
Theee two opposing positions retative to independent evaluation form the<br />
horns of a dilemm that periodically rear into view. ‘What’8 the eneuer?<br />
Whatever the Ansver, it mst first be recognized on both aider that tha<br />
qussti.on iteelf is recilly neither one of tralnirg nor of evaluation in an<br />
exclusive eense. Rather, it is a question of rcanpower utilization in A<br />
corcpreheneive renee and probably should be Approached Aa such. Whatever the<br />
answer, it should be identifiable neither as training policy nor a8 evaluation<br />
policy but as a amnpower management Qolicy. In short, it should be an Air<br />
Porte anmer to an Air Force question.<br />
Surnnary. And there It ie-one vereion of the mjor goale of specinlty<br />
knowledge testing in the Air Force. Ye have reviewed the goals of the SKT<br />
as an instrument of management in support of the airman pereonnel classification<br />
ayetern through quality control and standardization of knowledge and through<br />
systernstfc career progreeeion. We have alao reviewed the goals of the SXT<br />
in relation to training, not only to energize study but also to screen bypass<br />
epacialiste. Finally, we have scrutinized the icpace of ehe SKT on the training<br />
program And raised the question& independent evaluation, In 00 doing, we hnve<br />
euggested that, whatever the answer, it should be transcendingly Air Force in<br />
it8 spirit.<br />
. .
.<br />
,<br />
.<br />
.<br />
-<br />
.<br />
.<br />
Uses of NOS Evaluation Teot Results<br />
J. E. HMREITM<br />
US Army Enlieted Evaluation Center<br />
Dr. Bruner, a mathematician and geophyoicist, more renowned for hie<br />
contributions to hi8 fields than by the number of his publications, once<br />
asked his students to differentiate between mathematicians and the colculur.<br />
After a few minutes of their profound, evasive silence, he pointed out<br />
mathematicians use calculus to solve problems which neither can solve<br />
alone or which mathematicians can laboriouely solve with lees efficient<br />
methods. Test6 of occupational capability, like the calcu?u.s, are tools<br />
which can aid in the solution of per?onnel management proceasea only when<br />
they are capably used. TeGt users, like mathematicians, can more<br />
effectively solve their perconnel evaluation problems &en they make full<br />
and appropriate use of the tests available to them.<br />
The fir-et Army HOS Evaluatior Tests were edmlniotered in January 1959<br />
as a basis for the award of proficiency pay. They were called HOS Proficiency<br />
Tests and, for the most part, s-led the abil?ty of enlisted<br />
personnel in grades E-4 and above to recall the fundmcntols of their<br />
primary HOS training. The raw test 6corea vere converted to Arm; standard<br />
scores, weightnd, and added to weighted Commander’s Evaluation <strong>Report</strong><br />
rating scales co provide a composite “proficiency score.” The te6t acores<br />
. .., and proficiency acores were reported to the enlisted personnel concerned<br />
and their unit personnel officers on a form entitled “Proficiency Data<br />
Card.” Summaries of teat results were furnished to Headquarters,<br />
Department of the Army, and major cousnands. The minimum score for the<br />
award of proficiency pay for each Hilitery Occupational Specialty (!%3S)<br />
was determined by the training requirements and attrition rate for the<br />
MOS and the number of proficiency payments that could be made. Lists<br />
of minimum proficiency score8 were distributed throughout the Army for<br />
corrnnanders to use as a basis for individual proficiency pay awards. Ae<br />
you can readily see, the first MS Evaluation Tests (MOS Proficiency Teets)<br />
were used in two vaye: They were used by cormmurders and unit personnel<br />
officers to determine which of their enlisted personnel met or exceeded<br />
the minimum rcquiremcnts for the award of proficiency pay. The tests were<br />
also used by enlisted perrronnel and their officers to estimate how they<br />
ranked with all others tested in their MOS.<br />
Current trends in the Army MOS Evaluation Program are tovard increaeed<br />
emphasis on: sampling enlisted personnel’s LJilitieo to solve job<br />
problem, multipurpose scores, and improved reporting of results. Becauee<br />
of the wide variety of specialties in the Army and the large number of<br />
personnel tested, realization of these trends is necesearily more complete<br />
for some NOS than for others, though some progress has been made in all<br />
141<br />
.*
.<br />
-<br />
. . -.. ..- . .-. .._-,_.-&s -..-._..-..... a-.-r-..< .I.- -w--w ,, -- . -_ ..-. . ..- _ -...-<br />
nreat3, Aa new ldcns arb developed, or m6thoda of adapting old fdeeo are<br />
visualized, lmnediate goal5 .wi.ll ,ba extended and trcndb: m’dified accordtngly,’<br />
Since the account of things to came. mat evolve from the pteeenc f the scope<br />
of thie prcaentatfon io limited to currant hippcning~ with a faw indices<br />
of next steps md anticipated hazardo.<br />
The emphaafr in teat covarage haa been shifted from trafnfng materi<br />
content to the problemc encountered by pereonnal senlgncd to the HOS<br />
between their graduation from formal training md their prowtforr to in<br />
advanced ski?.1 level or MOS. Par HOS in which cepcciaily needed (Eandsmen,<br />
Typists, S:cnographere, Court Repotters, and Radio Code Operators) performonca<br />
tests of motor and sensory skills have been developad for<br />
standard adminietration world-wid6 and are used a~ supplements to papcrand-pencil<br />
tests to product more job-related composite ecorea. Job-semple<br />
problems, adapted for multiple-choice anawerfng, huv6 been developed and<br />
uRed in aeverel testr. (These problems are bneed upon repreaentativa<br />
aeaignments for the specialty skill level with coannonly encountered<br />
aituctlonal data presented in nnrrattve form, or recorder tapce, or<br />
drawicge, fully or partfully completed forma, or ocher vieuel meana.<br />
Representative eolutfono of outstanding and infer-lor specialiats are provided<br />
for choices.) Several test outlines have been radevelopcd along<br />
f!:nctional linea to produce eubncores which reflect the comparntfve<br />
abilitiee of examineee to perform the various duty pcnitionp within a<br />
sp6Cialty. Atl can readily be seen, the net result ie a ronvsreion of the<br />
primarily job knowledga tests developed in th.1 early stages of the Army<br />
Evaluation Program to more predcrinantly job-pI’oblem eolvfng abilitiee<br />
tests. When the converafon ie complete, all KOS Rvaluatfon Teet. ecores<br />
can be used with confidence to determine how well examfneee ten perform<br />
the current mafor dutiee of a military Bpecialty rat.her than how much the<br />
examlncoe knou about the fundamentals of their KOS.<br />
Tvo additional index scores ere developed and reported: the NOS<br />
qualiflcatfon acore and the promotioa qucliftcatfon acore. The KOS<br />
qualification score le a *minimum passing score used to determine vhich<br />
sxtminees should be retrained or reassigned to P mor6 appropriate<br />
rpecielty skill levsl. The score can be used to determine which examfneee<br />
with a critfcal primary MO5 will not be awarded the sllpplemental<br />
pay for their assigned opeclalty. The method for determining the HiXi<br />
qualification score is bacied upon the premise that experts can arrive<br />
at the absolute number of question8 in e teat which, if answered correctly,<br />
distinguieh between the minimally qualified and unqualified pereonnel<br />
aaefgned the H03. The Promotion Queliffcat:on Score nay be used by<br />
commandera, if they desire, es a requirement for the advancement of enlieted<br />
personnel within their commend to a higher pay grade or ckill level.<br />
It is the score attained or exceeded by one-third of the examinees arai.gned<br />
in the same pay grade in the HOS skill level teeted. Where such comparfaone<br />
142<br />
_.-
- .<br />
.<br />
‘..<br />
!<br />
_<br />
. I<br />
1.).<br />
--<br />
.<br />
.,2<br />
‘.<br />
L’<br />
.<br />
.<br />
:<br />
. I<br />
..-<br />
. .<br />
\<br />
_-<br />
.’<br />
‘s . . ,,<br />
‘.<br />
i<br />
c *<br />
:<br />
4<br />
\<br />
/<br />
:’<br />
. 1<br />
.<br />
.<br />
were made, this score comnly 1.~38 In the range of avcrcgr! wares made<br />
on the s&ne test by soldiers in the next higher pay grade.<br />
The Proficiency Data Card has been replaced by the WS Evaluation<br />
Data <strong>Report</strong> which, In addition to the MOS Eveluotfon Score, reflect8 on<br />
a five-point stole the proportion of the itcza in each QajGr test creil<br />
the examinre answered correctly. NOW, all exazzfnees and their 5upcrvioors<br />
have available a medium for determining how each excninee’a compooLce<br />
Bcore (test + commander’s rating) compare5 with the Bcores of all other<br />
Active Army exminees with that primary HOS 61~111 level and how each<br />
exazzinee succeeded or failed in answering the questiocq related to the<br />
major teot areas. Examinees can and should uBe their reported test scores<br />
88 a guide when preparing for retesting during the next scheduled evaluation<br />
period for the NOS. If they concentrate on improving their skills In the<br />
areaa In which they answered the snaller proportions of item8 correctly,<br />
they will broaden their overall MS ca-pahflltLes, and usefulness to the<br />
Army, more rapidly and efficiently than they can by divcrsftying their<br />
study cfforte. Supervisors and unit coxnanders can also identify the<br />
test areas in which their subordinates answered the smaller proportions<br />
of Items correctly by reviewing and summarizing the XOS Evnluction lIeports<br />
of their oubordlnates. The results of their reviews can and chouid i.!<br />
used to plan their training programs and training emphasis.<br />
Each major coz#nnnder is provided smary report5 of the teat acorcs<br />
attained by the enlisted personnel tested within his coumarnd. These reflect<br />
the distribution af the MS Evaluation Scor.e of the personnel tested<br />
by the comnand distributed according to NOS skill level, pny grade, and<br />
principal coaznand subdivisiona. From these reports the major conmondera’<br />
staff can determine: (1) the numbers, locatione, and pny grade5 of personnel<br />
tested who failed to attain the minimum score for the avnrd of a<br />
verified primary HOS; (2) the numbers, location, and pay grades of personnel<br />
teeted whose scorea are in the upper third of the acoree of all personnel<br />
assigned the primary WL!~ skill level tested; (3) the numbers, locations,<br />
and pay grades of the personnel who may be awarded specialty pay; (4) the<br />
numbers, location, and pay grades of the pereonnel who may be awarded<br />
superior performance pay. Theee reports supplement the strength reports<br />
of the command by providing information concerning the capabflitfee of<br />
personnel assigned enlleted specialties within the conmwd( The staff<br />
is no Longer limited to reports that there are X men with specialty Y<br />
In the command, The command twmary reports add: A of the X men are In<br />
the upper third of all of the Y speciallets In the Army; B of the X men<br />
failed to cylnlify for a verified primary HOS; C scored above any determined<br />
more point; the average score of personnel assigned to a given specialty<br />
ekfll level was , etc. Such data can be used for estimating training<br />
needs, locating specialists for critical aaeignments, and related administrative<br />
processee.<br />
143<br />
.<br />
i
During the post year, the fiOS Evaluation Tisting Program has been<br />
extended to the evaluation of the ability of Active P.my personnel to<br />
perform :he duties of their sccondery HOS and the ability of Reserve and<br />
National Guard personnel to perform the dutiec of the N0.S for thefr current<br />
duty positions. Uhile the fame tests and reporte are used for thece new<br />
purposes, evaluations are beaed upon teat acores without courrander’e<br />
evaluation ratings and are used only to identify those exmtneee who<br />
failed to attain the Active Army minimum qualifying test BCOI’C for nosignment<br />
and training purposes.<br />
fumy l?OS Evaluation Tests are also used in determining the pay grede<br />
and HOS of commissioned and warrant officers vho intend to enlist cr reenlist<br />
upon termination of his current active commissioned or varrant<br />
officer service, Eligible off tcers are permitted to take aa manv as three<br />
HOS teat6 appropriate to pay f rade E-5 or their prior enlisted temporary<br />
grade, if higher. At least c!-e of the tests must relate to an HDS which<br />
hue vacancies at their reqtleh!ed pry grade. None of the tests may involve<br />
an overstrength MOS. Finol tictcnninntion of pry grade and MS ia mnde by<br />
the Department of Army Grade Determfnetion Board based upon the teat<br />
acore6, cotprlander’s recommen;ations, and other pertinent, available<br />
data.<br />
In each of the current gpiicatl.ons of ;.rmy HOS TeLting Programs, it<br />
is the connander who must take the final action. The cocszander can give<br />
an enlisted man tiho fails to make a passing score and acquire a verified<br />
primary HOS a second chrnce to improve his skills before the next<br />
scheduled test session ior the HOS or reassign him to a more appropriate<br />
MOS and okill level, but must reclassify the man who fails twice, The<br />
connander can withhold Ruperior performance or opecialty pay from personnel<br />
who attain an eligible score if he determines the individual is not currently<br />
performing in a satisfactory manner. And major commandera may restrict<br />
promotion8 to those whose scores fall within the upper third of the acores<br />
for their specialty skill level.<br />
Several have euggested that MOS Evaluation Teat reeults be ueed to<br />
evaluate the effectiveness of service school training. At first glance,<br />
the proposal appears to have merit; but closer inspection reveala crucial<br />
f allaciee. In the first place, eervice school training is ordinarily<br />
designed to provide enlisted personnel with the basic vocabulary, theory,<br />
methods , and procedurea that enable them to begin their on-the-job training<br />
at their first unit of assignment. However, HOS Evaluation Teats are intended<br />
to cover the period from the completion of on-the-job training to<br />
advancement to a higher akill level or MOS. In the second place, the<br />
students vho fail the school NOS course are rarely, if ever, aoaigned the<br />
NOS where they might succeed or fail depending upon the degree to b%ich<br />
school training met on-the-job needs. In the third place, many of the<br />
144
enlisted personnel evaluated, particulnrly senior tcc!lnicfans and noncommissioned<br />
officers, have not attended a service school course for many<br />
years, if at all. Consequently, much of the material acquired during<br />
service school :raining by such personnel b.>uld not be covered by teuts<br />
restricted to curxent doctrine and matcriai. It should ail&o be obvious<br />
that the highly-motivated, rapid learners k&o do uell in school courses<br />
usually perform better on the job and NOS tests than slow learners and<br />
those not motivated to make the Army their career. Any correlation<br />
between schoo! grades and MS test acores in influenced by the quality<br />
of school training and by lear,ling abilities--motivnticn factore. The<br />
degree to which positive correlations are increased or decrascd by the<br />
quality of school training and its relation-hip to MOS requirements<br />
cannot be determined from the school grades--test score correlations<br />
directly, Additional studies would be required to determi:;e the extent -<br />
and direction of the influence of training upon job success, While<br />
there are probably other reasons why MOS Evaluation Test scores should<br />
not be used to evaluate the quality of service school courses, the four<br />
reasons offered suffice to negate the proposal.<br />
Others have suggested HOS Evaluation Tests be used to determine<br />
whether persons called to active duty in a mobilization may be assigned<br />
directly to a unit or whether they require preliminary training bc fore<br />
assignment to a specialty. The Reserve --National Guard testing program<br />
obviates the need for retesting of members of Reserve and National Guard<br />
units who will be assigned to Active Army units or issued current Active<br />
Army equipment. But HOS Evaluation Tests cannot be used to determine the<br />
abilities of reservists, guardsmen, and draftees to operate, maintain,<br />
and employ the limited-standard, demothballed ,laterial ubed in situations<br />
requiring major mobilization, because MOS Evaluation Tests cover only<br />
current, Active Amy doctrine and material. It would be necessary to<br />
develop additional tests of the abilitles of exeminees to perform the<br />
specific duties of the additional specialties to be used in the mobilized<br />
force and to establish minimum standards of performance for those<br />
specialties to determine which examinees require further training and<br />
which could be assigned directly to a unit. Tests of basic theories and<br />
principles would only rate the abilities of examinees to learn--not to<br />
do. It is possible that time limitations in major mobilizations would<br />
preclude the development of adequate placement tests and that the benefits<br />
derived from such a program would be less than those resulting from the<br />
refresher training of those who pass the tests.<br />
.<br />
Some have also suggested MS Evaluation Tests can be used to predict<br />
how well an examinee will succeed on the job. That is, how will his peers<br />
and supervisors regard him? Certainly, one must agree that such a goal<br />
is desirable, However, the question is: “Is it practical at this time?”<br />
145
I--<br />
,<br />
c<br />
Now, an MOS Evaluation Test. rates examinees’ ebilitiec to perform the<br />
full scope of the duties of an HOS skill level, weighted to fit the<br />
objective3 and concepts of relative importance of the standards setting<br />
level of the Army pro&ream manager3 staff. In other words, the testo are<br />
biased in favor of the broadly-skilled examinee rather than one who is<br />
exceptionally skilled in a pnrt of his HOS skill level reqafremcnts.<br />
The test score indicates the exazinee’s relative standing among all<br />
examineee ansigned the MOS skill level. If peers and supervisors can<br />
adjust their ratings of exr=,inces to comyenaate for their personal<br />
bioaes and influences, if they thoroughly understood and accepted the<br />
objectives and important concepts of the standards setting level of the<br />
program manager’s staff, and if they could simultaneously and comprehensively<br />
evaluate the abilities of all personnel assigned the primory<br />
MOS skill level, they would probably rate the exnminee as the test rated<br />
him, if the examinee reacted to them as he reacted to the ceet during the<br />
evaluation period. Ohviot~sly, peers, auperviuore, and exainees interreact<br />
differently, differ in their opinions a8 to the relative importance<br />
of tasks, have varying degrees of understanding of the specific objective3<br />
of top level standard setter3, and have rarely, if ever, had the opportunity<br />
to evaluate the abilities of 311 persons assigned to any MOS skill level,<br />
let alone nimultaneously. The test3 do provide.a basis for predicting how<br />
well an examinee could do on the ; Jb when permitted to do the whole job<br />
rather than a oubspecialty and when motivated in the acne degree and<br />
direction aa he was at the time he took the test. A greet deal more information<br />
concerning his interpcreonal relationships and attitudes along<br />
with those of his peer3 and supcrvisor3, their rating attitudes 6J d<br />
abilities, and the dynamics of the group would bc required before one<br />
could predict Low an examinee would be regarded oy his peers with<br />
reasonable accuracy.<br />
A few item writing and technical publications writing groups have<br />
cooperatively used test question response data as a cue to whether technical<br />
material has been dietribuced to and understood by examinees. With<br />
few exceptions, the test question analyses developed by the Enlisted<br />
Evoluation Center reflect how all of the personnel in each MOS skill level<br />
responded to each test question. Items relating to equipment or doctrinal<br />
changes are carefully reviewed when only a small proportion of the CXaminecs<br />
answer the questions to determine whether the queetione or the<br />
cxaminees are deficient. Checks are also made to determine when guides<br />
and training materials were distributed to exeminees and whether these<br />
material6 require clarification or amplification.<br />
Many will envision other u3es of PiOS Evaluation Tests. In planning<br />
the uses, one must bear in mind and insure:<br />
a. The te6t content and objectives are compatible with each<br />
planned use.<br />
trolled,<br />
b.<br />
All pertinent variables are identified, evaluated, and con-<br />
____ __- .--.--- --<br />
146<br />
-.-_- _._. _ ..-.._. -<br />
- - -_ - -.
,<br />
.<br />
c. Test-ocorss which only reprtaant uslotive rtandlngr of<br />
%x~~~fncer ore not logically summed, multiplied, aubracted, or divided<br />
by ordinary arf thnotic p~xmmt1s.<br />
d. The cmponmto and wcighttngo of cmqcsbta ecoreo era<br />
cczqatible with the plantlad u8e.<br />
e. A teat deeignusd for one purpose cannot noceesarily ba<br />
used fox whet appear8 tc be n related pcrpoee.<br />
147<br />
;;<br />
^f
f<br />
.<br />
Job P.na1ysi.r for Tart Development F’urpozars<br />
Prank H. Price, Chsi-n<br />
US Amy Enlisted Evaluation Center<br />
Today the grmt need in mfXitary teiting -- perticultrly job proiiciancy<br />
evaluation -- is a sound understmding of the baeic job. Gnly<br />
if the job to be evaluated lo known in all of itn detailed charactoristics,<br />
can teata be developed to adequately measure euccess in the job.<br />
fn other wrdn, we naad information about the job in vhich ue hope to<br />
ta8t proficiency.<br />
Job onalysia la a proteus of obtaining information about jobo. While<br />
fob an~1yri.s can serve vorioua uoeful peroonnel purposes, the most Important<br />
one for our consideration ia that of retting pereonnol rpecffications<br />
required in a particular job. For tent construction purposes, a job<br />
dercription is the end product of job analysis. It ir vital that the job<br />
dercription be complete fn every detail. Success or proficiency in a job<br />
cannot be properly wetueted unless tho nature of the job is fully knaan.<br />
There era a ambar of different agproachnm to enalyxing jobo and<br />
writing derr:iptiona of them. The papcro which Dr. Horah and Nr. McBride<br />
will present this morning mre illurtratPve of different approaches to<br />
obtaining job Infonaation. We believe you till find the papers intereotlng<br />
end inform.stivo; and we hope thay vili stimulate job anslyrir effort In<br />
your mm organIs~t1one.<br />
148<br />
-- --. . - c<br />
,<br />
.<br />
.-<br />
.
I .<br />
.<br />
. _ ,-,:. ._ .._ ., -.. --.. “., __ _. * * _<br />
.,. . .,<br />
New Perrpactives in Job Analysis<br />
JOSEPH E. WRSC<br />
6570th Personnel Research Laborctory, US Air Force<br />
. .<br />
As the result of an intensive research program during the past ftve or<br />
six years, the United States Air Force has dcveioped and applied a novel<br />
procedure for collecting, organizing, analyzing, and reporting comprehenafve<br />
job lnformatfon. The procedure cmbines features of the check list method<br />
with those of the open-ended questFonnaire and the observation interview<br />
into a single integrated procedure. I am certain that research findings and<br />
products obtained thus far have implication s for proficiency test development<br />
beyond rhoae that have been utilized. I piopose, therefore, to discuss the<br />
meth:? in some detail and to present some typical end products in the anticfpatfon<br />
that the potentialities of the method will be provocative of fdeae<br />
and will elicit from members of this symposium suggestions for future research<br />
and computer progrerusfng.<br />
Advantages of the Air Force Method<br />
The Air Por-.e method of job analysis has a number of advantages over<br />
traditional methods. The procedure LB simpie, economtcal, and flexible.<br />
It makes feasible the survey of large samples. It fs based on joint responsibility<br />
of job incumbents, test .:ontrol officers and unit coaxnanders. The<br />
job information fs obtained in etandardfzed , qusntffied or readfly quantifiable<br />
form. The information is current and has been found to be highly<br />
reliable.<br />
Job Anal*.rsis Operatfons<br />
The Air Force job analysis procedure involves a sequence of several discreet<br />
steps.<br />
a. Location and procurement of source materials.<br />
b. Construction of first draft of job inventory.<br />
c. Interview review of first draft by technical advisers.<br />
d. Revision of first draft of job inventory.<br />
e. Pield reviev of revised draft by senior incumbents.<br />
f. Construction of operational job inventory.<br />
g. Selection and location of survey sample.<br />
,<br />
149<br />
.
h.<br />
i.<br />
1.<br />
k.<br />
1.<br />
m.<br />
n.<br />
.<br />
Reproduction and mailing to eelected ‘X0’s<br />
Administration of the job inventory<br />
Responding to the job inventory<br />
Receiving, scanning, coding, and collating<br />
Key punching and verifying job inventory data<br />
Electronic computer analysis of frurvey data<br />
Distribution of survey remits<br />
Now let us look briefly at each of these steps.<br />
Source Materials<br />
The source material8 used in the construction of job inventories coneist<br />
of the specialty descriptions in Air Force Hanuals 36-l and 39-1, Job<br />
Training Standards, On-the-Job Training Package Programs, Training Course<br />
Outlines, <strong>Technical</strong> Orders, and any other pertinent publications. A reference<br />
library facility is being built up which provLdes current source<br />
materials pertaining to all airman career fields.<br />
Construction of First Draft<br />
An Air Force job inventory covers tasks performed by all skill levels<br />
of one airman career ladder from spprentice, through journeyman and supervi8or,<br />
to superintendent. Three persons work together In constructing the<br />
first draft of the inventory. A personnel technician or job annlyst select8<br />
duty and task statements from published source nateriale. Upon his judgment<br />
the quality l>f the inventory largely depends. A clerk-typist prepares successive<br />
drofte of the inventory and may derive preliminary task statements<br />
from selected sections of publications. A supervisor editor checks forn;nt,<br />
wording and organization of tasks statements into duty categories and coordinates<br />
the development of related inventories. Construction time for a<br />
job inventory varies with the complexity of the career ladder. For the less<br />
technical ladders, three to four weeks is adequate for writing the first draft.<br />
For the nore technically complex career ladders the period may be twice as<br />
lows -- six to eight weeks -- OP even lor.ger.<br />
Interview Review<br />
Fran three to eix technical advisers who are in the appropriate career<br />
ladder and are usually experienced senior NCO’a, are interviewed individually<br />
or as a group to obtain their constructive critfcism of the first draft of<br />
the inventory. These consultants are frequently the same subject matter<br />
specialiete who arc assigned on TDY to the Personnel Research Laboratory to<br />
build Specialty .Lowledgc Tests.<br />
150<br />
I<br />
_<br />
c<br />
.<br />
.
I<br />
,<br />
. .<br />
.<br />
.<br />
(,. ..> I. “._ .__^ - ,, ..*. ., ./<br />
Bevisicn of First Draft<br />
- -<br />
. .<br />
On the basis of the suggestions and recorenendatl.ons of the technical<br />
adticers the first drraft of the inventory is revised. Teaks which urre<br />
not lisC:d are added, dutfee oad tetzkn not performed nre deleted, and f3properly<br />
worded statements are corrected.<br />
Yield Revlev<br />
- - -<br />
.<br />
UcualSy frc-m 15 to 20 copies of the reviEed draft of the job invntoq<br />
are reproduced In booklet form. Theee booklets are then malled to Test<br />
Control Officers (TCOe) at rrelectcd baecs in different gf?ogPnphiCal arena<br />
la major air commands. The Test Control Officer8 distribute the inventorlea<br />
to senior job incumbents for review. The senior NW’s, like the technical<br />
advioere, are instructed to add duties aad tasks which are not listed, Co<br />
delete tasko not performed in the career ladder, to revise improperly worded<br />
statements, to meke reccoreeadatioas for improving the invcnioq and to return<br />
their suggesttoaa and comments to the Personnel Reeearch Laboratory.<br />
Construction of @peratloaal Job Lnventory<br />
- -<br />
Tasks statements added by senior job incumbents in the field cre extracted<br />
verbatim from the invector; booklets, clnsstffad by type, and grouped<br />
by duty category. After careful consideration and close inspection for overlapping<br />
atatemente, decision 5.6 made regarding the ecccptance or rejectf.oa<br />
of each ndded statement or susested modification. Accepted tcsk statements<br />
are coll.ated with statements Fn the inveatorj under their reepective duty<br />
headings. This second revision coastitutee the first operational form of<br />
the job iaventory.<br />
Sesnple Selection and Location<br />
_I_-<br />
Sample size depends upoa the aumber of incumbents available in the<br />
career lndder being surveyed. Since 2,000 ie the limit of the computer<br />
program capacity, thfe sets the maximum size of the sample. An attempt fe<br />
made to obtain anproximetely 500 incumbents fn each of the four skill levels<br />
in the career ladder. In order to insure having a stctistically adequate<br />
sRnple of each skill level, surveys us~.~elly have not been conducted in any<br />
career ladder where fever thaa 500 airmen are aseigned.<br />
Reproduction and Mailing<br />
TCO addresses end numbers aad locatfone of incumbents in the appropriate<br />
specialties are detenined from manning information suppliee by headquarters<br />
of the several ccmmande. The aumber of inventory booklet0 to be published<br />
may vary from about 600 to about 2,500 to allow for booklets not completed<br />
for one reason or another. Sufficient copies of the inventory for the portion<br />
.<br />
k<br />
;<br />
Ii<br />
:,<br />
)-’<br />
;<br />
of the sample under hi.8 jurivdicttoo are mailed to each participating TM.<br />
Included in the package are administrative directions and other fnotrucrfone<br />
for hendling and returning the booklets.<br />
151’<br />
/<br />
I<br />
I<br />
I
.<br />
.<br />
Administration of the Job Inventory<br />
,<br />
�<br />
Teat Control Officera conduct the group ndministrntioq of the job<br />
inventory in base testing rooms. They scan completed booklets for adherence<br />
to directions and return them to Khe Personnel Research Laboratory.<br />
A tyRica1 job inventory of some 300 task statemcnta require8 about<br />
two houra adminfatration time.<br />
Reeponding to the Job Inventory<br />
Job incumbents in the selected sample complete the inventory by first<br />
supplying certain identification and biographical information. They then<br />
check al! the tasks in the inventory which they perform and write in any<br />
tasks they do which are not listed. Each incumbent’s statements written in<br />
by incumbents during the survey are transcribed, classified by type, and<br />
grouped by duty category. The job inventory in then revised by adding the<br />
acceptable write-in statements. This final revision of the inventory ia<br />
prepared so that a current instrument will be ready whenever a reaurvey is<br />
required.<br />
Key Yunching and Verifying<br />
Upon completion of a survey, incumbents’ responses entered in the<br />
inventory booklets are key punched into electronic data processing cards<br />
and verified. For each incumbent in the sample there is required a “background<br />
information” card, a “position title” card ond several task response<br />
carda, the Ku-rber depending upon the number of tasks in the inventory. One<br />
such taek reaponsa card is required for each 69 tasks In the inventory.<br />
Computer Analysis<br />
And nw we come to the phase of the Air Force job analyefcl procedure<br />
which justifies the “new perepectivea” of my title. It is In the processing<br />
of occupational data by means of the high speed electronic computer that tho<br />
most recent major advance has been made. Computer progr8ma have been written<br />
for the publication of a job description of the work performed by any specified<br />
group of individuals. These groups may be identified in terms of current<br />
skill level, grade, command, time on the job, geographical location, kind of<br />
base, typa of previouo training, or any other variable deeircd.<br />
Routinely, the statistical analysis of the occupatfonal data includes<br />
for each ekill level, apprentice, journeyman, supervisor, and superintendent<br />
cmputatLon of the percent performing each task, Aloo computed are the<br />
average percent time #pent by members of each group who perform the task,<br />
�<br />
152<br />
_ . . __- _-__ ..-. -- .._...._ -. ; _-..<br />
I<br />
.
___ _- .__. _.._-._-. - ._.........I._.___._ _ _<br />
.<br />
.<br />
_<br />
. .<br />
. .<br />
I<br />
. ,. . .<br />
the average percent time spent by all mcrmbers of, the group, both performers<br />
and nonperformers of the task. The cumulative sum of the average percent<br />
tfne spent by all mmbera of the group is also shown so that for any group,<br />
tasks that consume 50 percent, 75 percent, or any other percentage of total<br />
tima can readily be identified. Tasks arc printed out in descending order<br />
of time spent on them.<br />
. Identiffcatlon of Job Types<br />
.<br />
.<br />
,<br />
Perhaps the most important statistfcal breakthrough, however, is the<br />
application, by Bottenberg and ChcLstal,of a hierarchical grouping computer<br />
progrsm,developed by Ward, to occupational data. This progrsm, which represents<br />
a major advancement in the state-of-the-art, groups together incumbents<br />
who perform essentially the same work activities regardless of<br />
skill level, grade, experfence, or assignment. In any career ladder there<br />
are many jobs which for all practical purposes are identfcal. The individuals<br />
who do these identical jobs are oaid to belong to the sane job type.<br />
In the grouping program,the ccmputer locates frm among perhsps 2,000 incumbents<br />
who have completed a job inventory for a particular career ladder,<br />
the two individuals vho have the most similar jobs. The computer does this<br />
by comparing every possible pair of fncumbents,ln the sample. A single job<br />
description for this pair is developed with accounts for their work tfmc<br />
with the least error. The ccxsputer then tests all possibilities of combining<br />
a job description of a third irdividual with the first accepted pair,<br />
or forms a new pair. This process is continued until finally the computer<br />
forss a group consisting of o,l members of the sample and reports the error.<br />
The iterative process may be terminated at any stage in the grouping<br />
program. The stopping point la a matter of jgldgfng when the error term<br />
resulting from merging somewhat dissinflar groups becomes unacceptably large,<br />
In a study involving 836 cases in the Personnel Career Ladder in vhich 35<br />
job types were identifled,the grouping process #as stopped at the 118 group<br />
stage. At this stage there were 27 groups containing five or more members.<br />
One of these groups which was composed of ttu, groups identified earlier in<br />
the program was listed as two separate job types and seven other job types<br />
were generated at later stages in the grouping process.<br />
In some job types work is concentrated upon a few tasks while the work<br />
of other job types is quite diverse. In general, it is found that the number<br />
of tasks performed is directly rclsted to skill level--the higher the level<br />
the more tasks are done. Certain supervisory job types can readily be differentiated<br />
fras technical job types, both in terms of specific tasks performed<br />
and in terms of skill lc*Jels of members forming the groups, Other<br />
job types cut across skill levels.<br />
Purposes of Job Analysis<br />
The Af.r Force method of job analysis has been designed, not for a<br />
speciffc purpose, but rather as R general procedure the results of which<br />
153<br />
.<br />
-, .I ! ’<br />
,]: ,/
.<br />
‘,‘,<br />
.<br />
:<br />
. . ;<br />
.<br />
.,---<br />
-.<br />
can be adapted to many uses. The Personnel Research Laboratory ROV has, or<br />
can develop, programs for producing repcrts to service the needs of many<br />
agencies. In connection with training, these program can be used to validate<br />
training standards, design training courses, deteminc which tasks should be<br />
taught in school and !&ich should be learned on the job, indicate which tnska<br />
should be taught early and which should be postponed, and so on. Proyrmas<br />
can be developed to validate qualitative personnel rcqiirementa information,<br />
to aid in the establishment of specialty qvalificntion requirements, and to<br />
identify the need for new specialLies and ahredouts. Job analysis results<br />
may be used to guide the developmen t of selection and classification testa,<br />
to improve assigcxnent procedures, to determine standards of job performance,<br />
to provide basis for job evaluation, and to contribute to manpower and organirational<br />
analysis.<br />
Job Analysis for Test Develoment<br />
In addition to the purposes outlined, one of the major functions of<br />
the Air Porte method of job analysis is that of providing data for teat<br />
development. Data derived frw computer programs may be used to maximize<br />
the content validity of Specialty Knowledge Teats, and to eatablish better<br />
measures of on-the-job proficiency. Results of many surveys now available<br />
show the percent of members of each skill level performing each task and<br />
tasks performed by the various skfll levels arranged in descending order of<br />
time spent on them. AE the grouping program becmes operational, tasks<br />
performed by various job types in each career ladder will be idectified.<br />
Hew Perspectives<br />
At the present time,vigorous research efforts are being directed toward<br />
improvement in flexibility and capability of the current computer grouping<br />
programs, Other research is devoted to the identffication of significant<br />
task rating factors and to the development of methods for obtaining other<br />
ancilliary job information from incumbents or their supervisors. Many<br />
possible factors are being considered. In studies now under way, the following<br />
task or job rating factors are being investigated:<br />
a. Frequency of task performance.<br />
b. Importance of task ccmpared with other tasks done*<br />
c. <strong>Technical</strong> assistance required.<br />
d. Difficulty of learning to do taek.<br />
e. On-the-job training required to perform task.<br />
.<br />
_.-..<br />
_<br />
.--‘, --- -<br />
_-_ _ _- - -.-.<br />
~
‘.<br />
. . ‘.<br />
.<br />
.<br />
.<br />
.i,<br />
.<br />
-,<br />
‘..<br />
‘,.<br />
‘. .<br />
- -<br />
-.<br />
i I. ,,<br />
i ,*<br />
.\ 1.<br />
.<br />
-_'<br />
1(<br />
.<br />
.<br />
j , _<br />
.<br />
_. * 1 I., ..-. _.<br />
f. Difficulty of learning to do task by OJT.<br />
g. Training emphasis task should have.<br />
h. Time apent in ~pccinl training for fob.<br />
i. Extent to t.hich job give& satisfac:ion.<br />
In Borne surveys the task rating factor used in addition to time spent<br />
is specifically tailored to fit s particular career ladder. Similarly,<br />
other information sought has specific reference to certain special coI>rscs<br />
or kinds of equipment used. In a recent survey of the Administrative Cnreer<br />
Ladder, for example, incumbents were asked to indicate admlnlstrativc ccurBe8<br />
they had attended. They were also required to gfve the number of hours per<br />
week usually spent in typing, and whether a menuai, electric, or both kinds<br />
of typewrlirr were used. Since the Training C-and was interested fn indications<br />
of the words-per-minute rates to which students should be‘ trained,<br />
incumbents were asked to check their typing speed on a six-point scale. The<br />
survey data have not let been analyzed but with an fntidental sample of 105<br />
inventories the following results were shown:<br />
AveraRe typing speed<br />
Under 15 W!?X 3<br />
15 - 24 wF?4 5<br />
25 - 34 WE4 9<br />
35 - 44 WFM 30<br />
.45 ” 54 WFM 36<br />
55 WPM or Over 17<br />
Total 100<br />
If the final survey, when validated, corroborates this am;le, it appeara<br />
that the results have obvious implications for training.<br />
Great advances have been made from the traditional observation, check<br />
list, questionnaire, and interview methods of job analyrris. The feastbility<br />
of the Air Force method has been azmply demonstrated in an operational setting.<br />
Some of the potentialities of the method have been examined but for<br />
the most part, the field of occupational analysis is comparatively unexploted<br />
in the light of modern scientific techniques. However, new peropcctlves have<br />
been revealed and at leant Borne of the problems have been identified and<br />
defined.<br />
155
.<br />
CL%TiS D. l!CYRIDZ<br />
US Army Artillery and Hissile School<br />
Tha US Army i* one of tha large&t producers of achievmant type tQRtR<br />
within tha United States today. During fiscal year 1964 over 1,000 different<br />
nchlsvemmt-type teats W~J produced by the US Amy. Them schievamrrctype<br />
tests mu referred to as KEi Evnluotion Teat8 within ths US Amy.<br />
The US Army fa organized under a decanttslfrcd concept for tha production<br />
of the reqc?rsd touting r=rtcrial becauee it r;ould rtquFre a very lerge<br />
and expensive teat construction rtcff end edmfnintrotivo crtatlirhment if<br />
all the teat materiel was produced at a ceotrclieed point.<br />
The US Amy Enlisted Zvrluation Center, Fort Benjamin liorriaon, Indiana,<br />
6CtO QO ths coordinating agency foi the ?lOS avaluatton testing program<br />
between the varicue rchoolr with the US Amy tie fumioh the r&w test<br />
mtarial and the Office of Perumnel Operfltiona, Department of the Army.<br />
When 8 dacantr~lfred orgtifaation of thie type in used in the productfon<br />
of test mter’ol, aavctsl problem ere created for US A:=y terting<br />
peraonnsl. One of them pioblm mea&!, MO3 Evclulttfon Teat outlimB, i8<br />
the topi.c for our discueolon.<br />
A teat plcn or by US Amy tsminology an MOS Evaluatfon Tcet outline<br />
for a opacific KOS Evaluation lert ia originally conrtructed by test .spacialietr<br />
at the US Army Enliatcd Evaluation Center- Fort Ben jm,fn Harrison,<br />
Indiana. The KCS Evelu*tion Test outline ir forwardad to the pertinent<br />
US Amy Schools for their review a.nd revisions. The US Army SChOO16 normally<br />
rely on porronnsl within the School dapartmanto fcr the teat outline<br />
review and revlofonr . After the test outlinas have been reviemd and rcvised<br />
by the US Amy Schoolo, they are returned to the US Army Enlisted<br />
Evaluation Center where a final revision ie performed, and the teat outiine<br />
ia then used by the US Amy Schoolr US a guide when constructing test Items.<br />
When this rycsten it~ ured in reviewing and revising l4CS Evaluation Teat<br />
outlined, it should be noted that ths review and ravleion have been c~ccmplished<br />
only by teat opecialist sod instructor personnel within the Army<br />
Schoolr. For the most part, no anllrted peraonnsl acturlly ?-*orking in the<br />
job hova been involved.<br />
Some of the foulto found in urfng a procedure of thio type for<br />
constructing KOS Evaluatf-n Tert outlines nrt a8 follovs:<br />
a. Publications uaed by test specialists as a baaia in deciding<br />
the aubjcct-matter � :eao to be included fn a teat outline arc in many<br />
carea not current. For exempla, one of the major publications uatd by<br />
toet opeciallst within the US Aczty ‘*en constructing a teat outline ie A job<br />
156
description pbliehed in AR 611-201. This job description lists all the<br />
skills and knowledge pertaining to a portlcular job (<strong>Military</strong> Occupational<br />
Speci al ty) , In many instances these job descriptions have not been upJnted<br />
for several years. There are many other publications relied upon by the<br />
teat specialist chat are net updated.<br />
b. The test specialist and reviewing and rcvistng personnel uho are<br />
constructing a test outline for a partfcular Hflitarj Cccupnticnai Specialty<br />
have had very lfnlted experience with the job in many cases; therefore, the<br />
test outlines might include areas to be tested that are not signiricant or<br />
it might exclude areas that definitely should be sampled. Lack of experience<br />
with the job also results in poo r weighting of the test outline.<br />
c. Some of the terminology used by the constructors of test outlines,<br />
who have had limited experience in the jobs for which the test outlfnea are<br />
being built, is not stated in terms that are understandable to the item<br />
writers and examinees.<br />
The US Army Artillery and MFasile School recognized some of the inherent<br />
weaknesses of reviewing a test outline through a decentralized organization<br />
and began to plan some means of improving test OuKlfne review<br />
prccedures. There was some thou;\rt of sending questionnaires to enlisted<br />
personnel and asking for suggestions on changes to the test outline. Another<br />
idea entertained was that of makin g visits to some Army nrganlrntions,<br />
talking with enlisted personnel,and asking for suggested improvements to<br />
the test outlines. These ideas seemed unfeasible for various r.,asons. It<br />
was finally ihcided that the US Amy Artillery and Missile School would use<br />
what it calls anHOS Evaluation Test Outline Seminar in an attempt to improve<br />
test outline review procedures.<br />
The underlying idea of the HOS Evaluation Test Outline Seminar is to<br />
gamer the thoughts of test specialists, experienced School instructors,<br />
and ffeld experienced anliated personnel at various skill levels and use<br />
theee thoughts when constructing, reviewing, or revising test outlines, thus<br />
creating a test outline balanced on academic school thought, test specialist<br />
thought, and job expertence thought.<br />
These seminars are still !n the experimental stage, To date, two !-lOS<br />
Evaluation Test Outline Seminars have been held. The success of the last<br />
two seminars has indicated that the US Army Artillery and Hissilc School<br />
will use seminars in the future as a regular part of its test outline review<br />
procedures. The last HOS Evaluation Test Outline Seminar was held on<br />
16 September 1964 at the US Army Artillery and Missile School for the pur-<br />
Tose of reviewing and revising the MOS Evaluation Test outline for HOS 142<br />
(Heavy and Very Heavy Field Artfllery Crewman). Those taking part in the<br />
seminar included 20 enlisted personnel, a test specialist from the Enlisted<br />
Evaluation Center, a senior instructor from the Gunnery Department, and test<br />
specialists from the US Army Artillery and Missile School.<br />
157
When a request was made for the names of enlisted peroonnel who would<br />
attend, the request specified that the individuals should be the “best”<br />
qualified in the 142 MOS and have considerable field experience. The request<br />
also indicated that 011 skill levels within the KG should attend.<br />
The active military service represented by the enlisted men who participated<br />
in the seminar rnnged from 10 to 29 years -- a combined total of 296.5 years,<br />
of which 116.5 years are associated with MOS 142. The active military<br />
service per individual averaged 14.8 years. The senior school instructor<br />
who attended had a total of 22 years’ experience in a field cloaaly related<br />
to the MOS. The test specialist attending had considerable experience fn<br />
the field of test construction.<br />
The individuals attending the seminar were thoroughly briefed on the<br />
role they were to play during the seminar and the procedures to be followed.<br />
Personnel were divided into two working groups with all skill levels represented<br />
in each group. The test specialists acted as monitors for the<br />
working groups.<br />
Each member of the working groups was given a copy of the 142 XOS<br />
Evaluation Test outline as proposed by the Enlisted Evaluation Center minus<br />
the weights. The morning session consisted of a review of the test outline<br />
subject-matter area descriptions with all members of the working groups<br />
discc.seing revisions, changes, or additions to the area descriptions,<br />
This allowed the view points of the enlisted personnel, the test specialist,<br />
and the senior school department instructor to be presented on an informal<br />
basis with the consensus determining what should or should not be included<br />
in the test outline subject-matter nreas. The afternoon session was devoted<br />
to the weighting of the test outlil,2 by the working groupe using the aamt)<br />
general procedures aa already discussed. A critique of the day’s work<br />
closed the MOS Evaluation Teat Outlina Seminar.<br />
After the seminar, the test specialists from the US Army Artillery and<br />
Missile School and the Enlisted Evaluation Center reviewed the results of<br />
the suggested deletions, revisions, and changes indicated by the two working<br />
groups during the Seminar and constructed a finalized test outline that<br />
was based on the combined thoughts of the three major groups involved. The<br />
finalized test outline was sent to the pertinent School departments at the<br />
US Army Artfllery and Missile School for their review and comments and then<br />
for-warded to the US Army Enlisted Evaluation Center for final review,<br />
An MCS Evaluatfon Teat outline that has been processed through an HOS<br />
Evaluation Test Outline Seminar results in a test outline that:<br />
a. Contains subject-matter areas significant to the job for which<br />
the examinee is being tested,<br />
158<br />
_-.-----__
. .<br />
b. Contains terminology that should be clear to anyone within the HOS<br />
because the terminology has been vrLtten in terms which has net the satisfaction<br />
of the individuals taktng part in the seminar.<br />
c. Contains properly weighted subject-matter areas because it represents<br />
the combined thoughts of the various types of groups &to were involved<br />
in the seminar.<br />
The main otrength of the HOS Evaluation Test Outline Seminar lies in<br />
its ability to create a test outline that contains a balance of thought<br />
among on-the-job experienced eniisted personnel, experienced school fnstructor<br />
personnel, and test specialists.<br />
The US Army Artillery and Hissile School feels that the HOS Evaluation<br />
Test Outline Seminar will develcp into a valuable tool to be used when<br />
constructing, revising, or relieving test outlines fn the future and it<br />
will provide a vsluable input on which to make recommendations to revise<br />
the NOS job descriptions and other related training publications.<br />
159
Item Writing Procedures for IncrasLng Validity<br />
I. J. WEwEilw, Chairman<br />
6570th Personnel Reeearch Lboratory, US Air Force<br />
The chairman introducad the subject by otatfng that the starting<br />
point for increasing the validity of item8 i6 in the fob analysis and<br />
* the outline. The firrt thing the vrftar must know i8 what he ir eup-<br />
’ I pored to be measuring. If he 10 measuring knowledge, one type of item<br />
.’ 18 required, and if he is mearurtng job akilla, another typa Is called<br />
for.<br />
*,<br />
The particlpantr brought out the vaafour problema they have in thla<br />
area dut to tha vsrfatfonr found in each of the oervlcer approach to<br />
thir problan. When the dircu@eion developed the great interest the<br />
earvices have in a proper job dercripcfon oud job analyaie to help rolvo<br />
thir problem, the choirman reminded the particlpnntc of the work being<br />
done in thfa area by the USAF. He tcuched on thio very briefly,<br />
reminding the group that a fuller triutnent was on the program et a<br />
later reseion.<br />
There aaaned to be agreement that the job analyoie and outYinr<br />
were the foundation of a eood teut item, the problem vipe in getting<br />
this enalysir .<br />
.<br />
160<br />
, .<br />
.<br />
.
\.’<br />
( ,<br />
r<br />
i<br />
I<br />
I<br />
I<br />
I<br />
. .<br />
Non-Fbpfrical Validation of Teat Itms<br />
The folloving paper conctituted n hando\:t. The groop dfncueoed<br />
the pofnte mder la, b, c, and d; nnd 2a. 0, and c on page 4 end 5 of<br />
the peper. No decisive conclusions were darivcd by the group dU@<br />
to<br />
time ?Lmitatione.<br />
The diocussion of the problem preorntsd in this paper shou Id be<br />
fiirther clarbflad, specific e?Xr%QieS furnf.shad, and then ehould ?Je<br />
prarented to the PfYX en u work RmLnar tit the next meting,<br />
161<br />
.;<br />
? !
-<br />
Preoentation to MTA Panel ConaLdering<br />
Non-EmpF;ical Velidatfan of Teet Items<br />
The fact that validity ia many-faced and difficult to determine<br />
does not lessen the importance of the problem. The test produced by<br />
all services here are of tremendous faport. (The results of theoe tears<br />
determine which enlisted personne? get additional pay for isuparior performence,<br />
and help determine which get promoted to the newt higher pay<br />
grade.) Thus, tha ccoaomic and leadership status and quality of the<br />
enlisted personnel of our Armed Forcer are effected etrongly by 0;1r te8tm.<br />
As e result, the morals and quality of our fighting forces ere directly<br />
Affected by the military testing program--our teeto.<br />
It ie our duty to aek ourselves, “How valid are our tertr? Do our<br />
teat identify the best informed and mst nkilled personnel in the euma<br />
compet~t~va occupa~el area? Do thcytruly FdentFfy the least lnfowed<br />
end lee8 rkilled?” Becauee these evaiuatfon testa do affect tha letdership<br />
and morale of the enlleted structure of our nation’s ff.ghttng forcec,<br />
we are interested fn determtnfng and fmprovfng the validity of our itema,<br />
and thus of our testa.<br />
The California Te8t Bureau has cotegorited the two bAsliC spproachrs<br />
to the detcrmlnAtion of validity according to the chr.;t on page 201.<br />
Thorndike end Hsgen (1961) list tha specific considerationa entering<br />
into evaluationa of test8 as (1) validity, (2) rellabFLlty, and (3) practfciality.<br />
Validity refers to the extent to which t test meaeuree what it<br />
18 intended to meascru. Reliability has to do with the accuracy and precision<br />
of a measurement procedure --conoirtency and reproducibility. Practicelity<br />
in concerned with Marty factora, ruch AP economy, convenience,<br />
and interpretabilfty--whfch determine vhether a teixt is practical for<br />
widespread use.<br />
A reading achievement tert requires people to select certain answer@<br />
to quertlonr. Penci.1 marka on enrwer rheeta determlqs each peraon’r<br />
more. Thio Gcore ir called their reading comprehenrlve #core, but the<br />
Icore fs NOT the compreheneion. It io A rfecord of 8ample~behavior.<br />
Any judgment regarding compreheorion ir an inference from thfc number<br />
of allegedly correct anowers. Its vs:fdity in not self-evident but rnuet<br />
be astsblfshed on the basis of BdequAta evidence.<br />
Thorndike and Hagen (1961) #Late, “A teet may be thought of ao cxraapondlng<br />
to some aspect of human bahavior In three sensea.” To identify<br />
these categories they use the terms (1) “reprerent,” (2) “predict,” and<br />
(3) “sfgnify.” Of these three type6 of evidences of validity, our topic<br />
ir prfnci+lly concerned with the fLrst type, KO wit: How well do our<br />
tests reprerment the standard8 (the practical derfred level of performance)<br />
.<br />
162<br />
_ _ .--
--<br />
. .a.,_..<br />
* .<br />
, /<br />
of the occupation or job? Xn proportton a6 the level of job perfornanco<br />
(achievemant of job standards) are accurately represented in the test,<br />
the test is valid. The proceso‘of obtaining thio proportfon of job<br />
performance to test contant is aesentfally a rational or judgmental<br />
one. This 8nalyofa 10 aometLmea spoksn of as rationai or 1ogLcal validity.<br />
Since the analyefs fr largely in terms of the content of the test, the<br />
term contenr validity fs sometimes used.<br />
We should not think of content too narrowly, because we arc Fnterestod<br />
in process as much aa fn atipla content. Thuo, in the test of a<br />
mechanic ve are concerned vl:h euch “content” elements a6 forms for<br />
requioftioning materfal8, of what materials rivets or bolts should be<br />
constructed, rules of safety, priwiples of expanrfon and contraction<br />
and remote reading gages. WC &r-e also interested in such “proceoa”<br />
eklllr as troubleohootFng and correcting defects of equipment, the<br />
ability to solve work aerignment problems, the ability to organize a<br />
repair crew for a particular job, and the use of correct procedurea in<br />
solvfng many other specific job proglanr.<br />
Thus, I submit the therlr that the problan of appraising content<br />
validity of a test starts vith the detailed listfng of the standards of<br />
job performance, and includes rucceo~lvaly ranking the content of the<br />
job standards, and canparing thl..a vith the blueprint of the test. After<br />
thFe task Lu completed, the item mu6t be compared vfth the test outline<br />
and thus vith the job rtandardu.<br />
The extent to vhfch the test outline areas (and thus the ‘CL standards)<br />
arc tapreoan: td in the teat 10 a crueLa fndicator of the validity of the<br />
te6t.<br />
Rovaver , ve cannot ignore predictfve and concurrent valfdity. Before<br />
ve consider techniques or methods of obtatning content validity of tests,<br />
it ir noteworthy to conbider the empirical method of valibting te(ltl),<br />
which Thorndike and Wagen (1961) clasrLfy a6 predfctive validity.<br />
As Thorndike and Hagen (1961) state, and I quote them verbatim, I’...<br />
predlctlve validity can be estimated by determining the correlation betvecn<br />
test scores and a euitablc criterion measure of euccesu on the job. The<br />
joker here ia the phraoe ‘suitable criterion maasure’....One of the moat<br />
difficult problems that the pcruonnel psychologist or educator faces is<br />
that of locating or creating a satisfactory mtaaure of job aucceae to<br />
eervc as � criterion mearure for test valLdation....All criterion measures<br />
are only partial fn that they measure only a part of success on the job,<br />
or only prelfminaries to actual job performances . . .an ultLmate criterion<br />
ir fnaccerrfble to us . ..and substitutes (intermediate criteria) are only<br />
partial and are never completely satisfactory . ..The problems of effectiva<br />
ratfng of paro3nnel arc dfscuasc,! in detafl in Chapter 13. It suffices<br />
to indfcate here that ratings are often unstable and are influenced by<br />
.<br />
163<br />
_ --.---- 3 --
--<br />
--<br />
.<br />
_ . _-._ ._- --- ..--._. - ._.<br />
many factor8 othsr than the proficiency of perronnarl being rated,” The<br />
authors Ilrt mwtn genaral limitations of rating proceduras (pp. 383-384).<br />
Thue, SU& retlnga should salden, if aver, be unod ee the sole or determining<br />
evidence of validity. For example, both logical enalyalo and<br />
raoaarch on ratinga indicate that peara end cuparvisors tend to rnte<br />
eomewhet different qualities end to weight these differently even when<br />
given exactly the same acelco end dafinltlonr.<br />
Whet I em proposing la related to an urea of ccnzmunicatfono rtrcarch<br />
celled “content enelyele.” In his book -Content - Analyaie fn Cmunicmtiona<br />
Rcoesrch, Dr. Bernard Birdson defines content enelyels as?‘...6 reoearch<br />
technique for the objective end quantitative description of the taenifert<br />
content of cmraunfcetlon.” Of courle, the test ltaa le the coazuunlcetion<br />
whooe menlfcet content interest8 ue.<br />
BcfDre proceeding further, perhaps we ten avoid quibbling on<br />
maentlc differencer by agrtclng to UIC English end Englioh (195B)<br />
deflnltlone of certain tat-ma bselc to our dfrcuesion.<br />
Dtflnftlone<br />
.I. Oblectlvt of test: To objectively neeeure the degree to which<br />
a person has mertered eJI1 tlglentr of a epecific occupetion (the vo:d<br />
degree impliea separating workere on the basis of dtffertng ebilftlte.)<br />
2. Job Kaatery: Such pr?fCcleney ln e epecfffc occupation thet<br />
certain definad standards of eccomplfshmsnt can be met perfectly (Englleh<br />
end English, 1958).<br />
3. Discrfmlnetfon: The proceos of detecting dfffcrencte (our teete<br />
must be able to measure differences in degree of job mestary emong pcopl~<br />
working ln the euae occupation).<br />
4. Standard: That which is expected; a practical, desirable level<br />
of ptrformsnce (Englleh and Engllrh, 1958).<br />
5. Job Standard: Army Regulation AR 611-2Gl (1961), pegs 6, atetee<br />
II .** d. Fzth digit . . ..Thara aklll level designations indfcate the level<br />
of proficiency required in e epeclflc job end the corresponding quellficetlon<br />
of an lndivlduel... (With the exception of HGS Code 718) NAVPERS<br />
18068 (1958) lieto the job � tendardo of Navy enlleted occupation groupr.<br />
Air Force Kenuel 35-l (1957) lleto the job standard8 for the Air Force<br />
enlirted occupatione. Slmtlar regulations list job etendardr for the Royal<br />
Cenadien Navy, the US Coert Cuerd, the Herchent Heri.ne Academy, and the<br />
tfarlne Corpr .<br />
.<br />
.<br />
164<br />
--
,<br />
‘ .<br />
. .<br />
_ -. . .<br />
For other deffnitlon&, English and Englierh (195Y), fn available<br />
for reference if neeP be during thir diecuosion.<br />
A teat itm la only valfd to the degrees that it represcntrc the<br />
job standard it la meant to rample and to thfa dqrea it contributes<br />
to the validity of the teat aa a whole.<br />
In order to guide ua In makin g useful prozrsoo Fn the ohort time<br />
available, I have llated major rteps which should help UQI to work orlt<br />
an acceptable proceao for logical validation of teat outlinen and tent<br />
items.<br />
1. For logical validation of test outlince vc muat accomp’iiah<br />
the following:<br />
a. Datamine what conrtitutto the panel of exports.<br />
b. Determine the criteria which the panel of experta plhould<br />
use in ranking element8 of job standards.<br />
c, Detel-mina the methode to usa in correlating the test outline<br />
with its aealgned weights -a8 it -axiete - with the rank order of job<br />
standardr arrived at by the panel of experto.<br />
d. What criteria, if any, ohould propored changee to teat<br />
outliner be required to meet before they are incorporated into cxirtfnp,<br />
validated teet outliner?<br />
2. For logical validation of teat Itemme, we uuat accomptfrh the<br />
following:<br />
a. Detennfne what conrtftuter a panel of experts. (Should<br />
they be the same peroonnel who validate job rtdndards?)<br />
b. Determine the criteria vhich the panel of experts ehould<br />
use in ranking teat Itema.<br />
c. Determine the methods to use in correlating teat ftmn<br />
vith the following:<br />
(1) Original tc6t outline 4rerc.<br />
(2) Rank order cf fob standards (tert outline) obtained<br />
fran the panel of experts.<br />
165<br />
____ _________ _ __.. --.___ ,_.___. __--- --<br />
I<br />
I<br />
I .
.-<br />
We probably ohould not conc1~rn ourcralvea here ~Lth tha tcchnlcnl<br />
accuracy of itakka in 43 test, EB rhie point rel.acwi to content validity<br />
uauafly fe validated by chockFug tha publication uaeci Fn raforencing<br />
the LeQm.<br />
A point which ahould bn coneidarsd ia that such rtrmrch BI has<br />
been conducLed hm failad to ffnd my eigntffcmt diffarauce In validity<br />
of ftcme ragardllaeo of uhothsr they conforra precFesPy to “Ltesn construction<br />
prfncfplar” or not.<br />
1. Fraquency of perfaming tho task: Vow hportant ie thio<br />
factor? What is tha Polorionehip of frequency, or routina, of job took<br />
to ehe crdtfcalit~ of ths tark?<br />
2. Criticality of performing the task: What fa the rerult if<br />
the tank ir HOT parfowed? if the task io ?JOT perfowod correctly?<br />
rafaly? Will man be killed? aquipmant ruined? miasfon fafl to be accompliohad?,<br />
etc.<br />
3. KncwZedgs ssaential to perform the feek.<br />
4. PTOC~YI, or use, of the knowladge to perform eha raako,<br />
5. Hew well do the problems in thirr ~1rm discriminate between<br />
anlisted perrono having high, svaregc,, and low ?svaln of job curtaty?<br />
Z’ha ocalar attached ara rugeasted for your use in l.le~lng auccfxtIy<br />
tha oseential criteria axparr judgea ohould u5?1 in cetegoriting job treks<br />
and teat itema atz one and of ehe scalao or in the middle. Scratch ; aper<br />
10 ateachod aloo for your coiweniance.<br />
E: Savaral copieo of the content validation etudy being conducted<br />
by Major David Culclasurs, MS Evaluation Teat Project Director, US Arq<br />
Medical Field Service School, Fort Smn Houston, TBXEO, ara availatlo to<br />
panel mexnbare for information and uce for suggaotloar do to procedurar.<br />
We invite Major Culclaaure to ditcues hia study with uil at thle point.<br />
166<br />
.
.<br />
-----<br />
We probably ahould not conc~~tn ourselvGa here with tho technical<br />
accuracy of item8 fn a test , us this pofnt ra?ot%d to content validity<br />
urually ir validated by checking :ha publication used In raferenclrq<br />
the It=.<br />
A point which ohould be considered ir that much raeearch aa hee<br />
bsen conducted harr foilad to fIn.3 sny oignificant difference In validity<br />
of ittwo ragardleor of whether they conform pracfaaly to “ita construction<br />
prfnciplae” or not,<br />
1. Frequency of parformIng the task: Ho3 inpsrtmt is thio<br />
factor? Whet Ia the relationehip of frequency, OT rcuclne, of job ttask<br />
to the criticality of the taok?<br />
2. Criticality of perfoming the task: What Is the rasolt if<br />
the Lark Ir NOT performed? If the task IO NOT parfamtd corrsctlyt<br />
� afaly? Wfll men be kIllad? squipmant ruined? miosion fail to ba accomplishcd?,<br />
etc.<br />
3. Knowladga eratntlal to parfom tha tack.<br />
4. Proceea, or UIQ, of the knowledga to perfom the taoks.<br />
5. How wall do the problems In this ares dIocrLmZnate between<br />
-lIlted pereons having high, average, and low levalrr of fob rnnetery?<br />
The rcnlar attached ara suggested for your uoa In listing ruccfnctly<br />
the essential criteria oxpert jud305 should use in categorialng job tmrko<br />
and toet Items at onr end of tha scales or In tha niddle. Scratch pager<br />
I8 attached alro for your convenience,<br />
NOTE : Several copier of the content valIdrt?an otudy being conducted<br />
by MaFDovtd Culclot~rti, MOS Ev&luatIon Test Project Director, US Amy<br />
Medical PIeld Service School, Fort Smn Rouoton, TEXAS, arc avaflatle to<br />
penal Iombarr for Infomatton and use for suggertions aa to procadureo.<br />
We fnvfte Hajor Culclavure to diecuos hia etudy with ua at thic point.<br />
166<br />
,<br />
c
. .<br />
.<br />
LOXCAL EKPIRICAL<br />
Content Valldfty<br />
- -<br />
Refers to how vcll the content of<br />
the tent samples the subject matter or<br />
� ituatioo about which conclusions are to<br />
ba dravn. Content Valid& is especially<br />
important Ln an schicvenent teat. Rxamplea<br />
: testbook analysis, description<br />
of the universe of itena. rdoquacy of<br />
the eI[Icple, represanta~:ivencss of the<br />
teat content, intarcorr?lationa of<br />
subocorcs, opinions of jury of experts.<br />
1tcm structure<br />
Includes (1) Corroborative evidence<br />
from item analysis rupporting the other<br />
characteristics of the teat; i.e.. interrclntionahips<br />
between items, bctvecn items<br />
and scores, and between items and criteria;<br />
(2) Item ccmposftion. For graphic items,<br />
it emphasizes perceptual clarity and rclated<br />
format functions. For verbal itms,<br />
i< mphasircs conceptual clarity in the<br />
cqra8oion of itema. For both graphic and<br />
verbal items), it Emphasizes functions of<br />
dietractors.<br />
I<br />
Construct Validtty<br />
I<br />
Concarns the psychological qt.-lities e teat<br />
mcaaurco. By both logical md empirical mcthodr<br />
the theory und’erlying the test is validated.<br />
Exmpleo: Corrciatims o f the teat scorea, factor<br />
malysir, uoc of inventories. studying rhc effect<br />
of speed on test scores.<br />
I<br />
A<br />
<<br />
. $, v<br />
-Cr-..-..~..--._-,---<br />
Predictive Validity<br />
Relates to hw veil predictions u.ade<br />
rfrom the test are confitued by data collactcd<br />
at a later time. -loo‘? Correlations<br />
of intelligence test8coras with<br />
cwrfe gradeo, corrclatioo of teat acores<br />
obtained at beginning of year with msrka<br />
earned at the md of the yaLr.<br />
Concurrent Vafidlty<br />
Refcra to hov veil teat mores natch<br />
mcasuree of contcmporsry criterion porfomance.<br />
Examplea: Cowparing of scores<br />
for men in an occupation vith these for<br />
men-in-general, correlation of pcraonaPity<br />
test scores with eotiwtcs of adjustments<br />
made in the mmnrcling interviews, correlation<br />
of end-of-course achieveac:rt or<br />
ability test scores with school okra.<br />
.<br />
\<br />
___~__^<br />
1<br />
I<br />
,,* - _,. . - - * -.<br />
.<br />
.
I<br />
Y--L.<br />
.<br />
.-<br />
-. *<br />
-- ,<<br />
.<br />
_.<br />
:<br />
.<br />
-<br />
: ,-<br />
\..L<br />
. -<br />
-=Yp<br />
,<br />
. .<br />
._ -*<br />
- .-<br />
L<br />
.C<br />
-._<br />
.
,<br />
__-____ 14_1- -...-. -- _.._.~<br />
_..<br />
..___ -..~ .- _.._ -___-<br />
___C__-..--<br />
169
,<br />
c<br />
References<br />
Birdaon, B. Content analyeir in coimunication~ reaearch.<br />
- - -<br />
Illfnoff3: Free Pfesn, 195X<br />
Glencoe,<br />
Californfa Teat Bureau. p- gloeeary of meaeumment tax-ma. Monterey,<br />
California.<br />
Chief, Bureau of Nuvol Peroonnel, US Navy, NAVPERS 18061. Washington,<br />
D. C., 1958.<br />
English, H. B., b English, A. C. Dictionary of psychological and<br />
psychoanalytical terms. New York: LongGiG & Company, 193K<br />
Headquarters, Department of the Air Force. Air Force Manual 35-l.. 1957.<br />
Headquarters, Department of the At-my. AR 611-201, Washington, D. C., 1961.<br />
horndike, K. L. & Hagen, Elfrabtth. Meaaurcnant and evaluation in<br />
psychology and education.<br />
Sons, Inc. ,196l.<br />
(2d ed.). New YorrJohn Wiley &-<br />
,<br />
170<br />
.
,<br />
.- -<br />
L<br />
L<br />
.<br />
-.<br />
.--Ly<br />
-_<br />
_. -<br />
. .<br />
\.<br />
\<br />
. .<br />
. j<br />
‘r<br />
:\<br />
\ I ;<br />
.<br />
. .<br />
Teet and Item Revision Technique8<br />
J. E . PARTTHCXOH , Chairmsn<br />
Teat And item rovisdom Are neceenuxy for A vcnriety of ressone.<br />
‘Zrc main reason for euch revisione ie to bring About improvement in<br />
measuring inotrumants. A test muat be revised if the job requiramcnte<br />
are changed And if it la not functioning properly AA A mcssurfng inatrumcnt,<br />
Among the pany raaAonA for reviaing itema Arc: obsoleaonca, too<br />
easy, too difficult, do not discriminate bctveen those who have HIsatered<br />
the job and those who have not, lack of validity, not job oriented,<br />
require rote memory to ancrwar.<br />
Techniques for Item Revision<br />
Statistical data, particularly the item anelyaio capl Aerve ca a tool<br />
or guide for indicating area8 vithin iteazre which Ara in need of revieion.<br />
It ie neceesAry for the teat psychologiet And the subject-matter expert<br />
to work together in analytfng And literally “taking apArt” the test item<br />
which According to tha statieticel Analyale hea not functioned in the<br />
way that it rhould. Evan though tha subject-matter expert May knc~a hio<br />
Aubjact,ha rosy not be Able to analyze the poorly-functioning item And<br />
bring about changes in it. it iA usually helpful for the teAt peychologist<br />
to ark the eubject-mattsr expert to deacriba in detail the protees<br />
or actions through vhfch tha eraminae must go to Bnevor tho particular<br />
Item. This should bring into clear focus for tha rubjact-mattcr<br />
axpart tha Apacific raquiratants on the part of tha exeminaa vhan he<br />
is facad with A problem which the Item presenta. When thio Approach<br />
iA tAkan to item revieion, it is oeu~lly a~sy for the eubjectwttar<br />
axpart to caA ths reAAon for tha poor functioning of the item.<br />
The following Actione or consideration of the following fsctorr<br />
rhould ba helpful in the revieion of test itatnr.<br />
A. ItemA rhould ba 80 written AA to make thafr dirtroctorr AA<br />
AttrACtiVa AO poasibla. . .<br />
b. Look for undatccted Ambiguity in fttxn diotractoro which<br />
mirlsad sxcrPrFnaea. Itaa Atems May also be ~mbiguour.<br />
c. Eliminate nonfunctioning dietractors.<br />
d. UIQ hints provided by the dAtA concorning mentA1 procarrso<br />
of axzunlnaes.<br />
I ---.---_.----___ ._ . . --. ---__<br />
171
‘7-.<br />
* -.<br />
,<br />
.<br />
--<br />
.<br />
_ ._<br />
e. If a diatractor ia discriminating in the wrong direction it<br />
may rapreoent an aspect th8.t cannot be r8movod without d88troying the<br />
whole point of the item. The iccm nny be covering the point about which<br />
there fo much mietnformntion; the dintractor should not be revlaed if<br />
the point of the item ie loat through ravieion.<br />
The following quotation from Devie’ ch8ptar on “Item Selection<br />
Techniques” in Lindquist’e Educational Measurement should be helpful<br />
when revieion technique8 8re applied to testa end item8. .-<br />
“It is interesting that mny invalid dietracters are found in items<br />
that how been carefully edited and checked by subject-matter urperts.<br />
Thi8 anph88ixes the well-kno-m fact thnt because a dfatracter ie dircriminative<br />
in the wrong direction we cannot conclude that it io too<br />
nearly correct from a factual point of view. Convereely, the fact that<br />
an incorrect choice ie too nearly a correct anawer doe8 not nacasaarily<br />
mean that it will turn out to be diocriminative in the wrong diraction<br />
when it ir subjected to item mnlyoie. Item analyois technique8 cunnot<br />
alone be relied upon to detect error8 and 8mbiguitieo; expert crfticiw<br />
and editing ore indiepenaable in te8t construction, The full value of<br />
item analyrie technique8 cannot be reelircd unlers criticiomo of the<br />
item8 by recognited authoritier are available for reference.”<br />
Referance<br />
Davio, P. B., Item selection technique8 in Lfndquiot, E. F. (ed.),<br />
Educational Measurement. Uarhlngton, D. C.: American Council<br />
on Educcltionx951, 308.<br />
.<br />
.<br />
\<br />
,<br />
172
._“...-<br />
.<br />
,<br />
._ I . . ..,_ . _, _ . . .-<br />
.<br />
__ .I ^., bL<br />
PAPEA<br />
Answers to Common Criticisms of Tests<br />
FRANK H. PR-ICE, JR.<br />
US Army Enlisted Evaluation Center<br />
It is appropriate at s conference of this type that we constder sane<br />
of the common criticisms of tests. This consideration is especially fitting<br />
since the mass testing movement was spawned as a result of the rush<br />
to military prcperdncss for World War I.<br />
The criticisms of testing come from many sources both within and<br />
without professional psychology. Generally, the psychologist critics are<br />
constructive while the lay critics are destructive. This morning we are<br />
concerned with these lay critics who have protested in the popular press<br />
and books such as Hoffmann’s The Tryanny of <strong>Testing</strong>, Harrington’s Life in<br />
the Crystnl Palace, Gross’s The Brain Watches, and Whyte’s Crgnnfzation<br />
&, indicating public distrust, uneasiness, and ignorance about which<br />
we mus’t be concerned,<br />
Most of our critics have leveled their blncts at so c6lled educational<br />
and industrial tests. Very seldom have our military testing programs been<br />
the direct victims cr’ such scath:ng attacks; but, just because we have not<br />
been the subjects a; eloquently set forth pronouncements does not mean<br />
that we are not discussed and “cussed” in the dayrooms and barracks of<br />
those we test and even in the offices and headquarters of those for whom<br />
we test.<br />
My remarks this morning will not be limited to testing in the military<br />
setting. Almost all of the recent protests against psychological<br />
testing are as applicable, if not more so, to the testing of military<br />
personnel as to the testing of school children, college students, or<br />
Industrial applicants and employees. And I hope that you will be able<br />
to apply this discussion of some of the criticisms in terms of the<br />
particular problems and Interests of your military programs.<br />
In this vein, I will devote particular attention to The Tyranny<br />
<strong>Testing</strong> (1962) primarily because Hoffmann appears to be more sophisticated<br />
atld-devious, but slightly less venemous than other of our critfcs. To<br />
give credit where credit is due, a symposium on this subject presented<br />
by Owens, Astin, Dunnette, and Albright at the 1963 meetings 4 t’,e Midwestern<br />
Psychological <strong>Association</strong> has provided much of the source material<br />
for this paper.<br />
First, let us examine some of the major assumptions found in<br />
Hoffmann’s detailed indictment of the multiple-choice test. He (p. 150)<br />
states, and here he is talking about multiple-choice tests -- the same<br />
type we construct and admfnfster -- “The tests deny the creative person<br />
a significant opportunity to.demonstrate his creativity and favor the<br />
shrewd and facile candidate over the one who has sometning to say.” The<br />
173<br />
, I<br />
_ . _ . _
.-<br />
/ .‘4 ‘._.<br />
.<br />
- --<br />
.<br />
problem with such an ammption is that there lo virtually no sotieftctory<br />
research defining creativeness. Usually the people who sccre high. on so<br />
called “creativityI’ tests are aimply the ones we call creative. In fact,<br />
there is considerable evidence to indicate that creativity teal& arc not<br />
actually measuring creativity as a personality trait (Thorndike, 1963).<br />
There is no evidence from carefully conducted and logically interpreted<br />
research to indicate that objective tests stifle the creative person.<br />
Roffmsnn’s charge is what he thinks should be fact rather than research<br />
data. In other words, Roffmann has the idea that merely because multiplechoice<br />
tests are highly structured, the examinee has nc opportunity tc<br />
expr.?ss himself. Nothing could be further from the truth, but the degree<br />
to which the examinee can express his knowledge depends on the skill and<br />
the data of the test writer.<br />
In his second assumption, Roffmann states that multiple choice tests,<br />
,I . ..penalize the candidate who perceives subtle points unnoticed by less<br />
able people including the test makers. They are apt to be superficial<br />
and intellectually dishonest with questions made ortifically difficult<br />
by means of ambiguity becnuse genuinely searching questions did not<br />
readily fit into the multiple choice format.” In this assmption the<br />
grent amount of careful research actually going into the construction<br />
and validation of a teat item is completely ignored. Naturally, dierractors<br />
are written purposely to “fool” the less knovledgoa’ble examinee.<br />
information about the responses to items m*de by persons of different<br />
levels of knowledge indicates without n doubt that the degree of ambiguity<br />
perceived by an examinee is inversely related to his knowLedge of the<br />
subject matter, This simply menns that the less one knows the more<br />
ambiguous the question appears. Yet, Hoffmann state6 (p. 67), The more<br />
one knows about the subject the uarc glaring the ambiguities become.”<br />
Of course Roffmann doea not support his assumption with evidence;<br />
nevertheless, this charge is the one with which we are moat often hit.<br />
He saye that the most serious consequence of test ambiguity is that it<br />
penalizes the gifted and talented examinee. In Hoffmann’ view, how does<br />
this discrimination occur? When first confronted vith the alternative<br />
answera to a question, the “deep” examinee, as Hoffmann calls the gifted<br />
and talented, analyzes the alternatives more carefully than does the<br />
*‘sup’ -ficial” exsminee. Naturally, such careful scrutinizing takes tfme,<br />
and the first penalty occurs.<br />
Secondly, the “deep” student is much more<br />
likely to perceive the ambiguities and, as a result, spends more time<br />
trying to determine exactly what the test author had in mind. Furthermore,<br />
according to Roffmann, the “deep” examinee’s motlvetion to perform<br />
well tends to be reduced as he sees more clearly the superficiality and<br />
ineptness of the test writer’s approach. Even more damaging, the gifted<br />
examinee is more likely to discover a “better” al.ternative than the keyed<br />
response.<br />
174<br />
.
c r-<br />
’<br />
,<br />
L-<br />
.’ ./’<br />
/ :‘..<br />
problem with such an assumption is that there i.8 virtually no satiafsctory<br />
research defining creativeness. Usually the peopie GAO Gcore high on co<br />
call.ed “creativity” test6 arc simply the one0 we call creative. In fact,<br />
there is considerable evidence to indicate that creativity testd are not<br />
actually measuring creativity as a pereonality trait (Thorndike, 1463).<br />
There is no evidence from carefully conducted and logically interpreted<br />
research to indicate that objective teats stifle the creative person.<br />
Hoffmann’s charge ie what he thinks should be fact rather than research<br />
data, In other words, Eoffnann ha8 the idea that merely because multiplechoice<br />
tests are highly structured, the exminee has no opportunity to<br />
expr.?ss himself. Nothing could be further fros the truth, but the degree<br />
to h%ich the examinee ten express his knowledge depends on the skill and<br />
the data of the test writer.<br />
In his second assumption, Hoffmann states that multiple choice tests,<br />
1, . ..penalize the candidate uho perceives subtle points unnoticed by less<br />
able people including the test makers. They are apt to be superficial<br />
and intellectually dishonest with questions made artifically difficult<br />
by means of ambiguity because genuinely searching questions did not<br />
readily fit into the rmrltiple choice format.” In this assumption the<br />
great amount of careful research actually going into the construction<br />
and validation of a teet item is cmpletely ignored. Natur.tlly, distr8ctor6<br />
are written purposely to “fool” the less knowledgeable examil.se.<br />
Information about the responses to iterns made by persons of different<br />
levels of knowledge indicates wlthout a doubt that the degree of ambiguity<br />
perceived by an exuainee is inversely related to his knowledge of the<br />
subject matter. ‘fiftl simply means that the less one knows the more<br />
ambiguoue the question appears. Yet, HoffmaM states (p. 67), “The more<br />
one knows about the subject the rare glaring the a;tilguitice become.”<br />
Qf course Hoffmann dccs not support hie assumption vith evidence;<br />
nevertheless, this charge is the one wit.h which we are most often hit.<br />
He says that the most serious consequence of test ambiguity is that it<br />
penalizes the gifted and talented examinee. In Hoffmann’s view, how does<br />
this discrimination occur? When first confronted with the alternative<br />
answers to a question, the “deep” examinee, as Hoffrrann calls the gifted<br />
and talented, analyzes the alternatives more carefu2ly than does the<br />
l’superficialt’ exnminee. Naturally, such careful scrutinizing takes time,<br />
and :-Be first penalty occurs, Secondly, the “deep” student is much more<br />
likely to perceive the rrmbiguitlee and, as a result, spends more time<br />
trying to determine exactly what the teat author had in mind. Furthermore,<br />
according to Hoffmann, the “deep” examince’s motivation to perform<br />
well tends to be reduced as he sees more clearly the superficiality and<br />
ineptness of the test writer’s approach. Even nore damaging, the gifted<br />
examinee is more likely to discover a “better” alternative than the keyed<br />
reeponse.<br />
%<br />
174<br />
.<br />
-____-- - _.<br />
.<br />
.<br />
.
f<br />
Ih<br />
i<br />
i<br />
Hoffmann attempts to document his reasoning -- and it in mere11<br />
U’arm-chairing” -- that multipie-choice test qucctitions are by neture<br />
ambiguous by citing Pample items from test manuals or more frequently<br />
by attacking illustrative items of his 0.m. A Hoffmann-type item C.W<br />
quickly illustrate the kind of rumination which forms the main content<br />
of his attack,<br />
i<br />
.<br />
I .<br />
Watt are the colors of the American flag?”<br />
(A) red, white, and blue<br />
i<br />
c<br />
, 0)<br />
((3<br />
say<br />
neither A nor B<br />
i The superficial examinee quickly selects answer “A,” red, white, and blue,<br />
and goes to the next question. The “deep” examinee begins to scrutinize<br />
and analyze the alternatives; he thinks, “A” is correct under some<br />
conditions, but “B,” gray, Is correct under some conditions too -- twilight,<br />
poor illumination, total color blindness. Both “A” and “B” could be correct<br />
under some conditions, but the other alternative, “neither A nor B,” could<br />
not be correct. Supposedly, ha wonders if the questfon is a trick, if<br />
the test wrftcr is malicious or just plain ignorant, and what was really<br />
wanted. Finally in desperation, t,e throvs up his hands and says, llThie<br />
is an absurd test nnd 1 don’t see how any intelligent person can be asked<br />
to take it seriouslylf’ And 1 have heard just that in the field on more<br />
than one occassion.<br />
This charge of ambiguity is one of most serious consequence to us<br />
and one on which we are most vunerable unless we consider the fundamental<br />
distinctions among the purposes of test and test items. I am referring<br />
to the distinction between tests which are used as criteria ant. tests<br />
which are u6eu as predictors. - -<br />
Criterion tests -- for example, achievement<br />
tests for comparing the effects of dffferent methods of teaching -may<br />
be open to Hoffmann’s criticisms. But his objections are lrreievant<br />
in the case of predictive tests -- for example, tests to select the most<br />
promising job applicant or the best qualified soldier for a special assignment.<br />
Basically a criterion test must be content valid -- that is,<br />
we must be able to defend the test on rational and logicalgrounds. All<br />
that fs required of predictive tests is that they sucessfully predict<br />
future performance.<br />
Now where does this leave those of us who use our tests for both<br />
purposes? Naturally we must attempt to demonstrate that vdt tests have<br />
both content validity and predictive or concurrent validity. It appears<br />
chet Hoffmann has never heard of empirical validation, but he manages<br />
to marshal1 a variety of defense against the evidence. He particularly<br />
attacks the criterion sgainst which we validate our tests. However) the<br />
value of a criterion is a very different question frocn the problem of<br />
whether a test can predict that criterion. Criticizing the tests simply<br />
175<br />
t
ecause we do not like the criteria is confusing the issue. Hoffmann’s<br />
concern rfth ambiguity is relevant only when the content vaLidity of a<br />
test is the primary constd.eratFon. Eve,1 80, tke customary item acreesing<br />
and selection in the constructicn of teats insurea a reaeonably satiafactory<br />
degree of content valfdity. This io particularly true in the<br />
case of our military tests which are designed to cover specific job areas.<br />
What about Hoffa%M'S cLaim that the “deep” examinee fs penalized<br />
by multipLe-choice iteM? What he seems to be saying is that the very<br />
dull examinee will fail the itea more frequently than the superficially<br />
bright examinee, but that the exceptLonally bright examinec vi11 have<br />
more trouble with the same item than the superficially bright examinee,<br />
Naturally, the brightest examinees are more apt to discover ambiguities<br />
than are other students. But Hoffmann’s ideas about the consequences of<br />
their perceptiveness do not hold up.<br />
To get back to Hoffmann’s assumptions, he states that tests,“... take<br />
account of only the choice of answer and not of quality of thought that<br />
lead to the choice,” and “They neglect skill and dtsciplfned expression.”<br />
This attribute he catls “quality of thought I’ is not defined, nor is any<br />
reliable and valid way of measuring it suggested. One mfght assume that<br />
Hoffmann would advocate use of the essay examination as a measure of his<br />
quality of thought. Such is not the case, for he outlines no less than<br />
four very convincing arguments why the essay test should not be used for<br />
that purpose. (L. Difficult to choose a topic fair to a% 2. difficult<br />
to determine whether the essay is actually revelant to the question; 3.<br />
difficult to overcome the problem of negatfvc halo due to poor handwriting,<br />
spelling, etc. ; 4. difficulty to maintain consistency within and between<br />
scores.) I often hear this quality of thought argument twisted around to<br />
say that just because a man scores high on the test does not necessarily<br />
mean that he can do the job. Most generaLly, those advancing this<br />
argument mean that the man will not do the job. While it may be true that<br />
a high score on an achievement or’ job knowledge type test does not<br />
automatically indicate a high degree of motivstion, it is eqtially true<br />
that the Low scorer cannot perform the fob, that is, he does not have<br />
the knowledge to perform regardless of hie motivation. Of course this<br />
depends on test validity and test purpose. It is up to us to define<br />
the purposes to which oar instruments may be put and to determine the<br />
validity, for various purposea, of these instruments.<br />
At this point we might conclude that Hoffmann does not like any kind<br />
of testing. We would probably be correct. He criticizes objective tests;<br />
yet he leaves no alternatives for mensurement. When he argues that &he<br />
results of an educational testing service st*)qy showing an essay teat<br />
was less good than an objective examinstion are silly and could not<br />
possibly have been obtained, he is simply confusing content with predfctive<br />
or concurrent validity.<br />
176<br />
,. ^ .._ -.. .__.-<br />
.
.<br />
. . . _. . - ._<br />
.<br />
. .<br />
Perhaps tha potentially mo6t damaging a66w$ion, rind the one which<br />
would be the mont difffcvlt for R0ffraan.n to 6u6tain, has to do with the<br />
effect of teatrr in the identification of individual mertt, He 6tate6,<br />
“They have a pernicious effect on education, and the recognition of merit.”<br />
Thi6 io cf cour6e -wholly without fcwundation. There have been too many<br />
6ucce66 6torie6 for u8 to even bother to refute hio claim.<br />
Now let u6 briefly turn to 6ome of our other critics. A major as-<br />
6umption found throughout the =fCfnpe of Packed, Barrum, Grose, tatlFte,<br />
Xarringtm, ard other6 i6 that “mind” cannot be meaeured. Tbcy seem to<br />
Consider ‘hind” a human “mystery 6y6tem” outeide the realm of rcientific<br />
otudy. This may be true, but if “Axi” is defined aa behavior, it indeed<br />
can be measured. In fact, the measurement of man’s fndividual differences<br />
perhapr ha6 been the greate6t sccwplishment of psychology thus far. We<br />
can anaeas the individuality of persons rind make pretty good predicitions<br />
about their future behevior.<br />
Our critics make tin assumption exactly opposite to the aerrtqtion<br />
I just mentioned. They se+- that test leads to conformity by picking<br />
perecmo who are all of the same type. This, of course, as6ume6 that<br />
“mind” can be measured only to well -- in fact, considerably better than<br />
we are able to measure it. Anyvay, there is considerable evidence to<br />
substantiate intra-individual trait variability. Thfa principle of<br />
p6ychologlcal testing 16 fgnored by these critics, The feet that people<br />
do differ vithfn themselves has long been recognized srnd submitted to<br />
careful etudy by peychologfste.<br />
It i6 complained that testing is an inva< ?on of prlvocy, and this<br />
criticicm m.6~ have 6ai~e merit. It i6 incumbent upon ‘:6 to demonstrate<br />
the validity of any item6 which might otherxiae be regarded a6 such an<br />
invarion. In the military eetting we arc seldom bothered by rhfs problem,<br />
especially in proficiency evaluation.<br />
The critic6 very rarely euggert alternative8 to p6yChOlOgiCal testing.<br />
Gardner, in hi6 book, Excellence, 6aye,“Anyone attacking the ueefulnerrr<br />
of te6t6 6w6t suggeet workable elternatlves. It ha6 been proven over and<br />
over again that the alternative method6 of evaluating ability arc subject<br />
to groso error6 and capable of producing grave injucticcr.” WC have only<br />
to look et our = rating ryrtem6 to realize how truly Gardaer rpoke.<br />
Or we could return to the day6 when perronnel decisions vere made on the<br />
baeir of hair color, family background, shape of the head, or sate other<br />
equally intuitive basis. The alternatives to testing when they are euggeeted,<br />
are cIearly rldfculous.<br />
So far thir morning,1 have attempted to point out the merits of our<br />
crltlc’6 charge6 agafnat testing, and we can conclude that the proterts<br />
tith oubatance are those about which we already knew. Hov we ahould take<br />
a look at the impact of our crftica. I do not believe that the Mllttary
--<br />
.,<br />
-__. . . .<br />
,Establlshment has become disillusioned with testing any more than I<br />
believe the general public. has. This conference is evidence that the<br />
Armed Forces are among the strongest supporters of testing. T!l2 protestors<br />
have not caused any detectable eifect in terms of reduced sale8<br />
of testfi or testing ecrvicee, The number of letters from educator6 or<br />
the general public (to reputable test publishers and the APA) critfciztcg<br />
teats has not increased. Generally, the poFuler pre6e hae not j-pad<br />
on the band wagon to foment a public outcry against u3. Apparently, the<br />
net effect has been little more than a few booka sold and a little high<br />
blood preesure BJXXXI~ psychologists. The latter at least may not be such<br />
a bad thing. Some of us need to be aroused.<br />
If this is the case, then why should we even bother to consider<br />
these self-styled protectors of a tyrannized society? The writings of<br />
these critics should make us stop, look, and listen. If we are to avoid<br />
future trouble and Improve the state of our nrt and aclence. we must<br />
improve the quality of our tests and services. He uwst adhere more<br />
etrictly to the ethical standards of our profession. We must give<br />
greater attention to the technfcal cha:acterfstice of our teats and<br />
criteria. We must continue our ef:orts to expose snd elimfnate the<br />
quacks and incompetenta who dwell about the fringe of psychological testing.<br />
And we muat improve our communications about testing 4th every<br />
segment of our public which we can reach. These cmdnicutions met be<br />
technically sound but they also must be written in underatandabl.. English.<br />
We might even form a committee withfn this association to draw up a pmphlet<br />
of general testing principles. and practices &ich could be diesemfnated<br />
within the <strong>Military</strong> Establishment.<br />
In conclusion, the basic assumptions of our critics are crroneoue<br />
and ftIliscfou8; they are generally baaed on lack of information, aa<br />
apparently is the case of Hoffmann, or, more seriously, on a refusal to<br />
accept the strong eolpirical evidence showing that fudfvfdurlity can be<br />
accurately assessed in such a way as to give better recognition toreal<br />
merit than has ever before been the case in our educstional, fnduetrinl,<br />
or rillitory lnetitution5.<br />
Actually, we know that standardized objective testing is ona of<br />
the great success stories of our day. This has been no better pointed<br />
out than by Gardner, in his book, Excellence (1961). Psychological testing<br />
for the firat tine enables us to look at the many facets of an individual<br />
rether than making judgmenta based on the oo-called “lump of dough”<br />
doctrine. Now we can truly measure and assess the individuality of each<br />
of our military personnel and through careful guidance help each individual<br />
realize hts potentialities ae indicated by our peychological<br />
.<br />
.<br />
178<br />
--_
\’ .<br />
c<br />
.<br />
,<br />
.<br />
* .<br />
. .<br />
testing iistmente. Tests provide us with the best u~ana avnilable for<br />
osseseimg individual profi.ciency and discovering md rewardfng indiv?dudl<br />
merit. We can help the cmadcr i:n t’he f!.eld indfviCuaII.y and different-,<br />
iaT!.y carei’ul.ly exnoine his men so ther he no longer mJSK depend cn his<br />
“feel-” or geatoult for pernonnei declaions, This ia cur gresteot strecgth.<br />
When the facts are Iaid it UnderstandabLy, it cannot be disputed or<br />
refuted.<br />
179<br />
-- -. ._^__ --._ _-... __ . - _. -. - _ ____---<br />
I
.<br />
6. .<br />
. .<br />
.i 4<br />
*<br />
.I<br />
.J ,I<br />
--.<br />
Dunnotte, H. D.<br />
prediction,<br />
Reforencrs<br />
Some methods for enhnncLng the vsltdita of prJycholoam<br />
PG read at ~%ympo~i~.~ ~Subgtoupin~Analyois a& AXI<br />
Approach to the Predfctton of lindfvidual Behavior, St. Lwia. MO,, 1962.<br />
Ethical otanderds of pcychologfete. Amer. Psychologist, 1963, 2, 56-60.<br />
Cdrdnor, Y. W. Excellence. New York:Horpar, 1961.<br />
Hoffman, B. The tyranny of teetine. New York:Crowell-Collier, 1962.<br />
X80<br />
I<br />
--1-- --...--w<br />
--
.<br />
Summary <strong>Report</strong> of the Steering Camnfttee<br />
The 1964 Steering Committee diacussfono prodklced the following<br />
resulto:<br />
a. The preaent mA emble3n Wit8 considered inadequate. After<br />
viewing a number of sump18 emblems, e bnafc deeign vas selected to be<br />
modified according to certain specificatr>na. It was agreed that the<br />
emblem would be drawn up and submitted to the Servfcea for further<br />
reccxzmendationo or acceptance.<br />
b. A draft copy of the WA Bylaw? was discussed in detail.<br />
The recanmended changer will be incorporated in the redraft of the<br />
Bylaws which wtll be submitted to the Servicee. It fe anticipated<br />
that a final copy will be presented to the membership at the 1965<br />
MTA conference.<br />
c. Major Frank I.. McLanethan of the Air Fcrce was elected<br />
to the Chairmanehip of the 1965 MTA Steering Colmnfttee.<br />
181<br />
..-----
-.<br />
Item Writers Aptitude Test Development Coornittec <strong>Report</strong><br />
Date of Meeting: 21 October 196t.<br />
Place of Meeting: EfCA Conference, Fort Benjamin HarrLdon, Indianapolis,<br />
Indiana.<br />
Committee Members: Mr. William W. !kr!ce, USA, Chainon<br />
Lt Harvey C. Gregoire, USAF<br />
Mr. fn:,rlte A, Nudoon, US?4<br />
I:r. John D. Kraf t, USA<br />
Hajor Joe R. Shafer, USAF<br />
#r. William M. Minter, USA<br />
hmary of Discussion:<br />
a. All services rsprejented gave a brief description of hov their<br />
test item writers are selected, None of the service8 currently u8es er.y<br />
special selection procedures other than subject-mhttar knowledge. The<br />
coernittee members felt that Subject-matter background was not a serious<br />
problem since the itcm uritars befng designated for test item construction<br />
duties are generally adequate in this respect.<br />
b. In general) the item urttero in all services perform skmilar<br />
duties. The Navy and Air Porte have their ftem writera come to a central<br />
location here they work directly with tC8t speclalist8. The Amy operates<br />
on a more decentralized basis in that the item writers are located mainly at<br />
Army Sertlice Schools and construct test itomr bared on wrtttcu speclffcotionr<br />
sent to them by the US Army Enlisted Evaluatior, Center. Test Sp.?Ciatt~tS<br />
from the latter locetfon make TDY trips to the item-writing agencies to<br />
coordinate the test development efforts,<br />
c. The ccmmittes felt that the followfng two major areas should be<br />
inveetigated as being most likely to provide valid fact ~8 which would<br />
asslat in screening the better from the poorer test item writers:<br />
(1) Personal hfatory items.<br />
(2) Test results obtained from measuring factoru, such a8 English<br />
usage, intelligence, and analytical reading ability. ,<br />
d. Some “brain storming” vas done by the cora3fttee to get some fndfcation<br />
of the types of personal history items or teets that might be used<br />
to asse8s item writera’ aptitude. However, the committee felt that a<br />
thorough and ccmprehenstve analysfe would need to be made of the f tern writer<br />
duties before the best groupfng of potential personal history item8 and<br />
tests could be compiled for use in fretting up an experimental desfgn appropriate<br />
for conducting a validation study.<br />
1. -<br />
.<br />
182<br />
_ . _ _.<br />
_.._.-_--.. -_--- -.--.--<br />
- - _^<br />
.-<br />
. .<br />
-<br />
.
e. Lt Gregolre, US&?, indicated that the S*KI (Specialty Knowledge<br />
Test) Branch at the Personnel Research Laboratory at Lsckland AI% was<br />
very much interested in the asseamnent of ?tes writer aptitudes. RC<br />
also fndicatcd that sufficient data and processing support very likely<br />
would be available at Lacklnnd AFB to conduct a preliminary study related<br />
to assessment of itm writers,<br />
f. Tentative plans wer
.<br />
i .<br />
--<br />
t<br />
!<br />
Ens Harold AdaL-an<br />
Hr. fied AlRrad<br />
Dr. Neal B. A~zdragg<br />
Lt Thumars 11. Atchley<br />
Mr. 841~ II. Baker<br />
Hr. Rqmond V. A&at<br />
Mr. Vernon H. Begge<br />
pi-r. Walter W. Rirdaill<br />
Dr. Warren S. Bluxmfald<br />
Mr. M&eel J. Bodi.<br />
Lt Barbara Bole<br />
Hr. John S. Brand<br />
Wr. Claude P. Brfdgee<br />
Mr. Donald A. Brown<br />
2d Lt Martin S. Brown<br />
Hre. Habal 0. Brunner<br />
ROSTER OF CONFEREEES<br />
Yx. Franklin S. Buckwmlter US Amy Quertermaatcr School<br />
Xaj Robert A. Burgesa<br />
Mr. Anthony Cmciglia<br />
u??lT ATTACHED<br />
- __yI--<br />
‘tlS Coset msrd Homdquartera<br />
US Coast &ard Training Cmto?:<br />
US Amy Security Agency<br />
us Am] w1itGy Polfce corpm<br />
US Naval ExDmFninS Cente:<br />
US Army Enlisted Evaluation Center<br />
6570th Personnel Research bborutowry<br />
US Air Force<br />
US Haval Exminlng Center<br />
US Naval Em~ining Center<br />
US Naval Peraonncl Research Activity<br />
Sm Diego<br />
US Amy Enli.ett:l Evaluation Center<br />
US Naval Exmlning Center<br />
US Army Enlisted Evaluation Center<br />
US Amy Enlisted Evsluetton Center<br />
US Amy Mr.dical Service, Fitzeinmons<br />
US Amy Enlf.sted Evaluation Center<br />
Headquarters, UZ Air Force<br />
US Army Enlfrrted Evaluation Center<br />
US Army Security Agency<br />
US Amy Enlisted Evaluation Center
_’<br />
. ‘-<br />
-.<br />
I<br />
I<br />
:<br />
e .<br />
I. .I<br />
--.. .._ _<br />
L<br />
C<br />
.<br />
c<br />
--_I.---<br />
. ..-A.<br />
Capt. JCE?C?B B. Carpenter<br />
Mr. Charles E. Caosidy<br />
:tt co1 c. f. chafenon<br />
Mr. Tbamz E. Chandler<br />
!XC William Q. Chcnn<br />
ILt Co1 Kent J. Ccllfngr<br />
lb. John W. Credlford<br />
Maj David P. Culclasure<br />
Cdr R. J. Dahlby<br />
M~aj Donald L. Dimand<br />
l.at Lt Duncan L. Dieterly<br />
Co1 Jamer C . Donaghey<br />
Dr. Henry J. Duel<br />
Mr. Erling A. Dukrrrchein<br />
?CSgt Raymond R. Durand<br />
1.6t Lt Jane* H. Durden<br />
Mr. Bernard J. Foley<br />
Mr. Step‘.:+- W. Pctlr<br />
Mr. John L. Pinucrnr<br />
Lt C. E. Csngenbneh<br />
Mr. Ronald K. Goodnight<br />
Cdt Robert J. Gray<br />
____.- _.... -.<br />
185<br />
6570th Personnel Research Laboratory<br />
US Air Force 0<br />
US Army Enlitited Evaluation Center<br />
US Army Enlisted Evaluation Center<br />
US hrmy Southeastern Signal School<br />
US Army Cmbat Survefllance Schocl<br />
US Army Enlisted Evaluation Center<br />
US Naval Zxamining Center<br />
US Amy Hedical Field Service<br />
US Coast Guard Training Center<br />
US t??rfne Corpr Inotftuta<br />
6570th Personnel Raecarch &sborrtory<br />
US Air Force<br />
US Army Enlfetcd Evaluation Center<br />
Headquarters, US Air Force<br />
US Naval Exmining Canter<br />
US Amy Security Agency Training Center<br />
US Army Ordnance Guided Hierilc School<br />
US Amy Security Agency Trafnlng Center<br />
6570th Personnel Research Laboratory<br />
US Air Force<br />
US Army Enlfrted Evaluation Center<br />
US Army Enliertb :;:lurtion Center<br />
US Naval Examining Center<br />
. -, -..-.. _ _<br />
__, y
1<br />
/<br />
. . .<br />
I<br />
.<br />
Cape Harry H. Craer, Jr.<br />
1st Lt Harvey Cregorie<br />
Ltjg Clyde A. Gronewold<br />
1st Lt Thomar H. Guback<br />
Mr. Robert L. Cup<br />
(Cdr Frederick J. Hancox<br />
Mr. Clayton B. Haradon<br />
I,t ‘Co1 Roy E. HaIrin, Jr.<br />
Capt R. H. Hayee<br />
F!r. Jock E . Hohrelter<br />
tfx. Fred 8. Hona<br />
Mr. John C. Houtr<br />
Hr. John J. Hcjbell<br />
Hr. Charles A. Hudson<br />
Hr. Clifford 2. Hutrlcy<br />
Mr. William L. Jackson<br />
HI :. E. C. Johnron<br />
Hr. L. W. Johnston<br />
Lt jg Katherine Kadenacy<br />
T.Edr Lawrence R. Kilty<br />
2d Lt Lloyd 0. Kimery<br />
Mr. Albert Kind<br />
UFXT ATTACREZD<br />
us Navy (Roti;ed)<br />
6570th Personnel Research Laboratory<br />
US Air Force<br />
US Navel Examining Center<br />
US Amy Defense Xnformotion School<br />
US Naval Examining Center<br />
US Coast Guard Headquarter6<br />
6570th Pcreonnel Research Laboratory<br />
US Air Force<br />
186<br />
Randolph Air Force Base<br />
US Naval Examining Center<br />
US Army Ealieted Evaluation Center<br />
US Army Enlisted F,valu:tFon Center<br />
US Naval Exemfning Center<br />
US Army AzzYor School<br />
US Naval Exeminlng Center<br />
US Army Signal School<br />
US Army Aviation School<br />
US Army Enlisted Evaluation Ctnter<br />
US Naval Examlnlng Center<br />
US Naval Examining Center<br />
US Naval Security Group Headquarter6<br />
US A- Training Ctnter, Englnoars<br />
US Army Combat Surveillance School<br />
. .<br />
. .<br />
%<br />
.<br />
_
_. . . i<br />
I<br />
.<br />
c<br />
Lt Co1 Albert S. Knauf<br />
Xr. Richard S. Kneirel<br />
Xr. John Kraft<br />
Cdr Ger?e L. Lane<br />
Mr. Larry J. LeBlanc<br />
LCdr Jack D. Lee<br />
Dr. C. L. John Lagere<br />
1st Lt Robert H. Lennevflle<br />
Lt Alexander A. Longo<br />
Xr, Charles J. Macaluao<br />
Capt Jack H. Harden<br />
Lt David R. Hnrkep<br />
Capt Joreph P. Kartin<br />
Hr. Curtis D. KcBrlde<br />
Hr. NflliuJ X. Hinter<br />
Dr. Joseph E. Harsh<br />
Mr. Ieadora J. Nevolpn<br />
Hr. W. Alcn Nicewander<br />
Ltfg Richard L. Olsen<br />
rtrj Herr11 R . Ower,<br />
1st Lt Arnold J. Pals<br />
ICdr Ralph Palverky<br />
-___ ..-<br />
. .<br />
6570th Personnel. Research Laboratory<br />
US Air Force<br />
US Amy Chamicnl Center and School<br />
OS Amy Enlfsted Eveluatlon Center<br />
Bureau of Naval Peroonnel<br />
Lackland Air Force Base<br />
US HAVA~ Examfning Center<br />
US Army Security Agency Training Center<br />
Navy School of Horic, Army Element<br />
US Naval Air <strong>Technical</strong> Training Center<br />
US NAVR~ Examining Center<br />
US Army Judge Advocate General School<br />
US Coast Guard Training Cer.ter<br />
US Coaat Guard Training Center<br />
US Axuy Artillery and Xiaeile School<br />
US Array Chemical Center and School<br />
6570th Personnel Research Laboratory<br />
US Air Force<br />
6570th Personnel Research LPboratory<br />
US Air Force<br />
US Army Enlirted Evaluation Center<br />
US Coart Guard Training Center<br />
US Amy Bnlieted Evaluation Center<br />
US Army Medical Center, Walter Reed<br />
Naval Examination Center, Royal Canadian<br />
N&V<br />
107<br />
.<br />
.<br />
_ I .- . . -<br />
I<br />
_.-- I .-__I .*.-a<br />
._- --- _ . . ^ e-e... .--<br />
1
.<br />
‘.<br />
. &<br />
.<br />
e<br />
.<br />
Fir. John J. Parke<br />
Mr. J. E. Ptrtingtcm<br />
Capt Kennuth A. Petri&<br />
M&j Jams N. Payne<br />
Hr. Henry W. Pepin<br />
Lt David L. Popple<br />
Mr. William Ii. pitman<br />
Uaf Joseph T, Polanrki<br />
Capt Carl R, Powers<br />
Mr. Frank R. Prfcs, Jr.<br />
Lt Co1 Robert A. Rerrnnyder<br />
\ Msj Clinton D. Regelia<br />
.<br />
I<br />
-..<br />
:<br />
-- ..---<br />
Mr. John H. Roth8<br />
Hr. Jack Rubak<br />
Hr. Carl Rudlnekl<br />
let L t Jumee L. Rueaell<br />
Maj William H. Sallejr<br />
Cept Clsrance D, Sspp<br />
Mrs. Genevieve K. Schulter<br />
Hoj Joe R. Shaafer<br />
Mr. Jean B. Sheppard<br />
Hr. Edwin C. Shtrkey<br />
SSgt Predrick J . Shunk<br />
.<br />
,<br />
,<br />
UN IT AYL’AC MD<br />
US Amj Ordnmce Guided Memile School<br />
US Army Enll$ted Evaluation Center<br />
US Amy Combat SurveilLaxe School<br />
US At-my Defenlre Atomic Support Agency<br />
US Amy Defame AtmEc Support Agency<br />
US Coast Guard Reserve Trainfng Center<br />
US Army Ordnance Center and School<br />
US Army <strong>Military</strong> Police School<br />
US Defense Atomic Support ABWICY<br />
US Army Enlisted Evaluation Center<br />
US Army Enlirttd Evaluation Center<br />
US Amy Defenm Information School<br />
Dept of Wavy, Bureau of Naval Personnel<br />
US Asq Dafanes Inforjletion School<br />
US Ar~rg S’qnal Cen.tar and School<br />
US Amy Infantry School<br />
US Army Intelligence School<br />
US Amy Air DL’fenoe School<br />
US laval Examining Center<br />
6570th Perronnel Rescsrch Laboratory<br />
US- Air Force<br />
188<br />
US Army Southeastern Signal School<br />
US Amy Enliattd Evaluation Center<br />
US Army <strong>Military</strong> Police School<br />
I<br />
‘.<br />
.<br />
. -<br />
,<br />
Y<br />
__. . ---<br />
.<br />
4<br />
.
.___....--<br />
i<br />
Mr. Willi6m P. SFm<br />
Capt Loren K. Smith<br />
j<br />
. Mr. Jane6 W. Smith<br />
1<br />
!<br />
) *<br />
-<br />
Mr. MAxon Ii. Smith<br />
Xdr Doncnld H. Tart<br />
Dr. June6 D. Teller<br />
ICdr Prrncea S. Turner<br />
Hr. Vern W, Urry<br />
Dr. Raymond 0. Weldkoetter<br />
Mt. FrUICi6 B . Waleh<br />
Mr. William W. W6nce<br />
Dr. Donald L. W666<br />
Mr. kdalbdrt U. Wafabrod<br />
Lt Berl R. Wll.li6ms<br />
Ens Thomas H. Wilson<br />
c Hr. C66tier S. Wlniwi66<br />
Dr. Michael A. Zaccarirr<br />
_ ._ ._ -, - _ . -<br />
.<br />
_.<br />
. .<br />
189<br />
UNIT ATTACHED<br />
US Arq Air Dafennse School<br />
US Army Intelligence School<br />
US Amy Ordnance Guided Miasfla School<br />
US 8rzz-y Ordnwce Gufdod Hiceile School<br />
Royal CanadiAn N6vy<br />
Headquartera, US Air Force<br />
US Coa6t Guard Training Center<br />
US Army Enlirted Evaluation Center<br />
US Army Enlieted Evuluation Center<br />
US Army Security Agency Training Center<br />
US Army Enlisted Evaluation Centur<br />
US Arq EnAl. l’6t6d Evaluatfon Center<br />
US Army Ccrpa of Engineer6<br />
US Coast Guard Training Center<br />
US Coast Guard Training Center<br />
US Naval ExaininLng Center<br />
Lockland Air Porca Baas Training Center<br />
.
6..<br />
,<br />
. .<br />
1%<br />
+ ,<br />
,*<br />
___<br />
.<br />
‘. *<br />
, ~ I<br />
, -I . . ;<br />
/ . . ‘x 4. . - . 1; \ .‘.), *.
UNCLASSIFIED/UNLIMITED<br />
PLEASE DO NOT RETURN<br />
THIS DOCUMENT TO DTIC<br />
EACH ACTIVITY IS RESPONSIBLE FOR DESTRUCTION OF THIS<br />
DOCUMENT ACCORDING TO APPLICABLE REGULATIONS.<br />
UNCLASSIFIED/UNLIMITED<br />
_. - .- ._. .-