(Revised) - NCRTM

EVALUATION: 

REHABILITATION 

APPLICATION GUIDELINES 

SOME 

(Revised) 

James A. Bitter and Don L. Goodyear 

Rehabilitatlon Research Institute 

Regional 

of Social and Rehabil1"tation Service 

Department 

of Special Education and Rehabilitation 

School 

of Northern Colorado 

University 

Greeley, Colorado 

840.031 

manual was supported, in part, by Research Grant Number 15-P-55254/8 

This 

the Social and Rehabilitation Service, Department of Health, Education, 

from 

and Wel fare. 

July, 1974

@ 1974 

Copyright 

A. Bitter 

James 

-11-

PREFACE 

manual was developed as an outgrowth of a Region VIII study group 

Thls 

program evaluation Initiated by the Regional Rehabilitation Research In- 

on 

at the Universlty of Northern Colorado. Early efforts by the study 

stitute 

in 1972 indicated that much of the program evaluation lterature in 

group 

emphasizes the nature of program evaluation, i.e. what it 

rehabilitation 

types of evaluation, etc. It was felt by study group members that a 

is, 

existed for application guidelines to program evaluation methodology 

need 

rehabilitation agency program evaluators. This manual, then, repre- 

for 

a beginning attempt to address this need and is offered as a technl- 

sents 

assistance effort to agencies by the Region VIII Regional Rehabilita- 

cal 

Research Institute. This revised edition of the manual is only In- 

tion 

as a working copy to be improved and further developed through co- 

tended 

operative efforts between researchers and practitioners 

acknowledgement is due many indivduals who helped to m- 

Grateful 

the February, 1974 draft of this document. Special recognition and 

prove 

s expressed to William G. Hills and Kenneth R. Reagles for de- 

gratitude 

examples for inclusion in this revision. Alo deeply appreciated 

veloping 

the many helpful suggestions offered by Dennis A. Gay, John P. 

were 

Joseph T Kunce and John E. Muthard relative to the manual's 

Giovannini, 

appropriateness of material, technical accuracy, organl- 

communicability, 

and format. Acknowledgement is also extended to Kevin G. Marshall, 

zation 

R. Rabbe and James E Tripp for their helpful assistance in edit- 

Stanley 

the February, 1974 draft. And a very special thank you is extended to 

ing 

D. Tanner for her patience and skill in editing and typing a diffl- 

Linnea 

t manuscrl pt. 

cul 

A. Btter 

James 

rector 

Di 

Rehabi litation Research Institute 

Regional 

of Northern Colorado 

University 

-111-

TABLE OF CONTENTS 

PREFACE III 

TABLE OF CONTENTS v 

PURPOSE 1 

INTRODUCTION 

Process 

Evaluation 

Evaluation Prlnclples 

Some 

11 ng 

Samp 

signlficance 

Statistical 

1- Random Numbers 

Table 

2 Sample Critical Values of Ch-square, the Correlation 

Table 

t and F at the .05 Probability Level 

Coefficient, 

Evaluation Objectives 

Sample 

PAR____T_T ONE E__X POST FACTO STUDY- DESIGN 

I 

EXAMPLE 

Sample 

Criteria 

Methodology 

Evaluation 6bjectlve 

Procedures Means, percentages 

Statistical 

Interpretation 

2 

EXAMPLE 

Evaluation Objective 

Sample 

Criterla 

Methodology 

Approac Questionnaire on client satisfaction 

Measurement 

Procedures One-scruple ohi-square test 

Statistical 


3 

EXAMIPLE 

Evaluatlon bet;'ve 

Sample 

teri a 

Cri 

Methodology 

Approach State Agency's Case Service Report 

Measurement 

data 

(R-300) 

Procedures" Chi-square test for two independent 

Statistical 

samples 

I nterp ret ati on 

-V- 

3 

3 

4 

4 

5 

6 

13 

15 

15 

15 

16 

20 

26 

33 

35 

35 

35 

36 

38 

46 

49 

51 

51 

51 

52 

52 

62

Eaiution betv 65 

Sample 

65 

Crlteria 

65 

Methodology 

Approach Fo llow-p survey of rehabilitated 

Measurement 

cases 

Procedures Product-moment correlation 

Statistical 

66 

coefficients 

Interpretati on 

Evaluation betv 79 

Sample 

79 

Criteria 

79 

Methodology 

Approach. Cas__.e__e Difficulty, Index 80 

Measurement 

Procedures One-way analysis of variance 82 

Statistical 

te rp retati on 

102 

In 

PART TWO" ONE GROUP PRETEST POSTTEST STUDY DESIGN 103 

Eaiuto )bletve 107 

Sample 

107 

Criterla 

107 

Methodology 

Approach" Goal Attainment Scalng 108 

Measurement 

Procedures t-es for correlated samples 112 

Statistical 

te rpret ati on 

122 

In 

Ezaiuton ble:tve 125 

Sample 

125 

Criteria 

Methodol ogy 

Approach. The Human Service Scale 126 

Measurement 

Procedures Linear regression 134 

Statistical 

In te rp re tati on 

PART THREE PRETEST POSTTEST CONTROL GROUP STUDY DESIGN 147 

8 149 

EXAMPLE Eaiuaton betlve 151 

Sample 

151 

Criteria 

thodo I ogy 151 

Me 

Approach Questionnaire for client knowledge 

Measurement 

the rehabilitation process 152 

of 

Procedures t-test for ndependent samples 154 

Statistical 

Interpretatl on 

-vl- 

66 

74 

125 

142 

160

9 

EXAMPLE Evaluto betiv 

Sample 

teria 

Crl 

Methodology 

Approach. The Work Austment Rating 

Measwrement 

Procedures Analysis of covariance 

Statistical 


PAR____T FOUR- SOLOMON FOUR_____-,G.ROUP STUDY DESIG______EN 201 

teri a 205 

Cri 

205 

Methodology 

Approach Test of counselor knowledge of 

Measurement 

mthods and resources 206 

placement 

Procedures. 2 x 2 factorial analysis of 

Statistical 

208 

variance 

Interpretatl on 220 

REFERENCES ............................. 221 

GLOSSARY OF SYMBOLS .................. inside back cover 

-vll 

161 

163 

163 

163 

164 

168 

196

PURPOSE 

Evaluatlon Some Application Guidellnes Is intended 

Rehabllitatlon 

be a working manual for rehabllitation practitioners responsible for, 

to 

interested in, program evaluar.ion and for graduate students of research 

or 

statistics It can be used i'or agency application, in-service train- 

and 

ing, or self-instruction. 

document attempts to be a how-to-do-t manual and workbook, thus, 

The 

the purpose of other documents which describe the nature of pro- 

extending 

evaluatlon. The manual offers 10 application examples for sample 

gram 

objectCves. Four evaluation designs which necessitate only a 

evaluation 

calculator are presented, i.e. ex post facto, one group pretest- 

hand 

pretest posttest control group, and Solomon four-group designs. 

posttest, 

manual is organized by parts corresponding to the four evaluatlon 

The 

and by examples. Each of the four parts is preceeded by a brief 

deslgns 

of the evaluation design, including its lmitations and/or 

description 

The content for each of the 10 examples is organized in the fol- 

value. 

format. 

lowing 

1 Sample evaluation objective for a rehabllitation program component. 

2 Criterla for evaluating the program component. 

Methodology for evaluation, including the general procedure, mea- 

3. 

approach (five recently developed procedures are presented in the 

surement 

manual ), and statistical procedures. 

4 Interpretation of the results for service delivery practice. 

statistical techniques, a dfferent one for each example, are pre- 

Ten 

in the followlng order" means and percentages, one-sample chi-square, 

sented 

test for two ndependent samples, product-moment correlation, 

chi-square 

analysis of variance, t-test for correlated samples, linear regres- 

one-way 

t-test for independent samples, analyss of covariance, and a 2 x 2 

sion, 

analysis of variance. 

factorial 

10 hypothetical examples are ntended to assist practitioners n 

The 

program evaluation, whether on a State agency, distrct office, 

conducting 

facility level. The materlal is offered only as worked examples or 

or 

for application of evaluation methodology. The user can sub- 

guidelines 

his own program evaluation objective and use this manual to help 

statute 

to understand and "walk" through the methodologCcal steps of evalua- 

him 

tlon and analysis. 

s, of course, the danger of oversimpllfcaton In a manual 

There 

as ths. It should be emphaslzed that ths manual is not intended 

such 

a substitute for knowledge and understanding about resear--E-and sta- 

as 

methodology. There are many textbooks which offer background 

tistical 

information and each user should supplement this manual with textbook n-

or quallfled consultants. Rather, It is the Intent of this gulde 

formation 

help rehabilitation practitioners apply evaluation methodology to their 

to 

Those using It are encouraged to make suggestions for its im- 

programs 

and development. 

provement 

some background n evaluation methodology, the user wll find a 

For 

by F. N Kerlnger entitled Foundations of behavioral research (New 

book 

Holt, Rinehart, & Winston, 'i964) to be an excellent source. Addl- 

York 

recommendatlons Include: J. L. Bruning & B. L. Keintz. Comput.a ,- 

tional 

handbook of statistics. Dallas Scott, Foresman and Co., 1968, 

tional 

J. R. Amos, F. L. Brown, & O. J. Mink. Statistical concepts- A_ 

and 

program. New York" Harper and Row, 1965 Other excellent resources 

bsic 

E. C. Bennett & M. Weisinger. Progr.am evaluation A resource handbook 

are 

vocational rehabilitation. New York ICD Rehabilita--tion and Research 

for 

1974, and H P. Harry, R. E. Winnie, & D. M. Fisk. Practical 9__.- 

Center, 

for state and local B.overnment officials. Washington, D.C. 

evaluation 

Institute, 1973. 

Urban 

-2-

INTRODUCTION 

evaluation is essentially an effort to determine what 

"Program 

occur as the result of a planned program by comparing 

changes 

changes (results) with desired changes (stated goals) 

actual 

by identifying the degree to which the activity (planned 

and 

is responsible for the changes." (Trantow, 1970) 

program) 

need for program evaluation has developed rapidly in the past 

The 

years. There are man), reasons for this trend, but specifically 

several 

70's have been termed the age of accountability a fact which has a 

the 

affect on all of the human services programs including rehabllita- 

direct 

welfare, education, public health, etc. No longer are publlc funds 

tion, 

without questions asked about program value to the public and 

available 

to target service populations. 

we look at the actions of legislators in the past few years, it 

If 

be evident that there is an increasing need to justify the "worth" 

should 

human service programs. Federal monies have increasingly been made 

of 

to State agencies to expand current evaluation activitles and 

available 

create new evaluation units in agencies where none existed. The Reha- 

to 

Act of 1973, PL 93-112, specifically requires rehabilitation 

billtation 

project evaluation. Title IV of the new Act (Sec. 401. (a) (1)) indi- 

and 

that: 

cates 

Evaluatl on 

Secretary shall measure and evaluate the impact of all 

"The 

authorized by this Act, in order to determine their 

programs 

In achleving stated goals, in general, and In 

effectiveness 

to their cost, their impact on related programs, and 

relation 

structure and mechanisms for delivery of serwces, in- 

their 

where approprlate, comparisons with appropriate con- 

cluding, 

groups composed of persons who have not participated in 

trol 

programs." 

such 

Process 

might be expected there are no deflnite rules for determining the 

As 

of program evaluatlon. Trantow (1970) suggests several basic 

structure 

in the evaluation process (a_) define specific goals or objectives, 

steps 

establish quantifiable criteria from which to evaluate these goals, (c_) 

(b_] 

methods of comparing program outcome to the criteria, (d) design 

develop 

evaluation, (e__) collect the information needed, (f_) analyze-the re- 

the 

sults; and (.) develop conclusions and recommendations. 

Suchman (1971) feels that prior to undertaking evaluation, 

Similarly, 

elements must be present- (a_] the establishment of objectives or 

three 

(b) a program of intervention capable of attaining the goal, and 

goals, 

a method of determining f the goal or objective has been obtained. 

(.) 

approach s llustrated by Collignon, Zawada, Thompson, and 

Another 

(1971). They suggest that the evaluator understand what exists, 

Markowitz 

-3-

the program Inputs, and understand the results of program actl- 

determlne 

tles and the program outcomes. In order to understand program outcomes, 

w 

recommend that the evaluator ask the following questions: has the 

they 

achieved the criteria set forth why was the program effective 

program 

or not effective? and, what actions need to be taken 

another perspective for evaluation is suggested by Denlston, 

Still 

and Getting (1968). They list four areas of concern which are 

Rosenstock 

expressed by program directors. (a) appropriateness has the pro- 

most 

undertaken been directed toward the important concerns, (b) adequacy 

gram 

what extent have the problem areas been eliminated, (c) ffectiveness 

to 

what degree have the program objectives been attained, and (d) effi- 

to 

ciency --what were the costs in achieving the objectives 

excellent book whlch describes how to specify goals and objectives 

An 

R. F. Mager. Preparing i.nstruction..al obectlves. Palo Alto, CA 

is- 

Publishers, 1962. Mager (1962) describes an objective as an ntent 

Fearon 

in performance terms and contains the following conslderations- 

stated 

identlfication of the terminal performance/outcome, (b) describes 

(a) 

appropriate, conditions under which performance/outcome is expected 

If 

occur, and (c) specifies the criteria for acceptable performance/ 

to 

outcome. 

Evaluatlon Prncipjles. 

Some 

is important, of course, that the design of a study, the measure- 

It 

approach, and the statistical procedures are appropriately applied. 

ment 

adequacy of interpretation of study results depends upon such assur- 

The 

Ex post facto studies (Examples 1 5), for example, have many 

ances. 

limitations for interpretation than do experlmental designs (Examples 

more 

10) which offer more control for the effects of unstudied variables 

8- 

must also be taken to assure that the measurement approach s valld 

Care 

reliable and the statistical procedures are appropriate. Positive 

and 

are relatively easy to interpret while negative findings are much 

findings 

difficult. Negative results may be due to nappropriate methodology, 

more 

and/or statistical procedures. On the other hand, if these 

measurement 

are adequate, then negative results can contribute to our 

consderatlons 

edge of program (Kerl inger, 1964). 

knowl 

When we sample, we are drawing a portion from some popu- 

Sampllng. 

of possible subjects. The main concern for the evaluator is to n- 

lation 

the representativeness of the sample. Large samples are more accu- 

sure 

than small samples. The smaller the sample, the greater s the pos- 

rate 

of error. However, it s often more costly to obtain larger sam- 

sibility 

more time consuming, and not always available A good pollcy to 

ples, 

is to use as large a sample as possible under the evaluator's 

follow 

rcumstances. 

ci 

are several approaches to sampling. If an evaluator wanted to 

There 

a sample of all the closed cases within his State during the past year 

use 

compare the personal characteristics of rehabilitated cases with the 

and 

cases, the evaluator could take a smple random sample. 

not-rehabilitated 

-4-

is a method of drawing a portlon of the populatlon (all closed cases 

Thls 

the fiscal year) so that each member of the population has an equal 

during 

of being selected, for example, drawing names from a hat or enter- 

chance 

a table of random numbers. To use a table of random numbers, merely 

ing 

a number to each member of the population, enter a table of random 

assign 

at any point (a sample appears as Table 1) and, moving n any pre- 

numbers 

direction, read the numbers of the individuals to be included 

determlned 

the sample. If our sample has been selected by a random method, we can 

in 

it is representative of the populatlon from which it was drawn. The 

say 

can then make inferences about the population from his sample. 

evaluator 

type of sampllng procedure Is systematic random sampl.ng. 

Another 

procedure could be used in doing a case rewew n which the evalua- 

Thls 

could take every tenth name from an alphabetical lsting or every nth 

tor 

in the file. If he chooses the latter procedure, he could select a 

case 

from 1 to 10, for example the number 6, to identify every nth case. 

number 

would then proceed to the files and pull case numbers 6, 16, 26, 36... 

He 

he has completed the selection of cases for hls study. 

until 

the other hand, an evaluator may wsh to study the comparative 

On 

of a particular training program on men and women. He would, then, 

mpact 

a representative sample of the number of men and women who went through 

want 

program. To insure such representativeness, he might use a stratified 

the 

sample. For example, if the proportion of men to women was 2 to 1, 

random 

sample should include a 2.1 proportion of men to women in order to be 

the 

considered representative. 

s.inificance. A level of statistical signlfcance s 

Statistical 

chosen in conductlng research In social science research com- 

generally 

levels of sgnificance are the .05 probability level and the .01 prob- 

mon 

level. This level refers to the number of times in 100 an obtained 

ability 

could occur merely by chance. For example, the .05 level means 

result 

a finding which is significant at this level could occur by chance 

that 

5 times in 100 trials. Generally, in published articles statistical 

only 

is dentified in the following manner- 

sgnficance 

9_ refers to "probabllty", < means "less than", and 05 and .01 refer 

where 

the probability levels. 

to 

critical values at the .05 probability level for chl-square, 

Sample 

correlation coefficient, t, and F statistical tests are gven n Table 2 

the 

the statistical test results equal or exceed the appropriate critical 

If 

the findings are said to be "significant" Table 2 gves critical 

value, 

only for examples used in this manual. More complete tables may 

values 

found in statistics textbooks. 

be 

-5-

1 Table 

Numbers 

Random 

00277 23723 27229 60600 27072 98180 44936 06731 58702 

55251 

80332 17163 93678 95418 03934 64810 43191 00792 61497 

35610 

46870 45244 88590 09738 49869 09608 96140 27241 80951 

62490 

98988 29169 33104 81989 67473 38782 31043 52586 29608 

15320 

95874 66629 19575 41256 72689 78437 89534 02501 35254 46750 

97797 86893 52474 71759 73424 16981 20350 93878 46078 

02355 

87757 24159 65268 20207 77187 32943 98105 75475 55627 

87802 

09042 50053 83910 56058 27350 30693 27593 50763 67489 

29026 

38973 24865 75755 05689 91453 43532 93256 34239 00556 

55914 

91057 74053 20835 89891 87082 99493 99167 38131 08877 52010 

10025 34448 54945 84908 44824 02226 18043 08241 90611 

82178 

76392 76037 78673 33227 01277 75409 46970 23682 34549 

69057 

25263 44620 13646 14226 80434 61480 47434 41334 46802 

01113 

22189 76654 44636 43451 23517 16904 07576 72731 65695 

69790 

29064 84525 31338 39175 72737 96858 43466 85200 77787 65988 

59031 65879 23226 71133 79899 83070 73786 45688 44753 

28904 

89245 89880 98245 79844 35192 97610 90245 97330 88783 

48379 

79387 29928 18678 80856 05069 85954 74623 50411 09027 

29325 

94092 85788 17796 02489 03399 01805 21021 02941 86703 

59029 

06192 20941 70417 45023 90896 91031 62830 89165 75397 44559 

35543 07683 45709 40591 23906 24101 48848 49240 85308 

07200 

18578 83597 77850 44418 51795 99619 61936 80796 04160 

67294 

16035 22564 41561 96559 27966 40835 13123 15966 38718 

26022 

58729 18571 49902 82539 11874 44595 89911 41204 91050 

47879 

23501 57801 32724 06785 04660 78417 40603 22602 84439 45927 

98259 50424 75466 47799 70064 59143 59231 43002 27199 

65139 

02174 58179 68393 92596 39349 26498 45450 47906 76317 

74442 

72281 25322 12956 27041 14839 80546 08719 95632 69105 

17478 

60992 64944 99329 72651 78197 91740 02277 51333 11367 

20664 

31772 62729 89920 76780 54712 22238 21460 04224 96783 73379 

29344 94207 33028 02422 15117 65932 28689 57077 73370 

36291 

44405 97704 11636 86533 29509 73252 83632 40990 19167 

83519 

47964 25765 04775 55801 89173 02722 80844 69856 07645 

92923 

82058 46808 98769 09570 41860 69274 93487 10305 22885 

95433 

83999 50298 74743 50424 76343 78247 98027 66897 24249 30608 

55721 86437 33564 05909 69041 68287 77580 14197 11526 

78964 

63862 17575 55461 22896 40835 42641 94695 24955 25430 

49278 

89191 04185 29797 40889 03305 87797 77840 90361 83638 

76883 

18734 53515 55274 10056 02608 78262 58530 56068 84681 

95431 

45503 99586 65880 41935 96575 60548 12842 22233 07973 88195 

Rand Corporation. A mllion random dlglts with 100,000 

Source 

deviates. New York. Free Press, 1955, reprinted wth permisslon 

nomal 

of the publisher. 

-6-

df 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

Sample 

Chisquare 

Table 2 

Values of Chi-square, the Correlation 

Critical 

and F at the .05 Probability Level 

t, 

3.84 

5.99 

7.82 

9.49 

11.07 

(X 2) 

ati on 

Correl 

(r) 

Coefficient 

.99 

.95 

.88 

-7- 

t 

12.71 

4.30 

3.18 

2.78 

2.57 

2.45 

2.37 

2.31 

2.26 

2.23 

2.20 

2.18 

2.16 

2.15 

2.13 

2.12 

2.11 

2.10 

Coefficient, 

F 

161.45 

18.51 

10.13 

7.71 

6.61 

5.99 

5.59 

5,32 

5.12 

4,96 

4.84 

4.75 

4 67 

4.60 

4.54 

4.49 

199.50 

19.00 

9.55 

6.94 

5.79 

5.14 

4.74 

4.46 

4.26 

4.10 

3.98 

3.88 

3.80 

3.74 

3.68 

3.63

S.ample Evaluatlon O.b.ectives 

following represent examples of evaluation objectives in reha- 

The 

agencies The list 1"s not exhaustive, of course; nor are some 

bilitatlon 

appropriate or of interest to all agencies. They are offered 

examples 

merely to orlent the manual user to some typlcal program evaluation 

here 

which can be measured. The first 10 sample evaluation objec- 

objectives 

correspond to the 10 worked examples of this manual. 

tives 

To determine client perceptions of rehabilitation agency servlces 

1. 

employment outcomes of clients. 

and 

To determlne client satisfaction with rehabilitation services for 

2. 

eligible cases closed during three fiscal years. 

all 

To determine differences for male and female clients in services 

3. 

or arranged for by the agency during the rehabilitation process. 

provided 

To determlne the relationships among some client and service varl- 

4. 

(age, number of counselor contacts, length of employment and weekly 

ables 

ncome) for clients closed as rehabilltated (status 26) for one fiscal year 

To compare effectiveness of rehabilitation counselor performance 

5. 

three rehabilitation offices based on the number and case difficulty 

n 

of successful closures. 

To compare the effectiveness of different service delivery tech- 

6. 

with a particular type of client. 

niques 

To determine the relatlonshlp between client and/or rehabilitation 

7 

variables, e.g. cost of training and change n client need satis- 

process 

faction. 

To compare the effectiveness of two dfferent methods of orienta- 

8. 

to rehabilitation services for new clients. 

tion 

To compare the effectiveness of three dfferent approaches to work 

9. 

training in a rehabilitation facility 

adjustment 

To determine the effectiveness of In-service tralning on counselors' 

10. 

of placement methods and resources. 

knowledge 

To compare service effectiveness wlth the same type of cllent of 

11. 

working with specialized caseloads and counselors with gener- 

counselors 

al I zed caseloads. 

To compare characteristics of clients that are closed successfully 

12 

after having received rehabilitation services with those who 

rehabilitated 

have not been successfully rehabilitated. 

-8-

To compare the employment stability of clients who have recelved 

13. 

job placement by the counselor with clients who found their own 

direct 

empl oymen t. 

To detemine the benefits of rehabilitatlon service, e g. weekly 

14. 

by comparing clents who had been prov1"ded servl'ces with a contrast 

income, 

group of eligible individuals who bad not rece'ved rehabilitation service. 

To compare the success rate of rehabilitated clients that were re- 

15. 

public assistance payments at application with rehabilitated cli- 

ceivlng 

ents who were not receiv1"ng public assistance. 

-9-

PART ONE 

EX POST FACTO STUDY DESIGN 

deslgn calls for a s±ngle group that is studed only 

Thls 

tme subsequent to services. The effects are presumed to 

one 

have been the result of service (X). 

The evaluator dent±f±es a group of sub3ects that 

Method. 

undergone services (X) and observes (0) ths group. He 

have 

attempts to nfer that the service had an effect on the 

then 

results. 

obta±ned 

Because the observation or measurement occurs 

Lmtat±ons. 

service has already been prov±ded, the des±gn does not 

after 

the evaluator to control or manipulate variables or 

allow 

(X). It also lacks the advantage of ob3ect±ve assign- 

serv±ces 

of cl±ents to serv±ces (randomization) to control for 

ment 

dfferences. Therefore, there ±s considerable rsk for 

clent 

nterpretatons snce some results may not be due to 

accurate 

serv±ces wh±ch were provided. 

the 

Kerlnger (1964) feels that desp±te the weaknesses 

Value. 

ex post facto studies, they are necessary n social science 

of 

quest±ons which do not lend themselves to controlled 

research 

nqury.

PART ONE 

EX POST FACTO STUDY DESIGN 

design calls for a single group that is studied only 

This 

tlme subsequent to services. The effects are presumed to 

one 

have been the result of servlce (X). 

The evaluator Identifies a group of sub3ects that 

Method. 

undergone services (X) and observes (0) thls group. He 

have 

attempts to infer that the service had an effect on the 

then 

results. 

obtained 

Because the observation or measurement occurs 

Limtatlons. 

service has already been provided, the design does not 

after 

the evaluator to control or manipulate variables or 

allow 

(X). It also lacks the advantage of ob3ectlve assign- 

services 

of clients to services (randomization) to control for 

ment 

differences. Therefore, there is considerable rlsk for 

client 

interpretations since some results may not be due to 

accurate 

services which were provided. 

the 

Kerlinger (1964) feels that despite the weaknesses 

Value. 

ex post facto studies, they are necessary in social science 

of 

questions which do not lend themselves to controlled 

research 

Inquiry.

i 

i 

EXAMPLE 

A. General Procedure: Ex post facto study deslgn. 

Measurement approach: The Consumer Measurement 

B. 

and Consumer Measurement Follow-up 

Scale 

Questlonnalre. 

C. Stat±stcal Procedures: Means, percentages. 

example was drafted by Wllam G. Hills, Assoclate 

iTh±s 

of Management and Publlc Admnstrat±on, Pennsylvania 

Professor 

Un±versty, M±ddletown. Dr. Hills was formerly the Drector 

State 

the Reg±onal Rehabltaton Research Inst±tute, the University 

of 

Oklahoma. The Consumer Measurement Scale and the Consumer 

of 

Follow-up Quest±onna±re are reprinted wlth permission 

Measurement 

of the authors.

EXAMPLE 11 

A. General Procedure: Ex post facto study design. 

Measurement approach: The Consumer Measurement 

B. 

and Consumer Measurement Follow-up 

Scale 

Questlonnalre. 

C. Statlstcal Procedures: Means, percentages. 

1 example was drafted by Wlllam G. Hllls, Assoclate 

Thls 

of Management and Publc Administration, Pennsylvania 

Professor 

University, Middletown. Dr. Hills was formerly the Dlrector 

State 

the Regional Rehabltaton Research Institute, the University 

of 

Oklahoma. The Consumer Measurement Scale and the Consumer 

of 

Follow-up Questionnaire are reprinted wlth perm±sson 

Measurement 

of the authors.

I Sample Evaluation Objectlve 

determine cllent perceptlons of rehabilitatlon agency services 

To 

employment outcomes of clients. 

and 

II Criterla 

of satisfactlon expressed by clients for nne agency service 

Levels 

related factors at the tlme of closure as determined by the Con 

delivery 

Measurement S..c.a]e and the employment status of clients slx months 

sume___._r 

as reported on the Consumer Measurement Follow-u_ Qu.estionnaire. 

later 

A. General Procedure s 

III Methodology 

Ex post facto study design XO 

Steps. 

X service/treatment 

where 

0 observation/measurement 

and 

Administer the Consumer Measurement Scal____e (CM__S) to all cll- 

i 

of the State agency as they are closed from servlce (status 26, 

ents 

30, 08) for an entlre fiscal year period. If maled, include a 

28, 

letter and a return addressed, stamped envelope. 

cover 

Mall the Consumer Measurement (C_M) Follow-u_ Questlo.n.nalre 

2 

the same closed clients sx months after closure wth a cover let- 

to 

ter and a return addressed, stamped envelope. 

Upon return, score and tabulate the results for the CMS and 

3 

responses for the C__M Follow-u_ Questlonnalre. 

tabulate 

D. T. & Stanley, J. C. Experimental and quas-expermental 

2Campbell, 

for research. Chicago Rand McNally, 1966. 

s 

-15-

Calculate means for the CMS total scores and for each of nlne 

4 

service delivery rlated factors 

CM_._S 

Develop a profile of the means for the nine factors 

5 

may also be developed relative to speclfc variables, e g. 

Profiles 

dlstrict, closure status, age, sex, disability, etc. 

Tabulate response category frequencies for the CM Follow-u 

6 

questions and determlne percentages of response for 

Q.u.estionnalre 

response category. If desired, a one-sample ch-square test 3 

each 

be applied to determlne if there is a sgnificant difference In 

may 

for each question (application procedures are described in 

responses 

Example 2). 

If deslred, compare scores for the nne CMS factors wlth the 

7. 

of the same nlne factors on the CM Follow- Questionnalre to 

scores 

agreement n satCsfaction perceptlon after six months. One 

assess 

for determining agreement is to calculate a product- 

possibility 

correlation coefficient (appICcation procedures are described 

moment 

Example 4). 

in 

If not all clients respond to the CMS or CM Follow-u_ (].- 

8. 

you may want to compare characterlstcs of the obtained 

tonnare, 

wth the total client population to determine if the sample 

sample 

representative of the population. One way this may be done is by 

is 

a ch-square test for two independent examples s (applicatlon 

using 

procedures are descrlbed in Example 3). 

Measurement Approach The Consumer Measurement Scale and the Consumer 

B. 

Follow-u_9 questionnaire 

Measurement 

Measurement Scale_. The purpose of the CQ.nsumer Measurement 

Consumer 

s to obtain an ndicatlon of client satisfaction/dssatsfacton 

Scale 

wth rehabilitation services (the CMS appears at the back of this example). 

S. N_onparametrlc statistics for the behavioral sciences. 

3Segel, 

York. McGraw-Hll Book Company, 1956, pp. 4T-47 

New 

G. A. Statlstical ana!,si s i._n and educatlon. 

WFerguson, 

York McGraw-Hill ,' 1966, pp 105-115 

New 

S. NQn.parametrc statistics for thee behavloral sciences 

SSiegel, 

York McGraw-Hill, 1956, pp. 104-111 

New 

W. G. & Viaille, H. V Consumer's measurement of vocational 

6Hills, 

Norman, OK. University of 'Oklahoma Press, 19---73. 

rehabilitatlon. 

-16-

WORK SHEET

CM__S contains 28 questions which represent nlne service dellvery 

The 

aspects (factors) of agency services. Through factor analysis, 

related 

questions were found to be relevant to each of eight factors, and 

three 

questions to the ninth. 

four 

The questions which relate to each of the nne factors are as follows 

Factor C.ues ti ons 

Speed of Service Delivery 

1. 

Medical Services 

2. 

Training Services 

3. 

Employment Satisfaction 

4. 

Participation in Planning 

5. 

Counselor Effort n Placement 

6. 

Agency Policies 

7. 

Physical Facilities 

8. 

9. Personal Treatment 

16, 22 

3, 

9, 28 

6, 

4, 10 

1, 

8, 26 

7, 

14, 15 

2, 

17, 23 

13, 

19, 21, 24 

11, 

12, 25 

5, 

18, 20, 27 

to CMS questions involves placing a check mark n the box 

Responding 

best explains how the respondent feels about each statement. For 

that 

example- 

1. the quality of training I received... 

Does 

Very Not 

Very 

Satlsfied Satisfied Neutral Dissatisfied Dissatisfied Apply 

of the questionna1"re involves the asslgnment of negative 

Scoring 

for responses indicating dCssatisfaction and positive points for 

polnts 

indicating satisfaction. Thus, a client's response to each 

responses 

is scored -2 f he is "very dissatCsfied", -1 if he is "dissat- 

question 

isfied" 0 if he is "neutral" +I f he s "satlsfied" or +2 f he s 

"very satisfied". 

describe the levels of satisfaction non-technically, these re- 

To 

are reported as "high", "medium", or "low" depending upon the de- 

sponses 

of satisfaction (or dissatisfactlon) expressed for each individual 

gree 

Low levels of satisfaction range from 0 to +.70, medium levels 

question. 

from +.71 to +1.40, and high levels range from +1.41 to +2.00. The 

range 

ranges, only negative, apply to the levels of dissatisfaction The 

same 

of consumer satsfaction/dlssatisfacton levels are shown in the 

continuum 

lowing chart. 

fol 

-18-

WORK SHEET

Continuum of Satisfaction/Dissatlsfaction Levels 

Di ssatisfactl on 

Satl sfactl on 

-2 -1.40 -.70 0 +.70 +1.40 +2 

HIGH MEDIUM LOW LOW MEDIUM HIGH 

questions pertaining to each factor are scored according to the 

The 

of respondents and are combined to yeld a mean (average) score 

replies 

each factor. For example, if every client rated all three questions 

for 

speed of service delivery as "satisfied", the mean score for 

concerning 

of service delivery would be speed 1 1 + 1 

+ 

or 1.00. The scores for all 

3 

factors are then profiled to portray a general picture of client sat- 

nne 

within an agency for a specified time interval (usually one fis- 

isfaction 

year). 

cal 

Measurement Follow-up questionnaire. The follow-up ques- 

Consumer 

seeks information about the former client's employment during 

tionnaire 

six months since closure. It contains ( six questions regarding 

the 

(b_) two related to the rehabilltation program, and (c_) nine 

employment, 

for again rating the service delivery related factors on the Co.__n- 

stems 

sumer Measurement Scale. 

C. Statistical Procedures 

1_. Score the CMS by computing response levels for each ques- 

Step 

and each of the nne factors. For example- 

tion 

Client 

I 

2 

3 

4 

Factor" Speed of Service Delivery 

Question 

16 22 

3 

1 0 1 

2 1 1 

1 0 -1 

-2 -1 0 

-1 1 0 

0 

5 

1 1 1 3 

Sum 

Sum 

2 

4 

0 

-3 

Mean 

.67 

1.33 

O0 

-1. O0 

Mean .20 ,20 .20 .20 

-20- 

.00

ORK SHEET

Client 

factor mean (.20) can be obtained in any of three ways: (a_) di- 

The 

the sum of all scores by the number of scores, e.g. 3 15 .20, 

viding 

dividing the sum of client means by the number of clients, e.g 

(b) 

+ 1.33 + O0 + (-1.00) + .00 

67 

20, or (c_) divlding the sum of ques- 

5 

+ .20 + .20 

.20 

tlon means by the number of questlons, e.g. 3 .20. 

mean satisfaction level for this sample of 5 clients for the fac- 

The 

"Speed of Service Dell"very" is .20 which suggests that this group was 

tor 

"low satisfied" with the speed of service delivery. 

same procedure s followed for each of the other eight factors 

The 

the CMS. 

of 

2_. Respondents' overall satisfaction with agency services is 

Step 

determined. For example" 

similarly 

FACTOR 

2 3 4 5 6 7 8 9 

1 

1.20 .40 .75 .90 1.00 .68 .70 .90 

.67 

1 33 1.47 .90 .89 .90 .49 1.00 .67 .45 

O0 70 1.30 -.60 .75 .89 91 -.91 .00 

-1.00 -.90 .87 45 1.25 1 54 .65 .78 .34 

1.00 -.98 .57 .48 1.15 -1.00 .19 .47 

O0 

1.00 2.07 2.49 2.06 4.28 5.07 2.24 1.43 2.16 

Sum 

Sum 

7.20 

8.10 

1 64 

3.98 

88 I. 

22.80 

Mean .20 41 .50 .41 86 1.01 .45 .29 .43 51 

in ths example the general level of satisfaction wlth rehabll- 

Thus, 

service delivery is posltve, but low (.51). 

ltaton 

A profile of factor means can then be made to provide a de- 

3_. 

visual picture of client satisfaction with services (See Figure 

scr!ptive 

1.1). 

4_ Similarly, a profile can be developed for any specific vari- 

Step 

e g. district, closure status, age, sex, disability, etc. 

able, 

-22- 

Mean 

.8O 

.90 

.18 

44 

21

WORK SHEET

-24--

WORK SHEET

5. Tabulate responses by category for each question on the CM 

Step 

questionnaire and determine percentages of response for each--- 

Follow-up 

questlon. For exmple" 

1. Are you employed at this time? 

390 (65%) 

Yes 

210 (35%) 

No 

one-sample chl-square test described in Example 2 may then be 

The 

if an agency w'shes to know 1"f significantly more former clients 

applled 

employed than unemployed. The same procedure is used for each of 

are 

other questions on the CM Follow-up ..u..e.stionna're. 

the 

6. An agency may also wish to know if former clients perceived 

Step 

services similarly at closure and six months after closure. 


such a case, the product moment correlation procedures described in 

In 

4 could appropriately be used to determine the degree of agreement 

Example 

each of the nine factors between scores on the CMS and the CM Follow- 

for 

questionnaire. 

up 

7_. Rarely is there 100% response to questionnaires. This is 

Step 

true for mailed questionnalres. If the agency has some doubts 

especially 

whether the respondents are representative of the population, t is 

about 

to compare the characteristics of the sample and the population which 

wise 

consldered important by the agency for interpretation and declsion 

are 

making. 

example, if the agency wanted to know f the proportion of males 

For 

females was the same between the respondents (sample) and all closed 

and 

(population), a chi-square test for two independent samples could 

clients 

appropriately conducted to determine if there was a significant dif- 

be 

between the two groups. Procedures for calculating a chi-square 

ference 

for two independent samples are described in Example 3. 

test 

IV Interpretation 

general mean for scores on the Consumer Measurement Scale was .51. 

The 

the descrlptlve continuum of satisfaction/dissatisfactl'on levels as 

Using 

guide, this value is in the low positive range of satisfaction. Compar- 

a 

the profile of factor means with the continuum satisfaction/dissatis- 

ing 

faction levels ndicates that seven of the service dellvery related factors 

-26-

ORK SHEET

in the low satisfaction range and two factors were in the medlum sat- 

were 

range. Clients were most sat1"sf'ed wth "Counselor Effort in 

isfaction 

and least satisfied with "Speed of Service Delivery" and "Phys- 

Placement" 

Facilities". Addit1"onally, 65% of all closed clients were employed 

ical 

months after closure as reported on the Consumer Measurement Follow- 

six 

Qqestl onnaire. 

u_ 

-28-

Consumer Measurement Scale 

]ER'S MEASUREIVIEHT OF VOCATIONAL REHABILITATION 

CONSU 

1973, Wllllam G. Hllls, Ph.D., Harold D. Vlallle, Ph.D.) 

(Copyrlght 

you for your time and assistance Your response to tls questionnaire m|l bo kept conhdent, ai and will be 

Thank 

only to ,denhfy the strengths and weaknesses of rehabilitation services. 

interpreted 

complete the following 28 questions while asking yourself" 

Please 

satlshed am with this aspect of rehabilitation service'; 

How 

sat means am very Very satisfied 

means am satisfied 

Sat 

means can't decide whether am N. satisfied not or 

means am dissatisfied 

Dissot 

dlssat means am very Very dssatlsfled 

N A. means this item does not apply to me 

D 

Please place a check mark in the box that best explains how you feel about eeck statement. 

2. My counselor's w,||n,aess to listen to my ideas 

s. , 0,,,. ..,,o. 

s. . emp..,, .0. ... 

m. x,.d ...,.,.o, ,.--,-" 

. vo.,,... R.bo,,,,..', ab,,,y,.... ..,,,0. 

. Tbo..,. ,,.. 

'. ,*-,. pan.,.o , - ,ab 

S. tbo ',--,' '00 ,* o.' ,. ,.--,--. -*- 

20 The pers.nol treatment re{e,ved from ,bo med,{.l 

21. Tbo nomr of fos and omoo.t of pape¢wrk 

26. How my employment hts my mental and phys,cai abd,t,es 

PLEASE TURN THE PAGE AND COMPLETE THE FOLLOWING QUESTIONS 

Very 

Very 

t. N D,$st. Dist D 14 A. 

t. 

Very 

Very 

Sat. N DiJsat. Dinsat. D N A 

Sat.

D,sabhty 

l.ast grade completed n 

school 

training you have received 

Other 

Wh,te [] Negro [ 

Race 

Sex Male 

Present Job 

Years completed ,n 

ind,en O sp ,,sh 0 Other 0 

Referral referred myself to VR [ Someone else referred me 

Please answer each of the following 

college 

(1) Would you return to Vocational Rehabll,tatlon (VR) should you again have need of VR services Yes [-1 No 

Do you think the serwces prowded by VR helped you get a better lob than you would have found without VR services 9 

(2) 

NoJ--] 

J" 

Yes 

(3) In your contact wsth VR, did you ever in any way experience d, scrlmmatlen+ Yes J"] 

How do you think rehabd,tutlon services could be improved 

ADDITIONAl COMMENTS 

are interested n knowing what happens to people after rehabilitation services end. Would you be willing to 

We 

another questionnaire for us about six months from now 

complete 

Yes/"] No 

yes, please print your name and address so we can send you the follow-up questionnaire 

Name 

Address., 

Street Cty State Ztp Code 

Please return this questionnaire in the enclosed envelope to 

Rehabdtatlon Research Institute 

Reglono| 

University of Oklahoma 

The 

District Number

Measurement Fol 1 ow-up questl onnal re 

Consumer 

|E SURE , E IT OF VOCATIO AL REHABILITATIO 

I 

(Copyrzght, 1973, Wzllzam G. HzlIs, Ph.D., Harold D. Vzazlle, Ph.D.) 

Today's Date 

1. Are you employed at th,s t,me Yes No. 

Note: If no, please skip questions 2, 3, and 4. 

2. Are you worksng for the same employer you were six months ago 

Yes No 

you do,ng the same k,nd of work you were doing sx months ago 

Are 

No 

Yes 

4. Now sat, sfted are you wtth your present 

Sat, stied Sat, slued 

Very 

Dssatsfled 

Very 

5. How many months during the last sx have you been employed ? 

0 2 3 4 5 6 

6. How many lobs have )ou had ,n the last sx months 9 

0 2.........._3 4 5 

7. Have you again appl,ed for rehabilitation services 9 

Yes. No 

Neutral Dssatshed 

6 or more 

What else could the Vocational Rehabilitation program have done that would have been of help to you n hnd- 

8. 

or keeping su,table employment 9 

Ing

Ask Yourself How satisfied am with this aspect of rehabilitation services'; 

sat. means am very Very satisfied 

means am satisfied 

Sat. 

means can't decde whether am N satisfied not or 

means am dissatisfied 

Dssat. 

dlssat, means am very dssatsfled 

Very 

A. means this tem does not apply to me 

D.N 

Please place a check mark In the box that best explains how you feel about each statement. 

Choose an answer for all statements. 

IN MY EXPERIENCE WITH VOCATIONAL REHABIL.ITATION, THIS IS THE WAY FEEL. ABOUT 

9 The tme it took to get the serwces started 

10 Results of medical services 

11 The quality of traanlng receIved 

12 

The employment I now have 

13 My counselor's w, llmgness to listen to my adeas and suggestions.. 

14 The part my counselor played In actually helpIng me get my lob. 

15. Vocational Rehab, lltatlon's ab, ity to make decisions 

16 l:ase with which could enter the office 

17 Personal treatment received from Vocational Rehabilitation 

Very 

Very 

Sat. N Dissat. Dissat. 

Sat. 

E] 

DE]DE] 

DE] 0 CI 

CI 

n D ODD 

OOO O 

...0 

O O O CI 

O 

O []] O O 

l,l 

mOO[]] 

O 

..mOOD 

put this cluestonnalre n the attached enveiope and ma, to the Regional Rehabilitation Research 

Please 

Thank you for your cooperation 

Institute 

REHABILITATION RESEARCH INSTITUTE 

REGIONAL 

Un:vers,ty of Oklahoma 

The 

Questionnaire Number 

DI stnct Number 

D.N.

EXAMPLE 2 

A. General Procedure: Ex post facto study des±gn. 

B. Measurement Approach: Questlonnalre on cllent satsfact±on. 

C. Statistical Procedures: One-sample ch±-square test.

EXAMPLE 2 


B. Measurement Approach: Questlonnare on clent satisfaction. 

C. Statlstlcal Procedures: One-sample ch-square test.

Sample Evaluation Objectlve 

determine cllent satisfaction with rehab111tation services for 

To 

eligible cases closed durlng three fiscal years. 

all 

II Criteria 

of clients expressing satsfactlon with rehabllitation serv- 

Number 

after having received servlce. 

ices 

A. General Procedure7 



Steps. 

X servlce/treatment 

where 

0 = observation/measurement 

and 

Identlfy clients who have been closed as rehabilitated or 

1. 

for the past 3 fiscal years. 

not-rehabilitated 

2 Develop a questonnalre to determine cllent satsfactlon 

Mal survey to former cllents explaining the survey and the 

3. 

for them to respond. 

need 

Send a follow-up letter again seeklng cooperation to all cli- 

4 

who did not respond. 

ents 

D. T. & Stanley, J. C. Experimental and quasi-experimental 

7Campbell, 

for research Chicago" Rand McNally, 1966.-'-- 

designs 

-35-

Tabulate results and do the necessary statistics to make the 

5. 

meaningful. One possibility Cs a one-sample chi-square test e. 

results 

B. Measurement Approach- Questionnalre on client satisfaction 

way to learn about client satisfaction with services provlded by 

One 

rehabilitatlon agency is to use a questionnaire with former clients. 

the 

typical related questions taken from a questionnaire used by one 

Some 

VIII State rehabilitation agency are 

Region 

How much do you feel your counselor and services provided by the 

1. 

agency helped you 


Great deal 

Only a little 

Not at all 

Were you satisfied wlth the help the counselor gave you In flnd- 

2. 

a job 

ing 

Yes 

No 

3. On the average how often dd you see your counselor 

2. etc. 

Once 

Every couple of weeks 

About once a month 

Every couple of months 

Less than every couple of months 

Never 

S Nonparametric statistics for the behavioral sciences. 

eSiegel, 

York- McGraw-Hill Book Company, 1956, pp. 42-47. 

New 

-36-

ORK $ttEET

C. Statlstical Procedures 

1. Tabulate frequencies for the various response categorles for 

Step 

question as a first step in the chi-square analysis. For example 

each 

How much do you feel your counselor and servlces provlded 

1 

the rehabllitation agency helped you 

by 

78 Great deal 

Only a little 

16 

Not at all 

2 

the 101 questlonnaire respondents, 5 dld not respond to 

Of 

question. 

tbls 

Were you satisfied wlth the help tne counselor gave you in 

2. 

a 3ob' 

finding 

29 Yes 

19 No 

the 101 questionnaire respondents, 52 did not respond to 

Of 

questlon. Ths may indicate that the question did not 

this 

former clients adequate categories to respond, e.g. 

allow 

satisfied nor dissatisfied. 

neither 

3. On the average bow often dd you see your counselor? 

7 Once 

26 Every couple of weeks 

30 About once a month 

19 Every couple of months 

12 Less than every couple of months 

2 Never 

-38-

WORK SHEET

the 101 questionnaire respondents, 5 dld not respond to 

Of 

question. 

this 

2. Ut111ze the tabulated frequencies in the chi-square formula, 

Step 

is as follows. 

which 

: z (o ,E), 

x 

E 

=1 

degrees of freedom 

df=k-1 

/¢ = number 

where 

classification 

of 

categories. 

r. directs us to sum the resultant calculation for 

where: 

each category over all categories (/c). 

=1 

0 = the observed frequency in a gven category. 

= the frequency we would expect in a given category. 

E 

a one-sample chi-square problem such as ths, 

In 

is expected that there will be equal dstribu- 

t 

of frequencles for all categories. 

tion 

for the responses to Question Number I the observed and expected 

Thus, 

would look llke this 

frequencles 

0 E 

78 Great deal 

Only a little 

Not at all 

Step 3_. Apply the chi-square formula to the data. 

: . (o E} 

x 

E 

=1 

-40-

WORK SHEET

Number 1 How much do you feel your counselor and servlces 

Questlon 

by the reha'bTlitation agency helped you 

provided 

78 Great deal 

16 Only a little 

2 Not at all 

the calculations necessary for the chi-square formula, 

Performing 

results can be shown in the following way: 

the 

dr= /¢- 1 

=3-I 

=2 

E O-E (O-E) 2 (O-E) 2 

0 

E 

78 32 46 2116 66.13 ÷ Major contributor 

16 32 -16 256 8.00 

2 32 -30 900 28.13 

X = 102.26 

a table of crit1"cal values for X (found in statistics text- 

Using 

for 2 degrees of freedom, the necessary chi-square value for sig- 

books) 

is 5.99 at the .05 probabil1"ty level. Th's means that only 5 

nificance 

out of 100 would we expect a X 2 of 5.99 or greater to happen by 

times 

There is, of course, the possibility of error, but when we select 

chance. 

probability level, we do so to make a decision, one way or the other, 

a 

significance. The X value of 102.26 for question 1 exceeds a X 2 

about 

5.99, therefore the 9_for ths calculate'on is less than .05 (in fact, 

of 

it is less than .001). 

The calculations for Question Number 2 are done n the same manner. 

-42-

WORK SHEET

Number 2: Were you satisfied with the help the counselor 

question. 

you in f1"nding a job 

gave 

29 Yes 

19 No 

df=/-I 

=2-1 

=1 

0 E O-E (O-E) 2 (O-E) 2 

29 24 5 25 1.04 

19 24 -5 25 1.04 

using a table of critical values, the X 2 value for 1 degree of 

Again 

is 3.84 at the .05 probability level. Thls means that only 5 times 

freedom 

of 100 would we expect a X 2 of 3.84 or greater to happen by chance. 

out 

the X 2 value of 2.08 s not significant. 

Therefore, 

calculations for Question Number 3 are done in the same manner 

The 

questions I and 2. 

as 

Number 3. On the average how often did you see your counselor? 

q.u.estion 

Once 

7 

26 Every couple of weeks 

30 About once a month 

19 Every couple of months 

12 Less than every couple of weeks 

2 Never 

-44- 

X 2 

E 

-- 

2.08

WORK SHEET

0 

7 

26 

3O 

19 

12 

2 

df=k-1 

=6-1 

:5 

O-E (O-E) 2 (O-E) 2 

E 

E 

16 -9 81 5.06 

16 10 100 6.25 

16 14 196 12.25 ÷ Maor contrlbutor 

16 3 9 .56 

16 -4 16 1.00 

16 -14 196 12.25 ÷ Maor contributor 

X 

= 

37.37 

using a table of crit1"cal values, the X 2 value for 5 degrees 

Again 

freedom is 11.07 at the .05 probabClty level The X 2 value of 37.37 

of 

indicates a probabil1"ty of less than .05. 


can be seen by the obtained X for Question 1, signCficant differ- 

As 

in response frequency from the expected were obtained at the .05 

ences 

level because the obtained X 2 value exceeded 5.99. (In fact, 

probability 

obtained X 2 value exceeded the critical value of 13.82 at the .001 prob- 

the 

level.) As can be observed from the calculation for this question 

ability 

much do you feel your counselor and services provided by the rehabil- 

"How 

agency helped you" the greatest contributor to the chi-square 

itation 

(102.26) was the response to the category "great deal" (66.13). More 

value 

(78) indicated "great deal" than expected (32) and fewer than 

respondents 

indicated "only a 11"ttle" (16) or "not at all" (2). Ths indi- 

expected 

that former clients were very satisfied (significantly) with their 

cates 

counselor and the services provided. 

outcome was different in Questlon 2 where the responses were not 

The 

different than expected at the .05 probabllity level. For 

signlficantly 

-46-

WORK SHEET

question many clients did not respond to the item (52 of them). For 

this 

that did respond, there was not sufficient difference between the 

those 

observed and the expected frequencies to suggest sgnCficance. 

question 3, statistical slgnfCcance was attained at the .05 prob- 

For 

level, indCcating that there was a dl'fference n the number of 

ability 

clients were seen by their counselors. The most frequent category 

times 

about once a month w'th very few clients never seen by their counselor. 

was 

-48-

EXAMPLE 3 

A. General Procedure: Ex post facto study des±gn. 

Measurement Approach: State Agency's Case Servlce Report 

B. 

data. 

(R-300) 

Statlstcal Procedures: Ch-square test for two ndepen- 

C. 

samples. 

dent

EXAMPLE 3 

A. General Procedure: Ex post facto study design. 

Bo 

Co 

Approach: State Agency's Case Servlce Report 

Measurement 

data. 

(R-300) 

Procedures: Ch-square test for two ndepen- 

Statistical 

samples. 

dent

(r- 1)(c- 1) 

df 

(2 1)(2 1) 

= 

=1 

a table of critical values for X 2 for 1 degree of freedom, the 

Using 

chi-square value for significance is 3.84 at the .05 level. Thus, 

necessary 

the X 2 value of 68.66 is significant at the .05 probability level. 


can be seen by the obtained X for example 1 relative to diagnos- 

As 

and evaluation services for males and females, sgnlficance was not 

tic 

at the .05 probability level because the obtained X value (.41) 

reached 

not exceed 3.84. Th1"s Ind'cates that there is not a signflcant dif- 

did 

between males and females who were prov'ded diagnostic and evalu- 

ference 

ation services. 

contrast, the outcome for example 2 relative to business school 

In 

college training indicates that significance was reached at the 05 

or 

A X value of 68.66 is greater than the needed critical value of 

level. 

for statistical significance. Thus, differences do ex1"st between 

3.84 

number of men and women who are sent to business school or college. 

the 

of the major contributors to the X value indicates that cells 

Inspection 

and D offer the most to the X 2 total. Fewer men and more women than 

B 

be expected by chance alone (except 5 times out of 100) are sent 

could 

to business school or college for training. 

-62-

WORK SHEET

= (r- 1)(c- 1) 

df 

(2- )(2 ) 

= 

=1 

()() 

a table of critical values for X 2 for 1 degree of freedom, the 

Using 

chi-square value for sgnficance is 3.84 at the .05 probability 

necessary 

level. Thus, the X 2 of .41 n our example is not significant. 

calculations for the second serw'ce category "Business School or 

The 

are done in the same manner: 

College" 

SEX 

Male 

Female 

A) 

'C)' 

Business School or College 

NO 

130 

Yes 

180 70 

Ce I l 0 E O-E (O-E) 

A 130 100.80 +29.2 852.64 

B 10 39.20 -29.2 852.64 

C 50 79.20 -29.3 852.64 

D 60 30.80 +29.2 852.64 

-60- 

X 2 

(O-E) 

E 

= 

140 

110 

250 

8.46 

21.75 ÷ Major contributor 

10.77 

27.68 ÷ Major contributor 

68.66

WORK SHEET

3_. Calculate the expected frequencies for each cell. This is 

Step 

using the following formula- 

accomplished 

Total x Column Total 

Row 

Grand Total 

E 

Example. Calculate the expected frequency for Cell A. 

x 85 

140 

250 

E 

the expected frequency for each cell, the Diagnostic & 

Calculating 

table would look like this- 

Evaluation 

A) 

c) 

Diagnostic & Evaluation 

No 

50 

47.60 

the calculations necessary for the chl-square formula, the 

Performing 

can be shown in the following way- 

results 

I 1 0 E O-E (O-E) 2 (O-E) 2 

Ce 

E 

A 50 47.60 +2.40 5.76 .12 

B 90 92.40 -2.40 5.76 .06 

C 35 37.40 -2.40 5.76 .15 

D 75 72.60 +2.40 5.76 .08 

-58- 

Yes 

90 

X 2 .41

WORK SHEET

dlrects us to sum the resultant calculation for 

where. 

category over all categories (/

WORK SHEET

SRS-RSA.300 

Form 

7-74 

Revtsed 

Agency Code 

Number 

Case 

Last Name 

A 

37 

th, Sml 

Street and Number 

Address 

Wr ,qht 

I992 

ty County 

C, 

Mes a 

SUc Sac No 

A 

SSDI Status. 0 

B. 

OF HEALTH, EDUCATION, AND WELFARE 

DEPARTMENT 

and Rehabl,tation 5erv,ce 

ocmi 

Center for Sac,el Stat, shcs 

Natmnel 

D C 20201 

Washington, 

SERVICE REPORT STATE.FEDERAL PROGRAM OF VOCATIONAL REHABILITATION 

CASE 

(TO 8E RECORDED AT TIME OF FIRST REFERRAL) 

PART 

371-00-3406 

SSl Status 

C. 

eo,.. 'i 

D. 

Date Referral 

E. 

Completed 

Process 

10-1-71 

Months ,n Statuses 00-02 2 

F 

Spomsh Surname 

G 

Outcome of Referral Process 

H 

ACCEPTED Reason_.._. 

NOT 

F-] from (IX)), 2 !"1 from (02) 

FOR 

ACCEPTED 

[] Extended Evaluat,on (06) 

04 

05VR Surv,ces (10) 

Data of Closure 

A 

Referred by Sac Sec Adm,n 

B 

Suctal Security C] at m Type 

C 

Federal Special Program Identtf, cat,on* 

D 

TF VET MAW PO WIN [SEC4 

None 

X 

001 002 '004 010 020 040 '100 

000 

Cost for Case Services 

E 

All Serv, cos- Total 

Rehab, l,tat,on Fac, ltJes Total 

2. 

Socal Security Trust Funds Total 

3 

First Name Imtial C. Referral Date 8--20-7. D 

Mary R. 

JZ,p Code 

(code*) 

182222 

71 

E Ag. 3_.!_7 dab 8-29-33,, F 

G D,sab, hty as Reported (descrmbe) 

a Mes 

2 (TO BE RECORDED AT COMPLETION OF REFERRAL PROCESS) 

PART 

0 

9-29-73 

2 F"]Yes 

k-No, 

4. Supplemental security Income Funds Total 

Date Ext Eval Completed (,f appl,coble) 

F 

SSDI Status 

G 

IH 

SISI Status 

CLOSED FROM STATUS 00, COMPLETE 

IF 

3 PART 

CLOSED FROM STATUS 02, COMPLETE 

IF 

THRU 2T AND PART 3ON 

ITEMS21 

2 PAGE 

ACCEPTED FOR EXTENDED EVALUATION 

IF 

VR SERVICES COMPLETE ITEMS 2.1 

OR 

2.T DO NOT COMPLETE PART 3 

THRU 

THIS TIME, 

AT 

sabhng Condit,ons (descr,be) 

D, 

Orthopedic 

Molar 

mpal 

upper llmb 

Secondary 

2 

appl cable 

Not 

rment, 

Code 

C,d, 

340 

999 

PART 3 (TO BE RECORDED AT TIME OF CLOSURE) 

SF 

SD 

2e0 4o0 

(Dollars) 

1474 

status 1 

o,k 

eekly Earn,ngs $ 40 

J 

ubi,c Ass,stance I Type 0 

K 

SSI) |Monthly Amount $ 

(tnclud,ng 

qccupot, on (.tie) ervce food shop L. 

Code 

Number of Months on Agency Rolls 

M. 

Extended Evaluahon (Status 06) 

In 

From A©cptance to Closure (Statuses 10-24) 2]. 

2 

In Tm,nmg (Status 18) 

3. 

4 Ready For or In Employment (Statuses 20-22) 

*These ,terns ore to be coded 

STATE AGENCY COPY 

3118 

42 

ct Code 

Dstr, 

lJm 

Referral Source 

Approved 

Form 

t¢0. 83-R0040 

OMB 

Sex [] Male, 2 [] Femai- 

Cod, 

699 

Prev,ous Closure wmthn 36 Months 

J 

C-Jl, Yes-Outcome Rahab. I- 2 

No 

Rehab [--3 

Not 

Yes-Months Stnce Last Closure.___._._ 

If 

Marital Status 

K 

Number of Dependents 

L 

Total Number ,n Family 

M. 

Highest Grade Completed 

N 

Work Status 

0 

Weekly Eam:ngs 

P 

Total Monthly Family Income 

Q 

earmngs) 

(,ncludng 

Pubhc ( Type 

R / Me Amt $ 

Assistance 

SSI) Time on P A 

(,ncluding 

Primary Source at Support 

S 

T Type of Institutmn 

Outcome of Extended Evaluation or VR 5erv,ces 

N 

Closed from Extended Eveluahon (Status 08) Reason 

[ 

Closed Rehabl,tated (status 26) 

2 

[] Closed Not Rehab,htoted (status 28) Reason 

3 

[] Closed Not Rahab, htated (Status 30) Reason 

4 

Serwces Provtded 

O 

of Service Provtded 

Type 

Arranged for by Agency 

or 

'D;ag'nst;c a.d 'Evaiuot,0n 

10 

Restorahon (Phys,cai or Mental) 

11 

12 T 

1611 

18 

G 

or Umvers,ty 

College 

cedem',c' (EIe. or MS) 

Other 

School or College 

Busme.ss 

School 

Vocatmnal 

On-the-Job 

& Voc Adlustment 

Personal 

'Mscellaneous 

Mm ntanance 

19 

Other Services 

20 

21 Services to Other Family Members 

P 5ate Agency Specml Program Identsf, cahon* 

None 

X 

Cost 

Status 

000 001 002 1,004 010 020 040 100 '200 

9-30-73 

of Report Counselor Sgrrature and 

Date 

i 

5 

12 

8 

000 

000 

serv, ca 

No 

th agen. 

W, 

only 

cost 

W,thout 

cost 

agency 

only 

and 

W,th 

thout 

w 

agency c 

SUM 

91 

Code 

L

WORK SHEET

B. Measurement Approach. State Agency's Case Service Report (R-300) data 

this example, the evaluator utilized case information from the 

For 

category entitled "Type of Service Provided or Arranged for by the 

R-300 

The lst of var'ous categories can be found on the R-300, Part 3, 

Agency." 

O, 10-21. (See sample SRS-RSA-300 form on next page.) 

Section 


1. Tabulate frequencles for the various categories according 

Step 

whether services were provided or were not provided for men and for 

to 

women. A convenl'ent way to record results is as follows- 

of service provided/arranged Male (N=140) Female (N=110) 

T_ 

b_z th__e agency No Yes No Yes 

fo___r 

1. Diagnostic & Evaluation 50 90 35 75 

2. Buslness School or College 125 15 70 40 

Two service categorles are included for demonstration purposes. 

2_. Utilize the tabulated frequencles n the chi-square formula, 

Step 

is as follows1: 

whlch 

i=1 

E 


df = (r- 1)(c- 1) 

r number of rows 

where 

c number of columns 

and 

In the special case of a 2 x 2 table, it is possible to cal- 

IZNote 

the X 2 from the raw data wthout calculating the expected values. 

culate 

formula for thls purpose offered by Ferguson, G. A. Statstlcal analy- 

A 

n psychology and education. New York McGraw-Hill, 1966, p 204 is- 

sis 

BC) 2 

Xe N(AD 

(A + B)(C + D)(A + C)(B + D) 

= 

in a 2 x 2 table if any of the expected frequencies are less 

Also, 

5, a Yate's correction for continuity may be applied. Ths results 

than 

a slightly more conservative X 2 value. In the case of very small ex- 

in 

frequencies, a FCsher exact probability test may be used (Ferguson, 

pected 

1966, pp. 206-210). 

-52-


determine differences for male and female clients in servlces pro- 

To 

or arranged for by the agency during the rehabilitation process. 

vided 

II Criteria 

of clients (by sex) who are provided services in any given 

Number 

category 

service 

A. General Procedure 

Ill Methodology 


Steps. 

X : service/treatment 

where 

0 : observation/measurement 

and 

Identify clients who bare been provided services and have 

1. 

closed during two fiscal years. Divide them nto two groups, 

been 

male and female. 

From the information relatlng to servlces contained on the 

2. 

agency's case service report (R-300), determine if services 

State 

were provided or not. 

3. Tabulate results for each group, 

Conduct necessary statistics to make results meaningful 

possi lity l's a chi-square test for two independent samples I°. 

b4 

D. T. & Stanley, J. C. Experimental and quas-experimental 

9Campbell, 

for research. Chicago- Rand McNally, 1966. 

designs 

S. Nonparametric statistics for the behavioral sciences. 

10Siegel, 

York- McGraw-Hill, 1956, pp 104-111. 

New 

-51- 

One

EXAMPLE 4 


Measurement Approach: Follow-up survey of rehab- 

B. 

cases. 

±ltated 

Statistical Procedures: Product-moment correlation 

C. 

coefflc±ents.

EXAMPLE 4 


Measurement Approach: Follow-up survey of rehab- 

B. 

cases. 

llltated 

Statlstcal Procedures: Product-moment correlation 

C. 

coefficients.

I Sample Evaluation Objective 

determine the relatlonships among some client and servlce varia- 

To 

(age, number of counselor contacts, months of employment, and weekly 

bles 

for clients closed as rehabilitated (Status 26) for one fiscal 

Income) 

year. 

I I Criterl a 

of relatlonship between varlables for clients successfully 

Degree 

I itated. 

rehabi 

A. General Procedure 12 

I I I Methodology 

Ex post facto study deslgn XO 

Steps. 


where 


and 

Identify clients who have been closed rehablltated (Status 

1. 

during one fiscal year. 

26) 

Develop a survey instrument to obtain the needed data con- 

2. 

variables you wish to investigate. 

cernlng 

3. Conduct survey by mat1, phone, or personal contact. 

D. T. & Stanley, J. C. E..xperimental and quasl-experlmental 

12Campbell, 

for research Chlcago: Rand McNally, 1966. 

designs 

-65-

Tabulate results and do necessary statstlcs to make the 

4. 

meanlngful. One ossib111ty Is to conduct product-moment 

results 

correlatlon coefficients I 

B. Measurement Approach Fol low-up survey of rehab 11 tared cases 

way to obtain the needed nformaton is to adminlster a question- 

One 

to former clients relatlve to their present employment and income. 

nacre 

Some questlons that might be asked are: 

How many months have you been employed snce rehablitatlon 

1. 

were terminated 

services 

2. What is yur present weekly ncome in dollars? 

addltional method of obtaining information is from the former cli- 

An 

case record. Some questions for which data would be available are: 

ent's 

Age of client at closure . 

I. 

Number of contacts made by the counselor durlng the rehabl- 

2. 

tatlon process ) 


1_. Tabulate the obtained information for each followed-up cli 

Step 

For example, if data was collected for 5 clients, it would be pre- 

ent. 

sented n the following format: 

of # 

Months of Weekly 

Counselor 

Cllent Age Contacts EmRloymept Income 

i 23 3 I 35.00 

2 29 8 6 60.00 

3 33 10 15 75.00 

4 42 7 25 80.00 

5 53 9 18 55.00 

G. A. Statlstcal analysls In psychology and education. 

13Ferguson, 

York McGraw-H II, 1966, pp. 105-115. 

New 

-66-

WORK SHEET

2. Calculate the correlation coefficient 14 for each combination 

Step 

variables (age, number of counselor contacts, months of employment, and 

of 

income) to determine degree of relationships. For example, to de- 

weekly 

the relationsh1"ps between age and @ncome, the follow'ng procedure 

termine 

could be util ized. 

Age 

Client 

X 

Example i 

of the Correlation Coefficient 

Calculatlon 

Age and Weekly Income 

for 

Income 

Weekl}, 

X 2 Y2 XY 

Y 

23 35. O0 529 1225. O0 805 

29 60.00 841 3600.00 1740 

33 75. O0 1089 5625.00 2475 

42 80. O0 1764 6400. O0 3360 

53 55. O0 2809 3025. O0 2915 

zX 180 7.X 305.00 7.X 7032 .Y 19875.00 XY 11295 

this instructs us to do is to identify age as the X variable 

What 

income as the Y variable. You then square each X value and enter the 

and 

under X . The same procedure is followed for the Y variable. To 

result 

XY we simply multlply each X by its corresponding Y value to obtain 

obtain 

XY. After you have done this, then sum each column. 

Utilize the previously calculated values to calculate the 

3_. 

correlation coefficients (r). The formula is: 

product-moment 

zXzY 

Ny.XY- 

=V f lNzX (zx)2l INzY (zY) I 

r 

In this example X represents the square of the value of X; 

14Note: 

chi-square. 

not 

-68-

WORK SHEET

where N total number of cases 

.XY : sum of the products of X tlmes Y 

sX sum of X values 

sY sum of Y values 

sX 2 sum of each X value squared 

sY2 sum of each Y value squared 

you are ready to substitute the prevlous obtained data into the 

Now 

a- 

formul 

180(305) 

..5(11295) 

=/(5 r 7032 1802 (5 x 19875 3052 

x 

54900 

_56475 

32400) (99375 93025) 

=/(35160- 

54900 

:(2760) 

56475 

(6350) 

1575 

17526000 

:/ 

1575 

4186.41 

determine the significance of r, one can use a table of cr%tical 

To 

found n statistics textbooks (a sample 1"s glven In Table 2 of the 

values 

To use the table, determine the degrees of freedom (df) 

Introduction). 

find the r value wh#ch Cs needed for sign1"ficance at the .05 probabil- 

and 

level. In th's case df N 2 or 5 2 3. For 3 df a critical 

ity 

of .88 or higher is necessary for a sgn1"ficant correlation. Thus, 

value 

.38 is not a statist'cally sign'f1"cant r at the .05 probability level. 

-70-

WORK SHEET

Cllent 

1 

2 

3 

4 

5 

of 

Months 

I oymen t 

Emp 

X 

1 

6 

15 

25 

18 

zX 65 

2 

Example 

of Correlation Coefficient for 

Calculation 

Months of Employment and Weekly Income 

Weekly 

Income 

y X 2 y2 

35.00 1 1225.00 

60.00 36 3600.00 

75.00 225 5625.00 

80.00 625 6400.00 

55.00 324 3025.00 

.Y = 305.00 zX 2 

sXzY 

NsXY 

:/INzX 2 (.x)l IN.Y 2 (sY)21 

r 

) 65(305) 

514510 

x 1211 652 (5 x 19875 3052 

"h/(5 

19825 

22550- 

(6055 4225) (99375 93025) 

" 

2725 

(1830) (6350) 

=,- 

2725 

11620500 

2725 

3408.89 

-72- 

XY 

35 

360 

1125 

2000 

990 

= 1211 sY 2 = 19875.00 .XY = 4510

3 df the r must equal or exceed 88 to be signlficant at the .05 

For 

level. Thus, there is not a statistically significant corre- 

probability 

lation between months of employment and weekly income. 

procedure Is followed for all comparisons. The addtlonal cor- 

This 

coefficients were calculated and results are presented in the 

relation 

correlation matrix table. This table allows us to conveniently 

following 

the inter-correlation of each of the four variables. 

examine 

Age 

of Counselor 

# 

Contacts 

of Em- 

Months 

I oyment 

p 

Weekly Income 


df:N-2 

=5-2 

=3 

Correlation Matrix Table 

of # 

Months of Weekly 

Counselor 

Contacts Employment Income 

.55 .79 .38 

.55 69 

a correlation coefficlent table (found in statistics textbooks) 

Using 

3 degrees of freedom, the necessary critical value of .88 was not 

for 

for a correlation coefficient significantly different from a zero 

reached 

at the .05 probability level for any of the relationships. 

correlate'on 


can be observed in the correlation matrix table, the highest cor- 

As 

found was between months of employment and weekly 1"ncome (.80). 

relation 

-74- 

.80

WORK SHEET

indicates a relatively high relationship between these two variables 

This 

O0 would be a perfect correlation) However, because of the small num- 

(1 

of subjects (5) for thls analysis, it as not high enough to be sig- 

ber 

different than a zero correlation. (.00 or no relationship at 

nificantly 

at the .05 probabllity level. Thus, we could not say that this rela- 

all) 

is significant and would occur by chance only 5 times out of 100 

tionship 

none of the other correlation coefficients were statistically 

Likewise, 

significant 

a correlation is not significantly different from zero, then sta- 

If 

t must be assumed that either no relatonshlp exists between 

tlstically 

or that the sample was not large enough to detect the difference. 

variables 

our example with 5 cases, a correlation coefficient of .88 is needed to 

In 

statistically significant. On the other hand, if we had 30 cases, a 

be 

of .36 would be significant at the .05 level, and with 100 

coefficient 

a coefficient of 20 would be significant. Whether a correlation 

cases 

statistically significant or not, the size of the correlation suggests 

is 

degree of relationship. Thus, correlations may also be viewed in terms 

the 

their practical meaning Borg & Merideth (1968) offer some general 

of 

for interpreting simple correlation coefficients assuming a 

guidelines 

of 100 or more subjects. According to Borg & Meredith Is, correla- 

sample 

ranging from .20 to 35 show a very slight relationship between 

tions 

and may have very limited meaning. Correlations from .35 to 65 

varlables 

crude predictions for groups but are of limlted value for ndl- 

provide 

dual predictions Correlations ranging from .65 to .85 are good for 

w 

group preductions, e g predicting the proportion of successful 

making 

Correlations over 85 indicate a very good relatonshlp between 

clients 

two variables and are useful for either groups or ndlwdual predictions 

the example the correlation coefficients between months of employ- 

In 

and age (.79), months of employment and weekly ncome (.80), and num- 

ment 

of counselor contacts and weekly ncome (.69) are good for making group 

ber 

The correlations between the number of counselor contacts 

predictions. 

age (55), number of counselor contacts and months of employment (.55), 

and 

weekly income and age (38) provide only crude predictions for groups. 

and 

of the correlation coefficients In our example were posltve. 

All 

suggests, for example, that the higher the months of employment of 

This 

clients, the higher the weekly income, the hgher the number of 

former 

contacts, the higher the weekly income of former clients, etc 

counselor 

the correlatlons were negative, then they would suggest that the hgher 

If 

months of employment, the lower the weekly Income. 

the 

a correlatlon coefficient does not reflect a cause-effect rela- 

Though 

it does suggest the possibillty of one. It can be used as an 

tionship, 

ndicator for improving service dellvery. 

W. R. & Merideth, D. G. Educational research- An introduc- 

ISBorg, 

(2nd Ed.) New York David McKay Company, Inc., 1963, pp. 357-360. 

tion. 

-76-

EXAMPLE 5 


B. Measurement Approach: Case Dlff!culty Index. 

C. Statistical Procedures: One-way analyss of variance.

EXAMPLE 5 


B. Measurement Approach: Case Dlffculty Index. 

C. Statistical Procedures: One-way analyss of varlance.


compare effectiveness of rehabilitatlon counselor performance in 

To 

rehabil itation offices based on the number and case dCfficulty of 

three 

successful closures. 

II Criteria 

effectiveness of rehabilitation counselors-in three 

Servcedelivery 

as determined by Case Di.ffculty Index scores for rehabilitated 

offices 

cases (Status 26). 

A. General Procedurel 6 



Steps. 


where 

0 = observation/measurement 

and 

Identify all clients closed as rehab11tated (Status 26) for 

1. 

in a given office. 

counselors 

Determine the service difficulty of each case by uslng the 

2. 

Difficulty Index. (CDI). 

Case 

Determine the Total Weighted Closure Index (TWCI) by adding 

3. 

CDI values for a gven counselor. 

all 

Divlde the TWCI by the number of successful closures for a 

4 

counselor to obtain the Average Weighted Closure Index (AWCl). 

given 

D. T. & Stanley, J. C. Experimental and quas'-experlmental 

16Cmnpbell, 

for resear.c.h. Chlcago Rand McNally, 1966. 

desl.qns 

-79-

Conduct necessary statlstlcs to make the results meanlngful 

5 

possibility Is a one-way analyss of variance test zT. 

One 

B. Measurement Approach Case Dfficult.y Inde___x 18 

Case Dffculty Index (CDI) was deslgned to expand and complement 

The 

number of successful closures as a measure of counselor performance 

the 

1972). In doing so, the CDI contains two additional dlmensons 

(Sermon, 

counselor performance. The first of these is the Total Weighted Clo- 

of 

Index (TWCl). Sermon (1972) defines the TWCl as a "quantity-tempered- 

sure 

measure of a counselor's performance and represents total 

wth-dfflculty" 

counselor effort. The second index is the Average Weighted Closure Index 

This reflects the average difficulty of all cases successfully 

(AWCl). 

by a counselor regardless of the actual number of cases closed 

closed 

index Is Intended to provide a quality dimension for counselor per- 

Thls 

The Cas___e Diffcu!t.y Index, then, allows us to look at counse- 

formance. 

performance n terms of cases successfully closed (quantity), the 

lor 

counselor effort represente by these closures (work effort), and 

total 

average difficulty of successfully closing these kinds of cases 

the 

(quality). 

Case Difficulty Index formula 

CDI (1 000 P) x 10 

CDI is the Cas___e Dfficulty Inde_.._x for 

where 

given impairment (represented 

a 

by 4). 

is the rehablltatlon success 

P 

experienced nationally 

rate 

way to express the same thing is CDI O x 10, where O is 

Another 

rehabilitation failure rate experienced nationally with an impairment 

the 

G A. Statlstlcal analysls in psychology and education 

17Ferguson, 

York McGraw-Hill, 1966, pp. New 21-297 

D. T. The dfficult>, index an expanded measure of coun- 

18Sermon, 

performance. St.---Paul" Minnesota Division of Vocational Rehablli- 

selor 

Research Monograph 1, 1972. 

tatlon, 

-80-

WORK SHEET

CDI; is then determined by taking the percentage of fallures 

category. 

28/30) for each impairment and multlplyng it by 10. 

(Status 

CDI indices (Table 3) on the next 4 pages are taken from natlonal 

The 

for FY 68, 69, and 70 as reported by the Statistical Analysis 

statistics 

Systems Branch of the Rehabilitation Services Administration in the 

and 

S. Department of Health, Education, and Welfare. For local comparisons 

U. 

agency could develop a similar scale based on local norms within a 

an 

or region. 

State 


1_. Convert the number of rehabilitated case closures to a Case 

Step 

Inde____x Score (See CD__.I table). For example 

Difficulty 

Counselor #1 

Disability 

hearing impair- 

Other 

ment 

220-229) 

(Disability Code 

and unspeci- 

Benlgn 

neoplasms (Dis- 

fied 

11 ty Code 609) 

ab 

retardat on- 

Mental 

(Disability 

m11d 

Code 530) 

of suc- 

No. 

closures CDI 

cessful 

.94 

.90 

1.96 

TWCI 

x .94 = 4.70 

5 

4 x .90 = 3.60 

1 x 1.96 = 1.96 

TWCI 

: 

AWCI 

10 FTTI 

10.26 

ths counselor has 10 successfully rehabilitated cases, a Total 

Thus, 

Closure Index of 10.26 and an Average Weighted Closure Index of 

Weighted 

1.03. 

-82-

WORK SHEET

IMPAIRMENTS 

VISUAL 

Both Eyes (No Light 

Blindness 

rcepti on 

Pe 

Both Eyes (Light 

Blindness 

Perception) 

i ndness One Eye Other Eye 

Bl 

Defective 

One Eye Other Eye 

Bllndness 

Good 

Visual Impal rments 

Other 

IMPAIRMENTS 

HEARING 

Unable to talk 

Deafness- 

-Able to talk 

Deafness 

Hearing Impairments 

Other 

IMPAIRMENTS 

ORTHOPEDIC 

3 or more limbs 

Involves 

re Body 

Ent 

1 upper and 1 lower 

Involves 

imb 

1 

1 or both upper llmbs 

Involves 

1 or both lower limbs 

Involves 

Ill Defined Impairments 

Other 

OR AMPUTATION OF MAJOR AND 

ABSENCE 

MEMBERS 

MINOR 

of at least 1 upper and 1 

Loss 

major extremity 

lower 

of both major upper 

Loss 

ti es 

extreml 

of 1 major upper extremity 

Loss 

of 1 or both major lower 

Loss 

t es 

extreml 

of other unspecified parts 

Loss 

TABLE 3 

CASE DIFFICULTY IBDEX 

FY FY FY 

Disability 

1968 1969 1970 

Code 

100-109 1.96 1.91 1.97 

110-119 1.70 1.84 1.75 

120-129 1.54 1.76 1.69 

1.55 1.59 1.37 

130-139 

1.44 1.41 1.21 

140-149 

I. 80 I. 86 1.82 

200-209 

1.10 1.18 1.18 

210-219 

220-229 .98 .99 .85 

300-319 2.79 2.99 2.75 

2.76 2.96 2.75 

320-339 

2.17 2.26 2.17 

340-359 

2.11 2.26 2.06 

360-379 

2.71 2.74 2.60 

380-399 

400-409 1.61 1.77 1.84 

1.50 1.68 2.15 

410-419 

1.36 1.67 1.51 

420-429 

1.55 1.68 1.51 

430-439 

2.12 2.49 1.94 

440-449 

-84- 

Dl ffl cul ty 

Case 

(Composite 

Index 

for the 3 years) 

1.94 

1.77 

1.67 

1.50 

1.35 

1.83 

1.16 

.94 

2.85 

82 2 

2.20 

2.15 

2.68 

1.73 

1.79 

1.51 

1.58 

2.20

PSYCHONEUROTIC, AND PER- 

MENTAL, 

DISORDERS 

SONALITY 

c 

Psychotl 

c 

Psychoneurotl 

Alcoholism 

Addiction 

Drug 

Character, Personality, and 

Other 

oral Disorders 

Behavl 

Retardatl on Mlld 

Mental 

Retardation Moderate 

Mental 

Mental Retardation Severe 

DISABLING CONDITIONS 

OTHER 

NOT KNOWN OR NOT APPROPRIATE 

ETIOLOGY 

FROM NEOPLASMS 

RESULTING 

from Malignant 

Colostomes 

I asms 

Neop 

from Mal gnant 

Laryngectomles 

I asms 

Neop 

and Aleukema 

Leukemia 

Mali gnant Neoplasms 

Other 

and Unspecified Neoplasms 

Benign 

ENDOCRINE SYSTEM, 

ALLERGIC, 

AND NUTRITIONAL DISEASES 

METABOLIC 

Fever and Asthma 

Hay 

A1 lergies 

Other 

abetes Mel i tus 

D 

Endocrlne System Dsorders 

Other 

tam nos s and Other Metabol c 

Aw 

seases 

D1 

OF THE BLOOD AND BLOOD 

DISEASES 

ORGANS 

FORMING 

I i a 

Hemoph 

and Other Dseases 

Anemia 

SPECIFIED DISORDERS OF THE 

OTHER 

SYSTEM 

NERVOUS 

lepsy 

Ep 

DIsorders 

Other 

TABLE 3 (Continued) 

sabi ty FY FY FY 

Di 

1968 1969 1970 

Code 

3.37 3.58 3.23 

500 

2.41 2.55 2.23 

510 

3.55 4.04 3.81 

520 

5.69 4.94 4.89 

521 

3.33 3.47 3.20 

522 

2.07 1.96 1.90 

530 

2 31 2.22 2.24 

532 

2.88 2.71 2.61 

534 

600 2.77 2.14 2.74 

4.31 3.47 3.25 

601 

5.56 

602 

2.30 

605 

.94 94 .82 

609 

2.03 2.13 1.98 

610 

1.84 1.91 1.65 

611 

2.21 2.20 1.94 

614 

1 46 1.47 1.18 

615 

619 3.19 2.33 2.18 

2.43 2.27 2.36 

620 

2.33 2.95 2.52 

629 

2.76 2.76 

630 

639 

-85- 

2.62 

D ff cul ty 

Cas____e 

(Compos te 

Index 

the 3 years) 

for 

3.40 

2.39 

3.84 

5.09 

3.33 

1.96 

2.26 

2.73 

2.48 

3 67 

.90 

2.05 

1.79 

11 2 

1.38 

2.54 

2.36 

2.62 

2.72

AND CIRCULATORY CONDITIONS 

CARDIAC 

Heart Disease 

Congenltal 

Fever and Chronic 

Rheumatic 

Heart Disease 

Rheumatic 

and Degenerative 

Arteriosclerotic 

DIsease 

Heart 

Diseases and Conditlons of 

Other 

Heart 

the 

Heart Disease 

Hypertensive 

Hypertensive Disease 

Other 

Veins and Hemorrhoids 

Varicose 

Conditions of the 

Other 

Circulatory System 

DISEASES 

RESPIRATORY 

ratory Tubercul os s 

Resp 

Emphysema 

and Aspestosis 

Pneumoconoiosis 

ectasl s 

Bronchi 

Bronchltis and Snusltus 

Chronic 

Dseases of Respiratory 

Other 

System 

OF THE DIGESTIVE SYSTEM 

DISORDERS 

of the Teeth and 

Conditions 

Structures 

Supporting 

of Stomach and Duodenum 

Ulcer 

Enteritis and Ulceratlve 

Chronic 

Colitis 

a Hernl 

other than from 

Colostomes 

gnant Neoplasms 

Mal 

Conditions of Digestlve 

Other 

Sys tern 

GENITO-URINARY SYSTEM CONDITIONS 

TABLE 3 (Continued) 

D ffl cul ty 

Case 

FY FY FY Index (compos te 

Disability 

Code 1968 1969 1970 for the 3 years) 

640 2.50 2.56 2.15 2 41 

641 2.68 2.75 2.40 2.62 

642 3.31 3.67 3.26 3.42 

2.97 2.93 3.01 2.97 

643 

3.17 3.41 3.31 3.30 

644 

2.56 2.40 2.53 2 51 

645 

1.38 1.25 1.16 1.27 

646 

649 2.82 2.74 2.48 2.68 

2.97 3.31 2.94 3.08 

650 

3.58 3.88 3.66 3.71 

651 

4.82 2.80 2.73 3.88 

652 

2.68 3.00 2.54 2.76 

653 

654 2.74 2.66 2.28 2.56 

659 2.60 2.41 2.64 2.55 

.73 .77 65 .71 

660 

1.81 1.71 1.58 1.70 

661 

1.26 2.40 2.20 2.28 

662 

1.05 1.14 .99 I. 06 

663 

664 1.87 1.96 1.49 1.76 

669 1.31 1.19 1.12 1.20 

670 .99 1.01 .92 .97 

-86-

IMPAIRMENTS 

SPEECH 

Palate and Harelip 

Cleft 

TABLE 3 (Contlnued) 

Dl ffl cul ty 

Case 

lty FY FY FY Index (Composite 

Dsabl 

1968 1969 1970 for the 3 years) 

Code 

Imperfections 680 1.32 1.72 1.39 1.47 

Speech 

and Stuttering 682 1.92 1.75 1.53 1.74 

Stammering 

from other than 

Laryngectomies 

i gnant Neoplasms 684 1.90 3.73 3.09 2 97 

Mal 

from Intercranial 

Aphasla 

Embolism or 

Hemorrhage, 

(Stroke) 685 3.50 3.2 2.78 3.16 

Thrombosis 

Speech Impairments 689 1.92 2.20 1.63 1.91 

Other 

DISEASES AND CONDITIONS 

DISABLING 

ELSEWHERE CLASSIFIED (N.E.C.) 

NOT 

Skin and Cellular Tssue 690 2.05 2.04 1.73 1.93 

Of 

(N.E.C.) 699 2.17 2.19 2.24 2.21 

Other 

from Duane T. Sermon, Minnesota D1wson of Vocatlonal Reha- 

Recelved 

St. Paul, Minnesota. 

b11taton, 

-87-

Counselor 

three counselors in each of three offices then, the CDI mlght 

For 

like this 

look 

Office #I 

Rehab 

TWCI AWCI 

Closures 

10 10.26 1.03 

9 11 36 1.26 

10 12.10 1.21 

29 33.72 1.17 

Office #2 

Rehab 

TWCI AWCI 

Closures 

5 23.76 4.75 

4 18.61 4.65 

5 19.00 3.80 

14 61.37 4.40 

Rehab. 

l os u res 

C 

9 

I0 

9 

28 

Office #3 

values 1.17, 4.40, and 1.31 represent AWCI means for the three 

The 

in each of the three offices. 

counselors 

2. Calculate F-test for a one-way ana]ysls of varlance. The 

Step 

tests the difference between the means of more than two groups. (In 

F 

TWCI AWCI 

12.00 1.33 

Ii.01 i. I0 

13.50 1.50 

36.51 1.31 

case the means of the AWCI for the three rehabilitation offices under 

this 

The F is represented by the ratlo of variance between groups 

evaluation) 

our example there are three groups) and the variance within groups. 

(in 

the variability between group means s large enough in comparison with 

If 

variability withln groups, a statistically significant F will result 

the 

we can infer that the groups are different The F-ratio is expressed 

and 

as 

follows- 

F = 

square between Is 

Mean 

square within 

Mean 

arrive at an F value, the following procedures should be followed 

To 

would look like ths for the AWCI comparisons. 

and 

#1 Office #2 Office #3 

Office 

X2 X3 

X1 

1.03 4.75 1.33 

1.26 4 65 1.10 

1.21 3.80 1.50 

G. A. Statis.tica.l analysis n psycholog}, and education 

19Ferguson, 

York- McGraw-Hill, i966, pp. 281-297. 

New 

-88-

WORK SHEET

Squaring each X value and surnaming the columns results In the following- 

#1 Office #2 Office #3 

Office 

XI 2 X2 X22 X3 X3 

Xz 

1.03 1.06 4.75 22.56 1.33 1.77 

1.26 1.59 4.65 21.62 1.10 1.21 

1.21 1.46 3.80 14.44 1.50 2.25 

3.50 4.11 13.20 58.62 3.93 5.23 

3 3 3 

n# 

3.50 13.20 3.93 

T# 

1.17 4.40 1.31 

[ 

nj 

. Xi# 2 4.11 58.62 5 23 

T z 

4.08 58.08 5.15 

N=9 

T = 20.63 

T/N 

425.60 

9 

47.29 

nj 

. Xi# 67 96 

z 

=i i=I 

k 

j=l n 

where the number of counselors in each column (office) 

total number of counselors In the evaluation 

67.31 

total of the AWCI in each column (e.g. in office #1" 

T# 

1.03 + 1.26 + 1.21 or 3.50) 

T# 

total of T#'s for all three offlces (e.g. T 3.50 + 

T 

+ 3.93) 

13.20 

# T_#_ or the average of the AWCI's for each office (e.g. 

: : 

-90-

WORK SHEET

the previous calculated T value squared dlvlded by the 

T2/N 

number of counselors. 

total 

T2/N 20.63 x 20.63 425.60 

9 = 9 47 29) 

(e.g. 

the sum of all three sXij2 (e.g. 4.11 + 58 62 + 5 23) 

s.Xij2 

T 

2 

take the value of T# for each office and multlply it by 

-#--= 

its own value (square it). Then divide by n# for each 

n# 

zT_= 

col umn. 

(e.g. for office #1 

T 3.50 x 3.50 12.25 4.08) 

j 3 3 

the three values calculated for and add them 

take 

together 

3. Using the values calculated in step 2, you can now deter- 

St 

the sum of the squares for between, within, and the total. The for- 

mlne 

and calculation would look like this. 

mulas 

Sum of Squares Between 

Degrees of freedom k 1 

:2 

3-1 

-92- 

k 

s 

T# 

#:1 T 2 

67.31 47.29 

12o,021 

where- k number of groups

WORK SHEET

of Squares Within s s X{# 2 

Sum 

j=l 

Degrees of freedom N k 

Sum of Squares Total 

:9-3 

=6 

Degrees of freedom N I 

=9-1 

--8 

-94- 

= 67 96 67 31 

k 

. 

i=I 

#=1 

67.96 47.29 

k 

s 

#=1 

T# 2 

N : total number of 

wheresubjects 

k number of groups 

If the sum of squares between 

120.,671 

within are added together, 

and 

value will equal the sum 

thelr 

squares total (Sum of 

of 

Total Sum of Squares 

Squares 

Between + Sum of Squares W1thln)

WORK SHEET

4_. Fill in the Summary Table and do the necessary calculatlons 

Step 

obtain an F. The Summary Table would appears as follows before cal- 

to 

culations are conducted. 

Source of varlation 

Between 

Within 

Total 

Summary Table 

Degrees of freedom 

N-1 

Sum of squares Mean squares 

the prewously calculated nformation in the summary 

Substituting 

results in the followlng- 

table 

Source of varation 

Between 

Within 

Total 

*p < .05. 

Summary Table 

Degrees of freedom Sum of ,squares 

20.02 

.65 

20.67 

B 

SS 

k-1 

SS 

W 

N-/< 

Mean Squares, 

10.01 

obtained F value for our example Is 91.00. Using a table of 

The 

values for F tests (found in statistics books) for 2 and 6 degrees 

critical 

-96- 

.11 

F 

B MS 

MS 

W 

91.00"

WORK SHEET

,"reedom, the necessary F value is 5.14 at the .05 probab111ty level. 

of 

means that only 5 times out of 100 would we expect a F of 5.14 or 

This 

greater to happen by chance alone. 

now know from our significant F value that there are significant 

We 

between our three groups. However, we do not know where the 

dlfferences 

exists. To determine this we must conduct tests between each 

difference 

of groups. This, of course, would not be necessary if a significant 

pair 

F was not obtained. 

5_. Run a Scheff test Step 2° determine which mean(s) of the three 

to 

belng studied is significantly different from the others. (Again, 

offices 

is only conducted if the obtained F is significant. The significant 

this 

tells us there is a difference between the three off'ces but does not 

F 

us where iIs difference Is.) Comparisons for this example would be 

tell 

follows: 

as 

mean I compared to mean II 

mean I compared to mean III 

mean II compared to mean 

To make these comparlsons It is necessary to use the following formula- 

Sw2("i + 2) 

SW2 mean square wthin (taken from 

where: 

sumary table) 

the 

of subjects n each group 

number 

examinati on. 

under 

k total number of groups. 

G. A. Statlstcal analysls in psychology and education. 

2°Ferguson, 

York. McGraw-Hll, 1966, pp. 295-297. 

New 

-98-

WORK SHEET

are as follows for the comparison between the means of 

Calculations 

#1 and office #2" 

office 

F 

4.40) 2 

(1.17 

(3 + 3)/(3)(3) 

(.11) 

(-3.23) 

.66/9 

10.43 

.07 

the other two comparisons are conducted with the fol]owlng 

Similarly, 

results 

Comparl s on 

F_ 

II 149.04 

I, 

I, III .28 

II, II! 136.40 

the determnaton of F values for the several comparisons, 

Following 

is necessary to consult a F table for the necessary crltical value re- 

it 

for significance at the .05 probabllty level for df /< 1 and 

qulred 

N k (2 and 6 in this example). For ou' example, a F of 5.14 is 

df 

necessary. 

calculate a new quantity F' which is /

WORK SHEET

compare the values of F and F' For any difference to be slg- 

Then, 

at the required .05 probability ievel, F must be equal to or 

nificant 

greater than F'. 

F_ F' 

Comparison 

II 149.04 10.28 

I 

I, III .28 10.28 

II, Ill 136.40 10.28 

F values for comparisons between off1"ce #I and office #2, and 

Te 

office #2 and office #3 are greater than the required F' value of 

for 

10.28 


can be observed by the obtained F value of 91.00 for counselors 

As 

three rehabilitation offices, significance was reached at the .05 prob- 

in 

level because the obtained F value exceeded 5.14. This indicates 

abllity 

differences exist between the Average Weighted Closure Index means 

that 

the three offices. However, it does not tell us where these differ- 

of 

are. To obtain this information, the evaluator must conduct a 

ences 

test to compare means two at a time. The Scheffe tests resulted 

Scheff 

slgnificant differences between offices #1and #2, and between offices 

in 

and #3, but no significant differences between offl'ces #I and #3. The 

#2 

again, were as follows. 

means, 

#1 1.17 

Office 

#2 4.40 

Office 

Office #3 1.31 

office #2 counselors, as a group, were significantly more ef- 

Thus, 

than both offices #1 and #3. Office #2 was more effective based 

fective 

a combination of effectiveness measures including quantity, work 

upon 

and quality of rehabilitations, even though they had fewer reha- 

effort, 

(14) than either office #1 (29 rehabilitations) or office #3 

b11itations 

rehabilitations). 

(28 

-102-

PART TWO 

One Group Pretest Posttest Study Des±gn 

01 X 0 2 

where: X service/treatment 

observat±on/measurement 

01 

service 

before 

observatlon/measurement 

02 

service 

after 

design calls for a smngle group that ms studied prior 

This 

services and again after services. The "gain" that is ob- 

to 

from 0 to 0 is presumed to have been a result of 

served 

serv±ce (). 2 

the 

The evaluator identlfmes a group of sub3ects 

Method. 

they are to undergo a particular servlce/treatment. 

before 

measures this group (0 I) prior to service. The evaluator 

He 

measures this group after service (02). From the 

again 

of the change from 01 to 02 he then attempts to 

results 

the effect of the service. 

infer 

This deslgn ms an improvement over the ex 

Limitations. 

facto study design because ±t enables the evaluator 

post 

manipulate services, and it permlts comparing a group 

to 

itself. Its weakness is that it does not allow the 

with 

the opportunity to know or test the effect of 

evaluator 

mnfluences in addlton to the provided service 

possible 

e.g. the effect of pretesting, maturation, other 

(X), 

etc. 

events,

PART TWO 

One Group Pretest Posttest Study Deslgn 

X 0 where: 

0 

2 

1 

X 

servlce/treatment 

observatzon/measurement 

01 

service 

before 

0 2 


servlce 

after 

deslgn calls for a sngle group that s studied prior 

Thls 

services and again after servlces. The "galn" that s ob- 

to 

from 0 to 0 s presumed to have been a result of 

served 

service (). 2 

the 

The evaluator dentfes a group of sub3ects 

Method. 

they are to undergo a partlcular service/treatment. 

before 

measures ths group (0 I) prior to servlce. The evaluator 

He 

measures thls group after service (0) From the 

agaln 

of change from 01 to 0_, he thn attempts to 

results the 

the effect of the 2 

service. 

nfer • 

Ths deslgn s an mprovement over the ex 

Lmtatons. 

facto study deslgn because t enables the evaluator 

post 

manipulate services, and t permits comparing a group 

to 

tself. Its weakness s that t does not allow the 

wlth 

the opportunity to know or test the effect of 

evaluator 

influences n addition to the provided service 

possible 

e.g. the effect of pretestlng, maturation, other 

(X), 

etc. 

events,

EXAMPLE 6 

General Procedure: One group pretest posttest 

A. 

deslgn. 

study 

B. Measurement Approach: Goal Attainment Scaling. 

Statlst±cal Procedures: t-test for correlated 

C. 

samples.

EXAMPLE 6 

General Procedure: One group pretest posttest 

A. 

deslgn. 

study 

B. Measurement Approach: 

Statlstlcal Procedures: 

C. 

samples. 

Goal Attainment Scalng. 

t-test for correlated

Sample Evaluatlon Objective 

compare the effectiveness of different service delivery technlques 

To 

a particular type of client. 

wltb 

I I Criteri a 

in ratings before and after service using Goal Attainment Scallng 

Change 

co res. 

s 

A. General Procedure zz 


One group pretest posttest study design 

Steps. 

O X Oz 

O1 observation/measurement 

where 

service 

before 

X : serwce/treatment 

on/measurement 

observati 

servlce 

after 

Identify clients that are to receive a particular servlce or 

I. 

i.e.; behavior modification, reality therapy, sheltered 

technique, 

workshop, etc. 

Measure the level of client functioning before he begins re- 

2. 

the service. 

ceiving 

D. T. & Stanley, J. C. Experlmental and quasi-experimental 

ZICampbell, 


designs 

-107-

3. Conduct the servlce. 

Measure the level of client functloning az termlnation of 

4. 

service. 

the 

Tabulate the results for the first and second measurements 

5. 

conduct the necessary statlstics to make the results meaningful. 

and 

One possibility is a t-test for correlated samples 22. 

Measurement Approach. Goal Attainment Scaling (G.A.S.) 23 

B. 

Program Evaluation 

for 

A Technlque 

Attalnment Scalng is a procedure for identlfying behaviorally 

Goal 

goals and outcomes for clients on a simple scale ranging from 

defined 

unfavorable outcome to best antlcpated outcome. The procedure has 

most 

following cbaracterlstlcs (a_) client objectives are devised for or 

the 

the individual, (b) there is a system for assigning weights among these 

by 

(c_) expected outcomes are identified for each objective, (d) 

objectives, 

is a quantifiable follow-up system for these outcomes, and (e_) a 

there 

can be obtained which summarizes outcome across all objectives. A 

score 

feature of this procedure is that goals are established for the 

positive 

client and he is compared with himself in terms of degree of 

individual 

success. 

problem or goal area s identlfied between counselor and clent 

Each 

weighted as to its importance n the overall service process. The 

and 

system allows the counselor and/or client to arb'trarily as- 

weighting 

a level of importance to each area One or two diglt numbers may 

sign 

used, the higher the number the more important is the goal. If no 

be 

are assigned, it is assumed that all problem areas identified are 

weights 

equal importance. After the identification and weighting of each of 

of 

problem areas, objective behavioral descriptions of outcome on fve 

these 

are scaled from the most unfavorable outcome likely (assigned a 

levels 

two) to the most favorable outcome likely (assigned a plus two). 

mnus 

expected outcome of treatment for each problem is also stated (asslgned 

An 

score of zero) and is at the middle of the scale. A sample Goal Attain- 

a 

Follow-up Guide appears on the next page (Figure 6 1) 

ment 

procedure for converting what are qualitative dmenslons to quan- 

The 

scores is computationally simple. It enables many comparisons 

titative 

transforming raw scores to standard scores ncluding comparisons be- 

by 

groups of disabled, types of servlce, and staff members. An advan- 

tween 

of the Goal Attainment Scaling procedure is its flexibility. It 

tage 

no restrlctions on posslble goals and glves freedom to assign rela- 

places 

tive weights which are appropriate for each client 

G.A. Statistlcal analysis in psychology and education. 

22Ferguson, 

York McGraw-Hill, 1966, pp. 169-171. 

New 

T J Goal attalnment scaling at a county mental health 

23Kiresuk, 

Evaluation, 1973, 1, 12-18. 

service 

-108-

WORK SHEET

z 

z 

o 

-ii0- 

> o 

X4-- 

o 

0 

o

WORK SHEET

The steps involved In developing goal attainment scales are as follows 

Collect information about the clleniC at the tlme of ntake or re- 

1 

and designate "problem areas" where change s desired to Insure a 

ferral 

rehabilitation. This can be done by (a_) counselor and client, 

successful 

cllent only, (c_) counselor only, and/or (d) others (e.g. family mem- 

(b) 

ber, etc. 

Identify these "problem areas" by giving each one a title. The 

2. 

s then placed across the top of the follow-up guide starting wlth 

tltle 

I for problem 1, Scale 2 for problem 2, etc., until each problem 

Scale 

Identlfied has been lsted. There is no minimum nor maximum number 

area 

areas necessary. These problem areas can then be weighted as to their 

of 

if so desired. 

mportance 

The counselor then states n behavioral terms under each "problem 

3. 

what he feels will be the outcomes, ranging on a 5-point scale, from 

area" 

most unfavorable service outcome thought likely to the best anticipated 

the 

with service. He also indicates the level of client functioning 

success 

in each problem area at the time of G.A.S. construction. 

The counselor then designates a follow-up date for determinlng 

4 

change. 

client 

At the deslgnated follow-up tme, a follow-up rater assesses goal 

5 

levels for each problem area for the client. 

attainment 

typical question that could be asked by a rehab11taton agency 

A 

How effective s a client-centered counseling approach wth clients 

s 

havlng drug abuse problems? 

ths example, the sample G.A.S. Follow-up Gulde was constructed 

For 

calculations carried out. Four additional cases were gven hypotheti- 

and 

cal pre and post scores for demonstration purposes. 

C Statistical Procedures 

Score each G A.S Follow-up Guide for "level at ntake" 

I 

--This score ndicates where the client was functioning prior 

(pretest) 

to counseling services 

procedure involves converting the G A.S. values to a standard T 

The 

(a T score has an average of 50 wlth a standard devlaton of 10) 

score 

The formula is 24 

T. J & Sherman, R. E Goal attainment scaling- A general 

24K1resuk, 

for evaluating comprehensive community mental health programs. 

method 

Community Mental Health Journal, 1968, 4_(6), 443-453. 

-112-

WORK SHEET

T=50+ 

ZWX 

I0 

+ 

%/(.7)ZWi2 

3(sWi)2 

W the weight asslgned to 

whereproblem 

area 

each 

level at which the cllent 

X 

functioning (ranges 

is 

from -2 to +2). 

For one client the information would appear lke thls 

Level at Intake 

Wi W 2 WiX 

X 

(Functionlng (Welght (Weight (Welght 

Scale 

Assigned) Squared) times level) 

Level) 

1 -2 3 9 -6 

2 -1 3 9 -3 

3 -I 2 4 -2 

4 -2 3 9 -6 

sW 11 zW 2 31 sWiX -17 

these values have been determlned, they can be placed n the T 

Once 

as follows" 

formula 

T= 50+ 

zWiXc 

10 

2 + .3(zW) 2 

(.7).Wc 

(-17) 

!0 

+ (.7)(31)+ (3)(11) 2 

50 

-114-

WORK SHEET

-170 

+21.7 50 (.3)(121) 

+ 

-170 

+%)2"i'.7 50 

36.3 

+ 

-170 

-170 

=50+ 

7.62 

50 + (-22.31) 

T I7 7 (level at intake) 

2. Calculate each G.A.S. follow-up score at the concluslon of 

Step 

The procedure Is the same as for "level of ntake" and would ap- 

servlce 

like this for the first client (see sample G.A.S. Follow-up Gulde for 

pear 

). 

scores 

Level at Fol l ow-up 

Scale X i Wi W 2 WX 

1 0 3 9 0 

2 +2 3 9 +6 

3 +1 2 4 +2 

4 +1 3 9 +3 

.Wc 11 zW 2 31 sWXi 11 

-116-

WORK SHEET

Convert follow-up ratlngs to T score as follows 

(11)_ 

10 

50 + ^,: 

7)(31) (.3)(11) 

T 

2 

110 

+ 21.7 + 36.3 

5O 

110 

=50+ 

7.62 

50 + 14.44 

T -64.¢I (level at follow-up) 

+ 

for this cllent and four addltional clients are given as 

Scores 

l ows 

fol 

Cl lent Pre-test Post-test 

1 27.7 64.4 

2 38.9 56.2 

3 40.0 59.1 

4 42.0 60.4 

5 35.3 63.6 

3 Conduct a statistlcal test to compare the pretest scores 

Step 

the posttest scores. A t-test for correlated samples is approprlate 

wlth 

because there are two observations for the same subjects) 25 

("correlated" 

formula is" 

The 

G.A. Statistlcal .a.nalsis n psychology an__d educatlon 

25Ferguson, 

York- McGraw-Hill, 1966, pp. 169-171. 

New 

-118-

WORK SHEET

Calculations are as follows- 

D 

--IN.D (zD)21/( t 1) 

N 

degrees of freedom N 1 

where D difference between 01 and 02 

Posttest D D z 

Pretest 

01 02 

Client 

1 27.7 64.4 36.7 1346.9 

2 38.9 56.2 17.3 299.3 

3 40.0 59.1 19.1 364.8 

4 42.0 60.4 18.4 338.6 

5 35.3 63.6 28.3 800.9 

83.9 303.7 

36.8 60.7 

119.8 3150.5 

these flgures in the t-test formula results In the 

Substltutlng 

I owing 

fo 

t INsD2_ (.D)I/(N 1) 

119.8 

(119.8)21/(5 1) 

15(3150.5) 

-120-

WORK SHEET

8 119 

14352 01/4 

115752.5 

119.8 

1400.5/4 

119.8 

350.1 

119.8 

18.7 

degrees of freedom = N 1 

=5-1 

=4 

a table of critical values for t-tests (found in statlstlcs 

Using 

for 4 degrees of freedom, the necessary t value for signifl- 

textbooks) 

cance at the 05 probability level is 2.78. 

IV Interpretati on 

can be observed by the obtalned t value, significance was reached 

As 

our example at the .05 probability level because the obtained t value 

in 

the critical value of 2.78. The mean of the pretest scores for 

exceeded 

five clients receiving the client-centered counseling service was 

the 

the mean of the posttest scores was 60.7. Thus, in this fictltous 

36.8, 

the client-centered counseling approach appears to be a signifl- 

example, 

effective technique for working with rehabilitation clients with 

cantly 

abuse problems. 

drug 

-122-

26 

7 

EXAMPLE 

General Proceudre: One-group pretest posttest 

A. 

deslgn. 

study 

B. Measurement Approcah: The Human Serv±ce Scale. 

C. Statst±cal Procedures: Lnear regress±on. 

example was developed by Kenneth W. Reagles, 

26Thls 

Dmrector, Reglonal Rehabllmtatmon Research Inst±tute 

Research 

the Unversmty of W±scons±n-Mad±son. The Human Servmce Scale 

at 

reprinted wmth permmsson of Human Service Systems, Inc. 

ms

26 

7 

EXAMPLE 

General Proceudre: One-group pretest posttest 

A. 

deslgn. 

study 

B. Measurement Approcah: The Human Servlce Scale. 

C. Statlstcal Procedures: Lnear regression. 

example was developed by Kenneth W. Reagles, 

26Thls 

Dlrector, Regional Rehabltaton Research Instltute 

Research 

the University of W±sconsln-Madson. The Human Service Scale 

at 

reprinted wth permlsson of Human Servlce Systems, Inc. 

is


determine the relationship between client and/or rehabilitation 

To 

variables, e.g. cost of trainingand change in client need sat- 

process 

I sfactl on. 

II Criteria 

of relationship between client/process variables and client 

Degree 

satisfaction as determined by the Human Service Scale. 

need 



One group pretest posttest study design 

Steps. 

01 X 02 

01 = observatlon/measurement 

where 

service 

before 

X = service/treatment 

on/measurement 

observati 

servi ce 

after 

Identify clients that are to receive a partlcular service or 

1. 

e.g. alcoholism treatment, personal-social counseling, 

training, 

training, physical restoration, marital counseling, 

pre-vocatonal 

etc. 

Admlnlster the Human Service Scale to all clients to ascer- 

2. 

the relatlve level of need satisfaction in seven need areas, 

taln 

as well as overall need satisfaction, prior to the receipt of services. 

D. T. & Stanley J. C. .xperimental an__d q.uasi-experlmental 

27Campbell, 


designs 

-125-

3 Implement and monitor the service plan. 

Re-administer the Human Service Scale at the completlon of 

4. 

service 

the 

Score the Human Service Scale and compile these scores for 

5 

flrst (pre) and second (post) aaministratlons of the scale. Con- 

the 

the necessary statlstical procedures to determine the relatlon- 

duct 

between cost of training and changes in need satisfaction. One 

ship 

possibility for determining relationship is linear regres- 

procedural 

2B. This is a process of determining the line which describes 

sion 

relationship between two variables. Its usefulness is n deter- 

the 

or predicting the value of one variable from knowledge of the 

mining 

other va ri ab I e. 

B. Measurement Approach The Human Service Scale29 

Human Service Scale (HSS____) was developed, in part, by the Univer- 

The 

of Wisconsin Regional Rehabilitation Research Institute (UW-RRRI). 

slty 

HSS s intended to measure the extent to which client needs are met 

The 

several life areas. 

in 

herarchy of basic human needs served as the underlying theo- 

Maslow's 

ratlonale for the development of the scale 0. From appropriate tem 

retical 

factor analyses at the UW-RRRI, seven factors, or subscales, were den- 

and 

These were labeled Physiological, Economic Securlty, Emotlonal, 

tfied. 

Social, Economic Self-Esteem, and Vocational Self-Actualizai:on. 

Family, 

a technique called "Smallest Space Analysis TM it was determlned 

Using 

the configuration of the need categorles was not a simple linear h- 

that 

but was a spherical configuration. This configuration was used 

erarchy 

construct a profile for reporting the scoring results Thus, a descrip- 

to 

of relative client need satisfaction can be presented graphcally for 

tion 

seven need dimensions (a HSS Scoring Profile form appears on the next 

the 

as Figure 7.1)o 

page 

HSS consists of 80 items (a copy of the scale appears at the back 

The 

thl s exam--'-ple). 

of 

R. B. Fundamental statistics four psyc.h..ology. New York. 

2BMcCall, 

Bruce and World, Inc., 1970, pp. 86-112. 

Harcourt, 

K. W., Wright, G. N., & Butler, A. J. Human Service Scale 

29Reagles, 

WI" Human Service Systems, Inc,, (P. O. Box 5551, Zip 53705), 

Madison, 

1973. 

A. H. Motivation and personality, 

3°Maslow, 

Harper and Row, 1970, pp. 97-105. 

IL- 

(2nd Ed.) Evanston, 

I. M. & Guttman, L. Smallest Space Analysts of Intel- 

31Schlesinger, 

and achievement test. Ps),choloi.cal Bulletin, 1969, 71, 95-100. 

lgence 

-126-

WORK SHEET

Z 

NEE 

"/IfNOS3

WORK SHEET

questions whlch relate to each of the seven subscales are as 

The 

follows 

Subscale 

1 Physlologlcal 

2. Emotlonal Security 

3 Economlc Security 

4 Famlly 

5 Socl al 

6 Economic Sel f-Esteem 

7. Vocational Self-Actuallzation 

ques tl ons 

10, 11, 13, 19, 26, 28, 31, 36, 37 

5, 

40, 41, 47, 52, 57. 

39, 

8, 9, 11, 12, 14, 16, 19, 20, 21, 

6, 

24, 25, 29, 33, 34, 35, 38, 41, 44, 

22, 

50, 59, 62. 

16, 18, 21, 48, 60, 61. 

7, 9, 17, 27, 30, 32, 42, 45, 49, 62 

15, 23, 43, 46, 51, 53, 54, 55, 58, 59. 

1, 2, 3, 4, 56, 78, 79, 80. 

64, 65, 66, 67, 68, 69, 70, 71, 72, 

63, 

74, 75, 76, 77, 78, 79, 80 

73, 

are a varlety of uses of the scale and a variety of statistical 

There 

in which the scores can be used 32 Scoring involves assigned 

techniques 

for each item response, weights range from one to seven A scoring 

weights 

is available, but hand-scoring is a laborious process at this time. 

key 

scale is machine-scorable and is returned to Human Service Systems, 

The 

for scoring. The scoring output provides basic identifying Informa- 

Inc. 

about the client, raw scores for the seven need areas and an overall 

tion 

and normative Information which compares the client's performance 

score, 

others with similar characteristics (e g. male alcoholics age 30-35 

to 

years 

scores for subscales are relatively independent of each other 

Raw 

lntercorrelatlon) and represent the extent of need satsfactlon. 

(low 

hgher the score, the greater the level of need satisfaction for that 

The 

The overall score does not represent a simple summation of the seven 

area 

scores, but s an independent welghtlng system. Raw scores may 

subscale 

used in pre and post comparisons as in this example However, the scoring 

be 

also translates raw scores into percentiles comparing the client 

output 

all subjects (approximately 4000) who have ever completed the HSS 

with 

Norm Group) In addition, the scale user may specify other com- 

(Total 

groups for obtaining more specific percentile informatlon. A sam- 

parison 

scoring output is presented as Figure 7.2. 

ple 

K. W., Kravetz, S Wright, G. N & Butler, A. J A 

32Reagles, 

yardstick The Human Servlce Scale. Madison The University of 

better 

-Regional Rehabilitation Research Institute, 1974 

Wisconsin 

-130-

WORK SHEET

2;: Z 

z 

-132- 

E

WORK SHEET


I_ Have each client's Human Service Scale scored by Human 

Step 

Systems, Inc. A sl'ngle pr'inted page of scoring information is 

Service 

produced by computer for each suboect. 

Tabulate the scores for each sbscale and the HSS overall 

Step2 

for each client for both pre and post administrations. For example, 

score 

and post scores for five clients for lust one subscale "Vocatlonal 

pre 

would appear as follows 

Self-Actualization" 

Vocational Self-Actual zat on 

Posttest Change 

Pretest 

Ient Oi 02 (dl ffe rence ) 

Cl 

1 63 67 4 

2 54 60 6 

3 57 65 8 

4 51 63 12 

5 39 53 14 

3_. Detemlne the linear regression for the cost of tralning on 

Step 

in vocational self-actualization. The followlng procedure can be 

change 

uti I ized- 

onal 

Vocat 

of Sel f-Actual izat on 

Cost 

ni ng Change 

Tra 

X Y 

Cllent 

1 $1oo 4 

2 150 6 

3 200 8 

4 300 12 

5 350 14 

N 5 ZX 1,100 .Y 44 

X 220 Y 8.80 

-134- 

X 2 y2 

10,000 16 

22,500 36 

40,000 64 

90,000 144 

122,500 196 

SX 2 285,000 456 

XY 

400 

900 

1,600 

3,600 

4,900 

sXY 11,400

WORK SHEET

instructs one to Identify cost of training as the X varlable 

This 

Vocational Self-Actualizatlon as the Y variable. One then squares 

and 

X value 33 and enters the result under X 2. The same procedure is fol- 

each 

for the Y variable. To obtain XY simply multlply each X by its cor- 

lowed 

responding Y value to obtain XY. After doing this, sum each column. 

4_. Utilize the previously computed values to calculate the 

Step 

relatlonships between cost of training and vocational self- 

lnear 

A linear relationship is an association between two varl- 

actualization. 

which can be represented on a graph by a straight line. The equa- 

ables 

tion Is" 

Y=X+a 

where Y variable Y 

of the lne 

lope 

constant value) 

(a 

X variable X 

at which the 

point 

Intercepts the 

line 

axis (a cons tant 

Y 

value) 

Thus, a linear relationship can be graphically represented as follows 

1 2 3 4 5 6 7 

X axis 

In this example X 2 represents the square of the value of X, 

33Note: 

chi-square. 

not 

-136-

WORK SHEET

Is now ready to substitute the calculated Infomation nto the 

One 

fomulas to detemine the values for /p and a. 

following 

(X)(ZY) 

N(XY)- 

2 (ZX) 2 

NZX 

c: Y -IpX 

For this example, the values n the formula for Zp are- 

= 

(1100)(44) 

5(11,400) 

(1100)(1100) 

5(285,0001- 

48,400 

57,000 

1,210,000 

1,425,000 

8600 

215,000 

]-,04 

Values In the formula for a are 

8.80 (.041(220) 

8.80 8 80 

-138-

WORK SHEET

the slope of the line (/p) Is .04 and the intercept (a) begins 

Thus, 

O. Presented graphically, it appears as follows: 

at 

sion 

N 

16 

14 

10 

4 

2 

#, 

s 

s 

#, 

s 

s 

S 

s 

s 

s 

s 

s 

S 

b .04 

i,,, I, 

$50 I00 150 200 250 300 350 400 

Cost of Training (X) 

7.3. Regresslon llne for cost of tralnlng on 

Figure 

in vocational self-actualization 

change 

Next, we substitute the resultant constants b and a nto the regres- 

equatlon 

Y=bX-cz 

Y .04X 0 

-140- 

s 

s 

•

WORK SHEET

this informatlon one can predict Y (vocational self-actualizatlon) 

With 

knowledge of X (cost of training). For example, if X $50, then 

from 

Y 2 as follows" 

.04(50) 0 

= 2.00 0 

2.00 

where = a predicted value for Y. 

f cost of training equals $700, one can predict the galn 

Similarly, 

vocational self-actualization satisfaction will equal 28, i e. 

in 

.04(700) 0 

28.00 0 

28.00 


shown in Figure 3, the regression line for the linear relation- 

As 

between cost of training and vocational self-actualization suggests 

shlp 

positive or direct relatlonship. This means that as the cost of traln- 

a 

ncreases so does the ga'n in client need satisfaction for vocational 

ng 

sel f-actua I i zati on. 

such relationshlps between two variables and the resul- 

Similarly, 

predictability can be conducted among other variables. Lnear re- 

tant 

is the basis for conducting multiple linear regresslon analysis 

gression 

relationships among more than two variables. 

nvolwng 

-142-

WORK SHEET

THIS PART TO BE COMPLETED BY THE PROFESSIONAL 

(First) (M 

(Last) 

^OIbRESS 

EPHONE 

(Street or R R 

JCIAL SECURITY NO L..i_J 

GE BIRTH DATE 

SEX, ® ® 

V 

(State) (Z,p) 

MO DAY YR 

-.LI ENT NO () ®I 

OUNSELOR NO 

r-] IIARITAL STATUS 

Married 

) 

W,dowed 

. 

£)vorced 

() 

Separated 

) 

Never Married 

_, 

Marriage Annulled 

C) 

" Unknown 

EDUCATION 

None 

grade 

7 

grade 

8th 

grade 

9 

School Diploma 

Hgh 

Techmcal wthout 

Vocational 

Lcensure/Certflcaton 

Techmcal wth 

Vocational 

censu re/Certfl cat on 

L 

College one or more 

Attended 

semesters 

Year College Degree 

Four 

Degree 

Graduate 

Unknown 

OF DEPENDENTS other 

NUMBER 

yourself 

than 

r- HERITAGE 

Wh=te 

_) 

Black 

C) 

Amencan Indian 

_( panlsh 

) 

Surname 

()ther 

LJnknown 

O 

FIRET EDITION COPYRIGHT ( 1973 

®®! 

®®i 

@®I 

@®I 

-®®®®@® 

®®®®®® 

@®®®®® 

®®®®®® 

®®®®®® 

@®®®@® 

®®®®®® 

®®@®®® 

@®@®®® 

ADMIN 

LIVING ARRANGEMENT 

Liv,ng alone 

0 

L,v,ng w, th spouse 

0 

L=vlng w,th one or both parents 

0 

step parents) 

(=ncludlng 

Liwng w=th non relat=ves 

O 

Other O 

r] WORK STATUS 

DISCOOE 

®@® 

®®® 

@®® 

@®® 

®@® 

@®@ 

®®® 

®®® 

®@® 

Wage or salaried worker (competlt=ve 

0 

market) 

labor 

Wage or salaried worker (sheltered 

O 

workshop) 

Self employed (except BEP) 

O 

State agency managed bus=ness 

O 

(BEP) 

enterprise 

Homemaker 

O 

Unpa=d famdy worker 

O 

Not work=ng student 

O 

O Unemployed 

r--] PRIMARY SOURCE OF SUPPORT 

0 Current earnings, nterest, 

Famdy and friends 

O 

Private rel=ef agency 

O 

Pubhc assistance, at least partly 

O 

Federal funds 

w=th 

d=v=dends, rent 

Pubhc assistance, without Federal funds 

O 

Pubhc nsttutlon-tax supported 

O 

Workmen's compensat=on 

O 

Soctai Security Disab=hty Insurance benefits 

O 

Other dsabl=ty, s=ckness, surwvors', or age- 

O 

benef=ts (except from private 

ret=rement 

=nsurance) unemployment nsurance 

Annu=ty or other non d=sab=hty 

O 

benefits (private nsurance) 

Cnsurance 

SPECIAL 

CODES 

@®@®® 

®®®®® 

®®®®® 

®®®®® 

®®®®® 

benefits 

D=sabl=ty or s=ckness benehts 

O 

Cnsurance) sav=ngs, other sources 

(private 

O Not reported 

r--] VETERAN 

®@®®® 

@®®®® 

Yes 0 No 0 Selective Serwce relectee 

0 

SERVICE SYSTEMS INC MADISON WI 53705 NCS Tran$ Optic F1706 5432 

HUMAN 

o-i111111111111111111111111111111111111111111111111 

ADD 

,0 

DROP 

I0 

to CHG 

Io 

OTHER

DI RECTIONS 

What IS your mal.__.n source of support 

( 

own earmngs (wages, workshop 

(your 

income from own bus 

payments 

hess) 

property or other invest 

()sawngs, 

ments 

of someone else m faro=Iv 

(earnmgs 

Security, pension payments, 

()Soc=al 

bnemployment Compensation 

or 

payments 

Public Assistance or Welfare pay 

() 

ments 

How much Public Assistance or Wel- 

2 

payments (but no.._t earnings, Social 

fare 

Pension Payments, or unem- 

Security, 

compensation payments) are 

ployment 

receiving per month> 

you 

Pubhc Welfare Ass=stance at this 

(No 

t=me 

to 75 dollars per month 

()1 


()76 


()151 

than 225 dollars per month 

(more 

How much do you earn 3 (wges, work- 

payments, income from own busi- 

shop 

savings, property or other nvest- 

ness, 

per week (nearest dollar);) 

ments) 

()none 

to 35 dollars per week 

()1 


(36 

DARKEN THE CIRCLE IN FRONT OF DESIRED RESPONSE FOR EACH QUESTION 

How often do you have trouble show=ng your feehngs to your famlly 

9 

very often () sometimes 

() 

often () hardly ever 

() 

as often as not 

() 


(as often as not 

()71 

dollars or more per week 16 How often do you 

()106 

How many jobs (either paid or unpaid 

4 

have you had ,n the last six months ;) 

work) 

(dd not work) 

()none 

job (2 jobs ()3 jobs (4 or more 

(1 

How often are you bothered by rap=d 

5 

beat 

heart 

very often () somet=mes 

() 


() 


( 

How often are you uncertain about 

6 

you make;) 

decisions 

often ( somet=mes 

()very 

() hardly ever 

()often 

often as not 

(as 

7 How often, when you need help, can 

find someone to help you 

you 

very often () somet= rues 

( 


(often 


( 

How often do you worry about grow- 

8 

old;) 

=ng 

often ( sometimes 

()very 

( hardly ever 

()often 


How often are you bothered by shortness of breath when not exercising;) 

10 

very often () somet mes 

() 

I) hardly ever 

()often 

often as not 

(as 

How often do you feel depressed, down, or very unhappy;) 

11 

very often () somet= mes 

() 

( hardly ever 

()often 

often as not J 

(as 

How often do you feel down or discouraged because your major problems cause yu 

12 

waste time;) 

to 

very often ( sometimes 

() 


() 


() 

How often do you become so sick you have to cut down on your usual actzvltes;) 

13 

very often ( somet=mes 

( 

( hardly ever 

()often 


() 

How often do you feel restless;) 

14 


() 


() 


( 

How often do you get together w=th 

15 

(go,ng out together or v=sit=ng ,n 

friends 

others' homeP 

each 

very often ( sometl mes 

() 


()often 

worry about the 

future;) 


() 


() 


( 

How often has your famzly faded to 

17 

you when you needed help? 

help 

very often () somet= mes 

() 


() 


( 

How often do you worry about your 

18 

havng enough money> 

famdy 

very often ( somet= rues 

() 

often ( hardly ever 

() 


() 

How often do you tend to go to 

19 

under pressure7 

pieces 


() 


()often 

often as not 

(as 

How often are you able to solve your 

20 

problems 

() 

own 

often () sometl rues 

very 


( 

often as not 

(as 

EDITION COPYRIGHT ( 1973 

i11111111111111111111111111111111 

FIRST 

How often do you worry about ge 

21 

ahead m the worid7 

tlng 


() 


( 


() 

How often do you worry about ge 

22 

along with your family;) 

tlng 

very often () somet=mes 

() 


() 


() 

How often do you become interested 

23 

something new7 

=n 

often () sometimes 

()very 


( 


( 

How often do you treat other people 

24 

badly;) 


() 


() 


() 

How often have you felt that you are 

25 

the kind of famdy member that you 

not 

like to be7 

would 


() 


() 


() 

How often are you bothered by 

26 

twitches, trembhng, or shakes> 

muscle 


() 


() 


( 

How often does your famdy accept 

27 

as you are 

you 

very often () someti mes 

() 


() 


( 

How often do you have headaches; 

28 


() 


() 


() 

lllllllllllUl

iil 

How many hours each week do you 

55 

on activities with other people in 

spend 

community7 

your 

hour or less () 14 to 19 hours 

() 

2 to 7 hours ( 20 hours or more 

() 

8 to 13 hours 

() 

How many weeks during the last six 

56 

were you unemployed7 

months 

none ( 17 24 weeks 

() 

8 weeks () 24 or more weeks 

() 

9 16 weeks 

( 

During the last six months, about how 

57 

days have your major problems kept 

many 

in bed all or most of the day> 

you 

none () 15-21 

() 

7 (E) 22 or more 

(E) 

8 14 

() 

How many people do you know 

5 

you feel free to talk to about per- 

whom 

things and ploblems7 

sonal 

very many () a few 

() 

many (E) none 

() 

some 

( 

How satisfied are you with your 

59 

hfe 

social 

very sattsfled 

() 

sat=stied 

() 

too satsf=ed but not too 

not 

dlssattsfled 

d=ssat=sfled 

( 

very dissatisfied 

(E) 

Whmh of the following statements 

60 

describes your present financial 

best 

situation7 

very good ( poor 

(E) 

good () very poor 

( 

average 

( 

Apart from mortgages on your house, 

61 

how many debts could you pay off n 

two months7 

next 

none of them 

() 

a few of them 

() 

some of them 

() 

of them 

(all 

have no debts 

( 

the 

Taking all things together, how would 

62 

descr=be your family hfe7 

you 

very happy 

() 

happy 

(E) 

not too happy but not too unhappy 

(E) 

unhappy 

( 

very unhappy 

( 

Which of the following best descr=bes 

63 

you are presently doing: 

what 

work for wages or salary or n own 

() 

including nvestments 

busness 

homemaker or housew=fe (housework 

(E) 

own family) 

for 

work in workshop or homebound 

() 

employment 

student or job tramng 

( 

unemployed 

IIII!1111111111 

YOUR ANSWER TO QUESTION 71 How often do you find t hard to 

IF 

63 WAS "E' (UNEMPLOYED), make friends w=th your present co- ork- 

NO 

HERE IF NOT, PLEASE CON- ere or people who are doing what y }u do ; 

STOP 

STUDENTS, PERSONS IN () very often () sometimes 

TINUE 

AND HOUSEWIVES () often ( hardly ever 

TRAINING, 

ANSWER THE FOLLOWING ( as often as not 

SHOULD 

ABOUT JOBS WITH 72 How often are you treated fa=rl y n 

QUESTIONS 

PRESENT ACTIVITY (SCHOOL, your present work 

THEIR 

OR HOUSEWORK) IN () very often () sometimes 

TRAINING, 

AS THEIR "WORK" AT THIS () often ( hardly ever 

MIND 

) as often as not 

TIME 

How often does your tresent vt fork 

73 

How often does your present work let you do something new each day 

64 

you make dems=ons on your own () very often () somet=mes 

let 

very often ( somet= mes (E) often () hardly ever 

(E) 

often ( hardly ever ( as often as not 

(E) 

as often as not 74 How often does your present ork 

( 

How often does your present work let you try out your own ideas7 

65 

you enough to do ( very often ( sometimes 

give 

very often ( somet, mes () often () hardly ever 

() 

often () hardly ever ( as often as not 

() 

as often as not 75 How often do you find that yo,u 

() 

During the last two weeks, how many really enjoy your [}resent work 

66 

of work did you m=ss due to a mnor () very often () somet=mes 

days 

such as a cold or sore throat () often () hardly ever 

smkness 

none ( 5 6 ( as often as not 

() 

2 7 or more 76 How often are you told m you=r 

() 

3 4 present work that you have done a 

() 

How many hours do you no_._w work good job 

67 

week7 () very often ( somet=mes 

each 

10 or less ( 31 to 40 hours () often () hardly ever 

() 

11 to 20 hours () over 40 hours () as often as not 

() 

21 to 30 hours 77 How often does your resent vork 

( 

How often do you learn new th=ngs give you a chance to make use of our 

68 

your present work abll=t=es 

from 

very often ( somet=mes (E) very often (E) somet=mesl 

(E) 

often (E) hardly ever often () hardly evel 

(E) 

as often as not as often as not 

() 

this list of activities that you 78 How steady is your present jot or 

Read 

take part n where you work the work you do 

may 

1 belong to some type of club or 

() very steady 

composed of people () steady 

organization 

whom work or who have reasonably steady 

with 

work unsteady 

s=m=lar 

belong to a un=on, attend umon () 2 unsteady 

very 

meet=ngs 

somahze after work hours with 79 What do other people th=nk of 

3 

workers your lob7 

fellow 

other actvlt=es related to your () they think it s a very good job 

4 

( they thnk =t s a good job 

work 

many of the above do you do : ( they think it s an average job 

How 

of them ( they think it is a poor job 

none 

one of them ( they thnk It is a very poor job 

() 

two of them 80 How does your present job (work) 

( 

three of them compare wth jobs you've had n the past 

( 

four or more of them () my present job =s much better 

() 

What =s the tota._._l number of hours () my present job is better 

70 

spend each week on the above actlv- () my present job s just as good 

you 

Choose one of the following ( my present job s worse 

itles 

2 hours or less (9 to 11 hours ( my present job s much worse 

() 

3 to 5 hours ( 12 hours or more STOP 

() 

( 6 to 8 hours 

THANK YOU FOR YOUR COOP.RATIO 

IIIIIIIIIIIIIIlilIIIlIiliI/ilL 

,=

9 

,/or 

30 

*all 

he 

3 

tor, 

often do things happen to make 

How 

angry 


ery 


often 

often as not 

as 

often do other members of the 

How 

talk to you about what went on 

ily 

the day' 

mg 


ry 


),ten 

often as not 

as 

often do you feel dizzy' 

How 

often sometimes 

very 


often 

often as not 

as 

speaking, how often do you 

Generally 

your famdy about what went on 

to 

the day' 

ing 


very 


often 


general, how often do you feel 

In 

Ipless' 


very 

( hardly ever 

often 

often as not 

as 

often have you consulted a doc- 

How 

psychiatrist, psychologzst, or anyone 

about a nervous problem' 

else 


(very 

often hard ly ever 

( 

often as not 

()as 

How often do your major problems 

35, 

you feel inferior' 

make 


( 


( 


() 

How often in the past year have you 

36, 

sePn a 

or been hosp,tahzed for 

doctor 

ur physmal problems' 

yo, 


) 


() 


( 

How often do you have general aches 

37 

pains' 

and 


() 


( 


( 


38 

it difficult for you to make friends' 

make 

very often () sometl rues 

() 


() 

How often have you felt that you are 

41 

to have a nervous breakdown> 

going 

very often ( somet mes 

@ 


() 


( 

About how much time a week do you 

42 

doing things together with your 

spend 

family' 

hours or less 

5 

6 to 11 hours 

() 

12 to 17 hours 

( 

18 to 23 hours 

() 

( 24 hours or more 

Read the list of clubs and orgamza- 

43 

to which people may belong 

t=ons 

parent-teachers group 

any 

church-connected groups (usher's 

2 

Ladies Aid, etc 

club, 

fraternal lodge or auxiliary 

3 

neighborhood clubs, commumty 

4 

(including YWCA, YMCA) 

center 

card clubs or social clubs 

5 

veteran's association 

6 

service club (Rotary, Lions, etc 

7 

cwc organizations (partm=patfon m 

8 

drives, Red Cross, etc 

charity 

sports team 

9 

partmfpaton m political activities 

10 

a 

club or party 

political 

many of the above organizations do 

How 

take an actwe part in' 

you 

none of them () 5 or 6 of them 

() 

or 2 of them () 7 or more of them 

() 

3 or 4 of them 

( 

How often do you feel bored' 

44 

very often ( sometl rues 

() 


( 


() 

the following list of things 

Read 

may do together 

Illes 

visit frmnds 

1 

go to a movie, bowling, sporting event, 

2 

some other entertainment 

or 

spend an evening just talking with 

3 

other 

each 

working on some household prolect 

4 

entertaining friends in your home 

5 

go shopping 

6 

have a good laugh together or share 

7 

joke 

a 

eat out in a restaurant 

8 

are affectionate toward each other 

9 

take a drive or go for a walk 

10 

help a family member solve some 

11 

problem 

often as not 12 take part in some religious act,vlty 

)as 

How often do you have a common How 

39 

or the flu 

cld 


) 


) 

often as not 

@as 

lww'eer;o you have sk|n rashes' 

' 


) 


many of these things does your 

do together' 

family 

of these things ()5 or 6 of them 

()none 

or 2 of them () 7 or more of 

() 

3 or 4 of them these things 

( 

In the last year, how many new 

46 

have you made' 

friends 

very many ( a few 

( 

many ( none 

( 

some 

() 

How often are you bothered by an 

47 

stomach' 

upset 


() 


() 


( 

How often do you worry about not 

48 

enough money' 

hav,ng 


() 


() 


( 

How often do you like spending 

49 

with your famdyY 

time 

very often 

() 

often 

() 


( 

sometimes 

( 

ever 

hardly 


50 

you from making use of your 

keep 

abilities' 


() 


()often 


() 

51 About how many people did you 

dunng the last year, other than 

meet 

you meet where you work, that 

those 

never met before' 

you 


() 

many none 

() 

some 

( 

How often do you worry about 

52 

healthy 

your 


() 


() 


() 

About how many friends do you 

53 

keep in touch withy 

usually 


() 

many ( none 

() 

some 

( 

Read this list of which 

activities you 

5j, 

take part In with other people in 

might 

your community 

sports football, basketball, tenms, 

1 

etc 

golf, 

outdoor activities hunting fishing, 

2 

etc 

hiking, 

indoor activities bowling, table 

3 

dancing, cards, etc 

tenms, 

4 other somai activities 

many of the above activities do you 

How 

part m with other people in your 

take 

communltyY 

none of them ( 3 of them 

() 

of them () 4 or more 

() 

2 of them of them 

( 

PLEASE CONTINUE ON NEXT PAGE 

often as not HUMAN SERVICE SYSTEMS INC MADISON WI 53705 

=(111111111111111111111111111111111111111111111111 

as

PART THREE 

Pretest Posttest Control Group Study Design 

R 01 X 0 2 

R random assignment 

where: 

sub3ects 

of 

X 

0 3 


0 4 

observatlon/measure- 

01 

before service 

ment 

experimental 

to 

group 

0 2 

0 3 

0 4 


service to experi- 

after 

mental group 

observation/measurement 

service for con- 

before 

trast group (no service) 



after 

trast group (no servlce) 

design calls for a sample of sub3ects that are randomly 

This 

to one of two groups. These groups are then examined 

assigned 

but only one receives service (X). The differences in 

twice, 

change from pre to post score measures of the two groups 

the 

is attributed to the effect of the service. 

The evaluator identifies a sample of sub3ects. From 

Method. 

sample he randomly assigns each of the members to one of 

this 

two groups, the sub3ects in both groups are then observed 

the 

measured. One group is given a service (exper±mental group) 

or 

the other is not (control/contrast group). After the 

while 

the groups are again measured. The difference between 

service 

measures for the two groups are then evaluated ststlstl- 

these 

cally. 

This design involves random assignment of sub3ects to 

Value. 

thus, permitting the evaluator to assume that the two 

groups; 

have been "equalized". This means the groups are 

groups 

to be equal, within chance limits, in all possible 

assumed 

Having a contrast group allows the evaluator 

characterlstcs. 

compare the change from pretest to posttest for the experi- 

to 

group with the before and after measures of the group 

mental 

the specfled service; thus, further ad3ust±ng for 

without 

to the experimental group during the evaluation 

cahnges 

The design's limitation is the lack of control by 

period. 

the evaluator for the affect of pretesting.

PART THREE 

Pretest Posttest Control Group Study Design 

0 X 0 

1 

R 

2 

0 0 

3 4 

R 

R random asslgnment 

where: 

sub3ects 

of 

X 


observatlon/measure- 

01 

before service 

ment 

exper:mental 

to 

group 

0 2 

0 3 

0 4 

bservaton/measurement 

service to experl- 

after 

mental group 



before 




after 


deslgn calls for a sample of sub3ects that are randomly 

Th:s 

to one of two groups. These groups are then examined 

assigned 

but only one receives service (X). The differences in 

twice, 

change from pre to post score measures of the two groups 

the 

is attributed to the effect of the service. 

The evaluator Identlfes a sample of sub3ects. From 

Method. 

sample he randomly assigns each of the members to one of 

thls 

two groups, the sub3ects in both groups are then observed 

the 

measured. One group s given a servlce (experimental group) 

or 

the other is not (control/contrast grouD). After the 

while 

the groups are again measured. The difference between 

service 

measures for the two groups are then evaluated ststst±- 

these 

cally. 

Ths design Involves random assignment of sub3ects to 

Value. 

thus, permttlng the evaluator to assume that the two 

groups; 

have been "equalized". Thls means the grouDs are 

groups 

to be equal, wlthn chance lmlts, in all possible 

assumed 

Having a contrast group allows the evaluator 

characterlstcs. 

compare the change from pretest to posttest for the experl- 

to 

group wlth the before and after measures of the group 

mental 

the specified service; thus, further ad3ustng for 

without 

to the experimental group during the evaluation 

cahnges 

The design's lmltatlon is the lack of control by 

period. 

the evaluator for the affect of pretesting.

ao 

So 

Co 

EXAMPLE 8 

Procedure: Pretest posttest control 

General 

study deslgn. 

group 

Approach: Questlonnare for cllent 

Measurement 

of the rehabltaton process. 

knowledge 

Procedures: t-test for ndependent 

Statlstcal 

samples.

EXAMPLE 8 

General Procedure: Pretest posttest control 

A. 

study deslgn. 

group 

Measurement Approach: Questlonnalre for clent 

B. 

of the rehabltatlon process. 

knowledge 

Statistical Procedures: t-test for ndependent 

C. 

samples.


compare the effectlveness of two dlfferent methods of orienta- 

To 

to rehabilitation services for new clients. 

tion 

II Criteria 

in scores of clients measured before and after a special 

Change 

program using a 10 item questionnaire for each group. 

orientation 



Pretest posttest control group study design 

Steps. 

R = random assignment of 

where 

to experlmental 

clients 

and contrast groups 

X = servlce/treatment 

0 : observation/measurement 

Identify clients who have been accepted for rehabltatlon 

1. 

but who have not recelved any orientation to the rehabil- 

services 

itation program. 


34Campbell, 

for research. Chicago: Rand McNally, 1966. 

designs 

-151-

Randomly asslgn these clients to one of the two groups uslng 

2. 

of the random sampling methods described in the Introduction of 

one 

this manual. 

Administer a measure to determine the level of knowledge of 

3. 

rehabilitation process to both groups before the orientation pro- 

the 

gram is initiated. 

Carry out the special orientation program wth the exper- 

4. 

group and give the usual orientation or none to the contrast 

mental 

group. 

Measure the level of client knowledge of the rehablltaton 

5. 

after the special orientation program with the same instru- 

process 

ment used prior to the program. 

Tabulate results and conduct the necessary statistics to make 

6. 

results meaningful. One possibility is a t-test for independent 

the 

samples 3s. 

Measurement Approach Questionnaire for client knowledge of the reha- 

B. 

process. 

bilitation 

way to measure cllent knowledge of the rehabilltation process is 

One 

administer a questionnalre to new clients that have been accepted for 

to 

prior to a speclal orientation program and again after the spe- 

services 

orientation program 

cial 

Some typical questions that might be included are 

Rehabilitation services are provided only when a client has a 

1. 

that creates a handicap to his belng employable. 

disability 

T 

F 

Cllents who qualify for rehabllitation servlces automatically 

2. 

eligible for welfare. 

become 

T 

F 

G. A. Statistical analysis in psychology and education. 

3SFerguson, 

York- McGraw-Hill, 1966, pp. 167-169. 

New 

-152-

WORK SHEET

One service every person recelves from the rehablltaton program 

3. 

vocational training. 

is 

1. etc. 


T 

F 

score each test given to clients making sure the pre-measures 

st__e_El, 

post-measures are kept separate for each of the two groups. 

and 

2. Determine the change in score from the pre-measure to the 

Step 

for each group. For example 

post-measure 

Hypothetical Scores for Two Groups of 10 Clients 

Group Contrast Group 

Experimental 

change 

Client pre-score post-score score 

1 2 9 +7 

2 4 10 +6 

3 3 8 +5 

4 1 9 +8 

5 3 10 +7 

6 2 8 +6 

7 4 9 +5 

8 1 7 +6 

9 3 8 +5 

10 5 10 +5 

-154- 

Clent pre-score post-score 

I 3 6 

2 2 5 

3 4 8 

4 1 4 

5 2 6 

6 3 5 

7 3 7 

8 1 3 

9 2 5 

10 5 7 

change 

score 

+3 

+3 

+4 

+3 

+4 

+2 

+4 

+2 

+3 

+2

WORK SHEET

3. Calculate the mean of the "change" scores for each group. 

Step 

the mean score from the scores for each subject and square this 

Subtract 

Then sum the squared values. Using the data from this example, 

value. 

calculations are as follows- 

the 

Change 

Score 

Cllent 

Experimental Groups 

X1 

1 7 +1 1 

2 6 0 0 

3 5 -1 1 

4 8 +2 4 

5 7 +1 1 

6 6 0 0 

7 5 -1 1 

8 6 0 0 

9 5 -1 1 

10 5 -1 1 

N 10 

.X 60 

X 6 

xq 

10 

Change 

lent Score 

C1 

Contrast Group 

X2 

1 3 

2 3 

3 4 

4 3 

5 

6 

7 

8 

9 

10 

I0 

30 

3 

0 0 

0 0 

+1 1 

0 0 

+1 1 

-I 1 

+1 1 

-1 1 

0 0 

-1 1 

4. Substltute the data obtalned from Step 3 into the t-test 

Step 

for independent samples 36 (Independent because the measures are 

formula 

J. P. Fundamental statistics l_D_n psycholog), and education. 

36Guilford, 

York McGraw-Hill, 1950, p. 228. 

New 

-156-

WORK SHEET

two dlfferent groups) and do the necessary calculatlons. The t-test 

for 

is as follows- 

formula 


df NI + N2 2 

XI X2 directs us to subtract the mean change 

wherefor 

the experimental group from the 

score 

mean change score of the contrast group. 

for the data In ths example, the t-test formula would look 

Thus, 

this" 

like 

t= 

6-3 

+10-2 

3 

r/ 89 (.20)' 

-158-

WORK SHEET

3 

178 

df=N+N-2 

3 

10 + 10 2 

18 

a table of critlcal values for t-tests wlth 18 degrees of free- 

Uslng 

the necessary t-value for significance at the .05 probabllity level 

dom, 

2 10 This means only 5 times out of 100 could we expect a t of 2.10 

is 

greater to happen by chance alone. 

or 

IV Interpretatl on 

can be observed by the obtained t value, sgnlfcance was reached 

As 

the 05 probab111ty level because the obtained t value exceeded the 

at 

value of 2.10. The mean change from pretest to posttest for 

critical 

experimental group was +6, the mean change for the contrast group was 

the 

In thls fictitious example, this indicates that the clients who were 

+3. 

assigned to the special orientation program scored significantly 

randomly 

in knowledge of the rehabilitation process than cllents who were 

higher 

nvolved n the orientation program. 

not 

-160-

EXAMPLE 9 

General Procedure: Pretest posttest control 

A. 

study deslgn. 

group 

Measurement Approach: The Work Adjustment 

B. 

Form. 

Rat±ng 

C. Stat±st±cal Procedures: Analysls of covar±ance.

Ao 

Bo 

EXAMPLE 9 

Procedure: Pretest posttest control 

General 

study deslgn. 

group 

Approach: The Work Aduustment 

Measurement 

Form. 

Ratlng 

C. Statlstcal Procedures: Analyss of covarance.


compare the effectiveness of three different approaches to work 

To 

trainlng In a rehabilitation facility. 

adjustment 

I I Crl teria 

delivery effectiveness of three different approaches to work 

Service 

training as measured by the Work Adjustment Rating Form. 

adjustment 



Pretest posttest control group study design 

R random asslgnment of 

where 

to experimental 

clients 

and contrast groups 

X = service/treatment 



37Campbell, 

for research. Chicago. Rand McNally, 1966. 

designs 

-163-

Steps. 

Randomly asslgn, if possible, clients to one of three dif- 

1. 

combinations of work adjustment training methodologies using 

ferent 

of the random sampling methods described in the Introduction of 

one 

manual. Three examples might be (a) productlon workshop assign- 

thls 

wth supportive counsellng, (b_] job-site assignment with com- 

ment 

employers and supportive counseling, and (c) a program nvolv- 

munity 

the workshop, job-sites, classroom nstruction and supportive 

ing 

ing. 

counsel 

Obtain Work Adjustment Rating Form ratings from two to four 

faclliy/job-site 2 

who are In a pion to observe the clients 

staff 

a work situation. These ratings are obtained for all clients n 

in 

three work adjustment trainlng groups during the first week in 

the 

the program (pretest). 

3. Carry out the work adjustment training programs. 

Again, obtain Work Adjustment Rating Form ratings from two 

4. 

four facility/job-siTstaff for all clien Tn the three work 

to 

training groups during the last week n the program 

adjustment 

(posttest). 

Tabulate results and conduct the necessary statistics to 

5. 

the results meaningful. One possibility is an analysis of 

make 

covariance 3B 

B Measurement Approach Work Adjustment Rating Form (WARF____) 3 

Work Adjustment Rating Form is a rating scale constructed pri- 

The 

for use by rehabllitation facility staff to assess work adjustment 

marily 

and limitations. It can be used to develop work adjustment 

strengths 

plans and assess progress toward job readiness. 

training 

WARF contains eight subscales, each having five Items, making a 

The 

of 40 i----tems The subscales are (a_) Amount of Supervision Requlred, 

total 

Realism of (b) Jo (c) Teamwork, (d_---) Acceptance of Rules/Authority, 

(e_---) 

Goals, 

Tolerance, (f__) PerSeverance n Wrk, (g) Extent Trainee Seeks 

Work 

Assistance, and (h) Importance Attached to Job Tralnlng. 

E. F. Deslgn and analysis of experiments in psychology 

3eLindquist, 

education. Boston Houghton Mifflin, '1953, pp. 317-339T-- 

and 

J. A. & Bolanovich, D. J. WARF. A scale for measuring 

gBitter, 

behaviors. American Journal of Mental Deficiency, 1970, 

job-readiness 

616-621. 

74, 

-164-

WORK SHEET

of the subscales is represented by items descrlbing flve df- 

Each 

levels of performance from low to high. The following five items 

ferent 

which represent the subscale "Teamwork" are an example of this- 

1. Trainee is unable to work effectively wth any others. 

Trainee can work wlth only one or two others whom he particularly 

2. 

kes. 

1 

3. Trainee can usually work with a few others without conflct. 

4. Trainee works effectively in small (2-3) groups. 

5. Trainee is a good team worker in any group sltuatlon. 

items are rated by checklng "yes" or "no" to each and they are 

The 

(Guttman, 1944) so that a posltve response to an item at any level 

scaled 

also give a positive response to all items below that level 4. 

should 

are scrambled and the level and scale of items are not known by 

They 

raters. 

WAR____F can be completed n 3 to 7 minutes depending on the rater 

The 

instructons for completien of the WARF are as follows- 

The 

Work Adjustment Rating __Form s a rating scale consisting 

"The 

40 items relating to observable work behaviors You are 

of 

to answer each item either 'yes' or 'no' for the trainee 

asked 

on the form. Some Items appear very slmllar. An- 

identified 

swer each item. Please do not omit any items. 

Mark your response with an 'X' over your choice." 

for the WARF consists of cumulative positive responses. A 

Scoring 

s a positive response for the items A3, A4, A5, B3, B4, B5, C3, 

"yes" 

C5, D3, D4, D5, E3, E4, E5, F3, F4, F5, G4, G5, H4, and H5. A "no" 

C4, 

a positive response for tems A1, A2, BI, B2, C1, C2, D1, D2, El, E2, 

s 

F1, F2, G1, G2, G3, H1, H2, and H3. 

if a key template s constructed, takes approximately fve 

Scoring, 

The range of total scores on the WARF is zero to 40. A profile 

mnutes. 

trainee strengths and llmtatons can be obtained by nspection of 

of 

subscale scores. A scorlng form wlth dentfication of scrambled 

WARF 

L A. A basis for scallng qualtatve data. American 

4OGuttman, 

Review, 1944, 9, 139-150. 

Sociological 

-166-

WORK SHEET

items in parentheses Is included wth the scale (a WARF appears at 

WARF 

back of this example) It is advlsable to obtaln more than one rater's 

the 

ratings for each client to effect control for rater bias 41 


i__. Score each WARF making sure the pre-measures and post- 

Step 

are kept separate for each of the three groups 

measures 

2. Tabulate each rater's total WARF score for pre and post 

Step 

for the three different approaches (methods) to work adjustment 

ratings 

Use the average score of the two to four raters for each cli- 

training 

Hypothetical average total scores for 9 clients are glven below 

ent. 

1. 

Method 

on Workshop 

Product 

2 

Method 

tes 

Job-si 

3 

Method 

Job-sl tes, 

Workshop, 

C l as s room 

Pre (X) pos.t. Pre (x) Pos.t Pre.(X) Post 

17 24 18 22 17 38 

20 25 21 29 18 38 

20 23 21 27 19 35 

Calculate the prellmlnary nformaton that Is needed to 

st__9_E3. 

out the analysis of covariance. Analysis of covarance s a method 

carry 

statistically controlling for differences on some variable (n ths 

of 

the pretest) by adjusting the criterion measure (the posttest n 

case 

example) Ths technique is partlcularly useful when subjects can- 

this 

be randomly assigned to groups so that differences between groups 

not 

be minimized In this fictitious example, subjects were randomly 

may 

to groups, thus, analysis of covarlance wll further ncrease 

assigned 

precision of the study 

the 

Calculations and explanations follow 

J. A. Bas effect on valldlty and relability of a rating 

41Bitter, 

Measurement and Evaluation _n Guidance, 1970, 3(2), 70-75 

scale. 

-168-

WORK SHEET

a 

TX 

Ty# 

TxaTY a 

.X#Y a 

Method 1 Method 2 Method 3 

X Y X Y X Y 

17 24 18 22 17 38 

20 25 21 29 18 38 

20 23 21 27 19 35 

3 3 3 

57 60 54 

19 

1083 

1089 

1368 

1368 

24 

1728 

1730 

72 78 111 

20 18 

26 

1200 972 

1560 

2028 

1206 974 

1572 

2054 

-170- 

1998 

1995 

37 

4107 

4113 

N=9 

T X 

171 

261 

Ty 

3249 

Tx2/N 

7569 

Ty2/N 

4959 

TxTy/N 

X X 

19 

Xy = 29 

. 

s 

na 

TxaTy a 

3255 

7863 

2 3269 

zzXj 

7897 

zzYa2 

r.r.X#Y,] 4935 

4926

WORK SHEET

where the number of cllents in each column 

N total number of cllents in the evaluatlon 

total of the WARF scores in each pretest column (e g 

for Method 1 T"# 17 + 20 + 20 57) 

TX# 

X total of the WARF pretest scores for all three methods 

T 

T X 570 + 54 171) 

(e.g. 

total of the WARF scores in each posttest column (e.g. 

for Method 1 ---# 24 + 25 + 23 72) 

Ty# 

total of the WARF posttest scores for all three methods 

Ty 

Ty 728 + 111 261). 

(e.g. 

the prewously calculated T X value squared dlvlded by 

Tx2/N 

total number of clents 

the 

Tx2/N 171 x 171 29,241 

9 9 3249) 

(e.g. 

the prewously calculated Ty value squared dlvided by 

Ty2/N 

total number of clents 

the 

g. Ty2/N 261 x 261 68,121 

9 9 7569) 

(e 

the prevlously calculated TX value multlplied by the 

TxTy/N 

calculated Ty value dvided by the total 

prewously 

of clients. 

number 

TxTy/N 171 x9 261 44,6319 4959) 

(e.g. 

TX 

XX n# 

or the average of the WARF pretest scores for each 

method (e.g. 57 

Method 1. XX = 19) 

for 

172-

WORK SHEET

Xy = 

average of the WARF pretest scores for all methods 

the 

+ 20 

(e.g. X 

19 

3 = 19) 

X 

or the average of the WARF posttest scores for 

each method (e.g. for Method 1 _vj 7--23 24) 

average of the WARF posttest scores for all methods 

the 

X- 24 + 26 

(e.g 

3 = 29) 

= take the value of TXj for each method (pretest) and 

multiply it by its own value (square it). Then dlwde 

by nj for each column. 

(e.g. for Method 1 

57 x 57 3249 

TX 

3 = 1083) 

n 

. n---_ take the three values calculated for -- and add them 

together. 

the value of Tyj for each method (posttest) and 

take 

it by Its own value (square t). The dvde 

multiply 

for each column. 

by 

2 T 

(e.g. for Method I 72 x 72 5184 

n 3 1728) 

T 2 

TY 

take the three values calculated for and add them 

nj 

z 

nj 

-174-

WORK SHEET

. 

Tx"Ty 

total of the WARF scores in each pretest column 

the 

by its corresponding total for WARF scores 

multlplled 

the posttest column and diwded by the er of 

in 

for that method. 

clients 

(e.g. for Method 1 

TxjTy#. 57 x 72 4104 

nj 3 - 1368) 

the three values calculated for 

TXJTY-# 

and add 

take 

nj 

nj 

together. 

them 

.XY = 

each WARF pretest score and sum for each column 

square 

for Me-- 1- sX 2 (17)(17) + (20)(20) + 

(e.g. 

(20)(20) = 1089) 

the sum of all three .X# 2 (e.g. 1089 + 1206 + 974 3269). 

square each WARF posttest score and sum for each column. 

sY2 

for Met----1 sY2 (24)(24) + (25)(25) + 

(e.g. 

= 1730) 

(23)(23) 

the sum of all three sYj (e.g. 1730 + 2054 + 4113 7897). 

X (pretest) score multlplied by Its corresponding 

each 

(posttest) score and summed for each method column. 

Y 

for Method 1" XY (17)(24) + (20)(25) + 

(e.g. 

1368) 

(20)(23) 

..XjY# the sum of all three sX#Y (e g. 1368 + 1572 + 1995 4935) 

Using the values calculated in Step 3, It is posslble to 

St__p_4 

sum of squares (a) total (SST), (b_) between (SSB) and (c) 

determine 

(SS W) for both pretest (X) and posttest (Y) scores Also, the (a) 

wthn 

of products (SP), (b_) adjusted sum of squares (SSV), and (c_) adjusted 

sum 

squares (MSV} can be computed. It is the adjusted mean squares for 

mean 

(MS#B) and within groups methods (MS which are mportant for calculating 

B) 

F value. The F-ratio tests the significance of the differences among 

the 

method means. 

the 

-176-

WORK SHEET

detemine the sum of squares total, between, and within, apply 

To 

following formulas- 

the 

of Squares Total (SSTx) XzXa2 PTx.._ 

Sum 

N 

Degrees of freedom = N 1 

:9-1 

=8 

Sum of Squares Between (SSBx) x: 

Degrees of freedom k- 1 

=3-1 

=2 

-178- 

= 3269 3249 

N total number 

wheresubjects 

of 

TX2 

TX 

N 

a 

3255- 3249 

where number of groups

WORK SHEET

Sum of Squares With1"n (SSWx) = 

Degrees of freedom N / 

=9-3 

:6 

SSTx 

SSBx 

=20-6 

SCmlarly, for posttest WARF scores (Y): 

of Squares Total (SSTy) ssyj2 TY__ 

Sum 

N 

Sum of Squares Between (SSBy) s 

-180- 

7897- 7569 

N = total number 

wheresubjects 

of 

Ty____ 

Tv2-- 

N 

nj 

7863- 7569 

number of groups

WORK SHEET

Sum of Squares Within (SSWy) SSTy SSBy 

Then, calculate the sum of products- 

Sum of Products Total (SP T) zzXjYj 

Sum of Products Between (SP B) : 

328 294 

4935 4959 

z 

TxYy 

N 

T X Ty TxT Y 

= 4926 4959 

Sum of Products W1thln (SP W) SP T SP B 

-182- 

(-24) (-33) 

N

WORK SHEET

using the prevlous values, calculate the adjusted sum of squares 

Now, 

posttest WARF scores (SS). 

for 

Sum of Squares Withln (SSy) SSWy 

Degrees of freedom N /¢ 1 

=9-3-1 

=5 

Sum of Squares Total (SSTy) = 

Degrees of freedom N 2 

=9 -2 

=7 

-184- 

SSTy 

(SPw)2 

X 

SSw 

34- (9)(9) 

14 

34 

81 

14 

34- 5.8 

12 8-2 


wheresubjects 

of 

(SPT)2 

.STy 

328 

328 

(-24)(-24) 

20 

576 

20 

328 28.8 

i299.2 

k = number of groups 


where: 

subjects 

of

WORK SHEET

Sum of Squares Between (SSy) SSy- SSy 

Degrees of freedom k 1 

=3-1 

=2 

299.2 28.2 

271.0 

where, k number of groups 

results of these computations can be summarlzed in a table as 

The 

lows. 

fol 

Sources df SS x SSy SP SSy df MSy 

Between 

Within 

2 6 294 -33 271.0 2 135.50 

6 14 34 9 28.2 5 5.64 

Total 8 20 328 -24 299.2 7 

adjusted mean squares (MS) for treatment and wthln groups are 

The 

by dividing the adjusted sum of squares (SSy) by the correspond- 

determined 

Ing degrees of freedom (dr) Thus, 

B 

SSy 

df 

MSYB 

-186- 

271.0 

2 

1135.50

WORK SHEET

W SSy 

MSYw_ and 

df 

=I5,.641 

Step 5_. Calculate the F-ratlo for analysis of covamance. 

F 

Square between 42 

Mean 

Square within 

Mean 

135.50 

5.64 

I'24 02 

obtained F value is 24.02 Using a table of crltcal values for 

The 

tests (found in statistics books) for 2 and 5 degrees of freedom, the 

F 

F value is 5.79 at the .05 probabllity level. This means that 

necessary 

5 times out of 100 would we expect a F of 5 79 or greater to happen 

only 

by chance alone. 

Is now known from the sgnlficant F value (24.02) that there are 

It 

differences between the three methods. However, a significant 

sgnlfcant 

doesn't indicate where the differences are among the three methods. 

F 

it is necessary to conduct tests between each par of groups. Ths 

Thus, 

would not be necessary, of course, if a significant F was not obtained 

6. To test dfferences for ndlvdual pairs of groups, It is 

Step 

to compute the adjusted means for methods. This adjustment Is 

necessary 

pretest differences between groups First, determine the wlthin-group 

for 

coefficient (b) as follows 

regression 

E. F. Design and analysis of experiments in psychology 

42Lindquist, 

education. Boston Houghton Mifflin, 1953, pp. 317-339'- 

and 

-188-

WORK SHEET

determine the adjusted posttest means for each group uslng the 

Next, 

formula- 

following 

Xy Xy (x Tx) 

Thus, for Method 1 the adjusted posttest mean is 

Y;a 24- .64(19 19) 

9 

24- 64(0) 

=24-0 

For Method 2 the adjusted posttest mean is 

Xya 26 -.64(20 19) 

26 .64(1) 

26 .64 

25.36 

-190-

WORK SHEET

Slmllarly, for Method 3 the adjusted posttest mean Is. 

yj 37 64(18- 19) 

37 64(-1) 

1'37.6'4 

detemne the error variance (o 2) for each of the comparisons 

Then, 

two groups. The fomula s- 

between 

2 : 

1 

+ + 

( 

S -wx ) 

The error varlance for comparing Method 1 wth Method 2 is as follows 

: [.73] 

.64 

-192- 

MSWy

WORK SHEET

Next, test the dlfference by 

24- 25.36 

I,..67 

other two comparlsons are conducted in the same manner wlth the 

The 

ng results- 

follow 

t 

Comparison 

II -.67 

I, 

I, III 6.72 

II, III 5 32 

a t table of critical values (found in statstlcs book) ndl- 

Uslng 

that a t value of 2.78 or greater Is necessary with 4 degrees of 

cates 

freedom (df n I 

+ n2 2) at the .05 probabillty level Thus, the df- 

between Methods 1 and 3 (t = 6.72) and between Methods 2 and 3 

ferences 

5.32) are sgnfcant. 

(t 

-194-

WORK SHEET

IV Interpretatl on 

can be observed by the obtalned F value of 24 02 for three dlf- 

As 

approaches to work adjustment tralnng in a rehabilitation facility, 

ferent 

was reached at the .05 probability level because the obtained 

sgnlflcance 

value exceeded 5.79. This indicates that differences exist among the 

F 

posttest means for the three methods after adjustment for pretest 

WARF 

To determine where these differences are among the three 

dfferences. 

the evaluator must compare n pars the adjusted WARF posttest 

methods, 

using a t-test. These comparisons resulted in signlficant differ- 

means 

between Methods 1 and 3, and between Methods 2 and 3, but no sig- 

ences 

difference between Methods 1 and 2. Again, the adjusted WARF 

nificant 

posttest means were 

1 (productlon workshop assignment wlth 24.00 

Method 

ve counsel ng) 

supportl 

2 (job-slte assignments with co,,nunity 25 36 

Method 

and supportive counseling) 

employers 

3 (workshop, job-sltes, classroom in- 37.64 

Method 

and supportive counseli}g) 

structon 

Method 3 involwng production workshop assignment, job-ste assign- 

Thus, 

with community employers, classroom nstructon and supportive coun- 

ments 

was sCgnificantly more effective in mprowng work adjustment of 

sellng 

as measured by the WARF than Methods 1 and 2 

clients 

-196-

TRAINEE 

RATER 

WORK ADJUSTMENT RATIN8 FORM 

Sub-scale 

Amount of Supervasaon 

A. 

Requared 

B. Realasm of Job Goals 

C. Teamwork 

Acceptance of 

D. 

Rules/Authoraty 

E. Work Tolerance 

F. Perseverance an Work 

Extent Traanee Seeks 

G. 

Assastance 

Importance Attached 

H. 

Job Tralnang 

to 

James A. Batter and D. J. Bolanovach 

SCORING FORM 

(25) 

(34) 

(19) 

(12) 

Level 

DATE 

1 2 3 5 

(5) 

(38) 

(23) 

(32) 

(33) 

(10) 

(3) 

(36) 

(21) 

(6) 

(31) 

(16) 

C1) 

(18) 

(35) 

(28) 

(13) 

(30) 

(39) 

(8) 

(17) 

(26) 

(ii) 

C4) 

(29) 

Note: Numbers an parentheses andcate atem numbers. 

(22) 

(15) 

(24) 

(9) 

(2) 

(27) 

(20) 

(37) 

(14) 

(7) 

(40) 

Total 

Total

INSTRUCT IONS : 

Work Adlustment Ratanq Form as a rating scale consastang 

The 

40 atems relatang to observable work behavlors. You are asked 

of 

answer each atem eather "yes" or "no" for the traanee adenta- 

to 

on the form. Some atems appear very samlar. Answer each 

fled 

atem. Please do not omlt any atems. 

l• 

i0. 

13. 

Mark your response wath an "X" over your choace. 

traanang and darectaon, traanee can work 

Wath 

under occasaonal supervasaon. 

andependently 

has developed realastac 3ob goals and 

Traanee 

seeks counsel an plannang. 

readaly 

can work wath only one or two others 

Traanee 

he partacularly lakes. 

whom 

knows and usually follows rules 

Traanee 

remlnder. 

wlthout 

becomes frustrated and gaves up 

Traanee 

on almost any 3ob- 

easaly 

generally stays at work but as 

Traanee 

dastracted and loses anterest. 

easaly 

seeks assastance only on bonafade 

Traanee 

and after attemptang to solve 

problems 

them hamself. 

desares work but does not do anythang 

Traanee 

to land at. 

hamself 

catches on easaly and does has work 

Traanee 

practacally no supervsaon. 

wlth 

consaders 3ob plans but they are not 

Traanee 

wth has abalataes. 

compatable 

Tralnee works effectively an small (2-3) groups. 

shows open hostalaty to authoraty 

Tralnee 

rules. 

and 

generally works at routine 3obs 

Trainee 

wthout resistance. 

readaly 

Y N 

Y N 

Y N 

Y N 

Y N 

Y N 

Y N 

Y 

Y 

Y 

N 

N

Traanee as a persastent worker on all assagn- 

14. 

even under adverse carcumstances. 

ments, 

Traanee generally handles own problems wlth 

15. 

occasaonal help. 

only 

Traanee expresses anterest an future work, 

16. 

not an a 3ob now. 

but 

Once shown what he must do, traanee applaes 

17. 

dalagently wlthout much supervlsaon. 

hamself 

Traanee has begun to thank about possible 

18. 

for hamself that are wathan has 

occupataons 

capabalatles. 

Trainee as unable to work effectavely wth 

19. 

others. 

any 

Traanee understands rules and regulations 

20. 

adheres to them conslstently. 

and 

Traanee traes sampler 3obs but usually becomes 

21. 

when he encounters changes an 

dascouraged 

routlne. 

Traanee applles hamself dalagently to almost 

22. 

kands of work. 

all 

Trainee frequently seeks help and attentaon 

23. 

personal as well as work-related problems. 

for 

Trainee wants a 3ob and seeks asslstance an 

24. 

to prepare for one. 

tryang 

Traanang works wath daffaculty, even under 

25. 

supervasaon and after gettang 

constant 

considerable traanang. 

Traanee accepts guadance n plannang for 

26. 

3obs, reallzng has lamltataons. 

future 

Traanee as a good team worker an any group 

27. 

a tuata on. 

s 

Traanee generally accepts anstructaons and 

28. 

but requares occasaonal rent%nder. 

rules 

Trainee gets personal satsfactaon from most 

29. 

gaven hlm. 

3obs 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

N 

N 

N 

N 

N 

N 

N 

N 

N

32. 

33. 

34. 

35. 

36. 

37. 

38. 

39. 

40. 

Tralnee usually works steadaly unless dasturbed. 

seeks assastance In manor problems 

Tralnee 

could solve hamself. 

he 

is not nterested an worklng now or 

Trainee 

n the future. 

• 

can work on hs own after thorough 

Trainee 

f hs work s frequently observed 

tranlng 

and checked. 

does not have plans for a 3ob and as 

Tralnee 

ready to consider any. 

not 

can usually work wlth a few others 

Traanee 

conflact. 

wlthout 

delaberately and knowangly vlolates 

Tralnee 

and instructaons on occasaons. 

rules 

s quate flexlble and enjoys any knd 

Tralnee 

work. 

of 

needs frequent relaef "breaks", cannot 

Tralnee 

at even sample tasks. 

keep 

Trainee often needs help and asks for t. 

makes active effort to seek a 3ob 

Trainee 

hmself. 

for 

Work Ad]ustment Ratang Form is covered by Copyright 1969 by 

The 

A. Btter. 

James 

Y 

N 

N 

N 

N 

N 

N 

N 

N 

N

PART FOUR 

Solomon Four-Group Study Deslgn 

0 X 0 

1 

R 

2 

0 0 

3 

R 

4 

R X 0 5 

R 0 6 

R random ass±gnment 

where: 

sub3ects 

of 

X 

service/treatment 

observation/measure- 

01 

before treat- 

ment 

to one experl- 

ment 

0 2 

0 3 

mental group 

observation/measureafter 

treatment 

ment 

one exper±mental 

to 

group 

observation/measurebefore 

treatment 

ment 

one contrast group 

for 

treatment) 

(no 

0 4 

observation 

treatment for one 

after 

group (no treat- 

contrast 

ment) 

observat±on/measurement 

0 

after 

5 

to one 

treatment 

0 6 

experlmental group 


treatment for one 

after 

group (no treat- 

contrast 

ment) 

des±gn calls for a group of sub3ects that are randomly 

Th±s 

to one of four groups. These groups are then examined 

ass±gned 

four dlfferent cond±tlons. The post measures of the four 

under 

are used to determ±ne the effects of pretests and treat- 

groups 

ment upon the sub3ects. 

The evaluator dentfes a sample of sub3ects. From 

Method. 

sample he randomly assigns the members to one of four groups. 

ths 

groups recelve a posttest. Groups one and two are also 

All 

a pretest (0). Groups one and three are the only ones to 

g±ven 

the treatment (X). After the treatment has been con- 

receive 

and the posttest g±ven to the four groups, the posttest 

cluded 

scores are compared. 

Ths des±gn, lke the pretest posttest control group 

Value. 

design, involves random ass±gnment of sub3ects to groups; 

study 

permitting the assumption of equal groups on all possible 

thus, 

It also has experimental and contrast groups 

characteristics. 

comparison. The advantage of the Solomon Four-Group deslgn 

for 

its control for the effects of pretesting. 

is

EXAMPLE 10 

A. General Procedures: Solomon four-group study des±gn. 

Measurement Approach: Test of counselor knowledge 

B. 

placement methods and resources. 

of 

Statlstlcal Procedures: 2 x 2 factorlal analyss 

C. 

varlance. 

of

EXAMPLE i0 

A. General Procedures: Solomon four-group study design. 

Measurement Approach: Test of counselor knowledge 

B. 

placement methods and resources. 

of 

Statlstcal Procedures: 2 x 2 factorial analyss 

C. 

variance. 

of

Sample Evaluatlon Objectlve 

determine the effectiveness of in-servlce tralning on counselors' 

To 

of placement methods and resources. 

knowledge 

II Criteria 

in posttest scores relative to counselor knowledge of 

Difference 

methods and resources by counselors involved in in-service traln- 

placement 

ing and counselors not involved in the training. 


Solomon four-group study design 


RO I 

XO 2 

R 03 04 

R XO s 

R OB 

R = random assignment of 

where 

to two exper- 

subjects 

and two contrast 

mental 

groups 

X service/treatment 

0 observatlon/measurement 


W3Campbell, 

for research. Chicago: Rand McNally, 196'6. 

designs 

-205-

Steps. 

Identify counselors to be included in the study 

1. 

them to one of the four groups. 

assign 

2. Administer a pretest to groups 1 and 2. 

Randomly 

3. Give in-service training (treatment) to groups 1 and 3. 

Administer posttest to all four groups when in-service traln- 

4 

has been completed. 

ing 

Tabulate the results for posttest analysis and apply statls- 

5. 

procedures. For this example, a 2 x 2 factorial analysls of 

tical 

is illustrated , and only the effects of treatment and pre- 

variance 

(main effects) are studied. 

testing 

Measurement Approach- Test of counselor knowledge of placement methods 

B. 

resources 

and 

way to measure the counselor's knowledge of placement methods 

One 

resources is to construct a test. For this example, a fictitious set 

and 

scores for four groups of 5 counselors each is used. Two groups re- 

of 

in-service training in placement methods and resources and two 

ceived 

did not recieve the in-service training. A knowledge test con- 

groups 

of 10 items (allowing scores from 0 10) was given to two groups 

slstlng 

a pretest and all four groups were given the posttest. As the pretest 

as 

are not necessary to determine the affect of pretestlng, they are 

scores 

included in this example. The scores on the posttest for the 20 coun- 

not 

involved n the study are as follows: 

selors 

1 Group 2.. Gr..oup 3 Group .4. 

Group 

6 9 4 

10 

9 5 8 7 

9 5 10 6 

I0 6 I0 6 

8 7 8 5 

E. F. Design and a.nalxsis of experiments in psychology 

4Lindquist, 

education. Boston: Houghton Mifflin, 1953, pp. 108-120. 

and 

-206-

WORK SHEET


1_. Tabulate the results of the posttest scores in an analysls 

Step 

variance double-entry table and calculate the mean of each group, the 

of 

means, column means, and general mean. The mean for each group (cell) 

row 

determined by summing the scores and divldlng by the number of subjects, 

is 

group 1 sum is 46, the number of subjects is 5, therefore, the mean 

e.x. 

9.2. The means for rows and columns are done in the same manner. For 

is 

amp I e 

ex 

PRETEST 

Yes 

No 

I0 

9 

9 

10 

8 

9 

8 

10 

10 

8 

TREATMENT 

(In-serw ce 

Training) 

Row 

No Means 

Yes 

Group 1 

Xi=92 

sXI 46 

Group 3 

X3 9 

zX3 = 45 

Group 2 

X2 5.8 

.X2 29 

Group 4 

X4 = 5.6 

sX 28 

o I umn 

C 

X 9.1 X 5 7 

Means 

': mean (average) 

7.5 

row 1 group' 

of 

= mean (average 

7.3 

row 2 group, 

of 

4 General Mean for 

7 

scores 

all 

2. Calculate the preliminary information that is needed to 

Step 

out the analyss of data. This would look llke the following when 

carry 

up (explanations for each of the symbols and sample calculations ap- 

set 

on pages 210 214). 

pear 

-208-

WORK SHEET

Row 1 T. 46 29 

Col 1 Col 2 T 

426 171 

423.2 168.2 

Row 2 T

WORK SHEET

T# Is the sum of scores for each group, e g. for group 

where 

(column one, row one) T# 10 + 9 + 9 + 10 + 

one 

8:46 

is the sum across rows, e.g. for row one T 46 + 

TC 

75 

29 

instructs us to square the T value prevlously cal- 

T2/m 

and divide by the number of subjects across 

culated 

rows, e.g. for row one T2/m; 75 x 75 10 562 5 

2 instructs us to take each X in a given group, square 

zX 

value, and then sum the results, e.g. 10 2 + 92 

that 

+ 92 + 10 2 + 82 : 426 

us to square the Tj value already cal- 

nstructs 

and then divide by mi# which is the number 

culated 

subjects in a given group, e.g for group one 

T2/mj of 

x 46 5 423.2 

46 

Is the sum of scores for columns, e g. for column 

T# 

T 46 + 45 91 

one 

us to square the prewously obtained T 

nstructs 

and dwde by the number of subjects n the 

value 

e.g. for column one T2/m 91 x 91 10 

column, 

8281 10 828.1 

2 instructs us to add all of the zX previously cal- 

zz.X 

values for the four groups together, e.g. 

culated 

426 + 171 + 409 + 162 1168.0 

us to add the four values for zT..?2/m 

instructs 

e.g. 423.2 + 168.2 + 405 0 + 156 8 1153 2 

together, 

is the value obtained from adding the two values 

zT2/m 

TC2/m together, e.g 562.5 + 532 9 1095 4 

for 

-212-

WORK SHEET

is the value obtained from adding the two values for 

F#2/m# 

zTjZ/nj 

e.g 828.1 + 324.9 1153 0 

together, 

instructs us to flnd the total value of all scores 

T2/N 

together, square that value, and divlde by 

added 

the total number of subjects, e.g. 1482 20 = 1095.2 

3_. Substitute the information obtalned in the previous calcula- 

Step 

nto the following sum of squares formulas 

tlons 

SSTotal = 

SSR°ws 

r c ni 

. X 2 T2/N 

z 

#=I 

i=1 

r Ti/n i T2/N 

Z 

i=1 

T2/n# T2/N 

SSCol z 

#=1 

umns 

SSCell s 

r 

s Ti2/ni T2/N 

s 

#=1 

i=1 

To determlne the followlng, use the above sum of squares informatlon 

SSRows x Columns SSCelIs SSRows SSColumns 

SSwithln SSTota I SSCell s 

-214-

WORK SHEET

For thls example, the results would appear as follows- 

SS T 

SS R 

SS c 

1168.0- 1095.2 72.8 

1095.4 1095.2 .2 

1153.0 1095.2 57.8 

SSCell s =1153.2 1095.2 58.0 

SSRc 58.0 .2 57.8 0 

SS W 

72.8 58.0 14.8 

Complete a Summary Table by substituting the above Informa- 

St__p_4 

In the followlng format and conduct the necessary calculations to 

ton 

arrive at F values. 

of 

Source 

at on 

Var 

Columns (C) 

Rows 

(Cells) 

(R) 

x Columns 

Rows 

(RC) 

thn Cells 

W 

(w) 

Total 

df 

c-1 

r-1 

(rc-1) 

(r-l)(c-1) 

N-rc 

N-1 

Summary Table 

of Squares 

Sum 

(SS) 

SS c z T#21n# T21N 

R Z Ti2/n i T2/N 

SS 

,=1 

o 

SSCells 

1 

s T#2/m# T2/N 

s 

i=1 j=l 

SSRc SSCell s 

SS W 

SS T 

SS R 

SSCell s 

SS C 

z z z X 2 T2/N 

SST 

;=1 #=1 

-216- 

Square 

Mean 

MS 

C 

SS 

c-1 

R 

SS 

r-1 

SSRc 

(r-1)(c-1) 

W 

SS 

N-rc 

C 

MS 

W MS 

MS 

R 

MSw

WORK SHEET

The calculations for the degrees of freedom are as follows. 

c-1 

r-1 

rc-1 

represents the number of columns less 1 (2 I 1) 

represents the number of rows less I (2 1 1) 

the number of rows tlmes the number of columns 

represents 

1 (2 x 2 i 3) 

less 

represents the number of rows less 1 tlmes the number 

(r-1)(c-1) 

columns less 1 I(2 1)(2 I) 11 

of 

represents the total number of subjects less rows tlmes 

N-rc 

120 -_(2 x 2) 161 

columns 

represents the total number of subjects less 1 

N-1 

1 19) 

(20- 

Completed summary table using example nformaton- 

of 

Source 

Variation 

Col umns 

Rows 

Cells 

x"Col umns 

Rows' 

( I nteractl ons ) 

df 

1 

of Squares 

Sum 

(SS) 

57.8 

2 

58 0 

MS 

57 8 

Wthn Cells 16 14 8 .9 

Total 19 72.8 

*9_

WORK SHEET

degrees of freedom for uslng a table of crltlcal values for F is 

The 

by the corresponding Sun=nary Table df used to calculate F. For 

determlned 

the F 64.22 was calculated from MS c (df 1) and MS W (df 16), 

example, 

the degrees of freedom used for looking up the critical value of F 

thus, 

1 and 16. The necessary F value for slgnificance at the .05 probability 

are 

is 4.49 This means that only 5 times out of 100 would we expect a 

level 

F of 4.49 or greater to happen by chance alone. 


can be observed by the obtained F values, sgnlficance was reached 

As 

better than the .05 probability level for treatment (column) effects 

at 

F value for pretest (row) effects was not sgnficant at the 05 prob- 

The 

level. Recall that the mean posttest scores for counselors re- 

abllity 

training was 9.1, and for those not receiwng training was 5.7. 

ceivlng 

in this fictitious example, the n-service training significantly 

Thus, 

performance on the test of counselor knowledge of placement 

Improved 

and resources. Giving a pretest did not significantly influence 

methods 

scores. The mean posttest score for those receiving a pretest 

posttest 

7.5, for those not pretested it was 7.3. 

was 

-220-

REFERENCES 

J. R., Brown, F. L., & Mink, O. J. Statistical concepts: A basic 

Amos, 

New York- Harper and Row, 1965. 

program. 

E. C. & Weisinger, M. Program evaluation- A resource handbook 

Bennett, 

vocational rehabilitation. New York: ICD Rehabilitation and 

for 

Research Center, 1974. 

J. A. Bias effect on validity and reliabClity of a rating scale. 

Bitter, 

and Evaluation in Guidance, 1970, 3_(2), 70-75. 

Measurement 

J. A. & Bolanovich, D. J. WARF- A scale for measuring job- 

Bitter, 

behaviors. American Journal of Mental Deficlenc, 74(5), 

readiness 

616-621. 

W. R. & Merideth, D. G. Educational research. An introduction. 

Borg, 

Ed.) New York: Davl'd McKay Company, Inc., 63. 

(2nd 

J. L. & Keintz, B. L. Computational handbook of statistics. 

Bruning, 

Scott, Foresman and Co., 1968. 

Dallas- 


Campbell, 

for research. Chlcago: Rand McNal--, 1966, 

designs 

F., Zawada, A., Thompson, B., & Markowitz, J. Guidelines and 

Collignon, 

for evaluating vocational rehabilitation programs: A dis- 

criteria 

paper for the prime study group on program evaluatlon, Tenth 

cussion 

on Rehabilitation Services, Berkeley: Institute of Urban 

Institute 

& Regional Development, University of Californla, 1972. 

O. L., Rosenstock, I. M., & Gettlng, V. A. Evaluation of pro- 

Deniston, 

effectiveness. Public Health Reports, 1968, 83(4), 323-335. 

gram 

G. A. Statistical analysls n psychology and education. New 

Ferguson, 

McGraw-Hi I l, 1966. 

York: 

J. P. Fundamental statstlcs n psychology and educatlon. New 

Guilford, 

McGraw-Hill, 19'0. 

York" 

L. A. A bass for scaling qualitative data. American Sociological 

Guttman, 

1944, 9, 139-150. 

Review, 

Hatry, H. P., Wnnie, R. E., & Fisk, D. M. Practical program evaluation 

state and local overnment officials'. Washington, D.C.: Urban 

for 

n s----ti tute ,73. 

I 

W. G. & V1alle, H. V. Consumer's measurement of vocational reha- 

H111s, 

Norman, OK. University of Oklahoma Pr--ess, 1973. 

bilitation. 

-221-

T. J. Goal attalnment scaling at a county mental health servlce. 

Kiresuk, 

1973, 1, 12-18. 

Evaluation, 

T. J. & Sherman, R. E. Goal attainment scallng. A general method 

Kresuk, 

evaluating comprehensive community mental health programs. Com- 

for 

munity Mental Health Journal, 1968, 4_(6), 443-453. 

F. N. Foundations of behavioral research. New York- Holt, 

Kerlinger, 

& Winston, 1964-- 

Rnehart, 

E. F. Design and anal)'sis of experiments In psychology and 

Lindquist, 

Boston Houghton-Mifflin, 1953 

education. 

R. F. Preparing instructlonal objectives. Palo Alto, CA- Feardon 

Mager, 

1962. 

Publishers, 

A. H. Motivation and personality. (2nd Ed.) Evanston, IL- Harper 

Maslow, 

Row, 1970. 

and 

R. B. Fundamental statlstics for psychology. New York. Harcourt, 

McCall, 

and World, Inc., 1970. 

Brace 

K. W Kravetz, S., Wright, G. N., & Butler, A. J. A better 

Reagles, 

The Human Service Scale. Madison. The University of 

yardstick" 

Wisconsin Regional Rehabilitation Research Institute, 1974. 

K. W., Wright, G. N.,& Butler, A.J. Human Serwce Scale. Madison, 

Reagles, 

Human Service Systems, Inc., 1973. 

WI: 

I. M. & Guttman, L. Smallest Space Analysis of Intellgence 

Schlesinger, 

achievement test. Psychological Bulletin, 1969, 71, 95-100. 

and 

D. T. The difficult), ndex an expanded measure of counselor 

Sermon, 

St Paul. Minnesota Divslon of Vocatiol' Rehabi'i'"ta- 

performance. 

tion, Research Monograph 1, 1972. 

S. N..onparametric statstlcs for the behavioral sclences. New 

Segel, 

McGraw-Hll, 1956. 

York- 

E. A. Action for what A crtlque of evaluation research. In 

Suchman, 

R. (Ed.). The organlzaton, management, and tactics of soclal 

O'Toole, 

research. Cambridgeassachusetts Schenkman Publishing Company, 1971. 

Institute on Rehab11tation Services. Program evaluation A begin- 

Tenth 

statement. Washington, D.C. Rehabilitation Servlces A-minlstra- 

ning 

U. S. Department of Health, Education, and Welfare, 1972. 

tion, 

D. J. An introduction to evaluatlon Program effectiveness and 

Trantow, 

need. Rehabilitation Literature, 1970, 31(1), 2-9, 12. 

connunity 

-222-

df 

SP 

SS 

SW2 

T 

0 

P 

r 

D 

F 

GLOSSARY OF SYMBOLS 

in a regression equation Indlcatlng the orlgln or inter- 

constant 

of a regression 11he with the Y axls 

cept 

regression welght reflecting the slope of a regression line. 

the number of columns 

a statistic whose value Is computed fr observed 

cht-square, 

e.g. from frequency data. 

data, 

of freedom. 

degrees 

between paired measurements. 

difference 

frequency in the calculation of X 2 

expected 

frequency 

the F test- analysis of variance. 

a subscript denoting rows. 

a subscript denoting columns 

number of groups or subclasses 

square (in analysis of variance) 

mean 

of observations in a subclass 

number 

number of observations n a sample, 

total 

frequency in the calculation of X 2. 

observed 

11ty estimate. 

probabl 

I n=o, o, row,, o, 

l, 

the Pearson product-moment correlation coefflcleet. 

of products (in analyss of covarlance) 

sum 

of squares (in analysis of variance). 

sum 

mean square wlthln (In a $cheffd test) 

t-test. 

the 

standardt°tal (inscore.analysis of variance), or 

}) 

indicating a level of mportance 

weight, 

expressed as scores 

varlables 

the mean (or average) of a sample of observations 

pred cted score. 

standard error or deviation of a population or sample 

sum of, symbols above and below deflne lmits of the sum- 

the 

on. mati 

a superscript denoting Instructlons to square a value, e.g. 4 2 16 

root' sign. 

'square 

s gn. 

equal 

than' s gn. 

greater 

than' sign. 

'less

(Revised) - NCRTM

Create successful ePaper yourself

Delete template?

Save as template?