2003 IMTA Proceedings - International Military Testing Association

Do we assess what we want to assess?
The appraisal dimensions at the Assessment Center for Professional Officers (ACABO)
Dr. Hubert Annen & Lic. phil. Barbara Kamer
Military Academy at the Swiss Federal Institute of Technology
Steinacherstrasse 101b, CH-8804 Au
hubert.annen@milak.ethz.ch
INTRODUCTION
The Assessment Center is a widely used tool for the selection and development of managers
in various organizations. Many studies have demonstrated that assessment center appraisals
predict a variety of important organizational criteria, such as training and job performance or
promotion (Gaugler, Rosenthal, Thornton & Bentson, 1987; McEvoy, Beaty, & Bernardin,
1987).
It appears therefore that the assessment center has a good predictive validity. However,
scientists seem to differ about what each individual dimension and what the tool itself really
measure or what the observers exactly assess. Because when it comes to construct validity,
most studies show a similar picture: ratings of multiple dimensions within a single exercise
correlate higher than do ratings of the same dimension across multiple exercises (Annen,
1995; Bycio, Alvares, & Hahn, 1987; Kleinmann, Kuptsch, & Köller, 1996; Robertson,
Gratton, & Sharpley, 1987; Sackett & Dreher, 1982; Turnage & Muchinsky, 1984).
There are many hypotheses which try to explain why the assessment center is nevertheless a
good predictor for future job success. Russel und Domm (1995) for example say that
Assessment Centers have such a high prognostic value because they measure attitudes which
are of importance for the future job. For a better understanding of the predictive value of the
assessment center, Shore, Thornton und Shore (1990) claim that the construct validity of
dimension ratings should be explored by building a nomological network of related
constructs. Their own studies showed that, during an assessment center, cognitive ability
correlate stronger with problem-solving dimensions and that personal traits have a stronger
connection with interpersonal dimensions. Other studies focus on the connection between
personality factors and cognitive ability respectively and the performance in the assessment
center, and they have produced significant results (Crawley, Pinder, & Herriot, 1990; Fleenor,
1996; Chan, 1996; Goffin, Rothstein & Johnston, 1996; Scholz & Schuler, 1993, Spector,
Schneider, Vance, Hezlett, 2000).
Taking into account the various studies made on the subject, it seems to be difficult to
establish a correlation between the results of assessment center and certain other criteria. Each
assessment center in an organization is tailored to persons with a specific job background.
Depending on the job profile which the successful candidate should meet other criteria of
observation are also used and operationalized according to the requirements of the job. It is
therefore of vital importance to have a clear idea of what is really measured through the given
dimensions and whether or not we really measure what we mean to measure.
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003
13