Core Abilities Assessment: Evidence of Reliability and ... - TalentLens

Core Abilities AssessmentEvidence of Reliability and Validity888-298-6227 • TalentLens.comCopyright © 2007 NCS Pearson, Inc. All rights reserved.

Copyright © 2007 by NCS Pearson, Inc.All rights reserved. No part of this publication may be reproduced or transmitted in any form or by anymeans, electronic or mechanical, including photocopy, recording, or any information storage and retrievalsystem, without permission in writing from the copyright owner.Pearson and TalentLens logos are trademarks, in the U.S. and/or other countries, ofPearson Education, Inc. or its affiliate(s).Portions of this work were previously published.Printed in the United States of America.

Core Abilities AssessmentEvidence of Reliability and ValidityInternal Consistency ReliabilityThe internal consistency reliability estimate for the Core Abilities Assessment total raw scorewas .85 in the standardization sample of 314 individuals. (See the Appendix for more detailsregarding the composition of the sample.) This reliability estimate indicates that the total rawscore on the Core Abilities Assessment possesses good internal consistency reliability asprovided in the guidelines of the U.S. Department of Labor (1999) for interpreting areliability coefficient.Content ValidityIn an employment setting, evidence of content validity is demonstrated by measuringcompetencies that are required for the job. The competencies measured by the Core AbilitiesAssessment (i.e., general mental ability, inductive and deductive reasoning, verbal reasoning,numerical ability, and abstract reasoning) are required for a broad range of jobs. For example,the Department of Labor’s O*NET database lists over 500 jobs for which inductive reasoningability was rated as being important, very important, or critical for success (O*NET OnLine,2007). To confirm the relevance of the Core Abilities Assessment for the job you are hiring for,we recommend that you compare your job description and other sources of job information tothe competencies measured by the assessment. For legal defensibility, the assessment solutionmust show relevance for the position for which it is used.Convergent ValidityEvidence of convergent validity is provided when scores on an assessment relate to scores onother assessments that claim to measure similar traits or constructs. Convergent validity for theCore Abilities Assessment is supported by part-whole correlations with the Differential AptitudeTests for Personnel and Career Assessment (DAT for PCA; Bennett, Seashore, & Wesman,1991), DAT for PCA convergent validity research, and results from the Core AbilitiesAssessment standardization study.- 1 -Copyright © 2007 NCS Pearson, Inc. All rights reserved.

Part-Whole Correlations with the DAT for PCAConvergent validity for the Core Abilities Assessment is supported by correlations among CoreAbilities Assessment items and the DAT for PCA tests that the items were drawn from. Aspresented in Table 1, these correlations range from .85 to .90, providing evidence that theabbreviated item sets from the Core Abilities Assessment measure the same constructs as thecorresponding DAT for PCA full-length tests.Table 1. Correlations between Core Abilities Assessment and DAT for PCA TestsCore Abilities AssessmentDAT for PCAItem Subset Number of Items Test Number of Items r part-wholeVerbal Reasoning 6 Verbal Reasoning 30 .87Numerical Ability 7 Numerical Ability 25 .90Abstract Reasoning 7 Abstract Reasoning 30 .85Note. n = 569 for Verbal Reasoning correlation; n = 1057 for Numerical Ability correlation; n = 1695 for AbstractReasoning correlation. All correlations are significant at the .001 level.DAT for PCA Convergent Validity StudiesYears of previous studies on the DAT for PCA support its convergent validity, and by extension,the convergent validity of the tests comprising the Core Abilities Assessment. For example,scores on the DAT Verbal Reasoning and Numerical Ability composite correlate in the range of.38 to .79 with SAT scores and .68 to .94 with ACT scores (Bennett, et al., 1991). Similarly, DATAbstract Reasoning scores correlate highly with general mental ability scores on assessmentssuch as the General Aptitude Test Battery (r = .64 with General Learning Ability) and the ArmedServices Vocational Aptitude Battery (r = .69 with the Armed Forces Qualification Test).- 2 -Copyright © 2007 NCS Pearson, Inc. All rights reserved.

Core Abilities Assessment StandardizationStudy — Convergent ValidityThe Core Abilities Assessment standardization study provides further evidence for theassessment’s convergent validity. In the standardization study, Core Abilities Assessmentscores correlated .84 with Wonderlic Personnel Test scores (Wonderlic, 1992; n = 205) and .56with Differential Aptitude Test Mechanical Reasoning scores (DAT MR; Bennett, et al., 1991; n =123). The Wonderlic Personnel Test, like the Core Abilities Assessment, measures generalmental ability, and the DAT MR measures mechanical aptitude, which typically correlatesmoderately to strongly with general mental ability (Bennett, 2006).Criterion-Related ValidityCriterion-related validity addresses the inference that individuals who score better on anassessment will be more successful on some criterion of interest. Criterion-related validity forthe Core Abilities Assessment is supported by three lines of research, including validitygeneralization, DAT for PCA criterion-related validity studies, and the Core Abilities Assessmentstandardization study.Validity GeneralizationCriterion-related validity for general mental ability tests like the Core Abilities Assessment issupported by validity generalization. The principle of validity generalization refers to the extentthat inferences from accumulated evidence of criterion-related validity from previous researchcan be generalized to a new situation.There is abundant evidence that measures of general mental ability, such as the Core AbilitiesAssessment, are significant predictors of overall performance across jobs. For example, in itspublication on the Principles for the Validation and Use of Personnel Selection Procedures, theSociety for Industrial and Organizational Psychology (2003) notes that validity generalization iswell-established for cognitive ability tests. Schmidt & Hunter (2004) provide evidence thatgeneral mental ability “predicts both occupational level attained and performance within one’schosen occupation and does so better than any other ability, trait, or disposition and better thanjob experience” (p. 162). Prien, Schippmann, and Prien (2003) observe that decades ofresearch “present incontrovertible evidence supporting the use of cognitive ability acrosssituations and occupations with varying job requirements” (p. 55). Many other studies provideevidence of the relationship between general mental ability and job performance (e.g., Kolz,- 3 -Copyright © 2007 NCS Pearson, Inc. All rights reserved.

McFarland, & Silverman, 1998; Kuncel, Hezlett, & Ones, 2004; Ree & Carretta, 1998; Salgado,et al., 2003; Schmidt & Hunter, 1998; Schmidt & Hunter, 2004).DAT for PCA Criterion-Related Validity StudiesIn addition to inferences based on validity generalization, findings from numerous studiesprovide evidence of criterion-related validity for the DAT for PCA tests that compose the CoreAbilities Assessment. For example, of the 36 uncorrected validity coefficients obtained across15 studies on DAT Verbal Reasoning (Bennett, et al., 1991), all met the U.S. Department ofLabor (DOL) criterion (1999) for an assessment that is “likely to be useful” (i.e., validitycoefficients ranging from .21 to .35) and 20 met the highest DOL standard, “very beneficial”(i.e., validity coefficients greater than .35). Similarly, for DAT Numerical Ability, 29 of 29uncorrected validity coefficients obtained across 13 studies met the DOL “likely to be useful”criterion, and 11 met the highest standard of “very beneficial.” For DAT Abstract Reasoning, 24of 24 uncorrected validity coefficients obtained across 21 studies met the DOL “likely to beuseful” criterion, and 16 met the highest standard of “very beneficial.” The DAT for PCA Manual(Bennett, et al., 1991) provides additional details on these studies.Core Abilities Assessment StandardizationStudy — Criterion-Related ValidityThe Core Abilities Assessment standardization study provides further evidence for theassessment’s criterion-related validity. In the standardization study, Core Abilities Assessmentscores correlated .32 (uncorrected) with supervisor ratings of problem-solving behavior for agroup of 115 plant mechanics at a national manufacturing company.Importance of Local ValidationThe validity information presented in this guide is not intended to serve as a substitute for locallyobtained validity data. Local validity studies, together with locally derived norms, provide asound basis for determining the most appropriate use of Core Abilities Assessment scores.Therefore, whenever technically feasible, users of the Core Abilities Assessment should studythe validity of the assessment at their own location or organization.- 4 -Copyright © 2007 NCS Pearson, Inc. All rights reserved.

Description of the SampleAppendixThe information provided in the following tables is based on Core Abilities Assessment datacollected during the period December, 2006 through June, 2007.Appendix 1. Composition of the Sample by Job Title (n = 288*)OccupationAdministrative Assistant 2.1%Clerk 2.1%Engineer 1.0%Human Resources Occupations 1.0%Information Technology Occupations 1.0%Maintenance 2.1%Manager 3.8%Operations Planner 1.0%Plant Mechanic 67.7%Purchasing 1.4%QC Specialist 1.4%Sales Representative (Non-Retail) 5.2%Supervisor 4.2%Team Leader 1.0%Vice President 1.0%Other 4.0%*Twenty-six individuals did not provide a response about their job title.- 5 -Copyright © 2007 NCS Pearson, Inc. All rights reserved.

Appendix 2. Composition of the Sample by Position Level (n = 288*)Position LevelExecutive; Director 1.4%Manager 3.8%Supervisor 4.2%Professional/Individual Contributor 13.9%Skilled Trades 71.9%Hourly/Entry-Level 4.9%*Twenty-six individuals did not provide a response about their position level.- 6 -Copyright © 2007 NCS Pearson, Inc. All rights reserved.

ReferencesBennett, G. (2006). Bennett Mechanical Comprehension Test manual. San Antonio, TX:Pearson Education, Inc.Bennett, G., Seashore, H., & Wesman, A. (1991). Technical manual for the DifferentialAptitude Tests for Personnel and Career Assessment. San Antonio, TX:The Psychological Corporation.Kolz, A. R., McFarland, L. A., & Silverman, S. B. (1998). Cognitive ability and job experience aspredictors of work performance. The Journal of Psychology, 132(5), 539–548.Kuncel, N. A., Hezlett, S. A., & Ones, D. S. (2004). Academic performance, career potential,creativity, and job performance: Can one construct predict them all? Journal ofPersonality and Social Psychology, 86(1), 148–161.O*NET OnLine (2007). Welcome to O*NET OnLine! Making occupational informationinteractive and accessible to all. Occupational Information Network: O*NET OnLine.Retrieved July 25, 2007 via O*NET OnLine Access: http://online.onetcenter.orgPrien, E. P., Schippmann, J. S., & Prien, K. O. (2003). Individual assessment as practiced inindustry and consulting. Mahwah, NJ: Erlbaum.Ree, M. J., & Carretta, T. R. (1998). General cognitive ability and occupational performance. InC. L. Cooper & I. T. Robertson (Eds.), International review of industrial andorganizational psychology (Vol. 13, pp. 159–184). Chichester, England: Wiley.Salgado, J., Anderson, N., Moscoso, S., Bertua, C., & Fruyt, F. (2003). International validitygeneralization of GMA and cognitive abilities: A European community meta-analysis.Personnel Psychology, 56, 573–605.Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnelpsychology: Practical and theoretical implications of 85 years of research findings.Psychological Bulletin, 124, 262–274.Schmidt, F., & Hunter, J. (2004). General mental ability in the world of work: Occupationalattainment and job performance. Journal of Personality and Social Psychology,86(1), 162–173.Society for Industrial and Organizational Psychology. (2003). Principles for the validation anduse of personnel selection procedures (4 th ed.). Bowling Green, OH: Author.- 7 -Copyright © 2007 NCS Pearson, Inc. All rights reserved.

U.S. Department of Labor. (1999). Testing and assessment: An employer’s guide to goodpractices. Washington, DC: Author.Wonderlic, E. F. (1992). Wonderlic Personnel Test & scholastic level exam user’s manual.Libertyville, IL: Wonderlic Personnel Test.- 8 -Copyright © 2007 NCS Pearson, Inc. All rights reserved.