Projects on Competencies in the OECD Context - Analysis - DeSeCo

Definition and Selection of Competencies<strong>Projects</strong> on Competencies in the OECDContextAnalysis of Theoretical and Conceptual FoundationsOFS BFS USTESSINeuchâtel, 1999

Definition and Selection of Competencies<strong>Projects</strong> on Competencies in theOECD ContextAnalysis of Theoretical and Conceptual FoundationsAuthorsLaura Hersh SalganikEducation Statistics Services Institute, American Institutes for ResearchDominique Simone RychenSwiss Federal Statistical OfficeUrs MoserUniversity of BernJohn W. KonstantEducation Statistics Services Institute, American Institutes for ResearchThis work has been supported by the United States Department of EducationNational Center for Education Statistics and the Swiss Federal Statistical Office.OFS BFS USTSwiss Federal Statistical OfficeNeuchâtel, 1999

Information: Heinz Gilomen (SFSO), 'Phone 41(0)32 713 68 38Dominique Simone Rychen (SFSO), Phone 41(0)32 713 61 60Editor:Swiss Federal Statistical Office (SFSO)Organisation for Economic Co-operation and Development (OECD)Education Statistics Services Institute, American Institutes forResearch (ESSI)Obtainable from: Swiss Federal Statistical Office (SFSO)CH-2010 Neuchâtel'Phone 41 (0)32 713 60 60 / Fax 41 (0)32 713 60 61OECD/OCDEMail Orders:2, rue André-Pascal75775 Paris Cedex 16'Phone 33 (0)1 45 24 82 00 / Fax 33 (0)1 49 10 42 76Internet: Compte.PUBSINQ@oecd.orgOrder nuber: 312-9900Price: Sfr 6.–Graphics/Layout: A. de Bruin, Publiduty (cover), SFSO (layout)Copyright: SFSO, Neuchâtel 1999Reproduction with mention of source authorized(except for commercial purposes)ISBN: 3-303-15197-0

Definition and Selection of CompetenciesContentForeword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Main Theses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Purpose of this Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Cross-Curricular Competencies Project (CCC) . . . . . . . . . . . . . . 13Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Discussion and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14International Adult Literacy Survey (IALS) . . . . . . . . . . . . . . . . 23Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Discussion and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Human Capital Indicators Project . . . . . . . . . . . . . . . . . . . . . . . . 31Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Discussion and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32Ongoing <strong>Projects</strong> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37International Life Skills Survey (ILSS) . . . . . . . . . . . . . . . . . . . . . . . . . 37Program for International Student Assessment (PISA) . . . . . . . . . . . 37CCC Problem Solving Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39IEA Civic Education Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Third International Mathematics and Science Study(TIMSS and TIMSS-R) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Annexe A: The INES Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Annexe B: Interview Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . 49SFSO OECD ESSI 3

Definition and Selection of Competencies4 ESSI OECD SFSO

Definition and Selection of CompetenciesForewordRapid changes in economic, social, and political life, including thoserelated to the advent of new technologies and on-going globalization,pose major challenges in today’s world. Individuals, communities, workorganizations and nations increasingly recognize that their future wellbeingdepends on high levels of knowledge, skills, and competencies.This recognition, together with an increased focus on the outcomes ofeducation, has led policymakers to seek out information about skill levelsof the population and about the effects of education, training, andinformal learning on those skills.To inform the policy-making process, OECD is developing internationallycomparable indicators of skills and competencies, and of theirroles in promoting individual, social, and economic well-being. To thisend, OECD Member countries are collaborating on the development ofa range of instruments designed to deliver reliable and policy-relevantmeasures on learning outcomes and on the distribution of skills in thepopulation.While the development of the empirical components of these effortsis well underway, there is a clear need to further advance thetheoretical underpinnings of the assessment of skills and competencies.As a contribution to such work, the Swiss Federal Statistical Office haslaunched, within the framework of the OECD indicators project, athree-year program, entitled Definition and Selection of Competencies:Theoretical and Conceptual Foundations (DeSeCo), with the aim ofidentifying a set of competencies that are needed by both children andadults to lead responsible and successful lives in a modern, democraticsociety and for society to face the challenges of the present and thefuture. The program also seeks to advance the development of a common,overarching theoretical framework for the identification of keycompetencies that can provide a basis for more accurate and appropriatemeasurement of competencies and interpretation of empirical results.International and interdisciplinary collaboration are key elements ofthe program; distinguished scientists, policymakers, and leading representativesof economic and social institutions are contributing to thework.SFSO OECD ESSI 5

Definition and Selection of CompetenciesThe first major activity of the DeSeCo program was an analysis ofprevious INES-related activities that address issues related to the selection,definition, and operationalization of skills and competencies: theCross-Curricular Competencies Project, the International Adult LiteracySurvey, and the Human Capital Indicators Project.This report presents the findings of the analysis, which was based ona review of project documentation as well as interviews with key actorsin each of these projects. It also briefly discusses several ongoing studies.Heinz GilomenAcknowledgmentsThe authors would like to express their thanks to the individualswho generously gave us their time and thoughtful observations: TomAlexander, Marilyn Binkley, Norberto Bottani, Denis Bonora, HelmutFend, Heinz Gilomen, Jeanne Griffith, Aletta Grisay, Tom Healy, DouglasHodgkinson, Walo Hutmacher, Scott Murray, Eugene Owen, JulesPeschar, Andreas Schleicher, Judith Torney-Purta, Uri Peter Trier, andAlbert Tuijnman.6 ESSI OECD SFSO

Definition and Selection of CompetenciesMain ThesesThis report is based on the study of three completed projects in theOECD context: the International Adult Literacy Survey, the Cross-CurricularCompetencies Project, and the Human Capital Indicators Project.While ongoing projects are also briefly described, the conclusions andtheses of the report reflect the analysis of the completed projects. Inseveral cases, the ongoing projects have displayed a sensitivity to theissues detailed below, and in that sense, reflect development beyondthe earlier projects. At the same time though, the authors believe thatthe conclusions and theses remain a key resource for the ongoingprojects as they continue their development and for future projects aswell.Decentralization. Growing demand by national authorities foroutput-oriented information about education has resulted in a numberof projects in the OECD context. Although their goals are related, theseprojects have operated more or less independently, with minimal conceptualand organizational coordination related to selecting and definingcompetencies. The process of definition and selection has beenlargely determined by individual initiatives and particular national interestsrather than an overarching strategy.Broad conception of indicators. Each of the studies adopted abroad conception of competencies and outcomes of education. This includesthe viewpoint that the desired outcomes of education arebroader than the acquisition of subject-related knowledge typicallytaught in school and that the idea of competencies extends beyond theschool context. The notion of preparation for life has become a leadingtheme in the process of definition and selection of competencies withinthe OECD context.Focus on indicators and measurement. Feasibility of measurementwas a major concern in the selection and definition of competenciesin each of these projects. Although not always the goal at the onset,each of the projects eventually gave priority to empirical testing ofconcepts by applying existing methodologies, instruments, and, whenpossible, existing data. Production of indicators was of prime importance.SFSO OECD ESSI 7

Definition and Selection of CompetenciesLimited theoretical and conceptual development. Theoreticaland conceptual issues were not at the forefront of these projects. Forthe CCC project, there was little scientific base from which to draw andthe question of whether it was feasible to develop measures took priorityover conceptual and theoretical issues. The scientific foundationpreviously developed in the United States was a key factor in the selection,conceptualization, and measurement of adult literacy for theIALS. The HCI project expanded the classical conception of human capital,but its priority was the development of indicators based on existingdata. In addition, studies were not concerned with how competenciesare interrelated. No strategy was developed for addressing theoreticaland conceptual issues.Accomplishments and the need for future involvement ofthe scientific community. Taken together, the studies succeeded inproviding a great deal of information for policymakers and in generatinginterest in competencies beyond the field of education and supportfor efforts that are currently underway. However, because limited resourceswere available on the international level for conceptual andtheoretical work, the studies drew mainly from the existing scientificbase. Because of the need to expand that base, close cooperation withthe scientific community in future efforts to define, conceptualize, andmeasure competencies is crucial to the future success of work in thedomain of competence.8 ESSI OECD SFSO

Definition and Selection of CompetenciesIntroductionBackgroundRecognizing an increasing demand for comparative statistical informationabout education in member countries, the OECD launched theINES project (Indicators of National Education Systems) in 1987. Theproject initiated a number of international working groups, each focusingon a particular area and providing a venue for exchanging viewpointsand achieving a common understanding of issues. The overallpurpose was to improve the gathering and reporting of informationabout education. One of these groups – Network A – was given thetask of developing indicators of learning outcomes (see Annex A foradditional information about INES Networks).It was apparent from the beginning of the work of INES and NetworkA that data providing direct measures of learning outcomes wereextremely limited. Indicators of the math achievement of 13-year olds,published in the first edition of Education at a Glance in 1992 (OECD,1992), were identified as provisional because of their experimental nature.Efforts such as those which provided data for these indicators (theIAEP study conducted by the United States and the Second InternationalMath Study conducted by the IEA 1 ) were conducted at irregularintervals and only in selected areas of the school curriculum. Further,these studies were defined by subjects in the school curriculum, and didnot reflect more general competencies applicable in multiple curricularareas or competencies needed for life outside of schools. Finally, directmeasures of the knowledge and skills of adults were nonexistent.At the same time, there was a growing demand from national authoritiesfor output-oriented information about schools. Whereas1The International Assessment of Educational Progress (IAEP) was a project conducted by the EducationTesting Service utilizing the methodology of the United States’ National Assessment of EducationalProgress (NAEP) on an international scale (Lapointe, et al. 1992a, 1992b ). The InternationalAssociation for the Evaluation of Educational Achievement (IEA) is an association of research institutions,universities, and ministries of education in over 45 countries. The IEA has conducted numerousstudies focusing on educational policies and practices. The following IEA studies were used in preparingthe 1992-1997 editions of Education at a Glance: The Second International MathematicsStudy (SIMS), the Reading Literacy Study, and the Third International Math and Science Study(TIMSS).SFSO OECD ESSI 9

Definition and Selection of Competenciespolicy-makers had previously relied on information about school enrollmentand attainment to monitor the performance of the educationsystem, there was now a call for information about system outputs.This situation presented several challenges. One was the need to developa regular source of data to provide indicators for the annual publicationof Education at a Glance. Further, whereas the broad categoriesof school curriculum are relatively well institutionalized and also relativelycommon across industrialized – and even non-industrialized –countries, the same cannot be said for competencies or outcomes outsideof curriculum categories. And although studies in a number ofOECD countries have conceptualized a set of skills needed for work,there is no general understanding about competencies needed for lifethat are recognized within most countries, let alone internationally.Thus, although there is a broad consensus that there are importantlearning outcomes that do not correspond directly to curriculum areas,that consensus does not extend to just what these outcomes are.The response to these challenges within the OECD context has beena decentralized one. Since the inception of INES, a number of differentprojects have worked towards conceptualizing and developing newmeasures of learning outcomes or competencies for both school-agedchildren and adults. These include the Cross-Curricular CompetenciesProject (CCC) within Network A, the International Adult Literacy Survey(IALS), the Human Capital Indicators (HCI) Project, and more recentprojects such as the International Life Skills Survey (ILSS) and the Programfor International Student Assessment (PISA). Although many individualshave participated in more than one of these projects, and someconceptualizations of particular competencies are included in morethan one project, there has been minimal conceptual or operationalcoordination across the range of projects.Purpose of this ReportThis report is an initial activity of a three-year OECD program entitledDefinition and Selection of Competencies: Theoretical and ConceptualFoundations (DeSeCo), which has been launched by the SwissFederal Statistical Office, which chairs the program with support fromthe U.S. National Center for Education Statistics (NCES). DeSeCo seeksto develop a theoretically grounded conceptual framework for under-10 ESSI OECD SFSO

Definition and Selection of Competenciesstanding the skills and competencies needed to lead a personally andsocially worthwhile life in a modern democratic state. The overarchinggoal is to provide general reference points for further theoretical developmentand future work in the measurement of competencies. Keyto the project is the idea that the work should be accomplished in aninternational and interdisciplinary environment, with the purpose ofcreating a common understanding of issues.The purpose of this report is to explore and clarify what DeSeCo canlearn from previous projects related to competencies that have evolvedwithin the INES framework: CCC, IALS, and HCI. 2 Toward that end, thereport provides a broad overview of the projects and attempts to identifythe theoretical and conceptual considerations that influenced theirdevelopment. The analysis includes both the explicit scientific theoriesand the cultural and normative assumptions that are embodied in theprojects. Further, the studies’ original intentions and how they evolvedthroughout development and implementation phases are discussed.The report concludes with a brief description and discussion of ongoingprojects, but does not present an evaluation of any of the projects.The research questions guiding the study cover the progress of theprojects from the initial selection of competencies through to theprojects’ completion. Major topics include:• Selection of Competencies: rationale for selection of competencies;advocacy of particular competencies by countries or politicalgroups;• Definition and Conceptualization of Competencies: Scientificand theoretical contexts and implicit assumptions influencing howthe competencies were conceptualized; and2Within the INES project, there were two other approaches to developing indicators going beyond oracross curriculum subject lines: the Goals Orientation and Attainment in Learning Systems (GOALS)project and a public opinion survey. The GOALS project aimed to identify a range of goals for educationsystems so that this information can be taken into account when developing measures of outcomesof education (Granheim and Pettersson 1995). The public opinion survey asked members ofthe public for their views about the importance of both school subjects and cross-curricular goalssuch as developing self-confidence and a desire to continue learning (OECD 1995b). Because theseprojects were not primarily concerned with the outputs of education, they are not analyzed in depthin this report.SFSO OECD ESSI 11

Definition and Selection of Competencies• Empirical Considerations: Measurement and operationalizationof concepts; frameworks for empirical results.MethodologyTwo types of sources provided information for this report. The firstwere the documents produced by the projects (see References for a listof these documents). The second were semi-structured interviews withkey actors. The following individuals were interviewed during thespring of 1998: Tom Alexander, OECD; Marilyn Binkley, National Centerfor Education Statistics, U.S. Department of Education (NCES); NorbertoBottani, Service de la Recherche en Education, Geneva, formerly OECD;Helmut Fend, University of Zurich; Jeanne Griffith, National ScienceFoundation (U.S.), formerly NCES; Tom Healy, OECD; Douglas Hodgkinson,British Columbia Ministry of Education; Walo Hutmacher, educationconsultant, formerly Service de la Recherche en Education, Geneva;Scott Murray, Statistics Canada; Eugene Owen, NCES; Jules Peschar, Universityof Groningen; Judith Torney-Purta, University of Maryland at CollegePark; Uri Trier, University of Bern; and Albert Tuijnman, Universityof Stockholm, formerly OECD (see Annex B for the interview protocol).For each project, we begin with a brief description and continuewith a discussion of the selection and definition process of the competenciesand empirical considerations. Evolution of the projects as theyprogressed from initial ideas to a working project and project accomplishmentsare discussed when appropriate within the major topicareas. We begin with two projects that began in the early 1990s, CCCand IALS. Then we discuss the Human Capital Indicators Project, whichwas initiated in 1996 and recently released a report. We conclude withbrief descriptions of five ongoing projects: the International Life SkillsSurvey (ILSS), the Program for International Student Assessment (PISA),the current work in the CCC project, the IEA Civic Education Study, andthe Third International Mathematics and Science Study (TIMSS andTIMSS-R).12 ESSI OECD SFSO

Definition and Selection of CompetenciesCross-Curricular Competencies Project(CCC)DescriptionThe term Cross-Curricular Competencies (CCC) used in the OECDcontext refers to a domain of competencies that includes knowledgeand skills related to outcomes of education in a broad sense, as respondingto needs from both the social and economic spheres of life.The underlying crucial question is “what do young adults who completedtheir formal education need in terms of skills so as to be able toplay a constructive role as a citizen in society?” (Trier 1991)Work in this general area was initiated in the early 1990’s in the INESproject as a reaction to the ongoing development of indicators of studentachievement based on available data in the domain of school andsubject-related learning outcomes. The proposed classification distinguishedtwo categories of achievement indicators: on the one hand,the so called curricula bounded knowledge and skills based on thequestion of what (and how much of it) do children learn in schools,and, on the other hand, what was initially called non-curriculabounded socio-cultural knowledge and skills (NOBS 3 ) based on thequestion of whether the knowledge and skills needed to live an individuallyworthy and socially valuable life in our societies are providedthrough education (and if so how much of them) (Trier 1992). At thetime, it was assumed that the indicators that were developed wouldmeasure the knowledge and skills of adults having finished formal education,e.g., at the age of 18.Within Network A, the group in INES responsible for the developmentof outcome indicators, there was broad support for pursuing thistrack and including the CCC issue in the planned activities. The mainobjective was to produce a set of empirically tested instruments to measureCCC. To operationalize the idea of cross-curricular competenciesthrough a set of instruments measuring a minimal level of competence,the concept of a survival kit was proposed. The suggested areas to becovered by the survival kit were orientation in the political, social, and3What initially was called NOBS has become CCC, the acronym for Cross-Curricular Competencies.SFSO OECD ESSI 13

Definition and Selection of Competencieseconomic world, problem solving capacity in everyday and critical keysituations, communication skills, degree of autonomy (measuredthrough self perception), and finally, perception of critical human values(e.g., prejudice versus tolerance, solidarity, etc.).In 1993, a study was launched to investigate whether it was feasibleto develop internationally comparable indicators of cross-curricularcompetencies using existing instruments and scales of satisfactory psychometricquality. As this was an untried idea at the time, it was not atall clear what the results of the study would be. The domains that wereeventually retained – Politics, Economics and Civics, Problem Solving,Self-Perception/Self-Concept, and Communication – were rationalizedby the availability of instruments. For political and technical/practicalreasons the referential age for the target population was lowered to 16years. The instruments were consequently adjusted to the target populationavailable in the school system. Thus, whereas the original conceptof the survival kit concerned young adults outside of school and aminimal level of competence, the CCC Feasibility Study focused on aschool-based population and a wider range of difficulty levels.The CCC Feasibility Study was undertaken between 1993 and 1996.Nine countries participated in the fieldwork. In a majority of the countries,the overall results were judged satisfactory for two out of fourdomains. Civics and Self-Concept met scientific standards, whereasProblem Solving and Communication needed further developmentalwork to meet the statistical standards. The results of the pilot studywere published in the 1997 OECD publication Prepared for Life (OECD1997a).Considerable developmental efforts are still necessary for theplanned integration of information on CCC into the ongoing PISA program,a large-scale study on educational achievement (see section onOngoing <strong>Projects</strong>).Discussion and AnalysisSelectionSince Network A is concerned with education outcome indicators,the general topic of competencies has been of interest to the Network14 ESSI OECD SFSO

Definition and Selection of Competenciesfrom the beginning and a feature of its activities. In the first phase ofthe work, the only large-scale achievement data available for calculatingstudent achievement indicators were from the IAEP (conducted byETS) and IEA studies, some as much as 10 years old. The availability ofdata became the dominant selection criterion for outcome indicators.Thus, in the early work of the network, the selection of an indicator setin the domain of competence was not based on a conceptual or theoreticalframework.Under the lead of United States, there was a quick development onthe operational level for these first indicators. A main objective was thedevelopment of standards for generating indicators of educationalachievement and the application of these standards to existing data inorder to test their reliability and validity. The first indicators identifiedwere related to major academic subjects taught in secondary school. Aset of three provisional and experimental indicators related to competencein mathematics 4 were published in the first edition of Educationat a Glance in 1992.This primarily subject-focused track within Network A, was criticized,and contested by several OECD members, especially representativesof European countries. It gave rise to a methodological, theoretical,and conceptual debate about outcome indicators and resulted inthe creation of two sub-projects within Network A – the Project GoalAttainment and Orientation in Education System (GOALS) and the CCCproject. Subsequently, they were integrated as major components inthe strategic framework of Network A for student achievement outcomeindicators.In these sub-projects conceptual concerns were expressed, such asthe difficulty of defining terms and the questions of what schools teachor are intended to teach. There was no consensus within Network A onthe type of indicators to eventually include in the scope of the selectedindicators. Both of the existing assessments, the international studies4Mathematics Achievement of 13-Year-Old Students (Data source: Second International MathematicsStudy carried out by IEA between 1980 and 1982), School Differences in Mathematics and StudentDifferences in Mathematics (Data source: Data from the Second International Assessment of EducationalProgress carried out by ETS in 1991).SFSO OECD ESSI 15

Definition and Selection of Competenciesby the IEA and the International Assessment of Educational Progress(IAEP), were considered to be too selective in their content and toorestricted to traditional subjects. This led to the issue of how desirableand feasible it was to attempt to measure a broad range of areas notspecifically related to particular subjects. There were also political concernsand interests involved, e.g., the benchmarking approach was contestedby several countries, mainly for political reasons. Lastly, therewas no consensus about whether the assessment was to be organizedwithin the school framework, placing priority on results of schooling,or within a general context focusing on educational outcomes as awhole.The GOALS project focused mainly on the question of how to takeinto account national specificities related to desired outcomes of learning.Since in some countries, student achievement indicators were notat the top level of educational goals, indicators only based on subjectmatter or on cognitive goals were considered misleading. Personalitydevelopment, cooperation, orientation skills, and the effects of emotionaleducation were judged just as crucial as cognitive goals. TheGOALS group, composed primarily of Scandinavian countries, concentratedits work in particular on curricular objectives of education inschools, as expressed by official documents. Through this they also triedto capture general aspects of competencies, comparability was not amain goal.The CCC project was another approach intended to define competenciesin a more general, and not only curricular-bounded way. Thisproject was aimed at a broader issue, trying to identify the needs comingfrom society and the economy to education. Although the conceptof the survival kit found easy acceptance within Network A, agreementwas limited to labeling. Due to methodological differences and a lackof resources and support, the decision was taken to begin work by testingthe ideas empirically, applying only existing instruments, withoutfirst developing a general framework. The Feasibility Study was intendedprimarily to give evidence that the CCC approach could be realizedin practice. The different conceptions regarding the definition ofthe concepts and the underlying theories and different normative assumptionshave never been made explicit and clarified. Priority wasgiven to empirical testing of the concepts.16 ESSI OECD SFSO

Definition and Selection of CompetenciesDefinition and ConceptualizationWhen Network A started its work, the notion of “competencies”was not used. The discussion about measuring outcome indicators in abroader sense was dominated by different concepts such as non-cognitivegoals, goal-directed outcome indicators, non-curriculum boundedknowledge and skills, and cross-curricular competencies.In the early work of the Network, the concept of non-curriculabounded skills and knowledge (NOBS) referred to competencies notincluded in the school curriculum. However, eventually there was a shiftaway from this concept and towards the concept of cross-curricularcompetencies (CCC), referring to competencies that are included inmultiple subject areas across the curriculum. This change in the focus ofthe network from one conceptualization (NOBS) to another (CCC) wasaccomplished without significant definition and clarification, reflectinga lack of theoretical reflection by the Network. In fact, recent trends incognitive psychology suggest a scepticism about whether it is valid tothink of competencies as spanning different content, proposing insteadthat cognitive competencies are always related to content.No overarching theory in the CCC domain was developed. There aresome traditions in the domain of outcomes of education that implicitlyassume that competencies learned through school include those representedby the CCC concepts, such as longitudinal models of status attainmentin sociology of education and the human capital approach inthe economics of education. But it was generally agreed that althoughthere was no general framework that included well-defined conceptsfrom which indicators could be deduced logically, the work could proceedin a more inductive way. Developing a broader concept of competencieswas not seen as crucial.There are four general ideas that are relevant for the better understandingof the concept of cross-curricular competencies: the context ofCCC, the notion of a survival kit, the conception of CCC as a preparationfor future life, and, related, the learning approach inherent in thatconceptualization. Each of these ideas influenced the development ofthe CCC project.SFSO OECD ESSI 17

Definition and Selection of CompetenciesCCC ContextOne crucial question for selecting and defining competencies iswhether the relevant domain is restricted to formal education or encompassesboth formal education and everyday life. It is widely recognizedthat the skills and competencies of young people are a result ofschooling and a result of everyday life – a result of formal educationand a result of informal education. Thus, not every outcome has beentaught by the school; it might be a product of everyday life, even if it isa product of everyday life within the school. Some of the outcomes relatedto CCC are neither a part of the official curriculum nor formallytaught, but relevant aspects for the individual learning process and thesocial capital of a society. Within the discussion of competencies, thisbroader understanding of education has an influence on selecting anddefining competencies.The inclusion of the whole learning context (not only the school butalso everyday life situations) is a normative decision which influencesthe conceptualization of competencies: it leads to curricular boundedcompetencies as well as competencies not bound by the school curriculum.The main interest in the CCC project is, therefore, not school outcomes,but education outcomes.Survival KitThe underlying hypothesis in the CCC project was that it was possibleto isolate a set of competencies common to all educational systems,and accepted as a common goal of compulsory school, becausethey were considered necessary for surviving. In the original conception,the indicator to be developed was supposed to measure not anoptimal, but a minimal level of competence judged necessary for individualsto live an individually worthy and socially valuable life. The conceptbecame known as the survival kit, including basic skills and fundamentalknowledge: life skills. The term itself turned out to be ambiguous.For some members it was thought to pertain to the survival ofindividuals while others referred to a kit necessary for the survival ofdemocratic societies, for the continuation of community in the sense ofDurkheim. Depending on the interpretation, the conception of the survivalkit varied.18 ESSI OECD SFSO

Definition and Selection of CompetenciesThe definition of the concept “survival kit” also depends on the borderbetween school and other life spheres, on the border between cognitiveand non-cognitive knowledge and skills, and finally, it depends ifit is the outcome of formal school education or informal education.Representative concepts are, for instance, numeracy and literacy on theone hand, and democratic values, tolerance, and the capacity to cooperateon the other hand. These issues were not resolved within the CCCproject group.CCC as a Preparation for LifeThe notion of preparing for life has become somewhat of a leadingthread in the definition process of cross-curricular competence. To prepareyounger generations for future demands is seen as the centralchallenge for society. It reflects the idea that schools or education systemsare not ends in themselves, but means for making students competentfor life, not just for school. In the CCC approach more generalaspects of capabilities are emphasized, such as coping with life situations.Preparing for life is just as much acquiring the capabilities forreading novels and listening to music as it is acquiring the abilities todeal with the economic or professional world, to solve social problems,to build up social relations, etc. There is also a reference to the socializationprocess linking the micro to the macro level. The psychological dispositionis seen as a product of history shaped partly through the family,partly through school, and partly through the conflict or cooperationbetween schools and other institutions.Originally, the objective was to come to a definition in an inductiveway by starting from factual, real life situations of young adults and byanalyzing what they need to be able to cope with life. This inductiveway was judged later as very interesting, but not successful at leadingto a broad conceptual framework for life skills.Learning ApproachMost of the already mentioned competencies are related to learning.It is well known that everything that the younger generation hasto be prepared for in the future cannot be anticipated. Every generationhas to handle unforeseen situations and life circumstances.SFSO OECD ESSI 19

Definition and Selection of CompetenciesThe ability to learn is seen as the best preparation for tackling futuresituations. This conception of competence assumes high quality learning.High quality learning is a concept associated with psychologicalnotions such as learning strategies and learning habits. The concept ofhigh quality learning comprises competencies (knowledge and skills)and motivation (beliefs, attitudes, values, habits, emotions, and allthose psychological constructs which regulate learning). This is a keyelement for understanding the CCC approach. The Self-Concept domain,which was included in the CCC Feasibility Study, is only a smallcomponent of an elaborate concept of learning competence. The domainof learning competence is thought to be extended through indicatorsof “mental health”, based on psychological and health research,and through indicators of “youth behavior”, based on the sociologicalmonitoring of youth.Empirical ConsiderationsFor certain areas of CCC, the theoretical link between concepts andscientific psychometrics can be done by referring to cognitive psychology.For example, motivation and coping strategies are rather clearlydefined scientific concepts. There is also a longstanding research traditionon self-concept, self-esteem, and self-efficacy as well as on problemsolving.Many other concepts of competencies in the CCC domain are, however,related to belief in ability to cope or motivation to achieve, conceptswhich have never been assessed in an international context.There is, therefore, very little experience on which to build.Since the CCC approach has been mainly empirically inductive, there isno general theoretical framework that could provide definitions to serveas a conceptual or theoretical reference for the development of measurementinstruments and the interpretation of results. This is, however,even more indispensable for complex indicators since they make senseonly when they are related to a theoretical framework. The developmentof an overriding theory and the construction of adequate instrumentsare key to progress in the competence related domain. However,this is possible only with considerable investment of resources: corebrainwork,involvement of highly qualified and experienced researchersin specific scientific fields, and substantial financial resources.20 ESSI OECD SFSO

Definition and Selection of CompetenciesThe CCC project succeeded in the sense that it demonstrated withthe Feasibility Study that the CCC approach was thinkable and doable,and it convinced the OECD countries that the ideas behind the CCCproject were interesting and relevant for the assessment of learningoutcomes. In addition, two of the scales in the CCC Feasibility Studycontributed to the development of later surveys. Work is proceeding tointegrate measures building on the self-concept scale into PISA, andthe experience gained from the civics scale is being used in the IEA CivicEducation Study (see Ongoing <strong>Projects</strong>).SFSO OECD ESSI 21

Definition and Selection of Competencies22 ESSI OECD SFSO

Definition and Selection of CompetenciesInternational Adult Literacy Survey (IALS)DescriptionThe basic goal of the International Adult Literacy Survey (IALS) wasto assess the literacy performance among people from different countriesusing measures specifically developed for cross-national comparisons.The first phase of the survey was conducted in 1994 in eight countriesand resulted in the 1995 OECD/Statistics Canada publication Literacy,Economy and Society. The second phase of the survey was conductedin four additional countries in 1995 and was used, along withdata from the first phase, to produce the 1997 OECD/Human ResourcesCanada publication Literacy Skills for the Knowledge Society. A thirdphase is currently collecting data in 12 countries.To collect the data for the study, all participating countries administereda household survey to a probability sample of adults aged 16 to65. Some countries included older adults also. The survey consisted of abackground questionnaire and booklets with literacy assessment items.Background questions covered socioeconomic status, educational attainment,literacy practices at work and at home, labor force status,adult education participation, and literacy self-assessment. The literacyitems were based on materials from both North American and Europeansources and were designed to assess skills in three areas:Prose literacy – the knowledge and skills needed to understand anduse information from texts including editorials, news stories, poems,and fiction;Document literacy – the knowledge and skills required to locate anduse information contained in various formats, including job applications,payroll forms, transportation schedules, maps, tables, andgraphics; andQuantitative literacy – the knowledge and skills required to applyarithmetic operations, either alone or sequentially, to numbers embeddedin printed materials, such as balancing a checkbook, figuringout a tip, completing an order form, or determining the amount ofinterest on a loan from an advertisement (OECD and StatisticsCanada 1995: 14).SFSO OECD ESSI 23

Definition and Selection of CompetenciesLiteracy scores for countries and subgroups were estimated usingitem response theory (see Empirical Considerations below). In additionto scores ranging from 0 to 500, the survey methodology allowed performanceto be characterized according to five levels, which rangefrom very basic skills to those that require high levels of inference andcomplex reasoning.Literacy Skills for a Knowledge Society is based primarily on the IALSdata collected in 12 countries. The report includes information aboutdistributions of literacy levels for different areas and within thoseareas, for people with different background characteristics, e.g., age,education, and gender; the economic benefits to individuals of literacyskills relative to the benefits of educational attainment and labor forceexperience; the relationship between literacy and characteristics suchas age (net of education) and parent education; literacy practices ofadults; and participation in adult education and training.Discussion and AnalysisSelectionUnlike the CCC project which conducted an exploratory study innew domains for national-level assessment, the IALS grew out of majorassessments of literacy conducted in two countries: Canada’s 1989 Surveyof Literacy Skills Used in Daily Activities and the United States’ 1990National Adult Literacy Survey (NALS). The conceptualization andmethodology for measuring literacy had been developed in the UnitedStates and used for the NALS. In addition, the Canadian survey hadalready successfully demonstrated that a literacy survey could be successfullyimplemented in two languages. From that point, the extensionof these surveys into the international arena reflected aconfluence of interest among a number of key actors.Reasons for interest in conducting an international assessment ofadult literacy varied from personal to political to scientific, dependingon the participating nation or the individual involved. For some, theprimary motivation was to investigate whether the NALS strategy forassessing literacy was feasible across languages and cultures, with newitems developed to reflect the range of reading material in differentcountries. The inclusion of a background questionnaire was also a keyfactor and generated interest among researchers and policymakers.24 ESSI OECD SFSO

Definition and Selection of CompetenciesThe background questionnaire allowed IALS to provide, for the firsttime, cross-national data on adult education and training and the relationof literacy skills to labor market outcomes, such as employmentearnings for workers in different occupations and industries.IALS was also attractive because it provided an opportunity to movethe assessment agenda beyond schools and schooling and beyond standardassessments of curricular-based skills. Its content represents activitiesfrom everyday life and is not organized around school subjects.Furthermore, IALS included the entire population, not just those inschools. This was particularly important for those interested in the skillsof the young population; previous measures of this group were limitedto those in school.Political support was generated because of the existing scientificbase for the study and the potential policy-relevance of the findings inthe context of issues such as unemployment and national competitivenessin the global economy. The pre-existence of a measurement instrumentalong with an established theoretical and conceptual framework(discussed below) created a notion of ease-of-implementation for IALSthat encouraged participation among countries. Although it was notthe driving force behind the study, political motivation in some quarterscame from a more basic interest in utilizing IALS simply to compareliteracy skills across the countries, what has been called “the horserace.” And for some, interest in literacy per se was secondary to thedesire for any comparable data on skills of the adult population, particularlyyoung adults out of school.Funding for the study, which came primarily from Canada, but alsofrom each of the participating countries, involved recognition on thepart of funding agencies that in spite of much rhetoric about the relationshipbetween worker skills and economic outcomes, there wasscant empirical evidence to support the claims. IALS was an opportunityto obtain this data and in the process advance other interests as well.IALS was initiated by the OECD and Statistics Canada, independently ofNetwork A and the CCC project.Definition and ConceptualizationThe definition of literacy adopted by IALS is “the ability to understandand employ printed information in daily activities at home, atSFSO OECD ESSI 25

Definition and Selection of Competencieswork and in the community – to achieve one’s goals, and to developone’s knowledge and potential” (OECD and Human Resources Canada1997: 14). Its theoretical foundation comes from the work of IrwinKirsch and Peter Mosenthal, which began in the early 1980s with thedevelopment of the Young Adult Literacy Survey in the United States(Kirsch and Jungeblut 1986), and was refined for subsequent studiesincluding the NALS (U.S. Department of Education 1993). The idea thatliteracy involves understanding any printed or written information, notjust prose which is typically associated with school reading, is key to thedefinition of literacy embodied in their work. In the IALS conceptualization,literacy involves the ability to complete tasks based on documentsencountered in daily life, including those involving arithmeticoperations encountered in everyday life, when presented in writtenform, as well as comprehension of prose selections. According to theview of literacy used in NALS and IALS, literacy can be subdivided intothree domains: prose literacy, document literacy, and quantitative literacydepending on the type of printed information.Further, literacy is conceptualized as a continuum ranging from themost basic decoding skills to understanding complex ideas presented inwritten form. This idea differs fundamentally from the concept traditionallyassociated with literacy, in which it is seen as a dichotomouscapacity that individuals either have or do not have. Kirsch andMosenthal’s theory of adult reading links reading difficulty to attributesof the text and the tasks the reader must perform. Difficulty isa function of task characteristics, such as how explicit is the relationshipbetween what the respondent is asked to do and the presentation ofthe information in the text; whether information appears in the textthat is similar to, but is not, the correct answer; whether the informationis concrete or abstract; and the structural complexity of the item.For quantitative literacy, it is also a function of the difficulty of settingup the problem and the type of calculation involved. In addition, difficultydepends on characteristics of the materials presented, such aslength of words and sentences, clarity of document labels, complexityof documents, and the amount of information presented in the item.An extension of the notion of literacy used in IALS is the idea thatindividuals’ literacy skills are not synonymous with their education andtraining qualifications; individuals may have identical qualificationsand yet have different literacy skills.26 ESSI OECD SFSO

Definition and Selection of CompetenciesEmpirical ConsiderationsEmpirical considerations were central to the development of themeasures of literacy used for NALS and, subsequently, for IALS. IALSrepresents the fourth study using the literacy theory and methodologythat was first developed for the Young Adult Literacy Survey in theUnited States, administered in 1985. (The second was a study conductedfor the U.S. Department of Labor (U.S. Department of Labor1992), and the third was NALS.) Each stage provided an opportunity toexamine the theory in the context of empirical information about theliteracy performance of individuals. For example, the theory of howcharacteristics of written material relate to reading difficulty was developedempirically by relating the task difficulty to a set of characteristicsfor each task. Regression analyses of the contribution of different characteristicsto predicting performance on the item were used to explorewhich characteristics contribute most heavily to difficulty, and to describeliteracy activities in terms of particular task characteristics. Initialnotions of difficulty have been refined based on the empirical relationshipsbetween characteristics of reading material and performance ofreaders.At the beginning of the development of IALS, it was recognized thatthere could be differences in the extent to which the items have similarcharacteristics and measure the same ability across countries, languagegroups, and cultural communities. In fact, one of the motivations forconducting the study was to investigate whether the methodologywould be workable on an international scale. Analyses conducted usingthe data collected for the study found, with the exception of a fewitems which were dropped from the study, that the items showed similarpatterns in relating to literacy proficiency of adults. 55IALS used the methodology of item response theory (IRT) to develop its three literacy scales. Thesemethods allow test developers to estimate the difficulty of individual items and then to estimatelatent traits or abilities based on performance on the items or an item subset. There were only 12(out of 114) items for which data for more than three countries showed a poor fit to item parameterscommon to other countries. These items were dropped from the analyses. In cases where datafrom one to three countries did not fit the common parameters, unique item parameters were estimatedfor the differing countries. In some cases, there were obvious explanations for the poor fit; inothers no reasons could be identified (U.S. Department of Education 1998).SFSO OECD ESSI 27

Definition and Selection of CompetenciesIALS was specifically designed to allow empirical tests of theoriesabout the relationship between literacy skills and educational attainmentand qualifications, and the relationship between literacy skillsand earnings – independent of educational attainment. Previously,educational attainment was the only measure available and had servedas a proxy for actual skills in research relating worker characteristics tolabor market outcomes. These analyses were key to the interest ofpolicymakers in the study.In addition, IALS provided an opportunity to further refine the theoriesdeveloped for NALS. The data were used to investigate the extentto which the constructs of prose, document, and quantitative literacyrepresent distinct dimensions in IALS countries, as proposed by thetheory guiding the work. The analyses found high correlations amongthe three scales, but concluded that there was “sufficient separationamong the three literacy scales to justify reporting these scales separately”(U.S. Department of Education 1998: 143).When the results of the first round of IALS were reported to participatingcountries, the extent to which the survey actually attained internationalcomparability was questioned by the French authorities. Theirconcerns included the appropriateness of the assessment instruments,the validity of the sampling procedures, and the reliability of the populationestimates (U.S. Department of Education 1998) 6 . In response, StatisticsCanada commissioned a review of the survey procedures by threestatisticians. Their review, which has been published in the IALS technicalreport, pointed out a number of problems mainly related to the surveydesign and procedures (U.S. Department of Education 1998).The review concluded that there is a serious need for more standardizedprocedures in future surveys, but that despite the methodologicalweaknesses, the survey results could be published. The authorsstated that all results should be interpreted with caution, and recommendedthat the reported results focus on analyses of the correlates ofliteracy and how these compare across countries rather than on overallliteracy levels. (Their position is that the former type of analysis is less6France had been an early participant in IALS and its representatives participated throughout thedevelopment of the survey.28 ESSI OECD SFSO

Definition and Selection of Competencieslikely to be affected by survey weaknesses than the latter.) At the sametime that it suggests that results of the first round of IALS should beaccompanied with cautions about interpretation, the report views theinstruments developed for IALS as an “important advance” and the resultsof the first round assessments as “a valuable contribution to thisfield” (U.S. Department of Education 1998: A4).France, however, confronted with very weak results in comparisonwith other countries, continued to question the comparability of theresults and decided to withdraw its results from the IALS publications.Consequently, two expert reports were initiated by the French Ministryof Education to examine methodological questions related to internationalsurveys in the domain of literacy 7 .7Blum, Alain and Guérin-Pace, France. 1997. De la difficulté d’entreprendre une enquêteinternationale sur l’illétrisme, Institut National d’études démographiques (INED).Dickes, Paul and Flieller, André. 1997. Analyses secondaires des données françaises de la premièreEnquête Internationale sur l’Alphabétisation des Adultes (enquête IALS), Université Nancy 2,Laboratoire de Psychologie.SFSO OECD ESSI 29

Definition and Selection of Competencies30 ESSI OECD SFSO

Definition and Selection of CompetenciesHuman Capital Indicators ProjectDescriptionIn 1996, the OECD Council of Ministers requested that OECD preparea report on what is known about human capital. The report was todevelop indicators based on existing data, identify gaps in internationallycomparable data, and discuss the costs of the development of datacollections for new measures of human capital. The request resulted inthe 1998 OECD publication Human Capital Investment: An InternationalComparison.Unlike the other projects discussed in this paper, the Human CapitalIndicators Project was for the most part an internal OECD effort, relyingon OECD staff and consultants. Its work built on the activities conductedby the INES project working group on measures of educationand labor market destinations (Network B), but the project did not involveparticipation by country representatives or national study managers.Human Capital Investment: An International Comparison recognizesthat the concept of human capital has been used in a number of differentways since its modern usage was initiated by economists in the sixties(Schultz 1961 and Becker 1964), and adopts a definition of humancapital as “the knowledge, skills, competences, and other attributesembodied in individuals that are relevant to economic activity” (OECD1998: 3). The report recognizes that social benefits of individual attributesextend beyond economic activity, but restricts human capitalto “assets with the capacity to enhance or support productivity, innovation,and employability (OECD 1998: 3).The first topic addressed by the report concerns the stock of humancapital, “the level of skills, knowledge, and competencies held at anyone time by individuals” (OECD 1998: 15). Data from IALS provide theonly direct measure of adult skills available for different ages. Qualificationsand years of schooling are also available for people of differentages, and analyzing educational attainment of different age groupsgives indications of changes in patterns across time. The value of humancapital is indirectly measured by using wage data and earning differentialsbetween those with different education and/or skills.SFSO OECD ESSI 31

Definition and Selection of CompetenciesIssues of investment and return on investment have become keypolicy topics within the OECD context. Human Capital Investment containschapters on these topics, with measures of investment based onexpenditures and participation in education and on training. Economicreturns to human capital investments are expressed in terms of employmentand earnings.Human Capital Investment concludes with chapters discussing theadequacy of the knowledge base and policy issues in human capitalinvestment. It recognizes that most available measures have focused onwhat is possible rather than what is desirable to measure and recommendsthat priority be placed on “more direct measures of life-relevantskills, of the value placed on them in the workplace and of the benefitsto individuals and enterprises of work-related training” (OECD 1998:89). Improved measures of human capital will allow for more rigorousanalysis of issues such as levels of investment, sharing of costs betweengovernments and the private sector, allocating expenses to areas wherethey will be most beneficial, and distributing investment equitablyamong the adult workforce.Discussion and AnalysisSelectionRecent OECD work in the area of human capital had its beginningsin the mid-1990’s when labor ministries from the United States andCanada expressed interest in revising official accounting methodologiesto recognize expenditures for training and labor force developmentas investments. According to their view, these expenditures representinvestment in human capital; not including them as investmentsin business and national accounting could result in decisions about theallocation of public and private resources that don’t maximize economicdevelopment. To accomplish a re-design of this magnitudewould have required significant political and financial support, and theeffort was never undertaken. However, the 1996 Council of Ministersrecognized this general concern when it requested that OECD preparea report presenting indicators of human capital for the ministerialmeeting in 1998.32 ESSI OECD SFSO

Definition and Selection of CompetenciesThe Council of Ministers’ request for a report on indicators of humancapital naturally reflects interests broader than the initial inquiries fromthe United States and Canada about accounting systems. In part, it canbe seen as developing from the OECD Jobs Study (OECD 1994). Thestudy concluded that adaptability – including individual adaptability –is a key dimension for economic stability and growth, an idea that hasclear ramifications for education throughout the lifespan. Further, theconcept of employability, as distinct from educational attainment andtraining, has gained increased acceptance by policymakers. In sum, therequest for the human capital report reflected a growing recognition inthe spheres of labor and finance ministries that issues related to thedevelopment of individual capabilities – human capital – are relevantfor economic policy.Indeed, one of the primary accomplishments of Human Capital Investmentwas providing the education arena with quantitative andqualitative analysis that relates human capital to issues of interest inmacroeconomics and labor economics. This in turns gives added weightto education policy vis-à-vis other policy areas, such as finance and labor.In fact, it appears that support for continuing the work begun inthe Human Capital Indicators Project was stronger in labor than in educationwithin OECD.Definition and ConceptualizationThe Human Capital Indicators Project provided an opportunity toclarify the conceptualization of “human capital” for the OECD context,including the meaning of the term itself and its position in a widertheoretical framework. A large number of theoretical and conceptualissues needed to be considered before arriving at the concise definitionpresented in Human Capital Investment and quoted above.While the final definition is quite brief, many interrelationshipsamong different social and individual characteristics are recognized inthe report. Although there is only brief mention of them, they include:• The relationship between education and training on the one hand,and “knowledge, skills, competencies, and other attributes” on theother;SFSO OECD ESSI 33

Definition and Selection of Competencies• The role of a range of life experiences in developing human capital;• The wide range of individual characteristics that contribute to humancapital;• The related concept of social capital; and• The extent to which non-economic sectors (e.g., social cohesion,health) contribute to the economic sector both at the individual andsociety-wide levels.The conceptualization proposed in Human Capital Investment challengesanalysts and policymakers to address issues that have traditionallyfallen outside of the domain of human capital. Whereas previousanalyses have focused on topics associated with classical economictheory, such as investment in and rates of return to formal educationand training, the report highlights other aspects of human capital. Thefirst is the notion that attributes of individuals extend beyond academicknowledge, encompassing both cross-curricular skills and attitudes. Secondis the idea that human capital is acquired by individuals not onlybefore they enter the workforce but throughout their lives. Third, humancapital is not acquired exclusively through the formal initial educationand training systems, but in a range of environments includingwork, work-based training, working life, and informal experiences.Lastly, human capital is relevant for individuals and society in non-economicarenas as well as in economic ones.It is important to note that representatives of different intellectualand policy areas participated in the discussions leading up to the report,and covered each of the related issues. By the time the final definitionwas agreed upon, they had gained a clearer understanding ofeach other’s perspectives and of the trade-offs inherent in the finaldefinition.At the same time that the report confirms a broad view of humancapital, many related questions are beyond its scope. Of key interest tothe DeSeCo project is the issue of what are the knowledge, skills, competencies,and attributes that constitute human capital. While affirmingthat human capital is important for individual and aggregate economicproductivity and further, that traditional curricular subject areasare inadequate to delineate characteristics that constitute human capi-34 ESSI OECD SFSO

Definition and Selection of Competenciestal, the project leaves the topic of specifying the knowledge, skills, competencies,and attributes of human capital to future investigators. Thereport also recognizes that further conceptual and theoretical developmentis necessary to take into account the interrelationships amonghuman capital, the economy, and non-economic sectors such as health,and the impact of human capital investments on non-economic sectors.Empirical ConsiderationsHuman Capital Investment takes a strong stand that most existingmeasures of human capital do not capture its full meaning. For example,traditional indicators based on the formal education system (enrollment,attainment, and costs) do not directly measure the capabilitiesof individuals. Nor do they reflect experiences in the range of environmentsoutside of the formal education systems that contribute toaccumulation of human capital in individuals. Indicators based on investmentsand wage differentials are even more indirect, and are basedon larger assumptions.The report recommends that priority be given to developing directmeasures of a range of individual attributes, with a viewpoint thatthese attributes are developed through institutions and experiencesbeyond primary, secondary, and tertiary education. In light of the factthat previous empirical investigations of human capital theory have reliedon more indirect measures with little recognition of their limitations,this understanding represents a new outlook in the empiricalanalyses of human capital.In spite of the recognized limitations of existing measures, the reportdevotes the majority of its attention to presenting indicators based onthem. Work is continuing in the Centre for Educational Research andInnovation at OECD on analysis of the relationship between humancapital and economic growth and social cohesion. Within this continuingwork, there is a focus on better conceptualisation of human capitalin relation to other forms of capital such as social capital as well as onevidence for broad social and macro-economic returns to investment inskills and knowledge.SFSO OECD ESSI 35

Definition and Selection of Competencies36 ESSI OECD SFSO

Definition and Selection of CompetenciesOngoing <strong>Projects</strong>International Life Skills Survey (ILSS)Building on the International Adult Literacy Survey (IALS), there ispresently an attempt underway to develop and conduct a new surveyto comparatively assess a range of life skills across the adult populationsof several OECD nations. The goal of this survey, the International LifeSkills Survey (ILSS), is to build on the IALS with a broader assessment,resulting in a more complete picture of individual life skills in the internationalcontext. ILSS will assess several domains, including prose literacyand document literacy (utilizing the same framework as IALS),numeracy, problem solving, teamwork, practical cognition, and computerfamiliarity. These skill areas will be assessed through direct performanceor through behavioral reports. A questionnaire to obtainparticipant background information will be included as well. Currently,survey planners are working towards conducting item feasibility studiesfor the domains in 1999, a full pilot survey in 2001, and the mainsurvey in 2002.This survey is patterned on IALS in multiple respects, and generallyseeks to take the lessons learned and procedures developed by IALS andapply them to the larger framework of ILSS. The current group of skillsincluded in the ILSS plans was developed by considering skills commonlymentioned in policy reports on employability along with recommendationsfrom experts in the field of international competence assessment.http://www.nces.ed.gov/ilssProgram for International Student Assessment(PISA)Between 1995 and 1997, the INES Network A group developed aData Strategy aimed “at the assessment of broadly defined knowledge,skills, and competencies embedded in the context of important contentdomains rather than the assessment of very narrowly defined subjectmatter knowledge” (OECD 1997b: 44-45). The Program for InternationalStudent Assessment (PISA) is a manifestation of this strategy designedto obtain indicators of student knowledge, skill, and compe-SFSO OECD ESSI 37

Definition and Selection of Competenciestency on a regular basis in the domains of reading, mathematics, andscience. The data collected will be presented as outcome indicators providinginformation in four key areas: student achievement, relationsbetween achievement and contextual variables, school effectiveness,and trend data.PISA is scheduled to operate on a three-year cycle. Each iteration willinclude all three domains, but focus on one. The first iteration of theassessment is planned for 2000, focusing on reading. Additionally, dependingon the progress of current developmental work, CCC domainssuch as Self-Concept and Problem Solving may be incorporated into theassessment (Self-Concept in 2000 and Problem Solving in 2003 (see below)).The domains of reading, mathematics, and science were selected becauseof the interest of policymakers, pre-existing measurement technology,established conceptual and empirical frameworks, and a historyof successful international assessments in the skill areas. In particular,PISA is building on the base developed for two recent IEA studies:the Reading Literacy Study and the Third International Mathematicsand Science Study (TIMSS). 8PISA is the result of a concerted effort on the part of survey plannersto move from an ad hoc basis to a cycle of regular data collection.The development of PISA has involved multiple compromises, notablyon the age of the sample (15-year-olds), and reflects a commitment byall participants to share the costs of the international assessment.Although the Data Strategy that resulted in PISA was originally developedwithin Network A, the effort has become independent of theNetwork. The Board of Participating Countries (BPC), with a representativefrom each country in the survey, is responsible for overall policydirection. PISA is managed by OECD through a consortium consisting ofThe Australian Council for Educational Research (ACER), The Nether-8Prior to TIMSS, IEA also conducted two mathematics and two science international assessments (seeFootnote 1).38 ESSI OECD SFSO

Definition and Selection of Competencieslands National Institute for Educational Measurement (CITO), WestatInc., and the Service de Pédagogie Expérimentale, Université de Liège(SPE).http://www.pisa.oecd.orgCCC Problem Solving ProjectPresently the INES Network A, as part of its overarching Data Strategy,and as a continuation of its CCC work, is planning an assessment inthe domain of problem solving. A six-year plan for devising and administeringan assessment in the competency by 2003, as a part of PISA, waspresented to the Network in the spring of 1998. The first step is thedefinition and exploration of the problem solving domain itself. Afterthis is complete, the next steps will involve establishing criteria for developingand/or selecting reliable and valid instruments for measuringproblem solving. Once an instrument has been developed it will undergofield testing before final survey implementation. The selection ofproblem solving is directly related to the earlier CCC activities of NetworkA, and this plan can be seen as an extension of that research.IEA Civic Education StudyThe IEA (see Footnote 1) is currently conducting an internationalCivic Education Study. The survey will assess the competency of 14-yearoldswith respect to civic and political knowledge based on items derivedfrom expert panels in participating nations. The study is based ona conceptual framework of civic knowledge developed specifically forthe assessment. Although unrelated to OECD and INES, this project ismentioned because it is building on the experience of the CCC FeasibilityStudy.Third International Mathematics and Science Study(TIMSS and TIMSS-R)Another IEA effort that is significant for work on competencies inthe OECD context is the Third International Mathematics and ScienceStudy (TIMSS). Building on earlier IEA studies of mathematics and sci-SFSO OECD ESSI 39

Definition and Selection of Competenciesence, TIMSS assessed mathematics and science knowledge and skills ofover a half-million students in 41 countries. Students at three differentlevels of schools were tested: midway through elementary school, midwaythrough lower secondary school, and at the end of upper secondaryschool. (Assessing the second population was required for participatingcountries; the others were optional.) In addition to the assessments,the study included school, teacher, and student questionnairesand a curriculum analysis. All aspects of the study design and instrumentationwere developed through international collaboration.Currently, 40 countries are participating in a replication of TIMSS(TIMSS-R) at the lower secondary level. This will make it possible to analyzetime trends during the 1990s for lower secondary students and toconduct analyses of the performance of the cohort of students originallytested midway through elementary school five years later, whenthey are in lower secondary school.40 ESSI OECD SFSO

Definition and Selection of CompetenciesConclusionsGrowing demand from national authorities for output-oriented informationabout education has resulted in a number of projects in theOECD context. Although their goals are related, these projects haveoperated more or less independently, with minimal conceptual and organizationalcoordination related to selecting and defining competencies.Nevertheless, the three major efforts profiled in this report – theCCC project and its Feasibility Study, IALS, and the Human Capital IndicatorsProject – have a number of elements in common.Each of the studies adopted a broad conception of competenciesand outcomes of education. This includes the viewpoint that the desiredoutcomes of education are broader than the acquisition of subject-relatedknowledge typically taught in school, and that the idea ofcompetencies extends beyond the school context. The areas covered inthe CCC Feasibility Study stretch across curricular boundaries and areviewed as skills for life. The IALS conceptualization of literacy includesperforming tasks based on a wide range of printed materials, and viewsliteracy as spanning the range from basic to advanced levels. Theconceptualization of human capital presented in the HCI report recognizesthat it is comprised of components acquired both in and out ofschool that contribute to economic productivity of individuals, and isconceptually distinct from attainment in formal education.Feasibility of measurement was a major concern in the selection anddefinition of competencies in each of these projects. Although not alwaysthe goal at the onset, each of the projects eventually gave priorityto empirical testing of concepts by applying existing methodologies,instruments, and, when possible, existing data. Competencies for theCCC Feasibility Study were selected on the basis of whether existinginstruments of sufficient quality were available; the goal of the projectwas to show that these instruments could produce interesting and usefulmeasures. In the case of IALS, the availability of a measure was keyto the selection of literacy for the first major international survey of anadult competency and to the use of the conceptualization of literacyfrom the National Adult Literacy Survey in the United States. For theHCI project, the request by the OECD Ministerial Council was to developindicators based on existing data.SFSO OECD ESSI 41

Definition and Selection of CompetenciesAs a result, theoretical and conceptual issues were not at the forefrontof these projects. In CCC, the reliance on existing instrumentscame at the expense of conceptualization of competencies, their relationshipsto the school curriculum, and their interrelationships to eachother. There is almost no discussion in Prepared for Life of conceptualissues related to the term “cross-curricular competencies” or to the specificcompetencies being assessed. The key initial question of the CCCproject – which knowledge and skills and attitudes are important forsurviving and leading a productive, socially worthy life – was set asideas the goal of the project changed. The conceptualization and measurementused in IALS is built on a theoretical base. However, as muchof this work has taken place over many years, it was not the focus of theIALS project. Further, IALS was developed in a national context forother projects (most directly, the United States National Adult LiteracySurvey), and thus its conceptualization of literacy is not the product ofan internationally collaborative process. Although the HCI project developeda distinctive conceptualization of human capital, the focus onthe work was on using existing data for indicators, rather than on refiningthe conceptualization or exploring its theoretical ramifications.Another result of the focus on measurement of individual competenciesis that the studies are not concerned with how competenciesare interrelated. The concept of human capital presented in the reportis not concerned with its components, in particular what knowledge,skills, competencies, and other attributes result in human capital; andhow they are related to each other. Likewise, the CCC Feasibility Studydid not deal with how the four skill areas are interrelated or with howcross-curricular skills relate to curricular skills. IALS and previous studiesbased on the literacy framework are not concerned with how literacyskills relate to, or are interconnected with, other knowledge, skills, andattributes, or how literacy fits into a larger framework of competencies.Each of these commonalties among the studies can be seen as followingfrom the fact that the overriding priority for work in the OECDcontext is providing information to the policy community. Taken together,the studies succeeded not only in providing a great deal of suchinformation, but also in generating both interest in competencies beyondthe field of education and support for efforts that are currentlyunderway. However, because limited resources were available on the42 ESSI OECD SFSO

Definition and Selection of Competenciesinternational level for conceptual and theoretical work, the studieswere obliged to draw from the existing scientific base. Close cooperationwith the scientific community in future efforts to define, conceptualize,and measure competencies is crucial to the success of this work.SFSO OECD ESSI 43

Definition and Selection of Competencies44 ESSI OECD SFSO

Definition and Selection of CompetenciesReferencesGranheim, M. and Pettersson, S. 1995. ”Goals Orientation and Attainmentin Learning Systems”. Measuring What Students Learn. Paris,France: OECD.Hirsch, D. 1995. ”From Collaboration to Co-ordination: The Past andFuture of the INES Networks.” Paris, France: OECD. (Prepared forGeneral Assembly of INES Project, GA-95-1).Kirsch, I. S. and Jungeblut, A. 1986. Literacy: Profiles of America’s YoungAdults. NAEP Report No. 16-PL-01. Princeton, New Jersey: EducationalTesting Service.Lapointe, A. E., Askew, Janice M., and Mead, Nancy A. 1992a. LearningMathematics. 22- CAEP-01. Princeton, NJ: Education Testing Service.– .Learning Science. 1992b. 22-CAEP-02. Princeton, NJ: Education TestingService.Materials. 1997. Informal Meeting of Experts. International Life SkillsSurvey. Paris, France.Materials and Minutes. 1996-1997. Selected Network A Sub- or WorkingGroup Meetings (Including CCC Subgroup Meetings). Assortedlocations.– .1996-1998. Network A Plenary Meetings. Assorted locations.Organization for Economic Cooperation and Development. 1994. TheOECD Jobs Study: Facts, Analysis, Strategies. Paris, France: OECD.– .1995a. Measuring What Students Learn. Paris, France: OECD.– .1995b. Public Expectations of the Final Stage of Compulsory Education.Paris, France: OECD.– .1997a. Prepared for Life? How to Measure Cross-Curricular Competencies.Paris, France: OECD.– .1997b. Terms of Reference for the Tendering Procedure of the DataStrategy for the Development of Student Achievement Indicators ona Regular Basis. Paris, France: OECD.– .1998. Human Capital Investment. Paris, France: OECD.SFSO OECD ESSI 45

Definition and Selection of CompetenciesOrganization for Economic Cooperation and Development and HumanResources Development Canada. 1997. Literacy Skills for the KnowledgeSociety: Further Results from the International Adult LiteracySurvey. Paris, France: OECD.Organization for Economic Cooperation and Development and StatisticsCanada. 1995. Literacy, Economy and Society: Results of the firstInternational Adult Literacy Survey. Paris, France: OECD.Trier, U.P. 1991. ”Non Curriculum Bound Outcomes,” Proposal presentedat the Network A Meeting. Paris, France.– .1992. ”Non-Curricula Bounded Socio-Cultural Knowledge and Skills:Some Remarks and Proposals” Paper presented at the Network AMeeting. Oslo, Norway.U.S. Department of Education. 1993. National Center for Education Statistics.Adult Literacy in America: A First Look at the Results of theNational Adult Literacy Survey, by Irwin S. Kirsch, Ann Jungeblut,Lynn Jenkins, and Andrew Kolstad. Washington, D.C.– . 1998. National Center for Education Statistics. Adult Literacy inOECD Countries: Technical Report on the First International Adult LiteracySurvey, NCES 98-053, by T. Scott Murray, Irwin S. Kirsch, andLynn B. Jenkins. Project Officer, Marilyn Binkley. Washington, D.C.U.S. Department of Labor. 1992. Employment and Training Administration.Beyond the School Doors: The Literacy Needs of Job SeekersServed by the U.S. Department of Labor, by Irwin S. Kirsch, AnnJungeblut, and Anne Campbell. Washington, D.C.46 ESSI OECD SFSO

Definition and Selection of CompetenciesAnnex A: The INES NetworksThe Centre for Educational Research and Innovation of OECD initiatedthe International Education Indicators Project (INES) in responseto increased demand from member countries for improved comparativeinformation about the quality of education systems. The projectgrew out of two preparatory conferences, an initial meeting in theUnited States in 1987, and a planing conference in France in 1988.In 1988, five Networks were set up, one in each of the followingareas: student flows, student outcomes, ecology of schools, finance,and attitudes and expectations. Funding for these Networks was providedby Australia, Austria, France, the Netherlands, and the UnitedStates.The Networks were restructured in 1991. The Networks on studentflows and finance were merged into a single Technical Group workingdirectly with the OECD Secretariat. Four Networks were formed: studentachievement outcomes (Network A), labour market destinations(Network B), school processes (Network C), and attitudes and expectations(Network D). These Networks were supported by the Netherlands,Scotland, Sweden, and the United States. At that time, a decision wasmade to begin publication of Education at a Glance. The first volumewas published in 1992. General Assemblies of all participants in thework have been held in 1989, 1991, and 1995.The Networks meet about twice a year for the purpose of workingcollaboratively toward the common goal of conceptualizing and producinginnovative education indicators. In between meetings, thegroups comment on proposals that are circulated in preparation for themeetings; Network sub-groups are assigned to specific tasks. Administrativeand scientific guidance is provided to the Networks by the OECDSecretariat (Hirsch, 1995).SFSO OECD ESSI 47

Definition and Selection of Competencies48 ESSI OECD SFSO

Definition and Selection of CompetenciesAnnexe B: Interview ProtocolThe interviews conducted for the purposes of this paper took placebetween March and June 1998. The interviews were conducted in bothSwitzerland and the United States, depending on the participants’schedules. An in-person interview was conducted in all cases but one,where a phone interview was conducted instead. The interviews wererecorded, transcribed, and shared among the authors. The followingquestionnaire was distributed to participants in advance of the interviewand was generally used as a guide for the entire interview.Questionnaire for the InterviewIn general, please provide answers only for projects that you havebeen personally involved with.1. What has been your role in OECD projects related to conceptualizingor measuring competence? Have you worked in other projects relatedto these topics?2. In your view, what factors contributed to the selection of the particularareas of competence associated with the project(s)? This may includethe sources of interest (educators, political groups, experts, thescientific community, particular countries), feasibility of measurement,availability of funding, and related events and conditions.3. For the sources of interest that you have identified, what were thereasons for their interest? What did they want the study to provide?4. How does the area(s) of competence as conceptualized or defined bythe projects(s) fit into a theoretical and scientific context? Whattheories have guided the conceptualization? Can it be identifiedwith a particular scientific context and with particular theorists andproponents?SFSO OECD ESSI 49

Definition and Selection of Competencies5. How was the conceptualization of the area(s) of competenceoperationalized into a measurement strategy (instrument, procedures,etc.)? What theoretical considerations were involved?6. What conceptual or theoretical frameworks were used to analyzethe empirical results? What is their scientific context?7. How did the original ideas in the project(s) evolve as the project progressed?What factors are associated with these changes (e.g., practical,scientific, political)?8. What do you view as the accomplishments of the projects? What arethe benefits and challenges associated with what has been learned?9. What do you see as next steps in the area of defining, selecting, andmeasuring competencies within the OECD context?50 ESSI OECD SFSO

In response to a growing demand from national authorities for output-orientedinformation about schools, several initiatives in the OECD context have worked towardsconceptualizing and developing new measures of learning outcomes for studentsand the general population. This report focuses on three such projects, theCross-Curricular Competencies Project, the International Adult Literary Survey, andthe Human Capital Indicators Project. These activities are examined at the theoreticaland conceptual level by analyzing the processes associated with the selection,definition and conceptualization of competencies, and the empirical considerationsapplied.This report is the initial activity of the OECD program Definition and Selection ofCompetencies: Theoretical and Conceptual Foundations (DeSeCo). Initiated in 1998,DeSeCo is a three-year program that seeks to develop, through international andinterdisciplinary collaboration, a framework for understanding the competencies thatan individual needs to lead a personally and socially successful life and for a democraticsociety to face the challenges of the present and the future.Order Number: Orders: Price:312-9900 032 713 60 60 Fax: 032 713 60 61 Sfr. 6.– ISBN 3-303-15197-0