Barry McGaw.pdf

nss.gov.au

Barry McGaw.pdf

Making maximum use of our data assets:

Knowledge and capability issues

Barry McGaw

Vice-Chancellor’s Fellow, University of Melbourne

Chair, Australian Curriculum, Assessment and Reporting Authority

NatStats 2013: A better informed Australia: the role

of statistics in building the nation

Brisbane

14 March 2013


Role of scientific evidence and big data in

informing decisions, research and debate


Data on educational achievement

International

‣ OECD Program for International Student Assessment (PISA)

‣ International Association for the Evaluation of Educational Achievement

• Trends in International Mathematics and Science Study (TIMSS)

• Progress in International Literacy Survey (PIRLS)

National

‣ National Assessment Program (NAP) Sample surveys

• Science Literacy

• Civics and Citizenship

• Information and Communication Technology Literacy

‣ National Assessment Program: Literacy and Numeracy – full cohort

‣ End of secondary education assessments


Message from international surveys about quality

Level

Yr 3

Yr 8

15yo

Relative

position

IEA 2011

OECD

PIRLS TIMSS PISA 2009

Reading Maths Science Reading Maths Science

Rank 27 19 24

No. ahead 21 17 18

No. same 6 6 9

No. behind 22 30 27

Rank 12 12

No. ahead 6 9

No. same 9 6

No. behind 30 30

Rank 9 15 10

No. ahead 7 12 6

No. same 3 4 7

No. behind 55 49 52


Reading literacy

Message from international surveys about equity

High

Two indices of relationship

Social gradient

Correlation or variance accounted for

Social gradient:

Magnitude of increment

in achievement

associated with an

increment in social

background

(on average)

Low

Correlation or variance accounted for:

How well the regression line

summarises the relationship

PISA Index of social background

Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 8.1, p.308

Social

Advantage


Reading

Social gradients for reading (PISA 2009)

550

High quality

Low equity

Korea

Finland

High quality

High equity

525

500

475

450

Canada

New Zealand

Japan

Australia

Belgium

Netherlands

Switzerland Norway

Germany USA Poland

France Hungary

Sweden Ireland

UK

Denmark

Italy

Slovenia Greece

Czech Rep Slovak Republic

Israel

Austria

Luxembourg

Chile

Estonia

Iceland

Portugal

Spain

Turkey

Low quality

Low equity

Mexico

425

-15 -10 -5 0 5 10 15

Social gradient (OECD regression slope – country regression slope)

OECD (2010) PISA 2009 Results: overcoming social background, Fig. II.3.2, p.55.

Low quality

High equity


Reading

Correlations for reading (PISA 2009)

550

525

500

475

450

High quality

Low equity

Hungary

High quality

Korea High equity

Finland

Canada

New Zealand

Japan

Australia

Belgium

Netherlands Norway

Poland Switzerland

Estonia

Germany USA Iceland

France

Sweden

UK

Ireland

Denmark Portugal

Slovenia

Italy

Greece

Slovak Republic Spain Czech Republic

Luxembourg Israel

Austria

Turkey

Chile

Low quality

Low equity

Mexico

425

-12 -10 -8 -6 -4 -2 0 2 4 6 8 10

Variance in reading accounted for by social background (OECD-country)

OECD (2010) PISA 2009 Results: overcoming social background, Fig. II.3.2, p.55.

Low quality

High equity


Variation in reading performance (PISA 2000)

120 Variation of performance

100

within schools

80

60

40

20

Australia

68%

32%

0

-20

-40

-60

-80

Explained by SES

Not explained by SES

Variation of performance

between schools

OECD,UNESCO (2003), Literacy skills for tomorrow’s world: further results from PISA 2000, Table 7.1a, p.357.


NAPLAN Performance

One national message about quality and equity

4.0

School B

2.0

School C

0.0

-2.0

School A

-4.0

800 900 1000 1100 1200

Index of socio-educational advantage (ICSEA)


Knowledge and capability challenges


Knowledge and capability challenges

Educational research community

‣ Few quantitative researchers being trained

‣ Challenges to validity of quantitative work – often uninformed

Best data sets have restricted access

‣ Need to protect confidentiality of students (but not schools or systems)

‣ Concerns about ‘misuse’ of data (e.g. inter-systemic comparisons)

‣ Data sets could be enriched with better connections to other sets

Data quality can be an issue

‣ Socio-educational advantage

• Get quality of ABS Census data without ecological fallacy when

characterising the homes of students in particular schools


Strategies for dealing with challenges


Knowledge and capability challenges

Training researchers

‣ Strengthen quantitative training for educational researchers

‣ Attract people from other fields

‣ Establish validity of quantitative work through utility

Improve access to data sets

‣ ACARA can grant access upon application

‣ Better connect data sets

A particular self-interested proposal

‣ Next census collect data on which schools children attend

• ABS Census data applied to schools – ecological fallacy

• Direct data from parents – validity higher but reliability may be dubious


Thank you.

More magazines by this user
Similar magazines