Chris Chapman, National Center for Education ... - Duke University

Item Changes in Repeating Studies
Sponsored by the National Center for
Education Statistics
Chris Chapman
National Center for Education Statistics (NCES)
Questionnaire Design Issues in Longitudinal and Repeated Cross-
Sectional Surveys
Duke University
February 18, 2011

This discussion is intended to promote the
exchange of ideas among researchers and policy
makers. The views expressed during discussion
and in these handouts are part of ongoing research
and analysis and do not necessarily reflect the
position of the U.S. Department of Education.

NCES Longitudinal Studies
Longitudinal studies of high school students
High School Longitudinal Study – HSLS (began fall 2009)
Longitudinal studies of college students
Beginning Postsecondary Study – BPS (began fall 2003)
Longitudinal studies of young children
Early Childhood Longitudinal Study – Kindergarten Class of
2010-11 – ECLS-K:2011 (began fall 2010)
Longitudinal studies of faculty
Beginning Teacher Longitudinal Study – BTLS (began fall
2007)
Please visit - http://nces.ed.gov

NCES Repeating Cross Sectional Studies
Studies of K-12 schools and teachers
Annual Common Core of Data – CCD
Schools and Staffing Survey – SASS (2007-08)
Studies of postsecondary schools and teachers
Annual Integrated Postsecondary Education Data System – IPEDS
National Survey of Postsecondary Faculty – NSOPF (2003-04)
Studies of pre-K – 12 students and families
National Assessment of Educational Progress – NAEP
National Household Education Surveys – NHES (2007)
Studies of postsecondary students
National Postsecondary Survey of Student Aid – NPSAS (2007-08)

NCES fields data collections on a wide range of
education topics
Focus on changes to Early Childhood Longitudinal
Studies of kindergarten cohorts and National
Household Education Surveys of pre-K through
grade 12 children
Changes by design
ECLS and NHES
Changes to improve item response and obtain more information
Changes to introduce new concepts and address new policies
Changes to reflect mode shifts

Planned Changes - ECLS Academic Assessments
Academic assessments were to measure growth
over the course of the Early Childhood Longitudinal
Study – Kindergarten Class of 1998-99(ECLS-K)
Needed to fit a math, a reading, and a social
studies/science assessment into about one hour
Did not have assessments that fit these parameters
Mode: Interviewer-directed, in-person, interactive assessment
kindergarten through grade 5, shifting to self administered paper and
pencil in grade 8

Planned Changes - ECLS Academic Assessments
Some of the items were borrowed or adapted, with
copyright permission, from
Peabody Individual Achievement Test-Revised (PIAT-R)
Peabody Picture Vocabulary Test-Revised (PPVT-R)
Primary Test of Cognitive Skills (PTCS)
Test of Early Reading Ability (TERA-2)
Test of Early Mathematics Ability (TEMA-2)
Woodcock-Johnson Tests of Achievement Revised (WJ-R)
Assessment experts including teachers developed
item pools starting from these existing sources

Planned Changes - ECLS Academic Assessments
Item pools were field tested with approximately
1,500 students
We reviewed results to select items that scaled
unidimensionally within a subject domain using IRT
assumptions
We also ran DIF analyses to determine if items did
not work properly

Increasing Item Response – Example from ECLS
How many of the children in each of your classes
demonstrated the following reading skills when they started
school this year?
WRITE NUMBER ON LINE.
a. Recognized letters ___
b. Read words ___
c. Read complete sentences ___
Mode: Self administered paper
Source: Early Childhood Longitudinal Study – Kindergarten Class of 1998-99 (ECLS-K)

Increasing Item Response and Expanding Content
Example from ECLS
What proportion of the children in each of your classes demonstrated
the following skills when they started school this year?
MARK ONE FOR EACH CLASS YOU TEACH.
Measures Response options
Recognized letters Less than ¼ of the children
Read words About ¼ of the children
Read complete sentences About ½ of the children
Mode: Self administered paper
About ¾ of the children
More than ¾ of the children
Source: Early Childhood Longitudinal Study – Kindergarten Class of 2010-11(ECLS-K:2011)

Increasing Item Response and Expanding Content
Example from ECLS
What proportion of the children in each of your classes
demonstrated the following skills when they started school this year?
MARK ONE FOR EACH CLASS YOU TEACH.
Measures Response options
Recognized letters Less than ¼ of the children
Read words About ¼ of the children
Read complete sentences About ½ of the children
Recognize numbers to 20 About ¾ of the children
Count to 20 More than ¾ of the children
Add or subtract two numbers
Source: Early Childhood Longitudinal Study – Kindergarten Class of 2010-11(ECLS-K:2011)

Adding Precision to Items – Watching TV in ECLS
ECLS-K, Kindergarten - Recorded as whole hours only.
Next, I have a few questions about {CHILD}'s television
viewing habits. How many hours a day does {CHILD}
usually watch TV or videos on school days?
Now think about the weekend. How many hours does
{CHILD} usually watch TV or videos on Saturday and
Sunday combined?
Mode: Telephone CATI
Source: Early Childhood Longitudinal Study – Kindergarten Class of 1998-99 (ECLS-K)

Adding Precision to Items – Watching TV in ECLS
ECLS-K, First grade and later - Recorded as hours and minutes.
On any given weekday, how many hours of television, videotapes, or
DVDs on average does {CHILD} watch at home? How about…
a. Before 8:00am?
b. Between 8:00am and 3:00pm?
c. Between 3:00pm and dinner time?
d. After dinner time?
How about on Saturday and Sunday? How many hours does {CHILD}
watch television, videotapes, or DVDs at home on…
a. Saturdays?
b. Sundays?

Adding Precision to Items – Watching TV in ECLS
Reported TV Viewing Time In ECLS-K for First Grade and
Kindergarten (in Minutes)
Grade of ECLS-K Study
First grade Kindergarten
WEEKDAY
Before 8am 15
8am – 3pm 5
3pm to dinner 54
After dinner 54
Total per weekday 127 113
WEEKEND
Saturday 165
Sunday 140
Sat. & Sun. total 305 301

Changing Content Focus of Items – ECLS Example
In a typical day, how much time does your class spend in the following
activities? CIRCLE ONE NUMBER ON EACH LINE.
a. Lunch
b. Recess
Response options:
1-15 minutes
16-30 minutes
31-45 minutes
Longer than 45 minutes
MODE: Self administered paper
Source: Early Childhood Longitudinal Study – Kindergarten Class of 1998-99 (ECLS-K)

Changing Content Focus of Items – ECLS Example
In a typical day, how much time do children in your class or classes
spend in the following activities? MARK ONE ON EACH ROW.
a. Lunch
b. Free play indoors
c. Free play outdoors (including recess)
Response options:
No time
1-15 minutes
16-30 minutes
31-45 minutes
Longer than 45 minutes
Source: Early Childhood Longitudinal Study – Kindergarten Class 2010-11 (ECLS-K:2011)

Integrating Administrative Data – NHES Example
During interview monitoring for NHES:2005, we heard
confusion about a school affiliation question:
- Is the school church-related or not church-related?
Some respondents were not clear about whether to
report schools related to religions that did not have
“churches”
We modified wording to broaden the concept for field
testing for NHES:2007
- Is (CHILD)’s school affiliated with a religion?

Integrating Administrative Data – NHES Example
Field test indicated new wording did not cause confusion
We ran a parallel test to determine
If parents would give us exact name of schools
If we could effectively match to our universe school- level
collections
- Please tell me the name of the school (he/she) attends.
Parallel test allowed us to identify exact school for over
95 percent of sampled children

Changing Mode – NHES Example
NHES was a CATI-based survey
We were able to develop complex skip patterns transparent to
interviewers and respondents
This allowed us to capture a full range of family structures
without significant response burden
Falling response and coverage rates led to a
redesign of NHES to a self-administered mail-out
mode
Can no longer use complex skip patterns possible in CATI

Changing Mode – NHES Example
Original skip patterns are too complicated to show
here, but they allowed us to capture:
Traditional family structure
Children being raised by grandparents
Children being raised by same-sex couples
Children being raised by other non-parent relatives
Children in foster families
Focus is on sampled children so family relations are
primarily in terms of relationships to reference child

Changing Mode – NHES Example

Changing Mode – NHES Example

Changing Mode – NHES Example
NHES Field Test experimenting with new household
and parent sections in split-ballot
Random sample of respondents will receiving
“alternate” wording
Field Test data will be analyzed to look at
Respondent difficulties with navigating the section
Respondent difficulties choosing who Parent 1 and Parent 2
should be
Frequency of traditional and nontraditional household structures
Consistency between parent items and family structure items

Chris Chapman - chris.chapman@ed.gov
Questions about the presentation
Gail Mulligan – gail.mulligan@ed.gov
Details on ECLS-K:2011
Andy Zukerberg – andrew.zukerberg@ed.gov
Details on new NHES
nces.ed.gov
For More Information
Link to our staff listings to find contacts for each study