validity of a self-assessment tool to measure - OBSSR

validity of a self-assessment tool to measure - OBSSR




Presenter: Rebekka M. Lee ScM

Karen M. Emmons PhD, Cassandra A. Okechukwu ScD MSN/MPH, Jessica L.

Barrett MPH, Erica L. Kenney MPH, Angie L. Cradock MS ScD, Catherine M.

Giles MPH, Madeleine deBlois MS, Steven L. Gortmaker PhD

Harvard School of Public Health

Department of Society, Human Development & Health


� American children do not meet nutrition & physical

activity guidelines

� Less than half are physically active for 60+ minutes per day

� Very few eat suggested amounts of fruits, vegetables, water

� Race & social class disparities in obesity, nutrition, &

physical activity

� Creating changes in environments where children spend

time—school, afterschool etc.—is a promising approach

for impacting behaviors

� Measurement has a large time & cost burden

� Many tools currently used for surveillance & assessment

of large scale initiatives have limited validity


� Increasing physical activity and improving healthy

eating among youth a major national priority

� Michelle Obama’s Let’s Move Campaign

� YMCA Partnership for a Healthier America commitment

� Simple, accurate tools are needed to assess current

practices and measure progress as programs &

interventions are disseminated broadly

The Out of School Nutrition & Physical

Activity Initiative (OSNAP)

� 5-year environmental & policy change initiative

� Group randomized control trial in 20 Boston afterschools

� Learning collaboratives delivered 2010-11 school year

� Community based participatory research approach

� Self-assessments collected at spring 2011 follow up

� Concurrent with:

� Observed plate waste (Snacks served & consumed)

� Child physical activity via direct observation &


� Observation of screen time offerings, staff behaviors, &

physical environments

� Staff questionnaires & interviews

The Out of School Nutrition & Physical

Activity Initiative (OSNAP)

� Nutrition program aims:

� Ban sugar-sweetened drinks brought in from outside the

snack program & from snacks served

� Offer water as a drink at snack every day

� Offer a fruit or vegetable option every day at snack

� Ban foods with trans fats from snacks served

� Physical activity & screen time program aims:

� Offer 30 or more minutes of physical activity for every child

every day

� Include vigorous activity at least 20 minutes 3x/week

� Limit computer time to

OSNAP criterion measures

� Observed plate waste + photography

�High interrater reliability & validity vs. weighed plate waste

- Large training time & staffing costs

� Physical activity direct observation

�High interrater reliability & moderate validity against


- Large training time & staffing costs

� Accelerometry

�High between day reliability & validity against doubly

labeled water & VO 2 max

- Expensive instruments, technically intricate

Aims & hypotheses

� Determine if a simple, low cost observational

measure completed by program staff can

accurately assess nutrition, physical activity, &

screen time within afterschool programs

1. The self-assessment tool will be a valid measure of

nutrition, physical activity, & screen time

2. Responses from site directors at intervention sites will be

more highly correlated with criterion measures than

responses from site directors at control programs


� OSNAP spring 2011 data collection

� Program self-assessment observation tool

� 25-item paper & pencil tool designed by study staff

� 20 site directors completed on 5 days

� Nutrition outcomes

� Snacks served observed on 5 days

� Observation of outside snacks & drinks on 5 days

� Snacks consumed observed on 2 days

� Physical activity & screen time outcomes

� Observations of program offerings on 5 days

� Structured scans via SOPLAY on 2 days

� Actigraph accelerometers during program time on 5 days

Program self-assessment observation tool

� Pilot tested at 4 Boston programs

� Introduced to directors on Monday of observation week

� Cover sheets includes:

� Instructions for completion

� Prompt to plan how to observe the day’s activities honestly

and accurately

� Glossary of nutrition and physical activity terms

� Prompt to answer all questions to the best of ability

� Informed consent

� $25 compensation for 5 days

� 5-10 minutes to complete each day

Physical activity & screen time items

� 5 yes/no items on physical activity offerings

� Did your program offer 30 or more minutes of physical

activity to any group of children?

� 3 yes/no items on screen time offerings

� Did your program show any broadcast or cable TV or

movies today?

Physical activity & screen time items

� A scaled item on physical activity participation

� How many children do you think were active when they

attended physical activity time?

� No PA offered

� None to ¼ of kids

� More than ¼ to half of kids

� More than half to ¾ of kids

� More than ¾ to all kids

Snack & drink items

� 6 yes/no items on foods & beverages served

� Fruits and vegetables, trans fats, grains, whole grains,

sugary drinks, water

� Was a fruit or vegetable served at snack?

� 2 scaled items on foods and sugary drinks brought

in from outside the snack program

� How many kids consumed sugary drinks from outside the snack

program (e.g. vending, home) during the afterschool day?

� None

� Few (1-5 kids)

� Some (6-10 kids)

� Many (> 10 kids)

Snack & drink items

� 4 scaled items on proportion of children served

� For the children who attended snack time, how many do

you think were served a fruit or vegetable?

� 4 scaled items on food and drink consumption

� For the children who were served a fruit or vegetable,

how much do you think they ate?

� None

� Some

� Most

� All


� Unit of analysis: weekly averages

� Pearson correlations between criterion measures &

self-assessment tool estimates of program nutrition,

physical activity, & screen time

� 1 vs. 5 days of self-assessment data

� Test for interactions with program characteristics

Characteristics of 20 Boston

afterschool programs

Affiliation # of programs


Boys & Girls Club 4

Boston Center for Youth & Families 4

Independent 4

Program characteristics

# of children enrolled (mean, range) 74.6 (32-320)

# staff (mean, range) 9.5 (3-40)

Free & reduced price lunch (%) 82%

English language learners (%) 25%

Characteristics of 20 site directors

Years of experience (mean, range) 3.5 (1 month – 17 yrs)

Age (mean, range) 35.7 (22-55)

Hours per week (mean, range) 33.0 (15.0 -47.5)

Female (%) 65%

Highest education level (%)

High school 0%

Some college 20%

College or higher 80%

Race/ethnicity (%)

White 35%

Black 40%

Hispanic/Latino 20%

Cape Verdean 5%

Baseline characteristics of consented

children (N = 596)

Age (Mean, SD) 7.82 (1.76)

Female (%) 49%

Race/ethnicity (%)

White non-Hispanic 9%

African American/Black 30%

Hispanic/Latino 30%

Asian 4%

Multiracial 5%

Cape Verdean 3%

Black Hispanic 3%

Missing 16%


� Data from 97/100 program days

� 9 items across physical activity, screen time, food,

and drink domains show strong criterion validity

with 5 days of data

� One day of self-assessment data may be enough to

validly assess fruits & vegetables, water, outside

foods & sugary drinks, and screen time

� No impact of intervention participation on accuracy

of self-reported nutrition and physical activity

Validity of nutrition & physical activity self-assessment tool among 20 afterschool programs

Self-assessment item





(1 day)



(5 days) P value


Was a fruit or vegetable offered at snack? Observation 0.52 0.84


� Limitations

� Small sample of 20 programs in one urban school system

� Generalizability

� Limited ability to test for hypothesized effect modifiers

� Subject to self-report bias

� Difficulty assessing complex nutrition outcomes

� Some physical activity items need improvement

� Strengths

� Easy to use in real world settings

� Unique attempt to merge siloed academic worlds of

measurement, health disparities & implementation research

Implications for implementation &

dissemination science

� Next steps:

� Revise physical activity items to more accurately reflect

construct of physical activity time

� Work with partners to roll out as a tool for surveillance

� Excellent tool to assess impacts as programs are

implemented & disseminated broadly

� Items closely linked to obesity & chronic disease risk are

valid compared to gold standard measures

� Limited training

� Few extra staffing needs

� Low cost

� Particularly useful for researchers & practitioners in low

resource settings where health disparities may be greatest


This research was supported by the Donald and Sue Pritzker Nutrition

and Fitness Initiative, Centers for Disease Control and Prevention

(Prevention Research Centers Grant U48DP001946 and U48DP000064).

Also a big THANK YOU to all the afterschool staff that participated in

this validation study!

Rebekka M. Lee



� Implementation science is an emerging field

� Looks beyond traditional focus of efficacy research

� Concerned with effectiveness of interventions in real

world settings

� What are the factors that influence intervention success?

� How can interventions be designed to become

embedded in organizations for sustained impact?

� Health equity is a solutions oriented approach

� Much public health research examines disparities in

health outcomes by race, social class, gender etc.

� Current project describes ways researchers can create

practical interventions that address these inequalities


� Health equity: the fair or just opportunity for health

for all members or groups of the population

� Diffusion: the process in which an intervention is

communicated through individuals in a social system

� Dissemination: the intentional process whereby

people manipulate the spread of an intervention

� Implementation: the way & degree to which an

intervention is taken up & put into practice within an


More magazines by this user
Similar magazines