Full report. - Social Research and Demonstration Corp

learn$ave Project: Final Reportparticularly unlikely were altered “by hand” based onvalues of other assets and income of the respondent inthe same survey as well as earlier surveys.Further to this, it was discovered that for propertyassets/debts in particular, MI of missing values led tosome impossible results, such as high mortgages withzero property value. Because property represents such alarge proportion of net worth and relatively few peopleactually own property, it was determined that the datashould be investigated further, to ensure there were no“impossible” values. Additional analysis determined thata source of the discrepancy lay in the share question.Respondents were asked if they shared the asset or debtand how much their share was, and this was not alwaysanswered well. Some indicated they shared the assetbut not the corresponding debt, or did not answer thequestion on asset after indicating they possessed thecorresponding asset, which led to large discrepancies.Each case was dealt with on an individual basis.Values of property assets and debts were furtherchecked to ensure logical consistency, i.e. no largenegative net values of property. MI was restricted toproduce non-negative net value of property. A few casesof problematic asset share and debt were analyticallycorrected using information from previous surveys. Therewere also a few cases of outliners of property values,which were checked against the property values aroundthe respondent’s neighbourhood to eliminate input errorsby the data collection agency. A record of edits made wasmaintained.Regression adjustment of impact estimatesThe main body of this report presents regressionadjustedresults. Unadjusted impacts were estimated bycalculating the difference between the mean outcomelevels of the program and control groups. Adjusted resultswere generated by estimating a regression in which theoutcome variable was modeled as a linear function ofthe respondents’ research group and a range of socioeconomicand demographic characteristics measuredbefore random assignment. Although random assignmentensures that there are no systematic differences betweenthe program and control groups, small differences can(and did) arise by chance — particularly in smallersamples. Also, as a result of survey attrition in the 54-month follow-up survey, small differences between theprogram and control groups were introduced.Regression was used to adjust the (unadjusted) impactestimates for differences between the program andcontrol group at baseline and at 18 months (that arosefrom unbalanced attrition). The reasons why linearregression could be used are twofold: the size of thesample was large and the co-variates in the adjustmentmodel had very little explanatory power over and abovethe research group variable. The adjusted estimate of theimpact was derived from the coefficient on the researchgroup variable in the estimated model. Two-tailed t-testswere used to determine the statistical significance of theestimates.There are two main advantages to regression-adjustedimpact estimates. First, given that observed baselinedifferences between the program and control groupscan be accounted for, the regression-adjusted impactestimates are potentially more accurate than the unadjustedmean differences in outcomes. Second, even in theabsence of differences at baseline, regression adjustmentcan improve the statistical precision of impact estimates.Standard errors of regression-adjusted estimates ofthe program’s impact may be lower (when correlationbetween the characteristics and the outcome is accountedfor in the regression), which results in improved statisticalpower.Regression adjustment has some drawbacks, however.One of the main disadvantages is that adjusted estimatesare not as well understood and not as easily interpretedas unadjusted results (the latter being simple differencesbetween the program and control groups). Also, formany outcomes, the improvement in statistical precisionachieved through regression adjustment is typicallyquite small (and was proved in this case). Nevertheless,because regression adjustment capitalizes on the wealthof information generated and brings greater precisionto the estimates, the decision was made to presentregression-adjusted impact estimates in this report.In total, each outcome variable observed at 54 months(the “dependent” variable, corresponding to each of theoutcomes discussed in Chapters 6 and 7 of the report)was regressed on a set of variables that comprised theresearch group plus 14 co-variates (the “independent”or explanatory variables). Both continuous and binaryexplanatory variables were included in the model, allof which were measured using the application formor the baseline survey administered prior to randomassignment.The explanatory variables, measured at baseline, comprisethe following:• Research group(learn$ave-plus, learn$ave-only, and control)• learn$ave site• GenderSocial Research and Demonstration Corporation Appendix D | 127