Is headspace making a difference to young people’s lives?

Appendix C
Data Cleaning and Analysis
This section describes the challenges that were encountered in the process of survey data cleaning.
While quality problems were present within single surveys, the need to integrate multiple surveys with
different formats added to the complexity of the process of data cleaning. The task of data cleaning
involved detecting and removing errors and inconsistencies from data in order to improve its quality
and minimise their impact on the analyses. Given the large scale and the complexities of the data
cleaning exercise, a number of quality assurance strategies were put in place to minimize the scope
for error. These included code walk-throughs to ensure that no errors were present, continuous
checking of data outputs and spot checks of individual records.
The following main problems were encountered as part of the data cleaning process:
Definition of key variables
Defining variables central to the evaluation involved a number of challenges associated with design
problems in the surveys. First, information on some of the key variables was incomplete (e.g. only
the study status of those at school could be captured in intervention group and 18-25 years old
comparison group surveys). As a result, such variables could not be meaningfully utilised in the
analysis. Second, some survey questions were included in one of the two waves only. For arguably
time-invariant variables, such as postcodes, information from one wave, where possible, was carried
over to the next one. However, time-varying variables could not be utilised in such instances (e.g.
questions on self-harm and suicidal intentions/attempts in YMM were asked in wave 1 only). Third,
information on some variables important for the analysis was not included in surveys and had to
be merged from external sources. These included information on remoteness and socio-economic
status of respondents’ residential areas, where it was assigned from external sources based on the
reported postcodes.
Representativeness of surveys
The representativeness of surveys is essential for generalising the results of the analysis. Survey
weights, if included in a survey, are commonly utilised means to achieve representativeness. While
the intervention group survey did not include weights, our comparisons across a range of observable
characteristics between the survey individuals and total headspace clients (as captured by the
hCSA dataset) confirmed that it can be used to make inferences on the headspace population as
a whole. Survey weights were provided with both comparison group surveys. YMM survey weights,
when applied, led to results supporting the representativeness of its participants over the general
population of 12-17 years olds as captured in the 2011 Census data. No such outcome has been
achieved for the comparison survey of 18-25 years olds (one potential problem is the use of limited
variables (age, state and gender) as benchmarks from which to construct the weights). The results
therefore need to be interpreted with this issue taken into account.
Alignment of surveys
The process of arriving at a single dataset based on multiple surveys involved a number of
complexities. The merging of eight surveys required attempts to resolve inconsistencies involving
data representations, units, measurement periods, etc. Additionally, correctly identifying individuals
across two waves was not a simple task due to some inconsistencies in identifiers that needed to
be resolved through alternative approaches, such as matching based on a number of observable
characteristics of individuals. The number of observations and variables included in the final merged
dataset had to be compromised in some cases due to inability of the evaluation team to satisfactorily
deal with some of these issues.
Treatment groups for survey data analysis (DID method)
Two treatment groups were selected to match to and draw comparisons to the ‘headspace treatment’
group - young people who received no treatment and young people that received another mental
health treatment. The ‘headspace treatment’ group comprises all persons within the headspace
intervention group survey who had not completed their treatment by the first wave of data collection.
This group was recruited from headspace centres over a 6 months period from 6 December 2013 to
Social Policy Research Centre 2015
headspace Evaluation Final Report
177