Is headspace making a difference to young people’s lives?

Appendix C
6 June 2014. The ‘no treatment’ group is drawn from the comparison surveys and comprises young
people who have not sought any substantial headspaces treatment and who also did not seek any
treatment from any other health professionals between the two survey waves. The ‘other treatment’
group comprises of young persons within the comparison surveys and includes all those who sought
support from a health professional(s) between Waves 1 and 2. A key limitation of this group is that
no information about the dose of the other treatment is known. Young people within the comparison
surveys that sought a substantive headspace treatment between Wave 1 and Wave 2 were excluded
from the analysis. This constitutes 90 observations.
Steps were also taken to match the headspace survey intervention group to administrative (hCSA)
data in order to assess the timing of the survey collection against the period of treatment at a
headspace centre. This process uncovered 340 observations (32.3%) that had completed their
headspace treatment prior to the Wave 1 collection date. These observations were excluded from the
analysis.
Difference-in-difference Method
The objective of the difference-in-difference (DID) method is to assess the changes in young
people’s mental health, physical health, drug and alcohol use and social inclusion outcomes after
using headspace services relative to other comparable young people that did not receive treatment at
headspace.
A difference-in-difference approach has been applied, using survey data for headspace treatment
and comparison groups that have been matched on a number of different factors. Differencein-differences
estimates are defined as the difference in the average outcome in the headspace
treatment group at two points in data collection, that is wave 1 and wave 2, minus the difference in
the average outcome in the matched comparison group. Comparison groups have been matched
using a propensity score matching (PSM) technique and have been divided into groups that have
received no mental health treatment between the survey waves and those that have received
treatment other than headspace. Both are described further below.
The differences in outcomes for headspace clients, for matched samples drawn from the
no-treatment and other-treatment comparison groups, and for the difference in differences between
headspace and either of these two matched comparison groups, are tested for their statistical
significance and using effect size measures.
Statistical significance is judged using an orthodox t-test for the mean differences between the
headspace treatment group and a propensity-score matched comparison group, and for the
difference in the differences between headspace and comparison groups.
Effect sizes can be expressed in a number of ways, with Cohen’s d commonly reported as a
standard indicator in clinical evaluation. The Cohen effect size measure presents a standardised
difference in means across the course of an intervention (ie. the ratio of mean difference to a pooled
standard deviation measure). In this regard it relates to classical measures of the significance of
mean differences. It is common practice to interpret effect sizes according to the magnitude of the
index – the normal ‘rule of thumb’ is to consider effects as small (0.2) through medium (0.5) to large
(0.8) when relating the size of the effects.
Propensity Score Matching
A complexity of the DID methodology stems from the fact that those in headspace treatment group
are often dissimilar in their observed and unobserved characteristics when compared to young
people within the comparison surveys. In order to mitigate these differences, a propensity score
matching technique has been applied to the datasets.
The implementation of propensity score methods requires that a set of characteristics are chosen as
the basis for alignment of the headspace treatment with a matched comparison group. It would seem
appropriate to match simultaneously on a large number of factors, however, the match on each single
characteristic becomes less precise as the number of matching characteristics increases. For the
purpose of this evaluation, four characteristics were chosen - age, gender, K10 score and the number
of non-functioning days ‘out of role’ to source a match between the headspace and comparison
Social Policy Research Centre 2015
headspace Evaluation Final Report
178