Prospective crime mapping in operational context Final report

Appendix 3. Detailed evaluation methodologyThe purpose of this appendix is to describe the evaluation techniques used that were not discussed inthe main body of the report. Some of these are novel and have a potentially wider application thanthis project and are thus discussed so that others might use or improve upon them. This section of thereport is intended to be self-contained and so there is some duplication with the text presented in themain report, although this repetition is minimal. The methods discussed consider changes observedover time, in space and in space and time, in that order.Analyses of change over timeWhich statistical method is most appropriate for establishing the statistical significance of changes inlevels of crime in a single area over time is the matter of some debate. A number of approaches exist.First are those that consider the overall difference in the volume of crime before and after intervention.The basic approach is to compare the change in the volume of crime for a particular unit of time (say12 months) before and after intervention in both an action and comparison area. If a reduction isobserved in the action but not comparator, or the reduction in the former exceeds that in the latter thena positive inference may be drawn. To determine whether the difference in the change between thetwo areas is significant, a measure of effect size and the associated standard error is derived.There are a number of approaches to computing effect sizes for single-case designs (Allison andGorman, 1993; Lipsey and Wilson, 2001). Here attention will be given to two techniques: one usedwithin the criminological literature and elsewhere, the other developed within the field of psychologyfor the analysis of behavioural change, but (variants) also used more widely within other fields ofinvestigation, such as economics.The simplest approach is to compute an odds ratio, which simply compares the change in theintervention and comparison areas before and after intervention. An odds ratio of one indicates thatthe changes in the two areas were commensurate, suggesting no impact of the scheme. An oddsratio of greater (less) than one suggests a reduction (increase) in the intervention area relative to thechange observed in the comparison area. The statistical significance of the odds ratio can also becomputed (see Lipsey and Wilson, 2001) by estimating the standard error of the value derived. Thistechnique, which is readily interpretable, has been frequently used in research concerned with whatworks in reducing crime (for examples, see Welsh and Farrington, 2006; Gill and Spriggs, 2005), but isnot without its critics for analyses conducted at the small area level (for which fluctuations over timemay occur even in the absence of intervention: Marchant, 2005). However, the problems articulatedabout this approach are likely to be less problematic for analyses conducted at the BCU level, forwhich the variation over time is likely to be relatively stable. Thus, the approach is used here not leastbecause it provides a simple assessment of how things changed in the pilot area compared to thecomparator.Two approaches are here used to compute the standard errors (since these are critical in determiningthe significance of the effect-size derived), one used by Farrington and colleagues (see Welsh andFarrington, 2006), the other by Gill and Spriggs (2005). 10 However, both approaches converged onsimilar estimates and hence only the former are presented.An alternative to using data which has been aggregated for two periods of time (before and afterintervention) is the analysis of time-series data. For this approach, data for a number of intervals areinstead analysed. This allows more complex patterns in the data to be identified and considered inthe analysis. For example, time-series analysis can help control for what is known as serialdependence in the data: that is, to control for the fact that the residual error for an observation at onetime point is likely to be highly related to that for an adjacent time point. If such dependence existswithin the data then failing to correct for it can increase the likelihood of Type I statistical error – thelikelihood of incorrectly rejecting the null hypothesis.10 Gill and Spriggs (2005) use a slightly different approach to calculate the standard by considering monthly fluctuation in thevolume of crime to reduce a problem known as over-dispersion.83