2000115-Strengthening-Communities-with-Neighborhood-Data

300 Strengthening Communities with Neighborhood Data
Evaluating the Impact of Neighborhood Initiatives and Action
In many instances questions arise about the degree to which a policy,
program, or practice has affected neighborhoods. Such questions may
be in conjunction with an evaluation of a government- or foundationsponsored
initiative that is aimed at community improvement, or they
may be in response to concerns raised about whether a policy or program
is having positive or negative effects at the neighborhood level. These
impact questions cannot be answered definitively by simply looking at
trends in selected neighborhood indicators. Instead, care must be taken
to craft an evaluation design that has a plausible counterfactual; that is,
the design must include a way of comparing what would have happened
if the initiative, program, or policy under investigation had not occurred.
Although the field of program evaluation is well developed with respect
to research designs that are valid for this purpose (Rossi, Lipsey, and Freeman
2004), these techniques have proved difficult to apply in practice
to questions of community or neighborhood change (Hollister and Hill
1995; Rossi 1999). Most obviously, this difficulty occurs because the triedand-true
methods of randomized controlled trials, which have become
the gold standard for human studies, have been difficult to apply when
the subjects are neighborhoods (or households nested within neighborhoods)
instead of unrelated individuals. In this section we lay out some
of the challenges that face evaluators in answering impact questions with
respect to neighborhoods and report on several examples of approaches
that have overcome some of these difficulties.
Because most program evaluations focus on individuals, it is first useful
to discuss some of the unique aspects of place-based program and
policy evaluation. An initial question is to consider whether the policy
or program is aimed at attributes of the place or at the behaviors or
characteristics of the people within the place. Although this sounds like
a simple distinction, it is often murky in practice. For example, community
development initiatives are often directed toward improving housing
conditions or increasing economic activity in the neighborhood.
At first glance, these would seem to be attributes of the neighborhood.
However, a deeper look suggests that such outcomes are difficult to separate
from attributes and behaviors of residents or investors who will be
called on to make the housing and business decisions that are necessary
to make the programs successful. Communities may also sponsor social
programs aimed directly at individuals in the neighborhood as part of a