5. Public Reporting as a Quality Improvement Strategy

public health, and quality improvement. While we attempted to adapt the methods recommended
by the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program, our
approach is only one of several, and others may be equally or more valid.
The majority of the research we identified focused on the intermediate and final outcomes
included in our analytic framework and correspond to Key Questions 1, 2, 3, and 4. We included
Key Questions 5 and 6 because understanding the implications of variation in the intervention
and content are important from a quality improvement perspective in which the desire is not just
to know if something works, but also who it works for and when. From this perspective the
results of our review were limited by the current state of the literature. Studies rarely reported
enough (if anything at all) about the public report or the context. Without this information it was
impossible to compare and contrast studies where public reporting had an impact to those where
it did not and hypothesize if the difference was due to specifics of the nature of the public reports
or the context. This leaves several important questions unanswered. For example, the current
research does not adequately address the correspondence between what patients want to know
and what is publicly provided or whether the information is assessable (e.g., do consumers have
the level of health literacy required to correctly interpret the information?). Similarly for
providers we do not know how well the publicly reported measure corresponds to what they
believe they can influence or their quality improvement goals.
There is a substantial amount of research on risk adjustment and creation of measures that
was not included in this review. The validity and acceptability of measures is essential if public
reporting is to have any impact. Research on the impact of public reporting often mentions the
importance of the validity of the measures and the research on risk adjustments will often discuss
the implications for public reporting, but research rarely links the two.
While we may not be sure if our review missed evaluations of public reports (see limitations
of the review above), we know that many more public reports exist than have been studied. An
AHRQ-maintained clearinghouse contained over 200 public reports and a recent study identified
263 public reports in 21 geographic areas. 233 This diversity of reports is not reflected in the
research literature. Public reports on cardiac surgery outcomes in three States (New York State,
Pennsylvania, and California) and Nursing Home Compare are the subject of just under half of
the total quantitative studies in this report. The fact that research has been narrowly focused on a
few public reporting initiatives may limit the generalizability and applicability of its results to
other reporting efforts and broader public policies.
Future Research
We identified a large number of studies in this review, but the return in terms of credible
guidance on how to maximize the impact of public reporting on quality of health care is
generally low. The reasons for this are translated into ideas for future research for both public
reporting and the synthesis of research on this type of intervention in this section. However this
is not an exhaustive inventory of future research needs; it is limited to what can be directly based
on or extrapolated from the experience of conducting this review, background materials
including many editorials and commentaries, as well as input from our expert panel, peer
reviewers, and public comments. These recommendations are limited in that they are not based
on a systematic interaction with stakeholders or prioritization of future research needs.
Research Reflecting the Diversity of Public Reporting. The research available on public
reporting does not appear to correspond to the current state of practice in several ways. Producers
131