Using evaluation data to manage, improve, market, and sustain ...

Promising Practices in Children’s Mental Health
Systems of Care - 2000 Series
report is that system of care services can result in dramatic decreases in residential placement costs, and this
analysis had a substantial impact on the community as well as broader audiences of children’s mental health
services.
In addition to these public evaluation products distributed to wide audiences, the evaluation team
made a concerted effort to produce publications and national conference presentations to supply information
about the efforts and accomplishments of the system of care to the academic community. After the first four
years of the grant, evaluation team members participated in over three dozen conference presentations and
published one dozen articles in educational and psychological journals in collaboration with family members
and community service providers. Also, approximately 15 to 20 newspaper articles appeared in local and
regional newspapers documenting the system’s progress throughout the project.
Recently, the MISC Family Program and evaluation team developed a customized Family Report,
specifically formatted for family members. The MISC Family Program selected data analyses from the
larger monthly report with particular relevance to families, included family-friendly interpretations, and
widely distributed the report to all MISC families.
Critical Analyses
According to the site director and evaluators, the analyses that brought about the most concrete
changes included: (1) an initial presentation of risk factors, and (2) longitudinal analyses concerning
“improvers” and “deprovers” in the system of care.
When presented in the first evaluation report, the documented levels of risk factors occurring within
the MISC children and their households—compared across ethnicities and to the aggregated national data,
were simultaneously alarming and instructive to the community. These analyses (see Exhibit IV-4: Analysis
of Risk Factors) had an early and instrumental effect on empirically supporting the system of care model
emphasizing family focus and collaboration. The data, according to the site director, definitively confirmed
that a single agency could not manage the multiple problems of the children and their families on their own,
thereby supporting the need for family partnership and cross-agency collaboration.
In addition, the analyses documenting differential outcomes (“Improvers and Deprovers”) forced
people to think about individuals for which the system of care might work best. An example of an analysis
to examine behavioral improvements and declines is shown in Exhibit IV-5: Improver/Deprover Data
Analysis. These analyses also were used within MISC to compare the differential progress of various ethnic
groups served by the system and to examine the cost-effectiveness of services. The Improver/Deprover
analyses have helped the MISC stakeholders to ask more sophisticated questions about their services such
as “For whom do wraparound services work best and at what cost?”
Volume II: Using Evaluation Data 71