The Matrix System at Work - Independent Evaluation Group - World ...

CHAPTER 4
DELIVERING QUALITY SERVICES
instead forced to rely on sub-regional or country-based “shadow managers,”
introducing an additional layer of non-technical review, as they are not always
experts within the sector.
4.26 Information from managers’ interviews validates these findings. Sector
managers typically described accountability for quality as high for the sector
manager and the task team leader, none for the sector director, and substantial for
the country director. Across all the Regions, sector directors’ accountability for
lending operations was rated as substantial (or very large) by only 13 percent (four
of 30) of sector managers and 9 percent (one of 11) of country directors interviewed.
Sector directors are deemed accountable only for cases where technical assistance
has substantial potential impact on country dialogue. In comparison, sector
managers are reported to have much greater accountability for lending, which was
rated substantial by 90 percent of the sector managers and 55 percent of country
directors interviewed. The staff survey yielded similar results.
4.27 Quality assurance systems are described as being clearer and more effective
for lending than for ESW, while accountability for NLTA is even weaker. Review
processes have been in place for lending operations since before the matrix. A twostage
review process was introduced for ESW following the 2001 evaluation of the
Strategic Compact, but until FY11 no Bank-wide requirement was in place for
review of NLTA quality. Managers and staff report that performance of the current
quality assurance system is relatively effective in ensuring fiduciary and safeguards
quality, but less so for assuring the quality of lending operations, even lower for the
quality of ESW, and lowest for NLTA. Feedback from country directors and sector
managers is fairly consistent. Managers find that the current incentive structure
results in attention to processes and reputational risk more than operational and
AAA quality. Many argued that, as one respondent put it, “too much attention on
safeguards and fiduciary takes away from time spent ensuring technical quality of
lending operations,” and that “the quality assurance system, such as it is, provides
some assurance for fiduciary/safeguard issues and ESW reports, but slight
[assurance] for quality of lending and none for NLTA.” The implications of weak
accountability of NLTA are especially serious for institutional interventions, such as
those in the public sector governance portfolio.
4.28 There is little opportunity for the quality assurance system to materially
improve many pieces of ESW. For nearly 40 percent of ESW that have decision
meetings—in which a study is put up for institutional scrutiny, and is commented
on by peer reviewers—those meetings occur less than 30 days before delivery to the
client, as shown in Figure 4.3. It is implausible that substantive peer review critiques
of a sophisticated piece of analytical work can be adequately addressed by such a
70