FINAL REPORT - International Joint Commission

FINAL REPORT
Evaluation and Screening Process
The evaluation of plans does not automatically indicate which plan is the most desirable. The Study Board
needed to compare the evaluation results to see which mix of outcomes it prefers.
In this Study, the Board decided to rank plans using both economic and non-economic metrics, a widely
accepted practice that reflects the idea that sustainable natural resource decisions must balance economic
and environmental consequences and must be equitable. Because there was no universal environmental
metric and because there was no presupposed translation between economic impacts and environmental
benefits, the Study Board used a two-year long process to refine and document the logic it applied to reject
plans because of their particular mix of economic and environmental impacts.
There has been a great deal of research and practice on the subject of making trade-offs. The “Trade-Off
Analysis Planning and Procedures Guidebook” (Yoe, 2002) presents a good summary of decision methods.
Yoe writes that “All multicriteria decision-making techniques are virtually identical in general concept.
They each include a set of alternatives, a set of criteria, weights for the criteria and a trade-off algorithm
(procedure)” and that they differ most in the last three steps of the decision-making process: the development
of weights, synthesis and decision making itself. In this Study, the Shared Vision Model allowed planners
to change plans and re-evaluate them within about an hour. That means there is a practical alternative to
making trade-offs, which is to create a new plan that reduces the need for trade-offs.
The Study Board concluded that there was a better way to develop options for the International Joint
Commission than using an analytical outranking approach. The plan ranking and rejection/selection
process was designed to be similar to the medical concept of “informed consent.” A dialogue over time
was used to exchange research conclusions produced by experts and values expressed by the Study Board
and the public in much the same way as a doctor provides expert medical knowledge and the patient tells
the doctor what is most relevant and important in his or her situation until an informed consent to a
particular approach is reached that both expert and impacted parties support.
The Study Board held seven “practice” workshops where it assessed evaluation results against its decision
guidelines and then gave direction to the plan formulators. This process allowed the Board and other
Study participants to explore the relative significance of different performance indicators to different
parties, and it helped the Study Board to determine where trade-offs were unavoidable because the
hydrologic conditions to produce benefits in different categories were truly incompatible. This philosophy
was supported by the Study Board and the Public Interest Advisory Group, which asked the Plan
Formulation and Evaluation Group to lead as many “practice” decision sessions to ensure that the real
decision would be better. Practicing the decision:
• Forced all parties to make sure the studies being conducted produced the information needed to
support a decision in a timely fashion;
• Stimulated debate about how to balance competing interests;
• Allowed the Study Board to focus on one part of the decision at a time;
• Gave the Study Board several opportunities to decide which are the key Study results that should be
tested and displayed;
• Provided the Study Board with an opportunity to test and refine its decision guidelines and
screening factors.
Each practice decision increased the Study Board’s understanding of how regulation would affect the
system and each led to improvements in the Study process. Full documentation of the highlights and
lessons learned from each of the practice workshops can be found in a report entitled “Lake Ontario-
St. Lawrence River Study Plan Evaluation, Ranking and Tradeoffs (Leger et al., 2005).
26 Options for Managing Lake Ontario and St. Lawrence River Water Levels and Flows