Practical Mechanism Design for Information Elicitation and ...

Practical Mechanism Design for
Information Elicitation and Aggregation
Blake Riley
University of Illinois Urbana-Champaign
Abstract
Good decisions depend on good information. Since decision-makers rarely possess all data relevant
to their situation, experts are called on to assess costs and benets, ll gaps in knowledge, provide
forecasts, and develop alternate courses of action. Expert judgment ranges from the personal, with
doctors advising patients, to the global, with climatologists assessing the impact of carbon emission
policies. However, purported experts might be misinformed, biased, or have conicts of interest,
leaving decision-makers with the new problem of deciding who to trust. Formal tools to si accurate
judgments from pools of advisers could substantially ease this burden and improve decision quality.
Formal procedures for eliciting judgments have a long history in economics, statistics, meteorology,
and articial intelligence, among other elds. Unfortunately, existing procedures tend to be
applicable in only a narrow range of situations, are too abstract for realistic implementation, or address
only one aspect of the entire process of elicitation, aggregation, and decision-making. New tools are
needed that:
• Encourage honesty from participants without the truth eventually being publicly observed,
• Acknowledge participants are not perfect Bayesians and might have systematic cognitive biases,
• Anticipate participants strategically misrepresenting their judgments to favor personal interests,
• Are exible about the space of possible hypotheses, how participants form their opinions, and
how much participants know about the beliefs of others,
• Can operate without payments to participants and have low transactions and communication
costs, and
• Can elicit and incorporate qualitative information.
In this dissertation, I develop mechanisms for making inferences and decisions that address the above
considerations, drawing on recent advancements in the theory of robust mechanism design. I emphasize
simplicity and robustness over optimality so the resulting mechanisms can be put into practice
with minimal barriers. Numerical simulations and experiments supplement theoretical results.
Proposal
Chapter 1: Eliciting and Aggregating Probabilities from Experts
I rst survey results on proper scoring rules, which form the foundation of incentive-compatible elicitation.
Proper scoring rules assign rewards conditional on an eventually observed outcome so that
agents maximize their expected scores by reporting their true beliefs about the likelihood of each event.
If probabilities are solicited from multiple experts, these must then be combined into a single distribution
to be useful. Aggregation methods might average expert beliefs, looking for consensus, or attempt
1

to combine the information implicit in each probability, treating each opinion as data in a larger inference
procedure or mimicking a process of Aumann agreement through discussion. Prediction markets
based on market scoring rules are one means of simultaneously eliciting and aggregating probabilities.
Chapter 2: Truthful Revelation without External Verication
I next survey work on peer-prediction mechanisms, including Drazen Prelec’s Bayesian truth serum.
As “peer-prediction” suggests, participants predict the answers of other agents in addition to stating
their own opinion. Comparing predictions against each other rather than against an exogenously
revealed outcome generates truthfulness internally, enabling the elicitation of judgments on vague,
hypothetical, or very long-term questions. As long as one is willing to assume agents share a common
prior, I show large classes of these mechanisms exist. Mechanisms can be easily tailored to the question
at hand, taking advantage of natural orderings or hierarchies in possible answers.
Chapter 3: Accounting for Preferences
Scoring rules, prediction markets, and peer-prediction mechanisms assume participants care only
about maximizing their score from the mechanism, oen given to the expert as a payment. However,
agents are also likely to care about how their responses are used. e principal-agent and informationtransmission
literature in economics describes how conicts in preferences can limit communication,
but yields little practical advice. Although preferences constrain what can be implemented, this comes
with a silver lining. If agents want the nal outcome to reect their opinion (though are still willing
to misrepresent their evidence in order to persuade), honest revelation can be achieved without relying
on payments. Since agents care about the conclusions drawn from reports, inference and decision
procedures must also satisfy incentive-compatibility constraints. I investigate maximally accurate inference
procedures in this setting.
Chapter 4: Robust Mechanism Design for Elicitation
A mechanism is robust if it doesn’t depend on a precise specication of agent preferences or beliefs.
In particular, robust mechanism design typically attempts to relax assumptions about the common
knowledge of participants. In common economic settings, higher-order beliefs of participants have
no relevance for how a good should be allocated, so a robust mechanism should not be sensitive to beliefs
about the beliefs of other agents. In an information elicitation setting, higher-order beliefs could
contain useful information and can be exploited with peer-prediction mechanisms. I argue for appropriate
notions of strategic robustness in elicitation, centering around the notion of rationalizability.
Chapter 5: Assessing Qualitative Information
Although tools for collecting and aggregating probabilistic beliefs are vital, participants might be unwilling
or unable to translate qualitative knowledge into quantitative judgments. In situations when
probabilities can’t be naturally interpreted as frequencies, mechanisms should be capable of handling
qualitative input. is design challenge is tied to developing practical mechanisms for innite hypothesis
or type spaces. One initial solution, proposed by Vincent Conitzer, separates agents into two
groups: one providing information and the other translating statements into probabilitiesis. I investigated
this proposal further.
2