2024 MIT IDE Annual Conference Event Report

TABLE OF CONTENTS
INTRODUCTION &
3 HIGHLIGHTED RESEARCHERS
8 GETTING GEEKY
David Verrill, IDE Executive Director, and
Albert Scerbo, IDE Associate Director
Technology-Driven Organizations and Digital
Culture Research Group
4
TESTING G AI IN THE FIELD
EN
9
AI’s IMPACT ON JOBS,
ACCESS AND INFLUENCE
Generative AI and Decentralization Research Group
Artificial Intelligence, Quantum and Beyond
Research Group
5
GUARDRAILS FOR AI
10
FIGHTING MISINFORMATION
& CONSPIRACY THEORIES
Building a Distributed Economy Research Group
Misinformation and Fake News Research Group
6
THE CASE FOR AI FRICTION
11
TAKING DATA ANALYTICS
TO THE NEXT LEVEL
Human-First AI Research Group
Data and Analytics Research Group
WATCHING AI LEARN
7 12 KEY TAKEAWAYS
AI Marketplaces and Labor Economics
Research Group
8 Big Ideas from the 2024 IDE Annual Conference

3
INTRODUCTION
“These topics are affecting pretty much everyone every day,”said IDE
Executive Director David Verrill in his opening remarks. That’s one truth
that can be trusted.
Access Video
Albert Scerbo
Generative AI is already answering
your search queries and your
customer calls, generating job posts
and resumes, and collaborating
with human teams on the job. But
can we trust its answers? What
regulations should be in place?
And is AI development being
dominated by commercial
interests? These are among the
topics being investigated by
researchers at the MIT Initiative
on the Digital Economy (IDE).
David Verrill
At the IDE’s 2024 Annual
Conference, an online membersonly
event during the week of May
20, speakers described AI projects
about trust, policy, productivity and
economics that they hope will
provide practical guidance to those
building and using GenAI. IDE
researchers also explained why, in
the rush to monetize and
implement AI, rigorous research
and careful consideration are
especially needed.
While GenAI is top-of-mind, the
conference also featured
presentations on other topics
vital to the digital economy. In
all, eight research group leaders,
postdocs and doctoral students
presented their cutting edge
studies. Their topics included
quantum computing, digital
culture, countering false
conspiracy theories online, and
the credibility of social media
platforms.
HIGHLIGHTED RESEARCHERS
Nur Ahmed
Michael Caosun
Patrick Connolly
Thomas Costello
Harang Ju
Shayne Longpre
Sahil Loomba
Robert Mahari
Benjamin Manning
Cameron Martel
Alex Moehring
Jonathan Ruane
Ana Trisovic
Emma Wiles
Yunhao Jerry Zhang

Generative AI and Decentralization Research Group
4
TESTING G ENAI
IN THE FIELD
Experiments measure productivity, human collaboration and trust.
To see how smart and reliable AI can be, tests are
moving from the laboratory to the real world. These
“road tests” can also help businesses estimate how
much productivity to expect from their AI investments.
Several efforts are underway at the IDE to explore AI
productivity, human collaboration and trust. IDE
Director Sinan Aral told attendees at the Annual
Conference, “We really want to get a handle on applied
GenAI and what it means for business.”
“The rubber meets the road” with large-scale
experiments, said Aral, who also heads the IDE’s
Generative AI and Decentralization Group.”
Two GenAI experiments are taking shape in Aral’s
research group. One involves an AI platform called
MindMeld developed by MIT Sloan postdoc Harang Ju
and doctoral student Michael Caosun. This online
platform pairs users and large language models (LLMs)
on tasks that include writing ad copy. The humans
work with either GenAI assistants or other humans to
measure collaboration and productivity.
The second experiment examines human trust in
generative search. Search results generated by an LLM
can include links to other references; sometimes, users
provide feedback on whether the results were helpful.
In the IDE field experiment, the key questions studied
were: Can we trust GenAI? Should we trust it? And
when do we trust it?
Nearly 5,000 test users were randomly given either
generative search results or conventional search results
to some 50,000 online queries. They were then asked
how much they trusted the results and how willing
they were to share that information.
Based on preliminary survey results, Aral reported that
people generally distrust the generative results versus
traditional search results when told which responses
are generated by AI. This holds true even when
traditional results and GenAI results are identical but
arranged differently.
Adding citations or references isn’t always the solution,
either. While people tend to trust AI more when
references are included, those additions can be
inaccurate.
“The veneer of rigor is more
important than rigor itself,” Aral
said. “It can give people trust in AI
even when it's not warranted.”
Sinan Aral
Overall, Aral noted that there are clear pros and cons
to generative information. On the one hand, GenAI can
be adaptable, flexible, specific, responsive and rigorous.
But GenAI can also “hallucinate,” making references to
research and papers that don’t exist. “It looks
authoritative,” Aral said, “but it’s really not.”
Access Video
Access Blog

Building a Distributed Economy Research Group
5
GUARDRAILS FOR AI
Researchers describe efforts to audit AI datasets, treat regulatory compliance as
a feature, not a bug.
Two researchers working with Alex
“Sandy” Pentland, the Faculty
Director of MIT Connection
Science and Lead for the IDE’s
Building a Distributed Economy
Group, spoke about AI policy issues
at the 2024 Annual Conference.
Shayne Longpre, a doctoral
candidate at the MIT Media Lab,
discussed a new AI topic known as
data provenance. He maintains that
the origin of a dataset’s ownership
is vital to the accuracy of AI training
data and important to those
building AI models. “This
information was not well
documented or understood,”
Longpre said. As a result, data may
be inappropriate for a given
application, or it may not represent
the right tasks, topics, domains or
languages. It may even be used
illegally.
That led Longpre and
representatives from 10
organizations, including MIT, to cofound
the Data Provenance
Initiative. It’s a collaborative effort
to audit the datasets used to train
large language models (LLMs). So
far, some 1,800 datasets have been
reviewed.
Some of what the group’s audits
turned up is disturbing.
They found that on HuggingFace, a
major platform that hosts datasets,
nearly two-thirds of the datasets
had incorrect or omitted licenses
that state access permissions.
To help, the team developed the
Data Provenance Explorer. It lets
users select subsets of languages
and licensing constraints, submit
their selection across different
criteria, and see information about
the underlying data.
Alex ‘Sandy’ Pentland
Another proactive AI project was
described by Robert Mahari,
Research Assistant at the MIT
Media Lab. Mahari’s approach is
called regulation by design, and it
involves embedding regulatory
objectives directly into a technical
design.
Regulation by design would give
people the confidence to use AI
systems knowing that do not violate
a law or regulation. To foster realworld
implementations, Mahari’s
group is working with the European
Union, World Bank, U.S. Copyright
Office and Singaporean Privacy
Agency.
“Compliance and
regulation by design
represent a riskmanagement
paradigm
that’s uniquely suited
for AI,” Mahari said.
“Through intelligent
technology design, it
can proactively prevent
failures and risks.”
Nearly 2/3
Share of datasets on
HuggingFace, a major
platform that hosts
datasets, found to have
incorrect or omitted
access permissions.
Access Video

Human-First AI Research Group
6
THE CASE FOR AI FRICTION
Research finds ‘beneficial friction’ keeps humans in the AI loop, boosting accuracy.
Artificial intelligence was the top
agenda item of the Annual
Conference session led by Renée
Richardson Gosline, head of the
IDE’s Human-First AI Group.
Gosline discussed the importance of
adding “beneficial friction”—
essentially digital speed bumps—to
AI systems to encourage users to be
more deliberative and, when
necessary, to change and correct
direction.
“Our goal,” Gosline told attendees, “is to amplify
the benefits of AI and minimize any potential
harm.” Adding questions or alerts, she added,
“ensures that humans are in the loop.”
As an example, OpenAI, the creator
of the popular ChatGPT system, has
added a pop-up message that says,
“Check your facts…ChatGPT may
give you inaccurate information.”
Another presenter, MIT Sloan
postdoc Yunhao “Jerry” Zhang,
explained findings from an
experiment he and Gosline
conducted to examine how humans
perceive AI. In their experiment,
both humans and AI systems wrote
advertising content for marketing
campaigns. The content was created
using four paradigms: human only,
AI only, augmented AI editor (the
human writes the first draft,
Renée Richardson Gosline
then edits using AI feedback), and
augmented human editor (the AI
writes the first draft, and then edits
using human feedback).
Some 1,200 participants were
randomly assigned to view the
content. Some didn’t know the
creator; others were partially
informed about the AI; and a third
group was fully informed about the
content’s origin.
The upshot: When people didn’t
know how the content had been
generated, they generally
considered the AI-generated
content to be valuable. But when
they did know, they favored the
content created by humans. “There
is evidence of human favoritism,
but not AI aversion,” Zhang said,
proving the value of human
inclusion.
Similarly, responsible AI was
highlighted by Patrick Connolly,
Global Responsible AI Lead at IDE
partner Accenture Research.
Connolly, Gosline and four other
researchers have co-written a
paper, Nudge Users to Catch
Generative AI Errors, featured in a
recent issue of the MIT Sloan
Management Review.
At the conference, Connolly
maintained that responsible AI is
essential to competitive advantage.
Technology alone won’t lead to
successful AI. Winning firms, he
said, will be those that “build
[responsible AI] into their core.”
“Traditional barriers of data, talent,
budgets and scaling proof-ofconcepts
are not holding back
companies today,” Connolly said.
Barriers now include intellectual
property, AI hallucinations and
cybersecurity. Responsible AI, he
added, will overcome these
concerns by being trustworthy,
accurate and high-performing.
Access Video

AI Marketplaces and Labor Economics Research Group
7
WATCHING AI LEARN
Two experiments move us closer to understanding AI intelligence.
The age of AI working side-by-side
with humans could be here sooner
than expected. Experiments at the
IDE already find AI acting like
humans—and in some cases
outperforming them.
AI as a writer and editor was the
subject of linked presentations by
John Horton, an Associate
Professor at MIT Sloan and lead of
the IDE’s AI Marketplaces and
Labor Economics Group, and
Emma Wiles, a doctoral candidate
at MIT Sloan.
In their experiment, a GenAI
system was prompted to write the
first drafts of job descriptions that
employers could then post online to
attract job candidates. Employers
with access to the AI-written drafts
were about 20% more likely to post
the descriptions than those without
the AI. The managers also spent
about 40% less time writing or
editing job posts than the control
group.
However, when it came to actual
hires, employers with access to AIwritten
job descriptions made
nearly 20% fewer hires than the
others.
The downturn in hiring among the
treatment group surprised the
researchers; they expected AI
would not only improve the
descriptions, but also increase the
number of hires.
While experimental
results don’t always
work out as planned,
Horton said, “it gives
us a road map for how
to improve these kinds
of features, which we
still think have an
enormous amount of
potential.”
John Horton
A presentation by MIT Sloan
doctoral candidate and IDE affiliate
Benjamin Manning explored
whether a large language model
(LLM) could complete the four
high-level tasks of a social scientist:
create a hypothesis, run an
experiment, analyze the results, and
then update the hypothesis.
That may sound like too much for
an AI system but, Manning said,
“that’s exactly what we did.”
To test the system, researchers
simulated bidding at an art auction.
The LLM hypothesized that the
higher the buyers’ budgets, the
higher the price of the final deal.
After running the simulation more
than 340 times, the LLM’s
hypothesis was generally correct.
When researchers fed the results of
the simulations back into the LLM
—the equivalent of a human social
scientist analyzing the results of
their experiment—the LLM
adjusted its predictions just as a
human would.
With each iteration,
“the model performed
much better,”
Manning said.
“It improved based on
experimentation on
itself—which is pretty
cool.”
Access Video

Technology-Driven Organizations and Digital Culture Research Group 8
GETTING GEEKY
Digital culture transforms management; quantum computing shows promise.
“A bunch of geeks” have figured
out a better way to run a business,
Andrew McAfee told attendees of
his Annual Conference
presentation. McAfee is Co-
Director of the IDE and the author
of the bestselling businessmanagement
book, The Geek
Way.
As McAfee explained, “geeky”
companies including Netflix and
SpaceX have developed new
management techniques that let
them overtake longer-standing
competitors. For example, it was
SpaceX—and not NASA or any
other national space agency—that
in 2022 managed 80% of all
satellite launches from the Earth,
McAfee said. Similarly, Netflix has
a market value over $270 billion,
more than double that of either
Disney or Warner Brothers.
Andrew McAfee
The Geeks adopted four new
norms, McAfee said:
1.Ownership
2.Openness
3. Science
4.Speed
Geek companies, McAfee
concluded, “move faster, are a lot
more egalitarian, give a great deal of
autonomy, and try to settle their
arguments via evidence. This is a lot
better than what we were doing
before.”
In a linked presentation, IDE
Research Scientist Jonathan Ruane
discussed quantum computing’s
progress. In contrast to
conventional computing’s binary
approach, quantum systems use
quantum bits (better known as
qubits) that exist in an
indeterminate state. Work on
quantum computing is intense.
IBM, for one, expects to have a
system with over 4,500 qubits by
next year.
While that’s impressive, Ruane
doesn’t expect a fully functional
quantum computer to become
available anytime soon.
$1.60
Amount spent by U.S. companies
on digital products and
services for every dollar they
spend on non-digital products
and services. This reverses the
ratio from 2007.
“There’s this enormous
chasm,” Ruane said,
“between where we are
today with the error
rates and how low we
need to be before we
can get into practical
applications.”
$270+ billion
The market valuation of
Netflix, which surpasses the
market valuation of either
Disney or Warner Brothers – two
of Netflix’s more traditional
competitors.
Access Video
Access Blog

Artificial Intelligence, Quantum and Beyond Research Group 9
AI’s IMPACT ON JOBS,
ACCESS AND INFLUENCE
Researchers study the economic and development implications of AI’s rapid pace
of advancement. Who will dominate?
With AI transforming business
and society, three important
questions are often overlooked:
Is AI really going to take jobs? Is
AI research being led by the right
parties? And is AI accessible
broadly enough?
Researchers from the AI,
Quantum and Beyond research
group addressed these and other
questions accompanying AI’s
rapid advancements.
Group leader Neil Thompson
presented the findings of his
recent study, which used AI
computer vision to measure
actual job replacement. The
study indicated that some of
what we’ve heard about AI and
jobs is “a little overblown,”
Thompson said.
While Thompson believes
excitement around AI is
warranted, he expects to see “a
much more gradual [adoption]
as it takes longer for costs to go
down and deployments to scale.”
Short-term, he added, businesses
will do cost-benefit analyses to
determine which tasks make
sense to automate with AI.
In a related presentation, IDE
postdoc Nur Ahmed explained the
results of his paper, co-written with
Thompson and a third researcher,
and published in Science. The
authors assert that AI research is
dominated by business interests,
which should have policymakers
worried.
Their paper also warns that business
influences could curtail both
research outcomes and future
products and applications. In his IDE
presentation, Ahmed described the
policy issues at stake:
commercialization, public interest
and a concentration of power.
Among the solutions he offered
were the establishment of a national
research cloud, the use of public
datasets, and greater support for
academic and international
collaboration.
In the session’s final presentation,
Ana Trisovic, a research scientist at
MIT Future Tech, asked whether AI
is accessible and usable by a broad
enough range of people and
organizations. This question, she
noted, has important implications for
regulatory policy, research practices
and societal equity.
“There is unequal
access to computational
resources and
technologies,” Trisovic
said. “This influences
who can participate in
the AI-driven
economy.”
Limited access, she added, can
restrict scientific benefits and
innovation potential to just a few,
well-resourced institutions. If that
happens, Trisovic said, expect to
see “a significant disparity in
research advancement.”
Neil Thompson
Access Video

Misinformation and Fake News Research Group 10
FIGHTING MISINFORMATION &
CONSPIRACY THEORIES
Misinformation flourishes online. Could technology help stem the tide?
False conspiracy theories and other
forms of misinformation spread
easily online. Too easily.
David Rand, an MIT Professor and
leader of the IDE’s Misinformation
and Fake News Group, explained
during his Annual Conference
presentation that while the content
may differ, most misinformationsharing
is driven by three common
factors:
1.A lack of attention
2.Message repetition
3. Dissemination by
partisan elites and
political parties
Given widespread concerns about
misinformation online, the hunt is
on for effective deterrents. One
hope is that the spread can be
curtailed with technology itself.
That’s the subject of experiments
described at the IDE conference by
two researchers working with
Rand: Thomas Costello, an MIT
postdoc, and Cameron Martel, a
doctoral candidate at MIT Sloan.
The problem is serious; the
researchers cited a recent poll
finding that fully half of all
Americans believe in at least one
conspiracy theory. What’s more,
dissuading people from believing
these false theories is extremely
difficult.
To help, Martel has been
researching the efficacy of
misinformation warnings. His
experiments, involving thousands
of participants, have found that
warning labels do work. They
lower users’ belief in
misinformation, even among
people who distrust human factcheckers.
To test whether AI technology
could combat misinformationspreading,
Costello and his
colleagues first asked human
subjects to describe a conspiracy
theory they believed to be true.
The researchers then prompted
ChatGPT to engage in a discussion
with these human subjects.
David Rand
During the chats, the AI system
would try to persuade people to
change their views; it did this by
showing their conspiracy theories
are unsupported by facts. A control
group also chatted with the AI
model, but on a banal topic.
To measure the results, the
researchers asked subjects to rate
their belief in their conspiracy
theories on a scale of 0 to 100,
both before and after the AI chats.
Overall, the AI interventions
decreased the subjects’ beliefs in
false conspiracy theories by about
20%. “Evidence and arguments can
change your beliefs about
conspiracy theories,” Costello said.
“Needs and motives don’t totally
blind you once you’re down the
rabbit hole.”
20%
The amount people’s belief
in false conspiracy
theories dropped after
chatting with a Generative
AI system.
Access Video

Data and Analytics Research Group 11
TAKING DATA ANALYTICS
TO THE NEXT LEVEL
Access Video
Researchers explore changes to ‘long ties,’ examine social media tradeoffs.
How do we know if data is accurate
and available to large populations of
users? This question is being
addressed by MIT Associate
Professor Dean Eckles and his IDE
Data and Analytics group.
Speaking at the annual conference,
Eckles discussed new methods his
group is developing to help
organizations make better decisions
using large-scale datasets. Eckles also
conducts digital experiments that
aim to make data more widely
available to both researchers and
businesses.
Eckles and two co-researchers
described three areas of study now
underway: geographically
aggregated network data; natural
experiments in social media; and
improving decision-making with
interventions.
The geodata research paper,
published last year, analyzed data
from postal codes in the United
States and Mexico to describe and
draw conclusions about
interrelationships known as “long
ties.” These connections, Eckles
explained, can be “predictive of
economic outcomes across different
places.”
Eckles also described how errors in
the estimates made from randomized
controlled trials (known as A/B tests)
can translate into good or bad
corporate decisions. A/B tests, widely
used to inform decisions, measure
the average effect of a new
intervention on various results, such
as revenue or engagement. However,
these estimates can be error-prone.
To reduce the errors, Eckles said,
researchers can adjust the
intervention or customer group, or
develop better decision tools.
Dean Eckles
A recent experiment using
aggregated network data was
described by Sahil Loomba, a
postdoctoral fellow in the IDE
research group. As Eckles noted,
aggregated treatment-effect data can
reflect estimate errors. As a result, it
can be too high or low. To help
correct these errors, Loomba studied
several aspects of social networks:
controlling the network structure;
experimenting with different
models-based solutions; and
considering social spillover behavior
and the role of sparsity.
Another aspect of large platform
data was discussed by MIT
doctoral candidate Alex Moehring:
personalized rankings and user
engagement, based on the Reddit
news site.
Moehring first assessed Reddit’s
ranking and recommendation
algorithms. He then considered
some of the fundamental tradeoffs
that firms make when implementing
ranking and recommendation
algorithms on social media
platforms.
Among other issues, Moehring
explored the impact of boosting
content ranking and engagement on
the credibility of promoted news
content. As a result of the ranking,
he found, a majority of users actually
became more discerning. However,
a subset of users instead saw much
more low-credibility content and
misinformation. It’s up to the
platforms, Moehring concluded, to
adjust their algorithms in ways that
boost credibility.

12
KEY TAKEAWAYS
8 Big Ideas from the 2024 IDE Annual Conference
1
HUMAN
FAVORITISM
leads people to
prefer content
created by humans
over that created by
AI. People also
distrust AI search
results. However,
when people don’t
know how content
was created, they
find AI-created
content to be of
high quality.
2
DATA
PROVENANCE
will gain acceptance.
The history of a dataset’s
ownership is becoming
vital to the accuracy of
AI training data and
important to those who
build AI models.
3
THE
DEMOCRATIZATION
OF AI HAS A LONG
WAY TO GO.
Even with OpenAI efforts, large
firms dominate over academia in
development and access to data.
4
AI JOB
DESCRIPTIONS
are a mixed bag.
Employers given access
to AI-written drafts
were more likely to
post the descriptions,
yet the AI group made
fewer hires.
5
“GEEK”-
MANAGED
COMPANIES
move faster than
traditional
organizations. They’re
also more egalitarian,
offer autonomy, and
settle debates with
evidence.
6
QUANTUM
COMPUTING
has the potential to
transform how business
applications are
processed, but not yet.
Progress is being slowed
by issues around
technology, funding and
security.
RESOURCES
The Data Provenance Institute: A
Large Scale Audit of Dataset
Licensing and Attribution in AI
Long Ties, Disruptive Life Events,
and Economic Prosperity
IDE, Accenture Develop a
Business Framework for Quantum
Computing
7
AI CAN FIGHT
FALSE CONSPIRACY
THEORIES ONLINE.
Beliefs in false theories
dropped by 20% when GenAI
prompted users to see how
the beliefs were unsupported
by facts.
Misinformation Warning Labels are Widely Effective
8
JOB LOSS
FROM AI
may not be as bad as
some fear. The number
of jobs that are actually
cost-effective to
automate is much
smaller than you might
expect. AI systems are
expensive!
Which Tasks are Cost-Effective to Automate with Computer Vision?
The Growing Influence of Industry in AI Research
New Research May Calm Some of the AI Job-Loss Clamor—For Now
Sending GenAI Into the Wild
Data Authenticity, Consent and Provenance for AI are All Broken
More, but Worse: The Impact of AI Writing Assistance on the Supply
and Quantity of Job Posts
6 New Studies Put AI to the Test
How Do People Regard AI-Generated Content?
Nudge Users to Catch Generative AI Errors

THANK YOU TO OUR LOYAL SUPPORTERS
CORPORATE MEMBERS:
FOUNDATIONS:
Ewing Marion Kauffman Foundation
Google.org
MIT-IBM Watson AI Lab
Nasdaq
New Venture Fund
TDF Foundation
Editorial Content: Paula Klein and Peter Krass
Design: Carrie Reynolds
INDIVIDUALS:
Nobuo N. Akiha
Joe Eastin
Michael Even
Wesley Chan
Junichi Hasegawa
Ellen and Bruce Herzfelder
Reid Hoffman
Richard B. Homonoff
Edward S. Hyman, Jr.
Gustavo Pierini
Gustavo Marini
Tom Pappas
Jeff and Liesl Wilke
Other individuals who prefer to remain anonymous