Hacker Bits, August 2016

Mortality vs. info disclosure, from Frederikson et al. The red line is patient mortality.
A tradeoff between
privacy and accuracy
Now obviously calculating the
total number of ice cream-loving
users on a system is a
pretty silly example. The neat
thing about DP is that the same
overall approach can be applied
to much more interesting functions,
including complex statistical
calculations like the ones
used by Machine Learning algorithms.
It can even be applied
when many different functions
are all computed over the same
database.
But there's a big caveat here.
Namely, while the amount of
"information leakage" from a
single query can be bounded by
a small value, this value is not
zero. Each time you query the
database on some function, the
total "leakage" increases and
can never go down. Over time,
as you make more queries, this
leakage can start to add up.
This is one of the more challenging
aspects of DP. It manifests
in two basic ways:
1. The more information you
intend to "ask" of your
database, the more noise
has to be injected in order
to minimize the privacy
leakage. This means that
in DP there is generally
a fundamental tradeoff
between accuracy and
privacy, which can be a
big problem when training
complex ML models.
2. Once data has been
leaked, it's gone. Once
you've leaked as much
data as your calculations
tell you is safe, you can't
keep going, at least not
without risking your users'
privacy. At this point, the
best solution may be to
just destroy the database
and start over,if such a
thing is possible.
The total allowed leakage is
often referred to as a "privacy
budget", and it determines how
many queries will be allowed
(and how accurate the results
will be). The basic lesson of DP
is that the devil is in the budget.
Set it too high, and you leak
your sensitive data. Set it too
low, and the answers you get
might not be particularly useful.
Now in some applications,
like many of the ones on our iPhones,
the lack of accuracy isn't
a big deal. We're used to our
phones making mistakes. But
sometimes when DP is applied
in complex applications, such as
training Machine Learning models,
this really does matter.
To give an absolutely crazy
example of how big the
tradeoffs can be, consider this
paper by Frederikson et al. from
2014. The authors began with a
public database linking warfarin
dosage outcomes to specific
genetic markers. They then used
ML techniques to develop a dosing
model based on their database,
but applied DP at various
privacy budgets while training
the model. Then they evaluated
both the information leakage
and the model's success at treating
simulated "patients."
The results showed that the
model's accuracy depends a lot
on the privacy budget on which
hacker bits
9