Smart Industry 1/2020

Says Who?
As AI turns to solving
problems further from
human experience, the
utility of explanations
will surely be called
into question. Concerns
around how explainable
these decisions are
bound to grow.
stand how the AI model functions in
its entirety. For example, users of an
AI model which classifies animals in a
zoo may want to drill down into how a
tiger is classified. This can tell them the
information that it uses to say what is
a tiger (perhaps the stripes, face, etc.),
but not how it classifies other animals,
or how it works generally. This allows
you to use a complex AI model, but focus
down into local models that drive
specific outputs where needed.
need to know what features in the
data it used to reach that decision. Of
course, not all decisions will be correct,
and that holds whether it’s a human
or a machine making the decision.
If AI gets 80% of calls on machine
maintenance right, compared to 60%
for human judgement, then it’s likely
a benefit worth having, even if the
decision-making isn’t perfect, or fully
understood.
On the other hand, there are many
situations where we do need to know
how the decision was made. There
may be legal or business requirements
to explain why a decision was taken,
such as why a loan was rejected. Banks
need to be able to see what specific
features in their data, or which combination
of features, led to the final decision,
for instance to grant a loan.
How Do We Know When
AI Decisions Are Right?
In other cases, it is important to know
why the decision is the right one; we
wouldn’t want a cancer diagnosis tool
to have the same flawed reasoning
as the husky AI. Medicine in particular
presents ethical gray areas. Let’s
imagine an AI model is shown to recommend
the right life-saving medical
treatment more often than doctors
do. Should we go with the AI even if
we don’t understand how it reached
the decision? Right now, completely
automating decisions like this is considered
a step too far.
And explainability is not just about
how AI reaches the right answer.
There may be times when we know
an AI model is wrong, for example if it
develops a bias against women, without
knowing why. Explaining how
the AI system has exploited inherent
biases in the data could give us the
understanding we need to improve
the model and remove the bias, rather
than throwing the whole thing out.
As with anything in AI, there are few
easy answers, but asking how explainable
you need your AI to be is a good
starting point.
If complete model transparency is vital,
then a white-box (as opposed to
a black-box) approach is important.
Transparent models which follow
simple sets of rules allow us to explain
which factors were used to make any
decision, and how they were used.
But there are trade-offs. Limiting AI
to simple rules also limits complexity,
which limits its ability to solve
complex problems, such as beating
world champions at complex games.
Where complexity brings greater accuracy,
there is a balance to be struck
between the best possible result and
understanding that result.
A compromise may be the ability to
get some understanding of particular
decisions, without needing to under-
As AI turns to
increasingly
challenging
problems
further from
human experience,
there will
still have to be
human experts
who can help
qualify the
explanations.
Who Should AI Be
Explainable To?
There is also the question of “explainable
to whom?” Explanations about
an animal classifier can be understood
by anyone: most people could appreciate
that if a husky is being classified
as a husky because there is snow in
the background, the AI is right for the
wrong reasons. But an AI which classifies,
say, cancerous tissue would need
to be assessed by an expert pathologist.
For many AI challenges, such as
automating human processes, there
will have to be human experts who
can help qualify the explanations.
However, as AI turns to increasingly
challenging problems further from
human experience, the utility of explanations
will surely come into question.
In the early days of mainstream AI,
many were satisfied with a black box
which gave answers. As AI is used
more and more for applications
where decisions need to be explainable,
the ability to look under the
hood of the AI model and understand
how those decisions are reached will
become more important.
There is no single definition of explainability:
it can be provided at
many different levels depending on
need and problem complexity. Organizations
need to consider issues
such as ethics, regulations, and customer
demand alongside the need
for optimization – in relation to the
business problem they are trying to
solve – before deciding whether and
how their AI decisions should be explainable.
Only then can they make
informed decisions about the role of
explainability when developing their
AI systems.
23