Smart Industry 1/2020

Smart Business Title Story: Smart Companies
In AI We Trust
When and How Should AI
explain ITS Decisions?
As artificial intelligence (AI) increasingly makes decisions, there are growing
concerns around AI decision-making, and how it reaches its answers.
n By Sam Genway
AI can be complex. Unlike
traditional algorithms, AI
does not follow a set of
predefined rules. Instead,
they learn to recognize patterns
– such as when a component of a
machine will fail or whether a transaction
is fraudulent – by building
their own rules from training data.
Once an AI model is shown to give
the right answers, it is set loose in
the real world.
However, getting the right answer
does not necessarily mean it was
reached in the right way. An AI model
could be successfully trained to recognize,
for instance, the difference
between wolves and huskies. However,
it might later transpire that the
AI model really learned to tell the dif-
22
Right now,
completely
automating
decisions is
considered a
step too far.
Sam Genway
Tessella
ference based on whether there was
snow in the background.
This approach will work most the
time, but as soon as it needs to spot a
source ©: Russell Publishing Limited
husky anywhere outside of its natural
habitat, it will presumably fail. If we
rely on AI (or indeed humans) being
right for the wrong reasons, it limits
where they can work effectively.
We may instinctively feel that any machine
decision must be understandable,
but that’s not necessarily the case.
We must distinguish between trust
(whether we are confident that our AI
gets the right answer) and explainability
(how it reached that answer). We
always need to have a level of trust
demonstrated when using an AI system,
but only sometimes do we need
to understand how it got there.
Take an AI model that decides whether
a machine needs maintenance to
avoid a failure. If we can show that the
AI is consistently right, we don’t even