Maintworld Magazine 3/2021

DIGITALISATION
NLP CAN BE CONSIDERED A BRANCH OF AI
AND IS ALL ABOUT MAKING SENSE
OF HUMAN LANGUAGE.
given all other words in that text (e.g. “It is [MASK] today.
Let’s go to the beach”). Knowing the conditional probability of
words is the basis for a variety of downstream NLP tasks such
as content creation, language translation, auto-completion,
question answering and text classification. Most language
models are pre-trained on a large dataset (such as Wikipedia)
and afterwards fine-tuned to perform a specific NLP task on a
smaller dataset. This process is called transfer learning.
What is GPT-3?
GPT-3 is the third version of the Generative Pre-trained
Transformer models developed by OpenAI, the AI specialist
co-founded by Elon Musk, although he is no longer onboard.
Nevertheless, there are no major breakthroughs in terms
of architecture, it is considered the most powerful language
model ever. Why? Because of its size!
The model has a stunning 175 billion parameters and was
pre-trained on a corpus of nearly half a trillion words, mainly
sourced from the internet. In fact, the model is so large that
no fine-tuning at all is required. It knows so much about
language that it can learn NLP tasks that it has never encountered
before by just giving it a few examples. This is called
few-shot learning. The idea of such a general model is very
tempting because it opens the path for democratizing AI and
making NLP accessible for everyone.
paragraph by revealing all the text was generated by GPT-3. A
few months ago, The Guardian generated some publicity with
a similar experiment.
There are a lot of other intriguing examples making GPT-3
a real buzz in the AI community and even beyond. But why is
it such a big deal for Natural Language Processing (NLP)?
NLP so far…
NLP can be considered a branch of AI and is all about making
sense of human language. NLP originates from the 1950s
but the last decade brought a real revolution. We went from
vectorizing words and analyzing word similarities (e.g. “man
is to boy” what “woman is to girl”) with word2vec in 2013 to
the Transformer model proposed in the “Attention Is All You
Need” paper released in 2017. Transformers leverage the use
of attention mechanisms to gather information about the
context of a given word and encode that information in the
vector representing the word (e.g. I have a “date” tonight vs
What “date” is it today?). Transformers are the basis for stateof-the-art
language models such as BERT and GPT.
"Attention please"
Language models are the new kids on the block in NLP. They
basically predict how likely one word will appear in a text,
Let’s get practical
The idea is tempting. Unfortunately only limited people
have access to the beta API for GPT-3 for now, but subscribing
to the premium version of AI Dungeon gives you a workaround
allowing you to catch a glimpse of what the model
really can do.
Let us have a look and see how it could help maintenance
professionals and operations managers in supporting their
day-to-day job. I gave the model a little context for warming
up and asked a question about pump failure modes.
The text in bold is fully generated by the AI system.
Centrifugal pumps are one of the world's most widely used
type of pump, having an extensive range of applications, from
food processing to water or sewage transportation. Problems
that arise within these machines decrease the flow of the fluid
within the pipelines, thus interrupting the production and
transport of the fluid to its destination within the process.
What are the most common failure modes that are found in
centrifugal pumps in the sewage industry?
A: The main cause for failures is a lack of lubrication.
If there is no oil or grease on the gears, then it will not
move smoothly and the machine may stop working altogether.
3/2021 maintworld 15