The Cyber Defense eMagazine November Edition for 2023

Artificial Intelligence or Mechanical Turk?
Walking the show floor at Black Hat, most vendors were pitching some sort of AI that would “revolutionize”
defense. I found some of these messages deceptive themselves, making promises the industry has heard
for years only to disappear into vaporware and disappointment. The advances in machine learning (ML)
and large language models (LLM) have been very promising over the past few years, although still a bit
overhyped as “AI” when in reality, these technologies require reliable data inputs, along with ongoing
human tuning and supervision.
Machine learning models are only as good as the data they are fed. As any data scientist will tell you, the
majority of their job is data prep and cleansing, this also makes these models themselves susceptible to
deception through data poisoning and model manipulation. The application of LLM through tools such as
ChatGPT has been a fantastic breakthrough in the application of data science, with the promise of
increasing productivity across many different industries. LLM, however, is a machine learning model that
uses Natural Language Processing (NLP) to scan massive amounts of text. Some companies have been
deceptive about how this technology works, confusing the industry.
Although LLM technology can magically create content from a prompt out of thin air, there is more to it
than meets the eye. LLMs rely on data inputs like any other model, so they leverage existing works,
whether articles, blog posts, art, or even code. So there should be no surprise that there are now mass
lawsuits against companies like ChatGPT from content creators claiming copyright infringement and
source code is no different, not to mention privacy implications.
LLM also has another negative side effect of “hallucinations” where it will spit out nonsense or untrue
content that could trick or confuse a person if they believe the content, which shows why even some of
the most advanced uses of “AI” require a human in the loop to verify content. Interestingly, we can be
deceived by this technology by accident; however, the same technology can and is being used offensively
to manipulate data models and people and, in many respects, is ahead of the defense.
Generative Deception
At Def Con I saw the other side of “AI” on the offensive side. Both the Social Engineering and
Misinformation Villages have grown over the years. The Social Engineering CTF was amazing to watch
as teams targeted employees at companies to see how they could gather valuable information from
targets for reconnaissance. This can now be taken a step further to manipulate people using voice
synthesis to mimic the voice of an authority figure, family member, or celebrity, for example, to gain the
target's trust.
The increasingly widespread use of this technology will pose a significant threat to organizations and
individuals, mainly as many non-tech-savvy folks are unaware of it, and the models become increasingly
convincing. In addition, the use of generative AI to create videos and images that are progressively
realistic is already finding its way into propaganda, fraud, and social engineering at a horrifying rate, and
most security awareness training programs and other defenses for these types of attacks are slow to
catch up.
Cyber Defense eMagazine – November 2023 Edition 22
Copyright © 2023, Cyber Defense Magazine. All rights reserved worldwide.