SELS Dialogues Journal Volume 3 Issue 1

Educational Technology
Will ChatGPT Get an A on My Test?
by Vincent Wong
ChatGPT is an advanced AI language model developed
by OpenAI capable of understanding and generating
human-like text responses across various subjects and
contexts. It is either your worst nightmare or a dream
come true whether you are a teacher or a student.
Regardless of which side you are on, whether ChatGPT
will ace your test should pique your interest. The answer
will either leave your jaw dropping in disbelief or have
you doing fist pumps in excitement. In this article, I will
share my discovery journey of finding out the answer to
that question for my assessments.
It was early July; my 8-year-old daughter and I were
both out of school. One afternoon, she came up with
this game where she tries to make ChatGPT guess an
object or a character she was thinking about. To her
amazement, ChatGPT was able to guess everything she
came up with, ranging from Rubrics Cube and Moana to
more obscure things like Tapioca and Gravity. After an
hour or so, she asked me “Baba…you think ChatGPTcan
get 100 on your test?” Challenge Accepted! To initiate
my experiment, I fed ChatGPT 3.5 one of my online
assessments, verbatim. It was a biology test with a mix
of knowledge-based multiple-choice (MC) questions,
application questions (App) in multiple-choice format
and multi-select (MS) questions that had more than
one correct answer. ChatGPT scored 83%, substantially
higher than the class average of 75%. The 70-minute
test only took the AI 15 minutes to complete. It
performed best with MC questions, producing a correct
answer 93% of the time. It performed significantly worse
in MS and application questions, scoring 72% and
63%, respectively. This raised the question: How could
I outsmart ChatGPT to the extent that it would stumble
or even fail? To assess the limitations of ChatGPT, I
changed different parameters for my test questions and
fed them to the AI model. Let’s start with parameters
that didn’t have any impact on the AI’s performances.
• Number of choices: For MC and MS questions,
doubling and even tripling the number of choices
for each question had no effects on the test scores.
ChatGPT took a few seconds longer to generate
the answer, but the extra choices did not seem to
confuse its programming.
• Different wordings: ChatGPT was able to answer
the questions correctly regardless of how the
question was phrased. I tried replacing certain
phrases with their definitions instead, but it had
no effect. For example, instead of asking for the
“expiratory reserved volume”, I asked for the
“maximum volume of air exhaled” or the “ERV”.
ChatGPT was able to identify them as having the
same meaning.
• Extra information: Including irrelevant information
in the question has limited effect. For example,
when I added information related to the heart in
a question related to respiration, it had no effect.
However, extra information for family members in
a genetic question caused ChatGPT to generate
the wrong answer. The AI model seems to be
good at discerning information that are distinctly
in different categories (e.g. heart and lung) but
struggles when the extra information is similar to
the relevant ones. Let’s look at some strategies
that will either help reduce ChatGPT use or
performance.
• Lock-Down Browser: This is perhaps the most
straightforward strategy. Lock-Down Browser will
prevent copy and pasting questions directly from
D2L into ChatGPT. The student can still type the
questions into ChatGPT on a separate device but
this will take significantly longer. This method
doesn’t prevent the use of ChatGPT but should
serve as a deterrent.
SELS DIALOGUES | 8