01-32 Backspace Oct20-FINAL-WEB

Newsletter of the European Chiropractors’ Union
Research
Research Corner:
Null-Hypothesis Statistical Testing and
the problem of inductive reasoning
Sidney Rubinstein, DC, PhD, Jonáh Stunt, PhD
IN THE previous research
corner, we discussed Null-
Hypothesis Statistical Testing
(NHST) and the p-value. In
this article we want to elaborate
on a fundamental problem with
this approach: the problem of
inductive reasoning.
Key points
• Null-Hypothesis Statistical
Testing (NHST) is based upon
inductive reasoning.
• This approach can introduce
error because conclusions are
based upon the probability of a
given observation, particularly
when the likelihood is low.
• NHST and p≤0.05 were
never meant to be a basis for
conclusions, but only meant as a
means to disprove a hypothesis.
In order to understand the situation
better, it is important to understand
what NHST-reasoning entails.
NHST is framed as a deductive
argument and follows a rule of
inference. An example, put simply:
• If it rains, the road is wet;
• The road is dry;
• Thus: it is not raining.
However, this approach
does not always lead to correct
conclusions, even if both
conditions (i.e. the first two
statements) are true. Suppose
you buy a lottery ticket and win.
We all know that the chances of
winning the jackpot are extremely
small. If one were to formulate
a hypothesis about the chances
of winning the jackpot, it would
take the following form, using the
modus tollens framework:
• If the lottery is fair (H 0 ) then
the chance of winning the
jackpot is very small (x);
• I won the jackpot (x);
• Thus: the lottery is unfair
(reject H 0 ). [whereby H 0 is the
null-hypothesis and ‘x’ is a given
probability]
Everyone would agree that it
would be incorrect to conclude
that the lottery is false based upon
the fact that you won it. Yet, we
follow this exact same line of
reasoning when we use NHST.
So, where does the NHST line
of reasoning go wrong? It has
to do with probabilities. To
understand this, let’s go back to
the experiment with the coin
which we used as an example in
the previous edition. If you want
“This
introduces the
importance of
understanding
that a theory
can never be
proven, only
disproven.”
to determine if a coin is fair (or
true), you toss the coin a number
of times, count the number of
heads and tails, and calculate the
probability of that observation
given your null hypothesis. The
probability of throwing heads or
tails is equal, so if you were to toss
the coin five times and the coin
lands on ‘heads’ every time, this
probability would be (0.5) 5 =0.031.
Quite a small chance, and you
would question the fairness of
the coin. Suppose we were to
toss the coin a thousand times
and every time the coin lands on
‘heads’. The probability of that
happening with a fair coin would
be (0.5) 1000 which is a probability
of practically null. This would be
very strong evidence for an unfair
coin. However, the probability
is not zero! Meaning, a fair
coin could result in a 1000 times
heads if we were to toss a coin an
infinite number of times, albeit
the probability is extremely low.
Following the NHST-reasoning,
we would conclude that the coin is
not fair and thus reject H 0 .
Even though the suspicion of
a false coin is high, observations
cannot provide definitive
proof. This is exactly the
problem surrounding inductive
reasoning and explains why
the modus tollens framework of
reasoning may lead to incorrect
conclusions with NHST. The
Scottish philosopher David
Hume identified this problem
in the 18th century. Simply put,
definitive proof cannot be attained
in empirical and inductive ‘open
systems’, which is in contrast to
deductive closed systems, such as
mathematics. In such systems as
the empirical sciences, definitive
proof is impossible because one
cannot sample ALL subjects
or ALL situations which fulfil
the study criteria. Consider, for
example, subjects with neck or low
back pain. It is simply impossible
to sample everyone on this planet.
Suppose you formulate the
hypothesis that all swans are
white. One would expect that
every swan you observe would
be white, and every observation
to be confirmation of this
hypothesis. However, one black
Sidney Rubinstein
Jonáh Stunt
swan would be enough to dispel
the hypothesis. This introduces the
importance of understanding that
a theory can never be proven, only
disproven.
In short, conclusions based
upon a limited number of
observations may be false.
However, just as it is impossible
to sample ALL swans, it is equally
impossible to sample ALL subjects
with neck and/or low back pain.
NHST relies upon probabilities;
however, as with the lottery
or coin tossing example, these
probabilities are not zero, and
therefore, potentially valid.
The founding father of NHST
(Fisher) never suggested that
NHST and p≤0.05 are a basis
for conclusions. He considered
a small p-value as an outcome
interesting enough to warrant
further research, NOT proof.
“The null hypothesis”, he said,
“is never proved or established,
but is possibly disproved, in the
course of experimentation. Every
experiment may be said to exist
only in order to give the facts
a chance of disproving the null
hypothesis.”
BACKspace www.chiropractic-ecu.org October 2020 15
01-32 Backspace Oct20.indd 15 08/10/2020 19:11