Statistical Methods in Physics Part VII Bayesian Statistics, Blindness ...

✬
Statistical Methods in Physics
✩
Statistical Methods in Physics
Part VII
Bayesian Statistics,
Blindness,
and Systematic errors
✫
February 25, 2013 Christian Walck, SU Page 1
✪
✬
Statistical Methods in Physics
✩
Some extras ....
Here we just want to briefly describe three statistical concepts important
to know about
• Bayesian statistics – very brief, Jan will hopefully find time to expand
a bit on this.
• Blindness
• Systematic errors
On the course web site I have included interesting references on all three
topics if you would like more information. Cowan mentions Bayesian
statistics on page 6 but do not seem to discuss the other two topics.
✫
February 25, 2013 Christian Walck, SU Page 2
✪

✬
Statistical Methods in Physics
✩
BAYESIAN STATISTICS
• Thomas Bayes (1702-1761) was an english matematician and a
Presbyterian minister.
• While Bayes’ theorem, which he formulated, is a simple consequence of
probability theory the use of so called Bayesian statistics in other
areas are very controversial.
• There are a lot of “philosophical” arguments and differences which are
quite interesting between this and the non-Bayesian (or frequentist)
approach
• A nice article is “Why isn’t every physicist a Bayesian?” by Robert D.
Cousins in American Journal of Physics 63 (1995) 398 (see course
homepage for link).
✫
February 25, 2013 Christian Walck, SU Page 3
✪
✬
Statistical Methods in Physics
Thomas Bayes (1702-1761)
✩
✫
February 25, 2013 Christian Walck, SU Page 4
✪

✬
Statistical Methods in Physics
✩
• Within the field of statistics there has been a clear division and
dispute between classical (or frequentistic) statisticians and Bayesian
statisticians since early last century.
• Basically Bayesian statistics uses prior knowledge (priors) with
information from earlier measurements/experiments.
• Frequentists argue that this “degree of belief” is too subjective and
hard to agree upon ...
• ... while Bayesians argue that even no prior (as a uniform distribution)
is anyway an assumption.
✫
February 25, 2013 Christian Walck, SU Page 5
✪
✬
Statistical Methods in Physics
✩
• Physicists often mix, even without realizing, between a frequentist and
a Bayesian approach. Since we are used to learn from past experiments
one may think that we should be inclined to be Bayesians but often a
more frequentistic approach is used (possibly making it easier to
combine different results).
• However, the use of Bayesian statistics in physics has increased in
popularity during the last years and there are many paper and even
books on the subject.
✫
February 25, 2013 Christian Walck, SU Page 6
✪

✬
Statistical Methods in Physics
✩
Robert Cousins on Bayesian statistics ....
Reference: American Journal of Physics 63 (1995) 398
✫
February 25, 2013 Christian Walck, SU Page 7
✪
✬
Statistical Methods in Physics
✩
BLINDNESS
• Within physics blind analysis was, at least previously, not regarded as
needed in order to assure quality of results.
• When having clear measurements and obvious effects this may be true.
On the other hand remember the comment by Ernest Rutherford “If
your experiment needs statistics, you ought to have done a better
experiment.”
• In contrast it is well known from fields as e.g. medicine or psychology
that blind or even double-blind techniques are necessary to avoid
influencing the results of a study.
✫
February 25, 2013 Christian Walck, SU Page 8
✪

✬
Statistical Methods in Physics
✩
• Indeed also in physics one has noted that unconsciencely one easily
affects results (while placing cuts, deciding on removing “bad”
measurements, having an idea on what the result “should be” etc)
• Or rephrasing it slightly: In a statistical analysis of experimental data
samples there are substantial risks to become biased. E.g. to be guided
by previous measurements, expectations or being biased by data itself.
• It has become quite common in physics analyses to use a blind
technique either using a subset for developing analysis methods or by
scrambling the data set.
• One should always keep in mind that it is easy to get fooled even by
ones own senses and effects are sometimes very subtle. As in the next
example ...
✫
February 25, 2013 Christian Walck, SU Page 9
✪
✬
Statistical Methods in Physics
Clever Hans (Der Kluge Hans)
✩
✫
February 25, 2013 Christian Walck, SU Page 10
✪

✬
Statistical Methods in Physics
✩
Clever Hans (contd.)
• Clever Hans (der Kluge Hans in German) was a horse that was though
to have special abilities. It was claimed, and demonstrated, that this
horse could make arithmetic calculations and solve other even much
more elaborate tasks.
• When the owner would ask a question the horse answered by tapping
his foot. Normally it would give the correct answer to everyones
amazement.
• This was in fact not done to fool anyone delibarately (as was shown in
later studies) but it was not understood how the horse could do this.
✫
February 25, 2013 Christian Walck, SU Page 11
✪
✬
Statistical Methods in Physics
✩
• A big investigation was initiated in 1907 to reveal a fraud or to find
out what was happening.
• This was made by making blind experiments where either the horse
did not see the examiner or none of them knew the correct answer etc.
• Exchanging the horse owner with someone else did not change the
positive results showing that it was not a deliberate fraud.
• The horse got the correct answer only when the questioner knew the
answer and the horse saw the questioner.
✫
February 25, 2013 Christian Walck, SU Page 12
✪

✬
Statistical Methods in Physics
✩
• In the investigation in 1907 it was found that the horse actually did
not solve the problems but was watching the human reactions.
Unconsciously the owner or other observers would make a minor
gesture when the true answer was approaching which the horse would
note.
• Even informing the examiner that he should not do any special
movement when the horse came to the correct answer did not help!
• One may say that the horse was very clever but not in the sense of
solving the problems but to note very subtle behaviour of humans!
✫
February 25, 2013 Christian Walck, SU Page 13
✪
✬
Statistical Methods in Physics
✩
• Again: If the horse could not see the questioner or the questioner did
not know the answer it failed to deliver correct answers.
• Similar things may happen in many every-day situations and the
example shows that one should always be sceptical to subjective
observations which easily could be interpreted as strange and fantastic.
• In physics we often have much more firm measurements and it may
seem unnecessary to take such precautions. However, it turns out that
even physicists hade prejudices which may influence the results of an
analysis.
• This is the reason why it has become common even in physics analysis
to adopt blind analysis.
✫
February 25, 2013 Christian Walck, SU Page 14
✪

✬
Statistical Methods in Physics
✩
• In my experiment, using the neutrino telecope IceCube at the South
Pole, we always perform blind analysis. In some cases blindness may
be achieved by scrambling (some variable in) data in other cases an
analysis is optimized on a subset of the data. In the first case the full
data sample may be used in the final analysis while in the second case
we would discard the subset used to optimize the analysis.
• A very nice reference describing many aspects of blind analysis is an
article in Annual Review of Nuclear and Particle Science by Joshua
Klein and Aaron Roodman from 2005.
✫
February 25, 2013 Christian Walck, SU Page 15
✪
✬
Statistical Methods in Physics
✩
Klein & Roodman on Blindness
✫
February 25, 2013 Christian Walck, SU Page 16
✪

✬
Statistical Methods in Physics
✩
Nobel Laureate Burton Richter on blindness...
Ref: AAAS 2007, arXiv:hep-ex/0702026
✫
February 25, 2013 Christian Walck, SU Page 17
✪
✬
Statistical Methods in Physics
The Unknown
As we know,
There are known knowns.
There are things we know we know.
We also know
There are known unknowns.
That is to say
We know there are some things
We do not know.
But there are also unknown unknowns,
The ones we don’t know
We don’t know.
✩
(Donald Rumsfeld, Feb. 12, 2002, Department of Defense news briefing)
✫
✪
February 25, 2013 Christian Walck, SU Page 18

✬
Statistical Methods in Physics
✩
SYSTEMATIC ERRORS
• An important consideration in all physics experiments is the
understanding and evaluating of systematic effects. For low statistics
(i.e. small samples) experiments statistical effects will normally
dominate but for high statistics the systematic effects will dominate.
(Not always true!)
• Intrinsically systematic errors are not well known. If we knew the
effects we could correct for it and take it into account in our analysis.
Some sources may be understood and investigated by simulations.
• Normally not easy to find reliable ways of evaluating systematic effects
...
✫
February 25, 2013 Christian Walck, SU Page 19
✪
✬
Statistical Methods in Physics
✩
• Systematic effects often have unknown distributions (if one even may
speak of a distribution in a unknown quantity). How much and
according to what distribution should we vary our inputs? Are they
correlated?
• A normal method is to vary parameters in a “reasonable range” and
study the influence on the final result
• Overcautious analysers may vary things too much thereby
overestimating their systematic errors
• ... but the “unknown unknows” often leads to underestimating them.
✫
February 25, 2013 Christian Walck, SU Page 20
✪

✬
Statistical Methods in Physics
✩
Heinrich & Lyons on Systematic Errors
Reference: Annu. Rev. Nucl. Part. Sci. 57 (2007) 145
✫
February 25, 2013 Christian Walck, SU Page 21
✪
✬
Statistical Methods in Physics
✩
A few “quotes” from the review:
• The nature of systematic effects is such that they may not cause
different answers when the experiment is repeated. Thus, a consistent
set of results does not imply the absence of systematics.
✫
In general, the reliable assessment of systematics requires much more
thought and work than for the corresponding statistical error.
• Some errors are clearly statistical (e.g., those associated with the
reading errors), and others are clearly systematic (e.g., the correction
of a measured quantity to another reference point). Others could be
regarded as either statistical or systematic (e.g., the uncertainty in the
recalibration of a ruler). Our attitude is that the type assigned to a
particular error is not crucial. What is important is that possible
correlations with other measurements are clearly understood.
February 25, 2013 Christian Walck, SU Page 22
✪

✬
Statistical Methods in Physics
✩
• The result of the experiment may be quoted as x ± σ stat ± σ syst , where
the statistical and systematic errors are shown separately. If a single
error is required, then typically σ stat and σ syst are combined in
quadrature, on the grounds that they are uncorrelated.
• At the other extreme, some or all of the systematic errors can be
shown individually. This would be useful in combining different
measurements, for which some of the systematic effects may be
correlated between the different measurements.
✫
February 25, 2013 Christian Walck, SU Page 23
✪