MOCA MMSE - Regional Geriatric Program of Eastern Ontario

THE M AIN EVENT
MMSE
MoCA
vs.
Regional Geriatric Program of Eastern Ontario
City Wide Rounds
Dr. Frank Molnar
April 17, 2009

Objectives
1. To learn how to review validation studies (studies of
sensitivity and specificity)
2. To use this new knowledge to review validation
studies for the use of the MOCA in dementia / MCI
3. To discuss the copyright issues of the MMSE (and
MOCA)
4. To discuss the optimal use of the MOCA
• For those with MMSE ≥ 26?
• For those in whom the MMSE is unrevealing?
• In place of the MMSE?

Disclosure
• I do not own stocks in nor have I been employed by
any company with a financial interest in either the
MMSE or MOCA.
• I have formal methodological training (i.e. MSc
Thesis) in the derivation and validation of cognitive
tests.
• I use both the MMSE and the MOCA in clinical
practice
• I am not a copyright lawyer

Acknowledgements
• Ian MacDowell
• Malcolm Hing
• Judy Willoughby
• My patients
• Everyone else who has argued with me about
these concepts

The Preliminary Event
• In order to truly understand the
results of the studies to be reviewed
we need to understand:
• The definitions of sensitivity and
specificity
• How sensitivity and specificity are
affected by:
• Cut-off values employed
• Overlap of cognitive scores
• Choice of test

Definitions
Sensitivity
• % of diseased persons identified as
diseased (score below cut-off)
Specificity
• % of normal persons identified as
normal (score above cut-off)

1. Sensitivity and specificity are
affected by the cut-off score
employed

Scores for persons
with normal
cognition
Specificity = 25%
xxx x x
x x x x x
xx x x x
x x x x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
30
20
10
0
xx x x
xx x
xxx x
x
xx xx
xx
xxx x
x x xx
xxx xx
x xx x x
Scores for persons
with dementia
Sensitivity = 100%

Scores for persons
with normal
cognition
Specificity = 50%
xxx x x
x x x x x
xx x x x
x x x x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
30
20
10
0
xx x x x
xxxx x
x xxx x
xxxxxx
xx xxx
xxxxx
xxx x
x x xx
x
Scores for persons
with dementia
Sensitivity = 87.5%

Scores for persons
with normal
cognition
Specificity = 75%
xxx x x
x x x x x
xx x x x
x x x x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
30
20
10
0
xx x x x
xxxx x
x xxx x
xxxxxx
xx xxx
xxxxx
xxx x
x x xx
x
Scores for persons
with dementia
Sensitivity = 75%

Scores for persons
with normal
cognition
Specificity = 100%
xxx xx x
x x x x x
xx x x x
x x x x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
30
20
10
0
xx x x x
xxxx x
x xxx x
xxxxxx
xx xxx
xxxxx
xxx x
x x xx
x
Scores for persons
with dementia
Sensitivity = 62.5%

Scores for persons
with normal
cognition
Specificity = 100%
xxx xx x
x x x x x
xx x x x
x x x x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
30
20
10
0
xx x x x
xxxx x
x xxx x
xxxxxx
xx xxx
xxxxx
xxx x
x x xx
x
Scores for persons
with dementia
Sensitivity = 35%

Take Home Message #1
Sensitivity and Specificity for any given
test are dependent on cut-off score
studied
For scales where high scores are good and
low scores are bad (MMSE, MOCA)
• When cut-off is lowered
• Sensitivity decreases
• Specificity increases
• When cut-off is raise
• Sensitivity increase
• Specificity decreases

Sensitivity vs. Specificity
100
90
80
70
60
50
40
30
20
10
0
10 20 30
MMSE and MoCA
specificity
sensitivity

2. Sensitivity and specificity are
affected by the population in
which the test is being used
- Overlap of cognitive scores
(spectrum of disease)

Scores for persons
with normal
cognition
Specificity = 75%
xxx x x
x x x x x
xx x x x
x x x x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
30
20
10
0
xx x x x
xxxx x
x xxx x
xxxxxx
xx xxx
xxxxx
xxx x
x x xx
x
Scores for persons
with dementia
Sensitivity = 62%

Scores for persons
with normal
cognition
Specificity = 75%
xxx x x
x x x x x
xx x x x
x x x x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
30
20
10
0
xx x x x
xxxx x
x xxx x
xxxxxx
xx xxx
xxxxx
xxx x
x x xx
x
Scores for persons
with dementia
Sensitivity = 75%

Scores for persons
with normal
cognition
Specificity = 75%
xxx x x
x x x x x
xx x x x
x x x x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
30
20
10
0
xx x x x
xxxx x
x xxx x
xxxxxx
xx xxx
xxxxx
xxx x
x x xx
x
Scores for persons
with dementia
Sensitivity = 87.5%

Scores for persons
with normal
cognition
Specificity = 75%
xxx x x
x x x x x
xx x x x
x x x x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
30
20
10
0
xx x x x
xxxx x
x xxx x
xxxxxx
xx xxx
xxxxx
xxx x
x x xx
x
Scores for persons
with dementia
Sensitivity = 100%

Scores for persons
with normal
cognition
Specificity = 100%
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by
test (i.e. % of normals that
score above cut-off)
MOCA or MMSE
xxxx x x x
x x x xxx x
xx x xxx
30
20
10
0
xx x x x
xxxx x
x xxx x
xxxxxx
xx xxx
xxxxx
xxx x
x x xx
x
Scores for persons
with dementia
Sensitivity = 100%

Less overlap – higher combined
sensitivity and specificity
0 10 20 30
Greater overlap – lower combined
sensitivity and specificity
0 10 20 30

Correct population distribution
0 10 20 30
Incorrect distribution resulting in exaggerated
sensitivity and specificity
0 10 20 30

Take Home Message #2
The sensitivity and specificity depend on the
amount of test score overlap between
normal and diseased
• Sensitivity and specificity depend on
sample / population
Since the populations we take care of
clinically are different from those in studies
• The Sensitivity and Specificity of a test in
clinical practice will likely not match that in
studies (we cannot know if it does)
• We can still compare the tests within 1 study

3 Take Home Messages
1. Sensitivity and Specificity for any given
test are dependent on cut-off score
2. Sensitivity and Specificity depend on
sample / population
Since the populations we take care of
clinically are different from those in studies
the Sensitivity and Specificity of a test in
clinical practice will likely not match that in
studies
3. Sensitivity and Specificity are dependent
on the test employed

3. Sensitivity and specificity
are affected by the test
employed

THE MAIN EVENT
MoCA
In this
corner…
MMSE

MOCA validation process
• Developed based on clinical intuition of main author
(ZN)
• Iterative modification based on 5 years of clinical use
• Tested on 46 MCI / AD with MMSE > 24 vs. 46
normal
• 5 items replaced & weighting adjusted
• Clinical distribution
• We are now in the stage of validation
• Ongoing process
• Main dementia / MCI articles to be
reviewed.

Search Strategy
• Emailed the author of the first article
(Ziad Nasreddine)
• MoCA validation studies kept up-to-date on
MoCA's website reference section at
www.mocatest.org
• RGPEO Librarian searched the literature
• Did not find any studies not found on above
website

3 MOCA Validation Studies in area of Dementia
1. Nasreddine et al. The Montreal Cognitive
Assessment, MOCA: A brief Screening Tool For
Mild Cognitive Impairment. Journal of the American
Geriatrics Society 2005; 53: 695-699
2. Smith et al. The Montreal Cognitive Assessment:
validity and Utility in a Memory Clinic Setting. The
Canadian Journal of psychiatry 2007; 52; 329-332
3. Luis et al. Cross validation of the Montreal Cognitive
Assessment in community dwelling older adults
residing in the Southern US. International Journal of
Geriatric Psychiatry 2008

Nasreddine et al - Design
• 94 MCI & 93 AD (90 MMSE ≥17),90 NC
at 2 Quebec MDCs
• Clinical diagnoses
• MCI – portions of Wechsler Memory Scale
• French and English MMSEs / MOCAs
• Note; language-based (verbal) tests must
be separately validated in each language
• I am not certain this was done

Nasreddine et al - Results
• MOCA (cut-off 25/26)
• 90% SENS to detect MCI
• 100% SENS to detect AD
• MMSE (cut-off 25/26)
• 18% SENS to detect MCI
• 78% SENS to detect AD
• MOCA seems to win on SENS
(particularly for MCI)

Nasreddine et al - Results
• SPEC = % Normals ≥ 26 (correctly
identified as normal
• MOCA (cut-off 25/26)
• 87% SPEC to normals
• Mislabelled 13% as impaired
• MMSE (cut-off 25/26)
• 100% SPEC to normals
• MMSE seems to win on SPECS

Nasreddine – Results (my interpretation)
• The results only describe part of
the story
• If you lowered the MOCA cut-off, its specificity
would improve and sensitivity will drop
• If you raise the MMSE cut-off, its sensitivity would
improve and specificity will drop
• SENS / SPEC are very dependent on cutoffs.
• In any case, we do not rely solely on
cut-off scores but look at which parts of
the test were failed (disaggregate
analysis)

Nasreddine et al – recommendations
• If patients have cognitive complaints
and functional impairment then likely
dementia
• MMSE first
• MOCA if MMSE ≥ 26 (MCI, Mild dementia)
• If patients have cognitive complaints
but no functional impairment then likely
normal or MCI
• MOCA first

Smith et al.- Design
◦ 6 month prospective study of 67 patients
with MMSE > 24 at UK MDC (mean MMSE
27.4). I will only look at initial
assessment – not 6 month F/U
• 32 dementia (ICD-10)
◦ 18 AD, 13 VaD, 1 PDD
• 23 MCI (Petersen Criteria)
• 12 MCC (normal memory or clearly identifiable
psychiatric disease causing subjective memory
complaint)

Smith et al.- Results (Table 2)
◦ MOCA (cut-off 25/26)
• 83% SENS to detect MCI
• 94% SENS to detect Dementia
◦ MMSE (cut-off 25/26)
• 17% SENS to detect MCI
• 25% SENS to detect Dementia
◦ MOCA seems to win on SENS (for
both MCI and Dementia) using
25/26 cut-offs

Smith et al.- Results (Table 2)
◦ SPEC = % MCC ≥ 26 (correctly
identified as not MCI / Dementia)
◦ MOCA (cut-off 25/26)
• 50% SPEC to MCC
◦ Mislabelled 50% as MCI / Dementia
◦ MMSE (cut-off 25/26)
• 100% SPEC to MCC
◦ MMSE seems to win on SPEC at
25/26 cut-offs

Smith et al.- Results (my interpretation)
◦ The results only describe part of
the story
• If you lowered the MOCA cut-off, its specificity
would improve and sensitivity will drop
• If you raise the MMSE cut-off, its sensitivity
would improve and specificity will drop
• SENS / SPEC are very dependent on cutoffs.
◦ In any case, we do not rely solely
on cut-off scores but look at which
parts of the test were failed
(disaggregate analysis)

Smith et al.- Conclusions
◦ Findings support conclusions of
Nasreddine et al
◦ Low specificity of MOCA may be
because used MCC rather than
normal controls
◦ MOCA is a useful screen in a
memory disorder clinic for people
who score > 25 on the MMSE

Luis et al - design
• 118 community-dwelling (Florida)
subjects involved in a prospective study
• 20 probable AD (NINCDS-ADRDA)
• 24 amnestic MCI (Petersen)
• 74 cognitively normal
• looked at sensitivity vs. specificity over a
range of cut-offs. This represents a
major improvement in analytic
methodology over the 2 previous studies.

Luis et al – Results (Table 2)
• MOCA (cut-off 26/27 – higher than prior studies)
• 100% SENS & 35% SPEC to detect MCI
• 97% SENS & 35% SPEC to detect MCI/AD
• MOCA (cut-off 23/24 – lower than prior studies)
• 96% SENS & 95% SPEC to detect MCI
• This is a suspect finding:
• too good – never see these values in cognitive tests
• see next slide
• MMSE (cut-off 27/28 – higher than prior studies)
• 58% SENS & 84% SPEC to detect MCI
• MMSE (cut-off 24/25 – lower than prior studies)
• 17% SENS & 96% SPEC to detect MCI
• 36% SENS & 96% SPEC to detect MCI/AD

Sensitivity vs. Specificity
100
90
80
70
60
50
40
30
20
10
0
10 20 30
MMSE and MoCA
specificity
sensitivity

Possible explanation
Incorrect distribution resulting in exaggerated
sensitivity and specificity

Luis et al – My Conclusions
• The results are questionable - likely due
to patient selection factors
• The sample size (20 AD) was likely too small
resulting in a non-representative sample with
spectrum bias
• I believe the results indicate that 96% of MCI
participants had a MOCA ≤ 23. Does this make
sense clinically or do you see MCI patients with
MOCA ≥ 24?
• The analytic approach (examining ROC curves to find the best
cut-offs) should be the standard – I used this approach for my
MSc thesis in the 1990s and it was standard back then.

MY SUMMARY of MOCA
studies
• First 2 validation studies did not employ best methodology
• 3 rd validation study likely has spectrum bias.
• In all tests, when sensitivity increases, specificity decreases (the
reverse is also true). This trade-off of sensitivity vs. specificity
means that the increased sensitivity of the MOCA comes
with the expected price – lower specificity resulting in more
normal people being labelled as impaired (higher false
positives).
• At a cut-off of 25/26 MOCA better at picking up AD than MMSE
(with cut-off of 25/26)
– This may not be true if MMSE cut-offs are altered
Despite the limitations with these studies, I believe the
MOCA better than MMSE at detecting MCI / early
dementia
– Likely a ceiling effect of the MMSE that increasing cut-off will
probably not compensate for.

Tests may have differential sensitivity in
different ranges of cognitive decline
MCI
Normal
MOCA
30
25
20
MMSE
30
Moderate dementia
Severe dementia
Mild dementia
15
10
5
25
20
15
10
5
0 0

NOW FOR SOMETHING
COMPLETELY DIFFERENT
• That was the science
• Now we move to the social /
legal environment in which the
MMSE and MOCA operate

http://www3.parinc.com/uploads/pdfs/MMSE_Copyright_PermReq.pdf
MMSE Copyright Permission Request Form
Permission is necessary if you want to use a modified version of the MMSE, including a modified format or
translation. If you plan to use the MMSE in its entirety and in its published form, you must purchase the
number of test protocols you will need for your purposes. To purchase the published form please
contact PAR Customer Support at 1.800.331.8378, or if outside the US and Canada at 813.968.3003.
• PAR will not grant permission to include the entire MMSE Test Form or scale in any publication, including
dissertations and theses. However, inclusion of up to four sample items may be approved.
• A per-copy royalty fee will be charged for all permissions granted for copies of translations of the MMSE or
modified versions of the test form. This fee does not include the cost of the User’s Guide, which is
necessary if you plan to obtain permission to use a translation or modified version of the test form.
There will be a permission fee charged for reproduction of up to four sample items from the MMSE Test Form to
be included in publications.
• Payment can be made in U.S. funds via check, money order, VISA, MasterCard, American Express, or
Discover. We also accept official institutional purchase orders from U.S. and Canadian residents.
Please complete the remainder of this form. Required sections are noted and highlighted. Please complete all
sections that are relevant to your request. If all of the required sections are not completed, we will not be
able to process your request. Once the form is completed, please send the form to the address below:
Copyright Permissions Department
16204 North Florida Avenue, Lutz, Florida 33549
Fax: Toll free 1.800.727.9329 (US & Canada). All other locations, fax 813.968.2598
E-mail: copyright@parinc.com
Upon submission of a completed form, your request will be reviewed and we will contact you

Is this copyright valid?
• Can one copyright something that has been
in the public domain for decades?
• Similar to ‘established right of way argument’
• Can one copyright something that has
appeared in numerous publications?
• Each publication holds copyright on its content

Does this copyright effect us (will it be
enforced if found to be valid)?
• Clinically?
• Are we individuals or a program / institution?
• Research?
• Can one publish research that involves the
MMSE?
• Will they go after ‘little fish’ like us?
• Is it worth the cost of litigation for our relatively
small scale use?
• Who wants to be the test case?

MOCA – also copyrighted
• Copyright not enforced (no demands for
payment).
• Will this be permanent?
• Note the concept of ‘stealth copyrighting’
• Allow a product to gain widespread use before
enforcing copyright as occurred with the MMSE
• Is any cognitive test immune from this process
in the future?

COST:
How to read studies & select tests:
Cut-off:
Sensitivity and Specificity for any given test are dependent on cut-off score
Objective:
- screen for MCI & dementia in community (high cut-off)
- screen for dementia (not MCI) in community (lower cut-off)
- NOT for diagnosis
- on inpatient setting can only screen for cognitive impairment (delirium,
depression, MCI, dementia)
Sample:
Sensitivity and Specificity depend on sample / population. Since the
populations we take care of clinically are different from those in studies the
Sensitivity and Specificity of a test in clinical practice will likely not match that
in studies
Test Characteristics:
Sensitivity and Specificity are dependent on the test employed. MOCA has
high sensitivity but low specificity (relative to MMSE)

Given all of the above - How
do I suggest using the MOCA?
• If it was used before to compare measures
• If seeing someone who is high functioning and likely in the normal /
MCI / very early dementia range
• No / few/ borderline functional problems
• If MMSE > 26
• If MMSE results are unrevealing
• If history suggests a need for a better measure of attention / frontal
executive function (e.g. Lewy Body Disease, Frontal Lobe dementia)
and language than MMSE provides
• Look at areas where points were lost and decide how important this is.
• If using in research or for large numbers of people I would consider the
MOCA given the MMSE enforced copyright (or would consider paying
to use the MMSE)