all-SSC outperforms participants on GSC

Ch<strong>all</strong>enge I vs. Ch<strong>all</strong>enge II
D. Rebholz-Schuhmann
16 th March 2011, CALBC Workshop II

CALBC workshop II
Mar 16, 2011, EBI
Stakes
� Corpus alignment is doable: different semantic types, single
document
� Different similarity measures for harmonisation: exact, nested, cos.
� <strong>SSC</strong> generation scales / generalisable: Iterative generation
� Partner-<strong>SSC</strong> <strong>outperforms</strong> partners on GSC
� <strong>all</strong>-<strong>SSC</strong> <strong>outperforms</strong> <strong>participants</strong> on GSC
� <strong>SSC</strong> is the best option when no GSC available
� Incremental improvement of <strong>SSC</strong> performance: Large number of
contributions leads to higher performance
� Performance of the <strong>SSC</strong> concerning the other types
� Normalisation of the <strong>SSC</strong>: concept id annotations for mentions
� Use cases of the <strong>SSC</strong>
2

CALBC workshop II
Mar 16, 2011, EBI
CALBC corpus development
December 2010
850K
June 2010
100K
December 2009
50K
June 2009
1,5K
3

1st Ch<strong>all</strong>enge Rollout (Overview)
CALBC workshop II
Mar 16, 2011, EBI
4

2nd Ch<strong>all</strong>enge Rollout (Overview)
CALBC workshop II
Mar 16, 2011, EBI
� <strong>SSC</strong>-I: result of the Pilot Project
� <strong>SSC</strong>-II: result of the first ch<strong>all</strong>enge
New
(850k + GSC)
5

Time
Timeline of the CALBC project
PMB
SAC
www.calbc.eu
Project
start
CALBC workshop II
Mar 16, 2011, EBI
Submission site
Test runs
Alignment of
Partners’ contributions
Alignment
solutions
available
Invitation
to ch<strong>all</strong>enge
Access to
test data +
submission
site
Harmonised
Corpus
available
Training
data
Active
participation
Final
Corpus
from
Pilot
6
40 <strong>participants</strong>
X submissions
Closing
First
Ch<strong>all</strong>enge
CALBC Silver Standard Corpus
6

Timeline of the CALBC project
15 <strong>participants</strong>
22 submissions
Time
Closing
1 st Ch<strong>all</strong>enge
CALBC workshop II
Mar 16, 2011, EBI
27
Participants
1 st
CALBC
Workshop
Invitation
to ch<strong>all</strong>enge
Access to
test data +
submission
site
Harmonised
Corpus
available
Training
data
Active
participation
Final
Corpus
from
1 st
Ch<strong>all</strong>enge
7
16 <strong>participants</strong>
54 submissions
Closing
2 nd Ch<strong>all</strong>enge
CALBC Silver Standard Corpus
7

CALBC workshop II
Mar 16, 2011, EBI
The Ch<strong>all</strong>enges
1 st
ch<strong>all</strong>enge
2 nd
ch<strong>all</strong>enge
# registered <strong>participants</strong> 40 28
# active <strong>participants</strong> 15 16
# submissions 18 54
8

Solution
Dictionary-based
concept
recognition
Indexing of
tokens and terms
Both, trained &
rule-based
solutions
Case-based
reasoning
CRF based,
trained NER
solution
CALBC workshop II
Mar 16, 2011, EBI
Partners & Participants
PPs |
CPs
Use of Training
Data
PRGE CHED DISO SPE
P01 [ / ] UniProtKb Jochem UMLS NCBI taxonomy
P02 [ / ]
P04 [ / ]
Different
resources incl.
UniProtKb,
EntrezGene
UniProtKb,
EntrezGene
Jochem UMLS NCBI taxonomy
Jochem
MeSH,
MedDRA, NCI,
SNOMED-CT
P06 [ / ] UMLS
P10 [ / ]
P13 [ / ]
UniProtKb,
EntrezGene
NCI, MeSH,
SNOMED-CT
NCBI taxonomy
P15 [ / ] UMLS UMLS UMLS UMLS
P03 [ / ]
UniProtKb,
EntrezGene
Jochem UMLS NCBI taxonomy
P09 [ / ] UMLS
P07 [ / ]
P16 [ / ] Genia UMLS
P11 YES [ / ] [ / ] [ / ] [ / ]
P12 YES [ / ] [ / ] [ / ] [ / ]
P14 YES [ / ] [ / ] [ / ] [ / ]
9

Rec<strong>all</strong>
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
CALBC workshop II
Mar 16, 2011, EBI
Assessment: PRGE, CHED
PRGE 0.0% 20.0% 40.0% 60.0% 80.0% 100.0%
Precision
Rec<strong>all</strong>
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
CHED
<strong>SSC</strong>-I
<strong>SSC</strong>-I
0.0% 20.0% 40.0% 60.0% 80.0% 100.0%
Precision
Rec<strong>all</strong>
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
PRGE
Series1 0.0% 20.0% 40.0% 60.0% 80.0% 100.0%
Rec<strong>all</strong>
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
CHED
<strong>SSC</strong>-II
Series1 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0%
Precision
Precision
<strong>SSC</strong>-II
10

F-Meas
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
CALBC workshop II
Mar 16, 2011, EBI
Assessment: PRGE, CHED
PRGE F-Meas CHED
<strong>SSC</strong>-I / PRGE <strong>SSC</strong>-II / PRGE
0 2 4 6 8 10
� Performance of the partners’ systems decrease for
PRGE/CHEM from <strong>SSC</strong>-I to <strong>SSC</strong>-II
� Performance of the <strong>participants</strong>’ systems increase
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
0 2 4 6 8
<strong>SSC</strong>-I / CHEM
<strong>SSC</strong>-II / CHEM
11

CALBC workshop II
Mar 16, 2011, EBI
Assessment: SPE, DISO
F-Meas <strong>SSC</strong>-I / SPE <strong>SSC</strong>-II / SPE
F-Meas
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Partners
SPE DISO
Part.
0 2 4 6 8 10
� For SPE/DISO the drop in performance for the PP’s
systems is not very big
� Performance for the <strong>participants</strong>’ systems increase
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
<strong>SSC</strong>-I / DISO <strong>SSC</strong>-II / DISO
0 2 4 6 8 10
12

CALBC workshop II
Mar 16, 2011, EBI
Partners & Participants
Nr. Of
anntoations
in the <strong>SSC</strong>-
I
Nr. Of CPs
Nr. Of
submissions
from CPs
Average
nr. Of
annotations
from <strong>all</strong>
CPs
Nr. Of
annotations
in the <strong>SSC</strong>-
II
CHED 228,622 6 11 233,398 238,431
PRGE 275,235 9 15 343,681 435,797
DISO 300,637 8 11 255,599 245,524
SPE 317,211 7 9 277,071 304,503
Cos-98 P12 P11 P03 P04 P01 P02 P10 P14 P08 P15 P09 P06 P07 P09 P13 P16
SPE 93% 93% 79% 83% 71% 69% 84% 69% 56% 42% 2%
DISO 87% 89% 71% 69% 82% 76% 78% 62% 51% 32% 3% 73%
CHEM 83% 84% 75% 82% 49% 68% 51% 20% 17% 3% 23%
PRGE 81% 73% 77% 66% 66% 59% 40% 52% 12% 18% 2% 50% 11% 28%
Avg. 86% 85% 76% 75% 67% 68% 68% 58% 35% 27% 2%
13

CALBC workshop II
Mar 16, 2011, EBI
Partners & Participants
Confusion Matrix, exact matching
Reference set
Assessment PGN DISO SPE CHEM
PGN 412,866 2,673 395 106,560
DISO 451,175 4,024 2,126
SPE 474,453 912
CHEM 414,798
Confusion Matrix, nested matching
Reference set
Assessment PGN DISO SPE CHEM
PGN 412,866 2,927 695 113,546
DISO 3,055 451,175 7,910 2,826
SPE 516 7,992 474,453 1,376
CHEM 107,436 2,859 1,155 414,798
14

CALBC workshop II
Mar 16, 2011, EBI
Stakes
� Corpus alignment is doable: different semantic types, single
document
� Different similarity measures for harmonisation: exact, nested, cos.
� <strong>SSC</strong> generation scales / generalisable: Iterative generation
� Partner-<strong>SSC</strong> <strong>outperforms</strong> partners on GSC
� <strong>all</strong>-<strong>SSC</strong> <strong>outperforms</strong> <strong>participants</strong> on GSC
� <strong>SSC</strong> is the best option when no GSC available
� Incremental improvement of <strong>SSC</strong> performance: Large number of
contributions leads to higher performance
� Performance of the <strong>SSC</strong> concerning the other types
� Normalisation of the <strong>SSC</strong>: concept id annotations for mentions
� Use cases of the <strong>SSC</strong>
15

CALBC workshop II
Mar 16, 2011, EBI
End
16