December 11-16 2016 Osaka Japan

Algorithm 1 Online Sentence Segmenter
Require: w 0 , w 1 , w 2 , . . .,
1: W ← []; S ← []
2: for w k in stream of words do
3: W ← W + [w k ] ⊲ assume W = [w 0 , w 1 , . . . , w k−1 , w k ]
4: s k−1 ← confidence of segmenting before w k
5: S ← S + [s k−1 ] ⊲ assume S = [s 0 , s 1 , . . . , s k−1 ]
6: B ← apply segmentation strategy to S ⊲ assume B = [b 0 , b 1 , . . . , b k−1 ]
7: if b i = 1 (0 ≤ i ≤ k − 1) then
8: output [w 0 , w 1 , . . . , w i ] as a segment
9: remove first i elements from W and S
10: end if
11: end for
segmentation strategy outputs no boundary, no action is taken (represented by (c) in Figure 2). Figure 2, a
segment will be output and the inner sequence will be updated accordingly (as in the process represented
by (d) in Figure 2).
The following two subsections describe the boundary confidence scores and segmentation strategies
in detail, respectively.
3.1 Segment Boundary Confidence Score
This confidence score is based on an N-gram language model. Suppose the language model order is n.
The confidence score represents the plausibility of placing a sentence boundary after the word w i ,
that is, converting the stream of words into · · · , w i−1 , w i , 〈/s〉, 〈s〉, w i+1 , · · · , where 〈/s〉 and 〈s〉 are
sentence start and end markers. The confidence score is based on the ratio of two probabilities arising
from two hypotheses defined below:
Hypothesis I : there is no sentence boundary after word w i . The corresponding Markov chain is,
P 〈I〉
i
= P left · P (wi+1 i+n−1 ) · P right
i+n−1
∏
= P left · p(w k |w k−1
k−n+1 ) · P right (1)
k=i+1
where p denotes the probability from the language model, P left and P right are the probabilities of the left
and right contexts, of the words wi+1 i+n−1 .
Hypothesis II : there is a sentence boundary after the word w i . The corresponding Markov chain is,
P 〈II〉
i
= P left · P (〈/s〉, 〈s〉, w i+n−1
i+1
) · P right
= P left · p(〈/s〉|w i i−n+2) · p(w i+1 |〈s〉) ·
i−n+1
∏
k=i+2
The confidence score is defined as the ratio of the probabilities P 〈II〉
i
s i = P 〈II〉
i
P 〈I〉
i
= p(〈/s〉|w i i−n+2) ·
p(w k |w k−1
i+1 , 〈s〉) · P right (2)
and P 〈I〉
i
, that is,
i−n+1
p(w i+1 |〈s〉)
p(w i+1 |wi−n−2 i ) ·
∏ p(w k |wi+1 k−1 , 〈s〉)
p(w
k=i+2 k |w k−1
k−n+1 ) (3)
142