Lexical variation in aggregate perspective

Lexical variation in aggregate perspective 101
ulation. We use the p-value from the Fisher Exact test as a filter for DCB .Wesetthe
dissimilarity between subcorpora at zero if p > 0.05, and we use DCB if p < 0.05. 2
If we now apply this step to the fictional data from Table 1 and 2, we must first
calculate the Fisher Exact p value for every concept, verifying that the absolute frequencies
for American and British English are sampled from different populations. For
SUBTERRANEAN PUBLIC TRANSPORT,thepvalueismuchsmallerthan0.05,sowecanac cept that British English is different from American English when it comes to this concept.
Therefore, we calculate the City-Block distance by means of Equation 5 for SUB-
TERRANEAN PUBLIC TRANSPORT. Filling in the equation, we get 0.5 × [(|0.875–0.286|) +
(|0.125–0.714|)] = 0.589. For the concept of a SMALL INSTRUMENT PLAYED WITH A BOW we
find a p value for the Fisher Exact test larger than 0.05, so we can say that British English
is statistically speaking not a different population than American English. Therefore,
we can set the distance between these varieties for this concept at zero.
To calculate the dissimilarity between subcorpora on the basis of many profiles,
we just sum the dissimilarities for the individual profiles. In other words, given a set of
profiles L1 to Lm , then the global dissimilarity D between two subcorpora V1 and VL2
on the basis of L1 up to Lm can be calculated as:
DCB (V1, V2) =
m
(L −i (V1, V2)W (Li )) (4)
i =1
The W in the formula is a weighting factor. We use weights to ensure that concepts
which have a relatively higher frequency (summed over the size of the two subcorpora
that are being compared) 3 also have a greater impact on the distance measurement. In
other words, in the case of a weighted calculation, concepts that are more common in
everyday life and language are treated as more important. Applying this to the fictional
example from Table 1, we can calculate the W per concept by dividing the sum of the
absolute frequencies of all variants for one concept by the sum of simply all variations.
For SUBTERRANEAN PUBLIC TRANSPORT this equals to (70+10+20+50)/(70+10+20+50+
50 + 40 + 30 + 35) = 0.492. There is no need to calculate the W for SMALL INSTRUMENT
PLAYED WITH A BOW as its distance is already set to zero. Filling out equation 4, we find
that the distance between British English and American English aggregated over both
concepts is (0.589 × 0.492) + 0 = 0.29.
Now, we put text categorization in contrast with the profile-based approach, which
incorporates probabilistic information of word choice. In text categorization, noncategorical
(probabilistic) word choice is well accounted for (unlike dialectometric ap-
2 If the frequency of the profile was lower than 30 in the two varieties that are being compared, that
profile was excluded from the comparison.
3 The size of the two subcorpora is not the actual amount of words in the two subcorpora, but the sum
of all profiles in these two subcorpora with a frequency higher than 30.