PDF (Online Text) - EURAC

2.1 Unicode and Characters: Uppercasing, Lowercasing and Sorting
Contrary to a common sense understanding of Unicode, the conceptual design of
Unicode avoids the notion of character, since this is a language-specific notion, and
languages are not covered by Unicode. Unicode refers instead to code elements (),
which frequently coincide with characters, but also contain combining characters
such as diacritics. Characters and code elements further differ, if ligatures (Dutch ‘ij’,
Spanish ‘ll’, ‘ch’, Belorussian Lacinka ‘dz’) are to be treated as one character in a
language. Uppercasing of ligatures is thus essentially undefined, and will produce from
‘xy’ uniformly either ‘Xy’ or ‘XY’, without knowing the requirements of the writing
system. It is thus obvious that specifying the character set of writing systems and
describing the mapping between the characters (e.g., for uppercasing and lowercasing)
is one principle task in XNLRDF, just as lowercasing, for example, is an important step
in the normalisation of a string (e.g., for a lexicon lookup or information retrieval).
Similarly, the sorting of characters, the second operation defined in Unicode (e.g.,
for the purpose of presenting dictionary entries or creating indices), depends on the
writing system, and can only be approximately defined on the basis of the script.
Thus, Unicode might successfully sort ‘a’ before ‘b’, but already the position of ‘á’
after ‘a’ or after ‘z’ is specific to each writing system. Another example is the Spanish
‘ll.’ Although it is no longer considered a character, it maintains its specific position
between ‘l’ and ‘m’ in a sorted list. Thus, sorting requires basic writing system-specific
information, which XNLRDF sets out to describe. What this example also shows is
that the definition of collating sequences for the characters of a writing system is
independent from the status of the character (base character, composed character,
contracted character, contracted non-character, context-sensitive character, foreign
character, swap character, etc.).
2.2 Linguistic Information: What Else?
The operations covered by Unicode are limited, and most NLP-applications would
require much more linguistic knowledge when processing documents in potentially
unknown writing systems. First, an application might need to identify the encoding
(e.g. KOI-R), the script (Cyrillic), the language (Russian), the standard (civil script),
and orthography (after 1917) of a document. Part of this information might be drawn
from the metadata available in the document, from the Unicode range, or the URL
of a document (in our example, http://xyz.xyz.ru), but filling in the remaining gaps,
(e.g., mapping from the encoding KOI-R to the language Russian, from the language
to potential scripts, or from a script to a language) requires background information
about the legacy encodings and writing systems. This background information is
193