The Corpus Thread - Det Danske Sprog- og Litteraturselskab

More documents

Recommendations

Info

4.2. Formatting text 88 4.2.3 Good: Formatting according to the requirements 4.2.3.1 From source version to base format Two of the consequences emerging from the requirements are that the text has to be mechanically segmented into basic tokens 3 and that it must be possible to unequivocally refer to these tokens, cf. Section 4.1.3. Mechanical text segmentation, or pre-tokenization, is carried out by certain textual surface items, i.e. characters, only. For segmentation purposes characters fall into three categories: ⊲ Letters and numbers, i.e. alpha-numeric characters ⊲ Whitespace characters ⊲ Punctuation characters Continuous sequences of alpha-numeric characters are considered ‘words’ even if these segments are not necessarily in accordance with a linguistic definition of a word. Linguistic interpretations are deliberately avoided at this point. ‘Words’ are put into elements. Whitespace and punctuation is put into elements – character by character – that can be of type “s” (space) or “p” (punctuation). The nonobligatory subtype attribute may specify some other characteristics of the character in question, e.g. the length of a whitespace. Specifications of the possible inventory of the subtype attribute are not given before it turns out that this attribute is really needed. Standard space characters (ASCII 32) are not explicitly denoted in the elements (i.e. they remain empty) whereas other whitespace characters such as tabs (coded as ) can be given in the element. and elements are the smallest segments (i.e. basic tokens) of a text. Each of them carries a unique xml:id that allows referencing to it from elsewhere. 4 The source example given in Section 4.2.1 would look like this after segmentation: To kendte russiske historikere Andronik 3 The segmentation process is called pre-tokenization, cf. Chapter 5. 4 Assigning IDs requires some sort of control that every ID is unique.
4.2. Formatting text 89 Mirganjan og Igor Klamkin tror ikke , at Rusland kan udvikles uden en " jernnæve " . This formatted version of the source text is called the text’s base format. The base format is the standard input format for all tools like tokenizers, sentence splitters, lemmatizers, and taggers of all kinds, see the motivation for a fixed text format in Section 4.1.1. As can be seen, markup above and level that is already present in the source version text, may be kept as long as the source version complies with the TEI specifications. In this case, the and tags were kept; tags may carry an xml:lang attribute that indicates the language of the paragraph by using a value from the languageId value set described in 3.3.2. Even though tags other than,,, and may be used as long as they are TEI-compliant, 5 this type of markup should be avoided and added as span groups instead, see the following section. 5 Among such elements is (line break) whereas the element, which was introduced by some other WP 2 projects, is not allowed in the body of a text. See also Section 4.2.3.4 on further examples.
Page 1 and 2:
The Corpus Thread “Reference corp
Page 3 and 4:
Chapter 1 Aim and concepts Sketchin
Page 5 and 6:
1.1. Aim of the project 5 Metadata
Page 7 and 8:
1.2. Project tasks and documentatio
Page 9 and 10:
Page 11 and 12:
Page 13 and 14:
1.3. Text collection, text bank, co
Page 15 and 16:
Chapter 2 The text bank A platform
Page 17 and 18:
2.2. Implementation 17 and metadata
Page 19 and 20:
Chapter 3 Text metadata What the he
Page 21 and 22:
3.1. Concepts 21 Section 3.3 starts
Page 23 and 24:
3.2. Header structure 23 (file des
Page 25 and 26:
3.2. Header structure 25 great verb
Page 27 and 28:
3.2. Header structure 27 The follow
Page 29 and 30:
3.2. Header structure 29 deprecated
Page 31 and 32:
3.2. Header structure 31 the text i
Page 33 and 34:
3.2. Header structure 33 Another el
Page 35 and 36:
3.2. Header structure 35 appDesc
Page 37 and 38: 3.2. Header structure 37 informal p
Page 39 and 40: 3.2. Header structure 39 3.2.3.4 Te
Page 41 and 42: 3.3. Filling in the header 41 revi
Page 43 and 44: 3.3. Filling in the header 43 samp
Page 45 and 46: 3.3. Filling in the header 45 and 9
Page 47 and 48: 3.3. Filling in the header 47 descr
Page 49 and 50: 3.3. Filling in the header 49 ⊲ a
Page 51 and 52: 3.3. Filling in the header 51 Legal
Page 53 and 54: 3.3. Filling in the header 53 ⊲ a
Page 61 and 62: 3.3. Filling in the header 61 Value
Page 65 and 66: 3.3. Filling in the header 65 ⊲ p
Page 67 and 68: 3.3. Filling in the header 67 Prope
Page 83 and 84: 3.3. Filling in the header 83 3.3.3
Page 85 and 86: 4.1. Basic considerations 85 Outlin
Page 87: 4.2. Formatting text 87 russiske hi
Page 91 and 92: 4.2. Formatting text 91 udvikle ude
Page 93 and 94: 4.2. Formatting text 93 4.2.3.4 Add
Page 95 and 96: 4.2. Formatting text 95 i dag . T
Page 97 and 98: Part III Collecting 97
Page 99 and 100: 5.1. Implementation 99 5.5.2 Implem
Page 101 and 102: 5.1. Implementation 101 [request c
Page 103 and 104: 5.2. Header constructor:make-header
Page 115 and 116: 5.3. Pre-tokenizer:pretokenize 115
Page 117 and 118: 5.3. Pre-tokenizer:pretokenize 117
Page 119 and 120: 5.4. Text id registry:register-text
Page 121 and 122: 5.4. Text id registry:register-text
Page 123 and 124: 5.5. id dispatcher:make-id 123 0 11
Page 125 and 126: Part IV Markup 125
Page 127 and 128: 6.1. Requirements 127 Outline of th
Page 129 and 130: 6.2. Survey 129 1. IMS’s TreeTagg
Page 131 and 132: 6.2. Survey 131 Code: Java. Assessm
Page 133 and 134: 6.2. Survey 133 9. alias-i’s Ling
Page 135 and 136: 6.3. Case study 135 lemmatization c
Page 137 and 138: 6.4. Further reading 137 disambigua
Page 139 and 140:
7.1. Modifications of the PAROLE Co
Page 141 and 142:
7.2. The ePOS tag set for Danish 14
Page 143 and 144:
Page 145 and 146:
Page 147 and 148:
Page 149 and 150:
Page 151 and 152:
Page 153 and 154:
Chapter 8 The full-form lexicon Sam
Page 155 and 156:
8.1. Enhancing existing material 15
Page 157 and 158:
8.1. Enhancing existing material 15
Page 159 and 160:
Part V Deployment 159
Page 161 and 162:
9.2. Text material 161 Type Source
Page 163 and 164:
10.2. Funktionaliteter 163 2. The u
Page 165 and 166:
10.2. Funktionaliteter 165 fortolke
Page 167 and 168:
10.2. Funktionaliteter 167 teknolog
Page 169 and 170:
10.2. Funktionaliteter 169 Implemen
Page 171 and 172:
10.2. Funktionaliteter 171 forbinde
Page 173 and 174:
10.2. Funktionaliteter 173 10.2.9 K
Page 175 and 176:
Bibliography Andersen, M. S., Asmus
show all

The Corpus Thread - Det Danske Sprog- og Litteraturselskab

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?