FoCL, Chapter 15: Corpus analysis) 251

15. Corpus analysis

15.1 Implementation and application of grammar systems

15.1.1 Parts of a grammar system

• Formal algorithm

• Linguistic method 15.1.2 Options for grammar system of word form recognition

• Formal algorithm: C- (Section 7.4), PS- (Section 8.1), or LA-grammar (Section 10.2).

• Linguistic method: Word form, morpheme, or allomorph method (cf. Section 13.5).

FoCL, Chapter 15: Corpus analysis) 252

15.1.3 Minimal standard of well-defined grammar systems A grammar system is well-defined only if it simultaneously allows

1. different applications in a given implementation, and

2. different implementations in a given application. 15.1.4 Modularity of a grammar system

grammar system different applications

different implementations 15.1.5 Different implementations of LA-morphology

1988 in LISP (Hausser & Todd Kaufmann) 1990 in C (Hausser & Carolyn Ellis) 1992 in C, ‘LAMA’ (Norbert Bröker) 1994 in C, ‘LAP’ (Gerald Schüller) 1995 in C, ‘Malaga’ (Björn Beutel)

15.1.6 Structural principles common to different LA-Morph implementations

• Specification of the allo- (cf. 14.1.1) and the combi-rules (cf. 14.4.1) on the basis of patterns which are matched onto the input.

• Storage of the analyzed allomorphs in a trie structure and their left-associative lookup with parallel pursuit of alternative hypotheses (cf. Section 14.3).

• Modular separation of motor, rule components, and lexicon, permitting a simple exchange of these parts, for example in the application of the system to new domains or languages.

• Use of the same motor and the same algorithm for the combi-rules of the morphological, syntactic, and semantic components during analysis.

• Use of the same rule components for analysis and generation in morphology, syntax, and semantics.

15.2 Subtheoretical variants

15.2.1 Combinatorics of the German determiner der

.^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}  ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}  ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}der (E′ MN′ S3)  schöne (E)  Baum ⇒ (M-G)  der schöne Baum (S3) der^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}  schönen  der schönen Bäume Frau ⇒ (EN′ F′ G) (EN′ F′ D)  (EN) (EN)  (F) (F)  der schönen Frau (G) (D) der  schönen (EN)  (G) Bäume ⇒ (P-D) (EN′ P-D′ G)

15.2.2 Agreement of adjective-ending with determiner

schöne  Baum  (cf. 15.2.1)^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}  (M-G) der  (S3) (ER′ MN′ S3)  Baum ⇒ ein  schöner  ein schöner Baum (ER)

15.2.3 Exhaustive versus distinctive categorization in deriving der schönen Frauen

der  ^{schönen}^{6}  ^{19 →} ·^{114}  ^{5}  ·  4 →^{20}^{+}  ^{1}^{=} ^{134}^{Frauen}  ^{number of input pairs}^{multiplication of exhaustive readings}^{number of input pairs}^{der}  ^{schönen} ·  ·^{+}  ^{1}^{=}^{Frauen}^{3}  1 →^{3}  ^{2}  1 →  ^{5}^{2}  ^{multiplication of distinctive readings}

FoCL, Chapter 15: Corpus analysis) 256

15.2.4 Representing lexical readings via different entries [der (E′ MN′ S3) DEF-ART] [der (EN′ F′ G&D) DEF-ART] [der (EN′ P-D′ G) DEF-ART]

15.2.5 Representing lexical readings via multicats [der ((E′ MN′ S3) (EN′ F′ G&D) (EN′ P-D′ G)) DEF-ART] 15.2.6 List-based matching (LAP)

.^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}  ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{input-output:}^{rule pattern:}  ^{(a}  ^{b}  ^{c}  ^{d)}^{(X}  ^{b}  ^{categorial operation}^{Y} ^{)} ss  nw^{(b)}^{(b) ⟹}  ^{(a}  ^{c}  ^{d)}^{(X}  ^{Y} ^{)} ss′

15.2.7 Feature-based matching (Malaga)

ss  nw  ss′
input-output:[\begin{matrix}mm1 = a \\ mm2 = b \\ mm3 = c \\ mm4 = d \end{matrix}] [mm5 = b]  [\begin{matrix}mm1 = a \\ mm3 = c \\ mm4 = d \end{matrix}]

rule pattern:[\begin{matrix}mm2 = b \\ X \end{matrix}] [mm5 = b] ⟹[X]

categorial operation

FoCL, Chapter 15: Corpus analysis) 258

15.3 Building corpora

15.3.1 Text genres of the Brown and the LOB corpus

Brown LOB

A Press: reportage  44  44
B Press: editorial  27  27
C Press: reviews  17  17
D Religion  17  17
E Skills, trade, and hobbies  36  38
F Popular lore  48  44
G Belle lettres, biography, essays  75  77

H Miscellaneous (government documents, foundation records, industry reports,

college catalogues, industry house organ)  30  38
J Learned and scientific writing  80  80
K General fiction  29  29
L Mystery and detective fiction  24  24
MScience fiction  6  6
N Adventure and western fiction  29  29
P Romance and love story  29  29
R Humour  9  9
Total  500  500

15.3.2 Kučera & Francis’ desiderata for the construction of corpora

1. Definite and specific delimitation of the language texts included, so that scholars using the Corpus may have a precise notion of the composition of the material.

2. Complete synchronicity; texts published in a single calendar year only are included.

3. A predetermined ratio of the various genres represented and a selection of individual samples through a random sampling procedure.

4. Accessibility of the Corpus to automatic retrieval of all information contained in it which can be formally identified.

5. An accurate and complete description of the basic statistical properties of the Corpus and of several subsets of the Corpus with the possibility of expanding such analysis to other sections or properties of the Corpus as may be required.

15.3.3 Difficulties with achieving a representative and balanced corpus

‘Genre’ is not a well-defined concept. Thus genres that have been distinguished so far have been identified on a purely intuitive basis. No empirical evidence has been provided for any of the genre distinctions that have been made.

N. Oostdijk 1988

FoCL, Chapter 15: Corpus analysis) 260

15.4 Distribution of word forms

15.4.1 Definition of rank The position of a word form in the frequency list 15.4.2 Definition of frequency class (F-class)

F-class =_{def} [frequency of types # number of types] There are much fewer F-classes in a corpus than ranks. In the BNC, for example, 655 270 ranks result in 5 301 F-classes. Thus, the number of the F-classes is only 0.8% of the number of ranks. Because of their comparatively small number the F-classes are well suited to bring the type-token correlation into focus.

FoCL, Chapter 15: Corpus analysis) 261

15.4.3 Type-token distribution in the BNC (surface-based)

0.000152 0.000152 0.000152 1 1 1 1 1 1 1 1 1 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 5293 5294 5295 5296 5297 5298 5299 5300  6577 7969 9973 12551 16701 24378 39691 85131 348145  59193 63752 69811 75306 83505 97512 119073 170262 348145  0.997620 1.208763 1.512736 1.903775 2.533260 3.697732 6.020456 12.912938 52.807732  0.065960 0.071040 0.077792 0.083915 0.093052 0.108660 0.132686 0.189727 0.387946 114731 108154 122700 132673 145224 161925 186303 225994 311125  114730 122699 132672 145223 161924 186302 225993 311124  0.000152 0.000152 0.000152 0.000152 1 1 1 1 4 5 17 38 517 1017 2171 3591 4536 5907 8332 10842 16012 44905  1017 2171  0.000152 3591  51 4536 5910 8336 10858 16049 45421  9 8 7 6 5 4 3 2 1 beginning 1 (the) 2 (of) 3 (and) 4 (to) 5 (a) 6 (in) 7 (is) 8 (that) 9 (was)  5776399 2789563 2421306 2332411 1957293 1746891 893368 891498 839967  0.000152 0.000152 0.000152 0.000152 0.000152  6.436776 3.108475 2.698118 2.599060 2.181057 1.946601 0.995501 0.993417 0.935995 (the first 9 F-classes) 9  19 648 696  0.001368 %  21.895 % sums F-class  tokens  types-%  tokens-% start r  end r  types end  (the last 9 F-classes) 5292  551 116  1 086 559  83.595012 % 1.210778 % sums  659269 middle 1000 2001 3000 3500 4000 4500 4750 5000 5250  9608 4560 2521 1857 5228 4005 9367 11438 26367  0.000607 0.000758 0.002579 0.005764 0.078420  0.010706 0.005081 0.002809 0.002069 0.005826 0.004463 0.010438 0.012746 0.029381 (9 samples)  tokens per type: 1307 801 551 301

15.4.4 Correlation of type and token frequency

Percentage of types^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0%Precentage of tokens

15.4.5 Semantic significance The higher the freqency, the lower the semantic significance. Examples: the, of, and, to, a, in, that, was The lower the freqency, the higher the semantic significance. Examples: audiophile, butternut, customhouse, dustheap 15.4.6 Hapaxlegomena Word forms in a corpus which occur only once.

\begin{matrix}c \\ ◯ \end{matrix}1999 Roland Hausser

FoCL, Chapter 15: Corpus analysis) 263

15.4.7 Zipf’s law

frequency · rank = constant 15.4.8 Illustration of Zipf’s law

3 251 ·  ^{=}^{2 870}  ^{9 330 370}^{word form}^{the}^{and}^{...}^{was}^{...}  ^{rank}^{1}^{2}^{9}^{holder}  ·  ^{7 559 703} ···  ^{frequency}^{5 776 399}^{2 789 563}^{839 967}  ^{=}^{=}^{=}^{=}  ^{constant}^{5 776 399}^{5 579 126}

15.5 Statistical tagging

15.5.1 Top of Brown corpus frequency list

69971-15-500 THE                      21341-15-500 IN
36411-15-500 OF                       10595-15-500 THAT
28852-15-500 AND                      10099-15-485 IS
26149-15-500 TO                        9816-15-466 WAS
23237-15-500 A                         9543-15-428 HE

The entry 9543-15-428 HE, for example, indicates that the word form HE occurs 9 543 times in the Brown corpus, in all 15 genres, and in 428 of the 500 sample texts.

15.5.2 Statistical tagging

is based on categorizing by hand – or half automatically with careful post-editing – a small part of the corpus, called the core corpus. The categories used for the classification are called tags or labels. After hand-tagging the core corpus, the probabilities of the transitions from one word form to the next are computed by means of Hidden Markov Models (HMMs).

15.5.3 Subset of the basic (C5) tagset

AJ0 Adjective (general or positive) (e.g. good, old, beautiful) CRD Cardinal number (e.g. one, 3, fifty-five, 3609) NN0 Common noun, neutral for number (e.g. aircraft, data, committee)

NN1 Singular common noun (e.g. pencil, goose, time, revelation) NN2 Plural common noun (e.g. pencils, geese, times, revelations) NP0 Proper noun (e.g. London, Michael, Mars, IBM)

UNC Unclassified items VVB The finite base form of lexical verbs (e.g. forget, send, live, return) VVD The past tense form of lexical verbs (e.g. forgot, sent, lived, returned)

VVG The -ing form of lexical verbs (e.g. forgetting, sending, living, returning)

VVI The infinitive form of lexical verbs (e.g. forget, send, live, return) VVN The past participle form of lexical verbs (e.g. forgotten, sent, lived, returned) VVZ The -s form of lexical verbs (e.g. forgets, sends, lives, returns)

15.5.4 Sample from the alphabetical word form list of the BNC

1 activ nn1-np0 1                8 activating aj0-nn1 6
1 activ np0 1                     47 activating aj0-vvg 22
2 activa nn1 1                   3 activating nn1-vvg 3
3 activa nn1-np0 1               14 activating np0 5
4 activa np0 2                   371 activating vvg 49
1 activatd nn1-vvb 1             538 activation nn1 93
21 activate np0 4                3 activation nn1-np0 3
62 activate vvb 42               2 activation-energy aj0 1
219 activate vvi 116             1 activation-inhibition aj0 1
140 activated aj0 48             1 activation-synthesis aj0 1
56 activated aj0-vvd 26         1 activation. nn0 1
52 activated aj0-vvn 34         1 activation/ unc 1
5 activated np0 3                282 activator nn1 30
85 activated vvd 56              6 activator nn1-np0 3
43 activated vvd-vvn 36         1 activator/ unc 1
312 activated vvn 144           1 activator/ unc 1
1 activatedness nn1 1           7 activator/tissue unc 1
88 activates vvz 60              61 activators nn2 18
5 activating aj0 5               1 activators np0 1

Each entry consists (i) of a number detailing the frequency of the tagged word form in the whole corpus, (ii) the surface of the word form, (iii) the label, and (iv) the number of texts in which the word form was found under the assigned label.

15.5.5 Error rates in statistical tagging

The error rate of CLAWS4 is quoted by Leech 1995 at 1.7%, which may seem very good. However, given that the last 1.2% of the low frequency tokens requires 83.6% of the types (cf. 15.4.4), an error rate of 1.7% may also represent a very bad result – namely that about 90% of the types are not analyzed or not analyzed correctly. This conclusion is born out by a closer inspection of sample 15.5.4.

15.5.6 Weaknesses of statistical tagging

1. The categorization is too unreliable to support rule-based syntactic parsing.

2. Word forms can be neither reduced to their base forms (lemmatization) nor segmented into their allomorphs or morphemes.

3. The overall frequency distribution analysis of a corpus is distorted by an artificial inflation of types (e.g., 37.5% in the BNC).

4. Even if the tagger is successfully improved as a whole, its results can never be more than probabilisticallybased conjectures.