Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
8.1.1 Corpus studies and first results
We first show how standard (freely distributed) gnu text utilities can be used to edit, sort and count things.
(These utilities are standardly provided in linux and unix implementations. In ms-windows systems, you can
get them with cygwin. In mac-osX systems, you can get them with fink.)
(5) Jane Austen’s Persuasion:
1%dir persu11.txt
460 -rw-r--r-- 1 es users 467137 Apr 30 18:00 persu11.txt
2%wc -l persu11.txt
8466 persu11.txt
3%wc -w persu11.txt
83309 persu11.txt
4%more persu11.txt
Persuasion by Jane Austen
Chapter 1
Sir Walter Elliot, of Kellynch Hall, in Somersetshire, was a man who,
for his own amusement, never took up any book but the Baronetage;
there he found occupation for an idle hour, and consolation in a
distressed one; there his faculties were roused into admiration and
respect, by contemplating the limited remnant of the earliest patents;
there any unwelcome sensations, arising from domestic affairs
changed naturally into pity and contempt as he turned over
the almost endless creations of the last century; and there,
if every other leaf were powerless, he could read his own history
with an interest which never failed. This was the page at which
the favorite volume always opened:
"ELLIOT OF KELLYNCH HALL.
"Walter Elliot, born March 1, 1760, married, July 15, 1784, Elizabeth,
daughter of James Stevenson, Esq. of South Park, in the county of
Gloucester, by which lady (who died 1800) he has issue Elizabeth,
q
In 1%, we get the number of bytes in the file.
In 2%, we get the number of lines in the file.
In 3%, we get the number of “words” – character sequences surrounded by spaces or newlines.
(6) HereweusetheGnuversionoftr. Check your man pages if your tr does not work the same way.
4%tr ’ ’ ’\012’ < persu11.txt > persu11.wds
6%more persu11.wds
Persuasion
by
Jane
Austen
Chapter
1
Sir
Walter
Elliot,
of
Kellynch
Hall,
in
Somersetshire,
was
a
man
who,
for
(7) 7%tr -sc ’A-Za-z’ ’\012’ < persu11.txt > persu11.wds
8%more persu11.wds
Persuasion
by
Jane
Austen
Chapter
Sir
Walter
Elliot
of
Kellynch
Hall
in
Somersetshire
was
a
man
118