Topics in Language Resources for Translation ... - ymerleksi - home

Chapter 11. Tagging and tracing Program Integrated Information 1895.4 ID tag generation programWe developed a Perl program to create a separate set of files with ID tags for eachset of PII files. Perl was selected in 2003 because of the maturity of its regularexpressions and its Unicode support. As of 2007, other programming languagescould be used to prepare such a program. We do not change the original PII files,but only create additional files to replace the original PII files. The additional fileshave no effect on the tested program and its PII files. These files are only used forthe TVT tests. The addid.pl supports the CATNls files and the Java properties files.The addid.pl program handles one directory of files at a time. A typical directoryis a set of English PII files or Japanese PII files. All of the CATIA PII files for alanguage are located in one directory. The addid.pl program does not support thenested directory structure of PII files, so the program must be run separately foreach subdirectory. Care is needed if the PII files have a nested directory structure.The addid.pl program has the following parameters. The default values areshown in the parentheses.-l Directory name of the PII files (Japanese)-f A filter for the file names that are processed (*)-p Prefix (Null)-e End of the ID tag (no)If the -e option (Position of the ID tag) is set to “yes”, then the ID tag is put atthe end of each PII string instead of at the beginning (though this option is rarelyused).IfweuseJ7dasaprefix,wewillcopytheJapanesedirectoryofCATIAPIIfilesinto the new J7d directory for the generated files. The command “addid.pl -l J7d-f *.CATNls -p J7d” generates the following output.J7dWithID: An output directory of files with ID tagsThis directory contains all of the copied files with the PII strings modified byadding the ID tags.J7dNumList.txt: A map file from the ID file numbers to the PII file names.This contains a CSV text mapping of the ID file numbers to the PII file names.J7dTextList.txt:AtextfilethathasallofthePIIThis text file contains all of the sets of PII (pairs of a key and a string) for all of thefiles in the J7d directory. We call this comprehensive index file the “Basic Form” inSection 5.5.We prepare a J7dIU directory that contains the IU directories copied andseparated for TM (TranslationManager) use. If addIUName.pl is executed with