Encyclopedia of Computer Science and Technology

parsing 361sis.) A token is normally defined as a series of one or morecharacters separated by “whitespace” (blanks, carriagereturns, and so on). A token is thus analogous to a word inEnglish.The series of tokens is then sent to the parser. The parser’sjob is to identify the significance of each token and togroup the tokens into properly formed statements. Generally,the parser first checks the tokens for keywords—wordssuch as “if” or “loop” that have a special meaning in aparticular programming language. (In the BASIC example,PRINT is a keyword: In many other languages such functionsare external rather than being part of the languageitself.) As keywords (and punctuation symbols such as thesemicolon used at the end of statements in C and Pascal)are identified, the parser uses a set of rules to determine theoverall structure of the statement. For example, a languagemight define an if statement as follows:If then else This means that when the parser encounters an “if” it willexpect to find between that word and “then” an expressionthat can be tested for being true or false (see Booleanoperators). Following “then,” it will expect to find a completestatement. If it finds the optional keyword “else,” thatword will be followed by an alternative statement. Thus inthe statementIf Total > Limit Print “Overflow” else PrintTotalThe elements would be broken down as follows:IfTotal > LimitPrint“Overflow”elsePrintTotalkeywordBoolean expressionkeywordString literal (charactersto be printed)keywordkeywordvariableWhen writing a parser, the programmer depends ona precise and exhaustive description of the possible legalconstructs in the language (see also Backus-Naur form).In turn, these rules are turned into procedures by whichthe parser can construct a representation of the relationshipsbetween the tokens. This representation is often representedas an upside-down tree, rather like the sentencediagrams used in English class.In general form, an expression, for example, can be diagrammedas consisting of one or more terms (variables,constants, or literal values) or other expressions separatedby operators.A parse tree for the statement A = B + C × D. Notice how theexpression on the right-hand side of the equals (assignment) sign iseventually parsed into the component identifiers and operators.Notice that these diagrams are often recursive. That is,the definition of an expression can include expressions.The number of levels that can be “nested” is usually limitedby the compiler if not by the definition of the language.The underlying rules must be constructed in such a waythat they are not ambiguous. That is, any given string oftokens must result in one, and only one parse tree.Once the elements have been extracted and classified,a compiler must also analyze the nonkeyword tokens tomake sure they represent valid data types, any variableshave been previously defined, and the language’s namingconventions have been followed (see compiler).Fortunately, people who are designing command processors,scripting languages, and other applications requiringparsers need not work from scratch. Tools such as YACC (agrammar definition compiler) and BISON and ANTLR (parsergenerators) are available for UNIX and other platforms.Further ReadingAho, Alfred V., et al. Compilers: Principles, Techniques, & Tools. 2nded. Boston: Pearson/Addison-Wesley, 2007.Bowen, Jonathan P., and Peter T. Breuer. “Razor: The Cutting Edgeof Parser Technology.” Oxford University Computing Laboratory.Available online. URL: http://www.jpbowen.com/pub/toulouse92.pdf. Accessed August 17, 2007.Donnelly, Charles, and Richard M. Stallman. Bison Manual: Usingthe YACC-Compatible Parser Generator. Boston, Mass.: GNUPress, 2003.