sample

First we define the special keywords of our language in a constant. It will be usedlater on in the tokenizing process to disambiguate an identifier (method name,local variable, etc.) from a keyword.7 KEYWORDS = ["def", "class", "if", "true", "false", "nil"]89 def tokenize(code)10 code.chomp! # Remove extra line breaks11 tokens = [] # This will hold the generated tokens12We need to know how deep we are in the indentation so we keep track of thecurrent indentation level we are in, and previous ones in the stack so that when wededent, we can check if we’re on the correct level.16 current_indent = 0 # number of spaces in the last indent17 indent_stack = []18Here is how to implement a very simple scanner. Advance one character at thetime until you find something to parse. We’ll use regular expressions to scan fromthe current position (i) up to the end of the code.23 i = 0 # Current character position24 while i < code.size25 chunk = code[i..-1]26Each of the following if/elsifs will test the current code chunk with a regularexpression. The order is important as we want to match if as a keyword, and not amethod name, we’ll need to apply it first.