Class: SrlRuby::Tokenizer
- Inherits:
-
Object
- Object
- SrlRuby::Tokenizer
- Defined in:
- lib/srl_ruby/tokenizer.rb
Overview
A tokenizer for the Simple Regex Language. Responsibility: break input SRL into a sequence of token objects. The tokenizer should recognize: Keywords: as, capture, letter Integer literals including single digit String literals (quote delimited) Single character literal Delimiters: parentheses '(' and ')' Separators: comma (optional)
Defined Under Namespace
Classes: ScanError
Constant Summary collapse
- @@lexeme2name =
attr_reader(:column)
{ '(' => 'LPAREN', ')' => 'RPAREN', ',' => 'COMMA' }.freeze
- @@keywords =
Here are all the SRL keywords (in uppercase)
%w[ ALL ALREADY AND ANY ANYTHING AS AT BACKSLASH BEGIN BETWEEN BY CAPTURE CARRIAGE CASE CHARACTER DIGIT EITHER END EXACTLY FOLLOWED FROM HAD IF INSENSITIVE LAZY LEAST LETTER LINE LITERALLY MORE MULTI MUST NEVER NEW NO NONE NOT NUMBER OF ONCE ONE OPTIONAL OR RAW RETURN STARTS TAB TIMES TO TWICE UNTIL UPPERCASE VERTICAL WHITESPACE WITH WORD ].map { |x| [x, x] } .to_h
Instance Attribute Summary collapse
-
#line_start ⇒ Object
readonly
Returns the value of attribute line_start.
-
#lineno ⇒ Object
readonly
Returns the value of attribute lineno.
-
#scanner ⇒ Object
readonly
Returns the value of attribute scanner.
Instance Method Summary collapse
-
#initialize(source) ⇒ Tokenizer
constructor
Constructor.
- #tokens ⇒ Object
Constructor Details
#initialize(source) ⇒ Tokenizer
Constructor. Initialize a tokenizer for SRL.
93 94 95 96 97 |
# File 'lib/srl_ruby/tokenizer.rb', line 93 def initialize(source) @scanner = StringScanner.new(source) @lineno = 1 @line_start = 0 end |
Instance Attribute Details
#line_start ⇒ Object (readonly)
Returns the value of attribute line_start.
20 21 22 |
# File 'lib/srl_ruby/tokenizer.rb', line 20 def line_start @line_start end |
#lineno ⇒ Object (readonly)
Returns the value of attribute lineno.
19 20 21 |
# File 'lib/srl_ruby/tokenizer.rb', line 19 def lineno @lineno end |
#scanner ⇒ Object (readonly)
Returns the value of attribute scanner.
18 19 20 |
# File 'lib/srl_ruby/tokenizer.rb', line 18 def scanner @scanner end |
Instance Method Details
#tokens ⇒ Object
99 100 101 102 103 104 105 106 107 |
# File 'lib/srl_ruby/tokenizer.rb', line 99 def tokens tok_sequence = [] until @scanner.eos? token = _next_token tok_sequence << token unless token.nil? end return tok_sequence end |