Class: Loxxy::FrontEnd::Scanner

Inherits:
Object
  • Object
show all
Defined in:
lib/loxxy/front_end/scanner.rb

Overview

A scanner (tokenizer) for the Lox language. Reference material: https://craftinginterpreters.com/the-lox-language.html Section 4.2.1 Token types Appendix A1.2 Lexical Grammar Responsibility: break input into a sequence of token objects. The tokenizer should recognize: Identifiers, Number literals including single digit String literals (quote delimited) Delimiters: e.g. parentheses '(', ')' Separators: e.g. comma

Constant Summary collapse

PATT_BLOCK_COMMENT_BEGIN = 
/\/\*/.freeze
PATT_BLOCK_COMMENT_END = 
/\*\//.freeze
PATT_COMPARISON = 
/[!=><]=?/.freeze
PATT_IDENTIFIER = 
/[a-zA-Z_][a-zA-Z_0-9]*/.freeze
PATT_LINE_COMMENT = 
/\/\/[^\r\n]*/.freeze
PATT_NEWLINE = 
/(?:\r\n)|\r|\n/.freeze
PATT_NUMBER = 
/\d+(?:\.\d+)?/.freeze
PATT_WHITESPACE = 
/[ \t\f]+/.freeze
Lexeme2name =

One or two special character tokens. These are enumerated in section 4.2.1 Token type

{
  '(' => 'LEFT_PAREN',
  ')' => 'RIGHT_PAREN',
  '{' => 'LEFT_BRACE',
  '}' => 'RIGHT_BRACE',
  ',' => 'COMMA',
  '.' => 'DOT',
  '-' =>  'MINUS',
  '+' => 'PLUS',
  ';' => 'SEMICOLON',
  '/' => 'SLASH',
  '*' => 'STAR',
  '!' => 'BANG',
  '!=' => 'BANG_EQUAL',
  '=' => 'EQUAL',
  '==' => 'EQUAL_EQUAL',
  '>' => 'GREATER',
  '>=' => 'GREATER_EQUAL',
  '<' => 'LESS',
  '<=' => 'LESS_EQUAL'
}.freeze
@@keywords =

Here are all the implemented Lox keywords These are enumerated in section 4.2.1 Token type

%w[
  and class else false fun for if nil or
  print return super this true var while
].map { |x| [x, x] }.to_h
@@escape_chars =

Single character that have a special meaning when escaped

Returns:

  • ({Char => String}) 
{
  ?a => "\a",
  ?b => "\b",
  ?e => "\e",
  ?f => "\f",
  ?n => "\n",
  ?r => "\r",
  ?s => "\s",
  ?t => "\t",
  ?v => "\v"
}.freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source = nil) ⇒ Scanner

Constructor. Initialize a tokenizer for Lox input.

Parameters:

  • source (String) (defaults to: nil) —

    Lox text to tokenize.



88
89
90
91
92
 
# File 'lib/loxxy/front_end/scanner.rb', line 88

def initialize(source = nil)
  reset
  input = source || ''
  @scanner = StringScanner.new(input)
end

Instance Attribute Details

#line_start ⇒ Integer (readonly)

Returns Position of last start of line in the input.

Returns:

  • (Integer) —

    Position of last start of line in the input



39
40
41
 
# File 'lib/loxxy/front_end/scanner.rb', line 39

def line_start
  @line_start
end

#lineno ⇒ Integer (readonly)

Returns The current line number.

Returns:

  • (Integer) —

    The current line number



36
37
38
 
# File 'lib/loxxy/front_end/scanner.rb', line 36

def lineno
  @lineno
end

#scanner ⇒ StringScanner (readonly)

Returns Low-level input scanner.

Returns:

  • (StringScanner) —

    Low-level input scanner



33
34
35
 
# File 'lib/loxxy/front_end/scanner.rb', line 33

def scanner
  @scanner
end

Instance Method Details

#start_with(source) ⇒ Object

Reset the tokenizer and make the given text, the current input.

Parameters:

  • source (String) —

    Lox text to tokenize.



96
97
98
99
 
# File 'lib/loxxy/front_end/scanner.rb', line 96

def start_with(source)
  reset
  @scanner.string = source
end

#tokens ⇒ Array<Rley::Lexical::Token>

Scan the source and return an array of tokens.

Returns:

  • (Array<Rley::Lexical::Token>) —

    | Returns a sequence of tokens



103
104
105
106
107
108
109
110
111
112
 
# File 'lib/loxxy/front_end/scanner.rb', line 103

def tokens
  tok_sequence = []
  until @scanner.eos?
    token = _next_token
    tok_sequence << token unless token.nil?
  end
  tok_sequence << build_token('EOF', nil)

  tok_sequence
end