Class: Ritex::Lexer

Inherits:
Object
  • Object
show all
Defined in:
lib/ritex/lexer.rb

Overview

The lexer splits an input stream into tokens. These are handed to the parser. Ritex::Parser takes care of setting up and configuring the lexer.

In order to support macros, the lexer maintains a stack of strings. Pushing a string onto the stack will cause #lex to yield tokens from that string, until it reaches the end, at which point it will discard the string and resume yielding tokens from the previous string.

To handle macros, the lexer is stateful. Normally it ignores all spacing. After hitting an ENV token it will start returning SPACE tokens for each space until it hits a ‘}’.

Constant Summary collapse

TOKENS =

passed as themselves

'+-\/\*|\.,;:<>=()#&\[\]^_!?~%\'{} '
OPERATOR_TOKENS =

things that can be 'd to become operators

' ,'
WORDS = 
%w(array arrayopts define left right rowopts cellopts colalign rowalign align padding equalcols equalrows rowlines collines
frame rowspan colspan)
WORDS_SEARCH =

passed as special tokens

WORDS.map { |w| [/\A\\(#{Regexp.escape w})\b/, w.upcase.intern] }

Instance Method Summary collapse

Constructor Details

#initialize(parser, s = nil) ⇒ Lexer

s is an initial string to push on the stack, or nil.



34
35
36
37
38
 
# File 'lib/ritex/lexer.rb', line 34

def initialize parser, s = nil
  @parser = parser
  @s = []
  push s unless s.nil?
end

Instance Method Details

#dlex ⇒ Object

For debugging purposes.



54
55
56
57
58
59
60
61
 
# File 'lib/ritex/lexer.rb', line 54

def dlex #:nodoc: 
  while true
    lex do |sym, val|
      puts "GOT: #{sym} => #{val.inspect}"
      return unless sym
    end
  end
end

#lex ⇒ Object

Yield token and value pairs from the string stack.



44
45
46
47
48
49
50
51
 
# File 'lib/ritex/lexer.rb', line 44

def lex #:yields: token, value
  ## actually this function does nothing right now except call
  ## lex_inner. if we switch to more stateful tokenization this
  ## might do something more.
  lex_inner do |sym, val|
    yield [sym, val]
  end
end

#push(s) ⇒ Object

push an additional string on to the stack.



41
 
# File 'lib/ritex/lexer.rb', line 41

def push s; @s.unshift [s, 0]; end