Class: Omnium::Lexer::Core

Inherits:
Object
  • Object
show all
Includes:
Common, TokenHelper
Defined in:
lib/omnium/lexer/core.rb

Overview

The lexer returns tokens for a given text.

Defined Under Namespace

Classes: LexerError

Constant Summary collapse

WHITESPACE = 
' '
COMMENT = 
'#'
NEWLINE = 
"\n"
DECIMAL_POINT = 
'.'
INTEGER = 
/[0-9]/.freeze
ALPHA = 
/[a-zA-Z]/.freeze
IDENTIFIER = 
/[a-zA-Z0-9_]/.freeze

Constants included from Common

Common::NIL_VALUE_TOKENS, Common::PARAMETERISED_TOKENS, Common::RESERVED_KEYWORDS, Common::TOKENS, Common::VALUE_BASED_TOKENS

Instance Method Summary collapse

Methods included from TokenHelper

define_new_token_method

Methods included from Common

define_token_predicate_method, define_token_type_method, token_entity

Constructor Details

#initialize(text) ⇒ Core

Returns a new instance of Core.



23
24
25
26
 
# File 'lib/omnium/lexer/core.rb', line 23

def initialize(text)
  @text = text
  @pointer = 0
end

Instance Method Details

#next_token ⇒ Object 



28
29
30
31
32
33
34
35
 
# File 'lib/omnium/lexer/core.rb', line 28

def next_token
  ignore
  return new_eof_token if eos?

  (@pointer...@text.length).each do
    return tokenise
  end
end