Class: Rley::Lexical::Token

Inherits:
Object
  • Object
show all
Defined in:
lib/rley/lexical/token.rb

Overview

In Rley, a (lexical) token is an object created by a lexer (tokenizer) and passed to the parser. Such token an object is created when a lexer detects that a sequence of characters(a lexeme) from the input stream is an instance of a terminal grammar symbol. Say, that in a particular language, the lexeme 'foo' is an occurrence of the terminal symbol IDENTIFIER. Then the lexer will return a Token object that states the fact that 'foo' is indeed an IDENTIFIER. Basically, a Token is a pair (lexeme, terminal): it asserts that a given lexeme is an instance of given terminal symbol.

Direct Known Subclasses

Literal

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(theLexeme, aTerminal, aPosition = nil) ⇒ Token

Constructor.

Parameters:

  • theLexeme (String) —

    the lexeme (= piece of text from input)

  • aTerminal (Syntax::Terminal, String) —

    The terminal symbol corresponding to the lexeme.

  • aPosition (Rley::Lexical::Position) (defaults to: nil) —

    position of the token in source file



41
42
43
44
45
46
47
 
# File 'lib/rley/lexical/token.rb', line 41

def initialize(theLexeme, aTerminal, aPosition = nil)
  raise 'Internal error: nil terminal symbol detected' if aTerminal.nil?

  @lexeme = theLexeme
  @terminal = aTerminal
  @position = aPosition
end

Instance Attribute Details

#lexeme ⇒ String (readonly)

The sequence of character(s) from the input stream that is an occurrence of the related terminal symbol.

Returns:

  • (String) —

    Input substring that is an instance of the terminal.



28
29
30
 
# File 'lib/rley/lexical/token.rb', line 28

def lexeme
  @lexeme
end

#position ⇒ Position

Returns The position -in "editor" coordinates- of the lexeme in the source file.

Returns:

  • (Position) —

    The position -in "editor" coordinates- of the lexeme in the source file.



34
35
36
 
# File 'lib/rley/lexical/token.rb', line 34

def position
  @position
end

#terminal ⇒ String (readonly)

Returns The name of terminal symbol matching the lexeme.

Returns:

  • (String) —

    The name of terminal symbol matching the lexeme.



31
32
33
 
# File 'lib/rley/lexical/token.rb', line 31

def terminal
  @terminal
end