Class: BaseTokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/support/base_tokenizer.rb

Overview

Simplistic tokenizer used mostly for testing purposes

Defined Under Namespace

Classes: ScanError

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source) ⇒ BaseTokenizer

Constructor. Initialize a tokenizer.

Parameters:

  • source (String) —

    Skeem text to tokenize.



21
22
23
24
 
# File 'lib/support/base_tokenizer.rb', line 21

def initialize(source)
  @scanner = StringScanner.new('')
  restart(source)
end

Instance Attribute Details

#line_start ⇒ Integer (readonly)

Returns position of start of current line in source text.

Returns:

  • (Integer) —

    position of start of current line in source text



15
16
17
 
# File 'lib/support/base_tokenizer.rb', line 15

def line_start
  @line_start
end

#lineno ⇒ Integer (readonly)

Returns current line number.

Returns:

  • (Integer) —

    current line number



12
13
14
 
# File 'lib/support/base_tokenizer.rb', line 12

def lineno
  @lineno
end

#scanner ⇒ StringScanner (readonly)

Returns:

  • (StringScanner) 


9
10
11
 
# File 'lib/support/base_tokenizer.rb', line 9

def scanner
  @scanner
end

Instance Method Details

#restart(source) ⇒ Object

Parameters:

  • source (String) —

    input text to tokenize.



27
28
29
30
31
 
# File 'lib/support/base_tokenizer.rb', line 27

def restart(source)
  @scanner.string = source
  @lineno = 1
  @line_start = 0
end

#tokens ⇒ Array<SkmToken>

Returns | Returns a sequence of tokens.

Returns:

  • (Array<SkmToken>) —

    | Returns a sequence of tokens



34
35
36
37
38
39
40
41
42
 
# File 'lib/support/base_tokenizer.rb', line 34

def tokens
  tok_sequence = []
  until @scanner.eos?
    token = _next_token
    tok_sequence << token unless token.nil?
  end

  return tok_sequence
end