Module: Jaina::Parser::Tokenizer Private

Defined in:
lib/jaina/parser/tokenizer.rb

Overview

This module is part of a private API. You should avoid using this module if possible, as it may be removed or be changed in the future.

Since:

  • 0.1.0

Defined Under Namespace

Classes: Token, TokenBuilder

Constant Summary collapse

TOKEN_SCAN_PATTERN =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Returns:

  • (Regexp)

Since:

  • 0.1.0

/\(|\)|[\<\=\-\>\:\"\'\.\,\w\[\]]+/.freeze
TOKEN_SPLITTER =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Returns:

  • (String)

Since:

  • 0.1.0

' '

Class Method Summary collapse

Class Method Details

.join(tokens) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Parameters:

Returns:

  • (String)

Since:

  • 0.1.0



38
39
40
 
# File 'lib/jaina/parser/tokenizer.rb', line 38

def join(tokens)
  raw_join(tokens.map(&:raw_token))
end

.raw_join(raw_tokens) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Parameters:

  • raw_tokens (Array<String>)

Returns:

  • (String)

Since:

  • 0.4.0



47
48
49
 
# File 'lib/jaina/parser/tokenizer.rb', line 47

def raw_join(raw_tokens)
  raw_tokens.join(TOKEN_SPLITTER)
end

.tokenize(program) ⇒ Array<Jaina::Parser::Tokenizer::Token>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Parameters:

  • program (String)

Returns:

Since:

  • 0.1.0



27
28
29
30
31
 
# File 'lib/jaina/parser/tokenizer.rb', line 27

def tokenize(program)
  program.scan(TOKEN_SCAN_PATTERN).map do |raw_token|
    TokenBuilder.build(raw_token)
  end
end