Class: Cambridge::Tokenizer

Inherits:

Object

• Object
• Cambridge::Tokenizer

Defined in:

lib/cambridge/tokenizer.rb

Instance Method Summary

• #initialize(source) ⇒ Tokenizer constructor

  A new instance of Tokenizer.

• #scanner ⇒ Object
• #to_ruby ⇒ Object
• #tokens ⇒ Object

Constructor Details

#initialize(source) ⇒ Tokenizer

Returns a new instance of Tokenizer.

# File 'lib/cambridge/tokenizer.rb', line 3

def initialize(source)
  @source = source
end

Instance Method Details

#scanner ⇒ Object

# File 'lib/cambridge/tokenizer.rb', line 15

def scanner
  @scanner ||= Scanner.new
end

#to_ruby ⇒ Object

# File 'lib/cambridge/tokenizer.rb', line 7

def to_ruby
  tokens.map(&:to_ruby).join("\n")
end

#tokens ⇒ Object

# File 'lib/cambridge/tokenizer.rb', line 11

def tokens
  @tokens ||= scanner.parse @source
end