Class: Klue::Langcraft::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/klue/langcraft/tokenizer-old-needs-revisit/tokenizer.rb

Overview

Tokenizer class

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(code) ⇒ Tokenizer

Returns a new instance of Tokenizer.



9
10
11
12
 
# File 'lib/klue/langcraft/tokenizer-old-needs-revisit/tokenizer.rb', line 9

def initialize(code)
  @code = code
  @tokens = []
end

Instance Attribute Details

#tokens ⇒ Object (readonly)

Returns the value of attribute tokens.



7
8
9
 
# File 'lib/klue/langcraft/tokenizer-old-needs-revisit/tokenizer.rb', line 7

def tokens
  @tokens
end

Instance Method Details

#tokenize ⇒ Object 



14
15
16
17
 
# File 'lib/klue/langcraft/tokenizer-old-needs-revisit/tokenizer.rb', line 14

def tokenize
  # Implement logic to convert code into tokens
  # Handle strings, symbols, keywords, and delimiters
end