Class: Keisan::Tokenizer
- Inherits:
-
Object
- Object
- Keisan::Tokenizer
- Defined in:
- lib/keisan/tokenizer.rb
Constant Summary collapse
- TOKEN_CLASSES =
[ Tokens::Null, Tokens::Boolean, Tokens::Word, Tokens::Number, Tokens::Assignment, Tokens::BitwiseShift, Tokens::LogicalOperator, Tokens::ArithmeticOperator, Tokens::BitwiseOperator, Tokens::Comma, Tokens::Colon, Tokens::Dot, Tokens::LineSeparator, Tokens::Unknown ]
- TOKEN_REGEX =
Regexp::new( TOKEN_CLASSES.map(&:regex).join("|") )
Instance Attribute Summary collapse
-
#expression ⇒ Object
readonly
Returns the value of attribute expression.
-
#tokens ⇒ Object
readonly
Returns the value of attribute tokens.
Instance Method Summary collapse
-
#initialize(expression) ⇒ Tokenizer
constructor
A new instance of Tokenizer.
Constructor Details
#initialize(expression) ⇒ Tokenizer
Returns a new instance of Tokenizer.
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
# File 'lib/keisan/tokenizer.rb', line 26 def initialize(expression) @expression = expression portions = StringAndGroupParser.new(expression).portions.reject do |portion| portion.is_a? StringAndGroupParser::CommentPortion end @tokens = portions.inject([]) do |tokens, portion| case portion when StringAndGroupParser::StringPortion tokens << Tokens::String.new(portion.escaped_string) when StringAndGroupParser::GroupPortion tokens << Tokens::Group.new(portion.to_s) when StringAndGroupParser::OtherPortion scan = portion.to_s.scan(TOKEN_REGEX) tokens += tokenize!(scan) end tokens end end |
Instance Attribute Details
#expression ⇒ Object (readonly)
Returns the value of attribute expression.
24 25 26 |
# File 'lib/keisan/tokenizer.rb', line 24 def expression @expression end |
#tokens ⇒ Object (readonly)
Returns the value of attribute tokens.
24 25 26 |
# File 'lib/keisan/tokenizer.rb', line 24 def tokens @tokens end |