Class: CodeRay::TokenStream

Inherits:
Tokens
  • Object
show all
Defined in:
lib/coderay/tokens.rb

Overview

TokenStream

The TokenStream class is a fake Array without elements.

It redirects the method << to a block given at creation.

This allows scanners and Encoders to use streaming (no tokens are saved, the input is highlighted the same time it is scanned) with the same code.

See CodeRay.encode_stream and CodeRay.scan_stream

Constant Summary

Constants inherited from Tokens

CodeRay::Tokens::ClassOfKind

Instance Attribute Summary collapse

Attributes inherited from Tokens

#scanner

Instance Method Summary collapse

Methods inherited from Tokens

#each, #each_text_token, #encode, #fix, #fix!, load, #method_missing, #optimize!, #split_into_lines, #split_into_lines!, #text, #to_s

Constructor Details

#initialize(&block) ⇒ TokenStream

Creates a new TokenStream that calls block whenever its << method is called.

Example:

require 'coderay'

token_stream = CodeRay::TokenStream.new do |kind, text|
  puts 'kind: %s, text size: %d.' % [kind, text.size]
end

token_stream << [:regexp, '/\d+/']
#-> kind: rexpexp, text size: 5.

Raises:

  • (ArgumentError) 


314
315
316
317
318
 
# File 'lib/coderay/tokens.rb', line 314

def initialize &block
  raise ArgumentError, 'Block expected for streaming.' unless block
  @callback = block
  @size = 0
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method in the class CodeRay::Tokens

Instance Attribute Details

#size ⇒ Object (readonly)

The Array is empty, but size counts the tokens given by <<.



298
299
300
 
# File 'lib/coderay/tokens.rb', line 298

def size
  @size
end

Instance Method Details

#<<(token) ⇒ Object

Calls block with token and increments size.

Returns self.



323
324
325
326
327
 
# File 'lib/coderay/tokens.rb', line 323

def << token
  @callback.call(*token)
  @size += 1
  self
end

#dump ⇒ Object

A TokenStream cannot be dumped. Use Tokens.

Raises:

  • (NotImplementedError) 


336
337
338
 
# File 'lib/coderay/tokens.rb', line 336

def dump
  raise NotImplementedError, 'A TokenStream cannot be dumped.'
end

#optimize ⇒ Object

A TokenStream cannot be optimized. Use Tokens.

Raises:

  • (NotImplementedError) 


341
342
343
 
# File 'lib/coderay/tokens.rb', line 341

def optimize
  raise NotImplementedError, 'A TokenStream cannot be optimized.'
end

#stream? ⇒ Boolean

Whether the object is a TokenStream.

Returns true.

Returns:

  • (Boolean) 


293
294
295
 
# File 'lib/coderay/tokens.rb', line 293

def stream?
  true
end

#text_size ⇒ Object

This method is not implemented due to speed reasons. Use Tokens.

Raises:

  • (NotImplementedError) 


330
331
332
333
 
# File 'lib/coderay/tokens.rb', line 330

def text_size
  raise NotImplementedError,
    'This method is not implemented due to speed reasons.'
end