Module: RDoc::TokenStream

Included in:
AnyMethod, Parser::Ruby
Defined in:
lib/rdoc/token_stream.rb

Overview

A TokenStream is a list of tokens, gathered during the parse of some entity (say a method). Entities populate these streams by being registered with the lexer. Any class can collect tokens by including TokenStream. From the outside, you use such an object by calling the start_collecting_tokens method, followed by calls to add_token and pop_token.

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.to_html(token_stream) ⇒ Object

Converts token_stream to HTML wrapping various tokens with <span> elements. The following tokens types are wrapped in spans with the given class names:

TkCONSTANT

‘ruby-constant’

TkKW

‘ruby-keyword’

TkIVAR

‘ruby-ivar’

TkOp

‘ruby-operator’

TkId

‘ruby-identifier’

TkNode

‘ruby-node’

TkCOMMENT

‘ruby-comment’

TkREGEXP

‘ruby-regexp’

TkSTRING

‘ruby-string’

TkVal

‘ruby-value’

Other token types are not wrapped in spans.



28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
 
# File 'lib/rdoc/token_stream.rb', line 28

def self.to_html token_stream
  token_stream.map do |t|
    next unless t

    style = case t
            when RDoc::RubyToken::TkCONSTANT then 'ruby-constant'
            when RDoc::RubyToken::TkKW       then 'ruby-keyword'
            when RDoc::RubyToken::TkIVAR     then 'ruby-ivar'
            when RDoc::RubyToken::TkOp       then 'ruby-operator'
            when RDoc::RubyToken::TkId       then 'ruby-identifier'
            when RDoc::RubyToken::TkNode     then 'ruby-node'
            when RDoc::RubyToken::TkCOMMENT  then 'ruby-comment'
            when RDoc::RubyToken::TkREGEXP   then 'ruby-regexp'
            when RDoc::RubyToken::TkSTRING   then 'ruby-string'
            when RDoc::RubyToken::TkVal      then 'ruby-value'
            end

    text = CGI.escapeHTML t.text

    if style then
      "<span class=\"#{style}\">#{text}</span>"
    else
      text
    end
  end.join
end

Instance Method Details

#add_tokens(*tokens) ⇒ Object Also known as: add_token

Adds tokens to the collected tokens



58
59
60
 
# File 'lib/rdoc/token_stream.rb', line 58

def add_tokens(*tokens)
  tokens.flatten.each { |token| @token_stream << token }
end

#collect_tokens ⇒ Object Also known as: start_collecting_tokens

Starts collecting tokens



67
68
69
 
# File 'lib/rdoc/token_stream.rb', line 67

def collect_tokens
  @token_stream = []
end

#pop_token ⇒ Object

Remove the last token from the collected tokens



76
77
78
 
# File 'lib/rdoc/token_stream.rb', line 76

def pop_token
  @token_stream.pop
end

#token_stream ⇒ Object

Current token stream



83
84
85
 
# File 'lib/rdoc/token_stream.rb', line 83

def token_stream
  @token_stream
end

#tokens_to_s ⇒ Object

Returns a string representation of the token stream



90
91
92
 
# File 'lib/rdoc/token_stream.rb', line 90

def tokens_to_s
  token_stream.map { |token| token.text }.join ''
end