Module: RDoc::TokenStream

Included in:
AnyMethod, Parser::Ruby
Defined in:
lib/rdoc/token_stream.rb

Overview

A TokenStream is a list of tokens, gathered during the parse of some entity (say a method). Entities populate these streams by being registered with the lexer. Any class can collect tokens by including TokenStream. From the outside, you use such an object by calling the start_collecting_tokens method, followed by calls to add_token and pop_token.

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.to_html(token_stream) ⇒ Object

Converts token_stream to HTML wrapping various tokens with <span> elements. Some tokens types are wrapped in spans with the given class names. Other token types are not wrapped in spans.



16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/rdoc/token_stream.rb', line 16

def self.to_html token_stream
  starting_title = false

  token_stream.map do |t|
    next unless t

    style = case t[:kind]
            when :on_const   then 'ruby-constant'
            when :on_kw      then 'ruby-keyword'
            when :on_ivar    then 'ruby-ivar'
            when :on_cvar    then 'ruby-identifier'
            when :on_gvar    then 'ruby-identifier'
            when '=' != t[:text] && :on_op
                             then 'ruby-operator'
            when :on_tlambda then 'ruby-operator'
            when :on_ident   then 'ruby-identifier'
            when :on_label   then 'ruby-value'
            when :on_backref, :on_dstring
                             then 'ruby-node'
            when :on_comment then 'ruby-comment'
            when :on_embdoc  then 'ruby-comment'
            when :on_regexp  then 'ruby-regexp'
            when :on_tstring then 'ruby-string'
            when :on_int, :on_float,
                 :on_rational, :on_imaginary,
                 :on_heredoc,
                 :on_symbol, :on_CHAR then 'ruby-value'
            when :on_heredoc_beg, :on_heredoc_end
                             then 'ruby-identifier'
            end

    comment_with_nl = false
    if :on_comment == t[:kind] or :on_embdoc == t[:kind] or :on_heredoc_end == t[:kind]
      comment_with_nl = true if "\n" == t[:text][-1]
      text = t[:text].rstrip
    else
      text = t[:text]
    end

    if :on_ident == t[:kind] && starting_title
      starting_title = false
      style = 'ruby-identifier ruby-title'
    end

    if :on_kw == t[:kind] and 'def' == t[:text]
      starting_title = true
    end

    text = CGI.escapeHTML text

    if style then
      "<span class=\"#{style}\">#{text}</span>#{"\n" if comment_with_nl}"
    else
      text
    end
  end.join
end

Instance Method Details

#add_token(token) ⇒ Object

Adds one token to the collected tokens



84
85
86
 
# File 'lib/rdoc/token_stream.rb', line 84

def add_token(token)
  @token_stream.push(token)
end

#add_tokens(tokens) ⇒ Object

Adds tokens to the collected tokens



77
78
79
 
# File 'lib/rdoc/token_stream.rb', line 77

def add_tokens(tokens)
  @token_stream.concat(tokens)
end

#collect_tokens ⇒ Object Also known as: start_collecting_tokens

Starts collecting tokens



91
92
93
 
# File 'lib/rdoc/token_stream.rb', line 91

def collect_tokens
  @token_stream = []
end

#pop_token ⇒ Object

Remove the last token from the collected tokens



100
101
102
 
# File 'lib/rdoc/token_stream.rb', line 100

def pop_token
  @token_stream.pop
end

#token_stream ⇒ Object

Current token stream



107
108
109
 
# File 'lib/rdoc/token_stream.rb', line 107

def token_stream
  @token_stream
end

#tokens_to_s ⇒ Object

Returns a string representation of the token stream



114
115
116
 
# File 'lib/rdoc/token_stream.rb', line 114

def tokens_to_s
  token_stream.compact.map { |token| token[:text] }.join ''
end