Module: RDoc::TokenStream

Included in:
AnyMethod
Defined in:
lib/rdoc/token_stream.rb

Overview

A TokenStream is a list of tokens, gathered during the parse of some entity (say a method). Entities populate these streams by being registered with the lexer. Any class can collect tokens by including TokenStream. From the outside, you use such an object by calling the start_collecting_tokens method, followed by calls to add_token and pop_token.

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.to_html(token_stream) ⇒ Object

Converts token_stream to HTML wrapping various tokens with elements. Some tokens types are wrapped in spans with the given class names. Other token types are not wrapped in spans.



18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
# File 'lib/rdoc/token_stream.rb', line 18

def self.to_html(token_stream)
  token_stream.map do |t|
    next unless t

    style = case t[:kind]
            when :operator   then 'ruby-operator'
            when :keyword    then 'ruby-keyword'
            when :constant   then 'ruby-constant'
            when :ivar       then 'ruby-ivar'
            when :comment    then 'ruby-comment'
            when :value      then 'ruby-value'
            when :string     then 'ruby-string'
            when :symbol     then 'ruby-value'
            when :x_string   then 'ruby-string'
            when :regexp     then 'ruby-regexp'
            when :identifier then 'ruby-identifier'
            end

    text = t[:text]

    text = CGI.escapeHTML text

    if style
      end_with_newline = text.end_with?("\n")
      text = text.chomp if end_with_newline
      "<span class=\"#{style}\">#{text}</span>#{"\n" if end_with_newline}"
    else
      text
    end
  end.join
end

Instance Method Details

#add_token(token) ⇒ Object

Adds one token to the collected tokens



60
61
62
 
# File 'lib/rdoc/token_stream.rb', line 60

def add_token(token)
  token_stream.push(token)
end

#add_tokens(tokens) ⇒ Object

Adds tokens to the collected tokens



53
54
55
 
# File 'lib/rdoc/token_stream.rb', line 53

def add_tokens(tokens)
  token_stream.concat(tokens)
end

#collect_tokens(language, loader: nil) ⇒ Object Also known as: start_collecting_tokens

Starts collecting tokens

The optional loader is called once on first access and its result is reused.



69
70
71
72
73
 
# File 'lib/rdoc/token_stream.rb', line 69

def collect_tokens(language, loader: nil)
  @token_stream = []
  @token_stream_loader = loader
  @token_stream_language = language
end

#pop_token ⇒ Object

Remove the last token from the collected tokens



80
81
82
 
# File 'lib/rdoc/token_stream.rb', line 80

def pop_token
  token_stream.pop
end

#source_language ⇒ Object

Returns the source language of the token stream as a string

Returns 'c' or 'ruby'



107
108
109
 
# File 'lib/rdoc/token_stream.rb', line 107

def source_language
  @token_stream_language == :c ? 'c' : 'ruby'
end

#token_stream ⇒ Object

Current token stream



87
88
89
90
91
92
93
 
# File 'lib/rdoc/token_stream.rb', line 87

def token_stream
  if @token_stream_loader
    @token_stream = @token_stream_loader.call
    @token_stream_loader = nil
  end
  @token_stream
end

#tokens_to_s ⇒ Object

Returns a string representation of the token stream



98
99
100
 
# File 'lib/rdoc/token_stream.rb', line 98

def tokens_to_s
  (token_stream or return '').compact.map { |token| token[:text] }.join ''
end