Class: YARD::Parser::SourceParser

Inherits:

Object

Object
YARD::Parser::SourceParser

show all

Defined in:: lib/yard/parser/source_parser.rb

Overview

Responsible for parsing a source file into the namespace. Parsing also invokes handlers to process the parsed statements and generate any code objects that may be recognized.

Custom Parsers

SourceParser allows custom parsers to be registered and called when a certain filetype is recognized. To register a parser and hook it up to a set of file extensions, call SourceParser.register_parser_type

See Also:

Constant Summary collapse

SHEBANG_LINE =

/\A\s*#!\S+/

ENCODING_LINE =

%r{\A(?:\s*#*!.*\r?\n)?\s*(?:#+|/\*+|//+).*coding\s*[:=]{1,2}\s*([a-z\d_\-]+)}i

FROZEN_STRING_LINE =

/frozen(-|_)string(-|_)literal:\s+(true|false)/i

DEFAULT_PATH_GLOB = The default glob of files to be parsed. Since: 0.9.0

["{lib,app}/**/*.rb", "ext/**/*.{c,cc,cxx,cpp,rb}"]

ENCODING_BYTE_ORDER_MARKS = Byte order marks for various encodings Since: 0.7.0

{
  'utf-8' => String.new("\xEF\xBB\xBF"),
  # Not yet supported
  # 'utf-16be' => "\xFE\xFF",
  # 'utf-16le' => "\xFF\xFE",
  # 'utf-32be' => "\x00\x00\xFF\xFE",
  # 'utf-32le' => "\xFF\xFE",
}

Parser Callbacks collapse

#contents ⇒ String readonly

The contents of the file to be parsed.
#file ⇒ String

The filename being parsed by the parser.
#globals ⇒ OpenStruct readonly

An open struct containing arbitrary global state shared between files and handlers.
#parser_type ⇒ Symbol readonly

The parser type associated with the parser instance.

Class Attribute Summary collapse

.parser_type ⇒ Symbol

The default parser type (defaults to :ruby).

Parser Callbacks collapse

.after_parse_file {|parser| ... } ⇒ Proc

Registers a callback to be called after an individual file is parsed.
.after_parse_file_callbacks ⇒ Array<Proc>

The list of callbacks to be called after parsing a file.
.after_parse_list {|files, globals| ... } ⇒ Proc

Registers a callback to be called after a list of files is parsed via SourceParser.parse.
.after_parse_list_callbacks ⇒ Array<Proc>

The list of callbacks to be called after parsing a list of files.
.before_parse_file {|parser| ... } ⇒ Proc

Registers a callback to be called before an individual file is parsed.
.before_parse_file_callbacks ⇒ Array<Proc>

The list of callbacks to be called before parsing a file.
.before_parse_list {|files, globals| ... } ⇒ Proc

Registers a callback to be called before a list of files is parsed via SourceParser.parse.
.before_parse_list_callbacks ⇒ Array<Proc>

The list of callbacks to be called before parsing a list of files.
#initialize(parser_type = SourceParser.parser_type, globals = nil) ⇒ SourceParser constructor

Creates a new parser object for code parsing with a specific parser type.
#parse(content = __FILE__) ⇒ Object^?

The main parser method.
#tokenize(content) ⇒ Array

Tokenizes but does not parse the block of code using the current #parser_type.

Class Method Summary collapse

.parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level) ⇒ void

Parses a path or set of paths.
.parse_string(content, ptype = parser_type) ⇒ Object

Parses a string content.
.parser_type_extensions=(value) ⇒ Object
.parser_type_for_extension(extension) ⇒ Symbol

Finds a parser type that is registered for the extension.
.parser_types=(value) ⇒ Object
.register_parser_type(type, parser_klass, extensions = nil) ⇒ void

Registers a new parser type.
.tokenize(content, ptype = parser_type) ⇒ Array

Tokenizes but does not parse the block of code.

Constructor Details

#initialize(parser_type = SourceParser.parser_type, globals = nil) ⇒ `SourceParser`

Returns a new instance of SourceParser.

Creates a new parser object for code parsing with a specific parser type.

Parameters:

parser_type (Symbol) (defaults to: SourceParser.parser_type) —

the parser type to use
globals (OpenStruct) (defaults to: nil) —

global state to be re-used across separate source files

# File 'lib/yard/parser/source_parser.rb', line 406

def initialize(parser_type = SourceParser.parser_type, globals1 = nil, globals2 = nil)
  globals = [true, false].include?(globals1) ? globals2 : globals1
  @file = '(stdin)'
  @globals = globals || OpenStruct.new
  self.parser_type = parser_type
end

Class Attribute Details

.parser_type ⇒ `Symbol`

Returns the default parser type (defaults to :ruby).

Returns:

(Symbol) —

the default parser type (defaults to :ruby)



85
86
87

# File 'lib/yard/parser/source_parser.rb', line 85

def parser_type
  @parser_type
end

Instance Attribute Details

#contents ⇒ `String` (readonly)

Returns the contents of the file to be parsed.

Returns:

(String) —

the contents of the file to be parsed

Since:

0.7.0



399
400
401

# File 'lib/yard/parser/source_parser.rb', line 399

def contents
  @contents
end

#file ⇒ `String`

Returns the filename being parsed by the parser.

Returns:

(String) —

the filename being parsed by the parser.



386
387
388

# File 'lib/yard/parser/source_parser.rb', line 386

def file
  @file
end

#globals ⇒ `OpenStruct` (readonly)

Returns an open struct containing arbitrary global state shared between files and handlers.

Returns:

(OpenStruct) —

an open struct containing arbitrary global state shared between files and handlers.

Since:

0.7.0



395
396
397

# File 'lib/yard/parser/source_parser.rb', line 395

def globals
  @globals
end

#parser_type ⇒ `Symbol`

Returns the parser type associated with the parser instance. This should be set by the constructor.

Returns:

(Symbol) —

the parser type associated with the parser instance. This should be set by the constructor.



390
391
392

# File 'lib/yard/parser/source_parser.rb', line 390

def parser_type
  @parser_type
end

Class Method Details

.after_parse_file {|parser| ... } ⇒ `Proc`

Registers a callback to be called after an individual file is parsed. The block passed to this method will be called on subsequent parse calls.

To register a callback that is called after the entire list of files is processed, see after_parse_list.

Examples:

Printing the length of each file after it is parsed

SourceParser.after_parse_file do |parser|
  puts "#{parser.file} is #{parser.contents.size} characters"
end
YARD.parse('lib/**/*.rb')
# prints:
"lib/foo.rb is 1240 characters"
"lib/foo_bar.rb is 248 characters"

Yields:

(parser) —

the yielded block is called once after each file that is parsed. This might happen many times for a single codebase.

Yield Parameters:

parser (SourceParser) —

the parser object that parsed the file.

Yield Returns:

(void) —

the return value for the block is ignored.

Returns:

(Proc) —

the yielded block

See Also:

Since:

0.7.0



324
325
326

# File 'lib/yard/parser/source_parser.rb', line 324

def after_parse_file(&block)
  after_parse_file_callbacks << block
end

.after_parse_file_callbacks ⇒ `Array<Proc>`

Returns the list of callbacks to be called after parsing a file. Should only be used for testing.

Returns:

(Array<Proc>) —

the list of callbacks to be called after parsing a file. Should only be used for testing.

Since:

0.7.0



352
353
354

# File 'lib/yard/parser/source_parser.rb', line 352

def after_parse_file_callbacks
  @after_parse_file_callbacks ||= []
end

.after_parse_list {|files, globals| ... } ⇒ `Proc`

Registers a callback to be called after a list of files is parsed via parse. The block passed to this method will be called on subsequent parse calls.

Examples:

Printing results after parsing occurs

SourceParser.after_parse_list do
  puts "Finished parsing!"
end
YARD.parse
# Prints "Finished parsing!" after parsing files

Yields:

(files, globals) —

the yielded block is called once before parsing all files

Yield Parameters:

files (Array<String>) —

the list of files that will be parsed.
globals (OpenStruct) —

a global structure to store arbitrary state for post processing (see Handlers::Processor#globals)

Yield Returns:

(void) —

the return value for the block is ignored.

Returns:

(Proc) —

the yielded block

See Also:

Since:

0.7.0



258
259
260

# File 'lib/yard/parser/source_parser.rb', line 258

def after_parse_list(&block)
  after_parse_list_callbacks << block
end

.after_parse_list_callbacks ⇒ `Array<Proc>`

Returns the list of callbacks to be called after parsing a list of files. Should only be used for testing.

Returns:

(Array<Proc>) —

the list of callbacks to be called after parsing a list of files. Should only be used for testing.

Since:

0.7.0



338
339
340

# File 'lib/yard/parser/source_parser.rb', line 338

def after_parse_list_callbacks
  @after_parse_list_callbacks ||= []
end

.before_parse_file {|parser| ... } ⇒ `Proc`

Registers a callback to be called before an individual file is parsed. The block passed to this method will be called on subsequent parse calls.

To register a callback that is called before the entire list of files is processed, see before_parse_list.

Examples:

Installing a simple callback

SourceParser.before_parse_file do |parser|
  puts "I'm parsing #{parser.file}"
end
YARD.parse('lib/**/*.rb')
# prints:
"I'm parsing lib/foo.rb"
"I'm parsing lib/foo_bar.rb"
"I'm parsing lib/last_file.rb"

Cancel parsing of any test_*.rb files

SourceParser.before_parse_file do |parser|
  return false if parser.file =~ /^test_.+\.rb$/
end

Yields:

(parser) —

the yielded block is called once before each file that is parsed. This might happen many times for a single codebase.

Yield Parameters:

parser (SourceParser) —

the parser object that will #parse the file.

Yield Returns:

(Boolean) —

if the block returns false, parsing for the file is cancelled.

Returns:

(Proc) —

the yielded block

See Also:

Since:

0.7.0



295
296
297

# File 'lib/yard/parser/source_parser.rb', line 295

def before_parse_file(&block)
  before_parse_file_callbacks << block
end

.before_parse_file_callbacks ⇒ `Array<Proc>`

Returns the list of callbacks to be called before parsing a file. Should only be used for testing.

Returns:

(Array<Proc>) —

the list of callbacks to be called before parsing a file. Should only be used for testing.

Since:

0.7.0



345
346
347

# File 'lib/yard/parser/source_parser.rb', line 345

def before_parse_file_callbacks
  @before_parse_file_callbacks ||= []
end

.before_parse_list {|files, globals| ... } ⇒ `Proc`

Registers a callback to be called before a list of files is parsed via parse. The block passed to this method will be called on subsequent parse calls.

Examples:

Installing a simple callback

SourceParser.before_parse_list do |files, globals|
  puts "Starting to parse..."
end
YARD.parse('lib/**/*.rb')
# prints "Starting to parse..."

Setting global state

SourceParser.before_parse_list do |files, globals|
  globals.method_count = 0
end
SourceParser.after_parse_list do |files, globals|
  puts "Found #{globals.method_count} methods"
end
class MyCountHandler < Handlers::Ruby::Base
  handles :def, :defs
  process { globals.method_count += 1 }
end
YARD.parse
# Prints: "Found 37 methods"

Using a global callback to cancel parsing

SourceParser.before_parse_list do |files, globals|
  return false if files.include?('foo.rb')
end

YARD.parse(['foo.rb', 'bar.rb']) # callback cancels this method
YARD.parse('bar.rb') # parses normally

Yields:

(files, globals) —

the yielded block is called once before parsing all files

Yield Parameters:

files (Array<String>) —

the list of files that will be parsed.
globals (OpenStruct) —

a global structure to store arbitrary state for post processing (see Handlers::Processor#globals)

Yield Returns:

(Boolean) —

if the block returns false, parsing is cancelled.

Returns:

(Proc) —

the yielded block

See Also:

Since:

0.7.0



234
235
236

# File 'lib/yard/parser/source_parser.rb', line 234

def before_parse_list(&block)
  before_parse_list_callbacks << block
end

.before_parse_list_callbacks ⇒ `Array<Proc>`

Returns the list of callbacks to be called before parsing a list of files. Should only be used for testing.

Returns:

(Array<Proc>) —

the list of callbacks to be called before parsing a list of files. Should only be used for testing.

Since:

0.7.0



331
332
333

# File 'lib/yard/parser/source_parser.rb', line 331

def before_parse_list_callbacks
  @before_parse_list_callbacks ||= []
end

.parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level) ⇒ `void`

This method returns an undefined value.

Parses a path or set of paths

Parameters:

paths (String, Array<String>) (defaults to: DEFAULT_PATH_GLOB) —

a path, glob, or list of paths to parse
excluded (Array<String, Regexp>) (defaults to: []) —

a list of excluded path matchers
level (Fixnum) (defaults to: log.level) —

the logger level to use during parsing. See Logger

# File 'lib/yard/parser/source_parser.rb', line 99

def parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level)
  log.debug("Parsing #{paths.inspect} with `#{parser_type}` parser")
  excluded = excluded.map do |path|
    case path
    when Regexp; path
    else Regexp.new(path.to_s, Regexp::IGNORECASE)
    end
  end
  files = [paths].flatten.
    map {|p| File.directory?(p) ? "#{p}/**/*.{rb,c,cc,cxx,cpp}" : p }.
    map {|p| p.include?("*") ? Dir[p].sort_by {|d| [d.length, d] } : p }.flatten.
    reject {|p| !File.file?(p) || excluded.any? {|re| p =~ re } }.
    map {|p| p.encoding == Encoding.default_external ? p : p.dup.force_encoding(Encoding.default_external) }

  log.enter_level(level) do
    parse_in_order(*files.uniq)
  end
end

.parse_string(content, ptype = parser_type) ⇒ `Object`

Parses a string content

Parameters:

content (String) —

the block of code to parse
ptype (Symbol) (defaults to: parser_type) —

the parser type to use. See parser_type.

Returns:

the parser object that was used to parse content



123
124
125

# File 'lib/yard/parser/source_parser.rb', line 123

def parse_string(content, ptype = parser_type)
  new(ptype).parse(StringIO.new(content))
end

.parser_type_extensions=(value) ⇒ `Object`

164	# File 'lib/yard/parser/source_parser.rb', line 164 def parser_type_extensions=(value) @@parser_type_extensions = value end

.parser_type_for_extension(extension) ⇒ `Symbol`

Finds a parser type that is registered for the extension. If no type is found, the default Ruby type is returned.

Returns:

(Symbol) —

the parser type to be used for the extension

Since:

0.5.6

# File 'lib/yard/parser/source_parser.rb', line 171

def parser_type_for_extension(extension)
  type = parser_type_extensions.find do |_t, exts|
    [exts].flatten.any? {|ext| ext === extension }
  end
  validated_parser_type(type ? type.first : :ruby)
end

.parser_types=(value) ⇒ `Object`

158	# File 'lib/yard/parser/source_parser.rb', line 158 def parser_types=(value) @@parser_types = value end

.register_parser_type(type, parser_klass, extensions = nil) ⇒ `void`

This method returns an undefined value.

Registers a new parser type.

Examples:

Registering a parser for “java” files

SourceParser.register_parser_type :java, JavaParser, 'java'

Parameters:

type (Symbol) —

a symbolic name for the parser type
parser_klass (Base) —

a class that implements parsing and tokenization
extensions (Array<String>, String, Regexp) (defaults to: nil) —

a list of extensions or a regex to match against the file extension

See Also:

Base

# File 'lib/yard/parser/source_parser.rb', line 146

def register_parser_type(type, parser_klass, extensions = nil)
  unless Base > parser_klass
    raise ArgumentError, "expecting parser_klass to be a subclass of YARD::Parser::Base"
  end
  parser_type_extensions[type.to_sym] = extensions if extensions
  parser_types[type.to_sym] = parser_klass
end

.tokenize(content, ptype = parser_type) ⇒ `Array`

Tokenizes but does not parse the block of code

Parameters:

content (String) —

the block of code to tokenize
ptype (Symbol) (defaults to: parser_type) —

the parser type to use. See parser_type.

Returns:

(Array) —

a list of tokens



132
133
134

# File 'lib/yard/parser/source_parser.rb', line 132

def tokenize(content, ptype = parser_type)
  new(ptype).tokenize(content)
end

Instance Method Details

#parse(content = FILE) ⇒ `Object`^?

The main parser method. This should not be called directly. Instead, use the class methods parse and parse_string.

Parameters:

content (String, #read, Object) (defaults to: __FILE__) —

the source file to parse

Returns:

(Object, nil) —

the parser object used to parse the source

# File 'lib/yard/parser/source_parser.rb', line 418

def parse(content = __FILE__)
  case content
  when String
    @file = File.cleanpath(content)
    content = convert_encoding(String.new(File.read_binary(file)))
    checksum = Registry.checksum_for(content)
    return if Registry.checksums[file] == checksum

    if Registry.checksums.key?(file)
      log.info "File '#{file}' was modified, re-processing..."
    end
    Registry.checksums[@file] = checksum
    self.parser_type = parser_type_for_filename(file)
  else
    content = content.read if content.respond_to? :read
  end

  @contents = content
  @parser = parser_class.new(content, file)

  self.class.before_parse_file_callbacks.each do |cb|
    return @parser if cb.call(self) == false
  end

  @parser.parse
  post_process

  self.class.after_parse_file_callbacks.each do |cb|
    cb.call(self)
  end

  @parser
rescue ArgumentError, NotImplementedError => e
  log.warn("Cannot parse `#{file}': #{e.message}")
  log.backtrace(e, :warn)
rescue ParserSyntaxError => e
  log.warn(e.message.capitalize)
  log.backtrace(e, :warn)
end

#tokenize(content) ⇒ `Array`

Tokenizes but does not parse the block of code using the current #parser_type

Parameters:

content (String) —

the block of code to tokenize

Returns:

(Array) —

a list of tokens

# File 'lib/yard/parser/source_parser.rb', line 462

def tokenize(content)
  @parser = parser_class.new(content, file)
  @parser.tokenize
end

Class: YARD::Parser::SourceParser

Overview

Custom Parsers

Constant Summary collapse

Parser Callbacks collapse

Class Attribute Summary collapse

Parser Callbacks collapse

Class Method Summary collapse

Constructor Details

#initialize(parser_type = SourceParser.parser_type, globals = nil) ⇒ SourceParser

Class Attribute Details

.parser_type ⇒ Symbol

Instance Attribute Details

#contents ⇒ String (readonly)

#file ⇒ String

#globals ⇒ OpenStruct (readonly)

#parser_type ⇒ Symbol

Class Method Details

.after_parse_file {|parser| ... } ⇒ Proc

Examples:

Printing the length of each file after it is parsed

.after_parse_file_callbacks ⇒ Array<Proc>

.after_parse_list {|files, globals| ... } ⇒ Proc

Examples:

Printing results after parsing occurs

.after_parse_list_callbacks ⇒ Array<Proc>

.before_parse_file {|parser| ... } ⇒ Proc

Examples:

Installing a simple callback

Cancel parsing of any test_*.rb files

.before_parse_file_callbacks ⇒ Array<Proc>

.before_parse_list {|files, globals| ... } ⇒ Proc

Examples:

Installing a simple callback

Setting global state

Using a global callback to cancel parsing

.before_parse_list_callbacks ⇒ Array<Proc>

.parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level) ⇒ void

.parse_string(content, ptype = parser_type) ⇒ Object

.parser_type_extensions=(value) ⇒ Object

.parser_type_for_extension(extension) ⇒ Symbol

.parser_types=(value) ⇒ Object

.register_parser_type(type, parser_klass, extensions = nil) ⇒ void

Examples:

Registering a parser for “java” files

.tokenize(content, ptype = parser_type) ⇒ Array

Instance Method Details

#parse(content = __FILE__) ⇒ Object?

#tokenize(content) ⇒ Array

#initialize(parser_type = SourceParser.parser_type, globals = nil) ⇒ `SourceParser`

.parser_type ⇒ `Symbol`

#contents ⇒ `String` (readonly)

#file ⇒ `String`

#globals ⇒ `OpenStruct` (readonly)

#parser_type ⇒ `Symbol`

.after_parse_file {|parser| ... } ⇒ `Proc`

.after_parse_file_callbacks ⇒ `Array<Proc>`

.after_parse_list {|files, globals| ... } ⇒ `Proc`

.after_parse_list_callbacks ⇒ `Array<Proc>`

.before_parse_file {|parser| ... } ⇒ `Proc`

.before_parse_file_callbacks ⇒ `Array<Proc>`

.before_parse_list {|files, globals| ... } ⇒ `Proc`

.before_parse_list_callbacks ⇒ `Array<Proc>`

.parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level) ⇒ `void`

.parse_string(content, ptype = parser_type) ⇒ `Object`

.parser_type_extensions=(value) ⇒ `Object`

.parser_type_for_extension(extension) ⇒ `Symbol`

.parser_types=(value) ⇒ `Object`

.register_parser_type(type, parser_klass, extensions = nil) ⇒ `void`

.tokenize(content, ptype = parser_type) ⇒ `Array`

#parse(content = FILE) ⇒ `Object`^?

#tokenize(content) ⇒ `Array`