Class: Datadog::SymbolDatabase::Extractor Private

Inherits:
Object
  • Object
show all
Defined in:
lib/datadog/symbol_database/extractor.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

Extracts symbol metadata from loaded Ruby modules and classes via introspection.

Instance created by Component with injected dependencies (logger, settings). All methods are instance methods accessing @logger, @settings directly — no parameter threading needed.

Uses Ruby's reflection APIs (Module#constants, Class#instance_methods, Method#parameters) to build hierarchical Scope structures representing code organization. Filters to user code only (excludes gems, stdlib, test files).

Extraction flow:

  1. ObjectSpace.each_object(Module) - Iterate all loaded modules/classes
  2. Filter to user code (user_code_module?)
  3. Build MODULE or CLASS scope with nested METHOD scopes
  4. Extract symbols: constants, class variables, method parameters

Called by: Component.extract_and_upload (during upload trigger) Produces: Scope objects passed to ScopeBatcher for batching File hashing: Calls FileHash.compute for MODULE scopes

Error handling strategy (defense-in-depth):

The extractor introspects arbitrary Ruby objects via ObjectSpace. Ruby's reflection APIs (Module#name, #instance_methods, #const_get, #source_location, #parameters) can fail unpredictably on third-party code: NameError from removed constants, LoadError from autoload, ArgumentError from overridden #name methods, SecurityError in restricted contexts, and more.

Rescue blocks are organized in three layers:

  1. Inner per-item rescues (bare rescue in const_get loops, method.name): Skip one constant or name lookup without aborting the enclosing collection. These are expected failures — no logging needed.

  2. Method-level rescues (rescue => e with logging): Catch failures in extract_method_scope, find_source_file, etc. Log at debug for post-hoc diagnosis, return nil or empty array. One bad method/module doesn't kill the entire class extraction.

  3. Top-level entry rescues (rescue => e with logging): extract() and extract_all() are the error boundaries. Any exception that escapes layers 1-2 is caught here and logged.

Constant Summary collapse

UNKNOWN_MIN_LINE =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Common Ruby core modules to exclude from included_modules extraction. These are ubiquitous mix-ins that don't provide meaningful context about the class structure. Kernel: Mixed into Object, appears in nearly all classes PP: Pretty-printing module, loaded by many tools JSON: JSON serialization module, loaded by many tools Enumerable: Core iteration protocol, extremely common Comparable: Core comparison protocol, extremely common Sentinel for unknown minimum line number. 0 means "available throughout the scope." Defined here (the only runtime consumer) so extractor.rb is self-contained. The parent module (lib/datadog/symbol_database.rb) defines the same values for documentation and external reference, but is not required by this file.

0
UNKNOWN_MAX_LINE =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

PostgreSQL signed INT_MAX (2^31 - 1). Means "entire file" or "unknown end."

2147483647
EXCLUDED_COMMON_MODULES =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future. 

["Kernel", "PP::", "JSON::", "Enumerable", "Comparable"].freeze
TARGETABLE_LINE_EVENTS =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

RubyVM::InstructionSequence#trace_points event types included when computing targetable lines on METHOD scopes. :line — any line with executable bytecode (primary line probe target) :return — last expression before method returns (DI instruments return events) :call excluded — method entry is handled by method probes, not line probes

[:line, :return].freeze
MODULE_NAME =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Cached UnboundMethod for Module#name — avoids resolving it on every safe_mod_name call. Some classes override .name (e.g. Faker::Travel::Airport), so we bind the original Module#name to get the real module name safely.

Module.instance_method(:name)

Instance Method Summary collapse

Constructor Details

#initialize(logger:, settings:) ⇒ Extractor

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of Extractor.

Parameters:



115
116
117
118
 
# File 'lib/datadog/symbol_database/extractor.rb', line 115

def initialize(logger:, settings:)
  @logger = logger
  @settings = settings
end

Instance Method Details

#extract(mod) ⇒ Scope?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Extract symbols from a single module or class. Returns nil if module should be skipped (anonymous, gem code, stdlib).

Returns a FILE scope wrapping the extracted CLASS or MODULE scope. The backend requires root-level scopes to be in ROOT_SCOPES (MODULE, JAR, ASSEMBLY, PACKAGE, FILE). FILE is the natural root for Ruby — one per source file.

For full extraction with proper FQN-based nesting and per-file method grouping, use extract_all instead. This method is kept for single-module extraction in tests.

Parameters:

  • mod (Module, Class) —

    The module or class to extract from

Returns:

  • (Scope, nil) —

    FILE scope wrapping extracted scope, or nil if filtered out



132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
 
# File 'lib/datadog/symbol_database/extractor.rb', line 132

def extract(mod)
  return nil unless Module === mod
  mod_name = safe_mod_name(mod)
  return nil unless mod_name

  return nil unless user_code_module?(mod)

  source_file = find_source_file(mod)
  return nil unless source_file

  inner_scope = if Class === mod
    extract_class_scope(mod)
  else
    extract_module_scope(mod)
  end

  wrap_in_file_scope(source_file, [inner_scope])
rescue Exception => e # standard:disable Lint/RescueException
  Datadog::DI.reraise_if_fatal(e)
  @logger.debug { "symdb: failed to extract #{mod_name || "<unknown>"}: #{e.class}: #{e.message}" }
  nil
end

#extract_all {|scope| ... } ⇒ Array<Scope>?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Extract symbols from all loaded modules and classes. Returns an array of FILE scopes with proper FQN-based nesting.

Two-pass algorithm: Pass 1 (build_per_file_index): iterate ObjectSpace once, building { file_path => [[mod_name, mod, [method_name_symbol, ...]], ...] }. Stores Symbol method names + Module refs only; no UnboundMethod retention between passes. Pass 2 (build_file_scope): for each file in the index, resolve UnboundMethods just-in-time, build the nested MODULE/CLASS scope tree from FQN splitting, and produce one FILE Scope. The per-file working set is released as soon as the FILE scope is yielded (or accumulated into the returned Array, in legacy mode).

This is the production path used by Component. Methods are split by source file, so a class reopened across two files produces two FILE scopes, each with only the methods defined in that file.

Memory profile (with a block):

- Pass 1 builds a per-file index containing only Symbol method names plus
Module references. No UnboundMethod objects are retained between passes.
- Pass 2 processes one file at a time. The peak per file is bounded by the
number of methods that live in that one file across all its modules
(typical Rails: tens of methods; pathological case: a single very large
source file). Once a FILE scope is yielded and the caller stops referencing
it, the entire per-file working set becomes garbage.

This is O(largest_file + batch_buffer), not O(total_classes).

Without a block, returns the full Array<Scope> (legacy form, used by specs). The Array itself still scales with the number of files, so block form is the one to use for production memory bounds.

Yield Parameters:

  • scope (Scope) —

    FILE scope for one source file

Returns:

  • (Array<Scope>, nil) —

    Array of FILE scopes when called without a block; nil when a block is given



190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'lib/datadog/symbol_database/extractor.rb', line 190

def extract_all
  index = build_per_file_index

  if block_given?
    # Drain the index destructively so each per-file entry becomes eligible for
    # collection as soon as its FILE scope is yielded and consumed. Hash#shift
    # returns [key, value] on a non-empty hash and nil when empty, so the
    # `while (pair = ...)` form is the drain. Indexing pair[0]/pair[1] rather
    # than destructuring avoids introducing names into method scope that would
    # then shadow the else-branch's block parameters.
    while (pair = index.shift)
      scope = build_file_scope(pair[0], pair[1])
      yield scope if scope
    end
    nil
  else
    # Legacy non-block form for specs. No memory bound — the full Array is
    # materialized.
    result = []
    index.each do |path, file_entries|
      scope = build_file_scope(path, file_entries)
      result << scope if scope
    end
    result
  end
rescue Exception => e # standard:disable Lint/RescueException
  Datadog::DI.reraise_if_fatal(e)
  @logger.debug { "symdb: error in extract_all: #{e.class}: #{e.message}" }
  block_given? ? nil : []
end