Class: Datadog::DI::CodeTracker Private

Inherits:
Object
  • Object
show all
Defined in:
lib/datadog/di/code_tracker.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

Tracks loaded Ruby code by source file and maintains a map from source file to the loaded code (instruction sequences). Also arranges for code in the loaded files to be instrumented by line probes that have already been received by the library.

The loaded code is used to target line trace points when installing line probes which dramatically improves efficiency of line trace points.

Note that, since most files will only be loaded one time (via the "require" mechanism), the code tracker needs to be global and not be recreated when the DI component is created.

Instance Method Summary collapse

Constructor Details

#initialize ⇒ CodeTracker

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of CodeTracker.



24
25
26
27
28
29
30
 
# File 'lib/datadog/di/code_tracker.rb', line 24

def initialize
  @registry = {}
  @per_method_registry = {}
  @trace_point_lock = Mutex.new
  @registry_lock = Mutex.new
  @compiled_trace_point = nil
end

Instance Method Details

#active? ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns whether this code tracker has been activated and is tracking.

Returns:

  • (Boolean) 


224
225
226
227
228
 
# File 'lib/datadog/di/code_tracker.rb', line 224

def active?
  trace_point_lock.synchronize do
    !!@compiled_trace_point
  end
end

#backfill_registry ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Populates the registry with iseqs for files that were loaded before code tracking started.

Uses the all_iseqs C extension to walk the Ruby object space and find instruction sequences for already-loaded code. Whole-file iseqs are stored in the main registry; per-method/block/class iseqs are stored in per_method_registry as fallback for files whose whole-file iseq was GC'd.

See docs/DynamicInstrumentationDevelopment.md "Iseq Lifecycle and GC" for which iseq types survive GC and implications for backfill.

Whole-file detection uses two strategies:

  • Ruby 3.1+: DI.iseq_type (wraps rb_iseq_type) returns :top for require/load and :main for the entry script. This is precise.
  • Ruby < 3.1: falls back to first_lineno == 0, which is true for whole-file iseqs from require/load (INT2FIX(0) in Ruby's rb_iseq_new_top and rb_iseq_new_main) and false for method/block/class definitions (first_lineno >= 1). InstructionSequence.compile passes first_lineno = 1 by default, so eval'd code is not matched. Both strategies produce the same result in practice.

Does not overwrite iseqs already in the registry (from :script_compiled), since those are guaranteed to be whole-file iseqs and are authoritative.



60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
 
# File 'lib/datadog/di/code_tracker.rb', line 60

def backfill_registry
  iseqs = DI.file_iseqs
  have_iseq_type = DI.respond_to?(:iseq_type)
  registry_lock.synchronize do
    iseqs.each do |iseq|
      path = iseq.absolute_path
      next unless path

      whole_file = if have_iseq_type
        type = DI.iseq_type(iseq)
        # Require first_lineno == 0 to exclude compile_file/compile
        # iseqs. These are :top type but have first_lineno == 1 and
        # produce iseq objects distinct from require-produced iseqs.
        # Targeted TracePoints are bound to the specific iseq object
        # — a probe on a compile_file iseq silently never fires when
        # the require-produced code runs.
        (type == :top || type == :main) && iseq.first_lineno == 0
      else
        iseq.first_lineno == 0
      end

      if whole_file
        # Ruby 3.2.9+ creates dummy profiler iseqs during require/load
        # (rb_iseq_alloc_with_dummy_path in iseq.c). These have type
        # :top, first_lineno == 0, and the same absolute_path as the
        # real iseq — but iseq_size == 0 (no bytecode). A targeted
        # TracePoint on a dummy iseq can't find child iseqs and raises
        # ArgumentError "can not enable any hooks". Filter them out:
        # a real top-level iseq always has at least one trace event.
        next if iseq.trace_points.empty?

        # Do not overwrite entries from :script_compiled — those are
        # captured at load time and are authoritative.
        next if registry.key?(path)

        registry[path] = iseq
      else
        # Skip top-level script iseqs (:top/:main) produced by
        # RubyVM::InstructionSequence.compile_file and .compile
        # (compile source to bytecode without executing it).
        # These represent the file body,
        # not a method or block. They pass the first_lineno check
        # (lineno != 0) but a targeted TracePoint bound to one
        # of these never fires for method-level code — the
        # user's probe silently produces no snapshots.
        #
        # On Ruby < 3.1 (no iseq_type), we cannot distinguish
        # these from method iseqs, so they leak into
        # per_method_registry. If iseq_for_line selects a leaked
        # top-level iseq instead of the real method iseq, the
        # probe installs but silently never fires — same failure
        # as above. This requires the application to call
        # compile_file and hold the result, which is rare outside
        # tooling like bootsnap (which discards it).
        next if have_iseq_type && (type == :top || type == :main)

        # Store per-method/block/class iseqs as fallback for files
        # whose whole-file iseq was GC'd. These can be used to
        # target line probes on lines within their range.
        (per_method_registry[path] ||= []) << iseq
      end
    end
  end
  nil
rescue Exception => exc # standard:disable Lint/RescueException
  Datadog::DI.reraise_if_fatal(exc)
  # Backfill is best-effort — if it fails, line probes on
  # pre-loaded code won't work but everything else is unaffected.
  if component = DI.current_component
    component.logger.debug { "di: backfill_registry failed: #{exc.class}: #{exc.message}" }
    component.telemetry&.report(exc, description: "backfill_registry failed")
  end
  nil
end

#clear ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Clears the stored mapping from paths to compiled code.

This method should normally never be called. It is meant to be used only by the test suite.



354
355
356
357
358
359
 
# File 'lib/datadog/di/code_tracker.rb', line 354

def clear
  registry_lock.synchronize do
    registry.clear
    per_method_registry.clear
  end
end

#iseq_for_line(suffix, line) ⇒ Array(String, RubyVM::InstructionSequence)?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a [path, iseq] pair for a line probe target, or nil.

First checks the whole-file iseq registry (via iseqs_for_path_suffix). If no whole-file iseq exists, searches the per-method iseq registry for an iseq whose trace_points include the target line.

Parameters:

  • suffix (String) —

    file path or suffix to match

  • line (Integer) —

    target line number

Returns:

  • (Array(String, RubyVM::InstructionSequence), nil) 


295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
 
# File 'lib/datadog/di/code_tracker.rb', line 295

def iseq_for_line(suffix, line)
  # Try whole-file iseq first — it always covers all lines.
  result = iseqs_for_path_suffix(suffix)
  return result if result

  # Fall back to per-method iseqs.
  registry_lock.synchronize do
    # Resolve the path using the per-method registry keys.
    path = resolve_path_suffix(suffix, per_method_registry.keys)
    return nil unless path

    iseqs = per_method_registry[path]
    return nil unless iseqs

    # Only match event types the instrumenter subscribes to
    # (:line, :return, :b_return — see hook_line). Lines that
    # only carry :call (e.g. a `def` line within the defined
    # method's own iseq, not the enclosing scope) have no
    # subscribed event at that position; TracePoint#enable
    # raises because it cannot bind an enabled event there.
    matches = iseqs.select do |iseq|
      iseq.trace_points.any? do |tp_line, event|
        tp_line == line && (event == :line || event == :return || event == :b_return)
      end
    end
    # When multiple iseqs contain the target line (e.g. a method
    # and an inline block sharing the same line), picking one
    # would silently miss executions in the other context.
    # Raise so the probe is recorded as failed with a clear error.
    if matches.length > 1
      raise Error::MultiplePathsMatch, "Multiple code locations match line #{line}"
    end
    matches.first ? [path, matches.first] : nil
  end
end

#iseqs_for_path_suffix(suffix) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns an array of RubVM::InstructionSequence (i.e. the compiled code) for the provided path.

The argument can be a full path to a Ruby source code file or a suffix (basename + one or more directories preceding the basename). The idea with suffix matches is that file paths are likely to be different between development and production environments and the source control system uses relative paths and doesn't have absolute paths at all.

Suffix matches are not guaranteed to be correct, meaning there may be multiple files with the same basename and they may all match a given suffix. In such cases, this method will return all matching paths (and all of these paths will be attempted to be instrumented by upstream code).

If the suffix matches one of the paths completely (which requires it to be an absolute path), only the exactly matching path is returned. Otherwise all known paths that end in the suffix are returned. If no paths match, an empty array is returned.



250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
 
# File 'lib/datadog/di/code_tracker.rb', line 250

def iseqs_for_path_suffix(suffix)
  registry_lock.synchronize do
    exact = registry[suffix]
    return [suffix, exact] if exact

    # Normalize Windows-style backslash separators (DEBUG-5111) upfront
    # so the suffix-shortening loop's "/+" regex can strip leading
    # components on probes whose sourceFile uses backslashes.
    suffix = Utils.normalize_windows_separators(suffix)

    # Per the design comment in utils.rb, attempt case-sensitive
    # matching first (steps 5-6) and only fall back to case-insensitive
    # matching (steps 7-8) when no case-sensitive match is found.
    [false, true].each do |case_insensitive|
      working_suffix = suffix.dup
      loop do
        inexact = []
        registry.each do |path, iseq|
          if Utils.path_matches_suffix?(path, working_suffix, case_insensitive: case_insensitive)
            inexact << [path, iseq]
          end
        end
        if inexact.length > 1
          raise Error::MultiplePathsMatch, "Multiple paths matched requested suffix"
        end
        if inexact.any?
          return inexact.first
        end
        break unless working_suffix.include?("/")
        working_suffix.sub!(%r{.*/+}, "")
      end
    end
    nil
  end
end

#start ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Starts tracking loaded code.

This method should generally be called early in application boot process, because any code loaded before code tracking is enabled will not be instrumentable via line probes.

Normally tracking should remain active for the lifetime of the process and would not be ever stopped.



143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
 
# File 'lib/datadog/di/code_tracker.rb', line 143

def start
  trace_point_lock.synchronize do
    # If this code tracker is already running, we can do nothing or
    # restart it (by disabling the trace point and recreating it).
    # It is likely that some applications will attempt to activate
    # DI more than once where the intention is to just activate DI;
    # do not break such applications by clearing out the registry.
    # For now, until there is a use case for recreating the trace point,
    # do nothing if the code tracker has already started.
    return if @compiled_trace_point

    # Note: .trace enables the trace point.
    @compiled_trace_point = TracePoint.trace(:script_compiled) do |tp|
      # Useful attributes of the trace point object here:
      # .instruction_sequence
      # .instruction_sequence.path (either absolute file path for
      #   loaded or required code, or for eval'd code, if filename
      #   is specified as argument to eval, then this is the provided
      #   filename, otherwise this is a synthesized
      #   "(eval at <definition-file>:<line>)" string)
      # .instruction_sequence.absolute_path (absolute file path when
      #   load or require are used to load code, nil for eval'd code
      #   regardless of whether filename was specified as an argument
      #   to eval on ruby 3.1+, same as path for eval'd code on ruby 3.0
      #   and lower)
      # .method_id
      # .path (refers to the code location that called the require/eval/etc.,
      #   not where the loaded code is; use .path on the instruction sequence
      #   to obtain the location of the compiled code)
      # .eval_script
      #
      # For now just map the path to the instruction sequence.
      path = tp.instruction_sequence.absolute_path
      # Do not store mapping for eval'd code, since there is no way
      # to target such code from dynamic instrumentation UI.
      # eval'd code always sets tp.eval_script.
      # When tp.eval_script is nil, code is either 'load'ed or 'require'd.
      # steep, of course, complains about indexing with +path+
      # without checking that it is not nil, so here, maybe there is
      # some situation where path would in fact be nil and
      # steep would end up saving the day.
      if path && !tp.eval_script
        registry_lock.synchronize do
          registry[path] = tp.instruction_sequence
        end

        # Also, pending line probes should only be installed for
        # non-eval'd code.
        DI.current_component&.probe_manager&.install_pending_line_probes(path)
      end
    # Since this method normally is called from customer applications,
    # rescue any exceptions that might not be handled to not break said
    # customer applications.
    rescue Exception => exc # standard:disable Lint/RescueException
      Datadog::DI.reraise_if_fatal(exc)
      # Code tracker may be loaded without the rest of DI,
      # in which case DI.component will not yet be defined,
      # but we will have DI.current_component (set to nil).
      if component = DI.current_component
        raise if component.settings.dynamic_instrumentation.internal.propagate_all_exceptions
        component.logger.debug { "di: unhandled exception in script_compiled trace point: #{exc.class}: #{exc.message}" }
        component.telemetry&.report(exc, description: "Unhandled exception in script_compiled trace point")
        # TODO test this path
      else
        # If we don't have a component, we cannot log anything properly.
        # Do not just print a warning to avoid spamming customer logs.
        # Don't reraise the exception either.
        # TODO test this path
      end
    end

    # Backfill the registry with iseqs for files that were loaded
    # before tracking started. This must happen after the trace
    # point is enabled so that any files loaded concurrently are
    # captured by the trace point (backfill won't overwrite them).
    backfill_registry
  end
end

#stop ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Stops tracking code that is being loaded.

This method should ordinarily never be called - if a file is loaded when code tracking is not active, this file will not be instrumentable by line probes.

This method is intended for test suite use only, where multiple code tracker instances are created, to fully clean up the old instances.



339
340
341
342
343
344
345
346
347
348
 
# File 'lib/datadog/di/code_tracker.rb', line 339

def stop
  # Permit multiple stop calls.
  trace_point_lock.synchronize do
    @compiled_trace_point&.disable
    # Clear the instance variable so that the trace point may be
    # reinstated in the future.
    @compiled_trace_point = nil
  end
  clear
end