Class: Datadog::SymbolDatabase::Component Private

Inherits:
Object
  • Object
show all
Defined in:
lib/datadog/symbol_database/component.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

Main coordinator for symbol database upload functionality.

Responsibilities:

  • Lifecycle management: Initialization, shutdown, upload triggering
  • Coordination: Connects Extractor → ScopeBatcher → Uploader
  • Remote config handling: start_upload called by Remote module on config changes
  • Debounce: extraction is deferred by EXTRACT_DEBOUNCE_INTERVAL seconds so reconfigurations during boot coalesce into a single extraction on the final Component instance.
  • Hot-load coverage: TracePoint :class hook captures classes loaded after initial extraction, enqueues them on a per-instance buffer; the scheduler drains the buffer on debounce and extracts each one via Extractor#extract, matching Java/Python/.NET continuous coverage.

Upload flow:

  1. Remote config sends upload_symbols: true (or force_upload mode)
  2. start_upload called — schedules extraction EXTRACT_DEBOUNCE_INTERVAL seconds in the future on a per-instance scheduler thread, and lazily installs the TracePoint :class hook if not already installed.
  3. When the timer fires (no further start_upload calls reset it), extract_and_upload runs. On the first call: ObjectSpace iteration → Extractor#extract_all. On subsequent calls: drain the hot-load buffer → Extractor#extract per module.
  4. ScopeBatcher batches and triggers Uploader.
  5. As new classes load throughout the process lifetime, the TracePoint hook fires and signals the scheduler — the next debounce window produces an incremental upload of just the new classes.

Created by: Components#initialize (in Core::Configuration::Components) Accessed by: Remote config receiver via Datadog.send(:components).symbol_database Requires: Remote config enabled (unless force mode)

API:

  • private

Constant Summary collapse

EXTRACT_DEBOUNCE_INTERVAL =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Debounce window for extraction. Multiple start_upload calls within this window coalesce; the timer fires once after the window of inactivity. Long enough to absorb reconfiguration cascades during Rails boot.

API:

  • private

5

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(settings, agent_settings, logger, telemetry: nil, di_active: nil) ⇒ Component

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Initialize component.

Parameters:

  • Tracer settings

  • Agent configuration

  • Logger instance

  • (defaults to: nil)

    Telemetry component for error reporting

  • (defaults to: nil)

    Predicate returning whether Dynamic Instrumentation is currently active

API:

  • private



102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/datadog/symbol_database/component.rb', line 102

def initialize(settings, agent_settings, logger, telemetry: nil, di_active: nil)
  @settings = settings
  @agent_settings = agent_settings
  @logger = logger
  @telemetry = telemetry
  @di_active = di_active

  @extractor = Extractor.new(logger: logger, settings: settings)
  @uploader = Uploader.new(settings: settings, agent_settings: agent_settings, logger: logger, telemetry: telemetry)
  @scope_batcher = ScopeBatcher.new(@uploader, logger: logger)

  @last_upload_time = nil
  @last_upload_scope_count = nil
  @mutex = Mutex.new
  @upload_in_progress = false
  @upload_in_progress_cv = ConditionVariable.new
  @shutdown = false
  # PID at construction time. Compared against Process.pid in shutdown!
  # to detect forked-child callers, whose inherited @upload_in_progress
  # snapshot is stale: the scheduler thread that would clear it lives
  # only in the parent. See shutdown! for details.
  @owner_pid = Process.pid

  # Signalled when @last_upload_time advances. wait_for_idle blocks on this
  # so short-lived scripts that trigger an upload can wait for an upload
  # attempt to complete without depending on a one-shot flag.
  @last_upload_time_cv = ConditionVariable.new

  # Per-instance scheduler state. The scheduler thread is started lazily
  # on the first start_upload call.
  @scheduler_mutex = Mutex.new
  @scheduler_cv = ConditionVariable.new
  @scheduled_at = nil
  @scheduler_signaled = false
  @scheduler_thread = nil

  # Hot-load coverage state. TracePoint :class hook is installed lazily on
  # the first start_upload call; classes defined after that point are
  # enqueued here and drained by the scheduler on debounce. Distinguishes
  # initial extraction (extract_all) from incremental (per-module extract).
  @hot_load_buffer = []
  @hot_load_buffer_mutex = Mutex.new
  @hot_load_tracepoint = nil
  @initial_extraction_done = false

  # Sticky record of "remote config (or force mode) wants symbols
  # uploaded", independent of whether DI is currently active. Set when an
  # upload is requested and either allowed or deferred by the DI gate;
  # cleared only when RC explicitly disables uploads (stop_upload). It
  # survives stop_for_di_disable's scheduler teardown, so
  # resume_pending_upload can restart uploads after a DI disable->re-enable
  # cycle even though RC does not re-dispatch the unchanged symbol-database
  # config.
  @upload_requested = false
end

Instance Attribute Details

#last_upload_scope_count ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

API:

  • private



94
95
96
 
# File 'lib/datadog/symbol_database/component.rb', line 94

def last_upload_scope_count
  @last_upload_scope_count
end

#last_upload_time ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

API:

  • private



94
95
96
 
# File 'lib/datadog/symbol_database/component.rb', line 94

def last_upload_time
  @last_upload_time
end

#logger ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

API:

  • private



94
95
96
 
# File 'lib/datadog/symbol_database/component.rb', line 94

def logger
  @logger
end

#settings ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

API:

  • private



94
95
96
 
# File 'lib/datadog/symbol_database/component.rb', line 94

def settings
  @settings
end

#upload_in_progress ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

API:

  • private



94
95
96
 
# File 'lib/datadog/symbol_database/component.rb', line 94

def upload_in_progress
  @upload_in_progress
end

Class Method Details

.build(settings, agent_settings, logger, telemetry: nil, di_active: nil) ⇒ Component?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Build a new Component if the runtime supports it and dependencies are met. The caller (Core::Configuration::Components) decides whether the feature is enabled and only invokes build when it is, so a disabled component is never constructed. This method gates only on symbol database's own requirements (supported platform, remote config availability).

Parameters:

  • Tracer settings

  • Agent configuration

  • Logger instance

  • (defaults to: nil)

    Telemetry component for error reporting

  • (defaults to: nil)

    Predicate returning whether Dynamic Instrumentation is currently active (started). Gates remote-config uploads in the nil-default case; nil means "no gate" (standalone force_upload contexts).

Returns:

  • Component instance or nil if requirements not met

API:

  • private



73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/datadog/symbol_database/component.rb', line 73

def self.build(settings, agent_settings, logger, telemetry: nil, di_active: nil)
  symdb_logger = SymbolDatabase::Logger.new(settings, logger)

  # Symbol database requires MRI Ruby 2.7+.
  # Configuration accessors (settings.symbol_database.*) remain available on all
  # platforms — only the component (upload) is disabled on unsupported engines/versions.
  # environment_supported? logs the specific reason (engine or version) internally.
  return nil unless environment_supported?(symdb_logger)

  # Requires remote config (unless force mode)
  if !settings.remote&.enabled && !settings.symbol_database.internal.force_upload
    symdb_logger.debug("symdb: remote config not available and force_upload not set, skipping")
    return nil
  end

  new(settings, agent_settings, symdb_logger, telemetry: telemetry, di_active: di_active).tap do |component|
    # Defer extraction if force upload mode — wait for app boot to complete
    component.schedule_deferred_upload if settings.symbol_database.internal.force_upload
  end
end

Instance Method Details

#after_fork! ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Reinitialize per-instance state in a forked child process.

Process.fork copies the parent's memory but only the forking thread survives in the child. Background threads (@scheduler_thread) are dead, mutexes and condition variables are copied without owner tracking (orphan-lock risk if the parent held a mutex at the fork instant), and the TracePoint hook is bound to the dead scheduler.

State reset (the child does its own initial extraction, then hot-load continues from there):

  • Hot-load buffer cleared — the child will rediscover via extract_all.
  • @initial_extraction_done = false — child has not extracted yet.
  • @hot_load_tracepoint = nil — start_upload reinstalls a fresh one bound to the child's component.
  • @scheduler_thread = nil, @scheduled_at = nil, @scheduler_signaled = false — scheduler restarts on next start_upload.
  • @upload_in_progress = false — parent may have been mid-upload at the fork instant; the child has no upload in flight.
  • @scope_batcher replaced with a fresh instance. The inherited batcher carries the parent's @uploaded_modules set, which add_scope uses to dedup by scope name. Without a fresh batcher, the child's re-extraction silently drops every scope whose name the parent already uploaded — under preload_app! that's most of the app.

Mutex/CV reinit (orphan-lock guard):

  • @scheduler_mutex, @scheduler_cv, @mutex, @upload_in_progress_cv, @last_upload_time_cv, @hot_load_buffer_mutex.

Force-upload mode: the parent's scheduled extraction is dead in the child, so re-register the deferred-upload callback. In Rails the :after_initialize hook has already fired (initialization happened in the parent), so the on_load block runs immediately and the child schedules its own upload. In non-Rails, this calls start_upload directly.

Cross-process upload deduplication is intentionally not handled here. Each forked Component does its own initial extraction. Workers in preload_app! + eager_load=true deployments hold identical code to the parent — backend dedup of identical-content uploads is the backend's responsibility, not the tracer's.

API:

  • private



402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
 
# File 'lib/datadog/symbol_database/component.rb', line 402

def after_fork!
  # Disable the inherited TracePoint before dropping the reference: fork
  # copies the enabled TP into the child, where it remains rooted by the
  # VM. Without an explicit disable, every subsequent class load in the
  # child would enqueue through the inherited hook in addition to the
  # fresh hook that start_upload installs.
  @hot_load_tracepoint&.disable
  @hot_load_buffer = []
  @hot_load_buffer_mutex = Mutex.new
  @hot_load_tracepoint = nil
  @initial_extraction_done = false

  @scheduler_mutex = Mutex.new
  @scheduler_cv = ConditionVariable.new
  @scheduled_at = nil
  @scheduler_signaled = false
  @scheduler_thread = nil

  @mutex = Mutex.new
  @upload_in_progress = false
  @upload_in_progress_cv = ConditionVariable.new
  @last_upload_time_cv = ConditionVariable.new

  # Fresh ScopeBatcher: the inherited one carries the parent's
  # @uploaded_modules set, against which add_scope dedups by name.
  @scope_batcher = ScopeBatcher.new(@uploader, logger: @logger)

  schedule_deferred_upload if @settings.symbol_database.internal.force_upload
end

#resume_pending_upload ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Re-attempt a symbol upload that remote config requested but that is not currently running because Dynamic Instrumentation was inactive — either deferred at request time, or suspended by stop_for_di_disable when DI was turned off. Called from the orchestration layer (Tracing::Remote) when DI is enabled via remote configuration (implicit enablement). No-op unless an upload was requested and not since disabled. Mirrors DI's replay_current_probes: RC does not re-dispatch the unchanged symbol-database config on DI re-enable, so the tracer restarts the upload from its own retained desire.

API:

  • private



265
266
267
268
269
 
# File 'lib/datadog/symbol_database/component.rb', line 265

def resume_pending_upload
  requested = @scheduler_mutex.synchronize { @upload_requested }
  start_upload if requested
  nil
end

#schedule_deferred_upload ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Schedule a deferred upload that waits for app boot to complete.

In Rails: registers ActiveSupport.on_load(:after_initialize). When the hook has already fired (e.g., this Component was built by a reconfigure after Rails finished initializing), the callback runs immediately.

In non-Rails: triggers start_upload immediately.

Each Component registers its own callback. Old Components that have been shut down short-circuit in start_upload via @shutdown. The hot-load hook handles classes loaded after this initial trigger, so under eager_load=false an under-extracted initial upload self-corrects as the app exercises code.

API:

  • private



173
174
175
176
177
178
179
180
181
182
183
184
185
 
# File 'lib/datadog/symbol_database/component.rb', line 173

def schedule_deferred_upload
  if defined?(::ActiveSupport) && defined?(::Rails::Railtie)
    # Capture self — on_load runs the block via instance_exec on the
    # loaded object (Rails::Application), so a bare `start_upload`
    # would resolve against it.
    component = self
    ::ActiveSupport.on_load(:after_initialize) do
      component.start_upload
    end
  else
    start_upload
  end
end

#shutdown! ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Shutdown component and cleanup resources. Disables the hot-load TracePoint so no events queue for a dead scheduler. Cancels the per-instance scheduler so any pending debounced extraction is dropped. Waits for an in-flight extraction to complete before returning. Does not touch any sibling Components, so a sibling Component built after shutdown can still upload. The TracePoint teardown sits inside the same @scheduler_mutex critical section as the @shutdown flag flip, so it is atomic against a concurrent start_upload (which installs the TracePoint under @scheduler_mutex). Without that, a shutdown interleaved with a start could leave an enabled TracePoint rooted by the VM — class loads would keep growing @hot_load_buffer for the rest of the process lifetime (enqueue_hot_load's @shutdown check skips re-scheduling but only after the buffer push).

API:

  • private



322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
 
# File 'lib/datadog/symbol_database/component.rb', line 322

def shutdown!
  @scheduler_mutex.synchronize do
    @hot_load_tracepoint&.disable
    @hot_load_tracepoint = nil
    @shutdown = true
    @scheduler_signaled = true
    @scheduler_cv.signal
  end
  @scheduler_thread&.join(5)
  @scheduler_thread = nil

  @mutex.synchronize do
    if @upload_in_progress
      if Process.pid == @owner_pid
        @upload_in_progress_cv.wait(@mutex, 5)
      else
        # We are in a forked child that inherited this Component but
        # never called start_upload here. The scheduler thread (the
        # only writer that clears @upload_in_progress and signals the
        # cv) lives only in the parent — fork carries only the calling
        # thread, so nothing in this process can ever signal us.
        # Waiting would burn the full 5s timeout for no benefit. Treat
        # the inherited @upload_in_progress as a stale snapshot and
        # proceed; the parent's shutdown! (running in the parent) is
        # authoritative. Child-owned uploads (where start_upload was
        # called in this process) take the PID-match branch above,
        # because start_upload claims @owner_pid for the current
        # process.
        @upload_in_progress = false
      end
    end
  end

  @scope_batcher.shutdown
end

#shutdown? ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Whether this component has been shut down.

Returns:

API:

  • private



189
190
191
 
# File 'lib/datadog/symbol_database/component.rb', line 189

def shutdown?
  @scheduler_mutex.synchronize { @shutdown }
end

#start_upload ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Schedule symbol upload (triggered by remote config or force mode). The actual extraction is debounced by EXTRACT_DEBOUNCE_INTERVAL seconds — subsequent calls within the window restart the timer. Thread-safe: can be called concurrently from multiple remote config updates.

API:

  • private



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
 
# File 'lib/datadog/symbol_database/component.rb', line 198

def start_upload
  @scheduler_mutex.synchronize do
    return if @shutdown

    unless upload_allowed?
      if deferred_by_di_gate?
        # nil-default case: Symbol Database mirrors Dynamic Instrumentation
        # and DI is not active. Record the desire and defer; resume_pending_upload
        # re-attempts when DI is enabled. Without this gate the tracer would
        # extract and upload symbols for applications that never enabled DI.
        @upload_requested = true
        @logger.debug("symdb: upload requested but Dynamic Instrumentation is not active; deferring until DI is enabled")
      else
        # Explicit symbol_database.enabled = false: the feature is disabled,
        # not merely waiting on DI. Clear the desire so resume_pending_upload
        # does not retry a disabled feature.
        @upload_requested = false
        @logger.debug("symdb: upload requested but symbol database upload is disabled; skipping")
      end
      return
    end
    @upload_requested = true

    if @owner_pid != Process.pid
      # Forked child: claim ownership and clear inherited
      # @upload_in_progress. The inherited flag was the parent's
      # snapshot; the parent's scheduler thread does not exist in this
      # process. Any upload starting now is child-owned and must be
      # waited on in shutdown! via the PID-match branch.
      @owner_pid = Process.pid
      @mutex.synchronize { @upload_in_progress = false }
    end

    install_hot_load_hook
    @scheduled_at = Datadog::Core::Utils::Time.get_time + EXTRACT_DEBOUNCE_INTERVAL
    @scheduler_signaled = true
    @scheduler_cv.signal
    ensure_scheduler_thread
  end
rescue Exception => e # standard:disable Lint/RescueException
  Datadog::DI.reraise_if_fatal(e)
  @logger.debug { "symdb: error scheduling upload: #{e.class}: #{e.message}" }
  @telemetry&.report(e, description: "symdb: error scheduling upload")
end

#stop_for_di_disable ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Stop uploading when Dynamic Instrumentation is disabled via remote configuration. Only the nil-default (follows-DI) case stops; an explicit symbol_database.enabled = true and force_upload are independent of DI and keep running. Called from the orchestration layer (Tracing::Remote) so Symbol Database's TracePoint and scheduler don't keep uploading after DI is turned off. No-op if uploads were never started.

API:

  • private



278
279
280
281
282
283
284
285
286
 
# File 'lib/datadog/symbol_database/component.rb', line 278

def stop_for_di_disable
  return if @settings.symbol_database.internal.force_upload
  return unless @settings.symbol_database.enabled.nil?

  # Suspend, don't stop: preserve @upload_requested so resume_pending_upload
  # restarts the upload when DI is re-enabled and RC never re-sends the
  # (unchanged) symbol-database config.
  suspend_scheduling
end

#stop_upload ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Stop symbol upload in response to remote config sending upload_symbols: false or deleting the config: the customer no longer wants uploads, so clear the sticky @upload_requested desire (a later resume_pending_upload must not restart it) and tear down the scheduler and hot-load hook. Thread-safe: can be called concurrently from multiple remote config updates.

API:

  • private



250
251
252
253
 
# File 'lib/datadog/symbol_database/component.rb', line 250

def stop_upload
  @scheduler_mutex.synchronize { @upload_requested = false }
  suspend_scheduling
end

#wait_for_idle(timeout: 30) ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Block until this Component finishes an extract+upload after this call, or until the timeout elapses. Used by short-lived scripts that trigger an upload via force_upload and need to wait before exiting. Tracks @last_upload_time advance — returns true once any upload attempt completes (success or failure), false on timeout.

Parameters:

  • (defaults to: 30)

    Maximum seconds to wait

Returns:

  • true if an upload completed; false on timeout

API:

  • private



295
296
297
298
299
300
301
302
303
304
305
306
 
# File 'lib/datadog/symbol_database/component.rb', line 295

def wait_for_idle(timeout: 30)
  deadline = Datadog::Core::Utils::Time.get_time + timeout
  @mutex.synchronize do
    start_time = @last_upload_time
    while @last_upload_time == start_time
      remaining = deadline - Datadog::Core::Utils::Time.get_time
      return false if remaining <= 0
      @last_upload_time_cv.wait(@mutex, remaining)
    end
  end
  true
end