Class: Datadog::SymbolDatabase::Uploader Private

Inherits:
Object
  • Object
show all
Defined in:
lib/datadog/symbol_database/uploader.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

Uploads symbol database payloads to the Datadog agent via HTTP multipart.

Handles the complete upload process:

  1. Wraps scopes in ServiceVersion (adds service/env/version metadata)
  2. Serializes to JSON
  3. Compresses with GZIP (always, ~40:1 ratio expected)
  4. Builds multipart form: event.json (metadata) + symbols_pid.json.gz (data)
  5. POSTs to agent at /symdb/v1/input via Core::Transport::HTTP No retries — single attempt. Any failure is logged at debug and discarded.

Uses Core::Transport::HTTP infrastructure (consistent with DI, Profiling, DataStreams). Headers: DD-API-KEY, Datadog-Container-ID, Datadog-Entity-ID (automatic from transport)

Called by: ScopeBatcher.perform_upload (when batch ready) Calls: Transport::HTTP for network, Zlib for compression

Constant Summary collapse

MAX_PAYLOAD_SIZE =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

50MB

50 * 1024 * 1024

Instance Method Summary collapse

Constructor Details

#initialize(settings:, agent_settings:, logger:, telemetry: nil) ⇒ Uploader

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Initialize uploader.

Parameters:

  • settings (Configuration::Settings) —

    Tracer settings (for service, env, version metadata)

  • agent_settings (Configuration::AgentSettings) —

    Agent connection settings

  • logger (Logger) —

    Logger instance

  • telemetry (Telemetry, nil) (defaults to: nil) —

    Optional telemetry component for error reporting



41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
 
# File 'lib/datadog/symbol_database/uploader.rb', line 41

def initialize(settings:, agent_settings:, logger:, telemetry: nil)
  @settings = settings
  @agent_settings = agent_settings
  @logger = logger
  @telemetry = telemetry

  @transport = Transport::HTTP.symbols(
    agent_settings: agent_settings,
    logger: @logger,
  )

  # Protects the lazy initialization of @upload_pid/@upload_id and the
  # @batch_num increment in next_upload_metadata. ScopeBatcher calls
  # upload_scopes outside its own mutex, so two threads (e.g. a size-
  # triggered flush and the timer/shutdown path) can race here.
  @metadata_mutex = Mutex.new
end

Instance Method Details

#upload_scopes(scopes) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Upload a batch of scopes to the agent. Wraps in ServiceVersion, serializes to JSON, compresses with GZIP, builds multipart form, and POSTs to /symdb/v1/input via transport. No retries — single attempt, matching Python behavior.

Parameters:

  • scopes (Array<Scope>) —

    Scopes to upload



65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
 
# File 'lib/datadog/symbol_database/uploader.rb', line 65

def upload_scopes(scopes)
  return if scopes.empty?

  upload_id, batch_num = next_upload_metadata
  json_data = build_symbol_payload(scopes, upload_id: upload_id, batch_num: batch_num)
  compressed_data = Zlib.gzip(json_data)

  # Emitted unconditionally so the rare oversized case is observable.
  @telemetry&.distribution(
    Tracing::Ext::TELEMETRY_METRICS_NAMESPACE,
    "symbol_database.payload_size",
    compressed_data.bytesize
  )

  # Symbols for very large applications (>50MB after gzip) are dropped:
  # the upload is skipped and the customer sees no autocomplete /
  # symbol probe results for those classes. Java handles the same case
  # by splitting the payload across multiple requests; we have not
  # implemented splitting here. Deferred for a post-MVP follow-up.
  if compressed_data.bytesize > MAX_PAYLOAD_SIZE
    @logger.debug { "symdb: payload too large: #{compressed_data.bytesize}/#{MAX_PAYLOAD_SIZE} bytes, skipping" }
    return
  end

  perform_http_upload(compressed_data, scopes.size, upload_id: upload_id, batch_num: batch_num)
rescue Exception => e # standard:disable Lint/RescueException
  Datadog::DI.reraise_if_fatal(e)
  @logger.debug { "symdb: upload failed: #{e.class}: #{e.message}" }
  @telemetry&.report(e, description: "symdb: upload failed")
end