Class: Sumologic::Search::RecordFetcher

Inherits:
Object
  • Object
show all
Defined in:
lib/sumologic/search/record_fetcher.rb

Overview

Fetches aggregation records (count by, group by results) with automatic pagination Uses Worker utility for concurrent page fetching when beneficial

Constant Summary collapse

PAGE_SIZE = 
10_000

Instance Method Summary collapse

Constructor Details

#initialize(http_client:, config:) ⇒ RecordFetcher

Returns a new instance of RecordFetcher.



12
13
14
15
16
17
18
19
 
# File 'lib/sumologic/search/record_fetcher.rb', line 12

def initialize(http_client:, config:)
  @http = http_client
  @config = config
  @worker = Utils::Worker.new(
    max_threads: @config.max_workers,
    request_delay: @config.request_delay
  )
end

Instance Method Details

#fetch_all(job_id, limit: nil) ⇒ Object

Fetch all records for a job with automatic pagination Used for aggregation queries (count by, group by, etc.) Single page: fetches directly Multiple pages: uses Worker for concurrent fetching



25
26
27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/sumologic/search/record_fetcher.rb', line 25

def fetch_all(job_id, limit: nil)
  # Fetch first page to check size
  first_batch_limit = calculate_batch_limit(limit, 0)
  return [] if first_batch_limit <= 0

  first_batch = fetch_page(job_id, 0, first_batch_limit)
  return [] if first_batch.empty?

  # Single page result? Return immediately
  return first_batch if first_batch.size < first_batch_limit || (limit && first_batch.size >= limit)

  # Multi-page result: calculate remaining pages and fetch in parallel
  fetch_all_pages(job_id, first_batch, limit)
end