Class: SidekiqIteration::CsvEnumerator

Inherits:
Object
  • Object
show all
Defined in:
lib/sidekiq_iteration/csv_enumerator.rb

Overview

CsvEnumerator makes it possible to write an Iteration job that uses CSV file as a collection to iterate.

Examples:

Enumerator to iterate on rows

def build_enumerator(cursor:)
  csv = CSV.open('tmp/files', { converters: :integer, headers: true })
  csv_enumerator(csv, cursor: cursor)
end

def each_iteration(row)
  ...
end

Enumerator to iterate on batches of rows

def build_enumerator(cursor:)
  csv = CSV.open('tmp/files', { converters: :integer, headers: true })
  csv_batches_enumerator(csv, cursor: cursor)
end

def each_iteration(row)
  ...
end

Instance Method Summary collapse

Constructor Details

#initialize(csv) ⇒ SidekiqIteration::CsvEnumerator

Constructs CsvEnumerator instance based on a CSV file.

Examples:

csv = CSV.open('tmp/files', { converters: :integer, headers: true })
SidekiqIteration::CsvEnumerator.new(csv).rows(cursor: cursor)

Parameters:

  • csv (CSV) —

    An instance of CSV object



38
39
40
41
42
43
44
 
# File 'lib/sidekiq_iteration/csv_enumerator.rb', line 38

def initialize(csv)
  unless defined?(CSV) && csv.instance_of?(CSV)
    raise ArgumentError, "CsvEnumerator.new takes CSV object"
  end

  @csv = csv
end

Instance Method Details

#batches(cursor:, batch_size: 100) ⇒ Enumerator

Constructs a enumerator on batches of CSV rows

Returns:

  • (Enumerator) —

    Enumerator instance



59
60
61
62
63
64
65
 
# File 'lib/sidekiq_iteration/csv_enumerator.rb', line 59

def batches(cursor:, batch_size: 100)
  @csv.lazy
    .each_slice(batch_size)
    .with_index
    .drop(cursor || 0)
    .to_enum { (count_of_rows_in_file.to_f / batch_size).ceil }
end

#rows(cursor:) ⇒ Enumerator

Constructs a enumerator on CSV rows

Returns:

  • (Enumerator) —

    Enumerator instance



49
50
51
52
53
54
 
# File 'lib/sidekiq_iteration/csv_enumerator.rb', line 49

def rows(cursor:)
  @csv.lazy
    .each_with_index
    .drop(cursor || 0)
    .to_enum { count_of_rows_in_file }
end