Class: Ductr::ETL::PaginatedSource

Inherits:
Source show all
Defined in:
lib/ductr/etl/controls/paginated_source.rb

Overview

Base class to implement paginated source.

Direct Known Subclasses

SequelBase::PaginatedSource

Instance Attribute Summary

Attributes inherited from Control

#adapter, #job_method, #options

Instance Method Summary collapse

Methods inherited from Control

#call_method, #initialize

Constructor Details

This class inherits a constructor from Ductr::ETL::Control

Instance Method Details

#each {|row| ... } ⇒ void

This method returns an undefined value.

Iterates over pages and calls #each_page.

Yields:

  • (row) —

    The row yielder



25
26
27
28
29
30
31
32
33
 
# File 'lib/ductr/etl/controls/paginated_source.rb', line 25

def each(&)
  @offset ||= 0

  loop do
    break unless each_page(&)

    @offset += page_size
  end
end

#each_page {|row| ... } ⇒ void

This method returns an undefined value.

Called once per pages.

Yields:

  • (row) —

    The row yielder

Raises:

  • (NotImplementedError) 


42
43
44
 
# File 'lib/ductr/etl/controls/paginated_source.rb', line 42

def each_page(&)
  raise NotImplementedError, "A paginated source must implement the `#each_page` method"
end

#page_size ⇒ Integer

The page size option, default to 10_000.

Returns:

  • (Integer) —

    The page size



14
15
16
 
# File 'lib/ductr/etl/controls/paginated_source.rb', line 14

def page_size
  @options[:page_size] || 10_000
end