Class: Ductr::SQLite::PaginatedSource

Inherits:
ETL::PaginatedSource
  • Object
show all
Defined in:
lib/ductr/sqlite/paginated_source.rb

Overview

A source control that allows to select a big number of rows by relying on pagination, registered as :paginated. Accept the :page_size option, default value is 10 000.

source :some_sqlite_database, :paginated, page_size: 4 def my_source(db, offset, limit) db[:items].offset(offset).limit(limit) end

Ensure to not select more rows than the configured page size, otherwise it will raise an InconsistentPaginationError.

Instance Method Summary collapse

Instance Method Details

#each_page { ... } ⇒ Boolean

Calls the job's method and iterate on the query result. Returns true if the page is full, false otherwise.

Yields:

  • The each block

Returns:

  • (Boolean) —

    True if the page is full, false otherwise.

Raises:

  • (InconsistentPaginationError) —

    When the query return more rows than the page size



29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
 
# File 'lib/ductr/sqlite/paginated_source.rb', line 29

def each_page(&)
  rows_count = 0

  call_method(adapter.db, @offset, page_size).each do |row|
    yield(row)
    rows_count += 1
  end

  if rows_count > page_size
    raise InconsistentPaginationError,
          "The query returned #{rows_count} rows but the page size is #{page_size} rows"
  end

  rows_count == page_size
end