Class: SisRuby::ResultEnumerable

Inherits:
TrickBag::Enumerables::BufferedEnumerable
  • Object
show all
Includes:
Enumerable, GetHelper
Defined in:
lib/sis_ruby/result_enumerable.rb

Overview

This class is an enumerator over a result set of SIS objects, as specified in a hash-like object (that is, any object that implements to_hash, such as a SisParams instance). It can be used as any Enumerable; that is, methods such as each, map, select, etc. can be called; an array of all results can be produced by calling to_a, and an Enumerator can be gotten by calling each without a chunk.

Internally it fetches chunks of records only when needed to serve the values to the caller. The chunk size can be specified by the caller.

By calling each or each_slice you can access some of the data before fetching all of it. This can be handy if:

1) there may not be enough available memory to process the entire result set

at once

2) you want to process some records while others are being fetched, by

using multiple threads, for example.

Constant Summary collapse

DEFAULT_CHUNK_RECORD_COUNT = 
5_000

Instance Method Summary collapse

Methods included from GetHelper

#create_headers, #typhoeus_get, #validate_response_success

Constructor Details

#initialize(endpoint_url, params = {}, chunk_size = DEFAULT_CHUNK_RECORD_COUNT) ⇒ ResultEnumerable

Returns a new instance of ResultEnumerable.

Parameters:

  • endpoint_url —

    the SIS endpoint URL string or an Endpoint instance

  • params (defaults to: {}) —

    any object that implements to_hash, e.g. a Params instance

  • chunk_size (defaults to: DEFAULT_CHUNK_RECORD_COUNT) —

    the maximum number of records to fetch from the server in a request



40
41
42
43
44
45
46
47
48
49
50
51
 
# File 'lib/sis_ruby/result_enumerable.rb', line 40

def initialize(endpoint_url, params = {}, chunk_size = DEFAULT_CHUNK_RECORD_COUNT)
  super(chunk_size)
  @endpoint_url = endpoint_url.is_a?(Endpoint) ? endpoint_url.url : endpoint_url
  @outer_params = params.is_a?(Params) ? params : Params.from_hash(params.to_hash)

  inner_limit = @outer_params.limit ? [chunk_size, @outer_params.limit].min : chunk_size
  @inner_params = @outer_params.clone.limit(inner_limit)

  @inner_params.offset ||= 0
  @position     = 0
  @total_count  = nil
end

Instance Method Details

#fetch ⇒ Object 



54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/sis_ruby/result_enumerable.rb', line 54

def fetch
  # This exits the fetching when the desired number of records has been fetched.
  if @outer_params.limit && @yield_count >= @outer_params.limit
    self.data = []
    return
  end

  request = Typhoeus::Request.new(@endpoint_url, params: @inner_params.to_hash, headers: create_headers(true))
  response = request.run
  validate_response_success(response)
  self.data = JSON.parse(response.body)
  @total_count = response.headers['x-total-count'].to_i
  @inner_params.offset(@inner_params.offset + chunk_size)

  # TODO: Deal with this
  # if @total_count > chunk_size && @inner_params.sort.nil?
  #   raise "Total count (#{@total_count}) exceeds chunk size (#{chunk_size})." +
  #       "When this is the case, a sort order must be specified, or chunk size increased."
  # end
end

#fetch_notify ⇒ Object 



82
83
84
 
# File 'lib/sis_ruby/result_enumerable.rb', line 82

def fetch_notify
  # puts "Fetch at #{Time.now}: chunk size: #{chunk_size}, yield count: #{yield_count}, total count = #{@total_count}"
end

#total_count ⇒ Object 



76
77
78
79
 
# File 'lib/sis_ruby/result_enumerable.rb', line 76

def total_count
  fetch unless @total_count
  @total_count
end