Method: AWS::Core::Collection::Simple#each_batch

Defined in:
lib/aws/core/collection/simple.rb

#each_batch(options = {}) {|batch| ... } ⇒ nil_or_next_token

Note:

If you require fixed size batches, see AWS::Core::Collection#in_groups_of.

Yields items from this collection in batches.

collection.each_batch do |batch|
  batch.each do |item|
    # ...
  end
end

Variable Batch Sizes

Each AWS service has its own rules on how it returns results. Because of this batch size may very based on:

  • Service limits (e.g. S3 limits keys to 1000 per response)

  • The size of response objects (SimpleDB limits responses to 1MB)

  • Time to process the request

Because of these variables, batch sizes may not be consistent for a single collection. Each batch represents all of the items returned in a single resopnse.

Parameters:

  • options (Hash) (defaults to: {})

Options Hash (options):

  • :limit (Integer) — default: nil —

    The maximum number of items to enumerate from this collection.

  • :next_token (next_token) — default: nil —

    Acts as an offset. :next_token may be returned by AWS::Core::Collection#each and #each_batch when a :limit is provided.

Yields:

  • (batch)

Returns:

  • (nil_or_next_token) —

    Returns nil if all items were enumerated. If some items were excluded because of a :limit option then a next_token is returned. Calling an enumerable method on the same collection with the next_token acts like an offset.



39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
 
# File 'lib/aws/core/collection/simple.rb', line 39

def each_batch options = {}, &block

  each_opts  = options.dup
  limit      = each_opts.delete(:limit)
  limit      = limit.to_i if limit
  next_token = each_opts.delete(:next_token)
  offset     = next_token ? next_token.to_i - 1 : 0
  total      = 0

  nil_or_next_token = nil

  batch = []
  _each_item(each_opts.dup) do |item|

    total += 1

    # skip until we reach our offset (derived from the "next token")
    next if total <= offset

    if limit

      if batch.size < limit
        batch << item
      else
        # allow _each_item to yield one more item than needed
        # so we can determine if we should return a "next token"
        nil_or_next_token = total
        break
      end

    else
      batch << item
    end

  end

  yield(batch)

  nil_or_next_token

end