Class: Dynamoid::Criteria::Chain

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/dynamoid/criteria/chain.rb

Overview

The criteria chain is equivalent to an ActiveRecord relation (and realistically I should change the name from chain to relation). It is a chainable object that builds up a query and eventually executes it by a Query or Scan.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source) ⇒ Chain

Create a new criteria chain.

Parameters:

  • source (Class)

    the class upon which the ultimate query will be performed.


19
20
21
22
23
24
25
26
27
# File 'lib/dynamoid/criteria/chain.rb', line 19

def initialize(source)
  @query = {}
  @source = source
  @consistent_read = false
  @scan_index_forward = true

  # we should re-initialize keys detector every time we change query
  @key_fields_detector = KeyFieldsDetector.new(@query, @source)
end

Instance Attribute Details

#consistent_readObject (readonly)

Returns the value of attribute consistent_read


13
14
15
# File 'lib/dynamoid/criteria/chain.rb', line 13

def consistent_read
  @consistent_read
end

#key_fields_detectorObject (readonly)

Returns the value of attribute key_fields_detector


13
14
15
# File 'lib/dynamoid/criteria/chain.rb', line 13

def key_fields_detector
  @key_fields_detector
end

#queryObject (readonly)

Returns the value of attribute query


13
14
15
# File 'lib/dynamoid/criteria/chain.rb', line 13

def query
  @query
end

#sourceObject (readonly)

Returns the value of attribute source


13
14
15
# File 'lib/dynamoid/criteria/chain.rb', line 13

def source
  @source
end

Instance Method Details

#allEnumerator::Lazy

Returns all the records matching the criteria.

Since where and most of the other methods return a Chain the only way to get a result as a collection is to call the all method. It returns Enumerator which could be used directly or transformed into Array

Post.all                            # => Enumerator
Post.where(links_count: 2).all      # => Enumerator
Post.where(links_count: 2).all.to_a # => Array

When the result set is too large DynamoDB divides it into separate pages. While an enumerator iterates over the result models each page is loaded lazily. So even an extra large result set can be loaded and processed with considerably small memory footprint and throughput consumption.

Returns:

  • (Enumerator::Lazy)

Since:

  • 0.2.0


149
150
151
# File 'lib/dynamoid/criteria/chain.rb', line 149

def all
  records
end

#batch(batch_size) ⇒ Dynamoid::Criteria::Chain

Set the batch size.

The batch size is a number of items which will be lazily loaded one by one. When the batch size is set then items will be loaded batch by batch of the specified size instead of relying on the default paging mechanism of DynamoDB.

Post.where(links_count: 2).batch(1000).all.each do |post|
  # process a post
end

It's useful to limit memory usage or throughput consumption


310
311
312
313
# File 'lib/dynamoid/criteria/chain.rb', line 310

def batch(batch_size)
  @batch_size = batch_size
  self
end

#consistentDynamoid::Criteria::Chain

Turns on strongly consistent reads.

By default reads are eventually consistent.

Post.where('size.gt' => 1000).consistent

125
126
127
128
# File 'lib/dynamoid/criteria/chain.rb', line 125

def consistent
  @consistent_read = true
  self
end

#countInteger

Returns the actual number of items in a table matching the criteria.

Post.where(links_count: 2).count

Internally it uses either `Scan` or `Query` DynamoDB's operation so it costs like all the matching items were read from a table.

The only difference is that items are read by DynemoDB but not actually loaded on the client side. DynamoDB returns only count of items after filtering.

Returns:

  • (Integer)

165
166
167
168
169
170
171
# File 'lib/dynamoid/criteria/chain.rb', line 165

def count
  if @key_fields_detector.key_present?
    count_via_query
  else
    count_via_scan
  end
end

#delete_allObject Also known as: destroy_all

Deletes all the items matching the criteria.

Post.where(links_count: 2).delete_all

If called without criteria then it deletes all the items in a table.

Post.delete_all

It loads all the items either with Scan or Query operation and deletes them in batch with BatchWriteItem operation. BatchWriteItem is limited by request size and items count so it's quite possible the deletion will require several BatchWriteItem calls.


228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
# File 'lib/dynamoid/criteria/chain.rb', line 228

def delete_all
  ids = []
  ranges = []

  if @key_fields_detector.key_present?
    Dynamoid.adapter.query(source.table_name, range_query).flat_map { |i| i }.collect do |hash|
      ids << hash[source.hash_key.to_sym]
      ranges << hash[source.range_key.to_sym] if source.range_key
    end
  else
    Dynamoid.adapter.scan(source.table_name, scan_query, scan_opts).flat_map { |i| i }.collect do |hash|
      ids << hash[source.hash_key.to_sym]
      ranges << hash[source.range_key.to_sym] if source.range_key
    end
  end

  Dynamoid.adapter.delete(source.table_name, ids, range_key: ranges.presence)
end

#each(&block) ⇒ Object

Allows to use the results of a search as an enumerable over the results found.

Post.each do |post|
end

Post.all.each do |post|
end

Post.where(links_count: 2).each do |post|
end

It works similar to the all method so results are loaded lazily.

Since:

  • 0.2.0


406
407
408
# File 'lib/dynamoid/criteria/chain.rb', line 406

def each(&block)
  records.each(&block)
end

#find_by_pages(&block) ⇒ Enumerator::Lazy

Iterates over the pages returned by DynamoDB.

DynamoDB has its own paging machanism and divides a large result set into separate pages. The find_by_pages method provides access to these native DynamoDB pages.

The pages are loaded lazily.

Post.where('views_count.gt' => 1000).find_by_pages do |posts, options|
  # process posts
end

It passes as block argument an Array of models and a Hash with options.

Options Hash contains only one option :last_evaluated_key. The last evaluated key is a Hash with key attributes of the last item processed by DynamoDB. It can be used to resume querying using the start method.

posts, options = Post.where('views_count.gt' => 1000).find_by_pages.first
last_key = options[:last_evaluated_key]

# ...

Post.where('views_count.gt' => 1000).start(last_key).find_by_pages do |posts, options|
end

If it's called without a block then it returns an Enumerator.

enum = Post.where('views_count.gt' => 1000).find_by_pages

enum.each do |posts, options|
  # process posts
end

Returns:

  • (Enumerator::Lazy)

445
446
447
# File 'lib/dynamoid/criteria/chain.rb', line 445

def find_by_pages(&block)
  pages.each(&block)
end

#first(*args) ⇒ Model|nil

Returns the first item matching the criteria.

Post.where(links_count: 2).first

Applies `record_limit(1)` to ensure only a single record is fetched when no non-key conditions are present and `scan_limit(1)` when no conditions are present at all.

If used without criteria it just returns the first item of some arbitrary order.

Post.first

Returns:

  • (Model|nil)

187
188
189
190
191
192
193
194
# File 'lib/dynamoid/criteria/chain.rb', line 187

def first(*args)
  n = args.first || 1

  return scan_limit(n).to_a.first(*args) if @query.blank?
  return super if @key_fields_detector.non_key_present?

  record_limit(n).to_a.first(*args)
end

#lastModel|nil

Returns the last item matching the criteria.

Post.where(links_count: 2).last

DynamoDB doesn't support ordering by some arbitrary attribute except a sort key. So this method is mostly useful during development and testing.

If used without criteria it just returns the last item of some arbitrary order.

Post.last

It isn't efficient from the performance point of view as far as it reads and loads all the filtered items from DynamoDB.

Returns:

  • (Model|nil)

212
213
214
# File 'lib/dynamoid/criteria/chain.rb', line 212

def last
  all.to_a.last
end

#pluck(*args) ⇒ Array

Select only specified fields.

It takes one or more field names and returns an array of either values or arrays of values.

Post.pluck(:id)                   # => ['1', '2']
Post.pluck(:title, :title)        # => [['1', 'Title #1'], ['2', 'Title#2']]

Post.where('views_count.gt' => 1000).pluck(:title)

There are some differences between pluck and project. pluck

  • doesn't instantiate models

  • it isn't chainable and returns Array instead of Chain

It deserializes values if a field type isn't supported by DynamoDB natively.

It can be used to avoid loading large field values and to decrease a memory footprint.

Returns:

  • (Array)

487
488
489
490
491
492
493
494
495
496
497
498
499
# File 'lib/dynamoid/criteria/chain.rb', line 487

def pluck(*args)
  fields = args.map(&:to_sym)
  @project = fields

  if fields.many?
    items.map do |item|
      fields.map { |key| Undumping.undump_field(item[key], source.attributes[key]) }
    end.to_a
  else
    key = fields.first
    items.map { |item| Undumping.undump_field(item[key], source.attributes[key]) }.to_a
  end
end

#project(*fields) ⇒ Dynamoid::Criteria::Chain

Select only specified fields.

It takes one or more field names and returns a collection of models with only these fields set.

Post.where('views_count.gt' => 1000).select(:title)
Post.where('views_count.gt' => 1000).select(:title, :created_at)
Post.select(:id)

It can be used to avoid loading large field values and to decrease a memory footprint.


462
463
464
465
# File 'lib/dynamoid/criteria/chain.rb', line 462

def project(*fields)
  @project = fields.map(&:to_sym)
  self
end

#record_limit(limit) ⇒ Dynamoid::Criteria::Chain

Set the record limit.

The record limit is the limit of evaluated items returned by the Query or Scan. In other words it's how many items should be returned in response.

Post.where(links_count: 2).record_limit(1000) # => 1000 models
Post.record_limit(1000)                       # => 1000 models

It could be very inefficient in terms of HTTP requests in pathological cases. DynamoDB doesn't support out of the box the limits for items count after filtering. So it's possible to make a lot of HTTP requests to find items matching criteria and skip not matching. It means that the cost (read capacity units) is unpredictable.

Because of such issues with performance and cost it's mostly useful in development and testing.

When called without criteria it works like scan_limit.


269
270
271
272
# File 'lib/dynamoid/criteria/chain.rb', line 269

def record_limit(limit)
  @record_limit = limit
  self
end

#scan_index_forward(scan_index_forward) ⇒ Dynamoid::Criteria::Chain

Reverse the sort order.

By default the sort order is ascending (by the sort key value). Set a false value to reverse the order.

Post.where(id: id, 'views_count.gt' => 1000).scan_index_forward(false)

It works only for queries with a partition key condition e.g. id: 'some-id' which internally performs Query operation.


356
357
358
359
# File 'lib/dynamoid/criteria/chain.rb', line 356

def scan_index_forward(scan_index_forward)
  @scan_index_forward = scan_index_forward
  self
end

#scan_limit(limit) ⇒ Dynamoid::Criteria::Chain

Set the scan limit.

The scan limit is the limit of records that DynamoDB will internally read with Query or Scan. It's different from the record limit as with filtering DynamoDB may look at N scanned items but return 0 items if none passes the filter. So it can return less items than was specified with the limit.

Post.where(links_count: 2).scan_limit(1000)   # => 850 models
Post.scan_limit(1000)                         # => 1000 models

By contrast with record_limit the cost (read capacity units) and performance is predictable.

When called without criteria it works like record_limit.


291
292
293
294
# File 'lib/dynamoid/criteria/chain.rb', line 291

def scan_limit(limit)
  @scan_limit = limit
  self
end

#start(start) ⇒ Dynamoid::Criteria::Chain

Set the start item.

When the start item is set the items will be loaded starting right after the specified item.

Post.where(links_count: 2).start(post)

It can be used to implement an own pagination mechanism.

Post.where(author_id: author_id).start(last_post).scan_limit(50)

The specified start item will not be returned back in a result set.

Actually it doesn't need all the item attributes to start - an item may have only the primary key attributes (partition and sort key if it's declared).

Post.where(links_count: 2).start(Post.new(id: id))

It also supports a Hash argument with the keys attributes - a partition key and a sort key (if it's declared).

Post.where(links_count: 2).start(id: id)

340
341
342
343
# File 'lib/dynamoid/criteria/chain.rb', line 340

def start(start)
  @start = start
  self
end

#where(args) ⇒ Dynamoid::Criteria::Chain

Returns a chain which is a result of filtering current chain with the specified conditions.

It accepts conditions in the form of a hash.

Post.where(links_count: 2)

A key could be either string or symbol.

In order to express conditions other than equality predicates could be used. Predicate should be added to an attribute name to form a key 'created_at.gt' => Date.yesterday

Currently supported following predicates:

  • gt - greater than

  • gte - greater or equal

  • lt - less than

  • lte - less or equal

  • ne - not equal

  • between - an attribute value is greater than the first value and less than the second value

  • in - check an attribute in a list of values

  • begins_with - check for a prefix in string

  • contains - check substring or value in a set or array

  • not_contains - check for absence of substring or a value in set or array

  • null - attribute doesn't exists in an item

  • not_null - attribute exists in an item

All the predicates match operators supported by DynamoDB's ComparisonOperator

Post.where('size.gt' => 1000)
Post.where('size.gte' => 1000)
Post.where('size.lt' => 35000)
Post.where('size.lte' => 35000)
Post.where('author.ne' => 'John Doe')
Post.where('created_at.between' => [Time.now - 3600, Time.now])
Post.where('category.in' => ['tech', 'fashion'])
Post.where('title.begins_with' => 'How long')
Post.where('tags.contains' => 'Ruby')
Post.where('tags.not_contains' => 'Ruby on Rails')
Post.where('legacy_attribute.null' => true)
Post.where('optional_attribute.not_null' => true)

There are some limitations for a sort key. Only following predicates are supported - gt, gte, lt, lte, between, begins_with.

where without argument will return the current chain.

Multiple calls can be chained together and conditions will be merged:

Post.where('size.gt' => 1000).where('title' => 'some title')

It's equivalent to:

Post.where('size.gt' => 1000, 'title' => 'some title')

But only one condition can be specified for a certain attribute. The last specified condition will override all the others. Only condition 'size.lt' => 200 will be used in following examples:

Post.where('size.gt' => 100, 'size.lt' => 200)
Post.where('size.gt' => 100).where('size.lt' => 200)

Internally where performs either Scan or Query operation.

Returns:

Since:

  • 0.2.0


94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
# File 'lib/dynamoid/criteria/chain.rb', line 94

def where(args)
  detector = IgnoredConditionsDetector.new(args)
  if detector.found?
    Dynamoid.logger.warn(detector.warning_message)
  end

  detector = OverwrittenConditionsDetector.new(@query, args)
  if detector.found?
    Dynamoid.logger.warn(detector.warning_message)
  end

  detector = NonexistentFieldsDetector.new(args, @source)
  if detector.found?
    Dynamoid.logger.warn(detector.warning_message)
  end

  query.update(args.symbolize_keys)

  # we should re-initialize keys detector every time we change query
  @key_fields_detector = KeyFieldsDetector.new(@query, @source, forced_index_name: @forced_index_name)

  self
end

#with_index(index_name) ⇒ Dynamoid::Criteria::Chain

Force the index name to use for queries.

By default allows the library to select the most appropriate index. Sometimes you have more than one index which will fulfill your query's needs. When this case occurs you may want to force an order. This occurs when you are searching by hash key, but not specifying a range key.

class Comment
  include Dynamoid::Document

  table key: :post_id
  range_key :author_id

  field :post_date, :datetime

  global_secondary_index name: :time_sorted_comments, hash_key: :post_id, range_key: , projected_attributes: :all
end

 Comment.where(post_id: id).with_index(:time_sorted_comments).scan_index_forward(false)

383
384
385
386
387
388
389
# File 'lib/dynamoid/criteria/chain.rb', line 383

def with_index(index_name)
  raise Dynamoid::Errors::InvalidIndex, "Unknown index #{index_name}" unless @source.find_index_by_name(index_name)

  @forced_index_name = index_name
  @key_fields_detector = KeyFieldsDetector.new(@query, @source, forced_index_name: index_name)
  self
end