Dynamoid Advanced Where (DAW)

Dynamoid Advanced where provides a more advanced query structure for selecting, and updating records. This is very much a work in progress and functionality is being added as it is needed.

This gem is tested against:

  • MRI 2.5, 2.6, 2.7, and 3.0-RC
  • Dynamoid 3.4, 3.5, 3.6, and git master

Installation

Add this line to your application's Gemfile:

gem 'dynamoid_advanced_where'

And then execute:

$ bundle

Upgrading

From pre 1.0

New where block format:

# Previously you had to do this to get access to certain scoped variables
local = getValue(123)
Model.where do
  field == local
end

# This is annoying, the new search block has deprecated the argument-less block, and now should be called
# with a single argument

Model.where do |r|
  r.field == getValue(123)
end

Existence checks have been changed:

# Old
Model.where{|r| r.field }

# New
Model.where{|r| r.field.exists? }

Usage

The HellowWorld usage for this app is basic search and retrieval. You can invoke DAW by calling where on a Dynamoid::Document (No relations yet) using a new block form.

class Foo
  include Dynamoid::Document

  field :bar
  field :baz
end

# Returns all records with `bar` equal to 'hello'
Foo.where{|r| r.bar == 'hello' }.all

# Advanced boolean logic is also supported

# Returns all records with `bar` equal to 'hello' and `baz` equal to 'dude'
x = Foo.where{|r| (r.baz == 'dude') & (r.bar == 'hello') }.all

Note: Those () are required, you do remember your operator precedence right?

Filtering

Filter can be applied to Queries (Searches by hash key), Scans, and update actions provided by this gem. Not all persistence actions make sense at the end of a filtering query, such as create.

Field Existence

Checks to see if a field is defined. See attribute_exists

Valid on field types: any

Example

where{|r| r.foo } or where{|r| r.foo.exists! }

Value Equality

The equality of a field can be tested using == and not equals tested using !=

Valid on field types: string

Example

where{|r| r.foo == 'bar' } and where{|r| r.foo != 'bar' }

Less than

The less than for a field can be tested using <

Valid on field types: numeric, and datetime (only when stored as a number)

Example

where{|r| r.foo < 123 } and where{|r| r.foo < Date.today }

Includes

This operator may be used to check if:

  • A string contains another substring
  • A set of String or Integers contain a given value

Valid on field types: string, or set/array of String / Integer

Example

where{|r| r.foo.includes?(123) } and where{|r| r.foo.includes?('foo') }

In?

This operator may be used to check if:

  • A string field is one of an enumerable set of values

Valid on field types: string

Example

where{|r| r.foo.in?(['foo', 'bar']) }

Working with Map and Raw types

When it comes to map and raw attribute types, DAW takes the approach of trusting you, since the exact format is not explicitly defined or enforced. You may specify the path to the value, as well as the value type and it will behave like any other top level attribute.

where do |r|
  (r.ratings.dig(:verified_reviews, :review_count, type: :number) > 100) &
    (r.ratings.dig(:verified_reviews, :average_review, type: :number) > 4) &
    (r.metadata.dig(:keywords, type: :set, of: :string).includes?('foo'))
end

If you have a nested array, you may access the elements by index by passing an integer into the dig command.

Custom Classes

The subfield dig works with CustomClasses if the classes store their data as a hash.

Example

CustomAttribute = Struct.new(:sub_field_a) do
  def self.dynamoid_dump(item)
    item.to_h
  end

  def self.dynamoid_load(data)
    new(**data.transform_keys(&:to_sym))
  end
end

class Foo
  include Dynamoid::Document
  field :bar, CustomAttribute
end

x = Foo.create(bar: CustomAttribute.new('b'))
Foo.where{|r| r.bar.dig(:sub_field_a, type: string).inclues?('b') }.all
# => [x]

Boolean Operators

Logical Operator Behavior Example
& and where{|r| (r.foo == 'bar') & (r.baz == 'nitch') }
| or where{|r| (r.foo == 'bar') | (r.baz == 'nitch') }
! negation where{|r| !( (r.foo == 'bar') & (r.baz == 'nitch')) }

Retrieving Records

Retrieving a pre-filtered set of records is a fairly obvious use case for the filtering abilities provided by DAW. Only a subset of what you may expect is provided, but enumerable is mixed in, and each provides an Enumerator.

Provided methods

  • all
  • first
  • each (and related enumerable methods)

Start

.start({ some_hash_key: some_value }) takes a hash argument that must match the key structure of the table (range key must be specified where valid). If passed an empty hash, results will start from the beginning of the table. Records before the specified start key will not be scanned or returned. This is useful when doing manual pagination.

Scaning backwards

.scan_index_forward(false) takes a boolean argument to indicate a query should be performed backwards. Errors if your query ends up scanning it will throw an error.

Scan vs Query

DAW will automatically preform a query when it determines it is possible, however if a query is determined to not be appropriate, a scan will be conduced instead. When ever possible, query do not scan. See the DynamoDB docs for why.

DAW will also extract filters on the range key whenever possible. In order to filter on a range key to be used for a query, it must be one of the allowed range key filters and at the top level of filters.

NOTE: Global Secondary Indices are not yet supported

How a query-able filter is identified

A scan will be performed when the search is not done via the hash key, with exact equality. DAW will examine the boolean logic to determine if a key condition may be extracted. For example, a query will be performed in the following examples:

  • where{|r| r.id == '123' }
  • where{|r| (r.id == '123') & (r.bar == 'baz') }

But it will not be performed in these scenarios

  • where{|r| r.id != '123' }
  • where{|r| !(r.id == '123') }
  • where{ (r.id == '123') | (r.bar == 'baz') }

Combination of Filters

Multiple DAW filters can be combined. This will provides the ability to compose filtering conditions to keep your code more readable and DRY.

Combining conditions with AND

class Foo
  include Dynamoid::Document

  field :bar
  field :baz
end

filter1 = Foo.where{|r| r.bar == 'abcd' }
filter2 = Foo.where{|r| r.baz == 'dude' }

# All of these produce the same results
combination1 = filter1.where(filter2)
combination2 = filter1.and(filter2)
combination3 = filter1.where{|r| r.baz == 'dude' }

Mutating Records

DAW provides the ability to modify records only if they meet the criteria defined by the where block.

Changes are also provided in batch form, so you may change multiple values with a single call. There may also be singleton methods provided for easy of use.

Batch Updates

Model.where{ conditions }.batch_update
  .set_values(field_name1: 'value', field_name2: 123)
  .append_to(arr_field_name: [1,2,3], set_field_name: %w[a b c])
  .apply(hash_key, range_key)

Like all conditional updates it will return the full record with the new data if it successfully updates. If it fails to update, it will return nil.

If the specified hash key, or hash/range key combination is not already present it will be inserted with the desired mutations (if possible).

Referencing a field

To identify the field to be updated, either through set, increment, decrement, or append you may just use the field name for top level keys. When you use the top level single symbol key DAW will use the built in Dynamoid dumper.

If you need to reference the sub-key of a map, or custom serialized object you may pass an array of keys. Since DAW looses context to the "type" it is up to you to ensure you are writing out the correct values. The only exception to this is if you are set the field to a class which implements dynamoid_dump.

Example

Model.where{ conditions }.batch_update
  .set_values([:map_or_custom_type, :sub_field, :foo] => 'value', [:map_or_custom_type2, :foo] => MyDumpableClass.new(test))
  .increment([:some_map, :attempts], by: 1)
  .decrement([:some_map, :attempts_remaining], by: 1)
  .apply(hash_key, range_key)

Setting a single field

The batch updated method set_values(attr_name: new_attr_value, other_atter: val)

Shortcut Method

You map perform an upsert using the .upsert method. This method performs a simple set on the provided hash and range key.

For example, consider the following example for conditionally updating a string field.

class Foo
  include Dynamoid::Document
  field :a_string
  field :a_number, number
end

item = Foo.create(a_number: 5, a_string: 'bar')

Foo.where{|r| r.a_number > 5 }.upsert(item.id, a_string: 'dude')

item.reload.a_string # => 'bar'

Foo.where{|r| r.a_number > 4 }.upsert(item.id, a_string: 'wassup')

item.reload.a_string # => 'wassup'

upsert can also create a record if an existing one is not found, if the hash key can be specified. By requiring the hash key be set, you can prevent an insert and force an update to occur.

Note: Upsert must be called with the hash as the first parameter, and the range key as the second parameter if required for the model.

Note:* Upsert will return nil if no records were found that matched the provided parameters

Appending values to a List or Set

You can append a set of values to an existing set or array by using the

append_to(
  array_field: [1,2,3],
  set_field: %w[foo bar],
)

If the fields are unset, it will still apply the changes to am empty array.

Increment / Decrement a value

You may increment or decrement a numeric value by using increment or decrement

increment(:field_one, :field_two)

decrement(:field_one, :field_two)

You may also provide an optional by: config to increment by more than one.

increment(:field_one, :field_two, by: 3)

decrement(:field_one, :field_two, by: 3)

If the value of the field is currently unset, it will initialize to zero

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

TODO:

Known issues

  • If you specify multiple term nodes for a query it will generate an invalid query
  • No support for custom types

Enhancements

  • Support Global Secondary Index
  • Conditions:
    • Equality
    • Partially implemented
    • Not Equals
    • less than
    • Implemented for numerics, datetimes, dates stored as integer
    • less than or equal to
    • greater than
    • greater than or equal to
    • between
    • in
    • attribute_not_exists
    • attribute_type
    • begins with
    • contains
    • size
  • Query enhancements
    • Range key conditions:
    • equality
    • less than
    • less than or equal to
    • greater than
    • greater than or equal to
    • between
    • begins with
    • convert to bulk query if multiple hash key terms are specified
  • Item mutation Docs
    • Update (without insert)
    • Upserting
    • Increment / Decrement number
    • Append item(s) to list
    • Prepend item(s) to list
    • Adding nested map attribute (low priority)
    • Set value if not set
    • Remove attributes

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/dynamoid-advanced-where.