Sideroo

Declarative and auditable object-oriented library for Redis

Sideroo is Object Oriented Redis (ooredis) spelled backward.

1. Motivations

This gem is aimed to provide

  • a declarative Redis key definition ```rb class TopStoriesCache < Sideroo::Set key_pattern 'top_stories:country:category' description 'Cache top stories by ID per country and category' example 'top_stories:us:romance' key_regex /^top_stories:(\w2):([^:]+)$/ # OPTIONAL - read docs below end

TopStoriesCache.dimensions # ['country', 'category']

- an **intuitive** Redis key initialization & `attr_accessor`
  ```rb
  cache = TopStoriesCache.new(country: 'us', category: 'romance')
  # instead of repeating key = "top_stories:#{country}:#{category}"

  cache.country # us
  cache.genre # romance
  • object-oriented methods for each Redis data type rb # Redis Set methods cache.sadd(story_id) # instead of redis.sadd(key, story_id) cache.smembers # instead of redis.smembers(key) cache.sismember(member) # instead of redis.sismember(key, member)
  • an auditable Redis key management ```rb TopStoriesCache.count # key count TopStoriesCache.all.map(&:key) # list all keys TopStoriesCache.flush # delete all keys of the same pattern

# Support where for key searching # each, map for enumerable TopStoriesCache.where(category: 'romance').each do |set| # ... end

- Potential **self-generated documentation** for Redis usage
  ```rb
  Sideroo.report # COMING SOON

All of these are done while maintaining a thin abstraction on top of redis gem.

2. Installation

Add this line to your application's Gemfile:

gem 'sideroo'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install sideroo

3. Usage

3.0. Configurations - REQUIRED

Sideroo.configure do |c|
  c.redis_client = Redis.new
end

Sideroo provides a thin OOP abstraction on top of redis-rb. Therefore, it's recommended to use redis-rb (with / without redis-namespace). Any Redis clients with the same interfaces are fine too.

Most of the usage are just to abstract the key argument into the internal state of the obj.

Examples

For Redis Set

# in Redis
key = "namespace:#{dimension_1}:#{dimension_2}"
redis_client.scard(key)
redis_client.sadd(key, member)

# in Sideroo
class MySet < Sideroo::Set
  key_pattern 'namespace:{dimension_1}:{dimension_2}'
end

sideroo_set = MySet.new(
  dimension_1: value_1,
  dimension_2: value_2,
)
sideroo_set.scard
sideroo_set.sadd(member)

3.1. Define a Redis usage

Each Redis usages usually

  • have a key pattern
  • use a certain Redis data type

Sideroo provides what you need and more.

There are some configurations you can specify for each use-cases.

key_pattern - String - required

Pattern used by the use-case

key_pattern 'top_stories:{country}:{category}'

key_regex - Regexp - optional

Regex for better pattern matching for search. See more in Section 5.

Can be 100% optional if key namespacing is done well.

key_regex /^top_stories\:(\w{2})\:([^\:]+)$/

description - String - optional

Provide description to the use-cases.

description 'Cache top stories per country and category'

example - String - optional

Example of actual Redis keys would be used. If specified, you can utilize example_valid? check to validate key_regex in your specs.

example 'top_stories:us:romance'

Dynamic initalization

When there are dynamic components inside the key pattern, e.g. top_stories:{country}:{category}, the constructor would detect and require country and category during initialization.

# Static cache key
class TopUserCache < Sideroo::List
  key_pattern 'top_users' # REQUIRED
  description 'Cache 50 top users worldwide'
end

# 1-dimension cache key
class CountryPolicyCache < Sideroo::String
  key_pattern 'policy:{country}' # REQUIRED
  key_regex /^page\:(\w{2})$/ # Optional. To resolve key conflicts with other usages if any.
  description 'Cache Policy page per country'
end

CountryPolicyCache.new # MissingKeys: Missing country
CountryPolicyCache.new(country: 'us') # Good
CountryPolicyCache.new(gender: 'us') # UnexpectedKeys: Unexpected keys gender

# 2-dimension cache key
class TopStoriesCache < Sideroo::List
  key_pattern 'top_stories:{country}:{category}'
  description 'Cache top stories by ID per country and category'
end

TopStoriesCache.new # MissingKeys: Missing country, category
TopStoriesCache.new(country: 'us') # MissingKeys: Missing category
TopStoriesCache.new(country: 'us', category: 'romance') # GOOD
TopStoriesCache.new(country: 'us', cateogry: 'romance', random_key: 'random_value') # UnexpectedKeys: Unexpected keys random_key

3.2. Object-oriented methods for each data type

class CountryPageCache < Sideroo::String
  key_pattern 'page:{country}'
  key_regex /^page\:(\w{2})$/
end

# The key-value params are auto detected
page_cache = CountryPageCache.new(country: country)
page_cache.get

class TopStoriesCache < Sideroo::List
  key_pattern 'top_stories:{country}:{category}'
  description 'Cache top stories by ID per country and category'
end

# The key-value params are auto detected
cache = TopStoryCache.new(country: 'sg', category: 10)
cache.lpush(story_id)
cache.set(story_id) # NoMethodError - since `set` is not a method of List type

3.3. Search and Enumerable

class TopStoriesCache < Sideroo::Set
  key_pattern 'top_stories:{country}:{category}'
  description 'Cache top stories by ID per country and category'
end
top_stories:sg:10
top_stories:sg:20
top_stories:us:10
top_stories:us:12
TopStoriesCache.all # Not recommended for large db
TopStoriesCache.all.to_a

TopStoriesCache.where(country: 'sg').to_a

# Loop through `top_stories:sg:*`
TopStoriesCache.where(country: 'sg').each do |list|
  list.key # top_stories:sg:10
  list.smembers # return story ids
  # ...
end

TopStoriesCache.where(country: 'sg').map do |list|
  #...
end

TopStoriesCache.where(country: 'sg').count

3.4. Report & Generate documentation - COMING SOON

Sideroo.report
TBD

3.5. Audit keys

TopStoriesCache.count # Scan and count

TopStoriesCache.all.to_a # NOT RECOMMENDED if there are too many keys

3.6. Flush keys

TopStoriesCache.flush # Delete all keys of TopStoriesCache

4. Data Types

Sideroo provides support for 7 main Redis data types.

All key-related Redis methods are supported by all below types.

class AnyRecord < Sideroo::Base
  # ...
end

record = AnyRecord.new(...)

record.del
record.dump
record.exists
record.expire(duration_in_seconds)
record.expireat(time_in_seconds)
record.persist
record.pexpire(duration_in_ms)
record.pexpireat(time_in_ms)
record.pttl
record.rename(new_key)
record.renamenx(new_key)
record.restore(ttl, serialized_value, options)
record.touch
record.ttl
record.type
record.unlink

4.1. Sideroo::String

Support all KEY-related methods and its own methods.

class MyStringCache < Sideroo::String
  # ...
end

string = MyStringCache.new(...)

string.append(value)
string.decr
string.decrby(value) # number
string.get
string.getbit(offset)
string.getrange(start, stop)
string.getset(value)
string.incr
string.incrby(value)
string.incrbyfloat(value)
string.psetex(ttl, value)
string.set(value)
string.setbit(offset, value)
string.setex(ttl, value)
string.setnx(value)
string.setrange(offset, value)
string.strlen

4.2. Sideroo::Hash

Support all KEY-related methods and its own methods.

class MyHash < Sideroo::Hash
  # ...
end

hash = MyHash.new(...)

hash.hdel(*fields)
hash.hexists(field)
hash.hget(field)
hash.hgetall
hash.hincrby(field, increment)
hash.hincrbyfloat(field, increment)
hash.hkeys
hash.hlen
hash.hmget(*fields, &blk)
hash.hmset(*attrs)
hash.hscan(cursor, options = {})
hash.hscan_each(options = {}, &block)
hash.hset(field, value)
hash.hsetnx(field, value)
hash.hvals
hash.mapped_hmget(*field)
hash.mapped_hmset(hash)

4.3. Sideroo::List

Support all KEY-related methods and its own methods.

class MyList < Sideroo::List
  # ...
end

list = MyList.new(...)

list.blpop(timeout:)
list.brpop(timeout:)
list.brpoplpush(destination, options = {})
list.lindex(index) # => String
list.linsert(where, pivot, value) # => Fixnum
list.llen # => Fixnum
list.lpop # => String
list.lpush(value) # => Fixnum
list.lpushx(value) # => Fixnum
list.lrange(start, stop) # => Array<String>
list.lrem(count, value) # => Fixnum
list.lset(index, value) # => String
list.ltrim(start, stop) # => String
list.rpop # => String
list.rpoplpush(source, destination) # => nil, String
list.rpush(value) # => Fixnum
list.rpushx(value) # => Fixnum

4.4. Sideroo::Set

Support all KEY-related methods and its own methods.

class SiteSet < Sideroo::Set
  # ...
end

set = SiteSet.new(...)

set.sadd(member) # => Boolean, Fixnum
set.scard
set.sdiff(*other_keys)
set.sinter(*other_keys)
set.sismember(member)
set.smembers
set.smove(destination, member)
set.spop(count = nil)
set.srandmember(count = nil)
set.srem(member)
set.sscan(cursor, options = {}) # => String+
set.sscan_each(options = {}, &block) # => Enumerator
set.sunion(*other_keys)
set.sdiffstore(destination, *other_keys)
set.sdiffstore!(*other_keys)
set.sinterstore(destination, *other_keys)
set.sinterstore!(*other_keys)
set.sunionstore(destination, *other_keys)
set.sunionstore!(*other_keys)

4.5. Sideroo::SortedSet

Support all KEY-related methods and its own methods.

class MySortedSet < Sideroo::SortedSet
  # ...
end

sorted_set = MySortedSet.new(...)

sorted_set.zadd(*args) # => Boolean, ...
sorted_set.zcard # => Fixnum
sorted_set.zcount(min, max) # => Fixnum
sorted_set.zincrby(increment, member) # => Float
sorted_set.zlexcount(min, max) # => Fixnum
sorted_set.zpopmax(count = nil) # => Array<String, Float>+
sorted_set.zpopmin(count = nil) # => Array<String, Float>+
sorted_set.zrange(start, stop, options = {}) # => Array<String>, Arra
sorted_set.zrangebylex(min, max, options = {}) # => Array<String>, Arra
sorted_set.zrangebyscore(min, max, options = {}) # => Array<String>, Arra
sorted_set.zrank(member) # => Fixnum
sorted_set.zrem(member) # => Boolean, Fixnum
sorted_set.zremrangebyrank(start, stop) # => Fixnum
sorted_set.zremrangebyscore(min, max) # => Fixnum
sorted_set.zrevrange(start, stop, options = {}) # => Object
sorted_set.zrevrangebylex(max, min, options = {}) # => Object
sorted_set.zrevrangebyscore(max, min, options = {}) # => Object
sorted_set.zrevrank(member) # => Fixnum
sorted_set.zscan(cursor, options = {}) # => String, Arra
sorted_set.zscan_each(options = {}, &block) # => Enumerator
sorted_set.zscore(member) # => Float
sorted_set.zinterstore(destination, *other_keys)
sorted_set.zinterstore!(*other_keys)
sorted_set.zunionstore(destination, *other_keys)
sorted_set.zunionstore!(*other_keys)

4.6. Sideroo::Bitmap

Support all KEY-related methods and its own methods.

class MyBitmap < Sideroo::Bitmap
  # ...
end

bitmap = MyBitmap.new(...)

bitmap.getbit(offset)
bitmap.setbit(offset, value)

4.7. Sideroo::HyperLogLog

Support all KEY-related methods and its own methods.

class MyHLL < Sideroo::HyperLogLog
  # ...
end

hll = MyHLL.new(...)

hll.pfadd(member)
hll.pfcount
hll.pfmerge(destination, *other_keys)
hll.pfmerge!(*other_keys)

5. Known issues

Redis search via keys and scan methods only support glob-style patterns.

  • h?llo matches hello, hallo and hxllo
  • h*llo matches hllo and heeeello
  • h[ae]llo matches hello and hallo, but not hillo
  • h[^e]llo matches hallo, hbllo, ... but not hello
  • h[a-b]llo matches hallo and hbllo

glob-style patterns are not as comprehensive as Regexp. This introduces conflicts for similar key patterns.

For examples,

  • users:{country}:{gender} would use search pattern users:*:*
  • users:{age} would use search pattern users:*

The second pattern does cover the data set of the first pattern. This could be avoid by having better namespacing in your applications.

e.g.ucg:{country}:{gender} vs. u:{user_id}.

Sideroo also provides an additional matching options called key_regex for each class. This would allow deeper key selection.

class TopCountryGenderUsersCache < Sideroo::Set
  key_pattern 'users:{country}:{gender}'
  key_regex /^users\:([a-z]{2})\:([mf])$/
  example 'users:sg:m'
  description 'Top users per country per gender'
end

class UserStoriesCache < Sideroo::Set
  key_pattern 'users:{user_id}'
  key_regex /^users\:\d+$/
  example 'users:12345'
  description 'Top stories per users'
end

6. Redis Clients

Redis clients can be customized at 3 levels

  • Global
  • Class
  • Instance

The lower level would inherit the config from parent level if a custom Redis client is not specified.

6.1. Global Sideroo config

Sideroo.configure do |c|
  c.redis_client = global_redis_client
end

6.2. Class level config

class UserStoriesCache < Sideroo::Set
  # ...
  redis_client class_redis_client
end

6.3. Instance level config

cache = UserStoriesCache.new(...)
cache.use_client(instance_redis_client)

7. Advanced usages

7.1. Dimension validations

To keep this gem thin, we have decided not to add explicit support for dimension validation.

However, Sideroo collaborates perfectly with ActiveModel::Validations. Please incorporate at your own needs.

class TopStoriesCache < Sideroo::Set
  include ActiveModel::Validations

  key_pattern 'top_stories:{country}:{category}'
  description 'Cache top stories by ID per country and category'
  example 'top_stories:us:romance'

  validates :country, length: 2
  validates :category, regex: /^[^:]+$/
end

8. Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

9. Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/sideroo. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.

10. License

The gem is available as open source under the terms of the MIT License.

11. Code of Conduct

Everyone interacting in the Sideroo project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.