Moneta: A unified interface for key/value stores
Moneta provides a standard interface for interacting with various kinds of key/value stores. Moneta is very feature rich:
- Supports a lot of backends (See below)
- Supports proxies (Similar to Rack middlewares)
- Custom serialization via
Moneta::Transformer
proxy (Marshal/JSON/YAML and many more) - Custom key transformation via
Moneta::Transformer
proxy - Value compression via
Moneta::Transformer
proxy (Zlib, Snappy, LZMA, ...) - Expiration for all stores (Added via proxy
Moneta::Expires
if not supported natively) - Atomic incrementation and decrementation for most stores (Method
#increment
) - Includes a very simple key/value server (
Moneta::Server
) and client (Moneta::Adapters::Client
) - Integration with Rails, Rack as cookie and session store and Rack-Cache
Moneta is tested thoroughly using Travis-CI.
Links
- Source: http://github.com/minad/moneta
- Bugs: http://github.com/minad/moneta/issues
- API documentation:
- Latest Gem: http://rubydoc.info/gems/moneta/frames
- GitHub master: http://rubydoc.info/github/minad/moneta/master/frames
Supported backends
Out of the box, it supports the following backends:
- Memory:
- In-memory store (
:Memory
) - LRU hash - prefer this over :Memory! (
:LRUHash
) - LocalMemCache (
:LocalMemCache
) - Memcached store (
:Memcached
,:MemcachedNative
and:MemcachedDalli
)
- In-memory store (
- Relational Databases:
- DataMapper (
:DataMapper
) - ActiveRecord (
:ActiveRecord
) - Sequel (
:Sequel
) - Sqlite3 (
:Sqlite
)
- DataMapper (
- Filesystem:
- PStore (
:PStore
) - YAML store (
:YAML
) - Filesystem directory store (
:File
) - Filesystem directory store which spreads files in subdirectories using md5 hash (
:HashFile
)
- PStore (
- Key/value databases:
- Berkeley DB (
:DBM
) - Cassandra (
:Cassandra
) - GDBM (
:GDBM
) - HBase (
:HBase
) - LevelDB (
:LevelDB
) - Redis (
:Redis
) - Riak (
:Riak
) - SDBM (
:SDBM
) - TokyoCabinet (
:TokyoCabinet
)
- Berkeley DB (
- Document databases:
- CouchDB (
:Couch
) - MongoDB (
:Mongo
)
- CouchDB (
- Other
- Moneta key/value server client (
:Client
works withMoneta::Server
) - Fog cloud storage which supports Amazon S3, Rackspace, etc. (
:Fog
) - Storage which doesn't store anything (
:Null
)
- Moneta key/value server client (
Some of the backends are not exactly based on key/value stores, e.g. the relational ones. These are useful if you already use the corresponding backend in your application. You get a key/value store for free then without installing any additional services and you still have the possibility to upgrade to a real key/value store.
Proxies
In addition it supports proxies (Similar to Rack middlewares) which add additional features to storage backends:
Moneta::Proxy
andMoneta::Wrapper
proxy base classesMoneta::Expires
to add expiration support to stores which don't support it natively. Add it in the builder usinguse :Expires
.Moneta::Stack
to stack multiple stores (Read returns result from first where the key is found, writes go to all stores). Add it in the builder usinguse :Stack
.Moneta::Transformer
transforms keys and values (Marshal, YAML, JSON, Base64, MD5, ...). Add it in the builder usinguse :Transformer
.Moneta::Cache
combine two stores, one as backend and one as cache (e.g.Moneta::Adapters::File
+Moneta::Adapters::Memory
). Add it in the builder usinguse :Cache
.Moneta::Lock
to make store thread safe. Add it in the builder usinguse :Lock
.Moneta::Logger
to log database accesses. Add it in the builder usinguse :Logger
.Moneta::Shared
to share a store between multiple processes. Add it in the builder usinguse :Shared
.
Supported serializers and compressors (Moneta::Transformer
)
Supported serializers:
- BEncode (
:bencode
) - BERT (
:bert
) - BSON (
:bson
) - JSON (
:json
) - Marshal (
:marshal
) - MessagePack (
:msgpack
) - Ox (
:ox
) - TNetStrings (
:tnet
) - YAML (
:yaml
)
Supported value compressors:
- LZMA (
:lzma
) - LZO (
:lzo
) - Snappy (
:snappy
) - QuickLZ (
:quicklz
) - Zlib (
:zlib
)
Special transformers:
- Digests (MD5, Shas, ...)
- Add prefix to keys (
:prefix
) - HMAC to verify values (
:hmac
, useful forRack::MonetaCookies
)
Moneta API
#initialize(options) options differs per-store, and is used to set up the store.
#[](key) retrieve a key. If the key is not available, return nil.
#load(key, options = {}) retrieve a key. If the key is not available, return nil.
#fetch(key, options = {}, &block) retrieve a key. If the key is not available, execute the
block and return its return value.
#fetch(key, value, options = {}) retrieve a key. If the key is not available, return the value,
#[]=(key, value) set a value for a key. If the key is already used, clobber it.
keys set using []= will never expire.
#store(key, value, options = {}) same as []=, but you can supply options.
#delete(key, options = {}) delete the key from the store and return the current value.
#key?(key, options = {}) true if the key exists, false if it does not.
#increment(key, amount = 1, options = {}) increment numeric value. This is a atomic operation
which is not supported by all stores. Returns current value.
#clear(options = {}) clear all keys in this store.
#close close database connection.
The Moneta API is purposely extremely similar to the Hash API. In order so support an identical API across stores, it does not support iteration or partial matches.
Creating a Store
There is a simple interface to create a store using Moneta.new
:
store = Moneta.new(:Memcached, :server => 'localhost:11211')
If you want to have control over the proxies, you have to use Moneta.build
:
store = Moneta.build do
# Adds expires proxy
use :Expires
# Transform key using Marshal and Base64 and value using Marshal
use :Transformer, :key => [:marshal, :base64], :value => :marshal
# Memory backend
adapter :Memory
end
Expiration
The Cassandra, Memcached and Redis backends supports expires values directly:
cache = Moneta::Adapters::Memcached.new
# Or using the builder...
cache = Moneta.build do
adapter :Memcached
end
# Expires in 60 seconds
cache.store(key, value, :expires => 60)
# Update expires time if value is found
cache.load(key, :expires => 30)
cache.key?(key, :expires => 30)
You can add the expires feature to other backends using the Expires proxy:
# Using the :expires option
cache = Moneta.new(:File, :dir => '...', :expires => true)
# or manually by using the proxy...
cache = Moneta::Expires.new(Moneta::Adapters::File.new(:dir => '...'))
# or using the builder...
cache = Moneta.build do
use :Expires
adapter :File, :dir => '...'
end
Incrementation and raw access
The stores support the #increment
which allows atomic increments of unsigned integer values. If you increment
a non existing value, it will be created. If you increment a non integer value an exception will be raised.
store.increment('counter') => 1 # counter created
store.increment('counter') => 2
store.increment('counter', -1) => 1
store.increment('counter', 13) => 14
store.increment('counter', 0) => 14
store['name'] = 'Moneta'
store.increment('name') => Exception
If you want to access the counter value you have to use raw access to the datastore. This is only important
if you have a Moneta::Transformer
somewhere in your proxy chain which transforms the values e.g. with Marshal
.
store.increment('counter') => 1 # counter created
store.load('counter', :raw => true) => '1'
store.store('counter', '10', :raw => true)
store.increment('counter') => 11
Fortunately there is a nicer way to do this using some syntactic sugar!
store.increment('counter') => 1 # counter created
store.raw['counter'] => '1'
store.raw.load('counter') => '1'
store.raw['counter'] = '10'
store.increment('counter') => 11
You can also keep the raw
store in a variable and use it like this:
counters = store.raw
counters.increment('counter') => 1 # counter created
counters['counter'] => '1'
counters.load('counter') => '1'
counters['counter'] = '10'
counters.increment('counter') => 11
Stores which support incrementation (you have to use Moneta::Lock
if you want to use the store in a multithreading environment.)
- ActiveRecord
- File
- HBase
- LRUHash
- LevelDB
- Memcached
- Memory
- Redis
- SDBM/DBM/GDBM
- Sequel
- Sqlite
- TokyoCabinet
- YAML/PStore
Stores which don't support incrementation:
- Cassandra
- Couch
- DataMapper
- Fog
- LocalMemCache
- Mongo
- Riak
Syntactic sugar and option merger
For raw data access as described before the class Moneta::OptionMerger
is used. It works like this:
# All methods after `with` get the options passed
store.with(:raw => true).load('key')
# You can also specify the methods
store.with(:raw => true, :only => :load).load('key')
store.with(:raw => true, :except => [:key?, :increment]).load('key')
# Syntactic sugar for raw access
store.raw.load('key')
# Access substore where all keys get a prefix
substore = store.prefix('sub')
substore['key'] = 'value'
store['key'] => nil
store['subkey'] => 'value'
# Set expiration time for all keys
short_lived_store = long_lived_store.expires(60)
short_lived_store['key'] = 'value'
Framework Integration
Inspired by redis-store there exist integration classes for Rails, Rack and Rack-Cache.
Rack session store
Use Moneta as a Rack session store:
require 'rack/session/moneta'
# Use only the adapter name
use Rack::Session::Moneta, :store => :Redis
# Use Moneta.new
use Rack::Session::Moneta, :store => Moneta.new(:Memory, :expires => true)
# Use the Moneta builder
use Rack::Session::Moneta do
use :Expires
adapter :Memory
end
Rack cache
Use Moneta as a Rack-Cache store:
require 'rack/cache/moneta'
use Rack::Cache,
:metastore => 'moneta://Memory?expires=true',
:entitystore => 'moneta://Memory?expires=true'
# Or used named Moneta stores
Rack::Cache::Moneta['named_metastore'] = Moneta.build do
use :Expires
adapter :Memory
end
use Rack::Cache,
:metastore => 'moneta://named_metastore',
:entity_store => 'moneta://named_entitystore'
Rack cookies
Use Moneta to store cookies in Rack. It uses the Moneta::Adapters::Cookie
. You might
wonder what the purpose of this store or Rack middleware is: It makes it possible
to use all the transformers on the cookies (e.g. :prefix
, :marshal
and :hmac
for value verification).
require 'rack/moneta_cookies'
use Rack::MonetaCookies, :domain => 'example.com', :path => '/path'
run lambda do |env|
req = Rack::Request.new(env)
req. #=> is now a Moneta store!
env['rack.request.cookie_hash'] #=> is now a Moneta store!
req.['key'] #=> retrieves 'key'
req.['key'] = 'value' #=> sets 'key'
req..delete('key') #=> removes 'key'
[200, {}, []]
end
Rails session store
Add the session store in your application configuration config/environments/*.rb
.
require 'moneta'
# Only by adapter name
config.cache_store :moneta_store, :store => :Memory
# Use Moneta.new
config.cache_store :moneta_store, :store => Moneta.new(:Memory)
# Use the Moneta builder
config.cache_store :moneta_store, :store => Moneta.build do
use :Expires
adapter :Memory
end
Rails cache store
Add the cache store in your application configuration config/environments/*.rb
. Unfortunately the
Moneta cache store doesn't support matchers. If you need these features use a different server-specific implementation.
require 'moneta'
# Only by adapter name
config.cache_store :moneta_store, :store => :Memory
# Use Moneta.new
config.cache_store :moneta_store, :store => Moneta.new(:Memory)
# Use the Moneta builder
config.cache_store :moneta_store, :store => Moneta.build do
use :Expires
adapter :Memory
end
Advanced - Build your own key value server
You can use Moneta to build your own key/value server which is shared between
multiple processes. If you run the following code in two different processes,
they will share the same data which will also be persistet in the database shared.db
.
require 'moneta'
store = Moneta.build do
use :Transformer, :key => :marshal, :value => :marshal
use :Shared do
use :Cache do
cache do
adapter :LRUHash
end
backend do
adapter :GDBM, :file => 'shared.db'
end
end
end
end
More information
- http://yehudakatz.com/2009/02/12/whats-the-point/
- http://yehudakatz.com/2009/02/12/initial-release-of-moneta-unified-keyvalue-store-api/
Alternatives
- Horcrux: Used at github, supports batch operations but only Memcached backend
- ToyStore: ORM mapper for key/value stores
- ToyStore Adapter: Adapter to key/value stores used by ToyStore, Moneta can be used directly with the ToyStore Memory adapter
Authors
- Daniel Mendler
- Hannes Georg
- Originally by Yehuda Katz and contributors