What is Oinky?

Oinky is an in-memory relational database library that supports fast serialization and deserialization. It is a tool for transforming the representation of data between the application (as a database) and network or storage (as a string of bytes). It is a database (like sqlite), but also an encoder (like Protocol Buffers).

The implementation is C++, and currently includes bindings for C and Ruby.

Applications

Oinky is a general purpose serialization library. It has tons of potential applications. Many excellent serialization libraries exist, but complex data structures like graphs can be cumbersome to encode with the best of them. Oinky's relational facility makes it well suited to such tasks, as relational schemas can naturally and efficiently represent data of arbitrary topology.

Microsharding

Microsharding is a scheme for deploying quasi-relational databases over distributed key-value storage platforms. Oinky provides the relational adapter. The rationale behind microsharding is described in more detail in a blog post.

Sample

The following sample is inspired by the Sequel sample at http://sequel.rubyforge.org

require 'oinky'

db = Oinky::DB.new

db.atomic {
  db.create_table(:items) { |tbl|
    tbl.add_column(:id, :int32, 0)
    tbl.add_column(:name, :string, '')
    tbl.add_column(:price, :int32, 0)

    tbl.add_index(:name=>'by_name', :unique=>true, :columns=>[:name])
  }
}

items = db[:items]

items.insert_row(:name => 'abc', :price => (rand * 100).to_i)
items.insert_row(:name => 'def', :price => (rand * 100).to_i)
items.insert_row(:name => 'ghi', :price => (rand * 100).to_i)

# print out the number of records
puts "Item count: #{items.row_count}"

puts "Items table contains:"
items.each{ |row|
  # row is a hash of colname=>value
  puts row.inspect
}

# print out the average over a specific column
avg = items.each.average(:price)
puts "The average price is: #{avg}"

# Using a lambda explicitly to compute something more complex 
# (and ideally less useless).
db[:items].each.average(lambda {|r| r[:price] + r[:name].size} )

# Serialize
str = db.serialize
puts "The serialized database consumes #{str.length} bytes"

# Create a clone
db2 = Oinky::DB.mount(str)

db2.alter_table(:items) { |tbl|
  tbl.add_index(:name=>'by_price', :unique=>false, :columns=>[:price])
}

puts "Items by price:"
db2[:items].indices[:by_price].each {|row|
  puts row.inspect
}

str2 = db2.serialize
puts "With the price index, the serialized database consumes #{str2.length} bytes"

Implementation

See the design document for an overview of behavior and implementation.

Performance

Oinky is designed to perform very well in specific scenarios. Design tradeoffs and performance properties are covered in the performance document.

Status

Oinky is fast and stable, but very new. The interface is raw, and the library is unsupported by the tools and ORMs that help make relational schemas so compelling in the first place.

Nevertheless, Oinky is highly functional as it is, and offers open-ended opportunities for improvement.

Installation

Oinky is configured with autoconf and depends on the boost C++ libraries. First you'll need to make sure boost and autotools are installed. Then you can generate the configure script as follows.

./boostrap.sh

If boost is installed in a nonstandard location, you'll need to provide that location to the configure script. Additionally, building the ruby gem is optional, as it depends on ruby and rake. The following command illustrates the available options.

./configure --with-boost-include-path=/usr/local/boost_1_47_0 \
    --with-boost-lib-path=/usr/local/boost_1_47_0/stage/lib \
    --enable-rubygem=yes --with-asserts=yes

Once you've configured, make will build the C library. make install will install the headers and C library. make check will build and run some sanity tests.

Ruby

Before installing the gem, you'll need to build it in (project_root)/lib/rubygem

The gem installer accepts the boost arguments in the same format as the configure script.

gem install oinky-0.1.0.gem -- --with-boost-include-path=/usr/local/boost_1_47_0 \
    --with-boost-lib-path=/usr/local/boost_1_47_0/stage/lib

The gem requires Ruby 1.9.x. Though the gem is built with FFI and is theoretically portable, only MRI currently works, due to differences in the FFI interface under JRuby and RBX.

Platforms

Oinky has been successfully built and tested on 64 bit OSX and 32/64 bit Linux with clang++(2.1) and g++(4.4 and 4.6) and boost versions 1.46 and 1.47. Various older compilers and boost versions have caused issues, though many may work. Oinky implements architecture-dependent byte-order conversion, but it's only been minimally tested.

Support

There is a mailing list where discussion may occur.

License

Oinky is published under the terms of the MIT license. See the LICENSE file.