Simple DBM-style key-value database using SQLite3

Description

dbmlite3 is a simple key-value store built on top of SQLite3 that provides a Hash-like interface. It is a drop-in replacement for DBM or YAML::DBM that uses SQLite3 to do the underlying storage.

Why?

Because DBM is really simple and SQLite3 is solid, reliable, ubiquitous, and file-format-compatible across all platforms. This gem gives you the best of both worlds.

Synopsis

require 'dbmlite3'

# Open a table in a database
settings = Lite3::DBM.new("config.sqlite3", "settings")

# You use it like a hash
settings["speed"] = 88
settings["date"] = Date.new(1955, 11, 5)  # Most Ruby types are allowed
settings["power_threshold"] = 2.2

puts settings['power_threshold']

settings.each{|k,v| puts "setting: #{k} = #{v}" }

# But you also have transactions
settings.transaction{
  settings["speed"] = settings["speed"] * 2
}

# You can open other tables in the same database if you want, as above
# or with a block
Lite3::DBM.open("config.sqlite3", "stats") { |stats|
  stats["max"] = 42

  # You can even open multiple handles to the same table if you need to
  Lite3::DBM.open("config.sqlite3", "stats") { |stats2|
    stats2["max"] += 1
  }

  puts "stats=#{stats["max"]}"
}

settings.close

Complete documentation is available in the accompanying rdoc.

Installation

dbmlite3 is available as a gem:

$ [sudo] gem install dbmlite3

Alternately, you can fetch the source code from GitLab and build it yourself:

$ git clone https://gitlab.com/suetanvil/dbmlite3
$ cd dbmlite3
$ rake

It depends on the gem sequel; previously, it used sqlite3.

Quirks and Hints

Remember that a `DBM` is a (potentially) shared file

It is important to keep in mind that while Lite3::DBM objects look like Hashes, they are accessing files on disk that other processes could modify at any time.

For example, an innocuous-looking expression like

db['foo'] = db['foo'] + 1

or its shorter equivalent

db['foo'] += 1

contains a race condition. If (e.g.) two copies of this script are running at the same time, it is possible for both to perform the read before one of them writes, losing the others' result.

There are two ways to deal with this. You can wrap the read-modify-write cycle in a transaction:

db.transaction { db['foo'] += 1 }

Or, of course, you could just design your script or program so that only one program accesses the table at a time.

Keys must be strings

While values may be any serializable type, keys must be strings. As a special exception, Symbols are also allowed but are transparently converted to Strings first. This means that while something like this will work:

db[:foo] = 42

a subseqent

db.keys.include?(:foo) or raise AbjectFailure.new

will raise an exception because the key :foo was turned into a string before being used. You will need to do this instead:

db.keys.include?('foo') or raise AbjectFailure.new

However, this

db.has_key?(:foo)

will work because has_key? does the conversion for us.

Transactions and performance

If you need to do a large number of accesses in a short amount of time (e.g. loading data from a file), it is significantly faster to do these in batches in one or more transactions.

Serialization Safety

Lite3::DBM stores Ruby data by first serializing values using the Marshal or Psych modules. This can pose a security risk if an untrusted third party has direct access to the underlying SQLite3 database. This tends to be pretty rare most of the time but if it is a concern, you can always configure Lite3::DBM to store its values as plain strings.

Forking safely

It is a documented limitation of SQLite3 that database objects cannot be carried across a process fork. Either the parent or the child process will keep the open handle and the other one must forget it completely.

For this reason, if you need both the parent and child process to be able to use Lite3::DBM after a fork, you must first call Lite3::SQL.close_all. Not only will this make it safe but it also lets the child and parent use the same Lite3::DBM objects.

`Lite3::DBM` objects act like file handles but are not

While it is generally safe to treat Lite3::DBM as a wrapper around a file handle (i.e. open and close work as expected), you should be aware that this is not precisely the way things actually work. Instead, the gem maintains a pool of database handles, one per file, and associates them with Lite3::DBM instances as needed. This is necessary for transactions to work correctly.

Mostly, you don't need to care about this. However, it affects you in the following ways:

Transactions are done at the file level and not the table level. This means that you can access separate tables in the same transaction, which is a Very Good Thing.
You can safely fork the current process and keep using existing DBM objects in both processes, provided you've invoked Lite3::SQL.close_all before the fork. This will have closed the actual database handles (which can't tolerate being carried across a fork) and opens new ones the next time they're needed.

DBM objects that go out of scope without first being closed will eventually have their underlying resources cleaned up. However, given that when when that happens depends on the vagaries of the garbage collector and various library internals, it's almost always a bad idea to not explicitly call close first.

Under the hood

Currently, Lite3::DBM uses Sequel to access the sqlite3 library. On JRuby, it goes through the jdbc interface. The previous version (1.0.0) used sqlite3 and only worked on MRI. However, you should make no assumptions about the underlying database libraries this gem uses. It may change in a future release.

All tables created by Lite3::DBM will have names beginning with dbmlite3_ and you should not modify them directly. It might be safe to put other tables in the same database file (e.g. via Sequel) provided that you don't make global changes or mix transactions across interfaces. However, I make no guarantees.