Oedipus: Sphinx 2 Search Client for Ruby
Oedipus is a client for the Sphinx search engine (>= 2.0.2), with support for real-time indexes and multi-dimensional faceted searches.
It is not a clone of the PHP API, rather it is written from the ground up, wrapping the SphinxQL API offered by searchd. Nor is it a plugin for ActiveRecord or DataMapper... though this will be offered in separate gems (see oedipus-dm).
Oedipus provides a level of abstraction in terms of the ease with which faceted search may be implemented, while remaining light and simple.
Data structures are managed using core ruby data types (Array and Hash), ensuring simplicity and flexibilty.
The current development focus is on supporting realtime indexes, where data is indexed from your application, rather than by running the indexer tool that comes with Sphinx. You may use indexes that are indexed with the indexer tool, but Oedipus does not (yet) provide wrappers for indexing that data via ruby [1].
Dependencies
- ruby >= 1.9
- sphinx >= 2.0.2
- mysql dev libs >= 4.1
Installation
Via rubygems:
gem install oedipus
Usage
The following features are all currently implemented.
Connecting to Sphinx
require "oedipus"
sphinx = Oedipus.connect('127.0.0.1:9306') # sphinxql host
NOTE: Don't connect to the named host 'localhost', since the MySQL library will try to use a UNIX socket instead of a TCP connection, which Sphinx doesn't currently support.
Connections can be re-used by calling Oedipus.connection
once connected.
sphinx = Oedipus.connection
If you're using Oedipus in a Rails application, you may wish to call connect
inside an initializer and then obtain that connection in your application, by
calling connection
.
If you need to manage multiple connections, you may specify names for each connection.
Oedipus.connect("other-host.tld:9306", :other)
sphinx = Oedipus.connection(:other)
Inserting (real-time indexes)
sphinx[:articles].insert(
7,
title: "Badgers in the wild",
body: "A big long wodge of text",
author_id: 4,
views: 102
)
Replacing (real-time indexes)
sphinx[:articles].replace(
7,
title: "Badgers in the wild",
body: "A big long wodge of text",
author_id: 4,
views: 102
)
Updating (real-time indexes)
sphinx[:articles].update(7, views: 103)
Deleting (real-time indexes)
sphinx[:articles].delete(7)
Fetching a known document (by ID)
record = sphinx[:articles].fetch(7)
# => { id: 7, views: 984, author_id: 3 }
Fulltext searching
You perform queries by invoking #search
on the index.
results = sphinx[:articles].search("badgers", limit: 2)
# Meta deta indicates the overall number of matched records, while the ':records'
# array contains the actual data returned.
#
# => {
# total_found: 987,
# time: 0.000,
# keywords: [ "badgers" ],
# docs: { "badgers" => 987 },
# records: [
# { id: 7, author_id: 4, views: 102 },
# { id: 11, author_id: 6, views: 23 }
# ]
# }
Fetching only specific attributes
sphinx[:articles].search(
"example",
attrs: [:id, :views]
)
Fetching additional attributes (including expressions)
Any valid field expression may be fetched. Be sure to alias it if you want to order by it.
sphinx[:articles].search(
"example",
attrs: [:*, "WEIGHT() AS wgt"]
)
Attribute filters
Result formatting is the same as for a fulltext search. You can add as many filters as you like.
# equality
sphinx[:articles].search(
"example",
author_id: 7
)
# less than or equal
sphinx[:articles].search(
"example",
views: -Float::INFINITY..100
)
sphinx[:articles].search(
"example",
views: Oedipus.lte(100)
)
# greater than
sphinx[:articles].search(
"example",
views: 100...Float::INFINITY
)
sphinx[:articles].search(
"example",
views: Oedipus.gt(100)
)
# not equal
sphinx[:articles].search(
"example",
author_id: Oedipus.not(7)
)
# between
sphinx[:articles].search(
"example",
views: 50..100
)
sphinx[:articles].search(
"example",
views: 50...100
)
# not between
sphinx[:articles].search(
"example",
views: Oedipus.not(50..100)
)
sphinx[:articles].search(
"example",
views: Oedipus.not(50...100)
)
# IN( ... )
sphinx[:articles].search(
"example",
author_id: [7, 22]
)
# NOT IN( ... )
sphinx[:articles].search(
"example",
author_id: Oedipus.not([7, 22])
)
Ordering
sphinx[:articles].search("badgers", order: { views: :asc })
Special handling is done for ordering by relevance.
sphinx[:articles].search("badgers", order: { relevance: :desc })
In the above case, Oedipus explicity adds WEIGHT() AS relevance
to the :attrs
option. You can manually set up the relevance sort if you wish to name the weighting
attribute differently.
Limits and offsets
Note that Sphinx applies a limit of 20 by default, so you probably want to specify
a limit yourself. You are bound by your max_matches
setting in sphinx.conf.
The meta data will still indicate the actual number of results that matched; you simply get a smaller collection of materialized records.
sphinx[:articles].search("bobcats", limit: 50)
sphinx[:articles].search("bobcats", limit: 50, offset: 150)
Faceted searching
A faceted search takes a base query and a set of additional queries that are variations on it. Oedipus makes this simple by allowing your facets to inherit from the base query.
Oedipus allows you to replace '%{query}'
in your facets with whatever was in
the original query. This can be useful if you want to provide facets that only
perform the search in the title of the document ("@title (%{query})"
) for
example.
Each facet is given a key, which is used to reference it in the results. This key is any arbitrary object that can be used as a key in a ruby Hash. You may, for example, use domain-specific objects as keys.
Sphinx optimizes the queries by figuring out what the common parts are. Currently it does two optimizations, though in future this will likely improve further, so using this technique to do your faceted searches is the correct approach.
results = sphinx[:articles].search(
"badgers",
facets: {
popular: { views: 100..10000 },
also_farming: "%{query} & farming",
popular_farming: ["%{query} & farming", views: 100..10000 ]
}
)
# => {
# total_found: 987,
# time: 0.000,
# records: [ ... ],
# facets: {
# popular: {
# total_found: 25,
# time: 0.000,
# records: [ ... ]
# },
# also_farming: {
# total_found: 123,
# time: 0.000,
# records: [ ... ]
# },
# popular_farming: {
# total_found: 2,
# time: 0.000,
# records: [ ... ]
# }
# }
# }
Multi-dimensional faceted search
If you can add facets to the root query, how about adding facets to the facets themselves? Easy:
results = sphinx[:articles].search(
"badgers",
facets: {
popular: {
views: 100..10000,
facets: {
in_title: "@title (%{query})"
}
}
}
)
# => {
# total_found: 987,
# time: 0.000,
# records: [ ... ],
# facets: {
# popular: {
# total_found: 25,
# time: 0.000,
# records: [ ... ],
# facets: {
# in_title: {
# total_found: 24,
# time: 0.000,
# records: [ ... ]
# }
# }
# }
# }
# }
In the above example, the nested facet :in_title
inherits the default
parameters from the facet :popular
, which inherits its parameters from
the root query. The result is a search for "badgers" limited only to the
title, with views between 100 and 10000.
There is no limit imposed in Oedipus for how deeply facets can be nested.
General purpose multi-search
If you want to execute multiple queries in a batch that are not related to each
other (which would be a faceted search), then you can use #multi_search
.
You pass a Hash of keyed-queries and get a Hash of keyed-resultsets.
results = sphinx[:articles].multi_search(
badgers: ["badgers", limit: 30],
frogs: "frogs & wetlands",
rabbits: ["rabbits | burrows", view_count: 20..100]
)
# => {
# badgers: {
# ...
# },
# frogs: {
# ...
# },
# rabbits: {
# ...
# }
# }
Disconnecting from Sphinx
Oedipus will automatically close connections that are idle, so you don't, generally speaking, need to close connections manually. However, before forking children, for example, it is important to dispose of any open resources to avoid sharing resources across processes. To do this:
Oedipus.connections.each { |key, conn| conn.close }
Running the specs
There are both unit tests and integration tests in the specs/ directory. By default they will both run, but in order for the integration specs to work, you need a locally installed copy of [Sphinx] [2]. You then execute the specs as follows:
SEARCHD=/path/to/bin/searchd bundle exec rake spec
If you don't have Sphinx installed locally, you cannot run the integration specs (they need to write config files and start and stop sphinx internally).
To run the unit tests alone, without the need for Sphinx:
bundle exec rake spec:unit
If you have made changes to the C extension, those changes will be compiled and installed (to the lib/ directory) before the specs are run.
Footnotes
[1]: In practice I find such an abstraction not to be very useful, as it assumes a single-server setup
[2]: You can build a local copy of sphinx without installing it on the system:
cd sphinx-2.0.4/
./configure
make
The searchd binary will be found in /path/to/sphinx-2.0.4/src/searchd.
Future Plans
- Integration ActiveRecord
- Support for re-indexing non-realtime indexes from ruby code
- Distributed index support (sharding writes between indexes)
- Query translation layer for Lucene-style AND/OR/NOT and attribute:value interpretation
- Fulltext query sanitization for unsafe user input (e.g. @@missing field)
Copyright and Licensing
Refer to the LICENSE file for details.