QRPC

QRPC currently implements queued JSON-RPC both client and server which works as normal RPC server, but through queue interface, so allows highly scalable, distributed and asynchronous remote API implementation and fast data processing.

It's based on eventmachine and the default implementation uses beanstalkd so it's fast and thread safe.

Protocol

It utilizes JSON-RPC protocol by default in versions both 1.1 and 2.0 altthough different protocols can be implemented. Adds special data member qrpc with few options appropriate for queue processing. Typicall request looks in Ruby hash notation like:

{
    "jsonrpc" => "2.0",
    "method" => "subtract",
    "params" => [2, 1],
    "id" => <some unique job id>,
    "qrpc" => {
        "version" => "1.0",
        "client" => <some unique client id>,
        "priority" => 30
    }
}

The last priority member is optional, others are expected to be present including them which are optional in classic JSON-RPC. Default priority is 50.

Typical response looks like:

{
    "jsonrpc" => "2.0",
    "result" => 1,
    "id" => <some unique job id>,
    "qrpc" => {
        "version" => "1.0",
    }
}

And in case of exception:

{
    "jsonrpc" => "2.0",
    "error" => {
        "code" => <some code>,
        "message" => <some message>,
        "data" => {
            "name" => <exception class name>,
            "message" => <exception message>,
            "backtrace" => <array of Base64 encoded strings>,
            "dump" => {
                "raw" => <Base64 encoded marshaled exception object>,
                "format" => "ruby"
            }
        }
    },

    "id" => <some unique job id>,
    "qrpc" => {
        "version" => "1.0",
    }
}

Both backtrace and dump members are optional.

Variety of Platforms

Transfer Agents (“Queues”)

QRPC uses the Unified Queues interface which ensures connecting to different queue interfaces using an unified API. It can be used on all platforms which have unified queues driver available by this way. By default, two of them are available:

beanstalkd queue protocol,
8 different queues-like in-memory objects (in fact, all priority queue implementations available for Ruby to date).

See the Unified Queues documentation.

Communication Protocols (“Protocols”)

Different protocols can be implemented provided that it will provide the required API. By default, two of them are available:

JSON-RPC protocol (default, see description above),
object protocol (in fact, the pure Ruby objects for in-memory communication)

Identification Generators (“Generators”)

The request identificators can be generated by various ways. It's possible to implement an own generator so have benefits for logging or in corporate use. By default two generators are available:

UUID generator (default),
Ruby object ID generator (fast, intended for in-memory communication in single Ruby instance).

Encoding Agents (“Serializers”)

The JSON-RPC protocol is intended for use with the JSON encoding, but can be encoded by various other methods. Various serializers support is part of the JSON-RPC Objects gem. See its documentation. The following protocols are supported for example, to date:

Server Usage

Usage is simple. Look example of Beanstalked JSON-RPC service encoded by the MsgPack format working synchronously:

require "qrpc/server"                           # server
require "qrpc/locator/em-jack"                  # queue transfer agent
require "qrpc/protocol/json-rpc"                # RPC protocol
require "json-rpc-objects/serializer/msgpack"   # serializer

class Foo
    def subtract(x, y)
        x - y
    end
end

serializer = JsonRpcObjects::Serializer::MsgPack::new
protocol = QRPC::Protocol::JsonRpc::new(:serializer => serializer)
locator = QRPC::Locator::EMJackLocator::new("test")

server = QRPC::Server::new(Foo::new, :synchronous, protocol)
server.listen! locator

This creates an instance of Foo which will serve as API, creates locator of the queue test at default server localhost:11300. Queue name will be remapped to the real name qrpc-test-input. After call to #listen!, it will run eventmachine and start listening for calls. If you want to run it inside already run eventmachine, simply call #start_listening with the same parameters.

Calls processing is thread safe because of eventmachine concept similar to fibers. Reponse will be put to the same queue server, to queue named qrpc-<client identifier>-output, with structure described above.

Server can work by synchronous or asynchronous way. In the first case, sends the returned values as result, in the other case yielded.

Client Usage

Client usage is simple too. Look example complement the example above:

require "eventmachine"

require "qrpc/client"                           # client 
require "qrpc/generator/uuid"                   # ID generator
require "qrpc/locator/em-jack"                  # queue transfer agent
require "qrpc/protocol/json-rpc"                # RPC protocol
require "json-rpc-objects/serializer/msgpack"   # serializer

EM::run do
    generator = QRPC::Generator::ObjectID::new
    locator = QRPC::Locator::EMJackLocator::new(:test)
    serializer = JsonRpcObjects::Serializer::JSON::new
    protocol = QRPC::Protocol::JsonRpc::new(:serializer => serializer)

    client = QRPC::Client::new(locator, generator, protocol) 
    client.subtract(2, 3) { |result| puts result }  # prints out -1
end

This connects to the test queue at default server localhost:11300, puts request to the real queue name qrpc-test-input and waits and then prints the result from qrpc-<client-id>-output queue. In case of multiple requests should be noted, results can arrive in any order because, of sure, QRPC is pseudo-fibered and asynchronous from its principle.

Client is implemented as evented too, but in case of need you can implement another one with non-evented interface using whatever beanstalkd client you like of sure or any other transfer agent.

Contributing

Fork it.
Create a branch (git checkout -b 20101220-my-change).
Commit your changes (git commit -am "Added something").
Push to the branch (git push origin 20101220-my-change).
Create an Issue with a link to your branch.
Enjoy a refreshing Diet Coke and wait.