You pipe from here, to a process that runs over there. Pipemaster forks, redirects and runs your command.

Why

I’ve got a short task I want to perform. To validate an incoming email, parse the message and store the results in the database(*). I setup Postfix to pipe incoming emails into a Ruby script.

Ruby processes are fairly cheap, until you get into loading the mail library, the database library, the ORM, the application logic, the … you get the picture.

I use Pipemaster to fire up the main process once, and the Pipemaster client to run commands on that server. Still have to make sure the code is light and fast, but I did eliminate the significant initialization overhead.

As you can guess, Pipemaster supports piping input and output streams, and uses forking (sorry Windows; for JRuby see Nailgun).

  • Processing emails as they come allows the application to reject unauthorized senders immediately by replying with an SMTP code. The alternative, accepting the email and later on sending a bounce, leads to backscatter (see en.wikipedia.org/wiki/Backscatter_(e-mail)).

Using Pipemaster

Step 1: Create a Pipemaster file. For example:

#!highlight/ruby
command :echo do |*args|
  first, rest = $stdin.read.split
  $stdout << [first, args, rest].flatten.join(" ")
end

Step 2: Fire up the Pipemaster server:

$ pipemaster
I, [2010-02-18T12:57:57.739230 #5460]  INFO -- : master process ready
I, [2010-02-18T12:57:57.739606 #5460]  INFO -- : listening on addr=127.0.0.1:7887 fd=3

Step 3: For a new shell, execute a command:

$ echo "Stand down!" | pipe echo upside
Stand upside down!

Pipemaster, Resque and Rails

This example uses Resque to queue and process tasks asynchronously, where the tasks are part of a larger Rails application (e.g. using ActiveRecord models, ActiveMailer).

This Pipefile loads the Rails application once during setup. It starts one Resque worker than polls for new jobs every 5 seconds.

#!/usr/bin/env ruby -S pipemaster
user  "nobody"

require "syslog_logger"
syslog = SyslogLogger.new("pipemaster")
class << syslog ; def close ; end ; end
logger syslog

setup do
  # Load RAILS.  Diz will take a while.
  require File.dirname(__FILE__) + '/config/environment'
  ActiveRecord::Base.connection.disconnect!
end
after_fork do |server, worker|
  ActiveRecord::Base.establish_connection
end

# Resque
background :resque do
  resque = Resque::Worker.new("*")
  resque.verbose = true
  trap(:QUIT) { resque.shutdown } # graceful
  resque.work(5)
end

Tips && tricks

Add this at the top of your Pipefile for Ruby syntax highlighting:

#!ruby -S pipemaster

License

Pipemaster is copyright of Assaf Arkin. It is heavily based on the awesome Unicorn Web server and therefore uses the same license.

Unicorn is copyright 2009 by all contributors (see logs in git). It is based on Mongrel and carries the same license.

Mongrel is copyright 2007 Zed A. Shaw and contributors. It is licensed under the Ruby license and the GPL2. See the included LICENSE file for details.

Pipemaster is 100% Free Software.