Pazuzu

Pazuzu is a supervisor daemon that manages pools of application processes as daemons. Pazuzu uses Heroku's Procfile format and relies on Linux cgroups to encapsulate groups of process.

                           ....
                          ......
                         ..=?$..               .
         ...            ...N~,....            .....
    .. ..... .          ...M8O....           ......
  ......NZ$,..        . ...DD8....          . $O$+....
     .$8~Z7$. .     ....:N88ZO:Z....        ..8ZOD=..
 . ...8OO7Z$+....  .  ..88Z:I=8$?..      ....ZO7I$M.....
 ....DZ+$ZZZ8... . .....?DN878=,~..... . ...N7ODOIDO...
....D=ZZ7Z$$?D.....  ..M$D8D$D8=I~....  ...N$88IDO8O:....
 ...OOZ$ZO$$?OO.... ...8D8ND8?MM,,... ....8+DZZ7$ZDOO..  .
....OZN7$~8Z$7$$.......88O88$~?DO.......=OZZOO8MDOZODN... ..
  .,D$7I7ZZ$?+$Z8,......ZDDDND8M,......M$OZNDO$NODM8+I..
  ..MDO$Z$DMD+~:Z8:......DDNMM?I,....8Z8Z8Z8N?Z$DDO8D...
  . ~+D7IN?Z=IOD$ONN.....N8N8~$D..IIOZ78Z$8Z$DO?ZMOOM...
   ...?7DNO=?IDOO8OONDNNNNNNN8OII$I=$Z8Z78D8NZZZZDDD.  ..
 .....MZ$DNDDN?ZZ8DNN8DDDNNNDO7$I$+++ZOOOOO8$NMDI8...
  ....:DOO8NOOZOO8DNNO88DO78O87~~77D=7OO$8OO?NZ?M...
     ...O8ODN$87O8DD8NNND8Z++II:77?7~$$OZOZ$7D+8. .
       .,8ONNDO~=Z8ZONNNNO=~==:8IN87I=OZZ7ID$7.,..
      ...,8OND88+DZ8DNNNNNDO87+D7D8$$:7Z7Z8Z?.....
   ...  ...DNN88NNDONNNNNNNNMZO7$DO8ZZ$7$O7, ...
       . ...MNDDDDO88DNNNNNDOOMD$DNN=77777,......
           ..,D8ZDO8NDNNNNDD88D78NND8O,$O$.....
           ...OONN8ODDNNNNDD8ZOI$DNDZ8Z7OO?..
        ... .O78O8OZ8DNNNNNNZOI=ODNDOZI~78+...
           ..8O$OZ88NNNNNNNDD8O+$ONN8DD$IZD...
           .ZDD8OZZDODNDDNNDD$8I+DDNND87O$$$...
        ....N7IOZ8O$O$DNNNNND7O$~,DNDOOZI+78...
         ..=8D8DDNDODZNNNDDNDZOZ:?8DNNZ$~O$8..  .
         ..8DN88DZDDDDNNNNDND8OOO$D8NN88$?=7?...
      . ...8OM8N8D8DDNNNNNNNDZO7Z+DDNN?I7IM8O...
      . ..MDDMODOND88NNN8NDMNO$$Z7ZINNMN$NNIZ,
        ..8O88888DDODDNNDNNNON78Z+Z8DMN=N:O8Z?..
       . .8888OD88M8DNNDONNNM+DOD,+Z$88OZIMD=I..
       . M8OOO8NDO+8NNNOOZZMDNN8DO?$ZZOO8IDDOM... .
     ....8878+8ZONNNN.DDOZ$..DDNDOIMI7888=8OOO+....
      ..MO88O8O=8D88..8DOOO..DNN8$.Z88Z=O8Z=DIZ.. .
      ..DO8ZO8Z$D=D8..ZD8Z. .NDNDM.?O8OZ$?8$8Z8....
   .  ..DIDDD8OO78D. .Z8DD...,DND$..8OOD87D$$IM=...
    ...8DDO7OI$D$MO. .OND8,...NDD.. .M88I~N7DIDI..
    ...DZDMOZ$O88.....DDODM...DDZ.....NZ77O?Z?88...
   ...8D~D7D8ZO: ....DDNDDM..DDD7......=O8D$DDOD...
    ..DOD8888....  ..8NNDD,.ZDD87...   ...DOZN+87...    .
   ...... ......... ,8DND8Z.D$MOO......  ...,O.$$...
     . ......... ....DNDDN+NDN8+8,...   . ........
           . . .  ...DDDDD.MDN,7Z$. .         ...
                 ....DDND:.MDMM.~Z.......     ..
              ......D8DDN..8DOZNIZ.........
        ... ... .7Z8DDDDDI,.NOD=O7.... ..
 ..    ..........:$$$NDNN87?MNI?IM..........
  .   ...........~D=?D$ODNNZD8=Z7=~++++++... .
       ..?????????8ZO7$Z8NNDN$O$?=++++++++....
      ...???????II$ZOZ7O+7OD$=OOI$=+++++++,...
      ...I+++++??I+D7O$8$?IM7:88=OO?++++++?..
      ...7++++++???$ZZZZZZ$Z$Z77~~+I+++++++:
    .....?+++++??I++8DO8DON87+,+7O$?++++++++...
    .....?+++++++??++?++?$ODZ$8$O$$Z=+++++++...

Introduction

Pazuzu controls applications. An application consists of workers. Each worker has a name and a command line. Each worker may have multiple instances of itself running.

For example, a typical application may consist of:

A Rails app, running under some web server like Thin or Unicorn.
A WebSockets app listening on a socket.
Some queue-processing daemons.
Some maintenance jobs that run now and then.

Applications are configured through a simple text format:

<key>:<command line>

For example:

main: bundle exec unicorn -c config/unicorn.rb config.ru
queue_processor: bundle exec ruby app/queue_processor.rb
web_socket_server: node lib/server.js

Pazuzu will control the lifecycle of each worker. Main points:

Workers can be started and stopped by issuing appropriate commands.
Workers will be monitored and respawned if they crash.
If the desired number of workers is changed, new worker instances will be spawned or terminated as needed.
Recent output from workers is kept so that it can be viewed.
If Pazuzu crashes, workers will continue to run, and will be reattached when Pazuzu is restarted.

Pazuzu leaves the following to external tools:

Resource limiting. Use the Linux cgroups tools to limit CPU and memory usage.
Scaling. Use external tools to monitor load, and invoke the pazuzu command to change the number of workers.

Installation

gem install pazuzu

Requirements

Ruby 1.9.1 or later.
Linux 2.6.24 or later with cgroups enabled in the kernel configuration.
The libcg binaries, which include cgcreate and cgexec.

Getting started

Create a minimal configuration /etc/pazuzu/pazuzu.conf:

applications:
  myapp:
    procfile: /srv/myapp

Create a minimal /srv/myapp/Procfile:

myworker: sleep 1h

Start Pazuzu (non-daemonized):

sudo pazuzud

Configuration file

log_path: Set the log file. Specify syslog to use syslog. Defaults to stderr.

socket_path: Where to create a socket for accepting commands. Defaults to /var/run/pazuzud.socket.

include: Include another configuration file or a number of files. May be either a single file name/glob pattern or an array of file names/glob patterns (eg., /etc/pazuzu/conf.d/*.conf). The included files's keys are merged into the current configuration in the order of inclusion.

cgroups: Has the following sub-keys:

hiearchy_root: Root of hierarchy for all cgroups created. Defaults to pazuzu.
subsystems: An array of cgroups subsystem to attach each cgroup to. Defaults to memory, cpu, cpuacct and blkio (essentially all subsystems at the time of writing).
fs_root: Where the cgroups file system is mounted. Defaults to /sys/fs/cgroup, which may or may not be the default for your Linux distribution.

applications: This is where applications are listed. Each application has its own key. For example:

applications:
  foo:
    procfile: /srv/foo

Each application has the following sub-keys:

procfile (required): The path to the procfile. If a directory, it's assumed that it contains a file named Procfile.
user: User name or UID to run the process as. Defaults to the user that Pazuzu runs as.
group: Group name or GID to run the process as. Defaults to the group that Pazuzu runs as.
workers: This key allows per-worker overrides. Each worker has its own key, which is the name specified in the Procfile.

For example:

applications:
  foo:
    procfile: /srv/foo
    workers:
      bar:
        num_instances: 2

Each worker has the following sub-keys:

command_line: Override the worker's command line.
num_instances: The number of instances to run. Defaults to 1.