unicorn-lockdown

unicorn-lockdown is a helper library for running Unicorn on OpenBSD with pledge, unveil, and fork+exec for increased security.

With unicorn-lockdown, unicorn should be started as the application user, which should be different than the user that owns the application’s files. unicorn will pledge the master process, then fork worker processes. The worker processes will re-exec (fork+exec), then load the application, then set unveil to restrict file system access, then pledge to limit the allowed system calls at runtime.

Assumptions

unicorn-lockdown assumes you are using OpenBSD 6.6+ with the nginx and rubyXY-unicorn and rubyXY-pledge packages installed, and that you have a unicorn symlink in the PATH to the appropriate unicornXY executable.

It also assumes you have a SMTP server listening on localhost port 25 to receive notification emails of worker crashes, if you are notifying for those.

Usage

unicorn-lockdown-setup

To start the process of setting up your system to use unicorn-lockdown, run the following as root after reading the file and understanding what it does.

unicorn-lockdown-setup

Briefly, the configuration this uses the following directories:

/var/www/sockets

Stores unix sockets that Unicorn listens on and Nginx uses.

/var/www/request-error-info

Stores temporary files for each request with request info, used for crash notifications

/var/log/unicorn

Stores unicorn log files, one per application

/var/log/nginx

Stores nginx log files, two per application, one for access and one for errors

This adds a _unicorn group that all application users will use as one of their groups, as well as a /etc/rc.d/rc.unicorn file that the application /etc/rc.d/unicorn_* files will use.

unicorn-lockdown-add

For each application you want to run with unicorn lockdown, run the following as root, again after reading the file and understanding what it does:

unicorn-lockdown-add -o $owner -u $user $app_name

Here’s the usage:

Usage: unicorn-lockdown-add -o owner -u user [options] app_name
Options:
    -c RACKUP_FILE                   rackup configuration file
    -d DIR                           application directory name
    -f UNICORN_FILE                  unicorn configuration file relative to application directory
    -o OWNER                         operating system application owner
    -u USER                          operating system user to run application
        --uid UID                    user id to use if creating the user when -U is specified
    -h, -?, --help                   Show this message

The -o and -u options are required. Default values for other options are:

-c

None, Unicorn will use config.ru by default.

-d

Same as app_name. The value provided should a relative path under /var/www.

-f

unicorn.conf. This file should be relative to dir.

--uid

The uid automatically generated by useradd.

The owner -o and the user -u should be different. The user is the user the application runs as, and should have read-only access to the application directory, other than locations where you want the application user to be able to modify files at runtime. The owner is the user that owns the application directory and can make modifications to the application.

unicorn-lockdown

unicorn-lockdown is the library required in your unicorn configuration file for the application, to handle configuring unicorn to run the app with fork+exec, unveil, and pledge.

When you run unicorn-lockdown-add, it will create the unicorn configuration file for the app if one does not already exist, looking similar to:

require 'unicorn-lockdown'

Unicorn.lockdown(self,
  :app=>"app_name",

  # Update this with correct email
  :email=>'root',

  # More pledges may be needed depending on application
  :pledge=>'rpath prot_exec inet unix flock',
  :master_pledge=>'rpath prot_exec cpath wpath inet proc exec',
  :master_execpledge=>'stdio rpath prot_exec inet unix cpath wpath unveil flock',

  # More unveils may be needed depending on application
  :unveil=>{
    'views'=>'r'
  },
  :dev_unveil=>{
    'models'=>'r'
  },
)

Unicorn.lockdown options:

:app

(required) a short string for the name of the application, used for socket/log file names and in notifications

:email

(optional) an email address to use for notifications when the worker process crashes or an unhandled exception is raised by the application or middleware.

:pledge

(required) a pledge string to limit the allowed system calls after privileges have been dropped

:master_pledge

(optional) The string to use when pledging the master process before spawning worker processes

:master_execpledge

(optional) The pledge string for processes spawned by the master process (i.e. worker processes before loading the app)

:unveil

(required) a hash of paths to limit file system access, passed to Pledge.unveil.

:dev_unveil

(optional) a hash of paths to limit file system, merged into the :unveil option paths if in the development environment. Useful if you are allowing more access in development, such as access needed for file reloading.

With this example pledge:

  • rpath is needed to read files

  • prot_exec is needed in most cases

  • unix is needed for the unix socket to nginx

  • inet is not needed in all cases, but most applications need some form of network access, and it is needed by default for emailing about exceptions that occur without process crashes. pf (OpenBSD’s firewall) should be used to limit access for the application’s operating system user to the minimum necessary access needed.

  • flock is needed in Ruby 3.1+ (not necessarily required in older Ruby versions).

With this example master pledge:

  • rpath is needed to read files

  • prot_exec is needed in most cases

  • cpath and wpath are needed to unlink the request error files

  • inet is needed to send emails for worker crashes

  • proc and exec are needed to spawn worker processes

With this examle master exec pledge:

  • stdio must be added because ruby-pledge doesn’t add it automatically to execpromises, and Ruby requires it

  • rpath, prot_exec, unix, inet are needed for the worker (see above)

  • cpath and wpath are needed to create the request error files

  • unveil is needed to restrict file system access

unicorn-lockdown has specific support for allowing emails to be sent for Unicorn worker crashes (e.g. pledge violations) and unhandled application exceptions (e.g. pledge violations). Additionally, unicorn-lockdown modifies unicorn’s process status line in a way that allows it to be controllable via OpenBSD’s rcctl program for stopping/starting/reloading/restarting daemons.

By default, Unicorn.lockdown limits the client_body_buffer_size to 11MB, with the expectation of an Nginx limit of 10MB, such that all client requests will be buffered in memory and unicorn will not need to write temporary files to disk. If this limit is not correct for your application, please call client_body_buffer_size after calling Unicorn.lockdown to set an appropriate limit. Note that rack still creates temporary files for file uploads by default, you’ll need to configure rack to disallow file uploads if your application does not need to accept uploaded files and you don’t want file upload attempts to cause pledge violations. With Roda, you can use the disallow_file_uploads plugin to prevent file upload attempts.

When Unicorn.lockdown is used with the :email option, if the worker process crashes, it will email the address using the contents specified by the request file. To make sure there is useful information to email in the case of a crash, you need to populate the request information for all requests. If you are using Roda, one way to do this is to use the error_email or error_mail plugins:

plugin :error_email, :from=>'[email protected]', :to=>'[email protected]',
  :prefix=>'[app_name]'
# or
plugin :error_mail, :from=>'[email protected]', :to=>'[email protected]',
  :prefix=>'[app_name]'

and then at the top of the route block, do:

if defined?(Unicorn) && Unicorn.respond_to?(:write_request)
  Unicorn.write_request(error_email_content("Unicorn Worker Process Crash"))
  # or
  Unicorn.write_request(error_mail_content("Unicorn Worker Process Crash"))
end

If you don’t have useful information in the request file, an email will still be sent for notification purposes, but it will not include request related information, which could make it difficult to diagnose the underlying problem.

roda-pg_disconnect

If you are using PostgreSQL as the database for the application, and using unix sockets to connect the application to the database, if the database is restarted, the application will no longer be able to connect to it unless you unveil the path the database socket (stored in /tmp by default). It can be a better approach security wise not to allow this, to prevent the application from being able to establish new database connections with potentially different credentials, as a mitigation in case the server is compromised.

To allow the application to handle cases where the database is disconnected, such as due to a restart of PostgreSQL, you can kill the worker process if a disconnect error is detected, and have the master process then spawn a new worker.

The roda-pg_disconnect plugin is a plugin for the roda web toolkit to kill the worker process after handling the connection if it detects the database connection has been lost. This plugin assumes the use of the Sequel database library and postgres adapter with the pg driver.

In your Roda application:

# Sometime before loading the error_handler plugin
plugin :pg_disconnect

To specifically restrict access to the database socket even when access to /tmp is allowed, you can unveil the database socket path with no permissions:

'/tmp/.s.PGSQL.5432'=>''

Note that there are potentially other security issues with unveiling access to /tmp beyond granting access to the database server, so it is recommended you do not unveil it. If the application needs a directory for temporary files (e.g. for handling uploaded files with rack), you can set the TMPDIR environment variable to an appropriate directory that is writable by the application user and not other users, and most web applications will respect that (assuming they use the tmpfile/tmpdir libraries in the standard library).

rack-email_exceptions

rack-email_exceptions is a rack middleware designed to wrap all other middleware and the application. It rescues unhandled exceptions raised by subsequent middleware or the application itself. Unicorn.lockdown will automatically setup this middleware if the :email option is used.

It is possible to use this middleware manually:

require 'rack/email_exceptions'
use Rack::EmailExceptions, "app_name", '[email protected]'

unveiler

unveiler is a library designed to help with testing applications that use pledge and unveil. If you are running your application pledged and unveiled, you want your tests to run pledged and unveiled to find problems.

unveiler assumes you are using minitest for testing. To use unveiler:

require 'minitest/autorun'
require 'unveiler'
at_exit do
  Unveiler.pledge_and_unveil('rpath prot_exec inet unix', 'views' => 'r')
end

autoload

As you’ll find out if you try to run your applications with unveil, autoload and other forms of runtime requires are the enemy. Both unicorn-lockdown and unveiler have support for handling common autoloaded constants in the rack and mail gems. If you use other gems that use autoload or runtime requires, you’ll have to add unveils for the appropriate gems:

Unicorn.lockdown(self,
  # ...
  :unveil=>{
    'views' => 'r',
    'gem-name' => :gem,
  }
)

Author

Jeremy Evans <[email protected]>