Promotion

Promotion makes it possible to repeatedly deploy an application in a fast and reliable way.

Staging

A key concept is the staging area. It is a folder (/var/staging by default) where the files for each application are loaded into a named sub-folder (eg. /var/staging/myapp). For example, a staging area may look like this:

/var
 |__ staging
   |__ webapp1        -- a web application
   |__ monitor        -- sysadmin tool to monitor availability
   |__ analytics      -- analytics app for admins
   |__ snort          -- a custom config for snort

In this example, note that we also have an area for an installed application like snort. Although the binaries are installed through normal package installation, you may have a lot of time and work invested in the configuration. These files can of course be backed up and restored manually, but the purpose of Promotion is to make deployment fast and reliable. One way to achieve that is to store your valuable configuration files in source control, so you can reliably checkout the latest version for deployment each and every time. So the snort staging area need only hold a few configuration files.

Aside: why not just use version control?

The important files are spread all over the file system, so the only way to restore them all in a single command is to make the root directory a subversion working folder, add selected files to version control and then have .svn folders spread all over the place. Moreover, you’d need to run svn as root.

Deployment Descriptor

Each application has an XML deployment descriptor (eg. /var/staging/myapp/deploy.xml) that describes the conditions necessary for the successful operation of the application.

The deployment descriptor describes how to move the files from staging to their correct locations in the file system. This simple requirement unrolls into a chain of others:

  • we need the folder to be there to put the file in

  • the permissions need to be correct on the file and folder

  • the ownerships must be set

  • that means we need the necessary user accounts created

  • which means we first need the right groups set up.

In addition to installing the application’s files, there are system-wide configuration files that need to be adjusted to enable an application to startup or run properly:

  • /etc/profile

  • /etc/rc.conf.local

  • /etc/sudoers

  • /var/cron/tabs/*

Deployment descriptors can include recommended settings for these sensitive system files, but it does not change them. That would be too intrusive. Instead, Promotion reads the files to check if the recommended entries are present, and if they are not it displays a message for the admin explaining what needs to be done.

An important aspect is that a deployment is idempotent - you can do it as many times as you like and the result will be the same. This means its safe to make a small change to an application and re-deploy it in a few seconds, ready for testing.

Promotion is useful in a testing environment when changes and redeployments are frequent, but it shines in a production environment by reducing maintenance downtime and risk.

Promotion was originally designed for use on an OpenBSD system, but is configurable enough to work on any *nix system. Just modify the promotion/config.rb configuration file to suit your system’s paths.

Installation

WARNING

Promotion will makes changes to your operating system.

DO NOT USE ON A PRODUCTION SYSTEM without rigorous testing in a virtual machine first.

Install the promotion gem, which displays a post-installation message:

$ sudo gem install promotion

To install the executables issue the following command:

$ sudo ruby -rubygems -e "require 'promotion/install'"

This will install the executables (promote, evolve, devolve, mkdeploy) and create the staging area /var/staging.

Now you are ready to use Promotion to deploy other applications.

Usage

For each application you want to deploy, create its staging folder:

$ cd /var/staging
$ sudo svn checkout https://hosted/path/to/project/dist myapp

The project (or perhaps a dist folder designed for deployment) is checked out into a folder with the given name.

Now you can promote the application:

$ sudo promote myapp
Promoting myapp...OK.

If a database is involved, you can also migrate the database schema:

$ sudo evolve myapp
Evolving the database to version 1023

which will execute any new schema migration scripts. If you execute it again, you will see:

$ sudo evolve myapp
Already at version 1023.

If you want to revert to an earlier version of the database schema:

$ sudo devolve myapp 1017
Devolving the database to version 1017

which will execute the schema migration scripts in the devolve folder, in reverse order from 1023 down to 1017.

To make it easier to create a deployment descriptor, use the mkdeploy command:

$ cd /var/staging
$ mkdir myapp
$ cd myapp
$ mkdeploy
Deployment descriptor template written to deploy.xml.

If you already have a deploy.xml file, mkdeploy will not overwrite it:

$ mkdeploy
deploy.xml already exists in the current directory. Exiting.

Deployment descriptor syntax

The deployment descriptor is typically named deploy.xml and placed at the top of the application’s staging folder (eg. /var/staging/myapp).

It’s structure is described by the XML schema which can be found at at /var/staging/promotion/promotion.xsd or at finalstep.com.au/schema/promotion.v100.xsd

Specification

The top level element is a Specification

<?xml version="1.0" encoding="UTF-8" ?>
<Specification Name="myapp" Title="My Test App"
  xmlns="http://finalstep.com.au/promotion/v100"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://finalstep.com.au/schema/promotion.v100.xsd ../promotion/promotion.xsd">

which has a Name matching the staging folder it resides in (myapp), and a fuller Title.

Description

A description element is the first child:

<Description>
  The promotion manager ensures that the environment for an application
  is set up correctly, including folder and file permissions and ownership,
  startup scripts, sudoers privileges and environment variables.
</Description>

Groups

The sequence is then driven by dependencies: you cannot set ownerships until you have users, and you cannot create a user without a group, so Groups come first. Each of these elements is optional however.

<Groups>
  <Group Gid="502" Name="_mysql" />
</Groups>

This creates a group with group ID 502 and the name _mysql.

Users

We can also create an admin and a user to run the MySQL daemon:

<Users>
  <User Name="richard" Class="staff" Gecos="Richard Kernahan" Uid="1002" Gid="1002"
    Home="/home/richard" Shell="/bin/ksh" Groups="wheel webadmin"/>
  <User Name="_mysql" Class="daemon" Gecos="MySQL Account" Uid="502"Gid="502"
    Home="/nonexistent" Shell="/sbin/nologin"  />
</Users>

  • Name, Uid, and Gid are mandatory

  • Gecos defaults to the Name

  • Home for richard defaults to /home/richard

  • Shell defaults to /sbin/nologin

  • Groups is an optional space-separated list of groups the user should be added to

Folders

Before we can move files into place, we need the Folders set up:

<Folders Mode="0750" Owner="root" Group="wheel">
  <Folder Owner="_mysql" Group="_mysql">/var/mysql</Folder>
  <Folder>/home/myapp</Folder>
  <Folder Clear="true">/var/myapp</Folder>
</Folders>

You can have several Folders elements, if desired. The defaults for all Folder child elements may be set as attributes of the parent Folders element. In this example, all the folders will have permissions rwxr-x---, and ownership of root:wheel. Note that the folder /var/mysql has a different owner and group. Any Folder can override the default Mode, Owner, and Group.

The optional Clear attribute will clear the contents of the folder (actually it completely removes the folder and recreates it). This happens before files are copied to it of course.

Files

Finally we can specify how to move the files into place:

<Files Owner="root" Group="wheel" Mode="0644" Source="conf">
  <File Backup="true">/etc/my.cnf</File>
  <File>/etc/myapp.conf</File>
</Files>
<Files Owner="root" Group="wheel" Mode="0750" Source="sbin">
  <File>/usr/local/sbin/up!</File>
  <File>/usr/local/sbin/down!</File>
  <File>/usr/local/sbin/push!</File>
</Files>

In this example, we have two sets of files. The first set is expected to be found under the conf folder. Looking at the first file, we see the destination path /etc/my.cnf. The source file should have the same filename my.cnf and be located in the conf folder. This results in Promotion executing a command such as:

cp -f /var/staging/myapp/conf/my.cnf /etc/my.cnf

This allows a flatter, more convenient project folder structure, since the deployment descriptor maps the files into their proper operating system locations.

Note that the file my.cnf has an extra Backup attribute. This makes a backup of the original file to my.cnf-original once only. This is a good idea for important original configuration files such as snort.conf or httpd.conf. They often have hundreds of comments that are useful for reference.

As for Folders, the Files element can define the default Owner, Group and Mode for all File children.

Allfiles

A convenient shorthand for copying all the files in a folder to another folder is the Allfiles element. In this example, we will copy everything from the conf folder to the destination folder specified.

<Allfiles Group="bin" Mode="0644" Owner="root" Source="conf">/var/axonsec/conf</Allfiles>

Daemon

To enable an application to startup automatically, a Daemon element is needed. The name of the startup script is expected to be /etc/rc.d/myapp, but may be overridden by the optional Name attribute.

<Daemon Flags="" Priority="10" User="" />
<Daemon Flags="-d $MYAPP_HOME" Priority="20" User="_myapp" Name="otherapp"/>

The Flags attribute contains the command line options provided to the executable by /etc/rc.conf.local.

The lower the Priority value, the earlier that script is run (high values start later).

The User is the user running the process, typically an unprivileged user named _myapp. Leave blank to run the startup script as root (eg. as when dropping privileges).

Crontab

Cron jobs are often needed for an application to run smoothly. 

<Crontab>
  <Schedule User="root" Hour="2" Minute="7"
    Comment="Backup the entire database at 2:07am each morning">
    <Command><![CDATA[/usr/local/sbin/myappbak]]></Command>
  </Schedule>
</Crontab>

In this example, we specify a job to be added to root’s crontab, to backup the database at 2:07am each morning. The time specification is as defined in crontab(5):

  • Minute

  • Hour

  • DayOfMonth

  • Month

  • DayOfWeek

The Command is best wrapped safely in a CDATA section in case of XML-unsafe characters like $.

The Comment will be inserted into the final crontab file just before the job specification.

Database

If the application has a database, we need to specify the path to the DBMS client command line tool.

<Database>/usr/local/bin/mysql</Database>

If using SQLite3, a Database attribute is also needed to specify the file to operate on:

<Database database="/var/myapp/data/orders.db">/usr/bin/sqlite3</Database>

Newsyslog

Log files can be automatically rotated with newsyslog. This element specifies how to perform the log rotation:

<Newsyslog Owner="root" Group="wheel" Mode="0640" Count="5"
  When="M1D1" Zip="false" Binary="true" Restart="myappd">/var/log/myapp.log</Newsyslog>

In this example, the log file at /var/log/myapp.log will be rotated at 1am on the first of each month (newsyslog.conf schedule format $M1D1). 5 backup copies will be kept in addition to the original. It will not be compressed with bzip and no log rotation message will be inserted (Binary=“false”). The Restart attribute causes a command to be executed after the log file has been rotated:

/etc/rc.d/myappd restart

so the application can reopen its log files.

Database schema migration

Four components are required to automate database schema migration:

  1. Database migration scripts, stored in the evolve and devolve sub-folders of the application’s staging folder. These are normal migration scripts you might apply manually.

  2. A <Database> element in the deployment descriptor, containing the path to the database client command line tool. In the case of SQLite3, it also needs a database attribute containing the path of the database file to operate on.

  3. Privileges to allow the user to apply the migration scripts to the database, or else it will of course fail.

  4. Credentials for the user stored privately in ~/.my.cnf or ~/.pgpass (unless using SQLite3). This allows the scripts to be executed in batch mode, without prompting for a password.