SSHotgun - Utility library for writing scripts to administer and provision multiple servers or server farms.

The SSHotgun project homepage is sshotgun.rubyforge.org

To get help, post your questions on forums on RubyForge at rubyforge.org/forum/?group_id=6867

The author of SSHotgun is Vick Perry (vick.perry @nospam@ gmail.com)

DESCRIPTION:

SShotgun is a utility library for writing Unix/Linux server management and provisioning scripts in Ruby. I use it for remotely managing machines in server clusters/farms.

SShotgun calls your locally installed SSH client. Before you do anything with SSHotgun, insure that your SSH public keys are installed correctly, the ssh-agent is running, that you’ve entered your passphrase via ssh-add and you can successfully log into the remote servers without getting a password prompt.

FEATURES:

• Uses OpenSSH compatible SSH client and keys already installed on your local machine

• Optional use of a gateway machine for access to servers behind a firewall

• Logs in to remote machines via your public key already installed on all remote machines

• Automatically logs to the console and to a timestamped log file

• stdout and stderr from each remote machine is displayed and logged

• Can run sudo commands (your sudo password must be same on all remote machines)

• Multithreaded with configurable thread pool to service multiple machines simultaneously

• At end of processing run, a status report is displayed indicating successes, failures, timeouts, incompletes, etc.

EXAMPLE:

You don’t need to know much about Ruby to write powerful SSHotgun scripts. Just copy the sample script below and edit the hostnames and the doProcessing method as necessary.

The steps for writing a SSHotgun Ruby script are:

1. Create an array of fully qualified hostnames (fqdn).

2. Write your own custom processing class and doProcess method that will be invoked on all remote servers. You can copy the example for this.

3. Instantiate a SSHotgun object in the script.

4. Run the script. A Ruby script can be invoked two different ways:

Run Ruby and specify the script

ruby ./myrubyscript.rb

or

include this line as the first line of the script

#!/usr/bin/env ruby

then make the script executable

chmod 755 myrubyscript.rb

and run it directly from the command line

./myrubyscript.rb

This is a simple example of a SSHotgun Ruby script:

#!/usr/bin/env ruby

# Required libraries
require "highline/import"
require 'sshotgun'

# Create an array to contain hosts for processing
hostlist = Array::new()

# Create a host definition
host = HostDefinition.new("localhost")

# Add the host definition to the array of hosts
hostlist << host

# Create and add more hosts...
host = HostDefinition.new("another.host.junk")
hostlist << host

# Create your own custom processing class that subclasses 
# BaseProcessingClass.  In this class, define your own doProcessing method 
# where you write commands to execute on each remote server. 
class MyProcessingClass < BaseProcessingClass
  def doProcessing
    # Execute a command on all remote servers. The run command has a return
    # object from which you can obtain the output and exit status. See the
    # user guide for more details.
    run "ls -aF"

    # There are other sshotgun commands available for copying files, running
    # sudo, invoking local commands, etc.
  end
end

# Create an instance of SSHotgun and pass it the list of hosts and your
# custom processing class.
sshotgun = SSHotgun.new(hostlist, MyProcessingClass)

# Start processing. The start call should be the very last line in a SSHotgun
# script.
sshotgun.start

REQUIREMENTS:

• You must have an OpenSSH-compatible SSH client installed on your local machine.

• You must have correctly installed your SSH public keys on all remote servers that you access.

• For most advanced administrative tasks, you’ll also need sudo privleges on the remote servers.

• You’ll need write permissions in your present working directory (pwd) since SSHotgun will need to write its logs.

INSTALL:

Use gem to install SSHotgun. Note that you probably need to run this command with sudo or as root.

gem install sshotgun

USER GUIDE:

Here are snippets of SSHotgun Ruby code to illustrate SSHotgun’s capabilities.

At a minimum, a host definition contains the fully qualified domain name (fqdn) of a machine. As a best practice, DNS aliases are not generally referenced - only the fqdn. There are also several optional fields in a host definition that can be used for more selectivity when running a script. For example, you may wish to do some additional configuration for those machines categorized as “ldap” or “proxy”

A host definition has the following fields: hostname, category, os, desc

• hostname is the primary hostname or fqdn and NOT an alias.

• category is a freeform identifier for the type of the host. e.g. “wiki”, “ldap” (optional)

• os is a freeform string indicating os name + server/workstation + version. e.g. ubuntu_server_7.10 (optional)

• desc is a one sentence string describing main purpose of the host (optional)

An example of a complete host definition is:

host = HostDefinition.new("localhost", "test", "ubuntu_server_7.10", "test vm")
hostlist << host

You can share a list of hosts among several SSHotgun scripts. To do this, create a Ruby module (not a class) in a separate file. The module will contain a method that returns an array of host definitions. The module is referenced by any script that needs it.

# When you create your own modules, name the filename and module name with a similar name.  
# Note: This module filename is 'listofhosts.rb' - this is the reference in the require statement in the calling script.
module ListOfHosts

  # Create a method named <modulename>.getHostlist
  def ListOfHosts.getHostlist
    # Define, load and return the hostlist array
    hostlist = []

    host = HostDefinition.new("localhost", "test", "ubuntu_server_7.10", "test vm")
    hostlist << host

    host = HostDefinition.new("other.host.junk", "test", "ubuntu_server_7.10", "test vm")
    hostlist << host

    # return the array
    return hostlist
  end

end

It is a good practice to name the module file name similar or identical to the module name.

Here is a SSHotgun Ruby script that uses the list of hosts:

#!/usr/bin/env ruby

# Required libraries
require "highline/import"
require 'sshotgun'

# Create array to contain hosts for processing
hostlist = Array::new()

# Read hosts from an external Ruby module.
require 'listofhosts'

# In the listofhosts.rb file, the ListOfHosts getHostlist method returns an
# array of hosts. Concatenate all of the multiple arrays into a single array of
# hosts.
hostlist = hostlist + ListOfHosts.getHostlist

class MyProcessingClass < BaseProcessingClass
  def doProcessing
    run "ls -aF"
  end
end

sshotgun = SSHotgun.new(hostlist, MyProcessingClass)
sshotgun.start

Here’s how to prompt for the sudo password (Don’t hardcode a password into your script). See the advanced example for details.

sshotgun.sudopassword = ask(">>> Enter your sudo password:  ") { |q| q.echo = "*" } # or q.echo = false

Below is an advanced example that illustrates additional capabilities.

#!/usr/bin/env ruby

# Required libraries
require "highline/import"
require 'sshotgun'

# Create array to contain hosts for processing
hostlist = Array::new()
host = HostDefinition.new("someserver.junk", "test", "ubuntu_server_7.10", "test vm")
hostlist << host

# Import an external (shared) list of host definitions
require 'listofhosts'
hostlist = hostlist + ListOfHosts.getHostlist

class MyProcessingClass < BaseProcessingClass

  # You can add your own Ruby instance variables. These are useful if you
  # want to share data across your own custom methods.
  attr_accessor :myInstanceVariable

  # You MUST define a doProcessing method. Note that to make the doProcessing
  # method less cluttered, you can also create your own methods that are
  # called from within the doProcessing method. You can use your own accessor
  # variables (see above) for data sharing between methods. The doProcess method
  # is the only method called by the SSHotgun framework for each host.
  def doProcessing
    # Log the beginning of this method. This marker is useful for debugging problems.
    log "[" + @hostdef.hostname + "] info: " + "Processing started"

    # Sometimes a remote server is unavailable. I like to include a check at
    # top of the doProcessing method to stop processing for that server. This
    # snippet works because if a server is down, unreachable or non-responsive
    # then the id lookup for yourself will fail - when this happens call
    # "return" to drop out of this method.
    # Ruby's ENV contains the current user (you)
    currentUser = ENV["USER"].to_s
    runstatus = run "id " + currentUser
    if runstatus.exitstatus != 0
      log "[" + @hostdef.hostname + "] info: " + "Server not available or you cannot log in. Stopping processing for this server"

      # Set the status field for this host to indicate an abort. This status
      # will be displayed upon termination of the script
      @hostdef.status = "aborted"

      # return will end the processing of this host
      return
    else
      # set status to indicate that processing for this host has started
      @hostdef.status = "started"
    end

    # Compound commands in many shells may be created via the ";" semicolon
    # separator.
    run "ls -alF; printenv"

    # Escape any quotes (single or double) that are included within the command
    # string. Surround the shell command string with single quotes or %q() in
    # the sshotgun script so that Ruby won't interpolate the escaped characters
    # before passing them to the your local ssh client.
    run 'ls -aF | grep -i \".bash*\"'
    run %q(ls -aF | grep -i \".bash*\")   

    # Run the command under sudo on the remote machine. If you run runSudo
    # commands then the SSHotgun.sudoPassword must be set - uncomment the user
    # prompt that sets the sudoPassword at the bottom of this script. See the
    # gotcha below when running compound commands via sudo.
    #runSudo "somecommand"

    # A gotcha with sudo in unix is that is that security is tightened by
    # restricting the process environment. This means that compound commands
    # that use built-in shell functions such as "cd" or "pushd" will fail.  To
    # overcome this, first spawn a new shell, then pass it the commands to run.
    # Note that because the command string contains escaped characters, you
    # must enclose the entire command string in single quotes.
    #runSudo 'bash -c \"hostname;pushd /tmp;hostname;ls -alf;popd\"'

    # THIS WON'T WORK - SUDO can't find pushd and popd unless you spawn a shell first
    #runSudo "hostname;pushd /tmp;hostname;ls -alf;popd"  

    # On the remote server, the run and runSudo commands are executed in a
    # non-interactive shell. This means that many of the environment variables
    # that are set when you log in interactively will be missing. For debugging
    # purposes, call 'printenv' to view the environment variables.
    #run "printenv"

    # A useful technique is set environment variables via a compound command.
    #run "export http_proxy=aaa.bbb.com/8080/;wget www.mysite.junk/mypage"

    # The runstatus object is returned from all run and runSudo calls.
    # runstatus contains information about exit status, stdout and stderr from
    # the remote server.
    runstatus = run "ls -aF .bash*"

    # Use the log method to write to the log (the console). Don't use the Ruby
    # 'puts' method to write to the console because it isn't threadsafe and may
    # mix and garble simultaneous output by multiple threads.
    log "[" + @hostdef.hostname + "] info: " + "The exit status of the run command is " + runstatus.exitstatus.to_s

    # Remote stderr and stdout are captured and stored in the stdoutstr and
    # stderrstr variables. Learn about Ruby's regular expressions for examining
    # the output from a remote command.
    runstatus = run "id rumplestiltskin"

    # Check for "No such user" contained in stdoutstr. That's what is displayed
    # when the unix "id" command can't find the specified user. Note that the
    # the exit status of a failed 'id' command is not equal to zero either (see
    # below). This is also a Ruby regex comparison.
    if /No such user/ =~ runstatus.stdoutstr
      log "I failed to find user rumplestiltskin on this server"
    end

    # In some cases a remote command fails and you must stop processing for
    # that one server. Stop the execution of the doProcessing method by calling
    # Ruby's "return" statement. Don't call Ruby's "exit" statement because
    # that will halt all running threads.
    #runstatus = run "id rumplestiltskin"
    #if runstatus.exitstatus > 0
    #  log "The exit code was not 0, therefore I failed to find user rumplestiltskin"
    #  return  # stop processing for this host
    #end
    #log "You'll never get here...unless you have a user named rumplestiltskin"

    # The @hostdef instance variable contains the host definition for the
    # current host being processed. If you set category, os and desc to
    # meaningful information then you can do fancier branching logic in your
    # script. The @hostdef.hostname is very useful in log statements. The other
    # @hostdef fields are useful for treating a category or os differently than
    # the others.
    log "The current host is: " + @hostdef.hostname
    log "The current host's category is: " + @hostdef.category
    log "The current host's os is: " + @hostdef.os
    log "The current host's description is: " + @hostdef.desc
    log "The current host's status is: " + @hostdef.status

    # Target a specified host. This is a Ruby string comparison.
    if @hostdef.hostname == "192.168.1.224"
      log "[" + @hostdef.hostname + "] info: " + "Special processing for this host"
    end

    # Run a command locally (on your local machine). The runstatus variable is
    # returned if you need the data - same as run or runSudo.
    # runLocal <command> 
    #runLocal "ls -alF"

    # Run a sudo command locally. The runstatus variable is returned if you
    # need data from it. Set a sudoPassword if you are calling runLocalSudo.
    # runLocalSudo <command> 
    #runLocalSudo "ls -alF"

    # Create a remote file from a string parameter.
    # createRemoteFile <content string>, <remotefile>
    #createRemoteFile "this is your content here", "/home/vickp/testfile.txt"

    # Create a remote file and set mode
    # createRemoteFile <content string>, <remotefile>, <mode string>
    #createRemoteFile "this is your content here", "/home/vickp/testfile.sh", "0755"

    # Copy a remote file to the local machine. Note that you should vary the
    # name of the incoming local file else it will be overwritten when each
    # host's file is copied.
    # copyRemoteFileToLocal <remotefile>, <localfile>
    #copyRemoteFileToLocal "/home/vickp/testfile.txt", "/home/vickp/temp/" + @hostdef.hostname + "_testfile.txt" 

    # Copy a remote file to the local machine and set the mode of local file.
    # copyRemoteFileToLocal <remotefile>, <localfile>, <mode string>
    #copyRemoteFileToLocal "/home/vickp/testfile.txt", "/home/vickp/temp/" + @hostdef.hostname + "_testfile.txt", "0644"

    # Copy a local file to the remote machine.
    # copyRemoteFileToLocal <localfile>, <remotefile>
    #copyLocalFileToRemote "/home/vickp/temp/testfile.txt", "/home/vickp/testfile2.txt"

    # Copy a local file to the remote machine and set the mode of remote file.
    # copyRemoteFileToLocal <localfile>, <remotefile>, <mode string>
    #copyLocalFileToRemote "/home/vickp/temp/testfile.txt", "/home/vickp/testfile2.txt", "0644"

    # Other helpful SSHotgun and Ruby tips below
    # ===========================================

    # Ruby allows you to concatenate multiple command lines
    # Note use of shell command separator ';'
    cmd = "hostname;"
    cmd << "ls -alF | grep -i .bash;"
    cmd << "ls -alF | grep -i .profile"
    run cmd

    # Delete a file on the remote server.
    # Note use of -f with rm command
    #run "rm -f /path/to/remote/file"

    # Determine if a directory exists on the remote server
    runstatus = run 'test -d \"/tmp\"'
    if runstatus.exitstatus > 0
      log "[" + @hostdef.hostname + "] info: " + "The /tmp directory does NOT exist"
    else
      log "[" + @hostdef.hostname + "] info: " + "The /tmp directory exists"
    end

    # Determine if a file exists on the remote server
    runstatus = run 'test -e \".profile\"'
    if runstatus.exitstatus > 0
      log "[" + @hostdef.hostname + "] info: " + ".profile does NOT exist"
    else
      log "[" + @hostdef.hostname + "] info: " + ".profile exists"
    end

    # Call your own methods to unclutter and better organize your doProcess
    # method code. In most of my SSHotgun scripts the doProcess method is
    # fairly sparse - most of the work is done is various custom methods that
    # are called within the doProcess method
    #myOwnMethod

    # The SSHotgun instance variables are visible in this method. 
    log "[" + @hostdef.hostname + "] info: " + "The script started at: " + @sshotgun.startTime.to_s

    # Sometimes I either install unsigned debian packages of my own
    # or need to force the installation of files (overwrite) that
    # belong to another package. Aptitude doesn't yet support forcing yes for
    # this sort of thing but apt-get does...
    #runSudo "apt-get install -y --force-yes myUntrustedPackage -o DPkg::options::='--force-overwrite'"

    # For double quoted strings, Ruby will do inline substitution for a
    # variable enbedded within "#{myvariablehere}". This is useful when you
    # want a variable substituted inside of double quotes
    #createRemoteFile "deb http://debrepo/global ./", "/home/#{currentUser}/sources.list", "0644"

    # Create a global variable if you need to prompt for additional information
    # (such as userid, date range, etc) and want to use that information in
    # your doProcessing method. Global variables begin with '$'. See bottom of
    # script where the variable is created. Note that "id" returns with an exit
    # status of 1 if the user does not exist.
    runstatus = run "id #{$userid}"
    if runstatus.exitstatus == 0
      log "[" + @hostdef.hostname + "] info: " + "Found user " + $userid
    else
      log "[" + @hostdef.hostname + "] info: " + "DID NOT find user " + $userid
    end

    # See example of how to update a debian/ubuntu sources.list package repository file
    # Also see use of case statement in the method body.
    #installSourcesListAndUpdate

    # You can set the exit code of this script
    #@sshotgun.exitCode = 3

    # Log the end of this method.
    log "[" + @hostdef.hostname + "] info: " + "Processing completed"

    # Set the status field for this host to indicate successful processing.
    @hostdef.status = "finished"
  end
end

# Your own custom methods can be called from within the doProcess method
def myOwnMethod
  log "[" + @hostdef.hostname + "] info: " + "Calling my own method"
end

# Update a debian/ubuntu /etc/apt/sources.list
def installSourcesListAndUpdate
  # WARNING: This only works if you fill in the category field in your hostdefs.
  #
  # Copy production sources.list into your home directory, then copy it into
  # place. Note this is a two step operation because you can't create the
  # remote file in a place where you don't have perms. The second step is to
  # use sudo and copy the file into place.
  s = ""
  case @hostdef.category
  when "production"
    f = File.new("/usr/local/bin/adminscripts_templates/production_ubuntuserver710i386.sources.list")
  when "devint"
    f = File.new("/usr/local/bin/adminscripts_templates/devint_ubuntuserver710i386.sources.list")
  when "qaint"
    f = File.new("/usr/local/bin/adminscripts_templates/qaint_ubuntuserver710i386.sources.list")
  when "staging"
    f = File.new("/usr/local/bin/adminscripts_templates/staging_ubuntuserver710i386.sources.list")
  else
    log "[" + @hostdef.hostname + "] ERROR: " + "Unknown hostdef category = " + @hostdef.category
    return
  end
  # read the file and concatenate it into a single string
  f.each_line do |line|
    s << line
  end
  createRemoteFile s, "/home/#{$currentUser}/sources.list", "0644"
  runSudo "cp /home/#{$currentUser}/sources.list /etc/apt/sources.list"
  run "rm /home/#{$currentUser}/sources.list"

  # after changing sources.list, update package manager to refresh package list
  runSudo "aptitude update"
end

# Create an instance of SSHotgun and pass it the list of hosts and your custom
# processing class.
sshotgun = SSHotgun.new(hostlist, MyProcessingClass)

# If you are calling a sudo command then prompt the user for the sudo password.
# I recommend that you NEVER HARDCODE A PASSWORD IN A SCRIPT!  Configure the
# ask command to hide the password as the user types it.  Note that the you
# must have the highline gem package installed for the "ask" command
#sshotgun.sudopassword = ask(">>> Enter your sudo password:  ") { |q| q.echo = "*" } # or q.echo = false

# If you need to forward all calls via a gateway ssh server, set it here.  Once
# set, all ssh connections go through the gateway machine
#sshotgun.gatewayhost = "someserver.somedomain.com"

# You can also use the 'ask' command to prompt the user for any other
# information. Configure the ask command to display the string as the user
# types it. Be sure to test password-less login from your gateway server to
# clear out any prompts asking you to verify the authenticity of the remote
# host.
#gatewayhost = ask(">>> Enter the hostname of the gateway server:  ") { |q| q.echo = true }
#sshotgun.gatewayhost = gatewayhost
#sshotgun.gatewayhost = "mygateway.hostname.junk"

# Create and use a global variable to get additional data into your doProcessing method.
$userid = ask(">>> Enter the userid to find:  ") { |q| q.echo = true }

# Number of simultaneous processing threads (hosts) to run. In terms of load to
# your local machine, each processing thread equates to about one ssh client
# running. Default is 50
# sshotgun.maxThreads = 30    

# A processing thread will be killed after this amount of time. Default is 30 minutes.
# sshotgun.threadTimeoutInSeconds = 900

# Period for displaying status information about processing threads that
# are still running. Default is 30
# sshotgun.monitorPollIntervalInSeconds = 20

# Any output is also written to a file logger. You can disable the file logging
# by setting isFileLogging to false.
# sshotgun.isFileLogging = false.

# Turn on debug mode
# sshotgun.isDebug = true

# What should the behavior be when a command returns a non-zero exit status? A
# non-zero exit status usually indicates failure of the command. Some sshotgun
# users wish to immediately stop the processing for that server and stop it's
# thread. Other people will check the exit status after each command and decide
# whether to continue or return from the processing method.  Note that if
# isStopOnNonZeroExit is true then the processing thread exits immediately and
# you never have the opportunity to check the exit status in your processing
# method code. The default is false (don't stop on non-zero exit).
# sshotgun.isStopOnNonZeroExit = true

# Start processing. The start call should be the very last line in a SSHotgun
# script.
sshotgun.start

LICENSE:

(Simplified BSD License)

Copyright © 2008, Vick Perry All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

• Neither the name of the <ORGANIZATION> nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.