Robocop Build Status

Robocop is a simple Rack middleware that inserts the X-Robots-Tag into the headers of all your responses.

The X-Robots-Tag can be used in place of a robots.txt file or meta tags to tell crawlers what they're allowed to do with your content. See this article for more information.

Tested against Ruby 1.8.7, 1.9.2, 1.9.3, and Rubinius & JRuby in both 1.8 and 1.9 mode.


The simplest way to install Robocop is to use Bundler.

Add Robocop to your Gemfile:

gem 'robocop'

Basic Usage


To use Robocop in your Rails application, add the following line to your application config file (config/application.rb for Rails 3, config/environment.rb for Rails 2):

config.middleware.use Robocop::Middleware do
  directives :all

Other Rack Applications (Sinatra, Padrino, etc.)

Simple add the following to your

use Robocop::Middleware do
  directives :all


The following directives can be passed in to Robocop's configuration:

  • all
  • noindex
  • nofollow
  • none
  • noarchive
  • nosnippet
  • noodp
  • notranslate
  • noimageindex

Directives (useragent agnostic)

If you just want to specify a list of directives for all useragents to follow, simply pass in a list of directives with the directive method:

config.middleware.use Robocop::Middleware do
  directives :noindex, :nofollow


If you want to give specific user agents unique sets of directives, you can do so by using the useragent method:

config.middleware.use Robocop::Middleware do
  useragent :googlebot do
    directive :all

  directives :noindex, :nofollow

It should be noted that if both the useragents and directives options are passed in, the useragents are output first in the header, followed by the generic directives.


  • Re-factor & DRY up code
  • Directive validation
  • Add support for unavailable_after directive.
  • Sanity checks for directives that are passed in. e.g. passing all, noindex, nofollow doesn't make any sense and should not be allowed.

Note on Patches / Pull Requests

  • Fork the project.
  • Code your feature addition or bug fix.
  • Add specs for it. This is important so we don't break it in a future version unintentionally.
  • Commit, do not mess with Rakefile or version number. If you want to have your own version, that's fine but bump version in a commit by itself so we can ignore it when merging.
  • Send a pull request. Bonus points for topic branches.

Copyright (c) 2012 PJ Kelly. See LICENSE for details.