Engineer

It makes engines, get it?

Explanation

Engines are rails apps which can be run from inside other rails apps.

Engineer is a small gem that lets a sufficiently normal rails application quickly become an engine. It provides necessary bits of engine anatomy, and also gives the engine author a way to publish changes (such as migrations) to users as the engine evolves. This has been a sticking point in the past, and is one of the few remaining areas of engine support not yet covered by rails itself.

Engineer targets rails 3 engines hosted inside rails 3 applications. If you are looking for rails 2.3 support, try the engines plugin (rails-engines.org).

Getting Started

Let’s say you have a new rails app named my_engine that you wish to package as an engine. Drop this in your Gemfile and do the bundler thing:

gem "engineer"

A new generator named engineer:install will be available; run it.

$ rails g engineer:install
      exist  lib
     create  lib/my_engine/engine.rb
     create  lib/my_engine.rb
     create  lib/generators/my_engine/install/install_generator.rb
     create  lib/generators/my_engine/install/templates/my_engine.rake
     create  lib/generators/my_engine/install/USAGE
     create  app/controllers/my_engine
     create  app/controllers/my_engine/application_controller.rb
     remove  app/controllers/application_controller.rb
     append  Rakefile
       gsub  config/routes.rb

The two major take-aways from this are

application_controller.rb has moved under my_engine.
Your Rakefile has grown a bit.

The Gory Details below explain more deeply, but for now let’s just look at the new Rakefile content:

$ cat Rakefile
 # ...

 Engineer::Tasks.new do |gem|
   gem.name = "my_engine"
   gem.summary = %Q{TODO: one-line summary of your engine}
   gem.description = %Q{TODO: longer description of your engine}
   gem.email = "TODO"
   gem.homepage = "TODO"
   gem.authors = ["TODO"]
   gem.require_path = 'lib'
   gem.files =  FileList[
     "[A-Z]*",
     "{app,config,lib,public,spec,test,vendor}/**/*",
     "db/**/*.rb"
   ]

   # Include Bundler dependencies
   Bundler.definition.dependencies.each do |dependency|
     next if dependency.name == "engineer"

     if (dependency.groups & [:default, :production]).any?
       gem.add_dependency dependency.name, *dependency.requirement.as_list
     else
       gem.add_development_dependency dependency.name, *dependency.requirement.as_list
     end
   end

   # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
 end

If you’ve used jeweler (github.com/technicalpickles/jeweler) before, this should look eerily familiar. Engineer engines are shipped as gems, so engineer throws a (very light) wrapper around jeweler for wrangling the gem-related tasks. Jeweler in turn keeps your gem’s metadata in the Rakefile, so here it is.

Unlike jeweler, dependencies should still be declared in Gemfile, and not here.

As you can see, your bundler dependencies will be included in the generated gemspec.

Let’s make a gem:

$ rake build
 (in /Users/phil/Public/code/my_engine)
 Expected VERSION or VERSION.yml to exist. See version:write to create an initial one.

Whoops. Our gem needs to know what version it is, and we haven’t told it. Start off at 0.0.0 by

$ rake version:write
 (in /Users/phil/Public/code/my_engine)
 Updated version: 0.0.0

$ rake build
 (in /Users/phil/Public/code/my_engine)
 Generated: my_engine.gemspec
 my_engine.gemspec is valid.
 rake aborted!
 "FIXME" or "TODO" is not an author

 (See full trace by running task with --trace)

Doh. Remember all those TODOs (summary, author, etc) in the Rakefile metadata? Go fill those out.

Once more:

$ rake build
 (in /Users/phil/Public/code/my_engine)
 Generated: my_engine.gemspec
 my_engine.gemspec is valid.
 WARNING:  no rubyforge_project specified
   Successfully built RubyGem
   Name: my_engine
   Version: 0.0.0
   File: my_engine-0.0.0.gem

There we go, there’s an engine gem sitting in pkg/, go nuts. See jeweler’s documentation for managing the version, pushing to gemcutter and other goodies.

Installing Engine Gems

How about the other side of the fence? To install an engine into a host application, the host author follows a similar workflow.

First, add a line to the Gemfile and call bundler:

gem "my_engine"

A new generator will be available, named my_engine:install; run it.

$ rails g my_engine:install
        exist  lib/tasks
       create  lib/tasks/my_engine.rake
         rake  my_engine:assets my_engine:db:schema my_engine:db:migrate
 rm -rf /.../host/public/my_engine
 ln -s /.../gems/my_engine-0.0.0/public /.../host/public/my_engine
 mkdir -p db/migrate
 cp /tmp/20100428232715_my_engine_schema_after_create_comments.rb /.../host/db/migrate/20100428232715_my_engine_schema_after_create_comments.rb

This includes the engine’s static assets in a subdirectory under the host’s public. Voodoo in the engine takes care of looking there for assets (see Gory Details below.) If your OS is allergic to symlinks, the files are copied instead.

The engine’s schema is also added as a new database migration without running it. The host author is free to take a peek at it before deciding to rake db:migrate for real.

Run your pending migrations and fire up rails s, you’re good to go.

Managing Engine Gems

After installation, some new rake tasks are available:

$ rake -T my_engine
 (in /Users/phil/Public/code/host)
 rake my_engine:assets[copy]               # Link (or copy) my_engine's static assets
 rake my_engine:db:migrate                 # Import my_engine's new db migrations
 rake my_engine:db:schema                  # Import my_engine's schema as a db migration
 rake my_engine:db:seed                    # Load my_engine's seed data
 rake my_engine:update                     # Import my_engine's assets and new db migrations

There are catch-all tasks as well:

$ rake -T engines
 (in /Users/phil/Public/code/host)
 rake engines:assets[copy]                 # Link (or copy) static assets from all engines
 rake engines:db:migrate                   # Import new migrations from all engines
 rake engines:db:seed                      # Load seed data from all engines
 rake engines:update                       # Import assets and new db migrations from all engines

These let the host author manage the newly-installed (or updated!) engine. If the host application revs the version of the engine gem, any new engine db migrations can be imported into the host app with:

$ rake my_engine:db:migrate
 (in /Users/phil/Public/code/host)
 mkdir -p db/migrate
 cp /tmp/20100428232715_my_engine_create_tags.rb /.../host/db/migrate/20100428232715_my_engine_create_tags.rb
 cp /tmp/20100428232716_my_engine_create_taggings.rb /.../host/db/migrate/20100428232716_my_engine_create_taggings.rb

As before, this doesn’t actually run any engine migrations but instead copies new ones (with mild munging) to db/migrate.

The engine’s static assets (stylesheets, images and so on) can be updated with:

$ rake my_engine:assets
 (in /Users/phil/Public/code/host)
 rm -rf /Users/phil/Public/code/host/public/my_engine
 ln -s /.../gems/my_engine-0.0.1/public /.../host/public/my_engine

Even when using soft-links, updating the assets is important: you need the symlink pointing into the correct gem version.

One can do both in a single shot with rake my_engine:update. Or with rake engines:update to hit all engines at once.

Gory Details

Rails’ engine support has become robust with the release of rails 3. There are still a few pain points which engineer tries to address. It does so by making some decisions for you, the engine author. In the spirit of openness, the introduced voodoo is not buried inside the engineer gem but copied into the engine app on installation. It’s your gem, feel free to tailor it to your needs.

Some intrepid adventurers will ask for an explanation of the generated engine’s internals; here is an overview.

Database Migrations

Engine database migrations are a tricky problem. Luckily for me, some very clever people already hammered out a workable idea, which engineer implements.

The migrations are packaged in the gem along with the rest of the engine. When the host author updates the gem, she runs the <engine_name>:db:migrate rake task (directly, or indirectly through <engine_name>:update.) This copies the engine migrations into the host application, changing a few things on the way.

Specifically:

The migration numbers are reassigned.
The engine name is inserted into the new migration name.

The numbers are reassigned so that the copied migrations preserve their relative order, and yet occur after all previous migrations in the host application. This guarantees that the host application has a linear schema history, by making that history explicit. Put another way, the final host migrations should reflect the evolution of the host schema, not the engine schema from which it was derived.

The engine name is inserted to avoid name collisions. It allows the engine to see what migrations have already been copied into the host app; it will not attempt to copy them again. Changing the migration name implies changing the contained migration class name as well: the rake task will normally take care of it.

Interested readers could also check out

The original discussion: rails.lighthouseapp.com/projects/8994/tickets/2058
James Adam’s related blog post: interblah.net/plugin-migrations

Schemas

It is recommended practice to avoid running a long string of migrations when setting up a new database, since the migration process can become slow and brittle. Analogously, engines should be able to (and can) create their schemas in the host database without running a long string of migrations.

It is a very real use case for a host application author to add a new engine after the host schema has been created and deployed to production. Deployment tools like cap and vlad know how to run pending migrations at the right time, but they don’t understand (out of the box) how to run an engine-specific db:schema:load. It would be really great if they didn’t have to.

Engineer engines satisfy these two requirements by importing the engine schema into the host application as a migration. A naming convention is used to identify engine migrations that are implied by the schema: the schema migration will be named something like my_engine_schema_after_create_posts. This indicates that all engine migrations before (and including) create_posts should be considered to be already run. If there are no engine migrations to skip (because the engine author removed them) then the name my_engine_schema is used instead.

No effort is made to make this schema migration reversible.

Seeding

Similar to loading a schema, seeding initial database rows is another task that should only be run once per database.

Engineer engines provide two ways to load the engine’s db/seeds.rb. The first is a rake task, which the host-level rake db:seed depends on. This is meant for new databases being set up after the engine’s schema has been incorporated into the host’s. All seed rows will be loaded together at the usual time, just as all the tables were created together.

The other way seed data can be loaded is in a migration generated on engine installation. This is appropriate for installing a new engine into an established application, such as in a production deploy.

Static Assets and Asset Helpers

Separation of assets is another issue facing engines. While it is certainly useful to harness the host’s layouts and styling, engines will inescapably need their own stylesheets, scripts, etc.

Rails provides conventions about where those assets live and what they are named, and backs those conventions up with helpers that “just work.” The problem is (for example), the host and engine master stylesheets can’t both live at public/stylesheets/application.css. So we need two separate places to keep assets, and a way of hiding this separation when it is convenient.

Many http servers serve static assets more efficiently than dynamic ones generated from frameworks like rails. Any serious solution to the problem will need to respect this optimization. So, assuming the web server wants to be dumb, the separate asset locations will be apparent in the URIs. The browser only fetches the URIs the application gives it, so the application must know to render engine asset URIs for engine assets, and host asset URIs otherwise. These URIs are created by asset tag helpers such as image_tag and stylesheet_link_tag.

An engine author could visit each tag and stick (for example) my_engine/ in front of the asset names, but that stinks. It also breaks the engine when run as a normal application. Instead, the asset helpers called by the engine must create engine asset URIs only when the engine is run as such.

Enter asset_path. This is an action_controller configuration hook provided by rails to control how asset URIs are generated. It can be set on a controller class, and it is inherited by subclasses. By default, it provides rails’ cache-busting ability (the ?123line-noise456 on the end of your stylesheet URIs.) Engineer hijacks it to provide asset separation.

Recall that engineer’s install generator moves application_controller.rb into an engine-specific namespace: this is why. On startup, a MyEngine::Engine initializer sets an asset_path on MyEngine::ApplicationController. Since this controller is no longer the global ApplicationController, we’re ensured this asset_path will affect not only all the engine controllers, but only them. The initializers in MyEngine::Engine are only run when the application is started as an engine. Thus when my_engine is fired up as a normal application, the custom asset_path is not used.

Almost there. asset_path can take as value either a string template (such as "/my_engine%s") or a lambda. If the engine and host authors were not interested in sharing layouts and other views, "/my_engine%s" would be enough. There, all asset tags rendered by any view from an engine controller will target the engine’s assets. This goes bad when an engine view wants to render with a host layout that includes a stylesheet: the stylesheet URI would point into the engine, not the host.

A little more (and arguably too much) leg work can save us. The flaw is that we want the customized URI not when requesting an engine controller, but when rendering an engine view. If you have one around, open up lib/my_engine/engine.rb. There is another initializer in there that duck punches ActionView::Template#render. When a template is rendered, its identifier is captured. For templates loaded from files, this is a file system path descending from a config.paths.app.views of either the host or the engine. Thus we can distinguish host templates from engine templates. Engine assets are generated for engine views and non-file system views.

Running the Tests

The (sparse) tests are written with cucumber (cukes.info) and can be run with just “rake”. You may need to install jeweler first.

Thanks

This tool would not exist if the road had not already been well-paved by many others. Specifically, James Adam’s herculean effort in the rails 2.3 engines plugin has illuminated many issues and solutions. Thanks also to the rails core team for incorporating many engine-centric ideas into the framework, vastly simplifying what this tool needs to do. And also for just making an awesome framework.

Finally, thanks to SEOmoz (seomoz.org) for letting me build this at my desk in the afternoons instead of on the couch in the middle of the night ^_^.