What is rake-compiler?

rake-compiler is first and foremost a productivity tool for Ruby developers. It’s goal is to make the busy developer’s life easier by simplifying the building and packaging of Ruby extensions by simplifying code and reducing duplication.

It follows *convention over configuration* by advocating a standardized build and package structure for both C and Java based RubyGems.

rake-compiler is the result of many hard-won experiences dealing with several diverse RubyGems that provided native extensions for different platforms and different user configurations in different ways. Details such as differences in code portability, differences in code clarity, and differences in project directory structure often made it very difficult for newcomers to those RubyGems.

From these challenges, rake-compiler was born with the single-minded goal of making the busy RubyGem developer’s life much less difficult.

Feature Overview

Some of the benefits rake-compiler provides include:

  • No custom rake tasks required. Less code duplication and errors.

  • Painlessly build extensions on different platforms (Linux, OSX and Windows).

  • Painlessly build extensions for different Ruby implementations (JRuby, Rubinius and MRI).

  • Allow multiple extensions to be compiled inside the same gem.

  • Painlessly build “fat” native gems for Windows users (from Linux or OSX).

  • Mimics RubyGems standard installation process, helping as a test environment.

  • Simplifies cross platform extension compilation (targeting Windows from Linux).

OK, I’m sold! Show me how to install it!

Simple:

$ gem install rake-compiler

That’s easy. How do I use it?

Now that you have installed rake-compiler, it’s time to give your project a standardized structure.

Using a standardized project structure

Let’s say you want to compile an extension called ‘hello_world’. Organizing your project’s code tree in the following way will help rake-compiler do its job:

|-- ext
|   `-- hello_world
|       |-- extconf.rb
|       |-- HelloWorldService.java
|       `-- hello_world.c
|-- lib
`-- Rakefile

TIP: Having a consistent project directory structure will help developers and newcomers find and understand your code, making it easier for them to contribute back to your project.

Adding the code to enable rake-compiler

Now the fun part. It’s time to introduce the code to your projects Rakefile to tell it to use rake-compiler to build your extension:

# File: Rakefile

require 'rake/extensiontask'

Rake::ExtensionTask.new('hello_world')

That’s it? Yes, that’s it! No other lines of code are needed for rake-compiler to work its magic.

If you want to do the same for a JRuby extension written in Java, it’s just as easy:

# File: Rakefile

require 'rake/javaextensiontask'

Rake::JavaExtensionTask.new('hello_world')

The simple process

Those two simple lines of code automatically added the Rake tasks needed to build your ‘hello_world’ extension. For example, checking the Rake tasks on MRI Ruby 1.8.x/1.9 returns something similar to:

$ rake -T
(in /home/user/my_extension)
rake compile                # Compile the extension(s)
rake compile:hello_world    # Compile just the hello_world extension

Simply calling compile like

$ rake compile

performs the entire compile and build process for you and places the resulting extension inside the lib directory of your project.

NOTE: Please be aware that building C extensions requires the proper development environment for your Platform, including libraries, headers and build tools. Check your distro / vendor documentation on how to install these development resources.

NOTE: Building Java extensions requires the javac, part of the Java Development Kit (JDK). This should be included by default on Mac OS X, and downloadable from java.sun.com for other operating systems.

Generating native RubyGems

A common usage scenario for rake-compiler is generating native gems that bundle your extensions. As mentioned above, if you have your development environment configured correctly, the following examples work even when building native gems on Windows systems.

Creating native gems is really easy with rake-compiler’s Rake::ExtensionTask:

# somewhere in your Rakefile, define your gem spec
spec = Gem::Specification.new do |s|
  s.name = "my_gem"
  s.platform = Gem::Platform::RUBY
  s.extensions = FileList["ext/**/extconf.rb"]
end

# add your default gem packing task
Gem::PackageTask.new(spec) do |pkg|
end

# feed the ExtensionTask with your spec
Rake::ExtensionTask.new('hello_world', spec)

As expected, you can still build your pure-ruby gem in the usual way (standard output) by running:

$ rake gem
(in /projects/oss/my_gem.git)
mkdir -p pkg
  Successfully built RubyGem
  Name: my_gem
  Version: 0.1.0
  File: my_gem-0.1.0.gem
mv my_gem-0.1.0.gem pkg/my_gem-0.1.0.gem

Plus, rake-compiler tasks give you the extra functionality needed to build native gems by running:

# rake native gem
(... compilation output ...)
mkdir -p pkg
  Successfully built RubyGem
  Name: my_gem
  Version: 0.1.0
  File: my_gem-0.1.0.gem
mv my_gem-0.1.0.gem pkg/my_gem-0.1.0.gem
  Successfully built RubyGem
  Name: my_gem
  Version: 0.1.0
  File: my_gem-0.1.0-x86-mingw32.gem
mv my_gem-0.1.0-x86-mingw32.gem pkg/my_gem-0.1.0-x86-mingw32.gem

Did you notice that you get two gems for the price of one? How’s that for a time saver?

Similarly, it’s just as easy to do the same thing for JRuby extensions:

# rake java gem
(... compilation output ...)
mkdir -p pkg
  Successfully built RubyGem
  Name: my_gem
  Version: 0.1.0
  File: my_gem-0.1.0.gem
mv my_gem-0.1.0.gem pkg/my_gem-0.1.0.gem
  Successfully built RubyGem
  Name: my_gem
  Version: 0.1.0
  File: my_gem-0.1.0-java.gem
mv my_gem-0.1.0-java.gem pkg/my_gem-0.1.0-java.gem

Great, but can I use a non-standard project structure?

Yes you can! While the conventional project structure is recommended, you may want, or need, to tweak those conventions. Rake-compiler allows you to customize several settings for Rake::ExtensionTask:

Rake::ExtensionTask.new do |ext|
  ext.name = 'hello_world'                # indicate the name of the extension.
  ext.ext_dir = 'ext/weird_world'         # search for 'hello_world' inside it.
  ext.lib_dir = 'lib/my_lib'              # put binaries into this folder.
  ext.config_script = 'custom_extconf.rb' # use instead of the default 'extconf.rb'.
  ext.tmp_dir = 'tmp'                     # temporary folder used during compilation.
  ext.source_pattern = "*.{c,cpp}"        # monitor file changes to allow simple rebuild.
  ext.config_options << '--with-foo'      # supply additional options to configure script.
  ext.gem_spec = spec                     # optionally indicate which gem specification
                                          # will be used.
end

Cross compilation - the future is now.

Rake-compiler also provides a standardized way to generate, from either Linux or OSX, extensions and gem binaries for your Windows users!

How can this be you say? Simple, rake-compiler’s cross compilation features take advantage of GCC’s host/target capabilities to build ‘target’ binaries on different ‘host’ OS’s.

How I do this from Linux or OSX?

In addition to having the development tool chain installed (GCC), you also need to install your platform’s mingw32 cross compilation package.

Installation depends upon your operating system/distribution. On Ubuntu and Debian host machines, a simple apt-get install mingw32 will be enough.

On Arch, mingw32 is installed by running pacman -S mingw32-gcc

On OSX, we no longer recommend the usage of MacPorts mingw32 package because it stagnated in GCC version 3.4.5.

Instead we recommend you download mingw-w64 automated build packages available at SourceForge:

sourceforge.net/downloads/mingw-w64/

Browse into Toolchains targetting Win32 and then Automated Builds.

Files will be ordered by recency, find the latest one with version 1.0 in it, like this one:

mingw-w32-1.0-bin_i686-darwin_20110422.tar.bz2

Download and extract. After that, make sure the bin directory is added to the PATH, eg:

export PATH=~/mingw-w64/w32/bin:$PATH

You can add this to your .profile to avoid the repitition.

I’ve got my tool-chain installed, now what?

First, you need to build Ruby for Windows on your Linux or OSX system.

Relax, no need to freak out! Let rake-compiler do all the heavy lifting for you:

rake-compiler cross-ruby

And you’re done. It will automatically download, configure and compile the latest stable version of Ruby for Windows, and place it into your ~/.rake-compiler directory.

This will create ~/.rake-compiler/config.yml file so that rake-compiler knows where to find the rbconfig.rb file that matches the Ruby version on the Windows host system you’re cross-compiling for. An example:

# File: ~/.rake-compiler/config.yml

rbconfig-1.8.6: /path/to/ruby-1.8.6/rbconfig.rb
rbconfig-1.8.7: /path/to/ruby-1.8.7/rbconfig.rb
rbconfig-1.9.2: /path/to/ruby-1.9.2/rbconfig.rb

If, instead, you want to build a different Ruby version than the default one, please supply a VERSION:

rake-compiler cross-ruby VERSION=1.8.6-p114

If you, like me, have multiple versions of MinGW packages installed, you can specify the HOST that will be used to cross compile Ruby:

rake-compiler cross-ruby HOST=i386-mingw32 # (OSX mingw32 port)

The host will vary depending on provider (mingw32 versus mingw-w64 projects). Please consult the documentation and website of the MinGW package provider before reporting any issues.

OK, let’s cross compile some gems!

Now, you only need specify a few additional options in your extension definition:

Rake::ExtensionTask.new('my_extension', gem_spec) do |ext|
  ext.cross_compile = true                # enable cross compilation (requires cross compile toolchain)
  ext.cross_platform = 'i386-mswin32'     # forces the Windows platform instead of the default one
                                          # configure options only for cross compile
  ext.cross_config_options << '--with-something'

  # perform alterations on the gemspec when cross compiling
  ext.cross_compiling do |gem_spec|
    gem_spec.post_install_message = "You installed the binary version of this gem!"
  end
end

By default, cross compilation targets ‘i386-mingw32’ which is the default GCC platform for Ruby.

To target gems for MRI Ruby’s current official distribution, please force the platform to the one (i386-mswin32) previously shown.

Cross compiling is still very simple:

rake cross compile

And now, building gems for your Windows users is just 5 more letters:

rake cross native gem

And you’re done, yeah.

But wait, there’s more

You can specify which version of Ruby to build the extension against:

rake cross compile RUBY_CC_VERSION=1.8.6

For example, if you installed 1.9.2, you can do:

rake cross compile RUBY_CC_VERSION=1.9.2

Even better, you can target multiple versions (ie. 1.8.6 and 1.9.2) in the same gem via:

rake cross compile RUBY_CC_VERSION=1.8.6:1.9.2

And better yet, you can bundle both binary extensions into one so-called “fat” gem via:

rake cross native gem RUBY_CC_VERSION=1.8.6:1.9.2

That will place binaries for both the 1.8 and 1.9 versions of your Ruby extensions inside your project’s lib_dir directory:

lib/1.8/my_extension.so
lib/1.9/my_extension.so

NOTE: building “fat” gems is currently only supported by rake-compiler when cross compiling from a Linux or OSX host. Patches are welcome if building “fat” gems from Windows hosts is desired, or natively for your platform :-)

Now is up to you to make your gem load the proper binary at runtime:

begin
  RUBY_VERSION =~ /(\d+.\d+)/
  require "#{$1}/my_extension"
rescue LoadError
  require "my_extension"
end

The above technique will lookup first for 1.8 or 1.9 version of the extension and when not found, will look for the plain extension.

This approach catch the cases of provided fat binaries or gems compiled by the end user installing the gem. It has also been implemented successfully in several projects.

What are you talking about? (Give me examples)

I know all the above sounds like a complete foreign language (it does even for me!). So, what if I show you some examples?

Check our wiki with links to the proper rake files used by many developers and projects and how they use rake-compiler.

github.com/luislavena/rake-compiler/wiki/projects-using-rake-compiler

Future

rake-compiler is a work in progress and we appreciate any and all feedback during the development of it! (and contributions too!)

You can find more information about rake-compiler:

Disclaimer

If you have any trouble, don’t hesitate to contact the author. As always, I’m not going to say “Use at your own risk” because I don’t want this library to be risky.

If you trip on something, I’ll share the liability by repairing things as quickly as I can. Your responsibility is to report the inadequacies.