simple_importer

Simple API for importing csv files.

Defining importers

The basic importer has a name, file, and a foreach block.

importer :items do
  file 'items.csv'

  foreach do |row|
    Item.create(:name => row[:name])
  end
end

simple_importer uses fastercsv, which replaces the csv standard lib in ruby1.9. Importers have configuration options that match those of fastercsv. The foreach block is called for each row in the csv file. The argument passed into the foreach block is a fastercsv row created with the specified configuration.

importer :items do
  file 'items.csv'

  # all fastercsv options are supported
  headers false
  converters :numeric
  # etc..

  foreach do |row|
    Item.create(:name => row[:name])
  end
end

Importers have a default configuration that matches the following importer.

importer :items do
  file 'items.csv'

  # default configuration
  col_sep ","
  row_sep :auto
  quote_char '"'
  field_size_limit nil
  converters :all
  unconverted_fields nil
  headers true
  return_headers false
  header_converters :symbol
  skip_blanks false
  force_quotes false
  # end default configuration

  foreach do |row|
    Item.create(:name => row[:name])
  end
end

Before callback

Define before_filter-style callbacks in the form of the before block. This block is run once before the importer runs.

importer :items do
  file 'items.csv'

  before do
    Item.delete_all
  end

  foreach do |row|
    Item.create(:name => row[:name])
  end
end

File processing

Passing an array of file paths to the file configuration field will process all the rows in each file.

importer :items do
  file Dir.glob("data/*.csv")

  foreach do |row|
    Item.create(:name => row[:name])
  end
end

Use the foreach_file block to add per-file behavior.

importer :items do
  file Dir.glob("data/*.csv")

  foreach_file do |file|
    List.create(:name => File.basename(file, ".csv"))
  end

  foreach do |row|
    Item.create(:name => row[:name])
  end
end

Nest the foreach block to give your row processing block access to foreach_file block variables.

importer :items do
  file Dir.glob("data/*.csv")

  foreach_file do |file|
    list = List.create(:name => File.basename(file, ".csv"))

    foreach do |row|
      Item.create(:name => row[:name], :list => list)
    end
  end
end

Loading and running importers

simple_importer provides a method that will find and load importer definitions in the importers, lib/importers, and app/importers directories.

# importer definition is in lib/importers/item.rb
SimpleImporter.find_importers

SimpleImporter now has a collection of importers and each importer can be run now that it has been loaded.

# the following will fire off the actual importers
SimpleImporter.importers.each do |importer|
  importer.run
end

Rake tasks and autoloaded importers

simple_importer provides rake tasks to run importers individually or all at once.

# add this to your rakefile
# lib/tasks/simple_importer.rake is a good place in your rails app
require 'simple_importer/tasks'

rake -T simple_importer will show a rake task for each importer as well as an import task which runs all of the importers. Descriptions defined in the importer will be used for the rake task description.

importer :items do
  file 'items.csv'
  desc 'Import all of the cool items'

  ...
end

Requirements

fastercsv (if you are not using ruby 1.9)

Note on Patches/Pull Requests

Fork the project.
Make your feature addition or bug fix.
Add tests for it. This is important so I don’t break it in a future version unintentionally.
Commit, do not mess with rakefile, version, or history. (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
Send me a pull request. Bonus points for topic branches.