RecordLoader provides a simple and standardized way of populating databases from information described in a series of organized yaml files. It is intended to be used to generate a number of idempotent tasks, which can be run in both your production and development environments.
While written with ActiveRecord/Rails in mind, it is possible to use RecordLoader in different environments.
- Produce testable, reproducible data migrations across multiple environments
- Organize data into multiple files to provide context
- Add development environment specific data with .dev.yml files
- Keep work-in-progress isolated with .wip.yml files
- Rails generators to quickly create new record loaders
Add this line to your application's Gemfile:
And then execute:
Or install it yourself as:
gem install record_loader
If you are using Rails, you do not need to make any further changes, and all necessary hooks will be installed when generating your first record loader.
RecordLoader provides a generator to automatically build a loader, specs and the yaml files necessary to use it. In addition, the first time you use it it will automatically install the necessary rake files and configuration. You can access this by running:
bundle exec rails g record_loader
Which will return the documentation:
An example loader
Suppose you want to create a loader to maintain a selection of product types. You'll first use the generator:
$ bundle exec rails g record_loader product_type exist create config/record_loader/product_types/default_records.yml create lib/record_loader/product_type_loader.rb create lib/record_loader/tasks/record_loader/product_type.rake create spec/data/record_loader/product_types/product_types_basic.yml create spec/lib/record_loader/product_type_loader_spec.rb skip lib/record_loader/application_record_loader.rb identical lib/tasks/record_loader.rake
This will create several files:
Adds the record_loader:all rake task which can be used to trigger all record loaders.
Application specific base class for customization.
Example yaml file to begin populating with your record information. Record Loaders will load all yaml files from within
this directory, so it is possible to separate your records into multiple different files for better organization.
In addition yaml files ending in
.wip.yml exhibit special behaviour.
See dev and wip files.
The actual loader. It will look something like this:
# frozen_string_literal: true # This file was automatically generated via `rails g record_loader` # RecordLoader handles automatic population and updating of database records # across different environments # @see https://rubydoc.info/github/sanger/record_loader/ module RecordLoader # Creates the specified plate types if they are not present class ProductTypeLoader < ApplicationRecordLoader config_folder 'product_types' def create_or_update!(name, ) ProductType.create_with().find_or_create_by!(name: name) end end end
config_folder specifies which directory under
config/record_loader will be used to source the yaml files.
create_or_update! will create the actual records, and should be idempotent (ie. calling it multiple times will
have the same effect as calling it once).
create_or_update! will be called once for each entry in the yaml files,
with the first argument being the key, and the second argument being the value, usually a hash of options.
This contains the
record_loader:product_type which will trigger the record loader, and also ensures that
record_loader:product_type will get invoked on calling
A basic configuration for testing the loader. Tests use a separate directory to avoid coupling your specs to the data.
A basic rspec spec file for testing your loader. By default this just confirms that your loader creates the expected number of records, and that it is idempotent.
Dev and Wip files
Each loader can have one or more yaml files contained within its config directory. Most files will be aggregated together and only serve to provide a means of organization. However it is possible to add extra behaviour:
.dev.yml files will only be loaded in development environments. This is useful for seeding data for quick testing, but
which will not be needed in production environments. This may include test user accounts, dummy projects or quick
.wip.yml files will only be loaded if explicitly enabled via a WIP environmental variable. For example the file
my_feature.wip.yml will run if the WIP env is set to
my_feature. Multiple WIP flags can be set at the same time by
providing a comma separated list. eg.
If you have an existing feature flag system you can use this instead by adding a
wip_list method to
RecordLoader::ApplicationRecordLoader which returns an array of enabled feature names. For example:
def wip_list FeatureFlags.active.pluck(:name) end
Sometimes one loader will be dependent on the output of another. If this is the case, you can simply configure its rake task to use the other as a pre-requisite. Rake's dependency handling is smart enough to ensure each task only gets run once.
namespace :record_loader do desc 'Automatically generate Dependent through DependentLoader' task dependent: [:environment, 'record_loader:prerequisite'] do RecordLoader::DependentLoader.new.create! end end
Triggering on deployment
It can be useful to set
rake record_loader:all to run automatically on deployment.
It is recommend you trigger this after migrations have run.
Within Sanger PSD
This section is relevant to developers within the Sanger only. Other users of the Gem should hook into their own deployment systems as they see fit.
In Production Software Development at the Sanger we have the Ansible deployment project configured to run
rake application:post_deploy after migrations for selected applications. If you wish to take advantage of this
uncomment the appropriate line in
lib/tasks/record_loader.rake. You should also ensure that your application has
the post_deploy tasks enabled by setting the post_deploy to true for your application.
Currently this behaviour is configured as part of the 'rails' role, and will need to be set-up independently for any non-Rails ruby applications.
Non Rails Environments
In non-rails environments you can use the RecordLoader::Adapter::Basic adapter to avoid Rails specific functionality. This is the default adapter for RecordLoader::Base, although it is still recommended that you create an application specific class that inherits from this to allow for customization. Your custom record loaders can then inherit from this class instead.
See RecordLoader::Adapter for information about custom adapters.
After checking out the repo, run
bin/setup to install dependencies. Then, run
rake spec to run the tests. You can
bin/console for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run
bundle exec rake install. To release a new version, update the
version number in
version.rb, ensure the CHANGELOG.md is updated and that everything is committed.
bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push
.gem file to rubygems.org.
Bug reports and pull requests are welcome on GitHub at https://github.com/sanger/record_loader.
The gem is available as open source under the terms of the MIT License.