BigShift

BigShift uploads your data to BigQuery.

Setup

$ bundle install

Run Specs

$ bundle exec rake

Regenerating VCR Cassettes

BigQuery Workspace

A BigQuery dataset is necessary to regenerate cassettes. The dataset may be named anything, but a table named insert_table is required and must have a field named field1 of type string.

If the cassettes have been regenerated against an existing dataset, the table named create_table will need to be deleted prior to generating vcr cassettes again.

Environment Variables

In order to regenerate vcr cassettes, some environment variables are required and should be added to a .env file in the root of the project:

BIG_SHIFT_PROJECT_ID           = [project-id]
BIG_SHIFT_DATASET_ID           = [dataset-id]
BIG_SHIFT_REFRESH_TOKEN        = [refresh-token]
BIG_SHIFT_GOOGLE_CLIENT_ID     = [google-client-id]
BIG_SHIFT_GOOGLE_CLIENT_SECRET = [google-client-secret]

Existing Values

Follow instructions here to retrieve values for each of the environment variables. You can also find instructions on finding a pre-defined list of variables there (sans the BIG_SHIFT_REFRESH_TOKEN as that is user-specific and you will be required to generate one).

Usage

All environment variables listed in the section on regenerating vcr cassettes must be set by any consumer of this gem.

Interface

All public endpoints are exposed in BigShift::Core.

Every response from the public API is wrapped in a Response object that will always have the same interface regardless of request. The Response#data attribute will be an object specific to the data requested.

create_table

This method requires only a schema to be specified:

schema = BigShift::Schema.new('MyTable')
  .add_field('field1', :string)
  .add_field('field2', :integer)
  .add_field('field3', :boolean)

BigShift.create_table schema

insert_rows

This method requires a table name and an array of objects to insert.

All objects are expected to respond to to_json.

rows = [{
  :field1 => 'field1-value-1',
  :field2 => 'field2-value-1',
}, {
  :field1 => 'field1-value-2',
  :field2 => 'field2-value-2',
}, {
  :field1 => 'field1-value-3',
  :field2 => 'field2-value-3',
}]

BigShift.insert_rows 'MyTable', rows

Deployment

This project makes use of branches to manage deployment. Pushing a new commit to the production branch will also build and push this gem to RubyGems.