Class: Remi::Loader::S3File

Inherits:
Remi::Loader show all
Includes:
DataSubject::S3File
Defined in:
lib/remi/data_subjects/s3_file.rb

Overview

S3 File loader Used to post files to Amazon S3

To use AWS KMS, supply a :ciphertext and optional :algorithm (default is AES256). The encrypted key stored in the ciphertext must be the same as that used for reading the file.

class MyJob < Remi::Job target :some_file do encoder Remi::Encoder::CsvFile.new loader Remi::Loader::S3File.new( credentials: { aws_access_key_id: ENV, aws_secret_access_key: ENV, region: 'us-west-2' }, bucket: 'itk-de-archive', remote_path: 'awesome.csv', kms_opt: { ciphertext: '' } ) end end

A ciphertext can be generated using the AWS SDK

require 'aws-sdk' require 'base64'

aws_credentials = Aws::Credentials.new( ENV, ENV )

kms = Aws::KMS::Client.new( region: 'us-west-2', credentials: aws_credentials )

See AWS docs for creating keys: http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html

data_key = kms.generate_data_key( key_id: 'alias/alias-of-kms-key', key_spec: 'AES_256' )

ciphertext = Base64.strict_encode64(data_key.ciphertext_blob) #=> "AQIDAHjmmRVcBAdMHsA9VUoJKgbW8niK2qL1qPcQ2OWEUlh5XAFw0vfl+QIgawB8cbAZ2OqXAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMIUIFFh++2w4d9al7AgEQgDvSRXQCOPLSMOjRS/lM5uxuyRV47qInlKKBIezIaYzXuFu1sRU+L46HqRyS0XqR4flFJ/fc8yEj3pU1UA=="

Examples:

Standard use


class MyJob < Remi::Job
  target :some_file do
    encoder Remi::Encoder::CsvFile.new
    loader Remi::Loader::S3File.new(
      credentials: {
        aws_access_key_id: ENV['AWS_ACCESS_KEY_ID'],
        aws_secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],
        region: 'us-west-2'
      },
      bucket: 'itk-de-archive',
      remote_path: 'awesome.csv'
    )
  end
end

job = MyJob.new
job.some_file.df = Daru::DataFrame.new(
  {
    numbers: [1,2,3],
    words: ['one', 'two', 'three']
  }
)
job.some_file.load

Using AWS KMS

Generating a ciphertext

Instance Attribute Summary collapse

Attributes included from DataSubject::S3File

#aws_credentials, #region

Attributes inherited from Remi::Loader

#context, #logger

Instance Method Summary collapse

Methods included from DataSubject::S3File

#encrypt_args, #init_aws_credentials, #init_kms, #s3

Methods inherited from Remi::Loader

#autoload, #fields

Constructor Details

#initialize(*args, **kargs, &block) ⇒ S3File

Returns a new instance of S3File.

Parameters:

  • bucket (String) —

    Name of S3 bucket containing the files

  • kms_opt (Hash) —

    Hash containing AWS KMS options

  • credentials (Hash) —

    Hash containing AWS credentials (must contain :aws_access_key_id, :aws_secret_access_key, :region)



241
242
243
244
 
# File 'lib/remi/data_subjects/s3_file.rb', line 241

def initialize(*args, **kargs, &block)
  super
  init_s3_loader(*args, **kargs, &block)
end

Instance Attribute Details

#bucket_name ⇒ Object (readonly)

Returns the value of attribute bucket_name.



247
248
249
 
# File 'lib/remi/data_subjects/s3_file.rb', line 247

def bucket_name
  @bucket_name
end

#remote_path ⇒ Object (readonly)

Returns the value of attribute remote_path.



246
247
248
 
# File 'lib/remi/data_subjects/s3_file.rb', line 246

def remote_path
  @remote_path
end

Instance Method Details

#load(data) ⇒ true

Copies data to S3

Parameters:

  • data (Object) —

    The path to the file in the temporary work location

Returns:

  • (true) —

    On success



252
253
254
255
256
257
258
 
# File 'lib/remi/data_subjects/s3_file.rb', line 252

def load(data)
  init_kms(@kms_opt)

  @logger.info "Writing file #{data} to S3 #{@bucket_name} as #{@remote_path}"
  s3.bucket(@bucket_name).object(@remote_path).upload_file(data, encrypt_args)
  true
end