Class: Cumulus::CSV::DataFileManager

Inherits:
Object
  • Object
show all
Defined in:
lib/cumulus_csv/data_file_manager.rb

Overview

DataFileManager is the gatekeeper for sending your data files to S3, and for iterating over them later.

In the constructor, It takes the same authentication parameters as aws-s3:

DataFileManager.new(:access_key_id => 'abc',:secret_access_key => '123')

For storing your csv data file on S3, you need to setup a controller to send your uploaded files through this interface:

DataFileManager.new(connection_params).store_uploaded_file!(params[:uploaded_file])

The file will be posted to S3 in a bucket set aside for this gem (it will be created upon connection if it doesn’t exist already)

When you’re ready to iterate over this csv file later in a background job (or wherever), you’ll use this:

DataFileManager.new(connection_params).each_row_of(name) {|row| #...whatever processing you need }

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(connect_params) ⇒ DataFileManager

Returns a new instance of DataFileManager.



23
24
25
26
27
28
29
 
# File 'lib/cumulus_csv/data_file_manager.rb', line 23

def initialize(connect_params)               
  params = {}
  allowed_keys = [:access_key_id, :secret_access_key, :server, :port, :use_ssl, :persistent, :proxy]  
  connect_params.select{|k,v| allowed_keys.index(k)}.each{|arr| params[arr[0]] = arr[1]}
  AWS::S3::Base.establish_connection!(params)
  cache_bucket
end

Instance Attribute Details

#bucket ⇒ Object (readonly)

Returns the value of attribute bucket.



21
22
23
 
# File 'lib/cumulus_csv/data_file_manager.rb', line 21

def bucket
  @bucket
end

Instance Method Details

#each_row_of(file_name) ⇒ Object 



47
48
49
50
 
# File 'lib/cumulus_csv/data_file_manager.rb', line 47

def each_row_of(file_name) 
  data = AWS::S3::S3Object.value(file_name,BUCKET_NAME)
  ::CSV::Reader.parse(data).each{|row| yield row }
end

#fetch_file(file_name) ⇒ Object 



37
38
39
40
41
42
43
44
45
 
# File 'lib/cumulus_csv/data_file_manager.rb', line 37

def fetch_file(file_name)
  File.open(file_name,'w') do |file|
    AWS::S3::S3Object.stream(file_name, BUCKET_NAME) do |chunk| 
      file.write chunk
    end
  end
   
  return File.open(file_name,'r')
end

#store_uploaded_file!(uploaded_file) ⇒ Object 



31
32
33
34
35
 
# File 'lib/cumulus_csv/data_file_manager.rb', line 31

def store_uploaded_file!(uploaded_file)        
  name = File.basename(uploaded_file.original_filename)   
  AWS::S3::S3Object.store(name,uploaded_file.read,BUCKET_NAME)    
  return name
end