Class: S3Publisher

Inherits:
Object
  • Object
show all
Defined in:
lib/s3-publisher.rb

Overview

You can either use the block syntax, or:

  • instantiate a class

  • queue data to be published with push

  • call run to actually upload the data to S3

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(bucket_name, opts = {}) ⇒ S3Publisher

Pass the publisher a bucket_name along with any of the following options:

  • base_path - path prepended to supplied file_name on upload

  • logger - a logger object to recieve ‘uploaded’ messages. Defaults to STDOUT.

  • workers - number of threads to use when pushing to S3. Defaults to 3.

Raises:

  • (ArgumentError) 


31
32
33
34
35
36
37
38
39
40
41
 
# File 'lib/s3-publisher.rb', line 31

def initialize bucket_name, opts={}
  @s3 =  RightAws::S3.new(AWSCredentials.access_key, AWSCredentials.secret_access_key, :multi_thread => true,
                                                                                       :protocol => 'http',
                                                                                       :port => 80, 
                                                                                       :logger => Logger.new(nil))
  @bucket_name, @base_path = bucket_name, opts[:base_path]
  raise ArgumentError, "#{bucket_name} doesn't seem to be a valid bucket on your account" if @s3.bucket(bucket_name).nil?
  @logger = opts[:logger] || $stdout
  @workers_to_use = opts[:workers] || 3
  @publish_queue = Queue.new
end

Instance Attribute Details

#base_path ⇒ Object (readonly)

Returns the value of attribute base_path.



15
16
17
 
# File 'lib/s3-publisher.rb', line 15

def base_path
  @base_path
end

#bucket_name ⇒ Object (readonly)

Returns the value of attribute bucket_name.



15
16
17
 
# File 'lib/s3-publisher.rb', line 15

def bucket_name
  @bucket_name
end

#logger ⇒ Object (readonly)

Returns the value of attribute logger.



15
16
17
 
# File 'lib/s3-publisher.rb', line 15

def logger
  @logger
end

#workers_to_use ⇒ Object (readonly)

Returns the value of attribute workers_to_use.



15
16
17
 
# File 'lib/s3-publisher.rb', line 15

def workers_to_use
  @workers_to_use
end

Class Method Details

.publish(bucket_name, opts = {}) {|p| ... } ⇒ Object

Block style. run is called for you on block close.

S3Publisher.publish('my-bucket') do |p|
  p.push('test.txt', '123abc')
end

Yields:

  • (p) 


21
22
23
24
25
 
# File 'lib/s3-publisher.rb', line 21

def self.publish bucket_name, opts={}, &block
  p = self.new(bucket_name, opts)
  yield(p)
  p.run
end

Instance Method Details

#inspect ⇒ Object 



84
85
86
 
# File 'lib/s3-publisher.rb', line 84

def inspect
  "#<S3Publisher:#{bucket_name}>"
end

#push(file_name, data, opts = {}) ⇒ Object

Pass:

  • file_name - name of file on S3. base_path will be prepended if supplied on instantiate.

  • data - data to be uploaded as a string

And one or many options:

  • :gzip (true|false) - gzip file contents? defaults to true.

  • :ttl - TTL in seconds for cache-control header. defaults to 5.

  • :cache_control - specify Cache-Control header directly if you don’t like the default

  • :content_type - no need to specify if default based on extension is okay. But if you need to force, you can provide :xml, :html, :text, or your own custom string.

  • :redundancy - by default objects are stored at reduced redundancy, pass :standard to store at full



54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
 
# File 'lib/s3-publisher.rb', line 54

def push file_name, data, opts={}
  headers = {}
  
  file_name = "#{base_path}/#{file_name}" unless base_path.nil?
  
  unless opts[:gzip] == false || file_name.match(/\.(jpg|gif|png|tif)$/)
    data = gzip(data)
    headers['Content-Encoding'] = 'gzip'
  end
  
  headers['x-amz-storage-class'] = opts[:redundancy] == :reduced ? 'REDUCED_REDUNDANCY' : 'STANDARD'
  headers['content-type']        = parse_content_type(opts[:content_type])  if opts[:content_type]

  if opts.has_key?(:cache_control)
    headers['Cache-Control'] = opts[:cache_control]
  else
    headers['Cache-Control'] = "max-age=#{opts[:ttl] || 5}"
  end

  @publish_queue.push({:key_name => file_name, :data => data, :headers => headers})
end

#run ⇒ Object

Process queued uploads and push to S3



77
78
79
80
81
82
 
# File 'lib/s3-publisher.rb', line 77

def run
  threads = []
  workers_to_use.times { threads << Thread.new { publish_from_queue } }
  threads.each { |t| t.join }
  true
end