Class: DerivativeRodeo::StorageLocations::S3Location

Inherits:
BaseLocation
  • Object
show all
Defined in:
lib/derivative_rodeo/storage_locations/s3_location.rb

Overview

Location to download and upload files to S3

Class Attributes collapse

Attributes inherited from BaseLocation

#file_uri

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from BaseLocation

build, #derived_file_from, #file_basename, #file_dir, #file_extension, #file_name, file_path_from_parts, from_uri, inherited, #initialize, load_location, location_name, locations, register_location, #tmp_file_dir, #with_new_extension, #with_new_tmp_path, #with_tmp_path

Constructor Details

This class inherits a constructor from DerivativeRodeo::StorageLocations::BaseLocation

Instance Attribute Details

#use_actual_s3_bucket ⇒ Object

When true , we are going to use a live S3 bucket. When false, we’ll use a fake local bucket.



16
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 16

class_attribute :use_actual_s3_bucket, default: true

Class Method Details

.adapter_prefix(bucket_name: config.aws_s3_bucket) ⇒ String

Parameters:

  • bucket_name (String, NilClass) (defaults to: config.aws_s3_bucket) —

    when given, use this as the bucket, otherwise, def

Returns:

  • (String) 


38
39
40
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 38

def self.adapter_prefix(bucket_name: config.aws_s3_bucket)
  "#{scheme}://#{bucket_name}.s3.#{config.aws_s3_region}.amazonaws.com"
end

.create_uri(path:, parts: 2) ⇒ String

Create a new uri of the classes type. Parts argument should have a default in implementing classes. Must support a number or the symbol :all

Parameters:

  • path (String)
  • parts (Integer, :all) (defaults to: 2) —

    , defaults to 2 for S3 which is parent_dir/file_name.ext

Returns:

  • (String) 


29
30
31
32
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 29

def self.create_uri(path:, parts: 2)
  file_path = file_path_from_parts(path: path, parts: parts)
  File.join("#{adapter_prefix}/#{file_path}")
end

.faux_bucket ⇒ AwsS3FauxBucket

A fake constructed fake bucket that confroms to the narrow S3 interface that we use.

Returns:

See Also:



147
148
149
150
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 147

def self.faux_bucket
  # We are not requiring this file; except in the spec context.
  @faux_bucket ||= AwsS3FauxBucket.new
end

Instance Method Details

#bucket ⇒ Object

See Also:



130
131
132
133
134
135
136
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 130

def bucket
  @bucket ||= if use_actual_s3_bucket?
                resource.bucket(bucket_name)
              else
                self.class.faux_bucket
              end
end

#bucket_name ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

long-term-video-storage.s3.eu-west-1.amazonaws.com/path1/path2/file.tld



123
124
125
126
127
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 123

def bucket_name
  @bucket_name ||= file_uri.match(%r{s3://(.+)\.s3})&.[](1)
rescue StandardError
  raise Errors::BucketMissingError.new(file_uri: file_uri)
end

#exist? ⇒ Boolean

Existance is futile

Returns:

  • (Boolean) 


61
62
63
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 61

def exist?
  bucket.objects(prefix: file_path).count.positive?
end

#file_path ⇒ Object 



138
139
140
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 138

def file_path
  @file_path ||= @file_uri.sub(%r{.+://.+?/}, '')
end

#matching_locations_in_file_dir(tail_regexp:) ⇒ Enumerable<DerivativeRodeo::StorageLocations::S3Location>

Note:

S3 allows searching on a prefix but does not allow for “wildcard” searches.

Parameters:

  • tail_regexp (Regexp)

Returns:

See Also:



73
74
75
76
77
78
79
80
81
82
83
84
85
86
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 73

def matching_locations_in_file_dir(tail_regexp:)
  uri = URI.parse(file_uri)
  scheme_and_host = "#{uri.scheme}://#{uri.host}"
  logger.debug("#{self.class}##{__method__} searching for matching files for " \
              "scheme_and_host: #{scheme_and_host.inspect} " \
              "file_dir: #{file_dir.inspect} " \
              "with tail_regexp: #{tail_regexp.inspect}.")
  bucket.objects(prefix: file_dir).each_with_object([]) do |object, accumulator|
    if tail_regexp.match(object.key)
      template = File.join(scheme_and_host, object.key)
      accumulator << derived_file_from(template: template)
    end
  end
end

#resource ⇒ Aws::S3::Resource

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns:

  • (Aws::S3::Resource) 


105
106
107
108
109
110
111
112
113
114
115
116
117
118
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 105

def resource
  # TODO: Are there instantiation considerations when running in Lambda?  In tests
  # initializing a resource is very slow (e.g. 3 seconds or so).  Should this be a class
  # method?  Can it be given the SpaceStone constraints?
  @resource ||= if DerivativeRodeo.config.aws_s3_access_key_id
                  Aws::S3::Resource.new(region: DerivativeRodeo.config.aws_s3_region,
                                        credentials: Aws::Credentials.new(
                                          DerivativeRodeo.config.aws_s3_access_key_id,
                                          DerivativeRodeo.config.aws_s3_secret_access_key
                                        ))
                else
                  Aws::S3::Resource.new
                end
end

#with_existing_tmp_path(&block) ⇒ String

download or copy the file to a tmp path deletes the tmp file after the block is executed

Returns:

  • (String) —

    the path to the tmp file



48
49
50
51
52
53
54
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 48

def with_existing_tmp_path(&block)
  with_tmp_path(lambda { |file_path, tmp_file_path, exist|
                  raise Errors::FileMissingError.with_info(method: __method__, context: self, file_path: file_path, tmp_file_path: tmp_file_path) unless exist
                  obj = bucket.object(file_path)
                  obj.download_file(tmp_file_path)
                }, &block)
end

#write ⇒ String

write the tmp file to the file_uri

Returns:

  • (String) —

    the file_uri



93
94
95
96
97
98
99
 
# File 'lib/derivative_rodeo/storage_locations/s3_location.rb', line 93

def write
  raise Errors::FileMissingError("Use write within a with__new_tmp_path block and fill the tmp file with data before writing") unless File.exist?(tmp_file_path)

  obj = bucket.object(file_path)
  obj.upload_file(tmp_file_path)
  file_uri
end