Class: S33r::Client
Overview
The client performs operations over the network, using the core to build request headers and content; only client-specific headers are managed here: other headers can be handled by the core. – TODO: need to wrap XML returned into object representation. TODO: use customisable thread pool for requests. –
Direct Known Subclasses
Constant Summary
Constants included from S33r
AWS_AUTH_HEADER_VALUE, AWS_HEADER_PREFIX, BUCKET_LIST_MAX_MAX_KEYS, CANNED_ACLS, DEFAULT_CHUNK_SIZE, DEFAULT_EXPIRY_SECS, HOST, INTERESTING_HEADERS, METADATA_PREFIX, METHOD_VERBS, NON_SSL_PORT, PORT, REQUIRED_HEADERS
Instance Attribute Summary collapse
-
#aws_access_key ⇒ Object
Returns the value of attribute aws_access_key.
-
#aws_secret_access_key ⇒ Object
Returns the value of attribute aws_secret_access_key.
-
#chunk_size ⇒ Object
Size of data chunk to be sent per request when putting data.
-
#client_headers ⇒ Object
Headers which should be sent with every request by default (unless overridden).
Class Method Summary collapse
-
.init(config_file) ⇒ Object
Initialise client from YAML configuration file (see load_config method for details of acceptable format).
-
.load_config(config_file) ⇒ Object
Load YAML config.
Instance Method Summary collapse
-
#add_client_headers(headers) ⇒ Object
Add any default headers which should be sent with every request from the client.
-
#bucket_exists?(bucket_name) ⇒ Boolean
Returns true if bucket exists.
-
#create_bucket(bucket_name, headers = {}) ⇒ Object
Create a bucket.
-
#delete_bucket(bucket_name, headers = {}, options = {}) ⇒ Object
Delete a bucket.
-
#delete_resource(bucket_name, resource_key, headers = {}) ⇒ Object
Delete a resource from S3.
-
#do_request(method, path, data = nil, headers = {}) ⇒ Object
Send a request over the wire.
-
#get_requester(method, path) ⇒ Object
Return an instance of an appropriate request class.
-
#get_resource(bucket_name, resource_key, headers = {}) ⇒ Object
Fetch a resource and return a fleshed-out S3Object instance.
-
#initialize(aws_access_key, aws_secret_access_key, options = {}) ⇒ Client
constructor
Configure either an SSL-enabled or plain HTTP client.
-
#list_bucket(bucket_name, query_params = {}) ⇒ Object
List entries in a bucket.
-
#list_buckets ⇒ Object
List all buckets.
-
#put_file(filename, bucket_name, resource_key = nil, headers = {}, options = {}) ⇒ Object
Put a file onto S3.
-
#put_resource(bucket_name, resource_key, data, headers = {}) ⇒ Object
Put some generic resource onto S3.
-
#put_text(string, bucket_name, resource_key, headers = {}) ⇒ Object
Put a string onto S3.
-
#use_ssl? ⇒ Boolean
Wrapper round embedded client
use_ssl
accessor.
Methods included from S33r
#add_default_headers, #bucket_name_valid?, #canned_acl_header, #generate_auth_header_value, #generate_canonical_string, #generate_querystring, #generate_signature, #guess_mime_type, keys_to_symbols, #metadata_headers, #s3_authenticated_url, #s3_public_url, #url_join
Constructor Details
#initialize(aws_access_key, aws_secret_access_key, options = {}) ⇒ Client
Configure either an SSL-enabled or plain HTTP client. (If using SSL, no verification of server certificate is performed.)
options
: hash of optional client config.:
-
:use_ssl => false
: only use plain HTTP for connections -
:dump_requests => true
: dump each request’s initial line and headers to STDOUT
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
# File 'lib/s33r/client.rb', line 32 def initialize(aws_access_key, aws_secret_access_key, ={}) if false == [:use_ssl] @client = HTTP.new(HOST, NON_SSL_PORT) @client.use_ssl = false else @client = HTTP.new(HOST, PORT) # turn off SSL certificate verification @client.verify_mode = OpenSSL::SSL::VERIFY_NONE @client.use_ssl = true end @dump_requests = (true == [:dump_requests]) # set default chunk size for streaming request body @chunk_size = DEFAULT_CHUNK_SIZE # Amazon S3 developer keys @aws_access_key = aws_access_key @aws_secret_access_key = aws_secret_access_key # headers sent with every request made by this client @client_headers = {} end |
Instance Attribute Details
#aws_access_key ⇒ Object
Returns the value of attribute aws_access_key.
18 19 20 |
# File 'lib/s33r/client.rb', line 18 def aws_access_key @aws_access_key end |
#aws_secret_access_key ⇒ Object
Returns the value of attribute aws_secret_access_key.
18 19 20 |
# File 'lib/s33r/client.rb', line 18 def aws_secret_access_key @aws_secret_access_key end |
#chunk_size ⇒ Object
Size of data chunk to be sent per request when putting data.
21 22 23 |
# File 'lib/s33r/client.rb', line 21 def chunk_size @chunk_size end |
#client_headers ⇒ Object
Headers which should be sent with every request by default (unless overridden).
24 25 26 |
# File 'lib/s33r/client.rb', line 24 def client_headers @client_headers end |
Class Method Details
.init(config_file) ⇒ Object
Initialise client from YAML configuration file (see load_config method for details of acceptable format).
58 59 60 61 |
# File 'lib/s33r/client.rb', line 58 def Client.init(config_file) aws_access_key, aws_secret_access_key, , _ = load_config(config_file) Client.new(aws_access_key, aws_secret_access_key, ) end |
.load_config(config_file) ⇒ Object
Load YAML config. file for a client. The config. file looks like this:
:include: test/files/namedbucket_config.yml
The options
section contains settings specific to Client and NamedClient instances; custom
contains extra settings specific to your application. options
and custom
sections can be omitted, but settings for AWS keys are required.
Returns an array [aws_access_key, aws_secret_access_key, options, custom]
, where options
and custom
are hashes.
73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
# File 'lib/s33r/client.rb', line 73 def Client.load_config(config_file) require 'yaml' config = YAML::load_file(config_file) aws_access_key = config['aws_access_key'] aws_secret_access_key = config['aws_secret_access_key'] = {} = S33r.keys_to_symbols(config['options']) if config['options'] custom = {} custom = S33r.keys_to_symbols(config['custom']) if config['custom'] [aws_access_key, aws_secret_access_key, , custom] end |
Instance Method Details
#add_client_headers(headers) ⇒ Object
Add any default headers which should be sent with every request from the client.
headers
is a hash of headers already set up. Any headers passed in here override the defaults in client_headers
.
Returns headers
with the content of client_headers
merged in.
270 271 272 |
# File 'lib/s33r/client.rb', line 270 def add_client_headers(headers) headers.merge!(client_headers) { |key, arg, default| arg } end |
#bucket_exists?(bucket_name) ⇒ Boolean
Returns true if bucket exists.
196 197 198 |
# File 'lib/s33r/client.rb', line 196 def bucket_exists?(bucket_name) do_head("/#{bucket_name}").ok? end |
#create_bucket(bucket_name, headers = {}) ⇒ Object
Create a bucket.
175 176 177 |
# File 'lib/s33r/client.rb', line 175 def create_bucket(bucket_name, headers={}) do_put("/#{bucket_name}", nil, headers) end |
#delete_bucket(bucket_name, headers = {}, options = {}) ⇒ Object
Delete a bucket.
options
hash can contain the following:
-
:force => true
: delete all keys within the bucket then delete the bucket itself
– TODO: maybe delete keys matching a partial path
184 185 186 187 188 189 190 191 192 193 |
# File 'lib/s33r/client.rb', line 184 def delete_bucket(bucket_name, headers={}, ={}) if true == [:force] _, bucket_listing = list_bucket(bucket_name) bucket_listing.contents.each_value do |obj| delete_resource(bucket_name, obj.key) end end do_delete("/#{bucket_name}", headers) end |
#delete_resource(bucket_name, resource_key, headers = {}) ⇒ Object
Delete a resource from S3.
260 261 262 |
# File 'lib/s33r/client.rb', line 260 def delete_resource(bucket_name, resource_key, headers={}) do_delete("/#{bucket_name}/#{resource_key}", headers) end |
#do_request(method, path, data = nil, headers = {}) ⇒ Object
Send a request over the wire.
This method streams data
if it responds to the stat
method (as files do).
– TODO: set timeout on requests in case S3 is unresponsive
99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 |
# File 'lib/s33r/client.rb', line 99 def do_request(method, path, data=nil, headers={}) req = get_requester(method, path) req.chunk_size = @chunk_size # add the S3 headers which are always required headers = add_default_headers(headers) # add any client-specific default headers headers = add_client_headers(headers) headers['Authorization'] = generate_auth_header_value(method, path, headers, @aws_access_key, @aws_secret_access_key) headers.each do |key, value| req[key] = value end if req.request_body_permitted? # for streaming files; NB Content-Length will be set by Net::HTTP # for character-based body content if data.respond_to?(:stat) req.body_stream = data req['Content-Length'] = data.stat.size.to_s data = nil end else data = nil end if @dump_requests puts req.to_s end @client.start do return @client.request(req, data) end end |
#get_requester(method, path) ⇒ Object
Return an instance of an appropriate request class.
139 140 141 142 |
# File 'lib/s33r/client.rb', line 139 def get_requester(method, path) raise S33rException::UnsupportedHTTPMethod, "The #{method} HTTP method is not supported" if !(METHOD_VERBS.include?(method)) eval("HTTP::" + method[0,1].upcase + method[1..-1].downcase + ".new('#{path}')") end |
#get_resource(bucket_name, resource_key, headers = {}) ⇒ Object
Fetch a resource and return a fleshed-out S3Object instance.
201 202 203 |
# File 'lib/s33r/client.rb', line 201 def get_resource(bucket_name, resource_key, headers={}) do_get("/#{bucket_name}/#{resource_key}", headers) end |
#list_bucket(bucket_name, query_params = {}) ⇒ Object
List entries in a bucket.
query_params
: hash of options on the bucket listing request, passed as querystring parameters to S3 (see docs.amazonwebservices.com/AmazonS3/2006-03-01/).
-
:prefix => 'some_string'
: restrict results to keys beginning with ‘some_string’ -
:marker => 'some_string'
: restict results to keys occurring lexicographically after ‘some_string’ -
:max_keys => 1000
: return at most this number of keys (maximum possible value is 1000) -
:delimiter => 'some_string'
: keys containing the same string between prefix and the delimiter are rolled up into a CommonPrefixes element inside the response
158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 |
# File 'lib/s33r/client.rb', line 158 def list_bucket(bucket_name, query_params={}) if query_params[:max_keys] max_keys = query_params[:max_keys].to_i raise S33rException::BucketListingMaxKeysError, "max_keys option to list bucket cannot be > #{BUCKET_LIST_MAX_MAX_KEYS}" \ if max_keys > BUCKET_LIST_MAX_MAX_KEYS # take out the max_keys parameter and move it to max-keys query_params['max-keys'] = query_params.delete(:max_keys) end resp = do_get("/#{bucket_name}" + generate_querystring(query_params)) bucket_listing = BucketListing.new(resp.body) [resp, bucket_listing] end |
#list_buckets ⇒ Object
List all buckets.
145 146 147 |
# File 'lib/s33r/client.rb', line 145 def list_buckets do_get('/') end |
#put_file(filename, bucket_name, resource_key = nil, headers = {}, options = {}) ⇒ Object
Put a file onto S3.
If resource_key
is nil, the filename is used as the key instead.
headers
sets some headers with the request; useful if you have an odd file type not recognised by the mimetypes library, and want to explicitly set the Content-Type header.
options
hash simplifies setting some headers with specific meaning to S3:
-
:render_as_attachment => true
: set the Content-Disposition for this file to “attachment” and set the default filename for saving the file (when accessed by a web browser) tofilename
; this turns the file into a download when opened in a browser, rather than trying to render it inline.
Note that this method uses a handle to the file, so it can be streamed in chunks to S3.
229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 |
# File 'lib/s33r/client.rb', line 229 def put_file(filename, bucket_name, resource_key=nil, headers={}, ={}) # default to the file path as the resource key if none explicitly set if resource_key.nil? resource_key = filename end # set Content-Disposition header if [:render_as_attachment] headers['Content-Disposition'] = "attachment; filename=#{File.basename(filename)}" end # content type is explicitly set in the headers, so apply to request if headers[:content_type] # use the first MIME type corresponding to this content type string # (MIME::Types returns an array of possible MIME types) mime_type = MIME::Types[headers[:content_type]][0] else mime_type = guess_mime_type(filename) end content_type = mime_type.simplified headers['Content-Type'] = content_type headers['Content-Transfer-Encoding'] = 'binary' if mime_type.binary? # the data we want to put (handle to file, so we can stream from it) File.open(filename) do |data| # send the put request put_resource(bucket_name, resource_key, data, headers) end end |
#put_resource(bucket_name, resource_key, data, headers = {}) ⇒ Object
Put some generic resource onto S3.
206 207 208 |
# File 'lib/s33r/client.rb', line 206 def put_resource(bucket_name, resource_key, data, headers={}) do_put("/#{bucket_name}/" + "#{CGI::escape(resource_key)}", data, headers) end |
#put_text(string, bucket_name, resource_key, headers = {}) ⇒ Object
Put a string onto S3.
211 212 213 214 |
# File 'lib/s33r/client.rb', line 211 def put_text(string, bucket_name, resource_key, headers={}) headers["Content-Type"] = "text/plain" put_resource(bucket_name, resource_key, string, headers) end |
#use_ssl? ⇒ Boolean
Wrapper round embedded client use_ssl
accessor.
89 90 91 |
# File 'lib/s33r/client.rb', line 89 def use_ssl? @client.use_ssl end |