Class: S33r::Client

Inherits:
Object
  • Object
show all
Includes:
S33r
Defined in:
lib/s33r/client.rb

Overview

The client performs operations over the network, using the core to build request headers and content; only client-specific headers are managed here: other headers can be handled by the core. – TODO: need to wrap XML returned into object representation. TODO: use customisable thread pool for requests. –

Direct Known Subclasses

NamedBucket

Constant Summary

Constants included from S33r

AWS_AUTH_HEADER_VALUE, AWS_HEADER_PREFIX, BUCKET_LIST_MAX_MAX_KEYS, CANNED_ACLS, DEFAULT_CHUNK_SIZE, DEFAULT_EXPIRY_SECS, HOST, INTERESTING_HEADERS, METADATA_PREFIX, METHOD_VERBS, NON_SSL_PORT, PORT, REQUIRED_HEADERS

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from S33r

#add_default_headers, #bucket_name_valid?, #canned_acl_header, #generate_auth_header_value, #generate_canonical_string, #generate_querystring, #generate_signature, #guess_mime_type, keys_to_symbols, #metadata_headers, #s3_authenticated_url, #s3_public_url, #url_join

Constructor Details

#initialize(aws_access_key, aws_secret_access_key, options = {}) ⇒ Client

Configure either an SSL-enabled or plain HTTP client. (If using SSL, no verification of server certificate is performed.)

options: hash of optional client config.:

  • :use_ssl => false: only use plain HTTP for connections

  • :dump_requests => true: dump each request’s initial line and headers to STDOUT



32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/s33r/client.rb', line 32

def initialize(aws_access_key, aws_secret_access_key, options={})
  if false == options[:use_ssl]
    @client = HTTP.new(HOST, NON_SSL_PORT)
    @client.use_ssl = false
  else
    @client = HTTP.new(HOST, PORT)
    # turn off SSL certificate verification
    @client.verify_mode = OpenSSL::SSL::VERIFY_NONE
    @client.use_ssl = true
  end
  
  @dump_requests = (true == options[:dump_requests])

  # set default chunk size for streaming request body
  @chunk_size = DEFAULT_CHUNK_SIZE

  # Amazon S3 developer keys
  @aws_access_key = aws_access_key
  @aws_secret_access_key = aws_secret_access_key

  # headers sent with every request made by this client
  @client_headers = {}
end

Instance Attribute Details

#aws_access_key ⇒ Object

Returns the value of attribute aws_access_key.



18
19
20
 
# File 'lib/s33r/client.rb', line 18

def aws_access_key
  @aws_access_key
end

#aws_secret_access_key ⇒ Object

Returns the value of attribute aws_secret_access_key.



18
19
20
 
# File 'lib/s33r/client.rb', line 18

def aws_secret_access_key
  @aws_secret_access_key
end

#chunk_size ⇒ Object

Size of data chunk to be sent per request when putting data.



21
22
23
 
# File 'lib/s33r/client.rb', line 21

def chunk_size
  @chunk_size
end

#client_headers ⇒ Object

Headers which should be sent with every request by default (unless overridden).



24
25
26
 
# File 'lib/s33r/client.rb', line 24

def client_headers
  @client_headers
end

Class Method Details

.init(config_file) ⇒ Object

Initialise client from YAML configuration file (see load_config method for details of acceptable format).



58
59
60
61
 
# File 'lib/s33r/client.rb', line 58

def Client.init(config_file)
  aws_access_key, aws_secret_access_key, options, _ = load_config(config_file)
  Client.new(aws_access_key, aws_secret_access_key, options)
end

.load_config(config_file) ⇒ Object

Load YAML config. file for a client. The config. file looks like this:

:include: test/files/namedbucket_config.yml

The options section contains settings specific to Client and NamedClient instances; custom contains extra settings specific to your application. options and custom sections can be omitted, but settings for AWS keys are required.

Returns an array [aws_access_key, aws_secret_access_key, options, custom], where options and custom are hashes.



73
74
75
76
77
78
79
80
81
82
83
84
85
86
 
# File 'lib/s33r/client.rb', line 73

def Client.load_config(config_file)
  require 'yaml'
  config = YAML::load_file(config_file)
  aws_access_key = config['aws_access_key']
  aws_secret_access_key = config['aws_secret_access_key']
  
  options = {}
  options = S33r.keys_to_symbols(config['options']) if config['options']
  
  custom = {}
  custom = S33r.keys_to_symbols(config['custom']) if config['custom']
  
  [aws_access_key, aws_secret_access_key, options, custom]
end

Instance Method Details

#add_client_headers(headers) ⇒ Object

Add any default headers which should be sent with every request from the client.

headers is a hash of headers already set up. Any headers passed in here override the defaults in client_headers.

Returns headers with the content of client_headers merged in.



270
271
272
 
# File 'lib/s33r/client.rb', line 270

def add_client_headers(headers)
  headers.merge!(client_headers) { |key, arg, default| arg }
end

#bucket_exists?(bucket_name) ⇒ Boolean

Returns true if bucket exists.

Returns:

  • (Boolean) 


196
197
198
 
# File 'lib/s33r/client.rb', line 196

def bucket_exists?(bucket_name)
  do_head("/#{bucket_name}").ok?
end

#create_bucket(bucket_name, headers = {}) ⇒ Object

Create a bucket.



175
176
177
 
# File 'lib/s33r/client.rb', line 175

def create_bucket(bucket_name, headers={})
  do_put("/#{bucket_name}", nil, headers)
end

#delete_bucket(bucket_name, headers = {}, options = {}) ⇒ Object

Delete a bucket.

options hash can contain the following:

  • :force => true: delete all keys within the bucket then delete the bucket itself

– TODO: maybe delete keys matching a partial path



184
185
186
187
188
189
190
191
192
193
 
# File 'lib/s33r/client.rb', line 184

def delete_bucket(bucket_name, headers={}, options={})
  if true == options[:force]
    _, bucket_listing = list_bucket(bucket_name)
    bucket_listing.contents.each_value do |obj|
      delete_resource(bucket_name, obj.key)
    end
  end
  
  do_delete("/#{bucket_name}", headers)
end

#delete_resource(bucket_name, resource_key, headers = {}) ⇒ Object

Delete a resource from S3.



260
261
262
 
# File 'lib/s33r/client.rb', line 260

def delete_resource(bucket_name, resource_key, headers={})
  do_delete("/#{bucket_name}/#{resource_key}", headers)
end

#do_request(method, path, data = nil, headers = {}) ⇒ Object

Send a request over the wire.

This method streams data if it responds to the stat method (as files do).

– TODO: set timeout on requests in case S3 is unresponsive



99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
 
# File 'lib/s33r/client.rb', line 99

def do_request(method, path, data=nil, headers={})
  req = get_requester(method, path)
  req.chunk_size = @chunk_size

  # add the S3 headers which are always required
  headers = add_default_headers(headers)

  # add any client-specific default headers
  headers = add_client_headers(headers)

  headers['Authorization'] = generate_auth_header_value(method, path, headers,
    @aws_access_key, @aws_secret_access_key)

  headers.each do |key, value|
    req[key] = value
  end

  if req.request_body_permitted?
    # for streaming files; NB Content-Length will be set by Net::HTTP
    # for character-based body content
    if data.respond_to?(:stat)
      req.body_stream = data
      req['Content-Length'] = data.stat.size.to_s
      data = nil
    end
  else
    data = nil
  end
  
  if @dump_requests
    puts req.to_s
  end

  @client.start do
    return @client.request(req, data)
  end

end

#get_requester(method, path) ⇒ Object

Return an instance of an appropriate request class.

Raises:



139
140
141
142
 
# File 'lib/s33r/client.rb', line 139

def get_requester(method, path)
  raise S33rException::UnsupportedHTTPMethod, "The #{method} HTTP method is not supported" if !(METHOD_VERBS.include?(method))
  eval("HTTP::" + method[0,1].upcase + method[1..-1].downcase + ".new('#{path}')")
end

#get_resource(bucket_name, resource_key, headers = {}) ⇒ Object

Fetch a resource and return a fleshed-out S3Object instance.



201
202
203
 
# File 'lib/s33r/client.rb', line 201

def get_resource(bucket_name, resource_key, headers={})
  do_get("/#{bucket_name}/#{resource_key}", headers)
end

#list_bucket(bucket_name, query_params = {}) ⇒ Object

List entries in a bucket.

query_params: hash of options on the bucket listing request, passed as querystring parameters to S3 (see docs.amazonwebservices.com/AmazonS3/2006-03-01/).

  • :prefix => 'some_string': restrict results to keys beginning with ‘some_string’

  • :marker => 'some_string': restict results to keys occurring lexicographically after ‘some_string’

  • :max_keys => 1000: return at most this number of keys (maximum possible value is 1000)

  • :delimiter => 'some_string': keys containing the same string between prefix and the delimiter are rolled up into a CommonPrefixes element inside the response



158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
 
# File 'lib/s33r/client.rb', line 158

def list_bucket(bucket_name, query_params={})
  if query_params[:max_keys]
    max_keys = query_params[:max_keys].to_i
    raise S33rException::BucketListingMaxKeysError, "max_keys option to list bucket cannot be > #{BUCKET_LIST_MAX_MAX_KEYS}" \
    if max_keys > BUCKET_LIST_MAX_MAX_KEYS

    # take out the max_keys parameter and move it to max-keys
    query_params['max-keys'] = query_params.delete(:max_keys)
  end
  
  resp = do_get("/#{bucket_name}" + generate_querystring(query_params))
  bucket_listing = BucketListing.new(resp.body)
  
  [resp, bucket_listing]
end

#list_buckets ⇒ Object

List all buckets.



145
146
147
 
# File 'lib/s33r/client.rb', line 145

def list_buckets
  do_get('/')
end

#put_file(filename, bucket_name, resource_key = nil, headers = {}, options = {}) ⇒ Object

Put a file onto S3.

If resource_key is nil, the filename is used as the key instead.

headers sets some headers with the request; useful if you have an odd file type not recognised by the mimetypes library, and want to explicitly set the Content-Type header.

options hash simplifies setting some headers with specific meaning to S3:

  • :render_as_attachment => true: set the Content-Disposition for this file to “attachment” and set the default filename for saving the file (when accessed by a web browser) to filename; this turns the file into a download when opened in a browser, rather than trying to render it inline.

Note that this method uses a handle to the file, so it can be streamed in chunks to S3.



229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
 
# File 'lib/s33r/client.rb', line 229

def put_file(filename, bucket_name, resource_key=nil, headers={}, options={})
  # default to the file path as the resource key if none explicitly set
  if resource_key.nil?
    resource_key = filename
  end
  
  # set Content-Disposition header
  if options[:render_as_attachment]
    headers['Content-Disposition'] = "attachment; filename=#{File.basename(filename)}"
  end

  # content type is explicitly set in the headers, so apply to request
  if headers[:content_type]
    # use the first MIME type corresponding to this content type string
    # (MIME::Types returns an array of possible MIME types)
    mime_type = MIME::Types[headers[:content_type]][0]
  else
    mime_type = guess_mime_type(filename)
  end
  content_type = mime_type.simplified
  headers['Content-Type'] = content_type
  headers['Content-Transfer-Encoding'] = 'binary' if mime_type.binary?

  # the data we want to put (handle to file, so we can stream from it)
  File.open(filename) do |data|
    # send the put request
    put_resource(bucket_name, resource_key, data, headers)
  end
end

#put_resource(bucket_name, resource_key, data, headers = {}) ⇒ Object

Put some generic resource onto S3.



206
207
208
 
# File 'lib/s33r/client.rb', line 206

def put_resource(bucket_name, resource_key, data, headers={})
  do_put("/#{bucket_name}/" + "#{CGI::escape(resource_key)}", data, headers)
end

#put_text(string, bucket_name, resource_key, headers = {}) ⇒ Object

Put a string onto S3.



211
212
213
214
 
# File 'lib/s33r/client.rb', line 211

def put_text(string, bucket_name, resource_key, headers={})
  headers["Content-Type"] = "text/plain"
  put_resource(bucket_name, resource_key, string, headers)
end

#use_ssl? ⇒ Boolean

Wrapper round embedded client use_ssl accessor.

Returns:

  • (Boolean) 


89
90
91
 
# File 'lib/s33r/client.rb', line 89

def use_ssl?
  @client.use_ssl
end