Class: ATT::Swift

Inherits:
Object
  • Object
show all
Defined in:
lib/att/swift.rb

Overview

A wrapper for OpenStack Object Storage v1 (aka Swift). Swift provides redundant storage similar to AWS S3. This gem is a wrapper around the Object # Storage RESTful API and provides the ability to manage containers (directories) and objects (files).

Example:

require 'pp'
require 'att/swift'

auth_uri = 'http://swift.example/auth/'
swift = ATT::Swift.new auth_uri, 'username', 'password'

pp swift.containers

See also docs.openstack.org/api/openstack-object-storage/1.0/content/

Defined Under Namespace

Classes: Error

Constant Summary collapse

VERSION =

The version of att-swift you are using

'1.0'

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(auth_uri, user, key) ⇒ Swift

Creates a new Swift object for communication with the server at auth_uri. user and key are your credentials.

For AT&T Cloud storage:

swift = ATT::Swift.new \
  'https://data.iad1.attstorage.com/auth/',
  'tennant:username', 'password'



70
71
72
73
74
75
76
77
78
79
80
81
 
# File 'lib/att/swift.rb', line 70

def initialize auth_uri, user, key
  auth_uri = URI auth_uri unless URI === auth_uri

  @auth_uri = auth_uri
  @user     = user
  @key      = key

  @http = Net::HTTP::Persistent.new 'att-swift'

  @storage_uri = nil
  @auth_token  = nil
end

Instance Attribute Details

#auth_token ⇒ Object (readonly)

:nodoc:



36
37
38
 
# File 'lib/att/swift.rb', line 36

def auth_token
  @auth_token
end

#http ⇒ Object

:nodoc:



34
35
36
 
# File 'lib/att/swift.rb', line 34

def http
  @http
end

#storage_uri ⇒ Object (readonly)

:nodoc:



37
38
39
 
# File 'lib/att/swift.rb', line 37

def storage_uri
  @storage_uri
end

Instance Method Details

#authenticate ⇒ Object

Authenticates you with the swift server. This method is called automatically by other API methods.



87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
 
# File 'lib/att/swift.rb', line 87

def authenticate
  return if @auth_token

  res = get @auth_uri + 'v1.0', 'X-Auth-User' => @user, 'X-Auth-Key' => @key

  case res
  when Net::HTTPSuccess
    @auth_token  = res['X-Auth-Token']

    storage_uri = res['X-Storage-Url']
    storage_uri << '/' unless storage_uri.end_with? '/'
    @storage_uri = URI storage_uri

    @http.override_headers['X-Auth-Token'] = @auth_token
  else
    raise Error.new(res, 'authentication failed')
  end
end

#chunk_objects(container, marker = nil, limit = 1_000) ⇒ Object

Like paginate_objects, but yields chunks of up to limit object information at a time from the swift server to the given block. marker may be used to start pagination after a particular entry.

swift.chunk_objects 'container' do |object_infos|
  object_infos.each do |object_info|
    # ...
  end
end



117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
 
# File 'lib/att/swift.rb', line 117

def chunk_objects container, marker = nil, limit = 1_000
  return enum_for __method__, container, marker, limit unless block_given?

  loop do
    chunk = objects container, marker, limit

    break if chunk.empty?

    yield chunk

    marker = chunk.last['name']
  end

  self
end

#containers(marker = nil, limit = nil) ⇒ Object

Retrieves the containers for your server. Example output:

[{"name" => "public_bucket", "count" => 2, "bytes" => 9},
 {"name" => "public_bucket_segments", "count" => 22, "bytes" => 21875}]



139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/att/swift.rb', line 139

def containers marker = nil, limit = nil
  authenticate

  params = { 'format' => 'json' }
  params['marker'] = marker if marker
  params['limit']  = limit  if limit

  uri = @storage_uri + "?#{escape_params params}"

  res = get uri

  case res
  when Net::HTTPSuccess then
    JSON.parse res.body
  else
    raise Error.new(res, 'error listing containers')
  end
end

#copy_object(source_container, source_object, destination_container, destination_object) ⇒ Object

Copies source_object in source_container to destination_object in destination_container.

Returns true if the copy was successful, false if the source object was not found or the destination container did not exist.



165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
 
# File 'lib/att/swift.rb', line 165

def copy_object source_container,      source_object,
                destination_container, destination_object
  authenticate

  source_path      = "#{source_container}/#{source_object}"
  destination_path = "#{destination_container}/#{destination_object}"

  uri = @storage_uri + destination_path

  req = Net::HTTP::Put.new uri.request_uri
  req['X-Copy-From'] = "/#{source_path}"
  req.body = ''

  res = @http.request uri, req

  case res
  when Net::HTTPSuccess then
    true
  when Net::HTTPNotFound
    false
  else
    raise
      Error.new(res, "error copying #{source_path} to #{destination_path}")
  end
end

#create_container(container) ⇒ Object

Creates a new container. Returns true if the container was created, false if it already existed.



202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
 
# File 'lib/att/swift.rb', line 202

def create_container container
  authenticate

  uri = @storage_uri + container

  res = put uri

  case res
  when Net::HTTPCreated then
    true
  when Net::HTTPSuccess then
    false
  else
    raise Error.new(res, 'error creating container')
  end
end

#delete(uri, headers = {}) ⇒ Object

HTTP DELETE



194
195
196
 
# File 'lib/att/swift.rb', line 194

def delete uri, headers = {} # :nodoc:
  request Net::HTTP::Delete, uri, headers
end

#delete_container(container) ⇒ Object

Deletes container. Returns true if the container existed, false if not. Raises an exception otherwise.



223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
 
# File 'lib/att/swift.rb', line 223

def delete_container container
  authenticate

  uri = @storage_uri + container

  res = delete uri

  case res
  when Net::HTTPNoContent then
    true
  when Net::HTTPNotFound then
    false
  when Net::HTTPConflict then
    raise Error.new(res, "container #{container} is not empty")
  else
    raise Error.new(res, "error deleting container")
  end
end

#delete_object(container, object) ⇒ Object

Deletes object from container Returns true if the object existed, false if not. Raises an exception otherwise.



246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
 
# File 'lib/att/swift.rb', line 246

def delete_object container, object
  authenticate

  uri = @storage_uri + "#{container}/#{object}"

  res = delete uri

  case res
  when Net::HTTPNoContent then
    true
  when Net::HTTPNotFound then
    false
  else
    raise Error.new(res, "error deleting #{object} in #{container}")
  end
end

#escape_params(params) ⇒ Object

:nodoc:



263
264
265
266
267
 
# File 'lib/att/swift.rb', line 263

def escape_params params # :nodoc:
  params.map do |k, v|
    [CGI.escape(k.to_s), CGI.escape(v.to_s)].join '='
  end.compact.join '&'
end

#get(uri, headers = {}, &block) ⇒ Object

HTTP GET



272
273
274
 
# File 'lib/att/swift.rb', line 272

def get uri, headers = {}, &block # :nodoc:
  request Net::HTTP::Get, uri, headers, &block
end

#head(uri, headers = {}) ⇒ Object

HTTP HEAD



279
280
281
 
# File 'lib/att/swift.rb', line 279

def head uri, headers = {} # :nodoc:
  request Net::HTTP::Head, uri, headers
end

#inspect ⇒ Object

:nodoc:



283
284
285
286
287
288
289
290
 
# File 'lib/att/swift.rb', line 283

def inspect # :nodoc:
  "#<%s:0x%x %s %s>" % [
    self.class,
    object_id,
    @auth_uri,
    @user
  ]
end

#object_info(container, object) ⇒ Object

Retrieves information for object in container including metadata. Example result:

{
  'metadata' => {
    'meat' => 'Bacon',
  },

  'content-length' => '3072',
  'content-type'   => 'application/octet-stream',
  'etag'           => 'a2a5648d09a83c6a85ddd62e4a22309c',
  'last-modified'  => 'Wed, 22 Aug 2012 22:51:28 GMT',
}



348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
 
# File 'lib/att/swift.rb', line 348

def object_info container, object
  authenticate

  uri = @storage_uri + "#{container}/#{object}"

  res = head uri

  case res
  when Net::HTTPSuccess then
    metadata = {}
    info     = { 'metadata' => metadata }

    res.each do |name, value|
      case name
      when /^accept/i, 'connection', 'date' then
        next
      when /^x-object-meta-(.*)/ then
        metadata[$1] = value
      when /^x-/ then
        next
      else
        info[name] = value
      end
    end

    info
  when Net::HTTPNotFound then
    nil
  else
    raise Error.new(res,
                    "error retrieving metadata for #{object} in #{container}")
  end
end

#object_metadata(container, object) ⇒ Object

Retrieves metadata for object in container. See also object_info. Example result:

'metadata' => {
  'meat' => 'Bacon',
}



390
391
392
393
394
395
396
 
# File 'lib/att/swift.rb', line 390

def object_metadata container, object
  info = object_info container, object

  return unless info

  info['metadata']
end

#objects(container, marker = nil, limit = nil) ⇒ Object

Lists objects in container. marker lists objects after the given name, limit restricts the object listing to that many items.

Example output:

[{"name"=>"test-0",
  "hash"=>"b1946ac92492d2347c6235b4d2611184",
  "bytes"=>6,
  "content_type"=>"application/octet-stream",
  "last_modified"=>"2012-08-22T19:24:03.102100"},
 {"name"=>"test-1",
  "hash"=>"802eeebb01b647913806a870cbb5394a",
  "bytes"=>52707,
  "content_type"=>"application/octet-stream",
  "last_modified"=>"2012-08-22T19:32:24.474980"},
 {"name"=>"test-2",
  "hash"=>"90affbd9a1954ec9ff029b7ad7183a16",
  "bytes"=>5,
  "content_type"=>"application/octet-stream",
  "last_modified"=>"2012-08-22T21:10:05.258080"}]



314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
 
# File 'lib/att/swift.rb', line 314

def objects container, marker = nil, limit = nil
  authenticate

  params = { 'format' => 'json' }
  params['marker'] = marker if marker
  params['limit']  = limit  if limit

  uri = @storage_uri + "#{container}?#{escape_params params}"

  res = get uri

  case res
  when Net::HTTPSuccess then
    JSON.parse res.body
  else
    raise Error.new(res, "error retrieving object list in #{container}")
  end
end

#paginate_objects(container, marker = nil, limit = 1_000) ⇒ Object

Like #objects, but only retrieves limit objects at a time from the swift server and yields the object information to the given block. marker may be used to start pagination after a particular entry.

swift.paginate_objects 'container' do |object_info|
  p object_info.name
end



407
408
409
410
411
412
413
414
415
416
417
 
# File 'lib/att/swift.rb', line 407

def paginate_objects container, marker = nil, limit = 1_000
  return enum_for __method__, container, marker, limit unless block_given?

  chunk_objects container, marker, limit do |chunk|
    chunk.each do |object_info|
      yield object_info
    end
  end

  self
end

#post(uri, headers = {}) ⇒ Object

HTTP POST



422
423
424
 
# File 'lib/att/swift.rb', line 422

def post uri, headers = {} # :nodoc:
  request Net::HTTP::Post, uri, headers
end

#put(uri, headers = {}) ⇒ Object

HTTP PUT



429
430
431
 
# File 'lib/att/swift.rb', line 429

def put uri, headers = {} # :nodoc:
  request Net::HTTP::Put, uri, headers
end

#read_object(container, object) ⇒ Object

Reads object from container. Unless the response is HTTP 200 OK an exception is raised.

If no block is given the response body is read, checked for a matching MD5 checksum and returned.

If a block is given the response is yielded for custom handling and no MD5 checksum is calculated.

Example:

swift.read_object 'test-container', 'test-object' do |res|
  open destination, 'wb' do |io|
    res.read_body do |chunk|
      io.write chunk
    end
  end
end



453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
 
# File 'lib/att/swift.rb', line 453

def read_object container, object
  authenticate

  uri = @storage_uri + "#{container}/#{object}"

  get uri do |res|
    case res
    when Net::HTTPOK then
      if block_given? then
        yield res
      else
        body = res.body

        if res['ETag'] != Digest::MD5.hexdigest(body) then
          raise Error.new(res,
                          "checksum retrieving object #{object} in #{container}")
        end

        body
      end
    when Net::HTTPNotFound then
      raise Error.new(res, "object #{object} in #{container} not found")
    else
      raise Error.new(res, "error retrieving object #{object} in #{container}")
    end
  end
end

#request(klass, uri, headers, &block) ⇒ Object

net-http-persistent request wrapper



484
485
486
487
488
489
490
 
# File 'lib/att/swift.rb', line 484

def request klass, uri, headers, &block # :nodoc:
  path = klass::REQUEST_HAS_BODY ? uri.path : uri.request_uri

  req = klass.new path, headers

  @http.request uri, req, &block
end

#set_object_metadata(container, object, metadata = {}) ⇒ Object

Sets the metadata for object in container. Existing metadata will be overwritten.



496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
 
# File 'lib/att/swift.rb', line 496

def set_object_metadata container, object, metadata = {}
  authenticate

  uri = @storage_uri + "#{container}/#{object}"

  headers = {}

  metadata.each do |name, value|
    headers["X-Object-Meta-#{name}"] = value
  end

  res = post uri, headers

  case res
  when Net::HTTPSuccess then
    true
  else
    raise Error.new(res,
                    "error setting metadata on #{object} in #{container}")
  end
end

#write_object(container, object, content = nil) ⇒ Object

Writes to object in container. content may be a String or IO-like object. If content is a String the response ETag is checked against the MD5 hash of the local copy.

A block may be given instead of content. You will be given a pipe you can write the content to:

r = Random.new

swift.write_object 'test-container', 'test-object' do |io|
  r.rand(10).times do
    io.write r.bytes r.rand 10_000
  end
end

In all cases the response ETag is returned when the content was successfully uploaded.

Raises:

  • (ArgumentError) 


537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
 
# File 'lib/att/swift.rb', line 537

def write_object container, object, content = nil
  raise ArgumentError, 'provide block or content' if
    content and block_given?

  if block_given? then
    content, write = IO.pipe

    Thread.start do
      begin
        yield write
      ensure
        write.close
      end
    end
  end

  authenticate

  uri = @storage_uri + "#{container}/#{object}"

  req = Net::HTTP::Put.new uri.path

  case content
  when String then
    req.body = content
    req['ETag'] = Digest::MD5.hexdigest content
  else
    req['Transfer-Encoding'] = 'chunked'
    req.body_stream = content
  end

  req.content_type = 'application/octet-stream'

  res = @http.request uri, req

  case res
  when Net::HTTPCreated then
    res['ETag']
  else
    raise Error.new(res,
                    "error creating object #{object} in container #{container}")
  end
end