Class: Aws::S3::DirectoryDownloader Private

Inherits:
Object
  • Object
show all
Defined in:
lib/aws-sdk-s3/directory_downloader.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

This is a one-shot class that downloads objects from a bucket to a local directory. This works as follows:

  • ObjectProducer runs in a background thread, calling list_objects_v2 and pushing entries into a SizedQueue (max: 100).
  • An internal executor pulls from that queue and posts work. Each task uses FileDownloader to download objects then signals completion via completion_queue.

We track how many tasks we posted, then pop that many times from completion_queue to wait for everything to finish.

Errors are collected in a mutex-protected array. On failure (unless ignore_failure is set), we call abort which closes the queue - the producer catches ClosedQueueError and exits cleanly.

API:

  • private

Defined Under Namespace

Classes: ObjectProducer

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ DirectoryDownloader

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of DirectoryDownloader.

API:

  • private



19
20
21
22
23
24
25
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 19

def initialize(options = {})
  @client = options[:client] || Client.new
  @executor = options[:executor] || DefaultExecutor.new
  @logger = options[:logger]
  @producer = nil
  @mutex = Mutex.new
end

Instance Attribute Details

#client ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

API:

  • private



27
28
29
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 27

def client
  @client
end

#executor ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

API:

  • private



27
28
29
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 27

def executor
  @executor
end

Instance Method Details

#abort ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

API:

  • private



29
30
31
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 29

def abort
  @producer&.close
end

#download(destination, bucket:, **options) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

API:

  • private



33
34
35
36
37
38
39
40
41
42
43
44
45
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 33

def download(destination, bucket:, **options)
  if File.exist?(destination)
    raise ArgumentError, 'invalid destination, expected a directory' unless File.directory?(destination)
  else
    FileUtils.mkdir_p(destination)
  end

  download_opts = build_download_opts(destination, options)
  @producer = ObjectProducer.new(build_producer_opts(destination, bucket, options))
  downloader = FileDownloader.new(client: @client, executor: @executor)
  downloads, errors = process_download_queue(downloader, download_opts)
  build_result(downloads, errors)
end