Class: Cabriolet::Parallel::BatchProcessor

Inherits:
Object
  • Object
show all
Defined in:
lib/cabriolet/parallel.rb

Overview

Parallel batch processor

Instance Method Summary collapse

Constructor Details

#initialize(workers: Extractor::DEFAULT_WORKERS) ⇒ BatchProcessor

Returns a new instance of BatchProcessor.



153
154
155
156
157
 
# File 'lib/cabriolet/parallel.rb', line 153

def initialize(workers: Extractor::DEFAULT_WORKERS)
  @workers = workers
  @stats = { total: 0, successful: 0, failed: 0 }
  @stats_mutex = Mutex.new
end

Instance Method Details

#process_all(archive_paths, output_base) {|archive_path, stats| ... } ⇒ Hash

Process multiple archives in parallel

Examples:

processor = Cabriolet::Parallel::BatchProcessor.new(workers: 8)
stats = processor.process_all(Dir.glob('*.cab'), 'output/')

Parameters:

  • archive_paths (Array<String>) —

    Paths to archives

  • output_base (String) —

    Base output directory

Yields:

  • (archive_path, stats) —

    Optional callback per archive

Returns:

  • (Hash) —

    Overall statistics



169
170
171
172
173
174
175
176
177
178
179
180
 
# File 'lib/cabriolet/parallel.rb', line 169

def process_all(archive_paths, output_base, &block)
  queue = Queue.new
  archive_paths.each { |path| queue << path }
  @workers.times { queue << :done }

  threads = Array.new(@workers) do
    Thread.new { process_loop(queue, output_base, &block) }
  end

  threads.each(&:join)
  @stats
end