Class: Range
Instance Method Summary collapse
-
#in_chunks(chunks, thread_limit = 40, min_per_chunk = 1) ⇒ Object
Supply a block taking |chunk_min_id, chunk_max_id|.
Instance Method Details
#in_chunks(chunks, thread_limit = 40, min_per_chunk = 1) ⇒ Object
Supply a block taking |chunk_min_id, chunk_max_id|. This will execute the block in parallel chunks, supplying the min and max id (inclusive) of the current chunk. Note that thread_limit is capped at the value of Jetpants.max_concurrency.
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
# File 'lib/jetpants/monkeypatch.rb', line 43 def in_chunks(chunks, thread_limit=40, min_per_chunk=1) per_chunk = ((max - min + 1) / chunks).ceil per_chunk = min_per_chunk if per_chunk < min_per_chunk min_id_queue = [] results = [] min.step(max, per_chunk) {|n| min_id_queue << n} min_id_queue.reverse! lock = Mutex.new group = ThreadGroup.new thread_count = Jetpants.max_concurrency thread_count = thread_limit if thread_limit && thread_limit < Jetpants.max_concurrency thread_count = min_id_queue.length if min_id_queue.length < thread_count thread_count.times do th = Thread.new do while min_id_queue.length > 0 do my_min = nil lock.synchronize {my_min = min_id_queue.pop} break unless my_min my_max = my_min + per_chunk - 1 my_max = max if my_max > max result = yield my_min, my_max lock.synchronize {results << result} end end group.add(th) end group.list.each {|th| th.join} results end |