Class: EventMachine::Iterator

Inherits:
Object
  • Object
show all
Includes:
IteratorWithArray, IteratorWithEnumerable, IteratorWithQueue
Defined in:
lib/em/iterator.rb

Overview

A simple iterator for concurrent asynchronous work.

Unlike ruby's built-in iterators, the end of the current iteration cycle is signaled manually, instead of happening automatically after the yielded block finishes executing. For example:

(0..10).each{ |num| }

becomes:

EM::Iterator.new(0..10).each{ |num,iter| iter.next }

This is especially useful when doing asynchronous work via reactor libraries and functions. For example, given a sync and async http api:

response = sync_http_get(url); ... async_http_get(url){ |response| ... }

a synchronous iterator such as:

responses = urls.map{ |url| sync_http_get(url) } ... puts 'all done!'

could be written as:

EM::Iterator.new(urls).map(proc{ |url,iter| async_http_get(url){ |res| iter.return(res) } }, proc{ |responses| ... puts 'all done!' })

Now, you can take advantage of the asynchronous api to issue requests in parallel. For example, to fetch 10 urls at a time, simply pass in a concurrency of 10:

EM::Iterator.new(urls, 10).each do |url,iter| async_http_get(url){ iter.next } end

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from IteratorWithQueue

#next_from_queue?

Methods included from IteratorWithArray

#next?, #setup_list

Methods included from IteratorWithEnumerable

#next?, #setup_list

Constructor Details

#initialize(list, concurrency = 1) ⇒ Iterator

Create a new parallel async iterator with specified concurrency.

i = EM::Iterator.new(1..100, 10)

will create an iterator over the range that processes 10 items at a time. Iteration is started via #each, #map or #inject



119
120
121
122
123
124
125
126
127
128
129
130
 
# File 'lib/em/iterator.rb', line 119

def initialize(list, concurrency = 1)
  if list.class == EventMachine::Queue
    @queue = list
    alias :next? :next_from_queue?
  else
    @list = setup_list(list)
  end
  @concurrency = concurrency

  @started = false
  @ended = false
end

Instance Attribute Details

#concurrency ⇒ Object

Returns the value of attribute concurrency.



141
142
143
 
# File 'lib/em/iterator.rb', line 141

def concurrency
  @concurrency
end

#next_item ⇒ Object (readonly)

Returns the value of attribute next_item.



106
107
108
 
# File 'lib/em/iterator.rb', line 106

def next_item
  @next_item
end

Instance Method Details

#each(foreach = nil, after = nil, &blk) ⇒ Object

Iterate over a set of items using the specified block or proc.

EM::Iterator.new(1..100).each do |num, iter| puts num iter.next end

An optional second proc is invoked after the iteration is complete.

EM::Iterator.new(1..100).each( proc{ |num,iter| iter.next }, proc{ puts 'all done' } )

Raises:

  • (ArgumentError) 


157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
 
# File 'lib/em/iterator.rb', line 157

def each(foreach=nil, after=nil, &blk)
  raise ArgumentError, 'proc or block required for iteration' unless foreach ||= blk
  raise RuntimeError, 'cannot iterate over an iterator more than once' if @started or @ended

  @started = true
  @pending = 0
  @workers = 0

  all_done = proc{
    after.call if after and @ended and @pending == 0
  }

  @process_next = proc{
    # p [:process_next, :pending=, @pending, :workers=, @workers, :ended=, @ended, :concurrency=, @concurrency, :list=, @list]
    unless @ended or @workers > @concurrency
      begin
        if next?
          item = next_item
          @pending += 1

          is_done = false
          on_done = proc{
            raise RuntimeError, 'already completed this iteration' if is_done
            is_done = true

            @pending -= 1

            if @ended
              all_done.call
            else
              EM.next_tick(@process_next)
            end
          }
          class << on_done
            alias :next :call
          end

          foreach.call(item, on_done)
        else
          @ended = true
          @workers -= 1
          all_done.call
        end
      rescue EventMachine::QueueIsEmpty
        EM.next_tick(@process_next)
      end
    else
      @workers -= 1
    end
  }

  spawn_workers

  self
end

#inject(obj, foreach, after) ⇒ Object

Inject the results of an asynchronous iteration onto a given object.

EM::Iterator.new(%w[ pwd uptime uname date ], 2).inject({}, proc{ |hash,cmd,iter| EM.system(cmd){ |output,status| hash[cmd] = status.exitstatus == 0 ? output.strip : nil iter.return(hash) } }, proc{ |results| p results })



262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
 
# File 'lib/em/iterator.rb', line 262

def inject(obj, foreach, after)
  each(proc{ |item,iter|
    is_done = false
    on_done = proc{ |res|
      raise RuntimeError, 'already returned a value for this iteration' if is_done
      is_done = true

      obj = res
      iter.next
    }
    class << on_done
      alias :return :call
      def next
        raise NoMethodError, 'must call #return on an inject iterator'
      end
    end

    foreach.call(obj, item, on_done)
  }, proc{
    after.call(obj)
  })
end

#map(foreach, after) ⇒ Object

Collect the results of an asynchronous iteration into an array.

EM::Iterator.new(%w[ pwd uptime uname date ], 2).map(proc{ |cmd,iter| EM.system(cmd){ |output,status| iter.return(output) } }, proc{ |results| p results })



223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
 
# File 'lib/em/iterator.rb', line 223

def map(foreach, after)
  index = 0

  inject([], proc{ |results,item,iter|
    i = index
    index += 1

    is_done = false
    on_done = proc{ |res|
      raise RuntimeError, 'already returned a value for this iteration' if is_done
      is_done = true

      results[i] = res
      iter.return(results)
    }
    class << on_done
      alias :return :call
      def next
        raise NoMethodError, 'must call #return on a map iterator'
      end
    end

    foreach.call(item, on_done)
  }, proc{ |results|
    after.call(results)
  })
end