Class: Anemone::Tentacle

Inherits:
Object
  • Object
show all
Defined in:
lib/anemone/tentacle.rb

Instance Method Summary collapse

Constructor Details

#initialize(link_queue, page_queue, opts = {}) ⇒ Tentacle

Create a new Tentacle



9
10
11
12
13
14
 
# File 'lib/anemone/tentacle.rb', line 9

def initialize(link_queue, page_queue, opts = {})
  @link_queue = link_queue
  @page_queue = page_queue
  @http = Anemone::HTTP.new(opts)
  @opts = opts
end

Instance Method Details

#run ⇒ Object

Gets links from @link_queue, and returns the fetched Page objects into @page_queue



20
21
22
23
24
25
26
27
28
29
30
 
# File 'lib/anemone/tentacle.rb', line 20

def run
  loop do
    link, referer, depth = @link_queue.deq

    break if link == :END

    @http.fetch_pages(link, referer, depth).each { |page| @page_queue << page }

    delay
  end
end