Class: SiteDiff::Fetch

Inherits:
Object
  • Object
show all
Defined in:
lib/sitediff/fetch.rb

Overview

SiteDiff Data Fetcher. TODO: Rename this to Fetcher.

Instance Method Summary collapse

Constructor Details

#initialize(cache, paths, interval, concurrency = 3, curl_opts = nil, debug: true, **tags) ⇒ Fetch

Cache is a cache object, see sitediff/cache Paths is a list of sub-paths Tags is a hash of tag names => base URLs.



13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
 
# File 'lib/sitediff/fetch.rb', line 13

def initialize(cache,
               paths,
               interval,
               concurrency = 3,
               curl_opts = nil,
               debug: true,
               **tags)
  @cache = cache
  @interval = interval
  @paths = paths
  @tags = tags
  @curl_opts = curl_opts || UriWrapper::DEFAULT_CURL_OPTS
  @concurrency = concurrency
  @debug = debug
end

Instance Method Details

#run(&block) ⇒ Object

Fetch all the paths, once per tag. When a path has been fetched for every tag, block will be called with the path, and a hash of tag => UriWrapper::ReadResult objects.



32
33
34
35
36
37
 
# File 'lib/sitediff/fetch.rb', line 32

def run(&block)
  @callback = block
  @hydra = Typhoeus::Hydra.new(max_concurrency: @concurrency)
  @paths.each { |path| queue_path(path) }
  @hydra.run
end