Module: FailureHandler

Included in:
ClearanceJobsComCrawler, ClearanceJobsComParser, ClearedJobsNetCrawler, ClearedJobsNetParser, SecurityClearedJobsComCrawler, SecurityClearedJobsComParser
Defined in:
lib/util/failure_handler.rb

Instance Method Summary collapse

Instance Method Details

#get_retry(url, requests, i = 0) ⇒ Object 



6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 
# File 'lib/util/failure_handler.rb', line 6

def get_retry(url, requests, i=0)
  puts "crawling "+url
  begin
    if requests
      return requests.get_page(url)
    else
      return File.read(open(url.gsub("[", "%5B").gsub("]", "%5D")))
    end
  rescue
    if i < 10
      i+=1
      sleep(i*rand(1..10))
      get_retry(url, requests, i)
    end
  end
end