Module: ScraperUtils::MiscUtils

Defined in:
lib/scraper_utils/misc_utils.rb

Overview

Misc Standalone Utilities

Constant Summary collapse

THROTTLE_HOSTNAME = 
"block"

Class Method Summary collapse

Class Method Details

.reset_throttler! ⇒ Object

Reset the internal throttler (useful in tests)



26
27
28
 
# File 'lib/scraper_utils/misc_utils.rb', line 26

def reset_throttler!
  @throttler = nil
end

.throttle_block ⇒ Object

Throttle block to be nice to servers we are scraping. Time spent inside the block (parsing, saving) counts toward the delay.



13
14
15
16
17
18
19
20
21
22
23
 
# File 'lib/scraper_utils/misc_utils.rb', line 13

def throttle_block
  throttler.before_request(THROTTLE_HOSTNAME)
  begin
    result = yield
    throttler.after_request(THROTTLE_HOSTNAME)
    result
  rescue StandardError => e
    throttler.after_request(THROTTLE_HOSTNAME, overloaded: HostThrottler.overload_error?(e))
    raise
  end
end

.will_pause_till ⇒ Object 



30
31
32
 
# File 'lib/scraper_utils/misc_utils.rb', line 30

def will_pause_till
  throttler.will_pause_till(THROTTLE_HOSTNAME)
end