Module: ScraperUtils::MiscUtils

Defined in:
lib/scraper_utils/misc_utils.rb

Overview

Misc Standalone Utilities

Constant Summary collapse

MAX_PAUSE = 
120.0

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.pause_duration ⇒ Object

Returns the value of attribute pause_duration.



9
10
11
 
# File 'lib/scraper_utils/misc_utils.rb', line 9

def pause_duration
  @pause_duration
end

Class Method Details

.throttle_block(extra_delay: 0.5) ⇒ Object

Throttle block to be nice to servers we are scraping



12
13
14
15
16
17
18
19
20
21
 
# File 'lib/scraper_utils/misc_utils.rb', line 12

def throttle_block(extra_delay: 0.5)
  if @pause_duration&.positive?
    puts "Pausing #{@pause_duration}s" if ScraperUtils::DebugUtils.trace?
    sleep(@pause_duration)
  end
  start_time = Time.now.to_f
  result = yield
  @pause_duration = (Time.now.to_f - start_time + extra_delay).round(3).clamp(0.0, MAX_PAUSE)
  result
end