Class: Mechanize::HTTP::Agent

Inherits:
Object
  • Object
show all
Defined in:
lib/grubby/mechanize/fetch_with_retry.rb

Overview

This monkey patch attempts to fix the insidious “too many connection resets” bug described here: github.com/sparklemotion/mechanize/issues/123

The code is taken and modified from this helpful blog article: scottwb.com/blog/2013/11/09/defeating-the-infamous-mechanize-too-many-connection-resets-bug/

Constant Summary collapse

MAX_CONNECTION_RESET_RETRIES = 
9
IDEMPOTENT_HTTP_METHODS = 
[:get, :head, :options, :delete]

Instance Method Summary collapse

Instance Method Details

#fetch_with_retry(uri, http_method = :get, headers = {}, params = [], referer = current_page, redirects = 0) ⇒ Object Also known as: fetch

Replacement for Mechanize::HTTP::Agent#fetch. When a “too many connection resets” error is encountered, this method retries the request (upto MAX_CONNECTION_RESET_RETRIES times).



14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 
# File 'lib/grubby/mechanize/fetch_with_retry.rb', line 14

def fetch_with_retry(uri, http_method = :get, headers = {}, params = [], referer = current_page, redirects = 0)
  retry_count = 0
  begin
    fetch_without_retry(uri, http_method, headers, params, referer, redirects)
  rescue Net::HTTP::Persistent::Error => e
    # raise if different type of error
    raise unless e.message.include?("too many connection resets")
    # raise if non-idempotent http method
    raise unless IDEMPOTENT_HTTP_METHODS.include?(http_method)
    # raise if we've tried too many times
    raise if retry_count >= MAX_CONNECTION_RESET_RETRIES

    # otherwise, shutdown the persistent HTTP connection and try again
    retry_count += 1
    $log.warn("#{e.message} (#{e.class}).  Retry in #{retry_count} seconds.")
    sleep(retry_count) # incremental backoff to allow server to self-correct
    $log.warn("Retry #{http_method.to_s.upcase} #{uri}")
    retry
  end
end