Class: Mechanize::HTTP::Agent

Inherits:
Object
  • Object
show all
Defined in:
lib/magnum-pi/gem_ext/mechanize/http/agent.rb

Constant Summary collapse

MAX_RESET_RETRIES = 
10

Instance Method Summary collapse

Instance Method Details

#fetch_with_retry(uri, method = :get, headers = {}, params = [], referer = current_page, redirects = 0) ⇒ Object Also known as: fetch

We need to replace the core Mechanize HTTP method:

Mechanize::HTTP::Agent#fetch

with a wrapper that handles the infamous “too many connection resets” Mechanize bug that is described here:

https://github.com/sparklemotion/mechanize/issues/123

The wrapper shuts down the persistent HTTP connection when it fails with this error, and simply tries again. In practice, this only ever needs to be retried once, but I am going to let it retry a few times (MAX_RESET_RETRIES), just in case.



22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
 
# File 'lib/magnum-pi/gem_ext/mechanize/http/agent.rb', line 22

def fetch_with_retry(
  uri,
  method    = :get,
  headers   = {},
  params    = [],
  referer   = current_page,
  redirects = 0
)
  action      = "#{method.to_s.upcase} #{uri.to_s}"
  retry_count = 0

  begin
    fetch_without_retry(uri, method, headers, params, referer, redirects)
  rescue Net::HTTP::Persistent::Error => e
    # Pass on any other type of error.
    raise unless e.message =~ /too many connection resets/

    # Pass on the error if we've tried too many times.
    if retry_count >= MAX_RESET_RETRIES
      puts "**** WARN: Mechanize retried connection reset #{MAX_RESET_RETRIES} times and never succeeded: #{action}"
      raise
    end

    # Otherwise, shutdown the persistent HTTP connection and try again.
    puts "**** WARN: Mechanize retrying connection reset error: #{action}"
    retry_count += 1
    self.http.shutdown
    retry
  end
end