Class: Mechanize::HTTP::Agent
- Inherits:
-
Object
- Object
- Mechanize::HTTP::Agent
- Defined in:
- lib/mechanize/http/agent.rb
Overview
An HTTP (and local disk access) user agent. This class is an implementation detail and is subject to change at any time.
Constant Summary collapse
- CREDENTIAL_HEADERS =
['Authorization']
- COOKIE_HEADERS =
['Cookie']
- POST_HEADERS =
['Content-Length', 'Content-MD5', 'Content-Type']
- RobotsKey =
:section: Robots
:__mechanize_get_robots__
Instance Attribute Summary collapse
-
#allowed_error_codes ⇒ Object
List of error codes (in String or Integer) to handle without raising Mechanize::ResponseCodeError, defaulted to an empty array.
-
#auth_store ⇒ Object
readonly
:section: HTTP Authentication.
-
#authenticate_methods ⇒ Object
readonly
:nodoc:.
-
#conditional_requests ⇒ Object
Disables If-Modified-Since conditional requests (enabled by default).
-
#content_encoding_hooks ⇒ Object
readonly
A list of hooks to call to handle the content-encoding of a request.
-
#context ⇒ Object
The context parses responses into pages.
-
#cookie_jar ⇒ Object
The cookies for this agent.
-
#digest_challenges ⇒ Object
readonly
:nodoc:.
-
#follow_meta_refresh ⇒ Object
Follow HTML meta refresh and HTTP Refresh.
-
#follow_meta_refresh_self ⇒ Object
Follow an HTML meta refresh that has no “url=” in the content attribute.
-
#gzip_enabled ⇒ Object
Is gzip compression of requests enabled?.
-
#history ⇒ Object
history of requests made.
-
#http ⇒ Object
readonly
:nodoc:.
-
#ignore_bad_chunking ⇒ Object
When set to true mechanize will ignore an EOF during chunked transfer encoding so long as at least one byte was received.
-
#keep_alive ⇒ Object
Set to false to disable HTTP/1.1 keep-alive requests.
-
#max_file_buffer ⇒ Object
Responses larger than this will be written to a Tempfile instead of stored in memory.
-
#open_timeout ⇒ Object
Length of time to wait until a connection is opened in seconds.
-
#pass ⇒ Object
OpenSSL key password.
-
#post_connect_hooks ⇒ Object
readonly
A list of hooks to call after retrieving a response.
-
#pre_connect_hooks ⇒ Object
readonly
A list of hooks to call before making a request.
-
#read_timeout ⇒ Object
Length of time to attempt to read data from the server.
-
#redirect_ok ⇒ Object
Controls how this agent deals with redirects.
-
#redirection_limit ⇒ Object
Maximum number of redirects to follow.
-
#request_headers ⇒ Object
A hash of request headers to be used for every request.
-
#robots ⇒ Object
When true, this agent will consult the site’s robots.txt for each access.
-
#robots_mutex ⇒ Object
readonly
Mutex used when fetching robots.txt.
-
#scheme_handlers ⇒ Object
Handlers for various URI schemes.
-
#user_agent ⇒ Object
The User-Agent header to send.
Instance Method Summary collapse
-
#add_auth(uri, user, password, realm = nil, domain = nil) ⇒ Object
Adds credentials
user
,pass
foruri
. -
#add_default_auth(user, password, domain = nil) ⇒ Object
USE OF add_default_auth IS NOT RECOMMENDED AS IT MAY EXPOSE PASSWORDS TO THIRD PARTIES.
-
#auto_io(name, read_size, input_io) ⇒ Object
Creates a new output IO by reading
input_io
inread_size
chunks. -
#back ⇒ Object
Equivalent to the browser back button.
-
#ca_file ⇒ Object
Path to an OpenSSL CA certificate file.
-
#ca_file=(ca_file) ⇒ Object
Sets the path to an OpenSSL CA certificate file.
-
#cert_store ⇒ Object
The SSL certificate store used for validating connections.
-
#cert_store=(cert_store) ⇒ Object
Sets the SSL certificate store used for validating connections.
-
#certificate ⇒ Object
The client X509 certificate.
-
#certificate=(certificate) ⇒ Object
Sets the client certificate to given X509 certificate.
-
#connection_for(uri) ⇒ Object
:section: Request.
-
#content_encoding_brotli(body_io) ⇒ Object
Decodes a Brotli-encoded
body_io
. -
#content_encoding_gunzip(body_io) ⇒ Object
Decodes a gzip-encoded
body_io
. -
#content_encoding_inflate(body_io) ⇒ Object
Decodes a deflate-encoded
body_io
. -
#content_encoding_zstd(body_io) ⇒ Object
Decodes a Zstd-encoded
body_io
. -
#current_page ⇒ Object
Returns the latest page loaded by the agent.
- #disable_keep_alive(request) ⇒ Object
- #enable_gzip(request) ⇒ Object
-
#fetch(uri, method = :get, headers = {}, params = [], referer = current_page, redirects = 0) ⇒ Object
Retrieves
uri
and parses it into a page or other object according to PluggableParser. -
#get_meta_refresh(response, uri, page) ⇒ Object
:section: Response.
-
#get_robots(uri) ⇒ Object
:nodoc:.
-
#hook_content_encoding(response, uri, response_body_io) ⇒ Object
:section: Hooks.
- #http_request(uri, method, params = nil) ⇒ Object
-
#idle_timeout ⇒ Object
Reset connections that have not been used in this many seconds.
-
#idle_timeout=(timeout) ⇒ Object
Sets the connection idle timeout for persistent connections.
- #inflate(compressed, window_bits = nil) ⇒ Object
-
#initialize(connection_name = 'mechanize') ⇒ Agent
constructor
The connection_name can be used to segregate SSL connections.
- #log ⇒ Object
- #make_tempfile(name) ⇒ Object
-
#max_history ⇒ Object
Returns the maximum size for the history stack.
-
#max_history=(length) ⇒ Object
Set the maximum size for the history stack.
-
#post_connect(uri, response, body_io) ⇒ Object
Invokes hooks added to post_connect_hooks after a
response
is returned and the responsebody
is handled. -
#pre_connect(request) ⇒ Object
Invokes hooks added to pre_connect_hooks before a
request
is made. -
#private_key ⇒ Object
An OpenSSL private key or the path to a private key.
-
#private_key=(private_key) ⇒ Object
Sets the client’s private key.
-
#proxy_uri ⇒ Object
URI for a proxy connection.
- #request_add_headers(request, headers = {}) ⇒ Object
- #request_auth(request, uri) ⇒ Object
- #request_auth_digest(request, uri, realm, base_uri, iis) ⇒ Object
- #request_cookies(request, uri) ⇒ Object
- #request_host(request, uri) ⇒ Object
- #request_language_charset(request) ⇒ Object
-
#request_log(request) ⇒ Object
Log specified headers for the request.
-
#request_referer(request, uri, referer) ⇒ Object
Sets a Referer header.
- #request_user_agent(request) ⇒ Object
- #reset ⇒ Object
- #resolve(uri, referer = current_page) ⇒ Object
- #resolve_parameters(uri, method, parameters) ⇒ Object
- #response_authenticate(response, page, uri, request, headers, params, referer) ⇒ Object
- #response_content_encoding(response, body_io) ⇒ Object
- #response_cookies(response, uri, page) ⇒ Object
- #response_follow_meta_refresh(response, uri, page, redirects) ⇒ Object
- #response_log(response) ⇒ Object
- #response_parse(response, body_io, uri) ⇒ Object
- #response_read(response, request, uri) ⇒ Object
- #response_redirect(response, method, page, redirects, headers, referer = current_page) ⇒ Object
-
#retry_change_requests ⇒ Object
Retry non-idempotent requests?.
-
#retry_change_requests=(retri) ⇒ Object
Retry non-idempotent requests.
-
#robots_allowed?(uri) ⇒ Boolean
Tests if this agent is allowed to access
url
, consulting the site’s robots.txt. -
#robots_disallowed?(url) ⇒ Boolean
Opposite of robots_allowed?.
-
#robots_error(url) ⇒ Object
Returns an error object if there is an error in fetching or parsing robots.txt of the site
url
. -
#robots_error!(url) ⇒ Object
Raises the error if there is an error in fetching or parsing robots.txt of the site
url
. -
#robots_reset(url) ⇒ Object
Removes robots.txt cache for the site
url
. - #save_cookies(uri, set_cookie) ⇒ Object
- #secure_resolve!(uri, referer = current_page) ⇒ Object
-
#set_proxy(addr, port = nil, user = nil, pass = nil) ⇒ Object
Sets the proxy address, port, user, and password.
-
#shutdown ⇒ Object
Closes all open connections for this agent.
-
#ssl_version ⇒ Object
SSL version to use.
-
#ssl_version=(ssl_version) ⇒ Object
Sets the SSL version to use.
- #use_tempfile?(size) ⇒ Boolean
-
#verify_callback ⇒ Object
A callback for additional certificate verification.
-
#verify_callback=(verify_callback) ⇒ Object
Sets the certificate verify callback.
-
#verify_mode ⇒ Object
How to verify SSL connections.
-
#verify_mode=(verify_mode) ⇒ Object
Sets the mode for verifying SSL connections.
-
#visited_page(url) ⇒ Object
Returns a visited page for the url passed in, otherwise nil.
- #webrobots ⇒ Object
Constructor Details
#initialize(connection_name = 'mechanize') ⇒ Agent
The connection_name can be used to segregate SSL connections. Agents with different names will not share the same persistent connection.
140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 |
# File 'lib/mechanize/http/agent.rb', line 140 def initialize(connection_name = 'mechanize') @allowed_error_codes = [] @conditional_requests = true @context = nil @content_encoding_hooks = [] @cookie_jar = Mechanize::CookieJar.new @follow_meta_refresh = false @follow_meta_refresh_self = false @gzip_enabled = true @history = Mechanize::History.new @ignore_bad_chunking = false @keep_alive = true @max_file_buffer = 100_000 # 5MB for response bodies @open_timeout = nil @post_connect_hooks = [] @pre_connect_hooks = [] @read_timeout = nil @redirect_ok = true @redirection_limit = 20 @request_headers = {} @robots = false @robots_mutex = Mutex.new @user_agent = nil @webrobots = nil # HTTP Authentication @auth_store = Mechanize::HTTP::AuthStore.new @authenticate_parser = Mechanize::HTTP::WWWAuthenticateParser.new @authenticate_methods = Hash.new do |methods, uri| methods[uri] = Hash.new do |realms, auth_scheme| realms[auth_scheme] = [] end end @digest_auth = Net::HTTP::DigestAuth.new @digest_challenges = {} # SSL @pass = nil @scheme_handlers = Hash.new { |h, scheme| h[scheme] = lambda { |link, page| raise Mechanize::UnsupportedSchemeError.new(scheme, link) } } @scheme_handlers['http'] = lambda { |link, page| link } @scheme_handlers['https'] = @scheme_handlers['http'] @scheme_handlers['relative'] = @scheme_handlers['http'] @scheme_handlers['file'] = @scheme_handlers['http'] @http = if defined?(Net::HTTP::Persistent::DEFAULT_POOL_SIZE) Net::HTTP::Persistent.new(name: connection_name) else # net-http-persistent < 3.0 Net::HTTP::Persistent.new(connection_name) end @http.idle_timeout = 5 @http.keep_alive = 300 end |
Instance Attribute Details
#allowed_error_codes ⇒ Object
List of error codes (in String or Integer) to handle without raising Mechanize::ResponseCodeError, defaulted to an empty array. Note that 2xx, 3xx and 401 status codes will be handled without checking this list.
83 84 85 |
# File 'lib/mechanize/http/agent.rb', line 83 def allowed_error_codes @allowed_error_codes end |
#auth_store ⇒ Object (readonly)
:section: HTTP Authentication
50 51 52 |
# File 'lib/mechanize/http/agent.rb', line 50 def auth_store @auth_store end |
#authenticate_methods ⇒ Object (readonly)
:nodoc:
51 52 53 |
# File 'lib/mechanize/http/agent.rb', line 51 def authenticate_methods @authenticate_methods end |
#conditional_requests ⇒ Object
Disables If-Modified-Since conditional requests (enabled by default)
19 20 21 |
# File 'lib/mechanize/http/agent.rb', line 19 def conditional_requests @conditional_requests end |
#content_encoding_hooks ⇒ Object (readonly)
A list of hooks to call to handle the content-encoding of a request.
46 47 48 |
# File 'lib/mechanize/http/agent.rb', line 46 def content_encoding_hooks @content_encoding_hooks end |
#context ⇒ Object
The context parses responses into pages
121 122 123 |
# File 'lib/mechanize/http/agent.rb', line 121 def context @context end |
#cookie_jar ⇒ Object
The cookies for this agent
112 113 114 |
# File 'lib/mechanize/http/agent.rb', line 112 def @cookie_jar end |
#digest_challenges ⇒ Object (readonly)
:nodoc:
52 53 54 |
# File 'lib/mechanize/http/agent.rb', line 52 def digest_challenges @digest_challenges end |
#follow_meta_refresh ⇒ Object
Follow HTML meta refresh and HTTP Refresh. If set to :anywhere
meta refresh tags outside of the head element will be followed.
58 59 60 |
# File 'lib/mechanize/http/agent.rb', line 58 def @follow_meta_refresh end |
#follow_meta_refresh_self ⇒ Object
Follow an HTML meta refresh that has no “url=” in the content attribute.
Defaults to false to prevent infinite refresh loops.
63 64 65 |
# File 'lib/mechanize/http/agent.rb', line 63 def @follow_meta_refresh_self end |
#gzip_enabled ⇒ Object
Is gzip compression of requests enabled?
22 23 24 |
# File 'lib/mechanize/http/agent.rb', line 22 def gzip_enabled @gzip_enabled end |
#history ⇒ Object
history of requests made
33 34 35 |
# File 'lib/mechanize/http/agent.rb', line 33 def history @history end |
#http ⇒ Object (readonly)
:nodoc:
123 124 125 |
# File 'lib/mechanize/http/agent.rb', line 123 def http @http end |
#ignore_bad_chunking ⇒ Object
When set to true mechanize will ignore an EOF during chunked transfer encoding so long as at least one byte was received. Be careful when enabling this as it may cause data loss.
128 129 130 |
# File 'lib/mechanize/http/agent.rb', line 128 def ignore_bad_chunking @ignore_bad_chunking end |
#keep_alive ⇒ Object
Set to false to disable HTTP/1.1 keep-alive requests
101 102 103 |
# File 'lib/mechanize/http/agent.rb', line 101 def keep_alive @keep_alive end |
#max_file_buffer ⇒ Object
Responses larger than this will be written to a Tempfile instead of stored in memory. Setting this to nil disables creation of Tempfiles.
116 117 118 |
# File 'lib/mechanize/http/agent.rb', line 116 def max_file_buffer @max_file_buffer end |
#open_timeout ⇒ Object
Length of time to wait until a connection is opened in seconds
104 105 106 |
# File 'lib/mechanize/http/agent.rb', line 104 def open_timeout @open_timeout end |
#pass ⇒ Object
OpenSSL key password
96 97 98 |
# File 'lib/mechanize/http/agent.rb', line 96 def pass @pass end |
#post_connect_hooks ⇒ Object (readonly)
A list of hooks to call after retrieving a response. Hooks are called with the agent and the response returned.
39 40 41 |
# File 'lib/mechanize/http/agent.rb', line 39 def post_connect_hooks @post_connect_hooks end |
#pre_connect_hooks ⇒ Object (readonly)
A list of hooks to call before making a request. Hooks are called with the agent and the request to be performed.
43 44 45 |
# File 'lib/mechanize/http/agent.rb', line 43 def pre_connect_hooks @pre_connect_hooks end |
#read_timeout ⇒ Object
Length of time to attempt to read data from the server
107 108 109 |
# File 'lib/mechanize/http/agent.rb', line 107 def read_timeout @read_timeout end |
#redirect_ok ⇒ Object
Controls how this agent deals with redirects. The following values are allowed:
- :all, true
-
All 3xx redirects are followed (default)
- :permanent
-
Only 301 Moved Permanently redirects are followed
- false
-
No redirects are followed
71 72 73 |
# File 'lib/mechanize/http/agent.rb', line 71 def redirect_ok @redirect_ok end |
#redirection_limit ⇒ Object
Maximum number of redirects to follow
74 75 76 |
# File 'lib/mechanize/http/agent.rb', line 74 def redirection_limit @redirection_limit end |
#request_headers ⇒ Object
A hash of request headers to be used for every request
25 26 27 |
# File 'lib/mechanize/http/agent.rb', line 25 def request_headers @request_headers end |
#robots ⇒ Object
When true, this agent will consult the site’s robots.txt for each access.
88 89 90 |
# File 'lib/mechanize/http/agent.rb', line 88 def robots @robots end |
#robots_mutex ⇒ Object (readonly)
Mutex used when fetching robots.txt
91 92 93 |
# File 'lib/mechanize/http/agent.rb', line 91 def robots_mutex @robots_mutex end |
#scheme_handlers ⇒ Object
Handlers for various URI schemes
131 132 133 |
# File 'lib/mechanize/http/agent.rb', line 131 def scheme_handlers @scheme_handlers end |
#user_agent ⇒ Object
The User-Agent header to send
28 29 30 |
# File 'lib/mechanize/http/agent.rb', line 28 def user_agent @user_agent end |
Instance Method Details
#add_auth(uri, user, password, realm = nil, domain = nil) ⇒ Object
Adds credentials user
, pass
for uri
. If realm
is set the credentials are used only for that realm. If realm
is not set the credentials become the default for any realm on that URI.
domain
and realm
are exclusive as NTLM does not follow RFC 2617. If domain
is given it is only used for NTLM authentication.
209 210 211 |
# File 'lib/mechanize/http/agent.rb', line 209 def add_auth uri, user, password, realm = nil, domain = nil @auth_store.add_auth uri, user, password, realm, domain end |
#add_default_auth(user, password, domain = nil) ⇒ Object
USE OF add_default_auth IS NOT RECOMMENDED AS IT MAY EXPOSE PASSWORDS TO THIRD PARTIES
Adds credentials user
, pass
as the default authentication credentials. If no other credentials are available these will be returned from credentials_for.
If domain
is given it is only used for NTLM authentication.
223 224 225 |
# File 'lib/mechanize/http/agent.rb', line 223 def add_default_auth user, password, domain = nil # :nodoc: @auth_store.add_default_auth user, password, domain end |
#auto_io(name, read_size, input_io) ⇒ Object
Creates a new output IO by reading input_io
in read_size
chunks. If the output is over the max_file_buffer size a Tempfile with name
is created.
If a block is provided, each chunk of input_io
is yielded for further processing.
1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 |
# File 'lib/mechanize/http/agent.rb', line 1254 def auto_io name, read_size, input_io out_io = StringIO.new.set_encoding(Encoding::BINARY) until input_io.eof? do if StringIO === out_io and use_tempfile? out_io.size then new_io = make_tempfile name new_io.write out_io.string out_io = new_io end chunk = input_io.read read_size chunk = yield chunk if block_given? out_io.write chunk end out_io.rewind out_io end |
#back ⇒ Object
Equivalent to the browser back button. Returns the most recent page visited.
366 367 368 |
# File 'lib/mechanize/http/agent.rb', line 366 def back @history.pop end |
#ca_file ⇒ Object
Path to an OpenSSL CA certificate file
1145 1146 1147 |
# File 'lib/mechanize/http/agent.rb', line 1145 def ca_file @http.ca_file end |
#ca_file=(ca_file) ⇒ Object
Sets the path to an OpenSSL CA certificate file
1150 1151 1152 |
# File 'lib/mechanize/http/agent.rb', line 1150 def ca_file= ca_file @http.ca_file = ca_file end |
#cert_store ⇒ Object
The SSL certificate store used for validating connections
1155 1156 1157 |
# File 'lib/mechanize/http/agent.rb', line 1155 def cert_store @http.cert_store end |
#cert_store=(cert_store) ⇒ Object
Sets the SSL certificate store used for validating connections
1160 1161 1162 |
# File 'lib/mechanize/http/agent.rb', line 1160 def cert_store= cert_store @http.cert_store = cert_store end |
#certificate ⇒ Object
The client X509 certificate
1165 1166 1167 |
# File 'lib/mechanize/http/agent.rb', line 1165 def certificate @http.certificate end |
#certificate=(certificate) ⇒ Object
Sets the client certificate to given X509 certificate. If a path is given the certificate will be loaded and set.
1171 1172 1173 1174 1175 1176 1177 1178 1179 |
# File 'lib/mechanize/http/agent.rb', line 1171 def certificate= certificate certificate = if OpenSSL::X509::Certificate === certificate then certificate else OpenSSL::X509::Certificate.new File.read certificate end @http.certificate = certificate end |
#connection_for(uri) ⇒ Object
:section: Request
429 430 431 432 433 434 435 436 |
# File 'lib/mechanize/http/agent.rb', line 429 def connection_for uri case uri.scheme.downcase when 'http', 'https' then return @http when 'file' then return Mechanize::FileConnection.new end end |
#content_encoding_brotli(body_io) ⇒ Object
Decodes a Brotli-encoded body_io
(Experimental, CRuby only) Although Mechanize will never request a Brotli-encoded response via ‘accept-encoding`, buggy servers may return brotli-encoded responses anyway. Let’s try to handle that case if the Brotli gem is loaded.
If you need to handle Brotli-encoded responses, install the ‘brotli’ gem and require it in your application. If the ‘Brotli` constant is defined, Mechanize will attempt to use it to inflate the response.
509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 |
# File 'lib/mechanize/http/agent.rb', line 509 def content_encoding_brotli(body_io) log.debug('deflate brotli body') if log unless defined?(::Brotli) raise Mechanize::Error, "cannot deflate brotli-encoded response. Please install and require the 'brotli' gem." end begin return StringIO.new(Brotli.inflate(body_io.read)) rescue Brotli::Error log.error("unable to brotli-inflate response") if log raise Mechanize::Error, "error inflating brotli-encoded response." end ensure body_io.close end |
#content_encoding_gunzip(body_io) ⇒ Object
Decodes a gzip-encoded body_io
. If it cannot be decoded, inflate is tried followed by raising an error.
447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 |
# File 'lib/mechanize/http/agent.rb', line 447 def content_encoding_gunzip body_io log.debug('gzip response') if log zio = Zlib::GzipReader.new body_io out_io = auto_io 'mechanize-gunzip', 16384, zio zio.finish return out_io rescue Zlib::Error => gz_error log.warn "unable to gunzip response: #{gz_error} (#{gz_error.class})" if log body_io.rewind body_io.read 10 begin log.warn "trying raw inflate on response" if log return inflate body_io, -Zlib::MAX_WBITS rescue Zlib::Error => e log.error "unable to inflate response: #{e} (#{e.class})" if log raise end ensure # do not close a second time if we failed the first time zio.close if zio and !(zio.closed? or gz_error) body_io.close unless body_io.closed? end |
#content_encoding_inflate(body_io) ⇒ Object
Decodes a deflate-encoded body_io
. If it cannot be decoded, raw inflate is tried followed by raising an error.
479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 |
# File 'lib/mechanize/http/agent.rb', line 479 def content_encoding_inflate body_io log.debug('deflate body') if log return inflate body_io rescue Zlib::Error log.error('unable to inflate response, trying raw deflate') if log body_io.rewind begin return inflate body_io, -Zlib::MAX_WBITS rescue Zlib::Error => e log.error("unable to inflate response: #{e}") if log raise end ensure body_io.close end |
#content_encoding_zstd(body_io) ⇒ Object
Decodes a Zstd-encoded body_io
(Experimental, CRuby only) Although Mechanize will never request a zstd-encoded response via ‘accept-encoding`, buggy servers may return zstd-encoded responses, or you might need to inform the zstd keyword on your Accept-Encoding headers. Let’s try to handle those cases if the Zstd gem is loaded.
If you need to handle Zstd-encoded responses, install the ‘zstd-ruby’ gem and require it in your application. If the ‘Zstd` constant is defined, Mechanize will attempt to use it to inflate the response.
538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 |
# File 'lib/mechanize/http/agent.rb', line 538 def content_encoding_zstd(body_io) log.debug('deflate zstd body') if log unless defined?(::Zstd) raise Mechanize::Error, "cannot deflate zstd-encoded response. Please install and require the 'zstd-ruby' gem." end begin return StringIO.new(Zstd.decompress(body_io.read)) rescue StandardError log.error("unable to zstd#decompress response") if log raise Mechanize::Error, "error decompressing zstd-encoded response." end ensure body_io.close end |
#current_page ⇒ Object
Returns the latest page loaded by the agent
373 374 375 |
# File 'lib/mechanize/http/agent.rb', line 373 def current_page @history.last end |
#disable_keep_alive(request) ⇒ Object
555 556 557 |
# File 'lib/mechanize/http/agent.rb', line 555 def disable_keep_alive request request['connection'] = 'close' unless @keep_alive end |
#enable_gzip(request) ⇒ Object
559 560 561 562 563 564 565 |
# File 'lib/mechanize/http/agent.rb', line 559 def enable_gzip request request['accept-encoding'] = if @gzip_enabled 'gzip,deflate,identity' else 'identity' end end |
#fetch(uri, method = :get, headers = {}, params = [], referer = current_page, redirects = 0) ⇒ Object
Retrieves uri
and parses it into a page or other object according to PluggableParser. If the URI is an HTTP or HTTPS scheme URI the given HTTP method
is used to retrieve it, along with the HTTP headers
, request params
and HTTP referer
.
The final URI to access is built with uri
and params
, the latter of which is formatted into a string using Mechanize::Util.build_query_string, which see.
redirects
tracks the number of redirects experienced when retrieving the page. If it is over the redirection_limit an error will be raised.
240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 |
# File 'lib/mechanize/http/agent.rb', line 240 def fetch uri, method = :get, headers = {}, params = [], referer = current_page, redirects = 0 referer_uri = referer ? referer.uri : nil uri = resolve uri, referer uri, params = resolve_parameters uri, method, params request = http_request uri, method, params connection = connection_for uri request_auth request, uri disable_keep_alive request enable_gzip request request_language_charset request request, uri request_host request, uri request_referer request, uri, referer_uri request_user_agent request request_add_headers request, headers pre_connect request # Consult robots.txt if robots && uri.is_a?(URI::HTTP) robots_allowed?(uri) or raise Mechanize::RobotsDisallowedError.new(uri) end # Add If-Modified-Since if page is in history if page = visited_page(uri) and last_modified = page.response['Last-Modified'] request['If-Modified-Since'] = last_modified end if @conditional_requests # Specify timeouts if supplied and our connection supports them if @open_timeout && connection.respond_to?(:open_timeout=) connection.open_timeout = @open_timeout end if @read_timeout && connection.respond_to?(:read_timeout=) connection.read_timeout = @read_timeout end request_log request response_body_io = nil # Send the request begin response = connection.request(uri, request) { |res| response_log res response_body_io = response_read res, request, uri res } rescue Mechanize::ChunkedTerminationError => e raise unless @ignore_bad_chunking response = e.response response_body_io = e.body_io end hook_content_encoding response, uri, response_body_io response_body_io = response_content_encoding response, response_body_io if request.response_body_permitted? post_connect uri, response, response_body_io page = response_parse response, response_body_io, uri response, uri, page = response, uri, page, redirects return if if robots && page.is_a?(Mechanize::Page) page.parser.noindex? and raise Mechanize::RobotsDisallowedError.new(uri) end case response when Net::HTTPSuccess page when Mechanize::FileResponse page when Net::HTTPNotModified log.debug("Got cached page") if log visited_page(uri) || page when Net::HTTPRedirection response_redirect response, method, page, redirects, headers, referer when Net::HTTPUnauthorized response_authenticate(response, page, uri, request, headers, params, referer) else if @allowed_error_codes.any? {|code| code.to_s == page.code} then page else raise Mechanize::ResponseCodeError.new(page, 'unhandled response') end end end |
#get_meta_refresh(response, uri, page) ⇒ Object
:section: Response
781 782 783 784 785 786 787 788 789 790 791 792 793 794 |
# File 'lib/mechanize/http/agent.rb', line 781 def response, uri, page return nil unless @follow_meta_refresh if page.respond_to?(:meta_refresh) and (redirect = page..first) then [redirect.delay, redirect.href] unless not @follow_meta_refresh_self and redirect.link_self elsif refresh = response['refresh'] delay, href, link_self = Mechanize::Page::MetaRefresh.parse refresh, uri raise Mechanize::Error, 'Invalid refresh http header' unless delay [delay.to_f, href] unless not @follow_meta_refresh_self and link_self end end |
#get_robots(uri) ⇒ Object
:nodoc:
1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 |
# File 'lib/mechanize/http/agent.rb', line 1079 def get_robots(uri) # :nodoc: robots_mutex.synchronize do Thread.current[RobotsKey] = true begin fetch(uri).body rescue Mechanize::ResponseCodeError => e case e.response_code when /\A4\d\d\z/ '' else raise e end rescue Mechanize::RedirectLimitReachedError '' ensure Thread.current[RobotsKey] = false end end end |
#hook_content_encoding(response, uri, response_body_io) ⇒ Object
:section: Hooks
394 395 396 397 398 |
# File 'lib/mechanize/http/agent.rb', line 394 def hook_content_encoding response, uri, response_body_io @content_encoding_hooks.each do |hook| hook.call self, uri, response, response_body_io end end |
#http_request(uri, method, params = nil) ⇒ Object
567 568 569 570 571 572 573 574 575 576 577 578 579 |
# File 'lib/mechanize/http/agent.rb', line 567 def http_request uri, method, params = nil case uri.scheme.downcase when 'http', 'https' then klass = Net::HTTP.const_get(method.to_s.capitalize) request ||= klass.new(uri.request_uri) request.body = params.first if params request when 'file' then Mechanize::FileRequest.new uri end end |
#idle_timeout ⇒ Object
Reset connections that have not been used in this many seconds
1235 1236 1237 |
# File 'lib/mechanize/http/agent.rb', line 1235 def idle_timeout @http.idle_timeout end |
#idle_timeout=(timeout) ⇒ Object
Sets the connection idle timeout for persistent connections
1240 1241 1242 |
# File 'lib/mechanize/http/agent.rb', line 1240 def idle_timeout= timeout @http.idle_timeout = timeout end |
#inflate(compressed, window_bits = nil) ⇒ Object
1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 |
# File 'lib/mechanize/http/agent.rb', line 1275 def inflate compressed, window_bits = nil inflate = Zlib::Inflate.new window_bits out_io = auto_io 'mechanize-inflate', 1024, compressed do |chunk| inflate.inflate chunk end inflate.finish out_io ensure inflate.close if inflate.finished? end |
#log ⇒ Object
1289 1290 1291 |
# File 'lib/mechanize/http/agent.rb', line 1289 def log @context.log end |
#make_tempfile(name) ⇒ Object
1333 1334 1335 1336 1337 1338 |
# File 'lib/mechanize/http/agent.rb', line 1333 def make_tempfile name io = Tempfile.new name io.unlink io.binmode io end |
#max_history ⇒ Object
Returns the maximum size for the history stack.
378 379 380 |
# File 'lib/mechanize/http/agent.rb', line 378 def max_history @history.max_size end |
#max_history=(length) ⇒ Object
Set the maximum size for the history stack.
383 384 385 |
# File 'lib/mechanize/http/agent.rb', line 383 def max_history=(length) @history.max_size = length end |
#post_connect(uri, response, body_io) ⇒ Object
Invokes hooks added to post_connect_hooks after a response
is returned and the response body
is handled.
Yields the context
, the uri
for the request, the response
and the response body
.
407 408 409 410 411 412 413 414 415 |
# File 'lib/mechanize/http/agent.rb', line 407 def post_connect uri, response, body_io # :yields: agent, uri, response, body @post_connect_hooks.each do |hook| begin hook.call self, uri, response, body_io.read ensure body_io.rewind end end end |
#pre_connect(request) ⇒ Object
Invokes hooks added to pre_connect_hooks before a request
is made. Yields the agent
and the request
that will be performed to each hook.
421 422 423 424 425 |
# File 'lib/mechanize/http/agent.rb', line 421 def pre_connect request # :yields: agent, request @pre_connect_hooks.each do |hook| hook.call self, request end end |
#private_key ⇒ Object
An OpenSSL private key or the path to a private key
1182 1183 1184 |
# File 'lib/mechanize/http/agent.rb', line 1182 def private_key @http.private_key end |
#private_key=(private_key) ⇒ Object
Sets the client’s private key
1187 1188 1189 1190 1191 1192 1193 1194 1195 |
# File 'lib/mechanize/http/agent.rb', line 1187 def private_key= private_key private_key = if OpenSSL::PKey::PKey === private_key then private_key else OpenSSL::PKey::RSA.new File.read(private_key), @pass end @http.private_key = private_key end |
#proxy_uri ⇒ Object
URI for a proxy connection
340 341 342 |
# File 'lib/mechanize/http/agent.rb', line 340 def proxy_uri @http.proxy_uri end |
#request_add_headers(request, headers = {}) ⇒ Object
581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 |
# File 'lib/mechanize/http/agent.rb', line 581 def request_add_headers request, headers = {} @request_headers.each do |k,v| request[k] = v end headers.each do |field, value| case field when :etag then request["ETag"] = value when :if_modified_since then request["If-Modified-Since"] = value when Symbol then raise ArgumentError, "unknown header symbol #{field}" else request[field] = value end end end |
#request_auth(request, uri) ⇒ Object
598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 |
# File 'lib/mechanize/http/agent.rb', line 598 def request_auth request, uri base_uri = uri + '/' base_uri.user &&= nil base_uri.password &&= nil schemes = @authenticate_methods[base_uri] if realm = schemes[:digest].find { |r| r.uri == base_uri } then request_auth_digest request, uri, realm, base_uri, false elsif realm = schemes[:iis_digest].find { |r| r.uri == base_uri } then request_auth_digest request, uri, realm, base_uri, true elsif realm = schemes[:basic].find { |r| r.uri == base_uri } then user, password, = @auth_store.credentials_for uri, realm.realm request.basic_auth user, password end end |
#request_auth_digest(request, uri, realm, base_uri, iis) ⇒ Object
614 615 616 617 618 619 620 621 |
# File 'lib/mechanize/http/agent.rb', line 614 def request_auth_digest request, uri, realm, base_uri, iis challenge = @digest_challenges[realm] uri.user, uri.password, = @auth_store.credentials_for uri, realm.realm auth = @digest_auth.auth_header uri, challenge.to_s, request.method, iis request['Authorization'] = auth end |
#request_cookies(request, uri) ⇒ Object
623 624 625 626 627 628 629 630 631 |
# File 'lib/mechanize/http/agent.rb', line 623 def request, uri return if @cookie_jar.empty? uri = @cookie_jar. uri return if .empty? request.add_field 'Cookie', .join('; ') end |
#request_host(request, uri) ⇒ Object
633 634 635 636 637 638 |
# File 'lib/mechanize/http/agent.rb', line 633 def request_host request, uri port = [80, 443].include?(uri.port.to_i) ? nil : uri.port host = uri.host request['Host'] = [host, port].compact.join ':' end |
#request_language_charset(request) ⇒ Object
640 641 642 |
# File 'lib/mechanize/http/agent.rb', line 640 def request_language_charset request request['accept-language'] = 'en-us,en;q=0.5' end |
#request_log(request) ⇒ Object
Log specified headers for the request
645 646 647 648 649 650 651 652 653 |
# File 'lib/mechanize/http/agent.rb', line 645 def request_log request return unless log log.info("#{request.class}: #{request.path}") request.each_header do |k, v| log.debug("request-header: #{k} => #{v}") end end |
#request_referer(request, uri, referer) ⇒ Object
Sets a Referer header. Fragment part is removed as demanded by RFC 2616 14.36, and user information part is removed just like major browsers do.
658 659 660 661 662 663 664 665 666 667 |
# File 'lib/mechanize/http/agent.rb', line 658 def request_referer request, uri, referer return unless referer return if 'https'.casecmp(referer.scheme) == 0 and 'https'.casecmp(uri.scheme) != 0 if referer.fragment || referer.user || referer.password referer = referer.dup referer.fragment = referer.user = referer.password = nil end request['Referer'] = referer end |
#request_user_agent(request) ⇒ Object
669 670 671 |
# File 'lib/mechanize/http/agent.rb', line 669 def request_user_agent request request['User-Agent'] = @user_agent if @user_agent end |
#reset ⇒ Object
1347 1348 1349 1350 |
# File 'lib/mechanize/http/agent.rb', line 1347 def reset @cookie_jar.clear @history.clear end |
#resolve(uri, referer = current_page) ⇒ Object
673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 |
# File 'lib/mechanize/http/agent.rb', line 673 def resolve(uri, referer = current_page) referer_uri = referer && referer.uri if uri.is_a?(URI) uri = uri.dup elsif uri.nil? if referer_uri return referer_uri end raise ArgumentError, "absolute URL needed (not nil)" else url = uri.to_s.strip if url.empty? if referer_uri return referer_uri.dup.tap { |u| u.fragment = nil } end raise ArgumentError, "absolute URL needed (not #{uri.inspect})" end url.gsub!(/[^#{0.chr}-#{126.chr}]/o) { |match| Mechanize::Util.uri_escape(match) } escaped_url = Mechanize::Util.html_unescape( url.split(/((?:%[0-9A-Fa-f]{2})+|#)/).each_slice(2).map { |x, y| "#{WEBrick::HTTPUtils.escape(x)}#{y}" }.join('') ) begin uri = URI.parse(escaped_url) rescue uri = URI.parse(WEBrick::HTTPUtils.escape(escaped_url)) end end uri.host = referer_uri.host if referer_uri && URI::HTTP === uri && uri.host.nil? scheme = uri.relative? ? 'relative' : uri.scheme.downcase uri = @scheme_handlers[scheme].call(uri, referer) if uri.relative? raise ArgumentError, "absolute URL needed (not #{uri})" unless referer_uri if referer.respond_to?(:bases) && referer.parser && (lbase = referer.bases.last) && lbase.uri && lbase.uri.absolute? base = lbase else base = nil end base = referer_uri + (base ? base.uri : referer_uri) # Workaround for URI's bug in that it squashes consecutive # slashes. See #304. if uri.path.match(%r{\A(.*?/)(?!/\.\.?(?!/))(/.*)\z}i) uri = URI((base + $1).to_s + $2) else uri = base + uri end # Strip initial "/.." bits from the path uri.path.sub!(/^(\/\.\.)+(?=\/)/, '') end unless ['http', 'https', 'file'].include?(uri.scheme.downcase) raise ArgumentError, "unsupported scheme: #{uri.scheme}" end case uri.path when nil raise ArgumentError, "hierarchical URL needed (not #{uri})" when ''.freeze uri.path = '/' end uri end |
#resolve_parameters(uri, method, parameters) ⇒ Object
764 765 766 767 768 769 770 771 772 773 774 775 776 777 |
# File 'lib/mechanize/http/agent.rb', line 764 def resolve_parameters uri, method, parameters case method when :head, :get, :delete, :trace then if parameters and parameters.length > 0 uri.query ||= '' uri.query << '&' if uri.query.length > 0 uri.query << Mechanize::Util.build_query_string(parameters) end return uri, nil end return uri, parameters end |
#response_authenticate(response, page, uri, request, headers, params, referer) ⇒ Object
796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 |
# File 'lib/mechanize/http/agent.rb', line 796 def response_authenticate(response, page, uri, request, headers, params, referer) www_authenticate = response['www-authenticate'] unless www_authenticate = response['www-authenticate'] then = 'WWW-Authenticate header missing in response' raise Mechanize::UnauthorizedError.new(page, nil, ) end challenges = @authenticate_parser.parse www_authenticate unless @auth_store.credentials? uri, challenges then = "no credentials found, provide some with #add_auth" raise Mechanize::UnauthorizedError.new(page, challenges, ) end if challenge = challenges.find { |c| c.scheme =~ /^Digest$/i } then realm = challenge.realm uri auth_scheme = if response['server'] =~ /Microsoft-IIS/ then :iis_digest else :digest end existing_realms = @authenticate_methods[realm.uri][auth_scheme] if existing_realms.include? realm = 'Digest authentication failed' raise Mechanize::UnauthorizedError.new(page, challenges, ) end existing_realms << realm @digest_challenges[realm] = challenge elsif challenge = challenges.find { |c| c.scheme == 'NTLM' } then existing_realms = @authenticate_methods[uri + '/'][:ntlm] if existing_realms.include?(realm) and not challenge.params then = 'NTLM authentication failed' raise Mechanize::UnauthorizedError.new(page, challenges, ) end existing_realms << realm if challenge.params then type_2 = Net::NTLM::Message.decode64 challenge.params user, password, domain = @auth_store.credentials_for uri, nil type_3 = type_2.response({ :user => user, :password => password, :domain => domain }, { :ntlmv2 => true }).encode64 headers['Authorization'] = "NTLM #{type_3}" else type_1 = Net::NTLM::Message::Type1.new.encode64 headers['Authorization'] = "NTLM #{type_1}" end elsif challenge = challenges.find { |c| c.scheme == 'Basic' } then realm = challenge.realm uri existing_realms = @authenticate_methods[realm.uri][:basic] if existing_realms.include? realm then = 'Basic authentication failed' raise Mechanize::UnauthorizedError.new(page, challenges, ) end existing_realms << realm else = 'unsupported authentication scheme' raise Mechanize::UnauthorizedError.new(page, challenges, ) end fetch uri, request.method.downcase.to_sym, headers, params, referer end |
#response_content_encoding(response, body_io) ⇒ Object
873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 |
# File 'lib/mechanize/http/agent.rb', line 873 def response_content_encoding response, body_io length = response.content_length || case body_io when Tempfile, IO then body_io.stat.size else body_io.length end return body_io if length.zero? out_io = case response['Content-Encoding'] when nil, 'none', '7bit', 'identity', "" then body_io when 'deflate' then content_encoding_inflate body_io when 'gzip', 'x-gzip' then content_encoding_gunzip body_io when 'br' then content_encoding_brotli body_io when 'zstd' then content_encoding_zstd body_io else raise Mechanize::Error, "unsupported content-encoding: #{response['Content-Encoding']}" end out_io.flush out_io.rewind out_io rescue Zlib::Error => e = String.new("error handling content-encoding #{response['Content-Encoding']}:") << " #{e.} (#{e.class})" raise Mechanize::Error, ensure begin if Tempfile === body_io and (StringIO === out_io or (out_io and out_io.path != body_io.path)) then body_io.close! end rescue IOError # HACK ruby 1.8 raises IOError when closing the stream end end |
#response_cookies(response, uri, page) ⇒ Object
919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 |
# File 'lib/mechanize/http/agent.rb', line 919 def response, uri, page if Mechanize::Page === page and page.body =~ /Set-Cookie/n page.search('//head/meta[@http-equiv="Set-Cookie"]').each do || (uri, ['content']) end end = response.get_fields 'Set-Cookie' return unless .each do || (uri, ) end end |
#response_follow_meta_refresh(response, uri, page, redirects) ⇒ Object
947 948 949 950 951 952 953 954 955 956 957 958 959 |
# File 'lib/mechanize/http/agent.rb', line 947 def response, uri, page, redirects delay, new_url = (response, uri, page) return nil unless delay new_url = new_url ? secure_resolve!(new_url, page) : uri raise Mechanize::RedirectLimitReachedError.new(page, redirects) if redirects + 1 > @redirection_limit sleep delay @history.push(page, page.uri) fetch new_url, :get, {}, [], Mechanize::Page.new, redirects + 1 end |
#response_log(response) ⇒ Object
961 962 963 964 965 966 967 968 969 970 |
# File 'lib/mechanize/http/agent.rb', line 961 def response_log response return unless log log.info("status: #{response.class} #{response.http_version} " \ "#{response.code} #{response.}") response.each_header do |k, v| log.debug("response-header: #{k} => #{v}") end end |
#response_parse(response, body_io, uri) ⇒ Object
972 973 974 |
# File 'lib/mechanize/http/agent.rb', line 972 def response_parse response, body_io, uri @context.parse uri, response, body_io end |
#response_read(response, request, uri) ⇒ Object
976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 |
# File 'lib/mechanize/http/agent.rb', line 976 def response_read response, request, uri content_length = response.content_length if use_tempfile? content_length then body_io = make_tempfile 'mechanize-raw' else body_io = StringIO.new.set_encoding(Encoding::BINARY) end total = 0 begin response.read_body { |part| total += part.length if StringIO === body_io and use_tempfile? total then new_io = make_tempfile 'mechanize-raw' new_io.write body_io.string body_io = new_io end body_io.write(part) log.debug("Read #{part.length} bytes (#{total} total)") if log } rescue EOFError => e # terminating CRLF might be missing, let the user check the document raise unless response.chunked? and total.nonzero? body_io.rewind raise Mechanize::ChunkedTerminationError.new(e, response, body_io, uri, @context) rescue Net::HTTP::Persistent::Error, Errno::ECONNRESET => e body_io.rewind raise Mechanize::ResponseReadError.new(e, response, body_io, uri, @context) end body_io.flush body_io.rewind raise Mechanize::ResponseCodeError.new(response, uri) if Net::HTTPUnknownResponse === response content_length = response.content_length unless Net::HTTP::Head === request or Net::HTTPRedirection === response then if content_length and content_length != body_io.length err = EOFError.new("Content-Length (#{content_length}) does not " \ "match response body length (#{body_io.length})") raise Mechanize::ResponseReadError.new(err, response, body_io, uri, @context) end end body_io end |
#response_redirect(response, method, page, redirects, headers, referer = current_page) ⇒ Object
1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 |
# File 'lib/mechanize/http/agent.rb', line 1035 def response_redirect(response, method, page, redirects, headers, referer = current_page) case @redirect_ok when true, :all # shortcut when false, nil return page when :permanent return page unless Net::HTTPMovedPermanently === response end log.info("follow redirect to: #{response['Location']}") if log raise Mechanize::RedirectLimitReachedError.new(page, redirects) if redirects + 1 > @redirection_limit redirect_method = method == :head ? :head : :get new_uri = secure_resolve!(response['Location'].to_s, page) @history.push(page, page.uri) # Make sure we are not copying over the POST headers from the original request POST_HEADERS.each do |key| headers.delete_if { |h| h.casecmp?(key) } end # Make sure we clear credential headers if being redirected to another site if new_uri.host == page.uri.host if new_uri.port != page.uri.port # https://datatracker.ietf.org/doc/html/rfc6265#section-8.5 # cookies are OK to be shared across ports on the same host CREDENTIAL_HEADERS.each { |ch| headers.delete_if { |h| h.casecmp?(ch) } } end else (COOKIE_HEADERS + CREDENTIAL_HEADERS).each { |ch| headers.delete_if { |h| h.casecmp?(ch) } } end fetch new_uri, redirect_method, headers, [], referer, redirects + 1 end |
#retry_change_requests ⇒ Object
Retry non-idempotent requests?
345 346 347 |
# File 'lib/mechanize/http/agent.rb', line 345 def retry_change_requests @http.retry_change_requests end |
#retry_change_requests=(retri) ⇒ Object
Retry non-idempotent requests
351 352 353 |
# File 'lib/mechanize/http/agent.rb', line 351 def retry_change_requests= retri @http.retry_change_requests = retri end |
#robots_allowed?(uri) ⇒ Boolean
Tests if this agent is allowed to access url
, consulting the site’s robots.txt.
1109 1110 1111 1112 1113 |
# File 'lib/mechanize/http/agent.rb', line 1109 def robots_allowed? uri return true if Thread.current[RobotsKey] webrobots.allowed? uri end |
#robots_disallowed?(url) ⇒ Boolean
Opposite of robots_allowed?
1117 1118 1119 |
# File 'lib/mechanize/http/agent.rb', line 1117 def robots_disallowed? url !robots_allowed? url end |
#robots_error(url) ⇒ Object
Returns an error object if there is an error in fetching or parsing robots.txt of the site url
.
1123 1124 1125 |
# File 'lib/mechanize/http/agent.rb', line 1123 def robots_error(url) webrobots.error(url) end |
#robots_error!(url) ⇒ Object
Raises the error if there is an error in fetching or parsing robots.txt of the site url
.
1129 1130 1131 |
# File 'lib/mechanize/http/agent.rb', line 1129 def robots_error!(url) webrobots.error!(url) end |
#robots_reset(url) ⇒ Object
Removes robots.txt cache for the site url
.
1134 1135 1136 |
# File 'lib/mechanize/http/agent.rb', line 1134 def robots_reset(url) webrobots.reset(url) end |
#save_cookies(uri, set_cookie) ⇒ Object
935 936 937 938 939 940 941 942 943 944 945 |
# File 'lib/mechanize/http/agent.rb', line 935 def (uri, ) return [] if .nil? if log = log() # reduce method calls @cookie_jar.parse(, uri, :logger => log) { |c| log.debug("saved cookie: #{c}") true } else @cookie_jar.parse(, uri) end end |
#secure_resolve!(uri, referer = current_page) ⇒ Object
752 753 754 755 756 757 758 759 760 761 762 |
# File 'lib/mechanize/http/agent.rb', line 752 def secure_resolve!(uri, referer = current_page) new_uri = resolve(uri, referer) if (referer_uri = referer && referer.uri) && referer_uri.scheme != 'file'.freeze && new_uri.scheme == 'file'.freeze raise Mechanize::Error, "insecure redirect to a file URI" end new_uri end |
#set_proxy(addr, port = nil, user = nil, pass = nil) ⇒ Object
Sets the proxy address, port, user, and password. addr
may be an HTTP URL/URI or a host name, port
may be a port number, service name or port number string.
1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 |
# File 'lib/mechanize/http/agent.rb', line 1298 def set_proxy addr, port = nil, user = nil, pass = nil case addr when URI::HTTP proxy_uri = addr.dup when %r{\Ahttps?://}i proxy_uri = URI addr when String proxy_uri = URI "http://#{addr}" when nil @http.proxy = nil return end case port when Integer proxy_uri.port = port when nil else begin proxy_uri.port = Socket.getservbyname port rescue SocketError begin proxy_uri.port = Integer port rescue ArgumentError raise ArgumentError, "invalid value for port: #{port.inspect}" end end end proxy_uri.user = user if user proxy_uri.password = pass if pass @http.proxy = proxy_uri end |
#shutdown ⇒ Object
Closes all open connections for this agent.
439 440 441 |
# File 'lib/mechanize/http/agent.rb', line 439 def shutdown http.shutdown end |
#ssl_version ⇒ Object
SSL version to use
1198 1199 1200 |
# File 'lib/mechanize/http/agent.rb', line 1198 def ssl_version @http.ssl_version end |
#ssl_version=(ssl_version) ⇒ Object
Sets the SSL version to use
1203 1204 1205 |
# File 'lib/mechanize/http/agent.rb', line 1203 def ssl_version= ssl_version @http.ssl_version = ssl_version end |
#use_tempfile?(size) ⇒ Boolean
1340 1341 1342 1343 1344 1345 |
# File 'lib/mechanize/http/agent.rb', line 1340 def use_tempfile? size return false unless @max_file_buffer return false unless size size >= @max_file_buffer end |
#verify_callback ⇒ Object
A callback for additional certificate verification. See OpenSSL::SSL::SSLContext#verify_callback
The callback can be used for debugging or to ignore errors by always returning true
. Specifying nil uses the default method that was valid when the SSLContext was created
1213 1214 1215 |
# File 'lib/mechanize/http/agent.rb', line 1213 def verify_callback @http.verify_callback end |
#verify_callback=(verify_callback) ⇒ Object
Sets the certificate verify callback
1218 1219 1220 |
# File 'lib/mechanize/http/agent.rb', line 1218 def verify_callback= verify_callback @http.verify_callback = verify_callback end |
#verify_mode ⇒ Object
How to verify SSL connections. Defaults to VERIFY_PEER
1223 1224 1225 |
# File 'lib/mechanize/http/agent.rb', line 1223 def verify_mode @http.verify_mode end |
#verify_mode=(verify_mode) ⇒ Object
Sets the mode for verifying SSL connections
1228 1229 1230 |
# File 'lib/mechanize/http/agent.rb', line 1228 def verify_mode= verify_mode @http.verify_mode = verify_mode end |
#visited_page(url) ⇒ Object
Returns a visited page for the url passed in, otherwise nil
388 389 390 |
# File 'lib/mechanize/http/agent.rb', line 388 def visited_page url @history.visited_page resolve url end |
#webrobots ⇒ Object
1138 1139 1140 |
# File 'lib/mechanize/http/agent.rb', line 1138 def webrobots @webrobots ||= WebRobots.new(@user_agent, :http_get => method(:get_robots)) end |