Class: Mechanize
- Inherits:
-
Object
- Object
- Mechanize
- Defined in:
- lib/mechanize.rb
Overview
The Mechanize library is used for automating interactions with a website. It can follow links and submit forms. Form fields can be populated and submitted. A history of URLs is maintained and can be queried.
Example
require 'mechanize'
require 'logger'
agent = Mechanize.new
agent.log = Logger.new "mech.log"
agent.user_agent_alias = 'Mac Safari'
page = agent.get "http://www.google.com/"
search_form = page.form_with :name => "f"
search_form.field_with(:name => "q").value = "Hello"
search_results = agent.submit search_form
puts search_results.body
Issues with mechanize
If you think you have a bug with mechanize, but aren’t sure, please file a ticket at github.com/tenderlove/mechanize/issues
Here are some common problems you may experience with mechanize
Problems connecting to SSL sites
Mechanize defaults to validating SSL certificates using the default CA certificates for your platform. At this time, Windows users do not have integration between the OS default CA certificates and OpenSSL. #cert_store explains how to download and use Mozilla’s CA certificates to allow SSL sites to work.
Problems with content-length
Some sites return an incorrect content-length value. Unlike a browser, mechanize raises an error when the content-length header does not match the response length since it does not know if there was a connection problem or if the mismatch is a server bug.
The error raised, Mechanize::ResponseReadError, can be converted to a parsed Page, File, etc. depending upon the content-type:
agent = Mechanize.new
uri = URI 'http://example/invalid_content_length'
begin
page = agent.get uri
rescue Mechanize::ResponseReadError => e
page = e.force_parse
end
Defined Under Namespace
Modules: ElementMatcher, Parser Classes: ChunkedTerminationError, ContentTypeError, Cookie, CookieJar, DirectorySaver, Download, Error, File, FileConnection, FileRequest, FileResponse, FileSaver, Form, HTTP, Headers, History, Image, Page, PluggableParser, RedirectLimitReachedError, RedirectNotGetOrHeadError, ResponseCodeError, ResponseReadError, RobotsDisallowedError, TestCase, UnauthorizedError, UnsupportedSchemeError, Util
Constant Summary collapse
- VERSION =
The version of Mechanize you are using.
'2.5'
- AGENT_ALIASES =
Supported User-Agent aliases for use with user_agent_alias=. The description in parenthesis is for informative purposes and is not part of the alias name.
-
Linux Firefox (3.6.1)
-
Linux Konqueror (3)
-
Linux Mozilla
-
Mac Firefox (3.6)
-
Mac Mozilla
-
Mac Safari (5)
-
Mac Safari 4
-
Mechanize (default)
-
Windows IE 6
-
Windows IE 7
-
Windows IE 8
-
Windows IE 9
-
Windows Mozilla
-
iPhone (3.0)
Example:
agent = Mechanize.new agent.user_agent_alias = 'Mac Safari'
-
{ 'Mechanize' => "Mechanize/#{VERSION} Ruby/#{ruby_version} (http://github.com/tenderlove/mechanize/)", 'Linux Firefox' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.1) Gecko/20100122 firefox/3.6.1', 'Linux Konqueror' => 'Mozilla/5.0 (compatible; Konqueror/3; Linux)', 'Linux Mozilla' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624', 'Mac FireFox' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6', 'Mac Mozilla' => 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030401', 'Mac Safari 4' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; de-at) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10', 'Mac Safari' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22', 'Windows IE 6' => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)', 'Windows IE 7' => 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)', 'Windows IE 8' => 'Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)', 'Windows IE 9' => 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)', 'Windows Mozilla' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6', 'iPhone' => 'Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1C28 Safari/419.3', }
Class Attribute Summary collapse
-
.html_parser ⇒ Object
Default HTML parser for all mechanize instances.
-
.log ⇒ Object
Default logger for all mechanize instances.
Instance Attribute Summary collapse
-
#agent ⇒ Object
readonly
:section: Utilities.
-
#default_encoding ⇒ Object
A default encoding name used when parsing HTML parsing.
-
#force_default_encoding ⇒ Object
Overrides the encodings given by the HTTP server and the HTML page with the default_encoding when set to true.
-
#history_added ⇒ Object
Callback which is invoked with the page that was added to history.
-
#html_parser ⇒ Object
The HTML parser to be used when parsing documents.
-
#keep_alive_time ⇒ Object
HTTP/1.0 keep-alive time.
-
#pluggable_parser ⇒ Object
readonly
The pluggable parser maps a response Content-Type to a parser class.
-
#proxy_addr ⇒ Object
readonly
The HTTP proxy address.
-
#proxy_pass ⇒ Object
readonly
The HTTP proxy password.
-
#proxy_port ⇒ Object
readonly
The HTTP proxy port.
-
#proxy_user ⇒ Object
readonly
The HTTP proxy username.
-
#watch_for_set ⇒ Object
The value of watch_for_set is passed to pluggable parsers for retrieved content.
Class Method Summary collapse
-
.inherited(child) ⇒ Object
:nodoc:.
Instance Method Summary collapse
-
#add_auth(uri, user, password, realm = nil, domain = nil) ⇒ Object
Adds credentials
user
,pass
foruri
. -
#auth(user, password, domain = nil) ⇒ Object
(also: #basic_auth)
NOTE: These credentials will be used as a default for any challenge exposing your password to disclosure to malicious servers.
-
#back ⇒ Object
Equivalent to the browser back button.
-
#ca_file ⇒ Object
Path to an OpenSSL server certificate file.
-
#ca_file=(ca_file) ⇒ Object
Sets the certificate file used for SSL connections.
-
#cert ⇒ Object
An OpenSSL client certificate or the path to a certificate file.
-
#cert=(cert) ⇒ Object
Sets the OpenSSL client certificate
cert
to the given path or certificate instance. -
#cert_store ⇒ Object
An OpenSSL certificate store for verifying server certificates.
-
#cert_store=(cert_store) ⇒ Object
Sets the OpenSSL certificate store to
store
. -
#certificate ⇒ Object
What is this?.
-
#click(link) ⇒ Object
If the parameter is a string, finds the button or link with the value of the string on the current page and clicks it.
-
#conditional_requests ⇒ Object
Are If-Modified-Since conditional requests enabled?.
-
#conditional_requests=(enabled) ⇒ Object
Disables If-Modified-Since conditional requests (enabled by default).
-
#content_encoding_hooks ⇒ Object
A list of hooks to call before reading response header ‘content-encoding’.
-
#cookie_jar ⇒ Object
A Mechanize::CookieJar which stores cookies.
-
#cookie_jar=(cookie_jar) ⇒ Object
Replaces the cookie jar with
cookie_jar
. -
#cookies ⇒ Object
Returns a list of cookies stored in the cookie jar.
-
#current_page ⇒ Object
(also: #page)
Returns the latest page loaded by Mechanize.
-
#delete(uri, query_params = {}, headers = {}) ⇒ Object
DELETE
uri
withquery_params
, and settingheaders
:. -
#download(uri, io_or_filename, parameters = [], referer = nil, headers = {}) ⇒ Object
GETs
uri
and writes it toio_or_filename
without recording the request in the history. -
#follow_meta_refresh ⇒ Object
Follow HTML meta refresh and HTTP Refresh headers.
-
#follow_meta_refresh=(follow) ⇒ Object
Controls following of HTML meta refresh and HTTP Refresh headers in responses.
-
#follow_meta_refresh_self ⇒ Object
Follow an HTML meta refresh and HTTP Refresh headers that have no “url=” in the content attribute.
-
#follow_meta_refresh_self=(follow) ⇒ Object
Alters the following of HTML meta refresh and HTTP Refresh headers that point to the same page.
-
#get(uri, parameters = [], referer = nil, headers = {}) {|page| ... } ⇒ Object
GET the
uri
with the given requestparameters
,referer
andheaders
. -
#get_file(url) ⇒ Object
GET
url
and return only its contents. -
#gzip_enabled ⇒ Object
Is gzip compression of responses enabled?.
-
#gzip_enabled=(enabled) ⇒ Object
Disables HTTP/1.1 gzip compression (enabled by default).
-
#head(uri, query_params = {}, headers = {}) {|page| ... } ⇒ Object
HEAD
uri
withquery_params
andheaders
:. -
#history ⇒ Object
The history of this mechanize run.
-
#idle_timeout ⇒ Object
Connections that have not been used in this many seconds will be reset.
-
#idle_timeout=(idle_timeout) ⇒ Object
Sets the idle timeout to
idle_timeout
. -
#ignore_bad_chunking ⇒ Object
When set to true mechanize will ignore an EOF during chunked transfer encoding so long as at least one byte was received.
-
#ignore_bad_chunking=(ignore_bad_chunking) ⇒ Object
When set to true mechanize will ignore an EOF during chunked transfer encoding.
-
#initialize {|_self| ... } ⇒ Mechanize
constructor
Creates a new mechanize instance.
-
#keep_alive ⇒ Object
Are HTTP/1.1 keep-alive connections enabled?.
-
#keep_alive=(enable) ⇒ Object
Disable HTTP/1.1 keep-alive connections if
enable
is set to false. -
#key ⇒ Object
An OpenSSL private key or the path to a private key.
-
#key=(key) ⇒ Object
Sets the OpenSSL client
key
to the given path or key instance. -
#log ⇒ Object
The current logger.
-
#log=(logger) ⇒ Object
Sets the
logger
used by this instance of mechanize. -
#max_file_buffer ⇒ Object
Responses larger than this will be written to a Tempfile instead of stored in memory.
-
#max_file_buffer=(bytes) ⇒ Object
Sets the maximum size of a response body that will be stored in memory to
bytes
. -
#max_history ⇒ Object
Maximum number of items allowed in the history.
-
#max_history=(length) ⇒ Object
Sets the maximum number of items allowed in the history to
length
. -
#open_timeout ⇒ Object
Length of time to wait until a connection is opened in seconds.
-
#open_timeout=(open_timeout) ⇒ Object
Sets the connection open timeout to
open_timeout
. -
#parse(uri, response, body) ⇒ Object
Parses the
body
of theresponse
fromuri
using the pluggable parser that matches its content type. -
#pass ⇒ Object
OpenSSL client key password.
-
#pass=(pass) ⇒ Object
Sets the client key password to
pass
. -
#post(uri, query = {}, headers = {}) ⇒ Object
POST to the given
uri
with the givenquery
. -
#post_connect_hooks ⇒ Object
A list of hooks to call after retrieving a response.
-
#pre_connect_hooks ⇒ Object
A list of hooks to call before retrieving a response.
-
#pretty_print(q) ⇒ Object
:nodoc:.
-
#put(uri, entity, headers = {}) ⇒ Object
PUT to
uri
withentity
, and settingheaders
:. -
#read_timeout ⇒ Object
Length of time to wait for data from the server.
-
#read_timeout=(read_timeout) ⇒ Object
Sets the timeout for each chunk of data read from the server to
read_timeout
. -
#redirect_ok ⇒ Object
(also: #follow_redirect?)
Controls how mechanize deals with redirects.
-
#redirect_ok=(follow) ⇒ Object
Sets the mechanize redirect handling policy.
-
#redirection_limit ⇒ Object
Maximum number of redirections to follow.
-
#redirection_limit=(limit) ⇒ Object
Sets the maximum number of redirections to follow to
limit
. -
#request_headers ⇒ Object
A hash of custom request headers that will be sent on every request.
-
#request_headers=(request_headers) ⇒ Object
Replaces the custom request headers that will be sent on every request with
request_headers
. -
#request_with_entity(verb, uri, entity, headers = {}) ⇒ Object
Makes an HTTP request to
url
using HTTP methodverb
. -
#retry_change_requests ⇒ Object
Retry POST and other non-idempotent requests.
-
#retry_change_requests=(retry_change_requests) ⇒ Object
When setting
retry_change_requests
to true you are stating that, for all the URLs you access with mechanize, making POST and other non-idempotent requests is safe and will not cause data duplication or other harmful results. -
#robots ⇒ Object
Will
/robots.txt
files be obeyed?. -
#robots=(enabled) ⇒ Object
When
enabled
mechanize will retrieve and obeyrobots.txt
files. -
#scheme_handlers ⇒ Object
The handlers for HTTP and other URI protocols.
-
#scheme_handlers=(scheme_handlers) ⇒ Object
Replaces the URI scheme handler table with
scheme_handlers
. -
#set_proxy(address, port, user = nil, password = nil) ⇒ Object
Sets the proxy
address
atport
with an optionaluser
andpassword
. -
#ssl_version ⇒ Object
SSL version to use.
-
#ssl_version=(ssl_version) ⇒ Object
Sets the SSL version to use to
version
without client/server negotiation. -
#submit(form, button = nil, headers = {}) ⇒ Object
Submits
form
with an optionalbutton
. -
#transact ⇒ Object
Runs given block, then resets the page history as it was before.
-
#user_agent ⇒ Object
The identification string for the client initiating a web request.
-
#user_agent=(user_agent) ⇒ Object
Sets the User-Agent used by mechanize to
user_agent
. -
#user_agent_alias=(name) ⇒ Object
Set the user agent for the Mechanize object based on the given
name
. -
#verify_callback ⇒ Object
A callback for additional certificate verification.
-
#verify_callback=(verify_callback) ⇒ Object
Sets the OpenSSL certificate verification callback.
-
#verify_mode ⇒ Object
the OpenSSL server certificate verification method.
-
#verify_mode=(verify_mode) ⇒ Object
Sets the OpenSSL server certificate verification method.
-
#visited?(url) ⇒ Boolean
(also: #visited_page)
Returns a visited page for the
url
passed in, otherwise nil.
Constructor Details
#initialize {|_self| ... } ⇒ Mechanize
Creates a new mechanize instance. If a block is given, the created instance is yielded to the block for setting up pre-connection state such as SSL parameters or proxies:
agent = Mechanize.new do |a|
a.proxy_host = 'proxy.example'
a.proxy_port = 8080
end
148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 |
# File 'lib/mechanize.rb', line 148 def initialize @agent = Mechanize::HTTP::Agent.new @agent.context = self @log = nil # attr_accessors @agent.user_agent = AGENT_ALIASES['Mechanize'] @watch_for_set = nil @history_added = nil # attr_readers @pluggable_parser = PluggableParser.new @keep_alive_time = 0 # Proxy @proxy_addr = nil @proxy_port = nil @proxy_user = nil @proxy_pass = nil @html_parser = self.class.html_parser @default_encoding = nil @force_default_encoding = false # defaults @agent.max_history = 50 yield self if block_given? @agent.set_proxy @proxy_addr, @proxy_port, @proxy_user, @proxy_pass end |
Class Attribute Details
.html_parser ⇒ Object
Default HTML parser for all mechanize instances
Mechanize.html_parser = Nokogiri::XML
553 554 555 |
# File 'lib/mechanize.rb', line 553 def html_parser @html_parser end |
.log ⇒ Object
Default logger for all mechanize instances
Mechanize.log = Logger.new $stderr
560 561 562 |
# File 'lib/mechanize.rb', line 560 def log @log end |
Instance Attribute Details
#agent ⇒ Object (readonly)
:section: Utilities
1156 1157 1158 |
# File 'lib/mechanize.rb', line 1156 def agent @agent end |
#default_encoding ⇒ Object
A default encoding name used when parsing HTML parsing. When set it is used after any other encoding. The default is nil.
568 569 570 |
# File 'lib/mechanize.rb', line 568 def default_encoding @default_encoding end |
#force_default_encoding ⇒ Object
Overrides the encodings given by the HTTP server and the HTML page with the default_encoding when set to true.
574 575 576 |
# File 'lib/mechanize.rb', line 574 def force_default_encoding @force_default_encoding end |
#history_added ⇒ Object
Callback which is invoked with the page that was added to history.
263 264 265 |
# File 'lib/mechanize.rb', line 263 def history_added @history_added end |
#html_parser ⇒ Object
The HTML parser to be used when parsing documents
579 580 581 |
# File 'lib/mechanize.rb', line 579 def html_parser @html_parser end |
#keep_alive_time ⇒ Object
HTTP/1.0 keep-alive time. This is no longer supported by mechanize as it now uses net-http-persistent which only supports HTTP/1.1 persistent connections
586 587 588 |
# File 'lib/mechanize.rb', line 586 def keep_alive_time @keep_alive_time end |
#pluggable_parser ⇒ Object (readonly)
The pluggable parser maps a response Content-Type to a parser class. The registered Content-Type may be either a full content type like ‘image/png’ or a media type ‘text’. See Mechanize::PluggableParser for further details.
Example:
agent.pluggable_parser['application/octet-stream'] = Mechanize::Download
598 599 600 |
# File 'lib/mechanize.rb', line 598 def pluggable_parser @pluggable_parser end |
#proxy_addr ⇒ Object (readonly)
The HTTP proxy address
603 604 605 |
# File 'lib/mechanize.rb', line 603 def proxy_addr @proxy_addr end |
#proxy_pass ⇒ Object (readonly)
The HTTP proxy password
608 609 610 |
# File 'lib/mechanize.rb', line 608 def proxy_pass @proxy_pass end |
#proxy_port ⇒ Object (readonly)
The HTTP proxy port
613 614 615 |
# File 'lib/mechanize.rb', line 613 def proxy_port @proxy_port end |
#proxy_user ⇒ Object (readonly)
The HTTP proxy username
618 619 620 |
# File 'lib/mechanize.rb', line 618 def proxy_user @proxy_user end |
#watch_for_set ⇒ Object
The value of watch_for_set is passed to pluggable parsers for retrieved content
995 996 997 |
# File 'lib/mechanize.rb', line 995 def watch_for_set @watch_for_set end |
Class Method Details
.inherited(child) ⇒ Object
:nodoc:
132 133 134 135 136 |
# File 'lib/mechanize.rb', line 132 def self.inherited(child) # :nodoc: child.html_parser ||= html_parser child.log ||= log super end |
Instance Method Details
#add_auth(uri, user, password, realm = nil, domain = nil) ⇒ Object
Adds credentials user
, pass
for uri
. If realm
is set the credentials are used only for that realm. If realm
is not set the credentials become the default for any realm on that URI.
domain
and realm
are exclusive as NTLM does not follow RFC 2617. If domain
is given it is only used for NTLM authentication.
653 654 655 |
# File 'lib/mechanize.rb', line 653 def add_auth uri, user, password, realm = nil, domain = nil @agent.add_auth uri, user, password, realm, domain end |
#auth(user, password, domain = nil) ⇒ Object Also known as: basic_auth
NOTE: These credentials will be used as a default for any challenge exposing your password to disclosure to malicious servers. Use of this method will warn. This method is deprecated and will be removed in mechanize 3.
Sets the user
and password
as the default credentials to be used for HTTP authentication for any server. The domain
is used for NTLM authentication.
630 631 632 633 634 635 636 637 638 639 640 641 |
# File 'lib/mechanize.rb', line 630 def auth user, password, domain = nil caller.first =~ /(.*?):(\d+).*?$/ warn <<-WARNING At #{$1} line #{$2} Use of #auth and #basic_auth are deprecated due to a security vulnerability. WARNING @agent.add_default_auth user, password, domain end |
#back ⇒ Object
Equivalent to the browser back button. Returns the previous page visited.
189 190 191 |
# File 'lib/mechanize.rb', line 189 def back @agent.history.pop end |
#ca_file ⇒ Object
Path to an OpenSSL server certificate file
1005 1006 1007 |
# File 'lib/mechanize.rb', line 1005 def ca_file @agent.ca_file end |
#ca_file=(ca_file) ⇒ Object
Sets the certificate file used for SSL connections
1012 1013 1014 |
# File 'lib/mechanize.rb', line 1012 def ca_file= ca_file @agent.ca_file = ca_file end |
#cert ⇒ Object
An OpenSSL client certificate or the path to a certificate file.
1019 1020 1021 |
# File 'lib/mechanize.rb', line 1019 def cert @agent.cert end |
#cert=(cert) ⇒ Object
Sets the OpenSSL client certificate cert
to the given path or certificate instance
1027 1028 1029 |
# File 'lib/mechanize.rb', line 1027 def cert= cert @agent.certificate = cert end |
#cert_store ⇒ Object
An OpenSSL certificate store for verifying server certificates. This defaults to the default certificate store for your system.
If your system does not ship with a default set of certificates you can retrieve a copy of the set from Mozilla here: curl.haxx.se/docs/caextract.html
(Note that this set does not have an HTTPS download option so you may wish to use the firefox-db2pem.sh script to extract the certificates from a local install to avoid man-in-the-middle attacks.)
After downloading or generating a cacert.pem from the above link you can create a certificate store from the pem file like this:
cert_store = OpenSSL::X509::Store.new
cert_store.add_file 'cacert.pem'
And have mechanize use it with:
agent.cert_store = cert_store
1053 1054 1055 |
# File 'lib/mechanize.rb', line 1053 def cert_store @agent.cert_store end |
#cert_store=(cert_store) ⇒ Object
Sets the OpenSSL certificate store to store
.
See also #cert_store
1062 1063 1064 |
# File 'lib/mechanize.rb', line 1062 def cert_store= cert_store @agent.cert_store = cert_store end |
#certificate ⇒ Object
What is this?
Why is it different from #cert?
1071 1072 1073 |
# File 'lib/mechanize.rb', line 1071 def certificate # :nodoc: @agent.certificate end |
#click(link) ⇒ Object
If the parameter is a string, finds the button or link with the value of the string on the current page and clicks it. Otherwise, clicks the Mechanize::Page::Link object passed in. Returns the page fetched.
290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 |
# File 'lib/mechanize.rb', line 290 def click link case link when Page::Link then referer = link.page || current_page() if @agent.robots if (referer.is_a?(Page) and referer.parser.nofollow?) or link.rel?('nofollow') then raise RobotsDisallowedError.new(link.href) end end if link.noreferrer? href = @agent.resolve(link.href, link.page || current_page) referer = Page.new else href = link.href end get href, [], referer when String, Regexp then if real_link = page.link_with(:text => link) click real_link else = nil form = page.forms.find do |f| = f.(:value => link) .is_a? Form::Submit end submit form, if form end else referer = current_page() href = link.respond_to?(:href) ? link.href : (link['href'] || link['src']) get href, [], referer end end |
#conditional_requests ⇒ Object
Are If-Modified-Since conditional requests enabled?
660 661 662 |
# File 'lib/mechanize.rb', line 660 def conditional_requests @agent.conditional_requests end |
#conditional_requests=(enabled) ⇒ Object
Disables If-Modified-Since conditional requests (enabled by default)
667 668 669 |
# File 'lib/mechanize.rb', line 667 def conditional_requests= enabled @agent.conditional_requests = enabled end |
#content_encoding_hooks ⇒ Object
A list of hooks to call before reading response header ‘content-encoding’.
The hook is called with the agent making the request, the URI of the request, the response an IO containing the response body.
256 257 258 |
# File 'lib/mechanize.rb', line 256 def content_encoding_hooks @agent.content_encoding_hooks end |
#cookie_jar ⇒ Object
A Mechanize::CookieJar which stores cookies
674 675 676 |
# File 'lib/mechanize.rb', line 674 def @agent. end |
#cookie_jar=(cookie_jar) ⇒ Object
Replaces the cookie jar with cookie_jar
681 682 683 |
# File 'lib/mechanize.rb', line 681 def @agent. = end |
#cookies ⇒ Object
Returns a list of cookies stored in the cookie jar.
688 689 690 |
# File 'lib/mechanize.rb', line 688 def @agent..to_a end |
#current_page ⇒ Object Also known as: page
Returns the latest page loaded by Mechanize
196 197 198 |
# File 'lib/mechanize.rb', line 196 def current_page @agent.current_page end |
#delete(uri, query_params = {}, headers = {}) ⇒ Object
DELETE uri
with query_params
, and setting headers
:
delete('http://example/', {'q' => 'foo'}, {})
372 373 374 375 376 |
# File 'lib/mechanize.rb', line 372 def delete(uri, query_params = {}, headers = {}) page = @agent.fetch(uri, :delete, headers, query_params) add_to_history(page) page end |
#download(uri, io_or_filename, parameters = [], referer = nil, headers = {}) ⇒ Object
GETs uri
and writes it to io_or_filename
without recording the request in the history. If io_or_filename
does not respond to #write it will be used as a file name. parameters
, referer
and headers
are used as in #get.
By default, if the Content-type of the response matches a Mechanize::File or Mechanize::Page parser, the response body will be loaded into memory before being saved. See #pluggable_parser for details on changing this default.
For alternate ways of downloading files see Mechanize::FileSaver and Mechanize::DirectorySaver.
340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 |
# File 'lib/mechanize.rb', line 340 def download uri, io_or_filename, parameters = [], referer = nil, headers = {} page = transact do get uri, parameters, referer, headers end io = if io_or_filename.respond_to? :write then io_or_filename else open io_or_filename, 'wb' end case page when Mechanize::File then io.write page.body else body_io = page.body_io until body_io.eof? do io.write body_io.read 16384 end end page ensure io.close if io and not io_or_filename.respond_to? :write end |
#follow_meta_refresh ⇒ Object
Follow HTML meta refresh and HTTP Refresh headers. If set to :anywhere
meta refresh tags outside of the head element will be followed.
696 697 698 |
# File 'lib/mechanize.rb', line 696 def @agent. end |
#follow_meta_refresh=(follow) ⇒ Object
Controls following of HTML meta refresh and HTTP Refresh headers in responses.
704 705 706 |
# File 'lib/mechanize.rb', line 704 def follow @agent. = follow end |
#follow_meta_refresh_self ⇒ Object
Follow an HTML meta refresh and HTTP Refresh headers that have no “url=” in the content attribute.
Defaults to false to prevent infinite refresh loops.
714 715 716 |
# File 'lib/mechanize.rb', line 714 def @agent. end |
#follow_meta_refresh_self=(follow) ⇒ Object
Alters the following of HTML meta refresh and HTTP Refresh headers that point to the same page.
722 723 724 |
# File 'lib/mechanize.rb', line 722 def follow @agent. = follow end |
#get(uri, parameters = [], referer = nil, headers = {}) {|page| ... } ⇒ Object
GET the uri
with the given request parameters
, referer
and headers
.
The referer
may be a URI or a page.
384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 |
# File 'lib/mechanize.rb', line 384 def get(uri, parameters = [], referer = nil, headers = {}) method = :get referer ||= if uri.to_s =~ %r{\Ahttps?://} Page.new else current_page || Page.new end # FIXME: Huge hack so that using a URI as a referer works. I need to # refactor everything to pass around URIs but still support # Mechanize::Page#base unless Mechanize::Parser === referer then referer = if referer.is_a?(String) then Page.new URI(referer) else Page.new referer end end # fetch the page headers ||= {} page = @agent.fetch uri, method, headers, parameters, referer add_to_history(page) yield page if block_given? page end |
#get_file(url) ⇒ Object
GET url
and return only its contents
416 417 418 |
# File 'lib/mechanize.rb', line 416 def get_file(url) get(url).body end |
#gzip_enabled ⇒ Object
Is gzip compression of responses enabled?
729 730 731 |
# File 'lib/mechanize.rb', line 729 def gzip_enabled @agent.gzip_enabled end |
#gzip_enabled=(enabled) ⇒ Object
Disables HTTP/1.1 gzip compression (enabled by default)
736 737 738 |
# File 'lib/mechanize.rb', line 736 def gzip_enabled=enabled @agent.gzip_enabled = enabled end |
#head(uri, query_params = {}, headers = {}) {|page| ... } ⇒ Object
HEAD uri
with query_params
and headers
:
head('http://example/', {'q' => 'foo'}, {})
425 426 427 428 429 430 431 |
# File 'lib/mechanize.rb', line 425 def head(uri, query_params = {}, headers = {}) page = @agent.fetch uri, :head, headers, query_params yield page if block_given? page end |
#history ⇒ Object
The history of this mechanize run
205 206 207 |
# File 'lib/mechanize.rb', line 205 def history @agent.history end |
#idle_timeout ⇒ Object
Connections that have not been used in this many seconds will be reset.
743 744 745 |
# File 'lib/mechanize.rb', line 743 def idle_timeout @agent.idle_timeout end |
#idle_timeout=(idle_timeout) ⇒ Object
Sets the idle timeout to idle_timeout
. The default timeout is 5 seconds. If you experience “too many connection resets”, reducing this value may help.
751 752 753 |
# File 'lib/mechanize.rb', line 751 def idle_timeout= idle_timeout @agent.idle_timeout = idle_timeout end |
#ignore_bad_chunking ⇒ Object
When set to true mechanize will ignore an EOF during chunked transfer encoding so long as at least one byte was received. Be careful when enabling this as it may cause data loss.
Net::HTTP does not inform mechanize of where in the chunked stream the EOF occurred. Usually it is after the last-chunk but before the terminating CRLF (invalid termination) but it may occur earlier. In the second case your response body may be incomplete.
765 766 767 |
# File 'lib/mechanize.rb', line 765 def ignore_bad_chunking @agent.ignore_bad_chunking end |
#ignore_bad_chunking=(ignore_bad_chunking) ⇒ Object
When set to true mechanize will ignore an EOF during chunked transfer encoding. See ignore_bad_chunking for further details
773 774 775 |
# File 'lib/mechanize.rb', line 773 def ignore_bad_chunking= ignore_bad_chunking @agent.ignore_bad_chunking = ignore_bad_chunking end |
#keep_alive ⇒ Object
Are HTTP/1.1 keep-alive connections enabled?
780 781 782 |
# File 'lib/mechanize.rb', line 780 def keep_alive @agent.keep_alive end |
#keep_alive=(enable) ⇒ Object
Disable HTTP/1.1 keep-alive connections if enable
is set to false. If you are experiencing “too many connection resets” errors setting this to false will eliminate them.
You should first investigate reducing idle_timeout.
791 792 793 |
# File 'lib/mechanize.rb', line 791 def keep_alive= enable @agent.keep_alive = enable end |
#key ⇒ Object
An OpenSSL private key or the path to a private key
1078 1079 1080 |
# File 'lib/mechanize.rb', line 1078 def key @agent.key end |
#key=(key) ⇒ Object
Sets the OpenSSL client key
to the given path or key instance. If a path is given, the path must contain an RSA key file.
1086 1087 1088 |
# File 'lib/mechanize.rb', line 1086 def key= key @agent.private_key = key end |
#log ⇒ Object
The current logger. If no logger has been set Mechanize.log is used.
798 799 800 |
# File 'lib/mechanize.rb', line 798 def log @log || Mechanize.log end |
#log=(logger) ⇒ Object
Sets the logger
used by this instance of mechanize
805 806 807 |
# File 'lib/mechanize.rb', line 805 def log= logger @log = logger end |
#max_file_buffer ⇒ Object
Responses larger than this will be written to a Tempfile instead of stored in memory. The default is 100,000 bytes.
A value of nil disables creation of Tempfiles.
815 816 817 |
# File 'lib/mechanize.rb', line 815 def max_file_buffer @agent.max_file_buffer end |
#max_file_buffer=(bytes) ⇒ Object
Sets the maximum size of a response body that will be stored in memory to bytes
. A value of nil causes all response bodies to be stored in memory.
Note that for Mechanize::Download subclasses, the maximum buffer size multiplied by the number of pages stored in history (controlled by #max_history) is an approximate upper limit on the amount of memory Mechanize will use. By default, Mechanize can use up to ~5MB to store response bodies for non-File and non-Page (HTML) responses.
See also the discussion under #max_history=
832 833 834 |
# File 'lib/mechanize.rb', line 832 def max_file_buffer= bytes @agent.max_file_buffer = bytes end |
#max_history ⇒ Object
Maximum number of items allowed in the history. The default setting is 50 pages. Note that the size of the history multiplied by the maximum response body size
214 215 216 |
# File 'lib/mechanize.rb', line 214 def max_history @agent.history.max_size end |
#max_history=(length) ⇒ Object
Sets the maximum number of items allowed in the history to length
.
Setting the maximum history length to nil will make the history size unlimited. Take care when doing this, mechanize stores response bodies in memory for pages and in the temporary files directory for other responses. For a long-running mechanize program this can be quite large.
See also the discussion under #max_file_buffer=
228 229 230 |
# File 'lib/mechanize.rb', line 228 def max_history= length @agent.history.max_size = length end |
#open_timeout ⇒ Object
Length of time to wait until a connection is opened in seconds
839 840 841 |
# File 'lib/mechanize.rb', line 839 def open_timeout @agent.open_timeout end |
#open_timeout=(open_timeout) ⇒ Object
Sets the connection open timeout to open_timeout
846 847 848 |
# File 'lib/mechanize.rb', line 846 def open_timeout= open_timeout @agent.open_timeout = open_timeout end |
#parse(uri, response, body) ⇒ Object
Parses the body
of the response
from uri
using the pluggable parser that matches its content type
1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 |
# File 'lib/mechanize.rb', line 1162 def parse uri, response, body content_type = nil unless response['Content-Type'].nil? data, = response['Content-Type'].split ';', 2 content_type, = data.downcase.split ',', 2 unless data.nil? end parser_klass = @pluggable_parser.parser content_type unless parser_klass <= Mechanize::Download then body = case body when IO, Tempfile, StringIO then body.read else body end end parser_klass.new uri, response, body, response.code do |parser| parser.mech = self if parser.respond_to? :mech= parser.watch_for_set = @watch_for_set if @watch_for_set and parser.respond_to?(:watch_for_set=) end end |
#pass ⇒ Object
OpenSSL client key password
1093 1094 1095 |
# File 'lib/mechanize.rb', line 1093 def pass @agent.pass end |
#pass=(pass) ⇒ Object
Sets the client key password to pass
1100 1101 1102 |
# File 'lib/mechanize.rb', line 1100 def pass= pass @agent.pass = pass end |
#post(uri, query = {}, headers = {}) ⇒ Object
POST to the given uri
with the given query
. The query is specified by either a string, or a list of key-value pairs represented by a hash or an array of arrays.
Examples:
agent.post 'http://example.com/', "foo" => "bar"
agent.post 'http://example.com/', [%w[foo bar]]
agent.post('http://example.com/', "<message>hello</message>",
'Content-Type' => 'application/xml')
446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 |
# File 'lib/mechanize.rb', line 446 def post(uri, query={}, headers={}) return request_with_entity(:post, uri, query, headers) if String === query node = {} # Create a fake form class << node def search(*args); []; end end node['method'] = 'POST' node['enctype'] = 'application/x-www-form-urlencoded' form = Form.new(node) query.each { |k, v| if v.is_a?(IO) form.enctype = 'multipart/form-data' ul = Form::FileUpload.new({'name' => k.to_s},::File.basename(v.path)) ul.file_data = v.read form.file_uploads << ul else form.fields << Form::Field.new({'name' => k.to_s},v) end } post_form(uri, form, headers) end |
#post_connect_hooks ⇒ Object
A list of hooks to call after retrieving a response. Hooks are called with the agent, the URI, the response, and the response body.
269 270 271 |
# File 'lib/mechanize.rb', line 269 def post_connect_hooks @agent.post_connect_hooks end |
#pre_connect_hooks ⇒ Object
A list of hooks to call before retrieving a response. Hooks are called with the agent, the URI, the response, and the response body.
277 278 279 |
# File 'lib/mechanize.rb', line 277 def pre_connect_hooks @agent.pre_connect_hooks end |
#pretty_print(q) ⇒ Object
:nodoc:
1189 1190 1191 1192 1193 1194 1195 1196 |
# File 'lib/mechanize.rb', line 1189 def pretty_print(q) # :nodoc: q.object_group(self) { q.breakable q.pp q.breakable q.pp current_page } end |
#put(uri, entity, headers = {}) ⇒ Object
PUT to uri
with entity
, and setting headers
:
put('http://example/', 'new content', {'Content-Type' => 'text/plain'})
477 478 479 |
# File 'lib/mechanize.rb', line 477 def put(uri, entity, headers = {}) request_with_entity(:put, uri, entity, headers) end |
#read_timeout ⇒ Object
Length of time to wait for data from the server
853 854 855 |
# File 'lib/mechanize.rb', line 853 def read_timeout @agent.read_timeout end |
#read_timeout=(read_timeout) ⇒ Object
Sets the timeout for each chunk of data read from the server to read_timeout
. A single request may read many chunks of data.
861 862 863 |
# File 'lib/mechanize.rb', line 861 def read_timeout= read_timeout @agent.read_timeout = read_timeout end |
#redirect_ok ⇒ Object Also known as: follow_redirect?
Controls how mechanize deals with redirects. The following values are allowed:
- :all, true
-
All 3xx redirects are followed (default)
- :permanent
-
Only 301 Moved Permanantly redirects are followed
- false
-
No redirects are followed
873 874 875 |
# File 'lib/mechanize.rb', line 873 def redirect_ok @agent.redirect_ok end |
#redirect_ok=(follow) ⇒ Object
Sets the mechanize redirect handling policy. See redirect_ok for allowed values
883 884 885 |
# File 'lib/mechanize.rb', line 883 def redirect_ok= follow @agent.redirect_ok = follow end |
#redirection_limit ⇒ Object
Maximum number of redirections to follow
890 891 892 |
# File 'lib/mechanize.rb', line 890 def redirection_limit @agent.redirection_limit end |
#redirection_limit=(limit) ⇒ Object
Sets the maximum number of redirections to follow to limit
897 898 899 |
# File 'lib/mechanize.rb', line 897 def redirection_limit= limit @agent.redirection_limit = limit end |
#request_headers ⇒ Object
A hash of custom request headers that will be sent on every request
904 905 906 |
# File 'lib/mechanize.rb', line 904 def request_headers @agent.request_headers end |
#request_headers=(request_headers) ⇒ Object
Replaces the custom request headers that will be sent on every request with request_headers
912 913 914 |
# File 'lib/mechanize.rb', line 912 def request_headers= request_headers @agent.request_headers = request_headers end |
#request_with_entity(verb, uri, entity, headers = {}) ⇒ Object
Makes an HTTP request to url
using HTTP method verb
. entity
is used as the request body, if allowed.
485 486 487 488 489 490 491 492 493 494 495 496 |
# File 'lib/mechanize.rb', line 485 def request_with_entity(verb, uri, entity, headers = {}) cur_page = current_page || Page.new headers = { 'Content-Type' => 'application/octet-stream', 'Content-Length' => entity.size.to_s, }.update headers page = @agent.fetch uri, verb, headers, [entity], cur_page add_to_history(page) page end |
#retry_change_requests ⇒ Object
Retry POST and other non-idempotent requests. See RFC 2616 9.1.2.
919 920 921 |
# File 'lib/mechanize.rb', line 919 def retry_change_requests @agent.retry_change_requests end |
#retry_change_requests=(retry_change_requests) ⇒ Object
When setting retry_change_requests
to true you are stating that, for all the URLs you access with mechanize, making POST and other non-idempotent requests is safe and will not cause data duplication or other harmful results.
If you are experiencing “too many connection resets” errors you should instead investigate reducing the idle_timeout or disabling keep_alive connections.
933 934 935 |
# File 'lib/mechanize.rb', line 933 def retry_change_requests= retry_change_requests @agent.retry_change_requests = retry_change_requests end |
#robots ⇒ Object
Will /robots.txt
files be obeyed?
940 941 942 |
# File 'lib/mechanize.rb', line 940 def robots @agent.robots end |
#robots=(enabled) ⇒ Object
When enabled
mechanize will retrieve and obey robots.txt
files
948 949 950 |
# File 'lib/mechanize.rb', line 948 def robots= enabled @agent.robots = enabled end |
#scheme_handlers ⇒ Object
The handlers for HTTP and other URI protocols.
955 956 957 |
# File 'lib/mechanize.rb', line 955 def scheme_handlers @agent.scheme_handlers end |
#scheme_handlers=(scheme_handlers) ⇒ Object
Replaces the URI scheme handler table with scheme_handlers
962 963 964 |
# File 'lib/mechanize.rb', line 962 def scheme_handlers= scheme_handlers @agent.scheme_handlers = scheme_handlers end |
#set_proxy(address, port, user = nil, password = nil) ⇒ Object
Sets the proxy address
at port
with an optional user
and password
1201 1202 1203 1204 1205 1206 1207 1208 |
# File 'lib/mechanize.rb', line 1201 def set_proxy address, port, user = nil, password = nil @proxy_addr = address @proxy_port = port @proxy_user = user @proxy_pass = password @agent.set_proxy address, port, user, password end |
#ssl_version ⇒ Object
SSL version to use. Ruby 1.9 and newer only.
1107 1108 1109 |
# File 'lib/mechanize.rb', line 1107 def ssl_version @agent.ssl_version end |
#ssl_version=(ssl_version) ⇒ Object
Sets the SSL version to use to version
without client/server negotiation. Ruby 1.9 and newer only.
1115 1116 1117 |
# File 'lib/mechanize.rb', line 1115 def ssl_version= ssl_version @agent.ssl_version = ssl_version end |
#submit(form, button = nil, headers = {}) ⇒ Object
Submits form
with an optional button
.
Without a button:
page = agent.get('http://example.com')
agent.submit(page.forms.first)
With a button:
agent.submit(page.forms.first, page.forms.first..first)
510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 |
# File 'lib/mechanize.rb', line 510 def submit(form, =nil, headers={}) form.() if case form.method.upcase when 'POST' post_form(form.action, form, headers) when 'GET' get(form.action.gsub(/\?[^\?]*$/, ''), form.build_query, form.page, headers) else raise ArgumentError, "unsupported method: #{form.method.upcase}" end end |
#transact ⇒ Object
Runs given block, then resets the page history as it was before. self is given as a parameter to the block. Returns the value of the block.
530 531 532 533 534 535 536 537 |
# File 'lib/mechanize.rb', line 530 def transact history_backup = @agent.history.dup begin yield self ensure @agent.history = history_backup end end |
#user_agent ⇒ Object
The identification string for the client initiating a web request
969 970 971 |
# File 'lib/mechanize.rb', line 969 def user_agent @agent.user_agent end |
#user_agent=(user_agent) ⇒ Object
Sets the User-Agent used by mechanize to user_agent
. See also user_agent_alias
977 978 979 |
# File 'lib/mechanize.rb', line 977 def user_agent= user_agent @agent.user_agent = user_agent end |
#user_agent_alias=(name) ⇒ Object
Set the user agent for the Mechanize object based on the given name
.
See also AGENT_ALIASES
986 987 988 989 |
# File 'lib/mechanize.rb', line 986 def user_agent_alias= name self.user_agent = AGENT_ALIASES[name] || raise(ArgumentError, "unknown agent alias #{name.inspect}") end |
#verify_callback ⇒ Object
A callback for additional certificate verification. See OpenSSL::SSL::SSLContext#verify_callback
The callback can be used for debugging or to ignore errors by always returning true
. Specifying nil uses the default method that was valid when the SSLContext was created
1127 1128 1129 |
# File 'lib/mechanize.rb', line 1127 def verify_callback @agent.verify_callback end |
#verify_callback=(verify_callback) ⇒ Object
Sets the OpenSSL certificate verification callback
1134 1135 1136 |
# File 'lib/mechanize.rb', line 1134 def verify_callback= verify_callback @agent.verify_callback = verify_callback end |
#verify_mode ⇒ Object
the OpenSSL server certificate verification method. The default is OpenSSL::SSL::VERIFY_PEER and certificate verification uses the default system certificates. See also cert_store
1143 1144 1145 |
# File 'lib/mechanize.rb', line 1143 def verify_mode @agent.verify_mode end |
#verify_mode=(verify_mode) ⇒ Object
Sets the OpenSSL server certificate verification method.
1150 1151 1152 |
# File 'lib/mechanize.rb', line 1150 def verify_mode= verify_mode @agent.verify_mode = verify_mode end |
#visited?(url) ⇒ Boolean Also known as: visited_page
Returns a visited page for the url
passed in, otherwise nil
235 236 237 238 239 |
# File 'lib/mechanize.rb', line 235 def visited? url url = url.href if url.respond_to? :href @agent.visited_page url end |