Class: Mechanize

Inherits:
Object
  • Object
show all
Defined in:
lib/mechanize.rb

Overview

The Mechanize library is used for automating interactions with a website. It can follow links and submit forms. Form fields can be populated and submitted. A history of URLs is maintained and can be queried.

Example

require 'mechanize'
require 'logger'

agent = Mechanize.new
agent.log = Logger.new "mech.log"
agent.user_agent_alias = 'Mac Safari'

page = agent.get "http://www.google.com/"
search_form = page.form_with :name => "f"
search_form.field_with(:name => "q").value = "Hello"

search_results = agent.submit search_form
puts search_results.body

Issues with mechanize

If you think you have a bug with mechanize, but aren't sure, please file a ticket at https://github.com/tenderlove/mechanize/issues

Here are some common problems you may experience with mechanize

Problems connecting to SSL sites

Mechanize defaults to validating SSL certificates using the default CA certificates for your platform. At this time, Windows users do not have integration between the OS default CA certificates and OpenSSL. #cert_store explains how to download and use Mozilla's CA certificates to allow SSL sites to work.

Problems with content-length

Some sites return an incorrect content-length value. Unlike a browser, mechanize raises an error when the content-length header does not match the response length since it does not know if there was a connection problem or if the mismatch is a server bug.

The error raised, Mechanize::ResponseReadError, can be converted to a parsed Page, File, etc. depending upon the content-type:

agent = Mechanize.new
uri = URI 'http://example/invalid_content_length'

begin
page = agent.get uri
rescue Mechanize::ResponseReadError => e
page = e.force_parse
end

Defined Under Namespace

Modules: ElementMatcher, Parser Classes: ContentTypeError, Cookie, CookieJar, DirectorySaver, Download, Error, File, FileConnection, FileRequest, FileResponse, FileSaver, Form, HTTP, Headers, History, Image, Page, PluggableParser, RedirectLimitReachedError, RedirectNotGetOrHeadError, ResponseCodeError, ResponseReadError, RobotsDisallowedError, TestCase, UnauthorizedError, UnsupportedSchemeError, Util

Constant Summary collapse

VERSION =

The version of Mechanize you are using.

'2.3'
AGENT_ALIASES =

Supported User-Agent aliases for use with user_agent_alias=. The description in parenthesis is for informative purposes and is not part of the alias name.

  • Linux Firefox (3.6.1)
  • Linux Konqueror (3)
  • Linux Mozilla
  • Mac Firefox (3.6)
  • Mac Mozilla
  • Mac Safari (5)
  • Mac Safari 4
  • Mechanize (default)
  • Windows IE 6
  • Windows IE 7
  • Windows IE 8
  • Windows IE 9
  • Windows Mozilla
  • iPhone (3.0)

Example:

agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
{
  'Mechanize' => "Mechanize/#{VERSION} Ruby/#{ruby_version} (http://github.com/tenderlove/mechanize/)",
  'Linux Firefox' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.1) Gecko/20100122 firefox/3.6.1',
  'Linux Konqueror' => 'Mozilla/5.0 (compatible; Konqueror/3; Linux)',
  'Linux Mozilla' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624',
  'Mac FireFox' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6',
  'Mac Mozilla' => 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030401',
  'Mac Safari 4' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; de-at) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10',
  'Mac Safari' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22',
  'Windows IE 6' => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)',
  'Windows IE 7' => 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)',
  'Windows IE 8' => 'Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)',
  'Windows IE 9' => 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)',
  'Windows Mozilla' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6',
  'iPhone' => 'Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1C28 Safari/419.3',
}

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize {|_self| ... } ⇒ Mechanize

Creates a new mechanize instance. If a block is given, the created instance is yielded to the block for setting up pre-connection state such as SSL parameters or proxies:

agent = Mechanize.new do |a|
a.proxy_host = 'proxy.example'
a.proxy_port = 8080
end

Yields:

  • (_self)

Yield Parameters:

  • _self (Mechanize) —

    the object that the method was called on



148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
 
# File 'lib/mechanize.rb', line 148

def initialize
  @agent = Mechanize::HTTP::Agent.new
  @agent.context = self
  @log = nil

  # attr_accessors
  @agent.user_agent = AGENT_ALIASES['Mechanize']
  @watch_for_set    = nil
  @history_added    = nil

  # attr_readers
  @pluggable_parser = PluggableParser.new

  @keep_alive_time  = 0

  # Proxy
  @proxy_addr = nil
  @proxy_port = nil
  @proxy_user = nil
  @proxy_pass = nil

  @html_parser = self.class.html_parser

  @default_encoding = nil
  @force_default_encoding = false

  # defaults
  @agent.max_history = 50

  yield self if block_given?

  @agent.set_proxy @proxy_addr, @proxy_port, @proxy_user, @proxy_pass
end

Class Attribute Details

.html_parser ⇒ Object

Default HTML parser for all mechanize instances

Mechanize.html_parser = Nokogiri::XML


553
554
555
 
# File 'lib/mechanize.rb', line 553

def html_parser
  @html_parser
end

.log ⇒ Object

Default logger for all mechanize instances

Mechanize.log = Logger.new $stderr


560
561
562
 
# File 'lib/mechanize.rb', line 560

def log
  @log
end

Instance Attribute Details

#agent ⇒ Object (readonly)

:section: Utilities



1109
1110
1111
 
# File 'lib/mechanize.rb', line 1109

def agent
  @agent
end

#default_encoding ⇒ Object

A default encoding name used when parsing HTML parsing. When set it is used after any other encoding. The default is nil.



568
569
570
 
# File 'lib/mechanize.rb', line 568

def default_encoding
  @default_encoding
end

#force_default_encoding ⇒ Object

Overrides the encodings given by the HTTP server and the HTML page with the default_encoding when set to true.



574
575
576
 
# File 'lib/mechanize.rb', line 574

def force_default_encoding
  @force_default_encoding
end

#history_added ⇒ Object

Callback which is invoked with the page that was added to history.



263
264
265
 
# File 'lib/mechanize.rb', line 263

def history_added
  @history_added
end

#html_parser ⇒ Object

The HTML parser to be used when parsing documents



579
580
581
 
# File 'lib/mechanize.rb', line 579

def html_parser
  @html_parser
end

#keep_alive_time ⇒ Object

HTTP/1.0 keep-alive time. This is no longer supported by mechanize as it now uses net-http-persistent which only supports HTTP/1.1 persistent connections



586
587
588
 
# File 'lib/mechanize.rb', line 586

def keep_alive_time
  @keep_alive_time
end

#pluggable_parser ⇒ Object (readonly)

The pluggable parser maps a response Content-Type to a parser class. The registered Content-Type may be either a full content type like 'image/png' or a media type 'text'. See Mechanize::PluggableParser for further details.

Example:

agent.pluggable_parser['application/octet-stream'] = Mechanize::Download


598
599
600
 
# File 'lib/mechanize.rb', line 598

def pluggable_parser
  @pluggable_parser
end

#proxy_addr ⇒ Object (readonly)

The HTTP proxy address



603
604
605
 
# File 'lib/mechanize.rb', line 603

def proxy_addr
  @proxy_addr
end

#proxy_pass ⇒ Object (readonly)

The HTTP proxy password



608
609
610
 
# File 'lib/mechanize.rb', line 608

def proxy_pass
  @proxy_pass
end

#proxy_port ⇒ Object (readonly)

The HTTP proxy port



613
614
615
 
# File 'lib/mechanize.rb', line 613

def proxy_port
  @proxy_port
end

#proxy_user ⇒ Object (readonly)

The HTTP proxy username



618
619
620
 
# File 'lib/mechanize.rb', line 618

def proxy_user
  @proxy_user
end

#watch_for_set ⇒ Object

The value of watch_for_set is passed to pluggable parsers for retrieved content



948
949
950
 
# File 'lib/mechanize.rb', line 948

def watch_for_set
  @watch_for_set
end

Class Method Details

.inherited(child) ⇒ Object

:nodoc:



132
133
134
135
136
 
# File 'lib/mechanize.rb', line 132

def self.inherited(child) # :nodoc:
  child.html_parser ||= html_parser
  child.log ||= log
  super
end

Instance Method Details

#auth(user, password, domain = nil) ⇒ Object Also known as: basic_auth

Sets the user and password to be used for HTTP authentication. sets the optional domain for NTLM authentication



624
625
626
627
628
 
# File 'lib/mechanize.rb', line 624

def auth(user, password, domain = nil)
  @agent.user     = user
  @agent.password = password
  @agent.domain = domain
end

#back ⇒ Object

Equivalent to the browser back button. Returns the previous page visited.



189
190
191
 
# File 'lib/mechanize.rb', line 189

def back
  @agent.history.pop
end

#ca_file ⇒ Object

Path to an OpenSSL server certificate file



958
959
960
 
# File 'lib/mechanize.rb', line 958

def ca_file
  @agent.ca_file
end

#ca_file=(ca_file) ⇒ Object

Sets the certificate file used for SSL connections



965
966
967
 
# File 'lib/mechanize.rb', line 965

def ca_file= ca_file
  @agent.ca_file = ca_file
end

#cert ⇒ Object

An OpenSSL client certificate or the path to a certificate file.



972
973
974
 
# File 'lib/mechanize.rb', line 972

def cert
  @agent.cert
end

#cert=(cert) ⇒ Object

Sets the OpenSSL client certificate cert to the given path or certificate instance



980
981
982
 
# File 'lib/mechanize.rb', line 980

def cert= cert
  @agent.certificate = cert
end

#cert_store ⇒ Object

An OpenSSL certificate store for verifying server certificates. This defaults to the default certificate store for your system.

If your system does not ship with a default set of certificates you can retrieve a copy of the set from Mozilla here: http://curl.haxx.se/docs/caextract.html

(Note that this set does not have an HTTPS download option so you may wish to use the firefox-db2pem.sh script to extract the certificates from a local install to avoid man-in-the-middle attacks.)

After downloading or generating a cacert.pem from the above link you can create a certificate store from the pem file like this:

cert_store = OpenSSL::X509::Store.new
cert_store.add_file 'cacert.pem'

And have mechanize use it with:

agent.cert_store = cert_store


1006
1007
1008
 
# File 'lib/mechanize.rb', line 1006

def cert_store
  @agent.cert_store
end

#cert_store=(cert_store) ⇒ Object

Sets the OpenSSL certificate store to store.

See also #cert_store



1015
1016
1017
 
# File 'lib/mechanize.rb', line 1015

def cert_store= cert_store
  @agent.cert_store = cert_store
end

#certificate ⇒ Object

What is this?

Why is it different from #cert?



1024
1025
1026
 
# File 'lib/mechanize.rb', line 1024

def certificate # :nodoc:
  @agent.certificate
end

#click(link) ⇒ Object

If the parameter is a string, finds the button or link with the value of the string on the current page and clicks it. Otherwise, clicks the Mechanize::Page::Link object passed in. Returns the page fetched.



290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
 
# File 'lib/mechanize.rb', line 290

def click link
  case link
  when Page::Link then
    referer = link.page || current_page()
    if @agent.robots
      if (referer.is_a?(Page) and referer.parser.nofollow?) or
         link.rel?('nofollow') then
        raise RobotsDisallowedError.new(link.href)
      end
    end
    if link.noreferrer?
      href = @agent.resolve(link.href, link.page || current_page)
      referer = Page.new
    else
      href = link.href
    end
    get href, [], referer
  when String, Regexp then
    if real_link = page.link_with(:text => link)
      click real_link
    else
      button = nil
      form = page.forms.find do |f|
        button = f.button_with(:value => link)
        button.is_a? Form::Submit
      end
      submit form, button if form
    end
  else
    referer = current_page()
    href = link.respond_to?(:href) ? link.href :
      (link['href'] || link['src'])
    get href, [], referer
  end
end

#conditional_requests ⇒ Object

Are If-Modified-Since conditional requests enabled?



635
636
637
 
# File 'lib/mechanize.rb', line 635

def conditional_requests
  @agent.conditional_requests
end

#conditional_requests=(enabled) ⇒ Object

Disables If-Modified-Since conditional requests (enabled by default)



642
643
644
 
# File 'lib/mechanize.rb', line 642

def conditional_requests= enabled
  @agent.conditional_requests = enabled
end

#content_encoding_hooks ⇒ Object

A list of hooks to call before reading response header 'content-encoding'.

The hook is called with the agent making the request, the URI of the request, the response an IO containing the response body.



256
257
258
 
# File 'lib/mechanize.rb', line 256

def content_encoding_hooks
  @agent.content_encoding_hooks
end

#cookie_jar ⇒ Object

A Mechanize::CookieJar which stores cookies



649
650
651
 
# File 'lib/mechanize.rb', line 649

def cookie_jar
  @agent.cookie_jar
end

#cookie_jar=(cookie_jar) ⇒ Object

Replaces the cookie jar with cookie_jar



656
657
658
 
# File 'lib/mechanize.rb', line 656

def cookie_jar= cookie_jar
  @agent.cookie_jar = cookie_jar
end

#cookies ⇒ Object

Returns a list of cookies stored in the cookie jar.



663
664
665
 
# File 'lib/mechanize.rb', line 663

def cookies
  @agent.cookie_jar.to_a
end

#current_page ⇒ Object Also known as: page

Returns the latest page loaded by Mechanize



196
197
198
 
# File 'lib/mechanize.rb', line 196

def current_page
  @agent.current_page
end

#delete(uri, query_params = {}, headers = {}) ⇒ Object

DELETE uri with query_params, and setting headers:

delete('http://example/', {'q' => 'foo'}, {})


372
373
374
375
376
 
# File 'lib/mechanize.rb', line 372

def delete(uri, query_params = {}, headers = {})
  page = @agent.fetch(uri, :delete, headers, query_params)
  add_to_history(page)
  page
end

#download(uri, io_or_filename, parameters = [], referer = nil, headers = {}) ⇒ Object

GETs uri and writes it to io_or_filename without recording the request in the history. If io_or_filename does not respond to #write it will be used as a file name. parameters, referer and headers are used as in #get.

By default, if the Content-type of the response matches a Mechanize::File or Mechanize::Page parser, the response body will be loaded into memory before being saved. See #pluggable_parser for details on changing this default.

For alternate ways of downloading files see Mechanize::FileSaver and Mechanize::DirectorySaver.



340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
 
# File 'lib/mechanize.rb', line 340

def download uri, io_or_filename, parameters = [], referer = nil, headers = {}
  page = transact do
    get uri, parameters, referer, headers
  end

  io = if io_or_filename.respond_to? :write then
         io_or_filename
       else
         open io_or_filename, 'wb'
       end

  case page
  when Mechanize::File then
    io.write page.body
  else
    body_io = page.body_io

    until body_io.eof? do
      io.write body_io.read 16384
    end
  end

  page
ensure
  io.close if io and not io_or_filename.respond_to? :write
end

#follow_meta_refresh ⇒ Object

Follow HTML meta refresh and HTTP Refresh headers. If set to :anywhere meta refresh tags outside of the head element will be followed.



671
672
673
 
# File 'lib/mechanize.rb', line 671

def follow_meta_refresh
  @agent.follow_meta_refresh
end

#follow_meta_refresh=(follow) ⇒ Object

Controls following of HTML meta refresh and HTTP Refresh headers in responses.



679
680
681
 
# File 'lib/mechanize.rb', line 679

def follow_meta_refresh= follow
  @agent.follow_meta_refresh = follow
end

#follow_meta_refresh_self ⇒ Object

Follow an HTML meta refresh and HTTP Refresh headers that have no "url=" in the content attribute.

Defaults to false to prevent infinite refresh loops.



689
690
691
 
# File 'lib/mechanize.rb', line 689

def follow_meta_refresh_self
  @agent.follow_meta_refresh_self
end

#follow_meta_refresh_self=(follow) ⇒ Object

Alters the following of HTML meta refresh and HTTP Refresh headers that point to the same page.



697
698
699
 
# File 'lib/mechanize.rb', line 697

def follow_meta_refresh_self= follow
  @agent.follow_meta_refresh_self = follow
end

#get(uri, parameters = [], referer = nil, headers = {}) {|page| ... } ⇒ Object

GET the uri with the given request parameters, referer and headers.

The referer may be a URI or a page.

Yields:



384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
 
# File 'lib/mechanize.rb', line 384

def get(uri, parameters = [], referer = nil, headers = {})
  method = :get

  referer ||=
    if uri.to_s =~ %r{\Ahttps?://}
      Page.new
    else
      current_page || Page.new
    end

  # FIXME: Huge hack so that using a URI as a referer works.  I need to
  # refactor everything to pass around URIs but still support
  # Mechanize::Page#base
  unless Mechanize::Parser === referer then
    referer = if referer.is_a?(String) then
                Page.new URI(referer)
              else
                Page.new referer
              end
  end

  # fetch the page
  headers ||= {}
  page = @agent.fetch uri, method, headers, parameters, referer
  add_to_history(page)
  yield page if block_given?
  page
end

#get_file(url) ⇒ Object

GET url and return only its contents



416
417
418
 
# File 'lib/mechanize.rb', line 416

def get_file(url)
  get(url).body
end

#gzip_enabled ⇒ Object

Is gzip compression of responses enabled?



704
705
706
 
# File 'lib/mechanize.rb', line 704

def gzip_enabled
  @agent.gzip_enabled
end

#gzip_enabled=(enabled) ⇒ Object

Disables HTTP/1.1 gzip compression (enabled by default)



711
712
713
 
# File 'lib/mechanize.rb', line 711

def gzip_enabled=enabled
  @agent.gzip_enabled = enabled
end

#head(uri, query_params = {}, headers = {}) {|page| ... } ⇒ Object

HEAD uri with query_params and headers:

head('http://example/', {'q' => 'foo'}, {})

Yields:



425
426
427
428
429
430
431
 
# File 'lib/mechanize.rb', line 425

def head(uri, query_params = {}, headers = {})
  page = @agent.fetch uri, :head, headers, query_params

  yield page if block_given?

  page
end

#history ⇒ Object

The history of this mechanize run



205
206
207
 
# File 'lib/mechanize.rb', line 205

def history
  @agent.history
end

#idle_timeout ⇒ Object

Connections that have not been used in this many seconds will be reset.



718
719
720
 
# File 'lib/mechanize.rb', line 718

def idle_timeout
  @agent.idle_timeout
end

#idle_timeout=(idle_timeout) ⇒ Object

Sets the idle timeout to idle_timeout. The default timeout is 5 seconds. If you experience "too many connection resets", reducing this value may help.



726
727
728
 
# File 'lib/mechanize.rb', line 726

def idle_timeout= idle_timeout
  @agent.idle_timeout = idle_timeout
end

#keep_alive ⇒ Object

Are HTTP/1.1 keep-alive connections enabled?



733
734
735
 
# File 'lib/mechanize.rb', line 733

def keep_alive
  @agent.keep_alive
end

#keep_alive=(enable) ⇒ Object

Disable HTTP/1.1 keep-alive connections if enable is set to false. If you are experiencing "too many connection resets" errors setting this to false will eliminate them.

You should first investigate reducing idle_timeout.



744
745
746
 
# File 'lib/mechanize.rb', line 744

def keep_alive= enable
  @agent.keep_alive = enable
end

#key ⇒ Object

An OpenSSL private key or the path to a private key



1031
1032
1033
 
# File 'lib/mechanize.rb', line 1031

def key
  @agent.key
end

#key=(key) ⇒ Object

Sets the OpenSSL client key to the given path or key instance. If a path is given, the path must contain an RSA key file.



1039
1040
1041
 
# File 'lib/mechanize.rb', line 1039

def key= key
  @agent.private_key = key
end

#log ⇒ Object

The current logger. If no logger has been set Mechanize.log is used.



751
752
753
 
# File 'lib/mechanize.rb', line 751

def log
  @log || Mechanize.log
end

#log=(logger) ⇒ Object

Sets the logger used by this instance of mechanize



758
759
760
 
# File 'lib/mechanize.rb', line 758

def log= logger
  @log = logger
end

#max_file_buffer ⇒ Object

Responses larger than this will be written to a Tempfile instead of stored in memory. The default is 100,000 bytes.

A value of nil disables creation of Tempfiles.



768
769
770
 
# File 'lib/mechanize.rb', line 768

def max_file_buffer
  @agent.max_file_buffer
end

#max_file_buffer=(bytes) ⇒ Object

Sets the maximum size of a response body that will be stored in memory to bytes. A value of nil causes all response bodies to be stored in memory.

Note that for Mechanize::Download subclasses, the maximum buffer size multiplied by the number of pages stored in history (controlled by #max_history) is an approximate upper limit on the amount of memory Mechanize will use. By default, Mechanize can use up to ~5MB to store response bodies for non-File and non-Page (HTML) responses.

See also the discussion under #max_history=



785
786
787
 
# File 'lib/mechanize.rb', line 785

def max_file_buffer= bytes
  @agent.max_file_buffer = bytes
end

#max_history ⇒ Object

Maximum number of items allowed in the history. The default setting is 50 pages. Note that the size of the history multiplied by the maximum response body size



214
215
216
 
# File 'lib/mechanize.rb', line 214

def max_history
  @agent.history.max_size
end

#max_history=(length) ⇒ Object

Sets the maximum number of items allowed in the history to length.

Setting the maximum history length to nil will make the history size unlimited. Take care when doing this, mechanize stores response bodies in memory for pages and in the temporary files directory for other responses. For a long-running mechanize program this can be quite large.

See also the discussion under #max_file_buffer=



228
229
230
 
# File 'lib/mechanize.rb', line 228

def max_history= length
  @agent.history.max_size = length
end

#open_timeout ⇒ Object

Length of time to wait until a connection is opened in seconds



792
793
794
 
# File 'lib/mechanize.rb', line 792

def open_timeout
  @agent.open_timeout
end

#open_timeout=(open_timeout) ⇒ Object

Sets the connection open timeout to open_timeout



799
800
801
 
# File 'lib/mechanize.rb', line 799

def open_timeout= open_timeout
  @agent.open_timeout = open_timeout
end

#parse(uri, response, body) ⇒ Object

Parses the body of the response from uri using the pluggable parser that matches its content type



1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
 
# File 'lib/mechanize.rb', line 1115

def parse uri, response, body
  content_type = nil

  unless response['Content-Type'].nil?
    data, = response['Content-Type'].split ';', 2
    content_type, = data.downcase.split ',', 2 unless data.nil?
  end

  parser_klass = @pluggable_parser.parser content_type

  unless parser_klass <= Mechanize::Download then
    body = case body
           when IO, Tempfile, StringIO then
             body.read
           else
             body
           end
  end

  parser_klass.new uri, response, body, response.code do |parser|
    parser.mech = self if parser.respond_to? :mech=

    parser.watch_for_set = @watch_for_set if
      @watch_for_set and parser.respond_to?(:watch_for_set=)
  end
end

#pass ⇒ Object

OpenSSL client key password



1046
1047
1048
 
# File 'lib/mechanize.rb', line 1046

def pass
  @agent.pass
end

#pass=(pass) ⇒ Object

Sets the client key password to pass



1053
1054
1055
 
# File 'lib/mechanize.rb', line 1053

def pass= pass
  @agent.pass = pass
end

#post(uri, query = {}, headers = {}) ⇒ Object

POST to the given uri with the given query. The query is specified by either a string, or a list of key-value pairs represented by a hash or an array of arrays.

Examples:

agent.post 'http://example.com/', "foo" => "bar"

agent.post 'http://example.com/', [%w[foo bar]]

agent.post('http://example.com/', "<message>hello</message>",
         'Content-Type' => 'application/xml')


446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
 
# File 'lib/mechanize.rb', line 446

def post(uri, query={}, headers={})
  return request_with_entity(:post, uri, query, headers) if String === query

  node = {}
  # Create a fake form
  class << node
    def search(*args); []; end
  end
  node['method'] = 'POST'
  node['enctype'] = 'application/x-www-form-urlencoded'

  form = Form.new(node)

  query.each { |k, v|
    if v.is_a?(IO)
      form.enctype = 'multipart/form-data'
      ul = Form::FileUpload.new({'name' => k.to_s},::File.basename(v.path))
      ul.file_data = v.read
      form.file_uploads << ul
    else
      form.fields << Form::Field.new({'name' => k.to_s},v)
    end
  }
  post_form(uri, form, headers)
end

#post_connect_hooks ⇒ Object

A list of hooks to call after retrieving a response. Hooks are called with the agent and the response returned.



269
270
271
 
# File 'lib/mechanize.rb', line 269

def post_connect_hooks
  @agent.post_connect_hooks
end

#pre_connect_hooks ⇒ Object

A list of hooks to call before making a request. Hooks are called with the agent and the request to be performed.



277
278
279
 
# File 'lib/mechanize.rb', line 277

def pre_connect_hooks
  @agent.pre_connect_hooks
end

#pretty_print(q) ⇒ Object

:nodoc:



1142
1143
1144
1145
1146
1147
1148
1149
 
# File 'lib/mechanize.rb', line 1142

def pretty_print(q) # :nodoc:
  q.object_group(self) {
    q.breakable
    q.pp cookie_jar
    q.breakable
    q.pp current_page
  }
end

#put(uri, entity, headers = {}) ⇒ Object

PUT to uri with entity, and setting headers:

put('http://example/', 'new content', {'Content-Type' => 'text/plain'})


477
478
479
 
# File 'lib/mechanize.rb', line 477

def put(uri, entity, headers = {})
  request_with_entity(:put, uri, entity, headers)
end

#read_timeout ⇒ Object

Length of time to wait for data from the server



806
807
808
 
# File 'lib/mechanize.rb', line 806

def read_timeout
  @agent.read_timeout
end

#read_timeout=(read_timeout) ⇒ Object

Sets the timeout for each chunk of data read from the server to read_timeout. A single request may read many chunks of data.



814
815
816
 
# File 'lib/mechanize.rb', line 814

def read_timeout= read_timeout
  @agent.read_timeout = read_timeout
end

#redirect_ok ⇒ Object Also known as: follow_redirect?

Controls how mechanize deals with redirects. The following values are allowed:

:all, true:: All 3xx redirects are followed (default) :permanent:: Only 301 Moved Permanantly redirects are followed

false

No redirects are followed



826
827
828
 
# File 'lib/mechanize.rb', line 826

def redirect_ok
  @agent.redirect_ok
end

#redirect_ok=(follow) ⇒ Object

Sets the mechanize redirect handling policy. See redirect_ok for allowed values



836
837
838
 
# File 'lib/mechanize.rb', line 836

def redirect_ok= follow
  @agent.redirect_ok = follow
end

#redirection_limit ⇒ Object

Maximum number of redirections to follow



843
844
845
 
# File 'lib/mechanize.rb', line 843

def redirection_limit
  @agent.redirection_limit
end

#redirection_limit=(limit) ⇒ Object

Sets the maximum number of redirections to follow to limit



850
851
852
 
# File 'lib/mechanize.rb', line 850

def redirection_limit= limit
  @agent.redirection_limit = limit
end

#request_headers ⇒ Object

A hash of custom request headers that will be sent on every request



857
858
859
 
# File 'lib/mechanize.rb', line 857

def request_headers
  @agent.request_headers
end

#request_headers=(request_headers) ⇒ Object

Replaces the custom request headers that will be sent on every request with request_headers



865
866
867
 
# File 'lib/mechanize.rb', line 865

def request_headers= request_headers
  @agent.request_headers = request_headers
end

#request_with_entity(verb, uri, entity, headers = {}) ⇒ Object

Makes an HTTP request to url using HTTP method verb. entity is used as the request body, if allowed.



485
486
487
488
489
490
491
492
493
494
495
496
 
# File 'lib/mechanize.rb', line 485

def request_with_entity(verb, uri, entity, headers = {})
  cur_page = current_page || Page.new

  headers = {
    'Content-Type' => 'application/octet-stream',
    'Content-Length' => entity.size.to_s,
  }.update headers

  page = @agent.fetch uri, verb, headers, [entity], cur_page
  add_to_history(page)
  page
end

#retry_change_requests ⇒ Object

Retry POST and other non-idempotent requests. See RFC 2616 9.1.2.



872
873
874
 
# File 'lib/mechanize.rb', line 872

def retry_change_requests
  @agent.retry_change_requests
end

#retry_change_requests=(retry_change_requests) ⇒ Object

When setting retry_change_requests to true you are stating that, for all the URLs you access with mechanize, making POST and other non-idempotent requests is safe and will not cause data duplication or other harmful results.

If you are experiencing "too many connection resets" errors you should instead investigate reducing the idle_timeout or disabling keep_alive connections.



886
887
888
 
# File 'lib/mechanize.rb', line 886

def retry_change_requests= retry_change_requests
  @agent.retry_change_requests = retry_change_requests
end

#robots ⇒ Object

Will /robots.txt files be obeyed?



893
894
895
 
# File 'lib/mechanize.rb', line 893

def robots
  @agent.robots
end

#robots=(enabled) ⇒ Object

When enabled mechanize will retrieve and obey robots.txt files



901
902
903
 
# File 'lib/mechanize.rb', line 901

def robots= enabled
  @agent.robots = enabled
end

#scheme_handlers ⇒ Object

The handlers for HTTP and other URI protocols.



908
909
910
 
# File 'lib/mechanize.rb', line 908

def scheme_handlers
  @agent.scheme_handlers
end

#scheme_handlers=(scheme_handlers) ⇒ Object

Replaces the URI scheme handler table with scheme_handlers



915
916
917
 
# File 'lib/mechanize.rb', line 915

def scheme_handlers= scheme_handlers
  @agent.scheme_handlers = scheme_handlers
end

#set_proxy(address, port, user = nil, password = nil) ⇒ Object

Sets the proxy address at port with an optional user and password



1154
1155
1156
1157
1158
1159
1160
1161
 
# File 'lib/mechanize.rb', line 1154

def set_proxy address, port, user = nil, password = nil
  @proxy_addr = address
  @proxy_port = port
  @proxy_user = user
  @proxy_pass = password

  @agent.set_proxy address, port, user, password
end

#ssl_version ⇒ Object

SSL version to use. Ruby 1.9 and newer only.



1060
1061
1062
 
# File 'lib/mechanize.rb', line 1060

def ssl_version
  @agent.ssl_version
end

#ssl_version=(ssl_version) ⇒ Object

Sets the SSL version to use to version without client/server negotiation. Ruby 1.9 and newer only.



1068
1069
1070
 
# File 'lib/mechanize.rb', line 1068

def ssl_version= ssl_version
  @agent.ssl_version = ssl_version
end

#submit(form, button = nil, headers = {}) ⇒ Object

Submits form with an optional button.

Without a button:

page = agent.get('http://example.com')
agent.submit(page.forms.first)

With a button:

agent.submit(page.forms.first, page.forms.first.buttons.first)


510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
 
# File 'lib/mechanize.rb', line 510

def submit(form, button=nil, headers={})
  form.add_button_to_query(button) if button

  case form.method.upcase
  when 'POST'
    post_form(form.action, form, headers)
  when 'GET'
    get(form.action.gsub(/\?[^\?]*$/, ''),
        form.build_query,
        form.page,
        headers)
  else
    raise ArgumentError, "unsupported method: #{form.method.upcase}"
  end
end

#transact ⇒ Object

Runs given block, then resets the page history as it was before. self is given as a parameter to the block. Returns the value of the block.



530
531
532
533
534
535
536
537
 
# File 'lib/mechanize.rb', line 530

def transact
  history_backup = @agent.history.dup
  begin
    yield self
  ensure
    @agent.history = history_backup
  end
end

#user_agent ⇒ Object

The identification string for the client initiating a web request



922
923
924
 
# File 'lib/mechanize.rb', line 922

def user_agent
  @agent.user_agent
end

#user_agent=(user_agent) ⇒ Object

Sets the User-Agent used by mechanize to user_agent. See also user_agent_alias



930
931
932
 
# File 'lib/mechanize.rb', line 930

def user_agent= user_agent
  @agent.user_agent = user_agent
end

#user_agent_alias=(name) ⇒ Object

Set the user agent for the Mechanize object based on the given name.

See also AGENT_ALIASES



939
940
941
942
 
# File 'lib/mechanize.rb', line 939

def user_agent_alias= name
  self.user_agent = AGENT_ALIASES[name] ||
    raise(ArgumentError, "unknown agent alias #{name.inspect}")
end

#verify_callback ⇒ Object

A callback for additional certificate verification. See OpenSSL::SSL::SSLContext#verify_callback

The callback can be used for debugging or to ignore errors by always returning true. Specifying nil uses the default method that was valid when the SSLContext was created



1080
1081
1082
 
# File 'lib/mechanize.rb', line 1080

def verify_callback
  @agent.verify_callback
end

#verify_callback=(verify_callback) ⇒ Object

Sets the OpenSSL certificate verification callback



1087
1088
1089
 
# File 'lib/mechanize.rb', line 1087

def verify_callback= verify_callback
  @agent.verify_callback = verify_callback
end

#verify_mode ⇒ Object

the OpenSSL server certificate verification method. The default is OpenSSL::SSL::VERIFY_PEER and certificate verification uses the default system certificates. See also cert_store



1096
1097
1098
 
# File 'lib/mechanize.rb', line 1096

def verify_mode
  @agent.verify_mode
end

#verify_mode=(verify_mode) ⇒ Object

Sets the OpenSSL server certificate verification method.



1103
1104
1105
 
# File 'lib/mechanize.rb', line 1103

def verify_mode= verify_mode
  @agent.verify_mode = verify_mode
end

#visited?(url) ⇒ Boolean Also known as: visited_page

Returns a visited page for the url passed in, otherwise nil

Returns:

  • (Boolean) 


235
236
237
238
239
 
# File 'lib/mechanize.rb', line 235

def visited? url
  url = url.href if url.respond_to? :href

  @agent.visited_page url
end