Method: Mechanize::HTTP::Agent#robots_allowed?

Defined in:
lib/mechanize/http/agent.rb

#robots_allowed?(uri) ⇒ Boolean

Tests if this agent is allowed to access url, consulting the site’s robots.txt.

Returns:

  • (Boolean)
[View source]

1116
1117
1118
1119
1120
 
# File 'lib/mechanize/http/agent.rb', line 1116

def robots_allowed? uri
  return true if Thread.current[RobotsKey]

  webrobots.allowed? uri
end