Class: CMSScanner::Model::RobotsTxt

Inherits:
InterestingFinding show all
Defined in:
app/models/robots_txt.rb

Overview

Robots.txt

Constant Summary

Constants included from Finders::Finding

Finders::Finding::FINDING_OPTS

Instance Attribute Summary

Attributes inherited from InterestingFinding

#url

Instance Method Summary collapse

Methods inherited from InterestingFinding

#==, #entries, #initialize, #type

Methods included from Finders::Finding

#<=>, #confidence, #confidence=, #confirmed_by, #eql?, included, #parse_finding_options

Constructor Details

This class inherits a constructor from CMSScanner::Model::InterestingFinding

Instance Method Details

#interesting_entries ⇒ Array<String>

TODO:

Better detection, currently everything not empty or / is returned

Returns The interesting Allow/Disallow rules detected.

Returns:

  • (Array<String>) —

    The interesting Allow/Disallow rules detected



15
16
17
18
19
20
21
22
23
24
25
26
27
28
 
# File 'app/models/robots_txt.rb', line 15

def interesting_entries
  results = []

  entries.each do |entry|
    next unless entry =~ /\A(?:dis)?allow:\s*(.+)\z/i

    match = Regexp.last_match(1)
    next if match == '/'

    results << match
  end

  results.uniq
end

#to_s ⇒ String

Returns:

  • (String) 


8
9
10
 
# File 'app/models/robots_txt.rb', line 8

def to_s
  @to_s ||= "robots.txt found: #{url}"
end