Module: Mdm::Host::OperatingSystemNormalization
- Included in:
- Mdm::Host
- Defined in:
- lib/mdm/host/operating_system_normalization.rb
Overview
Handle OS icon incompatiblities with new fingerprint names Note that VMWare ESX(i) was special cased before as well, make sure it still works 1) Cisco IOS -> IOS breaks the icon mapping in MSP/MSCE of /cisco/ 2) Ubuntu Linux -> Linux breaks the distro selection The real solution is to add os_vendor and take this into account for icons
Implement rspec coverage for normalize_os()
Implement smb.generic fingerprint database (replace #parse_windows_os_str?)
Implement Samba version matching for specific distributions and OS versions
Implement DD-WRT and various embedded device signatures currently missing
Correct inconsistencies in os_name use by removing the vendor string (Microsoft Windows -> Windows) This applies to MSF core and a handful of modules, not to mention some Recog fingerprints.
Rename host.os_sp to host.os_version
Add host.os_vendor
Add host.os_confidence
Add host.domain
Rules for operating system fingerprinting in Metasploit
The os.product
key identifies the common-name of a specific operating system
Examples include: Linux, Windows XP, Mac OS X, IOS, AIX, HP-UX, VxWorks
The os.version
key identifies the service pack or version of the operating system
Sometimes this means a kernel or firmware version when the distribution or OS
version is not available.
Examples include: SP2, 10.04, 2.6.47, 10.6.1
The os.vendor
key identifies the manufacturer of the operating system
Examples include: Microsoft, Ubuntu, Cisco, HP, IBM, Wind River
The os.family
key identifies the group of the operating system. This is often a
duplicate of os.product, unless a more specific product name is available.
Examples include: Windows, Linux, IOS, HP-UX, AIX
The os.edition
key identifies the specific variant of the operating system
Examples include: Enterprise, Professional, Starter, Evaluation, Home, Datacenter
An example breakdown of a common operating system is shown below
- Microsoft Windows XP Professional Service Pack 3 English (x86)
- os.product = 'Windows XP'
- os.edition = 'Professional'
- os.vendor = 'Microsoft'
- os.version = 'SP3'
- os.language = 'English'
- os.arch = 'x86'
These rules are then mapped to the Mdm::Host attributes below:
- os_name - Maps to a normalized os.product key
- os_flavor - Maps to a normalized os.edition key
- os_sp - Maps to a normalized os.version key (soon os_version)
- os_lang - Maps to a normalized os.language key
- arch - Maps to a normalized os.arch key
Additional rules include the following mappings:
- name - Maps to the host.name key
- mac - Maps to the host.mac key
The following keys are not mapped to Mdm::Host at this time (but should be):
- os.vendor
In order to execute these rules, this module is responsible for mapping various fingerprint sources to Mdm::Host values. This requires some ugly glue code to account for differences between each supported input (external scanners), the Recog gem and associated databases, and how Metasploit itself likes to handle these values. Getting a mapping wrong is often harmless, but can impact the automatic targetting capabilities of certain exploit modules.
In other words, this is a best-effort attempt to rationalize multiple competing
sources of information about a host and come up with the values representing a
normalized assessment of the system. The use of Recog
and multiple scanner
fingerprints can result in a comprehensive (and confident) identification of the
remote operating system and associated services.
Historically, there are direct conflicts between certain Metasploit modules, certain scanners, and external fingerprint databases in terms of how a particular OS and patch level is represented. This module attempts to fix what it can and serve as documentation and live workarounds for the rest.
Examples of known conflicts that are still in progress:
Metasploit defines an OS constant of 'win'/'windows' as Microsoft Windows
- Scanner modules report a mix of 'Microsoft Windows' and 'Windows'
- Nearly all exploit modules reference 'Windows
SP ' - Nmap (and other scanners) also prefix the vendor before Windows
Windows service packs represented as 'Service Pack X' or 'SPX'
- The preferred form is to set os.version to 'SPX'
- Many external scanners & Recog prefer 'Service Pack X'
Apple Mac OS X, Cisco IOS, IBM AIX, Ubuntu Linux, all reported with vendor prefix
- The preferred form is to remove the vendor from os.product
- Mdm::Host currently has no vendor field, so this information is lost today
- Many scanners report leading vendor strings and require normalization
- The os_flavor field is used in contradictory ways across Metasploit
- The preferred form is to be a 'display only' field
- Some Recog fingerprints still append the edition to os.product
- Many scanners report the edition as a trailing suffix to os.product
Maintenance:
- Ensure that the latest Recog gem is present and installed
- For new operating system releases, update relevant sections a) Windows releases will require updates to a few methods 1) parse_windows_os_str() 2) normalize_nmap_fingerprint() 3) normalize_nexpose_fingerprint() 4) Other scanner normalizers b) Mobile operating systems are minimally recognized
Constant Summary collapse
- MAX_NMAP_CERTAINTY =
Cap nmap certainty at 0.84 until we update it more frequently XXX: Without this, Nmap will beat the default certainty of recog matches and its less-confident guesses will take precedence over service-based fingerprints.
0.84
Instance Method Summary collapse
-
#apply_match_to_host(match) ⇒ Object
Examine the assertations of the merged best match and map these back to fields of Mdm::Host.
-
#get_arch_from_string(str) ⇒ Object
Return a normalized architecture based on patterns in the input string.
-
#guess_purpose_from_match(match) ⇒ Object
Loosely guess the purpose of a device based on available match values.
-
#normalize_fusionvm_fingerprint(data) ⇒ Object
Normalize data from FusionVM fingerprints.
-
#normalize_generic_fingerprint(data) ⇒ Object
Normalize data from generic fingerprints.
-
#normalize_match(m) ⇒ Object
Normalize matches in order to handle inconsistencies between fingerprint sources and our desired usage in Metasploit.
-
#normalize_match_family(m) ⇒ Object
Normalize matches in order to ensure that an os.family entry exists if we have enough data to put one together.
-
#normalize_mbsa_fingerprint(data) ⇒ Object
Normalize data from MBSA fingerprints.
-
#normalize_nessus_fingerprint(data) ⇒ Object
Normalize data from Nessus fingerprints.
-
#normalize_nexpose_fingerprint(data) ⇒ Object
Normalize data from Nexpose fingerprints.
-
#normalize_nmap_fingerprint(data) ⇒ Object
Normalize data from Nmap fingerprints.
-
#normalize_os ⇒ Object
Normalize the operating system fingerprints provided by various scanners (nmap, nexpose, retina, nessus, metasploit modules, and more!).
-
#normalize_qualys_fingerprint(data) ⇒ Object
Normalize data from Qualys fingerprints.
-
#normalize_retina_fingerprint(data) ⇒ Object
Normalize data from Retina fingerprints.
-
#normalize_scanner_fp(fp) ⇒ Object
Convert a host.os._fingerprint Note into a hash containing 'os.' and 'host.*' fields.
-
#normalize_session_fingerprint(data) ⇒ Object
Normalize data from Meterpreter's client.sys.config.sysinfo().
-
#parse_windows_os_str(str) ⇒ Object
Take a windows version string and return a hash with fields suitable for Host this object's version fields.
-
#recog_matches_for_note(note) ⇒ Array<Hash>
Recog matches for the fingerprint in
note
. -
#recog_matches_for_service(s) ⇒ Array<Hash>
Recog matches for the
s
service. -
#sanitize(text) ⇒ Object
Ensure that the host attribute is using ascii safe text and escapes any other byte value.
-
#service_banner_recog_filter_ssh(banner) ⇒ Object
Recog assumes that the protocol version of the SSH banner has been removed.
-
#validate_fingerprint_data(fp) ⇒ Object
Determine if the fingerprint data is readable.
Instance Method Details
#apply_match_to_host(match) ⇒ Object
Examine the assertations of the merged best match and map these back to fields of Mdm::Host. Take particular care not to leave related fields (os_*) in a conflicting state, leverage existing values where possible, and use the most confident values we have.
395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 395 def apply_match_to_host(match) host = self # These values in a match always override the current value unless # the host attribute has been explicitly locked by the user if match['host.mac'] && !host.attribute_locked?(:mac) host.mac = sanitize(match['host.mac']) end if match['host.name'] && !host.attribute_locked?(:name) host.name = sanitize(match['host.name']) end # Select the os architecture if available if match['os.arch'] && !host.attribute_locked?(:arch) host.arch = sanitize(match['os.arch']) end # Guess the purpose using some basic heuristics if ! host.attribute_locked?(:purpose) host.purpose = guess_purpose_from_match(match) end # # Map match fields from Recog fingerprint style to Metasploit style # # os.build: Examples: 9001, 2600, 7602 # os.device: Examples: General, ADSL Modem, Broadband router, Cable Modem, Camera, Copier, CSU/DSU # os.edition: Examples: Web, Storage, HPC, MultiPoint, Enterprise, Home, Starter, Professional # os.family: Examples: Windows, Linux, Solaris, NetWare, ProCurve, Mac OS X, HP-UX, AIX # os.product: Examples: Windows, Linux, Windows Server 2008 R2, Windows XP, Enterprise Linux, NEO Tape Library # os.vendor: Examples: Microsoft, HP, IBM, Sun, 3Com, Ricoh, Novell, Ubuntu, Apple, Cisco, Xerox # os.version: Examples: SP1, SP2, 6.5 SP3 CPR, 10.04, 8.04, 12.10, 4.0, 6.1, 8.5 # os.language: Examples: English, Arabic, German # linux.kernel.version: Examples: 2.6.32 # Metasploit currently ignores os.build, os.device, and os.vendor as separate fields. # Select the OS name from os.name, fall back to os.family if ! host.attribute_locked?(:os_name) # Try to fill this value from os.product first if it exists if match.has_key?('os.product') host.os_name = sanitize(match['os.product']) else # Fall back to os.family otherwise, if available if match.has_key?('os.family') host.os_name = sanitize(match['os.family']) end end end if match.has_key?('os.family') host.os_family = sanitize(match['os.family']) end # Select the flavor from os.edition if available if match.has_key?('os.edition') and ! host.attribute_locked?(:os_flavor) host.os_flavor = sanitize(match['os.edition']) end # Select an OS version as os.version, fall back to linux.kernel.version if ! host.attribute_locked?(:os_sp) if match['os.version'] host.os_sp = sanitize(match['os.version']) else if match['linux.kernel.version'] host.os_sp = sanitize(match['linux.kernel.version']) end end end # Select the os language if available if match.has_key?('os.language') and ! host.attribute_locked?(:os_lang) host.os_lang = sanitize(match['os.language']) end # Normalize MAC addresses to lower-case colon-delimited format if host.mac and ! host.attribute_locked?(:mac) host.mac = host.mac.downcase if host.mac =~ /^[a-f0-9]{12}$/ host.mac = host.mac.scan(/../).join(':') end end end |
#get_arch_from_string(str) ⇒ Object
Return a normalized architecture based on patterns in the input string. This will identify things like sparc, powerpc, x86_x64, and i686
937 938 939 940 941 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 937 def get_arch_from_string(str) res = Recog::Nizer.match("architecture", str) return unless (res and res['os.arch']) res['os.arch'] end |
#guess_purpose_from_match(match) ⇒ Object
Loosely guess the purpose of a device based on available match values. In the future, also take into account the exposed services and rename to guess_purpose_with_match()
487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 487 def guess_purpose_from_match(match) # some data that is sent to this is numeric; we do not want that pstr = "" # Go through each character of each value and make sure it is all # UTF-8 match.values.each do |i| if i.respond_to?(:encoding) i.each_char do |j| begin pstr << j.downcase.encode("UTF-8") rescue Encoding::UndefinedConversionError # rescue Encoding::UndefinedConversionError => e # this works in Framework, but causes a Travis CI error # elog("Found incompatible (non-ANSI) character in guess_purpose_from_match") end end end end # Loosely map keywords to specific purposes case pstr when /windows server|windows (nt|20)/ 'server' when /windows (xp|vista|[78]|10)/ 'client' when /printer|print server/ 'printer' when /router/ 'router' when /firewall/ 'firewall' when /linux/ 'server' else 'device' end end |
#normalize_fusionvm_fingerprint(data) ⇒ Object
Normalize data from FusionVM fingerprints
801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 801 def normalize_fusionvm_fingerprint(data) ret = {} case data[:os] when /Windows/ ret.update(parse_windows_os_str(data[:os])) when /Linux ([^[:space:]]*) ([^[:space:]]*) .* (\(.*\))/ ret['os.product'] = "Linux" ret['host.name'] = $1 ret['os.version'] = $2 ret['os.arch'] = get_arch_from_string($3) else ret['os.product'] = data[:os] end ret['os.arch'] = data[:arch] if data[:arch] ret['host.name'] = data[:name] if data[:name] [ ret ] end |
#normalize_generic_fingerprint(data) ⇒ Object
Normalize data from generic fingerprints
822 823 824 825 826 827 828 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 822 def normalize_generic_fingerprint(data) ret = {} ret['os.product'] = data[:os_name] || data[:os] || data[:os_fingerprint] || "Unknown" ret['os.arch'] = data[:os_arch] if data[:os_arch] ret['os.certainty'] = data[:os_certainty] || 0.5 [ ret ] end |
#normalize_match(m) ⇒ Object
Normalize matches in order to handle inconsistencies between fingerprint sources and our desired usage in Metasploit. This amounts to yet more duct tape, but the situation should improve as the fingerprint sources are updated and enhanced. In the future, this method will no longer be needed (or at least, doing less and less work)
316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 316 def normalize_match(m) # Normalize os.version strings containing 'Service Pack X' to just 'SPX' if m['os.version'] and m['os.version'].index('Service Pack ') == 0 m['os.version'] = m['os.version'].gsub(/Service Pack /, 'SP') end if m['os.product'] # Normalize Apple Mac OS X to just Mac OS X if m['os.product'] =~ /^Apple Mac/ m['os.product'] = m['os.product'].gsub(/Apple Mac/, 'Mac') m['os.vendor'] ||= 'Apple' end # Normalize Sun Solaris/Sun SunOS to just Solaris/SunOS if m['os.product'] =~ /^Sun (Solaris|SunOS)/ m['os.product'] = m['os.product'].gsub(/^Sun /, '') m['os.vendor'] ||= 'Oracle' end # Normalize Microsoft Windows to just Windows to catch any stragglers if m['os.product'] =~ /^Microsoft Windows/ m['os.product'] = m['os.product'].gsub(/Microsoft Windows/, 'Windows') m['os.vendor'] ||= 'Microsoft' end # Normalize Windows Server to just Windows to match Metasploit target names if m['os.product'] =~ /^Windows Server/ m['os.product'] = m['os.product'].gsub(/Windows Server/, 'Windows') end # Normalize OS Family m = normalize_match_family(m) end m end |
#normalize_match_family(m) ⇒ Object
Normalize matches in order to ensure that an os.family entry exists if we have enough data to put one together.
356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 356 def normalize_match_family(m) # If the os.family already exists, we don't need to do anything return m if m['os.family'].present? case m['os.product'] when /Windows/ m['os.family'] = 'Windows' when /Linux/ m['os.family'] = 'Linux' when /Solaris/ m['os.family'] = 'Solaris' when /SunOS/ m['os.family'] = 'SunOS' when /AIX/ m['os.family'] = 'AIX' when /HP-UX/ m['os.family'] = 'HP-UX' when /OS X/ m['os.family'] = 'OS X' end m end |
#normalize_mbsa_fingerprint(data) ⇒ Object
Normalize data from MBSA fingerprints
596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 596 def normalize_mbsa_fingerprint(data) ret = {} # :os_match=>"Microsoft Windows Vista SP0 or SP1, Server 2008, or Windows 7 Ultimate (build 7000)" # :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"7" :os_accuracy=>"100" ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy] ret['os.family'] = data[:os_family] if data[:os_family] ret['os.vendor'] = data[:os_vendor] if data[:os_vendor] if data[:os_family] and data[:os_version] ret['os.product'] = data[:os_family] + " " + data[:os_version] end ret['host.name'] = data[:hostname] if data[:hostname] [ ret ] end |
#normalize_nessus_fingerprint(data) ⇒ Object
Normalize data from Nessus fingerprints
701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 701 def normalize_nessus_fingerprint(data) ret = {} # :os=>"Microsoft Windows 2000 Advanced Server (English)" # :os=>"Microsoft Windows 2000\nMicrosoft Windows XP" # :os=>"Linux Kernel 2.6" # :os=>"Sun Solaris 8" # :os=>"IRIX 6.5" # Nessus sometimes jams multiple OS names together with a newline. oses = data[:os].split(/\n/) if oses.length > 1 # Multiple fingerprints means Nessus wasn't really sure, reduce # the certainty accordingly ret['os.certainty'] = 0.5 else ret['os.certainty'] = 0.8 end # Since there is no confidence associated with them, the best we # can do is just take the first one. case oses.first when /^(Microsoft |)Windows/ ret.update(parse_windows_os_str(data[:os])) when /(2\.[46]\.\d+[-a-zA-Z0-9]+)/ # Look for older Linux kernel versions ret['os.product'] = "Linux" ret['os.version'] = $1 when /^Linux Kernel ([\d\.]+)(.*)/ # Look for strings like "Linux Kernel 2.6 on Ubuntu 9.10 (karmic)" # Ex: Linux Kernel 2.2 on Red Hat Linux release 6.2 (Zoot) # Ex: Linux Kernel 2.6 on Ubuntu Linux 8.04 (hardy) ret['os.product'] = "Linux" ret['os.version'] = $1 vendor = $2.to_s # Try to snag the vendor name as well if vendor =~ /on (\w+|\w+ \w+|\w+ \w+ \w+) (Linux|\d)/ ret['os.vendor'] = $1 end when /(.*) ([0-9\.]+)$/ # Then we don't necessarily know what the os is, but this fingerprint has # some version information at the end, pull it off, treat the first part # as the OS, and the rest as the version. ret['os.product'] = $1.gsub("Kernel", '').strip ret['os.version'] = $2 else # TODO: Return each OS guess as a separate match ret['os.product'] = oses.first end ret['host.name'] = data[:hname] if data[:hname] [ ret ] end |
#normalize_nexpose_fingerprint(data) ⇒ Object
Normalize data from Nexpose fingerprints
617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 617 def normalize_nexpose_fingerprint(data) ret = {} # :family=>"Windows" :certainty=>"0.85" :vendor=>"Microsoft" :product=>"Windows 7 Ultimate Edition" # :family=>"Windows" :certainty=>"0.67" :vendor=>"Microsoft" :arch=>"x86" :product=>'Windows 7' :version=>'SP1' # :family=>"Linux" :certainty=>"0.64" :vendor=>"Linux" :product=>"Linux" # :family=>"Linux" :certainty=>"0.80" :vendor=>"Ubuntu" :product=>"Linux" # :family=>"IOS" :certainty=>"0.80" :vendor=>"Cisco" :product=>"IOS" # :family=>"embedded" :certainty=>"0.61" :vendor=>"Linksys" :product=>"embedded" ret['os.certainty'] = data[:certainty] if data[:certainty] ret['os.family'] = data[:family] if data[:family] ret['os.vendor'] = data[:vendor] if data[:vendor] case data[:product] when /^Windows/ # TODO: Verify Windows CE and Windows 8 RT fingerprints # Translate the version into the representation we want case data[:version].to_s # These variants are normalized to just 'Windows <Version>' when "NT", "2000", "95", "ME", "XP", "Vista", "7", "8", "8.1" ret['os.product'] = "Windows #{data[:version]}" # Service pack in the version field should be recognized when /^SP\d+/, /^Service Pack \d+/ ret['os.product'] = data[:product] ret['os.version'] = data[:version] # No version means the version is part of the product already when nil, '' # Trim any 'Server' suffix and use as it is ret['os.product'] = data[:product].sub(/ Server$/, '') # Otherwise, we assume a Server version of Windows else ret['os.product'] = "Windows Server #{data[:version]}" end # Extract the edition string if it is present if data[:product] =~ /(XP|Vista|\d+(?:\.\d+)) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/ ret['os.edition'] = $2 end when nil, 'embedded' # Use the family or vendor name when the product is empty or 'embedded' ret['os.product'] = data[:family] unless data[:family] == 'embedded' ret['os.product'] ||= data[:vendor] ret['os.version'] = data[:version] if data[:version] else # Default to using the product name reported by Nexpose ret['os.product'] = data[:product] if data[:product] end ret['os.arch'] = get_arch_from_string(data[:arch]) if data[:arch] ret['os.arch'] ||= get_arch_from_string(data[:desc]) if data[:desc] [ ret ] end |
#normalize_nmap_fingerprint(data) ⇒ Object
Normalize data from Nmap fingerprints
565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 565 def normalize_nmap_fingerprint(data) ret = {} # :os_vendor=>"Microsoft" :os_family=>"Windows" :os_version=>"2000" :os_accuracy=>"94" ret['os.certainty'] = ( data[:os_accuracy].to_f / 100.0 ).to_s if data[:os_accuracy] if (data[:os_vendor] == data[:os_family]) ret['os.product'] = data[:os_family] else ret['os.product'] = data[:os_family] ret['os.vendor'] = data[:os_vendor] end # Nmap places the type of Windows (XP, 7, etc) into the version field if ret['os.product'] == 'Windows' and data[:os_version] ret['os.product'] = ret['os.product'] + ' ' + data[:os_version].to_s else ret['os.version'] = data[:os_version] end ret['host.name'] = data[:hostname] if data[:hostname] if ret['os.certainty'] ret['os.certainty'] = [ ret['os.certainty'].to_f, MAX_NMAP_CERTAINTY ].min.to_s end [ ret ] end |
#normalize_os ⇒ Object
Normalize the operating system fingerprints provided by various scanners (nmap, nexpose, retina, nessus, metasploit modules, and more!)
These are stored as notes (instead of directly in the os_* fields) specifically for this purpose.
The goal is to infer as much as we can about the OS of the device and the various services offered using the Recog gem and some glue logic to determine the best weights. This method can result in changes to the recorded Mdm::Host#os_name, Mdm::Host#os_flavor, Mdm::Host#os_sp, Mdm::Host#os_lang, Mdm::Host#purpose, Mdm::Host#name, Mdm::Host#arch, and the service details.
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 151 def normalize_os host = self matches = [] # Note that we're already restricting the query to this host by using # host.notes instead of Note, so don't need a host_id in the # conditions. fingerprintable_notes = self.notes.where("ntype like '%%fingerprint'") fingerprintable_notes.each do |fp_note| matches += recog_matches_for_note(fp_note) end # XXX: This hack solves the memory leak generated by self.services.each {} fingerprintable_services = self.services.where("name is not null and name != '' and info is not null and info != ''") fingerprintable_services.each do |s| matches += recog_matches_for_service(s) end # # Look for generic fingerprint.match notes that generate a match hash from modules # This handles ad-hoc os.language, host.name, etc identifications # generated_matches = self.notes.where(ntype: 'fingerprint.match') generated_matches.each do |m| next unless (m.data and m.data.kind_of?(::Hash)) matches << m.data.dup end # Normalize matches for consistency during the ranking phase matches = matches.map{ |m| normalize_match(m) } # Calculate the best OS match based on fingerprint hits match = Recog::Nizer.best_os_match(matches) # Merge and normalize the best match to the host object apply_match_to_host(match) if match # Set some sane defaults if needed host.os_name ||= 'Unknown' host.purpose ||= 'device' host.save if host.changed? end |
#normalize_qualys_fingerprint(data) ⇒ Object
Normalize data from Qualys fingerprints
762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 762 def normalize_qualys_fingerprint(data) ret = {} # :os=>"Microsoft Windows 2000" # :os=>"Windows 2003" # :os=>"Microsoft Windows XP Professional SP3" # :os=>"Ubuntu Linux" # :os=>"Cisco IOS 12.0(3)T3" # :os=>"Red-Hat Linux 6.0" case data[:os] when /Windows/ ret.update(parse_windows_os_str(data[:os])) when /^(Cisco) (IOS) (\d+[^\s]+)/ ret['os.product'] = $2 ret['os.vendor'] = $1 ret['os.version'] = $3 when /^([^\s]+) (Linux)(.*)/ ret['os.product'] = $2 ret['os.vendor'] = $1 ver = $3.to_s.strip.split(/\s+/).first if ver =~ /^\d+\./ ret['os.version'] = ver end else parts = data[:os].split(/\s+/, 3) ret['os.product'] = "Unknown" ret['os.product'] = parts[0] if parts[0] ret['os.product'] << " " + parts[1] if parts[1] ret['os.version'] = parts[2] if parts[2] end [ ret ] end |
#normalize_retina_fingerprint(data) ⇒ Object
Normalize data from Retina fingerprints
682 683 684 685 686 687 688 689 690 691 692 693 694 695 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 682 def normalize_retina_fingerprint(data) ret = {} # :os=>"Windows Server 2003 (X64), Service Pack 2" case data[:os] when /Windows/ ret.update(parse_windows_os_str(data[:os])) else # No idea what this looks like if it isn't windows. Just store # the whole thing and hope for the best. # TODO: Add examples of non-Windows results ret['os.product'] = data[:os] if data[:os] end [ ret ] end |
#normalize_scanner_fp(fp) ⇒ Object
Convert a host.os._fingerprint Note into a hash containing 'os.' and 'host.*' fields
Also includes a os.certainty which is a float from 0 - 1.00 indicating the scanner's confidence in its fingerprint. If the particular scanner does not provide such information, default to 0.80.
837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 837 def normalize_scanner_fp(fp) hits = [] return hits if not validate_fingerprint_data(fp) case fp.ntype when /^host\.os\.(.*_fingerprint)$/ pname = $1 pmeth = 'normalize_' + pname if self.respond_to?(pmeth) hits = self.send(pmeth, fp.data) else hits = normalize_generic_fingerprint(fp.data) end end hits.each {|hit| hit['os.certainty'] ||= 0.80} hits end |
#normalize_session_fingerprint(data) ⇒ Object
Normalize data from Meterpreter's client.sys.config.sysinfo()
533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 533 def normalize_session_fingerprint(data) ret = {} case data[:os] when /Windows/ ret.update(parse_windows_os_str(data[:os])) # Switch to this code block once the multi-meterpreter code review is complete =begin when /^(Windows \w+)\s*\(Build (\d+)(.*)\)/ ret['os.product'] = $1 ret['os.build'] = $2 ret['os.vendor'] = 'Microsoft' possible_sp = $3 if possible_sp =~ /Service Pack (\d+)/ ret['os.version'] = 'SP' + $1 end =end when /Linux (\d+\.\d+\.\d+\S*)\s* \((\w*)\)/ ret['os.product'] = "Linux" ret['os.version'] = $1 ret['os.arch'] = get_arch_from_string($2) else ret['os.product'] = data[:os] end ret['os.arch'] = data[:arch] if data[:arch] ret['host.name'] = data[:name] if data[:name] [ ret ] end |
#parse_windows_os_str(str) ⇒ Object
Take a windows version string and return a hash with fields suitable for Host this object's version fields. This is used as a fall-back to parse external fingerprints and should eventually be replaced by per-source mappings.
A few example strings that this will have to parse: sessions Windows XP (Build 2600, Service Pack 3). Windows .NET Server (Build 3790). Windows 2008 (Build 6001, Service Pack 1). retina Windows Server 2003 (X64), Service Pack 2 nessus Microsoft Windows 2000 Advanced Server (English) qualys Microsoft Windows XP Professional SP3 Windows 2003
Note that this list doesn't include nexpose or nmap, since they are both kind enough to give us the various strings in seperate pieces that we don't have to parse out manually.
879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 879 def parse_windows_os_str(str) ret = {} # Set some reasonable defaults for Windows ret['os.vendor'] = 'Microsoft' ret['os.product'] = 'Windows' # Determine the actual Windows product name case str when /\.NET Server/ ret['os.product'] << ' Server 2003' when / (2000|2003|2008|2012)/ ret['os.product'] << ' Server ' + $1 when / (NT (?:3\.51|4\.0))/ ret['os.product'] << ' ' + $1 when /Windows (95|98|ME|XP|Vista|[\d\.]+)/ ret['os.product'] << ' ' + $1 else # If we couldn't pull out anything specific for the flavor, just cut # off the stuff we know for sure isn't it and hope for the best ret['os.product'] = (ret['os.product'] + ' ' + str.gsub(/(Microsoft )|(Windows )|(Service Pack|SP) ?(\d+)/i, '').strip).strip # Make sure the product name doesn't include any non-alphanumeric stuff # This fixes cases where the above code leaves 'Windows XX (Build 3333,)...' ret['os.product'] = ret['os.product'].split(/[^a-zA-Z0-9 ]/).first.strip end # Take a guess at the architecture arch = get_arch_from_string(str) ret['os.arch'] = arch if arch # Extract any service pack value in the string if str =~ /(Service Pack|SP) ?(\d+)/i ret['os.version'] = "SP#{$2}" end # Extract any build ID found in the string if str =~ /build (\d+)/i ret['os.build'] = $1 end # Extract the OS edition if available if str =~ /(\d+|\d+\.\d+) (\w+|\w+ \w+|\w+ \w+ \w+) Edition/ ret['os.edition'] = $2 else if str =~ /(Professional|Enterprise|Pro|Home|Start|Datacenter|Web|Storage|MultiPoint)/ ret['os.edition'] = $1 end end ret end |
#recog_matches_for_note(note) ⇒ Array<Hash>
Recog matches for the fingerprint in note
.
244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 244 def recog_matches_for_note(note) # Skip notes that are missing the correct structure or have been blacklisted return [] if not validate_fingerprint_data(note) # # These rules define the relationship between fingerprint note keys # and specific Recog databases for detailed matching. Notes that do # not match a rule are passed to the generic matcher. # fingerprint_note_match_keys = { 'smb.fingerprint' => { :native_os => [ 'smb.native_os' ], :native_lm => [ 'smb.native_lm' ], }, 'http.fingerprint' => { :header_server => [ 'http_header.server', 'apache_os' ], :header_set_cookie => [ 'http_header.cookie' ], :header_www_authenticate => [ 'http_header.wwwauth' ], # TODO: Candidates for future Recog support # :content => 'http_body' # :code => 'http_response_code' # :message => 'http_response_message' } } matches = [] # Look for a specific Recog database for this type and data key if fingerprint_note_match_keys.has_key?( note.ntype ) fingerprint_note_match_keys[ note.ntype ].each_pair do |k,rdbs| if note.data.has_key?(k) rdbs.each do |rdb| res = Recog::Nizer.match(rdb, note.data[k]) matches << res if res end end end else # Add all generic match results to the overall match array normalize_scanner_fp(note).each do |m| next unless m matches << m end end matches end |
#recog_matches_for_service(s) ⇒ Array<Hash>
Recog matches for the s
service.
199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 199 def recog_matches_for_service(s) # # We assume that the service.info field contains certain types of probe # replies and associate these with one or more Recog databases. The mapping # of service.name to a specific database only fits into so many places and # Mdm currently serves that role. # service_match_keys = { # TODO: Implement smb.generic fingerprint database # 'smb' => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match # 'netbios' => [ 'smb.generic' ], # Distinct from smb.fingerprint, use os.certainty to choose best match 'ssh' => [ 'ssh.banner' ], # Recog expects just the vendor string, not the protocol version 'http' => [ 'http_header.server', 'apache_os'], # The 'Apache' fingerprints try to infer OS/distribution from the extra information in the Server header 'https' => [ 'http_header.server', 'apache_os'], # XXX: verify vmware esx(i) case on https (TODO: normalize https to http, track SSL elsewhere, such as a new set of fields) 'snmp' => [ 'snmp.sys_description' ], 'telnet' => [ 'telnet.banner' ], 'smtp' => [ 'smtp.banner' ], 'imap' => [ 'imap4.banner' ], # Metasploit reports 143/993 as imap (TODO: normalize imap to imap4) 'pop3' => [ 'pop3.banner' ], # Metasploit reports 110/995 as pop3 'nntp' => [ 'nntp.banner' ], 'ftp' => [ 'ftp.banner' ], 'ssdp' => [ 'http_header.server' ] } matches = [] return matches unless service_match_keys.has_key?(s.name) service_match_keys[s.name].each do |rdb| = s.info if self.respond_to?("service_banner_recog_filter_#{s.name}") = self.send("service_banner_recog_filter_#{s.name}", ) end res = Recog::Nizer.match(rdb, ) matches << res if res end matches end |
#sanitize(text) ⇒ Object
Ensure that the host attribute is using ascii safe text and escapes any other byte value.
526 527 528 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 526 def sanitize(text) Rex::Text.ascii_safe_hex(text) end |
#service_banner_recog_filter_ssh(banner) ⇒ Object
Recog assumes that the protocol version of the SSH banner has been removed
381 382 383 384 385 386 387 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 381 def () if =~ /^SSH-\d+\.\d+-(.*)/ $1 else end end |
#validate_fingerprint_data(fp) ⇒ Object
Determine if the fingerprint data is readable. If not, it nearly always means that there was a problem with the YAML or the Marshal'ed data, so let's log that for later investigation.
295 296 297 298 299 300 301 302 303 304 305 306 307 |
# File 'lib/mdm/host/operating_system_normalization.rb', line 295 def validate_fingerprint_data(fp) if fp.data.kind_of?(Hash) and !fp.data.empty? return true elsif fp.ntype == "postgresql.fingerprint" # Special case postgresql.fingerprint; it's always a string, # and should not be used for OS fingerprinting (yet), so # don't bother logging it. TODO: fix os fingerprint finding, this # name collision seems silly. return false else return false end end |