Module: ThreatDetector

Defined in:
lib/threat_detector.rb,
lib/threat_detector/cli.rb,
lib/threat_detector/cache.rb,
lib/threat_detector/search.rb,
lib/threat_detector/scraper.rb,
lib/threat_detector/utility.rb,
lib/threat_detector/version.rb,
lib/threat_detector/downloader.rb

Overview

Check IPs, Hosts, Networks or URLs for possible threats using feeds from ThreatFeeds.io

ThreatDetector creates a local cache of threats by downloading ThreatFeeds.io feeds, parsing them, and storing them in [Trie](en.wikipedia.org/wiki/Trie) structures to allow for efficient and fast searches in this huge database of threats.

ThreatDetector tries to be smart in understanding that the database may not necessarily contain the exact search term, and often, employs logic in deciding whether a given IP, network, host or URL is a threat. For example, in addition to directly matching threats in this database, it knows that:

  • An IP is a threat if it belongs to an identified network.

  • A network is a threat if it belongs to a wider identified network.

  • An IP is a threat if the hostname it resolves to is marked as a threat.

  • A host can be a threat if the IP it resolves to is marked as a threat.

  • A url can be a threat if its hostname or IP is marked as a threat.

  • etc.

Furthermore, any threat that is identified is returned with a reason for the same, allowing you to also identify why a particular IP, host, network or URL was marked as a threat. You can, optionally, choose to disable smarter searching (only match directly with database), disable resolving IP/host to corresponding host/IP for matching, etc.

Defined Under Namespace

Modules: Utility Classes: CLI, Cache, Downloader, Error, Scraper, Search

Constant Summary collapse

ROOT =

Root path for this gem.

File.dirname(File.dirname(__FILE__))
DEFAULT_HOME =

Directory where downloaded feeds and cache will be saved by default. You can specify another directory to save the feeds and cache, by using `:working_directory` option when instantiating Scraper or Downloader. Note that, you must also specify this option when instanting Search in this case.

File.join(ENV['HOME'], '.threat_detector')
DEFAULT_CONFIG =

Path to default scraping configuration YAML file. This configuration is used by Scraper#parse, and can be changed to another file by using `:feeds_config_path` option when instantiating Scraper or Downloader

File.join(ROOT, 'feeds.yaml')
USER_AGENT =

Default User Agent for scrapers.

'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
DEFAULT_CURL_OPTIONS =

Default CURL options used by scrapers, used by Utility#fetch_page They can be overridden by passing relevant options when fetching the page. These options are passed to Curb library.

{
  timeout: 30,
  encoding: 'gzip',
  max_redirects: 10,
  follow_location: true,
  useragent: USER_AGENT
}.freeze
VERSION =

Current version for the gem.

'0.1.0'

Class Method Summary collapse

Class Method Details

.build_cache(options = {}, &block) ⇒ Object

Helper method to instantiate Cache with given options, and run the cache builder afterwards.


96
97
98
# File 'lib/threat_detector.rb', line 96

def self.build_cache(options = {}, &block)
  ThreatDetector::Cache.new(options).run(&block)
end

.download(options = {}, &block) ⇒ Object

Helper method to instantiate Downloader with given options, and run the downloader afterwards.


87
88
89
# File 'lib/threat_detector.rb', line 87

def self.download(options = {}, &block)
  ThreatDetector::Downloader.new(options).run(&block)
end

.search(keys, options = {}, &block) ⇒ Object

Helper method to instantiate Search with given options, and run the search on specified keys afterwards.

Parameters:

  • keys (Array{String,#read})

    search terms or files with search terms

  • opts (Hash)

    a customizable set of options

  • opts (Hash)

    options received from the user


111
112
113
# File 'lib/threat_detector.rb', line 111

def self.search(keys, options = {}, &block)
  ThreatDetector::Search.new(options).process(*keys, &block)
end