Class: ZyCrawler

Inherits:
Object
  • Object
show all
Defined in:
lib/zy_crawler.rb

Overview

ZyCrawler is from Jason zhao’s YzCrawler

Instance Method Summary collapse

Constructor Details

#initialize ⇒ ZyCrawler

Returns a new instance of ZyCrawler.



7
8
9
10
11
12
 
# File 'lib/zy_crawler.rb', line 7

def initialize
  @argument_parser = CommandLineArgumentParser.new
  @argument_parser.parse_arguments
  @spider = Spider.new
  @url_store = URLStore.new(@argument_parser.url_file)
end

Instance Method Details

#crawl ⇒ Object 



14
15
16
17
18
19
20
21
22
 
# File 'lib/zy_crawler.rb', line 14

def crawl
  if @argument_parser.crawl_type == CommandLineArgumentParser::WEB_CRAWLER
    @spider.crawl_web(@url_store.get_urls,
                      @argument_parser.crawl_depth,
                      @argument_parser.page_limit)
  else
    @spider.crawl_domain(@url_store.firt_url, @argument_parser.page_limit)
  end
end