Module: NewsCrawler

Defined in:

lib/news_crawler/url_helper.rb,
lib/news_crawler/config.rb,
lib/news_crawler/nc_logger.rb,
lib/news_crawler/downloader.rb,
lib/news_crawler/crawler_module.rb,
lib/news_crawler/storage/raw_data.rb,
lib/news_crawler/storage/url_queue.rb,
lib/news_crawler/storage/yaml_stor.rb,
lib/news_crawler/processing/structure_analysis.rb,
lib/news_crawler/storage/raw_data/mongo_storage.rb,
lib/news_crawler/storage/url_queue/mongo_storage.rb,
lib/news_crawler/storage/yaml_stor/mongo_storage.rb,
lib/news_crawler/storage/raw_data/raw_data_engine.rb,
lib/news_crawler/storage/url_queue/url_queue_error.rb,
lib/news_crawler/link_selector/same_domain_selector.rb,
lib/news_crawler/storage/url_queue/url_queue_engine.rb,
lib/news_crawler/storage/yaml_stor/yaml_stor_engine.rb

Overview

– NewsCrawler - a website crawler

Copyright © 2013 - Hà Quang Dương <contact@haqduong.net>

This file is part of NewsCrawler.

NewsCrawler is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

NewsCrawler is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with NewsCrawler. If not, see <www.gnu.org/licenses/>. ++

Defined Under Namespace

Modules: CrawlerModule, LinkSelector, Processing, Storage, URLHelper Classes: CrawlerConfig, Downloader, NCLogger