Module: NewsCrawler::Storage::RawData

Defined in:
lib/news_crawler/storage/raw_data.rb,
lib/news_crawler/storage/raw_data/mongo_storage.rb,
lib/news_crawler/storage/raw_data/raw_data_engine.rb

Overview

store raw data from website

Defined Under Namespace

Classes: MongoStorage, RawDataEngine

Class Method Summary collapse

Class Method Details

.add(url, body) ⇒ Object

Add entry to raw data collection

Parameters:

  • url (String)
  • body (String) 


53
54
55
 
# File 'lib/news_crawler/storage/raw_data.rb', line 53

def add(url, body)
  @engine.add(url, body)
end

.clear ⇒ Object 



68
69
70
 
# File 'lib/news_crawler/storage/raw_data.rb', line 68

def clear
  @engine.clear
end

.count ⇒ Object 



64
65
66
 
# File 'lib/news_crawler/storage/raw_data.rb', line 64

def count
  @engine.count
end

.find_by_url(url) ⇒ String?

Find document with correspond url

Parameters:

  • url (String)

Returns:

  • (String, nil) 


60
61
62
 
# File 'lib/news_crawler/storage/raw_data.rb', line 60

def find_by_url(url)
  @engine.find_by_url url
end

.set_engine(engine, *opts) ⇒ Object

Set RawData storage engine

Parameters:

  • engine (Symbol, Object) —

    specify database engine, pass an object for custom engine

  • opts (Hash) —

    options pass to engine This can be

    • ‘:mongo`, `:mongodb` for MongoDB backend



38
39
40
41
42
43
44
45
46
47
48
 
# File 'lib/news_crawler/storage/raw_data.rb', line 38

def set_engine(engine, *opts)
  if engine.respond_to? :intern
    engine = engine.intern
  end
  engine_class = RawDataEngine.get_engines[engine]
  if engine_class
    @engine = engine_class.new(*opts)
  else
    @engine = engine
  end
end