Class: Scraper

Inherits:
Object
  • Object
show all
Defined in:
lib/scrapin-a-livin.rb

Overview

Scrape the listings for the provided URL’s

Instance Method Summary collapse

Constructor Details

#initialize(urls) ⇒ Scraper

Returns a new instance of Scraper.



9
10
11
12
 
# File 'lib/scrapin-a-livin.rb', line 9

def initialize(urls)

     @urls = urls
end

Instance Method Details

#fetch ⇒ Object

Fetch the data for the provided urls

Parameters:

  • urls (Array, #read) —

    array of query urls



17
18
19
20
21
22
23
24
25
26
 
# File 'lib/scrapin-a-livin.rb', line 17

def fetch

     @urls.each { |u|

          # Write the listings
          get_listings(u).each { |l|
               write_listing(l)
          }
     }
end

#get_listings(url) ⇒ Object

Get the job listings for the url

Parameters:

  • url (String, #read) —

    the url to query

Returns:

  • the job listings

Raises:

  • ArgumentError



33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/scrapin-a-livin.rb', line 33

def get_listings(url)

     # Check the url for Dice
     if url.index("http://seeker.dice.com/") == 0
          DiceSearch.get_listings(url).sort_by{ |l| l.date }

     # Check the url for Hotjobs
     elsif url.index("http://hotjobs.yahoo.com/") == 0
          HotjobsSearch.get_listings(url).sort_by{ |l| l.date }
     
     # Else Error
     else
          raise ArgumentError.new "Url invalid or not supported"
     end
end

#write_listing(listing) ⇒ Object

Write the listing at the specified path

Parameters:

  • path (String, #read) —

    the base path for the listing



53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
 
# File 'lib/scrapin-a-livin.rb', line 53

def write_listing(listing)

     date = listing.add_date(JOBS_DIR)
     state = listing.add_state(date)
     city = listing.add_city(state)
     company = listing.add_company(city)
     name = listing.add_name(company)

     puts name

     # Write the listing to the directory
     File.open("#{name}.html", "w"){ |file|
          file << `curl #{listing.link}`
          file << "<!--\n"
          file << listing
          file << "-->"
     }
end