Class: CraigScrape::Posting

Inherits:
Scraper
  • Object
show all
Defined in:
lib/posting.rb

Overview

Posting represents a fully downloaded, and parsed, Craigslist post. This class is generally returned by the listing scrape methods, and contains the post summaries for a specific search url, or a general listing category

Constant Summary collapse

POST_DATE = 
/Date:[^\d]*((?:[\d]{2}|[\d]{4})\-[\d]{1,2}\-[\d]{1,2}[^\d]+[\d]{1,2}\:[\d]{1,2}[ ]*[AP]M[^a-z]+[a-z]+)/i
LOCATION = 
/Location\:[ ]+(.+)/
HEADER_LOCATION = 
/^.+[ ]*\-[ ]*[\$]?[\d]+[ ]*\((.+)\)$/
POSTING_ID = 
/PostingID\:[ ]*([\d]+)/
REPLY_TO = 
/(.+)/
PRICE = 
/((?:^\$[\d]+(?:\.[\d]{2})?)|(?:\$[\d]+(?:\.[\d]{2})?))/
USERBODY_PARTS =

NOTE: we implement the (?:) to first check the ‘old’ style format, and then the ‘new style’ (As of 12/03’s parse changes)

/^(.+)\<div id\=\"userbody\">(.+)\<br[ ]*[\/]?\>\<br[ ]*[\/]?\>(.+)\<\/div\>(.+)$/m
HTML_HEADER = 
/^(.+)\<div id\=\"userbody\">/m
IMAGE_SRC = 
/\<im[a]?g[e]?[^\>]*src=(?:\'([^\']+)\'|\"([^\"]+)\"|([^ ]+))[^\>]*\>/
REQUIRED_FIELDS =

This is used to determine if there’s a parse error

%w(contents posting_id post_time header title full_section)
XPATH_USERBODY = 
"//*[@id='userbody']"
XPATH_BLURBS = 
"//ul[@class='blurbs']"
XPATH_PICS = 
"//*[@class='tn']/a/@href"
XPATH_REPLY_TO = 
"//*[@class='dateReplyBar']/small/a"

Constants inherited from Scraper

Scraper::HTML_ENCODING, Scraper::HTML_TAG, Scraper::HTTP_HEADERS, Scraper::URL_PARTS

Instance Attribute Summary collapse

Attributes inherited from Scraper

#url

Instance Method Summary collapse

Methods inherited from Scraper

#downloaded?, #uri

Constructor Details

#initialize(*args) ⇒ Posting

Create a new Post via a url (String), or supplied parameters (Hash)



40
41
42
43
44
45
46
47
48
49
50
51
52
 
# File 'lib/posting.rb', line 40

def initialize(*args)
  super(*args)

  # Validate that required fields are present, at least - if we've downloaded it from a url
  if args.first.kind_of? String and is_active_post?
    unparsed_fields = REQUIRED_FIELDS.find_all{|f| 
      val = send(f)
      val.nil? or (val.respond_to? :length and val.length == 0)
    } 
    parse_error! unparsed_fields unless unparsed_fields.empty?
  end  

end

Instance Attribute Details

#href ⇒ Object (readonly)

This is really just for testing, in production use, uri.path is a better solution



37
38
39
 
# File 'lib/posting.rb', line 37

def href
  @href
end

Instance Method Details

#contents ⇒ Object

String, The full-html contents of the post



134
135
136
137
138
139
140
141
 
# File 'lib/posting.rb', line 134

def contents
  unless @contents
    @contents = user_body if html_source
    @contents = he_decode(@contents).strip if @contents
  end
  
  @contents
end

#contents_as_plain ⇒ Object

Returns the post contents with all html tags removed



305
306
307
 
# File 'lib/posting.rb', line 305

def contents_as_plain
  strip_html contents
end

#deleted_by_author? ⇒ Boolean

Returns true if this Post was parsed, and represents a ‘This posting has been deleted by its author.’ notice

Returns:

  • (Boolean) 


223
224
225
226
227
228
229
 
# File 'lib/posting.rb', line 223

def deleted_by_author?
  @deleted_by_author = (
    system_post? and header_as_plain == "This posting has been deleted by its author."
  ) if @deleted_by_author.nil?
  
  @deleted_by_author
end

#flagged_for_removal? ⇒ Boolean

Returns true if this Post was parsed, and merely a ‘Flagged for Removal’ page

Returns:

  • (Boolean) 


214
215
216
217
218
219
220
 
# File 'lib/posting.rb', line 214

def flagged_for_removal?
  @flagged_for_removal = (
    system_post? and header_as_plain == "This posting has been flagged for removal"
  ) if @flagged_for_removal.nil?
  
  @flagged_for_removal
end

#full_section ⇒ Object

Array, hierarchial representation of the posts section



77
78
79
80
81
82
83
84
85
86
87
 
# File 'lib/posting.rb', line 77

def full_section
  unless @full_section
    @full_section = []
    
    (html_head / "*[@class='bchead']//a").each do |a|
      @full_section << he_decode(a.inner_html) unless a['id'] and a['id'] == 'ef'
    end if html_head
  end

  @full_section
end

#has_img? ⇒ Boolean

true if post summary has ‘img(s)’. ‘imgs’ are different then pics, in that the resource is not hosted on craigslist’s server. This is always able to be pulled from the listing post-summary, and should never cause an additional page load

Returns:

  • (Boolean) 


282
283
284
 
# File 'lib/posting.rb', line 282

def has_img?
  img_types.include? :img
end

#has_pic? ⇒ Boolean

true if post summary has ‘pic(s)’. ‘pics’ are different then imgs, in that craigslist is hosting the resource on craigslist’s servers This is always able to be pulled from the listing post-summary, and should never cause an additional page load

Returns:

  • (Boolean) 


288
289
290
 
# File 'lib/posting.rb', line 288

def has_pic?
  img_types.include? :pic
end

#has_pic_or_img? ⇒ Boolean

true if post summary has either the img or pic label This is always able to be pulled from the listing post-summary, and should never cause an additional page load

Returns:

  • (Boolean) 


294
295
296
 
# File 'lib/posting.rb', line 294

def has_pic_or_img?
  img_types.length > 0
end

#header ⇒ Object

String, The contents of the item’s html body heading



56
57
58
59
60
61
62
63
 
# File 'lib/posting.rb', line 56

def header
  unless @header
    h2 = html_head.at 'h2' if html_head
    @header = he_decode h2.inner_html if h2
  end
  
  @header
end

#header_as_plain ⇒ Object

Returns the header with all html tags removed. Granted, the header should usually be plain, but in the case of a ‘system_post’ we may get tags in here



311
312
313
 
# File 'lib/posting.rb', line 311

def header_as_plain
  strip_html header
end

#images ⇒ Object

Array, urls of the post’s images that are not hosted on craigslist



181
182
183
184
185
186
187
188
189
190
 
# File 'lib/posting.rb', line 181

def images
  # Keep in mind that when users post html to craigslist, they're often not posting wonderful html...
  @images = ( 
    contents ? 
      contents.scan(IMAGE_SRC).collect{ |a| a.find{|b| !b.nil? } } :
      [] 
  ) unless @images
  
  @images
end

#img_types ⇒ Object

Array, which image types are listed for the post. This is always able to be pulled from the listing post-summary, and should never cause an additional page load



265
266
267
268
 
# File 'lib/posting.rb', line 265

def img_types
  @img_types || [ (images.length > 0) ? :img : nil, 
    (pics.length > 0) ? :pic : nil ].compact
end

#is_active_post? ⇒ Boolean

This is mostly used to determine if the post should be checked for parse errors. Might be useful for someone else though

Returns:

  • (Boolean) 


323
324
325
 
# File 'lib/posting.rb', line 323

def is_active_post?
  [flagged_for_removal?, posting_has_expired?, deleted_by_author?].none?
end

#label ⇒ Object

Returns The post label. The label would appear at first glance to be indentical to the header - but its not. The label is cited on the listings pages, and generally includes everything in the header - with the exception of the location. Sometimes there’s additional information ie. ‘(map)’ on rea listings included in the header, that aren’t to be listed in the label This is also used as a bandwidth shortcut for the craigwatch program, and is a guaranteed identifier for the post, that won’t result in a full page load from the post’s url.



253
254
255
256
257
258
259
260
261
 
# File 'lib/posting.rb', line 253

def label
  unless @label or system_post?
    @label = header
    
    @label = $1 if location and /(.+?)[ ]*\(#{location}\).*?$/.match @label
  end
  
  @label
end

#location ⇒ Object

String, the location of the item, as best could be parsed



144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
 
# File 'lib/posting.rb', line 144

def location
  if @location.nil? and html
   
    if html.at_xpath(XPATH_BLURBS)
      # This is the post-12/3/12 style:
      @location = $1 if html.xpath(XPATH_BLURBS).first.children.any?{|c| 
        LOCATION.match c.content}
    elsif craigslist_body
      # Location (when explicitly defined):
      cursor = craigslist_body.at 'ul' unless @location

      # This is the legacy style:
      # Note: Apa section includes other things in the li's (cats/dogs ok fields)
      cursor.children.each do |li|
        if LOCATION.match li.inner_html
          @location = he_decode($1) and break
          break
        end
      end if cursor

      # Real estate listings can work a little different for location:
      unless @location
        cursor = craigslist_body.at 'small'
        cursor = cursor.previous until cursor.nil? or cursor.text?
        
        @location = he_decode(cursor.to_s.strip) if cursor
      end
      
      # So, *sometimes* the location just ends up being in the header, I don't know why:
      @location = $1 if @location.nil? and HEADER_LOCATION.match header
    end
  end
  
  @location
end

#pics ⇒ Object

Array, urls of the post’s craigslist-hosted images



193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
 
# File 'lib/posting.rb', line 193

def pics
  unless @pics
    @pics = []
    
    if html 
      if html.at_xpath(XPATH_PICS)
        @pics = html.xpath(XPATH_PICS).collect(&:value)
      elsif craigslist_body
        # This is the pre-12/3/12 style:
        # Now let's find the craigslist hosted images:
        img_table = (craigslist_body / 'table').find{|e| e.name == 'table' and e[:summary] == 'craigslist hosted images'}
      
        @pics = (img_table / 'img').collect{|i| i[:src]} if img_table
      end
    end
  end
  
  @pics
end

#post_date ⇒ Object

Reflects only the date portion of the posting. Does not include hours/minutes. This is useful when reflecting the listing scrapes, and can be safely used if you wish conserve bandwidth by not pulling an entire post from a listing scrape.



242
243
244
245
246
 
# File 'lib/posting.rb', line 242

def post_date
  @post_date = post_time.to_date unless @post_date or post_time.nil?
  
  @post_date
end

#post_time ⇒ Object

Time, reflects the full timestamp of the posting



105
106
107
108
109
110
111
112
113
 
# File 'lib/posting.rb', line 105

def post_time
  unless @post_time
    cursor = html_head.at 'hr' if html_head
    cursor = cursor.next until cursor.nil? or POST_DATE.match cursor.to_s
    @post_time = Time.zone.parse $1 if $1
  end
  
  @post_time
end

#posting_has_expired? ⇒ Boolean

Returns true if this Post was parsed, and represents a ‘This posting has expired.’ notice

Returns:

  • (Boolean) 


232
233
234
235
236
237
238
 
# File 'lib/posting.rb', line 232

def posting_has_expired?
  @posting_has_expired = (
    system_post? and header_as_plain == "This posting has expired."
  ) if @posting_has_expired.nil?
  
  @posting_has_expired
end

#posting_id ⇒ Object

Integer, Craigslist’s unique posting id



116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
 
# File 'lib/posting.rb', line 116

def posting_id
  if @posting_id 

  elsif USERBODY_PARTS.match html_source
    # Old style:
    html_footer = $4
    cursor = Nokogiri::HTML html_footer, nil, HTML_ENCODING 
    cursor = cursor.next until cursor.nil? or 
    @posting_id = $1.to_i if POSTING_ID.match html_footer.to_s
  else
    # Post 12/3
    @posting_id = $1.to_i if POSTING_ID.match html.xpath("//*[@class='postingidtext']").to_s
  end

  @posting_id
end

#price ⇒ Object

Returns the best-guess of a price, judging by the label’s contents. Price is available when pulled from the listing summary and can be safely used if you wish conserve bandwidth by not pulling an entire post from a listing scrape.



300
301
302
 
# File 'lib/posting.rb', line 300

def price
  $1.tr('$','').to_f if header and PRICE.match header
end

#reply_to ⇒ Object

String, represents the post’s reply-to address, if listed



90
91
92
93
94
95
96
97
98
99
100
101
102
 
# File 'lib/posting.rb', line 90

def reply_to
  unless @reply_to
    if html.at_xpath(XPATH_REPLY_TO)
      @reply_to = html.at_xpath(XPATH_REPLY_TO).content
    else
      cursor = html_head.at 'hr' if html_head
      cursor = cursor.next until cursor.nil? or cursor.name == 'a'
      @reply_to = $1 if cursor and REPLY_TO.match he_decode(cursor.inner_html)
    end
  end
  
  @reply_to
end

#section ⇒ Object

Retrieves the most-relevant craigslist ‘section’ of the post. This is generally the same as full_section.last. However, this (sometimes/rarely) conserves bandwidth by pulling this field from the listing post-summary



272
273
274
275
276
277
278
 
# File 'lib/posting.rb', line 272

def section
  unless @section
    @section = full_section.last if full_section  
  end
  
  @section
end

#system_post? ⇒ Boolean

Some posts (deleted_by_author, flagged_for_removal) are common template posts that craigslist puts up in lieu of an original This returns true or false if that case applies

Returns:

  • (Boolean) 


317
318
319
 
# File 'lib/posting.rb', line 317

def system_post?
  [contents,posting_id,post_time,title].all?{|f| f.nil?}
end

#title ⇒ Object

String, the item’s title



66
67
68
69
70
71
72
73
74
 
# File 'lib/posting.rb', line 66

def title
  unless @title
    title_tag = html_head.at 'title' if html_head
    @title = he_decode title_tag.inner_html if title_tag
    @title = nil if @title and @title.length == 0
  end

  @title
end