Class: Feedzirra::Feed
- Inherits:
-
Object
- Object
- Feedzirra::Feed
- Defined in:
- lib/feedzirra/feed.rb
Constant Summary collapse
- USER_AGENT =
"feedzirra http://github.com/pauldix/feedzirra/tree/master"
Class Method Summary collapse
-
.add_common_feed_element(element_tag, options = {}) ⇒ Object
Makes all registered feeds types look for the passed in element to parse.
-
.add_common_feed_elements(element_tag, options = {}) ⇒ Object
Makes all registered feeds types look for the passed in elements to parse.
-
.add_common_feed_entry_element(element_tag, options = {}) ⇒ Object
Makes all registered entry types look for the passed in element to parse.
-
.add_common_feed_entry_elements(element_tag, options = {}) ⇒ Object
Makes all registered entry types look for the passed in elements to parse.
-
.add_feed_class(klass) ⇒ Object
Adds a new feed parsing class that will be used for parsing.
-
.add_feed_to_multi(multi, feed, feed_queue, responses, options) ⇒ Object
An abstraction for adding a feed by a Feed object to the passed Curb::multi stack.
-
.add_url_to_multi(multi, url, url_queue, responses, options) ⇒ Object
An abstraction for adding a feed by URL to the passed Curb::multi stack.
-
.call_on_each_feed_entry(method, *parameters) ⇒ Object
Call a method on all feed entries classes.
-
.decode_content(c) ⇒ Object
Decodes the XML document if it was compressed.
-
.determine_feed_parser_for_xml(xml) ⇒ Object
Determines the correct parser class to use for parsing the feed.
-
.etag_from_header(header) ⇒ Object
Determines the etag from the request headers.
-
.feed_classes ⇒ Object
Provides a list of registered feed parsing classes.
-
.fetch_and_parse(urls, options = {}) ⇒ Object
Fetches and returns the parsed XML for each URL provided.
-
.fetch_raw(urls, options = {}) ⇒ Object
Fetches and returns the raw XML for each URL provided.
-
.last_modified_from_header(header) ⇒ Object
Determines the last modified date from the request headers.
-
.parse(xml, &block) ⇒ Object
Takes a raw XML feed and attempts to parse it.
-
.setup_easy(curl, options) ⇒ Object
Setup curl from options.
-
.update(feeds, options = {}) ⇒ Object
Updates each feed for each Feed object provided.
Class Method Details
.add_common_feed_element(element_tag, options = {}) ⇒ Object
Makes all registered feeds types look for the passed in element to parse. This is actually just a call to element (a SAXMachine call) in the class.
Parameters
- element_tag<String>
-
The element tag
- options<Hash>
-
Valid keys are same as with SAXMachine
58 59 60 61 62 |
# File 'lib/feedzirra/feed.rb', line 58 def self.add_common_feed_element(element_tag, = {}) feed_classes.each do |k| k.element element_tag, end end |
.add_common_feed_elements(element_tag, options = {}) ⇒ Object
Makes all registered feeds types look for the passed in elements to parse. This is actually just a call to elements (a SAXMachine call) in the class.
Parameters
- element_tag<String>
-
The element tag
- options<Hash>
-
Valid keys are same as with SAXMachine
70 71 72 73 74 |
# File 'lib/feedzirra/feed.rb', line 70 def self.add_common_feed_elements(element_tag, = {}) feed_classes.each do |k| k.elements element_tag, end end |
.add_common_feed_entry_element(element_tag, options = {}) ⇒ Object
Makes all registered entry types look for the passed in element to parse. This is actually just a call to element (a SAXMachine call) in the class.
Parameters
- element_tag<String>
- options<Hash>
-
Valid keys are same as with SAXMachine
82 83 84 |
# File 'lib/feedzirra/feed.rb', line 82 def self.add_common_feed_entry_element(element_tag, = {}) call_on_each_feed_entry :element, element_tag, end |
.add_common_feed_entry_elements(element_tag, options = {}) ⇒ Object
Makes all registered entry types look for the passed in elements to parse. This is actually just a call to element (a SAXMachine call) in the class.
Parameters
- element_tag<String>
- options<Hash>
-
Valid keys are same as with SAXMachine
92 93 94 |
# File 'lib/feedzirra/feed.rb', line 92 def self.add_common_feed_entry_elements(element_tag, = {}) call_on_each_feed_entry :elements, element_tag, end |
.add_feed_class(klass) ⇒ Object
Adds a new feed parsing class that will be used for parsing.
Parameters
- klass<Constant>
-
The class/constant that you want to register.
Returns
A updated array of feed parser class names.
40 41 42 |
# File 'lib/feedzirra/feed.rb', line 40 def self.add_feed_class(klass) feed_classes.unshift klass end |
.add_feed_to_multi(multi, feed, feed_queue, responses, options) ⇒ Object
An abstraction for adding a feed by a Feed object to the passed Curb::multi stack.
Parameters
- multi<Curl::Multi>
-
The Curl::Multi object that the request should be added too.
- feed<Feed>
-
A feed object that you would like to be fetched.
- url_queue<Array>
-
An array of feed objects that are queued for request.
- responses<Hash>
-
Existing responses that you want the response from the request added to.
- feeds<String>
-
or <Array> A single feed object, or an array of feed objects.
- options<Hash>
-
Valid keys for this argument as as followed:
-
:on_success - Block that gets executed after a successful request.
-
:on_failure - Block that gets executed after a failed request.
-
all parameters defined in setup_easy
-
Returns
The updated Curl::Multi object with the request details added to it’s stack.
325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 |
# File 'lib/feedzirra/feed.rb', line 325 def self.add_feed_to_multi(multi, feed, feed_queue, responses, ) easy = Curl::Easy.new(feed.feed_url) do |curl| setup_easy curl, curl.headers["If-Modified-Since"] = feed.last_modified.httpdate if feed.last_modified curl.headers["If-Modified-Since"] = [:if_modified_since] if [:if_modified_since] && (!feed.last_modified || (Time.parse([:if_modified_since].to_s) > feed.last_modified)) curl.headers["If-None-Match"] = feed.etag if feed.etag curl.on_success do |c| begin add_feed_to_multi(multi, feed_queue.shift, feed_queue, responses, ) unless feed_queue.empty? updated_feed = Feed.parse(c.body_str){ || warn "Error while parsing [#{feed.feed_url}] #{}" } updated_feed.feed_url = c.last_effective_url updated_feed.etag = etag_from_header(c.header_str) updated_feed.last_modified = last_modified_from_header(c.header_str) feed.update_from_feed(updated_feed) responses[feed.feed_url] = feed [:on_success].call(feed) if .has_key?(:on_success) rescue Exception => e [:on_failure].call(feed, c.response_code, c.header_str, c.body_str) if .has_key?(:on_failure) end end curl.on_failure do |c, err| add_feed_to_multi(multi, feed_queue.shift, feed_queue, responses, ) unless feed_queue.empty? response_code = c.response_code if response_code == 304 # it's not modified. this isn't an error condition responses[feed.feed_url] = feed [:on_success].call(feed) if .has_key?(:on_success) else responses[feed.url] = c.response_code [:on_failure].call(feed, c.response_code, c.header_str, c.body_str) if .has_key?(:on_failure) end end end multi.add(easy) end |
.add_url_to_multi(multi, url, url_queue, responses, options) ⇒ Object
An abstraction for adding a feed by URL to the passed Curb::multi stack.
Parameters
- multi<Curl::Multi>
-
The Curl::Multi object that the request should be added too.
- url<String>
-
The URL of the feed that you would like to be fetched.
- url_queue<Array>
-
An array of URLs that are queued for request.
- responses<Hash>
-
Existing responses that you want the response from the request added to.
- feeds<String> or <Array>
-
A single feed object, or an array of feed objects.
- options<Hash>
-
Valid keys for this argument as as followed:
-
:on_success - Block that gets executed after a successful request.
-
:on_failure - Block that gets executed after a failed request.
-
all parameters defined in setup_easy
-
Returns
The updated Curl::Multi object with the request details added to it’s stack.
269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 |
# File 'lib/feedzirra/feed.rb', line 269 def self.add_url_to_multi(multi, url, url_queue, responses, ) easy = Curl::Easy.new(url) do |curl| setup_easy curl, curl.headers["If-Modified-Since"] = [:if_modified_since].httpdate if .has_key?(:if_modified_since) curl.headers["If-None-Match"] = [:if_none_match] if .has_key?(:if_none_match) curl.on_success do |c| add_url_to_multi(multi, url_queue.shift, url_queue, responses, ) unless url_queue.empty? xml = decode_content(c) klass = determine_feed_parser_for_xml(xml) if klass begin feed = klass.parse(xml, Proc.new{|| warn "Error while parsing [#{url}] #{}" }) feed.feed_url = c.last_effective_url feed.etag = etag_from_header(c.header_str) feed.last_modified = last_modified_from_header(c.header_str) responses[url] = feed [:on_success].call(url, feed) if .has_key?(:on_success) rescue Exception => e [:on_failure].call(url, c.response_code, c.header_str, c.body_str) if .has_key?(:on_failure) end else # puts "Error determining parser for #{url} - #{c.last_effective_url}" # raise NoParserAvailable.new("no valid parser for content.") (this would unfortunately fail the whole 'multi', so it's not really usable) [:on_failure].call(url, c.response_code, c.header_str, c.body_str) if .has_key?(:on_failure) end end curl.on_failure do |c, err| add_url_to_multi(multi, url_queue.shift, url_queue, responses, ) unless url_queue.empty? responses[url] = c.response_code if c.response_code == 304 # it's not modified. this isn't an error condition [:on_success].call(url, nil) if .has_key?(:on_success) else [:on_failure].call(url, c.response_code, c.header_str, c.body_str) if .has_key?(:on_failure) end end end multi.add(easy) end |
.call_on_each_feed_entry(method, *parameters) ⇒ Object
Call a method on all feed entries classes.
Parameters
- method<Symbol>
-
The method name
- parameters<Array>
-
The method parameters
101 102 103 104 105 106 107 108 109 110 111 112 |
# File 'lib/feedzirra/feed.rb', line 101 def self.call_on_each_feed_entry(method, *parameters) feed_classes.each do |k| # iterate on the collections defined in the sax collection k.sax_config.collection_elements.each_value do |vl| # vl is a list of CollectionConfig mapped to an attribute name # we'll look for the one set as 'entries' and add the new element vl.find_all{|v| (v.accessor == 'entries') && (v.data_class.class == Class)}.each do |v| v.data_class.send(method, *parameters) end end end end |
.decode_content(c) ⇒ Object
Decodes the XML document if it was compressed.
Parameters
- curl_request<Curl::Easy>
-
The Curl::Easy response object from the request.
Returns
A decoded string of XML.
211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 |
# File 'lib/feedzirra/feed.rb', line 211 def self.decode_content(c) if c.header_str.match(/Content-Encoding: gzip/i) begin gz = Zlib::GzipReader.new(StringIO.new(c.body_str)) xml = gz.read gz.close rescue Zlib::GzipFile::Error # Maybe this is not gzipped? xml = c.body_str end elsif c.header_str.match(/Content-Encoding: deflate/i) xml = Zlib::Inflate.inflate(c.body_str) else xml = c.body_str end xml end |
.determine_feed_parser_for_xml(xml) ⇒ Object
Determines the correct parser class to use for parsing the feed.
Parameters
- xml<String>
-
The XML that you would like determine the parser for.
Returns
The class name of the parser that can handle the XML.
29 30 31 32 |
# File 'lib/feedzirra/feed.rb', line 29 def self.determine_feed_parser_for_xml(xml) start_of_doc = xml.slice(0, 2000) feed_classes.detect {|klass| klass.able_to_parse?(start_of_doc)} end |
.etag_from_header(header) ⇒ Object
Determines the etag from the request headers.
Parameters
- header<String>
-
Raw request header returned from the request
Returns
A string of the etag or nil if it cannot be found in the headers.
368 369 370 371 |
# File 'lib/feedzirra/feed.rb', line 368 def self.etag_from_header(header) header =~ /.*ETag:\s(.*)\r/ $1 end |
.feed_classes ⇒ Object
Provides a list of registered feed parsing classes.
Returns
A array of class names.
48 49 50 |
# File 'lib/feedzirra/feed.rb', line 48 def self.feed_classes @feed_classes ||= [Feedzirra::Parser::RSSFeedBurner, Feedzirra::Parser::RSS, Feedzirra::Parser::GoogleDocsAtom, Feedzirra::Parser::AtomFeedBurner, Feedzirra::Parser::Atom, Feedzirra::Parser::ITunesRSS] end |
.fetch_and_parse(urls, options = {}) ⇒ Object
Fetches and returns the parsed XML for each URL provided.
Parameters
- urls<String> or <Array>
-
A single feed URL, or an array of feed URLs.
- options<Hash>
-
Valid keys for this argument as as followed:
-
:user_agent - String that overrides the default user agent.
-
:if_modified_since - Time object representing when the feed was last updated.
-
:if_none_match - String, an etag for the request that was stored previously.
-
:on_success - Block that gets executed after a successful request.
-
:on_failure - Block that gets executed after a failed request.
Returns
A Feed object if a single URL is passed.
A Hash if multiple URL’s are passed. The key will be the URL, and the value the Feed object.
190 191 192 193 194 195 196 197 198 199 200 201 202 203 |
# File 'lib/feedzirra/feed.rb', line 190 def self.fetch_and_parse(urls, = {}) url_queue = [*urls] multi = Curl::Multi.new responses = {} # I broke these down so I would only try to do 30 simultaneously because # I was getting weird errors when doing a lot. As one finishes it pops another off the queue. url_queue.slice!(0, 30).each do |url| add_url_to_multi(multi, url, url_queue, responses, ) end multi.perform return urls.is_a?(String) ? responses.values.first : responses end |
.fetch_raw(urls, options = {}) ⇒ Object
Fetches and returns the raw XML for each URL provided.
Parameters
- urls<String> or <Array>
-
A single feed URL, or an array of feed URLs.
- options<Hash>
-
Valid keys for this argument as as followed: :if_modified_since - Time object representing when the feed was last updated. :if_none_match - String that’s normally an etag for the request that was stored previously. :on_success - Block that gets executed after a successful request. :on_failure - Block that gets executed after a failed request.
-
all parameters defined in setup_easy
-
Returns
A String of XML if a single URL is passed.
A Hash if multiple URL’s are passed. The key will be the URL, and the value the XML.
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 |
# File 'lib/feedzirra/feed.rb', line 151 def self.fetch_raw(urls, = {}) url_queue = [*urls] multi = Curl::Multi.new responses = {} url_queue.each do |url| easy = Curl::Easy.new(url) do |curl| setup_easy curl, curl.headers["If-Modified-Since"] = [:if_modified_since].httpdate if .has_key?(:if_modified_since) curl.headers["If-None-Match"] = [:if_none_match] if .has_key?(:if_none_match) curl.on_success do |c| responses[url] = decode_content(c) end curl.on_failure do |c, err| responses[url] = c.response_code end end multi.add(easy) end multi.perform urls.is_a?(String) ? responses.values.first : responses end |
.last_modified_from_header(header) ⇒ Object
Determines the last modified date from the request headers.
Parameters
- header<String>
-
Raw request header returned from the request
Returns
A Time object of the last modified date or nil if it cannot be found in the headers.
379 380 381 382 |
# File 'lib/feedzirra/feed.rb', line 379 def self.last_modified_from_header(header) header =~ /.*Last-Modified:\s(.*)\r/ Time.parse($1) if $1 end |
.parse(xml, &block) ⇒ Object
Takes a raw XML feed and attempts to parse it. If no parser is available a Feedzirra::NoParserAvailable exception is raised. You can pass a block to be called when there’s an error during the parsing.
Parameters
- xml<String>
-
The XML that you would like parsed.
Returns
An instance of the determined feed type. By default a Feedzirra::Atom, Feedzirra::AtomFeedBurner, Feedzirra::RDF, or Feedzirra::RSS object.
Raises
Feedzirra::NoParserAvailable : If no valid parser classes could be found for the feed.
15 16 17 18 19 20 21 |
# File 'lib/feedzirra/feed.rb', line 15 def self.parse(xml, &block) if parser = determine_feed_parser_for_xml(xml) parser.parse(xml, block) else raise NoParserAvailable.new("No valid parser for XML.") end end |
.setup_easy(curl, options) ⇒ Object
Setup curl from options. Possible parameters:
-
:user_agent - overrides the default user agent.
-
:compress - any value to enable compression
-
:http_authentication - array containing http authentication parameters
-
:proxy_url - proxy url
-
:proxy_port - proxy port
-
:max_redirects - max number of redirections
-
:timeout - timeout
123 124 125 126 127 128 129 130 131 132 133 134 135 |
# File 'lib/feedzirra/feed.rb', line 123 def self.setup_easy curl, curl.headers["Accept-encoding"] = 'gzip, deflate' if .has_key?(:compress) curl.headers["User-Agent"] = ([:user_agent] || USER_AGENT) curl.userpwd = [:http_authentication].join(':') if .has_key?(:http_authentication) curl.proxy_url = [:proxy_url] if .has_key?(:proxy_url) curl.proxy_port = [:proxy_port] if .has_key?(:proxy_port) curl.max_redirects = [:max_redirects] if [:max_redirects] curl.timeout = [:timeout] if [:timeout] curl.ssl_verify_host = [:ssl_verify_host] if .has_key?(:ssl_verify_host) curl.follow_location = true end |
.update(feeds, options = {}) ⇒ Object
Updates each feed for each Feed object provided.
Parameters
- feeds<Feed> or <Array>
-
A single feed object, or an array of feed objects.
- options<Hash>
-
Valid keys for this argument as as followed:
-
:on_success - Block that gets executed after a successful request.
-
:on_failure - Block that gets executed after a failed request.
-
all parameters defined in setup_easy
-
Returns
A updated Feed object if a single URL is passed.
A Hash if multiple Feeds are passed. The key will be the URL, and the value the updated Feed object.
242 243 244 245 246 247 248 249 250 251 252 253 |
# File 'lib/feedzirra/feed.rb', line 242 def self.update(feeds, = {}) feed_queue = [*feeds] multi = Curl::Multi.new responses = {} feed_queue.slice!(0, 30).each do |feed| add_feed_to_multi(multi, feed, feed_queue, responses, ) end multi.perform responses.is_a?(Array)? responses.values : responses.values.first end |