Class: RightScraper::Scrapers::Base
- Inherits:
-
RegisteredBase
- Object
- RegisteredBase
- RightScraper::Scrapers::Base
- Defined in:
- lib/right_scraper/scrapers/base.rb
Overview
Base class for all scrapers. Subclasses should override #find_next which instantiates the resource from the file system.
Direct Known Subclasses
Instance Attribute Summary collapse
-
#resources ⇒ Object
readonly
Scraped resources.
Class Method Summary collapse
-
.registration_module ⇒ Module
Module for registered repository types.
-
.scraper(options) ⇒ Object
Initialize scraper.
Instance Method Summary collapse
-
#close ⇒ Object
Close any opened file descriptor.
-
#next_resource ⇒ Object
Return the next resource in the filesystem, or nil if none.
-
#scrape ⇒ Object
Do the scrape! Extract all resources from directory Call this method or call ‘next_resource’ to retrieve resources one by one (you must then call ‘close’ yourself) Fill @resources.
Methods inherited from RegisteredBase
query_registered_type, register_class, register_self, registered_types
Instance Attribute Details
#resources ⇒ Object (readonly)
Scraped resources
36 37 38 |
# File 'lib/right_scraper/scrapers/base.rb', line 36 def resources @resources end |
Class Method Details
.registration_module ⇒ Module
Returns module for registered repository types.
39 40 41 |
# File 'lib/right_scraper/scrapers/base.rb', line 39 def self.registration_module ::RightScraper::Scrapers end |
.scraper(options) ⇒ Object
Initialize scraper
Options
:kind
-
Scraper type, one of :cookbook or :workflow
:repo_dir
-
Required, path to directory containing files
to be scraped
:ignorable_paths
-
List of directory names that should
be ignored by scraper
:scanners
-
List of Scanner classes to use, optional
:builders
-
List of Builder classes to use, optional
Return
- scraper(Scrapers::Base)
-
Corresponding scraper instance
56 57 58 59 60 |
# File 'lib/right_scraper/scrapers/base.rb', line 56 def self.scraper() scraper_kind = .delete(:kind) scraper_class = query_registered_type(scraper_kind) scraper_class.new() end |
Instance Method Details
#close ⇒ Object
Close any opened file descriptor
Return
- true
-
Always return true
107 108 109 110 111 112 113 114 |
# File 'lib/right_scraper/scrapers/base.rb', line 107 def close @builder.finish if @stack && !@stack.empty? @stack.each {|s| s.close} @stack = [] end true end |
#next_resource ⇒ Object
Return the next resource in the filesystem, or nil if none. As a part of building the resources, invokes the builders. A resource can be a cookbook, a workflow, a RightScript etc.
Returns
- Object
-
next resource in filesystem, or nil if none.
90 91 92 93 94 95 96 97 98 99 100 101 |
# File 'lib/right_scraper/scrapers/base.rb', line 90 def next_resource @logger.operation(:next) do next nil if @next.nil? value = @next @next = search_dirs while @next.nil? && !@queue.empty? pop_queue end value end end |
#scrape ⇒ Object
Do the scrape! Extract all resources from directory Call this method or call ‘next_resource’ to retrieve resources one by one (you must then call ‘close’ yourself) Fill @resources
Return
- resources<Array>
-
List of all scraped resources
70 71 72 73 74 75 76 77 78 79 80 81 82 |
# File 'lib/right_scraper/scrapers/base.rb', line 70 def scrape @resources = [] begin resource = next_resource until resource.nil? @resources << resource resource = next_resource end ensure close end @resources end |