Class: Retriever::FetchFiles
Overview
recieves target url and RR options returns an array of all unique files (based on given filetype)
found on the target site
Constant Summary
Constants inherited from Fetch
Instance Attribute Summary
Attributes inherited from Fetch
Instance Method Summary collapse
- #autodownload ⇒ Object
- #download_file(path) ⇒ Object
-
#initialize(url, options) ⇒ FetchFiles
constructor
A new instance of FetchFiles.
Methods inherited from Fetch
#dump, #errlog, #good_response?, #lg, #start, #write
Constructor Details
#initialize(url, options) ⇒ FetchFiles
Returns a new instance of FetchFiles.
6 7 8 9 10 11 12 13 14 15 16 17 |
# File 'lib/retriever/fetchfiles.rb', line 6 def initialize(url, ) super start temp_file_collection = @page_one.parse_files(@page_one.parse_internal) @result.concat(temp_file_collection) if temp_file_collection.size > 0 lg("#{@result.size} new files found") async_crawl_and_collect # done, make sure progress bar says we are done @progressbar.finish if @progress @result.sort_by! { |x| x.length } end |
Instance Method Details
#autodownload ⇒ Object
32 33 34 35 36 37 38 39 40 41 42 |
# File 'lib/retriever/fetchfiles.rb', line 32 def autodownload # go through the fetched file URL collection and download each one. puts HR puts '### Initiating Autodownload...' puts HR puts "#{@result.count} - #{@file_ext}'s Located" puts HR move_to_download_dir iterate_thru_collection_and_download Dir.chdir('..') end |
#download_file(path) ⇒ Object
19 20 21 22 23 24 25 26 27 28 29 30 |
# File 'lib/retriever/fetchfiles.rb', line 19 def download_file(path) # given valid url, downloads file to current directory in /rr-downloads/ arr = path.split('/') shortname = arr.pop puts "Initiating Download of: #{shortname}" File.open(shortname, 'wb') do |saved_file| open(path) do |read_file| saved_file.write(read_file.read) end end puts ' SUCCESS: Download Complete' end |