Class: Util::Downloader

Inherits:
Object
  • Object
show all
Defined in:
lib/util/downloader.rb

Overview

A helper to safely dowload a file. Can be dowloaded to an actual file, or directly parsed by another gem (e. g. Nokogiri).

Examples:

require 'oga' # Must be required by the user, `Util` will not do it
url = 'https://www.perdu.com/'
html = Util::Downloader.new(url).set_dest(Oga)
html.download # => Util::Result.ok
html.data.at_css('h1').text # 'Perdu sur l\'Internet ?'

url = 'https://gitlab.com/uploads/-/system/user/avatar/5582173/avatar.png'
Util::Downloader.new(url).set_name('guillel.png').set_force.download

Constant Summary collapse

ALLOWED =

Do not document:

[:data, :force, :url]
DEFAULT_OPTIONS =

Do not document:

{
  :dest => '.',
  :name => '',
  :ref => '',
}

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(url, opts = {}) ⇒ self

Create a new helper to safely download a file.

Parameters:

  • url (String) —

    URL of the page to download

  • opts (#to_h) (defaults to: {}) —

    download options

Options Hash (opts):

  • :dest (String, Class) —

    the directory in which to download, or the class with which to parse it

  • :force (Boolean) —

    whether to replace an existing file

  • :name (String) —

    name under which to save the file

  • :ref (String) —

    referer to use while downloading



39
40
41
42
43
44
45
46
47
 
# File 'lib/util/downloader.rb', line 39

def initialize url, opts={}
  require 'util/args'
  @url, opts = Util::Args.check url, String, '', opts, Hash, {}
  opts[:force] === true ? set_force : unset_force

  DEFAULT_OPTIONS.each_pair do |k, v|
    self.method('set_' + k.to_s).call (opts[k].nil? ? v : opts[k])
  end
end

Instance Attribute Details

#data ⇒ Object (readonly)

Returns the value of attribute data.



14
15
16
 
# File 'lib/util/downloader.rb', line 14

def data
  @data
end

#dest ⇒ Object (readonly)

Returns the value of attribute dest.



15
16
17
 
# File 'lib/util/downloader.rb', line 15

def dest
  @dest
end

#force ⇒ Object (readonly)

Returns the value of attribute force.



16
17
18
 
# File 'lib/util/downloader.rb', line 16

def force
  @force
end

#name ⇒ Object (readonly)

Returns the value of attribute name.



17
18
19
 
# File 'lib/util/downloader.rb', line 17

def name
  @name
end

#ref ⇒ Object (readonly)

Returns the value of attribute ref.



18
19
20
 
# File 'lib/util/downloader.rb', line 18

def ref
  @ref
end

#url ⇒ Object (readonly)

Returns the value of attribute url.



19
20
21
 
# File 'lib/util/downloader.rb', line 19

def url
  @url
end

Instance Method Details

#[](key) ⇒ Object?

Return the value of one of the options, the URL, or the parsed web page.

Parameters:

  • key (#to_sym) —

    the attribute to read

Returns:

  • (Object) —

    if succesfull

  • (nil) —

    if key cannot be converted to a symbol, or is not part of the allowed attributes to read



97
98
99
100
101
102
103
 
# File 'lib/util/downloader.rb', line 97

def [] key
  require 'util/args'
  key = Util::Args.check key, Symbol, nil
  return nil if key.nil?
  return nil unless (DEFAULT_OPTIONS.keys + ALLOWED).include? key
  instance_variable_get('@' + key.to_s)
end

#download ⇒ Util::Result

Actually download the file, according to the given options.

Returns:



51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/util/downloader.rb', line 51

def download
  require 'open-uri'
  require 'util/result'

  return Util::Result.err :no_url, self if @url.empty?
  @name = File.basename URI(@url).path if @name.empty?
  @name = 'index.html' if @name.empty? or @name == '/'

  if @dest.is_a?(String) then
    return Util::Result.err :no_dir, self unless File.directory? @dest
    return Util::Result.err :file_exists, self if not @force \
      and File.exists?(File.join @dest, @name)
  end

  begin
    io = @ref.empty? ? URI.open(@url) : URI.open(@url, 'Referer' => @ref)
    case @dest.to_s
    when 'Magick' then @data = Magick::Image.from_blob io.read
    when 'Nokogiri' then @data = Nokogiri::HTML io
    when 'Nokogiri::HTML' then @data = Nokogiri::HTML io
    when 'Nokogiri::XML' then @data = Nokogiri::XML io
    when 'Oga' then @data = Oga.parse_html io
    when 'Oga::HTML' then @data = Oga.parse_html io
    when 'Oga::XML' then @data = Oga.parse_xml io
    when 'REXML' then @data = REXML::Document.new io
    else
      if @dest.respond_to? :parse then
        @data = @dest.parse io
      elsif @dest.respond_to? :read then
        @data = @dest.read io
      else
        IO.copy_stream(io, File.join(@dest, @name))
      end
    end
  rescue Exception => e
    return Util::Result.err e, self
  end

  Util::Result.ok
end

#set_dest(dir = ) ⇒ self

Set the directory in which to download the file, or the class with which to parse it. Currently accepted class are the following.

  • Magick (will try to parse a Magick::Image).

  • Nokogiri::HTML or its alias Nokogiri.

  • Nokogiri::XML.

  • Oga::HTML or its alias Oga.

  • Oga::XML.

  • REXML.

  • Any class that responds to parse.

  • Any class that responds to read.

Please note that with the last two, the methods are tried in this order, and the result might not be what would be expected, especially if the given method does not accept an IO object.

Parameters:

  • dir (Module) (defaults to: ) —

    path to the directory, or if the file shall not be saved, the class that shall parse it

Returns:

  • (self) 


121
122
123
124
 
# File 'lib/util/downloader.rb', line 121

def set_dest dir=DEFAULT_OPTIONS[:dest]
  @dest = dir.is_a?(Module) ? dir : dir.to_s
  self
end

#set_force ⇒ self

Set #download to replace the destination file if it already exists.

Returns:

  • (self) 


128
129
130
131
 
# File 'lib/util/downloader.rb', line 128

def set_force
  @force = true
  self
end

#set_name(n = ) ⇒ self

Set the name under which to save the file. If the given name is empty, defaults to the same name as in the remote location.

Parameters:

  • n (String) (defaults to: ) —

    the file name

Returns:

  • (self) 


144
145
146
147
 
# File 'lib/util/downloader.rb', line 144

def set_name n=DEFAULT_OPTIONS[:name]
  @name = n.to_s
  self
end

#set_ref(r = ) ⇒ self

Set the referer to use while downloading.

Parameters:

  • r (String) (defaults to: ) —

    the referer

Returns:

  • (self) 


152
153
154
155
 
# File 'lib/util/downloader.rb', line 152

def set_ref r=DEFAULT_OPTIONS[:ref]
  @ref = r.to_s
  self
end

#unset_force ⇒ self

Set #download to not replace the destination file if it already exists.

Returns:

  • (self) 


135
136
137
138
 
# File 'lib/util/downloader.rb', line 135

def unset_force
  @force = false
  self
end