Class: Unbreakable::DataStorage::FileDataStore

Inherits:
Dragonfly::DataStorage::FileDataStore
  • Object
show all
Includes:
Dragonfly::Loggable, Observable
Defined in:
lib/unbreakable/data_storage/file_data_store.rb

Overview

Stores files to the filesystem. To configure:

scraper.configure do |c|
  c.datastore = Unbreakable::DataStorage::FileDataStore.new(scraper,
    :decorators => [:timeout],        # optional
    :observers => [:log])             # optional
  c.datastore.root_path = '/path/dir' # default '/var/tmp/unbreakable'
  c.datastore.store_meta = true       # default false
end

Instance Method Summary collapse

Constructor Details

#initialize(app, opts = {}) ⇒ FileDataStore

Returns a new instance of FileDataStore.

Parameters:

  • app (Dragonfly::App)
  • opts (Hash) (defaults to: {})
  • options (Hash) —

    a customizable set of options



37
38
39
40
41
42
43
44
45
46
47
48
49
50
 
# File 'lib/unbreakable/data_storage/file_data_store.rb', line 37

def initialize(app, opts = {})
  use_same_log_as(app)
  use_as_fallback_config(app)
  if opts[:decorators]
    opts[:decorators].each do |decorator|
      extend Symbol === decorator ? Unbreakable::Decorators.const_get(decorator.capitalize) : decorator
    end
  end
  if opts[:observers]
    opts[:observers].each do |observer|
      add_observer Symbol === observer ? Unbreakable::Observers.const_get(observer.capitalize).new(self) : observer.new(self)
    end
  end
end

Instance Method Details

#clobber ⇒ Boolean, ...

Configure the datastore to overwrite files upon repeated download.

scraper.configure do |c|
  c.datastore.clobber = true # default false
end

Returns:

  • (Boolean, Proc, lambda) —

    whether to overwrite files upon repeated download



29
 
# File 'lib/unbreakable/data_storage/file_data_store.rb', line 29

configurable_attr :clobber, false

#defer_store(opts = {}, &block) ⇒ String

Stores a record in the datastore. This method does lazy evaluation of the record’s contents, e.g.:

defer_store(:path => 'index.html') do
  open('http://www.example.com/').read
end

The open method is called only if the record hasn’t already been downloaded or if the datastore has been configured to overwrite files upon repeated download.

Parameters:

  • opts (Hash) (defaults to: {})
  • block (Proc) —

    a block that yields the contents of the file

Options Hash (opts):

  • :meta (Hash) —

    any file metadata, e.g. bitrate

  • :path (String) —

    the relative path at which to store the file

Returns:

  • (String) —

    the relative path to the file

Raises:

  • (Dragonfly::DataStorage::UnableToStore) —

    if permission is denied

See Also:

  • Unbreakable::DataStorage::FileDataStore.[Dragonfly[Dragonfly::DataStorage[Dragonfly::DataStorage::FileDataStore[Dragonfly::DataStorage::FileDataStore#store]


70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
 
# File 'lib/unbreakable/data_storage/file_data_store.rb', line 70

def defer_store(opts = {}, &block)
  meta = opts[:meta] || {}
  relative_path = if opts[:path]
    opts[:path]
  else
    filename = meta[:name] || 'file'
    relative_path = relative_path_for(filename)
  end

  changed
  if empty?(relative_path) or clobber?(relative_path)
    begin
      path = absolute(relative_path)
      prepare_path(path)
      string = yield_block(relative_path, &block)
      Dragonfly::TempObject.new(string).to_file(path).close
      store_meta_data(path, meta) if store_meta
      notify_observers :store, relative_path, string
      relative(path)
    rescue InvalidRemoteFile => e
      log.error e.message
    rescue Errno::EACCES => e
      raise UnableToStore, e.message
    end
  else
    notify_observers :skip, relative_path
  end
end

#records(pattern = nil) ⇒ Array<String>

Returns all filenames matching a pattern, if given.

Parameters:

  • pattern (String, Regexp) (defaults to: nil) —

    a pattern to match filenames with

Returns:

  • (Array<String>) —

    an array of matching filenames



102
103
104
105
106
107
108
109
110
 
# File 'lib/unbreakable/data_storage/file_data_store.rb', line 102

def records(pattern = nil)
  if pattern
    Dir[File.join(root_path, '**', pattern)]
  else
    Dir[File.join(root_path, '**', '*')]
  end.map do |absolute_path|
    relative absolute_path
  end
end