Class: Defog::Proxy

Inherits:
Object
  • Object
show all
Defined in:
lib/defog/proxy.rb

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(opts = {}) ⇒ Proxy

Opens a Fog cloud storage connection to map to a corresponding proxy directory. Use via, e.g.,

defog = Defog::Proxy.new(:provider => :AWS, :aws_access_key_id => access_key, ...)

The :provider and its corresponding options must be specified as per Fog::Storage.new. Currently, only :local and :AWS are supported. When using :AWS, an additional option :bucket must be specified; all files proxied by this instance must be in a single bucket.

By default, each proxy’s root directory is placed in a reasonable safe place, under Rails.root/tmp if Rails is defined otherwise under Dir.tmpdir. (More details: within that directory, the root directory is disambiguated by #provider and #location, so that multiple Defog::Proxy instances can be created without collision.)

The upshot is that if you have no special constraints you don’t need to worry about it. But if you do care, you can specify the option:

:proxy_root => "/root/for/this/proxy/files"

You can turn on persistence of local proxy files by specifying

:persist => true

The persistence behavior can be overriden on a per-file basis when opening a proxy (see Defog::Handle#open)

You can enable cache management by specifying a max cache size in bytes, e.g.

:max_cache_size => 3.gigabytes

See the README for discussion. [Number#gigabytes is defined in Rails’ ActiveSupport core extensions]



47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/defog/proxy.rb', line 47

def initialize(opts={})
  opts = opts.keyword_args(:provider => :required,
                           :proxy_root => :optional,
                           :persist => :optional,
                           :max_cache_size => :optional,
                           :OTHERS => :optional)

  @proxy_root = Pathname.new(opts.delete(:proxy_root)) if opts.proxy_root
  @persist = opts.delete(:persist)
  @max_cache_size = opts.delete(:max_cache_size)
  @reserved_proxy_paths = Set.new

  @fog_wrapper = FogWrapper.connect(opts)

  @proxy_root ||= case
                  when defined?(Rails) then Rails.root + "tmp"
                  else Pathname.new(Dir.tmpdir)
                  end + "defog" + provider.to_s + location

end

Instance Attribute Details

#fog_wrapper ⇒ Object (readonly)

:nodoc:



12
13
14
 
# File 'lib/defog/proxy.rb', line 12

def fog_wrapper
  @fog_wrapper
end

#max_cache_size ⇒ Object (readonly)

Returns the value of attribute max_cache_size.



11
12
13
 
# File 'lib/defog/proxy.rb', line 11

def max_cache_size
  @max_cache_size
end

#persist ⇒ Object (readonly)

Returns the value of attribute persist.



10
11
12
 
# File 'lib/defog/proxy.rb', line 10

def persist
  @persist
end

#proxy_root ⇒ Object (readonly)

Returns the value of attribute proxy_root.



9
10
11
 
# File 'lib/defog/proxy.rb', line 9

def proxy_root
  @proxy_root
end

Instance Method Details

#each ⇒ Object

Iterate through the cloud storage, yielding a Defog::Handle for each remote file.

If no block is given, an enumerator is returned.



118
119
120
121
122
123
124
125
126
 
# File 'lib/defog/proxy.rb', line 118

def each
  if block_given?
    @fog_wrapper.fog_directory.files.all.each do |fog_model|
      yield file(fog_model.key)
    end
  else
    to_enum(:each)
  end
end

#file(key, mode = nil, opts = {}, &block) ⇒ Object

Proxy a remote cloud file. Returns or yields a Defog::Handle object that represents the file.

If a mode is given, opens a proxy file via Defog::Handle#open (passing it the mode and other options and optional block), returning or yielding instead the Defog::File object.

Thus 

proxy.file("key", mode, options, &block)

is shorthand for

proxy.file("key").open(mode, options, &block)



105
106
107
108
109
110
111
112
 
# File 'lib/defog/proxy.rb', line 105

def file(key, mode=nil, opts={}, &block)
  handle = Handle.new(self, key)
  case
  when mode then handle.open(mode, opts, &block) if mode
  when block then block.call(handle)
  else handle
  end
end

#fog_connection ⇒ Object

Returns the underlying Fog::Storage object for the cloud connection



83
84
85
 
# File 'lib/defog/proxy.rb', line 83

def fog_connection
  @fog_wrapper.fog_connection
end

#fog_directory ⇒ Object

Returns the Fog directory object for the root of the cloud files



88
89
90
 
# File 'lib/defog/proxy.rb', line 88

def fog_directory
  @fog_wrapper.fog_directory
end

#location ⇒ Object

Returns a ‘location’ handle to use in the default proxy root path, to disambiguate it from other proxies with the same provider. For :AWS it’s the bucket name, for :Local it’s derived from the local root path.



78
79
80
 
# File 'lib/defog/proxy.rb', line 78

def location
  @fog_wrapper.location
end

#manage_cache(want_size, proxy_path) ⇒ Object

:nodoc:

Raises:



140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
 
# File 'lib/defog/proxy.rb', line 140

def manage_cache(want_size, proxy_path) #:nodoc:
  return if max_cache_size.nil?
  return if want_size.nil?
  return if want_size <= 0

  # find available space (not counting current proxy)
  available = max_cache_size
  proxy_root.find { |path| available -= path.size if path.file? and path != proxy_path}
  return if available >= want_size

  space_needed = want_size - available

  # find all paths in the cache that aren't currently open (not
  # counting current proxy)
  candidates = []
  proxy_root.find { |path| candidates << path if path.file? and not @reserved_proxy_paths.include?(path) and path != proxy_path}

  # take candidates in LRU order until that would be enough space
  would_free = 0
  candidates = Set.new(candidates.sort_by(&:atime).take_while{|path| (would_free < space_needed).tap{|condition| would_free += path.size}})

  # still not enough...?
  raise Error::CacheFull, "No room in cache for #{proxy_path.relative_path_from(proxy_root)}: size=#{want_size} available=#{available} can_free=#{would_free} (max_cache_size=#{max_cache_size})" if would_free < space_needed

  # LRU order may have taken more than needed, if last file was a big
  # chunk.  So take another pass, eliminating files that aren't needed.
  # Do this in reverse size order, since we want to keep big files in
  # the cache if possible since they're most expensive to replace.
  candidates.sort_by(&:size).reverse.each do |path|
    if (would_free - path.size) > space_needed
      candidates.delete path
      would_free -= path.size
    end
  end

  # free the remaining candidates
  candidates.each(&:unlink)
end

#provider ⇒ Object

Returns the provider for this proxy. I.e., :local or :AWS



70
71
72
 
# File 'lib/defog/proxy.rb', line 70

def provider
  @fog_wrapper.provider
end

#release_proxy_path(proxy_path) ⇒ Object

:nodoc:



136
137
138
 
# File 'lib/defog/proxy.rb', line 136

def release_proxy_path(proxy_path) #:nodoc:
  @reserved_proxy_paths.delete proxy_path
end

#reserve_proxy_path(proxy_path) ⇒ Object

public-but-internal methods



132
133
134
 
# File 'lib/defog/proxy.rb', line 132

def reserve_proxy_path(proxy_path) #:nodoc:
  @reserved_proxy_paths << proxy_path
end