Module: MogileFS::NewFile

Defined in:
lib/mogilefs/new_file.rb

Overview

The MogileFS::MogileFS#new_file method is enhanced in v3.1.0+ to support the :largefile parameter. While we have always supported large files via the “store_file” method, streaming large amounts of content of an unknown length required the use of awkward APIs.

It is possible to stream large content of known length any WebDAV server. One example of this is for mirroring a large file from an existing HTTP server to MogileFS without letting it hit the local filesystem.

uri = URI('http://example.com/large_file')
Net::HTTP.start(uri.host, uri.port) do |http|
  req = Net::HTTP::Get.new(uri.request_uri)

  http.request(req) do |response|
    if len = response.content_length
      io = mg.new_file('key', :largefile => true, :content_length => len)
    else
      warn "trying to upload with Transfer-Encoding: chunked"
      warn "this is not supported by all WebDAV servers"
      io = mg.new_file('key', :largefile => :stream)
    end
    response.read_body { |buf| io.write(buf) }
    io.close
  end
end

If your WebDAV servers have chunked PUT support (e.g. Perlbal), you can stream a file of unknown length using “Transfer-Encoding: chunked”.

nf = mg.new_file("key", :largefile => :stream)
nf.write "hello"
nf.write ...
nf.close

If your WebDAV server has partial PUT support (e.g Apache), you can you can use multiple PUT requests with “Content-Range” support. This method is slower than Transfer-Encoding: chunked.

nf = mg.new_file("key", :largefile => :content_range)
nf.write "hello"
nf.write ...
nf.close

Finally, if your WebDAV servers does not support either partial nor nor chunked PUTs, you must buffer a large file of unknown length using a Tempfile:

nf = mg.new_file("key", :largefile => :tempfile)
nf.write "hello"
nf.write ...
nf.close

Defined Under Namespace

Modules: Writer

Class Method Summary collapse

Class Method Details

.new(dests, opts) ⇒ Object

avoiding autoload for new code since it’s going away in Ruby…



59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/mogilefs/new_file.rb', line 59

def self.new(dests, opts) # :nodoc:
  largefile = opts[:largefile]
  largefile = :stream if largefile && opts[:content_length]
  require "mogilefs/new_file/#{largefile}" if Symbol === largefile
  case largefile
  when nil, false
    MogileFS::HTTPFile
  when :stream
    Stream
  when :content_range
    ContentRange
  when :tempfile
    Tempfile
  else
    raise ArgumentError, "largefile: #{largefile.inspect} not understood"
  end.new(dests, opts)
end