Module: IMW::Schemes::HDFSFile

Defined in:
lib/imw/schemes/hdfs.rb

Overview

Defines methods for reading data from HDFS files.

Instance Method Summary collapse

Instance Method Details

#each {|String| ... } ⇒ Object

Iterate through each line of this HDFS resource.

Yields:

  • (String) —

    each line of the file



184
185
186
 
# File 'lib/imw/schemes/hdfs.rb', line 184

def each &block
  HDFS.fs(:cat, path, &block)
end

#io ⇒ StringIO

Return a handle on a StringIO object representing the content in this HDFS file.

Be VERY careful how you use this! It is a StringIO object so the whole HDFS file is read into a string before returning the handle.

Returns:

  • (StringIO) 


196
197
198
 
# File 'lib/imw/schemes/hdfs.rb', line 196

def io
  @io ||= StringIO.new(read)
end

#map {|String| ... } ⇒ Array

Map over the lines of this HDFS resource.

Yields:

  • (String) —

    each line of the file

Returns:

  • (Array) —

    the result of the block on each line



204
205
206
207
208
209
210
 
# File 'lib/imw/schemes/hdfs.rb', line 204

def map &block
  returning([]) do |output|
    HDFS.fs(:cat, path) do |line|
      output << block.call(line)
    end
  end
end

#read ⇒ String

Return the contents of this HDFS file as a string.

Be VERY careful how you use this!

Returns:



177
178
179
 
# File 'lib/imw/schemes/hdfs.rb', line 177

def read
  HDFS.fs(:cat, path)
end