Class: Rubydoop::JobDefinition

Inherits:
Object
  • Object
show all
Defined in:
lib/rubydoop/dsl.rb

Overview

Job configuration DSL.

Rubydoop.configure blocks are run within the context of an instance of this class. These are the methods available in those blocks.

Instance Method Summary collapse

Constructor Details

#initialize(context, job) ⇒ JobDefinition

Returns a new instance of JobDefinition.



82
83
84
85
 
# File 'lib/rubydoop/dsl.rb', line 82

def initialize(context, job)
  @context = context
  @job = job
end

Instance Method Details

#combiner(cls = nil) ⇒ Object Also known as: combiner=

Sets the combiner class.

The equivalent of calling setCombinerClass on a Hadoop job, but instead of a Java class you pass a Ruby class and Rubydoop will wrap it in a way that works with Hadoop.

A combiner should implement reduce, just like reducers.

Parameters:

  • cls (Class) (defaults to: nil) —

    The (Ruby) combiner class.

See Also:



237
238
239
240
241
242
243
244
 
# File 'lib/rubydoop/dsl.rb', line 237

def combiner(cls=nil)
  if cls
    @job.configuration.set(Rubydoop::CombinerProxy::RUBY_CLASS_KEY, cls.name)
    @job.set_combiner_class(Rubydoop::CombinerProxy)
    @combiner = cls
  end
  @combiner
end

#grouping_comparator(cls = nil) ⇒ Object Also known as: grouping_comparator=

Sets a custom grouping comparator.

The equivalent of calling setGroupingComparatorClass on a Hadoop job, but instead of a Java class you pass a Ruby class and Rubydoop will wrap it in a way that works with Hadoop.

Parameters:

  • cls (Class) (defaults to: nil) —

    The (Ruby) comparator class.

See Also:



278
279
280
281
282
283
284
285
 
# File 'lib/rubydoop/dsl.rb', line 278

def grouping_comparator(cls=nil)
  if cls
    @job.configuration.set(Rubydoop::GroupingComparatorProxy::RUBY_CLASS_KEY, cls.name)
    @job.set_grouping_comparator_class(Rubydoop::GroupingComparatorProxy)
    @grouping_comparator = cls
  end
  @grouping_comparator
end

#input(paths, options = {}) ⇒ Object

Sets the input paths of the job.

Calls setInputFormatClass on the Hadoop job and uses the static setInputPaths on the input format to set the job’s input path.

Parameters:

  • paths (String, Enumerable) —

    The input paths, either a comma separated string or an Enumerable of strings (which will be joined with a comma).

  • options (Hash) (defaults to: {})

Options Hash (options):

  • :format (JavaClass) —

    The input format to use, defaults to TextInputFormat

See Also:



98
99
100
101
102
103
104
105
106
107
108
109
110
111
 
# File 'lib/rubydoop/dsl.rb', line 98

def input(paths, options={})
  paths = paths.join(',') if paths.is_a?(Enumerable)
  format = options.fetch(:format, :text)
  unless format.is_a?(Class)
    class_name = format.to_s.gsub(/^.|_./) {|x| x[-1,1].upcase } + "InputFormat"
    format = Hadoop::Mapreduce::Lib::Input.const_get(class_name)
  end
  unless format <= Hadoop::Mapreduce::InputFormat
    @job.configuration.set(Rubydoop::InputFormatProxy::RUBY_CLASS_KEY, format.name)
    format = Rubydoop::InputFormatProxy
  end
  format.set_input_paths(@job, paths)
  @job.set_input_format_class(format)
end

#map_output_key(cls) ⇒ Object

Sets the mapper’s output key type.

Parameters:

  • cls (Class) —

    The mapper’s output key type

See Also:



342
 
# File 'lib/rubydoop/dsl.rb', line 342

class_setter :map_output_key

#map_output_value(cls) ⇒ Object

Sets the reducer’s output value type.

Parameters:

  • cls (Class) —

    The reducer’s output value type

See Also:



351
 
# File 'lib/rubydoop/dsl.rb', line 351

class_setter :map_output_value

#mapper(cls = nil) ⇒ Object Also known as: mapper=

Sets the mapper class.

The equivalent of calling setMapperClass on a Hadoop job, but instead of a Java class you pass a Ruby class and Rubydoop will wrap it in a way that works with Hadoop.

The class only needs to implement the method map, which will be called exactly like a Java mapper class’ map method would be called.

You can optionally implement setup and cleanup, which mirrors the methods of the same name in Java mappers.

Parameters:

  • cls (Class) (defaults to: nil) —

    The (Ruby) mapper class.

See Also:



190
191
192
193
194
195
196
197
 
# File 'lib/rubydoop/dsl.rb', line 190

def mapper(cls=nil)
  if cls
    @job.configuration.set(Rubydoop::MapperProxy::RUBY_CLASS_KEY, cls.name)
    @job.set_mapper_class(Rubydoop::MapperProxy)
    @mapper = cls
  end
  @mapper
end

#output(dir = nil, options = {}) ⇒ Object

Sets or gets the output path of the job.

Calls setOutputFormatClass on the Hadoop job and uses the static setOutputPath on the output format to set the job’s output path.

Parameters:

  • dir (String) (defaults to: nil) —

    The output path

  • options (Hash) (defaults to: {})

Options Hash (options):

  • :format (JavaClass) —

    The output format to use, defaults to TextOutputFormat

See Also:



123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
 
# File 'lib/rubydoop/dsl.rb', line 123

def output(dir=nil, options={})
  if dir
    if dir.is_a?(Hash)
      options = dir
      if options[:intermediate]
        dir = @job.job_name
      else
        raise ArgumentError, sprintf('neither dir nor intermediate: true was specified')
      end
    end
    dir = sprintf('%s-%010d-%05d', dir, Time.now, rand(1e5)) if options[:intermediate]
    @output_dir = dir
    format = options.fetch(:format, :text)
    unless format.is_a?(Class)
      class_name = format.to_s.gsub(/^.|_./) {|x| x[-1,1].upcase } + "OutputFormat"
      format = Hadoop::Mapreduce::Lib::Output.const_get(class_name)
    end
    format.set_output_path(@job, Hadoop::Fs::Path.new(@output_dir))
    @job.set_output_format_class(format)
    if options[:lazy]
      Hadoop::Mapreduce::Lib::Output::LazyOutputFormat.set_output_format_class(@job, format)
    end
  end
  @output_dir
end

#output_key(cls) ⇒ Object

Sets the reducer’s output key type.

Parameters:

  • cls (Class) —

    The reducer’s output key type

See Also:



360
 
# File 'lib/rubydoop/dsl.rb', line 360

class_setter :output_key

#partitioner(cls = nil) ⇒ Object Also known as: partitioner=

Sets a custom partitioner.

The equivalent of calling setPartitionerClass on a Hadoop job, but instead of a Java class you pass a Ruby class and Rubydoop will wrap it in a way that works with Hadoop.

The class must implement partition, which will be called exactly like a Java partitioner would.

Parameters:

  • cls (Class) (defaults to: nil) —

    The (Ruby) partitioner class.

See Also:



259
260
261
262
263
264
265
266
 
# File 'lib/rubydoop/dsl.rb', line 259

def partitioner(cls=nil)
  if cls
    @job.configuration.set(Rubydoop::PartitionerProxy::RUBY_CLASS_KEY, cls.name)
    @job.set_partitioner_class(Rubydoop::PartitionerProxy)
    @partitioner = cls
  end
  @partitioner
end

#raw {|job| ... } ⇒ Object

If you need to manipulate the Hadoop job in some that isn’t covered by this DSL, this is the method for you. It yields the Job, letting you do whatever you want with it.

Yield Parameters:

  • job (Hadoop::Mapreduce::Job) —

    The raw Hadoop Job instance

See Also:



314
315
316
 
# File 'lib/rubydoop/dsl.rb', line 314

def raw(&block)
  yield @job
end

#reducer(cls = nil) ⇒ Object Also known as: reducer=

Sets the reducer class.

The equivalent of calling setReducerClass on a Hadoop job, but instead of a Java class you pass a Ruby class and Rubydoop will wrap it in a way that works with Hadoop.

The class only needs to implement the method reduce, which will be called exactly like a Java reducer class’ reduce method would be called.

You can optionally implement setup and cleanup, which mirrors the methods of the same name in Java reducers.

Parameters:

  • cls (Class) (defaults to: nil) —

    The (Ruby) reducer class.

See Also:



216
217
218
219
220
221
222
223
 
# File 'lib/rubydoop/dsl.rb', line 216

def reducer(cls=nil)
  if cls
    @job.configuration.set(Rubydoop::ReducerProxy::RUBY_CLASS_KEY, cls.name)
    @job.set_reducer_class(Rubydoop::ReducerProxy)
    @reducer = cls
  end
  @reducer
end

#set(property, value) ⇒ Object

Sets a job property.

Calls set/setBoolean/setLong/setFloat on the Hadoop Job’s configuration (exact method depends on the type of the value).

Parameters:

  • property (String) —

    The property name

  • value (String, Numeric, Boolean) —

    The property value

See Also:



161
162
163
164
165
166
167
168
169
170
171
172
 
# File 'lib/rubydoop/dsl.rb', line 161

def set(property, value)
  case value
  when Integer
    @job.configuration.set_long(property, value)
  when Float
    @job.configuration.set_float(property, value)
  when true, false
    @job.configuration.set_boolean(property, value)
  else
    @job.configuration.set(property, value)
  end
end

#sort_comparator(cls = nil) ⇒ Object Also known as: sort_comparator=

Sets a custom sort comparator.

The equivalent of calling setSortComparatorClass on a Hadoop job, but instead of a Java class you pass a Ruby class and Rubydoop will wrap it in a way that works with Hadoop.

Parameters:

  • cls (Class) (defaults to: nil) —

    The (Ruby) comparator class.

See Also:



297
298
299
300
301
302
303
304
 
# File 'lib/rubydoop/dsl.rb', line 297

def sort_comparator(cls=nil)
  if cls
    @job.configuration.set(Rubydoop::SortComparatorProxy::RUBY_CLASS_KEY, cls.name)
    @job.set_sort_comparator_class(Rubydoop::SortComparatorProxy)
    @sort_comparator = cls
  end
  @sort_comparator
end