Class: Job

Inherits:
Object
  • Object
show all
Defined in:
lib/jobserver.rb

Overview

Use Job objects to generate a jobQueue array for the JobServer. See JobServer for an example.

Constant Summary collapse

WAITING = 
0
RUNNING = 
1
FAILED = 
2
SUCCESS = 
3
@@default_client_command = 
""
@@nicelevel =

default process priority for the command

19
@@jobNumber =

class variable to give each new job a new number

0
@@verbose = 
1

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(args) ⇒ Job

The arguments must be passed as a hash where the keys are the parameter names listed below. You need not write {}.

:name

is the string identifier of the job

:params

are the parameters for the command

:data

is an arbitrary object containing user data for the job. It is available in the pre_run_handler and can be accessed by the data attribute whenever a job object is given.

:dependencies

A job or an array of jobs that must be finished first. Default is nil. Prevent circular or unfulfillable dependencies.

:client_command

either a string with the name of the command to call in order to execute the job on the (remote) machine or a Proc object. In the latter case it receives the job as argument. Proc objects are executed on the server only. This should come in handy for some small tasks that depend on other jobs but note that it increases the load on the server. For Proc commands no output handler will be called. A default value can be set for all new jobs by default_client_command.

:pre_run_handler |job|

is a Proc object that is called before the command is run.

:output_handler |file,job|

is a Proc object that handles the output of the command output. You have to read the lines by yourself: (file.gets). The default handler puts the file’s lines to stdout. The user can collect/store the results in the job.results variable, that is true by default.

post_run_handler |job|

is a Proc object that is called after the command is run.

See JobServer for an example.

Raises:

  • (ArgumentError) 


130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
 
# File 'lib/jobserver.rb', line 130

def initialize(args)
  raise(ArgumentError, "Hash arguments expected.",caller) unless args.kind_of?(Hash)
  args.each_pair{|key,value| args[key.to_sym]= value} # allow string keys also
  @@jobNumber += 1
  @number = @@jobNumber
  @name = args[:name]
  @params = args[:params]
  @data = args[:data]
  @dependencies = args[:dependencies] || []
  @dependencies = [@dependencies] unless @dependencies.is_a?(Array)
  @client_command = args[:client_command] || @@default_client_command
  raise(ArgumentError, "No command specified for job #{name}", caller) if @client_command == ""
  @pre_run_handler = args[:pre_run_handler]
  @output_handler = args[:output_handler] || Proc.new {|file,job| puts file.gets }
  @post_run_handler = args[:post_run_handler]
  @host = nil  #hostname where the job is executed (nil if only scheduled)
  @worker_name = nil
  @working_directory = ""
  @numTries = 0
  @results = nil
  @filesToRemove = [] #list: (host, filename) of files to be removed after run of job
  @status = WAITING
  @failedOnHosts = []
end

Instance Attribute Details

#client_command ⇒ Object

The command to be executed on the (remote) machine. Set in #new. This is either a string or a Proc object (see #new).



80
81
82
 
# File 'lib/jobserver.rb', line 80

def client_command
  @client_command
end

#data ⇒ Object

Is an arbitrary object containing user data for the job. Set in #new.



68
69
70
 
# File 'lib/jobserver.rb', line 68

def data
  @data
end

#dependencies ⇒ Object

A single job object or an array of jobs that must finish before this job can run. (This means, they need all status == SUCCESS.)



90
91
92
 
# File 'lib/jobserver.rb', line 90

def dependencies
  @dependencies
end

#failedOnHosts ⇒ Object (readonly)

list of host names the job failed on



97
98
99
 
# File 'lib/jobserver.rb', line 97

def failedOnHosts
  @failedOnHosts
end

#host ⇒ Object (readonly)

A string with the name of the script/binary to call in order to execute the job. Set in run.



71
72
73
 
# File 'lib/jobserver.rb', line 71

def host
  @host
end

#name ⇒ Object (readonly)

Is the string identifier of the job. Set in #new.



66
67
68
 
# File 'lib/jobserver.rb', line 66

def name
  @name
end

#numTries ⇒ Object (readonly)

Number of times the job failed to run with success



92
93
94
 
# File 'lib/jobserver.rb', line 92

def numTries
  @numTries
end

#params ⇒ Object (readonly)

The parameters for the client_command, set in #new.



77
78
79
 
# File 'lib/jobserver.rb', line 77

def params
  @params
end

#results ⇒ Object

The results object that may be modified in the different handlers. It is initialized with true. If it is false or nil in the end, the job is regarded as FAILED, otherwise as SUCCESS.



84
85
86
 
# File 'lib/jobserver.rb', line 84

def results
  @results
end

#status ⇒ Object

Returns the status of the execution of the job in one of the values: WAITING, RUNNING, FAILED, SUCCESS



87
88
89
 
# File 'lib/jobserver.rb', line 87

def status
  @status
end

#working_directory ⇒ Object

Will be set when job is run and contains the working directory on the (remote) host. Set in run. You could change it in the pre_run_handler.



75
76
77
 
# File 'lib/jobserver.rb', line 75

def working_directory
  @working_directory
end

Class Method Details

.default_client_command ⇒ Object

Returns the default client command that will be used if none is specified in #new.



50
 
# File 'lib/jobserver.rb', line 50

def Job::default_client_command() @@default_client_command end

.default_client_command=(s) ⇒ Object

Sets the default client command that will be used if none is specified in #new.



52
 
# File 'lib/jobserver.rb', line 52

def Job::default_client_command=(s) @@default_client_command = s end

.nicelevel ⇒ Object

Returns the nice level (priority) used when the client is run. Default is 19. For further information about nice see the man-pages for nice.



55
 
# File 'lib/jobserver.rb', line 55

def Job::nicelevel() @@nicelevel end

.nicelevel=(v) ⇒ Object

Sets the nice level (priority) used when the client is run. Default is 19. If you set it to nil, the nice command will not be used.



58
 
# File 'lib/jobserver.rb', line 58

def Job::nicelevel=(v) @@nicelevel = v end

.verbose(v) ⇒ Object

Returns the verbose level. 0 means no output, > 0 means output.



63
 
# File 'lib/jobserver.rb', line 63

def Job::verbose(v) @@verbose end

.verbose=(v) ⇒ Object

Sets the verbose level. 0 means no output, > 0 means output. Default is 1.



61
 
# File 'lib/jobserver.rb', line 61

def Job::verbose=(v) @@verbose = v end

Instance Method Details

#number ⇒ Object 



93
94
95
 
# File 'lib/jobserver.rb', line 93

def number
  @number
end

#run(hostname, worker_name, working_directory) ⇒ Object

Calls the client_command on the given host. The job will be executed by a thread worker_name in working_directory. If hostname != “localhost”, the command is executed via ssh on the remote machine. Returns @results.



160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
 
# File 'lib/jobserver.rb', line 160

def run(hostname, worker_name, working_directory) # :nodoc:
  @host = hostname
  @worker_name = worker_name
  @working_directory = working_directory || ""
  @results = true # default result, can be changed by user
  @status = RUNNING
  @numTries += 1
  @pre_run_handler.call(self) if @pre_run_handler
  if @client_command.is_a?(Proc)
    @host = "localhost"
    @client_command.call(self) #run the Proc on the server (ignore that it should run remotely)
  else
    nice = if @@nicelevel and not (PLATFORM.downcase =~ /win/)
             "nice -#{@@nicelevel}"
           else
             "" # don't use nice
           end
    chdir = (@working_directory != "") ? "cd #{quote(@working_directory)};" : ""
    cmd = "#{chdir}#{nice} #{@client_command} #{@params}"
    cmd = "ssh #{hostname} #{quote(cmd)}" if hostname != "localhost"
    runCommand(cmd)
  end
  @post_run_handler.call(self) if @post_run_handler
  failedOnHosts |= [hostname] unless @results
  @status = @results ? Job::SUCCESS : Job::FAILED
  return @results
end

#runsOnHost(hostname) ⇒ Object

This is called by the jobserver in order to decide if the job should be run on the given host. Default behaviour is true (run on all hosts). You can override this function, e.g.:

require 'jobserver'

class Job
  def runsOnHost(hostname)
    case hostname
    when ...
    end
  end
end

Beware not to construct dead-lock situations when no host satisfies a constraint.



239
240
241
 
# File 'lib/jobserver.rb', line 239

def runsOnHost(hostname)
  true
end

#runsOnHosts(hosts) ⇒ Object

Returns a subset of the given hosts for which #runsOnHost is true.



244
245
246
 
# File 'lib/jobserver.rb', line 244

def runsOnHosts(hosts)
  hosts.select {|host| runsOnHost(host)}
end

#store_data(data, filename, temporary = true) ⇒ Object

Do you want to store the data of an object in a file that will be accessible to the command? If yes, then use this method.

data

is an object that can be written with puts

filename

is relative to working_directory

temporary

determines if the file should be removed after the job has run.

Note: you could use this method in pre_run_handler even to create the client script before executing it.



195
196
197
198
199
200
201
202
203
204
205
206
207
 
# File 'lib/jobserver.rb', line 195

def store_data(data, filename, temporary = true)
  filename = File.join(@working_directory, filename)
  if @host == "localhost" then
    File.open(filename,"w") { |file| file.puts data }
  else
    tempFile = Tempfile.new("#{File.basename(filename)}.tmp")
    tempFile.puts data
    tempFile.close
    copyToHost(tempFile.path, filename)
    tempFile.close(true) # remove local temp file
  end
  @filesToRemove << filename if temporary
end

#to_s ⇒ Object

Returns the current state of the job.



210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
 
# File 'lib/jobserver.rb', line 210

def to_s
  s = "#{@name}: "
  case status
  when WAITING
    s += "waiting"
  when RUNNING
    s += "running"
  when SUCCESS
    s += "finished"
  when FAILED
    s += "failed"
  end
  s += ", try #{@numTries}" if @numTries > 1
  return s
end