JobQueue - Let run your jobs in parallel

This package contains jobQueue:

* A simple class to parallalize your work on a user definend number of threads
* A ruby script, which uses the above class to run each line of a shell script

jobQueue can do the following things:

Run blocks, Procs and Lambdas
Run instance and class methods
Run shell commands
Respect user definded locks

I started a python2 implementation of this, which can be installed via pip (pypi.python.org/pypi/jobqueue). When its maature enough, I will use the ruby version numbers for it.

Installation

Gem Installation

Download and install jobQueue with the following.

gem install jobQueue

Requirements

JobQueue requires Ruby only, but versions 1.9.x are needed to make use of system threads.

Usage

Parallelize Ruby’s blocks, procs, lambdas and things

Create a JobQueue with nThreads worker with:

jq = JobQueue.new(nThreads)

Use its push method to put in something to do

For blocks:

jq.push do
  myObject.method0(...)
  myObject.method1(...)
  myObject.method3(...)
end

For procs and lambdas:
```
jp.push(myProc,arg0,arg1,...)
```

For object methods:

jq.push([myObject,[:method,arg0,arg1,...])

Same code can be used for class methods:

jq.push(myClass,[:myClassMethod,arg0,arg1,...])

To start the workers, call

jq.run

That’s it. You might have look at tests, that come with the jobQueue gem.

Parallelize system commands

Use a separate class for this and push string to it:

jq = SystemJobs.new(nThreads)
jq.push('find ./src -name "*.rb"','find ./downloads -name "*.rb"')

will start 2 parallel searches for ruby files. This is implemented in the prun.rb script, which comes with the jobQueue gem. To run each line of a shell script on 10 threads, use:

prun.rb -j 10 jobs.sh

The ‘-j’ switch is optional. Default is the maximum number of cores. The script accepts multiple files and processes one after another. Try ‘-h’ for documentation.

Support, Issues, Bugs, …

please use personal mail, ruby-lang mailing list or github

Changelog

1.0.11: prun.rb now ignores empty lines
1.0.10: more flexible logging control (new switches ‘-l’ and ‘-b’)
1.0.9: print out stdout and stderr from the jobs given to prun.rb, use ‘-D’ to avoid printing
1.0.8: support AIX for getting the maximum number of processors, improve processor count for jruby and rbx

Credits

Robert Klemme: For the first hints: www.ruby-forum.com/topic/68001#86298

License

jobQueue use the BSD License

:include LICENSE

Other stuff

Author: Ralf Mueller <[email protected]>
Requires: Ruby 1.9 or later
License: Copyright 2011-2013 by Ralf Mueller Released under BSD-style license. See the LICENSE file included in the distribution.