Module: Bio::Command

Included in:
Meme::Mast
Defined in:
lib/bio/command.rb

Overview

Bio::Command

Bio::Command is a collection of useful methods for execution of external commands or web applications. Any wrapper class for applications shall use this class.

Library internal use only. Users should not directly use it.

Defined Under Namespace

Classes: Tmpdir

Constant Summary collapse

UNSAFE_CHARS_UNIX = 
/[^A-Za-z0-9\_\-\.\:\,\/\@\x1b\x80-\xfe]/n
QUOTE_CHARS_WINDOWS = 
/[^A-Za-z0-9\_\-\.\:\,\/\@\\]/n
UNESCAPABLE_CHARS = 
/[\x00-\x08\x10-\x1a\x1c-\x1f\x7f\xff]/n

Class Method Summary collapse

Class Method Details

.call_command(cmd, options = {}, &block) ⇒ Object

Executes the program. Automatically select popen for Ruby 1.9 or Windows environment and fork for the others. A block must be given. An IO object is passed to the block.

Available options:

:chdir => "path" : changes working directory to the specified path.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) options: Hash

Returns

(undefined)



196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
 
# File 'lib/bio/command.rb', line 196

def call_command(cmd, options = {}, &block) #:yields: io
  if RUBY_VERSION >= "1.9.0" then
    return call_command_popen(cmd, options, &block)
  elsif no_fork? then
    call_command_popen(cmd, options, &block)
  else
    begin
      call_command_fork(cmd, options, &block)
    rescue NotImplementedError
      # fork(2) not implemented
      @@no_fork = true
      call_command_popen(cmd, options, &block)
    end
  end
end

.call_command_fork(cmd, options = {}) ⇒ Object

This method is internally called from the call_command method. In normal case, use call_command, and do not call this method directly.

Executes the program via fork (by using IO.popen(“-”)) and exec. A block must be given. An IO object is passed to the block.

See the document of call_command for available options.

Note for Ruby 1.8: In Ruby 1.8, from the view point of security, this method is recommended rather than call_command_popen. However, this method might have problems with multi-threads.

Note for Ruby 1.9: In Ruby 1.9, this method can not be used, because Thread.critical is removed. In Ruby 1.9, call_command_popen is safe and robust enough, and is the recommended way, because IO.popen is improved to get a command-line as an array without calling shell.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) options: Hash

Returns

(undefined)



295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
 
# File 'lib/bio/command.rb', line 295

def call_command_fork(cmd, options = {})
  dir = options[:chdir]
  cmd = safe_command_line_array(cmd)
  begin
  tc, Thread.critical, flag0, flag1 = Thread.critical, true, true, true
  IO.popen("-", "r+") do |io|
    if io then
      # parent
      flag0, Thread.critical, flag1 = false, tc, false
      yield io
    else
      # child
      Thread.critical = true # for safety, though already true
      GC.disable
      # chdir to options[:chdir] if available
      begin
        Dir.chdir(dir) if dir
      rescue Exception
        Process.exit!(1)
      end
      # executing the command
      begin
        Kernel.exec(*cmd)
      rescue Errno::ENOENT, Errno::EACCES
        Process.exit!(127)
      rescue Exception
      end
      Process.exit!(1)
    end
  end
  ensure
    # When IO.popen("-") raises error, Thread.critical will be set here.
    Thread.critical = tc if flag0 or flag1
    #warn 'Thread.critical might have wrong value.' if flag0 != flag1
  end
end

.call_command_open3(cmd) ⇒ Object

Executes the program via Open3.popen3 A block must be given. IO objects are passed to the block.

You would use this method only when you really need to get stderr.

Arguments:

  • (required) cmd: Array containing String objects

Returns

(undefined)



341
342
343
344
345
346
 
# File 'lib/bio/command.rb', line 341

def call_command_open3(cmd)
  cmd = safe_command_line_array(cmd)
  Open3.popen3(*cmd) do |pin, pout, perr|
    yield pin, pout, perr
  end
end

.call_command_popen(cmd, options = {}) ⇒ Object

This method is internally called from the call_command method. In normal case, use call_command, and do not call this method directly.

Executes the program via IO.popen for OS which doesn’t support fork. A block must be given. An IO object is passed to the block.

See the document of call_command for available options.

Note for Ruby 1.8: In Ruby 1.8, although shell unsafe characters are escaped. If inescapable characters exists, it raises RuntimeError. So, call_command_fork is normally recommended.

Note for Ruby 1.9: In Ruby 1.9, call_command_popen is safe and robust enough, and is the recommended way, because IO.popen is improved to get a command-line as an array without calling shell.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) options: Hash

Returns

(undefined)



235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
 
# File 'lib/bio/command.rb', line 235

def call_command_popen(cmd, options = {})
  if RUBY_VERSION >= "1.9.0" then
    # For Ruby 1.9 or later, using command line array with options.
    dir = options[:chdir]
    cmd = safe_command_line_array(cmd)
    if dir then
      cmd = cmd + [ { :chdir => dir } ]
    end
    r = IO.popen(cmd, "r+") do |io|
      yield io
    end
    return r
  end
  # For Ruby 1.8, using command line string.
  str = make_command_line(cmd)
  # processing options
  if dir = options[:chdir] then
    if windows_platform?
      # Unix-like dir separator is changed to Windows dir separator
      # by using String#gsub.
      dirstr = dir.gsub(/\//, "\\")
      chdirstr = make_command_line([ 'cd', '/D', dirstr ])
      str = chdirstr + ' && ' + str
    else
      # UNIX shell
      chdirstr = make_command_line([ 'cd', dir ])
      str = chdirstr + ' && ' + str
    end
  end
  # call command by using IO.popen
  IO.popen(str, "w+") do |io|
    io.sync = true
    yield io
  end
end

.escape_shell(str) ⇒ Object

Escape special characters in command line string.

Arguments:

  • (required) str: String

Returns

String object



121
122
123
124
125
126
127
 
# File 'lib/bio/command.rb', line 121

def escape_shell(str)
  if windows_platform? then
    escape_shell_windows(str)
  else
    escape_shell_unix(str)
  end
end

.escape_shell_unix(str) ⇒ Object

Escape special characters in command line string for UNIX shells.

Arguments:

  • (required) str: String

Returns

String object



110
111
112
113
114
 
# File 'lib/bio/command.rb', line 110

def escape_shell_unix(str)
  str = str.to_s
  raise 'cannot escape control characters' if UNESCAPABLE_CHARS =~ str
  str.gsub(UNSAFE_CHARS_UNIX) { |x| "\\#{x}" }
end

.escape_shell_windows(str) ⇒ Object

Escape special characters in command line string for cmd.exe on Windows.

Arguments:

  • (required) str: String

Returns

String object



95
96
97
98
99
100
101
102
103
 
# File 'lib/bio/command.rb', line 95

def escape_shell_windows(str)
  str = str.to_s
  raise 'cannot escape control characters' if UNESCAPABLE_CHARS =~ str
  if QUOTE_CHARS_WINDOWS =~ str then
    '"' + str.gsub(/\"/, '""') + '"'
  else
    String.new(str)
  end
end

.http_post_form(http, path, params = nil, header = {}) ⇒ Object

Same as:

http = Net::HTTP.new(...); http.post_form(path, params)

and it uses proxy if an environment variable (same as OpenURI.open_uri) is set. In addition, header can be set. (Note that Content-Type and Content-Length are automatically set by default.) uri must be a URI object, params must be a hash, and header must be a hash.

Arguments:

  • (required) http: Net::HTTP object or compatible object

  • (required) path: String

  • (optional) params: Hash containing parameters

  • (optional) header: Hash containing header strings

Returns

(same as Net::HTTP::post_form)



693
694
695
696
697
698
699
700
701
702
703
 
# File 'lib/bio/command.rb', line 693

def http_post_form(http, path, params = nil, header = {})
  data = make_cgi_params(params)

  hash = {
    'Content-Type'   => 'application/x-www-form-urlencoded',
    'Content-Length' => data.length.to_s
  }
  hash.update(header)

  http.post(path, data, hash)
end

.make_cgi_params(params) ⇒ Object

Builds parameter string for from Hash of parameters for application/x-www-form-urlencoded.

Arguments:

  • (required) params: Hash containing parameters

Returns

String



747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
 
# File 'lib/bio/command.rb', line 747

def make_cgi_params(params)
  data = ""
  case params
  when Hash
    data = params.map do |key, val|
      make_cgi_params_key_value(key, val)
    end.join('&')
  when Array
    case params.first
    when Hash
      data = params.map do |hash|
        hash.map do |key, val|
          make_cgi_params_key_value(key, val)
        end
      end.join('&')
    when Array
      data = params.map do |key, val|
        make_cgi_params_key_value(key, val)
      end.join('&')
    when String
      data = params.map do |str|
        key, val = str.split(/\=/, 2)
        if val then
          make_cgi_params_key_value(key, val)
        else
          CGI.escape(str)
        end
      end.join('&')
    end
  when String
    data = URI.escape(params.strip)
  end
  return data
end

.make_cgi_params_key_value(key, value) ⇒ Object

Builds parameter string for from a key string and a value (or values) for application/x-www-form-urlencoded.

Arguments:

  • (required) key: String

  • (required) value: String or Array containing String

Returns

String



790
791
792
793
794
795
796
797
798
799
800
801
 
# File 'lib/bio/command.rb', line 790

def make_cgi_params_key_value(key, value)
  result = []
  case value
  when Array
    value.each do |val|
      result << [key, val].map {|x| CGI.escape(x.to_s) }.join('=')
    end
  else
    result << [key, value].map {|x| CGI.escape(x.to_s) }.join('=')
  end
  return result
end

.make_command_line(ary) ⇒ Object

Generate command line string with special characters escaped.

Arguments:

  • (required) ary: Array containing String objects

Returns

String object



134
135
136
137
138
139
140
 
# File 'lib/bio/command.rb', line 134

def make_command_line(ary)
  if windows_platform? then
    make_command_line_windows(ary)
  else
    make_command_line_unix(ary)
  end
end

.make_command_line_unix(ary) ⇒ Object

Generate command line string with special characters escaped for UNIX shells.

Arguments:

  • (required) ary: Array containing String objects

Returns

String object



158
159
160
 
# File 'lib/bio/command.rb', line 158

def make_command_line_unix(ary)
  ary.collect { |str| escape_shell_unix(str) }.join(" ")
end

.make_command_line_windows(ary) ⇒ Object

Generate command line string with special characters escaped for cmd.exe on Windows.

Arguments:

  • (required) ary: Array containing String objects

Returns

String object



148
149
150
 
# File 'lib/bio/command.rb', line 148

def make_command_line_windows(ary)
  ary.collect { |str| escape_shell_windows(str) }.join(" ")
end

.mktmpdir(prefix_suffix = nil, tmpdir = nil, &block) ⇒ Object

Backport of Dir.mktmpdir in Ruby 1.9.

Same as Dir.mktmpdir(prefix_suffix) in Ruby 1.9.

Arguments:

  • (optional) prefix_suffix: String (or Array, etc.)

  • (optional) tmpdir: String: temporary directory’s path



495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
 
# File 'lib/bio/command.rb', line 495

def mktmpdir(prefix_suffix = nil, tmpdir = nil, &block)
  begin
    Dir.mktmpdir(prefix_suffix, tmpdir, &block)
  rescue NoMethodError
    # backported from Ruby 1.9.2-preview1.
    # ***** Below is excerpted from Ruby 1.9.2-preview1's lib/tmpdir.rb ****
    # ***** Be careful about copyright. ****
    case prefix_suffix
    when nil
      prefix = "d"
      suffix = ""
    when String
      prefix = prefix_suffix
      suffix = ""
    when Array
      prefix = prefix_suffix[0]
      suffix = prefix_suffix[1]
    else
      raise ArgumentError, "unexpected prefix_suffix: #{prefix_suffix.inspect}"
    end
    tmpdir ||= Dir.tmpdir
    t = Time.now.strftime("%Y%m%d")
    n = nil
    begin
      path = "#{tmpdir}/#{prefix}#{t}-#{$$}-#{rand(0x100000000).to_s(36)}"
      path << "-#{n}" if n
      path << suffix
      Dir.mkdir(path, 0700)
    rescue Errno::EEXIST
      n ||= 0
      n += 1
      retry
    end

    if block_given?
      begin
        yield path
      ensure
        remove_entry_secure path
      end
    else
      path
    end
    # ***** Above is excerpted from Ruby 1.9.2-preview1's lib/tmpdir.rb ****
  end
end

.new_http(address, port = 80) ⇒ Object

Same as:

Net::HTTP.new(address, port)

and it uses proxy if an environment variable (same as OpenURI.open_uri) is set.

Arguments:

  • (required) address: String containing host name or IP address

  • (optional) port: port (sanme as Net::HTTP::start)

Returns

(same as Net::HTTP.new except for proxy support)



663
664
665
666
667
668
669
670
671
672
673
 
# File 'lib/bio/command.rb', line 663

def new_http(address, port = 80)
  uri = URI.parse("http://#{address}:#{port}")
  # Note: URI#find_proxy is an unofficial method defined in open-uri.rb.
  # If the spec of open-uri.rb would be changed, we should change below.
  if proxyuri = uri.find_proxy then
    raise 'Non-HTTP proxy' if proxyuri.class != URI::HTTP
    Net::HTTP.new(address, port, proxyuri.host, proxyuri.port)
  else
    Net::HTTP.new(address, port)
  end
end

.post_form(uri, params = nil, header = {}) ⇒ Object

Same as: Net::HTTP.post_form(uri, params) and it uses proxy if an environment variable (same as OpenURI.open_uri) is set. In addition, header can be set. (Note that Content-Type and Content-Length are automatically set by default.) uri must be a URI object, params must be a hash, and header must be a hash.

Arguments:

  • (required) uri: URI object or String

  • (optional) params: Hash containing parameters

  • (optional) header: Hash containing header strings

Returns

(same as Net::HTTP::post_form)



722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
 
# File 'lib/bio/command.rb', line 722

def post_form(uri, params = nil, header = {})
  unless uri.is_a?(URI)
    uri = URI.parse(uri)
  end

  data = make_cgi_params(params)

  hash = {
    'Content-Type'   => 'application/x-www-form-urlencoded',
    'Content-Length' => data.length.to_s
  }
  hash.update(header)

  start_http(uri.host, uri.port) do |http|
    http.post(uri.path, data, hash)
  end
end

.query_command(cmd, query = nil, options = {}) ⇒ Object

Executes the program with the query (String) given to the standard input, waits the program termination, and returns the output data printed to the standard output as a string.

Automatically select popen for Ruby 1.9 or Windows environment and fork for the others.

Available options:

:chdir => "path" : changes working directory to the specified path.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) query: String

  • (optional) options: Hash

Returns

String or nil



364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
 
# File 'lib/bio/command.rb', line 364

def query_command(cmd, query = nil, options = {})
  if RUBY_VERSION >= "1.9.0" then
    return query_command_popen(cmd, query, options)
  elsif no_fork? then
    query_command_popen(cmd, query, options)
  else
    begin
      query_command_fork(cmd, query, options)
    rescue NotImplementedError
      # fork(2) not implemented
      @@no_fork = true
      query_command_fork(cmd, query, options)
    end
  end
end

.query_command_fork(cmd, query = nil, options = {}) ⇒ Object

This method is internally called from the query_command method. In normal case, use query_command, and do not call this method directly.

Executes the program with the query (String) given to the standard input, waits the program termination, and returns the output data printed to the standard output as a string.

Fork (by using IO.popen(“-”)) and exec is used to execute the program.

See the document of query_command for available options.

See the document of call_command_fork for the security and Ruby version specific issues.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) query: String

  • (optional) options: Hash

Returns

String or nil



429
430
431
432
433
434
435
436
437
438
 
# File 'lib/bio/command.rb', line 429

def query_command_fork(cmd, query = nil, options = {})
  ret = nil
  call_command_fork(cmd, options) do |io|
    io.sync = true
    io.print query if query
    io.close_write
    ret = io.read
  end
  ret
end

.query_command_open3(cmd, query = nil) ⇒ Object

Executes the program via Open3.popen3 with the query (String) given to the stain, waits the program termination, and returns the data from stdout and stderr as an array of the strings.

You would use this method only when you really need to get stderr.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) query: String

Returns

Array containing 2 objects: output string (or nil) and stderr string (or nil)



451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
 
# File 'lib/bio/command.rb', line 451

def query_command_open3(cmd, query = nil)
  errorlog = nil
  cmd = safe_command_line_array(cmd)
  Open3.popen3(*cmd) do |pin, pout, perr|
    perr.sync = true
    t = Thread.start { errorlog = perr.read }
    begin
      pin.print query if query
      pin.close
      output = pout.read
    ensure
      t.join
    end
    [ output, errorlog ]
  end
end

.query_command_popen(cmd, query = nil, options = {}) ⇒ Object

This method is internally called from the query_command method. In normal case, use query_command, and do not call this method directly.

Executes the program with the query (String) given to the standard input, waits the program termination, and returns the output data printed to the standard output as a string.

See the document of query_command for available options.

See the document of call_command_popen for the security and Ruby version specific issues.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) query: String

  • (optional) options: Hash

Returns

String or nil



398
399
400
401
402
403
404
405
406
407
 
# File 'lib/bio/command.rb', line 398

def query_command_popen(cmd, query = nil, options = {})
  ret = nil
  call_command_popen(cmd, options) do |io|
    io.sync = true
    io.print query if query
    io.close_write
    ret = io.read
  end
  ret
end

.read_uri(uri) ⇒ Object

Same as OpenURI.open_uri(uri).read and it uses proxy if an environment variable (same as OpenURI.open_uri) is set.

Arguments:

  • (required) uri: URI object or String

Returns

String



624
625
626
 
# File 'lib/bio/command.rb', line 624

def read_uri(uri)
  OpenURI.open_uri(uri).read
end

.remove_entry_secure(path, force = false) ⇒ Object

Same as FileUtils.remove_entry_secure after Ruby 1.8.3. In Ruby 1.8.2 or previous version, it only shows warning message and does nothing.

It is strongly recommended using Ruby 1.8.5 or later.

Arguments:

  • (required) path: String

  • (optional) force: boolean



477
478
479
480
481
482
483
484
 
# File 'lib/bio/command.rb', line 477

def remove_entry_secure(path, force = false)
  begin
    FileUtils.remove_entry_secure(path, force)
  rescue NoMethodError
    warn "The temporary file or directory is not removed because of the lack of FileUtils.remove_entry_secure. Use Ruby 1.8.3 or later (1.8.5 or later is strongly recommended): #{path}"
    nil
  end
end

.safe_command_line_array(ary) ⇒ Object

Returns an Array of command-line command and arguments that can be safely passed to Kernel.exec etc. If the given array is already safe (or empty), returns the given array.

Arguments:

  • (required) ary: Array

Returns

Array



169
170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/bio/command.rb', line 169

def safe_command_line_array(ary)
  ary = ary.to_ary
  return ary if ary.size >= 2 or ary.empty?
  if ary.size != 1 then
    raise 'Bug: assersion of ary.size == 1 failed'
  end
  arg0 = ary[0]
  begin
    arg0 = arg0.to_ary
  rescue NoMethodError
    arg0 = [ arg0, arg0 ]
  end
  [ arg0 ]
end

.start_http(address, port = 80, &block) ⇒ Object

Same as:

Net::HTTP.start(address, port)

and it uses proxy if an environment variable (same as OpenURI.open_uri) is set.

Arguments:

  • (required) address: String containing host name or IP address

  • (optional) port: port (sanme as Net::HTTP::start)

Returns

(same as Net::HTTP::start except for proxy support)



639
640
641
642
643
644
645
646
647
648
649
650
 
# File 'lib/bio/command.rb', line 639

def start_http(address, port = 80, &block)
  uri = URI.parse("http://#{address}:#{port}")
  # Note: URI#find_proxy is an unofficial method defined in open-uri.rb.
  # If the spec of open-uri.rb would be changed, we should change below.
  if proxyuri = uri.find_proxy then
    raise 'Non-HTTP proxy' if proxyuri.class != URI::HTTP
    http = Net::HTTP.Proxy(proxyuri.host, proxyuri.port)
  else
    http = Net::HTTP
  end
  http.start(address, port, &block)
end