Module: Bio::Command

Included in:
Meme::Mast
Defined in:
lib/bio/command.rb

Overview

Bio::Command

Bio::Command is a collection of useful methods for execution of external commands or web applications. Any wrapper class for applications shall use this class.

Library internal use only. Users should not directly use it.

Defined Under Namespace

Classes: Tmpdir

Constant Summary collapse

UNSAFE_CHARS_UNIX = 
/[^A-Za-z0-9\_\-\.\:\,\/\@\x1b\x80-\xfe]/n
QUOTE_CHARS_WINDOWS = 
/[^A-Za-z0-9\_\-\.\:\,\/\@\\]/n
UNESCAPABLE_CHARS = 
/[\x00-\x08\x10-\x1a\x1c-\x1f\x7f\xff]/n

Class Method Summary collapse

Class Method Details

._call_command_popen_jruby19(cmd, options = {}, &block) ⇒ Object

This method is internally called from the call_command method. In normal case, use call_command, and do not call this method directly.

Executes the program via IO.popen. A block must be given. An IO object is passed to the block.

See the document of call_command for available options.

The method is written for the workaround of the JRuby bugs:

  • JRUBY-6195 Process.spawn (and related methods) ignore option hash

  • JRUBY-6818 Kernel.exec, Process.spawn (and IO.popen etc.) raise error when program is an array containing two strings

This method may be removed after the bugs are resolved.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) options: Hash

Returns

(undefined)



340
341
342
343
344
345
346
 
# File 'lib/bio/command.rb', line 340

def _call_command_popen_jruby19(cmd, options = {}, &block)
  if !options.empty? or cmd.size == 1 then
    _call_command_popen_ruby18(cmd, options, &block)
  else
    _call_command_popen_ruby19(cmd, options, &block)
  end
end

._call_command_popen_ruby18(cmd, options = {}) ⇒ Object

This method is internally called from the call_command method. In normal case, use call_command, and do not call this method directly.

Executes the program via IO.popen. A block must be given. An IO object is passed to the block.

See the document of call_command for available options.

The method is written for Ruby 1.8.

In Ruby 1.8, although shell unsafe characters are escaped, if inescapable characters exists, it raises RuntimeError.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) options: Hash

Returns

(undefined)



265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
 
# File 'lib/bio/command.rb', line 265

def _call_command_popen_ruby18(cmd, options = {})
  # For Ruby 1.8, using command line string.
  str = make_command_line(cmd)
  # processing options
  if dir = options[:chdir] then
    if windows_platform?
      # Unix-like dir separator is changed to Windows dir separator
      # by using String#gsub.
      dirstr = dir.gsub(/\//, "\\")
      chdirstr = make_command_line([ 'cd', '/D', dirstr ])
      str = chdirstr + ' && ' + str
    else
      # UNIX shell
      chdirstr = make_command_line([ 'cd', dir ])
      str = chdirstr + ' && ' + str
    end
  end
  # call command by using IO.popen
  IO.popen(str, "w+") do |io|
    io.sync = true
    yield io
  end
end

._call_command_popen_ruby19(cmd, options = {}) ⇒ Object

This method is internally called from the call_command method. In normal case, use call_command, and do not call this method directly.

Executes the program via IO.popen. A block must be given. An IO object is passed to the block.

See the document of call_command for available options.

The method can be run only on Ruby (MRI) 1.9 or later versions.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) options: Hash

Returns

(undefined)



305
306
307
308
309
310
311
312
313
314
315
316
 
# File 'lib/bio/command.rb', line 305

def _call_command_popen_ruby19(cmd, options = {})
  # For Ruby 1.9 or later, using command line array with options.
  dir = options[:chdir]
  cmd = safe_command_line_array(cmd)
  if dir then
    cmd = cmd + [ { :chdir => dir } ]
  end
  r = IO.popen(cmd, "r+") do |io|
    yield io
  end
  return r
end

.call_command(cmd, options = {}, &block) ⇒ Object

Executes the program. Automatically select popen for Ruby 1.9 or Windows environment and fork for the others. A block must be given. An IO object is passed to the block.

Available options:

:chdir => "path" : changes working directory to the specified path.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) options: Hash

Returns

(undefined)



196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
 
# File 'lib/bio/command.rb', line 196

def call_command(cmd, options = {}, &block) #:yields: io
  if RUBY_VERSION >= "1.9.0" then
    return call_command_popen(cmd, options, &block)
  elsif no_fork? then
    call_command_popen(cmd, options, &block)
  else
    begin
      call_command_fork(cmd, options, &block)
    rescue NotImplementedError
      # fork(2) not implemented
      @@no_fork = true
      call_command_popen(cmd, options, &block)
    end
  end
end

.call_command_fork(cmd, options = {}) ⇒ Object

This method is internally called from the call_command method. In normal case, use call_command, and do not call this method directly.

Executes the program via fork (by using IO.popen(“-”)) and exec. A block must be given. An IO object is passed to the block.

See the document of call_command for available options.

Note for Ruby 1.8: In Ruby 1.8, from the view point of security, this method is recommended rather than call_command_popen. However, this method might have problems with multi-threads.

Note for Ruby 1.9: In Ruby 1.9, this method can not be used, because Thread.critical is removed. In Ruby 1.9, call_command_popen is safe and robust enough, and is the recommended way, because IO.popen is improved to get a command-line as an array without calling shell.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) options: Hash

Returns

(undefined)



373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
 
# File 'lib/bio/command.rb', line 373

def call_command_fork(cmd, options = {})
  dir = options[:chdir]
  cmd = safe_command_line_array(cmd)
  begin
  tc, Thread.critical, flag0, flag1 = Thread.critical, true, true, true
  IO.popen("-", "r+") do |io|
    if io then
      # parent
      flag0, Thread.critical, flag1 = false, tc, false
      yield io
    else
      # child
      Thread.critical = true # for safety, though already true
      GC.disable
      # chdir to options[:chdir] if available
      begin
        Dir.chdir(dir) if dir
      rescue Exception
        Process.exit!(1)
      end
      # executing the command
      begin
        Kernel.exec(*cmd)
      rescue Errno::ENOENT, Errno::EACCES
        Process.exit!(127)
      rescue Exception
      end
      Process.exit!(1)
    end
  end
  ensure
    # When IO.popen("-") raises error, Thread.critical will be set here.
    Thread.critical = tc if flag0 or flag1
    #warn 'Thread.critical might have wrong value.' if flag0 != flag1
  end
end

.call_command_open3(cmd) ⇒ Object

Executes the program via Open3.popen3 A block must be given. IO objects are passed to the block.

You would use this method only when you really need to get stderr.

Arguments:

  • (required) cmd: Array containing String objects

Returns

(undefined)



419
420
421
422
423
424
 
# File 'lib/bio/command.rb', line 419

def call_command_open3(cmd)
  cmd = safe_command_line_array(cmd)
  Open3.popen3(*cmd) do |pin, pout, perr|
    yield pin, pout, perr
  end
end

.call_command_popen(cmd, options = {}, &block) ⇒ Object

This method is internally called from the call_command method. In normal case, use call_command, and do not call this method directly.

Executes the program via IO.popen for OS which doesn’t support fork. A block must be given. An IO object is passed to the block.

See the document of call_command for available options.

Note for Ruby 1.8: In Ruby 1.8, although shell unsafe characters are escaped. If inescapable characters exists, it raises RuntimeError. So, call_command_fork is normally recommended.

Note for Ruby 1.9: In Ruby 1.9, call_command_popen is safe and robust enough, and is the recommended way, because IO.popen is improved to get a command-line as an array without calling shell.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) options: Hash

Returns

(undefined)



235
236
237
238
239
240
241
242
243
244
245
 
# File 'lib/bio/command.rb', line 235

def call_command_popen(cmd, options = {}, &block)
  if RUBY_VERSION >= "1.9.0" then
    if RUBY_ENGINE == 'jruby' then
      _call_command_popen_jruby19(cmd, options, &block)
    else
      _call_command_popen_ruby19(cmd, options, &block)
    end
  else
    _call_command_popen_ruby18(cmd, options, &block)
  end
end

.escape_shell(str) ⇒ Object

Escape special characters in command line string.

Arguments:

  • (required) str: String

Returns

String object



121
122
123
124
125
126
127
 
# File 'lib/bio/command.rb', line 121

def escape_shell(str)
  if windows_platform? then
    escape_shell_windows(str)
  else
    escape_shell_unix(str)
  end
end

.escape_shell_unix(str) ⇒ Object

Escape special characters in command line string for UNIX shells.

Arguments:

  • (required) str: String

Returns

String object



110
111
112
113
114
 
# File 'lib/bio/command.rb', line 110

def escape_shell_unix(str)
  str = str.to_s
  raise 'cannot escape control characters' if UNESCAPABLE_CHARS =~ str
  str.gsub(UNSAFE_CHARS_UNIX) { |x| "\\#{x}" }
end

.escape_shell_windows(str) ⇒ Object

Escape special characters in command line string for cmd.exe on Windows.

Arguments:

  • (required) str: String

Returns

String object



95
96
97
98
99
100
101
102
103
 
# File 'lib/bio/command.rb', line 95

def escape_shell_windows(str)
  str = str.to_s
  raise 'cannot escape control characters' if UNESCAPABLE_CHARS =~ str
  if QUOTE_CHARS_WINDOWS =~ str then
    '"' + str.gsub(/\"/, '""') + '"'
  else
    String.new(str)
  end
end

.http_post_form(http, path, params = nil, header = {}) ⇒ Object

Same as:

http = Net::HTTP.new(...); http.post_form(path, params)

and it uses proxy if an environment variable (same as OpenURI.open_uri) is set. In addition, header can be set. (Note that Content-Type and Content-Length are automatically set by default.) uri must be a URI object, params must be a hash, and header must be a hash.

Arguments:

  • (required) http: Net::HTTP object or compatible object

  • (required) path: String

  • (optional) params: Hash containing parameters

  • (optional) header: Hash containing header strings

Returns

(same as Net::HTTP::post_form)



775
776
777
778
779
780
781
782
783
784
785
 
# File 'lib/bio/command.rb', line 775

def http_post_form(http, path, params = nil, header = {})
  data = make_cgi_params(params)

  hash = {
    'Content-Type'   => 'application/x-www-form-urlencoded',
    'Content-Length' => data.length.to_s
  }
  hash.update(header)

  http.post(path, data, hash)
end

.make_cgi_params(params) ⇒ Object

Builds parameter string for from Hash of parameters for application/x-www-form-urlencoded.

Arguments:

  • (required) params: Hash containing parameters

Returns

String



829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
 
# File 'lib/bio/command.rb', line 829

def make_cgi_params(params)
  data = ""
  case params
  when Hash
    data = params.map do |key, val|
      make_cgi_params_key_value(key, val)
    end.join('&')
  when Array
    case params.first
    when Hash
      data = params.map do |hash|
        hash.map do |key, val|
          make_cgi_params_key_value(key, val)
        end
      end.join('&')
    when Array
      data = params.map do |key, val|
        make_cgi_params_key_value(key, val)
      end.join('&')
    when String
      data = params.map do |str|
        key, val = str.split(/\=/, 2)
        if val then
          make_cgi_params_key_value(key, val)
        else
          CGI.escape(str)
        end
      end.join('&')
    end
  when String
    data = URI.escape(params.strip)
  end
  return data
end

.make_cgi_params_key_value(key, value) ⇒ Object

Builds parameter string for from a key string and a value (or values) for application/x-www-form-urlencoded.

Arguments:

  • (required) key: String

  • (required) value: String or Array containing String

Returns

String



872
873
874
875
876
877
878
879
880
881
882
883
 
# File 'lib/bio/command.rb', line 872

def make_cgi_params_key_value(key, value)
  result = []
  case value
  when Array
    value.each do |val|
      result << [key, val].map {|x| CGI.escape(x.to_s) }.join('=')
    end
  else
    result << [key, value].map {|x| CGI.escape(x.to_s) }.join('=')
  end
  return result
end

.make_command_line(ary) ⇒ Object

Generate command line string with special characters escaped.

Arguments:

  • (required) ary: Array containing String objects

Returns

String object



134
135
136
137
138
139
140
 
# File 'lib/bio/command.rb', line 134

def make_command_line(ary)
  if windows_platform? then
    make_command_line_windows(ary)
  else
    make_command_line_unix(ary)
  end
end

.make_command_line_unix(ary) ⇒ Object

Generate command line string with special characters escaped for UNIX shells.

Arguments:

  • (required) ary: Array containing String objects

Returns

String object



158
159
160
 
# File 'lib/bio/command.rb', line 158

def make_command_line_unix(ary)
  ary.collect { |str| escape_shell_unix(str) }.join(" ")
end

.make_command_line_windows(ary) ⇒ Object

Generate command line string with special characters escaped for cmd.exe on Windows.

Arguments:

  • (required) ary: Array containing String objects

Returns

String object



148
149
150
 
# File 'lib/bio/command.rb', line 148

def make_command_line_windows(ary)
  ary.collect { |str| escape_shell_windows(str) }.join(" ")
end

.mktmpdir(prefix_suffix = nil, tmpdir = nil, &block) ⇒ Object

Backport of Dir.mktmpdir in Ruby 1.9.

Same as Dir.mktmpdir(prefix_suffix) in Ruby 1.9.

Arguments:

  • (optional) prefix_suffix: String (or Array, etc.)

  • (optional) tmpdir: String: temporary directory’s path



573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
 
# File 'lib/bio/command.rb', line 573

def mktmpdir(prefix_suffix = nil, tmpdir = nil, &block)
  begin
    Dir.mktmpdir(prefix_suffix, tmpdir, &block)
  rescue NoMethodError
    # backported from Ruby 1.9.2-preview1.
    # ***** Below is excerpted from Ruby 1.9.2-preview1's lib/tmpdir.rb ****
    # ***** Be careful about copyright. ****
    case prefix_suffix
    when nil
      prefix = "d"
      suffix = ""
    when String
      prefix = prefix_suffix
      suffix = ""
    when Array
      prefix = prefix_suffix[0]
      suffix = prefix_suffix[1]
    else
      raise ArgumentError, "unexpected prefix_suffix: #{prefix_suffix.inspect}"
    end
    tmpdir ||= Dir.tmpdir
    t = Time.now.strftime("%Y%m%d")
    n = nil
    begin
      path = "#{tmpdir}/#{prefix}#{t}-#{$$}-#{rand(0x100000000).to_s(36)}"
      path << "-#{n}" if n
      path << suffix
      Dir.mkdir(path, 0700)
    rescue Errno::EEXIST
      n ||= 0
      n += 1
      retry
    end

    if block_given?
      begin
        yield path
      ensure
        remove_entry_secure path
      end
    else
      path
    end
    # ***** Above is excerpted from Ruby 1.9.2-preview1's lib/tmpdir.rb ****
  end
end

.new_http(address, port = 80) ⇒ Object

Same as:

Net::HTTP.new(address, port)

and it uses proxy if an environment variable (same as OpenURI.open_uri) is set.

Arguments:

  • (required) address: String containing host name or IP address

  • (optional) port: port (sanme as Net::HTTP::start)

Returns

(same as Net::HTTP.new except for proxy support)



745
746
747
748
749
750
751
752
753
754
755
 
# File 'lib/bio/command.rb', line 745

def new_http(address, port = 80)
  uri = URI.parse("http://#{address}:#{port}")
  # Note: URI#find_proxy is an unofficial method defined in open-uri.rb.
  # If the spec of open-uri.rb would be changed, we should change below.
  if proxyuri = uri.find_proxy then
    raise 'Non-HTTP proxy' if proxyuri.class != URI::HTTP
    Net::HTTP.new(address, port, proxyuri.host, proxyuri.port)
  else
    Net::HTTP.new(address, port)
  end
end

.post_form(uri, params = nil, header = {}) ⇒ Object

Same as: Net::HTTP.post_form(uri, params) and it uses proxy if an environment variable (same as OpenURI.open_uri) is set. In addition, header can be set. (Note that Content-Type and Content-Length are automatically set by default.) uri must be a URI object, params must be a hash, and header must be a hash.

Arguments:

  • (required) uri: URI object or String

  • (optional) params: Hash containing parameters

  • (optional) header: Hash containing header strings

Returns

(same as Net::HTTP::post_form)



804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
 
# File 'lib/bio/command.rb', line 804

def post_form(uri, params = nil, header = {})
  unless uri.is_a?(URI)
    uri = URI.parse(uri)
  end

  data = make_cgi_params(params)

  hash = {
    'Content-Type'   => 'application/x-www-form-urlencoded',
    'Content-Length' => data.length.to_s
  }
  hash.update(header)

  start_http(uri.host, uri.port) do |http|
    http.post(uri.path, data, hash)
  end
end

.query_command(cmd, query = nil, options = {}) ⇒ Object

Executes the program with the query (String) given to the standard input, waits the program termination, and returns the output data printed to the standard output as a string.

Automatically select popen for Ruby 1.9 or Windows environment and fork for the others.

Available options:

:chdir => "path" : changes working directory to the specified path.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) query: String

  • (optional) options: Hash

Returns

String or nil



442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
 
# File 'lib/bio/command.rb', line 442

def query_command(cmd, query = nil, options = {})
  if RUBY_VERSION >= "1.9.0" then
    return query_command_popen(cmd, query, options)
  elsif no_fork? then
    query_command_popen(cmd, query, options)
  else
    begin
      query_command_fork(cmd, query, options)
    rescue NotImplementedError
      # fork(2) not implemented
      @@no_fork = true
      query_command_fork(cmd, query, options)
    end
  end
end

.query_command_fork(cmd, query = nil, options = {}) ⇒ Object

This method is internally called from the query_command method. In normal case, use query_command, and do not call this method directly.

Executes the program with the query (String) given to the standard input, waits the program termination, and returns the output data printed to the standard output as a string.

Fork (by using IO.popen(“-”)) and exec is used to execute the program.

See the document of query_command for available options.

See the document of call_command_fork for the security and Ruby version specific issues.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) query: String

  • (optional) options: Hash

Returns

String or nil



507
508
509
510
511
512
513
514
515
516
 
# File 'lib/bio/command.rb', line 507

def query_command_fork(cmd, query = nil, options = {})
  ret = nil
  call_command_fork(cmd, options) do |io|
    io.sync = true
    io.print query if query
    io.close_write
    ret = io.read
  end
  ret
end

.query_command_open3(cmd, query = nil) ⇒ Object

Executes the program via Open3.popen3 with the query (String) given to the stain, waits the program termination, and returns the data from stdout and stderr as an array of the strings.

You would use this method only when you really need to get stderr.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) query: String

Returns

Array containing 2 objects: output string (or nil) and stderr string (or nil)



529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
 
# File 'lib/bio/command.rb', line 529

def query_command_open3(cmd, query = nil)
  errorlog = nil
  cmd = safe_command_line_array(cmd)
  Open3.popen3(*cmd) do |pin, pout, perr|
    perr.sync = true
    t = Thread.start { errorlog = perr.read }
    begin
      pin.print query if query
      pin.close
      output = pout.read
    ensure
      t.join
    end
    [ output, errorlog ]
  end
end

.query_command_popen(cmd, query = nil, options = {}) ⇒ Object

This method is internally called from the query_command method. In normal case, use query_command, and do not call this method directly.

Executes the program with the query (String) given to the standard input, waits the program termination, and returns the output data printed to the standard output as a string.

See the document of query_command for available options.

See the document of call_command_popen for the security and Ruby version specific issues.

Arguments:

  • (required) cmd: Array containing String objects

  • (optional) query: String

  • (optional) options: Hash

Returns

String or nil



476
477
478
479
480
481
482
483
484
485
 
# File 'lib/bio/command.rb', line 476

def query_command_popen(cmd, query = nil, options = {})
  ret = nil
  call_command_popen(cmd, options) do |io|
    io.sync = true
    io.print query if query
    io.close_write
    ret = io.read
  end
  ret
end

.read_uri(uri) ⇒ Object

Same as OpenURI.open_uri(uri).read and it uses proxy if an environment variable (same as OpenURI.open_uri) is set.

Arguments:

  • (required) uri: URI object or String

Returns

String



706
707
708
 
# File 'lib/bio/command.rb', line 706

def read_uri(uri)
  OpenURI.open_uri(uri).read
end

.remove_entry_secure(path, force = false) ⇒ Object

Same as FileUtils.remove_entry_secure after Ruby 1.8.3. In Ruby 1.8.2 or previous version, it only shows warning message and does nothing.

It is strongly recommended using Ruby 1.8.5 or later.

Arguments:

  • (required) path: String

  • (optional) force: boolean



555
556
557
558
559
560
561
562
 
# File 'lib/bio/command.rb', line 555

def remove_entry_secure(path, force = false)
  begin
    FileUtils.remove_entry_secure(path, force)
  rescue NoMethodError
    warn "The temporary file or directory is not removed because of the lack of FileUtils.remove_entry_secure. Use Ruby 1.8.3 or later (1.8.5 or later is strongly recommended): #{path}"
    nil
  end
end

.safe_command_line_array(ary) ⇒ Object

Returns an Array of command-line command and arguments that can be safely passed to Kernel.exec etc. If the given array is already safe (or empty), returns the given array.

Arguments:

  • (required) ary: Array

Returns

Array



169
170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/bio/command.rb', line 169

def safe_command_line_array(ary)
  ary = ary.to_ary
  return ary if ary.size >= 2 or ary.empty?
  if ary.size != 1 then
    raise 'Bug: assersion of ary.size == 1 failed'
  end
  arg0 = ary[0]
  begin
    arg0 = arg0.to_ary
  rescue NoMethodError
    arg0 = [ arg0, arg0 ]
  end
  [ arg0 ]
end

.start_http(address, port = 80, &block) ⇒ Object

Same as:

Net::HTTP.start(address, port)

and it uses proxy if an environment variable (same as OpenURI.open_uri) is set.

Arguments:

  • (required) address: String containing host name or IP address

  • (optional) port: port (sanme as Net::HTTP::start)

Returns

(same as Net::HTTP::start except for proxy support)



721
722
723
724
725
726
727
728
729
730
731
732
 
# File 'lib/bio/command.rb', line 721

def start_http(address, port = 80, &block)
  uri = URI.parse("http://#{address}:#{port}")
  # Note: URI#find_proxy is an unofficial method defined in open-uri.rb.
  # If the spec of open-uri.rb would be changed, we should change below.
  if proxyuri = uri.find_proxy then
    raise 'Non-HTTP proxy' if proxyuri.class != URI::HTTP
    http = Net::HTTP.Proxy(proxyuri.host, proxyuri.port)
  else
    http = Net::HTTP
  end
  http.start(address, port, &block)
end