Module: Pkg::Util::Net

Defined in:
lib/packaging/util/net.rb

Overview

Utility methods for handling network calls and interactions

Class Method Summary collapse

Class Method Details

.add_param_to_uri(uri, param) ⇒ Object

Add a parameter to a given uri. If we were sane we’d use encode_www_form(params) of URI, but because we’re not, because that will http encode it, which isn’t what we want since we’re require the encoding provided by escapeHTML of CGI, since this is being transfered in the xml of a jenkins job via curl and DEAR JEEBUS WHAT HAVE WE DONE.



368
369
370
371
372
373
 
# File 'lib/packaging/util/net.rb', line 368

def add_param_to_uri(uri, param)
  require 'uri'
  uri = URI.parse(uri)
  uri.query = [uri.query, param].compact.join('&')
  uri.to_s
end

.check_host(host, options = { required: true }) ⇒ Object

Check that the current host matches the one we think it should



21
22
23
24
25
26
 
# File 'lib/packaging/util/net.rb', line 21

def check_host(host, options = { required: true })
  return true if hostname == host

  fail "Error: #{hostname} does not match #{host}" if options[:required]
  return nil
end

.check_host_gpg(hosts, gpg) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to check for gpg keys

    If the host needs a special username it should be passed in as user@host

  • gpg —

    • The gpg secret key to look for

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



49
50
51
52
53
54
55
56
57
58
 
# File 'lib/packaging/util/net.rb', line 49

def check_host_gpg(hosts, gpg)
  errs = []
  Array(hosts).flatten.each do |host|
    remote_execute(host, "gpg --list-secret-keys #{gpg} > /dev/null 2&>1",
                   { extra_options: '-oBatchMode=yes' })
  rescue StandardError
    errs << host
  end
  return errs
end

.check_host_ssh(hosts) ⇒ Object

Returns an array of hosts where ssh access failed. Empty array if successful.

Parameters:

  • hosts —

    • An array of hosts to try ssh-ing into

    If the host needs a special username it should be passed in as user@host

Returns:

  • an array of hosts where ssh access failed. Empty array if successful



33
34
35
36
37
38
39
40
41
 
# File 'lib/packaging/util/net.rb', line 33

def check_host_ssh(hosts)
  errs = []
  Array(hosts).flatten.each do |host|
    remote_execute(host, 'exit', { extra_options: '-oBatchMode=yes' })
  rescue StandardError
    errs << host
  end
  return errs
end

.curl_form_data(uri, form_data = [], options = {}) ⇒ Object

This is fairly absurd. We’re implementing curl by shelling out. What do I wish we were doing? Using a sweet ruby wrapper around curl, such as Curb or Curb-fu. However, because we’re using clean build systems and trying to make this portable with minimal system requirements, we can’t very well depend on libraries that aren’t in the ruby standard libaries. We could also do this using Net::HTTP but that set of libraries is a rabbit hole to go down when what we’re trying to accomplish is posting multi-part form data that includes file uploads to jenkins. It gets hairy fairly quickly, but, as they say, pull requests accepted.

This method takes three arguments 1) String - the URL to post to 2) Array - Ordered array of name=VALUE curl form parameters 3) Hash - Options to be set



247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
 
# File 'lib/packaging/util/net.rb', line 247

def curl_form_data(uri, form_data = [], options = {})
  curl = Pkg::Util::Tool.check_tool("curl")
  #
  # Begin constructing the post string.
  # First, assemble the form_data arguments
  #
  post_string = "-i "
  form_data.each do |param|
    post_string << "#{param} "
  end

  # Add the uri
  post_string << "'#{uri}'"

  # If this is quiet, we're going to silence all output
  begin
    stdout, _, retval = Pkg::Util::Execution.capture3("#{curl} #{post_string}")
    if options[:quiet]
      stdout = ''
    end
    return stdout, retval
  rescue RuntimeError => e
    puts e
    return false
  end
end

.escape_html(uri) ⇒ Object 



358
359
360
361
 
# File 'lib/packaging/util/net.rb', line 358

def escape_html(uri)
  require 'cgi'
  CGI.escapeHTML(uri)
end

.fetch_uri(uri, target) ⇒ Object

This simple method does an HTTP get of a URI and writes it to a file in a slightly more platform agnostic way than curl/wget



7
8
9
10
11
12
 
# File 'lib/packaging/util/net.rb', line 7

def fetch_uri(uri, target)
  require 'open-uri'
  if Pkg::Util::File.file_writable?(File.dirname(target))
    File.open(target, 'w') { |f| f.puts(URI.parse(uri).read) }
  end
end

.hostname ⇒ Object

Get the hostname of the current host



15
16
17
18
 
# File 'lib/packaging/util/net.rb', line 15

def hostname
  require 'socket'
  Socket.gethostname
end

.print_url_info(url_string) ⇒ Object

Use the provided URL string to print important information with ASCII emphasis



288
289
290
291
292
 
# File 'lib/packaging/util/net.rb', line 288

def print_url_info(url_string)
  puts "\n////////////////////////////////////////////////////////////////////////////////\n\n
  Build submitted. To view your build progress, go to\n#{url_string}\n\n
////////////////////////////////////////////////////////////////////////////////\n\n"
end

.remote_buildparams(host, build) ⇒ Object

Given a BuildInstance object and a host, send its params to the host. Return the remote path to the params.



405
406
407
408
409
410
411
 
# File 'lib/packaging/util/net.rb', line 405

def remote_buildparams(host, build)
  params_file = build.config_to_yaml
  params_file_name = File.basename(params_file)
  params_dir = Pkg::Util.rand_string
  Pkg::Util::Net.rsync_to(params_file, host, "/tmp/#{params_dir}/")
  "/tmp/#{params_dir}/#{params_file_name}"
end

.remote_bundle_install_command ⇒ Object 



395
396
397
398
399
400
401
 
# File 'lib/packaging/util/net.rb', line 395

def remote_bundle_install_command
  rvm_ruby_version = ENV['RVM_RUBY_VERSION'] || '3.1.1'
  export_packaging_location = "export PACKAGING_LOCATION='#{ENV['PACKAGING_LOCATION']}';" if ENV['PACKAGING_LOCATION'] && !ENV['PACKAGING_LOCATION'].empty?
  export_vanagon_location = "export VANAGON_LOCATION='#{ENV['VANAGON_LOCATION']}';" if ENV['VANAGON_LOCATION'] && !ENV['VANAGON_LOCATION'].empty?
  export_gem_source = "export GEM_SOURCE='#{ENV['GEM_SOURCE']}';" if ENV['GEM_SOURCE'] && !ENV['GEM_SOURCE'].empty?
  "source /usr/local/rvm/scripts/rvm; rvm use ruby-#{rvm_ruby_version}; #{export_gem_source} #{export_packaging_location} #{export_vanagon_location} bundle install --path .bundle/gems ;"
end

.remote_create_latest_symlink(package_name, dir, platform_ext, options = {}) ⇒ Object

Create a symlink indicating the latest version of a package

Parameters:

  • package_name (String) —

    The name of the package you want to symlink to, e.g. ‘puppet-agent’, ‘facter’, etc.

  • dir (String) —

    The directory you want to find the latest package and create the symlink in.

  • platform_ext (String) —

    The type of files you want to consider, e.g. ‘dmg’, ‘msi’, etc.

  • options (Hash) (defaults to: {}) —

    Additional optional params: @option :arch [String] Architecture you want to narrow your search by. @option :excludes [Array] Strings you want to exclude from your search,

    e.g. 'agent' if only searching for 'puppet'.



321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
 
# File 'lib/packaging/util/net.rb', line 321

def remote_create_latest_symlink(package_name, dir, platform_ext, options = {})
  ls_cmd = "ls -1 *.#{platform_ext} | grep -v latest | grep -v rc | grep -P '#{package_name}-\\d' "

  # store this in a separate var to avoid side affects
  full_package_name = String.new(package_name)

  if options[:arch]
    ls_cmd << "| grep #{options[:arch]}"
    full_package_name << "-#{options[:arch]}"
  end
  if options[:excludes]
    options[:excludes].each do |excl|
      ls_cmd << "| grep -v #{excl} "
    end
  end
  ls_cmd << '| sort --version-sort | tail -1'
  cmd = <<-CMD
    if [ ! -d '#{dir}' ] ; then
      echo "directory '#{dir}' does not exist, not creating latest package link"
      exit 0
    fi
    pushd '#{dir}'
    link_target=$(#{ls_cmd})
    if [ -z "$link_target" ] ; then
      echo "Unable to find a link target for '#{full_package_name}' in '#{dir}'; skipping link creation"
      exit 0
    fi
    echo "creating link to '$link_target'"
    ln -sf "$link_target" #{full_package_name}-latest.#{platform_ext}
  CMD

  _, err = Pkg::Util::Net.remote_execute(
    Pkg::Config.staging_server, cmd, { capture_output: true }
  )
  warn err
end

.remote_execute(target_host, command, user_options = {}) ⇒ Object 



60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
 
# File 'lib/packaging/util/net.rb', line 60

def remote_execute(target_host, command, user_options = {})
  option_defaults = {
    capture_output: false,
    extra_options: '',
    fail_fast: true,
    trace: false
  }
  options = option_defaults.merge(user_options)

  ssh = Pkg::Util::Tool.check_tool('ssh')

  # we pass some pretty complicated commands in via ssh. We need this to fail
  # if any part of the remote ssh command fails.
  shell_flags = ''
  shell_flags += 'set -e;' if options[:fail_fast]
  shell_flags += 'set -x;' if options[:trace]
  shell_commands = "#{shell_flags}#{command}"

  remote_command = "#{ssh} #{options[:extra_options]} -t #{target_host} " +
                   "'#{shell_commands.gsub("'", "'\\\\''")}'"

  # This is NOT a good way to support this functionality.
  if ENV['DRYRUN']
    puts "[DRY-RUN] Executing '#{command}' on #{target}"
    puts "[DRY-RUN] #{cmd}"
    return ''
  end

  # We're forced to make different calls depending on the capture_output option
  # because something about our #capture3 method screws up gpg. This should
  # be untangled.
  if options[:capture_output]
    stdout, stderr, exitstatus = Pkg::Util::Execution.capture3(remote_command)
    Pkg::Util::Execution.success?(exitstatus) or
      raise "Remote ssh command (\"#{remote_command}\") failed."
    return stdout, stderr
  end

  # Pkg::Util::Execution.capture3 reports its command but Kernel.system does not
  # Let's print it out for some amount of consistency.
  puts "Remote Execute: '#{remote_command}'"
  Kernel.system(remote_command)
  Pkg::Util::Execution.success? or
    raise "Remote ssh command (\"#{remote_command}\") failed."
end

.remote_set_immutable(host, files) ⇒ Object

Remotely set the immutable bit on a list of files



305
306
307
 
# File 'lib/packaging/util/net.rb', line 305

def remote_set_immutable(host, files)
  Pkg::Util::Net.remote_execute(host, "sudo chattr +i #{files.join(' ')}")
end

.remote_set_ownership(host, owner, group, files) ⇒ Object 



294
295
296
297
 
# File 'lib/packaging/util/net.rb', line 294

def remote_set_ownership(host, owner, group, files)
  remote_cmd = "for file in #{files.join(' ')}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chown #{owner}:#{group} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_execute(host, remote_cmd)
end

.remote_set_permissions(host, permissions, files) ⇒ Object 



299
300
301
302
 
# File 'lib/packaging/util/net.rb', line 299

def remote_set_permissions(host, permissions, files)
  remote_cmd = "for file in #{files.join(' ')}; do if [[ -d $file ]] || ! `lsattr $file | grep -q '\\-i\\-'`; then sudo chmod #{permissions} $file; else echo \"$file is immutable\"; fi; done"
  Pkg::Util::Net.remote_execute(host, remote_cmd)
end

.remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true, trace = false) ⇒ Object

Deprecated method implemented as a shim to the new ‘remote_execute` method



109
110
111
112
113
114
115
116
117
 
# File 'lib/packaging/util/net.rb', line 109

def remote_ssh_cmd(target, command, capture_output = false, extra_options = '', fail_fast = true, trace = false) # rubocop:disable Metrics/ParameterLists
  puts "Warn: \"remote_ssh_cmd\" call in packaging is deprecated. Use \"remote_execute\" instead."
  remote_execute(target, command, {
                   capture_output: capture_output,
                   extra_options: extra_options,
                   fail_fast: fail_fast,
                   trace: trace
})
end

.remote_unpack_git_bundle(host, treeish, tar_cmd = nil, tarball = nil) ⇒ Object

We take a tar argument for cases where ‘tar` isn’t best, e.g. Solaris. We also take an optional argument of the tarball containing the git bundle to use.



378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
 
# File 'lib/packaging/util/net.rb', line 378

def remote_unpack_git_bundle(host, treeish, tar_cmd = nil, tarball = nil)
  unless tar = tar_cmd
    tar = 'tar'
  end
  tarball ||= Pkg::Util::Git.bundle(treeish)
  tarball_name = File.basename(tarball).gsub('.tar.gz', '')
  Pkg::Util::Net.rsync_to(tarball, host, '/tmp')
  appendix = Pkg::Util.rand_string
  git_bundle_directory = File.join('/tmp', "#{Pkg::Config.project}-#{appendix}")
  command = <<~DOC
    #{tar} -zxvf /tmp/#{tarball_name}.tar.gz -C /tmp/ ;
    git clone --recursive /tmp/#{tarball_name} #{git_bundle_directory} ;
  DOC
  Pkg::Util::Net.remote_execute(host, command)
  return git_bundle_directory
end

.rsync_cmd(origin_path, opts = {}) ⇒ String

Construct a valid rsync command

Parameters:

  • origin_path (String, Pathname) —

    the path to sync from; if opts is not passed, then the parent directory of ‘origin_path` will be used to construct a target path to sync to.

  • opts (Hash) (defaults to: {}) —

    additional options that can be used to construct the rsync command.

Options Hash (opts):

  • :bin (String) — default: 'rsync' —

    the path to rsync (can be relative or fully qualified).

  • :origin_host (String) —

    the remote host to sync data from; cannot be specified alongside :target_host

  • :target_host (String) —

    the remote host to sync data to; cannot be specified alongside :origin_host.

  • :extra_flags (String) — default: ["--ignore-existing"] —

    extra flags to use when constructing an rsync command

  • :dryrun (String) — default: false —

    tell rsync to perform a trial run with no changes made.

Returns:

  • (String) —

    a rsync command that can be used in shell or ssh methods

Raises:

  • (ArgumentError) —

    if opts and opts names are both defined.

  • (ArgumentError) —

    if :origin_path exists without opts, opts, remote target is defined.



140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
 
# File 'lib/packaging/util/net.rb', line 140

def rsync_cmd(origin_path, opts = {})
  options = {
    bin: 'rsync',
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: false
}.merge(opts)
  origin = Pathname.new(origin_path)
  target = options[:target_path] || origin.parent

  raise(ArgumentError, "Cannot sync between two remote hosts") if
    options[:origin_host] && options[:target_host]

  raise(ArgumentError, "Cannot sync path '#{origin}' because both origin_host and target_host are nil. Perhaps you need to set TEAM=release ?") unless
    options[:origin_host] || options[:target_host]

  cmd = %W[
    #{options[:bin]}
    --recursive
    --hard-links
    --links
    --verbose
    --omit-dir-times
    --no-perms
    --no-owner
    --no-group
  ] + [*options[:extra_flags]]

  cmd << '--dry-run' if options[:dryrun]
  cmd << Pkg::Util.pseudo_uri(path: origin, host: options[:origin_host])
  cmd << Pkg::Util.pseudo_uri(path: target, host: options[:target_host])

  cmd.uniq.compact.join("\s")
end

.rsync_exec(source, opts = {}) ⇒ Object

A generic rsync execution method that wraps rsync_cmd in a call to Pkg::Util::Execution#capture3()



179
180
181
182
183
184
185
186
187
188
189
190
191
 
# File 'lib/packaging/util/net.rb', line 179

def rsync_exec(source, opts = {})
  options = {
    bin: Pkg::Util::Tool.check_tool('rsync'),
    origin_host: nil,
    target_path: nil,
    target_host: nil,
    extra_flags: nil,
    dryrun: ENV['DRYRUN']
}.merge(opts.merge(opts.delete_if { |_, value| value.nil? }))

  stdout, = Pkg::Util::Execution.capture3(rsync_cmd(source, options), true)
  stdout
end

.rsync_from(source, origin_host, dest, opts = {}) ⇒ Object

A wrapper method to maintain the existing interface for executing incoming rsync commands with minimal changes to existing code.



208
209
210
211
212
213
214
215
216
217
 
# File 'lib/packaging/util/net.rb', line 208

def rsync_from(source, origin_host, dest, opts = {})
  rsync_exec(
    source,
    origin_host: origin_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] }) ⇒ Object

A wrapper method to maintain the existing interface for executing outbound rsync commands with minimal changes to existing code.



195
196
197
198
199
200
201
202
203
204
 
# File 'lib/packaging/util/net.rb', line 195

def rsync_to(source, target_host, dest, opts = { extra_flags: ["--ignore-existing"] })
  rsync_exec(
    source,
    target_host: target_host,
    target_path: dest,
    extra_flags: opts[:extra_flags],
    dryrun: opts[:dryrun],
    bin: opts[:bin],
  )
end

.s3sync_to(source, target_bucket, target_directory = '', flags = []) ⇒ Object 



219
220
221
222
223
224
225
226
227
228
229
230
231
 
# File 'lib/packaging/util/net.rb', line 219

def s3sync_to(source, target_bucket, target_directory = '', flags = [])
  s3cmd = Pkg::Util::Tool.check_tool('s3cmd')
  s3cfg_path = File.join(ENV['HOME'], '.s3cfg')

  unless File.exist?(s3cfg_path)
    fail "#{s3cfg_path} does not exist. It is required to ship files using s3cmd."
  end

  sync_command = "#{s3cmd} sync #{flags.join(' ')} '#{source}' " \
                 "s3://#{target_bucket}/#{target_directory}/"

  Pkg::Util::Execution.capture3(sync_command, true)
end

.uri_status_code(uri) ⇒ Object 



274
275
276
277
278
279
280
281
282
283
284
 
# File 'lib/packaging/util/net.rb', line 274

def uri_status_code(uri)
  data = [
    '--request GET',
    '--silent',
    '--location',
    '--write-out "%{http_code}"',
    '--output /dev/null'
  ]
  stdout, = Pkg::Util::Net.curl_form_data(uri, data)
  stdout
end