Module: PuppetLitmus::RakeHelper

Included in:
PuppetLitmus
Defined in:
lib/puppet_litmus/rake_helper.rb

Overview

helper methods for the litmus rake tasks

Defined Under Namespace

Classes: LitmusTimeoutError

Constant Summary collapse

DEFAULT_CONFIG_DATA =

DEFAULT_CONFIG_DATA should be frozen for our safety, but it needs to work around github.com/puppetlabs/bolt/pull/1696

{ 'modulepath' => File.join(Dir.pwd, 'spec', 'fixtures', 'modules') }
SUPPORTED_PROVISIONERS = 
%w[abs docker docker_exp lxd provision_service vagrant vmpooler].freeze

Instance Method Summary collapse

Instance Method Details

#build_module(module_dir = nil, target_dir = nil) ⇒ String

Build the module in ‘module_dir` and put the resulting compressed tarball into `target_dir`.

Parameters:

  • opts —

    Hash of options to build the module

  • module_dir (String) (defaults to: nil) —

    The path of the module to build. If missing defaults to Dir.pwd

  • target_dir (String) (defaults to: nil) —

    The path the module will be built into. The default is <module_dir>/pkg

Returns:

  • (String) —

    The path to the built module



177
178
179
180
181
182
183
184
185
186
187
188
189
190
 
# File 'lib/puppet_litmus/rake_helper.rb', line 177

def build_module(module_dir = nil, target_dir = nil)
  require 'puppet/modulebuilder'

  module_dir ||= Dir.pwd
  target_dir ||= File.join(source_dir, 'pkg')

  puts "Building '#{module_dir}' into '#{target_dir}'"
  builder = Puppet::Modulebuilder::Builder.new(module_dir, target_dir, nil)

  # Force the metadata to be read. Raises if metadata could not be found
  _metadata = builder.metadata

  builder.build
end

#build_modules_in_dir(source_dir, target_dir = nil) ⇒ Array

Builds all the modules in a specified directory

Parameters:

  • source_dir (String) —

    the directory to get the modules from

  • target_dir (String) (defaults to: nil) —

    temporary location to store tarballs before uploading. This directory will be cleaned before use. The default is <source_dir>/pkg

Returns:

  • (Array) —

    an array of module tars’ filenames



197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
 
# File 'lib/puppet_litmus/rake_helper.rb', line 197

def build_modules_in_dir(source_dir, target_dir = nil)
  target_dir ||= File.join(Dir.pwd, 'pkg')
  # remove old build dir if exists, before we build afresh
  FileUtils.rm_rf(target_dir) if File.directory?(target_dir)

  module_tars = Dir.entries(source_dir).map do |entry|
    next if ['.', '..'].include? entry

    module_dir = File.join(source_dir, entry)
    next unless File.directory? module_dir

    build_module(module_dir, target_dir)
  end
  module_tars.compact
end

#build_modules_in_folder(source_folder) ⇒ Object

Deprecated.

Use ‘build_modules_in_dir` instead



214
215
216
 
# File 'lib/puppet_litmus/rake_helper.rb', line 214

def build_modules_in_folder(source_folder)
  build_modules_in_dir(source_folder)
end

#check_bolt_errors(result_set) ⇒ Hash

Parse out errors messages in result set returned by Bolt command.

Parameters:

  • result_set (Array) —

    result set returned by Bolt command.

Returns:

  • (Hash) —

    Errors grouped by target.



308
309
310
311
312
313
314
315
316
317
318
319
320
321
 
# File 'lib/puppet_litmus/rake_helper.rb', line 308

def check_bolt_errors(result_set)
  errors = {}
  # iterate through each error
  result_set.each do |target_result|
    status = target_result['status']
    # jump to the next one when there is not fail
    next if status != 'failure'

    target = target_result['target']
    # get some info from error
    errors[target] = target_result['value']
  end
  errors
end

#check_connectivity?(inventory_hash, target_node_name) ⇒ Boolean

Returns:

  • (Boolean) 


275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
 
# File 'lib/puppet_litmus/rake_helper.rb', line 275

def check_connectivity?(inventory_hash, target_node_name)
  # if we're only checking connectivity for a single node
  add_platform_field(inventory_hash, target_node_name) if target_node_name

  include ::BoltSpec::Run
  target_nodes = find_targets(inventory_hash, target_node_name)
  puts "Checking connectivity for #{target_nodes.inspect}"

  results = run_command('cd .', target_nodes, config: nil, inventory: inventory_hash)
  failed = []
  results.reject { |r| r['status'] == 'success' }.each do |result|
    puts "Failure connecting to #{result['target']}:\n#{result.inspect}"
    failed.push(result['target'])
  end
  raise "Connectivity has failed on: #{failed}" unless failed.empty?

  puts 'Connectivity check PASSED.'
  true
end

#configure_path(inventory_hash) ⇒ Object 



157
158
159
160
161
162
163
164
165
166
167
168
169
 
# File 'lib/puppet_litmus/rake_helper.rb', line 157

def configure_path(inventory_hash)
  results = []
  # fix the path on ssh_nodes
  unless inventory_hash['groups'].none? { |group| group['name'] == 'ssh_nodes' && !group['targets'].empty? }
    results << run_command('echo PATH="$PATH:/opt/puppetlabs/puppet/bin" > /etc/environment',
                           'ssh_nodes', config: nil, inventory: inventory_hash)
  end
  unless inventory_hash['groups'].none? { |group| group['name'] == 'winrm_nodes' && !group['targets'].empty? }
    results << run_command('[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\Program Files\Puppet Labs\Puppet\bin;C:\Program Files (x86)\Puppet Labs\Puppet\bin", "Machine")',
                           'winrm_nodes', config: nil, inventory: inventory_hash)
  end
  results
end

#get_metadata_operating_systems(metadata) ⇒ String

Gets a string representing the operating system and version.

Parameters:

  • metadata (Hash) —

    metadata to parse for operating system info

Returns:

  • (String) —

    the operating system string with version info for use in provisioning.



16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
 
# File 'lib/puppet_litmus/rake_helper.rb', line 16

def get_metadata_operating_systems(metadata)
  return unless metadata.is_a?(Hash)
  return unless metadata['operatingsystem_support'].is_a?(Array)

  metadata['operatingsystem_support'].each do |os_info|
    next unless os_info['operatingsystem'] && os_info['operatingsystemrelease']

    os_name = case os_info['operatingsystem']
              when 'Amazon', 'Archlinux', 'AIX', 'OSX'
                next
              when 'OracleLinux'
                'oracle'
              when 'Windows'
                'win'
              else
                os_info['operatingsystem'].downcase
              end

    os_info['operatingsystemrelease'].each do |release|
      version = case os_name
                when 'ubuntu', 'osx'
                  release.sub('.', '')
                when 'sles'
                  release.gsub(%r{ SP[14]}, '')
                when 'win'
                  release = release.delete('.') if release.include? '8.1'
                  release.sub('Server', '').sub('10', '10-pro')
                else
                  release
                end

      yield "#{os_name}-#{version.downcase}-x86_64".delete(' ')
    end
  end
end

#install_agent(collection, targets, inventory_hash) ⇒ Object 



130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
 
# File 'lib/puppet_litmus/rake_helper.rb', line 130

def install_agent(collection, targets, inventory_hash)
  include ::BoltSpec::Run
  params = if collection.nil?
             {}
           else
             { 'collection' => collection }
           end
  raise "puppet_agent was not found in #{DEFAULT_CONFIG_DATA['modulepath']}, please amend the .fixtures.yml file" \
    unless File.directory?(File.join(DEFAULT_CONFIG_DATA['modulepath'], 'puppet_agent'))

  # using boltspec, when the runner is called it changes the inventory_hash dropping the version field. The clone works around this
  bolt_result = run_task('puppet_agent::install', targets, params, config: DEFAULT_CONFIG_DATA, inventory: inventory_hash.clone)
  targets.each do |target|
    params = {
      'path' => '/opt/puppetlabs/bin'
    }
    node_facts = facts_from_node(inventory_hash, target)
    next unless node_facts['provisioner'] == 'vagrant'

    puts "Adding puppet agent binary to the secure_path on target #{target}."
    result = run_task('provision::fix_secure_path', target, params, config: DEFAULT_CONFIG_DATA, inventory: inventory_hash.clone)
    raise_bolt_errors(result, "Failed to add the Puppet agent binary to the secure_path on target #{target}.")
  end
  raise_bolt_errors(bolt_result, 'Installation of agent failed.')
  bolt_result
end

#install_module(inventory_hash, target_node_name, module_tar, module_repository = nil, ignore_dependencies = false) ⇒ Object

Install a specific module tarball to the specified target. This method installs dependencies using a forge repository.

Parameters:

  • inventory_hash (Hash) —

    the pre-loaded inventory

  • target_node_name (String) —

    the name of the target where the module should be installed

  • module_tar (String) —

    the filename of the module tarball to upload

  • module_repository (String) (defaults to: nil) —

    the URL for the forge to use for downloading modules. Defaults to the public Forge API.

  • ignore_dependencies (Boolean) (defaults to: false) —

    flag used to ignore module dependencies defaults to false.

Returns:

  • a bolt result



227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
 
# File 'lib/puppet_litmus/rake_helper.rb', line 227

def install_module(inventory_hash, target_node_name, module_tar, module_repository = nil, ignore_dependencies = false) # rubocop:disable Style/OptionalBooleanParameter
  # make sure the module to install is not installed
  # otherwise `puppet module install` might silently skip it
  module_name = File.basename(module_tar, '.tar.gz').split('-', 3)[0..1].join('-')
  uninstall_module(inventory_hash.clone, target_node_name, module_name, force: true)

  include ::BoltSpec::Run

  target_nodes = find_targets(inventory_hash, target_node_name)
  bolt_result = upload_file(module_tar, File.basename(module_tar), target_nodes, options: {}, config: nil, inventory: inventory_hash.clone)
  raise_bolt_errors(bolt_result, 'Failed to upload module.')

  module_repository_opts = "--module_repository '#{module_repository}'" unless module_repository.nil?
  install_module_command = "puppet module install #{module_repository_opts} #{File.basename(module_tar)}"
  install_module_command += ' --ignore-dependencies --force' if ignore_dependencies.to_s.casecmp('true').zero?

  bolt_result = run_command(install_module_command, target_nodes, config: nil, inventory: inventory_hash.clone)
  raise_bolt_errors(bolt_result, "Installation of package #{File.basename(module_tar)} failed.")
  bolt_result
end

#metadata_module_name ⇒ Object 



248
249
250
251
252
253
254
255
256
 
# File 'lib/puppet_litmus/rake_helper.rb', line 248

def metadata_module_name
  require 'json'
  raise 'Could not find metadata.json' unless File.exist?(File.join(Dir.pwd, 'metadata.json'))

  metadata = JSON.parse(File.read(File.join(Dir.pwd, 'metadata.json')))
  raise 'Could not read module name from metadata.json' if metadata['name'].nil?

  metadata['name']
end

#provision(provisioner, platform, inventory_vars) ⇒ Object 



66
67
68
69
70
71
72
73
74
75
76
77
78
79
 
# File 'lib/puppet_litmus/rake_helper.rb', line 66

def provision(provisioner, platform, inventory_vars)
  include ::BoltSpec::Run
  raise "the provision module was not found in #{DEFAULT_CONFIG_DATA['modulepath']}, please amend the .fixtures.yml file" unless
    File.directory?(File.join(DEFAULT_CONFIG_DATA['modulepath'], 'provision'))

  params = { 'action' => 'provision', 'platform' => platform, 'inventory' => File.join(Dir.pwd, 'spec', 'fixtures', 'litmus_inventory.yaml') }
  params['vars'] = inventory_vars unless inventory_vars.nil?

  task_name = provisioner_task(provisioner)
  bolt_result = run_task(task_name, 'localhost', params, config: DEFAULT_CONFIG_DATA, inventory: nil)
  raise_bolt_errors(bolt_result, "provisioning of #{platform} failed.")

  bolt_result
end

#provision_list(provision_hash, key) ⇒ Object 



81
82
83
84
85
86
87
88
89
 
# File 'lib/puppet_litmus/rake_helper.rb', line 81

def provision_list(provision_hash, key)
  provisioner = provision_hash[key]['provisioner']
  inventory_vars = provision_hash[key]['vars']
  # Splat the params into environment variables to pass to the provision task but only in this runspace
  provision_hash[key]['params']&.each { |k, value| ENV[k.upcase] = value.to_s }
  provision_hash[key]['images'].map do |image|
    provision(provisioner, image, inventory_vars)
  end
end

#provisioner_task(provisioner) ⇒ Object 



295
296
297
298
299
300
301
302
 
# File 'lib/puppet_litmus/rake_helper.rb', line 295

def provisioner_task(provisioner)
  if SUPPORTED_PROVISIONERS.include?(provisioner)
    "provision::#{provisioner}"
  else
    warn "WARNING: Unsupported provisioner '#{provisioner}', try #{SUPPORTED_PROVISIONERS.join('/')}"
    provisioner.to_s
  end
end

#raise_bolt_errors(result_set, error_msg) ⇒ Object

Parse out errors messages in result set returned by Bolt command. If there are errors, raise them.

Parameters:

  • result_set (Array) —

    result set returned by Bolt command.

  • error_msg (String) —

    error message to raise when errors are detected. The actual errors will be appended.



327
328
329
330
331
332
333
334
335
336
 
# File 'lib/puppet_litmus/rake_helper.rb', line 327

def raise_bolt_errors(result_set, error_msg)
  errors = check_bolt_errors(result_set)

  unless errors.empty?
    formatted_results = errors.map { |k, v| "  #{k}: #{v.inspect}" }.join("\n")
    raise "#{error_msg}\nResults:\n#{formatted_results}}"
  end

  nil
end

#run_local_command(command) ⇒ Object

Executes a command on the test runner.

Parameters:

  • command (String) —

    command to execute.

Returns:

  • (Object) —

    the standard out stream.



56
57
58
59
60
61
62
63
64
 
# File 'lib/puppet_litmus/rake_helper.rb', line 56

def run_local_command(command)
  require 'open3'
  stdout, stderr, status = Open3.capture3(command)
  error_message = "Attempted to run\ncommand:'#{command}'\nstdout:#{stdout}\nstderr:#{stderr}"

  raise error_message unless status.to_i.zero?

  stdout
end

#start_spinner(message) ⇒ Object 



338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
 
# File 'lib/puppet_litmus/rake_helper.rb', line 338

def (message)
  if (ENV['CI'] || '').casecmp('true').zero? || Gem.win_platform?
    puts message
     = Thread.new do
      # CI systems are strange beasts, we only output a '.' every wee while to keep the terminal alive.
      loop do
        printf '.'
        sleep(10)
      end
    end
  else
    require 'tty-spinner'
     = TTY::Spinner.new("[:spinner] #{message}")
    .auto_spin
  end
  
end

#stop_spinner(spinner) ⇒ Object 



356
357
358
359
360
361
362
 
# File 'lib/puppet_litmus/rake_helper.rb', line 356

def ()
  if (ENV['CI'] || '').casecmp('true').zero? || Gem.win_platform?
    Thread.kill()
  else
    .success
  end
end

#tear_down(node_name, inventory_hash) ⇒ Object 



119
120
121
122
123
124
125
126
127
128
 
# File 'lib/puppet_litmus/rake_helper.rb', line 119

def tear_down(node_name, inventory_hash)
  # how do we know what provisioner to use
  add_platform_field(inventory_hash, node_name)

  params = { 'action' => 'tear_down', 'node_name' => node_name, 'inventory' => File.join(Dir.pwd, 'spec', 'fixtures', 'litmus_inventory.yaml') }
  node_facts = facts_from_node(inventory_hash, node_name)
  bolt_result = run_task(provisioner_task(node_facts['provisioner']), 'localhost', params, config: DEFAULT_CONFIG_DATA, inventory: nil)
  raise_bolt_errors(bolt_result, "tear_down of #{node_name} failed.")
  bolt_result
end

#tear_down_nodes(targets, inventory_hash) ⇒ Object 



91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/puppet_litmus/rake_helper.rb', line 91

def tear_down_nodes(targets, inventory_hash)
  include ::BoltSpec::Run
  config_data = { 'modulepath' => File.join(Dir.pwd, 'spec', 'fixtures', 'modules') }
  raise "the provision module was not found in #{config_data['modulepath']}, please amend the .fixtures.yml file" unless File.directory?(File.join(config_data['modulepath'], 'provision'))

  results = {}
  targets.each do |node_name|
    #  next if local host or provisioner fact empty/not set (GH-421)
    next if node_name == 'litmus_localhost' || facts_from_node(inventory_hash, node_name)['provisioner'].nil?

    result = tear_down(node_name, inventory_hash)
    # Some provisioners tear_down targets that were created as a batch job.
    # These provisioners should return the list of additional targets
    # removed so that we do not attempt to process them.
    if result != [] && result[0]['value'].key?('removed')
      removed_targets = result[0]['value']['removed']
      result[0]['value'].delete('removed')
      removed_targets.each do |removed_target|
        targets.delete(removed_target)
        results[removed_target] = result
      end
    end

    results[node_name] = result unless result == []
  end
  results
end

#uninstall_module(inventory_hash, target_node_name, module_to_remove = nil, **opts) ⇒ Object

Uninstall a module from a specified target

Parameters:

  • inventory_hash (Hash) —

    the pre-loaded inventory

  • target_node_name (String) —

    the name of the target where the module should be uninstalled

  • module_to_remove (String) (defaults to: nil) —

    the name of the module to remove. Defaults to the module under test.

  • opts (Hash) —

    additional options to pass on to ‘puppet module uninstall`



263
264
265
266
267
268
269
270
271
272
273
 
# File 'lib/puppet_litmus/rake_helper.rb', line 263

def uninstall_module(inventory_hash, target_node_name, module_to_remove = nil, **opts)
  include ::BoltSpec::Run
  module_name = module_to_remove || metadata_module_name
  target_nodes = find_targets(inventory_hash, target_node_name)
  install_module_command = "puppet module uninstall #{module_name}"
  install_module_command += ' --force' if opts[:force]
  bolt_result = run_command(install_module_command, target_nodes, config: nil, inventory: inventory_hash)
  # `puppet module uninstall --force` fails if the module is not installed. Ignore errors when force is set
  raise_bolt_errors(bolt_result, "uninstalling #{module_name} failed.") unless opts[:force]
  bolt_result
end

#with_retries(options: { tries: Float::INFINITY }, max_wait_minutes: 15) ⇒ Object 



375
376
377
378
379
380
381
382
 
# File 'lib/puppet_litmus/rake_helper.rb', line 375

def with_retries(options: { tries: Float::INFINITY }, max_wait_minutes: 15)
  stop = Time.now + (max_wait_minutes * 60)
  Retryable.retryable(options.merge(not: [LitmusTimeoutError])) do
    raise LitmusTimeoutError if Time.now > stop

    yield
  end
end