Class: Beaker::AwsSdk

Inherits:
Hypervisor show all
Defined in:
lib/beaker/hypervisor/aws_sdk.rb

Overview

This is an alternate EC2 driver that implements direct API access using Amazon’s AWS-SDK library: aws.amazon.com/documentation/sdkforruby/

It is built for full control, to reduce any other layers beyond the pure vendor API.

Constant Summary collapse

ZOMBIE =

anything older than 3 hours is considered a zombie

3

Constants inherited from Hypervisor

Hypervisor::CHARMAP

Constants included from HostPrebuiltSteps

HostPrebuiltSteps::APT_CFG, HostPrebuiltSteps::DEBIAN_PACKAGES, HostPrebuiltSteps::ETC_HOSTS_PATH, HostPrebuiltSteps::ETC_HOSTS_PATH_SOLARIS, HostPrebuiltSteps::IPS_PKG_REPO, HostPrebuiltSteps::NTPSERVER, HostPrebuiltSteps::ROOT_KEYS_SCRIPT, HostPrebuiltSteps::ROOT_KEYS_SYNC_CMD, HostPrebuiltSteps::SLEEPWAIT, HostPrebuiltSteps::SLES_PACKAGES, HostPrebuiltSteps::TRIES, HostPrebuiltSteps::UNIX_PACKAGES, HostPrebuiltSteps::WINDOWS_PACKAGES

Instance Method Summary collapse

Methods inherited from Hypervisor

#configure, create, #generate_host_name, #validate

Methods included from HostPrebuiltSteps

#add_el_extras, #apt_get_update, #copy_file_to_remote, #copy_ssh_to_root, #disable_iptables, #disable_se_linux, #enable_root_login, #epel_info_for, #get_domain_name, #get_ip, #hack_etc_hosts, #package_proxy, #proxy_config, #set_etc_hosts, #sync_root_keys, #timesync, #validate_host

Methods included from DSL::Patterns

#block_on

Constructor Details

#initialize(hosts, options) ⇒ AwsSdk

Initialize AwsSdk hypervisor driver

Parameters:

  • hosts (Array<Beaker::Host>) —

    Array of Beaker::Host objects

  • options (Hash<String, String>) —

    Options hash



19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 19

def initialize(hosts, options)
  @hosts = hosts
  @options = options
  @logger = options[:logger]

  # Get fog credentials from the local .fog file
  creds = load_fog_credentials(@options[:dot_fog])

  config = {
    :access_key_id => creds[:access_key],
    :secret_access_key => creds[:secret_key],
    :logger => Logger.new($stdout),
    :log_level => :debug,
    :log_formatter => AWS::Core::LogFormatter.colored,
    :max_retries => 12,
  }
  AWS.config(config)

  @ec2 = AWS::EC2.new()
end

Instance Method Details

#add_tags ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Add metadata tags to all instances



280
281
282
283
284
285
286
287
288
289
290
291
292
293
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 280

def add_tags
  @hosts.each do |host|
    instance = host['instance']

    # Define tags for the instance
    @logger.notify("aws-sdk: Add tags for #{host.name}")
    instance.add_tag("jenkins_build_url", :value => @options[:jenkins_build_url])
    instance.add_tag("Name", :value => host.name)
    instance.add_tag("department", :value => @options[:department])
    instance.add_tag("project", :value => @options[:project])
  end

  nil
end

#backoff_sleep(tries) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Calculates and waits a back-off period based on the number of tries

Logs each backupoff time and retry value to the console.

Parameters:

  • tries (Number) —

    number of tries to calculate back-off period



355
356
357
358
359
360
361
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 355

def backoff_sleep(tries)
  # Exponential with some randomization
  sleep_time = 2 ** tries
  @logger.notify("aws-sdk: Sleeping #{sleep_time} seconds for attempt #{tries}.")
  sleep sleep_time
  nil
end

#cleanup ⇒ void

This method returns an undefined value.

Cleanup all earlier provisioned hosts on EC2 using the AWS::EC2 library

It goes without saying, but a #cleanup does nothing without a #provision method call first.



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 75

def cleanup
  @logger.notify("aws-sdk: Cleanup, iterating across all hosts and terminating them")
  @hosts.each do |host|
    # This was set previously during provisioning
    instance = host['instance']

    # Only attempt a terminate if the instance actually is set by provision
    # and the instance actually 'exists'.
    if !instance.nil? and instance.exists?
      instance.terminate
    end
  end

  nil #void
end

#configure_hosts ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Configure /etc/hosts for each node



317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 317

def configure_hosts
  @hosts.each do |host|
    etc_hosts = "127.0.0.1\tlocalhost localhost.localdomain\n"
    name = host.name
    domain = get_domain_name(host)
    ip = host['private_ip']
    etc_hosts += "#{ip}\t#{name} #{name}.#{domain} #{host['dns_name']}\n"
    @hosts.each do |neighbor|
      if neighbor == host
        next
      end
      name = neighbor.name
      domain = get_domain_name(neighbor)
      ip = neighbor['ip']
      etc_hosts += "#{ip}\t#{name} #{name}.#{domain} #{neighbor['dns_name']}\n"
    end
    set_etc_hosts(host, etc_hosts)
  end
end

#create_group(region, ports) ⇒ AWS::EC2::SecurityGroup

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Create a new security group

Parameters:

  • region (AWS::EC2::Region) —

    the AWS region control object

  • ports (Array<Number>) —

    an array of port numbers

Returns:

  • (AWS::EC2::SecurityGroup) —

    created security group



457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 457

def create_group(region, ports)
  name = group_id(ports)
  @logger.notify("aws-sdk: Creating group #{name} for ports #{ports.to_s}")
  group = region.security_groups.create(name,
                                        :description => "Custom Beaker security group for #{ports.to_a}")

  unless ports.is_a? Set
    ports = Set.new(ports)
  end

  ports.each do |port|
    group.authorize_ingress(:tcp, port)
  end

  group
end

#ensure_group(region, ports) ⇒ AWS::EC2::SecurityGroup

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return an existing group, or create new one

Parameters:

  • region (AWS::EC2::Region) —

    the AWS region control object

  • ports (Array<Number>) —

    an array of port numbers

Returns:

  • (AWS::EC2::SecurityGroup) —

    created security group



438
439
440
441
442
443
444
445
446
447
448
449
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 438

def ensure_group(region, ports)
  @logger.notify("aws-sdk: Ensure security group exists for ports #{ports.to_s}, create if not")
  name = group_id(ports)

  group = region.security_groups.filter('group-name', name).first

  if group.nil?
    group = create_group(region, ports)
  end

  group
end

#ensure_key_pair(region) ⇒ AWS::EC2::KeyPair

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns the KeyPair for this host, creating it if needed

Parameters:

  • region (AWS::EC2::Region) —

    region to create the key pair in

Returns:

  • (AWS::EC2::KeyPair) —

    created key_pair



401
402
403
404
405
406
407
408
409
410
411
412
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 401

def ensure_key_pair(region)
  @logger.notify("aws-sdk: Ensure key pair exists, create if not")
  key_pairs = region.key_pairs
  pair_name = key_name()
  kp = key_pairs[pair_name]
  unless kp.exists?
    ssh_string = public_key()
    kp = key_pairs.import(pair_name, ssh_string)
  end

  kp
end

#group_id(ports) ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return a reproducable security group identifier based on input ports

Parameters:

  • ports (Array<Number>) —

    array of port numbers

Returns:

  • (String) —

    group identifier



419
420
421
422
423
424
425
426
427
428
429
430
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 419

def group_id(ports)
  if ports.nil? or ports.empty?
    raise ArgumentError, "Ports list cannot be nil or empty"
  end

  unless ports.is_a? Set
    ports = Set.new(ports)
  end

  # Lolwut, #hash is inconsistent between ruby processes
  "Beaker-#{Zlib.crc32(ports.inspect)}"
end

#key_name ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Generate a reusable key name from the local hosts hostname

Returns:

  • (String) —

    safe key name for current host



383
384
385
386
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 383

def key_name
  safe_hostname = Socket.gethostname.gsub('.', '-')
  "Beaker-#{local_user}-#{safe_hostname}"
end

#kill_zombies(max_age = ZOMBIE, key = key_name) ⇒ Object

Shutdown and destroy ec2 instances idenfitied by key that have been alive longer than ZOMBIE hours.

Parameters:

  • max_age (Integer) (defaults to: ZOMBIE) —

    The age in hours that a machine needs to be older than to be considered a zombie

  • key (String) (defaults to: key_name) —

    The key_name to match for



116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 116

def kill_zombies(max_age = ZOMBIE, key = key_name)
  @logger.notify("aws-sdk: Kill Zombies! (keyname: #{key}, age: #{max_age} hrs)")
  #examine all available regions
  kill_count = 0
  volume_count = 0
  time_now = Time.now.getgm #ec2 uses GM time
  @ec2.regions.each do |region|
    @logger.debug "Reviewing: #{region.name}"
    # Note: don't use instances.each here as that funtion doesn't allow proper rescue from error states
    instances = @ec2.regions[region.name].instances
    instances.each do |instance|
      begin
        if (instance.key_name =~ /#{key}/)
          @logger.debug "Examining #{instance.id} (keyname: #{instance.key_name}, launch time: #{instance.launch_time}, status: #{instance.status})"
          if ((time_now - instance.launch_time) >  max_age*60*60) and instance.status.to_s !~ /terminated/
            @logger.debug "Kill! #{instance.id}: #{instance.key_name} (Current status: #{instance.status})"
              instance.terminate()
              kill_count += 1
          end
        end
      rescue AWS::Core::Resource::NotFound, AWS::EC2::Errors => e
        @logger.debug "Failed to remove instance: #{instance.id}, #{e}"
      end
    end
    # Occasionaly, tearing down ec2 instances leaves orphaned EBS volumes behind -- these stack up quickly.
    # This simply looks for EBS volumes that are not in use
    # Note: don't use volumes.each here as that funtion doesn't allow proper rescue from error states
    volumes = @ec2.regions[region.name].volumes.map { |vol| vol.id }
    volumes.each do |vol|
      begin
        vol = @ec2.regions[region.name].volumes[vol]
        if ( vol.status.to_s =~ /available/ )
          @logger.debug "Tear down available volume: #{vol.id}"
          vol.delete()
          volume_count += 1
        end
      rescue AWS::EC2::Errors::InvalidVolume::NotFound => e
        @logger.debug "Failed to remove volume: #{vol.id}, #{e}"
      end
    end
  end
  @logger.notify "#{key}: Killed #{kill_count} instance(s), freed #{volume_count} volume(s)"

end

#launch_all_nodes ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Launch all nodes

This is where the main launching work occurs for each node. Here we take care of feeding the information from the image required into the config for the new host, we perform the launch operation and ensure that the instance is properly tagged for identification.



170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 170

def launch_all_nodes
  # Load the ec2_yaml spec file
  ami_spec = YAML.load_file(@options[:ec2_yaml])["AMI"]

  # Iterate across all hosts and launch them, adding tags along the way
  @logger.notify("aws-sdk: Iterate across all hosts in configuration and launch them")
  @hosts.each do |host|
    amitype = host['vmname'] || host['platform']
    amisize = host['amisize'] || 'm1.small'

    # Use snapshot provided for this host
    image_type = host['snapshot']
    if not image_type
      raise RuntimeError, "No snapshot/image_type provided for EC2 provisioning"
    end
    ami = ami_spec[amitype]
    ami_region = ami[:region]

    # Main region object for ec2 operations
    region = @ec2.regions[ami_region]

    # Grab image object
    image_id = ami[:image][image_type.to_sym]
    @logger.notify("aws-sdk: Checking image #{image_id} exists and getting its root device")
    image = region.images[image_id]
    if image.nil? and not image.exists?
      raise RuntimeError, "Image not found: #{image_id}"
    end

    # Transform the images block_device_mappings output into a format
    # ready for a create.
    orig_bdm = image.block_device_mappings()
    @logger.notify("aws-sdk: Image block_device_mappings: #{orig_bdm.to_hash}")
    block_device_mappings = []
    orig_bdm.each do |device_name, rest|
      block_device_mappings << {
        :device_name => device_name,
        :ebs => {
          # This is required to override the images default for
          # delete_on_termination, forcing all volumes to be deleted once the
          # instance is terminated.
          :delete_on_termination => true,
        }
      }
    end

    # Launch the node, filling in the blanks from previous work.
    @logger.notify("aws-sdk: Launch instance")
    config = {
      :count => 1,
      :image_id => image_id,
      :monitoring_enabled => true,
      :key_pair => ensure_key_pair(region),
      :security_groups => [ensure_group(region, Beaker::EC2Helper.amiports(host['roles']))],
      :instance_type => amisize,
      :disable_api_termination => false,
      :instance_initiated_shutdown_behavior => "terminate",
      :block_device_mappings => block_device_mappings,
    }
    instance = region.instances.create(config)

    # Persist the instance object for this host, so later it can be
    # manipulated by 'cleanup' for example.
    host['instance'] = instance

    @logger.notify("aws-sdk: Launched #{host.name} (#{amitype}:#{amisize}) using snapshot/image_type #{image_type}")
  end

  nil
end

#load_fog_credentials(dot_fog = '.fog') ⇒ Hash<Symbol, String>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Return a hash containing the fog credentials for EC2

Parameters:

  • dot_fog (String) (defaults to: '.fog') —

    dot fog path

Returns:

  • (Hash<Symbol, String>) —

    ec2 credentials



479
480
481
482
483
484
485
486
487
488
489
490
491
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 479

def load_fog_credentials(dot_fog = '.fog')
  fog = YAML.load_file( dot_fog )

  default = fog[:default]

  creds = {}
  creds[:access_key] = default[:aws_access_key_id]
  creds[:secret_key] = default[:aws_secret_access_key]
  raise "You must specify an aws_access_key_id in your .fog file (#{dot_fog}) for ec2 instances!" unless creds[:access_key]
  raise "You must specify an aws_secret_access_key in your .fog file (#{dot_fog}) for ec2 instances!" unless creds[:secret_key]

  creds
end

#local_user ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns the local user running this tool

Returns:

  • (String) —

    username of local user



392
393
394
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 392

def local_user
  ENV['USER']
end

#log_instances(key = key_name, status = /running/) ⇒ Object

Print instances to the logger. Instances will be from all regions associated with provided key name and limited by regex compared to instance status. Defaults to running instances.

Parameters:

  • key (String) (defaults to: key_name) —

    The key_name to match for

  • status (Regex) (defaults to: /running/) —

    The regular expression to match against the instance’s status



95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 95

def log_instances(key = key_name, status = /running/)
  instances = []
  @ec2.regions.each do |region|
    @logger.debug "Reviewing: #{region.name}"
    @ec2.regions[region.name].instances.each do |instance|
      if (instance.key_name =~ /#{key}/) and (instance.status.to_s =~ status)
        instances << instance
      end
    end
  end
  output = ""
  instances.each do |instance|
    output << "#{instance.id} keyname: #{instance.key_name}, dns name: #{instance.dns_name}, private ip: #{instance.private_ip_address}, ip: #{instance.ip_address}, launch time #{instance.launch_time}, status: #{instance.status}\n"
  end
  @logger.notify("aws-sdk: List instances (keyname: #{key})")
  @logger.notify("#{output}")
end

#populate_dns ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Populate the hosts IP address from the EC2 dns_name



299
300
301
302
303
304
305
306
307
308
309
310
311
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 299

def populate_dns
  # Obtain the IP addresses and dns_name for each host
  @hosts.each do |host|
    @logger.notify("aws-sdk: Populate DNS for #{host.name}")
    instance = host['instance']
    host['ip'] = instance.ip_address
    host['private_ip'] = instance.private_ip_address
    host['dns_name'] = instance.dns_name
    @logger.notify("aws-sdk: name: #{host.name} ip: #{host['ip']} private_ip: #{host['private_ip']} dns_name: #{instance.dns_name}")
  end

  nil
end

#provision ⇒ void

This method returns an undefined value.

Provision all hosts on EC2 using the AWS::EC2 API



43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 43

def provision
  start_time = Time.now

  # Perform the main launch work
  launch_all_nodes()

  # Wait for each node to reach status :running
  wait_for_status(:running)

  # Add metadata tags to each instance
  add_tags()

  # Grab the ip addresses and dns from EC2 for each instance to use for ssh
  populate_dns()

  # Set the hostname for each box
  set_hostnames()

  # Configure /etc/hosts on each host
  configure_hosts()

  @logger.notify("aws-sdk: Provisioning complete in #{Time.now - start_time} seconds")

  nil #void
end

#public_key ⇒ String

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Retrieve the public key locally from the executing users ~/.ssh directory

Returns:

  • (String) —

    contents of public key



367
368
369
370
371
372
373
374
375
376
377
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 367

def public_key
  filename = File.expand_path('~/.ssh/id_rsa.pub')
  unless File.exists? filename
    filename = File.expand_path('~/.ssh/id_dsa.pub')
    unless File.exists? filename
      raise RuntimeError, 'Expected either ~/.ssh/id_rsa.pub or ~/.ssh/id_dsa.pub but found neither'
    end
  end

  File.read(filename)
end

#set_hostnames ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Set the hostname of all instances to be the hostname defined in the beaker configuration.



342
343
344
345
346
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 342

def set_hostnames
  @hosts.each do |host|
    host.exec(Command.new("hostname #{host.name}"))
  end
end

#wait_for_status(status) ⇒ void

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

This method returns an undefined value.

Waits until all boxes reach the desired status

Parameters:

  • status (Symbol) —

    EC2 state to wait for, :running :stopped etc.



246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
 
# File 'lib/beaker/hypervisor/aws_sdk.rb', line 246

def wait_for_status(status)
  # Wait for each node to reach status :running
  @logger.notify("aws-sdk: Now wait for all hosts to reach state #{status}")
  @hosts.each do |host|
    instance = host['instance']
    name = host.name

    @logger.notify("aws-sdk: Wait for status #{status} for node #{name}")

    # Here we keep waiting for the machine state to reach ':running' with an
    # exponential backoff for each poll.
    # TODO: should probably be a in a shared method somewhere
    for tries in 1..10
      begin
        if instance.status == status
          # Always sleep, so the next command won't cause a throttle
          backoff_sleep(tries)
          break
        elsif tries == 10
          raise "Instance never reached state #{status}"
        end
      rescue AWS::EC2::Errors::InvalidInstanceID::NotFound => e
        @logger.debug("Instance #{name} not yet available (#{e})")
      end
      backoff_sleep(tries)
    end

  end
end