Class: Dpl::Provider

Inherits:
Cl::Cmd
  • Object
show all
Extended by:
Dsl, Forwardable
Includes:
Assets, ConfigFile, Env, Interpolate, FileUtils, Memoize, Squiggle
Defined in:
lib/dpl/provider.rb,
lib/dpl/provider/status.rb,
lib/dpl/provider/dsl.rb

Overview

Base class for all concrete providers that ‘dpl` supports.

These are subclasses of ‘Cl::Cmd` which means they are going to be detected by the first argument passed to `dpl [provider]`, instantiated, and run.

Implementors are encouraged to use the provider DSL to declare various features, requirements, and attributes that apply to their provider, to implement any of the following stages (methods) according to their needs and semantics:

* init
* install
* login
* setup
* validate
* prepare
* deploy
* finish

The main logic should sit in the ‘deploy` stage.

If at any time the method ‘error` is called, or any exception raised the deploy process will be halted, and subsequent stages skipped. However, the stage `finish` will run even if previous stages have raised an error, giving the provider the opportunity to potentially clean up stage.

In addition to this the following methods will be called if implemented by the provider:

* run_cmd
* add_key
* remove_key

Like the ‘finish` stage, the method `remove_key` will be called even if previous stages have raised an error.

See the respective method’s documentation for details on these.

The following stages are not meant to be overwritten, but considered internal:

* before_install
* before_setup
* before_prepare
* before_finish

Dependencies declared as required, such as APT, NPM, or Python are going to be installed as part of the ‘before_install` stage .

Cleanup is run as part of the ‘before_prepare` stage if the option `–cleanup` was given. This will use `git stash –all` in order to reset the working directory to the committed state, and cleanup any left over artifacts from the build process. Providers can use the DSL method `keep` in order to declare known artifacts (such as CLI tooling installed to the working directory) that needs to be moved out of the way and restored after the cleanup process. (It is recommended to place such artifacts outside of the build working directory though, for example in `~/.dpl`).

The method ‘run_cmd` is called for each command specified using the `–run` option. By default, these command are going to be run as local shell commands, but providers can choose to overwrite this method in order to run the command on a remote machine.

providers (commands) are declared and run.

See Also:

Defined Under Namespace

Modules: Dsl Classes: Status

Constant Summary collapse

FOLDS =

Fold names to display in the build log.

{
  init:     'Initialize deployment',
  setup:    'Setup deployment',
  validate: 'Validate deployment',
  install:  'Install deployment dependencies',
  login:    'Authenticate deployment',
  prepare:  'Prepare deployment',
  deploy:   'Run deployment',
  finish:   'Finish deployment',
}
STAGES =

Deployment process stages.

In addition to the stages listed here the stage ‘finish` will be run at the end of the process.

Also, the methods ‘add_key` (called before `setup`), `remove_key` (called before `finish`), and `run_cmd` (called after `deploy`) may be of interest to implementors.

%i(
  init
  install
  login
  setup
  validate
  prepare
  deploy
)

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Dsl

apt, apt?, cmds, description, env, errs, full_name, gem, gem?, keep, move, msgs, needs, needs?, node_js, npm, npm?, opt, path, pip, pip?, python, ruby_pre?, ruby_version, runtimes, status, strs, summary, user_agent, vars

Methods included from Assets

#asset

Methods included from Env

included, #opts

Methods included from ConfigFile

included, #opts

Methods included from Interpolate

#interpolate, #obfuscate, #vars

Methods included from Memoize

included

Constructor Details

#initialize(ctx, *args) ⇒ Provider

Returns a new instance of Provider.



193
194
195
196
197
 
# File 'lib/dpl/provider.rb', line 193

def initialize(ctx, *args)
  @repo_name = ctx.repo_name
  @key_name = ctx.machine_name
  super
end

Instance Attribute Details

#key_name ⇒ Object (readonly)

Returns the value of attribute key_name.



191
192
193
 
# File 'lib/dpl/provider.rb', line 191

def key_name
  @key_name
end

#repo_name ⇒ Object (readonly)

Returns the value of attribute repo_name.



191
192
193
 
# File 'lib/dpl/provider.rb', line 191

def repo_name
  @repo_name
end

Class Method Details

.examples ⇒ Object 



88
89
90
 
# File 'lib/dpl/provider.rb', line 88

def examples
  @examples ||= super || Examples.new(self).cmds
end

.install_deps(ctx) ⇒ Object 



104
105
106
107
108
109
 
# File 'lib/dpl/provider.rb', line 104

def install_deps(ctx)
  ctx.apts_get(apt) if apt?
  ctx.gems_require(gem) if gem?
  npm.each { |npm| ctx.npm_install *npm } if npm?
  pip.each { |pip| ctx.pip_install *pip } if pip?
end

.install_deps? ⇒ Boolean

Returns:

  • (Boolean) 


100
101
102
 
# File 'lib/dpl/provider.rb', line 100

def install_deps?
  apt? || gem? || npm? || pip?
end

.move_files(ctx) ⇒ Object 



92
93
94
 
# File 'lib/dpl/provider.rb', line 92

def move_files(ctx)
  ctx.move_files(move) if move.any?
end

.unmove_files(ctx) ⇒ Object 



96
97
98
 
# File 'lib/dpl/provider.rb', line 96

def unmove_files(ctx)
  ctx.unmove_files(move) if move.any?
end

.validate_runtimes(ctx) ⇒ Object 



111
112
113
 
# File 'lib/dpl/provider.rb', line 111

def validate_runtimes(ctx)
  ctx.validate_runtimes(runtimes) if runtimes.any?
end

Instance Method Details

#before_finish ⇒ Object

Finalizes the deployment process.

This will:

  • Call the method ‘remove_key` if implemented by the provider, and if the feature `ssh_key` has been declared as required.

  • Revert the cleanup process, i.e. restore files moved out of the way during ‘cleanup`.

  • Remove the temporary directory ‘~/.dpl`



312
313
314
315
316
317
 
# File 'lib/dpl/provider.rb', line 312

def before_finish
  remove_key if needs?(:ssh_key) && respond_to?(:remove_key)
  uncleanup if cleanup?
  unmove_files(ctx)
  remove_dpl_dir
end

#before_init ⇒ Object

Initialize the deployment process.

This will:

  • Displays warning messages about the provider’s maturity status, and deprecated options used.

  • Setup a ~/.dpl working directory

  • Move files out of the way that have been declared as such



247
248
249
250
251
252
 
# File 'lib/dpl/provider.rb', line 247

def before_init
  warn status.msg if status && status.announce?
  deprecations.each { |(key, msg)| ctx.deprecate_opt(key, msg) }
  setup_dpl_dir
  move_files(ctx)
end

#before_install ⇒ Object

Install APT, NPM, and Python dependencies as declared by the provider.



255
256
257
258
259
260
 
# File 'lib/dpl/provider.rb', line 255

def before_install
  validate_runtimes(ctx)
  return unless install_deps?
  info :before_install
  install_deps(ctx)
end

#before_prepare ⇒ Object

Prepares the deployment by cleaning up the working directory.

See Also:



280
281
282
 
# File 'lib/dpl/provider.rb', line 280

def before_prepare
  cleanup if cleanup?
end

#before_setup ⇒ Object

Sets the build environment up for the deployment.

This will:

  • Setup a ~/.dpl working directory

  • Create a temporary, per build SSH key, and call ‘add_key` if the feature `ssh_key` has been declared as required.

  • Setup git config (email and user name) if the feature ‘git` has been declared as required.

  • Either set or unset the environment variable ‘GIT_HTTP_USER_AGENT` depending if the feature `git_http_user_agent` has been declared as required.



270
271
272
273
274
275
 
# File 'lib/dpl/provider.rb', line 270

def before_setup
  info :before_setup
  setup_ssh_key if needs?(:ssh_key)
  setup_git_config if needs?(:git)
  setup_git_http_user_agent
end

#chmod(perm, path) ⇒ Object 



625
626
627
 
# File 'lib/dpl/provider.rb', line 625

def chmod(perm, path)
  super(perm, expand(path))
end

#cleanup ⇒ Object

Resets the current working directory to the commited state.

Cleanup will use ‘git stash –all` in order to reset the working directory to the committed state, and cleanup any left over artifacts from the build process. Providers can use the DSL method `keep` in order to declare known artifacts (such as CLI tooling installed to the working directory) that needs to be moved out of the way and restored after the cleanup process.



327
328
329
330
331
332
 
# File 'lib/dpl/provider.rb', line 327

def cleanup
  info :cleanup
  keep.each { |path| shell "mv ./#{path} ~/#{path}", echo: false, assert: false }
  shell 'git stash --all'
  keep.each { |path| shell "mv ~/#{path} ./#{path}", echo: false, assert: false }
end

#cmd ⇒ Object

Looks up a shell command from the commands declared by the provider (using the class level DSL).

Not usually useful to be used by provider implementors directly. Use the method ‘shell` in order to execute shell commands.




# File 'lib/dpl/provider.rb', line 510

#compact(hash) ⇒ Object

Compacts the given hash by rejecting nil values.



596
597
598
 
# File 'lib/dpl/provider.rb', line 596

def compact(hash)
  hash.reject { |_, value| value.nil? }
end

#err ⇒ Object

Looks up an error message from the error messages declared by the provider (using the class level DSL), as needed by the option ‘assert` when passed to the method `shell`.




# File 'lib/dpl/provider.rb', line 517

#error ⇒ Object

Outputs an error message to stderr, and raises an error, halting the deployment process.

This method is intended to be used for all error conditions that require the deployment process to be aborted.

Messages support interpolation variables. See Dpl::Interpolate for details on interpolating variables.



503
504
505
506
507
508
 
# File 'lib/dpl/provider.rb', line 503

%i(print info warn error).each do |level|
  define_method(level) do |msg, *args|
    msg = interpolate(self.msg(msg), args) if msg.is_a?(Symbol)
    ctx.send(level, msg)
  end
end

#escape(str) ⇒ Object

Escapes the given string so it can be safely used in Bash.



550
551
552
 
# File 'lib/dpl/provider.rb', line 550

def escape(str)
  Shellwords.escape(str)
end

#expand(*args) ⇒ Object 



645
646
647
 
# File 'lib/dpl/provider.rb', line 645

def expand(*args)
  File.expand_path(*args)
end

#file?(path) ⇒ Boolean

Returns:

  • (Boolean) 


617
618
619
 
# File 'lib/dpl/provider.rb', line 617

def file?(path)
  File.file?(expand(path))
end

#finish? ⇒ Boolean

Returns:

  • (Boolean) 


220
221
222
 
# File 'lib/dpl/provider.rb', line 220

def finish?
  stage.size == STAGES.size
end

#fold(name, opts = {}, &block) ⇒ Object

Creates a log fold.

Folds any log output from the given block into a fold with the given name.



415
416
417
418
419
 
# File 'lib/dpl/provider.rb', line 415

def fold(name, opts = {}, &block)
  return yield unless fold?(name, opts)
  title = FOLDS[name] || "deploy.#{name}"
  ctx.fold(title, &block)
end

#fold?(name, opts = {}) ⇒ Boolean

Checks if the given stage needs to be folded.

Depends on the option ‘–fold`, also omits folds for the init and finish stages. Can be overwritten by passing `fold: false`.

Returns:

  • (Boolean) 


425
426
427
 
# File 'lib/dpl/provider.rb', line 425

def fold?(name, opts = {})
  !opts[:fold].is_a?(FalseClass) && super() && !%i(init).include?(name)
end

#info ⇒ Object

Outputs an info message to stdout

This method is intended to be used for default, info level messages that are supposed to show up in the build log.

Outputs an warning message to stderr

This method is intended to be used for warning messages that are supposed to show up in the build log, but do not qualify as errors that would abort the deployment process. The warning will be highlighted as red text. Use sparingly.

Messages support interpolation variables. See Dpl::Interpolate for details on interpolating variables.




# File 'lib/dpl/provider.rb', line 477

#mkdir_p(path) ⇒ Object 



621
622
623
 
# File 'lib/dpl/provider.rb', line 621

def mkdir_p(path)
  FileUtils.mkdir_p(expand(path))
end

#msg ⇒ Object

Looks up a message from the messages declared by the provider (using the class level DSL).

For example, a message declared on the class body like so:

```ruby
msgs commit_msg: 'Commit build artifacts on build %{build_number}'
```

could be used by the implementation like so:

```ruby
def commit_msg
  interpolate(msg(:commit_msg))
end
```

Note that the the method ‘interpolate` needs to be used in order to interpolate variables used in a message (if any).



542
543
544
545
546
547
 
# File 'lib/dpl/provider.rb', line 542

%i(cmd err msg str).each do |name|
  define_method(name) do |*keys|
    key = keys.detect { |key| key.is_a?(Symbol) }
    self.class.send(:"#{name}s")[key] if key
  end
end

#mv(src, dest) ⇒ Object 



629
630
631
 
# File 'lib/dpl/provider.rb', line 629

def mv(src, dest)
  super(expand(src), expand(dest))
end

#only(hash, *keys) ⇒ Object

Returns a new hash with the given keys selected from the given hash.



601
602
603
 
# File 'lib/dpl/provider.rb', line 601

def only(hash, *keys)
  hash.select { |key, _| keys.include?(key) }
end

#open(path, *args, &block) ⇒ Object 



637
638
639
 
# File 'lib/dpl/provider.rb', line 637

def open(path, *args, &block)
  File.open(expand(path), *args, &block)
end

#opt_for(key, opts = {}) ⇒ Object 



583
584
585
586
587
588
589
 
# File 'lib/dpl/provider.rb', line 583

def opt_for(key, opts = {})
  case value = send(key)
  when String then "#{opt_key(key, opts)}=#{value.inspect}"
  when Array  then value.map { |value| "#{opt_key(key, opts)}=#{value.inspect}" }
  else opt_key(key, opts)
  end
end

#opt_key(key, opts) ⇒ Object 



591
592
593
 
# File 'lib/dpl/provider.rb', line 591

def opt_key(key, opts)
  "#{opts[:prefix] || '--'}#{opts[:dashed] ? key.to_s.gsub('_', '-') : key}"
end

#opts_for(keys, opts = {}) ⇒ Object

Generate shell option strings to be passed to a shell command.

This generates strings like ‘–key=“value”` for the option keys passed. These keys are supposed to correspond to methods on the provider instance, which will be called in order to determine the option value.

If the returned value is an array then the option will be repeated multiple times. If it is a String then it will be double quoted. Otherwise it is assumed to be a flag that does not have a value.

Parameters:

  • prefix (Hash) —

    a customizable set of options

  • dashed (Hash) —

    a customizable set of options



578
579
580
581
 
# File 'lib/dpl/provider.rb', line 578

def opts_for(keys, opts = {})
  strs = Array(keys).map { |key| opt_for(key, opts) if send(:"#{key}?") }.compact
  strs.join(' ') if strs.any?
end

#print ⇒ Object

Prints a partial message to stdout

This method does not append a newline character to the given message, which usually is not the desired behaviour. The method is intended to be used if an initial, partial message is supposed to be printed, which will be completed later (using the method ‘info`).

For example:

print 'Starting a long running task ...'
run_long_running_task
info 'done.'

Messages support interpolation variables. See Dpl::Interpolate for details on interpolating variables.




# File 'lib/dpl/provider.rb', line 460

#quote(str) ⇒ Object

Double quotes the given string.



555
556
557
 
# File 'lib/dpl/provider.rb', line 555

def quote(str)
  %("#{str.to_s.gsub('"', '\"')}")
end

#read(path) ⇒ Object 



641
642
643
 
# File 'lib/dpl/provider.rb', line 641

def read(path)
  File.read(expand(path))
end

#remove_dpl_dir ⇒ Object

Remove the internal working directory ‘~/.dpl`.



347
348
349
 
# File 'lib/dpl/provider.rb', line 347

def remove_dpl_dir
  rm_rf '~/.dpl'
end

#rm_rf(path) ⇒ Object 



633
634
635
 
# File 'lib/dpl/provider.rb', line 633

def rm_rf(path)
  super(expand(path))
end

#run ⇒ Object

Runs all stages, all commands provided by the user, as well as the final stage ‘finish` (which will be run even if an error has been raised during previous stages).



202
203
204
205
206
207
208
209
210
211
212
213
 
# File 'lib/dpl/provider.rb', line 202

def run
  stages = stage.select { |stage| run_stage?(stage) }
  stages.each { |stage| run_stage(stage) }
  run_cmds
rescue Error
  raise
rescue Exception => e
  raise Error.new("#{e.message} (#{e.class})", backtrace: backtrace? ? e.backtrace : nil) unless test?
  raise
ensure
  run_stage(:finish, fold: false) if finish?
end

#run_cmd(cmd) ⇒ Object 



299
300
301
 
# File 'lib/dpl/provider.rb', line 299

def run_cmd(cmd)
  cmd.downcase == 'restart' ? restart : shell(cmd)
end

#run_cmds ⇒ Object

Runs each command as given by the user using the ‘–run` option.

For a command that matches ‘restart` the method `restart` will be called (which can be overwritten by providers, e.g. in order to restart service instances).

All other commands will be passed to the method ‘run_cmd`. By default this will be run as a shell command locally, but providers can choose to overwrite this method in order to run the command on a remote machine.



293
294
295
296
297
 
# File 'lib/dpl/provider.rb', line 293

def run_cmds
  Array(opts[:run]).each do |cmd|
    cmd.downcase == 'restart' ? restart : run_cmd(cmd)
  end
end

#run_stage(stage, opts = {}) ⇒ Object

Runs a single stage.

For each stage the base class has the opportunity to implement a ‘before` stage method, in order to apply default behaviour. Provider implementors are asked to not overwrite these methods.

Any log output from both the before stage and stage method is going to be folded in the resulting build log.



232
233
234
235
236
237
 
# File 'lib/dpl/provider.rb', line 232

def run_stage(stage, opts = {})
  fold(stage, opts) do
    send(:"before_#{stage}") if respond_to?(:"before_#{stage}")
    send(stage) if respond_to?(stage)
  end
end

#run_stage?(stage) ⇒ Boolean

Whether or not a stage needs to be run

Returns:

  • (Boolean) 


216
217
218
 
# File 'lib/dpl/provider.rb', line 216

def run_stage?(stage)
  respond_to?(:"before_#{stage}") || respond_to?(stage)
end

#script(name, opts = {}) ⇒ Object

Runs a script as a shell command.

Scripts can be stored as separate files (assets) in the directory ‘lib/dpl/assets/`.

This is meant for large shell commands that would be hard to read if embedded in Ruby code. Storing them as separate files helps with proper syntax highlighting etc in editors, and allows to execute them for testing purposes.

Scripts can have interpolation variables. See Dpl::Interpolate for details on interpolating variables.

See Ctx::Bash#shell for details on the options accepted.



443
444
445
446
 
# File 'lib/dpl/provider.rb', line 443

def script(name, opts = {})
  opts[:assert] = name if opts[:assert].is_a?(TrueClass)
  shell(asset(name).read, opts.merge(echo: false))
end

#setup_dpl_dir ⇒ Object

Creates the directory ‘~/.dpl` as an internal working directory.



340
341
342
343
344
 
# File 'lib/dpl/provider.rb', line 340

def setup_dpl_dir
  rm_rf '~/.dpl'
  mkdir_p '~/.dpl'
  chmod 0700, '~/.dpl'
end

#setup_git_config ⇒ Object

Setup git config

This adds the current user’s name and email address (as user@localhost) to the git config.



368
369
370
371
 
# File 'lib/dpl/provider.rb', line 368

def setup_git_config
  shell "git config user.email >/dev/null 2>/dev/null || git config user.email `whoami`@localhost", echo: false, assert: false
  shell "git config user.name  >/dev/null 2>/dev/null || git config user.name  `whoami`", echo: false, assert: false
end

#setup_git_http_user_agent ⇒ Object

Sets or unsets the environment variable ‘GIT_HTTP_USER_AGENT`.



389
390
391
392
393
 
# File 'lib/dpl/provider.rb', line 389

def setup_git_http_user_agent
  return ENV.delete('GIT_HTTP_USER_AGENT') unless needs?(:git_http_user_agent)
  info :setup_git_ua
  ENV['GIT_HTTP_USER_AGENT'] = user_agent(git: `git --version`[/[\d\.]+/])
end

#setup_git_ssh(key) ⇒ Object

Sets up ‘git-ssh` and the GIT_SSH env var



374
375
376
377
378
379
380
 
# File 'lib/dpl/provider.rb', line 374

def setup_git_ssh(key)
  info :setup_git_ssh
  path, conf = '~/.dpl/git-ssh', asset(:dpl, :git_ssh).read % expand(key)
  open(path, 'w+') { |file| file.write(conf) }
  chmod(0740, path)
  ENV['GIT_SSH'] = expand(path)
end

#setup_ssh_key ⇒ Object

Creates an SSH key, and sets up git-ssh if needed.

This will:

  • Create a temporary, per build SSH key.

  • Setup a ‘git-ssh` executable to use that key.

  • Call the method ‘add_key` if implemented by the provider.



358
359
360
361
362
 
# File 'lib/dpl/provider.rb', line 358

def setup_ssh_key
  ssh_keygen(key_name, '~/.dpl/id_rsa')
  setup_git_ssh('~/.dpl/id_rsa')
  add_key('~/.dpl/id_rsa.pub') if respond_to?(:add_key)
end

#shell(cmd, *args) ⇒ Object

Runs a single shell command.

Shell commands can have interpolation variables. See Dpl::Interpolate for details on interpolating variables.

See Ctx::Bash#shell for details on the options accepted.



454
455
456
457
458
 
# File 'lib/dpl/provider.rb', line 454

def shell(cmd, *args)
  opts = args.last.is_a?(Hash) ? args.pop : {}
  cmd = Cmd.new(self, cmd, opts)
  ctx.shell(cmd)
end

#sq(str) ⇒ Object

Outdents the given string.

See Also:

  • Squiggle


562
563
564
 
# File 'lib/dpl/provider.rb', line 562

def sq(str)
  self.class.sq(str)
end

#ssh_keygen(key, path) ⇒ Object

Generates an SSH key.



383
384
385
386
 
# File 'lib/dpl/provider.rb', line 383

def ssh_keygen(key, path)
  info :ssh_keygen
  ctx.ssh_keygen(key, expand(path))
end

#symbolize(obj) ⇒ Object

Deep symbolizes the given hash’s keys



606
607
608
609
610
611
612
613
614
615
 
# File 'lib/dpl/provider.rb', line 606

def symbolize(obj)
  case obj
  when Hash
    obj.map { |key, obj| [key.to_sym, symbolize(obj)] }.to_h
  when Array
    obj.map { |obj| symbolize(obj) }
  else
    obj
  end
end

#try_ssh_access(host, port) ⇒ Object

Tries to connect to the given SSH host and port.



406
407
408
409
 
# File 'lib/dpl/provider.rb', line 406

def try_ssh_access(host, port)
  info :ssh_try_connect
  shell "#{ENV['GIT_SSH']} #{host} -p #{port} 2>&1 | grep -c 'PTY allocation request failed' > /dev/null", echo: false, assert: false
end

#uncleanup ⇒ Object

Restore files that have been cleaned up.



335
336
337
 
# File 'lib/dpl/provider.rb', line 335

def uncleanup
  shell 'git stash pop', assert: false
end

#wait_for_ssh_access(host, port) ⇒ Object

Waits for SSH access on the given host and port.

This will try to connect to the given SSH host and port, and keep retrying 30 times, waiting a second inbetween retries.



399
400
401
402
403
 
# File 'lib/dpl/provider.rb', line 399

def wait_for_ssh_access(host, port)
  info :ssh_remote_host, host, port
  1.upto(20) { try_ssh_access(host, port) && break || sleep(3) }
  success? ? info(:ssh_connected) : error(:ssh_failed)
end

#warn ⇒ Object

Outputs an info message to stdout

This method is intended to be used for default, info level messages that are supposed to show up in the build log.

Outputs an warning message to stderr

This method is intended to be used for warning messages that are supposed to show up in the build log, but do not qualify as errors that would abort the deployment process. The warning will be highlighted as red text. Use sparingly.

Messages support interpolation variables. See Dpl::Interpolate for details on interpolating variables.




# File 'lib/dpl/provider.rb', line 477