Class: Google::Serverless::Exec

Inherits:
Object
  • Object
show all
Defined in:
lib/google/serverless/exec.rb,
lib/google/serverless/exec/tasks.rb,
lib/google/serverless/exec/gcloud.rb,
lib/google/serverless/exec/railtie.rb,
lib/google/serverless/exec/version.rb

Overview

Serverless execution tool

This class provides a client for serverless execution, allowing Serverless applications to perform on-demand tasks in the serverless environment. This may be used for safe running of ops and maintenance tasks, such as database migrations, that access production cloud resources.

About serverless execution tool

Serverless execution spins up a one-off copy of a serverless app, and runs a command against it. For example, if your app runs on Ruby on Rails, then you might use serverless execution to run a command such as bundle exec bin/rails db:migrate in production infrastructure (to avoid having to connect directly to your production database from a local workstation).

Serverless execution provides two strategies for generating that "one-off copy":

  • A deployment strategy, which deploys a temporary version of your app to a single backend instance and runs the command there.
  • A cloud_build strategy, which deploys your application image to Google Cloud Build and runs the command there.

Both strategies are generally designed to emulate the runtime environment on cloud VMs similar to those used by actual deployments of your app. Both provide your application code and environment variables, and both provide access to Cloud SQL connections used by your app. However, they differ in what version of your app code they run against, and in certain other constraints and performance characteristics. More detailed information on using the two strategies is provided in the sections below.

Apps deployed to the App Engine flexible environment and Cloud Run will use the cloud_build strategy by default. However, you can force an app to use the deployment strategy instead. (You might do so if you need to connect to a Cloud SQL database on a VPC using a private IP, because the cloud_build strategy does not support private IPs.) To force use of deployment, set the strategy parameter in the Exec constructor (or the corresponding GAE_EXEC_STRATEGY parameter in the Rake task). Note that the deployment strategy is usually significantly slower than cloud_build for apps in the flexible environment.

Apps deployed to the App Engine standard environment will always use the deployment strategy. You cannot force use of the cloud_build strategy.

Prerequisites

To use this tool, you will need:

  • An app deployed to Google serverless compute, of course!
  • The gcloud SDK installed and configured.
  • The serverless gem.

You may use the Google::Serverless::Exec class to run commands directly. However, in most cases, it will be easier to run commands via the provided rake tasks (see Tasks).

Using the "deployment" strategy

The deployment strategy deploys a temporary version of your app to a single backend App Engine instance, runs the command there, and then deletes the temporary version when it is finished.

This is the default strategy (and indeed the only option) for apps running on the App Engine standard environment. It can also be used for flexible environment apps, but this is not commonly done because deployment of flexible environment apps can take a long time.

Because the deployment strategy deploys a temporary version of your app, it runs against the current application code present where the command was initiated (i.e. the code currently on your workstation if you run the rake task from your workstation, or the current code on the branch if you are running from a CI/CD system.) This may be different from the code actually running in production, so it is important that you run from a compatible code branch.

Specifying the host application

The deployment strategy works by deploying a temporary version of your app, so that it has access to your app's project and settings in App Engine. In most cases, it can determine this automatically, but depending on how your app or environment is structured, you may need to give it some help.

By default, your Google Cloud project is taken from the current gcloud project. If you need to override this, set the :project parameter in the Exec constructor (or the corresponding GAE_PROJECT parameter in the Rake task).

By default, the service name is taken from the App Engine config file. Serverless execution will assume this file is called app.yaml in the current directory. To use a different config file, set the config_path parameter in the Exec constructor (or the corresponding GAE_CONFIG parameter in the Rake task). You may also set the service name directly, using the service parameter (or GAE_SERVICE in Rake).

Providing credentials

Your command will effectively be a deployment of your serverless app itself, and will have access to the same credentials. For example, App Engine provides a service account by default for your app, or your app may be making use of its own service account key. In either case, make sure the service account has sufficient access for the command you want to run (such as database admin credentials).

Other options

You may also provide a timeout, which is the length of time that serverless execution will allow your command to run before it is considered to have stalled and is terminated. The timeout should be a string of the form 2h15m10s. The default is 10m.

The timeout is set via the timeout parameter to the Exec constructor, or by setting the GAE_TIMEOUT environment variable when invoking using Rake.

Resource usage and billing

The deployment strategy deploys to a temporary instance of your app in order to run the command. You may be billed for that usage. However, the cost should be minimal, because it will then immediately delete that instance in order to minimize usage.

If you interrupt the execution (or it crashes), it is possible that the temporary instance may not get deleted properly. If you suspect this may have happened, go to the App Engine tab in the cloud console, under "versions" of your service, and delete the temporary version manually. It will have a name matching the pattern serverless-exec-<timestamp>.

Using the "cloud_build" strategy

The cloud_build strategy takes the application image that App Engine or Cloud Run is actually using to run your app, and uses it to spin up a copy of your app in Google Cloud Build (along with an emulation layer that emulates certain serverless services such as Cloud SQL connection sockets). The command then gets run in the Cloud Build environment.

This is the default strategy for apps running on the App Engine flexible and Cloud Run environments. (It is not available for standard environment apps.) Note that the cloud_build strategy cannot be used if your command needs to connect to a database over a VPC private IP address. This is because it runs on virtual machines provided by the Cloud Build service, which are not part of your VPC. If your database can be accessed only over a private IP, you should use the deployment strategy instead.

The Cloud Build log is output to the directory specified by "CLOUD_BUILD_GCS_LOG_DIR". (ex. "gs://BUCKET-NAME/FOLDER-NAME") By default, log directory name is "gs://[PROJECT_NUMBER].cloudbuild-logs.googleusercontent.com/".

Specifying the host application

The cloud_build strategy needs to know exactly which app, service, and version of your app, to identify the application image to use.

By default, your Google Cloud project is taken from the current gcloud project. If you need to override this, set the :project parameter in the Exec constructor (or the corresponding GAE_PROJECT parameter in the Rake task).

By default, the service name is taken from the App Engine config file. Serverless execution will assume this file is called app.yaml in the current directory. To use a different config file, set the config_path parameter in the Exec constructor (or the corresponding GAE_CONFIG parameter in the Rake task). You may also set the service name directly, using the service parameter (or GAE_SERVICE in Rake).

By default, the image of the most recently deployed version of your app is used. (Note that this most recently deployed version may not be the same version that is currently receiving traffic: for example, if you deployed with --no-promote.) To use a different version, set the version parameter in the Exec constructor (or the corresponding GAE_VERSION parameter in the Rake task).

Providing credentials

By default, the cloud_build strategy uses your project's Cloud Build service account for its credentials. Unless your command provides its own service account key, you may need to grant the Cloud Build service account any permissions needed to execute your command (such as access to your database). For most tasks, it is sufficient to grant Project Editor permissions to the service account. You can find the service account configuration in the IAM tab in the Cloud Console under the name [your-project-number]@cloudbuild.gserviceaccount.com.

Other options

You may also provide a timeout, which is the length of time that serverless execution will allow your command to run before it is considered to have stalled and is terminated. The timeout should be a string of the form 2h15m10s. The default is 10m.

The timeout is set via the timeout parameter to the Exec constructor, or by setting the GAE_TIMEOUT environment variable when invoking using Rake.

You can also set the wrapper image used to emulate the App Engine runtime environment, by setting the wrapper_image parameter to the constructor, or by setting the GAE_EXEC_WRAPPER_IMAGE environment variable. Generally, you will not need to do this unless you are testing a new wrapper image.

Resource usage and billing

The cloud_build strategy uses virtual machine resources provided by Google Cloud Build. Generally, a certain number of usage minutes per day is covered under a free tier, but additional compute usage beyond that time is billed to your Google Cloud account. For more details, see https://cloud.google.com/cloud-build/pricing

If your command makes API calls or utilizes other cloud resources, you may also be billed for that usage. However, the cloud_build strategy (unlike the deployment strategy) does not use actual App Engine instances, and you will not be billed for additional App Engine instance usage.

Defined Under Namespace

Modules: Gcloud, Tasks Classes: BadConfigFileFormat, BadParameter, ConfigFileNotFound, NoDefaultProject, NoSuchVersion, Railtie, UnsupportedStrategy, UsageError

Constant Summary collapse

APP_ENGINE = 
:app_engine
CLOUD_RUN = 
:cloud_run
VERSION =

The current version of this gem, as a string.

"0.2.0"

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(command, project: nil, service: nil, config_path: nil, version: nil, timeout: nil, wrapper_image: nil, strategy: nil, gcs_log_dir: nil, product: nil, region: nil) {|_self| ... } ⇒ Exec

Create an execution for the given command.

Parameters:

  • command (Array<String>) —

    The command in array form.

  • project (String, nil) (defaults to: nil) —

    ID of the project. If omitted, obtains the project from gcloud.

  • service (String, nil) (defaults to: nil) —

    Name of the service. If omitted, obtains the service name from the config file.

  • config_path (String, nil) (defaults to: nil) —

    App Engine config file to get the service name from if the service name is not provided directly. If omitted, defaults to the value returned by default_config_path.

  • version (String, nil) (defaults to: nil) —

    Version string. If omitted, defaults to the most recently created version of the given service (which may not be the one currently receiving traffic).

  • timeout (String, nil) (defaults to: nil) —

    Timeout string. If omitted, defaults to the value returned by default_timeout.

  • wrapper_image (String, nil) (defaults to: nil) —

    The fully qualified name of the wrapper image to use. (Applies only to the "cloud_build" strategy.)

  • strategy (String, nil) (defaults to: nil) —

    The execution strategy to use, or nil to choose a default based on the App Engine environment (flexible or standard) or Cloud Run environments. Allowed values are nil, "deployment" (which is the default for App Engine Standard), and "cloud_build" (which is the default for App Engine Flexible and Cloud Run).

  • gcs_log_dir (String, nil) (defaults to: nil) —

    GCS bucket name of the cloud build log when strategy is "cloud_build". (ex. "gs://BUCKET-NAME/FOLDER-NAME")

  • product (Symbol) (defaults to: nil) —

    The serverless product. If omitted, defaults to the value returns by #default_product. Allowed values are APP_ENGINE and CLOUD_RUN.

  • region (String) (defaults to: nil) —

    The region for the cloud run service. Required if the product is cloud run.

Yields:

  • (_self)

Yield Parameters:



433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
 
# File 'lib/google/serverless/exec.rb', line 433

def initialize command,
               project: nil,
               service: nil,
               config_path: nil,
               version: nil,
               timeout: nil,
               wrapper_image: nil,
               strategy: nil,
               gcs_log_dir: nil,
               product: nil,
               region: nil
  @command = command
  @service = service
  @config_path = config_path
  @version = version
  @timeout = timeout
  @project = project
  @wrapper_image = wrapper_image
  @strategy = strategy
  @gcs_log_dir = gcs_log_dir
  @product = product
  @region = region

  yield self if block_given?
end

Class Attribute Details

.default_config_path ⇒ String

Returns Path to default config file.

Returns:

  • (String) —

    Path to default config file.



347
348
349
 
# File 'lib/google/serverless/exec.rb', line 347

def default_config_path
  @default_config_path
end

.default_service ⇒ String

Returns Default service name if the config doesn't specify.

Returns:

  • (String) —

    Default service name if the config doesn't specify.



344
345
346
 
# File 'lib/google/serverless/exec.rb', line 344

def default_service
  @default_service
end

.default_timeout ⇒ String

Returns Default command timeout.

Returns:

  • (String) —

    Default command timeout.



341
342
343
 
# File 'lib/google/serverless/exec.rb', line 341

def default_timeout
  @default_timeout
end

.default_wrapper_image ⇒ String

Returns Docker image that implements the app engine wrapper.

Returns:

  • (String) —

    Docker image that implements the app engine wrapper.



350
351
352
 
# File 'lib/google/serverless/exec.rb', line 350

def default_wrapper_image
  @default_wrapper_image
end

Instance Attribute Details

#command ⇒ String+

The command to run.

Returns:

  • (String) —

    if the command is a script to be run in a shell.

  • (Array<String>) —

    if the command is a posix command to be run directly without a shell.



496
497
498
 
# File 'lib/google/serverless/exec.rb', line 496

def command
  @command
end

#config_path ⇒ String?

Returns:

  • (String) —

    Path to the config file.

  • (nil) —

    if the default of ./app.yaml should be used.



475
476
477
 
# File 'lib/google/serverless/exec.rb', line 475

def config_path
  @config_path
end

#product ⇒ Symbol

Returns The serverless product to use. Allowed values are APP_ENGINE and CLOUD_RUN.

Returns:



516
517
518
 
# File 'lib/google/serverless/exec.rb', line 516

def product
  @product
end

#project ⇒ String?

Returns:

  • (String) —

    The project ID.

  • (nil) —

    if the default gcloud project should be used.



463
464
465
 
# File 'lib/google/serverless/exec.rb', line 463

def project
  @project
end

#region ⇒ String

Returns The region for the cloud run service.

Returns:

  • (String) —

    The region for the cloud run service



521
522
523
 
# File 'lib/google/serverless/exec.rb', line 521

def region
  @region
end

#service ⇒ String?

Returns:

  • (String) —

    The service name.

  • (nil) —

    if the service should be obtained from the app config.



469
470
471
 
# File 'lib/google/serverless/exec.rb', line 469

def service
  @service
end

#strategy ⇒ String?

Returns:

  • (String) —

    The execution strategy to use. Allowed values are "deployment" and "cloud_build".

  • (nil) —

    to choose a default based on the App Engine (flexible or standard) or Cloud Run environments.



510
511
512
 
# File 'lib/google/serverless/exec.rb', line 510

def strategy
  @strategy
end

#timeout ⇒ String?

Returns:

  • (String) —

    The command timeout, in 1h23m45s format.

  • (nil) —

    if the default of 10m should be used.



487
488
489
 
# File 'lib/google/serverless/exec.rb', line 487

def timeout
  @timeout
end

#version ⇒ String?

Returns:

  • (String) —

    Service version of the image to use.

  • (nil) —

    if the most recent should be used.



481
482
483
 
# File 'lib/google/serverless/exec.rb', line 481

def version
  @version
end

#wrapper_image ⇒ String?

Returns:

  • (String) —

    Custom wrapper image to use.

  • (nil) —

    if the default should be used.



502
503
504
 
# File 'lib/google/serverless/exec.rb', line 502

def wrapper_image
  @wrapper_image
end

Class Method Details

.new_rake_task(name, args: [], env_args: [], service: nil, config_path: nil, version: nil, timeout: nil, project: nil, wrapper_image: nil, strategy: nil, gcs_log_dir: nil, product: nil) ⇒ Object

Create an execution for a rake task.

Parameters:

  • name (String) —

    Name of the task

  • args (Array<String>) (defaults to: []) —

    Args to pass to the task

  • env_args (Array<String>) (defaults to: []) —

    Environment variable settings, each of the form NAME=value.

  • service (String, nil) (defaults to: nil) —

    Name of the service. If omitted, obtains the service name from the config file.

  • config_path (String, nil) (defaults to: nil) —

    App Engine config file to get the service name from if the service name is not provided directly. If omitted, defaults to the value returned by default_config_path.

  • version (String, nil) (defaults to: nil) —

    Version string. If omitted, defaults to the most recently created version of the given service (which may not be the one currently receiving traffic).

  • timeout (String, nil) (defaults to: nil) —

    Timeout string. If omitted, defaults to the value returned by default_timeout.

  • wrapper_image (String, nil) (defaults to: nil) —

    The fully qualified name of the wrapper image to use. (Applies only to the "cloud_build" strategy.)

  • strategy (String, nil) (defaults to: nil) —

    The execution strategy to use, or nil to choose a default based on the App Engine (flexible or standard) or Cloud Run environments. Allowed values are nil, "deployment" (which is the default for App Engine Standard), and "cloud_build" (which is the default for App Engine Flexible and Cloud Run).

  • gcs_log_dir (String, nil) (defaults to: nil) —

    GCS bucket name of the cloud build log when strategy is "cloud_build". (ex. "gs://BUCKET-NAME/FOLDER-NAME")

  • product (Symbol) (defaults to: nil) —

    The serverless product to use. If omitted, defaults to the value returned by #default_product



381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
 
# File 'lib/google/serverless/exec.rb', line 381

def new_rake_task name, args: [], env_args: [],
                  service: nil, config_path: nil, version: nil,
                  timeout: nil, project: nil, wrapper_image: nil,
                  strategy: nil, gcs_log_dir: nil, product: nil
  escaped_args = args.map do |arg|
    arg.gsub(/[,\[\]]/) { |m| "\\#{m}" }
  end
  name_with_args =
    if escaped_args.empty?
      name
    else
      "#{name}[#{escaped_args.join ','}]"
    end
  new ["bundle", "exec", "rake", name_with_args] + env_args,
      service: service, config_path: config_path, version: version,
      timeout: timeout, project: project, wrapper_image: wrapper_image,
      strategy: strategy, gcs_log_dir: gcs_log_dir, product: product
end

Instance Method Details

#start ⇒ Object

Executes the command synchronously. Streams the logs back to standard out and does not return until the command has completed or timed out.



527
528
529
530
531
532
533
534
535
 
# File 'lib/google/serverless/exec.rb', line 527

def start
  resolve_parameters
  case @product
  when APP_ENGINE
    start_app_engine
  when CLOUD_RUN
    start_cloud_run
  end
end

#start_app_engine ⇒ Object 



537
538
539
540
541
542
543
544
545
 
# File 'lib/google/serverless/exec.rb', line 537

def start_app_engine
  app_info = version_info @service, @version
  resolve_strategy app_info["env"]
  if @strategy == "cloud_build"
    start_build_strategy app_info
  else
    start_deployment_strategy app_info
  end
end

#start_cloud_run ⇒ Object 



547
548
549
550
 
# File 'lib/google/serverless/exec.rb', line 547

def start_cloud_run
  app_info = version_info_cloud_run @service
  start_build_strategy app_info
end