Vara
A gem for building products to be consumed by Ops Manager.
Complementary tools
While Vara focuses on building a .pivotal from a complete product repository, you may find some of the following tools complement your workflow with Vara:
- Vöxtur to automatically update metadata_parts/binaries.yml from a built release tarball.
Usage
Build .pivotal file
In most circumstances, you will only need to run vara build-pivotal PRODUCT_DIR
, which calls other lower-level vara commands.
vara build-pivotal ~/workspace/p-runtime
will build a .pivotal file using the stemcell, release tarball, and compiled packages as specified by the product metadata. If the stemcell, release tarball, or compiled packages do not exist on disk, they will be downloaded.
The product file will be named #{product_name}-#{product_version}.pivotal
, where both variables come from the product metadata YAML.
The product file's contents will look like:
cf-1.1.0.0.pivotal
├── compiled_packages
│ └── cf-158.1-dev-bosh-vsphere-esxi-ubuntu-1471.2.tgz
├── content_migrations
│ └── migrations.yml
├── metadata
│ └── cf.yml
├── releases
│ └── cf-158.1-dev.tgz
└── stemcells
└── bosh-stemcell-1471_2-vsphere-esxi-ubuntu.tgz
The content_migrations/migrations.yml
is copied from the product folder's content_migrations/migrations.yml
.
The metadata/cf.yml
file is copied from the product folder's metadata/cf.yml
if it exists.
Otherwise the metadata YAML is assembled from metadata_parts/{handcraft,binaries}.yml
.
The release file is copied from the product folder's releases
folder.
The stemcell file is copied from the product folder's stemcells
folder.
The PRODUCT_DIR
Vara expects a product directory that resembles the layout of the .pivotal zip contents.
p-runtime/
├── .gitignore
├── compiled_packages
│ └── .gitkeep
├── content_migrations
│ └── .gitkeep
├── content_migrations_parts
│ ├── base.yml
│ └── migrations
│ ├── from_1.1.0.0.yml
│ └── from_1.2.0.0.yml
├── metadata
│ └── .gitkeep
├── metadata_parts
│ ├── binaries.yml
│ └── handcraft.yml
├── releases
│ └── .gitkeep
└── stemcells
└── .gitkeep
The contents of compiled_packages
, releases
, and stemcells
are large tarballs.
Large binary files are discouraged from being checked into git and other source control systems.
To work around this limitation, vara examines the metadata YAML and downloads these files from a URL if necessary.
The compiled_packages
, releases
, and stemcells
directories are populated during the download-artifacts
step of building a .pivotal.
The metadata
directory is populated during the build-metadata
step.
The content_migrations
directory is populated during the build-migrations
step.
The .gitignore file should include these entries to prevent large binary files from being committed by mistake. These entries also prevent vara from failing various operations due to a dirty git working tree.
releases/
stemcells/
compiled_packages/
*.pivotal
*.pivotal.yml
*.pivotal.md5
Re-generate Metadata .yml
Using metadata_parts
To only build the product metadata file from the two separate metadata_parts files:
vara build-metadata ~/workspace/p-runtime
This will create a metadata/${PRODUCT_NAME}.yml
file composed of the two files located in ${product_dir}/metadata_parts
:
- binaries.yml (includes stemcells, release, and optional compiled_packages information)
- handcraft.yml (includes everything else, including product name (e.g. cf), product_version (e.g. 1.2.0.0)
(where PRODUCT_NAME is derived from the product name entry in handcraft.yml)
*[These files were split in order to maintain the comments and the formatting of contained in the handcraft.yml file while allowing the information contained in binaries.yml to be updated by utilities such as vara-update-metadata
]
Include the following entry in .gitignore if vara dynamically generates the metadata .yml.
metadata/*.yml
Automatically increasing a product's prerelease version
You can set the product_version
to something like 1.2.3.4$PRERELEASE_VERSION$
and during metadata generation, vara will expand that into something like 1.2.3.4.alpha.88.dedbeef
where:
- alpha is the "cycle" as specified by the --cycle flag to
vara build-metadata
, which defaults to alpha - 88 is the number of commits in the current branch
- dedbeef is the short commit hash of the current revision on the current branch
Specifying a product version
You can set the product_version
to something specific like 1.2.3-build.1
or 1.2.3
by specifying the --version
flag, e.g.:
vara build-metadata --version=1.2.3-build.1 .
vara build-migrations --version=1.2.3 .
vara build-pivotal --version=1.2.3-rc.3 .
Specifying a final version
You can remove the $PRERELEASE_VERSION$
from the product version by using the --final
flag. So for a product version that is in
handcraft.yml
as 1.2.3.4$PRERELEASE_VERSION$
running
vara build-pivotal . --final
vara build- . --final
vara build-migrations . --final
would result in a product version of 1.2.3.4
.
Specifying an external release
You can update multiple releases when building the pivotal by passing one or more external release files using the --external-releases
flag.
vara build-pivotal . --external-releases=some-release.yml
vara build-pivotal . --external-release=some-release.yml,another-release.yml
An external release file contains an array of manifest snippets that replace entries in metadata_parts/binaries.yml
. For example a sample external release file could be the following. This would update the cf
entry in binaries.yml when running the pivotal.
- file: cf-214.tgz
name: cf
version: '214'
md5: 20cd54b93c23a7fb232d5c25b2082666
url: http://bosh.io/d/github.com/cloudfoundry/cf-release?v=214
The flag will use the specified files to replace the entries that matches the name
property of an already existing release inside of the binaries.yml
file.
Lint product metadata
Syntax
vara lint <Product-Metadata>
Description
Running vara lint ~/workspace/p-runtime/cf.yml
will inspect the product template for malformed data.
vara build-pivotal
executes vara lint
before running vara zip-pivotal
, therefore if malformed data exists,
the resulting .pivotal file will not be created.
Re-generate content migrations
To only compose the content migrations file from the files in the content_migrations_parts
directory:
vara build-migrations ~/workspace/p-runtime
This will create a content_migrations/migrations.yml
file composed of the files in ${product_dir}/content_migrations_parts
:
- base.yml
- Typically includes the
product
,installation_version
, andto_version
keys - Optionally includes the
migrations
key, which is allowed to have any number of entries underneath it
- Typically includes the
- migrations/from_1.2.3.4.yml
- Can be named anything you want with a .yml extension, but by convention is
from_${version}.yml
- Content is the hash of a single entry to be inserted under the migrations key (e.g. has
from_version
andrules
as top-level keys)
- Can be named anything you want with a .yml extension, but by convention is
A migration file in the migrations
folder may have a base.yml
file like:
product: example-product
installation_version: "1.1"
to_version: "1.2.0.0$PRERELEASE_VERSION$"
and a rule entry like:
from_version: 1.1.0.0
rules:
- type: update
selector: "product_version"
to: "1.2.0.0$PRERELEASE_VERSION$"
and then $PRERELEASE_VERSION$
will be replaced in both places using the same rules as above in the metadata file.
Include the following entry in .gitignore if vara dynamically generates the metadata .yml.
content_migrations/*.yml
A Warning on Auto-Generated Prerelease versions
Ops Manager does not allow wildcards in the from_version
field of migrations.
Therefore, if you have deployed a prerelease version, you will not be able to upgrade to another prerelease version unless you explicitly write a migration between the two.
We expect that the normal workflow will not include upgrading between prereleases.
Any prerelease products published externally will require an additional stanza in future migrations.yml for upgradability.
Download missing artifacts
Running vara download-artifacts ~/workspace/p-runtime/cf.yml
will download any missing compiled_packages, releases, and stemcells, inserting them into the respective directories.
For stemcells, vara assumes that the stemcell is a public bosh stemcell and infers the URL based on the file
attribute of the stemcell hash.
For releases and compiled packages, vara uses the url
attribute of the respective hashes in the product metadata.
For example:
releases:
- file: cf-172.tgz
name: cf
version: '172'
md5: 777fe352515612841a3d96af12054947
url: https://example.s3.amazonaws.com/cf-172.tgz
Checksum Validation
Artifacts can use either MD5
or SHA1
checksums for release and stemcell validation. For example:
releases:
- file: cf-172.tgz
name: cf
version: '172'
md5: 777fe352515612841a3d96af12054947
sha1: 7d7ea80ccb59ce57c047e0c91c865172d98f1ba0
url: https://example.s3.amazonaws.com/cf-172.tgz
You must specify at least one checksum.
AWS S3 support
Release artifacts can be downloaded from an S3 bucket. For example:
releases:
- file: cf-172.tgz
name: cf
version: '172'
md5: 777fe352515612841a3d96af12054947
sha1: 7d7ea80ccb59ce57c047e0c91c865172d98f1ba0
aws:
access_key_id: <some-access-key>
secret_access_key: <some-secret-key>
region: us-west-1
bucket_name: <some-bucket>
filename: path/to/s3/object.tgz
You should only specify url
or aws
, not both.
Zipping the .pivotal
If your metadata file has been generated and your artifacts have been downloaded, running vara zip-pivotal ~/workspace/p-runtime
will put all of those contents into a new .pivotal in your working directory.
LEGAL
Copyright (c) 2014-2015 Pivotal Software, Inc. All rights reserved. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.