The ruby GEM consul-templaterb is both a library and an executable that allows to generate files using data from Consul (Discovery and Key/Value Store) easily using ruby's erb templates. It also support launching programs and baby-sitting processes when rendering the files, thus notifying programs when data do change.
It is intended for user accustomed to expressiveness or Ruby templating (ERB), allowing for more flexibility and features than Go templates.
It also allows to use all of ruby language, especially useful for generating files in several formats (JSON, XML) for which text substitutions are hard to get right (escaping, attributes encoding...).
It also focuses on good performance and lightweight usage of bandwidth, especially for very large clusters and watching lots of services.
For complicated rendering of templates and large Consul Clusters, it usually renders faster with a more predictable way the template than the original consul-template.
It also allow to display a very nice and hi-performance HTML5 UI for Consul, see consul-ui for details.
Video introduction to consul-templaterb
Differences with HashiCorp's consul-template
Hashicorp's Consul Template inspired strongly the creation of this GEM and this executable wants to achieve better results in some use cases, especially for very large Consul clusters with many nodes and servers.
consul-template has more features regarding Consul support (for instance, it has support for Hashicorp's Vault), but consul-templaterb focuses on getting more power with the generation of templates and more performance.
Consul Template uses Go templates which is very limited in its set of features: it is complicated to sort, apply real transformations using code and even interact with the OS (ex: get the current date, format timestamps...).
The sort feature for instances allow you to create predictable output (i.e: meaning that the order of nodes is predictable), thus it might offer better performance since the reload of processes if happening ONLY when the files are binary different. Thus, if using consul-templaterb, you will reload less your haproxy or load-balancer than you would do with consul-template.
Compared to consul-template, consul-templaterb offers the following features:
- Hot-Reload of template files
- Bandwidth limitation per endpoint (will soon support dynamic bandwidth limiter)
- Supports launch and supervision of multiple child processes
- Supports launching commands when files do change on disk (reload commands...)
- Supports all Ruby features (ex: base64, real JSON/XML generation...)
- Information about bandwidth
The executable supports semantics and command line flags and options similar to
HashiCorp's Consul-template, so many flags you might use in consul-template will
work in a similar way. It also supports the same environment variable
CONSUL_HTTP_ADDR to find the Consul Agent to query and
get the token.
You might either use the executable directly OR use this GEM as a library by
adding this line to your application's
And then execute:
$ bundle [...]
Or install it yourself as:
$ gem install consul-templaterb [...]
If you simply want to use the executable on your preferred Linux distribution, you have to install first: ruby and ruby dev.
Quick install on Ubuntu-Linux
sudo apt-get install ruby ruby-dev && sudo gem install consul-templaterb
You can now use it directly using the binary
consul-templaterb in your path.
Quick install on Windows
On Windows, a bug exists with Ruby greater than 2.4.
In order to make it work, use this commands as explained in https://github.com/oneclick/rubyinstaller2/issues/96#issuecomment-434619796
gem install eventmachine consul-templaterb --platform ruby
Playing with the samples templates
Samples are installed with the GEM, you can either download them or simply use the ones installed with the gem. To figure out where the templates are installed:
$ gem contents consul-templaterb|grep samples [...]
Will output the path where the samples are being installed, you can copy the directory somewhere and then issue the command:
$ consul-templaterb samples/*.html.erb Using samples/checks.html output for samples/checks.html.erb [...]
It will render a full web site you may browse to look in real time the status of your Consul Cluster.
You can now have a look to the API Documentation to modify existing templates or write your owns, it is very easy!
Usage of consul-templaterb
$ consul-templaterb --help USAGE: consul-templaterb [[options]] -h, --help Show help -v, --version Show Version -g, --no-gzip-compression Disable GZIP compression in HTTP requests -c, --consul-addr=<address> Address of Consul, eg: http://localhost:8500 -l, --log-level=<log_level> Log level, default=info, any of none|error|info|debug --consul-token=<token> Use a token to connect to Consul -V, --vault-addr=<address> Address of Vault, eg: http://localhost:8200 --vault-token=<token> Token used to authenticate against vault. --[no-]vault-renew Control auto-renewal of the Vault token. Default: activated --vault-lease-duration-factor=<factor> Wait at least <factor> * lease time before updating a Vault secret. Default: 0.5 -w, --wait=<min_duration> Wait at least n seconds before each template generation -r, --retry-delay=<min_duration> Min Retry delay on Error/Missing Consul Index -k, --hot-reload=<behavior> Control hot reload behaviour, one of :[die (kill daemon on hot reload failure), keep (on error, keep running), disable (hot reload disabled)] -K, --sig-term=kill_signal Signal to send to next --exec command on kill, default=TERM -T, --trim-mode=trim_mode ERB Trim mode to use (- by default) -R, --sig-reload=reload_signal Signal to send to next --exec command on reload (NONE supported), default=HUP -M, --debug-memory-usage Display messages when RAM grows -e, --exec=<command> Execute the following command -d, --debug-network-usage Debug the network usage -t erb_file:[output]:[command]:[params_file], --template Add a erb template, its output and optional reload command -o, --once Do not run the process as a daemon
When launched with file arguments ending with .erb, the executable will assume
the file is a template and will render the corresponding file without the
It means that you can call consul-templaterb with
*.erb arguments, the shell
will then substitute all files and render it by removing the
.erb extension as
--template my_file.ext.erb:myfile.ext was used.
Generate multiple templates
In the same way as consul-template, consul-templaterb supports multiple templates and executing
commands when the files do change. The parameter
--template <ERB>:<DEST>:[reload_command]:params_file works
in the following way:
- ERB : the ERB file to use as a template
- DEST: the destination file
- reload_command: optional shell command executed whenever the file has been modified
- params_file: JSON or YAML file to load and to use as parameter for template (see param() function to retrieve the values)
The argument can be specified multiple times, ex:
Example of usage:
$ consul-templaterb \\ --template "samples/ha_proxy.cfg.erb:/opt/haproxy/etc/haproxy.cfg:sudo service haproxy reload" --template "samples/consul_template.erb:consul-summary.txt"
Process management and signalisation of configuration files
--exec argument (can be specified multiple times), consul-templaterb will launch
the process specified when all templates have been generated and will send a reload signal
if the content of any of the files do change (the signal will be sent atomically however,
meaning that if 2 results of templates are modified at the same time, the signal will be
sent only once (it is helpful for instance if your app is using several configurations
files that must be consistent all together).
If any template does return nothing (aka use
return nil in the code of a template),
consul-templaterb does consider that the template is not ready and will not launch the executable
nor write the file.
Signals can be customized per process. Two signals are supported with options
--sig-term. When the option is added, the next
--exec options to start a process will use the
given signal. By default, HUP will be sent to reload events (you can use NONE to avoid sending any
reload signal), TERM will be used when leaving consul-templaterb.
This is actually the original reason for the creation of this GEM: on Criteo's large clusters, consul-template generated several hundreds of Mb/s to the Consul-Agent which also generated several hundreds of Mb/s with the Consul servers.
By design, the GEM supports limiting the number of requests per endpoints (see code in
bin/consul-templaterb file). It avoids using too much network to fetch data from Consul
in large Consul Clusters (especially when watching lots of files).
The limitation is static for now, but fair dynamic bandwidth allocation will allow to limit the bandwidth used to get information for all services by capping the global bandwidth used by consul-templaterb.
Have a look into the samples/ directory to browse example files which contains those examples:
- List all nodes on Cluster
- Show all services in Cluster
- Show all Service Checks and their output
- Show all Key/Values nicely
- Show Choregraphies - work on content of K/V with JSON
- Services in XML
- Services in JSON
- Generate HaProxy Configuration
- Export Consul Statistics to Prometheus : count all services, their state, datacenters and nodes and export it to prometheus easily to trigger alerts.
- List all services/Nodes with their statuses for all datacenters
- Show all services/instances not passing on all DCs
If you want to test it quickly, you might try with (assuming your consul agent is listening on
$ be bin/consul-templaterb -c 'http://localhost:8500' samples/*.html.erb [...]
It will generate a full website in samples/ directory with lots of Consul information ready to use (website updated automagically when values to change).
All templates are validated using the Travis CI, so all should be working for your Consul Configuration.
We recommend using bundle using
bundle install, you can now run
bundle exec bin/consul-templaterb.
Help is available running
bundle exec bin/consul-templaterb --help
The following example will generate static HTML pages and JSON data for
bundle exec bin/consul-templaterb -c your.consul.agent:8500 samples/consul-ui/*.erb
If you need remote calls, you need an HTTP server. A simple way to have one is using Python's simple HTTP
server. Example for
cd samples/consul-ui python -m SimpleHTTPServer
To install this gem onto your local machine, run
bundle exec rake install. To release a new version, update the
version number in
version.rb, and then run
bundle exec rake release, which will create a git tag for the version,
push git commits and tags, and push the
.gem file to rubygems.org.
Here are the known bugs of the application:
- [ ]
render_filemight create an infinite recursion if a template includes itself indirectly.
- [ ] On Windows, the consul-templaterb cannot work with more than 2048 endpoints watched (see https://github.com/oneclick/rubyinstaller/issues/104#issuecomment-7681074)
Please consult CHANGELOG.md for fixed bugs.
- [x] Hashi's Vault support (EXPERIMENTAL)
- [ ] Implement automatic dynamic rate limit
- [x] More samples: apache, nginx, full website displaying consul information...
- [x] Optimize rendering speed at start-up: an iteration is done very second by default, but it would be possible to speed up rendering by iterating with higher frequency until the first write of result has been performed.
- [ ] Allow to tune bandwidth using a simple configuration file (while it should not be necessary for 90% of use-cases)
The gem is available as open source under the terms of the Apache v2 license. See LICENSE.txt file.