Hephaestus

A plug template for Yetto. It serves two purposes:

  1. Use this to quickly spin up new plugs with a set of defaults.
  2. Hephaestus also exists as an engine, and is responsible for the majority of a plug's Yetto communication logic.

Usage

First, install Hephaestus globally, on your system:

gem install hephaestus

Then, in the parent directory where you want your plug to be created, run:

hephaestus plug-app

Where app represents the name of the platform you'd like to interact with, like jira, notion, slack, etc.

Testing locally

If you're working on updating/testing this gem locally, you may also try:

rm -rf plug-app && DEBUG=1 hephaestus/bin/hephaestus plug-app

This way you can wipe the dir and quickly iterate on new changes.

Building upon the base

Most of the files which Hephaestus creates for you shouldn't need much modification. The information below is part-informative, part-guidelines on how to extend the generated code for your specific plug.

Controllers

ApplicationController

Your application controller inherits from Hephaestus::ApplicationController, and handles the basics of request/response cycles. Two common callbacks to add here are:

  • before_action :set_request_span_context, which lets you add data to the OpenTelemetry trace before a request occurs
  • after_action :set_response_span_context, similarly, lets you add to the trace after the response is sent

For examples on how to integrate with these, check out plug-email.

SettingsController

Typically, your settings inherit directly from Hephaestus. We'll discuss more about how this works in routes.rb. Either way, you'll define two files:

  • app/views/settings/new.json.jbuilder
  • app/views/settings/edit.json.jbuilder

Define the settings UI in these files. See plug-email for an example of a settings page with no customizations.

However, for multi-step plugs, it's common to expect customized steps. If you'd like to extend the settings controller, create a file with one of two methods:

  • new
  • edit

You only need to define these methods if you're actually overriding the action—often, this only means edit. If you do define methods in this controller, remember to add before_action :ensure_json_request at the top. For examples on settings pages which have some customization, see plug-github and plug-linear. For an example of a totally custom settings page, see plug-zendesk.

AppController

The logic to communicate with your platform sits in AppController. Every plug is unique, so the requirements for your platform varies!

At a minimum, though, you should place all the logic which authenticates requests in the Authable concern. See plug-github and plug-linear for practical examples.

YettoController

Place the logic which receives webhooks from Yetto in the YettoController. This controller must include Hephaestus::ValidatesFromYetto, and must define before_action :from_yetto?. Beyond that, you can respond to events as they come in, and fire jobs to perform the actions your platform needs to integrate with Yetto.

Jobs

Jobs are at the heart of plugs, because they perform time-sensitive work asynchronously. Since Hephaestus already creates an ApplicationJob which inherits from Hephaestus::ApplicationJob, there isn't much else you need to do, other than define your jobs.

Whenever you need to integrate with Yetto, just pass the appropriate arguments to Hephaestus::UpdateYettoJob.

The bulk of your plug logic should exist within these jobs. Keep the controllers as place to authenticate incoming requests and respond with appropriate status codes.

Constants

Place any and all constants in the file called lib/constants.rb.

Note that Hepheastus provides several convenience methods and constants for you. These are:

  • plug_shortname: the downcased version of the plug (e.g., Slack => slack, GitHub => github)
  • plug_name: The proper name of the plug (e.g., Slack, GitHub)
  • plug_url: The plug's URL (e.g., email{.plugs.yetto.app,.plugs.yetto.dev,.ngrok.io})

Application Configuration

The typical Rails' environment configs (config/environments/{development.rb,test.rb,production.rb}) are managed by Hephaestus. However, any plug can define their own environment files as well. These are added after Hephaestus' default configuration.

The test.rb file in plug-email provides an example of this.

Upgrading Rails

Upgrading Rails always introduces breaking changes, even between minor releases. To ensure that a plug still works after a Rails upgrade, first, add

rails "x"

to your plug's Gemfile (where "x" is the new Rails version.) Note that this means plugs can run Rails versions independently of what Hephaestus provides (although variance is not recommended, it can help during the upgrade process).

Next, call rails app:update:configs. You can pretty much ignore every changed file, except the one that starts with new_framework_defaults_. Use these values to slowly reintroduce the new configuration changes, as described in the Rails documentation.

Integrating the Plug with the Engine

There are certain times where the child application needs to "inject" data into Hephaestus. These are typically customizations that are unique to the plug. Place these customizations in config/application.rb, inside of an initializer :engines_blank_point do block.

At a minimum, you'll need to define your plug's API version, and the version of the Yetto API it's communicating with:

initializer :engines_blank_point do
  config.yetto_api_version = "2023-03-06"
  config.plug_api_version = "2023-03-06"
end

You can also define several other customizations:

  • config.tracing_body_filters: use this to scrub out any sensitive data coming in from your platform, to prevent it from leaking in our metrics aggregator. See the override in plug-email for an example. The default is to simply log everything (after running it through Rails' ParameterFilter, of course), which may not be a good idea!
  • config.enhance_update_yetto_job: use this to include any plug-specific logic which needs to occur as part of issuing calls to the Yetto API

For an example of both these customizations, see plug-email.

There's also middleware you can use to protect any endpoint with an OpenAPI schema. To do so, add:

require "hephaestus/middleware/openapi_validation"
config.middleware.use(Hephaestus::Middleware::OpenapiValidation, match_path: "/app/2023-03-06/path/", limit_methods_to: ["POST"], spec: Rails.root.join("lib/schemas/path/2023-03-06/openapi.json"))

In this example, any POST to match_path is protected by the OpenAPI spec provided.

Services

Place any HTTP communication necessary to communicate with your platform in this directory. All other code should which communicates to your plug should rely on methods defined in this file.

See any of the plugs for an example of how to define this interaction, particularly with httpsensible.

Routes

If you do not have any custom actions to perform in the settings controller, define your settings pages as:

# you must always include this line when referring to `hephaestus_settings`
HephaestusSettingsController = Hephaestus::SettingsController

get "/api/#{Rails.application.config.plug_api_version}/settings", to: "hephaestus_settings#new"
get "/api/#{Rails.application.config.plug_api_version}/settings/:organization_id/:inbox_id/:plug_installation_id/edit", to: "hephaestus_settings#edit"

Otherwise, be sure to refer to hephaestus_settings only for any default settings related actions:

# you must always include this line when referring to `hephaestus_settings`
HephaestusSettingsController = Hephaestus::SettingsController

get "/api/#{Rails.application.config.plug_api_version}/settings", to: "hephaestus_settings#new"
get "/api/#{Rails.application.config.plug_api_version}/settings/:organization_id/:inbox_id/:plug_installation_id/edit", to: "settings#edit"

Otherwise, you can add any other routes your plug needs to config/routes.rb.

Launching the server

First, you'll note that you have a script/ngrok file, which launches an ngrok server at https://${hostname}-plug-app.ngrok.io, which maps locally to http://localhost:6661. Setting up ngrok can be essential when testing the platform locally for the first time. (Keep in mind that you still need to run script/server to actually start the local server—this is just to help facilitate communication with the platform.)

Setting up routes

You should probably open up config/routes.rb to make modifications to any incoming (from the platform) or outgoing (for Yetto) HTTP flows.

Defining settings

Next, you'll want to open app/views/settings/new.json.jbuilder and modify the JSON structure of the Settings form page users will see when they first install the plug. Note that this adheres to a strict schema.

Accepting events

After a user submits a plug installation on the Yetto side, it'll send a POST payload to /api/:version/:event/:record_type--for example, /api/2023-03-06/after/plug_installation. Open up the yetto_controller.rb file and decide what happens next!

Creating services

Any code which communicates with the third party should be placed in the app/services directory. A generic HTTP service is included.

Writing tests

In addition to writing tests for your plug interacting with its platform, many of your tests will also need to cover your plug's interaction with Yetto.

Writing these tests well comes with time; it's not an easy thing to cover in a pithy paragraph. Regardless, here's a bit of hopefully helpful advice.

Be sure to tests requests before and after they occur. Typically, this takes the form of a stub_request and assert_requested. Similarly, be sure to:

  • include Hephaestus::Webmocks::SlackWebmock whenever a plug action is expected to error out to Slack
  • include Hephaestus::Webmocks::YettoWebmock whenever a plug needs to issue a request to Yetto

Use include Hephaestus::API::TestHelpers whenever you need to write a test which either

  • match "/#{plug_shortname} routes (by issuing calls to plug). These are routes which your external server is calling into.
  • match /api routes (by issuing calls to api). These are routes which Yetto is calling into.

For examples on when and how to use these methods, see plug-github.

Acknowledgements

The template generation for this project was heavily based on thoughtbot/suspenders.