Class: Webhookdb::LoggedWebhook

Inherits:
Object
  • Object
show all
Includes:
Appydays::Configurable
Defined in:
lib/webhookdb/logged_webhook.rb

Defined Under Namespace

Classes: Resilient

Constant Summary collapse

DELETE_UNOWNED = 
14.days
DELETE_SUCCESSES = 
90.days
TRUNCATE_SUCCESSES = 
7.days
DELETE_FAILURES = 
90.days
TRUNCATE_FAILURES = 
30.days
RETRY_HEADER =

When we retry a request, set this so we know not to re-log it.

"Whdb-Logged-Webhook-Retry"
NONOVERRIDABLE_HEADERS =

When we retry a request, these headers must come from the Ruby client, NOT the original request.

[
  "Accept-Encoding",
  "Accept",
  "Host",
  "Version",
].to_set
WEBHOST_HEADERS =

These headers have been added by Heroku/our web host, so should not be part of the retry.

[
  "Connection",
  "Connect-Time",
  "X-Request-Id",
  "X-Forwarded-For",
  "X-Request-Start",
  "Total-Route-Time",
  "X-Forwarded-Port",
  "X-Forwarded-Proto",
  "Via",
].to_set

Class Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Class Attribute Details

.available_resilient_database_urls ⇒ Object

Returns the value of attribute available_resilient_database_urls.



11
12
13
 
# File 'lib/webhookdb/logged_webhook.rb', line 11

def available_resilient_database_urls
  @available_resilient_database_urls
end

Class Method Details

.resilient_insert(**kwargs) ⇒ Object

Insert the logged webhook, and fall back to inserting into the configured available_resilient_database_urls. If none are inserted successfully, raise the error; otherwise, swallow the insert error and more on.

Note that these resilient inserts are MUCH slower than normal inserts; they require a separate database connection, CREATE TABLE call, etc. But it’s a reasonable way to handle when the database is down.



162
163
164
 
# File 'lib/webhookdb/logged_webhook.rb', line 162

def self.resilient_insert(**kwargs)
  Resilient.new.insert(kwargs)
end

.resilient_replay ⇒ Object

Replay and delete all rows in the resilient database tables.



167
168
169
 
# File 'lib/webhookdb/logged_webhook.rb', line 167

def self.resilient_replay
  Resilient.new.replay
end

.retry_logs(instances, truncate_successful: false) ⇒ Object

Send instances back in ‘through the front door’ of this API. Return is a partition of [logs with 2xx responses, others]. Generally you can safely call ‘truncate_logs(result)`, or pass in (truncate_successful: true).



105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
 
# File 'lib/webhookdb/logged_webhook.rb', line 105

def self.retry_logs(instances, truncate_successful: false)
  successes, failures = instances.partition do |lw|
    uri = URI(Webhookdb.api_url + lw.request_path)
    req = Net::HTTP::Post.new(uri.path, {"Content-Type" => "application/json"})
    req.body = lw.request_body
    # This is going to have these headers:
    # ["content-type", "accept-encoding", "accept", "user-agent", "host"]
    # We want to keep all of these, except if user-agent or content-type were set
    # in the original request; then we want to use those.
    # Additionally, there are a whole set of headers we'll find on our webserver
    # that are added by our web platform, which we do NOT want to include.
    lw.request_headers.each do |k, v|
      next if Webhookdb::LoggedWebhook::WEBHOST_HEADERS.include?(k)
      next if Webhookdb::LoggedWebhook::NONOVERRIDABLE_HEADERS.include?(k)
      req[k] = v
    end
    req[Webhookdb::LoggedWebhook::RETRY_HEADER] = lw.id
    resp = Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == "https") do |http|
      http.request(req)
    end
    resp.code.to_i < 400
  end
  self.truncate_logs(*successes) if truncate_successful
  return successes, failures
end

.trim(now: Time.now) ⇒ Object

Trim logged webhooks to keep this table to a reasonable size. The current trim algorithm and rationale is:

  • Logs that belong to inserts that were not part of an org are for our internal use only. They usually indicate an integration that was misconfigured, or is for an org that doesn’t exist. We keep these around for 2 weeks (they are always errors since they have no org). Ideally we investigate and remove them before that. We may need to ‘block’ certain opaque ids from being logged in the future, if for example we cannot get a client to turn off a misconfigured webhook.

  • Successful webhooks get their contents (request body and headers) truncated after 7 days (but the webhook row remains). Usually we don’t need to worry about these so in theory we can avoid logging verbose info at all.

  • Successful webhooks are deleted entirely after 90 days. Truncated webhooks are useful for statistics, but we can remove them earlier in the future.

  • Failed webhooks get their contents truncated after 30 days, but the webhook row remains. We have a longer truncation date so we have more time to investigate.

  • Error webhooks are deleted entirely after 90 days.



86
87
88
89
90
91
92
93
94
95
96
97
98
99
 
# File 'lib/webhookdb/logged_webhook.rb', line 86

def self.trim(now: Time.now)
  owned = self.exclude(organization_id: nil)
  unowned = self.where(organization_id: nil)
  successes = owned.where { response_status < 400 }
  failures = owned.where { response_status >= 400 }
  # Delete old unowned
  unowned.where { inserted_at < now - DELETE_UNOWNED }.delete
  # Delete successes first so they don't have to be truncated
  successes.where { inserted_at < now - DELETE_SUCCESSES }.delete
  self.truncate_dataset(successes.where { inserted_at < now - TRUNCATE_SUCCESSES })
  # Delete failures
  failures.where { inserted_at < now - DELETE_FAILURES }.delete
  self.truncate_dataset(failures.where { inserted_at < now - TRUNCATE_FAILURES })
end

.truncate_dataset(ds) ⇒ Object 



147
148
149
 
# File 'lib/webhookdb/logged_webhook.rb', line 147

def self.truncate_dataset(ds)
  return ds.update(request_body: "", request_headers: "{}", truncated_at: Time.now)
end

.truncate_logs(*instances) ⇒ Object

Truncate the logs id’ed by the given instances. Instances are NOT modified; you need to .refresh to see truncated values.



142
143
144
145
 
# File 'lib/webhookdb/logged_webhook.rb', line 142

def self.truncate_logs(*instances)
  ds = self.where(id: instances.map(&:id))
  return self.truncate_dataset(ds)
end

Instance Method Details

#replay_async ⇒ Object 



136
137
138
 
# File 'lib/webhookdb/logged_webhook.rb', line 136

def replay_async
  return self.publish_immediate("replay", self.id)
end

#retry_one(truncate_successful: false) ⇒ Object 



131
132
133
134
 
# File 'lib/webhookdb/logged_webhook.rb', line 131

def retry_one(truncate_successful: false)
  _, bad = self.class.retry_logs([self], truncate_successful:)
  return bad.empty?
end

#truncated? ⇒ Boolean

Returns:

  • (Boolean) 


151
152
153
 
# File 'lib/webhookdb/logged_webhook.rb', line 151

def truncated?
  return self.truncated_at ? true : false
end