Class: Amigo::Autoscaler

Inherits:
Object
  • Object
show all
Defined in:
lib/amigo/autoscaler.rb,
lib/amigo/autoscaler/heroku.rb

Defined Under Namespace

Classes: Heroku, InvalidHandler

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(poll_interval: 20, latency_threshold: 5, hostname_regex: /^web\.1$/, handlers: [:log], alert_interval: 120, latency_restored_threshold: latency_threshold, latency_restored_handlers: [:log], log: ->(level, message, params={}) { Amigo.log(nil, level, message, params) }) ⇒ Autoscaler

Returns a new instance of Autoscaler.

Raises:

  • (ArgumentError) 


96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
 
# File 'lib/amigo/autoscaler.rb', line 96

def initialize(
  poll_interval: 20,
  latency_threshold: 5,
  hostname_regex: /^web\.1$/,
  handlers: [:log],
  alert_interval: 120,
  latency_restored_threshold: latency_threshold,
  latency_restored_handlers: [:log],
  log: ->(level, message, params={}) { Amigo.log(nil, level, message, params) }
)

  raise ArgumentError, "latency_threshold must be > 0" if
    latency_threshold <= 0
  raise ArgumentError, "latency_restored_threshold must be >= 0" if
    latency_restored_threshold.negative?
  raise ArgumentError, "latency_restored_threshold must be <= latency_threshold" if
    latency_restored_threshold > latency_threshold
  @poll_interval = poll_interval
  @latency_threshold = latency_threshold
  @hostname_regex = hostname_regex
  @handlers = handlers.freeze
  @alert_interval = alert_interval
  @latency_restored_threshold = latency_restored_threshold
  @latency_restored_handlers = latency_restored_handlers.freeze
  @log = log
end

Instance Attribute Details

#alert_interval ⇒ Integer (readonly)

Only alert this often. For example, with poll_interval of 10 seconds and alert_interval of 200 seconds, we’d alert once and then 210 seconds later.

Returns:

  • (Integer) 


71
72
73
 
# File 'lib/amigo/autoscaler.rb', line 71

def alert_interval
  @alert_interval
end

#handlers ⇒ Array<String,Symbol,Proc,#call> (readonly)

Methods to call when alerting, as strings/symbols or procs. Valid string values are ‘log’ and ‘sentry’ (requires Sentry to be required already). Anything that responds to call will be invoked with:

  • Positional argument which is a Hash of ‘name => latency in seconds`

  • Keyword argument :depth: Number of alerts as part of this latency event. For example, the first alert has a depth of 1, and if latency stays high, it’ll be 2 on the next call, etc. depth can be used to incrementally provision additional processing capacity, and stop adding capacity at a certain depth to avoid problems with too many workers (like excessive DB load).

  • Keyword argument :duration: Number of seconds since this latency spike started.

  • Additional undefined keywords. Handlers should accept additional options, like via ‘**kw` or `opts={}`, for compatibility.

Returns:

  • (Array<String,Symbol,Proc,#call>) 


65
66
67
 
# File 'lib/amigo/autoscaler.rb', line 65

def handlers
  @handlers
end

#hostname_regex ⇒ Regexp (readonly)

What hosts/processes should this run on? Looks at ENV and Socket.gethostname for a match. Default to only run on ‘web.1’, which is the first Heroku web dyno. We run on the web, not worker, dyno, so we report backed up queues in case we, say, turn off all workers (broken web processes are generally easier to find).

Returns:

  • (Regexp) 


51
52
53
 
# File 'lib/amigo/autoscaler.rb', line 51

def hostname_regex
  @hostname_regex
end

#latency_restored_handlers ⇒ Array<String,Symbol,Proc,#call> (readonly)

Methods to call when a latency of latency_restored_threshold is reached (ie, when we get back to normal latency after a high latency event). Valid string values are ‘log’. Usually this handler will deprovision capacity procured as part of the alert handlers. Anything that responds to call will be invoked with:

  • Keyword :depth, the number of times an alert happened before the latency spike was resolved.

  • Keyword :duration, the number of seconds for the latency spike has been going on.

  • Additional undefined keywords. Handlers should accept additional options, like via ‘**kw`, for compatibility.

Returns:

  • (Array<String,Symbol,Proc,#call>) 


91
92
93
 
# File 'lib/amigo/autoscaler.rb', line 91

def latency_restored_handlers
  @latency_restored_handlers
end

#latency_restored_threshold ⇒ Object (readonly)

After an alert happens, what latency should be considered “back to normal” and latency_restored_handlers will be called? In most cases this should be the same as (and defaults to) latency_threshold so that we’re ‘back to normal’ once we’re below the threshold. It may also commonly be 0, so that the callback is fired when the queue is entirely clear. Note that, if latency_restored_threshold is less than latency_threshold, while the latency is between the two, no alerts will fire.



79
80
81
 
# File 'lib/amigo/autoscaler.rb', line 79

def latency_restored_threshold
  @latency_restored_threshold
end

#latency_threshold ⇒ Integer (readonly)

What latency should we alert on?

Returns:

  • (Integer) 


43
44
45
 
# File 'lib/amigo/autoscaler.rb', line 43

def latency_threshold
  @latency_threshold
end

#log ⇒ Object (readonly)

Proc/callable called with (level, message, params={}). By default, use Amigo.log (which logs to the Sidekiq logger).



94
95
96
 
# File 'lib/amigo/autoscaler.rb', line 94

def log
  @log
end

#poll_interval ⇒ Integer (readonly)

How often should Autoscaler check for latency?

Returns:

  • (Integer) 


40
41
42
 
# File 'lib/amigo/autoscaler.rb', line 40

def poll_interval
  @poll_interval
end

Instance Method Details

#alert_log(names_and_latencies, depth:, duration:) ⇒ Object 



250
251
252
 
# File 'lib/amigo/autoscaler.rb', line 250

def alert_log(names_and_latencies, depth:, duration:)
  self._log(:warn, "high_latency_queues", queues: names_and_latencies, depth: depth, duration: duration)
end

#alert_restored_log(depth:, duration:) ⇒ Object 



256
257
258
 
# File 'lib/amigo/autoscaler.rb', line 256

def alert_restored_log(depth:, duration:)
  self._log(:info, "high_latency_queues_restored", depth: depth, duration: duration)
end

#alert_sentry(names_and_latencies) ⇒ Object 



242
243
244
245
246
247
248
 
# File 'lib/amigo/autoscaler.rb', line 242

def alert_sentry(names_and_latencies)
  Sentry.with_scope do |scope|
    scope.set_extras(high_latency_queues: names_and_latencies)
    names = names_and_latencies.map(&:first).sort.join(", ")
    Sentry.capture_message("Some queues have a high latency: #{names}")
  end
end

#alert_test(_names_and_latencies, _opts = {}) ⇒ Object 



254
 
# File 'lib/amigo/autoscaler.rb', line 254

def alert_test(_names_and_latencies, _opts={}); end

#check ⇒ Object 



191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
 
# File 'lib/amigo/autoscaler.rb', line 191

def check
  now = Time.now
  skip_check = now < (@last_alerted + self.alert_interval)
  if skip_check
    self._log(:debug, "async_autoscaler_skip_check")
    return
  end
  self._log(:info, "async_autoscaler_check")
  high_latency_queues = Sidekiq::Queue.all.
    map { |q| [q.name, q.latency] }.
    select { |(_, latency)| latency > self.latency_threshold }.
    to_h
  if high_latency_queues.empty?
    # Whenever we are in a latency event, we have a depth > 0. So a depth of 0 means
    # we're not in a latency event, and still have no latency, so can noop.
    return if @depth.zero?
    # We WERE in a latency event, and now we're not, so report on it.
    @restored_methods.each do |m|
      m.call(depth: @depth, duration: (Time.now - @latency_event_started).to_f)
    end
    # Reset back to 0 depth so we know we're not in a latency event.
    @depth = 0
    @latency_event_started = Time.at(0)
    @last_alerted = now
    self.persist
    return
  end
  if @depth.positive?
    # We have already alerted, so increment the depth and when the latency started.
    @depth += 1
    duration = (Time.now - @latency_event_started).to_f
  else
    # Indicate we are starting a high latency event.
    @depth = 1
    @latency_event_started = Time.now
    duration = 0.0
  end
  # Alert each handler. For legacy reasons, we support handlers that accept
  # ({queues and latencies}) and ({queues and latencies}, {}keywords}).
  kw = {depth: @depth, duration: duration}
  @alert_methods.each do |m|
    if m.respond_to?(:arity) && m.arity == 1
      m.call(high_latency_queues)
    else
      m.call(high_latency_queues, **kw)
    end
  end
  @last_alerted = now
  self.persist
end

#polling_thread ⇒ Object 



123
124
125
 
# File 'lib/amigo/autoscaler.rb', line 123

def polling_thread
  return @polling_thread
end

#setup ⇒ Object 



127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/amigo/autoscaler.rb', line 127

def setup
  # Store these as strings OR procs, rather than grabbing self.method here.
  # It gets extremely hard ot test if we capture the method here.
  @alert_methods = self.handlers.map { |a| _handler_to_method("alert_", a) }
  @restored_methods = self.latency_restored_handlers.map { |a| _handler_to_method("alert_restored_", a) }
  @stop = false
  Sidekiq.redis do |r|
    @last_alerted = Time.at((r.get("#{namespace}/last_alerted") || 0).to_f)
    @depth = (r.get("#{namespace}/depth") || 0).to_i
    @latency_event_started = Time.at((r.get("#{namespace}/latency_event_started") || 0).to_f)
  end
end

#start ⇒ Object 



170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
 
# File 'lib/amigo/autoscaler.rb', line 170

def start
  raise "already started" unless @polling_thread.nil?

  hostname = ENV.fetch("DYNO") { Socket.gethostname }
  return false unless self.hostname_regex.match?(hostname)

  self._log(:info, "async_autoscaler_starting")
  self.setup
  @polling_thread = Thread.new do
    until @stop
      Kernel.sleep(self.poll_interval)
      self.check unless @stop
    end
  end
  return true
end

#stop ⇒ Object 



187
188
189
 
# File 'lib/amigo/autoscaler.rb', line 187

def stop
  @stop = true
end

#unpersist ⇒ Object

Delete all the keys that Autoscaler stores. Can be used in extreme cases where things need to be cleaned up, but should not be normally used.



151
152
153
154
155
156
157
 
# File 'lib/amigo/autoscaler.rb', line 151

def unpersist
  Sidekiq.redis do |r|
    r.del("#{namespace}/last_alerted")
    r.del("#{namespace}/depth")
    r.del("#{namespace}/latency_event_started")
  end
end