Class: Amigo::Autoscaler
- Inherits:
-
Object
- Object
- Amigo::Autoscaler
- Defined in:
- lib/amigo/autoscaler.rb,
lib/amigo/autoscaler/heroku.rb
Defined Under Namespace
Classes: Heroku, InvalidHandler
Instance Attribute Summary collapse
-
#alert_interval ⇒ Integer
readonly
Only alert this often.
-
#handlers ⇒ Array<String,Symbol,Proc,#call>
readonly
Methods to call when alerting, as strings/symbols or procs.
-
#hostname_regex ⇒ Regexp
readonly
What hosts/processes should this run on? Looks at ENV and Socket.gethostname for a match.
-
#latency_restored_handlers ⇒ Array<String,Symbol,Proc,#call>
readonly
Methods to call when a latency of
latency_restored_threshold
is reached (ie, when we get back to normal latency after a high latency event). -
#latency_restored_threshold ⇒ Object
readonly
After an alert happens, what latency should be considered “back to normal” and
latency_restored_handlers
will be called? In most cases this should be the same as (and defaults to)latency_threshold
so that we’re ‘back to normal’ once we’re below the threshold. -
#latency_threshold ⇒ Integer
readonly
What latency should we alert on?.
-
#log ⇒ Object
readonly
Proc/callable called with (level, message, params={}).
-
#poll_interval ⇒ Integer
readonly
How often should Autoscaler check for latency?.
Instance Method Summary collapse
- #alert_log(names_and_latencies, depth:, duration:) ⇒ Object
- #alert_restored_log(depth:, duration:) ⇒ Object
- #alert_sentry(names_and_latencies) ⇒ Object
- #alert_test(_names_and_latencies, _opts = {}) ⇒ Object
- #check ⇒ Object
-
#initialize(poll_interval: 20, latency_threshold: 5, hostname_regex: /^web\.1$/, handlers: [:log], alert_interval: 120, latency_restored_threshold: latency_threshold, latency_restored_handlers: [:log], log: ->(level, message, params={}) { Amigo.log(nil, level, message, params) }) ⇒ Autoscaler
constructor
A new instance of Autoscaler.
- #polling_thread ⇒ Object
- #setup ⇒ Object
- #start ⇒ Object
- #stop ⇒ Object
-
#unpersist ⇒ Object
Delete all the keys that Autoscaler stores.
Constructor Details
#initialize(poll_interval: 20, latency_threshold: 5, hostname_regex: /^web\.1$/, handlers: [:log], alert_interval: 120, latency_restored_threshold: latency_threshold, latency_restored_handlers: [:log], log: ->(level, message, params={}) { Amigo.log(nil, level, message, params) }) ⇒ Autoscaler
Returns a new instance of Autoscaler.
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 |
# File 'lib/amigo/autoscaler.rb', line 96 def initialize( poll_interval: 20, latency_threshold: 5, hostname_regex: /^web\.1$/, handlers: [:log], alert_interval: 120, latency_restored_threshold: latency_threshold, latency_restored_handlers: [:log], log: ->(level, , params={}) { Amigo.log(nil, level, , params) } ) raise ArgumentError, "latency_threshold must be > 0" if latency_threshold <= 0 raise ArgumentError, "latency_restored_threshold must be >= 0" if latency_restored_threshold.negative? raise ArgumentError, "latency_restored_threshold must be <= latency_threshold" if latency_restored_threshold > latency_threshold @poll_interval = poll_interval @latency_threshold = latency_threshold @hostname_regex = hostname_regex @handlers = handlers.freeze @alert_interval = alert_interval @latency_restored_threshold = latency_restored_threshold @latency_restored_handlers = latency_restored_handlers.freeze @log = log end |
Instance Attribute Details
#alert_interval ⇒ Integer (readonly)
Only alert this often. For example, with poll_interval of 10 seconds and alert_interval of 200 seconds, we’d alert once and then 210 seconds later.
71 72 73 |
# File 'lib/amigo/autoscaler.rb', line 71 def alert_interval @alert_interval end |
#handlers ⇒ Array<String,Symbol,Proc,#call> (readonly)
Methods to call when alerting, as strings/symbols or procs. Valid string values are ‘log’ and ‘sentry’ (requires Sentry to be required already). Anything that responds to call
will be invoked with:
-
Positional argument which is a
Hash
of ‘name => latency in seconds` -
Keyword argument
:depth
: Number of alerts as part of this latency event. For example, the first alert has a depth of 1, and if latency stays high, it’ll be 2 on the next call, etc.depth
can be used to incrementally provision additional processing capacity, and stop adding capacity at a certain depth to avoid problems with too many workers (like excessive DB load). -
Keyword argument
:duration
: Number of seconds since this latency spike started. -
Additional undefined keywords. Handlers should accept additional options, like via ‘**kw` or `opts={}`, for compatibility.
65 66 67 |
# File 'lib/amigo/autoscaler.rb', line 65 def handlers @handlers end |
#hostname_regex ⇒ Regexp (readonly)
What hosts/processes should this run on? Looks at ENV and Socket.gethostname for a match. Default to only run on ‘web.1’, which is the first Heroku web dyno. We run on the web, not worker, dyno, so we report backed up queues in case we, say, turn off all workers (broken web processes are generally easier to find).
51 52 53 |
# File 'lib/amigo/autoscaler.rb', line 51 def hostname_regex @hostname_regex end |
#latency_restored_handlers ⇒ Array<String,Symbol,Proc,#call> (readonly)
Methods to call when a latency of latency_restored_threshold
is reached (ie, when we get back to normal latency after a high latency event). Valid string values are ‘log’. Usually this handler will deprovision capacity procured as part of the alert handlers
. Anything that responds to call
will be invoked with:
-
Keyword
:depth
, the number of times an alert happened before the latency spike was resolved. -
Keyword
:duration
, the number of seconds for the latency spike has been going on. -
Additional undefined keywords. Handlers should accept additional options, like via ‘**kw`, for compatibility.
91 92 93 |
# File 'lib/amigo/autoscaler.rb', line 91 def latency_restored_handlers @latency_restored_handlers end |
#latency_restored_threshold ⇒ Object (readonly)
After an alert happens, what latency should be considered “back to normal” and latency_restored_handlers
will be called? In most cases this should be the same as (and defaults to) latency_threshold
so that we’re ‘back to normal’ once we’re below the threshold. It may also commonly be 0, so that the callback is fired when the queue is entirely clear. Note that, if latency_restored_threshold
is less than latency_threshold
, while the latency is between the two, no alerts will fire.
79 80 81 |
# File 'lib/amigo/autoscaler.rb', line 79 def latency_restored_threshold @latency_restored_threshold end |
#latency_threshold ⇒ Integer (readonly)
What latency should we alert on?
43 44 45 |
# File 'lib/amigo/autoscaler.rb', line 43 def latency_threshold @latency_threshold end |
#log ⇒ Object (readonly)
Proc/callable called with (level, message, params={}). By default, use Amigo.log
(which logs to the Sidekiq logger).
94 95 96 |
# File 'lib/amigo/autoscaler.rb', line 94 def log @log end |
#poll_interval ⇒ Integer (readonly)
How often should Autoscaler check for latency?
40 41 42 |
# File 'lib/amigo/autoscaler.rb', line 40 def poll_interval @poll_interval end |
Instance Method Details
#alert_log(names_and_latencies, depth:, duration:) ⇒ Object
250 251 252 |
# File 'lib/amigo/autoscaler.rb', line 250 def alert_log(names_and_latencies, depth:, duration:) self._log(:warn, "high_latency_queues", queues: names_and_latencies, depth: depth, duration: duration) end |
#alert_restored_log(depth:, duration:) ⇒ Object
256 257 258 |
# File 'lib/amigo/autoscaler.rb', line 256 def alert_restored_log(depth:, duration:) self._log(:info, "high_latency_queues_restored", depth: depth, duration: duration) end |
#alert_sentry(names_and_latencies) ⇒ Object
242 243 244 245 246 247 248 |
# File 'lib/amigo/autoscaler.rb', line 242 def alert_sentry(names_and_latencies) Sentry.with_scope do |scope| scope.set_extras(high_latency_queues: names_and_latencies) names = names_and_latencies.map(&:first).sort.join(", ") Sentry.("Some queues have a high latency: #{names}") end end |
#alert_test(_names_and_latencies, _opts = {}) ⇒ Object
254 |
# File 'lib/amigo/autoscaler.rb', line 254 def alert_test(_names_and_latencies, _opts={}); end |
#check ⇒ Object
191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 |
# File 'lib/amigo/autoscaler.rb', line 191 def check now = Time.now skip_check = now < (@last_alerted + self.alert_interval) if skip_check self._log(:debug, "async_autoscaler_skip_check") return end self._log(:info, "async_autoscaler_check") high_latency_queues = Sidekiq::Queue.all. map { |q| [q.name, q.latency] }. select { |(_, latency)| latency > self.latency_threshold }. to_h if high_latency_queues.empty? # Whenever we are in a latency event, we have a depth > 0. So a depth of 0 means # we're not in a latency event, and still have no latency, so can noop. return if @depth.zero? # We WERE in a latency event, and now we're not, so report on it. @restored_methods.each do |m| m.call(depth: @depth, duration: (Time.now - @latency_event_started).to_f) end # Reset back to 0 depth so we know we're not in a latency event. @depth = 0 @latency_event_started = Time.at(0) @last_alerted = now self.persist return end if @depth.positive? # We have already alerted, so increment the depth and when the latency started. @depth += 1 duration = (Time.now - @latency_event_started).to_f else # Indicate we are starting a high latency event. @depth = 1 @latency_event_started = Time.now duration = 0.0 end # Alert each handler. For legacy reasons, we support handlers that accept # ({queues and latencies}) and ({queues and latencies}, {}keywords}). kw = {depth: @depth, duration: duration} @alert_methods.each do |m| if m.respond_to?(:arity) && m.arity == 1 m.call(high_latency_queues) else m.call(high_latency_queues, **kw) end end @last_alerted = now self.persist end |
#polling_thread ⇒ Object
123 124 125 |
# File 'lib/amigo/autoscaler.rb', line 123 def polling_thread return @polling_thread end |
#setup ⇒ Object
127 128 129 130 131 132 133 134 135 136 137 138 |
# File 'lib/amigo/autoscaler.rb', line 127 def setup # Store these as strings OR procs, rather than grabbing self.method here. # It gets extremely hard ot test if we capture the method here. @alert_methods = self.handlers.map { |a| _handler_to_method("alert_", a) } @restored_methods = self.latency_restored_handlers.map { |a| _handler_to_method("alert_restored_", a) } @stop = false Sidekiq.redis do |r| @last_alerted = Time.at((r.get("#{namespace}/last_alerted") || 0).to_f) @depth = (r.get("#{namespace}/depth") || 0).to_i @latency_event_started = Time.at((r.get("#{namespace}/latency_event_started") || 0).to_f) end end |
#start ⇒ Object
170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 |
# File 'lib/amigo/autoscaler.rb', line 170 def start raise "already started" unless @polling_thread.nil? hostname = ENV.fetch("DYNO") { Socket.gethostname } return false unless self.hostname_regex.match?(hostname) self._log(:info, "async_autoscaler_starting") self.setup @polling_thread = Thread.new do until @stop Kernel.sleep(self.poll_interval) self.check unless @stop end end return true end |
#stop ⇒ Object
187 188 189 |
# File 'lib/amigo/autoscaler.rb', line 187 def stop @stop = true end |
#unpersist ⇒ Object
Delete all the keys that Autoscaler stores. Can be used in extreme cases where things need to be cleaned up, but should not be normally used.
151 152 153 154 155 156 157 |
# File 'lib/amigo/autoscaler.rb', line 151 def unpersist Sidekiq.redis do |r| r.del("#{namespace}/last_alerted") r.del("#{namespace}/depth") r.del("#{namespace}/latency_event_started") end end |