Class: Sidekiq::Middleware::Server::RetryJobs
- Inherits:
-
Object
- Object
- Sidekiq::Middleware::Server::RetryJobs
- Includes:
- Util
- Defined in:
- lib/sidekiq/middleware/server/retry_jobs.rb
Overview
Automatically retry jobs that fail in Sidekiq. Sidekiq’s retry support assumes a typical development lifecycle:
-
push some code changes with a bug in it
-
bug causes message processing to fail, sidekiq’s middleware captures the message and pushes it onto a retry queue
-
sidekiq retries messages in the retry queue multiple times with an exponential delay, the message continues to fail
-
after a few days, a developer deploys a fix. the message is reprocessed successfully.
-
if 3 never happens, sidekiq will eventually give up and throw the message away.
A message looks like:
{ 'class' => 'HardWorker', 'args' => [1, 2, 'foo'] }
The ‘retry’ option also accepts a number (in place of ‘true’):
{ 'class' => 'HardWorker', 'args' => [1, 2, 'foo'], 'retry' => 5 }
The job will be retried this number of times before giving up. (If simply ‘true’, Sidekiq retries 25 times)
We’ll add a bit more data to the message to support retries:
* 'queue' - the queue to use
* 'retry_count' - number of times we've retried so far.
* 'error_message' - the message from the exception
* 'error_class' - the exception class
* 'failed_at' - the first time it failed
* 'retried_at' - the last time it was retried
We don’t store the backtrace as that can add a lot of overhead to the message and everyone is using Airbrake, right?
Constant Summary collapse
- DEFAULT_MAX_RETRY_ATTEMPTS =
delayed_job uses the same basic formula
25
- DELAY =
proc { |count| (count ** 4) + 15 + (rand(30)*(count+1)) }
Constants included from Util
Instance Method Summary collapse
Methods included from Util
#hostname, #logger, #process_id, #redis, #watchdog
Methods included from ExceptionHandler
Instance Method Details
#call(worker, msg, queue) ⇒ Object
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
# File 'lib/sidekiq/middleware/server/retry_jobs.rb', line 48 def call(worker, msg, queue) yield rescue Exception => e raise e unless msg['retry'] max_retry_attempts = retry_attempts_from(msg['retry'], DEFAULT_MAX_RETRY_ATTEMPTS) msg['queue'] = queue msg['error_message'] = e. msg['error_class'] = e.class.name count = if msg['retry_count'] msg['retried_at'] = Time.now.utc msg['retry_count'] += 1 else msg['failed_at'] = Time.now.utc msg['retry_count'] = 0 end if msg['backtrace'] == true msg['error_backtrace'] = e.backtrace elsif msg['backtrace'].to_i != 0 msg['error_backtrace'] = e.backtrace[0..msg['backtrace'].to_i] end if count < max_retry_attempts delay = DELAY.call(count) logger.debug { "Failure! Retry #{count} in #{delay} seconds" } retry_at = Time.now.to_f + delay payload = Sidekiq.dump_json(msg) Sidekiq.redis do |conn| conn.zadd('retry', retry_at.to_s, payload) end else # Goodbye dear message, you (re)tried your best I'm sure. logger.debug { "Dropping message after hitting the retry maximum: #{msg}" } end raise e end |
#retry_attempts_from(msg_retry, default) ⇒ Object
86 87 88 89 90 91 92 |
# File 'lib/sidekiq/middleware/server/retry_jobs.rb', line 86 def retry_attempts_from(msg_retry, default) if msg_retry.is_a?(Fixnum) msg_retry else default end end |