Class: Sidekiq::Middleware::Server::RetryJobs

Inherits:
Object
  • Object
show all
Includes:
Util
Defined in:
lib/sidekiq/middleware/server/retry_jobs.rb

Overview

Automatically retry jobs that fail in Sidekiq. Sidekiq’s retry support assumes a typical development lifecycle:

  1. push some code changes with a bug in it

  2. bug causes message processing to fail, sidekiq’s middleware captures the message and pushes it onto a retry queue

  3. sidekiq retries messages in the retry queue multiple times with an exponential delay, the message continues to fail

  4. after a few days, a developer deploys a fix. the message is reprocessed successfully.

  5. if 3 never happens, sidekiq will eventually give up and throw the message away. If the worker defines a method called ‘retries_exhausted’, this will be called before throwing the message away. If the ‘retries_exhausted’ method throws an exception, it’s dropped and logged.

A message looks like:

{ 'class' => 'HardWorker', 'args' => [1, 2, 'foo'] }

The ‘retry’ option also accepts a number (in place of ‘true’):

{ 'class' => 'HardWorker', 'args' => [1, 2, 'foo'], 'retry' => 5 }

The job will be retried this number of times before giving up. (If simply ‘true’, Sidekiq retries 25 times)

We’ll add a bit more data to the message to support retries:

* 'queue' - the queue to use
* 'retry_count' - number of times we've retried so far.
* 'error_message' - the message from the exception
* 'error_class' - the exception class
* 'failed_at' - the first time it failed
* 'retried_at' - the last time it was retried

We don’t store the backtrace as that can add a lot of overhead to the message and everyone is using Airbrake, right?

The default number of retry attempts is 25. You can pass a value for the number of retry attempts when adding the middleware using the options hash:

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Middleware::Server::RetryJobs, :max_retries => 7
  end
end

Constant Summary collapse

DEFAULT_MAX_RETRY_ATTEMPTS =
25

Constants included from Util

Util::EXPIRY

Instance Method Summary collapse

Methods included from Util

#fire_event, #hostname, #identity, #logger, #redis, #watchdog

Methods included from ExceptionHandler

#handle_exception

Constructor Details

#initialize(options = {}) ⇒ RetryJobs

Returns a new instance of RetryJobs.



57
58
59
# File 'lib/sidekiq/middleware/server/retry_jobs.rb', line 57

def initialize(options = {})
  @max_retries = options.fetch(:max_retries, DEFAULT_MAX_RETRY_ATTEMPTS)
end

Instance Method Details

#call(worker, msg, queue) ⇒ Object



61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
# File 'lib/sidekiq/middleware/server/retry_jobs.rb', line 61

def call(worker, msg, queue)
  yield
rescue Sidekiq::Shutdown
  # ignore, will be pushed back onto queue during hard_shutdown
  raise
rescue Exception => e
  # ignore, will be pushed back onto queue during hard_shutdown
  raise Sidekiq::Shutdown if exception_caused_by_shutdown?(e)

  raise e unless msg['retry']
  max_retry_attempts = retry_attempts_from(msg['retry'], @max_retries)

  msg['queue'] = if msg['retry_queue']
    msg['retry_queue']
  else
    queue
  end
  msg['error_message'] = e.message
  msg['error_class'] = e.class.name
  count = if msg['retry_count']
    msg['retried_at'] = Time.now.to_f
    msg['retry_count'] += 1
  else
    msg['failed_at'] = Time.now.to_f
    msg['retry_count'] = 0
  end

  if msg['backtrace'] == true
    msg['error_backtrace'] = e.backtrace
  elsif msg['backtrace'] == false
    # do nothing
  elsif msg['backtrace'].to_i != 0
    msg['error_backtrace'] = e.backtrace[0..msg['backtrace'].to_i]
  end

  if count < max_retry_attempts
    delay = delay_for(worker, count)
    logger.debug { "Failure! Retry #{count} in #{delay} seconds" }
    retry_at = Time.now.to_f + delay
    payload = Sidekiq.dump_json(msg)
    Sidekiq.redis do |conn|
      conn.zadd('retry', retry_at.to_s, payload)
    end
  else
    # Goodbye dear message, you (re)tried your best I'm sure.
    retries_exhausted(worker, msg)
  end

  raise e
end