Module: Resque::Plugins::Retry

Includes:
Logging
Included in:
ExponentialBackoff
Defined in:
lib/resque/plugins/retry.rb,
lib/resque/plugins/retry/logging.rb

Overview

If you want your job to retry on failure, simply extend your module/class with this module:

class DeliverWebHook
  extend Resque::Plugins::Retry # allows 1 retry by default.
  @queue = :web_hooks

  def self.perform(url, hook_id, hmac_key)
    heavy_lifting
  end
end

Easily do something custom:

class DeliverWebHook
  extend Resque::Plugins::Retry
  @queue = :web_hooks

  @retry_limit = 8  # default: 1
  @retry_delay = 60 # default: 0

  # used to build redis key, for counting job attempts.
  def self.retry_identifier(url, hook_id, hmac_key)
    "#{url}-#{hook_id}"
  end

  def self.perform(url, hook_id, hmac_key)
    heavy_lifting
  end
end

Defined Under Namespace

Modules: Logging Classes: AmbiguousRetryStrategyException, RetryConfigurationException

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Logging

#log_message

Instance Attribute Details

#expire_retry_key_after ⇒ Number (readonly)

This method is abstract.

The number of seconds to set the TTL to on the resque-retry key in redis

Returns:

  • (Number) —

    number of seconds



294
295
296
 
# File 'lib/resque/plugins/retry.rb', line 294

def expire_retry_key_after
  @expire_retry_key_after
end

#fatal_exceptions ⇒ Array? (readonly)

This method is abstract.

Controls what exceptions may not be retried

Default: ‘nil` - this will retry all exceptions.

Returns:

  • (Array, nil) 


270
271
272
 
# File 'lib/resque/plugins/retry.rb', line 270

def fatal_exceptions
  @fatal_exceptions
end

Class Method Details

.extended(receiver) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Fail fast, when extended, if the “receiver” is misconfigured



54
55
56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/resque/plugins/retry.rb', line 54

def self.extended(receiver)
  retry_exceptions = nil
  retry_exceptions = receiver.instance_variable_get(:@retry_exceptions) \
    if receiver.instance_variable_defined?(:@retry_exceptions)

  fatal_exceptions = nil
  fatal_exceptions = receiver.instance_variable_get(:@fatal_exceptions) \
    if receiver.instance_variable_defined?(:@fatal_exceptions)

  if fatal_exceptions && retry_exceptions
    raise AmbiguousRetryStrategyException.new(%{You can't define both "@fatal_exceptions" and "@retry_exceptions"})
  end
end

Instance Method Details

#after_perform_retry(*args) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Resque after_perform hook

Deletes retry attempt count from Redis.



485
486
487
488
489
 
# File 'lib/resque/plugins/retry.rb', line 485

def after_perform_retry(*args)
  return if Resque.inline?
  log_message 'after_perform_retry, clearing retry key', args
  clean_retry_key(*args)
end

#before_perform_retry(*args) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Resque before_perform hook

Increments ‘@retry_attempt` count and updates the “retry_key” expiration time (if applicable)



461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
 
# File 'lib/resque/plugins/retry.rb', line 461

def before_perform_retry(*args)
  return if Resque.inline?
  log_message 'before_perform_retry', args
  @on_failure_retry_hook_already_called = false

  # store number of retry attempts.
  retry_key = redis_retry_key(*args)
  Resque.redis.setnx(retry_key, -1)
  @retry_attempt = Resque.redis.incr(retry_key)
  log_message "attempt: #{@retry_attempt} set in Redis", args

  # set/update the "retry_key" expiration
  if expire_retry_key_after
    log_message "updating expiration for retry key: #{retry_key}", args
    exception_class = Object.const_get(args[0]) rescue nil
    Resque.redis.expire(retry_key, retry_delay(exception_class) + expire_retry_key_after)
  end
end

#call_symbol_or_block(method, *args) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Helper to call functions that may be passed as Symbols or Procs. If a symbol, it is assumed to refer to a method that is already defined on this class.

Parameters:

  • method (Symbol|Proc)
  • *args (Object...)

Returns:

  • (Object) 


665
666
667
668
669
670
671
 
# File 'lib/resque/plugins/retry.rb', line 665

def call_symbol_or_block(method, *args)
  if method.is_a?(Symbol)
    send(method, *args)
  elsif method.respond_to?(:call)
    instance_exec(*args, &method)
  end
end

#clean_retry_key(*args) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Clean up retry state from redis once done



549
550
551
552
 
# File 'lib/resque/plugins/retry.rb', line 549

def clean_retry_key(*args)
  log_message 'clean_retry_key', args
  Resque.redis.del(redis_retry_key(*args))
end

#give_up(exception, *args) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

We failed and we’re not retrying.



449
450
451
452
453
 
# File 'lib/resque/plugins/retry.rb', line 449

def give_up(exception, *args)
  log_message 'retry criteria not sufficient for retry', args, exception
  run_give_up_callbacks(exception, *args)
  clean_retry_key(*args)
end

#give_up_callback(method = nil) {|exception, *args| ... } ⇒ Object

Register a give up callback that will be called when the job fails and is not retrying. Can be registered with a block or a symbol.

Examples:

Registering a callback with a block


give_up_callback do |exception, *args|
  logger.error(
    "Resque job received exception #{exception} and is giving up")
end

Registering a callback with a Symbol


give_up_callback :my_callback

Parameters:

  • method (Symbol?) (defaults to: nil)

Yields:

  • (exception, *args)

Yield Parameters:

  • exception (Exception) —

    the exception that was raised

  • args (Array) —

    job arguments



632
633
634
635
636
637
638
 
# File 'lib/resque/plugins/retry.rb', line 632

def give_up_callback(method=nil, &block)
  if method.is_a? Symbol
    give_up_callbacks << method
  elsif block_given?
    give_up_callbacks << block
  end
end

#give_up_callbacks ⇒ Array<Proc>

Returns the give up callbacks.

Returns:

  • (Array<Proc>) 


608
609
610
 
# File 'lib/resque/plugins/retry.rb', line 608

def give_up_callbacks
  @give_up_callbacks ||= []
end

#ignore_exceptions ⇒ Object 



652
653
654
 
# File 'lib/resque/plugins/retry.rb', line 652

def ignore_exceptions
  @ignore_exceptions ||= []
end

#inherited(subclass) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Copy retry criteria checks, try again callbacks, and give up callbacks on inheritance.



72
73
74
75
76
77
 
# File 'lib/resque/plugins/retry.rb', line 72

def inherited(subclass)
  super(subclass)
  subclass.instance_variable_set('@retry_criteria_checks', retry_criteria_checks.dup)
  subclass.instance_variable_set('@try_again_callbacks', try_again_callbacks.dup)
  subclass.instance_variable_set('@give_up_callbacks', give_up_callbacks.dup)
end

#instance_exec(*args, &block) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Used to perform retry criteria check blocks under the job instance’s context

Returns:

  • (Object) —

    return value of the criteria check



535
536
537
538
539
540
541
542
543
544
 
# File 'lib/resque/plugins/retry.rb', line 535

def instance_exec(*args, &block)
  mname = "__instance_exec_#{Thread.current.object_id.abs}"
  class << self; self end.class_eval{ define_method(mname, &block) }
  begin
    ret = send(mname, *args)
  ensure
    class << self; self end.class_eval{ undef_method(mname) } rescue nil
  end
  ret
end

#on_failure_retry(exception, *args) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This hook will only allow execution once per job perform attempt. This was added because Resque v1.20.0 calls the hook twice. IMO; this isn’t something resque-retry should have to worry about!

Resque on_failure hook

Checks if our retry criteria is valid, if it is we try again. Otherwise the retry attempt count is deleted from Redis.



501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
 
# File 'lib/resque/plugins/retry.rb', line 501

def on_failure_retry(exception, *args)
  return if Resque.inline?
  log_message 'on_failure_retry', args, exception
  if exception.is_a?(Resque::DirtyExit)
    # This hook is called from a worker processes, not the job process
    # that failed with a DirtyExit, so @retry_attempt wasn't set yet
    @retry_attempt = Resque.redis.get(redis_retry_key(*args)).to_i
  elsif instance_variable_defined?(:@on_failure_retry_hook_already_called) && \
    @on_failure_retry_hook_already_called
    log_message 'on_failure_retry_hook_already_called', args, exception
    return
  end

  # If we are "ignoring" the exception, then we decrement the retry
  # counter, so that the current attempt didn't count toward the retry
  # counter.
  if ignore_exceptions.include?(exception.class)
    @retry_attempt = Resque.redis.decr(redis_retry_key(*args))
  end

  if retry_criteria_valid?(exception, *args)
    try_again(exception, *args)
  else
    give_up(exception, *args)
  end

  @on_failure_retry_hook_already_called = true
end

#redis_retry_key(*args) ⇒ String

Builds the redis key to be used for keeping state of the job attempts.

Returns:

  • (String) —

    redis key



101
102
103
 
# File 'lib/resque/plugins/retry.rb', line 101

def redis_retry_key(*args)
  ['resque-retry', name, retry_identifier(*args)].compact.join(':').gsub(/\s/, '')
end

#retry_args(*args) ⇒ Array

This method is abstract.

Modify the arguments used to retry the job. Use this to do something other than try the exact same job again

Returns:

  • (Array) —

    new job arguments



208
209
210
 
# File 'lib/resque/plugins/retry.rb', line 208

def retry_args(*args)
  args.dup
end

#retry_args_for_exception(exception, *args) ⇒ Array

This method is abstract.

Modify the arguments used to retry the job based on the exception. Use this to do something other than try the exact same job again.

Returns:

  • (Array) —

    new job arguments



219
220
221
 
# File 'lib/resque/plugins/retry.rb', line 219

def retry_args_for_exception(exception, *args)
  retry_args(*args)
end

#retry_attempt ⇒ Fixnum

Number of retry attempts used to try and perform the job

The real value is kept in Redis, it is accessed and incremented using a before_perform hook.

Returns:

  • (Fixnum) —

    number of attempts



141
142
143
 
# File 'lib/resque/plugins/retry.rb', line 141

def retry_attempt
  @retry_attempt ||= 0
end

#retry_criteria_check(method = nil) {|exception, *args| ... } ⇒ Object

Register a retry criteria check callback to be run before retrying the job again. Can be registered with a block or a symbol.

If any callback returns ‘true`, the job will be retried.

Examples:

Registering a custom retry criteria check.


retry_criteria_check do |exception, *args|
  if exception.message =~ /InvalidJobId/
    # don't retry if we got passed a invalid job id.
    false
  else
    true
  end
end

retry_criteria_check :my_check

Parameters:

  • method (Symbol?) (defaults to: nil)

Yields:

  • (exception, *args)

Yield Parameters:

  • exception (Exception) —

    the exception that was raised

  • args (Array) —

    job arguments

Yield Returns:

  • (Boolean) —

    false == dont retry, true == can retry



375
376
377
378
379
380
381
 
# File 'lib/resque/plugins/retry.rb', line 375

def retry_criteria_check(method=nil, &block)
  if method.is_a? Symbol
    retry_criteria_checks << method
  elsif block_given?
    retry_criteria_checks << block
  end
end

#retry_criteria_checks ⇒ Array

Retry criteria checks

Returns:

  • (Array) 


329
330
331
 
# File 'lib/resque/plugins/retry.rb', line 329

def retry_criteria_checks
  @retry_criteria_checks ||= []
end

#retry_criteria_checks_pass?(exception, *args) ⇒ Boolean

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns true if any of the retry criteria checks pass. When a retry criteria check passes, the remaining ones are not executed.

Returns:

  • (Boolean) 


389
390
391
392
393
394
 
# File 'lib/resque/plugins/retry.rb', line 389

def retry_criteria_checks_pass?(exception, *args)
  retry_criteria_checks.each do |criteria_check|
    return true if !!call_symbol_or_block(criteria_check, exception, *args)
  end
  false
end

#retry_criteria_valid?(exception, *args) ⇒ Boolean

Test if the retry criteria is valid

Parameters:

  • exception (Exception)
  • args (Array) —

    job arguments

Returns:

  • (Boolean) 


303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
 
# File 'lib/resque/plugins/retry.rb', line 303

def retry_criteria_valid?(exception, *args)
  # if the retry limit was reached, dont bother checking anything else.
  if retry_limit_reached?
    log_message 'retry limit reached', args, exception
    return false
  end

  # We always want to retry if the exception matches.
  retry_based_on_exception = retry_exception?(exception)
  log_message "Exception is #{retry_based_on_exception ? '' : 'not '}sufficient for a retry", args, exception

  retry_based_on_criteria = false
  unless retry_based_on_exception
    # call user retry criteria check blocks.
    retry_based_on_criteria = retry_criteria_checks_pass?(exception, *args)
    log_message "user retry criteria is #{retry_based_on_criteria ? '' : 'not '}sufficient for a retry", args, exception
  end

  retry_based_on_exception || retry_based_on_criteria
end

#retry_delay(exception_class = nil) ⇒ Number

This method is abstract.

Number of seconds to delay until the job is retried If @retry_exceptions is a Hash and there is no delay defined for exception_class, looks for closest superclass and assigns it’s delay to @retry_exceptions

Returns:

  • (Number) —

    number of seconds to delay



153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
 
# File 'lib/resque/plugins/retry.rb', line 153

def retry_delay(exception_class = nil)
  if \
    !exception_class.nil? && \
    instance_variable_defined?(:@retry_exceptions) && \
    @retry_exceptions.is_a?(Hash)
    delay = @retry_exceptions[exception_class] ||= begin
      relevant_definitions = \
        @retry_exceptions.select { |ex| exception_class <= ex }
      relevant_definitions.any? ? relevant_definitions.sort.first[1] : 0
    end
    # allow an array of delays.
    delay.is_a?(Array) ? delay[retry_attempt] || delay.last : delay
  else
    @retry_delay ||= 0
  end
end

#retry_exception?(exception) ⇒ Boolean

Convenience method to test whether you may retry on a given exception

also be a Class

Parameters:

  • an (Exception) —

    instance of Exception. Deprecated: can

Returns:

  • (Boolean) 


232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
 
# File 'lib/resque/plugins/retry.rb', line 232

def retry_exception?(exception)
  # If both "fatal_exceptions" and "retry_exceptions" are undefined we are
  # done (we should retry the exception)
  #
  # It is intentional that we check "retry_exceptions" first since it is
  # more likely that it will be defined (over "fatal_exceptions") as it
  # has been part of the API for quite a while
  return true if retry_exceptions.nil? && fatal_exceptions.nil?

  # If "fatal_exceptions" is undefined interrogate "retry_exceptions"
  if fatal_exceptions.nil?
    retry_exceptions.any? do |ex|
      if exception.is_a?(Class)
        ex >= exception
      else
        ex === exception
      end
    end
  # It is safe to assume we need to check "fatal_exceptions" at this point
  else
    fatal_exceptions.none? do |ex|
      if exception.is_a?(Class)
        ex >= exception
      else
        ex === exception
      end
    end
  end
end

#retry_exceptions ⇒ Array?

This method is abstract.

Controls what exceptions may be retried

Default: ‘nil` - this will retry all exceptions.

Returns:

  • (Array, nil) 


280
281
282
283
284
285
286
 
# File 'lib/resque/plugins/retry.rb', line 280

def retry_exceptions
  if instance_variable_defined?(:@retry_exceptions) && @retry_exceptions.is_a?(Hash)
    @retry_exceptions.keys
  else
    @retry_exceptions ||= nil
  end
end

#retry_identifier(*args) ⇒ String

This method is abstract.

You may override to implement a custom retry identifier, you should consider doing this if your job arguments are many/long or may not cleanly convert to strings.

Builds a retry identifier using the job arguments. This identifier is used as part of the redis key

Parameters:

  • args (Array) —

    job arguments

Returns:

  • (String) —

    job identifier



90
91
92
93
 
# File 'lib/resque/plugins/retry.rb', line 90

def retry_identifier(*args)
  args_string = args.join('-')
  args_string.empty? ? nil : Digest::SHA1.hexdigest(args_string)
end

#retry_job_delegate ⇒ Object?

This method is abstract.

Specify another resque job (module or class) to delegate retry duties to upon failure

Returns:

  • (Object, nil) —

    class or module if delegate on failure, otherwise nil



187
188
189
 
# File 'lib/resque/plugins/retry.rb', line 187

def retry_job_delegate
  @retry_job_delegate ||= nil
end

#retry_limit ⇒ Fixnum

Maximum number of retrys we can attempt to successfully perform the job

A retry limit of 0 will never retry. A retry limit of -1 or below will retry forever.

The default value is: ‘1` or in the case of where `@retry_exceptions` is specified, and it contains one or more `Array` values, the maximum length will be used (e.g. `@retry_exceptions = { NetworkError => 30, SystemCallError => [120, 240] }` would return `2` because `SystemCallError` should be attempted at least twice to respect the specified configuration).

Returns:

  • (Fixnum) 


119
120
121
122
123
124
125
126
127
128
129
130
131
 
# File 'lib/resque/plugins/retry.rb', line 119

def retry_limit
  @retry_limit ||= begin
    default_retry_limit = 1
    if instance_variable_defined?(:@retry_exceptions) && @retry_exceptions.is_a?(Hash)
      @retry_exceptions.values.each do |value|
        if value.is_a?(Array) && value.length > default_retry_limit
          default_retry_limit = value.length
        end
      end
    end
    default_retry_limit
  end
end

#retry_limit_reached? ⇒ Boolean

Test if the retry limit has been reached

Returns:

  • (Boolean) 


338
339
340
341
342
343
344
345
346
 
# File 'lib/resque/plugins/retry.rb', line 338

def retry_limit_reached?
  if retry_limit == 0
    true
  elsif retry_limit > 0
    true if retry_attempt >= retry_limit
  else
    false
  end
end

#retry_queue(exception, *args) ⇒ Symbol

This method is abstract.

Specify the queue that the job should be placed in upon failure

Returns:

  • (Symbol) —

    Symbol representing queue that job should be placed in



197
198
199
 
# File 'lib/resque/plugins/retry.rb', line 197

def retry_queue(exception, *args)
  nil
end

#run_give_up_callbacks(exception, *args) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Runs all the give up callbacks.

Parameters:

  • exception (Exception)
  • args (Object...) 


646
647
648
649
650
 
# File 'lib/resque/plugins/retry.rb', line 646

def run_give_up_callbacks(exception, *args)
  give_up_callbacks.each do |callback|
    call_symbol_or_block(callback, exception, *args)
  end
end

#run_try_again_callbacks(exception, *args) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Runs all the try again callbacks.

Parameters:

  • exception (Exception)
  • args (Object...) 


597
598
599
600
601
 
# File 'lib/resque/plugins/retry.rb', line 597

def run_try_again_callbacks(exception, *args)
  try_again_callbacks.each do |callback|
    call_symbol_or_block(callback, exception, *args)
  end
end

#sleep_after_requeue ⇒ Number

This method is abstract.

Number of seconds to sleep after job is requeued

Returns:

  • (Number) —

    number of seconds to sleep



176
177
178
 
# File 'lib/resque/plugins/retry.rb', line 176

def sleep_after_requeue
  @sleep_after_requeue ||= 0
end

#try_again(exception, *args) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Retries the job



399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
 
# File 'lib/resque/plugins/retry.rb', line 399

def try_again(exception, *args)
  log_message 'try_again', args, exception
  run_try_again_callbacks(exception, *args)

  # some plugins define retry_delay and have it take no arguments, so rather than break those,
  # we'll just check here to see whether it takes the additional exception class argument or not
  # we also allow all job args to be passed to a custom `retry_delay` method
  retry_delay_arity = method(:retry_delay).arity

  temp_retry_delay = if [-2, 2].include?(retry_delay_arity)
    retry_delay(exception.class, *args)
  elsif [-1, 1].include?(retry_delay_arity)
    retry_delay(exception.class)
  else
    retry_delay
  end

  retry_job_class = retry_job_delegate ? retry_job_delegate : self

  retry_in_queue = retry_queue(exception, *args)
  retry_in_queue ||= Resque.queue_from_class(retry_job_class)

  log_message "retry delay: #{temp_retry_delay} for queue: #{retry_in_queue}", args, exception

  # remember that this job is now being retried. before_perform_retry will increment
  # this so it represents the retry count, and MultipleWithRetrySuppression uses
  # the existence of this to determine if the job should be sent to the
  # parent failure backend (e.g. failed queue) or not.  Removing this means
  # jobs that fail before ::perform will be both retried and sent to the failed queue.
  Resque.redis.setnx(redis_retry_key(*args), -1)

  retry_args = retry_args_for_exception(exception, *args)

  if temp_retry_delay <= 0
    # If the delay is 0, no point passing it through the scheduler
    Resque.enqueue_to(retry_in_queue, retry_job_class, *retry_args)
  else
    Resque.enqueue_in_with_queue(retry_in_queue, temp_retry_delay, retry_job_class, *retry_args)
  end

  # remove retry key from redis if we handed retry off to another queue.
  clean_retry_key(*args) if retry_job_delegate

  # sleep after requeue if enabled.
  sleep(sleep_after_requeue) if sleep_after_requeue > 0
end

#try_again_callback(method = nil) {|exception, *args| ... } ⇒ Object

Register a try again callback that will be called when the job fails but is trying again. Can be registered with a block or a symbol.

Examples:

Registering a callback with a block


try_again_callback do |exception, *args|
  logger.error(
    "Resque job received exception #{exception} and is trying again")
end

Registering a callback with a Symbol


try_again_callback :my_callback

Parameters:

  • method (Symbol?) (defaults to: nil)

Yields:

  • (exception, *args)

Yield Parameters:

  • exception (Exception) —

    the exception that was raised

  • args (Array) —

    job arguments



583
584
585
586
587
588
589
 
# File 'lib/resque/plugins/retry.rb', line 583

def try_again_callback(method=nil, &block)
  if method.is_a? Symbol
    try_again_callbacks << method
  elsif block_given?
    try_again_callbacks << block
  end
end

#try_again_callbacks ⇒ Array<Proc>

Returns the try again callbacks.

Returns:

  • (Array<Proc>) 


559
560
561
 
# File 'lib/resque/plugins/retry.rb', line 559

def try_again_callbacks
  @try_again_callbacks ||= []
end