Class: Gitlab::Database::WithLockRetries

Inherits:
Object
  • Object
show all
Defined in:
lib/gitlab/database/with_lock_retries.rb

Overview

This class provides a way to automatically execute code that relies on acquiring a database lock in a way designed to minimize impact on a busy production database.

A default timing configuration is provided that makes repeated attempts to acquire the necessary lock, with varying lock_timeout settings, and also serves to limit the maximum number of attempts.

Direct Known Subclasses

WithLockRetriesOutsideTransaction

Constant Summary collapse

AttemptsExhaustedError = 
Class.new(StandardError)
NULL_LOGGER = 
Gitlab::JsonLogger.new('/dev/null')
DEFAULT_TIMING_CONFIGURATION =

Each element of the array represents a retry iteration.

  • DEFAULT_TIMING_CONFIGURATION.size provides the iteration count.

  • First element: DB lock_timeout

  • Second element: Sleep time after unsuccessful lock attempt (LockWaitTimeout error raised)

  • Worst case, this configuration would retry for about 40 minutes.

[
  [0.1.seconds, 0.05.seconds], # short timings, lock_timeout: 100ms, sleep after LockWaitTimeout: 50ms
  [0.1.seconds, 0.05.seconds],
  [0.2.seconds, 0.05.seconds],
  [0.3.seconds, 0.10.seconds],
  [0.4.seconds, 0.15.seconds],
  [0.5.seconds, 2.seconds],
  [0.5.seconds, 2.seconds],
  [0.5.seconds, 2.seconds],
  [0.5.seconds, 2.seconds],
  [1.second, 5.seconds], # probably high traffic, increase timings
  [1.second, 1.minute],
  [0.1.seconds, 0.05.seconds],
  [0.1.seconds, 0.05.seconds],
  [0.2.seconds, 0.05.seconds],
  [0.3.seconds, 0.10.seconds],
  [0.4.seconds, 0.15.seconds],
  [0.5.seconds, 2.seconds],
  [0.5.seconds, 2.seconds],
  [0.5.seconds, 2.seconds],
  [3.seconds, 3.minutes], # probably high traffic or long locks, increase timings
  [0.1.seconds, 0.05.seconds],
  [0.1.seconds, 0.05.seconds],
  [0.5.seconds, 2.seconds],
  [0.5.seconds, 2.seconds],
  [5.seconds, 2.minutes],
  [0.5.seconds, 0.5.seconds],
  [0.5.seconds, 0.5.seconds],
  [7.seconds, 5.minutes],
  [0.5.seconds, 0.5.seconds],
  [0.5.seconds, 0.5.seconds],
  [7.seconds, 5.minutes],
  [0.5.seconds, 0.5.seconds],
  [0.5.seconds, 0.5.seconds],
  [7.seconds, 5.minutes],
  [0.1.seconds, 0.05.seconds],
  [0.1.seconds, 0.05.seconds],
  [0.5.seconds, 2.seconds],
  [10.seconds, 10.minutes],
  [0.1.seconds, 0.05.seconds],
  [0.5.seconds, 2.seconds],
  [10.seconds, 10.minutes]
].freeze

Instance Method Summary collapse

Constructor Details

#initialize(logger: NULL_LOGGER, allow_savepoints: true, timing_configuration: DEFAULT_TIMING_CONFIGURATION, klass: nil, env: ENV, connection:) ⇒ WithLockRetries

Returns a new instance of WithLockRetries.



64
65
66
67
68
69
70
71
72
73
 
# File 'lib/gitlab/database/with_lock_retries.rb', line 64

def initialize(logger: NULL_LOGGER, allow_savepoints: true, timing_configuration: DEFAULT_TIMING_CONFIGURATION, klass: nil, env: ENV, connection:)
  @logger = logger
  @klass = klass
  @allow_savepoints = allow_savepoints
  @timing_configuration = timing_configuration
  @env = env
  @current_iteration = 1
  @log_params = { method: 'with_lock_retries', class: klass.to_s }
  @connection = connection
end

Instance Method Details

#run(raise_on_exhaustion: false, &block) ⇒ Object

Executes a block of code, retrying it whenever a database lock can’t be acquired in time

When a database lock can’t be acquired, ActiveRecord throws ActiveRecord::LockWaitTimeout exception which we intercept to re-execute the block of code, until it finishes or we reach the max attempt limit. The default behavior when max attempts have been reached is to make a final attempt with the lock_timeout disabled, but this can be altered with the raise_on_exhaustion parameter.

Parameters:

  • raise_on_exhaustion (Boolean) (defaults to: false) —

    whether to raise ‘AttemptsExhaustedError` when exhausting max attempts

  • block (Proc) —

    of code that will be executed

See Also:



85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
 
# File 'lib/gitlab/database/with_lock_retries.rb', line 85

def run(raise_on_exhaustion: false, &block)
  raise 'no block given' unless block

  @block = block

  if lock_retries_disabled?
    log(message: 'DISABLE_LOCK_RETRIES environment variable is true, executing the block without retry')

    return run_block
  end

  begin
    run_block_with_lock_timeout
  rescue ActiveRecord::LockWaitTimeout
    if retry_with_lock_timeout?
      disable_idle_in_transaction_timeout if connection.transaction_open?
      wait_until_next_retry
      reset_db_settings

      retry
    else
      reset_db_settings

      raise AttemptsExhaustedError, 'configured attempts to obtain locks are exhausted' if raise_on_exhaustion

      run_block_without_lock_timeout
    end

  ensure
    reset_db_settings
  end
end