Module: CounterAttribute

Extended by:
ActiveSupport::Concern, AfterCommitQueue
Includes:
Gitlab::ExclusiveLeaseHelpers
Defined in:
app/models/concerns/counter_attribute.rb

Overview

Add capabilities to increment a numeric model attribute efficiently by using Redis and flushing the increments asynchronously to the database after a period of time (10 minutes). When an attribute is incremented by a value, the increment is added to a Redis key. Then, FlushCounterIncrementsWorker will execute `flush_increments_to_database!` which removes increments from Redis for a given model attribute and updates the values in the database.

@example:

class ProjectStatistics
  include CounterAttribute

  counter_attribute :commit_count
  counter_attribute :storage_size
end

To increment the counter we can use the method:

delayed_increment_counter(:commit_count, 3)

Constant Summary collapse

LUA_STEAL_INCREMENT_SCRIPT = 
<<~EOS.freeze
  local increment_key, flushed_key = KEYS[1], KEYS[2]
  local increment_value = redis.call("get", increment_key) or 0
  local flushed_value = redis.call("incrby", flushed_key, increment_value)
  if flushed_value == 0 then
    redis.call("del", increment_key, flushed_key)
  else
    redis.call("del", increment_key)
  end
  return flushed_value
EOS
WORKER_DELAY = 
10.minutes
WORKER_LOCK_TTL = 
10.minutes

Constants included from Gitlab::ExclusiveLeaseHelpers

Gitlab::ExclusiveLeaseHelpers::FailedToObtainLockError

Instance Method Summary collapse

Methods included from AfterCommitQueue

run_after_commit, run_after_commit_or_now

Methods included from Gitlab::ExclusiveLeaseHelpers

#in_lock

Instance Method Details

#counter_flushed_key(attribute) ⇒ Object 


103
104
105
 
# File 'app/models/concerns/counter_attribute.rb', line 103

def counter_flushed_key(attribute)
  counter_key(attribute) + ':flushed'
end

#counter_key(attribute) ⇒ Object 


99
100
101
 
# File 'app/models/concerns/counter_attribute.rb', line 99

def counter_key(attribute)
  "project:{#{project_id}}:counters:#{self.class}:#{id}:#{attribute}"
end

#counter_lock_key(attribute) ⇒ Object 


107
108
109
 
# File 'app/models/concerns/counter_attribute.rb', line 107

def counter_lock_key(attribute)
  counter_key(attribute) + ':lock'
end

#delayed_increment_counter(attribute, increment) ⇒ Object 


81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
 
# File 'app/models/concerns/counter_attribute.rb', line 81

def delayed_increment_counter(attribute, increment)
  return if increment == 0

  run_after_commit_or_now do
    if counter_attribute_enabled?(attribute)
      redis_state do |redis|
        redis.incrby(counter_key(attribute), increment)
      end

      FlushCounterIncrementsWorker.perform_in(WORKER_DELAY, self.class.name, self.id, attribute)
    else
      legacy_increment!(attribute, increment)
    end
  end

  true
end

#flush_increments_to_database!(attribute) ⇒ Object

This method must only be called by FlushCounterIncrementsWorker because it should run asynchronously and with exclusive lease. This will

1. temporarily move the pending increment for a given attribute
   to a relative "flushed" Redis key, delete the increment key and return
   the value. If new increments are performed at this point, the increment
   key is recreated as part of `delayed_increment_counter`.
   The "flushed" key is used to ensure that we can keep incrementing
   counters in Redis while flushing existing values.
2. then the value is used to update the counter in the database.
3. finally the "flushed" key is deleted.


64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
 
# File 'app/models/concerns/counter_attribute.rb', line 64

def flush_increments_to_database!(attribute)
  lock_key = counter_lock_key(attribute)

  with_exclusive_lease(lock_key) do
    increment_key = counter_key(attribute)
    flushed_key = counter_flushed_key(attribute)
    increment_value = steal_increments(increment_key, flushed_key)

    next if increment_value == 0

    transaction do
      unsafe_update_counters(id, attribute => increment_value)
      redis_state { |redis| redis.del(flushed_key) }
    end
  end
end