Module: Gitlab::SidekiqStatus

Defined in:
lib/gitlab/sidekiq_status.rb,
lib/gitlab/sidekiq_status/client_middleware.rb,
lib/gitlab/sidekiq_status/server_middleware.rb

Overview

The SidekiqStatus module and its child classes can be used for checking if a Sidekiq job has been processed or not.

To check if a job has been completed, simply pass the job ID to the completed? method:

job_id = SomeWorker.with_status.perform_async(...)

if Gitlab::SidekiqStatus.completed?(job_id)
  ...
end

If you do not use with_status, and the worker class does not declare status_expiration in its sidekiq_options, then this status will not be stored.

For each job ID registered a separate key is stored in Redis, making lookups much faster than using Sidekiq's built-in job finding/status API. These keys expire after a certain period of time to prevent storing too many keys in Redis.

Defined Under Namespace

Classes: ClientMiddleware, ServerMiddleware

Constant Summary collapse

STATUS_KEY = 
'gitlab-sidekiq-status:%s'
DEFAULT_EXPIRATION =

The default time (in seconds) after which a status key is expired automatically. The default of 30 minutes should be more than sufficient for most jobs.

30.minutes.to_i

Class Method Summary collapse

Class Method Details

.all_completed?(job_ids) ⇒ Boolean

Returns true if all the given job have been completed.

job_ids - The Sidekiq job IDs to check.

Returns true or false.

Returns:

  • (Boolean) 


68
69
70
 
# File 'lib/gitlab/sidekiq_status.rb', line 68

def self.all_completed?(job_ids)
  self.num_running(job_ids) == 0
end

.completed_jids(job_ids) ⇒ Object

Returns the JIDs that are completed

job_ids - The Sidekiq job IDs to check.

Returns an array of completed JIDs



125
126
127
128
129
130
131
132
133
134
 
# File 'lib/gitlab/sidekiq_status.rb', line 125

def self.completed_jids(job_ids)
  statuses = job_status(job_ids)

  completed = []
  job_ids.zip(statuses).each do |job_id, status|
    completed << job_id unless status
  end

  completed
end

.expire(jid, expire = DEFAULT_EXPIRATION) ⇒ Object

Refreshes the timeout on the key if it exists

jid = The Sidekiq job ID expire - The expiration time of the Redis key.



57
58
59
60
61
 
# File 'lib/gitlab/sidekiq_status.rb', line 57

def self.expire(jid, expire = DEFAULT_EXPIRATION)
  with_redis do |redis|
    redis.expire(key_for(jid), expire.to_i)
  end
end

.job_status(job_ids) ⇒ Object

Returns the job status for each of the given job IDs.

job_ids - The Sidekiq job IDs to check.

Returns an array of true or false indicating job completion. true = job is still running or enqueued false = job completed



102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
 
# File 'lib/gitlab/sidekiq_status.rb', line 102

def self.job_status(job_ids)
  return [] if job_ids.empty?

  keys = job_ids.map { |jid| key_for(jid) }

  status = with_redis do |redis|
    Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
      if Gitlab::Redis::ClusterUtil.cluster?(redis)
        Gitlab::Redis::ClusterUtil.batch_get(keys, redis)
      else
        redis.mget(*keys)
      end
    end
  end

  status.map { |result| !result.nil? }
end

.key_for(jid) ⇒ Object 



136
137
138
 
# File 'lib/gitlab/sidekiq_status.rb', line 136

def self.key_for(jid)
  STATUS_KEY % jid
end

.num_completed(job_ids) ⇒ Object

Returns the number of jobs that have completed.

job_ids - The Sidekiq job IDs to check.



91
92
93
 
# File 'lib/gitlab/sidekiq_status.rb', line 91

def self.num_completed(job_ids)
  job_ids.size - self.num_running(job_ids)
end

.num_running(job_ids) ⇒ Object

Returns the number of jobs that are running or enqueued.

job_ids - The Sidekiq job IDs to check.



82
83
84
85
86
 
# File 'lib/gitlab/sidekiq_status.rb', line 82

def self.num_running(job_ids)
  responses = self.job_status(job_ids)

  responses.count(&:present?)
end

.running?(job_id) ⇒ Boolean

Returns true if the given job is running or enqueued.

job_id - The Sidekiq job ID to check.

Returns:

  • (Boolean) 


75
76
77
 
# File 'lib/gitlab/sidekiq_status.rb', line 75

def self.running?(job_id)
  num_running([job_id]) > 0
end

.set(jid, expire = DEFAULT_EXPIRATION) ⇒ Object

Starts tracking of the given job.

jid - The Sidekiq job ID expire - The expiration time of the Redis key.



36
37
38
39
40
41
42
 
# File 'lib/gitlab/sidekiq_status.rb', line 36

def self.set(jid, expire = DEFAULT_EXPIRATION)
  return unless expire

  with_redis do |redis|
    redis.set(key_for(jid), 1, ex: expire.to_i)
  end
end

.unset(jid) ⇒ Object

Stops the tracking of the given job.

jid - The Sidekiq job ID to remove.



47
48
49
50
51
 
# File 'lib/gitlab/sidekiq_status.rb', line 47

def self.unset(jid)
  with_redis do |redis|
    redis.del(key_for(jid))
  end
end