Class: SidekiqUniqueJobs::Orphans::RubyReaper
- Defined in:
- lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb
Overview
this is a much slower version of the lua script but does not crash redis
Class DeleteOrphans provides deletion of orphaned digests
Constant Summary collapse
- RUN_SUFFIX =
Returns the suffix for :RUN locks.
":RUN"
Constants inherited from Reaper
SidekiqUniqueJobs::Orphans::Reaper::REAPERS
Instance Attribute Summary collapse
-
#digests ⇒ Object
readonly
Returns the value of attribute digests.
-
#retried ⇒ Object
readonly
Returns the value of attribute retried.
-
#scheduled ⇒ Object
readonly
Returns the value of attribute scheduled.
Attributes inherited from Reaper
Instance Method Summary collapse
-
#active?(digest) ⇒ Boolean
rubocop:disable Metrics/MethodLength, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity.
-
#belongs_to_job?(digest) ⇒ true, false
Checks if the digest has a matching job.
-
#call ⇒ Integer
Delete orphaned digests.
- #considered_active?(time_f) ⇒ Boolean
-
#enqueued?(digest) ⇒ true
Checks if the digest exists in a Sidekiq::Queue.
-
#entries(conn, queue, &block) ⇒ Object
rubocop:disable Metrics/MethodLength.
-
#in_sorted_set?(key, digest) ⇒ true, false
Checks a sorted set for the existance of this digest.
-
#initialize(conn) ⇒ RubyReaper
constructor
Initialize a new instance of DeleteOrphans.
- #match?(key_one, key_two) ⇒ Boolean
-
#orphans ⇒ Array<String>
Find orphaned digests.
-
#queues(conn) { ... } ⇒ void
Loops through all the redis queues and yields them one by one.
-
#retried?(digest) ⇒ true
Checks if the digest exists in the Sidekiq::RetrySet.
-
#scheduled?(digest) ⇒ true
Checks if the digest exists in the Sidekiq::ScheduledSet.
Methods inherited from Reaper
call, #config, #reaper, #reaper_count, #reaper_timeout
Methods included from JSON
dump_json, load_json, safe_load_json
Methods included from Logging
#build_message, included, #log_debug, #log_error, #log_fatal, #log_info, #log_warn, #logger, #logging_context, #with_configured_loggers_context, #with_logging_context
Methods included from Script::Caller
call_script, debug_lua, do_call, extract_args, max_history, now_f, redis_version
Methods included from Connection
Constructor Details
#initialize(conn) ⇒ RubyReaper
Initialize a new instance of DeleteOrphans
34 35 36 37 38 39 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 34 def initialize(conn) super(conn) @digests = SidekiqUniqueJobs::Digests.new @scheduled = Redis::SortedSet.new(SCHEDULE) @retried = Redis::SortedSet.new(RETRY) end |
Instance Attribute Details
#digests ⇒ Object (readonly)
Returns the value of attribute digests.
19 20 21 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 19 def digests @digests end |
#retried ⇒ Object (readonly)
Returns the value of attribute retried.
27 28 29 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 27 def retried @retried end |
#scheduled ⇒ Object (readonly)
Returns the value of attribute scheduled.
23 24 25 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 23 def scheduled @scheduled end |
Instance Method Details
#active?(digest) ⇒ Boolean
rubocop:disable Metrics/MethodLength, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 123 def active?(digest) # rubocop:disable Metrics/MethodLength, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity Sidekiq.redis do |conn| procs = conn.sscan_each("processes").to_a return false if procs.empty? procs.sort.each do |key| valid, workers = conn.pipelined do # TODO: Remove the if statement in the future if conn.respond_to?(:exists?) conn.exists?(key) else conn.exists(key) end conn.hgetall("#{key}:workers") end next unless valid next unless workers.any? workers.each_pair do |_tid, job| next unless (item = safe_load_json(job)) payload = safe_load_json(item[PAYLOAD]) return true if match?(digest, payload[LOCK_DIGEST]) return true if considered_active?(payload[CREATED_AT]) end end false end end |
#belongs_to_job?(digest) ⇒ true, false
Checks if the digest has a matching job.
1. It checks the scheduled set
2. It checks the retry set
3. It goes through all queues
78 79 80 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 78 def belongs_to_job?(digest) scheduled?(digest) || retried?(digest) || enqueued?(digest) || active?(digest) end |
#call ⇒ Integer
Delete orphaned digests
47 48 49 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 47 def call BatchDelete.call(orphans, conn) end |
#considered_active?(time_f) ⇒ Boolean
162 163 164 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 162 def considered_active?(time_f) (Time.now - reaper_timeout).to_f < time_f end |
#enqueued?(digest) ⇒ true
Checks if the digest exists in a Sidekiq::Queue
111 112 113 114 115 116 117 118 119 120 121 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 111 def enqueued?(digest) Sidekiq.redis do |conn| queues(conn) do |queue| entries(conn, queue) do |entry| return true if entry.include?(digest) end end false end end |
#entries(conn, queue, &block) ⇒ Object
rubocop:disable Metrics/MethodLength
179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 179 def entries(conn, queue, &block) # rubocop:disable Metrics/MethodLength queue_key = "queue:#{queue}" initial_size = conn.llen(queue_key) deleted_size = 0 page = 0 page_size = 50 loop do range_start = (page * page_size) - deleted_size range_end = range_start + page_size - 1 entries = conn.lrange(queue_key, range_start, range_end) page += 1 break if entries.empty? entries.each(&block) deleted_size = initial_size - conn.llen(queue_key) end end |
#in_sorted_set?(key, digest) ⇒ true, false
Checks a sorted set for the existance of this digest
210 211 212 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 210 def in_sorted_set?(key, digest) conn.zscan_each(key, match: "*#{digest}*", count: 1).to_a.any? end |
#match?(key_one, key_two) ⇒ Boolean
156 157 158 159 160 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 156 def match?(key_one, key_two) return false if key_one.nil? || key_two.nil? key_one.delete_suffix(RUN_SUFFIX) == key_two.delete_suffix(RUN_SUFFIX) end |
#orphans ⇒ Array<String>
Find orphaned digests
57 58 59 60 61 62 63 64 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 57 def orphans conn.zrevrange(digests.key, 0, -1).each_with_object([]) do |digest, memo| next if belongs_to_job?(digest) memo << digest break if memo.size >= reaper_count end end |
#queues(conn) { ... } ⇒ void
This method returns an undefined value.
Loops through all the redis queues and yields them one by one
175 176 177 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 175 def queues(conn, &block) conn.sscan_each("queues", &block) end |
#retried?(digest) ⇒ true
Checks if the digest exists in the Sidekiq::RetrySet
100 101 102 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 100 def retried?(digest) in_sorted_set?(RETRY, digest) end |
#scheduled?(digest) ⇒ true
Checks if the digest exists in the Sidekiq::ScheduledSet
89 90 91 |
# File 'lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb', line 89 def scheduled?(digest) in_sorted_set?(SCHEDULE, digest) end |