Class: Gitlab::SidekiqMiddleware::SizeLimiter::Validator

Inherits:
Object
  • Object
show all
Defined in:
lib/gitlab/sidekiq_middleware/size_limiter/validator.rb

Overview

Handle a Sidekiq job payload limit based on current configuration. This validator pulls the configuration from application settings:

  • limiter_mode: the current mode of the size limiter. This must be either ‘track` or `compress`.

  • compression_threshold_bytes: the threshold before the input job payload is compressed.

  • limit_bytes: the size limit in bytes.

In track mode, if a job payload limit exceeds the size limit, an event is sent to Sentry and the job is scheduled like normal.

In compress mode, if a job payload limit exceeds the threshold, it is then compressed. If the compressed payload still exceeds the limit, the job is discarded, and a ExceedLimitError exception is raised.

Constant Summary collapse

EXEMPT_WORKER_NAMES =

Avoid limiting the size of jobs for ‘BackgroundMigrationWorker` classes. We can’t read the configuration from ‘ApplicationSetting` for those jobs when migrating a path that modifies the `application_settings` table. Reading the application settings through `ApplicationSetting#current` causes a `SELECT` with a list of column names, but that list of column names might not match what the table currently looks like causing an error when scheduling background migrations.

The worker classes aren’t constants here, because that would force Application Settings to be loaded earlier causing failures loading the environment in rake tasks

%w[BackgroundMigrationWorker
BackgroundMigration::CiDatabaseWorker
Database::BatchedBackgroundMigrationWorker
Database::BatchedBackgroundMigration::CiDatabaseWorker
RedisMigrationWorker].to_set
JOB_STATUS_KEY = 
'size_limiter'
DEFAULT_SIZE_LIMIT = 
0
DEFAULT_COMPRESSION_THRESHOLD_BYTES =

100kb

100_000
MODES = 
[
  TRACK_MODE = 'track',
  COMPRESS_MODE = 'compress'
].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(worker_class, job) ⇒ Validator

Returns a new instance of Validator.



64
65
66
67
68
69
70
71
72
73
 
# File 'lib/gitlab/sidekiq_middleware/size_limiter/validator.rb', line 64

def initialize(worker_class, job)
  @worker_class = worker_class
  @job = job

  current_settings = Gitlab::CurrentSettings.current_application_settings

  @mode = current_settings.sidekiq_job_limiter_mode
  @compression_threshold = current_settings.sidekiq_job_limiter_compression_threshold_bytes
  @size_limit = current_settings.sidekiq_job_limiter_limit_bytes
end

Instance Attribute Details

#compression_threshold ⇒ Object (readonly)

Returns the value of attribute compression_threshold.



62
63
64
 
# File 'lib/gitlab/sidekiq_middleware/size_limiter/validator.rb', line 62

def compression_threshold
  @compression_threshold
end

#mode ⇒ Object (readonly)

Returns the value of attribute mode.



62
63
64
 
# File 'lib/gitlab/sidekiq_middleware/size_limiter/validator.rb', line 62

def mode
  @mode
end

#size_limit ⇒ Object (readonly)

Returns the value of attribute size_limit.



62
63
64
 
# File 'lib/gitlab/sidekiq_middleware/size_limiter/validator.rb', line 62

def size_limit
  @size_limit
end

Class Method Details

.validate!(worker_class, job) ⇒ Object 



42
43
44
45
46
47
 
# File 'lib/gitlab/sidekiq_middleware/size_limiter/validator.rb', line 42

def validate!(worker_class, job)
  return if EXEMPT_WORKER_NAMES.include?(worker_class.to_s)
  return if validated?(job)

  new(worker_class, job).validate!
end

.validated?(job) ⇒ Boolean

Returns:

  • (Boolean) 


49
50
51
 
# File 'lib/gitlab/sidekiq_middleware/size_limiter/validator.rb', line 49

def validated?(job)
  job.has_key?(JOB_STATUS_KEY)
end

Instance Method Details

#validate! ⇒ Object 



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/gitlab/sidekiq_middleware/size_limiter/validator.rb', line 75

def validate!
  @job[JOB_STATUS_KEY] = 'validated'

  job_args = compress_if_necessary(::Sidekiq.dump_json(@job['args']))

  return if @size_limit == 0
  return if job_args.bytesize <= @size_limit
  return if allow_big_payload?

  exception = exceed_limit_error(job_args)
  if compress_mode?
    @job.delete(JOB_STATUS_KEY)
    raise exception
  else
    @job[JOB_STATUS_KEY] = 'tracked'
    track(exception)
  end
end