Class: RocketJob::DirmonEntry

Inherits:
Object
  • Object
show all
Includes:
Plugins::Document, Plugins::StateMachine
Defined in:
lib/rocket_job/dirmon_entry.rb

Constant Summary collapse

@@default_archive_directory = 
'_archive'.freeze
@@whitelist_paths = 
Concurrent::Array.new

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.add_whitelist_path(path) ⇒ Object

Add a path to the whitelist Raises: Errno::ENOENT: No such file or directory



155
156
157
158
159
160
161
 
# File 'lib/rocket_job/dirmon_entry.rb', line 155

def self.add_whitelist_path(path)
  # Confirms that path exists
  path = Pathname.new(path).realpath.to_s
  @@whitelist_paths << path
  @@whitelist_paths.uniq!
  path
end

.counts_by_state ⇒ Object

Returns [Hash<String:Integer>] of the number of dirmon entries in each state. Note: If there are no workers in that particular state then the hash will not have a value for it.

Example dirmon entries in every state:

RocketJob::DirmonEntry.counts_by_state
# => {
       :pending => 1,
       :enabled => 37,
       :failed => 1,
       :disabled => 3
     }

Example no dirmon entries:

RocketJob::Job.counts_by_state
# => {}



188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
 
# File 'lib/rocket_job/dirmon_entry.rb', line 188

def self.counts_by_state
  counts = {}
  collection.aggregate([
    {
      '$group' => {
        _id:   '$state',
        count: {'$sum' => 1}
      }
    }
  ]
  ).each do |result|
    counts[result['_id'].to_sym] = result['count']
  end
  counts
end

.delete_whitelist_path(path) ⇒ Object

Deletes a path from the whitelist paths Raises: Errno::ENOENT: No such file or directory



165
166
167
168
169
170
171
 
# File 'lib/rocket_job/dirmon_entry.rb', line 165

def self.delete_whitelist_path(path)
  # Confirms that path exists
  path = Pathname.new(path).realpath.to_s
  @@whitelist_paths.delete(path)
  @@whitelist_paths.uniq!
  path
end

.whitelist_paths ⇒ Object

Security Settings

A whitelist of paths from which to process files. This prevents accidental or malicious ‘pattern`s from processing files from anywhere in the system that the user under which Dirmon is running can access.

All resolved ‘pattern`s must start with one of the whitelisted path, otherwise they will be rejected

Note:

  • If no whitelist paths have been added, then a whitelist check is not performed

  • Relative paths can be used, but are not considered safe since they can be manipulated

  • These paths should be assigned in an initializer and not editable via the Web UI to ensure that they are not tampered with

Default: [] ==> Do not enforce whitelists

Returns [Array<String>] a copy of the whitelisted paths



149
150
151
 
# File 'lib/rocket_job/dirmon_entry.rb', line 149

def self.whitelist_paths
  @@whitelist_paths.dup
end

Instance Method Details

#archive_pathname(file_pathname) ⇒ Object

Returns [Pathname] the archive_directory if set, otherwise the default_archive_directory Creates the archive directory if one is set



212
213
214
215
216
217
218
219
220
221
222
223
224
 
# File 'lib/rocket_job/dirmon_entry.rb', line 212

def archive_pathname(file_pathname)
  if archive_directory
    path = Pathname.new(archive_directory)
    begin
      path.mkpath unless path.exist?
    rescue Errno::ENOENT => exc
      raise(Errno::ENOENT, "DirmonJob failed to create archive directory: #{path}, #{exc.message}")
    end
    path.realpath
  else
    file_pathname.dirname.join(self.class.default_archive_directory).realdirpath
  end
end

#each(&block) ⇒ Object

Passes each filename [Pathname] found that matches the pattern into the supplied block



227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
 
# File 'lib/rocket_job/dirmon_entry.rb', line 227

def each(&block)
  SemanticLogger.named_tagged(dirmon_entry: id.to_s) do
    # Case insensitive filename matching
    Pathname.glob(pattern, File::FNM_CASEFOLD).each do |pathname|
      next if pathname.directory?
      pathname  = pathname.realpath
      file_name = pathname.to_s

      # Skip archive directories
      next if file_name.include?(self.class.default_archive_directory)

      # Security check?
      if (whitelist_paths.size > 0) && whitelist_paths.none? { |whitepath| file_name.to_s.start_with?(whitepath) }
        logger.error "Skipping file: #{file_name} since it is not in any of the whitelisted paths: #{whitelist_paths.join(', ')}"
        next
      end

      # File must be writable so it can be removed after processing
      unless pathname.writable?
        logger.error "Skipping file: #{file_name} since it is not writable by the current user. Must be able to delete/move the file after queueing the job"
        next
      end
      block.call(pathname)
    end
  end
end

#job_class ⇒ Object

Returns the Job to be queued



272
273
274
275
276
277
 
# File 'lib/rocket_job/dirmon_entry.rb', line 272

def job_class
  return if job_class_name.nil?
  job_class_name.constantize
rescue NameError
  nil
end

#later(pathname) ⇒ Object

Queues the job for the supplied pathname



280
281
282
283
284
285
286
287
288
289
290
291
 
# File 'lib/rocket_job/dirmon_entry.rb', line 280

def later(pathname)
  if klass = job_class
    logger.measure_info "Enqueued: #{name}, Job class: #{job_class_name}" do
      job = klass.new(properties)
      upload_file(job, pathname)
      job.save!
      job
    end
  else
    raise(ArgumentError, "Cannot instantiate a class for: #{job_class_name.inspect}")
  end
end

#set_exception(worker_name, exc_or_message) ⇒ Object

Set exception information for this DirmonEntry and fail it



255
256
257
258
259
260
261
262
263
264
265
266
267
 
# File 'lib/rocket_job/dirmon_entry.rb', line 255

def set_exception(worker_name, exc_or_message)
  if exc_or_message.is_a?(Exception)
    self.exception        = JobException.from_exception(exc_or_message)
    exception.worker_name = worker_name
  else
    build_exception(
      class_name:  'RocketJob::DirmonEntryException',
      message:     exc_or_message,
      backtrace:   [],
      worker_name: worker_name
    )
  end
end