Class: UkParliament::QueueManager

Inherits:
Object
  • Object
show all
Includes:
UkParliament
Defined in:
lib/uk_parliament/queue_manager.rb

Overview

Class to create/manage a queue for a set of items that will be scraped.

Constant Summary collapse

QUEUE_MAIN =

Unique identifier for the main work queue.

'main'
QUEUE_ERROR =

Unique identifier for the error queue.

'error'

Constants included from UkParliament

DATA_SOURCE_FILE, DATA_SOURCE_HTTP, VERSION

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from UkParliament

#configuration, configuration, log, #log

Constructor Details

#initialize(name = 'commons') ⇒ QueueManager

Set up queue states.



17
18
19
20
21
22
23
24
25
26
27
 
# File 'lib/uk_parliament/queue_manager.rb', line 17

def initialize(name = 'commons')
  config = configuration
  main_queue_file_name = File.join(config[:queue_file_path], "#{name}.queue")
  error_queue_file_name = File.join(config[:queue_file_path], "#{name}.error.queue")

  @main_queue = FileQueue.new(main_queue_file_name)
  @error_queue = FileQueue.new(error_queue_file_name)

  reset_main_queue
  set_active_queue
end

Instance Attribute Details

#active_queue ⇒ Object (readonly)

Instance data accessor(s).



14
15
16
 
# File 'lib/uk_parliament/queue_manager.rb', line 14

def active_queue
  @active_queue
end

#error_queue ⇒ Object (readonly)

Instance data accessor(s).



14
15
16
 
# File 'lib/uk_parliament/queue_manager.rb', line 14

def error_queue
  @error_queue
end

#main_queue ⇒ Object (readonly)

Instance data accessor(s).



14
15
16
 
# File 'lib/uk_parliament/queue_manager.rb', line 14

def main_queue
  @main_queue
end

Instance Method Details

#enqueue(members) ⇒ Object

Set up the queue, either with provided items, or from the error queue.



53
54
55
56
57
58
59
60
61
 
# File 'lib/uk_parliament/queue_manager.rb', line 53

def enqueue(members)
  if @active_queue == QUEUE_ERROR
    populate_from_error_queue
  else
    populate(members, 'id')
  end

  log.info("Populated queue with #{@main_queue.length} items...")
end

#error_queue_size ⇒ Object

Return the current size of the error queue.

This is a bit of a work around FileQueue. github.com/pezra/filequeue/pull/4



42
43
44
45
46
47
48
49
50
 
# File 'lib/uk_parliament/queue_manager.rb', line 42

def error_queue_size
  size = 0

  if File.exists?(@error_queue.file_name)
    size = @error_queue.length
  end

  size
end

#scrape_errors? ⇒ Boolean

Identify if there were errors from the last scrape.

Returns:

  • (Boolean) 


30
31
32
33
34
35
36
 
# File 'lib/uk_parliament/queue_manager.rb', line 30

def scrape_errors?
  if @active_queue == QUEUE_ERROR
    true
  else
    false
  end
end