Class: Arachni::Support::Database::Queue

Inherits:
Base show all
Defined in:
lib/arachni/support/database/queue.rb

Overview

Flat-file Queue implementation

Behaves pretty much like a Ruby Queue however it transparently serializes and saves its entries to the file-system under the OS's temp directory after a specified #max_buffer_size (for in-memory entries) has been exceeded.

It's pretty useful when you want to reduce memory footprint without having to refactor any code since it behaves just like a Ruby Queue implementation.

Author:

Constant Summary collapse

DEFAULT_MAX_BUFFER_SIZE =

Default #max_buffer_size.

100

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(*args) ⇒ Queue

Returns a new instance of Queue.

See Also:

  • Database::Base#initialize

41
42
43
44
45
46
47
# File 'lib/arachni/support/database/queue.rb', line 41

def initialize( *args )
    super( *args )
    @disk    = []
    @buffer  = []
    @waiting = []
    @mutex   = Mutex.new
end

Instance Attribute Details

#bufferArray<Object> (readonly)

Returns Objects stored in the memory buffer.

Returns:

  • (Array<Object>)

    Objects stored in the memory buffer.


34
35
36
# File 'lib/arachni/support/database/queue.rb', line 34

def buffer
  @buffer
end

#diskArray<String> (readonly)

Returns Paths to files stored to disk.

Returns:


38
39
40
# File 'lib/arachni/support/database/queue.rb', line 38

def disk
  @disk
end

#max_buffer_sizeInteger

Note:

Returns How many entries to keep in memory before starting to off-load to disk.

Returns:

  • (Integer)

    How many entries to keep in memory before starting to off-load to disk.


30
31
32
# File 'lib/arachni/support/database/queue.rb', line 30

def max_buffer_size
  @max_buffer_size
end

Instance Method Details

#<<(obj) ⇒ Object Also known as: push, enq

Parameters:

  • obj (Object)

    Object to add to the queue.


59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
# File 'lib/arachni/support/database/queue.rb', line 59

def <<( obj )
    synchronize do
        if @buffer.size < max_buffer_size
            @buffer << obj
        else
            @disk << dump( obj )
        end

        begin
            t = @waiting.shift
            t.wakeup if t
        rescue ThreadError
            retry
        end
    end
end

#buffer_sizeObject


108
109
110
# File 'lib/arachni/support/database/queue.rb', line 108

def buffer_size
    @buffer.size
end

#clearObject

Removes all objects from the queue.


125
126
127
128
129
130
131
132
133
134
135
# File 'lib/arachni/support/database/queue.rb', line 125

def clear
    synchronize do
        @buffer.clear

        while !@disk.empty?
            path = @disk.pop
            next if !path
            delete_file path
        end
    end
end

#disk_sizeObject


112
113
114
# File 'lib/arachni/support/database/queue.rb', line 112

def disk_size
    @disk.size
end

#empty?Bool

Returns true if the queue if empty, false otherwise.

Returns:

  • (Bool)

    true if the queue if empty, false otherwise.


118
119
120
121
122
# File 'lib/arachni/support/database/queue.rb', line 118

def empty?
    synchronize do
        internal_empty?
    end
end

#free_buffer_sizeObject


104
105
106
# File 'lib/arachni/support/database/queue.rb', line 104

def free_buffer_size
    max_buffer_size - buffer_size
end

#num_waitingObject


137
138
139
# File 'lib/arachni/support/database/queue.rb', line 137

def num_waiting
    @waiting.size
end

#pop(non_block = false) ⇒ Object Also known as: deq, shift

Returns Removes an object from the queue and returns it.

Returns:

  • (Object)

    Removes an object from the queue and returns it.


80
81
82
83
84
85
86
87
88
89
90
91
92
93
# File 'lib/arachni/support/database/queue.rb', line 80

def pop( non_block = false )
    synchronize do
        loop do
            if internal_empty?
                raise ThreadError, 'queue empty' if non_block
                @waiting.push Thread.current
                @mutex.sleep
            else
                return @buffer.shift if !@buffer.empty?
                return load_and_delete_file( @disk.shift )
            end
        end
    end
end

#sizeInteger Also known as: length

Returns Size of the queue, the number of objects it currently holds.

Returns:

  • (Integer)

    Size of the queue, the number of objects it currently holds.


99
100
101
# File 'lib/arachni/support/database/queue.rb', line 99

def size
    buffer_size + disk_size
end