Class: RWGet::Dupes

Inherits:
Object
  • Object
show all
Defined in:
lib/rwget/dupes.rb

Constant Summary collapse

SIZE = 
1_000_000

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ Dupes

Returns a new instance of Dupes.



8
9
10
 
# File 'lib/rwget/dupes.rb', line 8

def initialize(options = {})
  @bloom = BloomFilter.new(SIZE, 4, 1)
end

Instance Method Details

#dupe?(uri) ⇒ Boolean

Returns:

  • (Boolean) 


12
13
14
15
16
17
 
# File 'lib/rwget/dupes.rb', line 12

def dupe?(uri)
  key = uri.to_s
  return true if @bloom.include?(key)
  @bloom.insert(key)
  return false
end