Class: PEROBS::Store
- Inherits:
-
Object
- Object
- PEROBS::Store
- Defined in:
- lib/perobs/Store.rb
Overview
PEROBS::Store is a persistent storage system for Ruby objects. Regular Ruby objects are transparently stored in a back-end storage and retrieved when needed. It features a garbage collector that removes all objects that are no longer in use. A build-in cache keeps access latencies to recently used objects low and lazily flushes modified objects into the persistend back-end. The default back-end is a filesystem based database. Alternatively, an Amazon DynamoDB can be used as well. Adding support for other key/value stores is fairly trivial to do. See PEROBS::DynamoDB for an example
Persistent objects must be defined by deriving your class from PEROBS::Object, PERBOS::Array or PEROBS::Hash. Only instance variables that are declared via po_attr will be persistent. It is recommended that references to other objects are all going to persistent objects again. TO create a new persistent object you must call Store.new(). Don’t use the constructors of persistent classes directly. Store.new() will return a proxy or delegator object that can be used like the actual object. By using delegators we can disconnect the actual object from the delegator handle.
require ‘perobs’
class Person < PEROBS::Object
attr_persist :name, :mother, :father, :kids
# The contructor is only called for the creation of a new object. It is
# not called when the object is restored from the database. In that case
# only restore() is called.
def initialize(cf, name)
super(cf)
self.name = name
self.kids = @store.new(PEROBS::Array)
end
def restore
# In case you need to do any checks or massaging (e. g. for additional
# attributes) you can provide this method.
end
def to_s
"#{@name} is the child of #{self.mother ? self.mother.name : 'unknown'} " +
"and #{self.father ? self.father.name : 'unknown'}.
end
end
store = PEROBS::Store.new(‘family’) store = joe = store.new(Person, ‘Joe’) store = jane = store.new(Person, ‘Jane’) jim = store.new(Person, ‘Jim’) jim.father = joe joe.kids << jim jim.mother = jane jane.kids << jim store.exit
Instance Attribute Summary collapse
-
#cache ⇒ Object
readonly
Returns the value of attribute cache.
-
#class_map ⇒ Object
readonly
Returns the value of attribute class_map.
-
#db ⇒ Object
readonly
Returns the value of attribute db.
Instance Method Summary collapse
-
#[](name) ⇒ Object
Return the object with the provided name.
-
#[]=(name, obj) ⇒ PEROBS::Object
Store the provided object under the given name.
-
#_collect(id, ruby_object_id) ⇒ Object
Remove the object from the in-memory list.
-
#_construct_po(klass, id, *args) ⇒ BasicObject
For library internal use only! This method will create a new PEROBS object.
-
#_new_id ⇒ Integer
Internal method.
-
#_register_in_memory(obj, id) ⇒ Object
Internal method.
-
#check(repair = false) ⇒ Integer
This method can be used to check the database and optionally repair it.
-
#copy(dir, options = {}) ⇒ Object
Copy the store content into a new Store.
-
#delete_store ⇒ Object
Delete the entire store.
-
#each ⇒ Object
Calls the given block once for each object, passing that object as a parameter.
-
#exit ⇒ Object
Close the store and ensure that all in-memory objects are written out to the storage backend.
-
#gc ⇒ Integer
Discard all objects that are not somehow connected to the root objects from the back-end storage.
-
#initialize(data_base, options = {}) ⇒ Store
constructor
Create a new Store.
-
#names ⇒ Array of Symbols
Return a list with all the names of the root objects.
-
#new(klass, *args) ⇒ POXReference
You need to call this method to create new PEROBS objects that belong to this Store.
-
#object_by_id(id) ⇒ Object
Return the object with the provided ID.
-
#rename_classes(rename_map) ⇒ Object
Rename classes of objects stored in the data base.
-
#size ⇒ Integer
Return the number of object stored in the store.
-
#statistics ⇒ Object
This method returns a Hash with some statistics about this store.
-
#sync ⇒ Object
Flush out all modified objects to disk and shrink the in-memory list if needed.
-
#transaction ⇒ Object
This method will execute the provided block as an atomic transaction regarding the manipulation of all objects associated with this Store.
Constructor Details
#initialize(data_base, options = {}) ⇒ Store
Create a new Store.
:progressmeter : reference to a ProgressMeter object that receives
progress information during longer running tasks.
It defaults to ProgressMeter which only logs into
the log. Use ConsoleProgressMeter or a derived
class for more fancy progress reporting.
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 |
# File 'lib/perobs/Store.rb', line 146 def initialize(data_base, = {}) # Create a backing store handler @progressmeter = ([:progressmeter] ||= ProgressMeter.new) @db = ([:engine] || FlatFileDB).new(data_base, ) @db.open # Create a map that can translate classes to numerical IDs and vice # versa. @class_map = ClassMap.new(@db) # List of PEROBS objects that are currently available as Ruby objects # hashed by their ID. @in_memory_objects = {} # This objects keeps some counters of interest. @stats = Statistics.new # The Cache reduces read and write latencies by keeping a subset of the # objects in memory. @cache = Cache.new([:cache_bits] || 16) # The named (global) objects IDs hashed by their name unless (@root_objects = object_by_id(0)) PEROBS.log.debug "Initializing the PEROBS store" # The root object hash always has the object ID 0. @root_objects = _construct_po(Hash, 0) # Mark the root_objects object as modified. @cache.cache_write(@root_objects) end unless @root_objects.is_a?(Hash) PEROBS.log.fatal "Database corrupted: Root objects must be a Hash " + "but is a #{@root_objects.class}" end end |
Instance Attribute Details
#cache ⇒ Object (readonly)
Returns the value of attribute cache.
111 112 113 |
# File 'lib/perobs/Store.rb', line 111 def cache @cache end |
#class_map ⇒ Object (readonly)
Returns the value of attribute class_map.
111 112 113 |
# File 'lib/perobs/Store.rb', line 111 def class_map @class_map end |
#db ⇒ Object (readonly)
Returns the value of attribute db.
111 112 113 |
# File 'lib/perobs/Store.rb', line 111 def db @db end |
Instance Method Details
#[](name) ⇒ Object
Return the object with the provided name.
298 299 300 301 302 303 |
# File 'lib/perobs/Store.rb', line 298 def [](name) # Return nil if there is no object with that name. return nil unless (id = @root_objects[name]) POXReference.new(self, id) end |
#[]=(name, obj) ⇒ PEROBS::Object
Store the provided object under the given name. Use this to make the object a root or top-level object (think global variable). Each store should have at least one root object. Objects that are not directly or indirectly reachable via any of the root objects are no longer accessible and will be garbage collected.
270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 |
# File 'lib/perobs/Store.rb', line 270 def []=(name, obj) # If the passed object is nil, we delete the entry if it exists. if obj.nil? @root_objects.delete(name) return nil end # We only allow derivatives of PEROBS::Object to be stored in the # store. unless obj.is_a?(ObjectBase) PEROBS.log.fatal 'Object must be of class PEROBS::Object but ' + "is of class #{obj.class}" end unless obj.store == self PEROBS.log.fatal 'The object does not belong to this store.' end # Store the name and mark the name list as modified. @root_objects[name] = obj._id obj end |
#_collect(id, ruby_object_id) ⇒ Object
Remove the object from the in-memory list. This is an internal method and should never be called from user code. It will be called from a finalizer, so many restrictions apply!
520 521 522 523 524 |
# File 'lib/perobs/Store.rb', line 520 def _collect(id, ruby_object_id) if @in_memory_objects[id] == ruby_object_id @in_memory_objects.delete(id) end end |
#_construct_po(klass, id, *args) ⇒ BasicObject
For library internal use only! This method will create a new PEROBS object.
249 250 251 |
# File 'lib/perobs/Store.rb', line 249 def _construct_po(klass, id, *args) klass.new(Handle.new(self, id), *args) end |
#_new_id ⇒ Integer
Internal method. Don’t use this outside of this library! Generate a new unique ID that is not used by any other object. It uses random numbers between 0 and 2**64 - 1.
494 495 496 497 498 499 500 501 502 503 |
# File 'lib/perobs/Store.rb', line 494 def _new_id begin # Generate a random number. It's recommended to not store more than # 2**62 objects in the same store. id = rand(2**64) # Ensure that we don't have already another object with this ID. end while @in_memory_objects.include?(id) || @db.include?(id) id end |
#_register_in_memory(obj, id) ⇒ Object
Internal method. Don’t use this outside of this library! Add the new object to the in-memory list. We only store a weak reference to the object so it can be garbage collected. When this happens the object finalizer is triggered and calls _forget() to remove the object from this hash again.
512 513 514 |
# File 'lib/perobs/Store.rb', line 512 def _register_in_memory(obj, id) @in_memory_objects[id] = obj.object_id end |
#check(repair = false) ⇒ Integer
This method can be used to check the database and optionally repair it. The repair is a pure structural repair. It cannot ensure that the stored data is still correct. E. g. if a reference to a non-existing or unreadable object is found, the reference will simply be deleted.
393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 |
# File 'lib/perobs/Store.rb', line 393 def check(repair = false) stats = { :errors => 0, :object_cnt => 0 } # All objects must have in-db version. sync # Run basic consistency checks first. stats[:errors] += @db.check_db(repair) # We will use the mark to mark all objects that we have checked already. # Before we start, we need to clear all marks. @db.clear_marks @progressmeter.start("Checking object link structure", @db.item_counter) do @root_objects.each do |name, id| check_object(id, repair, stats) end end # Delete all broken root objects. if repair @root_objects.delete_if do |name, id| unless @db.check(id, repair) PEROBS.log.error "Discarding broken root object '#{name}' " + "with ID #{id}" stats[:errors] += 1 end end end if stats[:errors] > 0 if repair PEROBS.log.error "#{stats[:errors]} errors found in " + "#{stats[:object_cnt]} objects" else PEROBS.log.fatal "#{stats[:errors]} errors found in " + "#{stats[:object_cnt]} objects" end else PEROBS.log.debug "No errors found" end # Ensure that any fixes are written into the DB. sync if repair stats[:errors] end |
#copy(dir, options = {}) ⇒ Object
Copy the store content into a new Store. The arguments are identical to Store.new().
183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 |
# File 'lib/perobs/Store.rb', line 183 def copy(dir, = {}) # Make sure all objects are persisted. sync # Create a new store with the specified directory and options. new_db = Store.new(dir, ) # Clear the cache. new_db.sync # Copy all objects of the existing store to the new store. i = 0 each do |ref_obj| obj = ref_obj._referenced_object obj._transfer(new_db) obj._sync i += 1 end PEROBS.log.debug "Copied #{i} objects into new database at #{dir}" # Flush the new store and close it. new_db.exit true end |
#delete_store ⇒ Object
Delete the entire store. The store is no longer usable after this method was called. This is an alternative to exit() that additionaly deletes the entire database.
256 257 258 259 260 |
# File 'lib/perobs/Store.rb', line 256 def delete_store @db.delete_database @db = @class_map = @in_memory_objects = @stats = @cache = @root_objects = nil end |
#each ⇒ Object
Calls the given block once for each object, passing that object as a parameter.
460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 |
# File 'lib/perobs/Store.rb', line 460 def each @db.clear_marks # Start with the object 0 and the indexes of the root objects. Push them # onto the work stack. stack = [ 0 ] + @root_objects.values while !stack.empty? # Get an object index from the stack. id = stack.pop next if @db.is_marked?(id) unless (obj = object_by_id(id)) PEROBS.log.fatal "Database is corrupted. Object with ID #{id} " + "not found." end # Mark the object so it will never be pushed to the stack again. @db.mark(id) yield(obj.myself) if block_given? # Push the IDs of all unmarked referenced objects onto the stack obj._referenced_object_ids.each do |r_id| stack << r_id unless @db.is_marked?(r_id) end end end |
#exit ⇒ Object
Close the store and ensure that all in-memory objects are written out to the storage backend. The Store object is no longer usable after this method was called.
210 211 212 213 214 215 216 217 218 219 220 221 |
# File 'lib/perobs/Store.rb', line 210 def exit if @cache && @cache.in_transaction? @cache.abort_transaction @cache.flush @db.close if @db PEROBS.log.fatal "You cannot call exit() during a transaction: #{Kernel.caller}" end @cache.flush if @cache @db.close if @db @db = @class_map = @in_memory_objects = @stats = @cache = @root_objects = nil end |
#gc ⇒ Integer
Discard all objects that are not somehow connected to the root objects from the back-end storage. The garbage collector is not invoked automatically. Depending on your usage pattern, you need to call this method periodically.
338 339 340 341 342 |
# File 'lib/perobs/Store.rb', line 338 def gc sync mark sweep end |
#names ⇒ Array of Symbols
Return a list with all the names of the root objects.
307 308 309 |
# File 'lib/perobs/Store.rb', line 307 def names @root_objects.keys end |
#new(klass, *args) ⇒ POXReference
You need to call this method to create new PEROBS objects that belong to this Store.
231 232 233 234 235 236 237 238 239 240 241 |
# File 'lib/perobs/Store.rb', line 231 def new(klass, *args) unless klass.is_a?(BasicObject) PEROBS.log.fatal "#{klass} is not a BasicObject derivative" end obj = _construct_po(klass, _new_id, *args) # Mark the new object as modified so it gets pushed into the database. @cache.cache_write(obj) # Return a POXReference proxy for the newly created object. obj.myself end |
#object_by_id(id) ⇒ Object
Return the object with the provided ID. This method is not part of the public API and should never be called by outside users. It’s purely intended for internal use.
347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 |
# File 'lib/perobs/Store.rb', line 347 def object_by_id(id) if (ruby_object_id = @in_memory_objects[id]) # We have the object in memory so we can just return it. begin object = ObjectSpace._id2ref(ruby_object_id) # Let's make sure the object is really the object we are looking # for. The GC might have recycled it already and the Ruby object ID # could now be used for another object. if object.is_a?(ObjectBase) && object._id == id return object end rescue RangeError => e # Due to a race condition the object can still be in the # @in_memory_objects list but has been collected already by the Ruby # GC. In that case we need to load it again. In this case the # _collect() call will happen much later, potentially after we have # registered a new object with the same ID. @in_memory_objects.delete(id) end end if (obj = @cache.object_by_id(id)) PEROBS.log.fatal "Object #{id} with Ruby #{obj.object_id} is in cache but not in_memory" end # We don't have the object in memory. Let's find it in the storage. if @db.include?(id) # Great, object found. Read it into memory and return it. obj = ObjectBase::read(self, id) # Add the object to the in-memory storage list. @cache.cache_read(obj) return obj end # The requested object does not exist. Return nil. nil end |
#rename_classes(rename_map) ⇒ Object
Rename classes of objects stored in the data base.
486 487 488 |
# File 'lib/perobs/Store.rb', line 486 def rename_classes(rename_map) @class_map.rename(rename_map) end |
#size ⇒ Integer
Return the number of object stored in the store. CAVEAT: This method will only return correct values when it is separated from any mutating call by a call to sync().
327 328 329 330 331 |
# File 'lib/perobs/Store.rb', line 327 def size # We don't include the Hash that stores the root objects into the object # count. @db.item_counter - 1 end |
#statistics ⇒ Object
This method returns a Hash with some statistics about this store.
527 528 529 530 531 532 |
# File 'lib/perobs/Store.rb', line 527 def statistics @stats.in_memory_objects = @in_memory_objects.length @stats.root_objects = @root_objects.length @stats end |
#sync ⇒ Object
Flush out all modified objects to disk and shrink the in-memory list if needed.
313 314 315 316 317 318 319 320 321 |
# File 'lib/perobs/Store.rb', line 313 def sync if @cache.in_transaction? @cache.abort_transaction @cache.flush PEROBS.log.fatal "You cannot call sync() during a transaction: \n" + Kernel.caller.join("\n") end @cache.flush end |
#transaction ⇒ Object
This method will execute the provided block as an atomic transaction regarding the manipulation of all objects associated with this Store. In case the execution of the block generates an exception, the transaction is aborted and all PEROBS objects are restored to the state at the beginning of the transaction. The exception is passed on to the enclosing scope, so you probably want to handle it accordingly.
447 448 449 450 451 452 453 454 455 456 |
# File 'lib/perobs/Store.rb', line 447 def transaction @cache.begin_transaction begin yield if block_given? rescue => e @cache.abort_transaction raise e end @cache.end_transaction end |