Class: Replicate::Loader
Overview
Load replicants in a streaming fashion.
The Loader reads [type, id, attributes] replicant tuples and creates objects in the current environment.
Objects are expected to arrive in order such that a record referenced via foreign key always precedes the referencing record. The Loader maintains a mapping of primary keys from the dump system to the current environment. This mapping is used to properly establish new foreign key values on all records inserted.
Instance Attribute Summary collapse
-
#stats ⇒ Object
readonly
Stats hash.
Instance Method Summary collapse
-
#constantize(string) ⇒ Object
Turn a string into an object by traversing constants.
-
#feed(type, id, attrs) ⇒ Object
Feed a single replicant tuple into the loader.
-
#initialize ⇒ Loader
constructor
A new instance of Loader.
-
#load(type, id, attributes) ⇒ Object
Load an individual replicant into the underlying datastore.
-
#log_to(out = $stderr, verbose = false, quiet = false) ⇒ Object
Register a filter to write status information to the given stream.
-
#read(io) ⇒ Object
Read multiple [type, id, attrs] replicant tuples from io and call the feed method to load and filter the object.
-
#register_id(object, type, remote_id, local_id) ⇒ Object
Register an id in the keymap.
-
#translate_ids(attributes) ⇒ Object
Translate remote system id references in the attributes hash to their local system id values.
Methods inherited from Emitter
#complete, #emit, #listen, #use
Constructor Details
#initialize ⇒ Loader
Returns a new instance of Loader.
17 18 19 20 21 |
# File 'lib/replicate/loader.rb', line 17 def initialize @keymap = Hash.new { |hash,k| hash[k] = {} } @stats = Hash.new { |hash,k| hash[k] = 0 } super end |
Instance Attribute Details
#stats ⇒ Object (readonly)
Stats hash.
15 16 17 |
# File 'lib/replicate/loader.rb', line 15 def stats @stats end |
Instance Method Details
#constantize(string) ⇒ Object
Turn a string into an object by traversing constants.
127 128 129 130 131 |
# File 'lib/replicate/loader.rb', line 127 def constantize(string) namespace = Object string.split('::').each { |name| namespace = namespace.const_get(name) } namespace end |
#feed(type, id, attrs) ⇒ Object
Feed a single replicant tuple into the loader.
type - The class to create. Must respond to load_replicant. id - The remote system’s id for this object. attrs - Hash of primitively typed objects.
Returns the need object resulting from the load operation.
45 46 47 48 49 50 |
# File 'lib/replicate/loader.rb', line 45 def feed(type, id, attrs) type = type.to_s object = load(type, id, attrs) @stats[type] += 1 emit type, id, attrs, object end |
#load(type, id, attributes) ⇒ Object
Load an individual replicant into the underlying datastore.
type - Model class name as a String. id - Primary key id of the record on the dump system. This must be
translated to the local system and stored in the keymap.
attrs - Hash of attributes to set on the new record.
Returns the new object instance.
72 73 74 75 76 77 78 79 80 81 82 83 |
# File 'lib/replicate/loader.rb', line 72 def load(type, id, attributes) model_class = constantize(type) translate_ids attributes begin new_id, instance = model_class.load_replicant(type, id, attributes) rescue => boom warn "error: loading #{type} #{id} #{boom.class} #{boom}" raise end register_id instance, type, id, new_id instance end |
#log_to(out = $stderr, verbose = false, quiet = false) ⇒ Object
Register a filter to write status information to the given stream. By default, a single line is used to report object counts while the dump is in progress; dump counts for each class are written when complete. The verbose and quiet options can be used to increase or decrease verbository.
out - An IO object to write to, like stderr. verbose - Whether verbose output should be enabled. quiet - Whether quiet output should be enabled.
Returns the Replicate::Status object.
34 35 36 |
# File 'lib/replicate/loader.rb', line 34 def log_to(out=$stderr, verbose=false, quiet=false) use Replicate::Status, 'load', out, verbose, quiet end |
#read(io) ⇒ Object
Read multiple [type, id, attrs] replicant tuples from io and call the feed method to load and filter the object.
54 55 56 57 58 59 60 61 62 |
# File 'lib/replicate/loader.rb', line 54 def read(io) begin while object = Marshal.load(io) type, id, attrs = object feed type, id, attrs end rescue EOFError end end |
#register_id(object, type, remote_id, local_id) ⇒ Object
Register an id in the keymap. Every object loaded must be stored here so that key references can be resolved.
116 117 118 119 120 121 122 123 |
# File 'lib/replicate/loader.rb', line 116 def register_id(object, type, remote_id, local_id) @keymap[type.to_s][remote_id] = local_id c = object.class while !['Object', 'ActiveRecord::Base'].include?(c.name) @keymap[c.name][remote_id] = local_id c = c.superclass end end |
#translate_ids(attributes) ⇒ Object
Translate remote system id references in the attributes hash to their local system id values. The attributes hash may include special id values like this:
{ 'title' => 'hello there',
'repository_id' => [:id, 'Repository', 1234],
'label_ids' => [:id, 'Label', [333, 444, 555, 666, ...]]
... }
These values are translated to local system ids. All object references must be loaded prior to the referencing object.
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
# File 'lib/replicate/loader.rb', line 94 def translate_ids(attributes) attributes.each do |key, value| next unless value.is_a?(Array) && value[0] == :id type, value = value[1].to_s, value[2] local_ids = Array(value).map do |remote_id| if local_id = @keymap[type][remote_id] local_id else warn "error: #{type} #{remote_id} missing from keymap" end end if value.is_a?(Array) attributes[key] = local_ids else attributes[key] = local_ids[0] end end end |