Class: Jinx::Migrator

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/jinx/migration/migrator.rb

Overview

Migrates a CSV extract to a caBIG application.

Instance Method Summary collapse

Constructor Details

#initialize(opts) ⇒ Migrator

Creates a new Migrator from the given options.

Parameters:

  • opts ({Symbol => Object}) —

    the migration options

Options Hash (opts):

  • :target (Class) —

    the required target domain class

  • :mapping (<String>, String) —

    the required input field => caTissue attribute mapping file(s)

  • :input (String, Migration::Reader) —

    the required input file name or an adapter which implements the Jinx::Migration::Reader methods

  • :defaults (<String>, String) —

    the optional caTissue attribute => value default mapping file(s)

  • :filters (<String>, String) —

    the optional caTissue attribute input value => caTissue value filter file(s)

  • :shims (<String>, String) —

    the optional shim file(s) to load

  • :unique (String) —

    the optional flag which ensures that migrator calls the uniquify method on those migrated objects whose class includes the Unique module

  • :create (String) —

    the optional flag indicating that existing target objects are ignored

  • :bad (String) —

    the optional invalid record file

  • :extract (String, IO) —

    the optional extract file or object that responds to <<

  • :extract_headers (<String>) —

    the optional extract CSV field headers

  • :from (Integer) —

    the optional starting source record number to process

  • :to (Integer) —

    the optional ending source record number to process

  • :quiet (Boolean) —

    the optional flag which suppress output messages

  • :verbose (Boolean) —

    the optional flag to print the migration progress



42
43
44
45
46
47
 
# File 'lib/jinx/migration/migrator.rb', line 42

def initialize(opts)
  @rec_cnt = 0
  @mgt_mths = {}
  parse_options(opts)
  build
end

Instance Method Details

#add_defaults(obj, row, created) ⇒ Object (private)

Parameters:

  • the (Resource) —

    migration object

  • created (<Resource>) —

    (see #create)



672
673
674
675
676
677
678
679
680
 
# File 'lib/jinx/migration/migrator.rb', line 672

def add_defaults(obj, row, created)
  dh = @def_hash[obj.class] || return
  dh.each do |path, value|
    # fill the reference path
    ref = fill_path(obj, path[0...-1], row, created)
    # set the attribute to the default value unless there is already a value
    ref.merge_attribute(path.last.to_sym, value)
  end
end

#add_owners(hash) { ... } ⇒ Object (private)

Adds missing owner classes to the migration class path hash (with empty paths) for the classes in the given hash.

Parameters:

  • hash ({Class => Object}) —

    the class map

Yields:

  • the map entry for a new owner



242
243
244
 
# File 'lib/jinx/migration/migrator.rb', line 242

def add_owners(hash, &factory)
  hash.keys.each { |klass| add_owners_for(klass, hash, &factory) }
end

#add_owners_for(klass, hash) { ... } ⇒ Object (private)

Adds missing owner classes to the migration class path hash (with empty paths) for the given migration class.

Parameters:

  • klass (Class) —

    the migration class

  • hash ({Class => Object}) —

    the class map

Yields:

  • the map entry for a new owner



252
253
254
255
256
257
258
 
# File 'lib/jinx/migration/migrator.rb', line 252

def add_owners_for(klass, hash, &factory)
  owner = missing_owner_for(klass, hash) || return
  logger.debug { "The migrator is adding #{klass.qp} owner #{owner}..." }
  @owners << owner
  hash[owner] = yield
  add_owners_for(owner, hash, &factory)
end

#attribute_filter_hash(klass) ⇒ Property => Proc (private)

Builds the property => filter hash. The filter is specified in the --filter migration option. A Boolean property has a default String => Boolean filter which converts the input string to a Boolean as specified in the Jinx::Boolean to_boolean methods.

Parameters:

  • klass (Class) —

    the migration class

Returns:

  • (Property => Proc) —

    the filter migration methods



380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
 
# File 'lib/jinx/migration/migrator.rb', line 380

def attribute_filter_hash(klass)
  hash = @flt_hash[klass]
  fh = {}
  klass.each_property do |prop|
    pa = prop.attribute
    spec = hash[pa] if hash
    # If the property is boolean, then make a filter that operates on the parsed string input.
    if prop.type == Java::JavaLang::Boolean then
      fh[pa] = boolean_filter(spec)
      logger.debug { "The migrator added the default text -> boolean filter for #{klass.qp} #{pa}." }
    elsif spec then
      fh[pa] = Migration::Filter.new(spec)
    end
  end
  unless fh.empty? then
    logger.debug { "The migration filters were loaded for #{klass.qp} #{fh.keys.to_series}." }
  end
  fh
end

#attribute_method_hash(klass) ⇒ Object (private) 



356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
 
# File 'lib/jinx/migration/migrator.rb', line 356

def attribute_method_hash(klass)
  # the migrate methods, excluding the Migratable migrate_references method
  mths = klass.instance_methods(true).select { |mth| mth =~ /^migrate.(?!references)/ }
  # the attribute => migration method hash
  mh = {}
  mths.each do |mth|
    # the attribute suffix, e.g. name for migrate_name or Name for migrateName
    suffix = /^migrate(_)?(.*)/.match(mth).captures[1]
    # the attribute name
    attr_nm = suffix[0, 1].downcase + suffix[1..-1]
    # the attribute for the name, or skip if no such attribute
    pa = klass.standard_attribute(attr_nm) rescue next
    # associate the attribute => method
    mh[pa] = mth
  end
  mh
end

#boolean_filter(spec = nil) ⇒ Migration::Filter (private)

Returns the boolean property migration filter.

Parameters:

  • the (String, nil) —

    value filter, if any

Returns:



402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
 
# File 'lib/jinx/migration/migrator.rb', line 402

def boolean_filter(spec=nil)
  # break up the spec into two specs, one on strings and one on booleans
  bspec, sspec = spec.split { |k, v| Boolean === k } if spec
  bf = Migration::Filter.new(bspec) if bspec and not bspec.empty?
  sf = Migration::Filter.new(sspec) if sspec and not sspec.empty?
  # make the composite filter 
  Migration::Filter.new do |value|
    fv = sf.transform(value) if sf
    if fv.nil? then
      bv = Jinx::Boolean.for(value) rescue nil
      fv = bf.nil? || bv.nil? ? bv : bf.transform(bv)
    end
    fv
  end 
end

#build ⇒ Object (private)

Raises:



169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
 
# File 'lib/jinx/migration/migrator.rb', line 169

def build
  # the current source class => instance map
  raise MigrationError.new("No file to migrate") if @input.nil?

  # If the input is a file name, then make a CSV loader which only converts input fields
  # corresponding to non-String attributes.
  if String === @input then
    @reader = CsvIO.new(@input, &method(:convert))
    logger.debug { "Migration data input file #{@input} headers: #{@reader.headers.qp}" }
  else
    @reader = @input
  end
  
  # add shim modifiers
  load_shims(@shims)

  # create the class => path => default value hash
  @def_hash = @def_files ? load_defaults_files(@def_files) : {}
  # create the class => path => default value hash
  @flt_hash = @flt_files ? load_filter_files(@flt_files) : {}
  # the missing owner classes
  @owners = Set.new
  # create the class => path => header hash
  fld_map = load_field_map_files(@fld_map_files)
  # create the class => paths hash
  @cls_paths_hash = create_class_paths_hash(fld_map, @def_hash)
  # create the path => class => header hash
  @header_map = create_header_map(fld_map)
  # Order the creatable classes by dependency, owners first, to smooth the migration process.
  @creatable_classes = @cls_paths_hash.keys.sort do |klass, other|
    other.depends_on?(klass) ? -1 : (klass.depends_on?(other) ? 1 : 0)
  end
  # An abstract class cannot be instantiated.
  @creatable_classes.each do |klass|
    if klass.abstract? then
      raise MigrationError.new("Migrator cannot create the abstract class #{klass}; specify a subclass instead in the mapping file.")
    end
  end
  
  logger.info { "Migration creatable classes: #{@creatable_classes.qp}." }
  unless @def_hash.empty? then logger.info { "Migration defaults: #{@def_hash.qp}." } end
  
  # the class => attribute migration methods hash
  create_migration_method_hashes
  
  # Print the input field => attribute map and collect the String input fields for
  # the custom CSVLoader converter.
  @nonstring_headers = Set.new
  logger.info("Migration attributes:")
  @header_map.each do |path, cls_hdr_hash|
    prop = path.last
    cls_hdr_hash.each do |klass, hdr|
      type_s = prop.type ? prop.type.qp : 'Object'
      logger.info("  #{hdr} => #{klass.qp}.#{path.join('.')} (#{type_s})")
    end
    @nonstring_headers.merge!(cls_hdr_hash.values) if prop.type != Java::JavaLang::String
  end
end

#class_for_name(name) ⇒ Class (private)

Returns the corresponding class.

Parameters:

  • the (String) —

    class name to resolve in the context of this migrator

Returns:

  • (Class) —

    the corresponding class

Raises:

  • (NameError) —

    if the name cannot be resolved



946
947
948
 
# File 'lib/jinx/migration/migrator.rb', line 946

def class_for_name(name)
  context_module.module_for_name(name)
end

#clear(target) ⇒ Object (private)

Clears references to objects allocated for migration of a single row into the given target. This method does nothing. Subclasses can override.

This method is overridden by subclasses to clear the migration state to conserve memory, since this migrator should consume O(1) rather than O(n) memory for n migration records.



528
529
 
# File 'lib/jinx/migration/migrator.rb', line 528

def clear(target)
end

#context_module ⇒ Module (private)

The context module is given by the target class ResourceClass#domain_module.

Returns:

  • (Module) —

    the class name resolution context



939
940
941
 
# File 'lib/jinx/migration/migrator.rb', line 939

def context_module
  @target_class.domain_module
end

#convert(value, info) ⇒ Object (private)

Converts the given input field value as follows:

  • If the info header is a String field, then return the value unchanged.

  • Otherwise, return nil which will delegate to the generic CsvIO converter.

Parameters:

  • f —

    the input field value to convert

  • info —

    the CSV field info



233
234
235
 
# File 'lib/jinx/migration/migrator.rb', line 233

def convert(value, info)
  value unless @nonstring_headers.include?(info.header)
end

#create_attribute_path(path_s) ⇒ <Property> (private)

Returns the corresponding attribute metadata path.

Parameters:

  • path_s (String) —

    a period-delimited path string path_s in the form class(.attribute)+

Returns:

  • (<Property>) —

    the corresponding attribute metadata path

Raises:

  • (MigrationError) —

    if the path string is malformed or an attribute is not found



903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
 
# File 'lib/jinx/migration/migrator.rb', line 903

def create_attribute_path(path_s)
  names = path_s.split('.')
  # If the path starts with a capitalized class name, then resolve the class.
  # Otherwise, the target class is the start of the path.
  klass = names.first =~ /^[A-Z]/ ? class_for_name(names.shift) : @target_class
  # There must be at least one attribute.
  if names.empty? then
    raise MigrationError.new("Property entry in migration configuration is not in <class>.<attribute> format: #{path_s}")
  end
  
  # Build the attribute path.
  path = []
  names.inject(klass) do |parent, name|
    pa = name.to_sym
    prop = begin
      parent.property(pa)
    rescue NameError
      raise MigrationError.new("Migration field mapping attribute #{parent}.#{pa} not found - " + $!)
    end
    if prop.collection? then
      raise MigrationError.new("Migration field mapping attribute #{parent}.#{prop} is a collection, which is not supported")
    end
    path << prop
    prop.type
  end
  
  # Return the starting class and Property path.
  # Note that the starting class is not necessarily the first path attribute declarer, since the
  # starting class could be the concrete target class rather than an abstract declarer. this is
  # important, since the class must be instantiated.
  [klass, path]
end

#create_class_paths_hash(fld_map, def_map) ⇒ Object (private)

Returns a new class => [paths] hash from the migration fields configuration map.

Returns:

  • a new class => [paths] hash from the migration fields configuration map



951
952
953
954
955
956
 
# File 'lib/jinx/migration/migrator.rb', line 951

def create_class_paths_hash(fld_map, def_map)
  hash = {}
  fld_map.each { |klass, path_hdr_hash| hash[klass] = path_hdr_hash.keys.to_set }
  def_map.each { |klass, path_val_hash| (hash[klass] ||= Set.new).merge(path_val_hash.keys) }
  hash
end

#create_header_map(fld_map) ⇒ Object (private)

Returns a new path => class => header hash from the migration fields configuration map.

Returns:

  • a new path => class => header hash from the migration fields configuration map



959
960
961
962
963
964
965
 
# File 'lib/jinx/migration/migrator.rb', line 959

def create_header_map(fld_map)
  hash = LazyHash.new { Hash.new }
  fld_map.each do |klass, path_hdr_hash|
    path_hdr_hash.each { |path, hdr| hash[path][klass] = hdr }
  end
  hash
end

#create_instance(klass, row, created) ⇒ Resource (private)

Creates an instance of the given klass from the given row. The new klass instance and all intermediate migrated instances are added to the created set.

Parameters:

  • klass (Class)
  • row ({Symbol => Object}) —

    the input row

  • created (<Resource>) —

    the migrated instances for this row

Returns:

  • (Resource) —

    the new instance



637
638
639
640
641
642
643
644
645
 
# File 'lib/jinx/migration/migrator.rb', line 637

def create_instance(klass, row, created)
  # the new object
  logger.debug { "The migrator is building #{klass.qp}..." }
  created << obj = klass.new
  migrate_properties(obj, row, created)
  add_defaults(obj, row, created)
  logger.debug { "The migrator built #{obj}." }
  obj
end

#create_migration_method_hashes ⇒ Object (private)

Creates the class => migrate__<attribute>_ hash for the given klasses.



276
277
278
279
280
281
282
283
284
285
286
287
 
# File 'lib/jinx/migration/migrator.rb', line 276

def create_migration_method_hashes
  # the class => attribute => migration filter hash
  @attr_flt_hash = {}
  customizable_class_attributes.each do |klass, pas|
    flts = migration_filters(klass) || next
    @attr_flt_hash[klass] = flts
  end
  # print the migration shim methods
  unless @mgt_mths.empty? then
    logger.info("Migration shim methods:\n#{@mgt_mths.qp}")
  end
end

#create_reference(obj, property, row, created) ⇒ Object (private)

Sets the given migrated object’s reference attribute to a new referenced domain object.

Parameters:

  • obj (Resource) —

    the domain object being migrated

  • property (Property) —

    the property being migrated

Returns:

  • the new object



702
703
704
705
706
707
708
709
710
711
712
 
# File 'lib/jinx/migration/migrator.rb', line 702

def create_reference(obj, property, row, created)
  if property.type.abstract? then
    raise MigrationError.new("Cannot create #{obj.qp} #{property} with abstract type #{property.type}")
  end
  ref = property.type.new
  ref.migrate(row, Array::EMPTY_ARRAY)
  obj.send(property.writer, ref)
  created << ref
  logger.debug { "The migrator created #{obj.qp} #{property} #{ref}." }
  ref
end

#current_record ⇒ Object (private) 



757
758
759
 
# File 'lib/jinx/migration/migrator.rb', line 757

def current_record
  @rec_cnt + 1
end

#customizable_class_attributes ⇒ Object (private)

Returns the class => attributes hash for terminal path attributes which can be customized by migrate_ methods.

Returns:

  • the class => attributes hash for terminal path attributes which can be customized by migrate_ methods



290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
 
# File 'lib/jinx/migration/migrator.rb', line 290

def customizable_class_attributes
  # The customizable classes set, starting with creatable classes and adding in
  # the migration path terminal attribute declarer classes below.
  klasses = @creatable_classes.to_set
  # the class => path terminal attributes hash
  cls_attrs_hash = LazyHash.new { Set.new }
  # add each path terminal attribute and its declarer class
  @cls_paths_hash.each_value do |paths|
    paths.each do |path|
      prop = path.last
      type = prop.declarer
      klasses << type
      cls_attrs_hash[type] << prop
    end
  end
  
  # Merge each redundant customizable superclass into its concrete customizable subclasses. 
  klasses.dup.each do |cls|
    redundant = false
    klasses.each do |other|
      # cls is redundant if it is a superclass of other
      redundant = other < cls
      if redundant then
        cls_attrs_hash[other].merge!(cls_attrs_hash[cls])
      end
    end
    # remove the redundant class
    if redundant then
      cls_attrs_hash.delete(cls)
      klasses.delete(cls)
    end
  end
  
  cls_attrs_hash
end

#each {|target| ... } ⇒ Object

Yields:

  • (target) —

    iterate on each migration target

Yield Parameters:

  • the (Jinx::Resource) —

    migration target



98
99
100
 
# File 'lib/jinx/migration/migrator.rb', line 98

def each
  migrate { |tgt, row| yield tgt }
end

#fill_path(obj, path, row, created) ⇒ Object (private)

Fills the given reference Property path starting at obj.

Returns:

  • the last domain object in the path



687
688
689
690
691
692
693
 
# File 'lib/jinx/migration/migrator.rb', line 687

def fill_path(obj, path, row, created)
  # create the intermediate objects as needed (or return obj if path is empty)
  path.inject(obj) do |parent, prop|
    # the referenced object
    parent.send(prop.reader) or create_reference(parent, prop, row, created)
  end
end

#filter_for(obj, attribute) ⇒ Object (private) 



752
753
754
755
 
# File 'lib/jinx/migration/migrator.rb', line 752

def filter_for(obj, attribute)
  flts = @attr_flt_hash[obj.class] || return
  flts[attribute]
end

#filter_value(obj, property, value, row) ⇒ Object (private)

Calls the shim migrate_<attribute> method or config filter on the input value.

Parameters:

  • value —

    the input value

  • property (Property) —

    the property to set

Returns:

  • the input value, if there is no filter, otherwise the filtered value



742
743
744
745
746
747
748
749
750
 
# File 'lib/jinx/migration/migrator.rb', line 742

def filter_value(obj, property, value, row)
  flt = filter_for(obj, property.to_sym)
  return value if flt.nil?
  fval = flt.call(obj, value, row)
  unless value == fval then
    logger.debug { "The migration filter transformed the #{obj.qp} #{property} value from #{value.qp} to #{fval}." }
  end
  fval
end

#load_defaults_file(file, hash) ⇒ Object (private)

Loads the defaults config file into the given hash.

Parameters:

  • file (String) —

    the file to load

  • hash (<Class => <String => Object>>) —

    the class => path => default value entries



849
850
851
852
853
854
855
856
857
858
859
860
861
 
# File 'lib/jinx/migration/migrator.rb', line 849

def load_defaults_file(file, hash)
  begin
    config = YAML::load_file(file)
  rescue
    raise MigrationError.new("Could not read defaults file #{file}: " + $!)
  end
  # collect the class => path => value entries
  config.each do |path_s, value|
    next if value.nil_or_empty?
    klass, path = create_attribute_path(path_s)
    hash[klass][path] = value
  end
end

#load_defaults_files(files) ⇒ <Class => <String => Object>> (private)

Loads the defaults configuration files.

Parameters:

  • files (<String>, String) —

    the file or file array to load

Returns:

  • (<Class => <String => Object>>) —

    the class => path => default value entries



838
839
840
841
842
843
 
# File 'lib/jinx/migration/migrator.rb', line 838

def load_defaults_files(files)
  # collect the class => path => value entries from each defaults file
  hash = LazyHash.new { Hash.new }
  files.enumerate { |file| load_defaults_file(file, hash) }
  hash
end

#load_field_map_file(file, hash) ⇒ Object (private)

Parameters:

  • file (String) —

    the migration fields configuration file

  • hash ({Class => {Property => Symbol}}) —

    the class => path => header hash to populate from the loaded configuration



805
806
807
808
809
810
811
812
813
 
# File 'lib/jinx/migration/migrator.rb', line 805

def load_field_map_file(file, hash)
  # load the field mapping config file
  begin
    config = YAML.load_file(file)
  rescue
    raise MigrationError.new("Could not read field map file #{file}: " + $!)
  end
  populate_field_map(config, hash)
end

#load_field_map_files(files) ⇒ {Class => {Property => Symbol}} (private)

Returns the class => path => header hash loaded from the mapping files.

Parameters:

  • files (<String>, String) —

    the migration fields mapping file or file array

Returns:

  • ({Class => {Property => Symbol}}) —

    the class => path => header hash loaded from the mapping files



764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
 
# File 'lib/jinx/migration/migrator.rb', line 764

def load_field_map_files(files)
  map = LazyHash.new { Hash.new }
  files.enumerate { |file| load_field_map_file(file, map) }

  # include the target class
  map[@target_class] ||= Hash.new
  # add the default classes
  @def_hash.each_key { |klass| map[klass] ||= Hash.new }
  # add the owners
  add_owners(map) { Hash.new }

  # Include only concrete classes that are not a superclass of another migration class.
  classes = map.keys
  sub_hash = classes.to_compact_hash do |klass|
    subs = classes.select { |other| other < klass }
    subs.delete_if { |klass| subs.any? { |other| other < klass } }
  end
  
  # Merge the superclass paths into the subclass paths.
  sub_hash.each do |klass, subs|
    paths = map.delete(klass)
    # Add, but don't replace, path => header entries from the superclass.
    subs.each do |sub|
      map[sub].merge!(paths) { |key, old, new| old }
      logger.debug { "Migrator merged #{klass.qp} mappings into the subclass #{sub.qp}." }
    end
  end
  
  # Validate that there are no abstract classes in the mapping.
  map.each_key do |klass|
    if klass.abstract? then
      raise MigrationError.new("Cannot migrate to the abstract class #{klass}")
    end
  end

  map
end

#load_filter_file(file, hash) ⇒ Object (private)

Loads the filter config file into the given hash.

Parameters:

  • file (String) —

    the file to load

  • hash (<Class => <String => <Object => Object>>>) —

    the class => path => input value => caTissue value entries



878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
 
# File 'lib/jinx/migration/migrator.rb', line 878

def load_filter_file(file, hash)
  # collect the class => attribute => filter entries
  logger.debug { "Loading the migration filter configuration #{file}..." }
  begin
    config = YAML::load_file(file)
  rescue
    raise MigrationError.new("Could not read filter file #{file}: " + $!)
  end
  config.each do |path_s, flt|
    next if flt.nil_or_empty?
    klass, path = create_attribute_path(path_s)
    if path.empty? then
      raise MigrationError.new("Migration filter configuration path does not include a property: #{path_s}")
    elsif path.size > 1 then
      raise MigrationError.new("Migration filter configuration path with more than one property is not supported: #{path_s}")
    end
    pa = klass.standard_attribute(path.first.to_sym)
    flt_hash = hash[klass] ||= {}
    flt_hash[pa] = flt
  end
end

#load_filter_files(files) ⇒ <Class => <String => Object>> (private)

Loads the filter config files.

Parameters:

  • files (<String>, String) —

    the file or file array to load

Returns:

  • (<Class => <String => Object>>) —

    the class => path => default value entries



866
867
868
869
870
871
872
 
# File 'lib/jinx/migration/migrator.rb', line 866

def load_filter_files(files)
  # collect the class => path => value entries from each defaults file
  hash = {}
  files.enumerate { |file| load_filter_file(file, hash) }
  logger.debug { "The migrator loaded the filters #{hash.qp}." }
  hash
end

#load_shims(files) ⇒ Object (private)

Loads the shim files.

Parameters:

  • files (<String>, String) —

    the file or file array



421
422
423
424
425
426
427
 
# File 'lib/jinx/migration/migrator.rb', line 421

def load_shims(files)
  logger.debug { "Loading the migration shims with load path #{$:.pp_s}..." }
  files.enumerate do |file|
    load file
    logger.info { "The migrator loaded the shim file #{file}." }
  end
end

#migrate {|target, row| ... } ⇒ Object

Imports this migrator’s CSV file and calls the given block on each migrated target domain object. If no block is given, then this method returns an array of the migrated target objects.

Yields:

  • (target, row) —

    operates on the migration target

Yield Parameters:

  • target (Resource) —

    the migrated target domain object

  • row ({Symbol => Object}) —

    the migration source record



56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
 
# File 'lib/jinx/migration/migrator.rb', line 56

def migrate(&block)
  unless block_given? then
    return migrate { |tgt, row| tgt }
  end
  # If there is an extract, then wrap the migration in an extract
  # writer block.
  if @extract then
    if String === @extract then
      logger.debug { "Opening migration extract #{@extract}..." }
      FileUtils::mkdir_p(File.dirname(@extract))
      if @extract_hdrs then
        logger.debug { "Migration extract headers: #{@extract_hdrs.join(', ')}." }
        CsvIO.open(@extract, :mode => 'w', :headers => @extract_hdrs) do |io|
          @extract = io
          return migrate(&block)
        end
      else
        File.open(@extract, 'w') do |io|
          @extract = io
          return migrate(&block)
        end
      end
    end
    # Copy the extract into a local variable and clear the extract i.v.
    # prior to a recursive call with an extract writer block.
    io, @extract = @extract, nil
    return migrate do |tgt, row|
      res = yield(tgt, row)
      tgt.extract(io)
      res
    end
  end
  begin
    migrate_rows(&block)
  ensure
    @rejects.close if @rejects
    remove_migration_methods
  end
end

#migrate_properties(obj, row, created) ⇒ Object (private)

Migrates each input field to the associated domain object attribute. String input values are stripped. Missing input values are ignored.

Parameters:

  • the (Resource) —

    migration object

  • created (<Resource>) —

    (see #create)



653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
 
# File 'lib/jinx/migration/migrator.rb', line 653

def migrate_properties(obj, row, created)
  # for each input header which maps to a migratable target attribute metadata path,
  # set the target attribute, creating intermediate objects as needed.
  @cls_paths_hash[obj.class].each do |path|
    header = @header_map[path][obj.class]
    # the input value
    value = row[header]
    value.strip! if String === value
    next if value.nil?
    # fill the reference path
    ref = fill_path(obj, path[0...-1], row, created)
    # set the attribute
    migrate_property(ref, path.last, value, row)
  end
end

#migrate_property(obj, property, value, row) ⇒ Object (private)

Sets the given property value to the filtered input value. If there is a filter defined for the property, then that filter is applied. If there is a migration shim method with name migrate_attribute, then that method is called on the (possibly filtered) value. The target object property is set to the resulting filtered value.

Parameters:

  • obj (Migratable) —

    the target domain object

  • property (Property) —

    the property to set

  • value —

    the input value

  • row ({Symbol => Object}) —

    the input row



724
725
726
727
728
729
730
731
732
733
734
735
 
# File 'lib/jinx/migration/migrator.rb', line 724

def migrate_property(obj, property, value, row)
  # if there is a shim migrate_<attribute> method, then call it on the input value
  value = filter_value(obj, property, value, row)
  return if value.nil?
  # set the attribute
  begin
    obj.send(property.writer, value)
  rescue Exception
    raise MigrationError.new("Could not set #{obj.qp} #{property} to #{value.qp} - " + $!)
  end
  logger.debug { "Migrated #{obj.qp} #{property} to #{value}." }
end

#migrate_row(row) ⇒ Object (private)

Imports the given CSV row into a target object.

Parameters:

  • row ({Symbol => Object}) —

    the input row field => value hash

Returns:

  • the migrated target object if the migration is valid, nil otherwise



535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
 
# File 'lib/jinx/migration/migrator.rb', line 535

def migrate_row(row)
  # create an instance for each creatable class
  created = Set.new
  # the migrated objects
  migrated = @creatable_classes.map { |klass| create_instance(klass, row, created) }
  # migrate each object from the input row
  migrated.each do |obj|
    # First uniquify the object if necessary.
    if @unique and Unique === obj then
      logger.debug { "The migrator is making #{obj} unique..." }
      obj.uniquify
    end
    obj.migrate(row, migrated)
  end
  # the valid migrated objects
  @migrated = migrate_valid_references(row, migrated)
  # the candidate target objects
  tgts = @migrated.select { |obj| @target_class === obj }
  if tgts.size > 1 then
    raise MigrationError.new("Ambiguous #{@target_class} targets #{tgts.to_series}")
  end
  target = tgts.first || return
  
  logger.debug { "Migrated target #{target}." }
  target
end

#migrate_rows {|target, row| ... } ⇒ Object (private)

Migrates all rows in the input.

Yields:

  • (target, row) —

    operates on the migration target

Yield Parameters:

  • target (Resource) —

    the migrated target domain object

  • row ({Symbol => Object}) —

    the migration source record



433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
 
# File 'lib/jinx/migration/migrator.rb', line 433

def migrate_rows
  # open an CSV output for rejects if the bad option is set
  if @bad_file then
    @rejects = open_rejects(@bad_file)
    logger.info("Unmigrated records will be written to #{File.expand_path(@bad_file)}.")
  end
  
  @rec_cnt = mgt_cnt = 0
  logger.info { "Migrating #{@input}..." }
  puts "Migrating #{@input}..." if @verbose
  @reader.each do |row|
    # the one-based current record number
    rec_no = @rec_cnt + 1
    # skip if the row precedes the from option
    if rec_no == @from and @rec_cnt > 0 then
      logger.info("Skipped the initial #{@rec_cnt} records.")
    elsif rec_no == @to then
      logger.info("Ending the migration after processing record #{@rec_cnt}.")
      return
    elsif rec_no < @from then
      @rec_cnt += 1
      next
    end
    begin
      # migrate the row
      logger.debug { "Migrating record #{rec_no}..." }
      tgt = migrate_row(row)
      # call the block on the migrated target
      if tgt then
        logger.debug { "The migrator built #{tgt} with the following content:\n#{tgt.dump}" }
        yield(tgt, row)
      end
    rescue Exception => e
      logger.error("Migration error on record #{rec_no} - #{e.message}:\n#{e.backtrace.pp_s}")
      # If there is a reject file, then don't propagate the error.
      raise unless @rejects
      # try to clear the migration state
      clear(tgt) rescue nil
      # clear the target
      tgt = nil
    end
    if tgt then
      # replace the log message below with the commented alternative to detect a memory leak
      logger.info { "Migrated record #{rec_no}." }
      #memory_usage = `ps -o rss= -p #{Process.pid}`.to_f / 1024 # in megabytes
      #logger.debug { "Migrated rec #{@rec_cnt}; memory usage: #{sprintf("%.1f", memory_usage)} MB." }
      mgt_cnt += 1
      if @verbose then print_progress(mgt_cnt) end
      # clear the migration state
      clear(tgt)
    elsif @rejects then
      # If there is a rejects file then warn, write the reject and continue.
      logger.warn("Migration not performed on record #{rec_no}.")
      @rejects << row
      @rejects.flush
      logger.debug("Invalid record #{rec_no} was written to the rejects file #{@bad_file}.")
    else
      raise MigrationError.new("Migration not performed on record #{rec_no}")
    end
    # Bump the record count.
    @rec_cnt += 1
  end
  logger.info("Migrated #{mgt_cnt} of #{@rec_cnt} records.")
  if @verbose then
    puts
    puts "Migrated #{mgt_cnt} of #{@rec_cnt} records."
  end
end

#migrate_valid_references(row, migrated) ⇒ Array (private)

Sets the migration references for each valid migrated object.

Parameters:

  • migrated (Array) —

    the migrated objects

Returns:

  • (Array) —

    the valid migrated objects



567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
 
# File 'lib/jinx/migration/migrator.rb', line 567

def migrate_valid_references(row, migrated)
  # Split the valid and invalid objects. The iteration is in reverse dependency order,
  # since invalidating a dependent can invalidate the owner.
  ordered = migrated.transitive_closure(:dependents)
  ordered.keep_if { |obj| migrated.include?(obj) }.reverse!
  valid, invalid = ordered.partition do |obj|
    if migration_valid?(obj) then
      obj.migrate_references(row, migrated, @target_class, @attr_flt_hash[obj.class])
      true
    else
      obj.class.owner_attributes.each { |pa| obj.clear_attribute(pa) }
      false
    end
  end
  
  # Go back through the valid objects in dependency order to invalidate dependents
  # whose owner is invalid.
  valid.reverse.each do |obj|
    unless owner_valid?(obj, valid, invalid) then
      invalid << valid.delete(obj)
      logger.debug { "The migrator invalidated #{obj} since it does not have a valid owner." }
    end
  end
  
  # Go back through the valid objects in reverse dependency order to invalidate owners
  # created only to hold a dependent which was subsequently invalidated.
  valid.reject do |obj|
    if @owners.include?(obj.class) and obj.dependents.all? { |dep| invalid.include?(dep) } then
      # clear all references from the invalidated owner
      obj.class.domain_attributes.each { |pa| obj.clear_attribute(pa) }
      invalid << obj
      logger.debug { "The migrator invalidated #{obj.qp} since it was created solely to hold subsequently invalidated dependents." }
      true
    end
  end
end

#migration_filters(klass) ⇒ Object (private)

Discovers methods of the form migrate_attribute implemented for the paths in the given class => paths hash the given klass. The migrate method is called on the input field value corresponding to the path.



329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
 
# File 'lib/jinx/migration/migrator.rb', line 329

def migration_filters(klass)
  # the attribute => migration method hash
  mh = attribute_method_hash(klass)
  @mgt_mths[klass] = mh unless mh.empty?
  fh = attribute_filter_hash(klass)
  return if mh.empty? and fh.empty?
  # For each class path terminal attribute metadata, add the migration filters
  # to the attribute metadata => proc hash.
  klass.attributes.to_compact_hash do |pa|
    # the filter
    flt = fh[pa]
    # the migration shim method
    mth = mh[pa]
    # the filter proc
    Proc.new do |obj, value, row|
      # filter the value
      value = flt.transform(value) if flt and not value.nil?
      # apply the migrate_<attribute> method, if defined
      if mth then
        obj.send(mth, value, row) unless value.nil?
      else
        value
      end
    end
  end
end

#migration_valid?(obj) ⇒ Boolean (private)

Returns whether the migration is successful.

Parameters:

Returns:

  • (Boolean) —

    whether the migration is successful



620
621
622
623
624
625
626
627
 
# File 'lib/jinx/migration/migrator.rb', line 620

def migration_valid?(obj)
  if obj.migration_valid? then
    true
  else
    logger.debug { "The migrated #{obj.qp} is invalid." }
    false
  end
end

#missing_owner_for(klass, hash) ⇒ Class? (private)

Returns the missing class owner, if any.

Parameters:

  • klass (Class) —

    the migration class

  • hash ({Class => Object}) —

    the class map

Returns:

  • (Class, nil) —

    the missing class owner, if any



263
264
265
266
267
268
269
270
271
272
273
 
# File 'lib/jinx/migration/migrator.rb', line 263

def missing_owner_for(klass, hash)
  # check for an owner among the current migration classes
  return if klass.owners.any? do |owner|
    hash.detect_key { |other| other <= owner }
  end
  # Find the first non-abstract candidate owner that is a dependent
  # of a migrated class.
  klass.owners.detect do |owner|
    not owner.abstract? and hash.detect_key { |other| owner.depends_on?(other, true) }
  end
end

#open_rejects(file) ⇒ IO (private)

Makes the rejects CSV output file.

Parameters:

  • file (String) —

    the output file

Returns:

  • (IO) —

    the reject stream



506
507
508
509
510
511
 
# File 'lib/jinx/migration/migrator.rb', line 506

def open_rejects(file)
  # Make the parent directory.
  FileUtils.mkdir_p(File.dirname(file))
  # Open the file.
  FasterCSV.open(file, 'w', :headers => true, :header_converters => :symbol, :write_headers => true)
end

#owner_valid?(obj, valid, invalid) ⇒ Boolean (private)

Returns whether the given domain object satisfies at least one of the following conditions:

  • it does not have an owner among the invalid objects

  • it has an owner among the valid objects

Parameters:

  • obj (Resource) —

    the domain object to check

  • valid (<Resource>) —

    the valid migrated objects

  • invalid (<Resource>) —

    the invalid migrated objects

Returns:

  • (Boolean) —

    whether the owner is valid



612
613
614
615
616
 
# File 'lib/jinx/migration/migrator.rb', line 612

def owner_valid?(obj, valid, invalid)
  otypes = obj.class.owners
  invalid.all? { |other| not otypes.include?(other.class) } or
    valid.any? { |other| otypes.include?(other.class) }
end

#parse_options(opts) ⇒ Object (private) 



133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
 
# File 'lib/jinx/migration/migrator.rb', line 133

def parse_options(opts)
  @fld_map_files = opts[:mapping]
  if @fld_map_files.nil? then
    raise MigrationError.new("Migrator missing required field mapping file parameter")
  end
  @def_files = opts[:defaults]
  @flt_files = opts[:filters]
  shims_opt = opts[:shims] ||= []
  # Make a single shims file into an array.
  @shims = shims_opt.collection? ? shims_opt : [shims_opt]
  @unique = opts[:unique]
  @from = opts[:from] ||= 1
  @input = opts[:input]
  if @input.nil? then
    raise MigrationError.new("Migrator missing required source file parameter")
  end
  @target_class = opts[:target]
  if @target_class.nil? then
    raise MigrationError.new("Migrator missing required target class parameter")
  end
  @bad_file = opts[:bad]
  @extract = opts[:extract]
  @extract_hdrs = opts[:extract_headers]
  @create = opts[:create]
  logger.info("Migration options: #{printable_options(opts).pp_s}.")
  # flag indicating whether to print a progress monitor
  @verbose = opts[:verbose]
end

#populate_field_map(config, hash) ⇒ Object (private)

Parameters:

  • config ({String => String}) —

    the attribute => header specification

  • hash ({Class => {Property => Symbol}}) —

    the class => path => header hash to populate from the loaded configuration



817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
 
# File 'lib/jinx/migration/migrator.rb', line 817

def populate_field_map(config, hash)
  # collect the class => path => header entries
  config.each do |field, attr_list|
    next if attr_list.blank?
    # the header accessor method for the field
    header = @reader.accessor(field)
    if header.nil? then
      raise MigrationError.new("Field defined in migration configuration not found in input file #{@input} headers: #{field}")
    end
    # associate each attribute path in the property value with the header
    attr_list.split(/,\s*/).each do |path_s|
      klass, path = create_attribute_path(path_s)
      hash[klass][path] = header
    end
  end
end

#print_progress(count) ⇒ Object (private)

Prints a ‘+’ progress indicator after each migrated record to stdout.

Parameters:

  • count (Integer) —

    the migrated record count



516
517
518
519
520
521
 
# File 'lib/jinx/migration/migrator.rb', line 516

def print_progress(count)
  # If the line is 72 characters, then print a line break 
  puts if count % 72 == 0
  # Print the progress indicator
  print "+"
end

#printable_options(opts) ⇒ Object (private) 



162
163
164
165
166
167
 
# File 'lib/jinx/migration/migrator.rb', line 162

def printable_options(opts)
  popts = opts.reject { |option, value| value.nil_or_empty? }
  # The target class should be a simple class name rather than the class metadata.
  popts[:target] = popts[:target].qp if popts.has_key?(:target)
  popts
end

#remove_extract_method(klass) ⇒ Object (private) 



125
126
127
128
129
130
131
 
# File 'lib/jinx/migration/migrator.rb', line 125

def remove_extract_method(klass)
  if (klass.method_defined?(:extract)) then
    klass.module_eval { remove_method(:extract) }
    sc = klass.superclass
    remove_extract_method(sc) if sc < Migratable
  end
end

#remove_migration_methods ⇒ Object (private)

Cleans up after the migration by removing the methods injected by migration shims.



106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
 
# File 'lib/jinx/migration/migrator.rb', line 106

def remove_migration_methods
  # remove the migrate_<attribute> methods
  @mgt_mths.each do | klass, hash|
    hash.each_value do |sym|
      while klass.method_defined?(sym)
        klass.instance_method(sym).owner.module_eval { remove_method(sym) }
      end
    end
  end
  # remove the migrate method
  @creatable_classes.each do |klass|
    while (k = klass.instance_method(:migrate).owner) < Migratable
      k.module_eval { remove_method(:migrate) }
    end
  end
  # remove the target extract method
  remove_extract_method(@target) if @extract
end