Class: NcsNavigator::Warehouse::Contents

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/ncs_navigator/warehouse/contents.rb

Overview

Provides an Enumerable over all of the content for some or all of the tables in a warehouse instance. Provides streaming results over batches for large content sets.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(config, options = {}) ⇒ Contents

Create a new NcsNavigator::Warehouse::Contents.

Parameters:

  • config (Configuration) —

    the configuration for the warehouse from which to iterate over records.

  • options (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (options):

  • :filters (Array<#call>, #call) —

    a list of filters to use for this transformer

  • :block-size (Fixnum) — default: 5000 —

    the maximum number of records to load into memory before yielding them to the consumer. Reduce this to reduce the memory load of the emitter. Increasing it will probably not improve performance, even if you have sufficient memory to load more records.

  • :tables (Array<#to_s>) — default: all for current MDES version —

    the tables to include in the iteration.



40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
 
# File 'lib/ncs_navigator/warehouse/contents.rb', line 40

def initialize(config, options={})
  @configuration = config
  @record_count = 0
  @block_size = options[:'block-size'] || 5000

  @models =
    if options[:tables]
      options[:tables].collect { |t| t.to_s }.collect { |t|
        config.models_module.mdes_order.find { |model| model.mdes_table_name == t }
      }
    else
      config.models_module.mdes_order
    end

  filter_list = options[:filters]
  @filters = NcsNavigator::Warehouse::Filters::CompositeFilter.new(
    filter_list ? [*filter_list].compact : [])
end

Instance Attribute Details

#block_size ⇒ Numeric (readonly)

Returns the maximum number of records to load into memory before yielding them to the consumer.

Returns:

  • (Numeric) —

    the maximum number of records to load into memory before yielding them to the consumer.



23
24
25
 
# File 'lib/ncs_navigator/warehouse/contents.rb', line 23

def block_size
  @block_size
end

#filters ⇒ CompositeFilter (readonly)

Returns the filters in use on this transformer.

Returns:

  • (CompositeFilter) —

    the filters in use on this transformer.



18
19
20
 
# File 'lib/ncs_navigator/warehouse/contents.rb', line 18

def filters
  @filters
end

#models ⇒ Array<Class> (readonly)

Returns the warehouse models whose records will be enumerated by this instance.

Returns:

  • (Array<Class>) —

    the warehouse models whose records will be enumerated by this instance.



14
15
16
 
# File 'lib/ncs_navigator/warehouse/contents.rb', line 14

def models
  @models
end

Instance Method Details

#each ⇒ Object

Yields each instance of every configured #models in turn.



61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/ncs_navigator/warehouse/contents.rb', line 61

def each
  models.each do |model|
    key = model.key.first.name.to_sym
    count = model.count
    offset = 0
    while offset < count
      model.all(:limit => block_size, :offset => offset, :order => key.asc).each do |instance|
        filters.call([instance]).each do |filtered_record|
          yield filtered_record
        end
      end
      offset += block_size
    end
  end
end