Class: ETL::Control::Source

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/etl/control/source.rb

Overview

ETL source. Subclasses must implement the each method.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(control, configuration, definition) ⇒ Source

Initialize the Source instance

  • control: The control object

  • configuration: The configuration hash

  • definition: The source layout definition

Configuration options:

  • :store_locally: Set to false to not store source data locally (defaults to true)



38
39
40
41
42
43
44
45
 
# File 'lib/etl/control/source.rb', line 38

def initialize(control, configuration, definition)
  @control = control
  @configuration = configuration
  @definition = definition
  
  @store_locally = true
  @store_locally = configuration[:store_locally] unless configuration[:store_locally].nil?
end

Instance Attribute Details

#configuration ⇒ Object

The configuration Hash



11
12
13
 
# File 'lib/etl/control/source.rb', line 11

def configuration
  @configuration
end

#control ⇒ Object

The control object



8
9
10
 
# File 'lib/etl/control/source.rb', line 8

def control
  @control
end

#definition ⇒ Object

The definition Hash



14
15
16
 
# File 'lib/etl/control/source.rb', line 14

def definition
  @definition
end

#local_base ⇒ Object

Get the local base, defaults to ‘source_data’



58
59
60
 
# File 'lib/etl/control/source.rb', line 58

def local_base
  @local_base
end

#store_locally ⇒ Object

Returns true if the source data should be stored locally for archival Default behavior will return true.



18
19
20
 
# File 'lib/etl/control/source.rb', line 18

def store_locally
  @store_locally
end

Class Method Details

.class_for_name(name) ⇒ Object

Convert the name to a Source class.

For example if name is :database then this will return a DatabaseSource class



25
26
27
 
# File 'lib/etl/control/source.rb', line 25

def class_for_name(name)
  ETL::Control.const_get("#{name.to_s.camelize}Source")
end

Instance Method Details

#errors ⇒ Object

Get an array of errors that occur during reading from the source



48
49
50
 
# File 'lib/etl/control/source.rb', line 48

def errors
  @errors ||= []
end

#last_local_file ⇒ Object

Get the last fully written local file



86
87
88
 
# File 'lib/etl/control/source.rb', line 86

def last_local_file
  File.join(local_directory, File.basename(last_local_file_trigger, '.trig'))
end

#last_local_file_trigger ⇒ Object

Get the last local file trigger



91
92
93
 
# File 'lib/etl/control/source.rb', line 91

def last_local_file_trigger
  Dir.glob(File.join(local_directory, '*.trig')).last
end

#local_directory ⇒ Object

The local directory for storing. This method must be overriden by subclasses



67
68
69
 
# File 'lib/etl/control/source.rb', line 67

def local_directory
  raise "local_directory method is abstract"
end

#local_file(sequence = nil) ⇒ Object

Return the local file for storing the raw source data. Each call to this method will result in a timestamped file, so you cannot expect to call it multiple times and reference the same file

Optional sequence can be specified if there are multiple source files



76
77
78
79
80
81
82
83
 
# File 'lib/etl/control/source.rb', line 76

def local_file(sequence=nil)
  filename = timestamp.to_s
  filename += sequence.to_s if sequence
  
  local_dir = local_directory
  FileUtils.mkdir_p(local_dir)
  File.join(local_dir, "#{filename}.csv")
end

#local_file_trigger(file) ⇒ Object

Get the local trigger file that is used to indicate that the file has been completely written



97
98
99
 
# File 'lib/etl/control/source.rb', line 97

def local_file_trigger(file)
  Pathname.new(file.to_s + '.trig')
end

#read_locally ⇒ Object

Return true if the source should read locally.



102
103
104
 
# File 'lib/etl/control/source.rb', line 102

def read_locally
  Engine.read_locally
end

#timestamp ⇒ Object

Get a timestamp value as a string



53
54
55
 
# File 'lib/etl/control/source.rb', line 53

def timestamp
  Engine.timestamp
end