Class: DbBlaster::Configuration

Inherits:
Object
  • Object
show all
Defined in:
lib/db_blaster/configuration.rb

Overview

Configuration class for providing credentials, topics, and customizations. Either the ‘sns_topic` or `s3_bucket’ must be set

Constant Summary collapse

DEFAULT_BATCH_SIZE = 
100
DEFAULT_MAX_MESSAGE_SIZE_IN_KILOBYTES =

max size allowed by AWS SNS

256
DEFAULT_S3_KEY = 
'<batch_date>/<batch_time>/db_blaster/<table_name>/<uuid>.json'
DEFAULT_DATETIME_FORMAT = 
'%Y-%m-%dT%H:%M:%S.%LZ'
ATTRIBUTE_S3_META_FORMAT =

{ meta: :value, records: [] }

'attribute'
INLINE_S3_META_FORMAT =

records.collect{|record| record.merge(meta) }

'inline'
REQUIRED_FIELDS =

The required configuration fields

%i[aws_access_key aws_access_secret aws_region].freeze
EITHER_OR_FIELDS =

exactly one of these fields needs to be set

%i[sns_topic s3_bucket].freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#aws_access_key ⇒ Object

Returns the value of attribute aws_access_key.



24
25
26
 
# File 'lib/db_blaster/configuration.rb', line 24

def aws_access_key
  @aws_access_key
end

#aws_access_secret ⇒ Object

Returns the value of attribute aws_access_secret.



24
25
26
 
# File 'lib/db_blaster/configuration.rb', line 24

def aws_access_secret
  @aws_access_secret
end

#aws_region ⇒ Object

Returns the value of attribute aws_region.



24
25
26
 
# File 'lib/db_blaster/configuration.rb', line 24

def aws_region
  @aws_region
end

#batch_size ⇒ Object

Optional db_blaster will select and then publish ‘batch_size` rows at a time Default value is 100



88
89
90
 
# File 'lib/db_blaster/configuration.rb', line 88

def batch_size
  @batch_size
end

#extra_sns_message_attributes ⇒ Object

Optional Applicable only when ‘sns_topic` is set Extra [SNS message_attributes](docs.aws.amazon.com/sns/latest/dg/sns-message-attributes.html) Attributes set here will be included in every published message example: config.extra_sns_message_attributes = => {data_type: ‘String’, value: ‘061’}



31
32
33
 
# File 'lib/db_blaster/configuration.rb', line 31

def extra_sns_message_attributes
  @extra_sns_message_attributes
end

#ignore_source_tables ⇒ Object

Optional If set, ignore source tables specified. example: config.ignore_source_tables = [‘active_storage_blobs’]



76
77
78
 
# File 'lib/db_blaster/configuration.rb', line 76

def ignore_source_tables
  @ignore_source_tables
end

#ignored_column_names ⇒ Object

Global list of column names not to include in published SNS messages example: config.ignored_column_names = [‘email’, ‘phone_number’]



66
67
68
 
# File 'lib/db_blaster/configuration.rb', line 66

def ignored_column_names
  @ignored_column_names
end

#max_message_size_in_kilobytes ⇒ Object

Optional DbBlaster will publish no messages larger than this value Default value is 256



93
94
95
 
# File 'lib/db_blaster/configuration.rb', line 93

def max_message_size_in_kilobytes
  @max_message_size_in_kilobytes
end

#only_source_tables ⇒ Object

Optional If set, only publish tables specified. example: config.only_source_tables = [‘posts’, ‘tags’, ‘comments’]



71
72
73
 
# File 'lib/db_blaster/configuration.rb', line 71

def only_source_tables
  @only_source_tables
end

#s3_bucket ⇒ Object

The s3 bucket name



23
24
25
 
# File 'lib/db_blaster/configuration.rb', line 23

def s3_bucket
  @s3_bucket
end

#s3_key ⇒ Object

Optional Applicable only when ‘s3_bucket` is set The S3 key. The following values will get substituted: <batch_date_time> - date time when batch started <batch_date> - date when batch started <batch_time - time when batch started <date_time> - the datetime just before pushing to S3 <table_name> - the name of the table associated with the S3 body <uuid> - a universal identifier ’<batch_timestamp>/kcp-api/001/<table_name>/<uuid>.json’



56
57
58
 
# File 'lib/db_blaster/configuration.rb', line 56

def s3_key
  @s3_key
end

#s3_meta ⇒ Object

Optional Applicable only when ‘s3_bucket’ is set The value set here will be included in every payload pushed to S3 example: config.s3_meta = => ‘061’, ‘source_app’ => ‘kcp-api’}



37
38
39
 
# File 'lib/db_blaster/configuration.rb', line 37

def s3_meta
  @s3_meta
end

#s3_meta_format ⇒ Object

Optional Options: [‘attribute’, ‘inline’] Defaults to ‘attribute’ ‘attribute’ payload: { meta: ‘s3_meta`, records: [source_table_records] } ’inline’ payload: records.collect{|record| record.merge(meta) }



44
45
46
 
# File 'lib/db_blaster/configuration.rb', line 44

def s3_meta_format
  @s3_meta_format
end

#s3_tags ⇒ Object

Optional Applicable only when ‘s3_bucket` is set S3 Tags example: config.s3_tags = { infra_id: ’001’, source_app: ‘kcp-api’, source_table: ‘meetings’ }



62
63
64
 
# File 'lib/db_blaster/configuration.rb', line 62

def s3_tags
  @s3_tags
end

#sns_topic ⇒ Object

The topic to which messages will be published



21
22
23
 
# File 'lib/db_blaster/configuration.rb', line 21

def sns_topic
  @sns_topic
end

#source_table_options ⇒ Object

Optional Customize batch_size and/or ignored_columns example: config.source_table_options = [{ source_table_name: ‘posts’, batch_size: 100, ignored_column_names: [‘email’] },

{ source_table_name: 'comments', ignored_column_names: ['tags'] }]



83
84
85
 
# File 'lib/db_blaster/configuration.rb', line 83

def source_table_options
  @source_table_options
end

Instance Method Details

#verify! ⇒ Object

Raises error if a required field is not set



96
97
98
99
 
# File 'lib/db_blaster/configuration.rb', line 96

def verify!
  verify_required
  verify_either_or
end

#verify_either_or ⇒ Object 



101
102
103
104
105
106
 
# File 'lib/db_blaster/configuration.rb', line 101

def verify_either_or
  either_or = EITHER_OR_FIELDS.select do |attribute|
    send(attribute).nil? || send(attribute).strip.empty?
  end
  raise "only one of [#{either_or.join(', ')}] should be set" unless either_or.length == 1
end

#verify_required ⇒ Object 



108
109
110
111
112
113
 
# File 'lib/db_blaster/configuration.rb', line 108

def verify_required
  no_values = REQUIRED_FIELDS.select do |attribute|
    send(attribute).nil? || send(attribute).strip.empty?
  end
  raise "missing configuration values for [#{no_values.join(', ')}]" unless no_values.empty?
end