Class: Rimless::AvroUtils

Inherits:
Object
  • Object
show all
Defined in:
lib/rimless/avro_utils.rb

Overview

Due to dynamic constrains on the Apache Avro schemas we need to compile our schema templates to actual ready-to-consume schemas. The namespace part of the schemas and cross-references to other schemas must be rendered according to the dynamic namespace prefix which reflects the application environment. Unfortunately we need to mess around with actual files to support the Avro and AvroTurf gems.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ AvroUtil

Create a new instance of the AvroUtil class.



16
17
18
19
20
 
# File 'lib/rimless/avro_utils.rb', line 16

def initialize
  @namespace = ENV.fetch('KAFKA_SCHEMA_SUBJECT_PREFIX',
                         Rimless.topic_prefix).tr('-', '_').gsub(/\.$/, '')
  @env = @namespace.split('.').first
end

Instance Attribute Details

#env ⇒ Object (readonly)

Returns the value of attribute env.



11
12
13
 
# File 'lib/rimless/avro_utils.rb', line 11

def env
  @env
end

#namespace ⇒ Object (readonly)

Returns the value of attribute namespace.



11
12
13
 
# File 'lib/rimless/avro_utils.rb', line 11

def namespace
  @namespace
end

Instance Method Details

#base_path ⇒ Pathname

Return the base path of the Avro schemas on our project.

Returns:

  • (Pathname) —

    the Avro schemas base path



93
94
95
 
# File 'lib/rimless/avro_utils.rb', line 93

def base_path
  Rimless.configuration.avro_schema_path
end

#clear ⇒ Object

Clear previous compiled Avro schema files to provide a clean rebuild.



67
68
69
70
71
72
73
74
75
 
# File 'lib/rimless/avro_utils.rb', line 67

def clear
  # In a test environment, especially with parallel test execution the
  # recompiling of Avro schemas is error prone due to the deletion of the
  # configuration (output) directory. This leads to random failures due to
  # file read calls to temporary not existing files. So we just keep the
  # files and just overwrite them in place while testing.
  FileUtils.rm_rf(output_path) unless Rimless.env.test?
  FileUtils.mkdir_p(output_path)
end

#output_path ⇒ Pathname

Return the path to the compiled Avro schemas. This path must be consumed by the AvroTurf::Messaging constructor.

Returns:

  • (Pathname) —

    the compiled Avro schemas path



101
102
103
 
# File 'lib/rimless/avro_utils.rb', line 101

def output_path
  Rimless.configuration.compiled_avro_schema_path
end

#recompile_schemas ⇒ Object

Clean and recompile all templated Avro schema files to their respective output path.



24
25
26
27
 
# File 'lib/rimless/avro_utils.rb', line 24

def recompile_schemas
  clear
  Dir[base_path.join('**', '*.erb')].each { |src| render_file(src) }
end

#render_file(src) ⇒ Object

Render (compile) a single Avro schema template. The given source file path will serve to calculate the destination path. So even deep path’ed templates will keep their hierarchy.

Parameters:

  • src (String) —

    the Avro schema template file path



34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/rimless/avro_utils.rb', line 34

def render_file(src)
  # Convert the template path to the destination path
  dest = schema_path(src)

  # Allow parallel cleanups and execution
  with_retries(max_tries: 3, rescue: Errno::ENOENT) do
    # Create the deep path when not yet existing
    FileUtils.mkdir_p(File.dirname(dest))
    # Write the rendered file contents to the destination
    File.write(dest, ERB.new(File.read(src)).result(binding))
    # Check the written file for correct JSON
    validate_file(dest)
  end
end

#schema_path(src) ⇒ Pathname

Return the compiled Avro schema file path for the given Avro schema template.

Parameters:

  • src (String) —

    the Avro schema template file path

Returns:

  • (Pathname) —

    the resulting schema file path



82
83
84
85
86
87
88
 
# File 'lib/rimless/avro_utils.rb', line 82

def schema_path(src)
  # No trailing dot on the prefix namespace directory
  prefix = env.remove(/\.$/)
  # Calculate the destination path based on the source file
  Pathname.new(src.gsub(/^#{base_path}/, output_path.join(prefix).to_s)
                  .gsub(/\.erb$/, ''))
end

#validate_file(dest) ⇒ Object

Check the given file for valid JSON.

rubocop:disable Security/JSONLoad because we wrote the file contents

Parameters:

  • dest (Pathname, File, IO) —

    the file to check

Raises:

  • (JSON::ParserError) —

    when invalid



55
56
57
58
59
60
61
62
63
 
# File 'lib/rimless/avro_utils.rb', line 55

def validate_file(dest)
  JSON.load(dest)
rescue JSON::ParserError => e
  path = File.expand_path(dest.is_a?(File) ? dest.path : dest.to_s)
  prefix = "Invalid JSON detected: #{path}"
  Rimless.logger.fatal("#{prefix}\n#{e.message}")
  e.message.prepend("#{prefix} - ")
  raise e
end