Class: OctocatalogDiff::CatalogDiff::Differ

Inherits:
Object
  • Object
show all
Defined in:
lib/octocatalog-diff/catalog-diff/differ.rb

Overview

Calculate the difference between two Puppet catalogs.

It was necessary to write our own code for this, and not just use some existing gem, for two main reasons:

  1. There are things that we want to ignore when doing a Puppet catalog diff. For example we want to ignore ‘before’ and ‘require’ parameters (because those affect the order of operations only, not the end result) and we probably want to ignore ‘tags’ attributes and all classes. No existing code (that I could find at least) was capable of allowing you to skip stuff via arguments, without your own custom pre-processing.

  2. When using the ‘hashdiff’ gem, there is no distinguishing between an addition of an entire new key-value pair, or the addition of an element in a deeply nested array. By way of further explanation, consider these two data structures:

    a = { ‘foo’ => ‘bar’, ‘my_array’ => [ 1, 2, 3 ] } b = { ‘foo’ => ‘bar’, ‘my_array’ => [ 1, 2, 3, 4 ], ‘another_key’ => ‘another_value’

    The hashdiff gem would report the differences between a and b to be:

    + 4
+ another_key => another_value

    We want to distinguish (without a whole bunch of convoluted code) between these two situations. One was a true addition (adding a key) while one was a change (adding element to array). This distinction becomes even more important when considering top-level changes vs. changes to arrays or hashes nested within the catalog.

Therefore, the algorithm implemented here is as follows:

  1. Pre-process the catalog JSON files to:

    • Sort the ‘tags’ array, since the order of tags does not matter to Puppet

    • Pull out additions of entire key-value pairs (above, ‘another_key’ => ‘another_value’)

  2. Everything left consists of key-value pairs where the key exists in both old and new. Pass this to the ‘hashdiff’ gem.

  3. Filter any differences to remove attributes, types, or resources that have been explicitly ignored.

  4. Reformat any ‘+’ or ‘-’ reported by hashdiff to be changes to the keys, rather than outright additions.

The heavy lifting is still handled by ‘hashdiff’ but we’re pre-simplifying the input and post-processing the output to make it easier to deal with later.

Instance Method Summary collapse

Constructor Details

#initialize(opts, catalog1_in, catalog2_in) ⇒ Differ

Constructor

Parameters:



63
64
65
66
67
68
69
70
71
72
73
 
# File 'lib/octocatalog-diff/catalog-diff/differ.rb', line 63

def initialize(opts, catalog1_in, catalog2_in)
  @catalog1_raw = catalog1_in
  @catalog2_raw = catalog2_in
  @catalog1 = catalog_resources(catalog1_in, 'First catalog')
  @catalog2 = catalog_resources(catalog2_in, 'Second catalog')
  @logger = opts.fetch(:logger, Logger.new(StringIO.new))
  @diff_result = nil
  @ignore = Set.new
  ignore(opts.fetch(:ignore, []))
  @opts = opts
end

Instance Method Details

#catalog1 ⇒ Array<Resource Hashes>

Return catalog1 with filter_and_cleanups applied. This is in the public section because it’s called from spec tests as well as being called internally.

Returns:

  • (Array<Resource Hashes>) —

    Filtered resources in catalog



138
139
140
 
# File 'lib/octocatalog-diff/catalog-diff/differ.rb', line 138

def catalog1
  filter_and_cleanup(@catalog1)
end

#catalog2 ⇒ Array<Resource Hashes>

Return catalog2 with filter_and_cleanups applied. This is in the public section because it’s called from spec tests as well as being called internally.

Returns:

  • (Array<Resource Hashes>) —

    Filtered resources in catalog



146
147
148
 
# File 'lib/octocatalog-diff/catalog-diff/differ.rb', line 146

def catalog2
  filter_and_cleanup(@catalog2)
end

#diff ⇒ Array<Diff results>

Difference - calculates and then returns the diff of this objects Each diff result is an array like this:

[ <String> '+|-|~|!', <String> Key name, <Object> Old object, <Object> New object ]

Returns:

  • (Array<Diff results>) —

    Results of the diff



79
80
81
 
# File 'lib/octocatalog-diff/catalog-diff/differ.rb', line 79

def diff
  @diff_result ||= catdiff
end

#ignore(ignores = []) ⇒ OctocatalogDiff::CatalogDiff::Differ

Ignore - ignored items can be set by Type, Title, or Attribute; setting multiple in a hash is interpreted as AND. The collection of all ignored items is interpreted as OR.

Parameters:

  • ignore (Hash<type: xxx, title: yyy, attr: zzz>) —

    Ignore type/title/attr (can pass array also)

Returns:



87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
 
# File 'lib/octocatalog-diff/catalog-diff/differ.rb', line 87

def ignore(ignores = [])
  ignore_array = ignores.is_a?(Array) ? ignores : [ignores]
  ignore_array.each do |item|
    raise ArgumentError, "Argument #{item.inspect} to ignore is not a hash" unless item.is_a?(Hash)
    unless item.key?(:type) || item.key?(:title) || item.key?(:attr)
      raise ArgumentError, "Argument #{item.inspect} does not contain :type, :title, or :attr"
    end
    item[:type] ||= '*'
    item[:title] ||= '*'
    item[:attr] ||= '*'

    # Support wildcards in title
    if item[:title].is_a?(String) && item[:title] != '*' && item[:title].include?('*')
      item[:title] = Regexp.new("\\A#{Regexp.escape(item[:title]).gsub('\*', '.*')}\\Z", 'i')
    end

    @ignore.add(item)
  end
  self
end

#ignore_tags ⇒ Object

Handle –ignore-tags option, the ability to tag resources within modules/manifests and have catalog-diff ignore them.



110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
 
# File 'lib/octocatalog-diff/catalog-diff/differ.rb', line 110

def ignore_tags
  return unless @opts[:ignore_tags].is_a?(Array) && @opts[:ignore_tags].any?

  # Go through the "to" catalog and identify any resources that have been tagged with one or more
  # specified "ignore tags." Add any such items to the ignore list. The 'to' catalog has the authoritative
  # list of dynamic ignores.
  @catalog2_raw.resources.each do |resource|
    next unless tagged_for_ignore?(resource)
    ignore(type: resource['type'], title: resource['title'])
    @logger.debug "Ignoring type='#{resource['type']}', title='#{resource['title']}' based on tag in to-catalog"
  end

  # Go through the "from" catalog and identify any resources that have been tagged with one or more
  # specified "ignore tags." Only mark the resources for ignoring if they do not appear in the 'to'
  # catalog, thereby allowing the 'to' catalog to be the authoritative ignore list. This allows deleted
  # items that were previously ignored to continue to be ignored.
  @catalog1_raw.resources.each do |resource|
    next if @catalog2_raw.resource(type: resource['type'], title: resource['title'])
    next unless tagged_for_ignore?(resource)
    ignore(type: resource['type'], title: resource['title'])
    @logger.debug "Ignoring type='#{resource['type']}', title='#{resource['title']}' based on tag in from-catalog"
  end
end