Class: Chef::ChefFS::ChefFSDataStore

Inherits:

Object

Object
Chef::ChefFS::ChefFSDataStore

show all

Defined in:: lib/chef/chef_fs/chef_fs_data_store.rb

Overview

Translation layer between chef-zero’s DataStore (a place where it expects files to be stored) and ChefFS (the user’s repository directory layout).

chef-zero expects the data store to store files its way–for example, it expects get(“nodes/blah”) to return the JSON text for the blah node, and it expects get(“cookbooks/blah/1.0.0”) to return the JSON definition of the blah cookbook version 1.0.0.

The repository is defined the way the user wants their layout. These two things are very similar in layout (for example, nodes are stored under the nodes/ directory and their filename is the name of the node).

However, there are a few differences that make this more than just a raw file store:

Cookbooks are stored much differently.

- chef-zero places JSON text with the checksums for the cookbook at
  /cookbooks/NAME/VERSION, and expects the JSON to contain URLs to the
  actual files, which are stored elsewhere.
- The repository contains an actual directory with just the cookbook
  files and a metadata.rb containing a version #.  There is no JSON to
  be found.
- Further, if versioned_cookbooks is false, that directory is named
  /cookbooks/NAME and only one version exists.  If versioned_cookbooks
  is true, the directory is named /cookbooks/NAME-VERSION.
- Therefore, ChefFSDataStore calculates the cookbook JSON by looking at
  the files in the cookbook and checksumming them, and reading metadata.rb
  for the version and dependency information.
- ChefFSDataStore also modifies the cookbook file URLs so that they point
  to /file_store/repo/<filename> (the path to the actual file under the
  repository root).  For example, /file_store/repo/apache2/metadata.rb or
  /file_store/repo/cookbooks/apache2/recipes/default.rb).

Sandboxes don’t exist in the repository.

- ChefFSDataStore lets cookbooks be uploaded into a temporary memory
  storage, and when the cookbook is committed, copies the files onto the
  disk in the correct place (/cookbooks/apache2/recipes/default.rb).

Data bags:

- The Chef server expects data bags in /data/BAG/ITEM
- The repository stores data bags in /data_bags/BAG/ITEM

JSON filenames are generally NAME.json in the repository (e.g. /nodes/foo.json).
Org membership: chef-zero stores user membership in an org as a series of empty files. If an org has jkeiser and cdoherty as members, chef-zero expects these files to exist:
- ‘users/jkeiser` (content: ’{}‘)
- ‘users/cdoherty` (content: ’{}‘)
ChefFS, on the other hand, stores user membership in an org as a single file, ‘members.json`, with content:
```
```json
[
  { "user": { "username": "jkeiser" } },
  { "user": { "username": "cdoherty" } }
]
```
```
To translate between the two, we need to intercept requests to ‘users` like so:
- ‘list(users)` -> `get(/members.json)`
- ‘get(users/NAME)` -> `get(/members.json)`, see if it’s in there
- ‘create(users/NAME)` -> `get(/members.json)`, add name, `set(/members.json)`
- ‘delete(users/NAME)` -> `get(/members.json)`, remove name, `set(/members.json)`
Org invitations: chef-zero stores org membership invitations as a series of empty files. If an org has invited jkeiser and cdoherty (and they have not yet accepted the invite), chef-zero expects these files to exist:
- ‘association_requests/jkeiser` (content: ’{}‘)
- ‘association_requests/cdoherty` (content: ’{}‘)
ChefFS, on the other hand, stores invitations as a single file, ‘invitations.json`, with content:
```
```json
[
  { "id" => "jkeiser-chef", 'username' => 'jkeiser' },
  { "id" => "cdoherty-chef", 'username' => 'cdoherty' }
]
```
```
To translate between the two, we need to intercept requests to ‘users` like so:
- ‘list(association_requests)` -> `get(/invitations.json)`
- ‘get(association_requests/NAME)` -> `get(/invitations.json)`, see if it’s in there
- ‘create(association_requests/NAME)` -> `get(/invitations.json)`, add name, `set(/invitations.json)`
- ‘delete(association_requests/NAME)` -> `get(/invitations.json)`, remove name, `set(/invitations.json)`

Constant Summary collapse

BASE_DIRNAMES = The base directories in a Chef Repo; even when these don’t exist, a matching GET for these objects will return an empty list instead of a 404.

%w{
  clients
  cookbooks
  data
  environments
  nodes
  roles
  users
  containers
  groups
  policy_groups
  policies
}.freeze

Instance Attribute Summary collapse

#chef_fs ⇒ Object readonly

Returns the value of attribute chef_fs.
#repo_mode ⇒ Object readonly

Returns the value of attribute repo_mode.

Instance Method Summary collapse

#create(path, name, data, *options) ⇒ Object

If you want to get the contents of /data/x/y from the server, you say chef_fs.child(‘data’).child(‘x’).child(‘y’).read.
#create_dir(path, name, *options) ⇒ Object
#delete(path) ⇒ Object
#delete_dir(path, *options) ⇒ Object
#exists?(path) ⇒ Boolean
#exists_dir?(path) ⇒ Boolean
#get(path, request = nil) ⇒ Object
#initialize(chef_fs, chef_config = Chef::Config) ⇒ ChefFSDataStore constructor

Create a new ChefFSDataStore.
#list(path) ⇒ Object
#publish_description ⇒ Object
#set(path, data, *options) ⇒ Object

Constructor Details

#initialize(chef_fs, chef_config = Chef::Config) ⇒ `ChefFSDataStore`

Create a new ChefFSDataStore

Arguments

chef_fs: A ChefFS::FileSystem object representing the repository root. Generally will be a ChefFS::FileSystem::ChefRepositoryFileSystemRoot object, created from ChefFS::Config.local_fs.

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 157

def initialize(chef_fs, chef_config = Chef::Config)
  @chef_fs = chef_fs
  @memory_store = ChefZero::DataStore::MemoryStore.new
  @repo_mode = chef_config[:repo_mode]
end

Instance Attribute Details

#chef_fs ⇒ `Object` (readonly)

Returns the value of attribute chef_fs.



167
168
169

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 167

def chef_fs
  @chef_fs
end

#repo_mode ⇒ `Object` (readonly)

Returns the value of attribute repo_mode.



168
169
170

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 168

def repo_mode
  @repo_mode
end

Instance Method Details

#create(path, name, data, *options) ⇒ `Object`

If you want to get the contents of /data/x/y from the server, you say chef_fs.child(‘data’).child(‘x’).child(‘y’).read. It will make exactly one network request: GET /data/x/y And that will return 404 if it doesn’t exist.

ChefFS objects do not go to the network until you ask them for data. This means you can construct a /data/x/y ChefFS entry early.

Alternative: chef_fs.child(‘data’) could have done a GET /data preemptively, allowing it to know whether child(‘x’) was valid (GET /data gives you a list of data bags). Then child(‘x’) could have done a GET /data/x, allowing it to know whether child(‘y’) (the item) existed. Finally, we would do the GET /data/x/y to read the contents. Three network requests instead of 1.

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 202

def create(path, name, data, *options)
  if use_memory_store?(path)
    @memory_store.create(path, name, data, *options)

  elsif path[0] == "cookbooks" && path.length == 2
  # Do nothing.  The entry gets created when the cookbook is created.

  # /policy_groups/GROUP/policies/NAME
  elsif path[0] == "policy_groups" && path[2] == "policies"
    # Just set or create the proper entry in the hash
    update_json(to_chef_fs_path(path[0..1]), {}, *options) do |group|
      if policies.key?(path[3])
        raise ChefZero::DataStore::DataAlreadyExistsError.new(path, group)
      end

      group["policies"] ||= {}
      group["policies"][path[3]] = { "revision_id" => Chef::JSONCompat.parse(data) }
      group
    end

  # create [/organizations/ORG]/users/NAME (with content '{}')
  # Manipulate the `members.json` file that contains a list of all users
  elsif is_org? && path == [ "users" ]
    update_json("members.json", [], *options) do |members|
      # Format of each entry: { "user": { "username": "jkeiser" } }
      if members.any? { |member| member["user"]["username"] == name }
        raise ChefZero::DataStore::DataAlreadyExistsError.new(path, entry)
      end

      # Actually add the user
      members << { "user" => { "username" => name } }
    end

  # create [/organizations/ORG]/association_requests/NAME (with content '{}')
  # Manipulate the `invitations.json` file that contains a list of all users
  elsif is_org? && path == [ "association_requests" ]
    update_json("invitations.json", [], *options) do |invitations|
      # Format of each entry: { "id" => "jkeiser-chef", 'username' => 'jkeiser' }
      if invitations.any? { |member| member["username"] == name }
        raise ChefZero::DataStore::DataAlreadyExistsError.new(path)
      end

      # Actually add the user (TODO insert org name??)
      invitations << { "username" => name }
    end

  else
    unless data.is_a?(String)
      raise "set only works with strings"
    end

    with_parent_dir(path + [name], *options) do |parent, name|

      parent.create_child(name, data)
    rescue Chef::ChefFS::FileSystem::AlreadyExistsError => e
      raise ChefZero::DataStore::DataAlreadyExistsError.new(to_zero_path(e.entry), e)

    end
  end
end

#create_dir(path, name, *options) ⇒ `Object`

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 170

def create_dir(path, name, *options)
  if use_memory_store?(path)
    @memory_store.create_dir(path, name, *options)
  else
    with_parent_dir(path + [name], *options) do |parent, name|

      parent.create_child(name, nil)
    rescue Chef::ChefFS::FileSystem::AlreadyExistsError => e
      raise ChefZero::DataStore::DataAlreadyExistsError.new(to_zero_path(e.entry), e)

    end
  end
end

#delete(path) ⇒ `Object`

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 395

def delete(path)
  if use_memory_store?(path)
    @memory_store.delete(path)

  # DELETE /policy_groups/GROUP/policies/POLICY
  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 4
    update_json(to_chef_fs_path(path[0..1]), {}) do |group|
      unless group["policies"] && group["policies"].key?(path[3])
        raise ChefZero::DataStore::DataNotFoundError.new(path)
      end

      group["policies"].delete(path[3])
      group
    end

  # DELETE [/organizations/ORG]/users/NAME
  # Manipulates members.json
  elsif is_org? && path[0] == "users" && path.length == 2
    update_json("members.json", []) do |members|
      result = members.reject { |member| member["user"]["username"] == path[1] }
      if result.size == members.size
        raise ChefZero::DataStore::DataNotFoundError.new(path)
      end

      result
    end

  # DELETE [/organizations/ORG]/users/NAME
  # Manipulates members.json
  elsif is_org? && path[0] == "association_requests" && path.length == 2
    update_json("invitations.json", []) do |invitations|
      result = invitations.reject { |invitation| invitation["username"] == path[1] }
      if result.size == invitations.size
        raise ChefZero::DataStore::DataNotFoundError.new(path)
      end

      result
    end

  else
    with_entry(path) do |entry|

      if %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length >= 3
        entry.delete(true)
      else
        entry.delete(false)
      end
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)

    end
  end
end

#delete_dir(path, *options) ⇒ `Object`

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 449

def delete_dir(path, *options)
  if use_memory_store?(path)
    @memory_store.delete_dir(path, *options)

  # DELETE /policies/POLICY
  elsif path[0] == "policies" && path.length == 2
    with_entry(path[0..0]) do |policies|
      # /policies:
      #   - a-1.0.0.json
      #   - a-1.0.1.json
      #   - b-2.0.0.json
      found_policy = false
      policies.children.each do |policy|
        # We want to delete just the ones that == POLICY
        next unless policy.name.rpartition("-")[0] == path[1]

        policy.delete(false)
        FileSystemCache.instance.delete!(policy.file_path)
        found_policy = true
      end
      raise ChefZero::DataStore::DataNotFoundError.new(path) unless found_policy
    end

  else
    with_entry(path) do |entry|

      entry.delete(options.include?(:recursive))
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)

    end
  end
end

#exists?(path) ⇒ `Boolean`

Returns:

(Boolean)

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 584

def exists?(path)
  if use_memory_store?(path)
    @memory_store.exists?(path)

  # /policy_groups/NAME/policies/POLICYNAME
  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 4
    group = get_json(to_chef_fs_path(path[0..1]), {})
    group["policies"] && group["policies"].key?(path[3])

  else
    path_always_exists?(path) || Chef::ChefFS::FileSystem.resolve_path(chef_fs, to_chef_fs_path(path)).exists?
  end
end

#exists_dir?(path) ⇒ `Boolean`

Returns:

(Boolean)

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 598

def exists_dir?(path)
  if use_memory_store?(path)
    @memory_store.exists_dir?(path)

  elsif %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 2
    list([ path[0] ]).include?(path[1])

  # /policies/NAME
  elsif path[0] == "policies" && path.length == 2
    list([ path[0] ]).include?(path[1])

  # /policy_groups/NAME/policies
  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 3
    exists_dir?(path[0..1])

  else
    Chef::ChefFS::FileSystem.resolve_path(chef_fs, to_chef_fs_path(path)).exists?
  end
end

#get(path, request = nil) ⇒ `Object`

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 263

def get(path, request = nil)
  if use_memory_store?(path)
    @memory_store.get(path)

  elsif path[0] == "file_store" && path[1] == "repo"
    entry = Chef::ChefFS::FileSystem.resolve_path(chef_fs, path[2..].join("/"))
    begin
      entry.read
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)
    end

  # /policy_groups/NAME/policies/POLICYNAME: return the revision of the given policy
  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 4
    # Just set or create the proper entry in the hash
    policy_group = get_json(to_chef_fs_path(path[0..1]), {})
    if !policy_group["policies"] || !policy_group["policies"][path[3]]
      raise ChefZero::DataStore::DataNotFoundError.new(path, entry)
    end

    # The policy group looks like:
    # {
    #   "policies": {
    #     "x": { "revision_id": "10" }
    #   }
    # }
    Chef::JSONCompat.to_json_pretty(policy_group["policies"][path[3]]["revision_id"])

  # GET [/organizations/ORG]/users/NAME -> /users/NAME
  # Manipulates members.json
  elsif is_org? && path[0] == "users" && path.length == 2
    if get_json("members.json", []).any? { |member| member["user"]["username"] == path[1] }
      "{}"
    else
      raise ChefZero::DataStore::DataNotFoundError.new(path)
    end

  # GET [/organizations/ORG]/association_requests/NAME -> /users/NAME
  # Manipulates invites.json
  elsif is_org? && path[0] == "association_requests" && path.length == 2
    if get_json("invites.json", []).any? { |member| member["user"]["username"] == path[1] }
      "{}"
    else
      raise ChefZero::DataStore::DataNotFoundError.new(path)
    end

  # GET /cookbooks/NAME/VERSION or /cookbook_artifacts/NAME/IDENTIFIER
  elsif %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 3
    with_entry(path) do |entry|
      cookbook_type = path[0]
      result = nil
      begin
        result = Chef::CookbookManifest.new(entry.chef_object, policy_mode: cookbook_type == "cookbook_artifacts").to_h
      rescue Chef::ChefFS::FileSystem::NotFoundError => e
        raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)
      end

      result.each_pair do |key, value|
        if value.is_a?(Array)
          value.each do |file|
            if file.is_a?(Hash) && file.key?("checksum")
              relative = ["file_store", "repo", cookbook_type]
              if chef_fs.versioned_cookbooks || cookbook_type == "cookbook_artifacts"
                relative << "#{path[1]}-#{path[2]}"
              else
                relative << path[1]
              end
              relative += file[:path].split("/")
              file["url"] = ChefZero::RestBase.build_uri(request.base_uri, relative)
            end
          end
        end
      end

      if cookbook_type == "cookbook_artifacts"
        result["metadata"] = result["metadata"].to_h
        result["metadata"].delete_if do |key, value|
          value == [] ||
            (value == {} && !%w{dependencies attributes recipes}.include?(key)) ||
            (value == "" && %w{source_url issues_url}.include?(key)) ||
            (value == false && key == "privacy")
        end
      end

      Chef::JSONCompat.to_json_pretty(result)
    end

  else
    with_entry(path) do |entry|

      entry.read
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)

    end
  end
end

#list(path) ⇒ `Object`

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 483

def list(path)
  if use_memory_store?(path)
    @memory_store.list(path)

  # LIST /policies
  elsif path == [ "policies" ]
    with_entry([ path[0] ]) do |policies|

      policies.children.map { |policy| policy.name[0..-6].rpartition("-")[0] }.uniq
    rescue Chef::ChefFS::FileSystem::NotFoundError
      []

    end

  # LIST /policies/POLICY/revisions
  elsif path[0] == "policies" && path[2] == "revisions" && path.length == 3
    with_entry([ path[0] ]) do |policies|
      # /policies:
      #   - a-1.0.0.json
      #   - a-1.0.1.json
      #   - b-2.0.0.json
      revisions = []
      policies.children.each do |policy|
        name, dash, revision = policy.name[0..-6].rpartition("-")
        revisions << revision if name == path[1]
      end
      raise ChefZero::DataStore::DataNotFoundError.new(path) if revisions.empty?

      revisions
    end

  elsif path[0] == "policy_groups" && path.length == 2
    with_entry(path) do |entry|
      [ "policies" ]
    end

  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 3
    with_entry(path[0..1]) do |entry|
      policies = Chef::JSONCompat.parse(entry.read)["policies"] || {}
      policies.keys
    end

  elsif %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 1
    with_entry(path) do |entry|

      if path[0] == "cookbook_artifacts"
        entry.children.map { |child| child.name.rpartition("-")[0] }.uniq
      elsif chef_fs.versioned_cookbooks
        # /cookbooks/name-version -> /cookbooks/name
        entry.children.map { |child| split_name_version(child.name)[0] }.uniq
      else
        entry.children.map(&:name)
      end
    rescue Chef::ChefFS::FileSystem::NotFoundError
      # If the cookbooks dir doesn't exist, we have no cookbooks (not 404)
      []

    end

  elsif %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 2
    if chef_fs.versioned_cookbooks || path[0] == "cookbook_artifacts"
      result = with_entry([ path[0] ]) do |entry|
        # list /cookbooks/name = filter /cookbooks/name-version down to name
        entry.children.map { |child| split_name_version(child.name) }
          .select { |name, version| name == path[1] }
          .map { |name, version| version }
      end
      if result.empty?
        raise ChefZero::DataStore::DataNotFoundError.new(path)
      end

      result
    else
      # list /cookbooks/name = <single version>
      version = get_single_cookbook_version(path)
      [version]
    end

  else
    result = with_entry(path) do |entry|

      entry.children.map { |c| zero_filename(c) }.sort
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      # /cookbooks, /data, etc. never return 404
      if path_always_exists?(path)
        []
      else
        raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)
      end

    end

    # Older versions of chef-zero do not understand policies and cookbook_artifacts,
    # don't give that stuff to them
    if path == [] && ChefZero::VERSION.to_f < 4.4
      result.reject! { |child| %w{policies policy_data cookbook_artifacts}.include?(child) }
    end
    result
  end
end

#publish_description ⇒ `Object`



163
164
165

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 163

def publish_description
  "Reading and writing data to #{chef_fs.fs_description}"
end

#set(path, data, *options) ⇒ `Object`

# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 361

def set(path, data, *options)
  if use_memory_store?(path)
    @memory_store.set(path, data, *options)
  else
    unless data.is_a?(String)
      raise "set only works with strings: #{path} = #{data.inspect}"
    end

    # Write out the files!
    if %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 3
      write_cookbook(path, data, *options)

    # Handle /policy_groups/some_policy_group/policies/some_policy_name
    elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 4
      # Just set or create the proper entry in the hash
      update_json(to_chef_fs_path(path[0..1]), {}, *options) do |group|
        group["policies"] ||= {}
        group["policies"][path[3]] = { "revision_id" => Chef::JSONCompat.parse(data) }
        group
      end

    else
      with_parent_dir(path, *options) do |parent, name|
        child = parent.child(name)
        if child.exists?
          child.write(data)
        else
          parent.create_child(name, data)
        end
      end
    end
  end
end

Class: Chef::ChefFS::ChefFSDataStore

Overview

Constant Summary collapse

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(chef_fs, chef_config = Chef::Config) ⇒ ChefFSDataStore

Arguments

Instance Attribute Details

#chef_fs ⇒ Object (readonly)

#repo_mode ⇒ Object (readonly)

Instance Method Details

#create(path, name, data, *options) ⇒ Object

#create_dir(path, name, *options) ⇒ Object

#delete(path) ⇒ Object

#delete_dir(path, *options) ⇒ Object

#exists?(path) ⇒ Boolean

#exists_dir?(path) ⇒ Boolean

#get(path, request = nil) ⇒ Object

#list(path) ⇒ Object

#publish_description ⇒ Object

#set(path, data, *options) ⇒ Object