Class: Chef::ChefFS::ChefFSDataStore

Inherits:
Object
  • Object
show all
Defined in:
lib/chef/chef_fs/chef_fs_data_store.rb

Overview

Translation layer between chef-zero's DataStore (a place where it expects files to be stored) and ChefFS (the user's repository directory layout).

chef-zero expects the data store to store files its way--for example, it expects get("nodes/blah") to return the JSON text for the blah node, and it expects get("cookbooks/blah/1.0.0") to return the JSON definition of the blah cookbook version 1.0.0.

The repository is defined the way the user wants their layout. These two things are very similar in layout (for example, nodes are stored under the nodes/ directory and their filename is the name of the node).

However, there are a few differences that make this more than just a raw file store:

  1. Cookbooks are stored much differently.

    • chef-zero places JSON text with the checksums for the cookbook at /cookbooks/NAME/VERSION, and expects the JSON to contain URLs to the actual files, which are stored elsewhere.
    • The repository contains an actual directory with just the cookbook files and a metadata.rb containing a version #. There is no JSON to be found.
    • Further, if versioned_cookbooks is false, that directory is named /cookbooks/NAME and only one version exists. If versioned_cookbooks is true, the directory is named /cookbooks/NAME-VERSION.
    • Therefore, ChefFSDataStore calculates the cookbook JSON by looking at the files in the cookbook and checksumming them, and reading metadata.rb for the version and dependency information.
    • ChefFSDataStore also modifies the cookbook file URLs so that they point to /file_store/repo/ (the path to the actual file under the repository root). For example, /file_store/repo/apache2/metadata.rb or /file_store/repo/cookbooks/apache2/recipes/default.rb).

  2. Sandboxes don't exist in the repository.

    • ChefFSDataStore lets cookbooks be uploaded into a temporary memory storage, and when the cookbook is committed, copies the files onto the disk in the correct place (/cookbooks/apache2/recipes/default.rb).

  3. Data bags:

    • The Chef server expects data bags in /data/BAG/ITEM
    • The repository stores data bags in /data_bags/BAG/ITEM

  4. JSON filenames are generally NAME.json in the repository (e.g. /nodes/foo.json).

  5. Org membership: chef-zero stores user membership in an org as a series of empty files. If an org has jkeiser and cdoherty as members, chef-zero expects these files to exist:

  • users/jkeiser (content: '{}')
  • users/cdoherty (content: '{}')

ChefFS, on the other hand, stores user membership in an org as a single file, members.json, with content:

   ```json
   [
     { "user": { "username": "jkeiser" } },
     { "user": { "username": "cdoherty" } }
   ]
   ```

To translate between the two, we need to intercept requests to users like so:

  • list(users) -> get(/members.json)
  • get(users/NAME) -> get(/members.json), see if it's in there
  • create(users/NAME) -> get(/members.json), add name, set(/members.json)
  • delete(users/NAME) -> get(/members.json), remove name, set(/members.json)
  1. Org invitations: chef-zero stores org membership invitations as a series of empty files. If an org has invited jkeiser and cdoherty (and they have not yet accepted the invite), chef-zero expects these files to exist:
  • association_requests/jkeiser (content: '{}')
  • association_requests/cdoherty (content: '{}')

ChefFS, on the other hand, stores invitations as a single file, invitations.json, with content:

   ```json
   [
     { "id" => "jkeiser-chef", 'username' => 'jkeiser' },
     { "id" => "cdoherty-chef", 'username' => 'cdoherty' }
   ]
   ```

To translate between the two, we need to intercept requests to users like so:

  • list(association_requests) -> get(/invitations.json)
  • get(association_requests/NAME) -> get(/invitations.json), see if it's in there
  • create(association_requests/NAME) -> get(/invitations.json), add name, set(/invitations.json)
  • delete(association_requests/NAME) -> get(/invitations.json), remove name, set(/invitations.json)

Constant Summary collapse

BASE_DIRNAMES =

The base directories in a Chef Repo; even when these don't exist, a matching GET for these objects will return an empty list instead of a 404.

%w{
  clients
  cookbooks
  data
  environments
  nodes
  roles
  users
  containers
  groups
  policy_groups
  policies
}.freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(chef_fs, chef_config = Chef::Config) ⇒ ChefFSDataStore

Create a new ChefFSDataStore

==== Arguments

[chef_fs] A +ChefFS::FileSystem+ object representing the repository root. Generally will be a +ChefFS::FileSystem::ChefRepositoryFileSystemRoot+ object, created from +ChefFS::Config.local_fs+.



157
158
159
160
161
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 157

def initialize(chef_fs, chef_config = Chef::Config)
  @chef_fs = chef_fs
  @memory_store = ChefZero::DataStore::MemoryStore.new
  @repo_mode = chef_config[:repo_mode]
end

Instance Attribute Details

#chef_fs ⇒ Object (readonly)

Returns the value of attribute chef_fs.



167
168
169
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 167

def chef_fs
  @chef_fs
end

#repo_mode ⇒ Object (readonly)

Returns the value of attribute repo_mode.



168
169
170
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 168

def repo_mode
  @repo_mode
end

Instance Method Details

#create(path, name, data, *options) ⇒ Object

If you want to get the contents of /data/x/y from the server, you say chef_fs.child('data').child('x').child('y').read. It will make exactly one network request: GET /data/x/y And that will return 404 if it doesn't exist.

ChefFS objects do not go to the network until you ask them for data. This means you can construct a /data/x/y ChefFS entry early.

Alternative: chef_fs.child('data') could have done a GET /data preemptively, allowing it to know whether child('x') was valid (GET /data gives you a list of data bags). Then child('x') could have done a GET /data/x, allowing it to know whether child('y') (the item) existed. Finally, we would do the GET /data/x/y to read the contents. Three network requests instead of 1.



202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 202

def create(path, name, data, *options)
  if use_memory_store?(path)
    @memory_store.create(path, name, data, *options)

  elsif path[0] == "cookbooks" && path.length == 2
  # Do nothing.  The entry gets created when the cookbook is created.

  # /policy_groups/GROUP/policies/NAME
  elsif path[0] == "policy_groups" && path[2] == "policies"
    # Just set or create the proper entry in the hash
    update_json(to_chef_fs_path(path[0..1]), {}, *options) do |group|
      if policies.key?(path[3])
        raise ChefZero::DataStore::DataAlreadyExistsError.new(path, group)
      end

      group["policies"] ||= {}
      group["policies"][path[3]] = { "revision_id" => Chef::JSONCompat.parse(data) }
      group
    end

  # create [/organizations/ORG]/users/NAME (with content '{}')
  # Manipulate the `members.json` file that contains a list of all users
  elsif is_org? && path == [ "users" ]
    update_json("members.json", [], *options) do |members|
      # Format of each entry: { "user": { "username": "jkeiser" } }
      if members.any? { |member| member["user"]["username"] == name }
        raise ChefZero::DataStore::DataAlreadyExistsError.new(path, entry)
      end

      # Actually add the user
      members << { "user" => { "username" => name } }
    end

  # create [/organizations/ORG]/association_requests/NAME (with content '{}')
  # Manipulate the `invitations.json` file that contains a list of all users
  elsif is_org? && path == [ "association_requests" ]
    update_json("invitations.json", [], *options) do |invitations|
      # Format of each entry: { "id" => "jkeiser-chef", 'username' => 'jkeiser' }
      if invitations.any? { |member| member["username"] == name }
        raise ChefZero::DataStore::DataAlreadyExistsError.new(path)
      end

      # Actually add the user (TODO insert org name??)
      invitations << { "username" => name }
    end

  else
    unless data.is_a?(String)
      raise "set only works with strings"
    end

    with_parent_dir(path + [name], *options) do |parent, name|

      parent.create_child(name, data)
    rescue Chef::ChefFS::FileSystem::AlreadyExistsError => e
      raise ChefZero::DataStore::DataAlreadyExistsError.new(to_zero_path(e.entry), e)

    end
  end
end

#create_dir(path, name, *options) ⇒ Object 



170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 170

def create_dir(path, name, *options)
  if use_memory_store?(path)
    @memory_store.create_dir(path, name, *options)
  else
    with_parent_dir(path + [name], *options) do |parent, name|

      parent.create_child(name, nil)
    rescue Chef::ChefFS::FileSystem::AlreadyExistsError => e
      raise ChefZero::DataStore::DataAlreadyExistsError.new(to_zero_path(e.entry), e)

    end
  end
end

#delete(path) ⇒ Object 



395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 395

def delete(path)
  if use_memory_store?(path)
    @memory_store.delete(path)

  # DELETE /policy_groups/GROUP/policies/POLICY
  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 4
    update_json(to_chef_fs_path(path[0..1]), {}) do |group|
      unless group["policies"] && group["policies"].key?(path[3])
        raise ChefZero::DataStore::DataNotFoundError.new(path)
      end

      group["policies"].delete(path[3])
      group
    end

  # DELETE [/organizations/ORG]/users/NAME
  # Manipulates members.json
  elsif is_org? && path[0] == "users" && path.length == 2
    update_json("members.json", []) do |members|
      result = members.reject { |member| member["user"]["username"] == path[1] }
      if result.size == members.size
        raise ChefZero::DataStore::DataNotFoundError.new(path)
      end

      result
    end

  # DELETE [/organizations/ORG]/users/NAME
  # Manipulates members.json
  elsif is_org? && path[0] == "association_requests" && path.length == 2
    update_json("invitations.json", []) do |invitations|
      result = invitations.reject { |invitation| invitation["username"] == path[1] }
      if result.size == invitations.size
        raise ChefZero::DataStore::DataNotFoundError.new(path)
      end

      result
    end

  else
    with_entry(path) do |entry|

      if %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length >= 3
        entry.delete(true)
      else
        entry.delete(false)
      end
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)

    end
  end
end

#delete_dir(path, *options) ⇒ Object 



449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 449

def delete_dir(path, *options)
  if use_memory_store?(path)
    @memory_store.delete_dir(path, *options)

  # DELETE /policies/POLICY
  elsif path[0] == "policies" && path.length == 2
    with_entry(path[0..0]) do |policies|
      # /policies:
      #   - a-1.0.0.json
      #   - a-1.0.1.json
      #   - b-2.0.0.json
      found_policy = false
      policies.children.each do |policy|
        # We want to delete just the ones that == POLICY
        next unless policy.name.rpartition("-")[0] == path[1]

        policy.delete(false)
        FileSystemCache.instance.delete!(policy.file_path)
        found_policy = true
      end
      raise ChefZero::DataStore::DataNotFoundError.new(path) unless found_policy
    end

  else
    with_entry(path) do |entry|

      entry.delete(options.include?(:recursive))
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)

    end
  end
end

#exists?(path) ⇒ Boolean

Returns:

  • (Boolean) 


584
585
586
587
588
589
590
591
592
593
594
595
596
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 584

def exists?(path)
  if use_memory_store?(path)
    @memory_store.exists?(path)

  # /policy_groups/NAME/policies/POLICYNAME
  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 4
    group = get_json(to_chef_fs_path(path[0..1]), {})
    group["policies"] && group["policies"].key?(path[3])

  else
    path_always_exists?(path) || Chef::ChefFS::FileSystem.resolve_path(chef_fs, to_chef_fs_path(path)).exists?
  end
end

#exists_dir?(path) ⇒ Boolean

Returns:

  • (Boolean) 


598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 598

def exists_dir?(path)
  if use_memory_store?(path)
    @memory_store.exists_dir?(path)

  elsif %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 2
    list([ path[0] ]).include?(path[1])

  # /policies/NAME
  elsif path[0] == "policies" && path.length == 2
    list([ path[0] ]).include?(path[1])

  # /policy_groups/NAME/policies
  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 3
    exists_dir?(path[0..1])

  else
    Chef::ChefFS::FileSystem.resolve_path(chef_fs, to_chef_fs_path(path)).exists?
  end
end

#get(path, request = nil) ⇒ Object 



263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 263

def get(path, request = nil)
  if use_memory_store?(path)
    @memory_store.get(path)

  elsif path[0] == "file_store" && path[1] == "repo"
    entry = Chef::ChefFS::FileSystem.resolve_path(chef_fs, path[2..].join("/"))
    begin
      entry.read
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)
    end

  # /policy_groups/NAME/policies/POLICYNAME: return the revision of the given policy
  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 4
    # Just set or create the proper entry in the hash
    policy_group = get_json(to_chef_fs_path(path[0..1]), {})
    if !policy_group["policies"] || !policy_group["policies"][path[3]]
      raise ChefZero::DataStore::DataNotFoundError.new(path, entry)
    end

    # The policy group looks like:
    # {
    #   "policies": {
    #     "x": { "revision_id": "10" }
    #   }
    # }
    Chef::JSONCompat.to_json_pretty(policy_group["policies"][path[3]]["revision_id"])

  # GET [/organizations/ORG]/users/NAME -> /users/NAME
  # Manipulates members.json
  elsif is_org? && path[0] == "users" && path.length == 2
    if get_json("members.json", []).any? { |member| member["user"]["username"] == path[1] }
      "{}"
    else
      raise ChefZero::DataStore::DataNotFoundError.new(path)
    end

  # GET [/organizations/ORG]/association_requests/NAME -> /users/NAME
  # Manipulates invites.json
  elsif is_org? && path[0] == "association_requests" && path.length == 2
    if get_json("invites.json", []).any? { |member| member["user"]["username"] == path[1] }
      "{}"
    else
      raise ChefZero::DataStore::DataNotFoundError.new(path)
    end

  # GET /cookbooks/NAME/VERSION or /cookbook_artifacts/NAME/IDENTIFIER
  elsif %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 3
    with_entry(path) do |entry|
      cookbook_type = path[0]
      result = nil
      begin
        result = Chef::CookbookManifest.new(entry.chef_object, policy_mode: cookbook_type == "cookbook_artifacts").to_h
      rescue Chef::ChefFS::FileSystem::NotFoundError => e
        raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)
      end

      result.each_pair do |key, value|
        if value.is_a?(Array)
          value.each do |file|
            if file.is_a?(Hash) && file.key?("checksum")
              relative = ["file_store", "repo", cookbook_type]
              if chef_fs.versioned_cookbooks || cookbook_type == "cookbook_artifacts"
                relative << "#{path[1]}-#{path[2]}"
              else
                relative << path[1]
              end
              relative += file[:path].split("/")
              file["url"] = ChefZero::RestBase.build_uri(request.base_uri, relative)
            end
          end
        end
      end

      if cookbook_type == "cookbook_artifacts"
        result["metadata"] = result["metadata"].to_h
        result["metadata"].delete_if do |key, value|
          value == [] ||
            (value == {} && !%w{dependencies attributes recipes}.include?(key)) ||
            (value == "" && %w{source_url issues_url}.include?(key)) ||
            (value == false && key == "privacy")
        end
      end

      Chef::JSONCompat.to_json_pretty(result)
    end

  else
    with_entry(path) do |entry|

      entry.read
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)

    end
  end
end

#list(path) ⇒ Object 



483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 483

def list(path)
  if use_memory_store?(path)
    @memory_store.list(path)

  # LIST /policies
  elsif path == [ "policies" ]
    with_entry([ path[0] ]) do |policies|

      policies.children.map { |policy| policy.name[0..-6].rpartition("-")[0] }.uniq
    rescue Chef::ChefFS::FileSystem::NotFoundError
      []

    end

  # LIST /policies/POLICY/revisions
  elsif path[0] == "policies" && path[2] == "revisions" && path.length == 3
    with_entry([ path[0] ]) do |policies|
      # /policies:
      #   - a-1.0.0.json
      #   - a-1.0.1.json
      #   - b-2.0.0.json
      revisions = []
      policies.children.each do |policy|
        name, dash, revision = policy.name[0..-6].rpartition("-")
        revisions << revision if name == path[1]
      end
      raise ChefZero::DataStore::DataNotFoundError.new(path) if revisions.empty?

      revisions
    end

  elsif path[0] == "policy_groups" && path.length == 2
    with_entry(path) do |entry|
      [ "policies" ]
    end

  elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 3
    with_entry(path[0..1]) do |entry|
      policies = Chef::JSONCompat.parse(entry.read)["policies"] || {}
      policies.keys
    end

  elsif %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 1
    with_entry(path) do |entry|

      if path[0] == "cookbook_artifacts"
        entry.children.map { |child| child.name.rpartition("-")[0] }.uniq
      elsif chef_fs.versioned_cookbooks
        # /cookbooks/name-version -> /cookbooks/name
        entry.children.map { |child| split_name_version(child.name)[0] }.uniq
      else
        entry.children.map(&:name)
      end
    rescue Chef::ChefFS::FileSystem::NotFoundError
      # If the cookbooks dir doesn't exist, we have no cookbooks (not 404)
      []

    end

  elsif %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 2
    if chef_fs.versioned_cookbooks || path[0] == "cookbook_artifacts"
      result = with_entry([ path[0] ]) do |entry|
        # list /cookbooks/name = filter /cookbooks/name-version down to name
        entry.children.map { |child| split_name_version(child.name) }
          .select { |name, version| name == path[1] }
          .map { |name, version| version }
      end
      if result.empty?
        raise ChefZero::DataStore::DataNotFoundError.new(path)
      end

      result
    else
      # list /cookbooks/name = <single version>
      version = get_single_cookbook_version(path)
      [version]
    end

  else
    result = with_entry(path) do |entry|

      entry.children.map { |c| zero_filename(c) }.sort
    rescue Chef::ChefFS::FileSystem::NotFoundError => e
      # /cookbooks, /data, etc. never return 404
      if path_always_exists?(path)
        []
      else
        raise ChefZero::DataStore::DataNotFoundError.new(to_zero_path(e.entry), e)
      end

    end

    # Older versions of chef-zero do not understand policies and cookbook_artifacts,
    # don't give that stuff to them
    if path == [] && ChefZero::VERSION.to_f < 4.4
      result.reject! { |child| %w{policies policy_data cookbook_artifacts}.include?(child) }
    end
    result
  end
end

#publish_description ⇒ Object 



163
164
165
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 163

def publish_description
  "Reading and writing data to #{chef_fs.fs_description}"
end

#set(path, data, *options) ⇒ Object 



361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
 
# File 'lib/chef/chef_fs/chef_fs_data_store.rb', line 361

def set(path, data, *options)
  if use_memory_store?(path)
    @memory_store.set(path, data, *options)
  else
    unless data.is_a?(String)
      raise "set only works with strings: #{path} = #{data.inspect}"
    end

    # Write out the files!
    if %w{cookbooks cookbook_artifacts}.include?(path[0]) && path.length == 3
      write_cookbook(path, data, *options)

    # Handle /policy_groups/some_policy_group/policies/some_policy_name
    elsif path[0] == "policy_groups" && path[2] == "policies" && path.length == 4
      # Just set or create the proper entry in the hash
      update_json(to_chef_fs_path(path[0..1]), {}, *options) do |group|
        group["policies"] ||= {}
        group["policies"][path[3]] = { "revision_id" => Chef::JSONCompat.parse(data) }
        group
      end

    else
      with_parent_dir(path, *options) do |parent, name|
        child = parent.child(name)
        if child.exists?
          child.write(data)
        else
          parent.create_child(name, data)
        end
      end
    end
  end
end