Class: FSDB::Database

Inherits:
Object
  • Object
show all
Includes:
DirectoryIterators, Formats, PathUtilities
Defined in:
lib/fsdb/database.rb,
lib/fsdb/util.rb

Overview

A thread-safe, process-safe object database class which uses the native file system as its back end and allows multiple file formats.

Defined Under Namespace

Classes: AbortedTransaction, CacheEntry, CreateFileError, DirIsImmutableError, DirNotEmptyError, FormatError, MissingFileError, MissingObjectError, NotDirError, PathComponentError

Constant Summary collapse

MTIME_RESOLUTION =

Even when linux mounts FAT, the mtime granularity is 1 sec.

1.1
CLOCK_SKEW =

in seconds, adjust as needed for stability on NFS

0.0
DEFAULT_META_PREFIX =

Subclasses can change the defaults.

'..fsdb.meta.'
DEFAULT_LOCK_TYPE = 
:flock
LOCK_TYPES =

These must be methods of File.

[:flock]
FORMATS =

Subclasses can define their own list of formats, with specified search order

[TEXT_FORMAT, MARSHAL_FORMAT].freeze

Constants included from Formats

Formats::BINARY_FORMAT, Formats::DIR_FORMAT, Formats::DIR_LOAD, Formats::DIR_LOAD_FROM_PATH, Formats::DIR_PAT, Formats::HIDDEN_FILE_PAT, Formats::MARSHAL_FORMAT, Formats::TEXT_FORMAT, Formats::YAML_FORMAT

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from DirectoryIterators

#browse_dir, #browse_each_child, #delete_each_child, #edit_dir, #edit_each_child, #replace_each_child

Methods included from PathUtilities

#canonical, #canonical?, #directory?, #glob, #valid?, #validate

Constructor Details

#initialize(dir, opts = {}) ⇒ Database

Create a new database object that accesses dir. Makes sure that the directory exists on disk, but doesn’t create or open any other files. The opts hash can include:

:lock_type

:flock by default

:meta_prefix

'..fsdb.meta.' by default

:formats

nil by default, so the class’s FORMATS is used



143
144
145
146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/fsdb/database.rb', line 143

def initialize dir, opts = {}
  @dir = File.expand_path(dir)

  @lock_type = opts[:lock_type] || DEFAULT_LOCK_TYPE
  unless LOCK_TYPES.include? @lock_type
    raise "Unknown lock type: #{lock_type}"
  end
  
  @meta_prefix = opts[:meta_prefix] || DEFAULT_META_PREFIX

  @formats = opts[:formats]
  
  FileUtils.makedirs(@dir)
end

Class Attribute Details

.cache ⇒ Object (readonly)

:nodoc:



121
122
123
 
# File 'lib/fsdb/database.rb', line 121

def cache
  @cache
end

.cache_mutex ⇒ Object (readonly)

:nodoc:



121
122
123
 
# File 'lib/fsdb/database.rb', line 121

def cache_mutex
  @cache_mutex
end

Instance Attribute Details

#dir ⇒ Object (readonly)

The root directory of the db, to which paths are relative.



128
129
130
 
# File 'lib/fsdb/database.rb', line 128

def dir
  @dir
end

#lock_type ⇒ Object (readonly)

The lock type of the db, by default :flock.



131
132
133
 
# File 'lib/fsdb/database.rb', line 131

def lock_type
  @lock_type
end

Class Method Details

.[](path) ⇒ Object

Shortcut to create a new database at path.



159
160
161
 
# File 'lib/fsdb/database.rb', line 159

def Database.[](path)
  new(path)
end

.abort ⇒ Object

Same as #abort.

Raises:



445
 
# File 'lib/fsdb/database.rb', line 445

def self.abort; raise AbortedTransaction; end

Instance Method Details

#_get_file_id(abs_path) ⇒ Object

:nodoc:



195
196
197
198
 
# File 'lib/fsdb/database.rb', line 195

def _get_file_id(abs_path) # :nodoc:
  File.stat(abs_path) # just to generate the right exceptions
  abs_path # might not be unique, due to links, etc.
end

#abort ⇒ Object

Abort the current transaction (#browse, #edit, #replace, or #delete, roll back the state of the object, and return nil from the transaction.

In the #browse case, the only effect is to end the transaction.

Note that any exception that breaks out of the transaction will also abort the transaction, and be re-raised.

Raises:



442
 
# File 'lib/fsdb/database.rb', line 442

def abort;      raise AbortedTransaction; end

#absolute(path) ⇒ Object Also known as: absolute_path_to

Convert a relative path (relative to the db dir) to an absolute path. A directory path will have ‘/’ appended to it.



182
183
184
185
186
187
188
 
# File 'lib/fsdb/database.rb', line 182

def absolute(path)
  abs_path = File.expand_path(File.join(@dir, path))
  if File.directory?(abs_path)
    abs_path << "/" # prevent Errno::EINVAL on UFS
  end
  abs_path
end

#browse(path = "/") ⇒ Object

Browse the object. Yields the object to the caller’s block, and returns the value of the block.

Changes to the object are not persistent, but should be avoided (they will be seen by other threads, but only in the current process, and only until the cache is cleared). If you return the object from the block, or keep a reference to it in some other way, the object will no longer be protected from concurrent writers.



485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
 
# File 'lib/fsdb/database.rb', line 485

def browse(path = "/")                # :yields: object
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  
  ## put these outside method, and pass in params?
  do_when_first = proc do |cache_entry|
    raise if cache_entry.file_handle

    begin
      if PLATFORM_IS_WINDOWS_ME
        abs_path.sub!(/\/+$/, "")
      end
      f = File.open(abs_path, "r")
    rescue Errno::ENOENT
      raise MissingFileError
    rescue Errno::EINTR
      retry
    end

    cache_entry.file_handle = f
    f.lock_shared_fsdb(@lock_type)
    identify_file_type(f, path, abs_path)
      ## could avoid if cache_object says so
    object = cache_object(f, cache_entry)
  end
  
  do_when_last = proc do |cache_entry|
    # last one out closes the file
    f = cache_entry.file_handle
    if f
      f.close
      cache_entry.file_handle = nil
    end
  end
  
  object_shared(file_id, do_when_first, do_when_last) do |cache_entry|
    object = cache_entry.just_gimme_the_damn_object!
    yield object if block_given?
  end
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    raise if File::CAN_OPEN_DIR
    raise unless File.directory?(abs_path) ### redundant!
    yield Formats::DIR_LOAD_FROM_PATH[abs_path] if block_given?
  end
  clear_entry(file_id)
  default_browse(path) {|x| yield x if block_given?}
rescue AbortedTransaction
rescue Errno::EACCES
  raise if File::CAN_OPEN_DIR
  raise unless File.directory?(abs_path)
  # on some platforms, opening a dir raises EACCESS
  yield Formats::DIR_LOAD_FROM_PATH[abs_path] if block_given?
end

#cache ⇒ Object 



124
 
# File 'lib/fsdb/database.rb', line 124

def cache; Database.cache; end

#cache_mutex ⇒ Object 



125
 
# File 'lib/fsdb/database.rb', line 125

def cache_mutex; Database.cache_mutex; end

#clear_cache ⇒ Object

Can be called occasionally to reduce memory footprint, esp. if cached objects are large and infrequently used.



275
276
277
278
279
280
281
 
# File 'lib/fsdb/database.rb', line 275

def clear_cache
  cache_mutex.synchronize do
    cache.delete_if do |file_id, cache_entry|
      cache_entry.unused?
    end
  end
end

#clear_entry(file_id) ⇒ Object

For housekeeping, so that stale entries don’t result in unused, but uncollectable, CacheEntry objects.



264
265
266
267
268
269
270
271
 
# File 'lib/fsdb/database.rb', line 264

def clear_entry(file_id)
  if file_id
    cache_mutex.synchronize do
      cache_entry = cache[file_id]
      cache.delete(file_id) if cache_entry and cache_entry.unused?
    end
  end
end

#default_browse(path) ⇒ Object

Called when #browse doesn’t find anything at the path. The original caller’s block is available to be yielded to.



452
453
454
 
# File 'lib/fsdb/database.rb', line 452

def default_browse(path)
  object_missing(path) {|x| yield x}
end

#default_edit(path) ⇒ Object

Called when #edit doesn’t find anything at the path. The original caller’s block is available to be yielded to.



458
459
460
 
# File 'lib/fsdb/database.rb', line 458

def default_edit(path)
  object_missing(path) {|x| yield x}
end

#default_fetch(path) ⇒ Object

Called when #fetch doesn’t find anything at the path. Default definition just returns nil.



470
 
# File 'lib/fsdb/database.rb', line 470

def default_fetch(path); nil; end

#delete(path, load = true) ⇒ Object

Delete the object from the db. If a block is given, yields the object (or nil if none) before deleting it from the db (but before releasing the lock on the path), and returns the value of the block. Otherwise, just returns the object (or nil, if none). Raises DirNotEmptyError if path refers to a non-empty dir. If the dir is empty, it is deleted, and the returned value is true. The block is not yielded to. If the load argument is false, delete the object from the db without loading it or yielding, returning true.



670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
 
# File 'lib/fsdb/database.rb', line 670

def delete(path, load=true)                  # :yields: object
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  delete_later = false
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      if load
        object = cache_object(f, cache_entry)
        result = block_given? ? (yield object) : object
      else
        result = true
      end
      if File::CAN_DELETE_OPEN_FILE
        File.delete(abs_path)
      else
        delete_later = true
      end
      cache_entry.stale!
      del_version_of(f)
      result
    end
  end
rescue DirIsImmutableError
  begin
    Dir.delete(abs_path)
  rescue Errno::ENOENT
    # Someone else got it first.
  end
  true
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  if File.symlink?(abs_path) # get_file_id fails if target deleted
    File.delete(abs_path) rescue nil
  end
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    Dir.delete(abs_path)
  end
  nil
rescue Errno::ENOTEMPTY
  raise DirNotEmptyError, "Directory not empty - #{path} in #{inspect}"
rescue Errno::EACCES
  raise if File::CAN_OPEN_DIR
  raise unless File.directory?(abs_path)
  # on some platforms, opening a dir raises EACCESS
  Dir.delete(abs_path)
  true
rescue AbortedTransaction
ensure
  if delete_later
    begin
      File.delete(abs_path) rescue Dir.delete(abs_path)
    rescue Errno::ENOENT
    end
  end
  clear_entry(file_id)
end

#dump(object, f) ⇒ Object

Writes object to f (must be open for writing).



771
772
773
 
# File 'lib/fsdb/database.rb', line 771

def dump(object, f)
  f.format.dump(object, f)
end

#edit(path = "/") ⇒ Object

Edit the object in place. Changes to the yielded object made within the caller’s block become persistent. Returns the value of the block. Note that assigning to the block argument variable does not change the state of the object. Use destructive methods on the object.



546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
 
# File 'lib/fsdb/database.rb', line 546

def edit(path = "/")
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      object = cache_object(f, cache_entry)
      result = yield object if block_given?
      dump(object, f)
      cache_entry.update(f.mtime, inc_version_of(f, cache_entry), object)
      result
    end
  end
rescue DirIsImmutableError
  raise DirIsImmutableError, "Cannot edit dir #{path} in #{inspect}"
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  raise DirIsImmutableError if PLATFORM_IS_WINDOWS_ME and
          File.directory?(abs_path)
  clear_entry(file_id)
  default_edit(path) {|x| yield x if block_given?}
rescue AbortedTransaction
  clear_entry(file_id) # The cached object may have edits which are not valid.
  nil
rescue Exception
  clear_entry(file_id)
  raise
end

#fetch(path = "/") ⇒ Object Also known as: []

Fetch a copy of the object at the path for private use by the current thread/process. (The copy is a deep copy.)

Note that this is inherently less efficient than #browse, because #browse leaves the object in the cache, but, for safety, #fetch can only return a copy and wipe the cache, since the copy is going to be used outside of any transaction. Subsequent transactions will have to read the object again.



735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
 
# File 'lib/fsdb/database.rb', line 735

def fetch(path = "/")
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_read_lock(path) do |f|
      object = cache_object(f, cache_entry)
      cache_entry.stale!
      object
    end
  end
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    return Formats::DIR_LOAD_FROM_PATH[abs_path]
  end
  clear_entry(file_id)
  default_fetch(path)
rescue Errno::EACCES
  raise if File::CAN_OPEN_DIR
  raise unless File.directory?(abs_path)
  # on some platforms, opening a dir raises EACCESS
  return Formats::DIR_LOAD_FROM_PATH[abs_path]
ensure
  clear_entry(file_id) # the entry was recently marked stale anyway
end

#find_format(path, abs_path = absolute(path)) ⇒ Object 



799
800
801
802
803
804
805
806
 
# File 'lib/fsdb/database.rb', line 799

def find_format(path, abs_path = absolute(path))
  if DIR_FORMAT === abs_path
    DIR_FORMAT
  else
    path = path.sub(/^\//, "") # So that db['foo'] and db['/foo'] are same
    formats.find {|fmt| fmt === path}
  end
end

#formats ⇒ Object 



778
779
780
 
# File 'lib/fsdb/database.rb', line 778

def formats
  @formats || self.class::FORMATS
end

#formats=(fmts) ⇒ Object 



782
783
784
 
# File 'lib/fsdb/database.rb', line 782

def formats=(fmts)
  @formats = fmts
end

#get_file_id(abs_path) ⇒ Object

Convert an absolute path to a unique key for the cache, raising MissingFileError if the file does not exist.



208
209
210
211
212
213
214
215
216
217
 
# File 'lib/fsdb/database.rb', line 208

def get_file_id(abs_path)
  _get_file_id(abs_path)
rescue Errno::ENOTDIR
  # db['x'] = 0; db.edit 'x/' do end
  raise NotDirError
rescue Errno::ENOENT
  raise MissingFileError, "Cannot find file at #{abs_path}"
rescue Errno::EINTR
  retry
end

#identify_file_type(f, path, abs_path = absolute(path)) ⇒ Object

path is relative to the database, and initial ‘/’ is ignored



790
791
792
793
794
795
796
797
 
# File 'lib/fsdb/database.rb', line 790

def identify_file_type(f, path, abs_path = absolute(path))
  format = find_format(path, abs_path)
  unless format
    raise FormatError, "No format found for path #{path.inspect}"
  end
  f.binmode if format.binary?
  f.format = format
end

#insert(path, object) ⇒ Object Also known as: []=

Insert the object, replacing anything at the path. Returns the object. (The object remains a local copy, distinct from the one which will be returned when accessing the path through database transactions.)



627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
 
# File 'lib/fsdb/database.rb', line 627

def insert(path, object)
  abs_path = absolute(path)
  file_id = make_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      dump(object, f)
      inc_version_of(f, cache_entry)
      cache_entry.stale!
      object
    end
  end
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue FormatError
  File.delete(abs_path)
  raise
rescue PathComponentError
  raise PathComponentError, "Some component of #{path} in #{inspect} " +
      "already exists and is not a directory"
rescue CreateFileError
  if PLATFORM_IS_WINDOWS_ME and /\/$/ =~ path
    raise DirIsImmutableError
  else
    raise CreateFileError, "Cannot create file at #{path} in #{inspect}"
  end
rescue MissingFileError
  raise DirIsImmutableError if PLATFORM_IS_WINDOWS_ME
ensure
  clear_entry(file_id) # the entry was recently marked stale anyway
end

#inspect ⇒ Object 



178
 
# File 'lib/fsdb/database.rb', line 178

def inspect; "#<#{self.class}:#{dir}>"; end

#link(old_name, new_name) ⇒ Object

Create a hard link, using File.link. The names are relative to the database’s path.



130
131
132
 
# File 'lib/fsdb/util.rb', line 130

def link(old_name, new_name)
  File.link(absolute(old_name), absolute(new_name))
end

#load(f) ⇒ Object

Returns object read from f (must be open for reading).



766
767
768
 
# File 'lib/fsdb/database.rb', line 766

def load(f)
  f.format.load(f)
end

#make_file_id(abs_path) ⇒ Object

Convert an absolute path to a unique key for the cache, creating the file if it does not exist. Raises CreateFileError if it can’t be created.



233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
 
# File 'lib/fsdb/database.rb', line 233

def make_file_id(abs_path)
  dirname = File.dirname(abs_path)
  begin
    FileUtils.makedirs(dirname)
  rescue Errno::EEXIST
    raise PathComponentError
  end
  begin
    _get_file_id(abs_path)
  rescue Errno::EINTR
    retry
  end
rescue Errno::ENOTDIR
  # db['x'] = 0; db.replace 'x/' do end
  raise NotDirError
rescue Errno::ENOENT
  begin
    File.open(abs_path, "w") do |f|
      _get_file_id(abs_path)
    end
  rescue Errno::EISDIR
    raise DirIsImmutableError
  rescue Errno::EINVAL
    raise DirIsImmutableError # for windows
  rescue StandardError
    raise CreateFileError
  end
end

#object_missing(path) ⇒ Object

The default behavior of both #default_edit and #default_browse. Raises MissingObjectError by default, but it can yield to the original block.

Raises:



464
465
466
 
# File 'lib/fsdb/database.rb', line 464

def object_missing(path)
  raise MissingObjectError, "No object at #{path} in #{inspect}"
end

#replace(path) ⇒ Object

Replace the yielded object (or nil) with the return value of the block. Returns the object that was replaced. No object need exist at path.

Use replace instead of edit when accessing db over a drb connection. Use replace instead of insert if the path needs to be protected while the object is prepared for insertion.

Note that (unlike #edit) destructive methods on the object do not persistently change the state of the object, unless the object is the return value of the block.



585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
 
# File 'lib/fsdb/database.rb', line 585

def replace(path)
  abs_path = absolute(path)
  file_id = make_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      old_object = f.stat.zero? ? nil : cache_object(f, cache_entry)
      object = yield old_object if block_given?
      dump(object, f)
      cache_entry.update(f.mtime, inc_version_of(f, cache_entry), object)
      old_object
    end
  end
rescue DirIsImmutableError
  raise DirIsImmutableError, "Cannot replace dir #{path} in #{inspect}"
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue AbortedTransaction
  clear_entry(file_id) # The cached object may have edits which are not valid.
  nil
rescue FormatError
  clear_entry(file_id)
  File.delete(abs_path)
  raise
rescue PathComponentError
  raise PathComponentError, "Some component of #{path} in #{inspect} " +
      "already exists and is not a directory"
rescue CreateFileError
  raise CreateFileError, "Cannot create file at #{path} in #{inspect}"
rescue MissingFileError
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    raise DirIsImmutableError
  else
    raise NotDirError
  end
rescue Exception
  clear_entry(file_id)
  raise
end

#subdb(path) ⇒ Object

Create a new database object that accesses path relative to the database directory. A process can have any number of dbs accessing overlapping dirs. The FSDB concurrency protections apply to a file regardless of which db is used to access it. The cost of creating an additional db is very low; its state is just the dir and some options. Caching is done in structures owned by the Database class itself.



170
171
172
173
174
175
176
 
# File 'lib/fsdb/database.rb', line 170

def subdb path
  self.class.new(File.join(@dir, path),
    :lock_type => @lock_type,
    :meta_prefix => @meta_prefix,
    :formats => @formats && @formats.dup
  )
end

#symlink(old_name, new_name) ⇒ Object

Create a symbolic link, using File.symlink. The names are relative to the database’s path.



136
137
138
 
# File 'lib/fsdb/util.rb', line 136

def symlink(old_name, new_name)
  File.symlink(absolute(old_name), absolute(new_name))
end