Class: FSDB::Database

Inherits:
Object
  • Object
show all
Includes:
DirectoryIterators, Formats, PathUtilities
Defined in:
lib/fsdb/database.rb,
lib/fsdb/util.rb

Overview

A thread-safe, process-safe object database class which uses the native file system as its back end and allows multiple file formats.

Defined Under Namespace

Classes: AbortedTransaction, CacheEntry, CreateFileError, DirIsImmutableError, DirNotEmptyError, FormatError, MissingFileError, MissingObjectError, NotDirError, PathComponentError

Constant Summary collapse

MTIME_RESOLUTION =

Even when linux mounts FAT, the mtime granularity is 1 sec.

1.1
CLOCK_SKEW =

in seconds, adjust as needed for stability on NFS

0.0
DEFAULT_META_PREFIX =

Subclasses can change the defaults.

'..fsdb.meta.'
DEFAULT_LOCK_TYPE = 
:flock
LOCK_TYPES =

These must be methods of File.

[:flock]
FORMATS =

Subclasses can define their own list of formats, with specified search order

[TEXT_FORMAT, MARSHAL_FORMAT].freeze

Constants included from Formats

Formats::BINARY_FORMAT, Formats::DIR_FORMAT, Formats::DIR_LOAD, Formats::DIR_LOAD_FROM_PATH, Formats::DIR_PAT, Formats::HIDDEN_FILE_PAT, Formats::MARSHAL_FORMAT, Formats::TEXT_FORMAT, Formats::YAML_FORMAT

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from DirectoryIterators

#browse_dir, #browse_each_child, #delete_each_child, #edit_dir, #edit_each_child, #replace_each_child

Methods included from PathUtilities

#canonical, #canonical?, #directory?, #glob, #valid?, #validate

Constructor Details

#initialize(dir, opts = {}) ⇒ Database

Create a new database object that accesses dir. Makes sure that the directory exists on disk, but doesn’t create or open any other files. The opts hash can include:

:lock_type

:flock by default

:meta_prefix

'..fsdb.meta.' by default

:formats

nil by default, so the class’s FORMATS is used



140
141
142
143
144
145
146
147
148
149
150
151
152
153
 
# File 'lib/fsdb/database.rb', line 140

def initialize dir, opts = {}
  @dir = File.expand_path(dir)

  @lock_type = opts[:lock_type] || DEFAULT_LOCK_TYPE
  unless LOCK_TYPES.include? @lock_type
    raise "Unknown lock type: #{lock_type}"
  end
  
  @meta_prefix = opts[:meta_prefix] || DEFAULT_META_PREFIX

  @formats = opts[:formats]
  
  FileUtils.makedirs(@dir)
end

Class Attribute Details

.cache ⇒ Object (readonly)

:nodoc:



118
119
120
 
# File 'lib/fsdb/database.rb', line 118

def cache
  @cache
end

.cache_mutex ⇒ Object (readonly)

:nodoc:



118
119
120
 
# File 'lib/fsdb/database.rb', line 118

def cache_mutex
  @cache_mutex
end

Instance Attribute Details

#dir ⇒ Object (readonly)

The root directory of the db, to which paths are relative.



125
126
127
 
# File 'lib/fsdb/database.rb', line 125

def dir
  @dir
end

#lock_type ⇒ Object (readonly)

The lock type of the db, by default :flock.



128
129
130
 
# File 'lib/fsdb/database.rb', line 128

def lock_type
  @lock_type
end

Class Method Details

.[](path) ⇒ Object

Shortcut to create a new database at path.



156
157
158
 
# File 'lib/fsdb/database.rb', line 156

def Database.[](path)
  new(path)
end

.abort ⇒ Object

Same as #abort.

Raises:



435
 
# File 'lib/fsdb/database.rb', line 435

def self.abort; raise AbortedTransaction; end

Instance Method Details

#_get_file_id(abs_path) ⇒ Object

:nodoc:



189
190
191
192
 
# File 'lib/fsdb/database.rb', line 189

def _get_file_id(abs_path) # :nodoc:
  File.stat(abs_path) # just to generate the right exceptions
  abs_path # might not be unique, due to links, etc.
end

#abort ⇒ Object

Abort the current transaction (#browse, #edit, #replace, or #delete, roll back the state of the object, and return nil from the transaction.

In the #browse case, the only effect is to end the transaction.

Note that any exception that breaks out of the transaction will also abort the transaction, and be re-raised.

Raises:



432
 
# File 'lib/fsdb/database.rb', line 432

def abort;      raise AbortedTransaction; end

#absolute(path) ⇒ Object Also known as: absolute_path_to

Convert a relative path (relative to the db dir) to an absolute path.



176
177
178
179
180
181
182
 
# File 'lib/fsdb/database.rb', line 176

def absolute(path)
  abs_path = File.expand_path(File.join(@dir, path))
  if File.directory?(abs_path)
    abs_path << ?/ # prevent Errno::EINVAL on UFS
  end
  abs_path
end

#browse(path = "/") ⇒ Object

Browse the object. Yields the object to the caller’s block, and returns the value of the block.

Changes to the object are not persistent, but should be avoided (they will be seen by other threads, but only in the current process, and only until the cache is cleared). If you return the object from the block, or keep a reference to it in some other way, the object will no longer be protected from concurrent writers.



475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
 
# File 'lib/fsdb/database.rb', line 475

def browse(path = "/")                # :yields: object
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  
  ## put these outside method, and pass in params?
  do_when_first = proc do |cache_entry|
    raise if cache_entry.file_handle

    begin
      if PLATFORM_IS_WINDOWS_ME
        abs_path.sub!(/\/+$/, "")
      end
      f = File.open(abs_path, "r")
    rescue Errno::ENOENT
      raise MissingFileError
    rescue Errno::EINTR
      retry
    end

    cache_entry.file_handle = f
    f.lock_shared(@lock_type)
    identify_file_type(f, path, abs_path)
      ## could avoid if cache_object says so
    object = cache_object(f, cache_entry)
  end
  
  do_when_last = proc do |cache_entry|
    # last one out closes the file
    f = cache_entry.file_handle
    if f
      f.close
      cache_entry.file_handle = nil
    end
  end
  
  object_shared(file_id, do_when_first, do_when_last) do |cache_entry|
    object = cache_entry.just_gimme_the_damn_object!
    yield object if block_given?
  end
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    raise if File::CAN_OPEN_DIR
    raise unless File.directory?(abs_path) ### redundant!
    yield Formats::DIR_LOAD_FROM_PATH[abs_path] if block_given?
  end
  clear_entry(file_id)
  default_browse(path) {|x| yield x if block_given?}
rescue AbortedTransaction
rescue Errno::EACCES
  raise if File::CAN_OPEN_DIR
  raise unless File.directory?(abs_path)
  # on some platforms, opening a dir raises EACCESS
  yield Formats::DIR_LOAD_FROM_PATH[abs_path] if block_given?
end

#cache ⇒ Object 



121
 
# File 'lib/fsdb/database.rb', line 121

def cache; Database.cache; end

#cache_mutex ⇒ Object 



122
 
# File 'lib/fsdb/database.rb', line 122

def cache_mutex; Database.cache_mutex; end

#clear_cache ⇒ Object

Can be called occasionally to reduce memory footprint, esp. if cached objects are large and infrequently used.



269
270
271
272
273
274
275
 
# File 'lib/fsdb/database.rb', line 269

def clear_cache
  cache_mutex.synchronize do
    cache.delete_if do |file_id, cache_entry|
      cache_entry.unused?
    end
  end
end

#clear_entry(file_id) ⇒ Object

For housekeeping, so that stale entries don’t result in unused, but uncollectable, CacheEntry objects.



258
259
260
261
262
263
264
265
 
# File 'lib/fsdb/database.rb', line 258

def clear_entry(file_id)
  if file_id
    cache_mutex.synchronize do
      cache_entry = cache[file_id]
      cache.delete(file_id) if cache_entry and cache_entry.unused?
    end
  end
end

#default_browse(path) ⇒ Object

Called when #browse doesn’t find anything at the path. The original caller’s block is available to be yielded to.



442
443
444
 
# File 'lib/fsdb/database.rb', line 442

def default_browse(path)
  object_missing(path) {|x| yield x}
end

#default_edit(path) ⇒ Object

Called when #edit doesn’t find anything at the path. The original caller’s block is available to be yielded to.



448
449
450
 
# File 'lib/fsdb/database.rb', line 448

def default_edit(path)
  object_missing(path) {|x| yield x}
end

#default_fetch(path) ⇒ Object

Called when #fetch doesn’t find anything at the path. Default definition just returns nil.



460
 
# File 'lib/fsdb/database.rb', line 460

def default_fetch(path); nil; end

#delete(path, load = true) ⇒ Object

Delete the object from the db. If a block is given, yields the object (or nil if none) before deleting it from the db (but before releasing the lock on the path), and returns the value of the block. Otherwise, just returns the object (or nil, if none). Raises DirNotEmptyError if path refers to a non-empty dir. If the dir is empty, it is deleted, and the returned value is true. The block is not yielded to. If the load argument is false, delete the object from the db without loading it or yielding, returning true.



664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
 
# File 'lib/fsdb/database.rb', line 664

def delete(path, load=true)                  # :yields: object
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  delete_later = false
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      if load
        object = cache_object(f, cache_entry)
        result = block_given? ? (yield object) : object
      else
        result = true
      end
      if File::CAN_DELETE_OPEN_FILE
        File.delete(abs_path)
      else
        delete_later = true
      end
      cache_entry.stale!
      del_version_of(f)
      result
    end
  end
rescue DirIsImmutableError
  begin
    Dir.delete(abs_path)
  rescue Errno::ENOENT
    # Someone else got it first.
  end
  true
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  if File.symlink?(abs_path) # get_file_id fails if target deleted
    File.delete(abs_path) rescue nil
  end
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    Dir.delete(abs_path)
  end
  nil
rescue Errno::ENOTEMPTY
  raise DirNotEmptyError, "Directory not empty - #{path} in #{inspect}"
rescue Errno::EACCES
  raise if File::CAN_OPEN_DIR
  raise unless File.directory?(abs_path)
  # on some platforms, opening a dir raises EACCESS
  Dir.delete(abs_path)
  true
rescue AbortedTransaction
ensure
  if delete_later
    begin
      File.delete(abs_path) rescue Dir.delete(abs_path)
    rescue Errno::ENOENT
    end
  end
  clear_entry(file_id)
end

#dump(object, f) ⇒ Object

Writes object to f (must be open for writing).



763
764
765
 
# File 'lib/fsdb/database.rb', line 763

def dump(object, f)
  f.format.dump(object, f)
end

#edit(path = "/") ⇒ Object

Edit the object in place. Changes to the yielded object made within the caller’s block become persistent. Returns the value of the block. Note that assigning to the block argument variable does not change the state of the object. Use destructive methods on the object.



536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
 
# File 'lib/fsdb/database.rb', line 536

def edit(path = "/")
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      object = cache_object(f, cache_entry)
      result = yield object if block_given?
      dump(object, f)
      cache_entry.update(f.mtime, inc_version_of(f, cache_entry), object)
      result
    end
  end
rescue DirIsImmutableError
  raise DirIsImmutableError, "Cannot edit dir #{path} in #{inspect}"
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  raise DirIsImmutableError if PLATFORM_IS_WINDOWS_ME and
          File.directory?(abs_path)
  clear_entry(file_id)
  default_edit(path) {|x| yield x if block_given?}
rescue AbortedTransaction
  clear_entry(file_id) # The cached object may have edits which are not valid.
  nil
rescue Exception
  clear_entry(file_id)
  raise
end

#fetch(path = "/") ⇒ Object Also known as: []

Fetch a copy of the object at the path for private use by the current thread/process. (The copy is a deep copy.)

Note that this is inherently less efficient than #browse, because #browse leaves the object in the cache, but, for safety, #fetch can only return a copy and wipe the cache, since the copy is going to be used outside of any transaction. Subsequent transactions will have to read the object again.



729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
 
# File 'lib/fsdb/database.rb', line 729

def fetch(path = "/")
  abs_path = absolute(path)
  file_id = get_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_read_lock(path) do |f|
      object = cache_object(f, cache_entry)
      cache_entry.stale!
      object
    end
  end
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue MissingFileError
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    return Formats::DIR_LOAD_FROM_PATH[abs_path]
  end
  clear_entry(file_id)
  default_fetch(path)
rescue Errno::EACCES
  raise if File::CAN_OPEN_DIR
  raise unless File.directory?(abs_path)
  # on some platforms, opening a dir raises EACCESS
  return Formats::DIR_LOAD_FROM_PATH[abs_path]
end

#find_format(path, abs_path = absolute(path)) ⇒ Object 



791
792
793
794
795
796
797
798
 
# File 'lib/fsdb/database.rb', line 791

def find_format(path, abs_path = absolute(path))
  if DIR_FORMAT === abs_path
    DIR_FORMAT
  else
    path = path.sub(/^\//, "") # So that db['foo'] and db['/foo'] are same
    formats.find {|fmt| fmt === path}
  end
end

#formats ⇒ Object 



770
771
772
 
# File 'lib/fsdb/database.rb', line 770

def formats
  @formats || self.class::FORMATS
end

#formats=(fmts) ⇒ Object 



774
775
776
 
# File 'lib/fsdb/database.rb', line 774

def formats=(fmts)
  @formats = fmts
end

#get_file_id(abs_path) ⇒ Object

Convert an absolute path to a unique key for the cache, raising MissingFileError if the file does not exist.



202
203
204
205
206
207
208
209
210
211
 
# File 'lib/fsdb/database.rb', line 202

def get_file_id(abs_path)
  _get_file_id(abs_path)
rescue Errno::ENOTDIR
  # db['x'] = 0; db.edit 'x/' do end
  raise NotDirError
rescue Errno::ENOENT
  raise MissingFileError, "Cannot find file at #{abs_path}"
rescue Errno::EINTR
  retry
end

#identify_file_type(f, path, abs_path = absolute(path)) ⇒ Object

path is relative to the database, and initial ‘/’ is ignored



782
783
784
785
786
787
788
789
 
# File 'lib/fsdb/database.rb', line 782

def identify_file_type(f, path, abs_path = absolute(path))
  format = find_format(path, abs_path)
  unless format
    raise FormatError, "No format found for path #{path.inspect}"
  end
  f.binmode if format.binary?
  f.format = format
end

#insert(path, object) ⇒ Object Also known as: []=

Insert the object, replacing anything at the path. Returns the object. (The object remains a local copy, distinct from the one which will be returned when accessing the path through database transactions.)

If path ends in “/”, then object is treated as a collection of key-value pairs, and each value is inserted at the corresponding key under path. (You can omit the “/” if the dir already exists.) is this still true?



622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
 
# File 'lib/fsdb/database.rb', line 622

def insert(path, object)
  abs_path = absolute(path)
  file_id = make_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      dump(object, f)
      cache_entry.update(f.mtime, inc_version_of(f, cache_entry), object)
      object
    end
  end
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue FormatError
  File.delete(abs_path)
  raise
rescue PathComponentError
  raise PathComponentError, "Some component of #{path} in #{inspect} " +
      "already exists and is not a directory"
rescue CreateFileError
  if PLATFORM_IS_WINDOWS_ME and /\/$/ =~ path
    raise DirIsImmutableError
  else
    raise CreateFileError, "Cannot create file at #{path} in #{inspect}"
  end
rescue MissingFileError
  raise DirIsImmutableError if PLATFORM_IS_WINDOWS_ME
ensure
  clear_entry(file_id) # no one else can get this copy of object
end

#inspect ⇒ Object 



173
 
# File 'lib/fsdb/database.rb', line 173

def inspect; "#<#{self.class}:#{dir}>"; end

#link(old_name, new_name) ⇒ Object

Create a hard link, using File.link. The names are relative to the database’s path.



130
131
132
 
# File 'lib/fsdb/util.rb', line 130

def link(old_name, new_name)
  File.link(absolute(old_name), absolute(new_name))
end

#load(f) ⇒ Object

Returns object read from f (must be open for reading).



758
759
760
 
# File 'lib/fsdb/database.rb', line 758

def load(f)
  f.format.load(f)
end

#make_file_id(abs_path) ⇒ Object

Convert an absolute path to a unique key for the cache, creating the file if it does not exist. Raises CreateFileError if it can’t be created.



227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
 
# File 'lib/fsdb/database.rb', line 227

def make_file_id(abs_path)
  dirname = File.dirname(abs_path)
  begin
    FileUtils.makedirs(dirname)
  rescue Errno::EEXIST
    raise PathComponentError
  end
  begin
    _get_file_id(abs_path)
  rescue Errno::EINTR
    retry
  end
rescue Errno::ENOTDIR
  # db['x'] = 0; db.replace 'x/' do end
  raise NotDirError
rescue Errno::ENOENT
  begin
    File.open(abs_path, "w") do |f|
      _get_file_id(abs_path)
    end
  rescue Errno::EISDIR
    raise DirIsImmutableError
  rescue Errno::EINVAL
    raise DirIsImmutableError # for windows
  rescue StandardError
    raise CreateFileError
  end
end

#object_missing(path) ⇒ Object

The default behavior of both #default_edit and #default_browse. Raises MissingObjectError by default, but it can yield to the original block.

Raises:



454
455
456
 
# File 'lib/fsdb/database.rb', line 454

def object_missing(path)
  raise MissingObjectError, "No object at #{path} in #{inspect}"
end

#replace(path) ⇒ Object

Replace the yielded object (or nil) with the return value of the block. Returns the object that was replaced. No object need exist at path.

Use replace instead of edit when accessing db over a drb connection. Use replace instead of insert if the path needs to be protected while the object is prepared for insertion.

Note that (unlike #edit) destructive methods on the object do not persistently change the state of the object, unless the object is the return value of the block.



575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
 
# File 'lib/fsdb/database.rb', line 575

def replace(path)
  abs_path = absolute(path)
  file_id = make_file_id(abs_path)
  object_exclusive file_id do |cache_entry|
    open_write_lock(path) do |f|
      old_object = f.stat.zero? ? nil : cache_object(f, cache_entry)
      object = yield old_object if block_given?
      dump(object, f)
      cache_entry.update(f.mtime, inc_version_of(f, cache_entry), object)
      old_object
    end
  end
rescue DirIsImmutableError
  raise DirIsImmutableError, "Cannot replace dir #{path} in #{inspect}"
rescue NotDirError
  raise NotDirError, "Not a directory - #{path} in #{inspect}"
rescue AbortedTransaction
  clear_entry(file_id) # The cached object may have edits which are not valid.
  nil
rescue FormatError
  clear_entry(file_id)
  File.delete(abs_path)
  raise
rescue PathComponentError
  raise PathComponentError, "Some component of #{path} in #{inspect} " +
      "already exists and is not a directory"
rescue CreateFileError
  raise CreateFileError, "Cannot create file at #{path} in #{inspect}"
rescue MissingFileError
  if PLATFORM_IS_WINDOWS_ME and File.directory?(abs_path)
    raise DirIsImmutableError
  else
    raise NotDirError
  end
rescue Exception
  clear_entry(file_id)
  raise
end

#subdb(path) ⇒ Object

Create a new database object that accesses path relative to the database directory. A process can have any number of dbs accessing overlapping dirs. The cost of creating an additional db is very low; its state is just the dir and some options. Caching is done in structures owned by the Database class itself.



165
166
167
168
169
170
171
 
# File 'lib/fsdb/database.rb', line 165

def subdb path
  self.class.new(File.join(@dir, path),
    :lock_type => @lock_type,
    :meta_prefix => @meta_prefix,
    :formats => @formats && @formats.dup
  )
end

#symlink(old_name, new_name) ⇒ Object

Create a symbolic link, using File.symlink. The names are relative to the database’s path.



136
137
138
 
# File 'lib/fsdb/util.rb', line 136

def symlink(old_name, new_name)
  File.symlink(absolute(old_name), absolute(new_name))
end