Class: Pho::FileManagement::AbstractFileManager

Inherits:
Object
  • Object
show all
Defined in:
lib/pho/file_management.rb

Overview

Base class for the management of collections of files

A collection is considered to be a directory (and sub-directory) of files. Depending on the type of collection manager used (e.g. Pho::FileManagement::FileManager or Pho::FileManagement::RDFManager) these may be collections of any kind of file, or just RDF serialisations.

The basic mechanism has support for:

  • tracking whether a file has been successfully loaded into a store (or not)

  • identifying all new files, just those that have been successfully stored, or failed

  • identifying files that have been changed locally since last stored

  • storage of any of these classes of files either by uploading into the contentbox or storing in the metabox

  • resumable uploads

  • retrying uploads

  • traversal across a directory structure to manage a complete set of files

Files are tracked locally by keeping some hidden files in the same directory system as the files being tracked. These are currently all managed in a hidden directory (“.pho”) that is automatically created. To access error messages from uploads look for the corresponding “fail file” for the relevant file. E.g. /foo/file.txt will be tracked in /foo/.pho, and will have either an OK file (/foo/.pho/file.txt.ok) or a fail file (/foo/.pho/file.txt.fail).

Direct Known Subclasses

FileManager, RDFManager

Constant Summary collapse

OK = 
"ok"
FAIL = 
"fail"
TRACKING_DIR = 
".pho"

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(store, dir, ok_suffix = OK, fail_suffix = FAIL) ⇒ AbstractFileManager

Returns a new instance of AbstractFileManager.



38
39
40
41
42
43
 
# File 'lib/pho/file_management.rb', line 38

def initialize(store, dir, ok_suffix=OK, fail_suffix=FAIL)
  @store = store
  @dir = dir
  @ok_suffix = ok_suffix
  @fail_suffix = fail_suffix
end

Instance Attribute Details

#dir ⇒ Object (readonly)

Returns the value of attribute dir.



31
32
33
 
# File 'lib/pho/file_management.rb', line 31

def dir
  @dir
end

#store(recursive = false) ⇒ Object (readonly)

Store all files that match the file name in directory



46
47
48
 
# File 'lib/pho/file_management.rb', line 46

def store
  @store
end

Instance Method Details

#changed?(file) ⇒ Boolean

Has this file changed (i.e have a newer mtime) than its tracking file Allows files that have been updated in a collection to be easily identified New files (i.e untracked) are always considered to be changed

Returns:

  • (Boolean) 


106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
 
# File 'lib/pho/file_management.rb', line 106

def changed?(file)
  if !stored?(file)
    return true
  end
  ok_file = get_ok_file_for(file)
  fail_file = get_fail_file_for(file)
  f = File.new(file)        
  if ( 
      ( File.exists?(ok_file) && f.stat.mtime > File.new(ok_file).stat.mtime ) or
      ( File.exists?(fail_file) && f.stat.mtime > File.new(fail_file).stat.mtime )
     )
    return true
  end        
  return false        
end

#changed_files(recursive = false) ⇒ Object 



130
131
132
133
134
135
136
 
# File 'lib/pho/file_management.rb', line 130

def changed_files(recursive=false)
  changed = Array.new
  list(recursive).each do |file|
    changed << file if changed?(file)
  end
  return changed
end

#create_tracking_dir(filename) ⇒ Object 



190
191
192
193
 
# File 'lib/pho/file_management.rb', line 190

def create_tracking_dir(filename)
  path = filename.split("/")[0..-2].join("/")  
  Dir.mkdir("#{path}/#{TRACKING_DIR}") unless File.exists?("#{path}/#{TRACKING_DIR}")
end

#failures(recursive = false) ⇒ Object

List failures



148
149
150
151
152
153
154
 
# File 'lib/pho/file_management.rb', line 148

def failures(recursive=false)
  fails = Array.new
  list(recursive).each do |file|
    fails << file if File.exists?( get_fail_file_for(file) )
  end
  return fails
end

#get_fail_file_for(filename) ⇒ Object 



176
177
178
179
180
181
 
# File 'lib/pho/file_management.rb', line 176

def get_fail_file_for(filename)
  relative_path = filename.gsub(@dir, "")
  base = File.basename(filename)
  relative_path = relative_path.gsub(base, "#{TRACKING_DIR}/#{base}")
  return "#{@dir}#{relative_path}.#{@fail_suffix}"              
end

#get_ok_file_for(filename) ⇒ Object 



183
184
185
186
187
188
 
# File 'lib/pho/file_management.rb', line 183

def get_ok_file_for(filename)
  relative_path = filename.gsub(@dir, "")
  base = File.basename(filename)
  relative_path = relative_path.gsub(base, "#{TRACKING_DIR}/#{base}")
  return "#{@dir}#{relative_path}.#{@ok_suffix}"              
end

#new_files(recursive = false) ⇒ Object

List any new files in the directory



139
140
141
142
143
144
145
 
# File 'lib/pho/file_management.rb', line 139

def new_files(recursive=false)
  newfiles = Array.new
  list(recursive).each do |file|
    newfiles << file if !stored?(file)            
  end
  return newfiles
end

#reset(recursive = false) ⇒ Object

Reset the directory to clear out any previous statuses



79
80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/pho/file_management.rb', line 79

def reset(recursive=false)
  if recursive
    pattern = "**/#{TRACKING_DIR}/*"
  else
    pattern = "/#{TRACKING_DIR}/*"
  end        
  Dir.glob( File.join(@dir, "#{pattern}.#{@fail_suffix}") ).each do |file|
    File.delete(file)
  end
  Dir.glob( File.join(@dir, "#{pattern}.#{@ok_suffix}") ).each do |file|
    File.delete(file)
  end         
end

#retry_failures(recursive = false) ⇒ Object

Retry anything known to have failed



55
56
57
58
59
60
61
62
63
 
# File 'lib/pho/file_management.rb', line 55

def retry_failures(recursive=false)
  retries = failures(recursive)
  retries.each do |filename|
    File.delete( get_fail_file_for(filename) )
    #store it
    file = File.new(filename)
    store_file(file, filename)      
  end
end

#store_changes(recursive = false) ⇒ Object 



65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/pho/file_management.rb', line 65

def store_changes(recursive=false)
  changes = changed_files(recursive)
  changes.each do |filename|
    #cleanup tracking files
    f = get_fail_file_for(filename)
    File.delete( f ) if File.exists?(f)
    f = get_ok_file_for(filename)
    File.delete( f ) if File.exists?(f)
    file = File.new(filename)
    store_file(file, filename)
  end
end

#stored?(file) ⇒ Boolean

returns true if there is a fail or ok file, false otherwise

Returns:

  • (Boolean) 


94
95
96
97
98
99
100
101
 
# File 'lib/pho/file_management.rb', line 94

def stored?(file)
  ok_file = get_ok_file_for(file)
  fail_file = get_fail_file_for(file)        
  if ( File.exists?(ok_file) or File.exists?(fail_file) )
    return true
  end        
  return false
end

#stored_files(recursive = false) ⇒ Object 



122
123
124
125
126
127
128
 
# File 'lib/pho/file_management.rb', line 122

def stored_files(recursive=false)
  stored = Array.new
  list(recursive).each do |file|
    stored << file if stored?(file)
  end
  return stored
end

#successes(recursive = false) ⇒ Object

List successes



157
158
159
160
161
162
163
 
# File 'lib/pho/file_management.rb', line 157

def successes(recursive=false)
  successes = Array.new
  list(recursive).each do |file|
    successes << file if File.exists?( get_ok_file_for(file) )
  end
  return successes
end

#summary(recursive = false) ⇒ Object

Summarize the state of the collection to the provided IO object Creates a simple report



167
168
169
170
171
172
173
174
 
# File 'lib/pho/file_management.rb', line 167

def summary(recursive=false)
   failures = failures(recursive)
   successes = successes(recursive)
   newfiles = new_files(recursive)
   total = failures.size + successes.size + newfiles.size
   summary = "#{@dir} #{recursive ? " and sub-directories" : ""} contains #{total} files: #{successes.size} stored, #{failures.size} failed, #{newfiles.size} new"
   return summary     
end