Class: TaliaUtil::Util

Inherits:
Object show all
Defined in:
lib/talia_util/util.rb

Overview

Main utility functions for Talia

Class Method Summary collapse

Class Method Details

.clear_data ⇒ Object

Remove the data directories. Removes the data directory (configured as data_directory_location in talia_core.yml) and the iip directory (configured as iip_root_location in talia_core.yml).

This ignores non-existing directories without an error message.



141
142
143
144
145
146
 
# File 'lib/talia_util/util.rb', line 141

def clear_data
  data_dir = TaliaCore::CONFIG['data_directory_location']
  iip_dir = TaliaCore::CONFIG['iip_root_directory_location']
  FileUtils.rm_rf(data_dir) if(File.exist?(data_dir))
  FileUtils.rm_rf(iip_dir) if(File.exist?(iip_dir))
end

.do_migrations ⇒ Object

Runs the migrations for the main Talia tables. This will use the migrations from the “talia” generator, not the ones from the Rails application.



228
229
230
231
 
# File 'lib/talia_util/util.rb', line 228

def do_migrations
  migration_path = File.join("generators", "talia", "templates", "migrations")
  ActiveRecord::Migrator.migrate(migration_path, ENV["VERSION"] ? ENV["VERSION"].to_i : nil )
end

.fat_record_joins ⇒ Object

SQL join snippet for selecting “fat” records. See fat_record_select



258
259
260
261
262
263
 
# File 'lib/talia_util/util.rb', line 258

def fat_record_joins
  joins =  " LEFT JOIN active_sources AS obj_sources ON semantic_relations.object_id = obj_sources.id AND semantic_relations.object_type = 'TaliaCore::ActiveSource'"
  joins << " LEFT JOIN semantic_properties AS obj_props ON semantic_relations.object_id = obj_props.id AND semantic_relations.object_type = 'TaliaCore::SemanticProperty'"
  joins << " LEFT JOIN active_sources AS subject_sources ON semantic_relations.subject_id = subject_sources.id"
  joins
end

.fat_record_select ⇒ Object

SQL select portion selecting “fat” records from the semantic_relatations table. This will select all data needed to create all triples. Used for rewriting the rdf.



243
244
245
246
247
248
249
250
251
252
253
254
255
 
# File 'lib/talia_util/util.rb', line 243

def fat_record_select
  select = 'semantic_relations.id AS id, semantic_relations.created_at AS created_at, '
  select << 'semantic_relations.updated_at AS updated_at, '
  select << 'object_id, object_type, subject_id, predicate_uri, '
  select << 'obj_props.created_at AS property_created_at, '
  select << 'obj_props.updated_at AS property_updated_at, '
  select << 'obj_props.value AS property_value, '
  select << 'obj_sources.created_at AS object_created_at, '
  select << 'obj_sources.updated_at AS object_updated_at, obj_sources.type AS  object_realtype, '
  select << 'obj_sources.uri AS object_uri, '
  select << 'subject_sources.uri AS subject_uri'
  select
end

.flag?(the_flag) ⇒ Boolean

Check if the given flag is set on the command line. This will assert that the flag is set, otherwise it’s equivalent to String#yes? (from the core extensions) on the variable

Returns:

  • (Boolean) 


236
237
238
239
 
# File 'lib/talia_util/util.rb', line 236

def flag?(the_flag)
  assit_not_nil(the_flag)
  ENV[the_flag].yes?
end

.flush_db ⇒ Object

Flush the SQL database. This deletes all entries only from the main Talia tables in the db. Additional tables for user-defined models (e.g. translations) will not be touched.



124
125
126
127
128
 
# File 'lib/talia_util/util.rb', line 124

def flush_db
  [ 'active_sources', 'data_records', 'semantic_properties', 'semantic_relations', 'workflows'].reverse.each { |f| ActiveRecord::Base.connection.execute "DELETE FROM #{f}" }
  # Also remove the "unsaved cache" for the wrappers (may be important during testing)
  TaliaCore::SemanticCollectionWrapper.instance_variable_set(:'@unsaved_source_cache', {})
end

.flush_rdf ⇒ Object

Flush the RDF store. This clears the whole store, including triples that were added through other means than the Talia API.



132
133
134
 
# File 'lib/talia_util/util.rb', line 132

def flush_rdf
  ActiveRDF::ConnectionPool.write_adapter.clear
end

.full_reset ⇒ Object

Do a full reset of the data store. Equivalent to clearing the Talia SQL tables, the RDF store and the data directories. This will re-initialize the ontologies afterwards.



152
153
154
155
156
157
 
# File 'lib/talia_util/util.rb', line 152

def full_reset
  flush_db
  flush_rdf
  clear_data
  setup_ontologies
end

.get_files ⇒ Object

Rake task helper. Get the list of files the files environment variable (passed as file=[files]). Exits the runtime and prints an error message if the variable is not set.

This returns a FileList object with all the files that match the pattern.



14
15
16
17
18
19
20
21
22
23
 
# File 'lib/talia_util/util.rb', line 14

def get_files
  puts "Files given: #{ENV['files']}"
  unless(ENV['files'])
    puts("This task needs files to work. Pass them like this files='something/*.x'")
    print_options
    exit(1)
  end

  FileList.new(ENV['files'])
end

.init_talia ⇒ Object

Init the talia core system. Use to initialize the system for the rake tasks. This will just exit if the system is already initialized, e.g. through Rails.

See the rake:talia_core help task for options supported. Options include talia_root, environment, config, reset_db and reset_rdf



63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/talia_util/util.rb', line 63

def init_talia
  return if(TaliaCore::Initializer.initialized)

  # If we have Rails installed, just call the Rails config 
  # instead of running the manual init
  if(defined?(RAILS_ROOT) && File.exist?(File.join(RAILS_ROOT, 'config', 'environment.rb')))
    puts "\nInitializing Talia through Rails"
    load(File.join(RAILS_ROOT, 'config', 'environment.rb'))
  else

    # If options are not set, the initializer will fall back to the internal default
    TaliaCore::Initializer.talia_root = ENV['talia_root']
    TaliaCore::Initializer.environment = ENV['environment']

    config_file = ENV['config'] ? ENV['config'] : 'talia_core'

    # run the initializer
    TaliaCore::Initializer.run(config_file) do |config|
      unless(flag?('no_standalone'))
        puts "Always using standalone db from utilities."
        puts "Give the no_standalone option to override it."
        config['standalone_db'] = "true"
      end
    end
  end
  puts("\nTaliaCore initialized")

  # # Flush the database if requested
  if(flag?('reset_db'))
    flush_db
    puts "DB flushed"
  end

  # Flus the rdf if requested
  if(flag?('reset_rdf'))
    flush_rdf
    puts "RDF flushed"
  end
end

.load_all_models ⇒ Object

Force-loads all Talia related models. This will attempt to load all classes in RAILS_ROOT/app/models and in TALIA_CODE_ROOT/lib/talia_core/source_types.

Use this to make sure the whole hierarchy of ActiveSource subclasses is in memory..



284
285
286
287
288
289
290
291
292
 
# File 'lib/talia_util/util.rb', line 284

def load_all_models
  return if @models_loaded
  load_models_from File.join(RAILS_ROOT, 'app', 'models', '**', '*.rb') if(defined? RAILS_ROOT)
  load_models_from File.join(TALIA_CODE_ROOT, 'lib', 'talia_core', 'source_types',  '**',  '*.rb'), 'TaliaCore::SourceTypes::'
  TaliaCore::Source
  TaliaCore::Collection

  @models_loaded = true
end

.load_fixtures ⇒ Object

Load the database fixtures. This “manually” loads the database fixtures for the Talia tables for the core unit tests. The fixtures are those contained in the talia_core folder, not the ones from the application’s tests.



217
218
219
220
221
222
223
224
 
# File 'lib/talia_util/util.rb', line 217

def load_fixtures
  # fixtures = ENV['FIXTURES'] ? ENV['FIXTURES'].split(/,/) : Dir.glob(File.join(File.dirname(__FILE__), 'test', 'fixtures', '*.{yml,csv}'))  
  fixtures = [ 'active_sources', 'semantic_relations', 'semantic_properties' 'data_records']
  fixtures.reverse.each { |f| ActiveRecord::Base.connection.execute "DELETE FROM #{f}" }
  fixtures.each do |fixture_file|
    Fixtures.create_fixtures(File.join('test', 'fixtures'), File.basename(fixture_file, '.*'))  
  end  
end

.load_models_from(dir, prefix = '') ⇒ Object

Helper to load all classes from a given directory. This will attempt to instanciate all classes found in the dir. The name of the class to be instantiated will be taken from the file name, prepending the prefix (e.g. ‘ModuleName::’) if set.



297
298
299
300
301
302
303
304
305
306
307
308
309
310
 
# File 'lib/talia_util/util.rb', line 297

def load_models_from(dir, prefix='')
  # Appends a file system directory separator to the directory if needed.
  dir = File.join(dir, '')
  Dir[File.join(dir, '**', '*.rb')].each do |f| 
    # For every rb file we try to gues and instantiate the contained class.
    model_name = f.gsub(/#{dir}|\.rb/, '')
    begin
      (prefix + model_name).camelize.constantize
    rescue Exception => e
      # Errors at this point could be ignored, as there may be files that do not contain classes.
      TaliaCore.logger.warn "Could not load class #{(prefix + model_name).camelize}: #{e.message}"
    end
  end
end

.ontology_folder ⇒ Object

Gets the ontology_folder environment variable, as passed to the rake task with ontology_folder=[folder]

Defaults to RAILS_ROOT/ontologies if the variable is not set.



30
31
32
 
# File 'lib/talia_util/util.rb', line 30

def ontology_folder
  ENV['ontology_folder'] || File.join(RAILS_ROOT, 'ontologies')
end

.print_options ⇒ Object

Print the options for the rake tasks to the console.



266
267
268
269
270
271
272
273
274
275
276
277
278
 
# File 'lib/talia_util/util.rb', line 266

def print_options
  puts "\nGeneral options (not all options are valid for all tasks):"
  puts "files=<pattern>     - Files for the task (a pattern to match the files)"
  puts "talia_root=<path>   - Manually configure the TALIA_ROOT path"
  puts "                      (default:autodetect)"
  puts "reset_rdf={yes|no}  - Flush the RDF store (default:no)"
  puts "reset_db={yes|no}   - Flush the database (default:no)"
  puts "config=<filename>   - Talia configuration file (default: talia_core)"
  puts "environment=<env>   - Environment for configuration (default: development)"
  puts "data_dir=<dir>      - Directory for the data files"
  puts "verbose={yes|no}    - Show some additional info"
  puts ""
end

.rewrite_count ⇒ Object

The number of triples that would be rewritten with #rewrite_rdf



210
211
212
 
# File 'lib/talia_util/util.rb', line 210

def rewrite_count
  TaliaCore::SemanticRelation.count + TaliaCore::ActiveSource.count
end

.rewrite_rdf(turn_off_safety = false) ⇒ Object

Rewrite the RDF for the whole database. Erases the RDF store completely and re-builds the graph from the data in the SQL tables.

Warning: This will loose all information contained in the RDF that is not duplicated. This includes all SWICKY notebooks!

Unless the turn_off_safety flag is set, or the environment variable i_know_what_i_am_doing is set to “yes”, this method will print an error message and raise an exception.

For each triple written, this will yield to the block (if one is given) without parameters. For progress reporting, the overall number of triples that will be rewritten can be acquired with #rewrite_count



173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
 
# File 'lib/talia_util/util.rb', line 173

def rewrite_rdf(turn_off_safety=false)
  unless((ENV['i_know_what_i_am_doing'].yes?) || turn_off_safety)
    puts "WARNING: Rewriting the RDF will ERASE all data that does not come from the Talia API"
    puts "This includes ALL SWICKY notebooks"
    puts 
    puts "To proceed run this task again, and give the following option:"
    puts "i_know_what_i_am_doing=yes"
    raise ArgumentError, "Can't proceed without confirmation."
  end
  flush_rdf
  # We'll get all data from single query.
  fat_rels = TaliaCore::SemanticRelation.find(:all, :joins => fat_record_joins,
  :select => fat_record_select)
  fat_rels.each do |rec|
    subject = N::URI.new(rec.subject_uri)
    predicate = N::URI.new(rec.predicate_uri)
    object = if(rec.object_uri)
      N::URI.new(rec.object_uri)
    else
      rec.property_value
    end
    ActiveRDF::FederationManager.add(subject, predicate, object)
    yield if(block_given?)
  end

  # Rewriting all the "runtime type" rdf triples
  # We'll select the type as something else, so that it doesn't try to do
  # STI instantiation (which would cause this to blow for classes that
  # are defined outside the core.
  TaliaCore::ActiveSource.find(:all, :select => 'uri, type AS runtime_type').each do |src|
    type = (src.runtime_type || 'ActiveSource')
    ActiveRDF::FederationManager.add(src, N::RDF.type, N::TALIA + type)
    yield if(block_given?)
  end
end

.setup_ontologies ⇒ Object

Set up the ontologies from the ontology_folder. This clears the RDF context for the ontologies, if possible. Then it will load all ontologies in the ontology_folder into the store, and run the RdfUpdate::owl_to_rdfs update.



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/talia_util/util.rb', line 38

def setup_ontologies
  # Clear the ontologies from RDF, if possible
  adapter = ActiveRDF::ConnectionPool.write_adapter
  if(adapter.contexts?)
    TaliaCore::RdfImport.clear_file_contexts
  else
    puts "WARNING: Cannot remove old ontologies, adapter doesn't support contexts."
  end

  puts "Ontologies loaded from: #{ontology_folder}"
  files = Dir[File.join(ontology_folder, '*.{rdf*,owl}')]
  ENV['rdf_syntax'] ||= 'rdfxml'
  params = [ENV['rdf_syntax'], files]
  params << :auto if(adapter.contexts?)
  TaliaCore::RdfImport::import(*params)
  RdfUpdate::owl_to_rdfs
end

.talia_config ⇒ Object

Rake helper. Prints the talia configuration to the console.



105
106
107
108
109
110
111
112
113
 
# File 'lib/talia_util/util.rb', line 105

def talia_config
  puts "Talia configuration"
  puts ""
  puts "TALIA_ROOT: #{TALIA_ROOT}"
  puts "Environment: #{TaliaCore::CONFIG['environment']}"
  puts "Standalone DB: #{TaliaCore::CONFIG['standalone_db']}"
  puts "Data Directory: #{TaliaCore::CONFIG['data_directory_location']}"
  puts "Local Domain: #{N::LOCAL}"
end

.title ⇒ Object

Rake/Startup helper. Prints out the talia version/header to the the console.



116
117
118
119
 
# File 'lib/talia_util/util.rb', line 116

def title
  puts "\nTalia Digital Library system. Version: #{TaliaCore::Version::STRING}" 
  puts "http://www.muruca.org/\n\n"
end