Module: Krikri::Harvester

Extended by:
ActiveSupport::Concern
Includes:
SoftwareAgent
Included in:
Krikri::Harvesters::CouchdbHarvester, Krikri::Harvesters::OAIHarvester
Defined in:
lib/krikri/harvester.rb

Overview

Harvester is the abstract interface for aggregating records from a source. Harvesters need to be able to:

- Enumerate record ids (#record_ids)
- Enumerate records (#records)
- Retrieve individual records (#get_record)

Implementations of Enumerators in subclasses should be lazy, avoiding loading large numbers of records into memory or sending unnecessary requests to providers. The following example should be safe:

my_harvester.record_ids.take(100)

This parent class implements a few generic methods based on the services outlined above:

- #count. This assumes that lazy counting is implemented on the
  Enumerable returned by #record_ids. If not, it is strongly
  recommended to override this method in your subclass with
  an efficient implementation.
- #run. Wraps persistence of each record returned by #records.
  Runs a full harvest, inserting Original Records into the database,
  given the options passed to #initialize.

When including, add a call to Krikri::Harvester::Registry.register() to put it in the registry so that it can be looked up. See lib/krikri/engine.rb.

See Also:

Constant Summary collapse

Registry =

An application-wide registry of defined Harvesters.

Class.new(Krikri::Registry)

Constants included from SoftwareAgent

SoftwareAgent::Logger

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from SoftwareAgent

#agent_name, #log

Instance Attribute Details

#nameObject

Returns the value of attribute name


37
38
39
# File 'lib/krikri/harvester.rb', line 37

def name
  @name
end

#uriObject

Returns the value of attribute uri


37
38
39
# File 'lib/krikri/harvester.rb', line 37

def uri
  @uri
end

Class Method Details

.expected_optsObject

This method is abstract.

Return initialization options for the harvester. The harvester will expect to receive these upon instantiation (in the opts argument to #initialize), in the form:

key: <symbol for this harvester>,
opts: {
  option_name: {type: :type, required: <boolean>,
                multiple_ok: <boolean, default false>
}

} … where type could be :uri, :string, :int, etc. … and multiple_ok means whether it's allowed to be an array … for example, for OAI this might be: {key: :oai,

set: {type: :string, required: false, multiple_ok: true},
metadata_prefix: {type: string, required: true}}
TODO:

The actual type token values and how they'll be used is to be determined, but something should exist for providing validation guidelines to a client so it doesn't have to have inside knowledge of the harvester's code.

Note:

The options are going to vary between harvesters. Some options are going to be constant for the whole harvest, and some are going to be lists that get iterated over. For example, a set or collection. There will be an ingestion event where we want multiple jobs enqueued, one per set or collection. The one option (a list) that would vary from harvest job to harvest job might be 'set' (in the case of OAI). This method doesn't solve how that's going to happen, but simply provides, as a convenience, the options that the harvester wants to see.

Raises:

  • (NotImplementedError)

164
165
166
# File 'lib/krikri/harvester.rb', line 164

def self.expected_opts
  raise NotImplementedError
end

Instance Method Details

#countInteger

Note:

override if #record_ids does not implement a lazy Enumerable#count.

Returns A count of the records expected in the harvest.

Returns:

  • (Integer)

    A count of the records expected in the harvest.


88
89
90
# File 'lib/krikri/harvester.rb', line 88

def count
  record_ids.count
end

#get_record(_) ⇒ Krikri::OriginalRecord

This method is abstract.

Get a single record by identifier.

Parameters:

  • identifier (#to_s)

    the identifier for the record to be retrieved

Returns:

Raises:

  • (NotImplementedError)

104
105
106
# File 'lib/krikri/harvester.rb', line 104

def get_record(_)
  raise NotImplementedError
end

#initialize(opts = {}) ⇒ Object

Accepts options for a generic harvester:

uri: a URI for the harvest endpoint or provider
name: a name for the harvester or provider, SHOULD be supplied when the
      provider does not use universally unique identifiers (optional).
record_class: Class of records to generate (optional; defaults to
              Krikri::OriginalRecord).
id_minter: Module to create identifiers for generated records (optional;
           defaults to Krikri::Md5Minter)

Pass harvester specific options to inheriting classes under a key for that harvester. E.g. { uri: my_uri, oai: { metadata_prefix: :oai_dc } }

Parameters:

  • opts (Hash) (defaults to: {})

    a hash of options


63
64
65
66
67
68
# File 'lib/krikri/harvester.rb', line 63

def initialize(opts = {})
  @uri = opts.fetch(:uri)
  @name = opts.delete(:name)
  @record_class = opts.delete(:record_class) { Krikri::OriginalRecord }
  @id_minter = opts.delete(:id_minter) { Krikri::Md5Minter }
end

#record_idsEnumerable<String>

This method is abstract.

Provide a low-memory, lazy enumerable for record ids.

The following usage should be safe:

record_ids.each do |id|
   some_operation(id)
end

Returns:

  • (Enumerable<String>)

    The identifiers included in the harvest.

Raises:

  • (NotImplementedError)

80
81
82
# File 'lib/krikri/harvester.rb', line 80

def record_ids
  raise NotImplementedError
end

#recordsEnumerable<Krikri::OriginalRecord>

This method is abstract.

Provide a low-memory, lazy enumerable for records.

Returns The harvested records.

Returns:

Raises:

  • (NotImplementedError)

95
96
97
# File 'lib/krikri/harvester.rb', line 95

def records
  raise NotImplementedError
end

#run(activity_uri = nil) ⇒ Boolean

Run the harvest. This should be idempotent so it can be safely retried on errors.

Returns:

  • (Boolean)

113
114
115
116
117
118
119
120
121
122
123
124
125
# File 'lib/krikri/harvester.rb', line 113

def run(activity_uri = nil)
  log :info, 'harvest is running'
  records.each do |rec|
    begin
      rec.save(activity_uri)
    rescue => e
      log :error, "Error harvesting record:\n#{rec.content}\n\twith message:\n#{e.message}"
      next
    end
  end
  log :info, 'harvest is done'
  true
end