Module: Traject::Macros::Marc21Semantics

Defined in:
lib/traject/macros/marc21_semantics.rb

Overview

extracting various semantic parts out of a Marc21 record. Few of these come directly from Marc21 spec or other specs with no judgement, they are all to some extent opinionated, based on actual practice and actual data, some more than others. If it doens’t do what you want, don’t use it. But if it does, you can use it, and continue to get updates with future versions of Traject.

Constant Summary collapse

MarcExtractor =

shortcut

Traject::MarcExtractor
LCC_REGEX =

Looks up Library of Congress Classification (LCC) or NLM Medical Subject Headings (MeSH) from usual parts of the marc record. Maps them to high-level broad categories, basically just using the first part of the LCC. Note it’s just looking in bib-level locations for LCCs, you’re on your own with holdings.

Sanity checks to make sure the thing looks like an LCC with a regex, before mapping.

Will call it ‘Unknown’ if it’s got nothing else, or pass in :default => something else, or nil.

The categories output aren’t great, but they’re something.

/ *[A-Z]{1,3}[ .]*(?:(\d+)(?:\s*?\.\s*?(\d+))?).*/

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.get_sortable_author(record) ⇒ Object 



62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
 
# File 'lib/traject/macros/marc21_semantics.rb', line 62

def self.get_sortable_author(record)
  onexx = MarcExtractor.cached("100:110:111", :first => true).extract(record).first
  onexx = onexx.strip if onexx

  titles = []
  MarcExtractor.cached("240:245", :first => true).each_matching_line(record) do |field, spec|
    non_filing = field.indicator2.to_i

    str = field.subfields.collect {|sf| sf.value}.join(" ")
    str = str.slice(non_filing, str.length)
    titles << str
  end.first
  title = titles.first
  title = title.strip if title

  return "#{onexx}#{title}"
end

.get_sortable_title(record) ⇒ Object 



88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
 
# File 'lib/traject/macros/marc21_semantics.rb', line 88

def self.get_sortable_title(record)
  MarcExtractor.cached("245ab").collect_matching_lines(record) do |field, spec, extractor|
    str = extractor.collect_subfields(field, spec).first

    if str.nil?
      # maybe an APPM archival record with only a 'k'
      str = field['k']
    end
    if str.nil?
      # still? All we can do is bail, I guess
      return nil
    end

    non_filing = field.indicator2.to_i
    str = str.slice(non_filing, str.length)
    str = Marc21.trim_punctuation(str)

    str
  end.first
end

.oclcnum_extract(num) ⇒ Object

If a num begins with a known OCLC prefix, return it without the prefix. otherwise nil.



29
30
31
32
33
34
35
36
37
38
 
# File 'lib/traject/macros/marc21_semantics.rb', line 29

def self.oclcnum_extract(num)
  stripped = num.gsub(/\A(ocm)|(ocn)|(on)|(\(OCoLC\))/, '')
  if num != stripped
    # it had the prefix, which we've now stripped
    return stripped
  else
    # it didn't have the prefix
    return nil
  end
end

.publication_date(record, estimate_tolerance = 15, min_year = 500, max_year = (Time.new.year + 6)) ⇒ Object

See #marc_publication_date. Yeah, this is a holy mess. Maybe it should actually be extracted to it’s own class!



243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
 
# File 'lib/traject/macros/marc21_semantics.rb', line 243

def self.publication_date(record, estimate_tolerance = 15, min_year = 500, max_year = (Time.new.year + 6))
  field008 = MarcExtractor.cached("008").extract(record).first
  found_date = nil

  if field008 && field008.length >= 11
    date_type = field008.slice(6)
    date1_str = field008.slice(7,4)
    date2_str = field008.slice(11, 4) if field008.length > 15

    # for date_type q=questionable, we have a range.
    if (date_type == 'q')
      # make unknown digits at the beginning or end of range,
      date1 = date1_str.sub("u", "0").to_i
      date2 = date2_str.sub("u", "9").to_i
      # do we have a range we can use?
      if (date2 > date1) && ((date2 - date1) <= estimate_tolerance)
        found_date = (date2 + date1)/2
      end
    end
    # didn't find a date that way, and anything OTHER than date_type
    # n=unknown, q=questionable, try single date -- for some date types,
    # there's a date range between date1 and date2, yeah, we often take
    # the FIRST date then, the earliest. That's just what we're doing.
    if found_date.nil? && date_type != 'n' && date_type != 'q'
      # in date_type 'r', second date is original publication date, use that I think?
      date_str = (date_type == 'r' && date2_str.to_i != 0) ? date2_str : date1_str
      # Deal with stupid 'u's, which end up meaning a range too,
      # find midpoint and make sure our tolerance is okay.
      ucount = 0
      while (!date_str.nil?) && (i = date_str.index('u'))
        ucount += 1
        date_str[i] = "0"
      end
      date = date_str.to_i
      if ucount > 0 && date != 0
        delta = 10 ** ucount # 10^ucount, expontent
        if delta <= estimate_tolerance
          found_date = date + (delta/2)
        end
      elsif date != 0
        found_date = date
      end
    end
  end
  # Okay, nothing from 008, try 260
  if found_date.nil?
    v260c = MarcExtractor.cached("260c", :seperator => nil).extract(record).first
    # just try to take the first four digits out of there, we're not going to try
    # anything crazy.
    if v260c =~ /(\d{4})/
      found_date = $1.to_i
    end
  end

  # is it within our acceptable range?
  found_date = nil if found_date && (found_date < min_year || found_date > max_year)

  return found_date
end

Instance Method Details

#marc_era_facet ⇒ Object

Opinionated routine to create values for a chronology/era facet out of LCSH chron subdivisions. Does some normalization: for 651 with a chron facet fitting the form “aaaaa, yyyy-yyyy”, it will add in the $a. For instance: 651 a| United States x| History y| Civil War, 1861-1865 –> “United States: Civil War, 1861-1865”



403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
 
# File 'lib/traject/macros/marc21_semantics.rb', line 403

def marc_era_facet
  ordinary_fields_spec = "600y:610y:611y:630y:648ay:650y:654y:656y:690y"
  special_fields_spec = "651:691"
  seperator = ": "

  extractor_ordinary_fields = MarcExtractor.new(ordinary_fields_spec)
  extractor_special_fields  = MarcExtractor.new(special_fields_spec)

  lambda do |record, accumulator|
    # straightforward ones


    accumulator.concat( extractor_ordinary_fields.extract(record).collect do |v|
      # May have a period we have to remove, if it was at end of tag
      v.sub(/\. *\Z/, '')
    end)

    # weird ones
    extractor_special_fields.each_matching_line(record) do |field, spec, extractor|
      field.subfields.each do |sf|
        next unless sf.code == 'y'
        if sf.value =~ /\A\s*.+,\s+(ca.\s+)?\d\d\d\d?(-\d\d\d\d?)?( B\.C\.)?[.,; ]*\Z/
          # it's our pattern, add the $a in please
          accumulator << "#{field['a']}#{seperator}#{sf.value.sub(/\. *\Z/, '')}"
        else
          accumulator << sf.value.sub(/\. *\Z/, '')
        end
      end
    end
  end
end

#marc_geo_facet(options = {}) ⇒ Object

An opinionated method of making a geographic facet out of BOTH 048 marc codes, AND geo subdivisions in 6xx LCSH subjects.

The LCSH geo subdivisions are further normalized:

  • geo qualifiers in $z fields into parens, so “Germany – Berlin” becomes “Berlin (Germany)” (to be consistent with how same areas are written in $a fields – doesn’t

    get everything, but gets lots of em)

  • qualified regions like that are additionally ‘posted up’, so “Germany – Berlin” gets recorded additionally as “Germany”



349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
 
# File 'lib/traject/macros/marc21_semantics.rb', line 349

def marc_geo_facet(options = {})
  marc_geo_map = Traject::TranslationMap.new("marc_geographic")

  a_fields_spec = options[:geo_a_fields] || "651a:691a"
  z_fields_spec = options[:geo_z_fields] || "600:610:611:630:648:650:654:655:656:690:651:691"

  extractor_043a      = MarcExtractor.new("043a", :seperator => nil)
  extractor_a_fields  = MarcExtractor.new(a_fields_spec, :seperator => nil)
  extractor_z_fields  = MarcExtractor.new(z_fields_spec)

  lambda do |record, accumulator|

    accumulator.concat(
      extractor_043a.extract(record).collect do |code|
        # remove any trailing hyphens, then map
        marc_geo_map[code.gsub(/\-+\Z/, '')]
      end.compact
    )

    #LCSH 651a and 691a go in more or less normally.
    accumulator.concat(
      extractor_a_fields.extract(record).collect do |s|
        # remove trailing periods, which they sometimes have if they were
        # at end of LCSH.
        s.sub(/\. */, '')
      end
    )

    # fields we take z's from have a bit more normalization
    extractor_z_fields.each_matching_line(record) do |field, spec, extractor|
      z_fields = field.subfields.find_all {|sf| sf.code == "z"}.collect {|sf| sf.value }
      # depending on position in total field, may be a period on the end
      # we want to remove.
      z_fields.collect! {|s| s.gsub(/\. *\Z/, '')}

      if z_fields.length == 2
        # normalize subdivision as parenthetical
        accumulator << "#{z_fields[1]} (#{z_fields[0]})"
        # and 'post up'
        accumulator << z_fields[0]
      else
        # just add all the z's if there's 1 or more than 2.
        accumulator.concat z_fields
      end
    end
  end
end

#marc_instrument_codes_normalized(spec = "048") ⇒ Object

This weird one actually returns marc instrumentation codes, not humanized. But it normalizes them by breaking them down into a numeric and non-numeric version. For instance “ba01” will be indexed as both “ba01” and “ba”. ALSO, if the code is in a subfield b (soloist), it’ll be indexed additionally as “ba01.s” and “ba.s”.

This has proven useful for expert music librarian searching by hand; it could also be the basis of a GUI that executes searches behind the scenes for these codes.



189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'lib/traject/macros/marc21_semantics.rb', line 189

def marc_instrument_codes_normalized(spec = "048")
  soloist_suffix = ".s"

  extractor = MarcExtractor.new("048", :seperator => nil)

  return lambda do |record, accumulator|
    accumulator.concat(
      extractor.collect_matching_lines(record) do |field, spec, extractor|
        values = []

        field.subfields.each do |sf|
          v = sf.value
          # Unless there's at least two chars, it's malformed, we can
          # do nothing
          next unless v.length >= 2

          # Index both with and without number -- both with soloist suffix
          # if in a $b
          values << v
          values << "#{v}#{soloist_suffix}" if sf.code == 'b'
          if v.length >= 4
            bare = v.slice(0,2) # just the prefix
            values << bare
            values << "#{bare}#{soloist_suffix}" if sf.code == 'b'
          end
        end
        values
      end.uniq
    )
  end
end

#marc_instrumentation_humanized(spec = "048ab", options = {}) ⇒ Object

Takes marc 048ab instrument code, and translates it to human-displayable string. Takes first two chars of 048a or b, to translate (ignores numeric code)

Pass in custom spec if you want just a or b, to seperate soloists or whatever.



166
167
168
169
170
171
172
173
174
175
176
177
178
 
# File 'lib/traject/macros/marc21_semantics.rb', line 166

def marc_instrumentation_humanized(spec = "048ab", options = {})
  translation_map = Traject::TranslationMap.new(options[:translation_map] || "marc_instruments")

  extractor = MarcExtractor.new(spec, :seperator => nil)

  lambda do |record, accumulator|
    values = extractor.extract(record)
    human = values.collect do |value|
      translation_map[ value.slice(0, 2) ]
    end.uniq
    accumulator.concat human if human && human.length > 0
  end
end

#marc_languages(spec = "008[35-37]:041a:041d") ⇒ Object

maps languages, by default out of 008 and 041a and 041d

Can specify other spec if you want, say, 041b (lang of abstract) or 041e (lang of librettos), or 041h (lang of original) instead or in addition.

de-dups values so you don’t get the same one twice.

Exact spec of #marc_languages may change with new user data on what works best.



118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
 
# File 'lib/traject/macros/marc21_semantics.rb', line 118

def marc_languages(spec = "008[35-37]:041a:041d")
  translation_map = Traject::TranslationMap.new("marc_languages")

  extractor = MarcExtractor.new(spec, :seperator => nil)

  lambda do |record, accumulator|
    codes = extractor.collect_matching_lines(record) do |field, spec, extractor|
      if extractor.control_field?(field)
        (spec[:bytes] ? field.value.byteslice(spec[:bytes]) : field.value)
      else
        extractor.collect_subfields(field, spec).collect do |value|
          # sometimes multiple language codes are jammed together in one subfield, and
          # we need to seperate ourselves. sigh.
          unless value.length == 3
            value = value.scan(/.{1,3}/) # split into an array of 3-length substrs
          end
          value
        end.flatten
      end
    end
    codes = codes.uniq

    translation_map.translate_array!(codes)

    accumulator.concat codes
  end
end

#marc_lcc_to_broad_category(options = {}, spec = "050a:060a:090a:096a") ⇒ Object 



316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
 
# File 'lib/traject/macros/marc21_semantics.rb', line 316

def marc_lcc_to_broad_category( options = {}, spec="050a:060a:090a:096a")
  # Trying to match things that look like LCC, and not match things
  # that don't. Is tricky.
  lcc_regex = LCC_REGEX
  default_value = options.has_key?(:default) ? options[:default] : "Unknown"
  translation_map = Traject::TranslationMap.new("lcc_top_level")

  extractor = MarcExtractor.new(spec, :seperator => nil)

  lambda do |record, accumulator|
    candidates = extractor.extract(record)

    candidates.reject! do |candidate|
      !(candidate =~ lcc_regex)
    end

    accumulator.concat translation_map.translate_array!(candidates.collect {|a| a.lstrip.slice(0, 1)}).uniq

    if default_value && accumulator.empty?
      accumulator << default_value
    end
  end
end

#marc_publication_date(options = {}) ⇒ Object

An opinionated algorithm for getting a SINGLE publication date out of marc

  • Prefers using 008, but will resort to 260c

  • If 008 represents a date range, will take the midpoint of the range,

    only if range is smaller than estimate_tolerance, default 15 years.

  • Ignores dates below min_year (default 500) or above max_year (this year plus 6 years),

    because experience shows too many of these were in error.

Yeah, this code ends up ridiculous.



230
231
232
233
234
235
236
237
238
239
 
# File 'lib/traject/macros/marc21_semantics.rb', line 230

def marc_publication_date(options = {})
  estimate_tolerance  = options[:estimate_tolerance] || 15
  min_year            = options[:min_year] || 500
  max_year            = options[:max_year] || (Time.new.year + 6)

  lambda do |record, accumulator|
    date = Marc21Semantics.publication_date(record, estimate_tolerance, min_year, max_year)
    accumulator << date if date
  end
end

#marc_series_facet(spec = "440a:490a:800abcdt:810abcdt:811acdeft:830adfgklmnoprst") ⇒ Object

Adds in marc fields in spec (default is recommended series spec, but you can specify your own) – only trick is that 490’s are skipped of first indicator is 1 – if 490 first indicator is “1”, “series traced”, that means the series title mentioned here is already covered by another field we’re including, so we don’t want to double count it, possibly with slight variation.



151
152
153
154
155
156
157
158
159
 
# File 'lib/traject/macros/marc21_semantics.rb', line 151

def marc_series_facet(spec = "440a:490a:800abcdt:810abcdt:811acdeft:830adfgklmnoprst")
  extractor = MarcExtractor.new(spec)

  lambda do |record, accumulator|
    accumulator.concat( extractor.collect_matching_lines(record) do |field, spec, extractor|
      extractor.collect_subfields(field, spec) unless (field.tag == "490" && field.indicator1 == "1")
    end.compact)
  end
end

#marc_sortable_author ⇒ Object

A sortable author value, created by concatenating:

  • the main entry author, if there is one (fields 100, 110 or 111)

  • the main entry uniform title (240), if there is one - not including non-filing chars as noted in 2nd indicator of the 240

    • If no 240, the 245 title, not including non-filing chars as noted in ind 2 of the 245

Always returns a SINGLE string, based on concatenation.

Thanks SolrMarc for basic logic.

Note: You’ll want to pay attention to the Solr schema field definition you’re using, and have it do case-insensitivity or any other normalization you might want.

these probably should be taking only certain subfields, but we’re copying from SolrMarc that didn’t do so either and nobody noticed, so not bothering for now.



56
57
58
59
60
 
# File 'lib/traject/macros/marc21_semantics.rb', line 56

def marc_sortable_author
  lambda do |record, accumulator|
    accumulator << Marc21Semantics.get_sortable_author(record)
  end
end

#marc_sortable_title ⇒ Object

245 a and b, with non-filing characters stripped off



82
83
84
85
86
 
# File 'lib/traject/macros/marc21_semantics.rb', line 82

def marc_sortable_title
  lambda do |record, accumulator|
    accumulator << Marc21Semantics.get_sortable_title(record)
  end
end

#oclcnum(extract_fields = "035a") ⇒ Object

Extract OCLC numbers from, by default 035a’s by known prefixes, then stripped just the num, and de-dup.



16
17
18
19
20
21
22
23
24
25
26
 
# File 'lib/traject/macros/marc21_semantics.rb', line 16

def oclcnum(extract_fields = "035a")
  extractor = MarcExtractor.new(extract_fields, :seperator => nil)

  lambda do |record, accumulator|
    list = extractor.extract(record).collect! do |o|
      Marc21Semantics.oclcnum_extract(o)
    end.compact

    accumulator.concat list.uniq if list
  end
end