Class: RelatonCalconnect::DataFetcher

Inherits:
Object
  • Object
show all
Defined in:
lib/relaton_calconnect/data_fetcher.rb

Overview

Relaton-calconnect data fetcher

Constant Summary collapse

ENDPOINT =

DOMAIN = “standards.calconnect.org/” SCHEME, HOST = DOMAIN.split(%r:?/?/)

"https://standards.calconnect.org/relaton/index.yaml"

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(output, format) ⇒ DataFetcher

DATADIR = “data” DATAFILE = File.join DATADIR, “bibliography.yml” ETAGFILE = File.join DATADIR, “etag.txt”



15
16
17
18
19
20
21
22
 
# File 'lib/relaton_calconnect/data_fetcher.rb', line 15

def initialize(output, format)
  @output = output
  @etagfile = File.join output, "etag.txt"
  @format = format
  @ext = format.sub "bibxml", "xml"
  @files = []
  @index = Relaton::Index.find_or_create :CC, file: "index-v1.yaml"
end

Class Method Details

.fetch(output: "data", format: "yaml") ⇒ void

This method returns an undefined value.

Fetch all the documents from a source

Parameters:

  • output (String) (defaults to: "data") —

    directory to output documents, default: “data”

  • format (String) (defaults to: "yaml") —

    output format, default: “yaml”



32
33
34
35
36
37
38
39
40
 
# File 'lib/relaton_calconnect/data_fetcher.rb', line 32

def self.fetch(output: "data", format: "yaml")
  t1 = Time.now
  puts "Started at: #{t1}"
  FileUtils.mkdir_p output
  new(output, format).fetch
  t2 = Time.now
  puts "Stopped at: #{t2}"
  puts "Done in: #{(t2 - t1).round} sec."
end

Instance Method Details

#fetch ⇒ Object

fetch data form server and save it to file.



45
46
47
48
49
50
51
52
53
54
55
 
# File 'lib/relaton_calconnect/data_fetcher.rb', line 45

def fetch # rubocop:disable Metrics/AbcSize
  resp = Faraday.new(ENDPOINT, headers: { "If-None-Match" => etag }).get
  # return if there aren't any changes since last fetching
  return unless resp.status == 200

  data = YAML.safe_load resp.body
  all_success = true
  data["root"]["items"].each { |doc| all_success &&= parse_page doc }
  self.etag = resp[:etag] if all_success
  @index.save
end