Module: RelatonGb::GbScrapper
- Extended by:
- Scrapper
- Defined in:
- lib/relaton_gb/gb_scrapper.rb
Overview
National standard scrapper.
Constant Summary
Constants included from Scrapper
Class Method Summary collapse
-
.get_committee(doc, _ref) ⇒ Hash
-
:type [String] * :name [String].
-
- .scrape_doc(hit) ⇒ RelatonGb::GbBibliographicItem
- .scrape_page(text) ⇒ RelatonGb::HitCollection
Methods included from Scrapper
fetch_structuredidentifier, get_contributors, get_docid, get_status, get_titles, get_type, org, scrapped_data
Class Method Details
.get_committee(doc, _ref) ⇒ Hash
Returns * :type [String]
-
:name [String].
49 50 51 52 |
# File 'lib/relaton_gb/gb_scrapper.rb', line 49 def get_committee(doc, _ref) name = doc.at("//div[contains(., '归口单位') or contains(., '归口部门')]/following-sibling::div") { type: "technical", name: name.text.delete("\r\n\t\t") } end |
.scrape_doc(hit) ⇒ RelatonGb::GbBibliographicItem
36 37 38 39 40 41 42 |
# File 'lib/relaton_gb/gb_scrapper.rb', line 36 def scrape_doc(hit) src = "http://openstd.samr.gov.cn/bzgk/gb/newGbInfo?hcno=#{hit.pid}" doc = Nokogiri::HTML OpenURI.open_uri(src) GbBibliographicItem.new(**scrapped_data(doc, src, hit)) rescue OpenURI::HTTPError, SocketError, OpenSSL::SSL::SSLError, Net::OpenTimeout raise RelatonBib::RequestError, "Cannot access #{src}" end |
.scrape_page(text) ⇒ RelatonGb::HitCollection
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# File 'lib/relaton_gb/gb_scrapper.rb', line 17 def scrape_page(text) # rubocop:disable Metrics/MethodLength, Metrics/AbcSize host = "http://openstd.samr.gov.cn/bzgk/gb/std_list" search_html = OpenURI.open_uri("#{host}?p.p2=#{text}") result = Nokogiri::HTML search_html hits = result.xpath( "//table[contains(@class, 'result_list')]/tbody[2]/tr", ).map do |h| ref = h.at "./td[2]/a" pid = ref[:onclick].match(/[0-9A-F]+/).to_s rdate = h.at("./td[7]").text Hit.new pid: pid, docref: ref.text, scrapper: self, release_date: rdate end HitCollection.new hits.sort_by(&:release_date).reverse rescue OpenURI::HTTPError, SocketError, OpenSSL::SSL::SSLError, Net::OpenTimeout raise RelatonBib::RequestError, "Cannot access #{host}" end |