Module: Pikuri::Tool::Scraper

Defined in:
lib/pikuri/tool/scraper.rb

Overview

HTTP side of the web tools (WEB_SCRAPE and FETCH): GET the URL with a real-browser User-Agent, follow redirects, and hand the body to Extractor.extract with the response Content-Type as the hint. HTML/XHTML renders via Extractor::HTML, other text/* passes through verbatim, plug-in extractors extend the set (with pikuri-pdf, application/pdf extracts — by header or %PDF- magic, so a lying header still works); remaining types raise FetchError so the LLM observes the failure rather than an empty rendering.

Split into a thin HTTP fetch (Scraper.fetch) and the extraction wrapper (Scraper.visit) so tests drive each in isolation and Fetch can reuse the HTTP half. Nothing here knows about the LLM — the wrapping tools own caching, truncation, and turning the result into an observation.

Defined Under Namespace

Classes: FetchError, Fetched

Constant Summary collapse

USER_AGENT =

Returns User-Agent sent with each request; many sites reject a missing or obviously-bot UA.

Returns:

  • (String) —

    User-Agent sent with each request; many sites reject a missing or obviously-bot UA.

'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' \
'(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
ACCEPT =

Returns Accept header, so content-negotiating servers hand back something usable: HTML first, then application/pdf, then text/*.

Returns:

  • (String) —

    Accept header, so content-negotiating servers hand back something usable: HTML first, then application/pdf, then text/*.

'text/html,application/xhtml+xml,application/pdf,text/*;q=0.8'
MAX_REDIRECTS =

Returns maximum HTTP redirects to follow before giving up.

Returns:

  • (Integer) —

    maximum HTTP redirects to follow before giving up.

5
OPEN_TIMEOUT =

Returns connect timeout (s) for the Faraday request.

Returns:

  • (Integer) —

    connect timeout (s) for the Faraday request.

10
READ_TIMEOUT =

Returns read timeout (s) for the Faraday request.

Returns:

  • (Integer) —

    read timeout (s) for the Faraday request.

20
ERROR_BODY_EXCERPT =

Returns max chars of an error response body in a FetchError message — enough to show what came back (a Cloudflare/WAF page) without flooding the observation.

Returns:

  • (Integer) —

    max chars of an error response body in a FetchError message — enough to show what came back (a Cloudflare/WAF page) without flooding the observation.

200

Class Method Summary collapse

Class Method Details

.extract(fetched) ⇒ String

Render a Fetched response as Markdown via Extractor.extract, re-raising both extraction failure modes as FetchError. The content-type is passed verbatim (including "" for a missing header, which matches no text arm — a body without transport metadata is refused, not sniffed; only a strong magic sniff like %PDF- overrides).

Parameters:

Returns:

  • (String) —

    Markdown from the matched extractor

Raises:

  • (FetchError) —

    when no extractor matches, or extraction fails



77
78
79
80
81
82
83
 
# File 'lib/pikuri/tool/scraper.rb', line 77

def self.extract(fetched)
  Pikuri::Extractor.extract(StringIO.new(fetched.body), content_type: fetched.content_type)
rescue Pikuri::Extractor::Unsupported
  raise FetchError, "unsupported content-type #{fetched.content_type.inspect} for #{fetched.url}"
rescue Pikuri::Extractor::Error => e
  raise FetchError, e.message
end

.fetch(url, limit: MAX_REDIRECTS) ⇒ Fetched

Download url, manually following up to MAX_REDIRECTS redirects. All recoverable failures (HTTP 4xx/5xx, Faraday::Error, exhausted redirect budget, 3xx without Location) surface as FetchError. Error bodies are trimmed to ERROR_BODY_EXCERPT so a challenge page doesn't dump kilobytes into the observation.

Parameters:

  • url (String) —

    absolute HTTP(S) URL

  • limit (Integer) (defaults to: MAX_REDIRECTS) —

    redirects remaining; recurses with limit - 1 on 3xx

Returns:

  • (Fetched) —

    body, normalized content-type, final URL

Raises:

  • (FetchError) —

    on non-2xx/3xx, network error, redirect-loop exhaustion, or 3xx without Location



96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
 
# File 'lib/pikuri/tool/scraper.rb', line 96

def self.fetch(url, limit: MAX_REDIRECTS)
  raise FetchError, "too many redirects fetching #{url}" if limit.zero?

  response = begin
    Faraday.new(request: { open_timeout: OPEN_TIMEOUT, timeout: READ_TIMEOUT }).get(url) do |req|
      req.headers['User-Agent'] = USER_AGENT
      req.headers['Accept']     = ACCEPT
    end
  rescue Faraday::Error => e
    raise FetchError, "#{e.class.name.split('::').last} fetching #{url}: #{e.message}"
  end

  case response.status
  when 200..299
    Fetched.new(body: response.body, content_type: normalize_content_type(response.headers['content-type']), url: url)
  when 300..399
    location = response.headers['location']
    raise FetchError, "HTTP #{response.status} from #{url} with no Location header" if location.nil? || location.empty?

    fetch(URI.join(url, location).to_s, limit: limit - 1)
  else
    raise FetchError, "HTTP #{response.status} fetching #{url}: #{excerpt(response.body)}"
  end
end

.visit(url) ⇒ String

Fetch url and render its main content as Markdown. No caching — every call hits the network (wrap it to memoize; WebScrape.visit does). The output is +strip+'d (leading/trailing blank lines are common from PDF page-feeds and trailing newlines) but interior whitespace is preserved (paragraph breaks and code indentation are load-bearing).

Parameters:

  • url (String) —

    absolute HTTP(S) URL

Returns:

  • (String) —

    Markdown, whitespace-trimmed, otherwise uncapped — caller size-limits before feeding the LLM

Raises:

  • (FetchError) —

    on HTTP non-2xx, network failure, redirect loop, a 3xx without Location, an unrecognized response, or extraction failure



64
65
66
 
# File 'lib/pikuri/tool/scraper.rb', line 64

def self.visit(url)
  extract(fetch(url)).strip
end