Module: Pikuri::Extractor::Passthrough

Defined in:
lib/pikuri/extractor/passthrough.rb

Overview

The terminal plain-text arm of the registry: content that is already text passes through verbatim (forced to UTF-8, invalid bytes left in, as File.read does). Markdown, source files, JSON, robots.txt land here.

Matching splits on whether the transport gave a content-type:

  • With one (the web path): claim text/* only — a non-text type no earlier extractor claimed is not second-guessed by sniffing (+application/octet-stream+ gets the Unsupported refusal).
  • Without one (the local-file path, FileType.detect_mime → nil): claim anything passing FileType.binary? on the sample; opaque binaries stay unclaimed and surface as Unsupported.

Class Method Summary collapse

Class Method Details

.extract(io) ⇒ String

Returns the content, tagged UTF-8. Deliberately NOT derived from extract_lines — a passthrough stays verbatim (trailing newline, CRLF), which joining chomped lines would normalize away.

Parameters:

  • IO over the text content.

Returns:

  • the content, tagged UTF-8. Deliberately NOT derived from extract_lines — a passthrough stays verbatim (trailing newline, CRLF), which joining chomped lines would normalize away.



37
38
39
 
# File 'lib/pikuri/extractor/passthrough.rb', line 37

def self.extract(io)
  io.read.force_encoding(Encoding::UTF_8)
end

.extract_lines(io) ⇒ Enumerator::Lazy<String>

Lazy line stream for Pikuri::Extractor.extract_paged: read line-by-line, so a window over the head of a gigabyte log never loads the rest. The whole stream is a cheap sequential read — which is why paging counts this tail for an exact total_lines.

Parameters:

  • IO over the text content; must stay open while the enumerator is consumed.

Returns:

  • chomped lines, tagged UTF-8.



49
50
51
 
# File 'lib/pikuri/extractor/passthrough.rb', line 49

def self.extract_lines(io)
  io.each_line.lazy.map { |raw| raw.chomp.force_encoding(Encoding::UTF_8) }
end

.kind ⇒ Symbol

Returns Pikuri::Extractor::Page#kind tag.

Returns:



19
20
21
 
# File 'lib/pikuri/extractor/passthrough.rb', line 19

def self.kind
  :text
end

.matches?(sample:, content_type:) ⇒ Boolean

Parameters:

  • leading bytes of the content.

  • normalized content-type, nil when the transport has none.

Returns:



27
28
29
30
31
 
# File 'lib/pikuri/extractor/passthrough.rb', line 27

def self.matches?(sample:, content_type:)
  return content_type.start_with?('text/') unless content_type.nil?

  !FileType.binary?(sample)
end