Class: PromptSanitizer::Engines::NEREngine

Inherits:
Object
  • Object
show all
Defined in:
lib/prompt_sanitizer/engines/ner_engine.rb

Overview

NER Engine — Layer 2 of prompt-sanitizer (SMART / FULL mode only).

Detects context-dependent PII that regex cannot catch: person names, organisations, locations, and miscellaneous named entities.

Two backends are supported:

:informers — uses the `informers` gem with Xenova/distilbert-NER or
           Xenova/bert-base-NER (ONNX int8).  F1 92.17.  ~25–50 ms.
           Model auto-downloaded to ~/.cache/huggingface/ on first use.
           Recommended default.

:mitie     — uses the `mitie` gem with the MITIE C++ NER model.
           F1 88.10.  ~2 ms.  Requires separate model download (~600 MB).
           Use when sub-5 ms latency is required.

Both backends are optional runtime dependencies. When neither gem is installed, NEREngine#available? returns false and #detect returns []. The Sanitizer falls back to FAST mode silently in this case.

Thread safety: both ONNX Runtime sessions and MITIE models are immutable after loading — safe to share across Puma threads.

Constant Summary collapse

CHUNK_SIZE =

Maximum number of characters per chunk when splitting long prompts. distilbert / bert-base have a 512 subword-token limit; ~300 words ≈ 1,800 characters is a safe conservative ceiling.

1_800
CHUNK_OVERLAP =

overlap between chunks to avoid edge-case misses

200
TAG_MAP =

BIO tag → EntityType mapping (CoNLL-2003 schema)

{
  "PER"  => EntityType::PERSON,
  "ORG"  => EntityType::ORGANIZATION,
  "LOC"  => EntityType::LOCATION,
  "MISC" => EntityType::MISC,
}.freeze

Instance Method Summary collapse

Constructor Details

#initialize(backend: :informers, model: "distilbert") ⇒ NEREngine

Returns a new instance of NEREngine.

Parameters:

  • backend (Symbol) (defaults to: :informers) —

    :informers (default) or :mitie

  • model (String) (defaults to: "distilbert") —

    "distilbert" (default) or "bert-base" for the informers backend; path to ner_model.dat for the mitie backend.



48
49
50
51
52
53
54
 
# File 'lib/prompt_sanitizer/engines/ner_engine.rb', line 48

def initialize(backend: :informers, model: "distilbert")
  @backend = backend
  @model   = model
  @pipeline = nil
  @mutex    = Mutex.new
  _load_backend
end

Instance Method Details

#available? ⇒ Boolean

Returns true when the chosen backend gem is installed and the model is ready to use.

Returns:

  • (Boolean) 


58
59
60
 
# File 'lib/prompt_sanitizer/engines/ner_engine.rb', line 58

def available?
  !@pipeline.nil?
end

#detect(text) ⇒ Object

Detect named entities in text and return Array. Returns [] immediately when the backend is unavailable.

Long texts are automatically chunked and results are merged.



66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
 
# File 'lib/prompt_sanitizer/engines/ner_engine.rb', line 66

def detect(text)
  return [] unless available?

  safe_text = text.encode("UTF-8", invalid: :replace, undef: :replace, replace: "")
  return [] if safe_text.strip.empty?

  if safe_text.length > CHUNK_SIZE
    _detect_chunked(safe_text)
  else
    _detect_single(safe_text)
  end
rescue => e
  # Never let NER failures break the sanitizer — degrade gracefully.
  warn "[PromptSanitizer] NER error (#{@backend}): #{e.message}"
  []
end