Class: RubricLLM::Metrics::FactualAccuracy

Inherits:
Base
  • Object
show all
Defined in:
lib/rubric_llm/metrics/factual_accuracy.rb

Constant Summary collapse

SYSTEM_PROMPT = 
<<~PROMPT
  You are an evaluation judge. Compare the factual claims in the candidate answer against the reference answer.
  Identify contradictions in candidate factual claims against the reference. Do not penalize
  missing reference facts, which correctness measures. Score 1.0 if there are no contradictions,
  0.5 if minor factual contradictions affect part of the answer, and 0.0 if major contradictions
  undermine the answer. Use intermediate values for partial cases. Explain the score and list
  each contradiction with severity minor or major. If the reference does not establish whether
  a claim is true, do not call it a contradiction.

  Respond with JSON only:
  {
    "score": <float 0.0-1.0>,
    "discrepancies": [{"claim": "<candidate claim>", "reference": "<what reference says>", "severity": "minor|major"}],
    "reasoning": "<brief explanation>"
  }
PROMPT

Instance Attribute Summary

Attributes inherited from Base

#judge

Instance Method Summary collapse

Methods inherited from Base

#initialize, normalize_context, require_context!

Constructor Details

This class inherits a constructor from RubricLLM::Metrics::Base

Instance Method Details

#call(answer:, ground_truth: nil) ⇒ Object 



23
24
25
26
27
28
29
30
31
32
33
34
35
36
 
# File 'lib/rubric_llm/metrics/factual_accuracy.rb', line 23

def call(answer:, ground_truth: nil, **)
  return { score: nil, details: { error: "No ground truth provided" } } if ground_truth.nil?

  user_prompt = <<~PROMPT
    Candidate Answer: #{answer}

    Reference Answer: #{ground_truth}

    Compare the factual claims and identify any discrepancies.
  PROMPT

  result = judge_eval(system_prompt: SYSTEM_PROMPT, user_prompt:)
  normalize(result)
end