Class: RubricLLM::Metrics::Relevance

Inherits:
Base
  • Object
show all
Defined in:
lib/rubric_llm/metrics/relevance.rb

Constant Summary collapse

SYSTEM_PROMPT = 
<<~PROMPT
  You are an evaluation judge. Assess whether the answer is relevant to the question.
  A relevant answer directly addresses what was asked, regardless of its factual accuracy.
  Score 1.0 if it directly addresses all parts of the question, 0.5 if it addresses only
  part or is mostly tangential, and 0.0 if it does not address the question. Use intermediate
  values for partial cases and explain the score. Do not score correctness here.

  Respond with JSON only:
  {
    "score": <float 0.0-1.0>,
    "reasoning": "<brief explanation>"
  }
PROMPT

Instance Attribute Summary

Attributes inherited from Base

#judge

Instance Method Summary collapse

Methods inherited from Base

#initialize, normalize_context, require_context!

Constructor Details

This class inherits a constructor from RubricLLM::Metrics::Base

Instance Method Details

#call(question:, answer:) ⇒ Object 



20
21
22
23
24
25
26
27
28
29
30
31
 
# File 'lib/rubric_llm/metrics/relevance.rb', line 20

def call(question:, answer:, **)
  user_prompt = <<~PROMPT
    Question: #{question}

    Answer: #{answer}

    Evaluate how relevant the answer is to the question.
  PROMPT

  result = judge_eval(system_prompt: SYSTEM_PROMPT, user_prompt:)
  normalize(result)
end