Class: Langchain::Evals::Ragas::Faithfulness

Inherits:
Object
  • Object
show all
Defined in:
lib/langchain/evals/ragas/faithfulness.rb

Overview

Faithfulness refers to the idea that the answer should be grounded in the given context, ensuring that the retrieved context can act as a justification for the generated answer. The answer is faithful to the context if the claims that are made in the answer can be inferred from the context.

Score calculation: F = |V| / |S|

F = Faithfulness |V| = Number of statements that were supported according to the LLM |S| = Total number of statements extracted.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(llm:) ⇒ Faithfulness

Returns a new instance of Faithfulness.

Parameters:



21
22
23
 
# File 'lib/langchain/evals/ragas/faithfulness.rb', line 21

def initialize(llm:)
  @llm = llm
end

Instance Attribute Details

#llm ⇒ Object (readonly)

Returns the value of attribute llm.



18
19
20
 
# File 'lib/langchain/evals/ragas/faithfulness.rb', line 18

def llm
  @llm
end

Instance Method Details

#score(question:, answer:, context:) ⇒ Float

Returns Faithfulness score.

Parameters:

  • question (String) —

    Question

  • answer (String) —

    Answer

  • context (String) —

    Context

Returns:

  • (Float) —

    Faithfulness score



29
30
31
32
33
34
35
36
37
38
39
 
# File 'lib/langchain/evals/ragas/faithfulness.rb', line 29

def score(question:, answer:, context:)
  statements = statements_extraction(question: question, answer: answer)
  statements_count = statements
    .split("\n")
    .count

  verifications = statements_verification(statements: statements, context: context)
  verifications_count = count_verified_statements(verifications)

  (verifications_count.to_f / statements_count.to_f)
end