Class: Google::Cloud::DiscoveryEngine::V1beta::QualityMetrics

Inherits:
Object
  • Object
show all
Extended by:
Protobuf::MessageExts::ClassMethods
Includes:
Protobuf::MessageExts
Defined in:
proto_docs/google/cloud/discoveryengine/v1beta/evaluation.rb

Overview

Describes the metrics produced by the evaluation.

Defined Under Namespace

Classes: TopkMetrics

Instance Attribute Summary collapse

Instance Attribute Details

#doc_ndcg ⇒ ::Google::Cloud::DiscoveryEngine::V1beta::QualityMetrics::TopkMetrics

Returns Normalized discounted cumulative gain (NDCG) per document, at various top-k cutoff levels.

NDCG measures the ranking quality, giving higher relevance to top results.

Example (top-3): Suppose SampleQuery with three retrieved documents (D1, D2, D3) and binary relevance judgements (1 for relevant, 0 for not relevant):

Retrieved: [D3 (0), D1 (1), D2 (1)] Ideal: [D1 (1), D2 (1), D3 (0)]

Calculate NDCG@3 for each SampleQuery:

  • DCG@3: 0/log2(1+1) + 1/log2(2+1) + 1/log2(3+1) = 1.13
  • Ideal DCG@3: 1/log2(1+1) + 1/log2(2+1) + 0/log2(3+1) = 1.63
  • NDCG@3: 1.13/1.63 = 0.693.

Returns:

  • (::Google::Cloud::DiscoveryEngine::V1beta::QualityMetrics::TopkMetrics) —

    Normalized discounted cumulative gain (NDCG) per document, at various top-k cutoff levels.

    NDCG measures the ranking quality, giving higher relevance to top results.

    Example (top-3): Suppose SampleQuery with three retrieved documents (D1, D2, D3) and binary relevance judgements (1 for relevant, 0 for not relevant):

    Retrieved: [D3 (0), D1 (1), D2 (1)] Ideal: [D1 (1), D2 (1), D3 (0)]

    Calculate NDCG@3 for each SampleQuery:

    • DCG@3: 0/log2(1+1) + 1/log2(2+1) + 1/log2(3+1) = 1.13
    • Ideal DCG@3: 1/log2(1+1) + 1/log2(2+1) + 0/log2(3+1) = 1.63
    • NDCG@3: 1.13/1.63 = 0.693


198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'proto_docs/google/cloud/discoveryengine/v1beta/evaluation.rb', line 198

class QualityMetrics
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods

  # Stores the metric values at specific top-k levels.
  # @!attribute [rw] top_1
  #   @return [::Float]
  #     The top-1 value.
  # @!attribute [rw] top_3
  #   @return [::Float]
  #     The top-3 value.
  # @!attribute [rw] top_5
  #   @return [::Float]
  #     The top-5 value.
  # @!attribute [rw] top_10
  #   @return [::Float]
  #     The top-10 value.
  class TopkMetrics
    include ::Google::Protobuf::MessageExts
    extend ::Google::Protobuf::MessageExts::ClassMethods
  end
end

#doc_precision ⇒ ::Google::Cloud::DiscoveryEngine::V1beta::QualityMetrics::TopkMetrics

Returns Precision per document, at various top-k cutoff levels.

Precision is the fraction of retrieved documents that are relevant.

Example (top-5):

  • For a single SampleQuery, If 4 out of 5 retrieved documents in the top-5 are relevant, precision@5 = 4/5 = 0.8.

Returns:



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'proto_docs/google/cloud/discoveryengine/v1beta/evaluation.rb', line 198

class QualityMetrics
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods

  # Stores the metric values at specific top-k levels.
  # @!attribute [rw] top_1
  #   @return [::Float]
  #     The top-1 value.
  # @!attribute [rw] top_3
  #   @return [::Float]
  #     The top-3 value.
  # @!attribute [rw] top_5
  #   @return [::Float]
  #     The top-5 value.
  # @!attribute [rw] top_10
  #   @return [::Float]
  #     The top-10 value.
  class TopkMetrics
    include ::Google::Protobuf::MessageExts
    extend ::Google::Protobuf::MessageExts::ClassMethods
  end
end

#doc_recall ⇒ ::Google::Cloud::DiscoveryEngine::V1beta::QualityMetrics::TopkMetrics

Returns Recall per document, at various top-k cutoff levels.

Recall is the fraction of relevant documents retrieved out of all relevant documents.

Example (top-5):

  • For a single SampleQuery, If 3 out of 5 relevant documents are retrieved in the top-5, recall@5 = 3/5 = 0.6.

Returns:



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'proto_docs/google/cloud/discoveryengine/v1beta/evaluation.rb', line 198

class QualityMetrics
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods

  # Stores the metric values at specific top-k levels.
  # @!attribute [rw] top_1
  #   @return [::Float]
  #     The top-1 value.
  # @!attribute [rw] top_3
  #   @return [::Float]
  #     The top-3 value.
  # @!attribute [rw] top_5
  #   @return [::Float]
  #     The top-5 value.
  # @!attribute [rw] top_10
  #   @return [::Float]
  #     The top-10 value.
  class TopkMetrics
    include ::Google::Protobuf::MessageExts
    extend ::Google::Protobuf::MessageExts::ClassMethods
  end
end

#page_ndcg ⇒ ::Google::Cloud::DiscoveryEngine::V1beta::QualityMetrics::TopkMetrics

Returns Normalized discounted cumulative gain (NDCG) per page, at various top-k cutoff levels.

NDCG measures the ranking quality, giving higher relevance to top results.

Example (top-3): Suppose SampleQuery with three retrieved pages (P1, P2, P3) and binary relevance judgements (1 for relevant, 0 for not relevant):

Retrieved: [P3 (0), P1 (1), P2 (1)] Ideal: [P1 (1), P2 (1), P3 (0)]

Calculate NDCG@3 for SampleQuery:

  • DCG@3: 0/log2(1+1) + 1/log2(2+1) + 1/log2(3+1) = 1.13
  • Ideal DCG@3: 1/log2(1+1) + 1/log2(2+1) + 0/log2(3+1) = 1.63
  • NDCG@3: 1.13/1.63 = 0.693.

Returns:

  • (::Google::Cloud::DiscoveryEngine::V1beta::QualityMetrics::TopkMetrics) —

    Normalized discounted cumulative gain (NDCG) per page, at various top-k cutoff levels.

    NDCG measures the ranking quality, giving higher relevance to top results.

    Example (top-3): Suppose SampleQuery with three retrieved pages (P1, P2, P3) and binary relevance judgements (1 for relevant, 0 for not relevant):

    Retrieved: [P3 (0), P1 (1), P2 (1)] Ideal: [P1 (1), P2 (1), P3 (0)]

    Calculate NDCG@3 for SampleQuery:

    • DCG@3: 0/log2(1+1) + 1/log2(2+1) + 1/log2(3+1) = 1.13
    • Ideal DCG@3: 1/log2(1+1) + 1/log2(2+1) + 0/log2(3+1) = 1.63
    • NDCG@3: 1.13/1.63 = 0.693


198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'proto_docs/google/cloud/discoveryengine/v1beta/evaluation.rb', line 198

class QualityMetrics
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods

  # Stores the metric values at specific top-k levels.
  # @!attribute [rw] top_1
  #   @return [::Float]
  #     The top-1 value.
  # @!attribute [rw] top_3
  #   @return [::Float]
  #     The top-3 value.
  # @!attribute [rw] top_5
  #   @return [::Float]
  #     The top-5 value.
  # @!attribute [rw] top_10
  #   @return [::Float]
  #     The top-10 value.
  class TopkMetrics
    include ::Google::Protobuf::MessageExts
    extend ::Google::Protobuf::MessageExts::ClassMethods
  end
end

#page_recall ⇒ ::Google::Cloud::DiscoveryEngine::V1beta::QualityMetrics::TopkMetrics

Returns Recall per page, at various top-k cutoff levels.

Recall is the fraction of relevant pages retrieved out of all relevant pages.

Example (top-5):

  • For a single SampleQuery, if 3 out of 5 relevant pages are retrieved in the top-5, recall@5 = 3/5 = 0.6.

Returns:



198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'proto_docs/google/cloud/discoveryengine/v1beta/evaluation.rb', line 198

class QualityMetrics
  include ::Google::Protobuf::MessageExts
  extend ::Google::Protobuf::MessageExts::ClassMethods

  # Stores the metric values at specific top-k levels.
  # @!attribute [rw] top_1
  #   @return [::Float]
  #     The top-1 value.
  # @!attribute [rw] top_3
  #   @return [::Float]
  #     The top-3 value.
  # @!attribute [rw] top_5
  #   @return [::Float]
  #     The top-5 value.
  # @!attribute [rw] top_10
  #   @return [::Float]
  #     The top-10 value.
  class TopkMetrics
    include ::Google::Protobuf::MessageExts
    extend ::Google::Protobuf::MessageExts::ClassMethods
  end
end