Class: Langchain::Utils::TokenLength::CohereValidator

Inherits:
BaseValidator
  • Object
show all
Defined in:
lib/langchain/utils/token_length/cohere_validator.rb

Overview

This class is meant to validate the length of the text passed in to Cohere’s API. It is used to validate the token length before the API call is made

Constant Summary collapse

TOKEN_LIMITS = 
{
  # Source:
  # https://docs.cohere.com/docs/models
  "command-light" => 4096,
  "command" => 4096,
  "base-light" => 2048,
  "base" => 2048,
  "embed-english-light-v2.0" => 512,
  "embed-english-v2.0" => 512,
  "embed-multilingual-v2.0" => 256,
  "summarize-medium" => 2048,
  "summarize-xlarge" => 2048
}.freeze

Class Method Summary collapse

Methods inherited from BaseValidator

limit_exceeded_exception, validate_max_tokens!

Class Method Details

.token_length(text, model_name, options = {}) ⇒ Integer

Calculate token length for a given text and model name

Parameters:

  • text (String) —

    The text to calculate the token length for

  • model_name (String) —

    The model name to validate against

Returns:

  • (Integer) —

    The token length of the text



33
34
35
36
 
# File 'lib/langchain/utils/token_length/cohere_validator.rb', line 33

def self.token_length(text, model_name, options = {})
  res = options[:llm].tokenize(text: text)
  res["tokens"].length
end

.token_length_from_messages(messages, model_name, options) ⇒ Object 



43
44
45
 
# File 'lib/langchain/utils/token_length/cohere_validator.rb', line 43

def self.token_length_from_messages(messages, model_name, options)
  messages.sum { |message| token_length(message.to_json, model_name, options) }
end

.token_limit(model_name) ⇒ Object 



38
39
40
 
# File 'lib/langchain/utils/token_length/cohere_validator.rb', line 38

def self.token_limit(model_name)
  TOKEN_LIMITS[model_name]
end