Method: LlmMemory::Broca#tokenizer

Defined in:
lib/llm_memory/broca.rb

#tokenizer ⇒ Object

[View source]

107
108
109
 
# File 'lib/llm_memory/broca.rb', line 107

def tokenizer
  @tokenizer ||= Tokenizers.from_pretrained("gpt2")
end