Class: Boxcars::VectorStore::InMemory::BuildFromFiles

Inherits:
Object
  • Object
show all
Includes:
Boxcars::VectorStore
Defined in:
lib/boxcars/vector_store/in_memory/build_from_files.rb

Instance Method Summary collapse

Methods included from Boxcars::VectorStore

included

Constructor Details

#initialize(params) ⇒ Hash

initialize the vector store with the following parameters:

Parameters:

  • params (Hash) —

    A Hash containing the initial configuration.

Options Hash (params):

  • :embedding_tool (Symbol) —

    The embedding tool to use.

  • :training_data_path (String) —

    The path to the training data files.

  • :split_chunk_size (Integer) —

    The number of characters to split the text into.



15
16
17
18
19
20
21
22
 
# File 'lib/boxcars/vector_store/in_memory/build_from_files.rb', line 15

def initialize(params)
  @split_chunk_size = params[:split_chunk_size] || 2000
  @training_data_path = File.absolute_path(params[:training_data_path])
  @embedding_tool = params[:embedding_tool] || :openai

  validate_params(embedding_tool, training_data_path)
  @memory_vectors = []
end

Instance Method Details

#call ⇒ Hash

Returns vector_store: array of hashes with :content, :metadata, and :embedding keys.

Returns:

  • (Hash) —

    vector_store: array of hashes with :content, :metadata, and :embedding keys



25
26
27
28
29
30
31
32
33
34
35
 
# File 'lib/boxcars/vector_store/in_memory/build_from_files.rb', line 25

def call
  data = load_data_files(training_data_path)
  texts = split_text_into_chunks(data)
  vectors = generate_vectors(texts)
  add_vectors(vectors, texts)

  {
    type: :in_memory,
    vector_store: memory_vectors
  }
end