Class: AITextSplitter

Inherits:
Object
  • Object
show all
Defined in:
lib/cl/magic/common/ai_text_splitter.rb

Instance Method Summary collapse

Constructor Details

#initialize(max_chunk_size, logger) ⇒ AITextSplitter

Returns a new instance of AITextSplitter.



5
6
7
8
9
 
# File 'lib/cl/magic/common/ai_text_splitter.rb', line 5

def initialize(max_chunk_size, logger)
  @max_chunk_size = max_chunk_size
  @cut_off = (@max_chunk_size + (@max_chunk_size * 0.1)).floor
  @logger = logger
end

Instance Method Details

#split(data, split_as_markdown = false, separator) ⇒ Object 



11
12
13
14
 
# File 'lib/cl/magic/common/ai_text_splitter.rb', line 11

def split(data, split_as_markdown=false, separator)
  return markdown_to_array(data) if split_as_markdown
  return basic_split_then_reduce(data, separator)
end