Class: Langfuse::PromptCache

Inherits:
Object
  • Object
show all
Defined in:
lib/langfuse/prompt_cache.rb

Overview

Bounded, thread-safe cache of Prompt objects, keyed by name/version/label.

TTLs are measured on the monotonic clock so wall-clock jumps (NTP, DST) cannot extend or shorten an entry's lifetime. Entries keep insertion order and the oldest ones are evicted once the cache is full, so a long-running process with many prompt names cannot grow it without bound.

Defined Under Namespace

Classes: Entry

Constant Summary collapse

DEFAULT_MAX_ENTRIES = 
200

Instance Method Summary collapse

Constructor Details

#initialize(max_entries: DEFAULT_MAX_ENTRIES) ⇒ PromptCache

Returns a new instance of PromptCache.



23
24
25
26
27
 
# File 'lib/langfuse/prompt_cache.rb', line 23

def initialize(max_entries: DEFAULT_MAX_ENTRIES)
  @max_entries = [max_entries.to_i, 0].max
  @entries = Concurrent::Hash.new
  @mutex = Mutex.new
end

Instance Method Details

#length ⇒ Object 



55
56
57
 
# File 'lib/langfuse/prompt_cache.rb', line 55

def length
  @entries.length
end

#read(key, ttl_seconds) ⇒ Object

The cached prompt while it is still fresh, otherwise nil.



30
31
32
33
34
35
 
# File 'lib/langfuse/prompt_cache.rb', line 30

def read(key, ttl_seconds)
  entry = @entries[key]
  return nil unless entry&.fresh?(ttl_seconds, monotonic_time)

  entry.prompt
end

#read_stale(key) ⇒ Object

The cached prompt regardless of age. Used to keep serving prompts while the Langfuse API is unreachable.



39
40
41
 
# File 'lib/langfuse/prompt_cache.rb', line 39

def read_stale(key)
  @entries[key]&.prompt
end

#write(key, prompt) ⇒ Object

Writes are serialized so concurrent fetches cannot corrupt the hash while it is being trimmed. Returns the prompt for convenient chaining.



45
46
47
48
49
50
51
52
53
 
# File 'lib/langfuse/prompt_cache.rb', line 45

def write(key, prompt)
  @mutex.synchronize do
    @entries.delete(key)
    @entries[key] = Entry.new(prompt, monotonic_time)
    @entries.shift while @entries.length > @max_entries && !@entries.empty?
  end

  prompt
end