Module: Karafka::Pro::Cleaner

Defined in:
lib/karafka/pro/cleaner.rb,
lib/karafka/pro/cleaner/errors.rb,
lib/karafka/pro/cleaner/messages/message.rb,
lib/karafka/pro/cleaner/messages/messages.rb,
lib/karafka/pro/cleaner/messages/metadata.rb

Overview

Feature that introduces a granular memory management for each message and messages iterator

It allows for better resource allocation by providing an API to clear payload and raw payload from a message after those are no longer needed but before whole messages are freed and removed by Ruby GC.

This can be useful when processing bigger batches or bigger messages one after another and wanting not to have all of the data loaded into memory.

Can yield significant memory savings (up to 80%).

Defined Under Namespace

Modules: Errors, Messages

Class Method Summary collapse

Class Method Details

.post_fork(_config, _pre_fork_producer) ⇒ Object

This feature does not need any changes post-fork

Parameters:



53
54
55
 
# File 'lib/karafka/pro/cleaner.rb', line 53

def post_fork(_config, _pre_fork_producer)
  true
end

.post_setup(_config) ⇒ Object

Parameters:

  • root node config



45
46
47
 
# File 'lib/karafka/pro/cleaner.rb', line 45

def post_setup(_config)
  true
end

.pre_setup(_config) ⇒ Object

Parameters:

  • root node config



38
39
40
41
42
 
# File 'lib/karafka/pro/cleaner.rb', line 38

def pre_setup(_config)
  Karafka::Messages::Message.prepend(Messages::Message)
  Karafka::Messages::Metadata.prepend(Messages::Metadata)
  Karafka::Messages::Messages.prepend(Messages::Messages)
end