Class: Karafka::Routing::SubscriptionGroupsBuilder

Inherits:
Object
  • Object
show all
Defined in:
lib/karafka/routing/subscription_groups_builder.rb

Overview

rdkafka allows us to group topics subscriptions when they have same settings. This builder groups topics from a single consumer group into subscription groups that can be subscribed with one rdkafka connection. This way we save resources as having several rdkafka consumers under the hood is not the cheapest thing in a bigger system.

In general, if we can, we try to subscribe to as many topics with one rdkafka connection as possible, but if not possible, we divide.

Instance Method Summary collapse

Constructor Details

#initialize ⇒ SubscriptionGroupsBuilder

Returns a new instance of SubscriptionGroupsBuilder.



27
28
29
 
# File 'lib/karafka/routing/subscription_groups_builder.rb', line 27

def initialize
  @position = -1
end

Instance Method Details

#call(topics) ⇒ Array<SubscriptionGroup>

Returns all subscription groups we need in separate threads.

Parameters:

Returns:



34
35
36
37
38
39
40
41
42
 
# File 'lib/karafka/routing/subscription_groups_builder.rb', line 34

def call(topics)
  topics
    .map { |topic| [checksum(topic), topic] }
    .group_by(&:first)
    .values
    .map { |value| value.map(&:last) }
    .map { |topics_array| Routing::Topics.new(topics_array) }
    .map { |grouped_topics| SubscriptionGroup.new(@position += 1, grouped_topics) }
end