Class: Readyset::Health::Healthchecker

Inherits:
Object
  • Object
show all
Defined in:
lib/readyset/health/healthchecker.rb

Overview

Processes the given exceptions to determine whether ReadySet is currently unhealthy. If ReadySet is indeed unhealthy, a background task is spawned that periodically checks ReadySet’s health directly until a healthy state has been restored. While ReadySet is in an unhealthy state, ‘Healthchecker#healthy?` will return false.

Constant Summary collapse

UNHEALTHY_ERRORS = 
[::PG::UnableToSend, ::PG::ConnectionBad].freeze

Instance Method Summary collapse

Constructor Details

#initialize(config, shard:) ⇒ Healthchecker

Returns a new instance of Healthchecker.



15
16
17
18
19
20
21
22
23
24
25
 
# File 'lib/readyset/health/healthchecker.rb', line 15

def initialize(config, shard:)
  @healthy = Concurrent::AtomicBoolean.new(true)
  @healthcheck_interval = config.healthcheck_interval!
  @healthchecks = Health::Healthchecks.new(shard: shard)
  @lock = Mutex.new
  @shard = shard
  @window_counter = Readyset::Utils::WindowCounter.new(
    window_size: config.error_window_size!,
    time_period: config.error_window_period!,
  )
end

Instance Method Details

#healthy? ⇒ Boolean

Returns true only if the connection to ReadySet is healthy. ReadySet’s health is gauged by keeping track of the number of connection errors that have occurred over a given time period. If the number of errors in that time period exceeds the preconfigured threshold, ReadySet is considered to be unhealthy.

Returns:

  • (Boolean) —

    whether ReadySet is healthy



33
34
35
 
# File 'lib/readyset/health/healthchecker.rb', line 33

def healthy?
  healthy.true?
end

#process_exception(exception) ⇒ Object

Checks if the given exception is a connection error that occurred on a ReadySet connection, and if so, logs the error internally. If ReadySet is unhealthy, a background task is spawned that periodically tries to connect to ReadySet and check its status. When this task determines that ReadySet is healthy again, the task is shut down and the state of the healthchecker is switched back to “healthy”.

Parameters:

  • the (Exception) —

    exception to be processed



44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/readyset/health/healthchecker.rb', line 44

def process_exception(exception)
  is_readyset_connection_error = is_readyset_connection_error?(exception)
  window_counter.log if is_readyset_connection_error

  # We lock here to ensure that only one thread starts the healthcheck task
  lock.lock
  if healthy.true? && window_counter.threshold_crossed?
    healthy.make_false
    lock.unlock

    logger.warn('ReadySet unhealthy: Routing queries to their original destination until ' \
      'ReadySet becomes healthy again')

    disconnect_readyset_pool!
    task.execute
  end
ensure
  lock.unlock if lock.locked?
end