Class: KicksLiveness::Heartbeat

Inherits:
Object
  • Object
show all
Includes:
GenerationGuard
Defined in:
lib/kicks_liveness/heartbeat.rb

Overview

The liveness marks on the filesystem: written by the worker, read by the probe.

The directory must live on tmpfs — in Kubernetes, an emptyDir with medium: Memory. Put it on a real disk and the probe starts depending on the disk again, and a stalling disk is one of the most common causes of false restarts.

This file deliberately runs no require on the probe's path and refers to nothing else in the gem: the probe can load it alone, with no application code and no Rails behind it. That keeps the optional direct form at ~43 ms and lets it run on a bare interpreter with --disable-gems when the image allows it -- whether Bundler is in the picture depends on where the image put its gems, not on this file. The one require it does contain is lazy, in a branch only the worker reaches.

See Also:

Constant Summary collapse

DEFAULT_DIR =

Returns marks directory used when the environment says nothing.

Returns:

  • marks directory used when the environment says nothing

'/opt/app/tmp/health'.freeze
DEFAULT_MAX_AGE =

Returns seconds after which a mark is stale, by default.

Returns:

  • seconds after which a mark is stale, by default

45
ENV_NAMES =

Returns setting name to environment variable.

Returns:

  • setting name to environment variable

{
  dir: 'KICKS_LIVENESS_DIR',
  max_age: 'KICKS_LIVENESS_MAX_AGE',
  tick: 'KICKS_LIVENESS_TICK'
}.freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(dir: Heartbeat.env_dir, max_age: Heartbeat.env_max_age, generation: Heartbeat.container_generation) ⇒ Heartbeat

Returns a new instance of Heartbeat.

Parameters:

  • (defaults to: Heartbeat.env_dir)

    marks directory

  • (defaults to: Heartbeat.env_max_age)

    seconds after which a mark is considered stale

  • (defaults to: Heartbeat.container_generation)

    container incarnation; injected in specs



140
141
142
143
144
145
146
147
148
 
# File 'lib/kicks_liveness/heartbeat.rb', line 140

def initialize(
  dir: Heartbeat.env_dir,
  max_age: Heartbeat.env_max_age,
  generation: Heartbeat.container_generation
)
  @dir = dir
  @max_age = max_age
  @generation = generation
end

Instance Attribute Details

#dir ⇒ Object (readonly)

Returns the value of attribute dir.



150
151
152
 
# File 'lib/kicks_liveness/heartbeat.rb', line 150

def dir
  @dir
end

#max_age ⇒ Object (readonly)

Returns the value of attribute max_age.



150
151
152
 
# File 'lib/kicks_liveness/heartbeat.rb', line 150

def max_age
  @max_age
end

Class Method Details

.container_generation ⇒ String?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Linux exposes a stable identifier shared by a private-PID container and its exec probes. A restarted container gets a new identifier even though its Kubernetes emptyDir survives.

Returns:

  • current private-PID container incarnation, or nil off Linux

API:

  • private



92
93
94
 
# File 'lib/kicks_liveness/heartbeat.rb', line 92

def self.container_generation
  ContainerGeneration.current
end

.env_dir ⇒ String

Returns marks directory.

Returns:

  • marks directory



128
129
130
 
# File 'lib/kicks_liveness/heartbeat.rb', line 128

def self.env_dir
  env_raw(:dir) || DEFAULT_DIR
end

.env_int(key, default) ⇒ Integer

Values arrive from a ConfigMap, and a typo there is no reason to bring a worker down — garbage falls back to the default.

Both settings are durations, so a value has to be parseable and positive. Zero and negative numbers parse perfectly well and are the more dangerous half: a tick of zero turns the monitor into a hot loop, a negative one used to kill the monitor thread on its first sleep, and a negative max_age makes every mark stale on arrival, so the probe can never pass again.

Parameters:

Returns:

  • the value from the environment when it is a positive integer, the default otherwise



120
121
122
123
124
125
 
# File 'lib/kicks_liveness/heartbeat.rb', line 120

def self.env_int(key, default)
  value = Integer(env_raw(key) || default)
  value.positive? ? value : default
rescue ArgumentError, TypeError
  default
end

.env_max_age ⇒ Integer

Returns seconds after which a mark is considered stale.

Returns:

  • seconds after which a mark is considered stale



133
134
135
 
# File 'lib/kicks_liveness/heartbeat.rb', line 133

def self.env_max_age
  env_int(:max_age, DEFAULT_MAX_AGE)
end

.env_raw(key) ⇒ String?

An empty string counts as unset: in a ConfigMap that is what you get by declaring a key and leaving it blank.

Parameters:

Returns:



101
102
103
104
 
# File 'lib/kicks_liveness/heartbeat.rb', line 101

def self.env_raw(key)
  value = ENV.fetch(ENV_NAMES.fetch(key), nil)
  value unless value.nil? || value.empty?
end

Instance Method Details

#check(now: Time.now.utc) ⇒ Array(Boolean, String)

The probe side: is every declared fork's mark present and fresh?

The returned message names slots the way the files are named, so that a human reading the Unhealthy event knows which file to look at.

Parameters:

  • (defaults to: Time.now.utc)

    injected in specs

Returns:

  • health, and the reason to print on stdout



198
199
200
201
202
203
204
205
206
 
# File 'lib/kicks_liveness/heartbeat.rb', line 198

def check(now: Time.now.utc)
  processes = expected
  return [false, "no #{expected_path}: worker has not started yet"] unless processes&.positive?
  return [false, 'heartbeat belongs to a previous container: worker has not started yet'] unless current_generation?

  problems = (0...processes).filter_map { |slot| problem_for(slot, now) }

  problems.empty? ? [true, "#{processes} process(es) healthy"] : [false, problems.join('; ')]
end

#declare!(processes) ⇒ void

This method returns an undefined value.

Records how many forks the probe must wait for. Every fork writes the same value. Without it the probe would pass as soon as any single mark was fresh, while half the workers had not subscribed yet.

Written through a rename, because a plain write truncates first: a probe reading in that window finds the file empty and reports that the worker has not started. The window is real and recurring — every fork rewrites this file on every tick, while the liveness probe reads it on a schedule of its own. Rename is atomic on tmpfs.

Parameters:



164
165
166
167
168
 
# File 'lib/kicks_liveness/heartbeat.rb', line 164

def declare!(processes)
  make_dir
  atomic_write(generation_path, @generation) if @generation
  atomic_write(expected_path, processes)
end

#touch!(slot) ⇒ Integer

Refreshes this fork's mark.

The file is named by supervisor slot, not by PID: a fork killed with SIGKILL is respawned into the same slot and overwrites its own file. With a PID in the name that file would stay stale forever and the probe would fail permanently.

The timestamp, pid and slot exist for a human running kubectl exec ... cat. The generation is also checked by the probe: a Kubernetes emptyDir survives a container restart, so freshness alone cannot distinguish this process from the one that just exited.

Parameters:

  • supervisor slot of this fork

Returns:

  • bytes written



184
185
186
187
188
189
 
# File 'lib/kicks_liveness/heartbeat.rb', line 184

def touch!(slot)
  make_dir
  contents = "#{Time.now.utc.strftime('%FT%TZ')} pid=#{Process.pid} slot=#{slot}"
  contents = "#{contents} generation=#{@generation}" if @generation
  atomic_write(slot_path(slot), "#{contents}\n")
end