Design

Why this gem is shaped the way it is. If you only want to deploy it, read KUBERNETES.md instead.

The problem with the usual liveness probe

The common implementation is a rake task, invoked by the kubelet through exec, that boots the application and asks the broker for consumer_count on the queues it expects to be served. It has four defects, and three of them cannot be tuned away.

It is expensive. Booting a full Rails application every few seconds, inside the worker's own cgroup. A probe measured at 2.7 s of CPU per invocation on a 15 s period consumes 2.7 / 15 ≈ 0.18 of a core continuously, per pod, forever. Multiply by your replica count and compare it against the CPU request of the container; on a small worker fleet the probe can easily outweigh the work.

It depends on the disk. Loading code means reading files. A disk that stalls for a few seconds stretches the probe past timeoutSeconds, and the kubelet kills a pod that was perfectly healthy. A stalling disk is one of the most common causes of false restarts, and a probe that reads from it converts that cause into an outage.

It cascades. A liveness probe that checks an external dependency makes that dependency a participant in your restart logic. The broker hiccups for ten seconds, every replica fails its probe at the same moment, and the whole fleet restarts simultaneously — reconnecting all at once, which is exactly what a struggling broker least needs.

It measures the wrong thing. consumer_count is a property of a queue, not of a pod. With two replicas, one healthy consumer keeps the count above zero while the other replica is hung and consuming nothing. The metric that is supposed to detect a dead pod is structurally blind to it.

The inversion

Turn the data flow around. The worker knows its own state better than anything outside it can, so let the worker publish and let the probe read.

Every tick seconds (10 by default) the worker checks, in its own memory, that its consumers are still subscribed, and touches a file on tmpfs. The probe loads one dependency-free Ruby file, checks the marks' container generation and mtimes, and exits with 0 or 1.

The probe therefore performs no network I/O and boots no framework, and the state it reads — the generation and mark mtimes — comes from procfs and tmpfs. Be precise about the disk, though: starting the probe still loads the Ruby interpreter and two files of this gem from the image filesystem, and those reads are ordinary filesystem reads (usually served from page cache, but not guaranteed to be). The honest claim is not "no disk" but no application boot, and no disk on the path that decides the answer — which is what removes the defects above, not tuning.

The health predicate

Computed only over Bunny objects held in the worker process:

channel = worker.queue.channel
return false if channel.nil?

connection = channel.connection
return true if recovering?(connection)

channel.open? && connection.open? && channel.any_consumers?

State	Verdict	Why
consumers present, channel open	healthy	—
`channel` still `nil`	unhealthy	subscription has not happened yet; `startupProbe` holds the pod
Bunny recovering from a network failure	healthy	Bunny reconnects and re-subscribes on its own; restarting now cures nothing and adds a reconnect storm
connection open, no consumers	unhealthy	the worker silently stopped consuming — a restart is the only cure
not the whole set subscribed	unhealthy	a worker that quietly dropped out would otherwise leave the probe green

The last row is the reason the registry compares its size against an expected count rather than just checking that whatever registered is alive. Four healthy workers out of five look perfectly fine one object at a time.

Nothing here talks to the broker

Every call in the predicate reads process-local state. Verified against Bunny 3.2.0:

Call	Implementation
`worker.queue.channel`	`attr_reader` on `Sneakers::Queue`
`channel.connection`	`attr_reader` on `Bunny::Channel`
`connection.recovering_from_network_failure?`	`@recovery_mutex.synchronize { @recovering_from_network_failure }`
`channel.open?`	`@status == :open`
`connection.open?`	status under `@status_mutex`, then `@transport.open?` → `@socket && [email protected]?`
`channel.any_consumers?`	`@consumer_mutex.synchronize { @consumers.any? }`

IO#closed? reports the state of the IO object in this process; it is not a syscall and not a packet. By contrast, consumer_count is a genuine round trip — a passive Queue.Declare or an HTTP API call. That round trip is what this gem does not make.

The predicate learns that a consumer was cancelled because RabbitMQ sends basic.cancel and Bunny deletes the consumer from its own registry. That is a push, not a poll — which is what makes an in-memory predicate possible at all, and also where its limits come from. See LIMITATIONS.md.

recovering_from_network_failure? is marked @private in Bunny's own documentation, so an upgrade may remove it. It is therefore called through respond_to?: without that guard the call would raise NoMethodError, alive? would catch it and report unhealthy, and every pod would enter a restart loop on the next Bunny upgrade. What the guard cannot preserve is the exemption itself — if the method disappears, so does "healthy while reconnecting", silently. That is the thing to re-check when upgrading Bunny.

The mark files

<dir>/expected       how many forks the probe must wait for
<dir>/generation     which container incarnation wrote this heartbeat
<dir>/worker-<slot>  one per fork, refreshed every tick the fork is healthy
<dir>/attempt-<slot> starts of a slot that has not become healthy yet

expected closes a startup hole. Without it the probe would pass as soon as any single file was fresh: fork 0 has written, fork 1 has not, and the pod looks ready while half of it is not subscribed. Every fork writes the same value, so they cannot disagree.

Every fork rewrites it on every tick, not only at startup. Nothing else restores it: if the directory is wiped, touch! brings the slot marks back while expected stays missing, and the probe reports "worker has not started" for the rest of the pod's life. Rewriting it is also what lets a respawned set of forks correct a count that has been lowered.

generation closes a container-restart hole when the container owns PID 1. Kubernetes preserves an emptyDir when it restarts a container inside the same pod. That is useful for an application cache elsewhere in the volume, but a fresh heartbeat from the dead process must not let the new container pass its one-shot startupProbe. On Linux, the worker and exec probe independently derive the same incarnation from the container's mount namespace and PID 1 start time. Both generation and every slot mark carry it. Until the new container declares itself and every current fork publishes its own mark, files inherited from the previous container are rejected. Nothing outside the marks directory is removed.

This guarantee assumes Kubernetes' default container-private PID namespace. With shareProcessNamespace: true PID 1 belongs to the pod sandbox, and with hostPID: true it is the node init process; neither restarts with the worker container. Under either setting the guard can accept an inherited fresh mark. Do not enable them on a pod whose startup probe relies on this guarantee; see LIMITATIONS.md.

Files are named by supervisor slot, not by PID. A fork killed with SIGKILL is respawned into the same slot and overwrites its own file. Had the name contained a PID, that file would sit there stale forever and the probe would fail permanently, turning one dead fork into a pod that can never come back.

expected is written through a temporary file and a rename, not with a plain write. A plain write truncates first, and a probe reading in that window finds the file empty and reports that the worker has not started. The window is real and recurring: every fork rewrites this file on every tick, while the liveness probe reads it on a schedule of its own. rename is atomic on tmpfs, so the probe sees either the old value or the new one.

attempt-<slot> is the respawn counter, and it lives in this directory for the single reason that the directory outlives the fork: a monitor caught in a respawn loop is a brand-new object every few hundred milliseconds and can hold no counter of its own. The file is removed once the slot becomes healthy, so in steady state the directory holds expected, generation, and the worker-<slot> marks. What the counter is for is in LIMITATIONS.md.

The timestamp, pid, and slot in worker-<slot> exist for a human running kubectl exec ... cat. Its generation is part of the probe contract; freshness still comes from mtime.

The directory must be on tmpfs — in Kubernetes, an emptyDir with medium: Memory. Put it on a real disk and the probe starts depending on the disk again, which was defect number two.

Why the heartbeat file has no `require` of its own

The probe loads exactly one file and nothing else. Measured over 30 invocations each; absolute values depend on the hardware, but the ratios do not:

	per invocation
`ruby --disable-gems -e ''` — bare interpreter	10 ms
`ruby -e ''` — RubyGems initialized	38 ms
the probe, with `--disable-gems` and an explicit path	10.5 ms
the probe, found through RubyGems	43 ms
`bundle exec ruby -e ''`	300 ms

Subtract the baselines and the probe's own work is a few milliseconds. Everything else is interpreter startup, and 28 ms of that is RubyGems initializing.

Three consequences follow. bundle exec kicks-liveness is the portable default; most of its runtime is Bundler startup rather than the check itself. When the gem is installed in GEM_HOME, RubyGems can find it with a plain require for under 50 ms. Skipping RubyGems entirely is faster still, but it requires naming the path to the gem's lib directory; see KUBERNETES.md for when that trade is worth making.

Keeping the probe's path through the heartbeat file free of require statements — and the file free of references to the rest of the gem — is what makes all three possible. The single require the file does contain, fileutils, is lazy and sits in the directory-creation branch that only the worker ever reaches.

Where the hooks attach

Two prepends, installed by KicksLiveness.install! (automatically through a Railtie in Rails), covering four methods:

Sneakers::Worker#run — register in the registry after super, so a worker whose subscribe raised never registers and the probe honestly fails.
Sneakers::Worker#stop — deregister before super.
Sneakers::WorkerGroup#after_fork — start the monitor thread after super.
Sneakers::WorkerGroup#stop — record that the shutdown is deliberate, before super starts unsubscribing.

after_fork rather than run, even though run is where the monitor must ultimately live: Sneakers::WorkerGroup#run calls after_fork as its very first statement and only then resolves worker_classes, so hooking after_fork is the earliest point in the fork at which the application's own after_fork hook has already run. That matters because resolving the expected consumer count may run application code — under sneakers:active_job the set is a callable registry — and the standard use of after_fork is re-establishing the connections a fork inherited. Resolving the set first would run that code against a fork that is not ready yet, and the rescue around the start would then leave the pod with no monitor at all. A spec asserts this ordering against the real worker gem, so an upgrade that reorders run fails in CI rather than in production.

ServerEngine::Server#create_worker calls w.extend(@worker_module): the WorkerGroup module lands in the singleton class of the instance, not as an include in a class. A prepend to the module still sits ahead of it in that lookup chain, which is what makes the hook work; worker_id and config come from ServerEngine::Worker. Verified on serverengine 2.4.0, and covered by a spec that asserts the order in ancestors — this is the most fragile assumption in the gem, so it fails in CI rather than in production.

Slot numbers come from worker_id, which is the index into ServerEngine's @monitors array, and both start_new_worker(wid) and restart_worker(wid) reuse the same index. That is what the slot-not-PID scheme relies on.

The monitor thread is created inside the fork, because threads do not survive fork. Starting it earlier would accomplish nothing. For the same reason the registry initializes its array and mutex at file-load time: that happens once, single-threaded, before any fork, so no lazy initialization is needed and none can race.

In Rails the hooks are installed from a Railtie, and from to_prepare rather than an initializer. An application's lib is normally managed by Zeitwerk and reloadable, and reloadable constants must not be referenced while the application is initializing.

Starting the monitor is wrapped in a rescue: instrumentation has no right to prevent a worker from starting. Dir.mkdir raises Errno::EROFS if the marks directory was never mounted, and resolving the worker set runs application code. The worst outcome should be a missing mark and a pod restart, never a pod that stays up with silent queues.

The expected consumer count is not configured

It is derived from config[:worker_classes] — the very set from which the worker gem builds its workers. The queue list is therefore never duplicated and cannot drift from the application's configuration.

That value is an array of classes under sneakers:run and a callable registry under sneakers:active_job, so it is resolved with a respond_to?(:call) check.

What is configurable, and where

Setting	Where	Why
`logger`, `enabled`, `startup_grace_ticks`	application config block	only the worker needs them
`tick`	either, and the config block wins	only the worker reads it, so two sources cannot contradict each other
`dir`, `max_age`	environment variables only	the probe is a separate process

dir and max_age are deliberately not settable from the application. The probe is launched by the kubelet as its own process; it never reads, and cannot read, the application's initializer. If those two values were configurable in both places, a mismatch between what the worker writes and what the probe expects would pass unnoticed — and that mismatch is silent by nature: the worker would look fine and the probe would look broken. Making them read-only from the application removes the failure mode instead of documenting it. A spec asserts that Configuration does not respond to dir= or max_age=.

tick is the opposite case and may be set from either side, because only the worker ever reads it: with a single reader there is nothing for a mismatch to happen between. KICKS_LIVENESS_TICK supplies the initial value and the configuration block overrides it, so config.tick = 5 wins over the variable.

Values arriving from a ConfigMap are parsed defensively: garbage falls back to the default rather than crashing the worker, and an empty string counts as "unset" — in a ConfigMap that is what you get by declaring a key and leaving it blank.

max_age and tick are durations, so a value must be parseable and positive. Zero and negative numbers are the more dangerous half: they parse perfectly well, a tick of zero turns the monitor into a hot loop, and a negative max_age makes every mark stale on arrival, so the probe can never pass again. They fall back to the default too.

Setting tick from the application is held to a stricter standard: a non-positive value raises ArgumentError, as does a value greater than or equal to max_age: such a monitor would inevitably let a healthy mark go stale. The environment gets a silent fallback for values that cannot be parsed. If its effective tick is greater than or equal to max_age, configuration emits a warning on stderr and uses the default tick when it is safe, or half of max_age otherwise. A ConfigMap typo must not bring a worker down, whereas an initializer is code and should fail loudly at boot, where the developer is looking.

startup_grace_ticks must be a positive integer. It is compared directly with an integer counter; accepting zero, a float, or a string would silently disable the escalation that is supposed to diagnose a worker that never subscribes.

The logger defaults to Sneakers.logger but is resolved lazily, because at the time the configuration object is built it may not be set up yet.

Logging is diagnostic; the heartbeat is the liveness contract. An exception raised by a custom logger is therefore swallowed at the logging boundary. It cannot prevent a healthy tick from writing its mark, abort monitor startup, or escape the loop's error handler and kill the monitor thread. No fallback message is attempted through the same broken logger.

Logging: events, not the pulse

The pulse lives in the mtime of a file; writing a log line every tick would only add noise. What gets logged is transitions.

Event	Level
started, with the effective settings	INFO
waiting for consumers	INFO
became healthy	INFO
shutting down	INFO
still not healthy after `startup_grace_ticks` ticks	ERROR
was healthy, became unhealthy	ERROR
a slot respawning without ever becoming healthy, once per grace window	ERROR
the monitor thread caught an exception	ERROR
the monitor could not be started at all	ERROR

ERROR is reserved for genuine failure. A logger typically comes up with LOG_LEVEL defaulting to error, so anything important must survive that filter — while an ordinary pod start must not emit ERROR on every deploy, or an alert on ERROR logs fires on every rollout and is promptly muted.

A graceful shutdown is the other half of that rule. Sneakers::Worker#stop unsubscribes and then waits for the thread pool to drain, which takes as long as the longest in-flight job — and the registry empties at the start of it. Judged by the consumer count that is indistinguishable from a fault, so every single deploy would emit an ERROR. From WorkerGroup#stop onwards the predicate is therefore suspended: the monitor says so once at INFO and keeps the mark fresh, so a slow drain gets the full terminationGracePeriodSeconds instead of having the probe cut it short at max_age.

The startup grace exists because a slow start and a start that will never finish look identical for the first few ticks. Once the grace expires, staying silent would mean a pod that never comes up never says so — hence exactly one ERROR, not one per tick.