Every Binions daemon answers two health questions over a loopback port: “am I running?” and “am I actually ready to do work?” Those two answers — liveness and readiness — are what systemd uses to recycle a hung service, what binions-cliconsole status rolls up into a single table, and what your monitoring system should watch. This page explains the difference, shows you exactly where the endpoints live and what they return, and ends with how to wire them into an uptime monitor.
Good to know. Health responses are deliberately tiny and machine-readable: plain text, a short body, and an HTTP status code. There is no JSON to parse and nothing to authenticate against — the endpoints are bound to loopback, so only something on the host (or your monitoring agent reaching them over the LAN) can see them.
Binions separates “the process is alive” from “the process is doing its job.” They are different questions, and conflating them is the classic cause of false alarms and missed outages. Each daemon exposes both as separate endpoints.
| Endpoint | Question it answers | Returns 200 when… |
|---|---|---|
/health/live | Liveness — is the process up? | The process is running and serving requests. It returns 200 for as long as the daemon is alive, regardless of whether its dependencies are healthy. |
/health/ready | Readiness — is it actually working? | The daemon has finished starting and it has shown recent activity: either it has processed events, or its background Redis ping succeeded, within the last 30 seconds. |
The distinction matters because the two are wired to different consumers:
Restart= policy described below, not on an HTTP poll./health/ready turns that into a 503 so your monitoring catches it, while liveness still reports a healthy 200.The rule of thumb. Use
/health/liveto ask “should this be restarted?” and/health/readyto ask “should this be paging me?” Point your uptime checks at/health/ready— it is the one that catches a daemon that is alive but no longer working.
Each daemon serves its health and metrics endpoints on a single loopback port — one port per daemon, bound to 127.0.0.1 and never exposed off the host. The three paths (/health/live, /health/ready, and the Prometheus /metrics scrape) all share that one port. The map is fixed:
# Each daemon's health/metrics port (127.0.0.1 loopback only)
logger 9100
database 9102
traefiklinker 9103
aiinjector 9104
datatransporter 9105
dataanaliser 9106
mailbox 9107
playbook 9108
webhookcaller 9109
scheduler 9110
showman 9111
modbus 9112The responses are plain text (content-type: text/plain), not JSON. The body is a short human-readable string and the status code carries the verdict — so a one-line curl tells you everything:
# Liveness: is the mailbox daemon's process up?
curl -s -w ' [%{http_code}]\n' http://127.0.0.1:9107/health/live
# -> OK [200]
# Readiness: is it started AND showing fresh activity?
curl -s -w ' [%{http_code}]\n' http://127.0.0.1:9107/health/ready
# -> OK [200]When a daemon is alive but not ready, /health/ready returns 503 with a short reason in the body — for example, it is still starting up, or its activity and Redis pings have both gone stale:
# A daemon that is up but not yet ready
curl -s -w ' [%{http_code}]\n' http://127.0.0.1:9108/health/ready
# -> not yet ready [503]
# A daemon that has gone quiet (no events and Redis ping both stale > 30 s)
curl -s -w ' [%{http_code}]\n' http://127.0.0.1:9109/health/ready
# -> activity + redis stale [503]Don’t parse it as JSON. These endpoints return plain text, not a JSON document. Match on the HTTP status code (
200vs503), not on a parsed field. The body string is for humans; the status code is for your tooling.
You rarely want to curl twelve ports by hand. The binions-cliconsole tool does it for you: it auto-discovers every daemon, polls each one’s /health/ready, and prints a single readiness table.
binions-cliconsole statusThe table has four columns — the service name, the readiness URL it polled, the resulting state, and the round-trip latency. The states map directly onto what the endpoint returned:
| State | What it means |
|---|---|
UP (200) | The daemon answered /health/ready with 200 — started and showing fresh activity. This is the healthy state. |
DEGRADED (code) | The daemon answered, but with a non-200 code (typically 503) — it is alive but not ready. Look at its logs and its Redis instance. |
DOWN (refused) | Nothing is listening on the port — the connection was refused. The daemon’s process is not running. |
DOWN (timeout) | The port accepted the connection but never answered in time — the daemon is hung. systemd’s watchdog should be recycling it. |
The CLI itself is skipped in its own table — it is an on-demand tool, not a long-running daemon, so it has no readiness state of its own.
Latency of 600–1100 ms is normal. The readiness check pings Redis as part of deciding whether a daemon is working, so each row takes a little longer than a bare HTTP round-trip. A latency in that band is expected, not a warning sign — only a missing or hung daemon should worry you.
Underneath the HTTP endpoints, systemd has its own, lower-level view of each daemon’s health — and it is systemd, not the /health endpoints, that actually restarts a broken service. Three settings do the work.
Type=notify. Every daemon unit is a notify-type service. The daemon does not report itself as started the instant the process spawns — it sends READY=1 to systemd only after it has brought up its internal machinery: its event consumers, its transactional outbox, and its health server. Until then, systemd considers the unit still “activating,” which keeps dependent units waiting for a service that is genuinely ready to work.WatchdogSec=30. The watchdog is active on every daemon. While the daemon is healthy and not deadlocked, its framework sends a watchdog ping to systemd roughly every ten seconds — about one third of the 30-second window. If those pings stop — because the daemon has hung — systemd notices the missed deadline and kills and restarts the unit. This is what turns a silently stuck process into a recovered one without anyone watching.Restart=. When a daemon exits or is killed (including by the watchdog), systemd restarts it. The exact policy varies by daemon: some restart only on failure (with a short back-off pause before retrying), while others restart unconditionally. Either way, the practical result is the same — a crashed or hung daemon comes back on its own.You can see all of this through the normal systemd tooling — the unit’s active state, its restart count, and the last watchdog activity:
# Liveness, restart count, and watchdog status from systemd's side
systemctl status binions-mailbox.service
# Follow a daemon's log to watch READY=1 and any restarts
journalctl -u binions-mailbox.service -fTwo safety nets, not one. The watchdog catches a daemon that is hung but not dead (it has stopped pinging), and the restart policy catches a daemon that has actually exited. Together they mean a single stuck or crashed daemon recovers itself — while readiness, exposed over HTTP, is what tells you it happened.
Readiness is not just a runtime signal — it shapes how the platform starts. A daemon stays not-ready until the dependencies it needs are actually up, and systemd’s ordering rules make sure those dependencies start first.
/health/ready stays at 503.database waits for its configured data store. The database daemon fronts the configured data store. When that store is an external server (such as PostgreSQL, MySQL, or MS SQL Server), the unit is ordered after it and will not report ready until that server is accepting connections. A local embedded backend such as SQLite has no external dependency and becomes ready immediately.playbook waits on database. Because the playbook engine relies on the database layer, its unit is ordered after database, so the dependency chain settles in the right order on every boot.The upshot: during a fresh boot or a full restart, it is normal to see daemons report 503 from /health/ready for a short window while their Redis instances — and any external database server, for database — come online. A binions-cliconsole status run during that window will show some rows as DEGRADED until the dependencies settle. That is the system behaving correctly, not a fault.
To monitor Binions from the outside, point your uptime checks at each daemon’s /health/ready endpoint and alert on anything that is not a 200. Because the endpoints live on the host’s loopback, your monitoring agent needs to reach them locally or over your LAN — an off-host probe will not see them. Two complementary signals are worth watching.
/health/ready per daemon: treat 200 as healthy and anything else (a 503, a connection refused, or a timeout) as a problem. This is the single most useful check, and it mirrors exactly what binions-cliconsole status shows./metrics. The same port also serves Prometheus metrics. Two age gauges tell you how stale a daemon has gone before it ever flips to 503 — useful for catching a daemon that is drifting toward unhealthy:# Scrape the freshness gauges from a daemon's metrics endpoint
curl -s http://127.0.0.1:9107/metrics | grep -E 'last_activity_age|last_redis_ok_age'
# binions_healthcheck_last_activity_age_seconds ...
# binions_healthcheck_last_redis_ok_age_seconds ...In a Prometheus scrape config, point one job at each daemon’s port and let the metrics flow into your usual dashboards and alert rules:
scrape_configs:
- job_name: binions
metrics_path: /metrics
static_configs:
- targets:
- 127.0.0.1:9100 # logger
- 127.0.0.1:9107 # mailbox
- 127.0.0.1:9108 # playbook
# ...one target per daemon from the port map aboveA practical alerting policy: page on a non-200 from /health/ready that persists past the normal boot window, and raise a warning when either age gauge climbs well beyond the 30-second readiness threshold. Both gauges resetting to low values is the signal that a daemon has recovered.
Keep health internal. The
/healthand/metricsendpoints are loopback-bound on purpose — they are operational signals, not a public API. Reach them from the host or across your own LAN/cluster; never publish them to the internet or put them behind the public edge.
Note: only daemons run as long-lived services have a port and a readiness state. binions-cliconsole is an on-demand command, so it never appears as a monitored target — it is the tool you run to read the others.