Binions tells you two things about itself at all times: whether it is healthy, and — if you ask — exactly what it is doing. Every daemon serves a tiny health and metrics endpoint on a loopback address, so liveness, readiness, and Prometheus metrics are always available with zero configuration. Distributed tracing is opt-in: switch it on for a daemon and its work flows into a bundled Jaeger UI you can browse on your own network, with every log line and every span sharing the same trace id.
Good to know. Health checks and metrics are always on — you never enable them. Tracing is opt-in per daemon, costs nothing until you turn it on, and adds no overhead while it is off.
Day-to-day monitoring of Binions comes down to two different questions, answered by two different mechanisms:
You can run Binions perfectly well on health and metrics alone. Tracing is the tool you reach for when something is slow or behaving oddly and you want to follow a single request end to end.
Every long-running daemon exposes a Prometheus /metrics endpoint, plus its health endpoints, on a single loopback port in the 91xx range. These ports listen on 127.0.0.1 only — they are never reachable from off the host. That keeps your operational metrics private by default while still letting a local Prometheus, a sidecar exporter, or a quick curl read them.
# Read one daemon's metrics directly (loopback only)
curl -s http://127.0.0.1:9100/metrics | grep '^binions_'The metrics carry a binions_ prefix. The platform emits a small, deliberate set — just enough to answer “is each daemon keeping up, and is its outbox draining?” without drowning you in cardinality:
| Metric | Type | What it tells you |
|---|---|---|
binions_events_processed_total | Counter | Events handled, labelled by kind, event_type, and status (handled / unhandled / duplicate / failed). Your throughput and error rate. |
binions_event_processing_duration_seconds | Histogram | How long each event takes to process, by kind and event_type. Your latency distribution. |
binions_outbox_pending_size | Gauge | How many events are queued in the daemon’s transactional outbox waiting to be published. A steadily rising value means a downstream daemon is falling behind. |
binions_healthcheck_last_activity_age_seconds | Gauge | Seconds since the daemon last did real work. Feeds readiness; a good liveness signal. |
binions_healthcheck_last_redis_ok_age_seconds | Gauge | Seconds since the daemon last confirmed its state store was reachable. Rising means the daemon has lost its backing store. |
Stream lag is not a metric. Binions does not publish a per-consumer lag metric. To measure backlog or queue depth on a stream, ask the state store directly with
XLEN(total entries) andXPENDING(entries claimed but not yet acknowledged). Thebinions_outbox_pending_sizegauge is the closest always-on signal of a building backlog.
Each daemon owns exactly one of these endpoints. The full loopback port map:
| Daemon | Endpoint | Daemon | Endpoint |
|---|---|---|---|
| logger | 127.0.0.1:9100 | dataanaliser | 127.0.0.1:9106 |
| database | 127.0.0.1:9102 | mailbox | 127.0.0.1:9107 |
| traefiklinker | 127.0.0.1:9103 | playbook | 127.0.0.1:9108 |
| aiinjector | 127.0.0.1:9104 | webhookcaller | 127.0.0.1:9109 |
| datatransporter | 127.0.0.1:9105 | scheduler | 127.0.0.1:9110 |
| showman | 127.0.0.1:9111 | ||
| modbus | 127.0.0.1:9112 | ||
The on-demand cliconsole tool is not a long-running daemon and has no metrics endpoint; 9101 is held in reserve for a future console interface.
Alongside /metrics, every daemon serves two health endpoints on the same loopback port. They answer two distinct questions, and they return plain text — not JSON:
| Endpoint | Answers | 200 OK when… | 503 when… |
|---|---|---|---|
/health/live | Is the process alive? | Always, while the process is running. | Never returns 503 — if the process is down, the connection is simply refused. |
/health/ready | Can it actually serve? | It has started and either real work or a background state-store ping has happened within the last 30 seconds. | Activity and the state-store ping have both gone stale — the daemon is up but not making progress. |
The split matters: /health/live is what an orchestrator uses to decide “restart this”, while /health/ready is what you alert on when a daemon is technically running but wedged. For the full readiness model, the roll-up table from the console, and how the systemd watchdog ties in, see Health checks.
Tracing is off until you ask for it. You enable it per daemon by adding an [otel] block to that daemon’s application.toml. With the block present, the daemon exports OpenTelemetry spans over OTLP/gRPC to the bundled Jaeger collector on the loopback address; with no block, the daemon logs as usual and pays zero tracing cost.
# In a daemon's application.toml — enable tracing for that daemon
[otel]
endpoint = "http://127.0.0.1:4317" # bundled Jaeger collector, loopback only
sample_rate = 1.0 # 1.0 = keep every trace; lower to sampleSet sample_rate to 1.0 while you are actively debugging so nothing is dropped, then dial it down on a busy host once you have what you need. Restart the daemon to apply the change. Sampling is parent-based, so turning a busy daemon down never breaks a cross-daemon trace — a span that belongs to an already-sampled playbook run is always kept, and only each daemon’s own root traffic is sampled at its rate.
Spans land in the bundled Jaeger UI, which listens on port 16686. Like the admin interfaces, it is reachable on your own LAN or cluster only — never published to the internet — while the collector ingest and Jaeger’s own admin stay on loopback. The same UI is also published through the platform edge at https://<your-host>:8443/, which fronts it with HTTPS and Basic authentication — open it at the root of the port (there is no sub-path), and see Network requirements for the full set of edge entry points. Traces are kept for roughly 24 hours, which is the right window for “something went wrong overnight, let me look” without turning Jaeger into a long-term store.
LAN-only, by design. The Jaeger UI is for operators on your network, not for end users. It is never exposed publicly. Treat its retention as short-lived diagnostics, not an audit log — for durable history, rely on the JSONL event logs.
When a playbook runs, the trace is the fastest way to see which steps succeeded and which failed at a glance. Open the Jaeger UI, pick playbook-service from the Service dropdown, and open the run. A single playbook renders as one connected waterfall, each step nested under the run.
The shape of the waterfall depends on the playbook’s execution mode. Playbooks default to saga mode, in which every run: step waits for its response fact before the next step begins. In saga mode you see a purely sequential chain, one step at a time, each span completing before the next opens:
execute_playbook (playbook-service) # the whole run
execute_one_step (playbook-service) # step 1: dispatch + wait for result
dispatch (mailbox-service) # the daemon that did the work
execute_one_step (playbook-service) # step 2
dispatch (webhookcaller-service)In async mode, run: steps are fire-and-forget: the orchestrator dispatches each action immediately without waiting for a result. The waterfall looks different — multiple execute_one_step spans overlap in time rather than forming a strict sequence. A wait_for: step then appears as its own span, joining one specific response by causation id. Expect a wider, overlapping waterfall for async playbooks, with the wait_for span marking the explicit join point:
execute_playbook (playbook-service) # the whole run
execute_one_step [cls] (playbook-service) # fire-and-forget dispatch
dispatch (aiinjector-service)
execute_one_step (playbook-service) # fires immediately — does not wait for cls
dispatch (database-service)
wait_for [cls_result] (playbook-service) # explicit join: blocks for Fact.AI.ClassifiedReading the waterfall top to bottom tells you the health of the run regardless of mode:
execute_one_step span with the target daemon’s dispatch span nested inside, both completing cleanly. The width of a bar is how long that step took — an unusually wide bar is where the time went.execute_one_step with no child dispatch — the action did not arrive (wrong target, or that daemon was down).wait_for: that timed out shows the wait_for span with an error tag and the duration equal to its configured timeout; the run will have ended with a failure.Two filters in the UI do most of the work:
5s) to surface only the runs that took too long, then open the widest span.correlation_id tag to pull up every span of a single business run across all daemons.An empty Jaeger means “nothing ran recently”, not “nothing works”. Spans appear only when a daemon actually does something, and they expire after about 24 hours, so a blank Service list on an idle host is normal — trigger a playbook and refresh. To check that the daemons themselves are alive, use
/health/readyorbinions-cliconsole status, never the trace view.
The reason logs and traces work so well together in Binions is that they share one identifier. As an event moves between daemons, the platform injects a W3C traceparent into the event metadata, and the trace id is the middle segment of that value. The same trace id appears both in Jaeger and in the structured log lines the logger writes, so you can pivot in either direction:
Error in the JSONL logs. Pull the trace id out of its traceparent and paste it into Jaeger to see the whole run that produced it.# Find the trace id (middle segment of W3C traceparent) for a failed run
jq -r 'select(.payload.level=="Error") | .metadata.traceparent' \
/var/log/binions/playbook/events.jsonl
# Then pull every log line that belongs to one trace id across all daemons
grep -rl "<trace-id>" /var/log/binions/*/events.jsonlBecause every envelope also carries a correlation_id, you can follow a single business operation through the logs even when tracing is switched off. See Logs for the JSONL format and field reference.
Binions does not ship its own dashboards or alertmanager — it exposes clean Prometheus endpoints and gets out of your way. Point your existing Prometheus at the loopback targets on the host. Because the endpoints are loopback-only, your Prometheus needs to run on the same host (or scrape through a local agent):
# prometheus.yml — scrape the Binions daemons on the host itself
scrape_configs:
- job_name: binions
metrics_path: /metrics
scrape_interval: 15s
static_configs:
- targets:
- 127.0.0.1:9100 # logger
- 127.0.0.1:9102 # database
- 127.0.0.1:9103 # traefiklinker
- 127.0.0.1:9104 # aiinjector
- 127.0.0.1:9105 # datatransporter
- 127.0.0.1:9106 # dataanaliser
- 127.0.0.1:9107 # mailbox
- 127.0.0.1:9108 # playbook
- 127.0.0.1:9109 # webhookcaller
- 127.0.0.1:9110 # scheduler
- 127.0.0.1:9111 # showman
- 127.0.0.1:9112 # modbusFor alerting, build rules around the always-on signals rather than trying to derive stream lag. Three rules cover most real incidents:
/health/ready probe fails — the clearest sign a daemon is up but stuck.binions_healthcheck_last_activity_age_seconds or binions_healthcheck_last_redis_ok_age_seconds climbs well past 30 seconds — the daemon has lost its state store or stopped progressing.binions_outbox_pending_size trends upward without recovering, then confirm the stream depth with XLEN / XPENDING.# A Prometheus alerting rule for a stalled daemon
groups:
- name: binions
rules:
- alert: BinionsDaemonStalled
expr: binions_healthcheck_last_activity_age_seconds > 120
for: 2m
labels: { severity: warning }
annotations:
summary: "A Binions daemon has not done work in over two minutes"Tip. Pair this with the systemd watchdog. The watchdog recycles a daemon that has actually hung, while your Prometheus alerts catch the subtler “running but not progressing” cases the watchdog can’t see — together they give you full coverage.