If Binions feels slow or a backlog keeps growing, the cause is almost always a blocked consumer or a resource cap — not raw processing speed. Binions is designed to run the whole platform on a single host with concurrent, non-blocking processing throughout, so “making it faster” usually means measuring where the time goes, giving a daemon the memory it needs, and clearing whatever is jamming the event flow. This page walks through that, symptom by symptom.
Single-host by design. Binions is not a horizontally-clustered system. “Scaling” means a bigger host and right-sizing each daemon, rather than adding more nodes. See Scaling for the full picture.
Understanding the execution model helps you read the diagnostics correctly. Every daemon dispatches incoming actions through a bounded concurrent worker pool. A slow operation — a long network call, a large database write, an SSH command — does not block independent actions on the same daemon. Actions targeting the same resource (for example, the same schedule name or the same Traefik route) are still serialised in order so no two operations race on one resource; everything else runs in parallel.
The playbook engine works the same way. It decouples accepting triggers from running them and can have many playbook runs in flight at once, bounded by configurable concurrency caps. A slow playbook run does not delay unrelated runs.
The practical implication for diagnostics: a backlog or rising duration on one action type does not automatically mean all work is blocked. Check whether the slowness is isolated to a specific resource or daemon before concluding the whole platform is behind.
/metrics tell you?Every daemon serves a Prometheus /metrics endpoint on a loopback address. Before changing anything, look at three numbers per daemon — they tell you whether you have a speed problem or a flow problem:
| Metric | What it tells you |
|---|---|
binions_event_processing_duration_seconds | How long a daemon takes to handle one event (handler latency). Rising here means real work is getting slower. |
binions_outbox_pending_size | How many results are queued to publish but haven’t gone out yet. A climbing value means the daemon can’t hand work off — usually a downstream Redis is unreachable. |
binions_events_processed_total{status} | A running count of processed events, split by outcome — useful to see throughput and the share of failures over time. |
You can scrape these from the host:
# Pull a daemon's metrics (replace 91xx with that daemon's health port)
curl -s http://127.0.0.1:91xx/metrics | grep binions_Each daemon runs under systemd with MemoryHigh and MemoryMax caps, so one runaway service can never starve the rest of the host. The trade-off is that a daemon doing more than its default cap allows will be restarted when it hits MemoryMax. The signs are a non-zero, climbing restart counter:
systemctl show -p NRestarts --value binions-<name> # climbing = a crash/OOM loop
systemctl status binions-<name> # check the memory line and last exitThe fix is to give it more headroom — either raise that daemon’s memory cap or add RAM to the host. The mailbox daemon ships with the largest default cap because it buffers messages; most other workers are deliberately small. Adjust the cap with a systemd drop-in:
sudo systemctl edit binions-<name>[Service]
MemoryMax=512M
MemoryHigh=448Msudo systemctl daemon-reload
sudo systemctl restart binions-<name>Right-size, don’t over-size. Caps exist on purpose. Raise the cap for the one daemon that needs it rather than removing limits everywhere — that keeps a single busy service from affecting the rest of the platform.
Binions deliberately exposes few throughput knobs, but there are several worth knowing. The platform distinguishes between daemon-level concurrency and playbook-engine concurrency — these are separate levers.
Every daemon processes actions through a bounded concurrent worker pool. The pool size caps how many actions a daemon handles in parallel. When a daemon is the bottleneck and the underlying resource (database, network, external API) has headroom, increasing the worker-pool size allows the daemon to issue more concurrent requests and reduces wall-clock latency. The setting is in the daemon’s configuration file under the worker or concurrency stanza.
The playbook engine exposes two concurrency caps:
These caps are configured in the playbook service configuration. If runs are queuing even though the host has spare CPU, raising the global cap (or the relevant per-playbook cap) is the correct lever.
Two daemons also expose domain-specific throughput settings:
[stream] and [batching] settings for how it reads and writes log events.If a daemon other than these is slow and the worker-pool size and playbook caps are already well-sized, the answer is a resource cap (memory, above) or a blocked flow (backlog, below).
A growing backlog almost always means a stuck consumer, not that Binions is too slow to keep up. Events are carried on Redis streams, and each daemon has its own Redis. To see the real depth, read it straight from Redis — the platform does not emit a queue-lag metric:
redis-cli -p <port> XLEN <stream> # how many events are waiting in the stream
redis-cli -p <port> XPENDING <stream> <group> # un-acked (in-flight) entriesBecause daemons process independent actions concurrently, a high XPENDING for one action type does not necessarily mean every action is stalled — it may mean only actions targeting a particular resource are waiting. Check which stream and group are accumulating before concluding the entire daemon is blocked.
From there, work through the usual causes:
binions-cliconsole status and start whatever shows DOWN.binions_outbox_pending_size means a daemon can’t publish its results — usually the downstream daemon’s Redis is unreachable. Check systemctl status redis-binions-<name> and restart it.XPENDING is high but binions_event_processing_duration_seconds is normal, the consumer is blocked on a dependency rather than churning through work — trace it through its logs.Backlog is a clue, not the disease. Throwing more memory or CPU at a stuck consumer won’t help. Find the daemon or Redis that’s down or unreachable first — the backlog drains on its own once the path is clear.