Almost every “degraded daemon” problem in Binions comes down to one of its two datastores. Each daemon keeps its own working state in a private Redis instance, and a single daemon — database — sits in front of your SQL or document store (PostgreSQL by default). When a service reports DEGRADED or returns 503, the cause is usually a Redis that has gone quiet or a database backend it can no longer reach. This page walks the common symptoms from the surface down to the fix.
One Redis per daemon. There is no shared cache. Every daemon has a dedicated
redis-binions-<name>.servicethat holds only that daemon’s state, its transactional outbox, and its idempotency keys. So when you restart Redis, you only ever restart the one belonging to the affected daemon.
Symptom. binions-cliconsole status shows a daemon as DEGRADED, or its readiness probe returns 503 instead of 200 "OK".
Likely cause. A daemon’s readiness depends on a fresh ping to its own Redis. If that Redis is down, slow, or restarting, the ping goes stale and the daemon drops out of the ready state — even though the process itself is still running.
Fix. Check and restart the matching Redis instance, then re-check the daemon:
systemctl status redis-binions-<name>
systemctl restart redis-binions-<name>
binions-cliconsole statusThe daemon picks the connection back up on its own once Redis answers. Each Redis instance persists its data (append-only file plus periodic saves) and runs with a small memory cap and a noeviction policy, so it will not silently discard your outbox or state.
Symptom. binions-database stays DEGRADED/DOWN, and anything that relies on it (such as binions-playbook) is unhappy too.
Likely cause. The database daemon is multi-backend — it can sit in front of PostgreSQL (the default), SQLite, MySQL/MariaDB, MongoDB, or Microsoft SQL Server. If its configured backend is not running, or the daemon’s configured connection is wrong, database can never reach the ready state.
Fix. Confirm the configured backend is up, then confirm the daemon’s connection is valid. For the default PostgreSQL backend:
systemctl status postgresql
systemctl restart binions-database
binions-cliconsole statusCheck the backend first. Restarting
binions-databaseachieves nothing while its backend is down. Bring the backend back, then restart the daemon — not the other way around. If you run a non-PostgreSQL backend, check that one’s own reachability instead: a remote MySQL, MongoDB, or MSSQL server withnc -z <host> <port>, or, for a file-based SQLite store, that the database file exists and is readable. A SQLite backend has no server to start.
Symptom. Work seems to be piling up — events arrive faster than they clear, or results stop appearing downstream.
Likely cause. Binions does not emit a queue-lag metric, so a backlog is not visible in the readiness table. You read it directly from the Redis stream instead. Each daemon’s Redis listens on its own loopback port, shown here as <port>:
redis-cli -p <port> XLEN <stream> # how many entries are in the stream
redis-cli -p <port> XPENDING <stream> <group> # un-acked, in-flight entriesA high or climbing XLEN with many XPENDING entries points to a consumer that is stuck rather than to raw throughput. Look at whether the downstream daemon (or its Redis) is healthy.
Symptom. A daemon is processing events but its output never reaches the next daemon.
Likely cause. The metric binions_outbox_pending_size — exposed on each daemon’s /metrics endpoint — is growing. That means the daemon has produced results but cannot hand them off, almost always because a downstream Redis is unreachable.
Fix. Find the downstream daemon that should receive the work, check its Redis, and restart it if it is down. Once the path clears, the pending outbox drains on its own.
Connection errors name the culprit. A connection-refused error in a daemon’s log always names the exact Redis URL that failed. Read the log line, identify the instance, and restart that one Redis — you rarely have to guess.