When a playbook misbehaves, work from the validator outward. Most playbook problems are caught before anything runs: a misspelled verb, a daemon that doesn’t exist, or an operation that isn’t implemented yet. The steps below take you from a static check, through reloading an edited playbook, to watching a live run and tracing exactly where it failed.
Good to know. A playbook is deliberately small: a
trigger, an optionalmode:(sagaorasync), and a list ofsteps. Each step is exactly one ofrun:,parallel:,loop:, orwait_for:. There are no conditionals or free-form expressions — but bounded loops and async joins are supported — so when something goes wrong it is usually a typo, a missing resource, a downstream daemon, or a join that never matched, not hidden branching logic.
Before you debug, it helps to know exactly what a step can be — a surprising number of “the playbook didn’t do what I expected” reports come down to using the wrong shape. Every step is one of these four:
run: | A single daemon operation, e.g. database.write, optionally with with: {…} arguments and an expect: response fact. This is the workhorse step. |
parallel: | A list of run: steps that fire together. Single level only — you cannot nest a parallel: or a loop: inside it. |
loop: | Bounded counted iteration — for: a counter, from:/to: a range, a mandatory max: cap, an optional until: early-exit, and a do: body of plain run: steps. |
wait_for: | A join used in async mode: block until one specific response fact arrives, matched by causation id. Needs event: and match:. |
If the validator complains that a step “must use exactly one of run, parallel, loop or wait_for,” you have either combined two of these in one step or misspelled the keyword. Each list item picks exactly one.
A playbook’s top-level mode: changes how steps are scheduled, and it is a common source of confusion when debugging timing or ordering:
saga (the default). Steps run in order. Every run: step waits for its own response fact before the next step starts, and ${prev} carries that fact’s payload forward. If a playbook has no mode: line, it is a saga — so an existing playbook keeps behaving exactly as before.async. Each run: step is fire-and-forget: it emits its action and moves on immediately without waiting for a result. To collect a result you add an explicit wait_for: step that joins on the earlier step’s causation id. This lets one run fan out many actions at once and join only the results it needs.Tip. A
wait_for:step is only meaningful undermode: async. If a join never resolves, first confirm the playbook actually declaresmode: async, then confirm thematch: { causation: ${<run-id>} }points at the right fire-and-forget step.
Start every change with the validator. It checks each playbook’s grammar — including that each step uses exactly one of the four shapes above — and confirms that every run: verb actually exists:
binions-cliconsole validateThe validator is careful to tell different problems apart:
loop: missing its max: cap, or a wait_for: missing its event: or match: { causation: … } — each is reported with the offending step.Fix everything the validator reports before moving on — a playbook that fails validation will not behave the way you expect.
Editing a playbook file does not require restarting any daemon, but the platform does not auto-detect the change — after saving, you trigger a reload explicitly:
binions-cliconsole emit-control Control.Playbook.ReloadThe platform picks up the new YAML immediately. A file in provisioning/ re-runs on its next trigger; one in business/ simply waits for its next event.
Tip. If an edit seems to have no effect, check two things in order: that you saved the file into the playbooks directory the service loads from, and that you ran the reload above. Validate the file first with
binions-cliconsole validateso a parse error doesn’t silently keep the old version in play.
With the playbook reloaded, trigger it (or wait for its trigger) and watch the events flow. Two views cover almost every case:
binions-cliconsole ls events
journalctl -u binions-playbook -n 50 --no-pagerThe event listing shows what the playbook engine is emitting in near-real time; the journal shows the playbook daemon’s own log lines, including any errors it hit while dispatching a step.
Good to know. Daemons and the playbook engine now process work concurrently. A slow step — a long network scan or a remote command — no longer blocks other actions, and the engine can have many runs in flight at once. So in a busy listing you will see events from several runs interleaved. Per-resource ordering is still preserved (actions targeting the same resource stay in order), so interleaving across runs is expected, not a bug.
When a step fails, the platform doesn’t silently drop it. The run halts at that step and the engine emits a Fact.Playbook.Failed event — carrying the failing step, the error, and the run’s correlation id — onto the event bus and the playbook daemon’s audit log, so you can see exactly which step failed and why:
# the most recent failed runs
jq 'select(.type=="Fact.Playbook.Failed")' \
/var/log/binions/playbook-service/events.jsonl | tailEach event tells you the failing step and the error that stopped it — the fastest way to turn “the playbook didn’t finish” into a concrete cause. There is no step retry and no separate dead-letter file: a run stops at its first failed step, recorded as the event above.
A loop: runs its do: body once per count and exposes the counter to the body as ${loop.<for>}. Two failure modes are specific to loops:
max: field is a mandatory hard ceiling (1–10000). If the resolved from:/to: range is larger than max:, the loop aborts loudly rather than silently clamping — check that to: (often an interpolated value resolved once at loop entry) really is what you expect.do: body stops the loop at that iteration and emits Fact.Playbook.Failed. The event names the failing step; the counter value at the time of failure is in the run’s correlated events.A wait_for: step blocks until a specific response fact arrives. When a join “hangs” or fails, check these in order:
async mode? A wait_for: only makes sense under mode: async. In a saga, steps already wait for their own results, so a stray join is a sign the mode is wrong.match: point at the right step? The join matches on the causation id of an earlier fire-and-forget run: step — usually match: { causation: ${<run-id>} }. A typo in that id means the awaited fact never matches.wait_for: has a timeout (defaulting to the executor’s step timeout). If the awaited fact never arrives in time, the timeout fails the run and emits Fact.Playbook.Failed — exactly like a failed run: step — so look for that event, then confirm the action you expected to produce the fact actually fired.Every event that belongs to one playbook run shares a single correlation id. That id is the thread you pull to follow a workflow from its trigger, through each step, to its result — even when several daemons were involved, and even when the engine was running other workflows concurrently. Grab the id from the Fact.Playbook.Failed event or the event listing, then filter the structured logs by it to reconstruct the run end to end. See Reading logs & traces for the exact jq filters.
If a step needs logic. Playbooks have no conditionals and no branching on a step’s result. Bounded, counted repetition is expressed with a
loop:step — that much does live in the YAML. But arbitrary looping and decision-making belong inside a daemon operation. If a step seems to want anif/when, that decision belongs in the daemon, not the playbook — keep playbooks describing what, and let daemons handle how.