The logger-service is the platform's central log sink — the one place where everything that happens on your host is written down. Every other daemon, while it works, emits short log records; the platform delivers those records onto an internal event stream, and the logger-service consumes that stream and appends each record as a single line to one structured log file. It also keeps that file tidy on its own — rotating, compressing and pruning it without any outside helper — and, if you opt in, it will raise a platform fact when one daemon starts erroring unusually often. There is nothing to wire up and no “log” step to add to a playbook — logging is automatic. This page is about how to read what the logger-service collects, and how to tune its built-in housekeeping.
Good to know. Unlike most daemons, the logger-service takes no instructions from playbooks. It does not register any actions you can call — it only listens to the internal log stream and writes it down. You will never address it from a playbook — though a playbook can listen to it: with error-rate alerting switched on, the logger-service emits one kind of fact that playbooks can trigger on (see below).
Binions is event-driven: the daemons talk to each other over a fast internal event bus, and every interesting thing — a playbook step running, an email being sent, a query failing — is described as an event riding in a standard envelope. Among those events are log records. The logger-service subscribes to the stream that carries them and turns it into a durable, machine-readable log file you can search, tail, and ship elsewhere.
Here is the journey of a single log line, end to end:
info, warn, error…), a human-readable message, and a bag of structured fields (which order, which mailbox, how many rows).The big idea. Every daemon writes its diagnostics to its own local journal as it runs, but the logger-service gives you one consolidated, structured stream of the whole platform — so you can answer “what happened across the system at 09:14 this morning?” from a single file instead of stitching together a dozen separate logs.
| What it is | The platform's central log sink — consumes the internal log stream and writes it to one file |
| What you do with it | Read it — and, optionally, react to its error-rate alerts. There are no playbook verbs and no actions to call; logging happens automatically |
| Output format | Newline-delimited JSON (JSON Lines) — one complete event per line |
| Where the log lives | A single JSON Lines file under the logger-service log directory (full path below) |
| Retention | Built in ([retention]) — the daemon rotates its own event log by size and by UTC day, gzips old files, and prunes them by age and total size |
| Error-rate alerting | Opt-in ([alerting]) — emits one Fact.Logs.ErrorRateExceeded per producer when errors spike, for playbooks to react to |
| Bad records | Quarantined to a sibling dead-letter file — never silently dropped |
| Service | Its own hardened systemd service, with a dedicated Redis instance for its state |
| Health & metrics | A local-only HTTP endpoint exposing liveness, readiness, and metrics |
| Package | binions-logger — one of the 13 binions in a set |
Each line in events.jsonl is a complete event envelope — the same shape every daemon on the platform uses. That uniformity is what makes the log easy to filter: the fields are always in the same place, whatever produced the line.
| Field | What it tells you |
|---|---|
event_id | A unique id for this single event. |
event_kind | Always "Log" for log lines — handy when you mix logs with other event records. |
event_type | The specific record type within that kind. |
producer | Which daemon wrote it — e.g. the mailbox, scheduler, or webhook caller service. |
correlation_id | The thread that ties one run together. Every event from the same playbook run shares it — grep this to reconstruct an entire workflow. |
payload | The log itself: level (severity), message (the human text), and fields (structured context). |
metadata | Carries the trace_id — the link to distributed tracing (see below). |
A single line, pretty-printed for readability (in the file it is all on one line):
{
"event_id": "01J9Z6K2C4P0M7Q3R8S5T1V2W3",
"event_kind": "Log",
"event_type": "service.log",
"producer": "mailbox-service",
"correlation_id": "01J9Z6K1A0B2C4D6E8F0G2H4J6",
"payload": {
"level": "error",
"message": "failed to fetch message",
"fields": { "mailbox": "invoices", "uid": 4821, "reason": "timeout" }
},
"metadata": { "trace_id": "7f3c2a9b1e4d5c6a8b0d1e2f3a4b5c6d" }
}Because the log is JSON Lines, the standard Unix toolchain works beautifully — tail to follow it, jq to filter and reshape it, grep to pull a single thread. Here are the recipes you will reach for most.
# Watch every event as it lands, pretty-printed
tail -f /var/log/binions/logger-service/events.jsonl | jq .# Just the error lines, compact
tail -f /var/log/binions/logger-service/events.jsonl \
| jq -c 'select(.payload.level == "error")'
# Errors and warnings together
jq -c 'select(.payload.level == "error" or .payload.level == "warn")' \
/var/log/binions/logger-service/events.jsonl# Every line produced by the mailbox service, as "time level message"
jq -r 'select(.producer == "mailbox-service")
| "\(.payload.level)\t\(.payload.message)"' \
/var/log/binions/logger-service/events.jsonlEvery event from one run carries the same correlation_id. Once you have it from any line, you can pull the complete story of that run — across every daemon that took part — in chronological order.
# Replace the id with the correlation_id you are chasing
CID=01J9Z6K1A0B2C4D6E8F0G2H4J6
jq -c --arg cid "$CID" 'select(.correlation_id == $cid)' \
/var/log/binions/logger-service/events.jsonlIf you would rather look at the raw event stream before it is written to file — for a quick sanity check that logs are flowing — the platform's command-line console can read it. Plain ls events prints the most recent entries; add --follow and it keeps streaming new ones as they land, exactly like tail -f for the bus:
# Print the latest events, then tail the live internal stream
binions-cliconsole ls events --followStructured fields are the payoff of JSON logs. Instead of writing fragile text patterns, filter on the data itself — e.g. jq 'select(.payload.fields.mailbox == "invoices")' — and every daemon's context becomes queryable in the same way.
Each log line carries a trace_id in its metadata. If you enable the platform's optional tracing, that same id appears on the distributed traces your daemons export — the timing breakdown of a request as it hops from one service to the next, viewable in a tracing tool such as Jaeger. That shared id is the bridge between the two views of your system:
events.jsonl to read exactly what each step logged while it ran.trace_id, and open the matching trace to see where in the chain the time went or the failure occurred.Tracing is optional — logging works fully on its own. Enable tracing and your log lines and traces share a trace_id; leave it off and the logs are exactly as complete, just without the cross-link to a trace viewer.
Reading the log tells you what happened. The opt-in [alerting] block goes one step further: it makes the logger-service notice when a daemon starts failing loudly, so something — or someone — can react. Switched on, the logger-service watches the error rate per producer as it writes: when more than threshold error-level records arrive from a single producer within window_secs, it emits one Fact.Logs.ErrorRateExceeded onto its own event stream. The fact carries producer, count, window_secs and threshold — everything a playbook needs to say who is failing and how hard.
Two properties keep it safe and quiet:
cooldown_secs — a sustained failure becomes one alert per cooldown window, not an alert storm.Because the alert is an ordinary platform fact (see the event catalog), any playbook can trigger on it — the classic use is “when a daemon is erroring hard, tell a human”:
name: error-rate-to-ops-mail
description: Business — a daemon is erroring hard; page operations.
trigger:
event: Fact.Logs.ErrorRateExceeded
steps:
- id: page
run: mail.send
with:
from_alias: ops-out
to: [ "ops@example.com" ]
subject: "High error rate: ${trigger.producer}"
body_text: |
${trigger.producer} logged ${trigger.count} errors within
${trigger.window_secs} seconds (threshold: ${trigger.threshold}).
Pull its lines from the event log to see what is failing.The logger-service reads a single application.toml in its package configuration directory. Most installs never need to touch it — the defaults are sensible — but here is the full shape so you know what each knob does.
# Connection to this daemon's dedicated Redis instance
[redis]
host = "127.0.0.1"
port = 6390
password_file = "/etc/binions/secrets/logger-redis.pass"
# The internal stream of log events, read as a consumer group
[stream]
name = "events:logs"
consumer_group = "logger-service"
consumer_name = "logger-1"
block_ms = 200 # how long to wait for new events before looping
# Batch many events before each write, so the disk is touched efficiently
[batching]
batch_size = 1000 # flush once this many events are buffered...
flush_interval_ms = 500 # ...or at least this often, whichever comes first
# Where the consolidated log file is written
[file]
path = "/var/log/binions/logger-service/events.jsonl"
# Built-in retention — the logger-service rotates its own event log
[retention]
enabled = true
rotate_max_bytes = 104857600 # rotate at 100 MiB
rotate_daily = true # also rotate after a UTC day change
max_age_days = 7 # prune rotated files older than 7 days
max_total_bytes = 1073741824 # hard cap (active + rotated); oldest go first
compress = true # gzip rotated files
check_interval_secs = 300 # housekeeping sweep cadence
# Opt-in error-rate alerting (see above)
[alerting]
enabled = true
window_secs = 60 # the sliding window watched per producer
threshold = 25 # more errors than this within the window fires...
cooldown_secs = 300 # ...at most one fact per producer per cooldown
# Local health and metrics HTTP endpoint
[healthcheck]
listen_addr = "127.0.0.1:9100"
# Optional: export traces so log trace_ids line up with a trace viewer
[otel]
endpoint = "http://127.0.0.1:4317"
sample_rate = 1.00 disables that dimension; enabled = false hands the file back to external tools. The full behaviour is described under Running & health below.[alerting] is opt-in. Leave it out (or set enabled = false) and the logger-service never emits anything; switch it on and you get the error-rate fact described above — with no impact on the write path either way.[otel] block is optional. Omit it and tracing is simply off; logging is unaffected.password_file, in line with how every daemon handles credentials — see Secrets & credentials.The logger-service is very nearly a pure consumer — it is the end of the road for log records, and it exposes no actions for playbooks to call. The one event it can produce is the opt-in error-rate alert.
| Direction | Stream | What it carries |
|---|---|---|
| Consumes | events:logs | Every Log envelope emitted by every daemon on the host, delivered via each daemon's transactional outbox. |
| Emits | events:logger | Only with [alerting] enabled: a single Fact.Logs.ErrorRateExceeded when one producer crosses the configured error-rate threshold. Nothing else — and never any playbook responses. |
For the full anatomy of the envelope every line shares, see The event envelope.
Like every binion, the logger-service runs as its own hardened systemd service under a dedicated, unprivileged loggersvc user, alongside its own Redis instance. It is installed and enabled the usual way:
# Install the package and start both units
sudo apt install binions-logger
sudo systemctl enable --now redis-binions-logger binions-logger
# Check status
systemctl status binions-loggerThe service is Type=notify with a watchdog, so systemd knows when it is truly ready and restarts it on failure. You can confirm it is healthy and watch its throughput from the local health endpoint:
# Liveness and readiness
curl -s http://127.0.0.1:9100/health/live
curl -s http://127.0.0.1:9100/health/ready
# Prometheus-style metrics (events consumed, written, quarantined, lag)
curl -s http://127.0.0.1:9100/metricsThe logger-service owns the housekeeping of its own event log — there is no external rotation job to set up for events.jsonl. On a periodic sweep (every check_interval_secs, five minutes by default) it:
rotate_max_bytes (100 MiB by default), and — with rotate_daily — whenever the UTC day changes;compress is on — compression runs off the hot path, so it never delays a log line;max_age_days (7 by default), and enforces max_total_bytes (1 GiB by default) across the active file and everything rotated — when the cap is hit, the oldest rotated files go first.Rotation is a flush-and-reopen, so it never drops or truncates a line, and the defaults mirror the rotation behaviour installs have always had. Setting a size or age limit to 0 switches that dimension off entirely. And if you would rather manage the file with your own tools, set enabled = false under [retention] — the logger-service then leaves rotation to the outside world, and you signal the flush-and-reopen yourself whenever you rotate:
# Flush buffered events and reopen the log file
sudo systemctl reload binions-loggerEither way, the logrotate configuration shipped with the packages covers only the daemons' plain-text *.log files — the structured event log is the logger-service's own responsibility.
If a record arrives malformed and cannot be written as valid JSON, the logger-service does not discard it and does not stop. It quarantines the offending record to a dead-letter file — events.dlq.jsonl, sitting right next to the main log — and carries on. A non-empty dead-letter file is your signal to investigate — the good logs keep flowing the whole time.
Watch the dead-letter file. Lines in
events.dlq.jsonlmean something upstream produced a record the sink could not parse. It is harmless to logging itself, but worth looking into — see Reading logs & traces for how to triage it.