The dataanaliser-service is Binions’ in-flight data processor — it takes the records a playbook has just produced and turns them into something the next step can act on: summary statistics, a filtered subset, freshly derived fields, one event per item, anomaly flags, rankings, and short-term forecasts. It is the daemon you reach for when a playbook has data in hand — rows from a database query, a batch of sensor readings, a set of order totals — and you want to summarise it, keep only the rows that matter, reshape or convert its fields, hand every element to its own workflow, spot the odd value, sort it into a leaderboard, or project the trend forward. You hand it the data inline in the playbook step and it hands back the result, all over the platform’s internal event bus.
Good to know. The dataanaliser-service is one of Binions’ background services — we call them daemons. You never call it directly; you describe the calculation in a playbook and the daemon does the work. New to the platform? Start with Core concepts.
Most automations reach a point where raw data isn’t enough — you need an answer, or the data needs shaping before the next step can use it. How many orders came in last night, and what did they total? Which of these rows are the VIP customers? How do I turn "1 234,56" into a number I can add up? Who gets an email about their own overdue invoice? Is this temperature reading wildly out of line with the rest? Which five products sold the most? Where is this trend heading next week? The dataanaliser-service answers exactly those kinds of question, with seven operations:
The most important thing to understand is that the daemon is caller-driven. It does not connect to a database, watch a folder, or hold any data of its own. You pass the records to it inline in the playbook step — almost always by reusing the output of a previous step, such as the rows from a database query. That keeps the daemon tiny, stateless, and predictable: the same input always gives the same answer.
The pattern. An earlier step produces a list of records; you bind that list into a
data:(oritems:) argument; the dataanaliser-service computes the answer or reshapes the data; and a later step acts on it — saves it, mails it, charts it, or raises an alert. It is a calculator and a shaping tool on the event bus, not a data store.
| What it is | A lightweight, stateless engine for in-flight data processing in playbooks |
| Playbook prefix | analytics. — e.g. analytics.calculate_stats |
| Operations | 7 — statistics, filtering, deriving, per-item fan-out, anomaly detection, ranking, forecasting |
| How you feed it data | Inline — you pass the records in the step, usually from a previous query |
| Holds state? | No — it stores nothing and reads no database or files of its own |
| Footprint | Around 128 MB of RAM — one of the lighter daemons |
| Licensing | One of the 13 binions in a set — £1 each, £13 per host (free for one set on one host, non-commercial) |
Every operation works on the same shape of input, so it is worth learning once. You supply data — a list of records (rows). The numeric operations — statistics, anomaly detection, forecasting — also take a field naming the column to analyse; filtering, deriving, and ranking work on whole records, and the fan-out operation calls its list items. The daemon walks the list and works on what it finds.
input_count (rows you supplied), and the numeric operations add a numeric_count (rows that actually contributed), so you always know how much of your data was usable.values map on statistics, the single row a derive pipeline can reduce to, next and last on a forecast — so the next step reads the answer with one short path like ${prev.values.avg}.Because the data is passed inline, the daemon pairs naturally with any step that yields a list of records. The most common partner is the database-service: a database.query step emits its result as a rows array, which you bind straight into the analytics step with data: ${steps.<id>.rows}.
You drive the daemon from playbook steps. Each step names an operation with the analytics. prefix and passes its arguments under with:. Behind the scenes that becomes an Action.Analytics.* event, and the daemon replies with a matching Fact.Analytics.* event your playbook can wait on.
| Operation | What it does | Key arguments |
|---|---|---|
analytics.calculate_stats | Compute summary statistics over a numeric column, for the whole set or per group. | data, field, ops, group_by? |
analytics.filter | Keep only the records that match a condition. | data, where |
analytics.derive | Reshape records with a pipeline of named operations. | data, ops |
analytics.emit_items | Explode a list into one event per element. | items |
analytics.detect_anomaly | Flag values that don’t fit the rest of the series. | data, field, method, threshold? |
analytics.rank | Sort records by one or more fields and tag each with its position. | data, criteria, top_n? |
analytics.forecast | Project a numeric series forward a number of steps. | data, field, method, horizon, window? |
analytics.calculate_statsPass the records in data, the column to summarise in field, and the list of statistics you want in ops. Each entry in ops is a small object with a name (and, for percentile, a p). You can ask for several at once in a single step:
| Op name | What you get |
|---|---|
count | How many numeric values were found. |
sum | The total of the values. |
avg (alias mean) | The arithmetic average. |
min / max | The smallest / largest value. |
median | The middle value — robust to outliers. |
stdev / variance | The spread of the data. Sample statistics (divided by n−1); both need at least two values. |
percentile | The value below which a given percentage falls. Requires a p between 0 and 100 — e.g. p: 95 for the 95th percentile. |
steps:
- id: nightly_totals
run: analytics.calculate_stats
with:
data: ${steps.orders.rows} # rows from an earlier database.query
field: amount_gross
ops:
- { name: count }
- { name: sum }
- { name: avg }
- { name: median }
- { name: percentile, p: 95 }The StatsComputed fact reports the answer twice: as the detailed results, and as a flat, slug-keyed values map built for composing — each requested statistic sits under its own key, so a later step reads it directly with ${prev.values.avg} or ${steps.nightly_totals.values.sum}. Keys are always safe to use in a path: a 95th percentile lands under p95, and even a 99.9th percentile stays addressable as the dot-safe p99_9.
To compute the same statistics per category, add group_by — a list of fields to group on. The fact then also carries groups: one entry per distinct combination of the grouping fields, in order of first appearance, each with a key object (the grouping values) and its own flat values map. The whole-set figures are still reported exactly as before, so adding group_by never disturbs an existing consumer of the same step.
steps:
- id: regional
run: analytics.calculate_stats
with:
data: ${steps.orders.rows}
field: amount_gross
ops:
- { name: sum }
- { name: avg }
group_by: [region] # groups[] = one {key, values} entry per regionanalytics.filterPass data and a where condition, and only the matching records come back. The condition is written in exactly the same grammar as a playbook trigger filter — and the daemon evaluates it with the same code, so the two can never drift apart. Keys are column.operator pairs (amount.gt, status.ne, email.endswith…); a bare key means equals; the column part may be a dot-path reaching into nested fields; and all top-level entries must match.
steps:
- id: vips
run: analytics.filter
with:
data: ${steps.overdue.rows}
where:
segment: vip # bare key = equals
amount.gt: 1000 # column.operator keyThe reply is Fact.Analytics.Filtered with the surviving rows, their count, and the input_count you supplied. An empty result is a perfectly normal outcome — count: 0 and the playbook carries on. Only a malformed condition (an unknown operator, a shape the grammar doesn’t allow) is an error, reported as Fact.Analytics.OperationFailed with reason bad_filter.
analytics.derivePass data and ops — a pipeline of 1 to 64 operations applied in order to every record. Each operation is a small object: fn names the function, as names the field to write, of names the input(s). Between them the functions cover most everyday shaping jobs:
| Family | Functions |
|---|---|
| Numeric | add, sub, mul, div, round, abs, coalesce |
| Conversions | to_number, to_string, to_bool |
| Text | concat, upper, lower, trim, replace, substring, split, pad, length, regex_extract, regex_match |
| Object | get, set, rename, pick, omit, merge |
| Array | unique, sort, slice, flatten, count_by, join |
Operands follow three simple rules: a bare string is a dot-path into the record (a numeric segment indexes into an array); a number or boolean is a literal; and when you need to be explicit, {lit: X} forces a literal and {field: p} forces a path. There is deliberately no expression language — the pipeline is decoded up front, so a mistyped function name or a malformed argument fails at decode time, before any data is touched.
steps:
- id: shape
run: analytics.derive
with:
data: "${steps.q.rows}"
ops:
- { fn: to_number, as: net, of: amount_raw }
- { fn: mul, as: gross, of: [net, 1.23] }
- { fn: round, as: gross, of: gross, decimals: 2 }to_number is built for real-world input: it understands both European and US separator conventions (1 234,56 and 1,234.56) and normalises no-break spaces, thin spaces, and apostrophe grouping before parsing.
Failures are split by scope. A miss at record level — a missing field, a value that won’t convert — yields null for that record and the pipeline continues; pair it with coalesce to supply defaults. An error at action level — an invalid regex, wrong arity, more than 64 operations, or more than 10 000 records — fails the whole request with Fact.Analytics.OperationFailed. The regex functions run on a linear-time engine, so no pattern written in a playbook can stall the daemon with catastrophic backtracking.
The reply is Fact.Analytics.Derived with the reshaped rows, their count, and the input_count. When the pipeline leaves exactly one record — after a join, say — the fact also carries it flat as row, so a later step reads a field directly with ${prev.row.digest}.
analytics.emit_itemsPass items — a list — and the daemon emits one Fact.Analytics.ItemEmitted per element, each carrying the item itself, its index, the total, and a shared batch_id, followed by a summary Fact.Analytics.ItemsEmitted. The summary is what completes your playbook step; the per-item facts are there for other playbooks to trigger on. This is the platform’s per-item iteration pattern: one playbook produces and explodes the list, and a second, separate playbook handles exactly one item per run — with a trigger filter that uses dot-paths to reach into the item, such as item.status.eq: overdue.
steps:
- id: fan_out
run: analytics.emit_items
with:
items: ${steps.vips.rows}There is a hard cap of 1,000 items per call. A larger list fails loudly — the daemon never silently truncates — so trim the set first (a limit on the query, or an analytics.filter step) when volumes can grow. Worked example 3 below shows the full producer/consumer pair.
analytics.detect_anomalyPass data, the field to inspect, and a method. An optional threshold tunes how strict the flagging is. The daemon returns one entry per anomaly, each carrying the offending value, a signed score showing how far — and in which direction — it sits from normal, the record_index of the row in the data you supplied, and the original record itself. Because the index refers to your input list rather than the cleaned-up numeric series, rows skipped as non-numeric can never shift the mapping — the record you get back is always exactly the one that misbehaved, ready to drop into an alert.
z_score — flags a value whose distance from the mean, measured in standard deviations, exceeds the threshold. The default threshold is 2.5; the method needs at least two values.iqr — flags a value that falls outside Q1 − k·IQR or Q3 + k·IQR, where the threshold k defaults to 1.5. This method needs at least four values to establish the quartiles.steps:
- id: scan_readings
run: analytics.detect_anomaly
with:
data: ${steps.readings.rows}
field: temperature_c
method: iqr # interquartile range; threshold k defaults to 1.5
analytics.rankPass data and a list of criteria, each naming a field and a direction (asc or desc; desc by default). The sort is a stable multi-key sort — ties are broken by the next criterion, then by the original order — and every record comes back with a _rank field counting from 1. Supply an optional top_n to keep only the leaders. Rows whose ranking field is missing sort last when the direction is ascending.
steps:
- id: best_sellers
run: analytics.rank
with:
data: ${steps.products.rows}
criteria:
- { field: units_sold, direction: desc }
- { field: revenue, direction: desc } # tie-break
top_n: 5analytics.forecastPass data, the field to project, a method, and a horizon (how many steps ahead, at least 1). Each forecast point comes back with a 1-based step and a projected value.
linear — fits a least-squares straight line through the series and extends it. The result also reports the line’s slope, intercept, and R² (how well the line fits), so you can judge whether the trend is real.moving_avg — averages the most recent values to project the next ones. The number of values averaged is set by window (default 3).For composing, the Forecasted fact also exposes the projection as flat next and last fields — the first and the final projected values — so a follow-up step can read ${prev.next} without walking the list of points.
steps:
- id: project_signups
run: analytics.forecast
with:
data: ${steps.daily_signups.rows}
field: signups
method: linear
horizon: 7 # project the next seven daysThe dataanaliser-service rarely works alone — it sits in the middle of a playbook, between a step that produces data and a step that acts on the answer. The examples below show that shape. Each starts from a schedule (a Fact.Schedule.Fired event from the scheduler-service) and feeds on the rows array emitted by a database-service query; the last two go further and chain several analytics operations together.
About these examples: the platform does not ship any ready-made analytics playbooks — the snippets below are illustrative, written to show the wiring. Treat them as starting points to adapt, not files to copy verbatim. The vocabulary and grammar are real; the table names, fields, and thresholds are made up for the example.
Every night, total up yesterday’s orders and record the headline figures — count, revenue, average order value, median, and the 95th percentile. The flat values map makes the write step trivial: each figure is one short path.
name: nightly-sales-summary
description: "Summarise yesterday's orders and store the figures."
trigger:
event: Fact.Schedule.Fired
filter:
name.eq: nightly-sales # a schedule registered with the scheduler
steps:
- id: orders
run: database.query
with:
table: orders_yesterday # e.g. a view holding yesterday's orders
columns: [amount_gross]
limit: 10000
- id: stats
run: analytics.calculate_stats
with:
data: ${steps.orders.rows} # Fact.Database.QueryResult carries a `rows` array
field: amount_gross
ops:
- { name: count }
- { name: sum }
- { name: avg }
- { name: median }
- { name: percentile, p: 95 }
- id: record
run: database.write
with:
table: sales_daily
row:
day: ${trigger.fired_at}
orders: ${steps.stats.values.count}
revenue: ${steps.stats.values.sum}
avg_order: ${steps.stats.values.avg}
p95: ${steps.stats.values.p95}Pull the last hour of sensor readings, scan them for outliers with the interquartile rule, and post the flagged values to a webhook (for example, a Slack channel) so someone can take a look. Each anomaly entry carries the original record, so the alert shows the actual row that misbehaved, not just a number.
name: reading-anomaly-scan
description: "Flag out-of-range readings and alert a webhook."
trigger:
event: Fact.Schedule.Fired
filter:
name.eq: hourly-readings
steps:
- id: readings
run: database.query
with:
table: readings_last_hour # e.g. a view of the trailing hour
columns: [value]
- id: scan
run: analytics.detect_anomaly
with:
data: ${steps.readings.rows}
field: value
method: iqr
threshold: 1.5
- id: alert
run: webhook.send
with:
endpoint: ops-alerts
body:
text: "Anomaly scan flagged values"
anomalies: ${steps.scan.anomalies}A daily schedule reads the open invoices, keeps the VIP segment with analytics.filter, and explodes the survivors with analytics.emit_items. A second, separate playbook then triggers once per emitted item — its filter reaches into the item with dot-paths — and sends that customer their own reminder. No loops in either playbook.
name: overdue-scan-daily
description: "Read open invoices, keep the VIP segment, fan out per item."
trigger:
event: Fact.Schedule.Fired
filter:
name.eq: overdue-scan-daily
steps:
- id: overdue
run: database.query
with:
table: invoices
where:
status: overdue
limit: 500
- id: vips
run: analytics.filter
with:
data: ${steps.overdue.rows}
where:
segment: vip
- id: fan_out
run: analytics.emit_items
with:
items: ${steps.vips.rows}
---
name: overdue-remind-one
description: "Runs once per emitted item; the filter reaches into the item."
trigger:
event: Fact.Analytics.ItemEmitted
filter:
item.invoice_no.is_not_null: true
item.status.eq: overdue
steps:
- id: remind
run: mail.send
with:
from_alias: crm-outbound
to:
- ${trigger.item.customer_email}
subject: "Payment reminder — invoice ${trigger.item.invoice_no}"
body_text: |
Our records show invoice ${trigger.item.invoice_no}
(amount: ${trigger.item.amount}) is still unpaid.
Batch ${trigger.batch_id}, item ${trigger.index}.A nightly digest built with one analytics.derive chain: pull the invoice number out of each subject line with a regex, convert the raw amount to a number, compute and round the gross value, build a display line per record, sort by amount, and join everything into a single text block. Because join leaves exactly one record, the mail step reads the digest straight from the flat row.
name: overdue-invoice-digest
description: "Nightly digest: derive typed fields, sort, join, mail."
trigger:
event: Fact.Schedule.Fired
filter:
name.eq: nightly-invoice-digest
steps:
- id: q
run: database.query
with:
table: mail_inbox
where:
days_overdue.gt: 0
limit: 200
- id: shape
run: analytics.derive
with:
data: "${steps.q.rows}"
ops:
- { fn: regex_extract, as: invoice_no, of: subject,
pattern: "INV-[0-9]{4}-[0-9]+" }
- { fn: to_number, as: net, of: amount_raw }
- { fn: mul, as: gross, of: [net, 1.23] }
- { fn: round, as: gross, of: gross, decimals: 2 }
- { fn: concat, as: line,
of: [invoice_no, { lit: " — " }, gross, { lit: " PLN, " },
days_overdue, { lit: " days overdue" }] }
- { fn: sort, by: [ { field: gross, direction: desc } ] }
- { fn: join, as: digest, of: line, separator: "\n" }
- id: mail
run: mail.send
with:
from_alias: ops-out
to: [ "ksiegowosc@example.com" ]
subject: "Overdue invoices digest"
body_text: |
Overdue as of tonight (gross, sorted desc):
${steps.shape.row.digest}The dataanaliser-service reads a small TOML file. There is nothing to tune for the analytics themselves — the operations, statistics, and thresholds are chosen per request in each playbook step, never in config. The file only covers the plumbing: the daemon’s private state store, its health endpoint, and optional tracing.
[redis]
host = "127.0.0.1"
port = 6396
password_file = "/opt/binions/dataanaliser-service/config/redis.password"
[healthcheck]
listen_addr = "127.0.0.1:9106"
# Optional — export traces to a collector such as Jaeger
[otel]
endpoint = "http://127.0.0.1:4317"Field by field:
redis — the daemon’s own private state store, used for its transactional outbox and duplicate-event protection. The password is read from a file, never written inline.healthcheck.listen_addr — the local address that answers liveness and readiness probes and serves metrics.otel — optional tracing. Point it at a collector to follow a request end to end across the platform; leave it out and tracing is simply off.Because all of its work is in-memory processing, the daemon stays small — it runs comfortably in about 128 MB of RAM. For the full picture of how daemons are configured and where secrets live, see Daemon configuration and Secrets.
Like every Binions daemon, the dataanaliser-service speaks in events. You ask it to do something with an Action, and it announces what happened with a Fact. Every event carries a correlation id, so you can trace a request from the triggering event all the way to the answer.
| You send (Action) | The daemon replies (Fact) |
|---|---|
Action.Analytics.CalculateStats | Fact.Analytics.StatsComputed |
Action.Analytics.Filter | Fact.Analytics.Filtered |
Action.Analytics.Derive | Fact.Analytics.Derived |
Action.Analytics.EmitItems | Fact.Analytics.ItemsEmitted — preceded by one Fact.Analytics.ItemEmitted per element |
Action.Analytics.DetectAnomaly | Fact.Analytics.AnomaliesDetected |
Action.Analytics.Rank | Fact.Analytics.Ranked |
Action.Analytics.Forecast | Fact.Analytics.Forecasted |
| Any operation that fails | Fact.Analytics.OperationFailed |
When a request can’t be honoured, the daemon does not fall over — it emits a Fact.Analytics.OperationFailed that names the operation, gives a short reason (such as bad_input for a missing field, empty_series when no numeric values were found, or bad_filter for a malformed condition), carries a human-readable message, and references the originating action so you can correlate it.
Two of these facts double as trigger events: a playbook can be started by Fact.Analytics.ItemEmitted (the per-item pattern in worked example 3) or by Fact.Analytics.OperationFailed (an analytics-failure alert playbook). To understand the shape these events share, see the event envelope reference.
The dataanaliser-service installs and runs like every other daemon: a managed system service that starts on boot, restarts on failure, and runs under its own locked-down service account with a watchdog. Install it as part of a Binions set and enable it alongside its companion state store:
sudo systemctl enable --now binions-dataanaliser redis-binions-dataanaliser
systemctl status binions-dataanaliserThe platform validates the configuration file before the daemon is allowed to start, so a typo in your TOML is caught immediately instead of after deploy. You can confirm liveness and readiness with the daemon’s health endpoints:
curl http://127.0.0.1:9106/health/live
curl http://127.0.0.1:9106/health/readyGeneral operational guidance lives in Service operations, Health checks, and Monitoring & tracing.
Licensing note. The dataanaliser-service is one of the 13 binions in a set. During the current alpha, the boot-time licence check defaults to off, so daemons start regardless. When enforcement is enabled, an unlicensed binion will refuse to boot — see First boot for how the check works.
* The alpha packages are for 64-bit x86 (amd64) only. Raspberry Pi and other 64-bit ARM hardware are supported from the public 1.0 release.