The data-transfer integration moves, shares, parses, and transforms files across storage systems — object stores, SFTP and FTP servers, the local disk, MongoDB GridFS, and HTTP push endpoints — from inside your playbooks. One background service handles it all: you point it at a backend, then a playbook can upload a generated report, download an incoming file, list what is in a bucket, copy or move an object between any two backends, mint a time-limited share link, lift the text out of a stored PDF, or convert a file between JSON, CSV, and Excel. This page is the integration view; for the full service reference see the data transporter service.
Good to know. Every storage operation is one playbook step. You never write storage SDK code — you call a small set of generic verbs (such as “upload” or “list objects”) and the service talks to whichever backend that bucket is wired to. Swapping S3 for SFTP later changes the backend, not your playbooks.
Binions treats storage as a single integration with several interchangeable backends. There is one broker for data transfer, and a backend: field selects where the bytes actually live. The playbook verbs are identical no matter which backend you choose (the write-only HTTP sink being the one exception), so a playbook written against the bundled object store works unchanged against an SFTP server once you re-point the bucket.
| Playbook prefix | data. |
| Event domain | Data — actions emit Fact.Data.* on success and a unified Fact.Data.OperationFailed on error |
| Backends | Seven: s3, minio, sftp, ftp, local_fs, mongo (GridFS), http_ingest (write-only push) |
| Default backend | The bundled object store (minio); a local filesystem path is the other zero-dependency option |
| What it moves | Any object — text or binary — addressed by a bucket and a key |
| File hand-off | Every file-carrying fact exposes flat bucket and key fields (plus sha256 and content type), and every data. verb accepts them |
Each backend is described once during setup; from then on a playbook simply names the bucket. The seven backends share one programming model — a bucket holds objects, each object has a key — so the same verbs apply across all of them, with one deliberate exception: the HTTP ingest backend only accepts writes.
| Backend | What it is | Typical use |
|---|---|---|
s3 / minio | S3-compatible object storage. The default is the bundled MinIO object store that ships with Binions; s3 points the same protocol at AWS S3 or any compatible endpoint. | Reports, attachments, AI outputs, anything you want addressable by URL |
local_fs | A directory on the host’s own filesystem, exposed through the same bucket-and-key model. | Simple, dependency-free storage when you do not need an object store |
sftp | A remote SFTP server over SSH. | Exchanging files with suppliers, partners, or legacy systems |
ftp | A remote FTP / FTPS server. | Integrating with older endpoints that only speak FTP |
mongo | MongoDB GridFS. The bucket maps to a GridFS bucket inside a MongoDB database (default binions), reached through a connection string you keep in a secret. Re-uploading an existing key overwrites it — old revisions are dropped, not stacked. | Archiving documents next to application data that already lives in MongoDB |
http_ingest | A write-only push sink. Every upload becomes one HTTP POST of a multipart/form-data request — a file part whose filename is the key, plus key and sha256 form fields — to a fixed endpoint, with an optional bearer token. Reads, listings, and deletes refuse loudly. | Feeding a RAG or search-indexing service: every file written to the bucket is pushed straight to your indexer |
MinIO and S3 speak the same protocol — the only practical difference is the endpoint URL — so a bucket can move from the bundled store to a managed S3 service without touching the playbooks that use it. SFTP and FTP buckets are addressed with a host, a remote root directory, and credentials. Two backend-specific notes: the GridFS backend connects and pings MongoDB eagerly at registration (an unreachable database fails the setup step, not your first transfer), and neither mongo nor http_ingest can host a watched folder — a registration that combines them with a watch directory is rejected.
Credentials stay on the host. Backend credentials — access keys, an SFTP username and password, a MongoDB connection string, an ingest bearer token — are configured on your machine and used by the service directly. Nothing about a transfer is sent to a third party; the bytes go straight between your host and the backend you named.
A playbook step names a verb under the data. prefix and supplies its arguments. The family counts eleven verbs: the nine operational ones below, plus two provisioning verbs covered under Registering a backend. Each verb performs one storage operation and, on success, emits a fact your next step can react to.
| Verb | What it does | Success fact |
|---|---|---|
data.upload | Write an object to a bucket | Fact.Data.Transported |
data.download | Read an object back out | Fact.Data.Downloaded |
data.list_objects | List the objects under a prefix, optionally sorted | Fact.Data.Listed |
data.delete_object | Remove an object | Fact.Data.Deleted |
data.copy | Copy an object between any two buckets | Fact.Data.Copied |
data.move | Copy an object to another bucket, then delete the source | Fact.Data.Moved |
data.presign | Mint a time-limited download URL for a stored object | Fact.Data.Presigned |
data.parse | Extract the text from a stored PDF, DOCX, or text file | Fact.Data.Parsed |
data.transform | Convert a body between JSON, CSV, and Excel | Fact.Data.Transformed |
If an operation fails — a missing key, an unreachable backend, a malformed body, a read against the write-only ingest sink — the service emits Fact.Data.OperationFailed instead, carrying the operation name and a reason, so a playbook can branch on the failure rather than stalling.
data.upload takes key and a body — body for text or body_b64 for binary — plus optional bucket and content_type. Omit bucket and it writes to the default object store.data.download takes key (and optional bucket); add as_b64: true to receive a binary object as base64 instead of text.data.list_objects takes an optional prefix to filter, max_keys to cap the result count, and sort (name, mtime, or size) with order (asc / desc). The result carries the listing plus a flat first object — sort by mtime descending and ${prev.first.key} is the newest file, in one step. Objects without a timestamp always sort last.data.delete_object takes key (and optional bucket).data.copy and data.move take from_bucket, from_key, to_bucket, and to_key — only from_key is required, and to_key defaults to the source key.data.presign takes key (and optional bucket) plus expires_secs — default 3600 (one hour), maximum 604800 (seven days).data.parse takes a stored object (bucket + key) or inline bytes as content_b64, plus optional format and max_text_bytes.data.transform takes format_from, format_to, and the body to convert.Storage backends would be silos if every pair needed its own bridge. They do not: every file-carrying fact in the platform exposes the same flat bucket and key fields (plus sha256 and content type where known), and every data. verb accepts them. Two verbs turn that convention into a universal file bus:
data.copy reads the object from the source bucket and writes it to the destination through the bucket registry — so any backend pair composes, with no per-pair plumbing.data.move does the same and then deletes the source object. If that final delete fails after a successful copy, the step fails loudly with Fact.Data.OperationFailed — you get an alertable fact, never a silent duplicate living in two places.On success they emit Fact.Data.Copied / Fact.Data.Moved, carrying the source and destination coordinates plus the byte count and checksum — and both are trigger events, so a further playbook can chain on a completed hop. Typical hops: a handled file from a watched folder moves to processed/; a mail attachment moves into a GridFS archive; a markdown note is copied into the RAG feed. Here is the archive hop — mail attachments are offloaded to storage and announced with Fact.Data.Transported, and this playbook picks up every DOCX among them:
name: mail-docx-to-archive
description: |
Business — every offloaded .docx mail attachment moves to the archive
(source key vanishes; to_key defaults to the source key).
trigger:
event: Fact.Data.Transported
filter:
key.startswith: "mail/"
key.endswith: ".docx"
steps:
- id: archive
run: data.move
with:
from_bucket: ${trigger.bucket}
from_key: ${trigger.key}
to_bucket: edi-archivedata.presign turns a stored object into a time-limited download URL that works for anyone who holds it — no Binions login, no storage credentials. Name the object and how long the link should live: expires_secs defaults to 3600 (one hour) and caps at 604800 (seven days, the signing protocol’s limit). The result fact Fact.Data.Presigned carries the url and its expires_at timestamp, ready to drop into a mail or a webhook. Pre-signing is an object-store capability: it works on s3 and minio buckets only, and asking for a link on any other backend fails loudly with Fact.Data.OperationFailed rather than handing you a URL that could never work.
name: report-share-link
description: |
Business — every object landing under reports/ is turned into a
24-hour pre-signed link and mailed to operations.
trigger:
event: Fact.Data.Transported
filter:
key.startswith: "reports/"
steps:
- id: link
run: data.presign
with:
bucket: ${trigger.bucket}
key: ${trigger.key}
expires_secs: 86400
- id: notify
run: mail.send
with:
from_alias: ops-mail
to: ["ops@example.com"]
subject: "Report ready: ${trigger.key}"
body_text: |
The nightly report is ready. Download (valid until ${steps.link.expires_at}):
${steps.link.url}data.parse lifts the text out of a stored document so the rest of the platform can work on it. Point it at a stored object (bucket + key) or pass the bytes inline as content_b64; it answers with Fact.Data.Parsed, carrying the extracted text along with the detected format, text_bytes, truncated, and source. Format detection is automatic — file extension first, then the file’s magic bytes, then plain text — or pin it with format: pdf / docx / text. Long documents are cut at max_text_bytes (default 1 MiB) on a character boundary, with truncated: true set so you know. A malformed file produces a parse-failure fact you can react to; it never wedges the service.
Text layer only — no OCR.
data.parsereads the text a PDF already contains; a scanned PDF with no text layer fails with reasonno_text_layer. Route scans to an external OCR service instead — call it withwebhook.sendandexpect_json: true, then read the recognised text back from the response.
The natural consumer is AI extraction: parse the text layer, hand it to ai.extract for typed fields, persist the result. A mail whose PDF attachment was offloaded to storage becomes database rows without the document ever leaving the platform — here with a two-provider fallback chain and a pre-flight cost cap on the AI step:
name: invoice-pdf-to-sql
description: |
Business — an offloaded PDF attachment lands in the store; parse its
text layer in-platform, extract typed fields with a fallback provider
chain and a $0.05 pre-flight cost cap, persist to SQL.
trigger:
event: Fact.Data.Transported
filter:
key.startswith: "mail/"
key.endswith: ".pdf"
steps:
- id: parse
run: data.parse
with:
bucket: ${trigger.bucket}
key: ${trigger.key}
max_text_bytes: 262144
- id: extract
run: ai.extract
with:
text: ${steps.parse.text}
providers: [primary, local]
max_cost_usd: 0.05
on_mismatch: fail
fields:
- { name: no, type: text }
- { name: total, type: decimal }
- { name: sender, type: text }
- id: persist
run: database.write
with:
table: invoices
row:
no: ${steps.extract.result.no}
total: ${steps.extract.result.total}
sender: ${steps.extract.result.sender}data.transform is a format converter: it parses a body in one tabular format and re-serialises it in another, driven entirely by format_from and format_to. The supported formats are json, csv, and xlsx (Excel), in any combination. Pass the source in body for the text formats (JSON, CSV) or in body_b64 for the binary XLSX format; the result comes back the same way — a text body for JSON or CSV, base64 for XLSX.
- run: data.transform
with:
format_from: csv
format_to: xlsx
body: |
invoice,amount,currency
INV-001,420.00,GBP
INV-002,99.50,GBPTransform is purely about file format — CSV to Excel, JSON to CSV, and so on. It is not a field-extraction or record-splitting step. To reshape the records inside a payload — derive fields, convert types, or split a list into per-item facts — use the analytics verbs; to lift text out of a document, use data.parse; to reshape a whole file’s format, use data.transform.
Wiring a bucket to a backend is itself a playbook verb: data.register_bucket gives a bucket_id a backend type, an endpoint or host or root, and credentials, and answers with Fact.Data.BucketRegistered (data.unregister_bucket reverses it, with Fact.Data.BucketUnregistered). Registration is provisioning, not business logic — the convention is a provisioning playbook triggered on Fact.System.Boot, so the wiring re-applies on every start. The default object store is already wired out of the box, so uploads and downloads work immediately without registering anything; you register additional buckets to reach an SFTP or FTP server, an external S3 service, a specific local directory, a MongoDB GridFS archive, or an HTTP ingest endpoint. Connection strings and tokens come from the secrets store via ${secret.*}, so they never sit in the playbook file:
name: register-notes-file-bus
description: Provisioning — watched inbox (local_fs) + two sinks.
trigger:
event: Fact.System.Boot
filter:
component.eq: playbook-service
steps:
- id: inbox
run: data.register_bucket
with:
bucket_id: notes-inbox
backend: local_fs
root: /var/lib/binions/datatransporter
watch_dir: notes-incoming
- id: rag
run: data.register_bucket
with:
bucket_id: rag-ingest
backend: http_ingest
endpoint: "http://127.0.0.1:8899/api/ingest"
credentials:
access_key: bearer
secret_key: ${secret.RAG_INGEST_TOKEN}
- id: mongo_archive
run: data.register_bucket
with:
bucket_id: docs-archive
backend: mongo
dsn: ${secret.MONGO_APP_DSN}
database: binionsBeyond on-demand transfers, the service can act as a source of events. A bucket registration can name a watch_dir — for example an incoming directory on an SFTP server, or the local drop folder registered above — and the service emits a Fact.Data.FileDiscovered for each new file that appears there, so a playbook reacts to deliveries instead of polling for them. This turns a supplier’s drop folder into a trigger: a file lands, a fact fires, your playbook takes it from there. The two push-oriented backends are the exception — mongo and http_ingest buckets cannot be watched.
Watched folders and the file bus compose naturally. Continuing the registration example: every markdown note dropped into the watched inbox fans out to both sinks in parallel — pushed to the RAG indexer and archived in GridFS:
name: markdown-to-rag-and-mongo
description: Business — per discovered .md file: feed the RAG and archive, in parallel.
trigger:
event: Fact.Data.FileDiscovered
filter:
bucket_id.eq: notes-inbox
key.endswith: ".md"
steps:
- parallel:
- run: data.copy
with:
from_bucket: ${trigger.bucket_id}
from_key: ${trigger.key}
to_bucket: rag-ingest
- run: data.copy
with:
from_bucket: ${trigger.bucket_id}
from_key: ${trigger.key}
to_bucket: docs-archive1. Save a generated report. A scheduled step produces a report and uploads it to the default object store under a dated key:
- run: data.upload
with:
key: reports/2026-05/summary.txt
content_type: text/plain
body: |
Daily summary
Orders processed: 128
Revenue: GBP 4,210.002. Convert, then store. Turn a CSV export into an Excel workbook and upload the result — two steps, the second consuming the first:
- id: convert
run: data.transform
with:
format_from: csv
format_to: xlsx
body: |
invoice,amount,currency
INV-001,420.00,GBP
INV-002,99.50,GBP
- run: data.upload
with:
key: exports/orders.xlsx
content_type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
body_b64: ${steps.convert.body_b64}3. Fetch the newest delivery. Sort the listing by modification time and the flat first object is the newest file — no loop, no date arithmetic — then download it for processing:
- id: newest
run: data.list_objects
with:
bucket: supplier-sftp
prefix: incoming/
sort: mtime
order: desc
- run: data.download
with:
bucket: supplier-sftp
key: ${steps.newest.first.key}The data-transfer service is wired into the same event bus as every other integration, so storage is rarely the whole story — it is usually one step in a larger flow. A mailbox attachment becomes an upload; an AI extraction becomes a stored result; a discovered file becomes the start of a pipeline. The platform also leans on this service automatically: when another service produces a result that is too large to pass around inline, that payload is offloaded to the object store and announced with Fact.Data.Transported, keeping the event bus lean while the bytes stay close at hand. And because every such announcement carries the same flat bucket and key, the pipelines compose end to end: the offloaded attachment can be parsed, the parsed text extracted into typed fields and persisted, the original moved to an archive, and a pre-signed link mailed out — all as ordinary steps. See the integration model for how actions and facts connect, and example workflows for end-to-end pipelines.