datatransporter-service is the platform's object-storage and file-transfer broker — the daemon that moves, parses, and transforms files across storage backends. When a playbook needs to put a file somewhere durable, pull one back, list what is there, move or copy it between backends, delete an object, extract the text of a document, mint a time-limited download link, or convert a payload from one tabular format to another, this is the daemon that does it. It speaks to several different storage systems — an S3-compatible object store, the local filesystem, an SFTP or FTP server, a MongoDB GridFS store, or an HTTP ingest endpoint — behind one small, consistent set of operations.
Good to know. This daemon handles files and objects — reports, exports, attachments, backups, supplier drops. It is the right place to send a generated document or to fetch one for a later step. It is not a database (that is database-service) and it is not an analytics engine (that is dataanaliser-service). Think of it as the platform's filing room and loading dock.
Binions is event-driven: its daemons talk over a fast internal event bus, and everything that happens is described as an event riding in a standard envelope. datatransporter-service turns storage and file handling into a small vocabulary of operations your playbooks can call, and announces the result back on the bus for later steps to use.
Files live in buckets — named storage locations the daemon knows about. A bucket points at one storage backend: an S3-compatible object store (the bundled default is MinIO), a directory on the local filesystem, an SFTP or FTP server, a MongoDB GridFS store, or a write-only HTTP ingest endpoint. Because the backend is a property of the bucket rather than of the operation, the same data.upload step works whether the file ends up in object storage or on a remote server.
data.move and data.copy are a get→put (→delete) through the bucket registry, so any backend pair composes: a watched directory drains into a processed/ prefix in object storage, an offloaded mail attachment lands in an archive, a markdown note is copied to a RAG service.data.parse extracts the text of a text-layer PDF or a DOCX (or passes plain text through) so the next step — typically ai.extract — can work on it. Honest limit: no OCR; a scanned PDF fails loudly instead of returning nonsense.data.presign mints a time-limited download URL for an object in S3/MinIO — hand a report to a person or an external system without shipping bytes or credentials.data.transform operation converts a payload between JSON, CSV, and XLSX — so you can turn a query result into a spreadsheet, or a CSV export into JSON, without leaving the platform.data.register_bucket and data.unregister_bucket. Runtime bindings are held in memory — a daemon restart forgets them — so durable registrations belong in a provisioning playbook triggered by Fact.System.Boot, which re-runs on every boot and heals the registry automatically.Fact.Data.Transported when an upload completes — and every file-carrying fact exposes a flat bucket + key pair (plus sha256 and content_type where known) that every data.* operation accepts, so the next step picks up exactly where this one left off.The big idea. Storing a file, fetching it back, moving it on, and reshaping it are just more actions a playbook can take. That means “receive an invoice PDF by mail, parse it, extract the figures, and file both the data and the document” is one short workflow built from the same small set of verbs as everything else on the platform.
| What it is | The object-storage & file-transfer broker — move, parse, and transform files across storage backends |
| Playbook prefix | data. |
| Operations | Upload, download, list objects, delete object, move, copy, pre-sign, parse, transform format, register bucket, deregister bucket |
| Storage backends | S3 / MinIO (the bundled default), local filesystem, SFTP, FTP, MongoDB GridFS, HTTP ingest (write-only) |
| Format conversion | JSON ↔ CSV ↔ XLSX via data.transform |
| Document parsing | Text-layer PDF, DOCX, and plain text via data.parse — no OCR |
| Share links | data.presign — time-limited download URLs for S3/MinIO objects (up to 7 days) |
| Headline event | Fact.Data.Transported — emitted when an upload completes, with a reference to the stored object |
| Service | binions-datatransporter.service with a dedicated redis-binions-datatransporter.service |
| Health endpoint | 127.0.0.1:9105 — /health/live, /health/ready, /metrics |
In a playbook you address the daemon with the lowercase verb form run: data.<operation>, which the platform turns into the daemon's internal Action.Data.<Verb>. There are eleven operations — six that put, fetch, list, delete, move, and copy objects; one that extracts a document's text; one that mints a time-limited download link; one that converts a payload's format; and two that manage bucket registration at runtime:
| Operation | What it does | Arguments |
|---|---|---|
data.upload | Store a file or payload under a key in a bucket. Emits Fact.Data.Transported with a reference to the stored object. | key, and exactly one of body (text), body_b64 (base64 bytes) or body_json (structured JSON, stored verbatim — handy for ${prev.rows}); optional bucket and content_type |
data.download | Fetch an object back into the playbook so a later step can use its contents. | key; optional bucket; as_b64 to receive binary content base64-encoded |
data.list_objects | List the objects in a bucket, optionally narrowed to a key prefix and sorted the way you need. The Fact.Data.Listed result mirrors the first entry as a flat first{} — so “take the newest file” is sort: mtime, order: desc and ${prev.first.key}. | optional bucket, prefix, max_keys; sort (name, mtime or size) with order (asc or desc) — objects with no timestamp always sort last |
data.delete_object | Remove an object from a bucket. | key; optional bucket |
data.move | Move an object to another key, another bucket, or both — on any backend pair — in one step: a copy through the bucket registry followed by a source delete. Emits Fact.Data.Moved. | from_key; optional from_bucket, to_bucket, to_key (defaults to from_key) |
data.copy | Copy an object to another key or bucket, leaving the source in place. Emits Fact.Data.Copied. | same as data.move |
data.presign | Mint a time-limited download URL for a stored object so people and systems outside the platform can fetch it without credentials. Emits Fact.Data.Presigned with url and expires_at. S3/MinIO buckets only — other backends fail loudly with unsupported. | key; optional bucket; expires_secs (default 3600 — one hour; maximum 604 800 — seven days) |
data.parse | Extract a document's text — a text-layer PDF, a DOCX, or plain text — from storage or from inline bytes, ready for ai.extract or a template. Emits Fact.Data.Parsed. No OCR: a scanned PDF fails loudly instead of returning nonsense. | key (with optional bucket) or content_b64; optional format (auto — the default — pdf, docx, text) and max_text_bytes (default 1 MiB — longer text is truncated and flagged) |
data.transform | Convert a payload between tabular formats — JSON, CSV, and XLSX — without touching storage. A pure data step you typically place between a source and an upload. | format_from, format_to, and the data as body or body_b64 |
data.register_bucket | Register a storage backend at runtime, making it available to subsequent steps under the given bucket name. Use for backends not pre-configured in the daemon's settings file — and re-register from a boot-triggered provisioning playbook, because the registry is in-memory. | bucket_id, backend (one of s3, minio, sftp, ftp, local_fs, mongo, http_ingest), plus backend-specific details: endpoint (S3/MinIO; the target URL for HTTP ingest), host/port/root (SFTP/FTP), root (local filesystem), dsn + database (MongoDB — pass the DSN as a ${secret.…} reference; database defaults to binions), credentials (access_key + secret_key; for HTTP ingest the secret_key becomes the bearer token), optional watch_dir (rejected on mongo and http_ingest) |
data.unregister_bucket | Deregister a previously registered bucket, removing it from the daemon's active configuration (its watch poller stops). The underlying storage is not deleted. | bucket_id (bucket name to remove) |
Each operation that omits bucket uses the daemon's default bucket (configured below), so most playbooks only name a bucket when they need a non-default one. Two operations stand slightly apart: data.transform never touches storage — it takes data in one format and hands it back in another, which you then pass to the next step (often a data.upload) — and data.parse can likewise work on inline content_b64 bytes without a stored object. data.move and data.copy are the opposite: pure storage plumbing — a get from the source and a put to the destination through the bucket registry, with move deleting the source afterwards — which is exactly what lets any backend pair compose. And because a move must never quietly become a duplicate, a source delete that fails after a successful copy raises a loud Fact.Data.OperationFailed instead of passing silently.
What
data.transformis — and is not. It is a format converter driven byformat_fromandformat_toacross JSON, CSV, and XLSX. It does not take atransform: { kind: … }block, and there is nojson_decodeorsplit_segmentstep — decoding and field-level reshaping belong in your playbook's own templating, not in this operation. Useformat_from/format_toas shown in the examples below.
One convention ties the whole family together: every file-carrying fact exposes a flat bucket + key pair (plus sha256 and content_type where known), and every data.* operation accepts exactly those fields. A mail attachment offloaded to storage, a file discovered in a watched folder, the destination echoed by a move — each is a reference the next step can use verbatim: bucket: ${trigger.bucket}, key: ${trigger.key}. That is the platform's file bus: facts carry the reference, verbs accept it, and files flow between daemons without custom glue.
Static buckets — the backends an operator wants always available — are defined once in the daemon's application.toml. Dynamic registration via data.register_bucket is for cases where a playbook needs to reach a backend that was not known at deploy time: a per-customer SFTP drop, a tenant's own S3 bucket, a document-ingest endpoint, or any backend that changes at runtime. Dynamic bindings live in memory only — register them from a provisioning playbook on Fact.System.Boot so a restart heals the registry, and remember that registering binds a backend; it does not create the physical bucket. (A MongoDB bucket is the eager exception: it connects and pings its server at registration, so a bad DSN fails the registration step rather than your first upload.) Credentials passed to data.register_bucket should be passed by reference from a secret rather than written inline. See Secrets & credentials and Storage & data transfer for how operators set up both static and dynamic buckets.
A scheduled export queries the day's metrics and writes them to storage under a dated key. Because no bucket is given, the file lands in the default bucket, and the upload emits Fact.Data.Transported so anything watching can react to the new file:
name: publish-daily-report
trigger:
event: Fact.Schedule.Fired
filter: { name.eq: daily-report }
steps:
- id: report
run: database.query
with:
table: daily_metrics
limit: 100
- run: data.upload
with:
key: "reports/${trigger.received_at}/metrics.json"
content_type: application/json
body_json: "${steps.report.rows}"The rows travel as body_json and are stored verbatim as a JSON document under a predictable key. To upload a binary file instead — a PDF or an image generated upstream — you would pass it as body_b64 (base64-encoded bytes) and set the matching content_type; plain text goes in body.
The two-step pattern that makes the format converter useful: a weekly query result is turned into an XLSX workbook with data.transform, then the result is written to storage with data.upload. The conversion never touches storage on its own — it hands the bytes to the upload step, which does:
name: export-orders-as-xlsx
trigger:
event: Fact.Schedule.Fired
filter: { name.eq: weekly-orders-export }
steps:
- id: orders
run: database.query
with:
table: orders
limit: 500
- id: workbook
run: data.transform
with:
format_from: json
format_to: xlsx
body: "${steps.orders.rows}"
- run: data.upload
with:
key: "exports/orders-${trigger.received_at}.xlsx"
content_type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
body_b64: "${steps.workbook.body_b64}"Because XLSX is a binary format, the converted workbook comes back as base64 bytes and is uploaded with body_b64. The same operation runs in the other direction too — set format_from: xlsx (or csv) and format_to: json to read a spreadsheet a supplier sent you into structured data your playbook can work with.
When you need to act on whatever arrived most recently, ask the listing to sort for you. sort and order compose with the flat first{} that the Fact.Data.Listed result carries: “the newest file” is simply the first entry of a listing sorted by mtime, descending. Here a scheduled scan fetches exactly that file from a supplier's SFTP bucket:
name: process-newest-drop
trigger:
event: Fact.Schedule.Fired
filter: { name.eq: supplier-drop-scan }
steps:
- id: drop
run: data.list_objects
with:
bucket: supplier-sftp
prefix: "incoming/"
sort: mtime
order: desc
- id: file
run: data.download
with:
bucket: supplier-sftp
key: "${steps.drop.first.key}"
as_b64: trueThe download step reads ${steps.drop.first.key} (in the immediately following step, ${prev.first.key} works too) — no loop, no glue code. The ordering is honest even when a backend cannot report a timestamp for every entry: objects without one always sort last, so they never masquerade as “newest”. The downloaded bytes are now available to subsequent steps — for instance handing the contents to aiinjector-service to extract figures, or to data.transform to reshape them.
When the target storage endpoint is not known at deploy time — for example a per-customer SFTP directory whose connection details ride in the schedule's payload — a playbook can register the bucket itself, use it, and deregister it when done:
name: deliver-to-customer-sftp
trigger:
event: Fact.Schedule.Fired
filter: { name.eq: nightly-delivery }
steps:
- id: reg
run: data.register_bucket
with:
bucket_id: "customer-sftp-${trigger.payload.customer_id}"
backend: sftp
host: "${trigger.payload.sftp_host}"
port: 22
root: "/drop"
credentials:
access_key: "${trigger.payload.sftp_user}"
secret_key: ${secret.CUSTOMER_SFTP_PASSWORD}
- id: report
run: database.query
with:
table: deliveries
where:
customer_id: ${trigger.payload.customer_id}
limit: 1000
- run: data.upload
with:
bucket: "customer-sftp-${trigger.payload.customer_id}"
key: "report-${trigger.received_at}.json"
content_type: application/json
body_json: "${steps.report.rows}"
- run: data.unregister_bucket
with:
bucket_id: "customer-sftp-${trigger.payload.customer_id}"Deregistering the bucket when the delivery is done keeps the daemon's active configuration clean. The underlying remote directory is not affected — only the daemon's reference to it is removed.
data.move turns “file this somewhere else” into one step. Here, every .docx attachment that mailbox-service has offloaded into storage (the offload announces itself as Fact.Data.Transported with a key under mail/) is moved into an archive bucket — which may live on a completely different backend, because the move is a get→put through the bucket registry:
name: mail-docx-to-archive
trigger:
event: Fact.Data.Transported
filter:
key.startswith: "mail/"
key.endswith: ".docx"
steps:
- id: archive
run: data.move
with:
from_bucket: ${trigger.bucket}
from_key: ${trigger.key}
to_bucket: edi-archiveWith to_key left out, the object keeps its key in the new bucket. The move deletes the source only after the copy has succeeded — and if that delete fails, the daemon raises a loud Fact.Data.OperationFailed rather than leaving a silent duplicate behind. The same shape drains a watched directory into a processed/ prefix, or ships anything a fact points at to wherever it belongs next.
data.copy is the same plumbing with the source left in place — and combined with the MongoDB GridFS and HTTP ingest backends it turns the daemon into a small file bus. A provisioning playbook registers a watched local inbox and two sinks; a business playbook fans every discovered markdown note out to both, in parallel:
name: register-notes-file-bus
trigger:
event: Fact.System.Boot
filter:
component.eq: playbook-service
steps:
- id: inbox
run: data.register_bucket
with:
bucket_id: notes-inbox
backend: local_fs
root: /var/lib/binions/datatransporter
watch_dir: notes-incoming
- id: rag
run: data.register_bucket
with:
bucket_id: rag-ingest
backend: http_ingest
endpoint: "http://127.0.0.1:8899/api/ingest"
credentials:
access_key: bearer
secret_key: ${secret.RAG_INGEST_TOKEN}
- id: mongo_archive
run: data.register_bucket
with:
bucket_id: docs-archive
backend: mongo
dsn: ${secret.MONGO_APP_DSN}
database: binions
---
name: markdown-to-rag-and-mongo
trigger:
event: Fact.Data.FileDiscovered
filter:
bucket_id.eq: notes-inbox
key.endswith: ".md"
steps:
- parallel:
- run: data.copy
with:
from_bucket: ${trigger.bucket_id}
from_key: ${trigger.key}
to_bucket: rag-ingest
- run: data.copy
with:
from_bucket: ${trigger.bucket_id}
from_key: ${trigger.key}
to_bucket: docs-archiveThe HTTP ingest bucket needs nothing but an endpoint and (optionally) a bearer token — each copy into it becomes a multipart POST the receiving API can accept as-is. The GridFS bucket archives the same bytes inside MongoDB, overwriting any earlier revision of the key. Note that the discovery fact names its bucket as bucket_id; data.copy takes it straight into from_bucket.
When a person or an external system needs a stored file, data.presign mints a download URL that expires on its own — no credentials shipped, no bytes attached to a mail. Here every object landing under reports/ becomes a 24-hour link mailed to operations:
name: report-share-link
trigger:
event: Fact.Data.Transported
filter:
key.startswith: "reports/"
steps:
- id: link
run: data.presign
with:
bucket: ${trigger.bucket}
key: ${trigger.key}
expires_secs: 86400
- id: notify
run: mail.send
with:
from_alias: ops-mail
to: ["ops@example.com"]
subject: "Report ready: ${trigger.key}"
body_text: |
The nightly report is ready. Download (valid until ${steps.link.expires_at}):
${steps.link.url}Links default to one hour and can live at most seven days — the ceiling of the S3 signature scheme — and the Fact.Data.Presigned fact carries both the url and the exact expires_at, ready for the message body. Pre-signing is an S3/MinIO capability: asking for a link on any other backend fails loudly with unsupported rather than producing a URL that would not work.
data.parse closes the classic mail-to-data pipeline entirely inside the platform: mailbox-service offloads the attachment to storage, data.parse reads its text layer, aiinjector-service turns the text into typed fields, and database-service stores the result — no external parsing service in the loop:
name: invoice-pdf-to-sql
trigger:
event: Fact.Data.Transported
filter:
key.startswith: "mail/"
key.endswith: ".pdf"
steps:
- id: parse
run: data.parse
with:
bucket: ${trigger.bucket}
key: ${trigger.key}
max_text_bytes: 262144
- id: extract
run: ai.extract
with:
text: ${steps.parse.text}
providers: [primary, local]
max_cost_usd: 0.05
on_mismatch: fail
fields:
- { name: no, type: text }
- { name: total, type: decimal }
- { name: sender, type: text }
- id: persist
run: database.write
with:
table: invoices
row:
no: ${steps.extract.result.no}
total: ${steps.extract.result.total}
sender: ${steps.extract.result.sender}Format detection is automatic: the key's extension first, then magic bytes (%PDF-, or the zip signature that opens a DOCX), then plain text; you can also force format: pdf, docx, or text. PDF parsing reads the embedded text layer only — there is no OCR, so a pure scan fails loudly with reason no_text_layer instead of feeding garbage to the extraction step; route scans to an external OCR service with webhook.send (expect_json: true) and carry on with its response_json. DOCX text comes from the document's XML text runs, and anything already textual passes through as UTF-8. Output longer than max_text_bytes (1 MiB unless you set it) is cut on a character boundary and flagged truncated: true, and a malformed document produces a clean parse_error failure fact rather than a poisoned queue.
datatransporter-service is a multi-backend broker: one set of operations works across six different storage systems, and a bucket's backend selects which one it uses. Only the bucket changes between them, never the playbook steps.
| Backend | What it is for |
|---|---|
| S3 / MinIO (default) | S3-compatible object storage — the natural home for reports, exports, and backups, and the only backend that can mint pre-signed download links. MinIO is the object store bundled with the platform; see Shared infrastructure. |
| Local filesystem | A directory on the host. Ideal for files that stay on the machine, or for a simple setup with no separate object store. |
| SFTP | A remote SFTP server — for exchanging files with partners and suppliers over SSH. |
| FTP | A remote FTP server — for integrating with systems that still expect classic FTP delivery. |
| MongoDB GridFS | Files stored inside a MongoDB database via GridFS — useful when Mongo is already your system of record and you want documents next to your data. The bucket name is the GridFS bucket; re-uploading a key replaces the file (earlier revisions are dropped), so a key always names exactly one current file. The connection is opened and pinged eagerly at registration, so a bad DSN fails there and not on first use. |
| HTTP ingest | A write-only sink: every upload or copy into it becomes a multipart/form-data POST to the configured endpoint — the file part is named file with the key as its filename, alongside key and sha256 fields — optionally authenticated with a bearer token. That is the generic shape document-ingest APIs (a RAG indexer, for example) expect. Reads, listings, and deletes fail loudly as unsupported. |
S3/MinIO and the local filesystem are available out of the box; the other backends are enabled per bucket — statically in application.toml or at runtime via data.register_bucket (in-memory — re-registered on boot by a provisioning playbook). Whichever a bucket uses, the playbook-facing operations are the same, with two honest exceptions the daemon enforces loudly rather than papering over: pre-signed links exist only on S3/MinIO, and an HTTP ingest bucket accepts writes only. MongoDB GridFS and HTTP ingest buckets also cannot carry a watch_dir — registration rejects it.
Beyond on-demand transfers, the daemon can act as a drop point: a registered S3/MinIO, SFTP, FTP, or local-filesystem bucket can carry a watch_dir, and the daemon polls it (every 30 seconds by default, tunable via the [watch] config block) and announces each new file as a Fact.Data.FileDiscovered event (bucket_id, watch_dir, key, size, discovered_at) your playbooks trigger on, exactly like the inbound-message pattern used elsewhere on the platform. The first listing after a registration announces the existing backlog, so files that arrived while the daemon was down are not missed. A discovered file usually flows straight into data.move, data.copy, or data.parse — the fact's bucket and key are exactly the reference those operations accept, as the markdown fan-out example above shows.
Configuration lives in the daemon's application.toml. The [redis] block points at the dedicated Redis instance that holds the daemon's state; [healthcheck] exposes the local health and metrics server; and the [minio] block holds the connection details for the default object store, including the default bucket used when an operation does not name one. An optional [reactor] block tunes how large a payload may be before it is streamed rather than handled inline, and an optional [otel] block enables distributed tracing. Credentials are always read from a file, never written inline.
# /opt/binions/datatransporter-service/config/application.toml
[redis]
port = 6395
password_file = "/etc/binions/secrets/datatransporter-redis.pass"
[healthcheck]
listen_addr = "127.0.0.1:9105"
# Default S3-compatible object store (MinIO is bundled)
[minio]
endpoint = "http://127.0.0.1:9000"
region = "us-east-1"
credentials_file = "/etc/binions/secrets/minio.creds"
default_bucket = "binions-data-transport"
force_path_style = true
# Optional: threshold above which a payload is streamed rather than
# held inline in memory (roughly 32 KiB)
# [reactor]
# inline_max_bytes = 32768
# Optional: send traces to a collector for end-to-end visibility
# [otel]
# endpoint = "http://127.0.0.1:4317"With force_path_style set, the daemon addresses buckets by path — the form MinIO and most self-hosted S3 stores expect. The default_bucket is what data.upload, data.download, and the other operations use whenever a playbook leaves bucket unset. As with every daemon, secrets are referenced by file — see Secrets & credentials for how the credential files are laid out.
datatransporter-service emits a family of Fact.Data.* events — one per operation, plus a discovery event for watched folders and a loud failure fact — and consumes data-producing events from other daemons so it can act as the final, storing step of a pipeline.
| Direction | Event | Meaning |
|---|---|---|
| Emits | Fact.Data.Transported | An upload completed. Carries a reference to the stored object (flat bucket + key); this is the fact later steps and other playbooks key off — a mail attachment offloaded to storage announces itself this way too. |
| Emits | Fact.Data.Downloaded | An object was fetched from storage. |
| Emits | Fact.Data.Listed | A bucket was listed — optionally under a prefix and in the requested order — with the first entry mirrored as a flat first{} for easy interpolation. |
| Emits | Fact.Data.Deleted | An object was removed from a bucket. |
| Emits | Fact.Data.Moved / Fact.Data.Copied | An object was moved (source deleted) or copied (source kept) — with from_bucket, from_key, to_bucket, to_key, bytes, sha256, the destination uri, and the content_type where known. |
| Emits | Fact.Data.Presigned | A time-limited download link was minted — carries url and expires_at. |
| Emits | Fact.Data.Parsed | A document's text was extracted — carries format, text, text_bytes, truncated, and the source it came from. |
| Emits | Fact.Data.Transformed | A payload was converted from one format to another. |
| Emits | Fact.Data.BucketRegistered / Fact.Data.BucketUnregistered | A storage backend was registered or deregistered at runtime. |
| Emits | Fact.Data.FileDiscovered | A new file appeared in a watched folder — the drop-point trigger. |
| Emits | Fact.Data.OperationFailed | An operation failed, with a concrete reason — unsupported for a capability the backend does not have, no_text_layer for a scanned PDF, parse_error for a malformed document, or a source delete that failed after a successful copy. Loud by design, and a perfectly good trigger for an alerting playbook. |
| Consumes | Fact.Database.QueryResult, Fact.AI.Generated | Data produced by other daemons — a query result, an AI-generated payload — which a playbook can route into a transform or an upload. |
Facts from this daemon are also things playbooks trigger on: the archive and share-link examples above key off Fact.Data.Transported, the markdown fan-out keys off Fact.Data.FileDiscovered, and downstream automation can just as well react to Fact.Data.Moved or Fact.Data.Copied. Every event rides in the platform's standard envelope with a correlation id, so a discovery, the parse that read it, the extraction that typed it, and the write that stored it all share one thread you can follow end to end. For the full anatomy of that envelope, see The event envelope.
The daemon runs as binions-datatransporter.service under a dedicated, unprivileged datatransportersvc user, alongside its own redis-binions-datatransporter.service which holds its state. Bring both up together:
sudo systemctl enable --now redis-binions-datatransporter binions-datatransporter
systemctl status binions-datatransporterThe service is a Type=notify unit with a watchdog (WatchdogSec=30): it must report liveness within its watchdog window or systemd restarts it, so a wedged transfer heals itself. Check health directly over the local endpoint:
curl -s http://127.0.0.1:9105/health/ready
curl -s http://127.0.0.1:9105/health/live
curl -s http://127.0.0.1:9105/metricsConfirm the object store is reachable from the host and that the credentials in the configured file are current, then check that the bucket exists and the key is spelled as your playbook expects. For SFTP or FTP buckets, verify the remote server is reachable and its credentials are valid. For buckets registered at runtime with data.register_bucket, confirm that the credentials file and connection details passed in the with: block are correct and accessible at the time the step runs. The /health/ready endpoint reports whether the daemon considers its storage connection healthy.
A few failures are by design and tell you exactly what to change. A reason of unsupported means you asked a backend for something it cannot do — a pre-signed link outside S3/MinIO, or a read from a write-only HTTP ingest bucket. A data.parse failure with no_text_layer means the PDF is a scan — there is no OCR on the platform, so route it to an external parser via webhookcaller-service and continue from its JSON reply. And a failure after a data.move means the copy landed but the source delete did not — the file exists in both places until you remove the original.