Dependencies & failure modes
What this covers / who it's for. The platform's runtime dependencies, what each one powers, the user-visible symptom when it is slow or down, and the first place to look. This is the triage map: match the symptom, find the dependency, check the named surface. Deep operational detail lives in Observability & ops.
The dependency map
| Dependency | What it powers | User-visible symptom when slow / down | First thing to check |
|---|---|---|---|
| Conduit API (outbound) | Every provider-executed action: onboarding/KYC submission, wallet + virtual-account creation, payouts, orders, the reconciliation sweep (Providers & rails) | Reads keep working — pages serve from our local mirrors. Writes fail with neutral errors after up to the 30s provider timeout; each failed attempt lands in the vendor request log. Reconciliation stops healing | Conduit status; the vendor request log (per-attempt outcomes, newest rows) |
| Conduit webhooks (inbound) | Status truth arriving: KYB progression, payout/order/deposit status, co-sign confirmation | Statuses freeze at last-known values while the provider moves on. Actions still submit fine. The hourly reconciliation sweep heals coarse statuses; webhook-only sub-states wait for the provider's retry ladder | Is the webhooks queue draining (Horizon)? Then the inbound webhook-call log |
| Queue workers / Horizon | All background work, per queue — see the queue table below | Depends entirely on which queue — a queue with no consumer silently stalls just its function, everything else looks healthy | Horizon dashboard: per-queue wait times + failed jobs list |
| PostgreSQL | Everything — all domain data, all requests | Total outage: the platform is down. Degraded: everything is uniformly slow | Database health; this is the one dependency with no partial mode |
| Object storage (S3 / MinIO) | Statement + export artifacts; the disabled-by-default surfaces (document direct transfer, audit WORM export) when enabled | Export/statement downloads fail after a successful poll (succeeded but the link errors). Document upload/download unaffected today (bytes proxy through the app on the local-disk store) | Storage reachability; then the failed-jobs list for artifact writers |
| Mail transport (SMTP/SES/…) | The email leg of every notification | Inbox notifications keep appearing (they are DB rows); emails silently stop arriving — queued mail jobs retry/fail in the background. No domain action is ever blocked | Failed jobs on the default queue; the mail provider's own status |
| Valkey / Redis | Cache, all queues, Horizon state, the FX rate cache (15s TTL behind the ≤20s freshness guarantee) | Down: every queue stops (all async symptoms at once) and rate lookups can't serve — treasury/FX views degrade. Flushed: queued-but-unprocessed jobs are lost; caches rebuild | Valkey health; then Horizon (it runs on the same store) |
| The scheduler | Every recurring job: reconciliation, SLA scans, retention prunes, rate warming, chain verification, monthly statements, forward re-drives, stuck-task detection | Nothing fails loudly — the platform stops self-healing: mirrors go stale past an hour, SLA breaches stop escalating, rates go stale (>20s), prunes stop, statements don't generate on the 1st | Is the scheduler process running? Last-run times of scheduled commands |
The queues (one function each — a stalled queue stalls exactly that)
Six queues, each with its own worker pool; a queue nobody consumes stalls its function
silently while the rest of the platform looks healthy (config/horizon.php — every queue must
have a supervisor):
| Queue | What stalls if it stops | How it surfaces |
|---|---|---|
default | Email sending, general async work | Emails stop; inbox unaffected |
webhooks | Provider status ingestion | Statuses freeze (Providers & rails) |
documents | Document forwarding to the rail (and scanning, when enabled) | Uploads succeed but never reach the provider; KYB stalls provider-side |
audit | Asynchronous audit-event persistence | Audit trail lags actions (writes are queued post-commit) |
dlp | Sensitive-read / egress logging pipeline | DLP trail lags; incidents open late |
tasks | All fleet work: reconciliation units, monthly statements, forward re-drives | Sweeps start but never finish; task ledger shows stuck runs (flagged after 120 min) |
Reading a symptom backwards
- "A payout has said pending for an hour" → webhooks first (queue draining?), then the hourly reconciliation (did it heal others?), then Conduit itself (the vendor request log).
- "Export succeeded but the download fails" → object storage, then the link's short TTL (re-poll for a fresh link — Async by design).
- "Customers say no emails arrive" → mail transport /
defaultqueue; confirm inbox rows exist (if they do, the action layer is healthy and it is purely a delivery problem — Communications delivery). - "Everything async is stuck at once" → Valkey or the worker fleet, not any single feature.
- "Things drift and nothing corrects itself" → the scheduler.
What degrades gracefully vs what doesn't
Built to degrade gracefully: provider outage (reads from mirrors, explicit write failures), mail outage (actions unaffected), single-queue stalls (isolated function), object-storage outage (only artifact bytes). Not graceful: PostgreSQL (everything), Valkey (all async at once). The scheduler is the quiet one — nothing errors, entropy just accumulates; the every-minute SLA scan and 15-second rate warming are the fastest-decaying casualties.