Dependencies & failure modes
What this covers / who it's for. The platform's runtime dependencies, what each one powers, the user-visible symptom when it is slow or down, and the first place to look. This is the triage map: match the symptom, find the dependency, check the named surface. Deep operational detail lives in Observability & ops.
The dependency map
| Dependency | What it powers | User-visible symptom when slow / down | First thing to check |
|---|---|---|---|
| Conduit API (outbound) | Every provider-executed action: onboarding/KYC submission, wallet + virtual-account creation, payouts, orders, the reconciliation sweep (Providers & rails) | Reads keep working — pages serve from our local mirrors. Writes fail with neutral errors after up to the 30s provider timeout; each failed attempt lands in the vendor request log. Reconciliation stops healing | Conduit status; the vendor request log (per-attempt outcomes, newest rows) |
| Conduit webhooks (inbound) | Status truth arriving: KYB progression, payout/order/deposit status, co-sign confirmation | Statuses freeze at last-known values while the provider moves on. Actions still submit fine. The hourly reconciliation sweep heals coarse statuses; webhook-only sub-states wait for the provider's retry ladder | Is the webhooks queue draining (Horizon)? Then the inbound webhook-call log |
| Sumsub API + webhooks (onboarding, GB/EU only) | KYB/KYC verification and the beneficial-owner graph for GB + EU tenants (Providers & rails) — code-live but pre-product | Scoped to GB/EU onboarding. Down/slow: GB/EU verification submissions fail (writes) while onboarding reads keep serving from mirrors; verification decisions arrive by webhook, so a webhook backlog freezes GB/EU KYB/KYC progression. ZA/NG/KE/US onboarding (on Conduit) and all money flows are unaffected | Is this a GB/EU tenant? Then the webhooks queue + the vendor request log (Sumsub attempts) |
| Utila API (custody — routed, pre-product) | Crypto custody wallet/transaction writes for all six jurisdictions (Providers & rails); custody now routes to Utila (D98) with Conduit retained only as an override-only escape hatch, not an automatic fallback | No live impact today — custody is routed to Utila in code (D98) but deploy-gated + pre-product, so no real custody traffic flows yet. Once live, an outage would fail custody wallet/transaction writes while reads serve from mirrors | Confirm whether the custody go-live deploy steps (Utila vaults + secrets) have landed; then the vendor request log |
| Fingerprint / SEON (fraud / device-risk — routed, pre-product) | Device-risk screening on login, sign-up, and payout for all six jurisdictions (Providers & rails); Fingerprint primary + SEON permitted (D102), deploy-gated | No live impact today — screening ships default-disabled, so no vendor call fires. Once enabled, a screening outage is asymmetric by design: login / sign-up fail open (proceed, no verdict recorded), while a payout fails closed to step-up and durably records a never-silent skipped-screening row (money never slides through unscreened). A deny blocks the action and opens a fraud compliance case. An inbound webhook that can't be verified or mapped is dropped-and-acked (bounded), never parked; webhook replay protection is now durable for the assessment-retention window, not the 30-day webhook prune. One optional lever: the default-off fraud.hint_reuse_enabled — when an operator turns it ON, a login / sign-up (never a payout, never a cached deny) may reuse a same-tenant cached low-risk verdict and skip the fresh vendor call, a fail-open trade that trusts a same-tenant client device claim | Is the fraud.device-risk feature enabled at all? Then the vendor request log (Fingerprint/SEON attempts) + the webhooks queue |
| Queue workers / Horizon | All background work, per queue — see the queue table below | Depends entirely on which queue — a queue with no consumer silently stalls just its function, everything else looks healthy | Horizon dashboard: per-queue wait times + failed jobs list |
| PostgreSQL | Everything — all domain data, all requests | Total outage: the platform is down. Degraded: everything is uniformly slow | Database health; this is the one dependency with no partial mode |
| Object storage (S3 / MinIO) | Statement + export artifacts; the disabled-by-default surfaces (document direct transfer, audit WORM export) when enabled | Export/statement downloads fail after a successful poll (succeeded but the link errors). Document upload/download unaffected today (bytes proxy through the app on the local-disk store) | Storage reachability; then the failed-jobs list for artifact writers |
| Mail transport (SMTP/SES/…) | The email leg of every notification | Inbox notifications keep appearing (they are DB rows); emails silently stop arriving — queued mail jobs retry/fail in the background. No domain action is ever blocked | Failed jobs on the default queue; the mail provider's own status |
| Valkey / Redis | Cache, all queues, Horizon state, the FX rate cache (15s TTL behind the ≤20s freshness guarantee) | Down: every queue stops (all async symptoms at once) and rate lookups can't serve — treasury/FX views degrade. Flushed: queued-but-unprocessed jobs are lost; caches rebuild | Valkey health; then Horizon (it runs on the same store) |
| The scheduler | Every recurring job: reconciliation, SLA scans, retention prunes, rate warming, chain verification, monthly statements, forward re-drives, stuck-task detection | Nothing fails loudly — the platform stops self-healing: mirrors go stale past an hour, SLA breaches stop escalating, rates go stale (>20s), prunes stop, statements don't generate on the 1st | Is the scheduler process running? Last-run times of scheduled commands |
The queues (one function each — a stalled queue stalls exactly that)
Six queues, each with its own worker pool; a queue nobody consumes stalls its function
silently while the rest of the platform looks healthy (config/horizon.php — every queue must
have a supervisor):
| Queue | What stalls if it stops | How it surfaces |
|---|---|---|
default | Email sending, general async work | Emails stop; inbox unaffected |
webhooks | Provider status ingestion | Statuses freeze (Providers & rails) |
documents | Document forwarding to the rail (and scanning, when enabled) | Uploads succeed but never reach the provider; KYB stalls provider-side |
audit | Asynchronous audit-event persistence | Audit trail lags actions (writes are queued post-commit) |
dlp | Sensitive-read / egress logging pipeline | DLP trail lags; incidents open late |
tasks | All fleet work: reconciliation units, monthly statements, forward re-drives | Sweeps start but never finish; task ledger shows stuck runs (flagged after 120 min) |
Reading a symptom backwards
- "A payout has said pending for an hour" → webhooks first (queue draining?), then the hourly reconciliation (did it heal others?), then Conduit itself (the vendor request log).
- "Export succeeded but the download fails" → object storage, then the link's short TTL (re-poll for a fresh link — Async by design).
- "Customers say no emails arrive" → mail transport /
defaultqueue; confirm inbox rows exist (if they do, the action layer is healthy and it is purely a delivery problem — Communications delivery). - "Everything async is stuck at once" → Valkey or the worker fleet, not any single feature.
- "Things drift and nothing corrects itself" → the scheduler.
What degrades gracefully vs what doesn't
Built to degrade gracefully: provider outage (reads from mirrors, explicit write failures), mail outage (actions unaffected), single-queue stalls (isolated function), object-storage outage (only artifact bytes). Not graceful: PostgreSQL (everything), Valkey (all async at once). The scheduler is the quiet one — nothing errors, entropy just accumulates; the every-minute SLA scan and 15-second rate warming are the fastest-decaying casualties.