Observability & Ops

What this covers / who it's for. The operational surfaces of the running system: Horizon and its queue topology, the complete scheduled-work table, failed-job handling, the retention/prune regime, log correlation, and the audit-chain duties. For which async platform to build on, see Async platforms; for the product-facing retention commitments, see Data lifecycle guarantees.

Horizon & the queue topology

Horizon supervises the Redis (Valkey) queues. The dashboard lives at /horizon (HORIZON_PATH), registered by the backoffice module's Modules\Backoffice\Providers\HorizonServiceProvider; its viewHorizon gate ships with an empty email allow-list, so in non-local environments nobody can view it until operators are added (local is open via Horizon's default). Supervisors, from config/horizon.php:

Supervisor	Queue	tries	timeout	Why it exists
supervisor-1	default	1	60s	Everything not listed below
supervisor-webhooks	webhooks	5	120s	Inbound webhook processing — spatie 200-acks before ProcessInboundWebhookJob runs, so transient failures must be retried, not lost (D58)
supervisor-documents	documents	5	120s	ForwardDocumentToProviderJob — store-before-forward to the rail; retries before the failed() alert (D43/D58)
supervisor-audit	audit	5	120s	PersistAuditEventJob — the async success audit path; a dropped append leaves no chain gap, so retry first (D58)
supervisor-dlp	dlp	5	120s	PersistDataAccessLogJob — classified-read access logging (D44/D57)
supervisor-tasks	tasks	1	300s	Task-runner fleet sweeps; the deliberately low worker count is the DB-load throttle for cross-tenant batches; failures are contained in the task ledger, not queue-retried (D93)

The non-consumed-queue trap. A Redis queue with no consumer doesn't error — jobs just silently accumulate and the function it serves stalls: no success audit rows, no access logs, no document forwards. Every specialized supervisor must appear in every Horizon environment block, and locally the docker worker service runs plain queue:work on default only — the specialized queues are consumed locally only under make artisan ARGS=horizon. LongWaitDetected thresholds (horizon.waits) are 60s per queue, except tasks at 900s — sweep waits are expected; only a genuinely stuck sweep fires.

Scheduled work — the complete table

Every entry, registered in the owning module's ServiceProvider boot() and driven by the scheduler container (schedule:work). All are onOneServer()/withoutOverlapping() where the comment trail demands it; gates are config keys checked at registration time.

Command	Cadence	What it does	Module
compliance:scan-sla-breaches	every minute	Escalates every SLA-breached, non-closed compliance case (gated compliance.schedule_sla_scan, default on)	compliance
treasury:warm-rates	every 15 seconds	Pre-warms the FX rate cache ahead of its 15s TTL (freshness is TTL-bound; this keeps the read path warm)	treasury
dlp:evaluate-rules	every 15 minutes	Evaluates DLP tripwire rules over classified-access logs, opening incidents	dlp
dlp:evaluate-egress-rules	every 15 minutes	Evaluates egress rules over download/export events	dlp
mirrors:reconcile-stale	hourly	Re-fetches provider mirrors that have gone stale	provider-mirror
audit:verify-chain --checkpoint	hourly	Recomputes every audit partition's hash chain, detects tamper, anchors a checkpoint (gated audit.schedule_verification, default on)	audit
documents:redrive-forward-failed	hourly	Redrives documents stranded by transient forward failures (D2/D59)	documents
features:roll-periods	hourly	Rolls usage-metering periods for entitlement quotas	features
tasks:detect-stuck	hourly	Flags task-run units stuck in running past stuck_after_minutes	task-runner
documents:verify-integrity	daily	Integrity sweep of the encrypted document vault	documents
documents:prune-accesses	daily	Class-aware access-trail retention prune (D78) — safe-by-default: prunes nothing until a signed-off window is configured	documents
foundation:prune-command-dedup-records	daily	Prunes expired bus idempotency/dedup records	foundation
notifications:prune-log	daily	Class-aware notification-log retention prune (D77)	notification
rails:prune-vendor-logs	daily	Prunes scrubbed vendor request/response logs	rails
model:prune (InboundWebhookCall)	daily	Prunes stored inbound webhook call rows	webhooks
mirrors:prune-drift	daily	Prunes resolved mirror-drift rows	provider-mirror
audit:export-worm	daily	Exports closed audit chain segments to the WORM object-storage disk (gated audit.worm_export.enabled, default off)	audit
audit:verify-worm	daily	Reconciles the WORM archive against the DB chain — coverage gaps, HMACs, closing hashes (same gate, default off)	audit
async-requests:prune	daily 03:30	Prunes terminal async-request rows past their retention window	async-requests
tasks:prune	daily 03:45	Prunes old task_runs/task_run_units ledger rows	task-runner
queue:prune-batches --hours=48 --unfinished=168 --cancelled=168	daily 04:00	Prunes Laravel job_batches rows underneath the task ledger	task-runner
payments:generate-monthly-statements	monthly, 1st at 02:00	Generates the previous month's tenant statements	payments

Notably not scheduled: documents:shred-document (RTBF crypto-shred) is manual, per-approved-request only.

Failed work — where to look, how to recover

Surface	Tooling
Failed queue jobs	make queue-failed (list), make queue-retry (retry all), make queue-flush (discard); Horizon retains failed jobs 7 days (horizon.trim)
Task-runner runs/units	tasks:status (inspect), tasks:retry (re-dispatch failed units), tasks:cancel; tasks:detect-stuck runs hourly; backoffice read surface at GET /api/v1/backoffice/task-runs
Stranded document forwards	documents:redrive-forward-failed (also scheduled hourly); exhausted retries raise a visible alert via the job's failed() handler
Inbound webhooks	The webhooks queue retries (tries 5); the stored InboundWebhookCall row is the replay source

Retention & pruning — the regime at a glance

The prune commands in the schedule table above are the whole regime: async-requests, task ledger + job batches, notification log (D77), document access trail (D78, inert until a window is signed off), vendor logs, webhook calls, mirror drift, bus dedup records. Two deliberate exceptions: the audit chain is append-only and never pruned (growth is handled by the config-gated WORM offload), and documents are never auto-deleted (RTBF is the manual crypto-shred). The product-facing view of these commitments — what users and regulators are promised — is Data lifecycle guarantees.

Log correlation (D74)

Modules\Foundation\Http\Middleware\AssignCorrelationId establishes three ids per request and shares them via Laravel Context, so they dehydrate into queued jobs/listeners and appear on every log line (via CorrelationProcessor; ShareActorContext adds the actor's public uuid + tenant):

• request_id — always server-minted (UUIDv7), unique to this hop; inbound values are never trusted.
• correlation_id — stable across the flow: adopted from W3C traceparent, else a sanitized X-Correlation-Id, else seeded from request_id.
• trace_id — set when a valid traceparent is presented.

Both ids are echoed as response headers and surface in the JSON:API meta block, so a support report can be joined to logs, the CQRS audit trail, and traces on one id.

Audit-chain duties

Ops owns the tamper-evidence loop: the hourly audit:verify-chain --checkpoint recomputes and anchors the chain (a failure is a tamper signal, not noise), and make audit-verify-chain runs it on demand. The WORM export/verify pair (make audit-export-worm / make audit-verify-worm) is config-gated off today — production enables it deliberately against an Object-Lock bucket. Until then there is no off-database copy of the chain; enabling it is a deploy decision, not a code change. Deep dive: the audit-trail skill and app-modules/audit/README.md.

---

← Engineering wiki index