Skip to main content

D93 — Durable async-task execution: the task-runner module — Laravel batches with one job per tenant, tracked on a domain-level run/unit ledger (RESOLVED)

Architecture decision record, relocated verbatim from the retired single-file docs/tenancy/ decision log. Status, thematic clusters, and how to record a new ADR: the decision log index.

Fleet sweeps ran as long serial console loops (payments:generate-monthly-statements, mirrors:reconcile-stale): one process iterates every tenant with tenancy()->initialize/end, per-tenant try/catch, and NO durable record of which tenants failed — retry means "the whole sweep runs again and skips what already worked" and the only telemetry is audit-channel log lines. Laravel already ships the execution mechanics (queues, Bus::batch, queue middleware); what was missing is the DOMAIN ledger (Laravel's job_batches is infra-level: pruned by queue:prune-batches, no per-unit outcome, no operator surface) and a reusable, module-agnostic way to adopt the pattern.

The task-runner module (Modules\TaskRunner) is that framework — infrastructure like the foundation bus, depending on Foundation only (never Tenancy, never a domain module: tenant linkage is an OPAQUE plain bigint carried on unit rows, D53/D54):

  • Fundamental pattern — a batch with one job per UNIT, where a unit is ANY domain key. The per-tenant fleet sweep is the flagship shape, not the only one: the same framework runs a batch over documents/files WITHIN one tenant (every unit carries the same tenantId), over central/internal work items (reports, model refreshes, dataset crunching — tenantId: null), or over any other enumerable key set — the framework never interprets unit_key. A consumer implements Integration\Contracts\TaskDefinition (type() dot-key, units(context) iterable of TaskUnit{key, tenantId?} — for a fleet sweep, the Tenant::sweepEligible cursor mapped to units — makeJob(unit, context), options()), registers it in its provider boot (TaskTypeRegistry::register(...) — the static MirrorReconciliationRegistry idiom), and calls TaskRunner::start(definition, context, initiator) (the Integration\Contracts\TaskRunner seam). The runner persists a task_runs row + one task_run_units row per unit, then dispatches a real Bus::batch() (allowFailures(), named after the type, chunked Batch::add() for large fleets) of Batchable per-unit jobs. Per-unit tenancy is EXPLICIT in the job (tenancy()->initialize in try / end() in finally — the RollTenantFeaturePeriods primitive-key idiom), because batches are built in the central context so QueueTenancyBootstrapper has nothing to serialize.
  • Data model (CENTRAL tables, D53/D54). task_runs (identifiers, type, status enum running|completed|completed_with_failures|failed|cancelled — creation and dispatch are one motion, so there is no observable run-level "pending", batch_id nullable — the Laravel batch uuid, null for single-job runs, initiator, redaction-safe context jsonb, unit counts, started/finished timestamps, blamable + soft-deletes) and task_run_units (task_run_id FK, unit_key — an OPAQUE identifier (a uuid), never PII: it is echoed by tasks:status and carried on tasks.unit.failed — PLAIN bigint tenant_id nullable NO FK — registered in CentralTableInverseLeakTest — status enum pending|running|completed|failed|skipped, attempts, REDACTED last_error (see the containment bullet), timestamps; the per-run unit-key unique is a WHERE deleted_at IS NULL partial, the repo's SoftDeletes convention). Central because operators/console read runs ACROSS tenants and most runs span the fleet; unit rows are the retry ledger.
  • Durability mechanism — unit rows are the truth; batch callbacks are NOT used. Read against the installed framework (Illuminate\Bus\Batch/PendingBatch/DatabaseBatchRepository): batch then/catch/finally callbacks are SerializableClosures persisted in job_batches.options — they die with the pruned row, catch fires only on the FIRST failure, then never fires once a job failed, and an EMPTY dispatched batch never finishes (nothing ever decrements pending_jobs). So the framework tracks state where it is durable: the TrackTaskUnit job middleware (via the InteractsWithTaskRun trait) marks the unit running/completed/skipped/failed around handle(), and finalization is a re-check after EVERY terminal unit transition (row-lock the run, recount units, close it when none are pending/running) — so a run closes correctly regardless of job_batches pruning, queue:retry re-drives, or worker crashes, and the zero-unit run is closed synchronously at start. task_runs unit counts are a finalization snapshot reconciled FROM unit rows; Laravel's job_batches counters stay infra plumbing (progress introspection only), never a second source of truth.
  • Failure containment (fleet mode) vs queue-native retries (single mode). Fleet-sweep unit jobs run with containTaskUnitFailures = true: the middleware catches the unit's Throwable, records failed + attempts + a REDACTED last_error, report()s it, and DOES NOT rethrow — one tenant's failure never aborts or cancels the batch (the exact semantics of the old per-tenant try/catch; on the sync driver a rethrow would abort the whole dispatch loop). The redaction contract (every ledgered error + the tasks.unit.failed event, which carries the SAME redacted reason): a DOMAIN exception — one implementing the Integration\Contracts\TaskReportableError marker, or any consumer-authored Modules\* exception class — keeps its truncated (500 chars) message; ANY other throwable (vendor/framework — a QueryException carries SQL + bound values, an HTTP exception may carry tokened URLs) is stored class-name-only (unhandled <FQCN> — see logs), the full exception still report()ed to the log stack. Defense in depth over the "author your own operator-safe message" obligation (tracking row 5): kept domain messages additionally pass the Integration\Contracts\ScrubsLedgerText seam before truncation — a passthrough no-op by default (task-runner stays standalone), rebound by the dlp module to its PAN/IBAN/SSN-detector masker (LedgerTextScrubber; dlp → task-runner, the consumer-direction edge), so value-shaped secrets chained into a domain message are masked; non-value-shaped secrets (tokened URLs, SQL fragments) still rely on the authoring obligation. Recovery is tasks:retry or the next scheduled run. A batch-dispatch Throwable AFTER the run committed as running (a broken definition, the queue backend down) is a wedge guard, not a stuck run: the pending units fail with the redacted dispatch reason, the run settles terminally, and the original throwable rethrows to the caller. Tracked SINGLE jobs keep their own queue retry semantics (tries/backoff): the middleware records the attempt + error and rethrows; the trait's failed() hook (or one explicit recordTaskUnitFailure($e) call from a job that overrides failed()) marks the unit failed on TERMINAL failure only (JobFailed/failed() fire once, on final failure — verified in Illuminate\Queue\Jobs\Job::fail).
  • Single tracked jobs adopt with the trait + a contract, no dispatch-site rewrites. A fire-and-forget job that wants a domain ledger implements TracksAsTaskRun (taskType()/taskUnitKey()) + uses InteractsWithTaskRun. The run+unit row is created lazily on FIRST execution and re-resolved on retries by (type, unit_key, non-terminal) — deliberate deviation from "create on dispatch": the queue payload is serialized BEFORE JobQueueing fires (verified in Illuminate\Queue\Queue::enqueueUsing), so dispatch-time stamping would require rewriting every dispatch site; the deterministic-key reattach gives the same ledger with trait-only adoption. Re-attach is dispatch-scoped (tracking row 4): the queue payload uuid — stable across releases/retries of one dispatch, distinct per dispatch, available in the tracked execute AND in failed() — is stamped into the run's context.dispatch_uuid, so retries of ONE dispatch re-attach while an OVERLAPPING same-key dispatch opens its own run+unit (per-dispatch attempt/error fidelity; the per-run (task_run_id, unit_key) partial unique never conflicts across runs; a terminal failure settles only its own dispatch's unit). Trade-off accepted + documented: a job lost before its first attempt has no row. First adopter: documents.forward_to_provider (ForwardDocumentToProviderJob).
  • Throttling — build on Laravel, no bespoke semaphore. Per-type TaskOptions{queue, ratePerMinute}: the provider registers a named limiter task-runner.<type> (RateLimiter::for + Limit::perMinute) for every registered type with a rate, and the trait's middleware() appends Illuminate\Queue\Middleware\RateLimited (an unregistered limiter name is a vendor-guaranteed pass-through). Queue-level concurrency: per-type queue (default tasks) + a dedicated low-worker Horizon supervisor (infra register). The limiter store is the default cache (Valkey) — a Redis-backed dependency, recorded in docs/tracking/infra-requests.md. Throttle releases vs tries — the containment/release distinction. Containment covers a unit whose EXECUTE throws; it does NOT cover a throttled unit, which RateLimited release()s back BEFORE the tracked execute (roughly once per limiter window — availableIn + 3s, vendor) — and a release INCREMENTS the queue job's attempts, so under the tasks supervisor's tries = 1 any fleet larger than ratePerMinute would fail its throttled tail MaxAttemptsExceededException (its failed() settles the unit failed with no work done). Batch units therefore carry a payload retryUntil drain horizon (InteractsWithTaskRun::retryUntil(), dispatch time + task-runner.retry_horizon_hours, default 6h): retryUntil SUPERSEDES maxTries when present (vendor Worker::markJobAsFailedIfAlreadyExceedsMaxAttempts checks it first), so throttled units are held for up to the drain window (~fleet_size / ratePerMinute minutes of ~1 release/min) instead of dying. SAFE with containment: a contained failure is ledgered and never rethrown, so the queue job completes and the horizon can never re-run a genuinely failed unit (a redelivered already-terminal unit is skipped by the UnitStart::Skip gate); the horizon only extends throttle releases and worker-timeout redeliveries (units are idempotent by contract). Tracked SINGLE jobs return a null retryUntil — their own tries/backoff stay in charge.
  • Operator surface (console-first). tasks:status [run-uuid] (recent runs / run detail + failed units), tasks:retry <run-uuid> [--failed-only] (re-dispatches failed(+pending) units as a NEW batch linked to the SAME run — attempts accumulate; refuses unregistered types, i.e. single-job runs, whose retry belongs to their own queue semantics), tasks:cancel <run-uuid> (cancels the Laravel batch + the run; in-flight units mark skipped), tasks:prune (task-runner.retention_days, terminal runs only) — and the module schedules queue:prune-batches daily (it was scheduled NOWHERE before; the infra rows must not outlive the domain ledger's usefulness window). The D93 deferrals (serial-sweep conversions, the backoffice HTTP read surface, stuck-running unit detection, per-dispatch single-job separation, the ledgered-error scrub seam) are all resolved — evidence per row in docs/tracking/task-runner.md.
  • Events (strictly typed, catalogued): tasks.run.started, tasks.run.completed, tasks.run.failed (terminal with failures/cancelled), tasks.unit.failed — AbstractDomainEvent subclasses carrying primitives, in docs/EVENTS.md via events:catalog. A per-unit COMPLETED event was considered and dropped (per-tenant noise with no listener use case; the run-level events carry the counts).
  • Module, not a packages/stables package (evaluated honestly). A Composer package would be reusable across apps and enforce domain-agnosticism by physical separation — but it would sit OUTSIDE the arch guardrails (ModuleBoundaryTest/CqrsBoundaryTest/StrictTypes/ModelConventions and the RLS schema tests that keep its central tables honest), duplicate the Foundation conventions (identifiers/blamable macros, DomainEvent base) it should reuse, and this repo has exactly one consumer app today. The module keeps the guardrails and the conventions; the seam surface (Integration\Contracts + Integration\Data + enums) is already the extraction boundary if a second app ever needs it — the recorded path is "module now, package extraction when a second consumer exists".
  • Relationship to async-requests (sibling, not overlap): async_requests is the CLIENT-facing, tenant-scoped poll surface ("is my export ready?"); task_runs is the INTERNAL, central execution ledger ("which units of the sweep failed and why"). Neither depends on the other; one job may feed both (the statements unit job COMPLETEs/FAILs its tenant's async request through the AsyncRequestPlatform seam while the task middleware tracks its unit) — the two records answer different audiences and prune on different clocks.
  • Upstream swap seams (where hand-rolled parts yield to Laravel if it grows them): (1) durable batch callbacks / batch events with guaranteed delivery would replace the middleware finalization re-check (the seam is Support\TaskRunLedger — the ONE class that touches the ledger models); (2) a first-class per-queue/per-job concurrency limiter (beyond RateLimited) would replace the named-limiter registration in the provider (the seam is TaskOptions → middleware()); (3) if job_batches ever gains domain-grade retention + per-job outcomes, task_run_units could shrink to a view over it (the seam is again the ledger — nothing else reads the tables); (4) progressive dispatch uses vendor Batch::add() chunks already — a future lazy-batch upstream API slots in at Support\LaravelTaskRunner::dispatchBatch().

Proven by conversion: payments:generate-monthly-statements now dispatches a payments.monthly_statements run (MonthlyStatementsTask definition; the eligibility gate — STANDING + FINAL-statement statuses on status_changed_at, unchanged and still pinned by unit test — moved verbatim into units(); the per-tenant generation moved verbatim into GenerateTenantMonthlyStatementJob, watermark idempotency + async-request complete/fail intact), one throttleable job per tenant instead of a serial loop. The command keeps --period and stays the schedule's entry point. ForwardDocumentToProviderJob is the tracked-single-job adopter (trait + one recordTaskUnitFailure line in its existing failed()).

Deferred (recorded in docs/tracking/task-runner.md — all rows since resolved; see the dated notes below): a backoffice HTTP read surface for runs; stuck-running unit detection (worker hard-kill between attempts) beyond tasks:retry; converting mirrors:reconcile-stale and the other fleet sweeps (the tenancy skill's "Sweeping all tenants" section now points at this framework as the HOW for new sweeps); concurrent same-key single-job dispatches sharing a unit. (Per-unit progress events were considered and DROPPED — per-tenant noise with no listener use case — not deferred.)

2026-07-05 — operator surface delivered (tracking rows 2 + 3): the backoffice READ surface (GET /api/v1/backoffice/task-runs[/{uuid}] — CQRS queries over new TaskRunLedger read methods, cursor pagination, capped failed/skipped/stale unit triage, uuid-only) shipped behind the module's own task-runner.runs.view web-guard permission (D36 per-module provider, granted to platform-support) — the ONE sanctioned amendment to this decision's "Foundation only" dependency rule: task-runner may now import the Authorization permission seam (the async-requests posture; ModuleBoundaryTest 2s² narrows accordingly; Tenancy and consumers stay forbidden). Stuck-running detection also shipped: the hourly tasks:detect-stuck sweep + the tasks.unit.stuck event + tasks:status/HTTP stale marking, thresholded by task-runner.stuck_after_minutes (per-type TaskOptions->stuckAfterMinutes), deduplicated per episode via task_run_units.stuck_notified_at — detection only, healing stays explicit (queue:retry/tasks:retry).

2026-07-05 — the last two deferrals delivered (tracking rows 4 + 5): single-job re-attach is now DISPATCH-SCOPED (the queue payload uuid stamped into the run's context.dispatch_uuid) — see the amended single-job bullet above; and the ledgered-error redaction gained the optional ScrubsLedgerText content-scrubbing seam (passthrough default inside task-runner; the dlp module rebinds its PAN/IBAN/SSN LedgerTextScrubber — dlp → task-runner is a sanctioned consumer-direction edge, dlp's ModuleBoundaryTest entry narrowed to the internals) — see the amended containment bullet above. docs/tracking/task-runner.md is now fully 🟢.

← Decision log index