Skip to main content

D74 — Observability app-code slices: log-Context enrichment (tenant/actor uuids), outbound trace propagation, and the HTTP client-idempotency table (distinct from the bus dedup)

Architecture decision record, relocated verbatim from the retired single-file docs/tenancy/ decision log. Status, thematic clusters, and how to record a new ADR: the decision log index.

The three app-code observability slices from docs/tracking/observability-and-request-identity.md (#1, #2, #6) land together (the OpenTelemetry-SDK / true span export, #3, stays deferred).

Log-Context enrichment (#6). Every log line (not only the CQRS audit channel) now carries tenant_id + actor_id — both the PUBLIC uuid, never the internal bigint (D25). Modules\Tenancy\Listeners\ShareTenantContext stamps the tenant uuid into Laravel Context on TenancyInitialized (forgets it on TenancyEnded), wired into TenancyServiceProvider::events() beside the stancl BootstrapTenancy/RevertToCentralContext listeners; Modules\Foundation\Listeners\ShareActorContext stamps the actor uuid on the framework Authenticated event (guard-agnostic: web User + tenant ClientUser, read off the model's uuid), registered in FoundationServiceProvider::boot(). Both keys were added to CorrelationProcessor::KEYS. No clash with the bus audit line, which records the bigint tenant_id/actor id in its own log context array (a distinct mechanism), not in Laravel Context.

Outbound trace propagation (#1). Modules\Rails\Logging\PropagateTraceContext — a named (propagate-trace-context) Saloon onRequest middleware attached (config-by-class, guarded by pipe-name, the RecordVendorRequest precedent) to every enabled finance connector in RailsServiceProvider. It forwards X-Correlation-Id = Context::get('correlation_id') and a W3C traceparent (00-<trace-id>-<span-id>-01) whose trace-id is adopted from the Context trace_id (or freshly minted 32-hex when absent) and whose span-id is ALWAYS a fresh 16-hex (bin2hex(random_bytes())) — the outbound leg is its own span. It never clobbers an already-present traceparent/Idempotency-Key/x-api-key (the SDK's auth + idempotency headers win).

HTTP client idempotency (#2) — a table DISTINCT from command_dedup_records. client_idempotency_keys (the new tenant-scoped, RLS-forced table; mandatory ClientIdempotencyKeyIsolationTest) backs Modules\Foundation\Http\Middleware\EnforceClientIdempotency (alias idempotency-key) via Modules\Foundation\Idempotency\ClientIdempotencyStore (a sanctioned out-of-bus DB-facade user by exact class in CqrsBoundaryTest, the CommandDeduplicator precedent; the Http middleware itself touches no DB). It is the client-facing, Stripe-style HTTP idempotency: opt-in per the Idempotency-Key header, keyed on (tenant_id, idempotency_key, route_name) with a sha256(method|path|body) fingerprint, replaying the stored HTTP response and releasing the claim on a non-2xx so a failed request stays retryable. This is deliberately DISTINCT from the internal bus command_dedup_records (D28/D56): that is INSIDE the command transaction, keyed on the command class, replaying the handler RESULT, and a rollback removes the claim. Mounted on the payments money-movement POSTs (orders create/cancel; payouts create/approve/cancel; registered-address create; whitelist-recipient create), after tenancy-init + auth:sanctum. A fingerprint mismatch is the new ErrorCode::IdempotencyKeyConflict (409); an in-flight duplicate reuses DuplicateRequest (409). RETENTION/PRUNING of client_idempotency_keys is a DEFERRED follow-up needing an owner-set window, mirroring foundation:prune-command-dedup-records (the partial unique index is WHERE deleted_at IS NULL so a pruned key can be re-used).

D74.1 — HTTP client-idempotency hardening: actor-scoped claim + self-healing in-flight TTL + header validation + replay/OpenAPI observability​

Follow-on to D74 (this branch). Four fixes harden the client_idempotency_keys slice:

  1. Actor-scoped claim (security). The claim gained an actor_id column — the authenticated actor's PUBLIC uuid (guard-agnostic: web User + tenant ClientUser both expose uuid) — and the partial unique index is now (tenant_id, actor_id, idempotency_key, route_name) WHERE deleted_at IS NULL. Previously keyed on (tenant_id, idempotency_key, route_name) only, a DIFFERENT actor in the SAME tenant presenting the SAME key would replay the FIRST actor's stored response WITHOUT their own bus authorize()/step-up — a cross-actor authorization-bypass. Now such an actor claims FRESH and runs their own request. The middleware reads $request->user()?->getAttribute('uuid') and threads it into claim()/complete()/release()/ classify()/row(). RLS is unchanged (still on tenant_id); the isolation test is extended for the actor dimension.
  2. Self-healing in-flight TTL (availability). A claim is inserted before the request runs and only completed on a 2xx (released on failure); a crash between claim and complete/release previously wedged the key at 409 (in-flight) forever. classify() now treats an uncompleted claim older than config('foundation.idempotency_inflight_ttl_seconds') as ABANDONED and reclaims it (Proceed) via an optimistic created_at-guarded in-place update that serialises concurrent retries; a still-recent uncompleted claim stays InFlight. This unwedges stale claims only — pruning of COMPLETED rows remains the D74 deferred follow-up.
  3. Header validation (robustness). The middleware validates the Idempotency-Key BEFORE the store — ≤255 chars (matching the column) of [A-Za-z0-9._~-] — rejecting a hostile/over-long key with a coded 400 IDEMPOTENCY_KEY_INVALID instead of a column-overflow 500.
  4. Observability. A REPLAY carries an Idempotency-Replayed: true response header. The optional Idempotency-Key header + the two idempotency 409s (DUPLICATE_REQUEST / IDEMPOTENCY_KEY_CONFLICT) + the 400 (IDEMPOTENCY_KEY_INVALID) are now in the OpenAPI contract via a #[SupportsIdempotencyKey] marker attribute + the IdempotencyContractExtension Scramble operation extension (all invisible to controller inference because they are middleware-thrown).

Round-2 hardening (this branch). Two correctness fixes on the self-healing reclaim:

  • Fencing token (money-movement correctness). The reclaim (finding 2) cannot distinguish a CRASHED original from a SLOW-BUT-ALIVE one — a provider call can hang past the TTL without accruing PHP max_execution_time. With complete()/release() previously matching only (tenant_id, actor_id, key, route), a slow original returning AFTER a retry reclaimed its row would hard-DELETE or overwrite the CURRENT owner's (the retry's) row — re-opening duplicate execution on money movement. FIX: a claim_token (uuid) column, minted afresh on the initial insert AND on every reclaim (the same in-place classifyInFlight() update that already fences the reclaim optimistically). claim() RETURNS the owning token (IdempotencyResult::proceed(string $claimToken)); the middleware threads it into complete()/release(), which add WHERE claim_token = :token. A superseded original holds the OLD token → affects ZERO rows. (Migration edited in place — the table is unshipped on this branch, mirroring how actor_id was added.)
  • TTL raised + invariant documented. idempotency_inflight_ttl_seconds default raised 90 → 900 and the INVARIANT documented in config/foundation.php + the store docblock: it MUST exceed the longest possible request lifetime (provider timeouts + retries), so a live request is essentially never reclaimed within its own lifetime; the fencing token is the correctness backstop if it ever is.

← Decision log index